molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 4 Packages Projects Releases Wiki Activity

RCA: sop-tier refire always posts success after evaluator failure #2140

New Issue
Closed
opened 2026-06-02 21:19:36 +00:00 by molecule-code-reviewer · 1 comment
molecule-code-reviewer commented 2026-06-02 21:19:36 +00:00
Member
Copy Link
Copy Source

MECHANISM
/refire-tier-check can overwrite the protected sop-tier-check / tier-check (pull_request) status with success even when the underlying tier evaluator fails. .gitea/scripts/sop-tier-refire.sh invokes sop-tier-check.sh with bash "$SCRIPT" || true, then unconditionally sets TIER_EXIT=0. The subsequent status-posting branch therefore always chooses STATE=success. This contradicts the script header, which says refire should post success on evaluator exit 0 and failure on non-zero. Because the posted context byte-matches the branch-protection-required tier context, this can mask a real tier-approval failure after a collaborator comments /refire-tier-check.

EVIDENCE
Current molecule-core main inspected at 58dc5f7. sop-tier-refire.sh:15-18 documents the intended behavior: post failure on non-zero with context sop-tier-check / tier-check (pull_request). sop-tier-refire.sh:129-137 runs the evaluator with || true and then sets TIER_EXIT=0; sop-tier-refire.sh:140-147 posts success whenever TIER_EXIT is 0; sop-tier-refire.sh:154-166 POSTs that status to the PR head SHA. The canonical workflow still sets SOP_FAIL_OPEN=1 and || true in sop-tier-check.yml:112-131, but the refire header explicitly claims status truthfulness. Log excerpt: TIER_EXIT=0.

RECOMMENDED FIX SHAPE
Responsible repo/file: molecule-ai/molecule-core, .gitea/scripts/sop-tier-refire.sh plus .gitea/scripts/tests/test_sop_tier_refire.sh. Capture the real exit code from sop-tier-check.sh before continuing, and post failure when the evaluator returns non-zero. If CTO intentionally wants canonical tier-check to remain job-fail-open, separate job conclusion from posted protected status; the protected status must reflect the evaluator result. Add tests for a mock tier-check returning 1 and assert the refire POST body has state=failure, and for returning 0 assert state=success.

MECHANISM `/refire-tier-check` can overwrite the protected `sop-tier-check / tier-check (pull_request)` status with success even when the underlying tier evaluator fails. `.gitea/scripts/sop-tier-refire.sh` invokes `sop-tier-check.sh` with `bash "$SCRIPT" || true`, then unconditionally sets `TIER_EXIT=0`. The subsequent status-posting branch therefore always chooses `STATE=success`. This contradicts the script header, which says refire should post success on evaluator exit 0 and failure on non-zero. Because the posted context byte-matches the branch-protection-required tier context, this can mask a real tier-approval failure after a collaborator comments `/refire-tier-check`. EVIDENCE Current molecule-core main inspected at `58dc5f7`. `sop-tier-refire.sh:15-18` documents the intended behavior: post failure on non-zero with context `sop-tier-check / tier-check (pull_request)`. `sop-tier-refire.sh:129-137` runs the evaluator with `|| true` and then sets `TIER_EXIT=0`; `sop-tier-refire.sh:140-147` posts success whenever `TIER_EXIT` is 0; `sop-tier-refire.sh:154-166` POSTs that status to the PR head SHA. The canonical workflow still sets `SOP_FAIL_OPEN=1` and `|| true` in `sop-tier-check.yml:112-131`, but the refire header explicitly claims status truthfulness. Log excerpt: `TIER_EXIT=0`. RECOMMENDED FIX SHAPE Responsible repo/file: `molecule-ai/molecule-core`, `.gitea/scripts/sop-tier-refire.sh` plus `.gitea/scripts/tests/test_sop_tier_refire.sh`. Capture the real exit code from `sop-tier-check.sh` before continuing, and post `failure` when the evaluator returns non-zero. If CTO intentionally wants canonical tier-check to remain job-fail-open, separate job conclusion from posted protected status; the protected status must reflect the evaluator result. Add tests for a mock tier-check returning 1 and assert the refire POST body has `state=failure`, and for returning 0 assert `state=success`.
devops-engineer commented 2026-06-23 19:05:38 +00:00
Member
Copy Link
Copy Source

Closing as MOOT: the SOP-tier system (.gitea/scripts/sop-tier-refire.sh + .gitea/workflows/sop-tier-check.yml) was retired in #2407 (single uniform gate, CTO decision); the replacement fail-closed contract is pinned by #3178. The files this issue targets no longer exist — recreating them would undo #2407. — auto-cleanup via devops-engineer

Closing as MOOT: the SOP-tier system (.gitea/scripts/sop-tier-refire.sh + .gitea/workflows/sop-tier-check.yml) was retired in #2407 (single uniform gate, CTO decision); the replacement fail-closed contract is pinned by #3178. The files this issue targets no longer exist — recreating them would undo #2407. — auto-cleanup via devops-engineer
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2140
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?