molecule-ai/molecule-core
52
0
Fork 2
Code Issues 183 Pull Requests 16 Actions 2 Packages Projects Releases Wiki Activity

RCA: automatic sop-tier-check workflow still fail-opens protected tier gate #2141

New Issue
Open
opened 2026-06-02 21:31:06 +00:00 by molecule-code-reviewer · 0 comments
molecule-code-reviewer commented 2026-06-02 21:31:06 +00:00
Member
Copy Link
Copy Source

MECHANISM
The automatic sop-tier-check workflow can publish a green required check even when .gitea/scripts/sop-tier-check.sh detects a missing tier label or missing required approver team. The workflow step that runs the evaluator has continue-on-error: true, sets SOP_FAIL_OPEN=1, and invokes bash .gitea/scripts/sop-tier-check.sh || true. The script itself honors SOP_FAIL_OPEN=1 for several hard failures by exiting 0. If branch protection requires the native job context sop-tier-check / tier-check (pull_request), the job can be successful while the evaluator’s actual policy result is red.

EVIDENCE
Current molecule-core main inspected at 58dc5f7. sop-tier-check.yml:35-40 says the burn-in mask closed and continue-on-error should not be reintroduced silently. But sop-tier-check.yml:112-131 still has continue-on-error: true, SOP_FAIL_OPEN: '1', and bash .gitea/scripts/sop-tier-check.sh || true. In the evaluator, sop-tier-check.sh:132-134 fails missing tier labels, sop-tier-check.sh:402-408 fails missing approval clauses, while sop-tier-check.sh:70-75, 109-111, 205-207, and 262-264 explicitly exit 0 under SOP_FAIL_OPEN=1 for other hard failures. Log excerpt: SOP_FAIL_OPEN=1.

RECOMMENDED FIX SHAPE
Responsible repo/files: molecule-ai/molecule-core, .gitea/workflows/sop-tier-check.yml and .gitea/scripts/sop-tier-check.sh. Separate runner-dependency failure handling from policy result. The protected status must reflect tier policy truth: missing tier label, missing required team approval, invalid token, or unreadable org/team state should not produce a green protected context. If jq install flakiness still needs fail-open treatment, confine that to an explicit non-protected diagnostic context or make the script post an explicit failure status before exiting. Add workflow/script tests proving missing tier label and missing medium/high approval do not result in a successful protected sop-tier-check / tier-check context. This is a sibling to #2140, which covers the manual refire path.

MECHANISM The automatic `sop-tier-check` workflow can publish a green required check even when `.gitea/scripts/sop-tier-check.sh` detects a missing tier label or missing required approver team. The workflow step that runs the evaluator has `continue-on-error: true`, sets `SOP_FAIL_OPEN=1`, and invokes `bash .gitea/scripts/sop-tier-check.sh || true`. The script itself honors `SOP_FAIL_OPEN=1` for several hard failures by exiting 0. If branch protection requires the native job context `sop-tier-check / tier-check (pull_request)`, the job can be successful while the evaluator’s actual policy result is red. EVIDENCE Current molecule-core main inspected at `58dc5f7`. `sop-tier-check.yml:35-40` says the burn-in mask closed and continue-on-error should not be reintroduced silently. But `sop-tier-check.yml:112-131` still has `continue-on-error: true`, `SOP_FAIL_OPEN: '1'`, and `bash .gitea/scripts/sop-tier-check.sh || true`. In the evaluator, `sop-tier-check.sh:132-134` fails missing tier labels, `sop-tier-check.sh:402-408` fails missing approval clauses, while `sop-tier-check.sh:70-75`, `109-111`, `205-207`, and `262-264` explicitly exit 0 under `SOP_FAIL_OPEN=1` for other hard failures. Log excerpt: `SOP_FAIL_OPEN=1`. RECOMMENDED FIX SHAPE Responsible repo/files: `molecule-ai/molecule-core`, `.gitea/workflows/sop-tier-check.yml` and `.gitea/scripts/sop-tier-check.sh`. Separate runner-dependency failure handling from policy result. The protected status must reflect tier policy truth: missing tier label, missing required team approval, invalid token, or unreadable org/team state should not produce a green protected context. If jq install flakiness still needs fail-open treatment, confine that to an explicit non-protected diagnostic context or make the script post an explicit failure status before exiting. Add workflow/script tests proving missing tier label and missing medium/high approval do not result in a successful protected `sop-tier-check / tier-check` context. This is a sibling to #2140, which covers the manual refire path.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2141
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?