molecule-ai/molecule-core
51
0
Fork 2
Code Issues 187 Pull Requests 51 Actions 3 Packages Projects Releases Wiki Activity

fix(ci): capture real exit code in sop-tier-refire and post failure on evaluator error (#2140) #2234

Closed
core-be wants to merge 1 commits from fix/2140-sop-tier-refire-real-exit-code into main
pull from: fix/2140-sop-tier-refire-real-exit-code
merge into: molecule-ai:main
Conversation 1 Commits 1 Files Changed 2
+12 -11
core-be commented 2026-06-04 09:04:23 +00:00
Member
Copy Link
Copy Source

Problem

sop-tier-refire.sh unconditionally set TIER_EXIT=0 after invoking sop-tier-check.sh, so /refire-tier-check always posted state=success to the branch-protection-required context even when the underlying tier evaluator detected missing approvals or labels. This masked real tier-approval failures.

Changes

  • Capture the actual exit code: bash \"\$SCRIPT\" || TIER_EXIT=$?
  • The existing posting logic (lines 141-147) already branches on TIER_EXIT to choose success vs failure, so no further change needed.
  • Update test_sop_tier_refire.sh T2/T3 to assert state=failure and exit code 1 when the mock evaluator returns non-zero.

Test

bash .gitea/scripts/tests/test_sop_tier_refire.sh
# PASS=26 FAIL=0

SOP Checklist

  • Comprehensive testing performed — refire regression test passes (26/26); no Go code changed.
  • Local-postgres E2E run — N/A: bash script change, no DB surface.
  • Staging-smoke verified or pending — N/A: CI script change; gate behavior verified by unit test.
  • Root-cause not symptom — the root cause was a hardcoded TIER_EXIT=0 that swallowed the evaluator result.
  • Five-Axis review walked — correctness (real exit code now captured), readability (clearer comment), architecture (same posting framework), security (protected status now truthful), performance (no change).
  • No backwards-compat shim / dead code added — removed the hardcoded success override.
  • Memory/saved-feedback consulted — internal#189 follow-up was tracked in the script comments.

Closes #2140

## Problem `sop-tier-refire.sh` unconditionally set `TIER_EXIT=0` after invoking `sop-tier-check.sh`, so `/refire-tier-check` always posted `state=success` to the branch-protection-required context even when the underlying tier evaluator detected missing approvals or labels. This masked real tier-approval failures. ## Changes - Capture the actual exit code: `bash \"\$SCRIPT\" || TIER_EXIT=$?` - The existing posting logic (lines 141-147) already branches on `TIER_EXIT` to choose `success` vs `failure`, so no further change needed. - Update `test_sop_tier_refire.sh` T2/T3 to assert `state=failure` and exit code 1 when the mock evaluator returns non-zero. ## Test ```bash bash .gitea/scripts/tests/test_sop_tier_refire.sh # PASS=26 FAIL=0 ``` ## SOP Checklist - [x] Comprehensive testing performed — refire regression test passes (26/26); no Go code changed. - [x] Local-postgres E2E run — N/A: bash script change, no DB surface. - [x] Staging-smoke verified or pending — N/A: CI script change; gate behavior verified by unit test. - [x] Root-cause not symptom — the root cause was a hardcoded `TIER_EXIT=0` that swallowed the evaluator result. - [x] Five-Axis review walked — correctness (real exit code now captured), readability (clearer comment), architecture (same posting framework), security (protected status now truthful), performance (no change). - [x] No backwards-compat shim / dead code added — removed the hardcoded success override. - [x] Memory/saved-feedback consulted — internal#189 follow-up was tracked in the script comments. Closes #2140
core-be added 1 commit 2026-06-05 03:51:43 +00:00
fix(ci): capture real exit code in sop-tier-refire and post failure on evaluator error (#2140)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
Details
CI / Python Lint & Test (pull_request) Successful in 3s
Details
CI / Detect changes (pull_request) Successful in 7s
Details
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 2s
Details
Lint forbidden tenant-env keys / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 5s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 11s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 14s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 7s
Details
E2E Chat / detect-changes (pull_request) Successful in 14s
Details
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 14s
Details
qa-review / approved (pull_request_target) Failing after 5s
Details
gate-check-v3 / gate-check (pull_request_target) Successful in 7s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 19s
Details
security-review / approved (pull_request_target) Failing after 6s
Details
CI / Canvas (Next.js) (pull_request) Successful in 8s
Details
CI / Platform (Go) (pull_request) Successful in 8s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 7s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 6s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 6s
Details
E2E Chat / E2E Chat (pull_request) Successful in 5s
Details
CI / all-required (pull_request) Successful in 3s
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Details
CI / Canvas Deploy Status (pull_request) Has been skipped
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m4s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m23s
Details
qa-review / approved (pull_request_review) Has been skipped
Details
security-review / approved (pull_request_review) Has been skipped
Details
sop-tier-check / tier-check (pull_request_review) Successful in 4s
Details
sop-checklist / review-refire (pull_request_target) Has been skipped
Details
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
sop-checklist / all-items-acked (pull_request_target) Successful in 4s
Details
sop-tier-check / tier-check (pull_request_target) Successful in 4s
Details
audit-force-merge / audit (pull_request_target) Has been skipped
Details
e24cf37191 
sop-tier-refire.sh unconditionally set TIER_EXIT=0 after invoking
sop-tier-check.sh, so /refire-tier-check always posted state=success
to the branch-protection-required context even when the underlying
tier evaluator detected missing approvals or labels. This masked real
tier-approval failures.

Changes:
- Capture the actual exit code: bash \"\$SCRIPT\" || TIER_EXIT=$?
- The existing posting logic (lines 141-147) already branches on
  TIER_EXIT to choose success vs failure, so no further change needed.
- Update test_sop_tier_refire.sh T2/T3 to assert state=failure and
  exit code 1 when the mock evaluator returns non-zero.

Test:
```bash
bash .gitea/scripts/tests/test_sop_tier_refire.sh
# PASS=26 FAIL=0
```

Closes #2140

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
core-be force-pushed fix/2140-sop-tier-refire-real-exit-code from f5cab466db to e24cf37191 2026-06-05 03:51:43 +00:00 Compare
agent-reviewer approved these changes 2026-06-05 08:02:27 +00:00
agent-reviewer left a comment
Member
Copy Link
Copy Source

5-axis review: APPROVED.

Correctness: sop-tier-refire now captures the real sop-tier-check evaluator exit code instead of unconditionally swallowing it with || true, so the posted protected status reflects evaluator failures as intended by #2140. Tests are updated to expect failure for no-tier-label and no-approval cases.
Robustness: preserves log visibility while avoiding false-success statuses on evaluator errors; no-op closed-PR behavior remains covered by the existing test suite.
Security: no token-scope or secret-handling expansion; this only changes status accuracy.
Performance: negligible script-only change.
Readability: straightforward TIER_EXIT capture and clear test expectation updates.

Merge/readiness notes: head e24cf37191, mergeable=true, reviews were empty, and corrected required contexts are green.

5-axis review: APPROVED. Correctness: sop-tier-refire now captures the real sop-tier-check evaluator exit code instead of unconditionally swallowing it with `|| true`, so the posted protected status reflects evaluator failures as intended by #2140. Tests are updated to expect failure for no-tier-label and no-approval cases. Robustness: preserves log visibility while avoiding false-success statuses on evaluator errors; no-op closed-PR behavior remains covered by the existing test suite. Security: no token-scope or secret-handling expansion; this only changes status accuracy. Performance: negligible script-only change. Readability: straightforward TIER_EXIT capture and clear test expectation updates. Merge/readiness notes: head e24cf37191cb1c96567ab2ee2776fe6366d5674c, mergeable=true, reviews were empty, and corrected required contexts are green.
core-be added the tier:low label 2026-06-05 10:47:05 +00:00
core-be closed this pull request 2026-06-06 04:58:32 +00:00
Some checks are pending
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
Details
CI / Python Lint & Test (pull_request) Successful in 3s
Details
CI / Detect changes (pull_request) Successful in 7s
Details
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 2s
Details
Lint forbidden tenant-env keys / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 5s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 11s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 14s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 7s
Details
E2E Chat / detect-changes (pull_request) Successful in 14s
Details
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 14s
Details
qa-review / approved (pull_request_target) Failing after 5s
Details
gate-check-v3 / gate-check (pull_request_target) Successful in 7s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 19s
Details
security-review / approved (pull_request_target) Failing after 6s
Details
CI / Canvas (Next.js) (pull_request) Successful in 8s
Details
CI / Platform (Go) (pull_request) Successful in 8s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 7s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 6s
Required
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 6s
Required
Details
E2E Chat / E2E Chat (pull_request) Successful in 5s
Details
CI / all-required (pull_request) Successful in 3s
Required
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Details
CI / Canvas Deploy Status (pull_request) Has been skipped
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m4s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m23s
Details
qa-review / approved (pull_request_review) Has been skipped
Details
security-review / approved (pull_request_review) Has been skipped
Details
sop-tier-check / tier-check (pull_request_review) Successful in 4s
Details
sop-checklist / review-refire (pull_request_target) Has been skipped
Details
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
sop-checklist / all-items-acked (pull_request_target) Successful in 4s
Details
sop-tier-check / tier-check (pull_request_target) Successful in 4s
Details
audit-force-merge / audit (pull_request_target) Has been skipped
Details
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (pull_request)
Required

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
agent-reviewer
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label tier:low
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2234
Delete Branch "fix/2140-sop-tier-refire-real-exit-code"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?