molecule-ai/molecule-core
49
0
Fork 2
Code Issues 145 Pull Requests 157 Actions 5 Packages Projects Releases Wiki Activity

docs/infra: document or auto-handle the compensating-status pattern for stale CI umbrellas #1780

New Issue
Open
opened 2026-05-24 05:25:51 +00:00 by hongming · 3 comments
hongming commented 2026-05-24 05:25:51 +00:00
Owner
Copy Link
Copy Source

Summary

Multiple recent merges used the "compensating-status" pattern: when CI / all-required reports failure due to a propagation/timing race despite the underlying CI sub-jobs being green, POST a corrected success status via the Gitea API to unblock the merge gate.

This is now load-bearing recovery. Used twice in CTO-bypass session 2026-05-24:

  • #1737: all 5 sub-jobs success, umbrella stale; compensating-status posted → merged d5941906
  • #1759: 4/5 sub-jobs success (5th was inherited-from-main failure — see #1778); compensating-status posted with honest description → merged 220a04b1

The pattern parallels what status-reaper.yml does for default-branch (push) status drift, but applied to PR umbrellas instead of main-branch contexts.

Proposed actions

  1. Document the pattern in internal/runbooks/dev-sop.md (or wherever recovery playbooks live) so future operators know the legitimate use-case and the API call to make.

  2. Auto-recovery option: extend status-reaper or write a sibling umbrella-reaper.yml that periodically scans for PRs where umbrella=failure AND all sub-jobs=success AND age > 5min, then posts compensating success. Same gate logic, automated.

  3. Resolved-by-prevention: if #1779 (ci-meta move) eliminates the dispatch deadlock, the propagation race might also vanish in practice (umbrella completes before sub-jobs are cancelled). Reconsider this issue after #1779 ships.

Acceptance

  • dev-sop has a "umbrella stale, sub-jobs green" recovery playbook OR
  • umbrella-reaper auto-handles the pattern OR
  • #1779 demonstrably eliminates the need

Discovered during

Same session as #1778 and #1779.

## Summary Multiple recent merges used the "compensating-status" pattern: when `CI / all-required` reports failure due to a propagation/timing race despite the underlying CI sub-jobs being green, POST a corrected success status via the Gitea API to unblock the merge gate. This is now load-bearing recovery. Used twice in CTO-bypass session 2026-05-24: - #1737: all 5 sub-jobs success, umbrella stale; compensating-status posted → merged `d5941906` - #1759: 4/5 sub-jobs success (5th was inherited-from-main failure — see #1778); compensating-status posted with honest description → merged `220a04b1` The pattern parallels what `status-reaper.yml` does for default-branch `(push)` status drift, but applied to PR umbrellas instead of main-branch contexts. ## Proposed actions 1. **Document the pattern** in `internal/runbooks/dev-sop.md` (or wherever recovery playbooks live) so future operators know the legitimate use-case and the API call to make. 2. **Auto-recovery option**: extend status-reaper or write a sibling `umbrella-reaper.yml` that periodically scans for PRs where umbrella=failure AND all sub-jobs=success AND age > 5min, then posts compensating success. Same gate logic, automated. 3. **Resolved-by-prevention**: if #1779 (ci-meta move) eliminates the dispatch deadlock, the propagation race might also vanish in practice (umbrella completes before sub-jobs are cancelled). Reconsider this issue after #1779 ships. ## Acceptance - [ ] dev-sop has a "umbrella stale, sub-jobs green" recovery playbook OR - [ ] umbrella-reaper auto-handles the pattern OR - [ ] #1779 demonstrably eliminates the need ## Discovered during Same session as #1778 and #1779.
hongming commented 2026-05-24 06:14:42 +00:00
Author
Owner
Copy Link
Copy Source

Docs sub-task landed in commit 2e027df8 (PR #1782): docs/runbooks/ci-umbrella-stale-compensating-status.md. Covers when-to-use, when-NOT, diagnose, recover, why-it-happens, and prevent.

Leaving this issue open for the auto-recovery sub-task (build umbrella-reaper.yml or extend status-reaper.yml to handle PR umbrellas). My recommendation is to defer that work until we have data on whether #1779's ci-meta fix actually eliminates the propagation race in normal load. If we go ~2 weeks without needing the compensating-status runbook, auto-recovery is unnecessary engineering. If we hit the pattern again within a week, build the reaper.

Tracking signal: every time the runbook gets exercised, post a brief comment here. We close this when one of:

  • 2 weeks pass with no exercise (close as obsolete, runner-pool fix sufficed)
  • Pattern recurs 3+ times → green-light the reaper build
Docs sub-task landed in commit `2e027df8` (PR #1782): `docs/runbooks/ci-umbrella-stale-compensating-status.md`. Covers when-to-use, when-NOT, diagnose, recover, why-it-happens, and prevent. Leaving this issue open for the **auto-recovery sub-task** (build `umbrella-reaper.yml` or extend `status-reaper.yml` to handle PR umbrellas). My recommendation is to defer that work until we have data on whether #1779's ci-meta fix actually eliminates the propagation race in normal load. If we go ~2 weeks without needing the compensating-status runbook, auto-recovery is unnecessary engineering. If we hit the pattern again within a week, build the reaper. Tracking signal: every time the runbook gets exercised, post a brief comment here. We close this when one of: - 2 weeks pass with no exercise (close as obsolete, runner-pool fix sufficed) - Pattern recurs 3+ times → green-light the reaper build
agent-researcher commented 2026-05-26 02:20:33 +00:00
Member
Copy Link
Copy Source

RCA — root cause\nThis issue is partly resolved on the documentation path, but not on the automation path. The repo now has a dedicated compensating-status runbook for stale PR umbrellas; status-reaper.py remains scoped to default-branch push/status-shadow repair and does not implement the requested “PR umbrella failed while all required sub-jobs are green” auto-reaper.\n\n## Evidence\n- docs/runbooks/ci-umbrella-stale-compensating-status.md:1 — the runbook exists and documents stale CI / all-required (pull_request) recovery.\n- docs/runbooks/ci-umbrella-stale-compensating-status.md:37 — recovery is manual: verify all required sub-jobs succeeded, then POST a compensating success status.\n- docs/runbooks/ci-umbrella-stale-compensating-status.md:63 — the runbook explicitly points back to #1780 for umbrella-reaper.yml automation if the pattern remains frequent.\n- .gitea/scripts/status-reaper.py:534 — current automation only compensates pull-request-shadow statuses when the matching push context succeeded on the same SHA, which is a narrower condition than PR sub-job umbrella recovery.\n\n## Suggested fix\nClose the documentation acceptance item if maintainers consider the runbook sufficient. If automation is still desired, implement a separate umbrella-reaper.yml that queries action job rows for the PR head, requires all five umbrella sub-jobs to be success, refuses compensation if any required job failed/missing, and posts a success status with an explicit compensating description. Do not fold this into the existing default-branch status reaper without a separate guardrail/test set; its safety proof is different.\n\n## Confidence\nHigh — the runbook and status-reaper code directly show current coverage and the remaining automation gap.

## RCA — root cause\nThis issue is partly resolved on the documentation path, but not on the automation path. The repo now has a dedicated compensating-status runbook for stale PR umbrellas; `status-reaper.py` remains scoped to default-branch push/status-shadow repair and does not implement the requested “PR umbrella failed while all required sub-jobs are green” auto-reaper.\n\n## Evidence\n- `docs/runbooks/ci-umbrella-stale-compensating-status.md:1` — the runbook exists and documents stale `CI / all-required (pull_request)` recovery.\n- `docs/runbooks/ci-umbrella-stale-compensating-status.md:37` — recovery is manual: verify all required sub-jobs succeeded, then POST a compensating success status.\n- `docs/runbooks/ci-umbrella-stale-compensating-status.md:63` — the runbook explicitly points back to #1780 for `umbrella-reaper.yml` automation if the pattern remains frequent.\n- `.gitea/scripts/status-reaper.py:534` — current automation only compensates pull-request-shadow statuses when the matching push context succeeded on the same SHA, which is a narrower condition than PR sub-job umbrella recovery.\n\n## Suggested fix\nClose the documentation acceptance item if maintainers consider the runbook sufficient. If automation is still desired, implement a separate `umbrella-reaper.yml` that queries action job rows for the PR head, requires all five umbrella sub-jobs to be `success`, refuses compensation if any required job failed/missing, and posts a success status with an explicit compensating description. Do not fold this into the existing default-branch status reaper without a separate guardrail/test set; its safety proof is different.\n\n## Confidence\nHigh — the runbook and status-reaper code directly show current coverage and the remaining automation gap.
agent-pm commented 2026-05-27 20:06:12 +00:00
Member
Copy Link
Copy Source

Auto-handling implementation in PR #1964.

Auto-handling implementation in PR #1964.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1780
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?