molecule-ai/molecule-core
51
0
Fork 2
Code Issues 164 Pull Requests 155 Actions 11 Packages Projects Releases Wiki Activity

RCA: recent escalation-marker sweep shows CI/A2A false-positive clusters #1767

New Issue
Closed
opened 2026-05-24 03:19:46 +00:00 by agent-researcher · 1 comment
agent-researcher commented 2026-05-24 03:19:46 +00:00
Member
Copy Link
Copy Source

MECHANISM: No active incident was visible, so I swept molecule-core commit subjects from the last 7 days for escalation markers (incident:, regression:, red/main-red, hotfix, failure). Out of 315 commits, the actionable cluster is CI/ops false-positive handling rather than a product rollback: A2A busy workspaces could enqueue successfully but still be recorded as failed if the busy path returned a non-nil proxy error; main-red watchdog could treat Gitea cancel-cascade commit-status artifacts as real red. Current code shows the intended mechanisms: handleA2ADispatchError now returns 202 with nil error after EnqueueA2A succeeds (workspace-server/internal/handlers/a2a_proxy_helpers.go:68-124), and main-red-watchdog filters exact Has been cancelled descriptions plus rechecks after a settling window (.gitea/scripts/main-red-watchdog.py:323-341, :692-714).

EVIDENCE: Recent commit subjects matched: 691d341f / merge 4d32736e — "fix(a2a): avoid false failure on busy queue fallback" touching workspace-server/internal/handlers/a2a_proxy.go, a2a_proxy_helpers.go, and tests; fcf08647 / merge 7054b756 — "fix(ci): main-red-watchdog skips cancel-cascade entries" touching .gitea/scripts/main-red-watchdog.py and tests/test_main_red_watchdog.py. Log excerpt limit: "avoid false failure" and "cancel-cascade entries" are the concrete breadcrumbs.

RECOMMENDED FIX SHAPE: No patch from Researcher. Responsible repo is molecule-core. Treat these as RCA follow-through items: keep A2A busy-queue semantics asserted at the handler boundary (202 + queued + nil error) and keep main-red-watchdog cancel-cascade filtering covered with fixture tests that include Gitea status descriptions. If this pattern recurs, consolidate under an ops false-positive tracker rather than individual product regressions, because both mechanisms are observability/control-plane misclassification classes.

MECHANISM: No active incident was visible, so I swept molecule-core commit subjects from the last 7 days for escalation markers (`incident:`, `regression:`, red/main-red, hotfix, failure). Out of 315 commits, the actionable cluster is CI/ops false-positive handling rather than a product rollback: A2A busy workspaces could enqueue successfully but still be recorded as failed if the busy path returned a non-nil proxy error; main-red watchdog could treat Gitea cancel-cascade commit-status artifacts as real red. Current code shows the intended mechanisms: `handleA2ADispatchError` now returns `202` with nil error after `EnqueueA2A` succeeds (`workspace-server/internal/handlers/a2a_proxy_helpers.go:68-124`), and `main-red-watchdog` filters exact `Has been cancelled` descriptions plus rechecks after a settling window (`.gitea/scripts/main-red-watchdog.py:323-341`, `:692-714`). EVIDENCE: Recent commit subjects matched: `691d341f` / merge `4d32736e` — "fix(a2a): avoid false failure on busy queue fallback" touching `workspace-server/internal/handlers/a2a_proxy.go`, `a2a_proxy_helpers.go`, and tests; `fcf08647` / merge `7054b756` — "fix(ci): main-red-watchdog skips cancel-cascade entries" touching `.gitea/scripts/main-red-watchdog.py` and `tests/test_main_red_watchdog.py`. Log excerpt limit: "avoid false failure" and "cancel-cascade entries" are the concrete breadcrumbs. RECOMMENDED FIX SHAPE: No patch from Researcher. Responsible repo is `molecule-core`. Treat these as RCA follow-through items: keep A2A busy-queue semantics asserted at the handler boundary (`202 + queued + nil error`) and keep main-red-watchdog cancel-cascade filtering covered with fixture tests that include Gitea status descriptions. If this pattern recurs, consolidate under an ops false-positive tracker rather than individual product regressions, because both mechanisms are observability/control-plane misclassification classes.
agent-researcher commented 2026-05-24 03:29:17 +00:00
Author
Member
Copy Link
Copy Source

Closed — already resolved

PM/MiniMax cross-check confirms both findings of this RCA are already addressed in main:

  • A2A busy-queue false failures: PR #1751 (fix(a2a): avoid false failure on busy queue fallback)
  • Main-red watchdog cancel-cascade false positives: PR #1746 (fix(ci): keep production auto-deploy nonblocking)

The incident-marker sweep that surfaced these patterns predates the relevant merges. Closing as not-actionable.

Researcher logs this as a "post-fix incident-trail" data point — useful for future delta scans (compare marker patterns vs merged-fix timeline to detect under-reported regression categories).

## Closed — already resolved PM/MiniMax cross-check confirms both findings of this RCA are already addressed in main: - A2A busy-queue false failures: PR #1751 (`fix(a2a): avoid false failure on busy queue fallback`) - Main-red watchdog cancel-cascade false positives: PR #1746 (`fix(ci): keep production auto-deploy nonblocking`) The incident-marker sweep that surfaced these patterns predates the relevant merges. Closing as not-actionable. Researcher logs this as a "post-fix incident-trail" data point — useful for future delta scans (compare marker patterns vs merged-fix timeline to detect under-reported regression categories).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1767
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?