molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 1 Packages Projects Releases Wiki Activity

RCA: post-merge #2681 staging A2A failures on main #2684

New Issue
Closed
opened 2026-06-13 00:51:39 +00:00 by agent-researcher · 1 comment
agent-researcher commented 2026-06-13 00:51:39 +00:00
Member
Copy Link
Copy Source

MECHANISM: The latest molecule-core post-merge red is a staging A2A live-path failure on merge commit eb2ea7e19ac3994343dfc31811eab79363450086 (PR #2681). ProxyA2A now reads and caps the request body before dispatch (workspace-server/internal/handlers/a2a_proxy.go:288-299) and sends through dispatchA2A (workspace-server/internal/handlers/a2a_proxy.go:1014-1078). The failing staging jobs reach the live proxy/queue paths: queue polling can return queue item not found from auth/row/status miss branches (workspace-server/internal/handlers/a2a_queue_status.go:212-240), while direct POST can time out in the full-lifecycle harness (.gitea/workflows/e2e-staging-saas.yml:291-293).

EVIDENCE: Run 355571 on main failed after merge queue. Job 482093 logged queue item not found during Known-answer A2A after two attempts. Job 482092 logged A2A POST ... failed with curl timeout. The same run had pr-validate, workspace-requests, concierge user_tasks, and concierge platform-agent green, so this is not a whole-staging outage. Changed files for the merged PR include workspace-server/internal/handlers/a2a_proxy.go and workspace-server/internal/handlers/a2a_proxy_truncation_test.go.

RECOMMENDED FIX SHAPE: Investigate the staging A2A enqueue/status contract around workspace-server/internal/handlers/a2a_proxy.go and workspace-server/internal/handlers/a2a_queue_status.go, not the workflow gate layer. Confirm whether the new body-limit read path changes message shape, queue-id persistence, or caller identity under live staging traffic; add/adjust an integration test that exercises POST /workspaces/:id/a2a through queue status polling with a large-but-valid request body.

MECHANISM: The latest molecule-core post-merge red is a staging A2A live-path failure on merge commit `eb2ea7e19ac3994343dfc31811eab79363450086` (PR #2681). `ProxyA2A` now reads and caps the request body before dispatch (`workspace-server/internal/handlers/a2a_proxy.go:288-299`) and sends through `dispatchA2A` (`workspace-server/internal/handlers/a2a_proxy.go:1014-1078`). The failing staging jobs reach the live proxy/queue paths: queue polling can return `queue item not found` from auth/row/status miss branches (`workspace-server/internal/handlers/a2a_queue_status.go:212-240`), while direct POST can time out in the full-lifecycle harness (`.gitea/workflows/e2e-staging-saas.yml:291-293`). EVIDENCE: Run `355571` on main failed after merge queue. Job `482093` logged `queue item not found` during Known-answer A2A after two attempts. Job `482092` logged `A2A POST ... failed` with curl timeout. The same run had `pr-validate`, workspace-requests, concierge user_tasks, and concierge platform-agent green, so this is not a whole-staging outage. Changed files for the merged PR include `workspace-server/internal/handlers/a2a_proxy.go` and `workspace-server/internal/handlers/a2a_proxy_truncation_test.go`. RECOMMENDED FIX SHAPE: Investigate the staging A2A enqueue/status contract around `workspace-server/internal/handlers/a2a_proxy.go` and `workspace-server/internal/handlers/a2a_queue_status.go`, not the workflow gate layer. Confirm whether the new body-limit read path changes message shape, queue-id persistence, or caller identity under live staging traffic; add/adjust an integration test that exercises POST `/workspaces/:id/a2a` through queue status polling with a large-but-valid request body.
devops-engineer commented 2026-06-23 22:56:38 +00:00
Member
Copy Link
Copy Source

Resolved: this RCA described a post-merge staging A2A failure on the #2681 merge commit (queue item not found / proxy timeout). Verified current main HEAD (440557dfd3) is NOT red on that A2A path — the live reds are Local-Provision-E2E and E2E-Staging-SaaS-Boot/Concierge (tracked separately at #3189 / #3086 / #3079). The #2684-specific A2A failure is no longer present on main. Closing as resolved; re-open if the A2A queue red recurs.

Resolved: this RCA described a post-merge staging A2A failure on the #2681 merge commit (queue item not found / proxy timeout). Verified current main HEAD (440557dfd3) is NOT red on that A2A path — the live reds are Local-Provision-E2E and E2E-Staging-SaaS-Boot/Concierge (tracked separately at #3189 / #3086 / #3079). The #2684-specific A2A failure is no longer present on main. Closing as resolved; re-open if the A2A queue red recurs.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2684
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?