molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 1 Packages Projects Releases Wiki Activity

Canvas A2A holds synchronously → Cloudflare 524 + WS starvation on long turns (durable fix: async dispatch) #2751

New Issue
Open
opened 2026-06-13 12:26:07 +00:00 by core-devops · 1 comment
core-devops commented 2026-06-13 12:26:07 +00:00
Member
Copy Link
Copy Source

Root cause (DevTools on JRS, long migrate turn ~300s)

The canvas→agent /a2a POST is held open synchronously for the entire agent turn. On turns longer than Cloudflare's ~100s edge limit this produces:

Failed to load /workspaces/<id>/a2a → 524   (CF 'A Timeout Occurred')
WebSocket connection to wss://<tenant>/ws failed   (concurrent)
  • 524: CF gives up at ~100s while the proxy still holds the connection waiting for the agent. Raising server-side timeouts (#2727 idle, #2749 ResponseHeaderTimeout) does NOT help — CF caps at 100s first.
  • WS failures: coincide with the held turn. Ruled out CORS (CheckOrigin passes for the tenant origin — bad origin→403, correct origin→not-403; CORS_ORIGINS is set to the tenant URL in provisioner ec2.go). The WS drop appears to be connection-path saturation/starvation during the long synchronous hold, not an auth/routing bug.

Mitigations already shipped (stop the user-facing symptom)

  • #2745: clear the 'unreachable' banner whenever the agent is thinking.
  • #2750: treat a 524/522/504 as 'still processing' (not 'unreachable'); reply arrives via WS.
    These stop the false banner, but live reply delivery still depends on the WS, and the 524 still aborts the held request.

Durable fix (the real one — needs design review)

Async canvas dispatch: /a2a from the canvas should return promptly (well under 100s) with {status:"queued"} and deliver the agent's reply via the AGENT_MESSAGE WebSocket event — the SAME contract already used for poll-mode/external workspaces. Today push-mode workspaces (those with a URL, e.g. JRS SEO agent) hold the HTTP connection for the whole turn; switching the canvas path to always-async removes the 100s ceiling entirely and frees the connection path (which should also resolve the concurrent WS failures).

Secondary: confirm WS reliability under load + a polling fallback for chat-history so a dropped WS still surfaces the reply without a manual reload.

This is a core chat-flow change — flagging for CTO review rather than a hot rewrite. It supersedes the timeout-raise workarounds (#2727/#2749) for the canvas path.

## Root cause (DevTools on JRS, long migrate turn ~300s) The canvas→agent `/a2a` POST is **held open synchronously for the entire agent turn**. On turns longer than **Cloudflare's ~100s edge limit** this produces: ``` Failed to load /workspaces/<id>/a2a → 524 (CF 'A Timeout Occurred') WebSocket connection to wss://<tenant>/ws failed (concurrent) ``` - **524**: CF gives up at ~100s while the proxy still holds the connection waiting for the agent. Raising server-side timeouts (#2727 idle, #2749 ResponseHeaderTimeout) does NOT help — CF caps at 100s first. - **WS failures**: coincide with the held turn. Ruled out CORS (`CheckOrigin` passes for the tenant origin — bad origin→403, correct origin→not-403; `CORS_ORIGINS` is set to the tenant URL in provisioner `ec2.go`). The WS drop appears to be connection-path saturation/starvation during the long synchronous hold, not an auth/routing bug. ## Mitigations already shipped (stop the user-facing symptom) - #2745: clear the 'unreachable' banner whenever the agent is `thinking`. - #2750: treat a 524/522/504 as 'still processing' (not 'unreachable'); reply arrives via WS. These stop the false banner, but live reply delivery still depends on the WS, and the 524 still aborts the held request. ## Durable fix (the real one — needs design review) **Async canvas dispatch**: `/a2a` from the canvas should return promptly (well under 100s) with `{status:"queued"}` and deliver the agent's reply via the `AGENT_MESSAGE` WebSocket event — the SAME contract already used for poll-mode/external workspaces. Today push-mode workspaces (those with a URL, e.g. JRS SEO agent) hold the HTTP connection for the whole turn; switching the canvas path to always-async removes the 100s ceiling entirely and frees the connection path (which should also resolve the concurrent WS failures). Secondary: confirm WS reliability under load + a polling fallback for chat-history so a dropped WS still surfaces the reply without a manual reload. This is a core chat-flow change — flagging for CTO review rather than a hot rewrite. It supersedes the timeout-raise workarounds (#2727/#2749) for the canvas path.
core-devops commented 2026-06-13 19:16:50 +00:00
Author
Member
Copy Link
Copy Source

Design proposal — async canvas A2A dispatch (cap-and-queue)

Grounded findings (this codebase):

  • The canvas /a2a POST is held synchronously by proxyA2ARequest for the whole agent turn; a turn > Cloudflare's ~100s edge limit returns 524 (the recurring "Failed to send").
  • The agent's reply reaches the canvas via the events.Broadcaster (AGENT_MESSAGE over WS) independently of the held HTTP response — proven by applyIdleTimeout keying off broadcaster activity, and by the existing client-side contract (useChatSend abandons at 120s and the reply still arrives via WS). So dropping the held connection does NOT lose the reply.

Proposal: for canvas callers (callerID == ""), cap the synchronous wait at a CF-safe window (A2A_CANVAS_SYNC_BUDGET, default ~90s). If the agent hasn't returned headers by then, return {status:"queued", delivery_mode:"poll"} to the canvas (the exact shape the client already handles), while the dispatch to the agent continues on a detached context (NOT the request context) so the turn is never cancelled. The reply lands via the existing AGENT_MESSAGE broadcast. Turns < 90s are unchanged (inline reply as today).

The one real risk to validate (the crux): detaching the agent dispatch from the request context without (a) cancelling the in-flight turn, (b) double-delivering (inline AND broadcast), or (c) leaking goroutines. Mitigations: background the upstream call with a context derived from a long-lived parent (idle/ceiling-bounded, as today) rather than the gin request ctx; suppress the inline reply once we've returned queued; reuse the existing dedup (the client already dedups AGENT_MESSAGE by messageId).

Test + CI wiring: unit tests in a2a_proxy_test.go (queued-at-budget, no-cancel, no-double-deliver) in the blocking Go gate; an e2e-chat scenario for a >90s turn asserting queued-then-WS-reply. Removes the need for the #2727/#2749 timeout raises on the canvas path (they stay as the agent-to-agent backstop).

Decision needed: approve this approach (cap-and-queue + detached dispatch) before I implement — it touches the core chat path for every turn. Alternatives considered: (1) raise CF's edge timeout — not possible on the current plan; (2) chunked/streaming 200 early from the runtime — a runtime change, heavier, can layer on later.

cc @ for sign-off. Filed from the long-turn-timeout work (#2723/#2727/#2749/#2745/#2750).

## Design proposal — async canvas A2A dispatch (cap-and-queue) **Grounded findings (this codebase):** - The canvas `/a2a` POST is held synchronously by `proxyA2ARequest` for the whole agent turn; a turn > Cloudflare's ~100s edge limit returns **524** (the recurring "Failed to send"). - The agent's reply reaches the canvas via the **`events.Broadcaster`** (AGENT_MESSAGE over WS) **independently** of the held HTTP response — proven by `applyIdleTimeout` keying off broadcaster activity, and by the existing client-side contract (`useChatSend` abandons at 120s and the reply still arrives via WS). **So dropping the held connection does NOT lose the reply.** **Proposal:** for **canvas callers** (`callerID == ""`), cap the synchronous wait at a CF-safe window (`A2A_CANVAS_SYNC_BUDGET`, default ~90s). If the agent hasn't returned headers by then, return `{status:"queued", delivery_mode:"poll"}` to the canvas (the exact shape the client already handles), while the dispatch to the agent **continues on a detached context** (NOT the request context) so the turn is never cancelled. The reply lands via the existing AGENT_MESSAGE broadcast. Turns < 90s are unchanged (inline reply as today). **The one real risk to validate (the crux):** detaching the agent dispatch from the request context without (a) cancelling the in-flight turn, (b) double-delivering (inline AND broadcast), or (c) leaking goroutines. Mitigations: background the upstream call with a context derived from a long-lived parent (idle/ceiling-bounded, as today) rather than the gin request ctx; suppress the inline reply once we've returned queued; reuse the existing dedup (the client already dedups AGENT_MESSAGE by messageId). **Test + CI wiring:** unit tests in `a2a_proxy_test.go` (queued-at-budget, no-cancel, no-double-deliver) in the blocking Go gate; an `e2e-chat` scenario for a >90s turn asserting queued-then-WS-reply. Removes the need for the #2727/#2749 timeout raises on the canvas path (they stay as the agent-to-agent backstop). **Decision needed:** approve this approach (cap-and-queue + detached dispatch) before I implement — it touches the core chat path for every turn. Alternatives considered: (1) raise CF's edge timeout — not possible on the current plan; (2) chunked/streaming 200 early from the runtime — a runtime change, heavier, can layer on later. cc @ for sign-off. Filed from the long-turn-timeout work (#2723/#2727/#2749/#2745/#2750).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2751
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?