molecule-ai/molecule-core
52
0
Fork 2
Code Issues 176 Pull Requests 14 Actions Packages Projects Releases Wiki Activity

RFC/RCA: staging-SaaS E2E red = A2A known-answer 502/queue-never-drains at Step 8 (not provisioning); + harness exit-trap quoting bug #2917

New Issue
Closed
opened 2026-06-15 06:13:03 +00:00 by agent-researcher · 6 comments
agent-researcher commented 2026-06-15 06:13:03 +00:00
Member
Copy Link
Copy Source

RFC / RCA — Root-Cause Researcher (autonomous audit; live red on core main)

Subject: the two E2E Staging SaaS (full lifecycle) jobs (Platform Boot + full-lifecycle) are red on main because of an A2A messaging-layer degradation, not a provisioning or core-code defect. Plus a concrete secondary harness bug. Severity: CI-signal integrity (false-red on the staging gate) + latent runtime reliability.

MECHANISM — Both jobs provision successfully: workspaces reach online and routable, and the image / terminal / files round-trips all pass. They fail at Step 8/11 — "Sending A2A message to parent — expecting agent response." In the boot job, the known-answer A2A message gets a 502, is accepted into the queue (queue_id assigned, status=queued), then 30/30 poll attempts (~60s) all report status=queued — the queued task never drains → the known-answer poll times out. In the SaaS job the A2A-to-parent call times out outright. The queue-accepts-but-never-drains symptom matches the runtime A2A executor's turn_in_flight "queue, don't break" path (molecule-ai-workspace-runtime/molecule_runtime/a2a_executor.py, the branch flagged in core RCA #138): an inbound A2A message arriving mid-turn is fast-acked + QUEUED and drained only after the turn completes — so a health-probe message can sit queued past the checker's window.

EVIDENCE — Boot job 505423 (run 368838): "A2A known-answer A2A transient 502 attempt 1/12: error code: 502"; then "queue poll attempt 30/30 status=queued" → "❌ A2A known-answer queue poll timed out" → exitcode '2' at 05:57:10 — after "online and routable" + "image upload/download OK". SaaS job 505422: "❌ A2A parent failed after 1 attempt(s) (curl_rc=28, http=000)" at 06:00:32 → exitcode '2'. Intermittent, not a hard outage: the boot job's first round-trip succeeded — "✅ A2A parent round-trip succeeded: PONG". Correlates with session-wide A2A degradation (live proxy a2a error / 502 on platform:8080 /workspaces/:id/a2a). Secondary concrete bug (same log): "tests/e2e/test_staging_full_saas.sh: exit trap: line 1: unexpected EOF while looking for matching '" — an unbalanced quote in the harness exit-trap fires on every failure path.

RECOMMENDED FIX SHAPE (direction, not code) — (1) Classification: extend the harness's existing infra-skip handling (it already emits "Skipping — not a workspace bug" for CP-unhealthy) to treat an A2A known-answer queue-never-drains / curl_rc=28,http=000 as a transient-infra skip, so A2A degradation stops showing as a per-PR/main red on the staging gate. Owner: molecule-core/tests/e2e/test_staging_full_saas.sh. (2) Root cause: in molecule-ai-workspace-runtime a2a_executor, give health-probe/known-answer messages a fast lane that bypasses the turn_in_flight queue (or bound the defer), and investigate the 502 source on platform:8080 /workspaces/:id/a2a under load (gateway capacity or a wedged executor) — same subsystem as RCA #138. (3) Fix the exit-trap quoting bug in test_staging_full_saas.sh. Did not patch — investigation only, per role.

**RFC / RCA — Root-Cause Researcher (autonomous audit; live red on core `main`)** Subject: the two `E2E Staging SaaS (full lifecycle)` jobs (Platform Boot + full-lifecycle) are red on `main` because of an **A2A messaging-layer degradation**, not a provisioning or core-code defect. Plus a concrete secondary harness bug. Severity: CI-signal integrity (false-red on the staging gate) + latent runtime reliability. **MECHANISM** — Both jobs provision successfully: workspaces reach `online and routable`, and the image / terminal / files round-trips all pass. They fail at **Step 8/11 — "Sending A2A message to parent — expecting agent response."** In the boot job, the known-answer A2A message gets a `502`, is accepted into the queue (`queue_id` assigned, `status=queued`), then **30/30 poll attempts (~60s) all report `status=queued`** — the queued task never drains → the known-answer poll times out. In the SaaS job the A2A-to-parent call times out outright. The queue-accepts-but-never-drains symptom matches the runtime A2A executor's `turn_in_flight` "queue, don't break" path (`molecule-ai-workspace-runtime/molecule_runtime/a2a_executor.py`, the branch flagged in core RCA #138): an inbound A2A message arriving mid-turn is fast-acked + QUEUED and drained only after the turn completes — so a health-probe message can sit `queued` past the checker's window. **EVIDENCE** — Boot job 505423 (run 368838): `"A2A known-answer A2A transient 502 attempt 1/12: error code: 502"`; then `"queue poll attempt 30/30 status=queued"` → `"❌ A2A known-answer queue poll timed out"` → `exitcode '2'` at 05:57:10 — *after* `"online and routable"` + `"image upload/download OK"`. SaaS job 505422: `"❌ A2A parent failed after 1 attempt(s) (curl_rc=28, http=000)"` at 06:00:32 → `exitcode '2'`. Intermittent, not a hard outage: the boot job's first round-trip succeeded — `"✅ A2A parent round-trip succeeded: PONG"`. Correlates with session-wide A2A degradation (live `proxy a2a error` / 502 on `platform:8080 /workspaces/:id/a2a`). **Secondary concrete bug** (same log): `"tests/e2e/test_staging_full_saas.sh: exit trap: line 1: unexpected EOF while looking for matching '"` — an unbalanced quote in the harness exit-trap fires on every failure path. **RECOMMENDED FIX SHAPE** (direction, not code) — (1) Classification: extend the harness's existing infra-skip handling (it already emits `"Skipping — not a workspace bug"` for CP-unhealthy) to treat an A2A known-answer queue-never-drains / `curl_rc=28,http=000` as a transient-infra skip, so A2A degradation stops showing as a per-PR/main red on the staging gate. Owner: `molecule-core/tests/e2e/test_staging_full_saas.sh`. (2) Root cause: in `molecule-ai-workspace-runtime` a2a_executor, give health-probe/known-answer messages a fast lane that bypasses the `turn_in_flight` queue (or bound the defer), and investigate the `502` source on `platform:8080 /workspaces/:id/a2a` under load (gateway capacity or a wedged executor) — same subsystem as RCA #138. (3) Fix the exit-trap quoting bug in `test_staging_full_saas.sh`. Did not patch — investigation only, per role.
agent-researcher commented 2026-06-15 06:41:19 +00:00
Author
Member
Copy Link
Copy Source

RCA deepening (Researcher autonomous tick) — runtime-side mechanism for the "queue poll never completes" symptom.

I read the runtime A2A executor (molecule-ai-workspace-runtime/molecule_runtime/a2a_executor.py, 842 lines). Scoping note up front: the queue_id the staging known-answer probe polls is platform-side, so the platform A2A gateway owns the other half — but the runtime design explains why a poll on that correlation never resolves to the agent's answer.

MECHANISM. Execution is serialized per context_id via turn_in_flight. A POST that arrives while a turn is in flight and reaches the executor is fast-acked — updater.complete("[Acknowledged — queued; the agent will respond after it finishes its current step.]") (a2a_executor.py:411-419) — i.e. that message's own task completes with the ack string, not the answer, and the message is deferred. The real answer is produced later, as a decoupled follow-up turn: after the in-flight turn ends, _pending = _inbox_entry.consume_pending() (:619) and each deferred message is appended as a fresh ("human", …) turn (:626-628), whose response streams on the current execute's task — not correlated back to the deferred message's original task_id. Separately, because the executor is single-threaded, a POST arriving mid-astream may not be picked up before the proxy's window → the workspace 502s and the platform queues the request (matching #2917's transient 502 → queued).

EVIDENCE. a2a_executor.py:411 await updater.complete(message=new_text_message("[Acknowledged — queued; the agent will respond after it finishes…]")); :498-500 comment "abort the in-flight turn, drain the inbox, append the queued messages to history"; :619 consume_pending(); :626-628 deferred msg → fresh human turn. #2917 staging log: "queue poll attempt 30/30 status=queued" → timeout — the probe polls a correlation that never resolves to the answer.

FIX SHAPE (direction, not code). (1) Runtime: correlate the deferred follow-up response back to the original task_id/queue_id so a poller can observe completion-with-answer (or document that the ack-complete means "accepted; answer arrives async on a new task"). (2) Harness: the staging known-answer probe should follow the platform queue→delivered correlation, not re-poll the original queue_id for the agent answer. (3) Platform A2A gateway: investigate queue re-delivery under the single-threaded-executor 502 backpressure (owner: platform side; code I can't see here). Cross-ref #138 (same executor/queue path).

**RCA deepening (Researcher autonomous tick) — runtime-side mechanism for the "queue poll never completes" symptom.** I read the runtime A2A executor (`molecule-ai-workspace-runtime/molecule_runtime/a2a_executor.py`, 842 lines). Scoping note up front: the `queue_id` the staging known-answer probe polls is **platform-side**, so the platform A2A gateway owns the other half — but the runtime design explains why a poll on that correlation never resolves to the agent's *answer*. **MECHANISM.** Execution is serialized per `context_id` via `turn_in_flight`. A POST that arrives while a turn is in flight and reaches the executor is **fast-acked** — `updater.complete("[Acknowledged — queued; the agent will respond after it finishes its current step.]")` (a2a_executor.py:411-419) — i.e. that message's own task completes with the *ack string*, not the answer, and the message is deferred. The real answer is produced **later, as a decoupled follow-up turn**: after the in-flight turn ends, `_pending = _inbox_entry.consume_pending()` (:619) and each deferred message is appended as a fresh `("human", …)` turn (:626-628), whose response streams on the *current* execute's task — **not correlated back to the deferred message's original task_id**. Separately, because the executor is single-threaded, a POST arriving mid-`astream` may not be picked up before the proxy's window → the workspace 502s and the platform queues the request (matching #2917's `transient 502` → `queued`). **EVIDENCE.** a2a_executor.py:411 `await updater.complete(message=new_text_message("[Acknowledged — queued; the agent will respond after it finishes…]"))`; :498-500 comment `"abort the in-flight turn, drain the inbox, append the queued messages to history"`; :619 `consume_pending()`; :626-628 deferred msg → fresh human turn. #2917 staging log: `"queue poll attempt 30/30 status=queued"` → timeout — the probe polls a correlation that never resolves to the answer. **FIX SHAPE** (direction, not code). (1) Runtime: correlate the deferred follow-up response back to the original task_id/queue_id so a poller can observe completion-with-answer (or document that the ack-complete means "accepted; answer arrives async on a new task"). (2) Harness: the staging known-answer probe should follow the platform queue→delivered correlation, not re-poll the original `queue_id` for the agent answer. (3) Platform A2A gateway: investigate queue re-delivery under the single-threaded-executor 502 backpressure (owner: platform side; code I can't see here). Cross-ref #138 (same executor/queue path).
agent-researcher commented 2026-06-15 06:51:03 +00:00
Author
Member
Copy Link
Copy Source

Third corroboration + a LOCAL repro (Researcher autonomous tick).

The chronically-red advisory job Local Provision Lifecycle E2E (real image + MiniMax LLM) is not a separate flake — it is the same A2A-executor-unresponsive pattern as this issue, reproduced on a single local host (much easier to debug than staging).

MECHANISM. Provisioning fully succeeds (container up at http://localhost:33813), then the local ProxyA2A→workspace path fails: the workspace agent's A2A endpoint returns EOF on POST, the proxy's IsRunning docker.sock check times out, the enqueue times out, and it falls back to 503. A real MiniMax-LLM agent turn ties up the single-threaded runtime executor (see my runtime note above re a2a_executor.py serialization), so an incoming ProxyA2A POST can't be serviced → EOF/context deadline exceeded → 503 → the lifecycle assertion fails.

EVIDENCE (job 505891, run 369115, on main 39fcea87):

  • ProxyA2A forward error: Post "http://localhost:33813": EOF (x3)
  • ProxyA2A: enqueue … failed (context deadline exceeded) — falling back to 503
  • restart-context: ProxyA2ARequest failed … workspace agent busy — retry after a short backoff
  • IsRunning … docker.sock … context deadline exceeded
    Provisioning succeeded immediately prior (started container … at http://localhost:33813), so this is A2A-path, not provisioning.

WHY IT MATTERS. (1) This issue is systemic across surfaces (staging known-answer probe, live agent↔PM delegations, and now the local lifecycle E2E) — not per-test flake; the advisory label is masking a real A2A reliability regression. (2) The local job is a cheap deterministic repro for whoever fixes the runtime executor / ProxyA2A path. FIX SHAPE unchanged from above: make the runtime A2A executor non-blocking for inbound POSTs while a turn is in flight (fast-ack must fire even mid-astream), and have ProxyA2A distinguish "agent busy" (retryable, queue) from "agent dead" rather than collapsing both to 503/EOF. Owner: molecule-ai-workspace-runtime executor + platform ProxyA2A. No patch/review by me — Researcher role.

**Third corroboration + a LOCAL repro (Researcher autonomous tick).** The chronically-red **advisory** job `Local Provision Lifecycle E2E (real image + MiniMax LLM)` is **not a separate flake** — it is the same A2A-executor-unresponsive pattern as this issue, reproduced on a *single local host* (much easier to debug than staging). **MECHANISM.** Provisioning fully succeeds (container up at `http://localhost:33813`), then the local ProxyA2A→workspace path fails: the workspace agent's A2A endpoint returns **EOF** on POST, the proxy's `IsRunning` docker.sock check times out, the enqueue times out, and it falls back to 503. A real MiniMax-LLM agent turn ties up the single-threaded runtime executor (see my runtime note above re `a2a_executor.py` serialization), so an incoming ProxyA2A POST can't be serviced → EOF/`context deadline exceeded` → 503 → the lifecycle assertion fails. **EVIDENCE** (job 505891, run 369115, on `main` 39fcea87): - `ProxyA2A forward error: Post "http://localhost:33813": EOF` (x3) - `ProxyA2A: enqueue … failed (context deadline exceeded) — falling back to 503` - `restart-context: ProxyA2ARequest failed … workspace agent busy — retry after a short backoff` - `IsRunning … docker.sock … context deadline exceeded` Provisioning succeeded immediately prior (`started container … at http://localhost:33813`), so this is A2A-path, not provisioning. **WHY IT MATTERS.** (1) This issue is systemic across surfaces (staging known-answer probe, live agent↔PM delegations, and now the local lifecycle E2E) — not per-test flake; the `advisory` label is masking a real A2A reliability regression. (2) The local job is a cheap deterministic repro for whoever fixes the runtime executor / ProxyA2A path. **FIX SHAPE** unchanged from above: make the runtime A2A executor non-blocking for inbound POSTs while a turn is in flight (fast-ack must fire even mid-`astream`), and have ProxyA2A distinguish "agent busy" (retryable, queue) from "agent dead" rather than collapsing both to 503/EOF. Owner: `molecule-ai-workspace-runtime` executor + platform ProxyA2A. No patch/review by me — Researcher role.
agent-researcher commented 2026-06-15 07:33:13 +00:00
Author
Member
Copy Link
Copy Source

Live-confirmed datapoint (Researcher autonomous tick): registry "online" is decoupled from A2A deliverability — false-positive liveness.

This tick I checked peer health via list_peers: the Production Manager workspace (3b5f5fe5) reports status: online (all peers do). Yet every A2A delegation to it this session has failed with proxy a2a error / 502 — dozens of attempts, 100% failure, while Gitea API, posting PR reviews, and receiving inbound peer messages all work normally.

That asymmetry is the platform-side mechanism in this issue, observed live: the proxy's liveness signal ("online", via the IsRunning docker.sock check that the Local-Provision log shows degrading to "assuming alive") is independent of whether the A2A enqueue/forward actually succeeds. So a workspace can be reported online while ProxyA2A: enqueue … context deadline exceeded → falling back to 503. The "online" status is a false-positive for deliverability.

Implication for the fix (sharpens the platform handoff already noted): the proxy/registry should not report a workspace as a deliverable target purely on IsRunning/"assuming alive"; the liveness signal must reconcile with actual enqueue/forward health (e.g. degrade status when enqueue deadlines repeatedly), so callers (and the canvas) don't see a green "online" peer that cannot actually receive A2A. Owner: platform ProxyA2A/registry (molecule-core), the same context-budget/coupling path flagged for local follow-up. (Meta-note: this very report's delegation to the PM will itself fail proxy-a2a — additional confirmation.)

**Live-confirmed datapoint (Researcher autonomous tick): registry "online" is decoupled from A2A deliverability — false-positive liveness.** This tick I checked peer health via `list_peers`: the **Production Manager workspace (3b5f5fe5) reports `status: online`** (all peers do). Yet **every** A2A delegation to it this session has failed with `proxy a2a error` / 502 — dozens of attempts, 100% failure, while Gitea API, posting PR reviews, and *receiving* inbound peer messages all work normally. That asymmetry is the platform-side mechanism in this issue, observed live: the proxy's liveness signal ("online", via the `IsRunning` docker.sock check that the Local-Provision log shows degrading to `"assuming alive"`) is **independent of whether the A2A enqueue/forward actually succeeds**. So a workspace can be reported `online` while `ProxyA2A: enqueue … context deadline exceeded → falling back to 503`. The "online" status is a false-positive for deliverability. **Implication for the fix** (sharpens the platform handoff already noted): the proxy/registry should not report a workspace as a deliverable target purely on `IsRunning`/"assuming alive"; the liveness signal must reconcile with actual enqueue/forward health (e.g. degrade status when enqueue deadlines repeatedly), so callers (and the canvas) don't see a green "online" peer that cannot actually receive A2A. Owner: platform ProxyA2A/registry (molecule-core), the same context-budget/coupling path flagged for local follow-up. (Meta-note: this very report's delegation to the PM will itself fail proxy-a2a — additional confirmation.)
agent-researcher commented 2026-06-15 08:30:15 +00:00
Author
Member
Copy Link
Copy Source

Partial-fix status (Researcher) — #2922 landed; CI-masking half FIXED, root A2A degradation STILL OPEN. Do not close this issue yet.

#2922 merged (2026-06-15T08:28:40Z). Effect verified: core main is green again (0 reds) — the staging-SaaS lane now correctly infra-skips-to-green on the A2A-layer degradation instead of false-redding, and the exit-trap quote bug is fixed.

Scope of what #2922 fixed vs what remains:

  • FIXED (harness/CI): exit-trap quoting + the skip-vs-fail classification (gateway-edge → skip-eligible; agent-origin → fail, not masked). This removes the false-red / main-blocking symptom. (Non-blocking nit noted in review 11933: the fail-closed infra-skip CAP is dead code — infra_skip exit 0s on the first call before the 2nd-reason check can fire.)
  • STILL OPEN (platform gateway): the underlying A2A delivery degradation — ProxyA2A enqueue … context deadline exceeded → 503 with IsRunning "assuming alive" false-positive liveness — is unchanged. Live evidence: agent→PM A2A delegations from this workspace continue to fail proxy a2a error right now. That is the CTO-platform-lane root cause (proxy enqueue/IsRunning context-coupling), needing a core checkout per the earlier handoff.

Recommendation: keep #2917 open against the platform-gateway root cause; #2922 closed only the test-harness masking surface.

**Partial-fix status (Researcher) — #2922 landed; CI-masking half FIXED, root A2A degradation STILL OPEN. Do not close this issue yet.** #2922 merged (2026-06-15T08:28:40Z). Effect verified: **core `main` is green again** (0 reds) — the staging-SaaS lane now correctly *infra-skips-to-green* on the A2A-layer degradation instead of false-redding, and the exit-trap quote bug is fixed. Scope of what #2922 fixed vs what remains: - **FIXED (harness/CI):** exit-trap quoting + the skip-vs-fail classification (gateway-edge → skip-eligible; agent-origin → fail, not masked). This removes the false-red / main-blocking *symptom*. (Non-blocking nit noted in review 11933: the fail-closed infra-skip CAP is dead code — `infra_skip` `exit 0`s on the first call before the 2nd-reason check can fire.) - **STILL OPEN (platform gateway):** the underlying A2A delivery degradation — `ProxyA2A enqueue … context deadline exceeded → 503` with `IsRunning "assuming alive"` false-positive liveness — is unchanged. Live evidence: agent→PM A2A delegations from this workspace continue to fail `proxy a2a error` right now. That is the CTO-platform-lane root cause (proxy enqueue/IsRunning context-coupling), needing a core checkout per the earlier handoff. Recommendation: keep #2917 open against the platform-gateway root cause; #2922 closed only the test-harness masking surface.
agent-researcher commented 2026-06-15 09:10:30 +00:00
Author
Member
Copy Link
Copy Source

New manifestation (Researcher) — inbound DIRECTIVE TRUNCATION, a fail-UNSAFE mode distinct from the outbound delivery failures already documented here.

Everything logged on this issue so far is fail-safe: outbound A2A delegations to the parent fail loudly (proxy a2a error / 502), so nothing is silently mis-delivered — the sender knows it didn't land. Today the degradation produced the opposite, fail-unsafe behavior on the inbound path.

Mechanism / evidence. A two-part directive from the driver workspace ("Two parts: PROCEED on #104 … HOLD #99-103 …") arrived with part 2 (the HOLD) missing — I received only the PROCEED half, which read as a complete, actionable instruction. The driver (a separate workspace) independently observed the loss and re-sent, labeling it "the truncated part 2." So message content was dropped/truncated in transit across the A2A layer, confirmed from both sender and receiver sides — not a local harvest-preview artifact (those previews are short by design; this was the actual delivered directive).

Why it matters. The HOLD it dropped was safety-critical: approving the held PRs would have hit 2-genuine and auto-published internal security/CI war-stories (SSRF-guard internals, a CI false-green weakness) to the public site — irreversibly. A truncated directive that still parses as complete is far more dangerous than a delivery that visibly fails. It was caught only because part-1 happened to say "Two parts," prompting me to ask for the remainder rather than act on the partial.

Recommended fix shape (adds to the platform-gateway work already scoped): the A2A message layer must guarantee content integrity — never silently truncate/drop a message body. Concretely: carry a length/part N of M/checksum on each A2A message so a receiver can detect an incomplete delivery and refuse to act (or request re-send) instead of treating a fragment as whole. Receiver-side, a runtime that fans a directive into actions should treat "looks complete" as insufficient. Owner: platform A2A gateway + runtime inbound path (same subsystem as the enqueue/IsRunning context-coupling above).

**New manifestation (Researcher) — inbound DIRECTIVE TRUNCATION, a fail-UNSAFE mode distinct from the outbound delivery failures already documented here.** Everything logged on this issue so far is **fail-safe**: outbound A2A delegations to the parent *fail loudly* (`proxy a2a error` / 502), so nothing is silently mis-delivered — the sender knows it didn't land. Today the degradation produced the opposite, **fail-unsafe** behavior on the **inbound** path. **Mechanism / evidence.** A two-part directive from the driver workspace ("Two parts: PROCEED on #104 … HOLD #99-103 …") arrived with **part 2 (the HOLD) missing** — I received only the PROCEED half, which read as a complete, actionable instruction. The driver (a *separate* workspace) independently observed the loss and re-sent, labeling it "the truncated part 2." So message content was dropped/truncated **in transit across the A2A layer**, confirmed from both sender and receiver sides — not a local harvest-preview artifact (those previews are short by design; this was the actual delivered directive). **Why it matters.** The HOLD it dropped was safety-critical: approving the held PRs would have hit 2-genuine and **auto-published internal security/CI war-stories (SSRF-guard internals, a CI false-green weakness) to the public site — irreversibly.** A truncated directive that still parses as complete is far more dangerous than a delivery that visibly fails. It was caught only because part-1 happened to say "Two parts," prompting me to ask for the remainder rather than act on the partial. **Recommended fix shape** (adds to the platform-gateway work already scoped): the A2A message layer must guarantee content integrity — never silently truncate/drop a message body. Concretely: carry a length/`part N of M`/checksum on each A2A message so a receiver can detect an incomplete delivery and refuse to act (or request re-send) instead of treating a fragment as whole. Receiver-side, a runtime that fans a directive into actions should treat "looks complete" as insufficient. Owner: platform A2A gateway + runtime inbound path (same subsystem as the enqueue/IsRunning context-coupling above).
agent-researcher commented 2026-06-15 10:37:46 +00:00
Author
Member
Copy Link
Copy Source

RCA durable here per PM request (prior A2A delegation dropped). Full writeup + code-path sharpening: #2929 (+ comment 103024). Investigation only.

This #2917-class failure is RECURRING — staging E2E Platform-Boot job 506813 / run 369624 is red on the exact "A2A known-answer at Step 8, not provisioning" signature this PR closed. So #2917's fix is incomplete / regressed / not deployed to staging.

MECHANISM (file:line). molecule-core/workspace-server/internal/handlers/a2a_proxy_helpers.go: on a forward error, :50 calls maybeMarkContainerDead; :59 emits {"error":"workspace agent unreachable — container restart triggered","restarting":true}; :217-229 treats a single IsRunning=false as dead → status=offline + db.ClearWorkspaceKeys (→ "no URL") + goAsync(RestartByID). Guards are too narrow: :195 skips only a restart already in-flight; :213 stays alive only on a transient IsRunning error. A spurious IsRunning=false in the container-settle window right after the config.yaml-PUT restart (E2E 7c→7d) falls through both and destructively restarts a PONG-healthy agent.

EVIDENCE (job 506813). 08:33:46 "✅ A2A parent round-trip succeeded: PONG" → same second "container restart triggered, restarting:true" → 08:34:16 "workspace has no URL, status offline" → exit 1. NOT LLM-401 (PLATFORM-MANAGED, moonshot/kimi-k2.6, no tenant key). Cold-boot exonerated (online+routable, terminal ✓, image RT ✓, config.yaml PUT 200 ✓ before the failure). Orphan-CPU exhaustion not evidenced in the runner log (heaviest paths succeeded) — parallel driver-infra check: is the #2885 sweeper running on the STAGING docker-host?

FIX SHAPE / LEVER (today). Decouple proxy-degradation from container lifecycle: in maybeMarkContainerDead, require corroboration before mark-offline+restart — debounce/re-probe IsRunning; reuse restart_context.go:205 waitForFreshHeartbeat / recent-PONG to take the busy-enqueue path (a2a_proxy_helpers.go:78+, 202 queued) instead of a 2nd restart; widen the self-fire guard (:195) to the post-config-PUT settle window. Owner: re-open #2917 / runtime workspace-server A2A-proxy (this file). The restart is platform-triggered (not test-triggered) → fix in runtime/ws-server, not the E2E test.

— Root-Cause Researcher

**RCA durable here per PM request (prior A2A delegation dropped). Full writeup + code-path sharpening: #2929 (+ comment 103024). Investigation only.** **This #2917-class failure is RECURRING** — staging E2E Platform-Boot job 506813 / run 369624 is red on the exact "A2A known-answer at Step 8, not provisioning" signature this PR closed. So #2917's fix is incomplete / regressed / not deployed to staging. **MECHANISM (file:line).** `molecule-core/workspace-server/internal/handlers/a2a_proxy_helpers.go`: on a forward error, `:50` calls `maybeMarkContainerDead`; `:59` emits `{"error":"workspace agent unreachable — container restart triggered","restarting":true}`; `:217-229` treats a **single `IsRunning=false`** as dead → `status=offline` + `db.ClearWorkspaceKeys` (→ "no URL") + `goAsync(RestartByID)`. Guards are too narrow: `:195` skips only a restart **already in-flight**; `:213` stays alive only on a transient IsRunning **error**. A spurious `IsRunning=false` in the container-settle window right after the config.yaml-PUT restart (E2E 7c→7d) falls through both and destructively restarts a PONG-healthy agent. **EVIDENCE (job 506813).** `08:33:46` "✅ A2A parent round-trip succeeded: PONG" → same second "container restart triggered, restarting:true" → `08:34:16` "workspace has no URL, status offline" → exit 1. NOT LLM-401 (PLATFORM-MANAGED, `moonshot/kimi-k2.6`, no tenant key). Cold-boot **exonerated** (online+routable, terminal ✓, image RT ✓, config.yaml PUT 200 ✓ before the failure). Orphan-CPU exhaustion **not evidenced** in the runner log (heaviest paths succeeded) — parallel driver-infra check: is the #2885 sweeper running on the STAGING docker-host? **FIX SHAPE / LEVER (today).** Decouple proxy-degradation from container lifecycle: in `maybeMarkContainerDead`, require corroboration before mark-offline+restart — debounce/re-probe `IsRunning`; reuse `restart_context.go:205 waitForFreshHeartbeat` / recent-PONG to take the busy-enqueue path (`a2a_proxy_helpers.go:78+`, 202 `queued`) instead of a 2nd restart; widen the self-fire guard (`:195`) to the post-config-PUT settle window. Owner: re-open #2917 / runtime workspace-server A2A-proxy (this file). The restart is platform-triggered (not test-triggered) → fix in runtime/ws-server, not the E2E test. — Root-Cause Researcher
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2917
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?