Canary failing: staging SaaS smoke #2737

Closed
opened 2026-06-13 09:05:24 +00:00 by gitea-actions · 447 comments

Smoke run failed at 2026-06-13T09:05:23Z.

Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358499

This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.

Smoke run failed at 2026-06-13T09:05:23Z. Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358499 This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358642
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358776
Member

MECHANISM: Fresh RCA tick on current main 6163f6636fc8 shows this staging smoke red has two live sub-failures. Platform Boot reaches tenant provisioning/running, then tests/e2e/test_staging_full_saas.sh:1105-1170 polls GET /workspaces/:id/a2a/queue/:qid for the known-answer A2A item until the 30-poll budget expires; the queue transitions through queued/dispatched but never returns completed. The full-SaaS job separately reaches A2A and fails with _ResultError, but the runtime response is still opaque, so the true provider/runtime stderr is not exposed yet.

EVIDENCE: run 358767/job 487990 logs A2A known-answer queue poll attempt 30/30 status=dispatched followed by queue poll timed out waiting for ... to complete. run 358767/job 487989 logs A2A — STAGING LLM/BACKEND/RUNTIME FAILURE (_ResultError) and Agent error (_ResultError) — see workspace logs for details. This is distinct from #2743's concierge user_tasks create timeout, which fails before user_tasks assertions.

RECOMMENDED FIX SHAPE: For Platform Boot, route to the A2A queue/worker readiness lane: add diagnostics that report queue row timestamps/status/worker owner when the poll budget expires, and verify the runtime consumes dispatched rows after restart/bring-up. For the full-SaaS _ResultError, land/fix the workspace-runtime sanitized error-detail path first (runtime #132 currently has a security RC for bare sk-... redaction), then rerun so the harness can classify provider/runtime failures without leaking secrets.

MECHANISM: Fresh RCA tick on current main `6163f6636fc8` shows this staging smoke red has two live sub-failures. Platform Boot reaches tenant provisioning/running, then `tests/e2e/test_staging_full_saas.sh:1105-1170` polls `GET /workspaces/:id/a2a/queue/:qid` for the known-answer A2A item until the 30-poll budget expires; the queue transitions through queued/dispatched but never returns `completed`. The full-SaaS job separately reaches A2A and fails with `_ResultError`, but the runtime response is still opaque, so the true provider/runtime stderr is not exposed yet. EVIDENCE: run 358767/job 487990 logs `A2A known-answer queue poll attempt 30/30 status=dispatched` followed by `queue poll timed out waiting for ... to complete`. run 358767/job 487989 logs `A2A — STAGING LLM/BACKEND/RUNTIME FAILURE (_ResultError)` and `Agent error (_ResultError) — see workspace logs for details`. This is distinct from #2743's concierge `user_tasks` create timeout, which fails before user_tasks assertions. RECOMMENDED FIX SHAPE: For Platform Boot, route to the A2A queue/worker readiness lane: add diagnostics that report queue row timestamps/status/worker owner when the poll budget expires, and verify the runtime consumes dispatched rows after restart/bring-up. For the full-SaaS `_ResultError`, land/fix the workspace-runtime sanitized error-detail path first (runtime #132 currently has a security RC for bare `sk-...` redaction), then rerun so the harness can classify provider/runtime failures without leaking secrets.
Member

Post-#132 / current-main recheck:

MECHANISM: runtime#132 is merged and its runtime publish/propagate jobs succeeded at 109bb74d, but the latest failed Platform Boot datapoint (core run 358888 / job 488231) started at 10:24:37Z, before runtime propagation completed at 10:26:57Z. That run still dies in tests/e2e/test_staging_full_saas.sh inside a2a_send_or_poll_queue after the known-answer send returns queued and all 30 queue-status polls remain status=queued (timeout path at lines 1159-1170 on main 0dd269e8). Because the queue never reaches a terminal failure response, #132's external sanitized error detail has not yet produced a legible payload for this specific red.

EVIDENCE: molecule-ai-workspace-runtime run 358907: publish-runtime / propagate success at 10:26:57Z. molecule-core run 358888 / job 488231: A2A known-answer queue poll attempt 30/30 status=queued then queue poll timed out. Newer post-propagation scheduled runs 358940 and 358944 are still in progress and have not yet reached the A2A assertion/error-detail phase in the logs I checked. Separately, #2743 is verified fixed: run 358888 E2E Staging Concierge user_tasks passed in 2m25s after #2746.

RECOMMENDED FIX SHAPE: do not route a new code fix from the pre-propagation red alone. Re-check the first post-#132 run that reaches the A2A known-answer phase; if it still queues forever, investigate the queue worker/runtime execution path rather than the external error sanitizer. If it returns a terminal failure with error_detail, use that sanitized detail as the next RCA anchor.

Post-#132 / current-main recheck: MECHANISM: runtime#132 is merged and its runtime publish/propagate jobs succeeded at `109bb74d`, but the latest failed Platform Boot datapoint (`core` run 358888 / job 488231) started at 10:24:37Z, before runtime propagation completed at 10:26:57Z. That run still dies in `tests/e2e/test_staging_full_saas.sh` inside `a2a_send_or_poll_queue` after the known-answer send returns queued and all 30 queue-status polls remain `status=queued` (timeout path at lines 1159-1170 on main `0dd269e8`). Because the queue never reaches a terminal failure response, #132's external sanitized error detail has not yet produced a legible payload for this specific red. EVIDENCE: `molecule-ai-workspace-runtime` run 358907: `publish-runtime / propagate` success at 10:26:57Z. `molecule-core` run 358888 / job 488231: `A2A known-answer queue poll attempt 30/30 status=queued` then `queue poll timed out`. Newer post-propagation scheduled runs 358940 and 358944 are still in progress and have not yet reached the A2A assertion/error-detail phase in the logs I checked. Separately, #2743 is verified fixed: run 358888 `E2E Staging Concierge user_tasks` passed in 2m25s after #2746. RECOMMENDED FIX SHAPE: do not route a new code fix from the pre-propagation red alone. Re-check the first post-#132 run that reaches the A2A known-answer phase; if it still queues forever, investigate the queue worker/runtime execution path rather than the external error sanitizer. If it returns a terminal failure with `error_detail`, use that sanitized detail as the next RCA anchor.
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358940
Member

MECHANISM: #132 is published, but the current staging smoke is still running a template pinned to runtime 0.3.23, so the new sanitized error_detail path is not active in the fresh claude-code workspace. molecule-ai-workspace-runtime main has #132 at 109bb74d, but molecule-ai-workspace-template-claude-code main .runtime-version is still 0.3.23 (the 0.3.24 bump is open as template #120). The core harness then fails in tests/e2e/test_staging_full_saas.sh around the A2A known-answer assertion/diagnostic path instead of receiving the new detail.

EVIDENCE: core schedule run 358940 / job 488344 is post-runtime-propagation and reaches the workspace online path, then prints the same opaque text: Agent error (_ResultError). The diagnostic burst contains no error_detail. Runtime publish/propagate for #132 succeeded on 109bb74d, but template main still reports .runtime-version = 0.3.23; template #120 head 165675c0 changes that to 0.3.24 and is still open. #2743 remains verified fixed in the same core head: user_tasks passed on run 358888.

RECOMMENDED FIX SHAPE: merge the runtime-version propagation PRs for all affected templates before expecting #132 to de-opaque staging canaries. For this claude-code smoke path, the responsible file is molecule-ai-workspace-template-claude-code/.runtime-version; sibling template bumps should land consistently where those templates are used. After the template bump deploys, rerun #2737 and use the sanitized detail if the A2A failure persists.

MECHANISM: #132 is published, but the current staging smoke is still running a template pinned to runtime `0.3.23`, so the new sanitized `error_detail` path is not active in the fresh claude-code workspace. `molecule-ai-workspace-runtime` main has #132 at `109bb74d`, but `molecule-ai-workspace-template-claude-code` main `.runtime-version` is still `0.3.23` (the `0.3.24` bump is open as template #120). The core harness then fails in `tests/e2e/test_staging_full_saas.sh` around the A2A known-answer assertion/diagnostic path instead of receiving the new detail. EVIDENCE: core schedule run 358940 / job 488344 is post-runtime-propagation and reaches the workspace online path, then prints the same opaque text: `Agent error (_ResultError)`. The diagnostic burst contains no `error_detail`. Runtime publish/propagate for #132 succeeded on `109bb74d`, but template main still reports `.runtime-version = 0.3.23`; template #120 head `165675c0` changes that to `0.3.24` and is still open. #2743 remains verified fixed in the same core head: user_tasks passed on run 358888. RECOMMENDED FIX SHAPE: merge the runtime-version propagation PRs for all affected templates before expecting #132 to de-opaque staging canaries. For this claude-code smoke path, the responsible file is `molecule-ai-workspace-template-claude-code/.runtime-version`; sibling template bumps should land consistently where those templates are used. After the template bump deploys, rerun #2737 and use the sanitized detail if the A2A failure persists.
Member

Post-propagation classification check: not yet sampled.

MECHANISM: the runtime propagation prerequisite is now satisfied for the claude-code smoke path: molecule-ai-workspace-template-claude-code main merged #120 at e2bd6f77 on 2026-06-13T10:46:26Z and .runtime-version is now 0.3.24 (runtime #132 sanitizer included). However, the currently available failing staging runs still predate that merge: core run 358940 started 10:30:26Z and synthetic run 358944 started 10:32:25Z. Those failures therefore still sampled the old template/runtime path and cannot classify the now-legible post-#132 error.

EVIDENCE: run 358940 / job 488344 still shows the opaque payload Agent error (_ResultError), with no error_detail, but it began before #120 merged. Template main now reports 0.3.24; openclaw #74 also merged at 0b0cfde5 on 2026-06-13T10:46:27Z. The only later scheduled core runs in the API list are sweep jobs (358961/358962), not staging-full-saas smoke runs.

RECOMMENDED FIX SHAPE: wait for or trigger the first staging-full-saas smoke after 2026-06-13T10:46:26Z. If the A2A failure recurs, classify from the sanitized error_detail: provider auth/quota/upstream if it names MiniMax/Anthropic HTTP/auth; runtime if it points at agent subprocess/tooling; backend if it points at Molecule registry/A2A/control-plane handling. Current classification remains pending because no post-template-bump failure sample exists yet.

Post-propagation classification check: not yet sampled. MECHANISM: the runtime propagation prerequisite is now satisfied for the claude-code smoke path: `molecule-ai-workspace-template-claude-code` main merged #120 at `e2bd6f77` on 2026-06-13T10:46:26Z and `.runtime-version` is now `0.3.24` (runtime #132 sanitizer included). However, the currently available failing staging runs still predate that merge: core run 358940 started 10:30:26Z and synthetic run 358944 started 10:32:25Z. Those failures therefore still sampled the old template/runtime path and cannot classify the now-legible post-#132 error. EVIDENCE: run 358940 / job 488344 still shows the opaque payload `Agent error (_ResultError)`, with no `error_detail`, but it began before #120 merged. Template main now reports `0.3.24`; openclaw #74 also merged at `0b0cfde5` on 2026-06-13T10:46:27Z. The only later scheduled core runs in the API list are sweep jobs (358961/358962), not staging-full-saas smoke runs. RECOMMENDED FIX SHAPE: wait for or trigger the first staging-full-saas smoke after 2026-06-13T10:46:26Z. If the A2A failure recurs, classify from the sanitized `error_detail`: provider auth/quota/upstream if it names MiniMax/Anthropic HTTP/auth; runtime if it points at agent subprocess/tooling; backend if it points at Molecule registry/A2A/control-plane handling. Current classification remains pending because no post-template-bump failure sample exists yet.
Member

MECHANISM: #2737 is a platform A2A queue-drain fault, not a provider/runtime terminal error and not a #132 sanitizer issue. In run 358888 the parent workspace is online/routable, the first known-answer A2A POST hits a transient 502/busy path, and the retry successfully inserts a2a_queue row 205f85c4-4330-4653-8789-6ec4cd6b3d53. From there the only production drainer is RegistryHandler.Heartbeat calling drainQueue when payload.ActiveTasks < max_concurrent (workspace-server/internal/handlers/registry.go:1128-1150), which invokes WorkspaceHandler.DrainQueueForWorkspace / DequeueNext (workspace-server/internal/handlers/a2a_queue.go:217-257,359-407). The row never leaves status=queued before the E2E poll budget expires, so the queue item is not being claimed/drained on the readiness heartbeat boundary.

EVIDENCE: run 358888 / job 488231: parent 375ad60e-977d-4625-b46d-5e0782df7325 is online and routable; log then shows A2A transient 502, then A2A queued (queue_id=205f85c4...), then 30 polls of status=queued, ending with queue poll timed out. Sibling job E2E Staging Concierge user_tasks passed, so #2746's create retry is validated and this red is isolated to Platform Boot's queued A2A drain. The code path confirms enqueue is synchronous (a2a_proxy_helpers.go:78-128), while drain is heartbeat-gated rather than driven by an independent queue worker.

RECOMMENDED FIX SHAPE: treat this as a queue/dispatch readiness-boundary bug. In workspace-server/internal/handlers, add a deterministic drain trigger for newly queued A2A rows (or a bounded background queue worker/sweeper) so a queued item is claimed even if the next heartbeat does not report spare capacity quickly enough. Keep heartbeat-driven drain, but do not make it the only progress mechanism. Also update tests/e2e/test_staging_full_saas.sh to log queue attempts/last_error from GetA2AQueueStatus, so future queued stalls distinguish "never claimed" from "claimed then requeued" without platform log scraping.

MECHANISM: #2737 is a platform A2A queue-drain fault, not a provider/runtime terminal error and not a #132 sanitizer issue. In run 358888 the parent workspace is online/routable, the first known-answer A2A POST hits a transient 502/busy path, and the retry successfully inserts `a2a_queue` row `205f85c4-4330-4653-8789-6ec4cd6b3d53`. From there the only production drainer is `RegistryHandler.Heartbeat` calling `drainQueue` when `payload.ActiveTasks < max_concurrent` (`workspace-server/internal/handlers/registry.go:1128-1150`), which invokes `WorkspaceHandler.DrainQueueForWorkspace` / `DequeueNext` (`workspace-server/internal/handlers/a2a_queue.go:217-257,359-407`). The row never leaves `status=queued` before the E2E poll budget expires, so the queue item is not being claimed/drained on the readiness heartbeat boundary. EVIDENCE: run 358888 / job 488231: parent `375ad60e-977d-4625-b46d-5e0782df7325` is `online and routable`; log then shows `A2A transient 502`, then `A2A queued (queue_id=205f85c4...)`, then 30 polls of `status=queued`, ending with `queue poll timed out`. Sibling job `E2E Staging Concierge user_tasks` passed, so #2746's create retry is validated and this red is isolated to Platform Boot's queued A2A drain. The code path confirms enqueue is synchronous (`a2a_proxy_helpers.go:78-128`), while drain is heartbeat-gated rather than driven by an independent queue worker. RECOMMENDED FIX SHAPE: treat this as a queue/dispatch readiness-boundary bug. In `workspace-server/internal/handlers`, add a deterministic drain trigger for newly queued A2A rows (or a bounded background queue worker/sweeper) so a queued item is claimed even if the next heartbeat does not report spare capacity quickly enough. Keep heartbeat-driven drain, but do not make it the only progress mechanism. Also update `tests/e2e/test_staging_full_saas.sh` to log queue `attempts`/`last_error` from `GetA2AQueueStatus`, so future queued stalls distinguish "never claimed" from "claimed then requeued" without platform log scraping.
Member

Refinement on the opaque SaaS canary _ResultError thread (separate from the Platform Boot queue-stuck thread):

MECHANISM: CI does not establish that the failing canary workspace actually ran the promoted 0.3.24 claude-code image. In run 358888, the core SaaS job started at 10:47:28Z and tenant provisioning completed at 10:48:21Z. The claude-code template publish for #120 built the 0.3.24 image from 10:46:33Z to 10:48:55Z and only promoted the staging runtime_image_pin at 10:48:57-10:48:58Z. The workspace provisioning step began later (10:49:37Z), but the tenant/provisioning workflow had already started before the staging pin was promoted, and the CI log does not print a runtime package version from inside the workspace. So from CI/logs only, this is not a clean proof that the canary process was on 0.3.24.

EVIDENCE: run 358888 / job 488230 shows Tenant provisioning complete at 10:48:21Z, parent workspace provision at 10:49:37Z, then the opaque response Agent error (_ResultError) — see workspace logs for details. at 10:54:19Z. Template #120 publish run 358969 shows Build & push workspace-template-claude-code image completed at 10:48:55Z and Promote runtime_image_pins ... staging completed at 10:48:58Z. Runtime 109bb74d does contain #132: a2a_executor.py:752-755 calls sanitize_agent_error(exc=e, stderr=error_detail_for_external(e)); executor_helpers.py:685-691 returns the exact opaque string when stderr/detail is empty.

RECOMMENDED FIX SHAPE: do not re-bump 0.3.24. First add/enable an explicit runtime-version/build stamp in the staging harness or workspace heartbeat/register diagnostics so the canary can prove which runtime wheel/image it is executing. Then rerun after the 0.3.24 staging pin is definitely active. If the exact opaque string persists on a proven-0.3.24 workspace, classify it as sanitizer fallback with an empty-detail _ResultError, not pre-sanitizer; the next runtime fix should teach error_detail_for_external how to unwrap _ResultError (or the A2A SDK result exception it wraps) into a safe message.

Refinement on the opaque SaaS canary `_ResultError` thread (separate from the Platform Boot queue-stuck thread): MECHANISM: CI does not establish that the failing canary workspace actually ran the promoted 0.3.24 claude-code image. In run 358888, the core SaaS job started at 10:47:28Z and tenant provisioning completed at 10:48:21Z. The claude-code template publish for #120 built the 0.3.24 image from 10:46:33Z to 10:48:55Z and only promoted the staging runtime_image_pin at 10:48:57-10:48:58Z. The workspace provisioning step began later (10:49:37Z), but the tenant/provisioning workflow had already started before the staging pin was promoted, and the CI log does not print a runtime package version from inside the workspace. So from CI/logs only, this is not a clean proof that the canary process was on 0.3.24. EVIDENCE: run 358888 / job 488230 shows `Tenant provisioning complete` at 10:48:21Z, parent workspace provision at 10:49:37Z, then the opaque response `Agent error (_ResultError) — see workspace logs for details.` at 10:54:19Z. Template #120 publish run 358969 shows `Build & push workspace-template-claude-code image` completed at 10:48:55Z and `Promote runtime_image_pins ... staging` completed at 10:48:58Z. Runtime `109bb74d` does contain #132: `a2a_executor.py:752-755` calls `sanitize_agent_error(exc=e, stderr=error_detail_for_external(e))`; `executor_helpers.py:685-691` returns the exact opaque string when `stderr`/detail is empty. RECOMMENDED FIX SHAPE: do not re-bump 0.3.24. First add/enable an explicit runtime-version/build stamp in the staging harness or workspace heartbeat/register diagnostics so the canary can prove which runtime wheel/image it is executing. Then rerun after the 0.3.24 staging pin is definitely active. If the exact opaque string persists on a proven-0.3.24 workspace, classify it as sanitizer fallback with an empty-detail `_ResultError`, not pre-sanitizer; the next runtime fix should teach `error_detail_for_external` how to unwrap `_ResultError` (or the A2A SDK result exception it wraps) into a safe message.
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358987
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359032
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359080
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359177
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359237
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359262
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359280
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359305
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359323
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359346
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359364
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359388
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359406
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359429
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359447
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359480
Member

#2737 RCA update — A2A queue-drain fault is reproduced in code

MECHANISM. Current main confirms the hypothesis, with one extra stuck-dispatched gap. Busy A2A POSTs enqueue at workspace-server/internal/handlers/a2a_proxy_helpers.go:78-124; the only normal queue drain trigger is RegistryHandler.Heartbeat, which calls drainQueue only when payload.ActiveTasks < maxConcurrent (registry.go:1128-1150). The persisted busy value is not server-derived: heartbeat overwrites workspaces.active_tasks directly from the runtime payload (registry.go:783-811). DequeueNext then claims exactly one queued item by changing it to dispatched (a2a_queue.go:217-257), and DrainQueueForWorkspace only completes/fails/requeues after proxyA2ARequest returns (a2a_queue.go:359-407). Repro from code: workspace active_tasks=max_concurrent, enqueue A/B/C, continue heartbeating with active_tasks=1; no drain fires, so the rows remain queued. If a drain did claim one row and the runtime never receives/finishes the POST, it stays dispatched until the proxy path returns.

EVIDENCE. Runtime-side normal errors are not supposed to leak busy state: both generic runtime and claude-code set active in a try and clear in finally (molecule_runtime/a2a_executor.py:423-430,739-760; claude_sdk_executor.py:1649-1656,1840-1841), and runtime tests assert a hung tool releases active_tasks after timeout (tests/integration/test_agent_responsiveness_under_load.py:597-639). So a live stuck-busy engine means the executor never reaches that finally, the clear heartbeat is not observed, or an in-flight task is genuinely still running. Separate but matching today’s “dispatched never landed” symptom: the delegation ledger sweeper is wired (cmd/server/main.go:464-470) but delegation_sweeper.go:242-247 explicitly ignores in-flight rows with NULL last_heartbeat; default ledger deadline is 6h (delegation_ledger.go:71-88). A task recorded as dispatched but never started/heartbeat’d therefore is not marked stuck by the 10m threshold.

RECOMMENDED FIX SHAPE. Core-be should add a bounded background queue/dispatch sweeper independent of heartbeat spare-capacity: periodically find a2a_queue rows stuck in dispatched past a short dispatch lease and requeue/fail them using the existing MarkQueueItemFailed semantics (a2a_queue.go:281-295), and trigger drain for online workspaces with queued depth using DB state plus a per-workspace concurrency lease, not solely the next idle heartbeat. Also tighten DelegationSweeper: NULL last_heartbeat on queued/dispatched should become stuck after a first-start grace (not wait for 6h deadline). Immediate unwedge: soft-restart the affected engine workspace(s) to reset runtime busy state and force fresh heartbeats; for already-claimed queue rows, operator can requeue stale a2a_queue.status='dispatched' rows or mark affected delegations stuck/failed so callers retry cleanly. Do not conflate this with the earlier engine _ResultError: this path is queue/dispatch starvation or a claimed POST that never reaches completion, not an LLM execution error.

#2737 RCA update — A2A queue-drain fault is reproduced in code **MECHANISM.** Current main confirms the hypothesis, with one extra stuck-dispatched gap. Busy A2A POSTs enqueue at `workspace-server/internal/handlers/a2a_proxy_helpers.go:78-124`; the only normal queue drain trigger is `RegistryHandler.Heartbeat`, which calls `drainQueue` only when `payload.ActiveTasks < maxConcurrent` (`registry.go:1128-1150`). The persisted busy value is not server-derived: heartbeat overwrites `workspaces.active_tasks` directly from the runtime payload (`registry.go:783-811`). `DequeueNext` then claims exactly one queued item by changing it to `dispatched` (`a2a_queue.go:217-257`), and `DrainQueueForWorkspace` only completes/fails/requeues after `proxyA2ARequest` returns (`a2a_queue.go:359-407`). Repro from code: workspace `active_tasks=max_concurrent`, enqueue A/B/C, continue heartbeating with active_tasks=1; no drain fires, so the rows remain queued. If a drain did claim one row and the runtime never receives/finishes the POST, it stays `dispatched` until the proxy path returns. **EVIDENCE.** Runtime-side normal errors are not supposed to leak busy state: both generic runtime and claude-code set active in a `try` and clear in `finally` (`molecule_runtime/a2a_executor.py:423-430,739-760`; `claude_sdk_executor.py:1649-1656,1840-1841`), and runtime tests assert a hung tool releases active_tasks after timeout (`tests/integration/test_agent_responsiveness_under_load.py:597-639`). So a live stuck-busy engine means the executor never reaches that `finally`, the clear heartbeat is not observed, or an in-flight task is genuinely still running. Separate but matching today’s “dispatched never landed” symptom: the delegation ledger sweeper is wired (`cmd/server/main.go:464-470`) but `delegation_sweeper.go:242-247` explicitly ignores in-flight rows with NULL `last_heartbeat`; default ledger deadline is 6h (`delegation_ledger.go:71-88`). A task recorded as `dispatched` but never started/heartbeat’d therefore is not marked stuck by the 10m threshold. **RECOMMENDED FIX SHAPE.** Core-be should add a bounded background queue/dispatch sweeper independent of heartbeat spare-capacity: periodically find `a2a_queue` rows stuck in `dispatched` past a short dispatch lease and requeue/fail them using the existing `MarkQueueItemFailed` semantics (`a2a_queue.go:281-295`), and trigger drain for online workspaces with queued depth using DB state plus a per-workspace concurrency lease, not solely the next idle heartbeat. Also tighten `DelegationSweeper`: NULL `last_heartbeat` on `queued`/`dispatched` should become `stuck` after a first-start grace (not wait for 6h deadline). Immediate unwedge: soft-restart the affected engine workspace(s) to reset runtime busy state and force fresh heartbeats; for already-claimed queue rows, operator can requeue stale `a2a_queue.status='dispatched'` rows or mark affected `delegations` stuck/failed so callers retry cleanly. Do not conflate this with the earlier engine `_ResultError`: this path is queue/dispatch starvation or a claimed POST that never reaches completion, not an LLM execution error.
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359624
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/359844
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360101
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360220
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360294
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360462
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360731
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/360894
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/361079
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/361255
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/361483
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/361602
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/361773
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362070
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362307
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362486
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362658
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362685
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362764
Member

MECHANISM: Current main 9595757a has a shared staging-create failure, not a tenant/LLM execution failure. Both scheduled lanes stop at tests/e2e/test_staging_full_saas.sh:349-351 while creating the org via POST /cp/admin/orgs; CP /health is green, then the create returns HTTP 400 before any provisioning/A2A step runs. The harness currently loses the actionable 400 body because CREATE_RESP=$(admin_call ...) runs under set -e and admin_call uses curl --fail-with-body (tests/e2e/test_staging_full_saas.sh:227,339-344), so curl exits 22 and the later raw-body diagnostic block is skipped.

EVIDENCE: staging-smoke run 362764/job 494927 and continuous-synth run 362810/job 495016 both fail at the same line sequence: CP reachable, then Creating org ... via /cp/admin/orgs, then curl: (22) The requested URL returned error: 400. Smoke has been persistent on this issue (#2737) from at least 2026-06-14T00:37Z through 04:00Z. The slug-collision class is unlikely here: slugs include the collision-proof random suffix (e2e-smoke-20260614-smoke-362764-760ff49d, e2e-20260614-synth-362810-eee4ee61) and teardown finds no orphan EC2.

RECOMMENDED FIX SHAPE: Owner should first inspect staging CP logs for /cp/admin/orgs 400 validation/body on the two run slugs; this is likely staging CP contract/config drift at the org-create boundary, not a molecule-core tenant code regression. In parallel, fix tests/e2e/test_staging_full_saas.sh org-create handling to temporarily disable set -e around admin_call, capture HTTP status + body, and print the body for any non-2xx (same spirit as the slug 409 diagnostic), so future 400s are self-diagnosing in CI.

MECHANISM: Current main `9595757a` has a shared staging-create failure, not a tenant/LLM execution failure. Both scheduled lanes stop at `tests/e2e/test_staging_full_saas.sh:349-351` while creating the org via `POST /cp/admin/orgs`; CP `/health` is green, then the create returns HTTP 400 before any provisioning/A2A step runs. The harness currently loses the actionable 400 body because `CREATE_RESP=$(admin_call ...)` runs under `set -e` and `admin_call` uses `curl --fail-with-body` (`tests/e2e/test_staging_full_saas.sh:227,339-344`), so curl exits 22 and the later raw-body diagnostic block is skipped. EVIDENCE: staging-smoke run 362764/job 494927 and continuous-synth run 362810/job 495016 both fail at the same line sequence: CP reachable, then `Creating org ... via /cp/admin/orgs`, then `curl: (22) The requested URL returned error: 400`. Smoke has been persistent on this issue (#2737) from at least 2026-06-14T00:37Z through 04:00Z. The slug-collision class is unlikely here: slugs include the collision-proof random suffix (`e2e-smoke-20260614-smoke-362764-760ff49d`, `e2e-20260614-synth-362810-eee4ee61`) and teardown finds no orphan EC2. RECOMMENDED FIX SHAPE: Owner should first inspect staging CP logs for `/cp/admin/orgs` 400 validation/body on the two run slugs; this is likely staging CP contract/config drift at the org-create boundary, not a molecule-core tenant code regression. In parallel, fix `tests/e2e/test_staging_full_saas.sh` org-create handling to temporarily disable `set -e` around `admin_call`, capture HTTP status + body, and print the body for any non-2xx (same spirit as the slug 409 diagnostic), so future 400s are self-diagnosing in CI.
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/362960
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363100
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363262
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363372
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363641
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363758
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/363893
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364031
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364153
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364277
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364391
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364574
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364868
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364953
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/364972
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365054
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365120
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365155
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365248
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365337
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365496
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365665
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365741
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/365933
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366084
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366277
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366489
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366600
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366745
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366891
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/366956
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367099
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367246
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367308
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367396
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367487
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367645
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367770
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367869
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/367952
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368050
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368251
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368312
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368354
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368484
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368512
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368607
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368634
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368661
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368746
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/368864
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369062
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369203
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369400
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369509
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369635
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369705
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369853
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/369965
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370037
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370137
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370267
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370369
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370478
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370622
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370752
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/370968
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371086
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371217
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371305
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371458
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371561
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371652
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371781
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371865
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/371971
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372183
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372270
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372431
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372657
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372799
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/372864
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373137
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373214
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373268
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373366
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373475
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373554
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373675
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373820
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373870
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/373975
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374218
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374288
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374388
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374438
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374462
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374491
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374509
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374534
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374555
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374583
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374602
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374629
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374648
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374673
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374691
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374715
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374733
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374761
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374779
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374805
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374823
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374849
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374867
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374925
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/374961
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375076
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375111
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375135
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375182
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375209
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375243
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375330
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375447
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375620
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375710
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375890
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375924
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/375964
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376102
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376312
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376450
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376630
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376674
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376698
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376716
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376740
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376758
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376821
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376840
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376878
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/376971
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377073
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377128
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377174
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377230
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377267
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377303
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377333
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377351
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377441
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377524
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377595
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377717
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377741
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377760
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377787
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377805
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377831
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377850
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377875
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377893
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/377972
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378009
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378034
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378052
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378076
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378106
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378249
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378284
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378398
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378453
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378590
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378660
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378731
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378817
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378860
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378878
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378902
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378920
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378950
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378968
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/378992
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379073
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379097
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379200
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379311
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379452
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379573
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379646
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379719
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379737
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379768
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379786
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379811
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379832
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379860
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379879
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/379923
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380005
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380088
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380182
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380257
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380302
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380330
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380348
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380374
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380392
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380418
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380436
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380460
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380478
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380503
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380521
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380547
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380566
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380614
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380632
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380692
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380794
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380863
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380900
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380925
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/380984
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381047
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381090
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381114
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381151
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381286
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381366
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381401
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381433
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381521
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381586
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381618
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/381709
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382006
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382090
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382341
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382601
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382736
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382901
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/382989
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383034
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383053
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383083
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383105
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383219
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383370
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383574
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383704
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383760
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383817
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383927
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/383977
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384005
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384023
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384047
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384065
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384090
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384124
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384148
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384170
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384198
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384218
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384250
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384268
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384297
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384325
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384458
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/384729
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385003
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385162
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385285
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385359
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385425
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385487
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385595
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385675
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385806
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385846
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385875
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385894
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/385959
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386004
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386046
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386065
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386095
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386165
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386199
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386226
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386279
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386341
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386394
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386436
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386461
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386479
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386503
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386521
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386549
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386567
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386593
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386611
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386637
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386655
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386679
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386697
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386722
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386740
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386800
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386854
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386905
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/386977
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387029
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387055
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387084
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387102
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387126
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387177
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387257
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387437
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387551
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387611
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387647
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387699
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387723
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387741
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387784
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387901
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/387959
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388009
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388080
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388145
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388253
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388339
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388415
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388472
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388526
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388639
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388829
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/388893
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389030
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389134
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389255
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389329
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389495
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389649
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389762
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/389934
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390121
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390217
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390262
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390282
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390306
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390396
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390421
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390439
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390463
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390482
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390509
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390527
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390551
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390569
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390594
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390612
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390636
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390654
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390678
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390696
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390720
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390767
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390929
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/390980
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/391224
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/391296
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/391412
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/391618
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/391801
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392068
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392451
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392637
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392667
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392760
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392813
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392865
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392890
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392933
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/392996
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393026
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393150
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393202
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393243
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393261
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393285
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393334
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393388
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393408
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393474
Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/393493

Smoke recovered at 2026-06-22T14:37:18Z. Closing.

Smoke recovered at 2026-06-22T14:37:18Z. Closing.
gitea-actions bot closed this issue 2026-06-22 14:37:23 +00:00
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2737