molecule-ai/molecule-core
52
0
Fork 2
Code Issues 189 Pull Requests 18 Actions 3 Packages Projects Releases Wiki Activity

Watchdog double-provision race wedges fresh workspaces: two boxes fight one bootstrap-register mint → permanent 401 + online-but-braindead agents #2611

New Issue
Open
opened 2026-06-11 22:44:26 +00:00 by core-devops · 0 comments
core-devops commented 2026-06-11 22:44:26 +00:00
Member
Copy Link
Copy Source

Live incident on enter-os (2026-06-11 ~22:19Z), third first-run blocker of the day (after #2608/#2609):

Observed chain (all evidence in CP logs + on-box docker logs):

  1. Fresh workspaces never heartbeated (separate trigger), so the stall watchdog fired a recreate — and CP provisioned TWO instances for the same workspace within 1s (i-03097c05832d392b8 + i-090da08abea9119ef, two bootstrap-watchers, DNS A-record flapping between them at 22:20:02/03). Every deprovision/provision log line in that window is doubled → two concurrent restart cycles raced.
  2. SaaS provision revokes all workspace tokens and relies on the register handler's bootstrap path ("first successful register mints"). With two boxes racing, one consumes the bootstrap mint; the other then presents nothing/stale and gets HTTP 401 — and the runtime treats register 401 as terminal ("client error — not retrying"). On-box log: Register: HTTP 401 ... proceeding (heartbeat backfill is the recovery path) — but heartbeats need the token too, so there is no recovery.
  3. Result: workspace shows online (canary), ingests messages (a2a_receive logged), agent never processes anything — user chats into a void. Watchdog keeps cycling, sometimes doubling again. Also hit SnapshotCreationPerVolumeRateExceeded from the churn.
  4. Recovery that worked: ONE controlled restart per workspace → single box → bootstrap register 200 → .auth_token + .platform_inbound_secret persisted → agent_log activity resumed.

Fixes needed (durable, not operator ritual):

  • Mutex the recreate path per workspace (CP): a provision/recreate for a workspace must be single-flight; a second concurrent request joins or no-ops, never launches a second instance. (The watchdog + auto-recovery + admin restart can currently overlap.)
  • Register 401 must be retryable when the workspace has no live token: either the bootstrap gate re-opens (presented-but-invalid token while zero live tokens exist → treat as bootstrap), or the runtime retries register with backoff instead of giving up forever.
  • delegation status=completed means DELIVERED, not processed — during the outage my diagnostic delegations reported completed while the target agent was dead. Either rename or add a processed/acked state; operators and agents both misread this.
  • Watchdog recreating a workspace whose register has been failing should surface that on the workspace record (ties into the last_error surfacing gap from #2608/cp#729).

Refs core#2530 (the 401-starves-chat class — its "restart to recover" hint is defeated by the double-provision race), #2608, #2609, cp#729.

Live incident on `enter-os` (2026-06-11 ~22:19Z), third first-run blocker of the day (after #2608/#2609): **Observed chain (all evidence in CP logs + on-box docker logs):** 1. Fresh workspaces never heartbeated (separate trigger), so the stall watchdog fired a recreate — and CP provisioned **TWO instances for the same workspace within 1s** (`i-03097c05832d392b8` + `i-090da08abea9119ef`, two bootstrap-watchers, DNS A-record flapping between them at 22:20:02/03). Every deprovision/provision log line in that window is doubled → two concurrent restart cycles raced. 2. SaaS provision **revokes all workspace tokens** and relies on the register handler's bootstrap path ("first successful register mints"). With two boxes racing, one consumes the bootstrap mint; the other then presents nothing/stale and gets **HTTP 401** — and the runtime treats register 401 as terminal ("client error — not retrying"). On-box log: `Register: HTTP 401 ... proceeding (heartbeat backfill is the recovery path)` — but heartbeats need the token too, so there is no recovery. 3. Result: workspace shows `online` (canary), ingests messages (`a2a_receive` logged), **agent never processes anything** — user chats into a void. Watchdog keeps cycling, sometimes doubling again. Also hit `SnapshotCreationPerVolumeRateExceeded` from the churn. 4. Recovery that worked: ONE controlled restart per workspace → single box → bootstrap register 200 → `.auth_token` + `.platform_inbound_secret` persisted → agent_log activity resumed. **Fixes needed (durable, not operator ritual):** - **Mutex the recreate path per workspace** (CP): a provision/recreate for a workspace must be single-flight; a second concurrent request joins or no-ops, never launches a second instance. (The watchdog + auto-recovery + admin restart can currently overlap.) - **Register 401 must be retryable when the workspace has no live token**: either the bootstrap gate re-opens (presented-but-invalid token while zero live tokens exist → treat as bootstrap), or the runtime retries register with backoff instead of giving up forever. - **`delegation status=completed` means DELIVERED, not processed** — during the outage my diagnostic delegations reported `completed` while the target agent was dead. Either rename or add a processed/acked state; operators and agents both misread this. - Watchdog recreating a workspace whose register has been failing should surface that on the workspace record (ties into the `last_error` surfacing gap from #2608/cp#729). Refs core#2530 (the 401-starves-chat class — its "restart to recover" hint is defeated by the double-provision race), #2608, #2609, cp#729.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2611
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?