feat(registry): reconcile online workspaces against real EC2 state — auto-heal terminated instances (core#2261) #2266

Merged

hongming merged 1 commits from feat/core2261-instance-state-reconciler into main 2026-06-05 00:28:32 +00:00

Conversation 0 Commits 1 Files Changed 3

+479

hongming commented 2026-06-05 00:21:44 +00:00

Owner

Copy Link

Copy Source

Root cause (core#2247)

Every existing liveness sweep in workspace-server keys off a proxy for "is this workspace alive?":

• StartLivenessMonitor — Redis TTL expiry (agent stopped heartbeating)
• StartHealthSweep (Docker pass) — local Docker daemon, only when prov != nil
• StartHealthSweep (remote pass) — last_heartbeat_at freshness for runtime='external'
• StartCPOrphanSweeper — status='removed' rows with a stray instance_id

A SaaS claude-code workspace whose EC2 was terminated/stopped out from under us (manual AWS action, spot reclaim, CP-side reap) falls through all of them: it isn't removed, isn't external, and on a pure-SaaS front-door prov == nil so the Docker pass never runs. The registry kept status=online pointing at a dead instance_id forever. CTO framing: "it shouldn't be pointing at a dead one at all."

The fix

New StartCPInstanceReconciler (workspace-server/internal/registry/cp_instance_reconciler.go): a 60s sweep that asks the one authoritative question the others lack — CPProvisioner.IsRunning, which ultimately asks the control-plane "is this EC2 actually running?" (DescribeInstances-equivalent). On a clean "not running" it feeds the workspace into the existing offline + auto-heal machinery via the same onWorkspaceOffline closure the other sweeps use — no new healing path, just real ground truth driving the one we already have.

onWorkspaceOffline flips the row offline and go wh.RestartByID(...), which reprovisions with the existing volume.

Query (online + SaaS EC2 only)

SELECT id::text FROM workspaces
 WHERE status = 'online'
   AND instance_id IS NOT NULL AND instance_id != ''
   AND COALESCE(runtime, '') <> 'external'
 ORDER BY updated_at DESC
 LIMIT 200

runtime='external' rows are owned by the remote-heartbeat pass; paused/hibernated/removed/provisioning/awaiting_agent are excluded by the status filter.

Guardrails

• Fail-safe: IsRunning returns (true, err) on any transient DB/transport error and (false, nil) only when CP genuinely reports not-running. The reconciler acts strictly on (false, nil); any error short-circuits to "leave it online" so a CP blip never cascades healthy workspaces into reprovision. Covered by TestReconcileOnce_TransientError_DoesNotFlip.
• Online + SaaS only (above).
• Per-cycle LIMIT 200 + per-workspace 10s timeout so one slow CP call can't stall the sweep.
• nil-checker / nil-DB tolerant (matches the sibling CP sweeper).

Wiring

Gated identically to cp-orphan-sweeper in cmd/server/main.go, reusing the same onWorkspaceOffline closure:

if cpProv != nil {
    go supervised.RunWithRecover(ctx, "cp-instance-reconciler", func(c context.Context) {
        registry.StartCPInstanceReconciler(c, cpProv, onWorkspaceOffline, 60*time.Second)
    })
}

Tests

cp_instance_reconciler_test.go (sqlmock + fake checker), mirroring cp_orphan_sweeper_test.go: not-running→flip, running→no-flip, transient-error→no-flip (fail-safe), query-scope excludes external/non-online, mixed batch, query-error, nil-DB, nil-checker disabled, runs-once-and-exits-on-cancel.

go build ./..., go vet ./internal/registry/..., go test ./internal/registry/... all green; gofmt clean on touched files only.

Refs core#2261, core#2247.

DO NOT MERGE — heavy core SOP gate.

🤖 Generated with Claude Code

## Root cause (core#2247) Every existing liveness sweep in workspace-server keys off a **proxy** for "is this workspace alive?": - `StartLivenessMonitor` — Redis TTL expiry (agent stopped heartbeating) - `StartHealthSweep` (Docker pass) — local Docker daemon, only when `prov != nil` - `StartHealthSweep` (remote pass) — `last_heartbeat_at` freshness for `runtime='external'` - `StartCPOrphanSweeper` — `status='removed'` rows with a stray `instance_id` A SaaS `claude-code` workspace whose EC2 was terminated/stopped out from under us (manual AWS action, spot reclaim, CP-side reap) falls through **all** of them: it isn't `removed`, isn't `external`, and on a pure-SaaS front-door `prov == nil` so the Docker pass never runs. The registry kept `status=online` pointing at a dead `instance_id` forever. CTO framing: *"it shouldn't be pointing at a dead one at all."* ## The fix New `StartCPInstanceReconciler` (`workspace-server/internal/registry/cp_instance_reconciler.go`): a 60s sweep that asks the **one authoritative question** the others lack — `CPProvisioner.IsRunning`, which ultimately asks the control-plane "is this EC2 actually running?" (DescribeInstances-equivalent). On a clean "not running" it feeds the workspace into the **existing** offline + auto-heal machinery via the same `onWorkspaceOffline` closure the other sweeps use — no new healing path, just real ground truth driving the one we already have. `onWorkspaceOffline` flips the row offline and `go wh.RestartByID(...)`, which reprovisions **with the existing volume**. ### Query (online + SaaS EC2 only) ```sql SELECT id::text FROM workspaces WHERE status = 'online' AND instance_id IS NOT NULL AND instance_id != '' AND COALESCE(runtime, '') <> 'external' ORDER BY updated_at DESC LIMIT 200 ``` `runtime='external'` rows are owned by the remote-heartbeat pass; paused/hibernated/removed/provisioning/awaiting_agent are excluded by the status filter. ## Guardrails - **Fail-safe**: `IsRunning` returns `(true, err)` on any transient DB/transport error and `(false, nil)` **only** when CP genuinely reports not-running. The reconciler acts strictly on `(false, nil)`; any error short-circuits to "leave it online" so a CP blip never cascades healthy workspaces into reprovision. Covered by `TestReconcileOnce_TransientError_DoesNotFlip`. - **Online + SaaS only** (above). - Per-cycle `LIMIT 200` + per-workspace 10s timeout so one slow CP call can't stall the sweep. - nil-checker / nil-DB tolerant (matches the sibling CP sweeper). ## Wiring Gated identically to `cp-orphan-sweeper` in `cmd/server/main.go`, reusing the **same** `onWorkspaceOffline` closure: ```go if cpProv != nil { go supervised.RunWithRecover(ctx, "cp-instance-reconciler", func(c context.Context) { registry.StartCPInstanceReconciler(c, cpProv, onWorkspaceOffline, 60*time.Second) }) } ``` ## Tests `cp_instance_reconciler_test.go` (sqlmock + fake checker), mirroring `cp_orphan_sweeper_test.go`: not-running→flip, running→no-flip, **transient-error→no-flip (fail-safe)**, query-scope excludes external/non-online, mixed batch, query-error, nil-DB, nil-checker disabled, runs-once-and-exits-on-cancel. `go build ./...`, `go vet ./internal/registry/...`, `go test ./internal/registry/...` all green; gofmt clean on touched files only. Refs core#2261, core#2247. DO NOT MERGE — heavy core SOP gate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hongming added 1 commit 2026-06-05 00:21:45 +00:00

feat(registry): reconcile online workspaces against real EC2 state — auto-heal terminated instances (core#2261) ... 48aebdfcc4

Root cause (core#2247): every existing liveness sweep keys off a PROXY
(Redis TTL, agent heartbeat, local Docker, or runtime='external'). A SaaS
claude-code workspace whose EC2 was terminated/stopped falls through ALL
of them and stays status=online pointing at a dead instance_id forever.

Adds StartCPInstanceReconciler: a 60s sweep that asks the ONE
authoritative question the others lack — CPProvisioner.IsRunning (CP
DescribeInstances-equivalent) — for each online SaaS row, and on a clean
"not running" feeds it into the existing onWorkspaceOffline closure
(status flip + RestartByID reprovision, existing volume).

Guardrails: fail-safe (IsRunning is (true, err) on any transient error →
never flip); online + SaaS-EC2 only (runtime <> 'external'); per-cycle
LIMIT 200 + per-workspace timeout.

Refs core#2261, core#2247.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

hongming added the tier:medium label 2026-06-05 00:23:28 +00:00

core-qa approved these changes 2026-06-05 00:23:28 +00:00

core-qa left a comment

Member

Copy Link

Copy Source

QA (core#2261 instance reconciler). Independently ran the 9 registry tests — all pass incl. the fail-safe (transient IsRunning err → no flip) and query-scope (online+SaaS only, excludes external/non-online). Logic verified: acts ONLY on (false,nil), per-workspace timeout, LIMIT 200, reuses onWorkspaceOffline auto-heal. Approve.

QA (core#2261 instance reconciler). Independently ran the 9 registry tests — all pass incl. the fail-safe (transient IsRunning err → no flip) and query-scope (online+SaaS only, excludes external/non-online). Logic verified: acts ONLY on (false,nil), per-workspace timeout, LIMIT 200, reuses onWorkspaceOffline auto-heal. Approve.

core-security approved these changes 2026-06-05 00:23:30 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Security (core#2261). No new external surface or creds. Auto-heal reuses the existing onWorkspaceOffline→RestartByID path (existing-volume reprovision); fail-safe IsRunning prevents flipping a healthy workspace on transient errors; scope strictly online+SaaS (excludes paused/hibernated/removed). DoS-safe: per-cycle LIMIT + per-workspace timeout. Approve.

Security (core#2261). No new external surface or creds. Auto-heal reuses the existing onWorkspaceOffline→RestartByID path (existing-volume reprovision); fail-safe IsRunning prevents flipping a healthy workspace on transient errors; scope strictly online+SaaS (excludes paused/hibernated/removed). DoS-safe: per-cycle LIMIT + per-workspace timeout. Approve.

hongming merged commit 8812285932 into main 2026-06-05 00:28:32 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

core-qa

core-security

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label tier:medium

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2266