molecule-ai/molecule-core
52
0
Fork 2
Code Issues 179 Pull Requests 49 Actions Packages Projects Releases Wiki Activity

PR #3029: CP orphan sweeper closes deprovision split-write race #2053

Closed
core-be wants to merge 0 commits from pr-3029 into staging
pull from: pr-3029
merge into: molecule-ai:staging
Conversation 0 Commits - Files Changed -
core-be commented 2026-06-01 03:37:09 +00:00
Member
Copy Link
Copy Source
No description provided.
core-be added 1 commit 2026-06-01 03:37:09 +00:00
fix(workspace-server): CP orphan sweeper closes deprovision split-write race (#2989)
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Details
Harness Replays / Harness Replays (pull_request) Blocked by required conditions
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Blocked by required conditions
Details
sop-checklist / review-refire (pull_request) Waiting to run
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 14s
Details
branch-protection drift check / Branch protection drift (pull_request) Successful in 4s
Details
cascade-list-drift-gate / check (pull_request) Successful in 5s
Details
Check merge_group trigger on required workflows / Required workflows have merge_group trigger (pull_request) Successful in 4s
Details
Check migration collisions / Migration version collision check (pull_request) Successful in 9s
Details
CodeQL / Analyze (${{ matrix.language }}) (go) (pull_request) Failing after 2s
Details
CodeQL / Analyze (${{ matrix.language }}) (javascript-typescript) (pull_request) Failing after 3s
Details
CodeQL / Analyze (${{ matrix.language }}) (python) (pull_request) Failing after 4s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 7s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 10s
Details
Harness Replays / detect-changes (pull_request) Successful in 5s
Details
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 5s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 10s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Failing after 28s
Details
qa-review / approved (pull_request) Successful in 5s
Details
security-review / approved (pull_request) Successful in 3s
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 41s
Details
Runtime Pin Compatibility / PyPI-latest install + import smoke (pull_request) Successful in 1m30s
Details
gate-check-v3 / gate-check (pull_request) Successful in 3s
Details
sop-checklist / all-items-acked (pull_request) Successful in 4s
Details
sop-tier-check / tier-check (pull_request) Successful in 6s
Details
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
Details
CI / Platform (Go) (pull_request) Has been cancelled
Details
CI / Canvas (Next.js) (pull_request) Has been cancelled
Details
CI / Python Lint & Test (pull_request) Has been cancelled
Details
CI / Detect changes (pull_request) Successful in 8s
Details
CI / Shellcheck (E2E scripts) (pull_request) Has been cancelled
Details
091a24dd70 
The deprovision path marks `workspaces.status='removed'` BEFORE calling
the controlplane DELETE. If that CP call fails (transient 5xx, network
hiccup, AWS provider error), the DB row stays at 'removed' with
`instance_id` populated and there's no retry — the EC2 lives forever.
9 prod orphans accumulated over 3 days under this bug.

Adds a SaaS-mode counterpart to the existing Docker `orphan_sweeper`:
- 60s tick (matches the Docker sweeper cadence)
- LIMIT 100 per cycle so a sustained CP outage drains over multiple
  cycles without blowing the request timeout
- Re-issues `cpProv.Stop` for any workspace at status='removed' with a
  non-NULL `instance_id`. Stop is idempotent (AWS terminate on
  already-terminated is a no-op; CP's Deprovision tolerates already-
  deleted DNS) so retries are safe.
- On Stop success, NULLs `instance_id` so the next cycle skips the row.
- On Stop failure, leaves `instance_id` populated for next cycle.

The existing Docker sweeper is gated on `prov != nil`; the new sweeper
is gated on `cpProv != nil`. SaaS tenants get exactly one of the two,
self-hosted tenants get the Docker one — no overlap.

Why this shape over option A (CP-first ordering) or B (durable outbox):
the existing inline path already returns a loud 500 to the user when
CP fails — the only missing piece is automatic retry, which a 60s
sweeper provides without protocol changes, new tables, or new workers.
~30 LOC of production code vs. ~400 for an outbox. RFC discussion in
#2989 comment chain.

Tests:
- 9 unit tests covering happy path, Stop failure, UPDATE failure,
  multiple orphans (one-fails-others-still-process), DB query error,
  nil-DB defense, nil-reaper short-circuit, and the boot-immediate-then-
  tick cadence contract.
- Mutation-tested: status='running' substitution and removed-UPDATE-
  block both fail at least one test.

Out of scope:
- Backfilling the 9 named orphans — they'll heal automatically on the
  first sweep cycle after this lands; no manual cleanup needed.
- Long-term durable-outbox architecture — separate RFC.
core-be changed target branch from main to staging 2026-06-01 03:40:33 +00:00
core-be force-pushed pr-3029 from 091a24dd70 to 7e130daef2 2026-06-01 04:27:26 +00:00 Compare
core-be closed this pull request 2026-06-01 04:36:12 +00:00
Some required checks failed
Block internal-flavored paths / Block forbidden paths (push) Successful in 7s
Details
CI / Detect changes (push) Successful in 14s
Details
CI / Shellcheck (E2E scripts) (push) Successful in 26s
Details
E2E API Smoke Test / detect-changes (push) Successful in 23s
Details
E2E Chat / detect-changes (push) Successful in 17s
Details
Handlers Postgres Integration / detect-changes (push) Successful in 7s
Details
Runtime PR-Built Compatibility / detect-changes (push) Successful in 21s
Details
Secret scan / Scan diff for credential-shaped strings (push) Successful in 19s
Details
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 2s
Details
CI / Platform (Go) (push) Failing after 5m59s
Details
CI / all-required (push) Failing after 4m21s
Details
E2E Chat / E2E Chat (push) Successful in 4s
Details
CI / Canvas (Next.js) (push) Successful in 7m12s
Details
CI / Python Lint & Test (push) Successful in 7m17s
Details
CI / Canvas Deploy Reminder (push) Has been cancelled
Details
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m49s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 2m56s
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Waiting to run
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Waiting to run
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Waiting to run
Details
gate-check-v3 / gate-check (pull_request) Waiting to run
Details
qa-review / approved (pull_request) Waiting to run
Details
security-review / approved (pull_request) Waiting to run
Details
sop-tier-check / tier-check (pull_request) Waiting to run
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
audit-force-merge / audit (pull_request) Waiting to run
Details
publish-runtime-autobump / bump-and-tag (pull_request) Waiting to run
Details
cascade-list-drift-gate / check (pull_request) Waiting to run
Details
Check migration collisions / Migration version collision check (pull_request) Waiting to run
Details
MCP Stdio Transport Regression / MCP stdio with regular-file stdout (pull_request) Waiting to run
Details
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (pull_request) Waiting to run
Details
E2E Staging External Runtime / E2E Staging External Runtime (pull_request) Waiting to run
Details
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Waiting to run
Details
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Waiting to run
Details
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Waiting to run
Details
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Waiting to run
Details
lint-mask-pr-atomicity / lint-mask-pr-atomicity (pull_request) Waiting to run
Details
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Waiting to run
Details
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Waiting to run
Details
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Waiting to run
Details
publish-runtime-autobump / pr-validate (pull_request) Waiting to run
Details
review-check-tests / review-check.sh regression tests (pull_request) Waiting to run
Details
Runtime Pin Compatibility / PyPI-latest install + import smoke (pull_request) Waiting to run
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Waiting to run
Details
sop-checklist / all-items-acked (pull_request) Waiting to run
Required
Details
sop-checklist / review-refire (pull_request) Waiting to run
Details
E2E API Smoke Test / detect-changes (pull_request) Has been cancelled
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Has been cancelled
Details
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
Details
CI / Detect changes (pull_request) Has been cancelled
Details
CI / Canvas (Next.js) (pull_request) Has been cancelled
Details
CI / Platform (Go) (pull_request) Has been cancelled
Details
CI / Shellcheck (E2E scripts) (pull_request) Has been cancelled
Details
CI / Python Lint & Test (pull_request) Has been cancelled
Details
CI / all-required (pull_request) Has been cancelled
Required
Details
E2E Chat / E2E Chat (pull_request) Has been cancelled
Details
E2E Chat / detect-changes (pull_request) Has been cancelled
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Has been cancelled
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Has been cancelled
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Has been cancelled
Details
Handlers Postgres Integration / detect-changes (pull_request) Has been cancelled
Details
Harness Replays / detect-changes (pull_request) Has been cancelled
Details
Harness Replays / Harness Replays (pull_request) Has been cancelled
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Has been cancelled
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2053
Delete Branch "pr-3029"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?