molecule-ai/molecule-core
52
0
Fork 2
Code Issues 178 Pull Requests 36 Actions 2 Packages Projects Releases Wiki Activity

[bug] Orphan-workspace leak: auto_reap_at stamped but never enforced (detector is dry-run-only) #2642

New Issue
Closed
opened 2026-06-12 11:24:40 +00:00 by claude-ceo-assistant · 1 comment
claude-ceo-assistant commented 2026-06-12 11:24:40 +00:00
Owner
Copy Link
Copy Source

Orphan-workspace leak — root cause + fix (2026-06-12)

ROOT CAUSE: the CP stamps every workspace/tenant EC2 with an auto_reap_at kill-deadline tag, but NOTHING enforced it. aws-orphan-ec2-detector.sh is DRY-RUN/emit-only (its header literally says "never terminates" — it only emits a Loki metric that pings a human cleanup workflow nobody ran). The intended enforcer, staging-tenant-reap (operator-config#96), is not running on the operator (no cron/timer/script). Result: 31 of 32 running boxes were OVERDUE (oldest 8 days, ~$1.5k/mo bleed).

RESOLVED: terminated 25 confirmed-dead e2e/test orphans (overdue + aged/>6h-past-deadline). Held 3 ambiguous (tenant-cp455, 2x tenant-cncrg-dbg) — past-deadline but possibly intentional debug tenants → need owner confirm.

PREVENT (operator stopgap, AWS-only): deployed /usr/local/bin/auto-reap-enforcer.sh + cron molecule-auto-reap-enforcer (every 2h, --apply). Enforces auto_reap_at with 6h grace, scoped to ws-tenant-(e2e|gcp-test|hz) names, skips no_reap/keep=true, honors /etc/molecule-bootstrap/auto-reap-enforcer.disabled, logs to /var/log/auto-reap-enforcer.log.

PROPER DURABLE FIX (this issue tracks it): the enforcement belongs in the CONTROL PLANE (it knows live-vs-orphaned + covers Hetzner/GCP too, which the operator AWS stopgap does not). Needs: (1) a CP reconciler that terminates its own workspaces past auto_reap_at across all providers; (2) restore/replace operator-config#96; (3) an e2e regression gate (ties to #2615). The dry-run detector stays as the observability backstop.

**Orphan-workspace leak — root cause + fix (2026-06-12)** ROOT CAUSE: the CP stamps every workspace/tenant EC2 with an `auto_reap_at` kill-deadline tag, but NOTHING enforced it. `aws-orphan-ec2-detector.sh` is **DRY-RUN/emit-only** (its header literally says "never terminates" — it only emits a Loki metric that pings a human cleanup workflow nobody ran). The intended enforcer, **staging-tenant-reap (operator-config#96)**, is not running on the operator (no cron/timer/script). Result: 31 of 32 running boxes were OVERDUE (oldest 8 days, ~$1.5k/mo bleed). RESOLVED: terminated 25 confirmed-dead e2e/test orphans (overdue + aged/>6h-past-deadline). Held 3 ambiguous (`tenant-cp455`, 2x `tenant-cncrg-dbg`) — past-deadline but possibly intentional debug tenants → need owner confirm. PREVENT (operator stopgap, AWS-only): deployed `/usr/local/bin/auto-reap-enforcer.sh` + cron `molecule-auto-reap-enforcer` (every 2h, --apply). Enforces auto_reap_at with 6h grace, scoped to `ws-tenant-(e2e|gcp-test|hz)` names, skips `no_reap`/`keep`=true, honors `/etc/molecule-bootstrap/auto-reap-enforcer.disabled`, logs to /var/log/auto-reap-enforcer.log. PROPER DURABLE FIX (this issue tracks it): the enforcement belongs in the CONTROL PLANE (it knows live-vs-orphaned + covers Hetzner/GCP too, which the operator AWS stopgap does not). Needs: (1) a CP reconciler that terminates its own workspaces past auto_reap_at across all providers; (2) restore/replace operator-config#96; (3) an e2e regression gate (ties to #2615). The dry-run detector stays as the observability backstop.
claude-ceo-assistant commented 2026-06-12 15:43:43 +00:00
Author
Owner
Copy Link
Copy Source

Durable cross-provider fix merged: molecule-controlplane#748 (AutoReapEnforcer across AWS/Hetzner/GCP, 6h grace, no_reap/keep opt-out, live-row cross-check, suspended-org guard restored, e2e-gated via internal/staginge2e + internal/sweep tests). Also root-caused why operator-config#96 silently died (cron-glob mismatch + binary never installed). Operator AWS-only stopgap remains as belt-and-suspenders. Closing.

Durable cross-provider fix merged: molecule-controlplane#748 (AutoReapEnforcer across AWS/Hetzner/GCP, 6h grace, no_reap/keep opt-out, live-row cross-check, suspended-org guard restored, e2e-gated via internal/staginge2e + internal/sweep tests). Also root-caused why operator-config#96 silently died (cron-glob mismatch + binary never installed). Operator AWS-only stopgap remains as belt-and-suspenders. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2642
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?