molecule-ai/molecule-core
52
0
Fork 2
Code Issues 193 Pull Requests 65 Actions 4 Packages Projects Releases Wiki Activity

POST /workspaces/:id/pause: implicit recursive cascade to descendants — make opt-in via ?cascade=true #1991

New Issue
Closed
opened 2026-05-28 07:06:44 +00:00 by hongming · 0 comments
hongming commented 2026-05-28 07:06:44 +00:00
Owner
Copy Link
Copy Source

Problem

Pause (workspace_restart.go:878) recursively collects all descendants via WITH RECURSIVE descendants SQL and stops them all. The handler's response field paused_count is the only signal to the caller that more than 1 workspace was affected.

Callers expecting Pause to affect only the named workspace will silently disrupt a whole subtree. Combined with the destructive backup-then-terminate path on subsequent Stop+Provision, this amplifies blast radius from 1 -> N.

Confirmed today 2026-05-28 06:50Z

Pause(deedcb61) returned {"paused_count":5} and terminated all 5 EC2-backed agents-team workspaces (Production Manager + 4 descendants in the parent tree). 4 of those were working; all 4 are now in status=failed. See incident internal#722.

Proposed fix

  • Default behavior: Pause/Resume affect ONLY the named workspace. If descendants exist and cascade=true is not set, return 409 with a body listing the descendants so the caller can decide.
  • New query param: ?cascade=true opts into the recursive behavior. ?cascade=false (default) treats the operation as single-workspace.
  • Migration: a deprecation period where the legacy implicit cascade is still allowed but emits a warning header X-Molecule-Implicit-Cascade: deprecated, then flip default after one minor version.

Acceptance criteria

  • Pause without cascade=true on a parent workspace returns 409 + descendants list.
  • Pause?cascade=true behaves as today.
  • Resume mirrors Pause's contract.
  • Test: parent-with-2-children setup; Pause(parent) without cascade returns 409; Pause(parent)?cascade=true returns 200 with paused_count=3.

Cross-refs

  • internal#722 — original incident.
  • The destructive Stop+Provision underneath Pause is tracked separately (linked to /restart fail-closed issue).
## Problem `Pause` (`workspace_restart.go:878`) recursively collects all descendants via `WITH RECURSIVE descendants` SQL and stops them all. The handler's response field `paused_count` is the only signal to the caller that more than 1 workspace was affected. Callers expecting Pause to affect only the named workspace will silently disrupt a whole subtree. Combined with the destructive backup-then-terminate path on subsequent Stop+Provision, this amplifies blast radius from 1 -> N. ## Confirmed today 2026-05-28 06:50Z Pause(deedcb61) returned `{"paused_count":5}` and terminated all 5 EC2-backed agents-team workspaces (Production Manager + 4 descendants in the parent tree). 4 of those were working; all 4 are now in `status=failed`. See incident internal#722. ## Proposed fix - Default behavior: Pause/Resume affect ONLY the named workspace. If descendants exist and `cascade=true` is not set, return 409 with a body listing the descendants so the caller can decide. - New query param: `?cascade=true` opts into the recursive behavior. `?cascade=false` (default) treats the operation as single-workspace. - Migration: a deprecation period where the legacy implicit cascade is still allowed but emits a warning header `X-Molecule-Implicit-Cascade: deprecated`, then flip default after one minor version. ## Acceptance criteria - Pause without cascade=true on a parent workspace returns 409 + descendants list. - Pause?cascade=true behaves as today. - Resume mirrors Pause's contract. - Test: parent-with-2-children setup; Pause(parent) without cascade returns 409; Pause(parent)?cascade=true returns 200 with paused_count=3. ## Cross-refs - internal#722 — original incident. - The destructive Stop+Provision underneath Pause is tracked separately (linked to /restart fail-closed issue).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1991
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?