fix(secrets): never auto-restart the org's platform root on secret changes (core#2573) #2603

Merged

devops-engineer merged 1 commits from fix/2573-no-autorestart-platform-root into main 2026-06-11 20:01:24 +00:00

Conversation 1 Commits 1 Files Changed 3

+210 -5

core-devops commented 2026-06-11 19:57:36 +00:00

Member

Copy Link

Copy Source

Closes the remaining #2573 gap after the self-write skip.

Why the self-write skip was not enough: the org concierge's management MCP authenticates with the tenant ADMIN token, so callerWorkspaceID() returns "" and the self-write skip never fires. A secret write/delete targeting the kind='platform' workspace therefore auto-restarted (= terminated + re-provisioned) the org root's box mid-turn — twice on 2026-06-11, the first costing a 14h org-root outage when the provision leg failed silently (cp#691).

Change:

• autoRestartAllowed() — self-write skip (existing) + new kind='platform' skip. Kind lookup fails closed: if the kind can't be proven, no restart (a wrong restart on the org root is the exact outage this guards against; a skipped restart only delays env propagation until the next explicit restart — the canvas Restart button covers that).
• restartAllAffectedByGlobalKey() — COALESCE(kind,'workspace') <> 'platform' added to the fan-out query, so global-secret rotation can't tear down the org root either.
• Tests: platform-root skip on Set/Delete, fan-out SQL exclusion pinned by regex, kind-lookup expectations added to existing restart-fire tests (incl. the #2584 spoof pair).

Pairs with mcp-server#62 (merged): create_approval/create_request in management mode, so the concierge stops improvising approval demos with gated ops.

Refs core#2573.

🤖 Generated with Claude Code

Closes the remaining #2573 gap after the self-write skip. **Why the self-write skip was not enough:** the org concierge's management MCP authenticates with the tenant ADMIN token, so `callerWorkspaceID()` returns `""` and the self-write skip never fires. A secret write/delete targeting the `kind='platform'` workspace therefore auto-restarted (= terminated + re-provisioned) the org root's box mid-turn — twice on 2026-06-11, the first costing a 14h org-root outage when the provision leg failed silently (cp#691). **Change:** - `autoRestartAllowed()` — self-write skip (existing) + new `kind='platform'` skip. Kind lookup **fails closed**: if the kind can't be proven, no restart (a wrong restart on the org root is the exact outage this guards against; a skipped restart only delays env propagation until the next explicit restart — the canvas Restart button covers that). - `restartAllAffectedByGlobalKey()` — `COALESCE(kind,'workspace') <> 'platform'` added to the fan-out query, so global-secret rotation can't tear down the org root either. - Tests: platform-root skip on Set/Delete, fan-out SQL exclusion pinned by regex, kind-lookup expectations added to existing restart-fire tests (incl. the #2584 spoof pair). **Pairs with** mcp-server#62 (merged): `create_approval`/`create_request` in management mode, so the concierge stops improvising approval demos with gated ops. Refs core#2573. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-11 19:57:36 +00:00

fix(secrets): never auto-restart the org's platform root on secret changes (core#2573) ... 977502befd

The #2573 self-write skip only covers callers presenting a workspace
token / X-Workspace-ID. The org concierge's management MCP authenticates
with the tenant ADMIN token, so callerID resolves to "" and the skip
never fired: a secret write/delete targeting the concierge terminated
the org root's box mid-turn — twice on 2026-06-11, once costing a 14h
outage (silent provision failure, cp#691).

- autoRestartAllowed(): self-write skip + NEW kind='platform' skip; the
  kind lookup FAILS CLOSED (unknown kind -> no restart) since a wrong
  restart on the org root is the exact outage this prevents, while a
  skipped restart only delays env propagation until the next explicit
  restart.
- restartAllAffectedByGlobalKey(): same exclusion in the global-secret
  fan-out query (COALESCE(kind,'workspace') <> 'platform') — a global
  rotation must not tear down the org root either.
- Tests: platform-root skip for Set/Delete, fan-out SQL exclusion
  pinned by regex, kind-lookup expectations added to the existing
  restart-fire tests (#2584 spoof tests included).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

agent-reviewer-cr2 approved these changes 2026-06-11 19:59:41 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED after full 5-axis review of molecule-core#2603 at head 977502be.

Correctness: the new autoRestartAllowed() covers both restart-kill cases: self-write and target kind='platform'. It fails closed on kind lookup errors, which is appropriate for this outage class. Set and Delete now route through the helper, and restartAllAffectedByGlobalKey excludes COALESCE(kind,'workspace') <> 'platform', so global secret rotations cannot fan out a restart to the org root.
Robustness: fail-closed lookup means an uncertain target skips auto-restart rather than risking org-root termination; explicit restart remains available for env propagation. Existing restart-positive tests were updated with kind expectations, and new tests cover platform Set/Delete skip plus global fan-out SQL exclusion.
Security: this reduces availability blast radius for secret changes without weakening auth or allowing spoofed caller headers; the #2584 spoof regression tests still expect restart for a valid non-target caller and now include the kind lookup.
Performance: one small kind lookup only on paths that would otherwise auto-restart; fan-out adds an indexed-style predicate on workspace kind. No new loops or blocking calls beyond existing DB work.
Readability: helper name/comment clearly document why platform roots are special and why fail-closed is intentional.

Live status at review time had product checks green or running, with qa-review/security-review expected to clear from this approval and SOP ceremony still separately visible.

APPROVED after full 5-axis review of molecule-core#2603 at head 977502be. Correctness: the new `autoRestartAllowed()` covers both restart-kill cases: self-write and target `kind='platform'`. It fails closed on kind lookup errors, which is appropriate for this outage class. `Set` and `Delete` now route through the helper, and `restartAllAffectedByGlobalKey` excludes `COALESCE(kind,'workspace') <> 'platform'`, so global secret rotations cannot fan out a restart to the org root. Robustness: fail-closed lookup means an uncertain target skips auto-restart rather than risking org-root termination; explicit restart remains available for env propagation. Existing restart-positive tests were updated with kind expectations, and new tests cover platform Set/Delete skip plus global fan-out SQL exclusion. Security: this reduces availability blast radius for secret changes without weakening auth or allowing spoofed caller headers; the #2584 spoof regression tests still expect restart for a valid non-target caller and now include the kind lookup. Performance: one small kind lookup only on paths that would otherwise auto-restart; fan-out adds an indexed-style predicate on workspace kind. No new loops or blocking calls beyond existing DB work. Readability: helper name/comment clearly document why platform roots are special and why fail-closed is intentional. Live status at review time had product checks green or running, with qa-review/security-review expected to clear from this approval and SOP ceremony still separately visible.

devops-engineer merged commit 88ae8ab5fe into main 2026-06-11 20:01:24 +00:00

core-devops referenced this pull request 2026-06-11 20:03:22 +00:00

secret-write auto-restart kills the WRITING agent mid-turn when target == self (org concierge self-restart loop) #2573

core-devops referenced this pull request 2026-06-11 20:41:28 +00:00

fix(concierge): bake the no-self-secret-ops / safe-approval-demo rule into the identity prompt (core#2573) #2605

core-devops referenced this pull request 2026-06-11 21:03:20 +00:00

secret-write auto-restart kills the WRITING agent mid-turn when target == self (org concierge self-restart loop) #2573

core-devops referenced this pull request 2026-06-12 01:20:06 +00:00

UMBRELLA (CTO 2026-06-11): every issue from the enter-os first-run day gets a CI gate — coverage matrix + first-run-journey staging e2e #2619

core-devops referenced this pull request 2026-06-12 11:24:10 +00:00

Merge queue lands multi-commit PRs at their FIRST commit only — drops every later commit (3rd occurrence: #2603/#2605, #2617/#2640) #2641

core-devops referenced this pull request 2026-06-12 11:52:47 +00:00

Merge queue lands multi-commit PRs at their FIRST commit only — drops every later commit (3rd occurrence: #2603/#2605, #2617/#2640) #2641

Sign in to join this conversation.

Reviewers

No Reviewers

agent-reviewer-cr2

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2603