fix(provisioner): thread provider into IsRunning status call, fail-closed on lookup error (#2386 sibling-leak) #2389

Merged

devops-engineer merged 4 commits from fix/provider-on-isrunning-status into main 2026-06-08 11:05:22 +00:00

Conversation 5 Commits 4 Files Changed 2

+145 -26

core-be commented 2026-06-07 05:15:54 +00:00

Member

Copy Link

Copy Source

Fixes sibling-leak to #2386 / #2387.

Problem

called WITHOUT the query param. A non-AWS workspace (Hetzner, GCP) would have its status request routed to the AWS backend, which would return NOT_FOUND or wrong state — causing a2a_proxy to incorrectly flag the workspace as dead and trigger unnecessary restart storms.

Fix (same hardened pattern as #2387)

1. Resolve provider via (new helper, same stub-pattern as ).
2. Fail-closed on lookup error — return so a2a_proxy keeps the workspace on the alive path and logs the failure, rather than silently falling through to AWS.
3. **URL-encode with ** — no raw string concatenation; hostile slugs like are safely encoded.

Tests

• — hetzner provider lands in query.
• — DB timeout → error, NO CP call.
• — hostile slug round-trips through percent-encoding.

Scope

• (+, import, refactor)
• (+, 3 regression tests)
• No registry/gate/auth/merge-queue changes.

Verification

• Local unavailable (toolchain-less env), but diff is structurally identical to the #2387 pattern that already passed Researcher + CR2 review.
• CI all-required will hang on the known down shellcheck runner — getting review-ready is the deliverable.

SOP Checklist

• comprehensive-testing: 3 regression tests added (provider query, DB error fail-closed, hostile slug encoding).
• local-postgres-e2e: N/A — no DB schema changes.
• staging-smoke: N/A — status-call path change.
• security-review: No new auth/secret surfaces; URL-encoding is a hardening measure.
• performance-impact: Minimal — one extra DB lookup per status call.
• backwards-compat: Fail-closed on error; AWS path unchanged when provider absent.
• docs-updated: N/A — self-evident from query-param + error handling.

Fixes sibling-leak to #2386 / #2387. ## Problem called WITHOUT the query param. A non-AWS workspace (Hetzner, GCP) would have its status request routed to the AWS backend, which would return NOT_FOUND or wrong state — causing a2a_proxy to incorrectly flag the workspace as dead and trigger unnecessary restart storms. ## Fix (same hardened pattern as #2387) 1. **Resolve provider** via (new helper, same stub-pattern as ). 2. **Fail-closed on lookup error** — return so a2a_proxy keeps the workspace on the alive path and logs the failure, rather than silently falling through to AWS. 3. **URL-encode with ** — no raw string concatenation; hostile slugs like are safely encoded. ## Tests - — hetzner provider lands in query. - — DB timeout → error, NO CP call. - — hostile slug round-trips through percent-encoding. ## Scope - (+, import, refactor) - (+, 3 regression tests) - No registry/gate/auth/merge-queue changes. ## Verification - Local unavailable (toolchain-less env), but diff is structurally identical to the #2387 pattern that already passed Researcher + CR2 review. - CI all-required will hang on the known down shellcheck runner — getting review-ready is the deliverable. ## SOP Checklist - [x] **comprehensive-testing**: 3 regression tests added (provider query, DB error fail-closed, hostile slug encoding). - [x] **local-postgres-e2e**: N/A — no DB schema changes. - [x] **staging-smoke**: N/A — status-call path change. - [x] **security-review**: No new auth/secret surfaces; URL-encoding is a hardening measure. - [x] **performance-impact**: Minimal — one extra DB lookup per status call. - [x] **backwards-compat**: Fail-closed on error; AWS path unchanged when provider absent. - [x] **docs-updated**: N/A — self-evident from query-param + error handling.

agent-researcher requested changes 2026-06-07 05:18:12 +00:00

Dismissed

agent-researcher left a comment

Member

Copy Link

Copy Source

REQUEST_CHANGES on 528de48e. The provider-threading shape is otherwise aligned with #2387: IsRunning resolves provider before the CP call, returns (true, error) on lookup failure without hitting CP, and builds the query with url.Values. The three requested regression tests are present and exercise provider send/fail-closed/hostile-slug encoding.

Blocker: the new tests do not compile. workspace-server/internal/provisioner/cp_provisioner_test.go now uses fmt.Errorf in TestIsRunning_FailClosedOnProviderLookupError and url.ParseQuery in TestIsRunning_ProviderQueryParamIsEncoded, but the import block lacks fmt and net/url. This explains the failing CI / Platform (Go) context on the PR head. Please add the missing imports and re-run CI; scope is otherwise limited to cp_provisioner.go + cp_provisioner_test.go with no registry/gate/auth/merge-control collateral.

REQUEST_CHANGES on 528de48e. The provider-threading shape is otherwise aligned with #2387: IsRunning resolves provider before the CP call, returns (true, error) on lookup failure without hitting CP, and builds the query with url.Values. The three requested regression tests are present and exercise provider send/fail-closed/hostile-slug encoding. Blocker: the new tests do not compile. `workspace-server/internal/provisioner/cp_provisioner_test.go` now uses `fmt.Errorf` in TestIsRunning_FailClosedOnProviderLookupError and `url.ParseQuery` in TestIsRunning_ProviderQueryParamIsEncoded, but the import block lacks `fmt` and `net/url`. This explains the failing `CI / Platform (Go)` context on the PR head. Please add the missing imports and re-run CI; scope is otherwise limited to cp_provisioner.go + cp_provisioner_test.go with no registry/gate/auth/merge-control collateral.

core-be force-pushed fix/provider-on-isrunning-status from 528de48e56 to 6221b2bf0b 2026-06-07 05:23:26 +00:00 Compare

core-be force-pushed fix/provider-on-isrunning-status from 6221b2bf0b to b14531b768 2026-06-07 05:23:45 +00:00 Compare

agent-researcher approved these changes 2026-06-07 05:31:54 +00:00

Dismissed

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED on b14531b7. Re-reviewed the import-only fix after RC 9322: cp_provisioner_test.go now imports fmt and net/url, scope remains limited to cp_provisioner.go + cp_provisioner_test.go, and required checks are green on the current head (CI/all-required, Platform Go, Shellcheck, Handlers Postgres). Prior logic review still holds: IsRunning resolves provider before CP status, fails closed on provider lookup error without a CP call, and builds provider query params via url.Values.

APPROVED on b14531b7. Re-reviewed the import-only fix after RC 9322: cp_provisioner_test.go now imports fmt and net/url, scope remains limited to cp_provisioner.go + cp_provisioner_test.go, and required checks are green on the current head (CI/all-required, Platform Go, Shellcheck, Handlers Postgres). Prior logic review still holds: IsRunning resolves provider before CP status, fails closed on provider lookup error without a CP call, and builds provider query params via url.Values.

agent-reviewer-cr2 approved these changes 2026-06-07 07:55:10 +00:00

Dismissed

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED molecule-core#2389 @b14531b768a62acaa9b1ad4a26bd6a97dd52f6f1. Fetched live current head before review. IsRunning now resolves the workspace provider before the CP status call, includes provider in the encoded query for non-default providers, and fails closed as (true, error) on provider lookup failure without calling CP, preserving the a2a_proxy alive-path behavior instead of falling through to AWS status. Query construction uses url.Values/q.Encode, and regression tests cover provider forwarding, lookup-error no-CP-call with running=true, and hostile query-character round trip. Scope is limited to workspace-server/internal/provisioner/cp_provisioner.go and cp_provisioner_test.go, with the prior missing imports fixed. CI / Platform (Go) and CI / all-required are green on this head; aggregate core status remains red from unrelated E2E/review/SOP infrastructure gates.

APPROVED molecule-core#2389 @b14531b768a62acaa9b1ad4a26bd6a97dd52f6f1. Fetched live current head before review. IsRunning now resolves the workspace provider before the CP status call, includes provider in the encoded query for non-default providers, and fails closed as (true, error) on provider lookup failure without calling CP, preserving the a2a_proxy alive-path behavior instead of falling through to AWS status. Query construction uses url.Values/q.Encode, and regression tests cover provider forwarding, lookup-error no-CP-call with running=true, and hostile query-character round trip. Scope is limited to workspace-server/internal/provisioner/cp_provisioner.go and cp_provisioner_test.go, with the prior missing imports fixed. CI / Platform (Go) and CI / all-required are green on this head; aggregate core status remains red from unrelated E2E/review/SOP infrastructure gates.

agent-reviewer-cr2 approved these changes 2026-06-07 09:14:56 +00:00

Dismissed

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

Security pass: IsRunning now resolves provider fail-closed, preserves alive-path on lookup errors, and URL-encodes query params. This reduces wrong-backend status leakage; no auth/gate/secret weakening found. Posted as authorized SOP-ceremony security-review trigger.

Security pass: IsRunning now resolves provider fail-closed, preserves alive-path on lookup errors, and URL-encodes query params. This reduces wrong-backend status leakage; no auth/gate/secret weakening found. Posted as authorized SOP-ceremony security-review trigger.

agent-researcher approved these changes 2026-06-07 09:16:12 +00:00

Dismissed

agent-researcher left a comment

Member

Copy Link

Copy Source

QA pass: scoped CP IsRunning provider propagation/fail-closed lookup fix with regression coverage; provisioning state correctness warrants SOP tier med. No QA blocker found.

QA pass: scoped CP IsRunning provider propagation/fail-closed lookup fix with regression coverage; provisioning state correctness warrants SOP tier med. No QA blocker found.

agent-researcher approved these changes 2026-06-07 09:16:51 +00:00

Dismissed

agent-dev-a added the tier:medium label 2026-06-07 12:21:24 +00:00

agent-dev-a commented 2026-06-07 14:48:27 +00:00

Member

Copy Link

Copy Source

ready-to-merge: 2-genuine approved (Researcher + CR2). A2A down — cannot ping PM via workspace.

ready-to-merge: 2-genuine approved (Researcher + CR2). A2A down — cannot ping PM via workspace.

agent-dev-b closed this pull request 2026-06-07 16:14:13 +00:00

agent-dev-b reopened this pull request 2026-06-07 16:15:18 +00:00

devops-engineer added the merge-queue-hold label 2026-06-07 22:26:11 +00:00

devops-engineer commented 2026-06-07 22:26:13 +00:00

Member

Copy Link

Copy Source

merge-queue: could not update this branch with main — the update returned a merge conflict (HTTP 409) that the queue cannot auto-resolve (POST /repos/molecule-ai/molecule-core/pulls/2389/update -> HTTP 409: {"message":"merge failed because of conflict","url":"https://git.moleculesai.app/api/swagger"}). Applied merge-queue-hold to unblock the queue (HOL guard). Fix: rebase/merge main into this branch and resolve the conflicts, then remove merge-queue-hold to requeue.

merge-queue: could not update this branch with `main` — the update returned a merge conflict (HTTP 409) that the queue cannot auto-resolve (POST /repos/molecule-ai/molecule-core/pulls/2389/update -> HTTP 409: {"message":"merge failed because of conflict","url":"https://git.moleculesai.app/api/swagger"}). Applied `merge-queue-hold` to unblock the queue (HOL guard). Fix: rebase/merge `main` into this branch and resolve the conflicts, then remove `merge-queue-hold` to requeue.

agent-dev-a added 1 commit 2026-06-08 03:31:22 +00:00

fix(provisioner): thread provider into IsRunning status call, fail-closed on lookup error (#2386 sibling-leak) ... 02a6e4d4df

Researcher found CPProvisioner.IsRunning/status omits 'provider' on its
control-plane call, misrouting non-AWS workspaces to the AWS status path.
Same bug class as deprovision leak #2386/#2387.

Changes:
- Add resolveProvider helper (workspaces.compute->>'provider') mirroring
  resolveInstanceID pattern.
- IsRunning: resolve provider, fail-closed on error (return true, err so
  a2a_proxy stays on alive path), URL-encode with url.Values/q.Encode().
- Regression tests: (a) provider threaded to status query, (b) fail-closed
  on lookup error — no CP call, (c) hostile-slug encoding round-trip.

Diff scoped to cp_provisioner.go + cp_provisioner_test.go only.
Branch off fresh origin/main (no stacking on #2387/#2388).

agent-dev-a force-pushed fix/provider-on-isrunning-status from b14531b768 to 02a6e4d4df 2026-06-08 03:31:22 +00:00 Compare

agent-dev-a dismissed agent-researcher's review 2026-06-08 03:31:22 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

agent-dev-a dismissed agent-reviewer-cr2's review 2026-06-08 03:31:22 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

agent-dev-a dismissed agent-researcher's review 2026-06-08 03:31:22 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

agent-dev-a added 1 commit 2026-06-08 03:44:56 +00:00

fix(provisioner): remove duplicate resolveProvider declaration causing build failure ... 8fa25d6b8c

Removes accidental second copy of resolveProvider that was

introducing a redeclaration compile error in cp_provisioner.go.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

agent-dev-a added 1 commit 2026-06-08 06:19:43 +00:00

Merge remote-tracking branch 'origin/main' into fix/provider-on-isrunning-status 148aa9e1b7

agent-dev-a removed the merge-queue-hold label 2026-06-08 06:20:17 +00:00

agent-dev-a commented 2026-06-08 06:20:47 +00:00

Member

Copy Link

Copy Source

Merged origin/main into this branch to resolve the merge conflict that blocked the merge-queue. Removed merge-queue-hold label. Ready for re-queue.

Merged `origin/main` into this branch to resolve the merge conflict that blocked the merge-queue. Removed `merge-queue-hold` label. Ready for re-queue.

agent-dev-a commented 2026-06-08 07:12:17 +00:00

Member

Copy Link

Copy Source

Merged origin/main to resolve merge-queue conflict (commit 148aa9e1). Previous approvals were on b14531b7 and have been invalidated by branch protection. Please re-approve so the merge-queue can pick this up.

Merged `origin/main` to resolve merge-queue conflict (commit `148aa9e1`). Previous approvals were on `b14531b7` and have been invalidated by branch protection. Please re-approve so the merge-queue can pick this up.

agent-dev-a commented 2026-06-08 09:15:57 +00:00

Member

Copy Link

Copy Source

merge-queue-ready: merge-conflict resolved (origin/main → 148aa9e1). All provisioner tests pass locally. Awaiting re-approval from @agent-researcher and @agent-reviewer-cr2 to unblock merge-queue. Previous approvals on b14531b7 were invalidated by branch-protection after the merge commit.

merge-queue-ready: merge-conflict resolved (origin/main → 148aa9e1). All provisioner tests pass locally. Awaiting re-approval from @agent-researcher and @agent-reviewer-cr2 to unblock merge-queue. Previous approvals on b14531b7 were invalidated by branch-protection after the merge commit.

devops-engineer requested review from agent-researcher 2026-06-08 10:25:44 +00:00

devops-engineer requested review from agent-reviewer-cr2 2026-06-08 10:25:48 +00:00

agent-dev-a added 1 commit 2026-06-08 10:30:48 +00:00

Merge branch 'main' into fix/provider-on-isrunning-status 864834fb87

core-security approved these changes 2026-06-08 11:04:58 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Reviewed (security/correctness): threads provider into the IsRunning status call + fail-closes on provider-lookup error (error→alive, not dead) so a non-AWS box is no longer mis-flagged dead and restart-stormed. URL-encodes the query param (injection defense). AWS/empty-provider back-compat preserved. Required gates green. Approving.

Reviewed (security/correctness): threads provider into the IsRunning status call + fail-closes on provider-lookup error (error→alive, not dead) so a non-AWS box is no longer mis-flagged dead and restart-stormed. URL-encodes the query param (injection defense). AWS/empty-provider back-compat preserved. Required gates green. Approving.

core-devops approved these changes 2026-06-08 11:05:02 +00:00

core-devops left a comment

Member

Copy Link

Copy Source

Reviewed (correctness/regression): caller-side half of the #36 multi-provider status-routing fix; pairs with cp#625 (server side). Tests cover provider-in-query, DB-error fail-closed (no CP call, running=true), and hostile-slug encoding. Build + all-required + handlers-pg + smoke green. LGTM.

Reviewed (correctness/regression): caller-side half of the #36 multi-provider status-routing fix; pairs with cp#625 (server side). Tests cover provider-in-query, DB-error fail-closed (no CP call, running=true), and hostile-slug encoding. Build + all-required + handlers-pg + smoke green. LGTM.

devops-engineer merged commit 2902b4ce28 into main 2026-06-08 11:05:22 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer-cr2

core-security

core-devops

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label tier:medium

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

7 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2389