fix(platform-agent): pin LLM_PROVIDER=platform when concierge MODEL is empty #3160

Merged

devops-engineer merged 1 commits from fix/concierge-provider-empty-model into main 2026-06-22 17:27:57 +00:00

Conversation 0 Commits 1 Files Changed 2

+66 -4

hongming commented 2026-06-22 16:47:53 +00:00

Owner

Copy Link

Copy Source

Root cause

Newly-created platform-agent orgs (e.g. test9) 401 on every LLM call ("Incorrect API key"). The concierge's claude-code sends an inherited tenant CLAUDE_CODE_OAUTH_TOKEN (sk-ant-oat01) as the bearer to the CP LLM proxy, which authenticates callers against the per-workspace org_instances.admin_token — an OAuth bearer can never match it -> 401.

That OAuth token should have been stripped because the concierge pins LLM_PROVIDER=platform. But ensureConciergeProvider gated the pin on MODEL having the platform-managed prefix; on a rebuilt-from-DB provision payload MODEL is empty, so HasPrefix("", ...) is false and the pin was skipped -> no LLM_PROVIDER -> the runtime could not drop the inherited OAuth token.

Fix

Treat an empty MODEL as platform-managed (pin platform). Empty is never a BYOK signal — BYOK concierges carry a stored LLM_PROVIDER (early-return) or an explicit non-platform MODEL (still skipped). Only the unresolved/empty case now pins.

Tests

TestEnsureConciergeProvider_EmptyModelPins: empty MODEL pins platform; explicit BYOK model still skips. go test green, go vet clean.

Paired with the runtime defense-in-depth (un-gated OAuth drop when base URL is the CP proxy): branch fix/llm-auth-drop-oauth-on-cp-proxy in molecule-ai-workspace-runtime.

🤖 Generated with Claude Code

## Root cause Newly-created platform-agent orgs (e.g. test9) 401 on every LLM call ("Incorrect API key"). The concierge's claude-code sends an inherited tenant `CLAUDE_CODE_OAUTH_TOKEN` (sk-ant-oat01) as the bearer to the CP LLM proxy, which authenticates callers against the per-workspace `org_instances.admin_token` — an OAuth bearer can never match it -> 401. That OAuth token should have been stripped because the concierge pins `LLM_PROVIDER=platform`. But `ensureConciergeProvider` gated the pin on `MODEL` having the platform-managed prefix; on a rebuilt-from-DB provision payload `MODEL` is empty, so `HasPrefix("", ...)` is false and the pin was skipped -> no `LLM_PROVIDER` -> the runtime could not drop the inherited OAuth token. ## Fix Treat an empty `MODEL` as platform-managed (pin `platform`). Empty is never a BYOK signal — BYOK concierges carry a stored `LLM_PROVIDER` (early-return) or an explicit non-platform `MODEL` (still skipped). Only the unresolved/empty case now pins. ## Tests `TestEnsureConciergeProvider_EmptyModelPins`: empty MODEL pins platform; explicit BYOK model still skips. go test green, go vet clean. Paired with the runtime defense-in-depth (un-gated OAuth drop when base URL is the CP proxy): branch `fix/llm-auth-drop-oauth-on-cp-proxy` in molecule-ai-workspace-runtime. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hongming added 1 commit 2026-06-22 16:47:54 +00:00

fix(platform-agent): pin LLM_PROVIDER=platform when concierge MODEL is empty ... 4b7fb6ecfc

ensureConciergeProvider gated the LLM_PROVIDER=platform pin on the effective MODEL
having the platform-managed prefix. On a rebuilt-from-DB provision payload the
MODEL is empty, so HasPrefix("", ...) was false and the pin was skipped — the
concierge booted without LLM_PROVIDER. The runtime then could not drop the
inherited tenant CLAUDE_CODE_OAUTH_TOKEN, so claude-code sent the OAuth bearer to
the CP LLM proxy (which authenticates via the per-workspace admin_token) and 401'd
every call. Root cause of the test9 / platform-agent "Incorrect API key" outage.

Treat an empty MODEL as platform-managed (pin platform): empty is never a BYOK
signal — BYOK concierges carry a stored LLM_PROVIDER (early-return) or an explicit
non-platform MODEL (still skipped). Only the unresolved/empty case now pins.

Regression test: empty MODEL pins platform; explicit BYOK model still skips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

molecule-code-reviewer approved these changes 2026-06-22 16:57:29 +00:00

molecule-code-reviewer left a comment

Member

Copy Link

Copy Source

Reviewed the gate change directly: !HasPrefix(model,prefix) -> model != "" && !HasPrefix(...) correctly pins platform for an empty/unresolved MODEL (the rebuilt-from-DB concierge case that 401'd). BYOK safe: stored LLM_PROVIDER early-returns above the gate (verified in source), non-empty non-platform MODEL still skips; both covered by the new regression test. APPROVE (code). Note: PR is not green yet — qa-review/security-review/gate-check-v3/sop-checklist + E2E Staging Boot need resolving before merge.

Reviewed the gate change directly: `!HasPrefix(model,prefix)` -> `model != "" && !HasPrefix(...)` correctly pins platform for an empty/unresolved MODEL (the rebuilt-from-DB concierge case that 401'd). BYOK safe: stored LLM_PROVIDER early-returns above the gate (verified in source), non-empty non-platform MODEL still skips; both covered by the new regression test. APPROVE (code). Note: PR is not green yet — qa-review/security-review/gate-check-v3/sop-checklist + E2E Staging Boot need resolving before merge.

core-security approved these changes 2026-06-22 16:57:37 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Independent adversarial pass (security focus): central claims verified in SOURCE, not just tests — the stored-LLM_PROVIDER early-return precedes the changed gate, so no BYOK regression from the empty-MODEL pin. Two pre-existing, non-blocking follow-ups noted for tracking: (1) readStoredProviderSecret fail-open could clobber a BYOK provider on decrypt-failure+empty-MODEL; (2) an inherited OAuth token arriving via ANTHROPIC_AUTH_TOKEN (not CLAUDE_CODE_OAUTH_TOKEN) under CP-proxy routing would re-route to native Anthropic (silent-billing shape) — neither introduced by this PR. APPROVE.

Independent adversarial pass (security focus): central claims verified in SOURCE, not just tests — the stored-LLM_PROVIDER early-return precedes the changed gate, so no BYOK regression from the empty-MODEL pin. Two pre-existing, non-blocking follow-ups noted for tracking: (1) readStoredProviderSecret fail-open could clobber a BYOK provider on decrypt-failure+empty-MODEL; (2) an inherited OAuth token arriving via ANTHROPIC_AUTH_TOKEN (not CLAUDE_CODE_OAUTH_TOKEN) under CP-proxy routing would re-route to native Anthropic (silent-billing shape) — neither introduced by this PR. APPROVE.

agent-researcher approved these changes 2026-06-22 17:26:28 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED — genuine 5-axis review on current head 4b7fb6ecfc.

Correctness: the platform pin now fires for the failing rebuilt-from-DB/empty MODEL path while preserving the existing stored LLM_PROVIDER early-return and the explicit non-platform MODEL skip, so legitimate BYOK/self-host model overrides are not forced to platform. Security/token precedence: this restores the provider pin needed for runtime auth stripping and avoids the inherited Claude OAuth token winning against the CP LLM proxy. Regression risk: narrow one-condition change in ensureConciergeProvider; no production logic outside concierge provider seeding. Tests: direct unit coverage proves empty MODEL pins and explicit BYOK model does not. Blast radius: platform-agent concierge provisioning only; appropriate for the live test9/cd62fe70 401 incident.

APPROVED — genuine 5-axis review on current head 4b7fb6ecfcab99d870ae9ce99922a4465f482d39. Correctness: the platform pin now fires for the failing rebuilt-from-DB/empty MODEL path while preserving the existing stored LLM_PROVIDER early-return and the explicit non-platform MODEL skip, so legitimate BYOK/self-host model overrides are not forced to platform. Security/token precedence: this restores the provider pin needed for runtime auth stripping and avoids the inherited Claude OAuth token winning against the CP LLM proxy. Regression risk: narrow one-condition change in ensureConciergeProvider; no production logic outside concierge provider seeding. Tests: direct unit coverage proves empty MODEL pins and explicit BYOK model does not. Blast radius: platform-agent concierge provisioning only; appropriate for the live test9/cd62fe70 401 incident.

agent-reviewer-cr2 approved these changes 2026-06-22 17:27:20 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED current-head incident review.

5-axis check:

• Correctness: empty MODEL for the platform concierge now pins LLM_PROVIDER=platform instead of returning early, closing the CP LLM proxy 401 path. Explicit non-empty non-platform/BYOK models still skip the pin.
• Robustness/regression: respects existing stored provider early-return and persists the platform pin for rebuild/fresh payloads; no broad behavior change outside concierge provider seeding.
• Security: restores admin-token precedence by ensuring runtime can classify platform traffic; no secret values added to logs.
• Performance: no new hot-loop or blocking behavior beyond the existing single provider-secret check/persist path.
• Readability/tests: direct regression test covers empty MODEL pin and explicit BYOK non-platform skip.

APPROVED current-head incident review. 5-axis check: - Correctness: empty MODEL for the platform concierge now pins LLM_PROVIDER=platform instead of returning early, closing the CP LLM proxy 401 path. Explicit non-empty non-platform/BYOK models still skip the pin. - Robustness/regression: respects existing stored provider early-return and persists the platform pin for rebuild/fresh payloads; no broad behavior change outside concierge provider seeding. - Security: restores admin-token precedence by ensuring runtime can classify platform traffic; no secret values added to logs. - Performance: no new hot-loop or blocking behavior beyond the existing single provider-secret check/persist path. - Readability/tests: direct regression test covers empty MODEL pin and explicit BYOK non-platform skip.

devops-engineer merged commit 9589d5373f into main 2026-06-22 17:27:57 +00:00

hongming referenced this pull request 2026-06-22 19:02:26 +00:00

readStoredProviderSecret fails open on decrypt error -> BYOK concierge can be mis-pinned to platform #3162

Sign in to join this conversation.

Reviewers

No Reviewers

molecule-code-reviewer

core-security

agent-researcher

agent-reviewer-cr2

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

5 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3160