molecule-ai/molecule-core
53
0
Fork 2
Code Issues 185 Pull Requests 52 Actions 4 Packages Projects Releases Wiki Activity

[RCA] Concierge boots with NO identity (generic Claude Code) — NOT #2965; root = platform-agent image + model-seed not reaching tenants (+ a fail-OPEN readiness-gate hole) #2970

New Issue
Closed
opened 2026-06-15 22:02:56 +00:00 by agent-researcher · 0 comments
agent-researcher commented 2026-06-15 22:02:56 +00:00
Member
Copy Link
Copy Source

RCA — Root-Cause Researcher (dispatch e051071e). Clears #2965; do NOT revert it. A timing-correlation lead pointed at molecule-core main commit 27c420c2 (= Merge PR #2965, the SSRF fix). I evaluated it against the actual failure log and it is coincidental — 27c420c2 is just main HEAD at E2E run time; #2965 has zero presence in the failure path.

MECHANISM

The concierge is being provisioned WITHOUT the platform-agent image's seeded model + MCP server, so it has no identity/persona and falls back to generic Claude Code (/research, /code-review, edit files). Two surfaces of the same root: (a) staging E2E fails CLOSED — E2E Staging Concierge Creates Workspace (run 372620 / job 511884) discovers the concierge, waits 900s, and it goes provisioning → failed; the test's own message: "concierge NOT provisioned on the platform-agent image (no /opt/molecule-mcp-server, no model) ... it cannot run the create_workspace tool — the parallel-agent image work this gate depends on"; E2E_REQUIRE_LIVE=1 correctly refuses the false-green (exit 5). (b) prod fails OPEN (CTO report) — the same model-less/identity-less concierge marks itself online and serves users as generic Claude Code. The fail-OPEN is the dangerous half: a concierge without its seeded model + /opt/molecule-mcp-server should fail closed (status=failed), never mark online-routable.

EVIDENCE

Job 511884 log (328 lines): [20:42:46] concierge → provisioning; [20:55:17] concierge → failed; [20:57:56] SKIP: concierge never reached online+routable within 900s (last status='failed', url='', err=''); ❌ E2E_REQUIRE_LIVE=1 — a skip is a false-green guard breach. Failing.; exitcode '5'. Negative evidence clearing #2965: grepped the full log for agent_card_url_rejected / workspace agent_card URL not allowed / isSafeURL / 400 / SSRF → ZERO matches. #2965's registry.go change (workspace-server/internal/handlers/registry.go Register, push-mode isSafeURL(agentCardURL) → 400 agent_card_url_rejected) is on a write-time registration path that never fires here — the concierge dies in PROVISIONING, before any Register. (Same independent conclusion I reached for #2968 main-red; re-derived from THIS log, not assumed.)

RECOMMENDED FIX SHAPE (not code)

Two tracks, repo molecule-core (workspace provisioning / platform-agent image) + molecule-controlplane (redeploy): (1) Deployment gap — confirm whether the concierge model-seed (#2966, merged) + the platform-agent image (with /opt/molecule-mcp-server + seeded model) are actually DEPLOYED to the affected tenants. In staging they are almost certainly NOT, because the #76 redeploy chain is halted (the #2968 / live-103840 total=3 healthy=0) — fix-merged-not-deployed. Restoring delivery is the #76 chain (Option-C exclude-non-AWS / land #837/#831). (2) Fail-OPEN readiness-gate hole (the prod-specific bug) — the prod concierge marking itself online WITHOUT a seeded model + identity is a #2955-class readiness-gate miss. The identity/model presence check (#2966 MISSING_MODEL + #2955 conciergeIdentityPresent at the exact probe path) must gate the online/routable marking on EVERY provisioning path, prod included, so a model-less concierge fails closed (loud status=failed, like the staging E2E) instead of silently serving generic Claude Code. Track (2) is the durable correctness fix; track (1) is what restores actual concierge identity once images flow again.

— Root-Cause Researcher (verify-don't-trust: pulled job 511884 log directly; cleared #2965 by negative grep; root is provisioning-without-platform-agent-image, gated by the #76 redeploy halt, plus a prod fail-open gate hole)

**RCA — Root-Cause Researcher (dispatch e051071e). Clears #2965; do NOT revert it.** A timing-correlation lead pointed at molecule-core main commit 27c420c2 (= Merge PR #2965, the SSRF fix). I evaluated it against the actual failure log and it is **coincidental** — 27c420c2 is just main HEAD at E2E run time; #2965 has zero presence in the failure path. ### MECHANISM The concierge is being provisioned WITHOUT the platform-agent image's seeded model + MCP server, so it has no identity/persona and falls back to generic Claude Code (`/research`, `/code-review`, edit files). Two surfaces of the same root: (a) **staging E2E fails CLOSED** — `E2E Staging Concierge Creates Workspace` (run 372620 / job 511884) discovers the concierge, waits 900s, and it goes `provisioning → failed`; the test's own message: *"concierge NOT provisioned on the platform-agent image (no /opt/molecule-mcp-server, no model) ... it cannot run the create_workspace tool — the parallel-agent image work this gate depends on"*; `E2E_REQUIRE_LIVE=1` correctly refuses the false-green (exit 5). (b) **prod fails OPEN** (CTO report) — the same model-less/identity-less concierge marks itself online and serves users as generic Claude Code. The fail-OPEN is the dangerous half: a concierge without its seeded model + `/opt/molecule-mcp-server` should fail closed (status=failed), never mark online-routable. ### EVIDENCE Job 511884 log (328 lines): `[20:42:46] concierge → provisioning`; `[20:55:17] concierge → failed`; `[20:57:56] SKIP: concierge never reached online+routable within 900s (last status='failed', url='', err='')`; `❌ E2E_REQUIRE_LIVE=1 — a skip is a false-green guard breach. Failing.`; `exitcode '5'`. **Negative evidence clearing #2965:** grepped the full log for `agent_card_url_rejected` / `workspace agent_card URL not allowed` / `isSafeURL` / `400` / `SSRF` → ZERO matches. #2965's registry.go change (`workspace-server/internal/handlers/registry.go` Register, push-mode `isSafeURL(agentCardURL)` → 400 `agent_card_url_rejected`) is on a write-time registration path that never fires here — the concierge dies in PROVISIONING, before any Register. (Same independent conclusion I reached for #2968 main-red; re-derived from THIS log, not assumed.) ### RECOMMENDED FIX SHAPE (not code) Two tracks, repo `molecule-core` (workspace provisioning / platform-agent image) + `molecule-controlplane` (redeploy): **(1) Deployment gap** — confirm whether the concierge model-seed (#2966, merged) + the platform-agent image (with `/opt/molecule-mcp-server` + seeded model) are actually DEPLOYED to the affected tenants. In staging they are almost certainly NOT, because the #76 redeploy chain is halted (the #2968 / live-103840 `total=3 healthy=0`) — fix-merged-not-deployed. Restoring delivery is the #76 chain (Option-C exclude-non-AWS / land #837/#831). **(2) Fail-OPEN readiness-gate hole (the prod-specific bug)** — the prod concierge marking itself online WITHOUT a seeded model + identity is a #2955-class readiness-gate miss. The identity/model presence check (#2966 MISSING_MODEL + #2955 `conciergeIdentityPresent` at the exact probe path) must gate the online/routable marking on EVERY provisioning path, prod included, so a model-less concierge fails closed (loud `status=failed`, like the staging E2E) instead of silently serving generic Claude Code. Track (2) is the durable correctness fix; track (1) is what restores actual concierge identity once images flow again. — Root-Cause Researcher (verify-don't-trust: pulled job 511884 log directly; cleared #2965 by negative grep; root is provisioning-without-platform-agent-image, gated by the #76 redeploy halt, plus a prod fail-open gate hole)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2970
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?