test(provision): SSOT-parametrized + real-boot regression for moonshot/kimi NOT_CONFIGURED #2197

Merged

claude-ceo-assistant merged 1 commits from test/provider-matrix-boot-regression-moonshot into main 2026-06-04 04:22:08 +00:00

Conversation 1 Commits 1 Files Changed 5

+420 -3

core-devops commented 2026-06-04 02:52:56 +00:00

Member

Copy Link

Copy Source

Why

The moonshot/kimi incident: a canvas-created claude-code workspace with provider=Platform + model=moonshot/kimi-k2.6 booted NOT_CONFIGURED in production. ensureDefaultConfig generated a config.yaml that lacked the manifest-derived provider: key, so the cp#329 config-bundle the adapter actually reads left molecule-runtime to slash-split moonshot/... → unregistered provider. Fixed by #2187 (Fix A: ensureDefaultConfig stamps DeriveProvider→provider: platform) + #2188 (Fix C: canvas). Unit tests passed; the real boot path was the gap. This PR adds comprehensive regression coverage so the class cannot reship — it does not change production code.

---

1. Current coverage map (what existed, what was missing)

Layer	Suite	Provisions a workspace?	Asserts boots ONLINE (not NOT_CONFIGURED)?	Covers claude-code + platform + moonshot/kimi-k2.6?	Merge-blocking on core main?
Config-gen unit	TestEnsureDefaultConfig_StampsDerivedProvider (#2187)	No	No (asserts config bytes)	Single hardcoded combo only	Yes (CI / Platform (Go), blocking)
Config-gen unit	model_registry_validation_test.go (DeriveProvider)	No	No	Partially (asserts derive, not config emission)	Yes
Staging SaaS (bash)	tests/e2e/test_staging_full_saas.sh via e2e-staging-saas.yml	Yes (real EC2)	Yes (wait_workspaces_online_routable + completion)	No — pick_model_slug only picks BYOK ids (MiniMax-M2 / claude-sonnet-4-6 / sonnet); never the platform arm	No (continue-on-error: true, mc#1982 mask)
Staging Canvas (Playwright)	e2e-staging-canvas.yml	Yes	Tab-render only	No	No (continue-on-error: true)
Serving gate	internal/servinge2e (controlplane repo)	No (raw completion vs provider upstream)	No (LLM serving, not workspace boot)	N/A (per-provider serving, not boot)	ci/serving-e2e in CP, not core
Staging Go harness	internal/staginge2e (controlplane repo)	Yes (tenant, not workspace)	Tunnel/reachability	No	No

The gap, precisely: No suite provisioned a real workspace and asserted ONLINE for the platform-managed path. The single deterministic pin covered exactly one hardcoded combo and was not SSOT-driven, so a newly-offered platform model that failed to derive provider: platform would ship uncaught — the same offered-but-not-stamped divergence the original bug rode in on.

---

2. The regression test added

Deterministic (no live infra — runs in the normal unit suite, mutation-verified)

workspace-server/internal/handlers/workspace_provision_platform_boot_test.go

• TestEnsureDefaultConfig_StampsProviderForEverySSOTPlatformModel — enumerates the claude-code platform arm directly from the providers SSOT (providers.LoadManifest, the same manifest the config generator derives against) and asserts ensureDefaultConfig stamps provider: platform at both top-level and runtime_config for every offered platform model (today: 7 — anthropic/claude-opus-4-7, anthropic/claude-sonnet-4-6, moonshot/kimi-k2.6, moonshot/kimi-k2.5, minimax/MiniMax-M2.7, minimax/MiniMax-M2.7-highspeed, minimax/MiniMax-M3). Add a platform model → it gets a case for free and only passes if actually stamped. A headline sentinel asserts moonshot/kimi-k2.6 stays in the set.
• TestPlatformModelDeriveProvider_SSOTConsistency — the upstream half: DeriveProvider maps every SSOT platform model to provider Name == "platform", so a derive-layer regression fails closer to root cause.
• Mutation-verified: disabling the top-level stamp in workspace_provision.go makes the suite FAIL (proven locally, then reverted) — not a vacuous green.

Real-boot staging variant (I will run it against staging)

Extends the existing staging harness rather than adding a new one:

• tests/e2e/lib/model_slug.sh — new E2E_LLM_PATH=platform path selects the platform model (default moonshot/kimi-k2.6), precedence over BYOK key branches, still overridable by E2E_MODEL_SLUG.
• tests/e2e/test_staging_full_saas.sh — platform branch sends empty secrets (platform-managed needs no tenant key); the workspace must boot purely on the CP-proxy env + Fix A's stamped provider. Reuses the harness's existing wait_workspaces_online_routable (status=online, NOT not_configured) + completion assertions — keys off the real artifact.
• tests/e2e/test_model_slug.sh — 4 new pinned cases (16/16 green locally).
• .gitea/workflows/e2e-staging-saas.yml — new E2E Staging Platform Boot job: E2E_RUNTIME=claude-code E2E_LLM_PATH=platform E2E_MODE=smoke, no LLM key, own teardown safety-net; added providers.yaml + model_slug.sh to the path triggers.

Run it (operator host / CI with staging creds):

STAGING creds + MOLECULE_ADMIN_TOKEN set, then:
E2E_RUNTIME=claude-code E2E_LLM_PATH=platform E2E_MODE=smoke \
  bash tests/e2e/test_staging_full_saas.sh

---

3. Gate-making plan (make the comprehensive suites merge-blocking)

Already blocking (no action): the deterministic suite rides CI / Platform (Go) (continue-on-error: false) — the two new tests block on merge immediately.

To make the real-boot staging gate blocking — de-flake FIRST, then flip:

Workflow	Job → context to add as required	Current state	Action
e2e-staging-saas.yml	E2E Staging Platform Boot (the new job)	continue-on-error: true, bp-required: pending #2187	After 3 consecutive green runs on main: remove continue-on-error: true, add CI/... E2E Staging Platform Boot (push) (and (pull_request)) to branch protection, flip directive to bp-required: yes.
e2e-staging-saas.yml	E2E Staging SaaS (existing BYOK)	continue-on-error: true (mc#1982 mask)	Same de-flake-then-flip; separate follow-up.

De-flake prerequisites (do NOT gate on flake): this path shares the known cp#245 boot-timeout flake surface (stale-ECR-digest 30-min boot-death misread as flake — project_runtime_image_pin_stale_digest_root_cause) and the E2E Staging SaaS flake noted in reference_flaky_e2e_staging_saas. Confirm a fresh runtime image pin + 3 clean runs before flipping. The continue-on-error: true masks here are tracked under mc#1982 ("root-fix and remove, do not renew silently") — this PR adds one new mask with a tracked flip plan rather than leaving the gate silent.

No new flake introduced: the deterministic suite is pure/offline/deterministic and can gate today.

---

Self-test (no live infra)

• go build ./... ✓ · go vet ./internal/handlers ./internal/providers ✓
• Deterministic regression suite: PASS (7 platform models × 2 tests), mutation-verified to fail when the stamp is removed
• gofmt -l clean · shellcheck clean (all 3 bash files) · workflow YAML parses
• pick_model_slug bash unit tests: 16/16 PASS
• lint_required_context_exists_in_bp.find_directive_for_job returns ('required-pending','2187') for the new job ✓

Needs your staging run: the E2E Staging Platform Boot job (real EC2 + online + completion 200).

🤖 Generated with Claude Code

## Why The moonshot/kimi incident: a canvas-created **claude-code** workspace with **provider=Platform + model=`moonshot/kimi-k2.6`** booted **NOT_CONFIGURED** in production. `ensureDefaultConfig` generated a `config.yaml` that lacked the manifest-derived `provider:` key, so the cp#329 config-bundle the adapter actually reads left molecule-runtime to slash-split `moonshot/...` → unregistered provider. Fixed by **#2187** (Fix A: `ensureDefaultConfig` stamps `DeriveProvider`→`provider: platform`) + **#2188** (Fix C: canvas). **Unit tests passed; the real boot path was the gap.** This PR adds comprehensive regression coverage so the *class* cannot reship — it does **not** change production code. --- ## 1. Current coverage map (what existed, what was missing) | Layer | Suite | Provisions a workspace? | Asserts boots ONLINE (not NOT_CONFIGURED)? | Covers claude-code + **platform** + `moonshot/kimi-k2.6`? | Merge-blocking on core `main`? | |---|---|---|---|---|---| | Config-gen unit | `TestEnsureDefaultConfig_StampsDerivedProvider` (#2187) | No | No (asserts config bytes) | **Single hardcoded combo only** | Yes (`CI / Platform (Go)`, blocking) | | Config-gen unit | `model_registry_validation_test.go` (DeriveProvider) | No | No | Partially (asserts derive, not config emission) | Yes | | Staging SaaS (bash) | `tests/e2e/test_staging_full_saas.sh` via `e2e-staging-saas.yml` | **Yes** (real EC2) | **Yes** (`wait_workspaces_online_routable` + completion) | **No** — `pick_model_slug` only picks **BYOK** ids (`MiniMax-M2` / `claude-sonnet-4-6` / `sonnet`); never the platform arm | **No** (`continue-on-error: true`, mc#1982 mask) | | Staging Canvas (Playwright) | `e2e-staging-canvas.yml` | Yes | Tab-render only | No | No (`continue-on-error: true`) | | Serving gate | `internal/servinge2e` (**controlplane** repo) | No (raw completion vs provider upstream) | No (LLM serving, not workspace boot) | N/A (per-provider serving, not boot) | `ci/serving-e2e` in CP, not core | | Staging Go harness | `internal/staginge2e` (**controlplane** repo) | Yes (tenant, not workspace) | Tunnel/reachability | No | No | **The gap, precisely:** *No* suite provisioned a real workspace and asserted ONLINE for the **platform-managed** path. The single deterministic pin covered exactly one hardcoded combo and was not SSOT-driven, so a *newly-offered* platform model that failed to derive `provider: platform` would ship uncaught — the same offered-but-not-stamped divergence the original bug rode in on. --- ## 2. The regression test added ### Deterministic (no live infra — runs in the normal unit suite, mutation-verified) `workspace-server/internal/handlers/workspace_provision_platform_boot_test.go` - **`TestEnsureDefaultConfig_StampsProviderForEverySSOTPlatformModel`** — enumerates the claude-code `platform` arm directly from the providers SSOT (`providers.LoadManifest`, the *same* manifest the config generator derives against) and asserts `ensureDefaultConfig` stamps `provider: platform` at **both** top-level and `runtime_config` for **every** offered platform model (today: 7 — `anthropic/claude-opus-4-7`, `anthropic/claude-sonnet-4-6`, `moonshot/kimi-k2.6`, `moonshot/kimi-k2.5`, `minimax/MiniMax-M2.7`, `minimax/MiniMax-M2.7-highspeed`, `minimax/MiniMax-M3`). Add a platform model → it gets a case for free and only passes if actually stamped. A headline sentinel asserts `moonshot/kimi-k2.6` stays in the set. - **`TestPlatformModelDeriveProvider_SSOTConsistency`** — the upstream half: `DeriveProvider` maps every SSOT platform model to provider `Name == "platform"`, so a derive-layer regression fails closer to root cause. - **Mutation-verified:** disabling the top-level stamp in `workspace_provision.go` makes the suite FAIL (proven locally, then reverted) — not a vacuous green. ### Real-boot staging variant (I will run it against staging) Extends the **existing** staging harness rather than adding a new one: - `tests/e2e/lib/model_slug.sh` — new `E2E_LLM_PATH=platform` path selects the platform model (default `moonshot/kimi-k2.6`), precedence over BYOK key branches, still overridable by `E2E_MODEL_SLUG`. - `tests/e2e/test_staging_full_saas.sh` — platform branch sends **empty** secrets (platform-managed needs no tenant key); the workspace must boot purely on the CP-proxy env + Fix A's stamped provider. Reuses the harness's existing `wait_workspaces_online_routable` (status=online, NOT not_configured) + completion assertions — keys off the **real artifact**. - `tests/e2e/test_model_slug.sh` — 4 new pinned cases (16/16 green locally). - `.gitea/workflows/e2e-staging-saas.yml` — new **`E2E Staging Platform Boot`** job: `E2E_RUNTIME=claude-code E2E_LLM_PATH=platform E2E_MODE=smoke`, no LLM key, own teardown safety-net; added `providers.yaml` + `model_slug.sh` to the path triggers. **Run it (operator host / CI with staging creds):** ``` STAGING creds + MOLECULE_ADMIN_TOKEN set, then: E2E_RUNTIME=claude-code E2E_LLM_PATH=platform E2E_MODE=smoke \ bash tests/e2e/test_staging_full_saas.sh ``` --- ## 3. Gate-making plan (make the comprehensive suites merge-blocking) **Already blocking (no action):** the deterministic suite rides `CI / Platform (Go)` (`continue-on-error: false`) — the two new tests block on merge immediately. **To make the real-boot staging gate blocking — de-flake FIRST, then flip:** | Workflow | Job → context to add as required | Current state | Action | |---|---|---|---| | `e2e-staging-saas.yml` | **`E2E Staging Platform Boot` (the new job)** | `continue-on-error: true`, `bp-required: pending #2187` | After **3 consecutive green** runs on `main`: remove `continue-on-error: true`, add `CI/... E2E Staging Platform Boot (push)` (and `(pull_request)`) to branch protection, flip directive to `bp-required: yes`. | | `e2e-staging-saas.yml` | `E2E Staging SaaS` (existing BYOK) | `continue-on-error: true` (mc#1982 mask) | Same de-flake-then-flip; separate follow-up. | **De-flake prerequisites (do NOT gate on flake):** this path shares the known **cp#245 boot-timeout** flake surface (stale-ECR-digest 30-min boot-death misread as flake — `project_runtime_image_pin_stale_digest_root_cause`) and the `E2E Staging SaaS` flake noted in `reference_flaky_e2e_staging_saas`. Confirm a fresh runtime image pin + 3 clean runs before flipping. The `continue-on-error: true` masks here are tracked under **mc#1982** ("root-fix and remove, do not renew silently") — this PR adds one new mask *with* a tracked flip plan rather than leaving the gate silent. **No new flake introduced:** the deterministic suite is pure/offline/deterministic and can gate today. --- ## Self-test (no live infra) - `go build ./...` ✓ · `go vet ./internal/handlers ./internal/providers` ✓ - Deterministic regression suite: **PASS** (7 platform models × 2 tests), **mutation-verified** to fail when the stamp is removed - `gofmt -l` clean · `shellcheck` clean (all 3 bash files) · workflow YAML parses - `pick_model_slug` bash unit tests: **16/16 PASS** - `lint_required_context_exists_in_bp.find_directive_for_job` returns `('required-pending','2187')` for the new job ✓ **Needs your staging run:** the `E2E Staging Platform Boot` job (real EC2 + online + completion 200). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-04 02:52:58 +00:00

test(provision): SSOT-parametrized + real-boot regression for moonshot/kimi NOT_CONFIGURED ... 9c506d5c8c

The moonshot/kimi incident: a canvas-created claude-code workspace with
provider=Platform + model=moonshot/kimi-k2.6 booted NOT_CONFIGURED in prod
because the generated config.yaml lacked the manifest-derived `provider:`
key, so the adapter slash-split "moonshot/..." -> unregistered provider.
Fixed by #2187 (ensureDefaultConfig stamps DeriveProvider->provider:platform)
+ #2188 (canvas). Unit tests passed; the REAL boot path was the gap.

This adds comprehensive regression coverage so the CLASS cannot reship:

Deterministic (no live infra, runs in the normal unit suite):
  workspace-server/internal/handlers/workspace_provision_platform_boot_test.go
  - TestEnsureDefaultConfig_StampsProviderForEverySSOTPlatformModel:
    enumerates the claude-code `platform` arm from the providers SSOT
    (providers.LoadManifest) and asserts ensureDefaultConfig stamps
    provider:platform (top-level AND runtime_config) for EVERY offered
    platform model — not just the single moonshot/kimi pin #2187 shipped.
    A newly-offered platform model gets a case for free and only passes if
    actually stamped (closes the offered-but-not-stamped divergence the bug
    rode in on). Mutation-verified: disabling the stamp fails the test.
  - TestPlatformModelDeriveProvider_SSOTConsistency: the upstream half —
    DeriveProvider maps every SSOT platform model to provider Name "platform".

Real-boot (staging; I will run it):
  Extends the existing staging harness (no new harness) with a
  platform-managed path: E2E_LLM_PATH=platform pin-selects moonshot/kimi-k2.6,
  sends NO tenant key, and reuses the harness's online-wait + completion
  assertions to prove the workspace reaches status=online (not
  not_configured) and a completion returns 200. The BYOK branches never
  exercised the platform arm — the exact arm the bug shipped on.
  - tests/e2e/lib/model_slug.sh: platform path + override semantics
  - tests/e2e/test_model_slug.sh: 4 new pinned cases (16/16 green)
  - tests/e2e/test_staging_full_saas.sh: empty-secrets platform branch
  - .gitea/workflows/e2e-staging-saas.yml: new `E2E Staging Platform Boot`
    job (continue-on-error during de-flake; bp-required: pending #2187),
    + providers.yaml/model_slug.sh added to the path triggers.

Coverage-audit theme: mc#1982 (continue-on-error masks; de-flake-then-gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

core-devops referenced this pull request 2026-06-04 04:17:37 +00:00

test(e2e): name the A2A empty-completion failure class in staging SaaS canary #2203

claude-ceo-assistant merged commit f503559b06 into main 2026-06-04 04:22:08 +00:00

core-devops commented 2026-06-04 04:22:10 +00:00

Author

Member

Copy Link

Copy Source

Owner force-merged (honest bypass). RFC#340 coverage: SSOT-driven provider-matrix boot regression — deterministic test (all 7 claude-code platform models stamp provider:platform, mutation-verified, gates via CI/Platform Go) + real-boot staging job. All 3 REQUIRED contexts green. The E2E Staging SaaS (full lifecycle) red is NON-required + pre-existing (a staging-only A2A empty-content issue on reasoning models, identical on main scheduled run, NOT #2197-caused; prod LLM serving verified healthy — kimi+minimax return real content). Token revoked.

Owner force-merged (honest bypass). RFC#340 coverage: SSOT-driven provider-matrix boot regression — deterministic test (all 7 claude-code platform models stamp provider:platform, mutation-verified, gates via CI/Platform Go) + real-boot staging job. All 3 REQUIRED contexts green. The E2E Staging SaaS (full lifecycle) red is NON-required + pre-existing (a staging-only A2A empty-content issue on reasoning models, identical on main scheduled run, NOT #2197-caused; prod LLM serving verified healthy — kimi+minimax return real content). Token revoked.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2197