test(providers): SSOT-driven DeriveProvider routing matrix — every offered runtime×provider gated (keyless, required-lane) #2292

Merged

core-devops merged 1 commits from harden/derive-provider-matrix-e2e into main 2026-06-05 08:18:37 +00:00

Conversation 0 Commits 1 Files Changed 1

+336

core-devops commented 2026-06-05 08:00:05 +00:00

Member

Copy Link

Copy Source

What

SSOT-driven, KEYLESS, REQUIRED-lane-gateable test that asserts every offered (runtime × model/provider arm) in the providers SSOT resolves to the exact correct provider via DeriveProvider — closing the provider-routing-correctness coverage hole for ALL ~29 providers + every BYOK arm without needing any LLM key.

Closes the gap flagged in the regression-coverage audit: many offered (runtime → provider) pairs — hermes's 17 name-only BYOK arms, claude-code's zai/deepseek/xiaomi-mimo, openclaw's byok-openai/byok-minimax/groq/openrouter/custom, codex's byok-minimax, etc. — are pure prefix-routing (resolved by DeriveProvider(runtime, modelId)) and had zero test. A regression in the routing table (wrong provider, dropped arm, bad regex) shipped silently and wedged tenant agents at boot.

Why keyless / why it gates in the required lane

DeriveProvider + ModelPrefixMatch resolve a model id to a provider with no upstream call — a pure function of the merged registry. So the entire offered routing table is gateable in the REQUIRED CI / all-required lane with zero secrets.

Approach — SSOT-driven, not hardcoded

workspace-server/internal/providers/derive_provider_matrix_test.go iterates LoadManifest().Runtimes (the same registry production reads). For every runtime × every offered arm it asserts: (a) DeriveProvider resolves to the exact expected provider (computed from the SSOT), (b) the (runtime, model) is registration-valid (the validateRegisteredModelForRuntime predicate: on the platform menu OR DeriveProvider resolves), (c) no offered id silently misroutes or falls through.

• exact-listed arms: every model id iterated off the SSOT; expected provider computed from native declaration order (first-declared wins the codex/anthropic "one id, two auth arms" shape). A newly-added model is auto-covered.
• name-only arms (zero models — pure prefix BYOK): each probed with a representative BYOK id its regex must own. The matrix requires a representative for every name-only arm in the SSOT — "added an arm, forgot routing/sample" fails RED. A dead representative also fails RED.

Coverage

5 runtimes, 43 (runtime×provider) arms across 29 distinct providers, 53 exact-listed (runtime×model) assertions + 29 name-only BYOK routing probes (82 routing assertions total).

Known-tricky forms pinned explicitly

So a regression names its class, not just a generic cell:

• the #2263/#2274 colon-vs-slash-vs-bare MiniMax triple on claude-code: bare MiniMax-M2.7→minimax, minimax/…→platform, minimax:…→unregistered (adapter can't strip minimax:)
• #2265 openai-namespaced rejected on claude-code (no native openai arm)
• groq:→groq; openclaw minimax:→byok-minimax
• hermes anthropic//gemini//openai:/minimax: → byok-* (NOT platform — cp#529 tenant-key billing safety)
• codex gpt-* no-auth→openai-subscription vs OPENAI_API_KEY→openai-api
• google-adk platform:→platform vs bare gemini-…→google

Watch-it-fail proof

Adding minimax:MiniMax-M2.7 to claude-code's platform arm (pointing the colon BYOK form at platform) reds the matrix naming the exact mismatch:

DeriveProvider("claude-code", "minimax:MiniMax-M2.7", []) = "platform", want an unregistered/unrouteable ERROR

Reverted → green. This proves real coverage of the #2263/#2274 routing class.

Verification

• go test ./internal/providers/ -run 'DeriveProvider|Matrix|Routing' -count=1 — all green
• go build ./... — clean
• go vet ./internal/providers/ — clean

Test-only; additive; no production path changed.

🤖 Generated with Claude Code

## What SSOT-driven, KEYLESS, REQUIRED-lane-gateable test that asserts **every offered (runtime × model/provider arm)** in the providers SSOT resolves to the **exact** correct provider via `DeriveProvider` — closing the provider-routing-correctness coverage hole for ALL ~29 providers + every BYOK arm without needing any LLM key. Closes the gap flagged in the regression-coverage audit: many offered (runtime → provider) pairs — hermes's 17 name-only BYOK arms, claude-code's zai/deepseek/xiaomi-mimo, openclaw's byok-openai/byok-minimax/groq/openrouter/custom, codex's byok-minimax, etc. — are pure prefix-routing (resolved by `DeriveProvider(runtime, modelId)`) and had **zero** test. A regression in the routing table (wrong provider, dropped arm, bad regex) shipped silently and wedged tenant agents at boot. ## Why keyless / why it gates in the required lane `DeriveProvider` + `ModelPrefixMatch` resolve a model id to a provider with **no upstream call** — a pure function of the merged registry. So the entire offered routing table is gateable in the REQUIRED `CI / all-required` lane with zero secrets. ## Approach — SSOT-driven, not hardcoded `workspace-server/internal/providers/derive_provider_matrix_test.go` iterates `LoadManifest().Runtimes` (the same registry production reads). For every runtime × every offered arm it asserts: (a) `DeriveProvider` resolves to the **exact** expected provider (computed from the SSOT), (b) the (runtime, model) is registration-valid (the `validateRegisteredModelForRuntime` predicate: on the platform menu OR DeriveProvider resolves), (c) no offered id silently misroutes or falls through. - **exact-listed arms**: every model id iterated off the SSOT; expected provider computed from native declaration order (first-declared wins the codex/anthropic "one id, two auth arms" shape). A newly-added model is **auto-covered**. - **name-only arms** (zero models — pure prefix BYOK): each probed with a representative BYOK id its regex must own. The matrix **requires** a representative for every name-only arm in the SSOT — "added an arm, forgot routing/sample" fails RED. A dead representative also fails RED. ## Coverage **5 runtimes, 43 (runtime×provider) arms across 29 distinct providers, 53 exact-listed (runtime×model) assertions + 29 name-only BYOK routing probes** (82 routing assertions total). ## Known-tricky forms pinned explicitly So a regression names its class, not just a generic cell: - the **#2263/#2274** colon-vs-slash-vs-bare MiniMax triple on claude-code: bare `MiniMax-M2.7`→`minimax`, `minimax/…`→`platform`, `minimax:…`→**unregistered** (adapter can't strip `minimax:`) - **#2265** openai-namespaced rejected on claude-code (no native openai arm) - `groq:`→`groq`; openclaw `minimax:`→`byok-minimax` - hermes `anthropic/`/`gemini/`/`openai:`/`minimax:` → `byok-*` (**NOT** platform — cp#529 tenant-key billing safety) - codex `gpt-*` no-auth→`openai-subscription` vs `OPENAI_API_KEY`→`openai-api` - google-adk `platform:`→`platform` vs bare `gemini-…`→`google` ## Watch-it-fail proof Adding `minimax:MiniMax-M2.7` to claude-code's `platform` arm (pointing the colon BYOK form at platform) reds the matrix naming the exact mismatch: ``` DeriveProvider("claude-code", "minimax:MiniMax-M2.7", []) = "platform", want an unregistered/unrouteable ERROR ``` Reverted → green. This proves real coverage of the #2263/#2274 routing class. ## Verification - `go test ./internal/providers/ -run 'DeriveProvider|Matrix|Routing' -count=1` — all green - `go build ./...` — clean - `go vet ./internal/providers/` — clean Test-only; additive; no production path changed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-05 08:00:06 +00:00

test(providers): SSOT-driven DeriveProvider routing matrix — every offered runtime×provider gated (keyless, required-lane) ... c3fd113780

Closes the provider-routing-correctness coverage hole identified in the
regression-coverage audit: many offered (runtime → provider) pairs — hermes's
17 name-only BYOK arms, claude-code's zai/deepseek/xiaomi-mimo, openclaw's
byok-openai/byok-minimax/groq/openrouter/custom, codex's byok-minimax, etc. —
are pure prefix-routing resolved by DeriveProvider(runtime, modelId) and had
ZERO test. A regression in the routing table (wrong provider, dropped arm, bad
regex) shipped silently and wedged tenant agents at boot.

DeriveProvider + ModelPrefixMatch resolve a model id to a provider with NO
upstream call — fully keyless — so the ENTIRE offered routing table is gateable
in the REQUIRED CI / all-required lane with zero secrets.

derive_provider_matrix_test.go is SSOT-DRIVEN (not hardcoded): it iterates
LoadManifest().Runtimes (the same registry production reads) and, for every
runtime × every offered model/provider arm, asserts (a) DeriveProvider resolves
to the EXACT expected provider (computed from the SSOT), (b) the (runtime, model)
is registration-valid (the validateRegisteredModelForRuntime predicate), and
(c) no offered id silently resolves to the wrong arm or falls through.

  - exact-listed arms: every model id iterated off the SSOT, expected provider
    computed from native declaration order (first-declared wins the codex/
    anthropic "one id, two auth arms" shape). A newly-added model is auto-covered.
  - name-only arms (zero models, pure prefix BYOK): each probed with a
    representative BYOK id its regex must own. The matrix REQUIRES a representative
    for every name-only arm in the SSOT — "added an arm, forgot routing/sample"
    fails RED. A dead representative (provider removed) also fails RED.

Coverage: 5 runtimes, 43 (runtime×provider) arms across 29 distinct providers,
53 exact-listed (runtime×model) assertions + 29 name-only BYOK routing probes.

Known-tricky forms pinned as explicit assertions so a regression names its class:
the #2263/#2274 colon-vs-slash-vs-bare MiniMax triple on claude-code (bare→minimax,
slash→platform, colon→unregistered), openai-namespaced-rejected-on-claude-code
(#2265 class), groq→groq, hermes anthropic//gemini//openai://minimax: →
byok-* (NOT platform — cp#529 billing safety), codex gpt default→openai-subscription
vs OPENAI_API_KEY→openai-api, google-adk platform: vs bare gemini.

Watch-it-fail proven: adding minimax:MiniMax-M2.7 to claude-code's platform arm
(pointing the colon BYOK form at platform) reds the matrix naming the exact
mismatch ("= platform, want an unregistered/unrouteable ERROR"); reverted → green.

go build ./... and go vet ./internal/providers/ clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-ceo-assistant approved these changes 2026-06-05 08:02:39 +00:00

claude-ceo-assistant left a comment

Owner

Copy Link

Copy Source

Reviewed: SSOT-driven DeriveProvider routing matrix — 82 assertions across 29 providers + every offered runtime×provider arm + name-only BYOK probes, keyless in the required internal/providers lane. Iterates LoadManifest().Runtimes so new providers auto-covered; requires a representative for every name-only arm (catches added-arm-but-no-routing). Pins the tricky classes (#2263/#2274 minimax colon/slash/bare, #2265, hermes BYOK billing-safety cp#529, codex auth-split, groq). Watch-it-fail proven (pointed minimax-colon at platform → RED). Test-only, additive, go build/vet/test green. Big coverage win for the no-regression goal. Approve.

Reviewed: SSOT-driven DeriveProvider routing matrix — 82 assertions across 29 providers + every offered runtime×provider arm + name-only BYOK probes, keyless in the required internal/providers lane. Iterates LoadManifest().Runtimes so new providers auto-covered; requires a representative for every name-only arm (catches added-arm-but-no-routing). Pins the tricky classes (#2263/#2274 minimax colon/slash/bare, #2265, hermes BYOK billing-safety cp#529, codex auth-split, groq). Watch-it-fail proven (pointed minimax-colon at platform → RED). Test-only, additive, go build/vet/test green. Big coverage win for the no-regression goal. Approve.

agent-reviewer approved these changes 2026-06-05 08:05:14 +00:00

agent-reviewer left a comment

Member

Copy Link

Copy Source

5-axis review: APPROVED.

Correctness: This is a test-only SSOT-driven DeriveProvider routing matrix that walks LoadManifest().Runtimes rather than duplicating a hand-maintained table. It covers exact-listed model arms, name-only BYOK/provider-prefix arms through required representatives, registration-validity, and explicit historically tricky routing forms including minimax slash/colon/bare, hermes BYOK vendor arms, codex auth-env selection, and google-adk platform/BYOK split.

Robustness: The representative coverage is bidirectional: missing samples for new name-only arms fail, and stale representative entries fail. The floor guards prevent an accidentally empty matrix from passing. Security: no secrets or live provider calls; this is pure keyless registry/provider routing validation. Performance: bounded unit-test matrix over the manifest with pure DeriveProvider calls, acceptable for the required lane. Readability: verbose, but the comments document why each guard exists and the deterministic ordering keeps failures stable.

Required-context review: head c3fd113780 is mergeable; E2E API Smoke and Handlers PG are green, and the red combined state is from SOP-tier ceremony, not a code/required-context failure.

5-axis review: APPROVED. Correctness: This is a test-only SSOT-driven DeriveProvider routing matrix that walks LoadManifest().Runtimes rather than duplicating a hand-maintained table. It covers exact-listed model arms, name-only BYOK/provider-prefix arms through required representatives, registration-validity, and explicit historically tricky routing forms including minimax slash/colon/bare, hermes BYOK vendor arms, codex auth-env selection, and google-adk platform/BYOK split. Robustness: The representative coverage is bidirectional: missing samples for new name-only arms fail, and stale representative entries fail. The floor guards prevent an accidentally empty matrix from passing. Security: no secrets or live provider calls; this is pure keyless registry/provider routing validation. Performance: bounded unit-test matrix over the manifest with pure DeriveProvider calls, acceptable for the required lane. Readability: verbose, but the comments document why each guard exists and the deterministic ordering keeps failures stable. Required-context review: head c3fd113780bd256a2920afbfb12a9c9522c3769f is mergeable; E2E API Smoke and Handlers PG are green, and the red combined state is from SOP-tier ceremony, not a code/required-context failure.

core-devops merged commit d3f93efabf into main 2026-06-05 08:18:37 +00:00

claude-ceo-assistant referenced this pull request 2026-06-05 17:17:14 +00:00

fix(e2e): claude-code MiniMax slug must be bare MiniMax-M2.7 not colon (internal#718 UNREGISTERED) #2311

Sign in to join this conversation.

Reviewers

No Reviewers

claude-ceo-assistant

agent-reviewer

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2292