molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 3 Packages Projects Releases Wiki Activity

[no-regression] e2e coverage matrix + gaps: runtimes × providers × features #2332

New Issue
Open
opened 2026-06-06 04:24:13 +00:00 by claude-ceo-assistant · 0 comments
claude-ceo-assistant commented 2026-06-06 04:24:13 +00:00
Owner
Copy Link
Copy Source

e2e coverage matrix — runtimes × providers × features (no-regression audit, 2026-06-05)

Evidence-based audit (read from origin/main of core+CP+templates; local checkouts were stale). Goal: every supported runtime + provider + feature has a gating e2e, so no silent regressions.

Supported set (SSOT = controlplane providers.yaml)

  • Runtimes (5): claude-code, codex, hermes, openclaw, google-adk. (crewai/deepagents/gemini-cli/autogen/langgraph removed via migrations 031/034/035.)
  • Providers×auth gated live by ci / serving-e2e (REQUIRED, CP): anthropic-api, anthropic-oauth, openai-api, kimi-coding, minimax, google, vertex(WIF, conditional), platform, byok-{anthropic,openai,minimax,gemini}.

CONFIRMED COVERAGE GAPS (prioritized)

P0 — gated-cell illusions / shipped-but-uncovered:

  1. google-adk has NO live e2e arm. Core test_priority_runtimes_e2e.sh default set = mock claude-code codex hermes openclaw minimax — google-adk absent. Also not asserted in CP runtimes_test.go SSOT map. We just registered its models (cp#568/#2327) with zero serving/runtime e2e. → add google-adk arm + SSOT assertion.
  2. openai-subscription (Codex OAuth) serving is gated NOWHERE. CP serving-e2e arm #4 hard-skips it (serving_e2e_test.go:233); core priority-runtimes codex arm skips without E2E_OPENAI_API_KEY. Looks covered, isn't. → real Codex-backend serving e2e.
  3. groq: zero serving coverage (no arm; key now provisioned in Infisical /shared/serving-e2e-keys) → add arm.

P0 — feature gaps that map to ACTUAL recent incidents:
4. Context-overflow autoheal / session-reset: ZERO e2e. This is exactly what wedged Kimi tonight (262k overflow → manual restart). Must ship an e2e WITH fix/context-overflow-autoheal.
5. Workspace data-volume survives-recreate + snapshot-before-swap (/home/agent wipe): no e2e. Past incident feedback_workspace_container_swap_wipes_home_agent.
6. SM/secrets.d real round-trip (two-sided IAM): no live e2e (only userdata-string unit). Past incident cp#358.
7. Desktop reconnect + 300s lease renewal (core#2216): no e2e.

P1 — registry/SSOT drift (real regressions):
8. Core providers registry mirror STALE on vertex — fingerprint 9d129c96(CP) vs e457249e(core); core lacks the keyless-WIF vertex (auth_mode wif_adc + endpoint_vars). Re-byte-sync providers.yaml + regen. (This is the #561 vertex-SSOT work.)
9. catalog-vs-runtime over-offer (billing-only model ids) + haiku id mismatch (claude-haiku-4-5 vs -4-5-20251001) + no template_registry row for codex/google-adk (not List-API-discoverable).

P1 — operational surface (impl exists, unit-only):
10. Soft-restart/pause/resume/hibernate against a real container; live channel send (telegram/slack/discord/lark/email — email has no adapter test at all); channel discover; workspace data-prune (RFC#734).

Structural finding

Core's 3 REQUIRED gates (CI/all-required, E2E API Smoke, Handlers Postgres) include no live-SaaS/staging e2e — E2E Chat / Staging SaaS / Peer Visibility / Canvas / External / Reconciler are all advisory. The only live-runtime assurance in the required set is the mock backbone (real-LLM arms skip without secrets). Promote-to-required candidates exist (e2e-peer-visibility is already continue-on-error:false).

Owners: P0.1/P0.3/P1.8 → in-flight PRs (this issue). P0.2/P0.4-7/P1.10 → scoped follow-up PRs, each adding a GATING test. Tracking umbrella for the no-regression e2e expansion.

## e2e coverage matrix — runtimes × providers × features (no-regression audit, 2026-06-05) Evidence-based audit (read from origin/main of core+CP+templates; local checkouts were stale). Goal: every supported runtime + provider + feature has a **gating** e2e, so no silent regressions. ### Supported set (SSOT = controlplane providers.yaml) - **Runtimes (5):** claude-code, codex, hermes, openclaw, google-adk. (crewai/deepagents/gemini-cli/autogen/langgraph removed via migrations 031/034/035.) - **Providers×auth gated live by `ci / serving-e2e` (REQUIRED, CP):** anthropic-api, anthropic-oauth, openai-api, kimi-coding, minimax, google, vertex(WIF, conditional), platform, byok-{anthropic,openai,minimax,gemini}. ### CONFIRMED COVERAGE GAPS (prioritized) **P0 — gated-cell illusions / shipped-but-uncovered:** 1. **google-adk has NO live e2e arm.** Core `test_priority_runtimes_e2e.sh` default set = `mock claude-code codex hermes openclaw minimax` — google-adk absent. Also not asserted in CP `runtimes_test.go` SSOT map. We *just* registered its models (cp#568/#2327) with zero serving/runtime e2e. → add google-adk arm + SSOT assertion. 2. **openai-subscription (Codex OAuth) serving is gated NOWHERE.** CP serving-e2e arm #4 hard-skips it (`serving_e2e_test.go:233`); core priority-runtimes codex arm skips without E2E_OPENAI_API_KEY. Looks covered, isn't. → real Codex-backend serving e2e. 3. **groq: zero serving coverage** (no arm; key now provisioned in Infisical /shared/serving-e2e-keys) → add arm. **P0 — feature gaps that map to ACTUAL recent incidents:** 4. **Context-overflow autoheal / session-reset: ZERO e2e.** This is exactly what wedged Kimi tonight (262k overflow → manual restart). Must ship an e2e WITH `fix/context-overflow-autoheal`. 5. **Workspace data-volume survives-recreate + snapshot-before-swap (`/home/agent` wipe): no e2e.** Past incident `feedback_workspace_container_swap_wipes_home_agent`. 6. **SM/secrets.d real round-trip (two-sided IAM): no live e2e** (only userdata-string unit). Past incident cp#358. 7. **Desktop reconnect + 300s lease renewal (core#2216): no e2e.** **P1 — registry/SSOT drift (real regressions):** 8. **Core providers registry mirror STALE on `vertex`** — fingerprint 9d129c96(CP) vs e457249e(core); core lacks the keyless-WIF vertex (auth_mode wif_adc + endpoint_vars). Re-byte-sync providers.yaml + regen. (This is the #561 vertex-SSOT work.) 9. catalog-vs-runtime over-offer (billing-only model ids) + haiku id mismatch (claude-haiku-4-5 vs -4-5-20251001) + no template_registry row for codex/google-adk (not List-API-discoverable). **P1 — operational surface (impl exists, unit-only):** 10. Soft-restart/pause/resume/hibernate against a real container; live channel send (telegram/slack/discord/lark/email — email has no adapter test at all); channel discover; workspace data-prune (RFC#734). ### Structural finding Core's 3 REQUIRED gates (CI/all-required, E2E API Smoke, Handlers Postgres) include **no live-SaaS/staging e2e** — E2E Chat / Staging SaaS / Peer Visibility / Canvas / External / Reconciler are all advisory. The only live-runtime assurance in the required set is the `mock` backbone (real-LLM arms skip without secrets). Promote-to-required candidates exist (e2e-peer-visibility is already `continue-on-error:false`). Owners: P0.1/P0.3/P1.8 → in-flight PRs (this issue). P0.2/P0.4-7/P1.10 → scoped follow-up PRs, each adding a GATING test. Tracking umbrella for the no-regression e2e expansion.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2332
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?