test(e2e): keyless required-lane coverage for mock runtime + terminal/webhooks/budget/checkpoints/audit/traces/session-search/rescue/billing-mode/resume/hibernate + wire orphaned secrets-dispatch #2293

Merged

core-devops merged 1 commits from harden/keyless-feature-e2e-coverage into main 2026-06-05 08:18:47 +00:00

Conversation 0 Commits 1 Files Changed 2

+347

core-devops commented 2026-06-05 08:05:09 +00:00

Member

Copy Link

Copy Source

What

Keyless, REQUIRED-lane (E2E API Smoke Test) e2e coverage for the CTO goal "e2e covers every runtime and feature, no regressions" — the feature endpoints + lifecycle that ship without an LLM key and had no e2e assertion in the required lane, plus wiring an orphaned keyless contract test.

New script — tests/e2e/test_keyless_feature_contracts_e2e.sh

Self-contained, hermetic (one runtime=external fixture, NO LLM key). For each endpoint it asserts the real HTTP contract and a meaningful failure mode so a regression goes RED, not silently green:

Endpoint	Tier	Happy	Failure
GET /workspaces/:id/terminal/diagnose	wsAuth	200 report w/ workspace_id+steps[]	401 no-auth
POST /webhooks/:type	public	200 ignored (non-message)	400 bad-json, 404 unknown type
GET /workspaces/:id/budget + PATCH	wsAuth / admin	periods view; set+persist monthly	400 empty, 400 unknown-period, 401
/workspaces/:id/checkpoints*	wsAuth	upsert→latest→list→delete	404 after delete, 400 missing wfid, 401
GET /workspaces/:id/audit	wsAuth	total:0 + chain_valid:null	400 bad from, 401
GET /workspaces/:id/traces	wsAuth	200 [] without Langfuse	401
GET /workspaces/:id/session-search	wsAuth	q-filter hit / [] miss	401
GET /workspaces/:id/rescue	wsAuth	fail-closed 503 (no MOLECULE_ORG_ID)	401
GET/PUT /admin/workspaces/:id/llm-billing-mode	admin	flip byok + readback	400 bad-UUID, 400 missing-mode, 400 unknown-mode
Lifecycle pause→resume + hibernate	wsAuth	transitions	404 wrong-state, 401

Auth mirrors wsauth_middleware.go: WorkspaceAuth is strict (401 without bearer once a token exists); AdminAuth accepts the platform ADMIN_TOKEN or the workspace bearer (Tier-3). The script resolves its admin bearer from MOLECULE_ADMIN_TOKEN/ADMIN_TOKEN if set, else the minted workspace token — so it is green in both the current no-ADMIN_TOKEN CI shape and the post-#2286 ADMIN_TOKEN shape.

Mock runtime

The mock-runtime A2A canned round-trip is owned by #2286's mock arm (run_mock in test_priority_runtimes_e2e.sh) — intentionally not duplicated here.

Wire orphaned test

tests/e2e/test_secrets_dispatch.sh was referenced by NO workflow. Added as a required-lane step. It is hermetic (extracts + runs the SECRETS_JSON branch-order block in isolation — no platform, no bearer, no network), guarding the 2026-05-03 "wrong LLM-key shape wins" incident class.

Coordination with #2286 (open)

#2286 (harden/enforce-ci-gates-core-v2) owns e2e-api.yml's admin-auth wiring, _lib.sh's e2e_admin_auth_args, test_api.sh's auth helpers, and the test_priority_runtimes runtime arms. This PR touches none of those — it only adds two run: steps to e2e-api.yml and adds one new file. Rebases cleanly whether #2286 lands first or this does.

Proof

Local PG + Redis + platform-server (CI shape):

• post-#2286 shape (ADMIN_TOKEN set): 48/48 green
• current shape (no ADMIN_TOKEN): 48/48 green
• test_secrets_dispatch.sh: 10/10 green; existing test_api.sh unchanged 61/61 green
• bash -n + shellcheck clean (only the suite-standard SC1091 source-follow info)

No "flaky": every assertion keys off a deterministic state (fresh-workspace zero-rows, fail-closed status, fixed status transitions, sorted/keyed responses).

Not keyless-coverable (flagged for staging tier)

• GET /workspaces/:id/terminal itself is a WebSocket upgrade, not HTTP-assertable in this lane — its pure-HTTP sibling /terminal/diagnose is covered instead.
• /rescue happy path (200 bundle) needs a captured rescue bundle + MOLECULE_ORG_ID; only the fail-closed 503 contract is keyless here. Full bundle round-trip belongs in the staging-saas tier.
• /resume 200-vs-503 depends on whether a provisioner is wired; the test accepts either valid contract.

🤖 Generated with Claude Code

## What Keyless, REQUIRED-lane (`E2E API Smoke Test`) e2e coverage for the CTO goal "e2e covers every runtime and feature, no regressions" — the feature endpoints + lifecycle that ship **without** an LLM key and had **no** e2e assertion in the required lane, plus wiring an orphaned keyless contract test. ## New script — `tests/e2e/test_keyless_feature_contracts_e2e.sh` Self-contained, hermetic (one `runtime=external` fixture, NO LLM key). For each endpoint it asserts the real HTTP contract **and** a meaningful failure mode so a regression goes RED, not silently green: | Endpoint | Tier | Happy | Failure | |---|---|---|---| | `GET /workspaces/:id/terminal/diagnose` | wsAuth | 200 report w/ `workspace_id`+`steps[]` | 401 no-auth | | `POST /webhooks/:type` | public | 200 `ignored` (non-message) | 400 bad-json, 404 unknown type | | `GET /workspaces/:id/budget` + `PATCH` | wsAuth / admin | periods view; set+persist monthly | 400 empty, 400 unknown-period, 401 | | `/workspaces/:id/checkpoints*` | wsAuth | upsert→latest→list→delete | 404 after delete, 400 missing wfid, 401 | | `GET /workspaces/:id/audit` | wsAuth | `total:0` + `chain_valid:null` | 400 bad `from`, 401 | | `GET /workspaces/:id/traces` | wsAuth | 200 `[]` without Langfuse | 401 | | `GET /workspaces/:id/session-search` | wsAuth | q-filter hit / `[]` miss | 401 | | `GET /workspaces/:id/rescue` | wsAuth | fail-closed 503 (no `MOLECULE_ORG_ID`) | 401 | | `GET/PUT /admin/workspaces/:id/llm-billing-mode` | admin | flip `byok` + readback | 400 bad-UUID, 400 missing-mode, 400 unknown-mode | | Lifecycle `pause`→`resume` + `hibernate` | wsAuth | transitions | 404 wrong-state, 401 | Auth mirrors `wsauth_middleware.go`: WorkspaceAuth is strict (401 without bearer once a token exists); AdminAuth accepts the platform `ADMIN_TOKEN` **or** the workspace bearer (Tier-3). The script resolves its admin bearer from `MOLECULE_ADMIN_TOKEN`/`ADMIN_TOKEN` if set, else the minted workspace token — so it is green in **both** the current no-`ADMIN_TOKEN` CI shape **and** the post-#2286 `ADMIN_TOKEN` shape. ## Mock runtime The mock-runtime A2A canned round-trip is **owned by #2286's `mock` arm** (`run_mock` in `test_priority_runtimes_e2e.sh`) — intentionally **not** duplicated here. ## Wire orphaned test `tests/e2e/test_secrets_dispatch.sh` was referenced by NO workflow. Added as a required-lane step. It is hermetic (extracts + runs the `SECRETS_JSON` branch-order block in isolation — no platform, no bearer, no network), guarding the 2026-05-03 "wrong LLM-key shape wins" incident class. ## Coordination with #2286 (open) #2286 (`harden/enforce-ci-gates-core-v2`) owns `e2e-api.yml`'s admin-auth wiring, `_lib.sh`'s `e2e_admin_auth_args`, `test_api.sh`'s auth helpers, and the `test_priority_runtimes` runtime arms. This PR touches **none** of those — it only **adds two `run:` steps** to `e2e-api.yml` and **adds one new file**. Rebases cleanly whether #2286 lands first or this does. ## Proof Local PG + Redis + `platform-server` (CI shape): - post-#2286 shape (`ADMIN_TOKEN` set): **48/48** green - current shape (no `ADMIN_TOKEN`): **48/48** green - `test_secrets_dispatch.sh`: **10/10** green; existing `test_api.sh` unchanged **61/61** green - `bash -n` + `shellcheck` clean (only the suite-standard SC1091 source-follow info) No "flaky": every assertion keys off a deterministic state (fresh-workspace zero-rows, fail-closed status, fixed status transitions, sorted/keyed responses). ## Not keyless-coverable (flagged for staging tier) - `GET /workspaces/:id/terminal` itself is a **WebSocket upgrade**, not HTTP-assertable in this lane — its pure-HTTP sibling `/terminal/diagnose` is covered instead. - `/rescue` **happy path** (200 bundle) needs a captured rescue bundle + `MOLECULE_ORG_ID`; only the **fail-closed 503** contract is keyless here. Full bundle round-trip belongs in the staging-saas tier. - `/resume` 200-vs-503 depends on whether a provisioner is wired; the test accepts either valid contract. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-05 08:05:10 +00:00

test(e2e): keyless required-lane coverage for mock runtime + terminal/webhooks/budget/checkpoints/audit/traces/session-search/rescue/billing-mode/resume/hibernate + wire orphaned secrets-dispatch ... d3d108a636

Closes coverage-audit gaps for CI-coverable, keyless feature endpoints that
had NO e2e assertion in the required `E2E API Smoke Test` lane.

New: tests/e2e/test_keyless_feature_contracts_e2e.sh — a self-contained,
hermetic script (runtime=external fixture, NO LLM key) asserting the real
HTTP contract + a meaningful failure mode for each endpoint:

  * GET  /workspaces/:id/terminal/diagnose  — 200 report / 401 no-auth
    (the /terminal WS-upgrade sibling that is HTTP-assertable keyless)
  * POST /webhooks/:type (public)           — 200 ignored / 400 bad-json / 404 unknown
  * GET  /workspaces/:id/budget + PATCH      — periods view / set+persist / 400 / 401
  * /workspaces/:id/checkpoints*             — upsert→latest→list→delete→404 / 400 / 401
  * GET  /workspaces/:id/audit               — total0+chain_valid null / 400 bad-from / 401
  * GET  /workspaces/:id/traces              — 200 [] without Langfuse / 401
  * GET  /workspaces/:id/session-search      — q-filter hit / [] miss / 401
  * GET  /workspaces/:id/rescue              — fail-closed 503 (no MOLECULE_ORG_ID) / 401
  * GET/PUT /admin/workspaces/:id/llm-billing-mode — flip byok+readback / 400 ×3
  * Lifecycle pause→resume + hibernate       — transitions / 404 wrong-state / 401

Auth model mirrors wsauth_middleware.go: WorkspaceAuth is strict (401 without
bearer once a token exists), AdminAuth accepts the platform ADMIN_TOKEN OR the
workspace bearer (Tier-3) — so the script is green in BOTH the current
no-ADMIN_TOKEN CI shape and the post-#2286 ADMIN_TOKEN shape (proven locally,
48/48 each). Mock-runtime A2A canned round-trip is left to #2286's mock arm
(not duplicated). Does not touch e2e-api.yml admin-auth wiring or
test_priority_runtimes runtime arms (#2286 owns those) — only adds run steps.

Wire: tests/e2e/test_secrets_dispatch.sh was orphaned (no workflow ran it).
Added as a required-lane step. It is hermetic (extracts + runs the SECRETS_JSON
branch-order block in isolation; no platform/bearer/network), guarding the
2026-05-03 "wrong LLM-key shape wins" incident class.

Proof: local PG+Redis+platform-server (CI shape), all three scripts GREEN in
lane order under both auth shapes; bash -n + shellcheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-ceo-assistant approved these changes 2026-06-05 08:07:19 +00:00

claude-ceo-assistant left a comment

Owner

Copy Link

Copy Source

Reviewed: keyless feature-contract coverage — 48 assertions across 11 endpoints (terminal-diagnose, webhooks, budget, checkpoints, audit, traces, session-search, rescue fail-closed, billing-mode flip, pause/resume/hibernate) each with happy + failure case, in the required E2E API Smoke lane; wires the orphaned secrets-dispatch test in. New self-contained script (no conflict with #2286 helpers). Locally proven 48/48 both auth shapes. Closes the keyless feature-coverage gap for the no-regressions goal. Approve.

Reviewed: keyless feature-contract coverage — 48 assertions across 11 endpoints (terminal-diagnose, webhooks, budget, checkpoints, audit, traces, session-search, rescue fail-closed, billing-mode flip, pause/resume/hibernate) each with happy + failure case, in the required E2E API Smoke lane; wires the orphaned secrets-dispatch test in. New self-contained script (no conflict with #2286 helpers). Locally proven 48/48 both auth shapes. Closes the keyless feature-coverage gap for the no-regressions goal. Approve.

agent-reviewer approved these changes 2026-06-05 08:08:52 +00:00

agent-reviewer left a comment

Member

Copy Link

Copy Source

5-axis review: APPROVED.

Correctness: This adds required-lane keyless E2E coverage for the feature endpoints named in the PR: terminal diagnose, webhooks, budget, checkpoints, audit, traces, session search, rescue, LLM billing mode, pause/resume, and hibernate. Each path has a happy-path assertion plus a meaningful failure mode, and the workflow now also runs the previously orphaned secrets-dispatch contract test.

Robustness: The fixture uses an external workspace and token/admin-auth handling that works across the pre/post ADMIN_TOKEN E2E shapes. The script accumulates assertion failures and exits non-zero at the end, so multiple contract breaks are visible in one run. Security: no LLM keys or live provider secrets are required; the tests explicitly validate auth rejection and fail-closed behavior. Performance: bounded curl-based E2E coverage in the existing E2E API Smoke lane; no unbounded loops or external provider calls. Readability: the script is long, but endpoint sections are named and the comments explain the key auth and fail-closed contracts.

Required-context review: head d3d108a636 is mergeable; CI/all-required, E2E API Smoke, and Handlers PG are green. Combined red is not used as the gate.

5-axis review: APPROVED. Correctness: This adds required-lane keyless E2E coverage for the feature endpoints named in the PR: terminal diagnose, webhooks, budget, checkpoints, audit, traces, session search, rescue, LLM billing mode, pause/resume, and hibernate. Each path has a happy-path assertion plus a meaningful failure mode, and the workflow now also runs the previously orphaned secrets-dispatch contract test. Robustness: The fixture uses an external workspace and token/admin-auth handling that works across the pre/post ADMIN_TOKEN E2E shapes. The script accumulates assertion failures and exits non-zero at the end, so multiple contract breaks are visible in one run. Security: no LLM keys or live provider secrets are required; the tests explicitly validate auth rejection and fail-closed behavior. Performance: bounded curl-based E2E coverage in the existing E2E API Smoke lane; no unbounded loops or external provider calls. Readability: the script is long, but endpoint sections are named and the comments explain the key auth and fail-closed contracts. Required-context review: head d3d108a6364b7c343accaa4facb3e5bd6ca3871e is mergeable; CI/all-required, E2E API Smoke, and Handlers PG are green. Combined red is not used as the gate.

core-devops merged commit 9efd06034c into main 2026-06-05 08:18:47 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

claude-ceo-assistant

agent-reviewer

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2293