molecule-core/tests/e2e
Hongming Wang 4c49ff75f6 test(e2e): canary classifies provider-quota 429 as operator-action, not platform regression
The staging canary's A2A step has a ladder of specific regression
classifiers (hermes-agent down, model_not_found, Invalid API key,
etc.) followed by a generic "error|exception" catch-all. Provider-
side OpenAI 429 quota errors fell through to the catch-all, so the
canary issue body and CI log just said "A2A returned an error-shaped
response" — which is technically true but obscures the actual
operator action.

This adds a 7th classifier above the catch-all for "exceeded your
current quota" / "insufficient_quota" — both terms appear in
OpenAI's quota-exhaustion 429 response. When matched, the failure
message names the operator action directly (top up MOLECULE_STAGING_OPENAI_KEY
or rotate the secret) and links to #2578.

Why this is correct, not "lowering the bar":
- Steps 0–7 of the canary cover full platform health (CP up, tenant
  provisioned, DNS+TLS reachable, workspace booted, A2A delivered).
- Reaching step 8 with a provider-side 429 means the platform IS
  healthy — the failure is downstream of all platform invariants.
- The canary still exits 1 (CI stays red, threshold-3 alarm still
  fires); only the failure message changes.
- All 6 existing specific classifiers run BEFORE this one, so any
  real platform regression is still caught with its specific message.

Verification:
- Regex tested against the actual 429 string from canary run 25291517608:
    "API call failed after 3 retries: HTTP 429: You exceeded your current quota..."
  → matches 
- Negative tests: "PONG", "hermes-agent unreachable" → no match 
- bash -n syntax check passes
- shellcheck -S error clean

Tracking: #2593 (canary), #2578 (root cause)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 15:18:42 -07:00
..
lib test(e2e): pin pick_model_slug behavior with bash unit tests 2026-05-03 12:04:12 -07:00
_extract_token.py chore: apply round-7 review nits 2026-04-13 17:08:45 -07:00
_lib.sh feat(platform): GET /admin/workspaces/:id/test-token for E2E (#6) 2026-04-14 09:35:26 -07:00
STAGING_SAAS_E2E.md feat(e2e): pivot to admin-bearer-only auth + add sanity self-check workflow 2026-04-21 04:34:11 -07:00
test_2307_peer_visibility_staging.sh test(e2e): add staging peer-visibility harness for #2307 2026-04-29 13:26:24 -07:00
test_a2a_e2e.sh initial commit — Molecule AI platform 2026-04-13 11:55:37 -07:00
test_activity_e2e.sh chore: apply code-review round-6 suggestions 2026-04-13 17:08:45 -07:00
test_api.sh fix(e2e): stop asserting current_task on public workspace GET (#966) 2026-04-19 02:19:15 -07:00
test_chat_attachments_e2e.sh feat(canvas+platform): chat attachments, model selection, deploy/delete UX 2026-04-24 13:27:51 -07:00
test_chat_attachments_multiruntime_e2e.sh feat(canvas+platform): chat attachments, model selection, deploy/delete UX 2026-04-24 13:27:51 -07:00
test_chat_upload_e2e.sh feat(chat_files): rewrite Upload as HTTP-forward to workspace (RFC #2312, PR-C) 2026-04-29 14:26:37 -07:00
test_claude_code_e2e.sh chore: final open-source cleanup — binary, stale paths, private refs 2026-04-18 00:38:55 -07:00
test_comprehensive_e2e.sh fix(e2e): make provisioning-status assertions robust to CI environment 2026-04-13 17:31:07 -07:00
test_dev_mode.sh fix(quickstart): hotfixes discovered during live testing session 2026-04-23 14:57:18 -07:00
test_harness_rc_normalization.sh fix(e2e-sanity): normalize unexpected curl exit codes in cleanup trap (#2159) 2026-04-27 02:55:44 -07:00
test_model_slug.sh test(e2e): pin pick_model_slug behavior with bash unit tests 2026-05-03 12:04:12 -07:00
test_notify_attachments_e2e.sh test(notify): pre-sweep prior workspaces so interrupted runs don't pile up 2026-04-26 20:55:13 -07:00
test_poll_mode_e2e.sh fix(e2e): use real UUIDs for poll-mode test workspace ids 2026-04-29 23:10:36 -07:00
test_priority_runtimes_e2e.sh feat(e2e): extend priority-runtimes test to cover all 8 templates 2026-04-27 05:57:59 -07:00
test_saas_tenant.sh chore: final open-source cleanup — binary, stale paths, private refs 2026-04-18 00:38:55 -07:00
test_staging_external_runtime.sh test(e2e): read delivery_mode from register response, not GET 2026-04-30 10:35:21 -07:00
test_staging_full_saas.sh test(e2e): canary classifies provider-quota 429 as operator-action, not platform regression 2026-05-03 15:18:42 -07:00