forked from molecule-ai/molecule-core
7cc1c39c49
Closes #9. Three pieces, all small: 1. **docs/e2e-coverage.md** — source of truth for which E2E suites guard which surfaces. Today three were running but informational only on staging; that's how the org-import silent-drop bug shipped without a test catching it pre-merge. Now the matrix shows what's required where + a follow-up note for the two suites that need an always-emit refactor before they can be required. 2. **tools/branch-protection/apply.sh** — branch protection as code. Lets `staging` and `main` required-checks live in a reviewable shell script instead of UI clicks that get lost between admins. This PR's net change: add `E2E API Smoke Test` and `Canvas tabs E2E` as required on staging. Both already use the always-emit path-filter pattern (no-op step emits SUCCESS when the workflow's paths weren't touched), so making them required can't deadlock unrelated PRs. 3. **branch-protection-drift.yml** — daily cron + drift_check.sh that compares live protection against apply.sh's desired state. Catches out-of-band UI edits before they drift further. Fails the workflow on mismatch; ops re-runs apply.sh or updates the script. Out of scope (filed as follow-ups): - e2e-staging-saas + e2e-staging-external use plain `paths:` filters and never trigger when paths are unchanged. They need refactoring to the always-emit shape (same as e2e-api / e2e-staging-canvas) before they can be required. - main branch protection mirrors staging here; if main wants the E2E SaaS / External added later, do it in apply.sh and rerun. Operator must apply once after merge: bash tools/branch-protection/apply.sh The drift check picks it up from there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
59 lines
4.1 KiB
Markdown
59 lines
4.1 KiB
Markdown
# E2E coverage matrix
|
|
|
|
This document is the source of truth for which E2E suites guard which surfaces and which gates are wired up where. Read this before adding a new E2E or moving a check between branches.
|
|
|
|
## Suites
|
|
|
|
| Workflow file | Job (= required-check name) | What it covers | Cron |
|
|
|---|---|---|---|
|
|
| `e2e-api.yml` | `E2E API Smoke Test` | A2A handshake, registry/register, /workspaces/:id/a2a forward, structured-event emission. Lightweight enough to run on every PR. | — |
|
|
| `e2e-staging-canvas.yml` | `Canvas tabs E2E` | Canvas-tab Playwright UX checks against staging — config tab, secrets tab, agent-card tab, Activity hydration. | weekly Sun 08:00 UTC |
|
|
| `e2e-staging-saas.yml` | `E2E Staging SaaS` | Full lifecycle: org creation → workspace provision (CP path) → A2A delegation → status/heartbeat → workspace delete → EC2 termination. The integration test that catches the silent-drop bug class (#2486 / #2811 / #2813 / #2814). | daily 07:00 UTC |
|
|
| `e2e-staging-external.yml` | `E2E Staging External Runtime` | External-runtime registration + heartbeat staleness sweep + `/registry/peers` resolution. Validates the OSS-templated workspace path. | daily 07:30 UTC |
|
|
| `e2e-staging-sanity.yml` | `Intentional-failure teardown sanity` | Inverted assertion — the run MUST fail. Validates the leak-detection self-check itself; not for general gating. | weekly Mon 06:00 UTC |
|
|
| `continuous-synth-e2e.yml` | `Synthetic E2E against staging` | Standing background coverage between PR runs. Catches drift in production-like staging that PR-time E2Es miss. | every 15 min |
|
|
|
|
## Required-check status (branch protection)
|
|
|
|
| Suite | staging required | main required |
|
|
|---|---|---|
|
|
| `E2E API Smoke Test` | ✅ this PR | ✅ |
|
|
| `Canvas tabs E2E` | ✅ this PR | (see follow-up) |
|
|
| `E2E Staging SaaS` | ❌ — needs always-emit refactor | ❌ |
|
|
| `E2E Staging External Runtime` | ❌ — needs always-emit refactor | ❌ |
|
|
| `Intentional-failure teardown sanity` | ❌ inverted assertion, never required | ❌ |
|
|
| `Synthetic E2E against staging` | ❌ cron-only, not a per-PR gate | ❌ |
|
|
|
|
## Why the always-emit pattern matters
|
|
|
|
Branch protection requires a *check name* to land at SUCCESS for every PR. Workflows with `paths:` filters that exclude a PR never run, so the check name never appears, and the PR sits BLOCKED forever.
|
|
|
|
The pattern that supports being required is:
|
|
|
|
1. Workflow always triggers on push/PR to the protected branch.
|
|
2. A `detect-changes` job uses `dorny/paths-filter` to decide if real work runs.
|
|
3. The protected job runs unconditionally and either (a) does real work when paths matched, or (b) emits a no-op SUCCESS step when paths skipped.
|
|
|
|
`e2e-api.yml` and `e2e-staging-canvas.yml` already have this shape. `e2e-staging-saas.yml` and `e2e-staging-external.yml` use plain `paths:` filters and need the refactor before they can be required (filed as follow-up).
|
|
|
|
## Adding a new E2E suite
|
|
|
|
1. Pick a verb: smoke test, full lifecycle, fault-injection, drift detection. Pre-existing suites split along these lines.
|
|
2. Use the always-emit shape so the check name can be made required.
|
|
3. Add a row to the matrix above.
|
|
4. Decide cron cadence based on cost + how fast drift would otherwise be caught.
|
|
5. If you want it required, add to the relevant branch protection via `tools/branch-protection/apply.sh` (this PR adds the script).
|
|
|
|
## When to break glass — temporarily skip a required E2E
|
|
|
|
Don't. If an E2E is intermittently flaky, fix the test or move it out of required. The point of a required check is that it's load-bearing; bypassing one with admin override teaches the next operator the gate is optional.
|
|
|
|
If a Production incident requires bypassing, document the override in the incident postmortem with a same-week followup to either fix the test or rip the check out of required.
|
|
|
|
## Related issues / PRs
|
|
|
|
- #2486 — silent-drop bug class that the SaaS E2E now catches
|
|
- PR #2811 — `provisionWorkspaceAuto` consolidation (org-import SaaS gate)
|
|
- PR #2824 — `StopWorkspaceAuto` mirror (closes #2813 + #2814)
|
|
- Follow-up: refactor `e2e-staging-saas` + `e2e-staging-external` to always-emit (so they can be required)
|