Hongming Wang 7cc1c39c49 ci: e2e coverage matrix + branch-protection-as-code

Closes #9.

Three pieces, all small:

1. **docs/e2e-coverage.md** — source of truth for which E2E suites
   guard which surfaces. Today three were running but informational
   only on staging; that's how the org-import silent-drop bug shipped
   without a test catching it pre-merge. Now the matrix shows what's
   required where + a follow-up note for the two suites that need an
   always-emit refactor before they can be required.

2. **tools/branch-protection/apply.sh** — branch protection as code.
   Lets `staging` and `main` required-checks live in a reviewable
   shell script instead of UI clicks that get lost between admins.
   This PR's net change: add `E2E API Smoke Test` and `Canvas tabs E2E`
   as required on staging. Both already use the always-emit path-filter
   pattern (no-op step emits SUCCESS when the workflow's paths weren't
   touched), so making them required can't deadlock unrelated PRs.

3. **branch-protection-drift.yml** — daily cron + drift_check.sh
   that compares live protection against apply.sh's desired state.
   Catches out-of-band UI edits before they drift further. Fails the
   workflow on mismatch; ops re-runs apply.sh or updates the script.

Out of scope (filed as follow-ups):
- e2e-staging-saas + e2e-staging-external use plain `paths:` filters
  and never trigger when paths are unchanged. They need refactoring
  to the always-emit shape (same as e2e-api / e2e-staging-canvas)
  before they can be required.
- main branch protection mirrors staging here; if main wants the
  E2E SaaS / External added later, do it in apply.sh and rerun.

Operator must apply once after merge:
  bash tools/branch-protection/apply.sh
The drift check picks it up from there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-04 20:21:59 -07:00

4.1 KiB

Raw Blame History

E2E coverage matrix

This document is the source of truth for which E2E suites guard which surfaces and which gates are wired up where. Read this before adding a new E2E or moving a check between branches.

Suites

Workflow file	Job (= required-check name)	What it covers	Cron
`e2e-api.yml`	`E2E API Smoke Test`	A2A handshake, registry/register, /workspaces/:id/a2a forward, structured-event emission. Lightweight enough to run on every PR.	—
`e2e-staging-canvas.yml`	`Canvas tabs E2E`	Canvas-tab Playwright UX checks against staging — config tab, secrets tab, agent-card tab, Activity hydration.	weekly Sun 08:00 UTC
`e2e-staging-saas.yml`	`E2E Staging SaaS`	Full lifecycle: org creation → workspace provision (CP path) → A2A delegation → status/heartbeat → workspace delete → EC2 termination. The integration test that catches the silent-drop bug class (#2486 / #2811 / #2813 / #2814).	daily 07:00 UTC
`e2e-staging-external.yml`	`E2E Staging External Runtime`	External-runtime registration + heartbeat staleness sweep + `/registry/peers` resolution. Validates the OSS-templated workspace path.	daily 07:30 UTC
`e2e-staging-sanity.yml`	`Intentional-failure teardown sanity`	Inverted assertion — the run MUST fail. Validates the leak-detection self-check itself; not for general gating.	weekly Mon 06:00 UTC
`continuous-synth-e2e.yml`	`Synthetic E2E against staging`	Standing background coverage between PR runs. Catches drift in production-like staging that PR-time E2Es miss.	every 15 min

Required-check status (branch protection)

Suite	staging required	main required
`E2E API Smoke Test`	✅ this PR	✅
`Canvas tabs E2E`	✅ this PR	(see follow-up)
`E2E Staging SaaS`	❌ — needs always-emit refactor	❌
`E2E Staging External Runtime`	❌ — needs always-emit refactor	❌
`Intentional-failure teardown sanity`	❌ inverted assertion, never required	❌
`Synthetic E2E against staging`	❌ cron-only, not a per-PR gate	❌

Why the always-emit pattern matters

Branch protection requires a check name to land at SUCCESS for every PR. Workflows with paths: filters that exclude a PR never run, so the check name never appears, and the PR sits BLOCKED forever.

The pattern that supports being required is:

Workflow always triggers on push/PR to the protected branch.
A detect-changes job uses dorny/paths-filter to decide if real work runs.
The protected job runs unconditionally and either (a) does real work when paths matched, or (b) emits a no-op SUCCESS step when paths skipped.

e2e-api.yml and e2e-staging-canvas.yml already have this shape. e2e-staging-saas.yml and e2e-staging-external.yml use plain paths: filters and need the refactor before they can be required (filed as follow-up).

Adding a new E2E suite

Pick a verb: smoke test, full lifecycle, fault-injection, drift detection. Pre-existing suites split along these lines.
Use the always-emit shape so the check name can be made required.
Add a row to the matrix above.
Decide cron cadence based on cost + how fast drift would otherwise be caught.
If you want it required, add to the relevant branch protection via tools/branch-protection/apply.sh (this PR adds the script).

When to break glass — temporarily skip a required E2E

Don't. If an E2E is intermittently flaky, fix the test or move it out of required. The point of a required check is that it's load-bearing; bypassing one with admin override teaches the next operator the gate is optional.

If a Production incident requires bypassing, document the override in the incident postmortem with a same-week followup to either fix the test or rip the check out of required.

#2486 — silent-drop bug class that the SaaS E2E now catches
PR #2811 — provisionWorkspaceAuto consolidation (org-import SaaS gate)
PR #2824 — StopWorkspaceAuto mirror (closes #2813 + #2814)
Follow-up: refactor e2e-staging-saas + e2e-staging-external to always-emit (so they can be required)

4.1 KiB Raw Blame History