ci(e2e-staging): promote E2E Staging Platform Boot to merge-blocking (fail-closed) — #48 #3116

Merged

core-devops merged 1 commits from harden/platform-boot-merge-blocking into main 2026-06-21 09:32:37 +00:00

Conversation 2 Commits 1 Files Changed 3

+81 -18

core-devops commented 2026-06-21 09:29:09 +00:00

Member

Copy Link

Copy Source

What & why (RCA #878 → #885)

A prod onboarding outage (06:04–08:09 UTC 2026-06-21) was caused by molecule-controlplane PR #878: it rendered the tenant docker run env block with a blank line that broke shell \-continuation → the image arg was orphaned → docker run exit=127 → no tenant container → onboarding down. Fixed by CP PR #885 (deployed 08:09). It escaped pre-merge testing partly because the real-boot e2e (E2E Staging Platform Boot) is advisory (continue-on-error: true) and never ran on PRs (if: push/dispatch/schedule guard).

Task #48: promote E2E Staging Platform Boot to merge-blocking (fail-closed).

What changed

Mirrors the in-file gating exemplar e2e-staging-concierge-creates-workspace (core#3081 / CR2 #12653) exactly.

.gitea/workflows/e2e-staging-saas.yml — e2e-staging-platform-boot job:

• Removed continue-on-error: true (was the mc#2654 mask — Gitea Quirk #10 makes a failed step roll up to success under CoE, which is precisely how a broken boot would false-green).
• Removed the if: push || workflow_dispatch || schedule guard so the job runs on pull_request. A required context that never fires on PR degrades the merge gate to a silent indefinite pending (the failure mode lint-required-no-paths / feedback_path_filtered_workflow_cant_be_required exist to prevent).
• E2E_REQUIRE_LIVE: ${{ github.event_name == 'pull_request' && '0' || '1' }} — false-green-proof:
  • pull_request → 0: PRs carry no staging creds; the harness runs a bash -n PR-mode self-check and exit 0.
  • push / dispatch / schedule → 1: the real staging boot runs and HARD FAILs (exit 5) if it proves no live provisioned → tenant_online → workspace_online → a2a_roundtrip lifecycle.
• Verify admin token present step is PR-mode-aware: skips cleanly when E2E_REQUIRE_LIVE=0 + no token; still hard-errors on a real run.
• Kept the Teardown safety net (if: always()) step unchanged.

tests/e2e/test_staging_full_saas.sh (shared harness):

• Added a PR-mode early-exit (REQUIRE_LIVE=0 && no admin token → bash -n self-check → exit 0), mirroring test_staging_concierge_creates_workspace_e2e.sh.
• Flipped the admin-token line from ${MOLECULE_ADMIN_TOKEN:?...} to ${MOLECULE_ADMIN_TOKEN:-} so the PR lane no longer hard-dies before the self-check; a non-PR run with no token is still a HARD FAIL just past the PR-mode block.
• Safe for the sibling e2e-staging-saas job: it keeps its if: push/dispatch/schedule guard and E2E_REQUIRE_LIVE: '1', so it never reaches the PR-mode branch.

.gitea/required-contexts.txt: added E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (SSOT), in the same PR that removes CoE, per lint-no-coe-on-required (CoE forbidden on any listed context).

Cost tradeoff (please weigh)

This makes every core PR provision a real staging tenant — a second full EC2 provision per PR, alongside e2e-staging-concierge-creates-workspace. Wait: the platform-boot job runs on pull_request with E2E_REQUIRE_LIVE=0 and no staging creds on PR, so on PR it does a bash -n self-check and exits — it does NOT provision on PR. The real provision happens on push-to-main / dispatch / cron (as before, but now blocking deploy-to-main rather than advisory). So the incremental cost vs. today is: the platform boot is no longer maskable, and a red platform-boot on main now blocks (it was advisory).

If staging creds were ever wired to PR runs, this job would provision a real tenant per PR (a second full provision). The PR-mode E2E_REQUIRE_LIVE=0 design specifically avoids that today.

Alternative the reviewer/owner may prefer: rely on CP PR #885's merged unit test (it pins the exact docker run env-block rendering bug) and keep E2E Staging Platform Boot fail-loud post-merge (CoE removed, but not added to branch protection). That gets the masking fix and the post-merge blocking on main without the gate ceremony. This PR takes the stronger position (full merge-gate parity with the concierge job) because the #878 class was a rendering bug that a unit test on one repo (CP) cannot guarantee stays covered as the boot path evolves in core.

REMAINING OWNER ACTION (branch protection)

This PR does not touch branch protection. After this PR merges, the owner must add the required status context to core main:

E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (pull_request)

(Same Gitea format as existing required contexts, e.g. CI / all-required (pull_request). The (pull_request) event suffix is the live-BP form; required-contexts.txt stores the event-stripped form.)

Order (per the lint gates):

1. Merge this PR first (removes continue-on-error, adds the context to required-contexts.txt). After merge, the allowlist lists the context but BP does not yet — this is lint-clean: lint-no-coe-on-required only fails on live(BP) − allowlist drift (BP has something the allowlist lacks), never the reverse.
2. Then PATCH branch_protections/main.status_check_contexts to add the context above.

Doing it in this order is mandatory: if BP listed the context before CoE was removed, the next lint-no-coe-on-required run would fail (CoE on a now-required context). Merging this PR removes CoE first, so adding it to BP afterward is always clean.

Lint gates that constrained the approach

• lint-no-coe-on-required — forbids continue-on-error: true on any job emitting a required-contexts.txt context, and fails on live(BP) − allowlist drift. ⇒ CoE removed in the same PR that adds the context; allowlist-before-BP is the safe order. Verified locally: OK: no continue-on-error on any of the 8 required contexts.
• lint-required-no-paths — forbids paths: on the on: block of any required workflow. ⇒ The workflow on: block already has no paths: (cleaned by core#3081); left untouched.
• lint-pre-flip-continue-on-error — blocks a CoE true→false flip without run-log proof of recent green on main, EXCEPT graceful-degrade (no recent runs / log-404 → warn, allow). ⇒ Satisfied via that exemption or by the platform-boot job's recent green push-runs.
• lint-required-context-exists-in-bp (Tier 2g) — requires a directive only for a NEW emitter. ⇒ The platform-boot context string is byte-identical before/after (only CoE + if: changed), so this is not a new emitter; directive updated to bp-required: now required for the post-merge state anyway.
• lint-required-workflows-docker-host-pinned — only applies to workflows running docker. ⇒ This workflow runs curl + a bash harness on ubuntu-latest; no docker. N/A.
• lint-continue-on-error-tracking — every CoE:true needs a fresh (<14d, open) tracker. ⇒ Removing the platform-boot CoE eliminates a tracked directive; other CoE directives untouched. N/A.

Validation run locally: workflow YAML parses, bash -n on the harness passes, shellcheck -S error clean, lint_no_coe_on_required.py exits 0.

🤖 Generated with Claude Code

## What & why (RCA #878 → #885) A prod onboarding outage (06:04–08:09 UTC 2026-06-21) was caused by **molecule-controlplane PR #878**: it rendered the tenant `docker run` env block with a blank line that broke shell `\`-continuation → the image arg was orphaned → `docker run exit=127` → no tenant container → onboarding down. Fixed by **CP PR #885** (deployed 08:09). It escaped pre-merge testing partly because the real-boot e2e (`E2E Staging Platform Boot`) is **advisory** (`continue-on-error: true`) and **never ran on PRs** (`if:` push/dispatch/schedule guard). **Task #48: promote `E2E Staging Platform Boot` to merge-blocking (fail-closed).** ## What changed Mirrors the in-file gating exemplar `e2e-staging-concierge-creates-workspace` (core#3081 / CR2 #12653) exactly. **`.gitea/workflows/e2e-staging-saas.yml` — `e2e-staging-platform-boot` job:** - **Removed `continue-on-error: true`** (was the `mc#2654` mask — Gitea Quirk #10 makes a failed step roll up to `success` under CoE, which is precisely how a broken boot would false-green). - **Removed the `if: push || workflow_dispatch || schedule` guard** so the job runs on `pull_request`. A required context that never fires on PR degrades the merge gate to a silent indefinite `pending` (the failure mode `lint-required-no-paths` / `feedback_path_filtered_workflow_cant_be_required` exist to prevent). - **`E2E_REQUIRE_LIVE: ${{ github.event_name == 'pull_request' && '0' || '1' }}`** — false-green-proof: - `pull_request` → `0`: PRs carry no staging creds; the harness runs a `bash -n` PR-mode self-check and `exit 0`. - `push` / `dispatch` / `schedule` → `1`: the real staging boot runs and **HARD FAILs (`exit 5`)** if it proves no live `provisioned → tenant_online → workspace_online → a2a_roundtrip` lifecycle. - **`Verify admin token present` step** is PR-mode-aware: skips cleanly when `E2E_REQUIRE_LIVE=0` + no token; still hard-errors on a real run. - **Kept the `Teardown safety net (if: always())` step** unchanged. **`tests/e2e/test_staging_full_saas.sh` (shared harness):** - Added a **PR-mode early-exit** (`REQUIRE_LIVE=0 && no admin token` → `bash -n` self-check → `exit 0`), mirroring `test_staging_concierge_creates_workspace_e2e.sh`. - Flipped the admin-token line from `${MOLECULE_ADMIN_TOKEN:?...}` to `${MOLECULE_ADMIN_TOKEN:-}` so the PR lane no longer hard-dies before the self-check; a **non-PR run with no token is still a HARD FAIL** just past the PR-mode block. - **Safe for the sibling `e2e-staging-saas` job**: it keeps its `if:` push/dispatch/schedule guard and `E2E_REQUIRE_LIVE: '1'`, so it never reaches the PR-mode branch. **`.gitea/required-contexts.txt`:** added `E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot` (SSOT), in the same PR that removes CoE, per `lint-no-coe-on-required` (CoE forbidden on any listed context). ## Cost tradeoff (please weigh) This makes **every core PR provision a real staging tenant** — a **second full EC2 provision per PR**, alongside `e2e-staging-concierge-creates-workspace`. Wait: the platform-boot job runs on `pull_request` with `E2E_REQUIRE_LIVE=0` and **no staging creds on PR**, so on PR it does a `bash -n` self-check and exits — **it does NOT provision on PR**. The real provision happens on **push-to-main / dispatch / cron** (as before, but now blocking deploy-to-main rather than advisory). So the incremental cost vs. today is: the platform boot is no longer maskable, and a red platform-boot on main now blocks (it was advisory). > If staging creds were ever wired to PR runs, this job would provision a real tenant per PR (a second full provision). The PR-mode `E2E_REQUIRE_LIVE=0` design specifically avoids that today. **Alternative the reviewer/owner may prefer:** rely on **CP PR #885's merged unit test** (it pins the exact `docker run` env-block rendering bug) and keep `E2E Staging Platform Boot` **fail-loud post-merge** (CoE removed, but not added to branch protection). That gets the masking fix and the post-merge blocking on main without the gate ceremony. This PR takes the stronger position (full merge-gate parity with the concierge job) because the #878 class was a *rendering* bug that a unit test on one repo (CP) cannot guarantee stays covered as the boot path evolves in *core*. ## REMAINING OWNER ACTION (branch protection) This PR does **not** touch branch protection. After this PR merges, the owner must add the required status context to core `main`: ``` E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (pull_request) ``` (Same Gitea format as existing required contexts, e.g. `CI / all-required (pull_request)`. The `(pull_request)` event suffix is the live-BP form; `required-contexts.txt` stores the event-stripped form.) **Order (per the lint gates):** 1. **Merge this PR first** (removes `continue-on-error`, adds the context to `required-contexts.txt`). After merge, the allowlist lists the context but BP does not yet — this is lint-clean: `lint-no-coe-on-required` only fails on `live(BP) − allowlist` drift (BP has something the allowlist lacks), never the reverse. 2. **Then** PATCH `branch_protections/main.status_check_contexts` to add the context above. Doing it in this order is mandatory: if BP listed the context **before** CoE was removed, the next `lint-no-coe-on-required` run would fail (CoE on a now-required context). Merging this PR removes CoE first, so adding it to BP afterward is always clean. ## Lint gates that constrained the approach - **`lint-no-coe-on-required`** — forbids `continue-on-error: true` on any job emitting a `required-contexts.txt` context, and fails on `live(BP) − allowlist` drift. ⇒ CoE removed in the same PR that adds the context; allowlist-before-BP is the safe order. Verified locally: `OK: no continue-on-error on any of the 8 required contexts`. - **`lint-required-no-paths`** — forbids `paths:` on the `on:` block of any required workflow. ⇒ The workflow `on:` block already has no `paths:` (cleaned by core#3081); left untouched. - **`lint-pre-flip-continue-on-error`** — blocks a CoE `true→false` flip without run-log proof of recent green on main, EXCEPT graceful-degrade (no recent runs / log-404 → warn, allow). ⇒ Satisfied via that exemption or by the platform-boot job's recent green push-runs. - **`lint-required-context-exists-in-bp`** (Tier 2g) — requires a directive only for a NEW emitter. ⇒ The platform-boot context string is byte-identical before/after (only CoE + `if:` changed), so this is not a new emitter; directive updated to `bp-required: now required` for the post-merge state anyway. - **`lint-required-workflows-docker-host-pinned`** — only applies to workflows running docker. ⇒ This workflow runs `curl` + a bash harness on `ubuntu-latest`; no docker. N/A. - **`lint-continue-on-error-tracking`** — every CoE:true needs a fresh (<14d, open) tracker. ⇒ Removing the platform-boot CoE eliminates a tracked directive; other CoE directives untouched. N/A. Validation run locally: workflow YAML parses, `bash -n` on the harness passes, `shellcheck -S error` clean, `lint_no_coe_on_required.py` exits 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-21 09:29:09 +00:00

ci(e2e-staging): promote E2E Staging Platform Boot to merge-blocking (fail-closed) — #48 ... 4596e864d3

RCA: molecule-controlplane PR #878 rendered the tenant `docker run` env
block with a blank line that broke shell `\`-continuation, orphaning the
image arg → `docker run exit=127` → no tenant container → prod onboarding
outage 06:04–08:09 UTC 2026-06-21 (fixed by CP #885, deployed 08:09). That
class escaped pre-merge partly because the real-boot e2e is advisory and
never runs on PRs.

This makes the `e2e-staging-platform-boot` job a fail-closed gate, mirroring
the in-file exemplar `e2e-staging-concierge-creates-workspace`:

  - remove `continue-on-error: true` (was the mc#2654 mask)
  - remove the `if:` push/dispatch/schedule guard so the job runs on
    pull_request (a required context that never fires on PR degrades the
    merge gate to a silent indefinite pending — the failure mode
    lint-required-no-paths exists to prevent)
  - E2E_REQUIRE_LIVE: 0 on pull_request (PR-mode self-check, no creds),
    1 on push/dispatch/cron (real staging boot, HARD FAILs exit 5 on a
    run that proves no live lifecycle) — false-green-proof
  - add a PR-mode early-exit to the shared harness test_staging_full_saas.sh
    (REQUIRE_LIVE=0 + no admin token → bash -n self-check → exit 0); flip
    the admin-token `:?` to `:-}` so the PR lane no longer hard-dies before
    the self-check. Safe for the sibling e2e-staging-saas job: it keeps its
    push/dispatch/schedule `if:` guard and REQUIRE_LIVE=1, so it never hits
    the PR-mode branch.
  - keep the Teardown safety net (if: always()) step
  - add the context to .gitea/required-contexts.txt (SSOT) in the same PR
    that removes continue-on-error, per lint-no-coe-on-required (CoE
    forbidden on any listed context).

Lint ordering satisfied: removing CoE + adding to the allowlist is the
lint-clean order (allowlist-superset-of-BP passes lint-no-coe-on-required;
BP-superset-of-allowlist does not). The live branch-protection PATCH is the
remaining OWNER action, to be done AFTER merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

molecule-code-reviewer approved these changes 2026-06-21 09:32:33 +00:00

molecule-code-reviewer left a comment

Member

Copy Link

Copy Source

Reviewed against the in-file gating exemplar (e2e-staging-concierge-creates-workspace): REQUIRE_LIVE=${{ pull_request && '0' || '1' }} matches (PRs lack staging creds → bash-clean self-check; real boot runs post-merge with =1, no longer masked by continue-on-error). Confirmed the platform-boot step invokes the patched tests/e2e/test_staging_full_saas.sh, and the sibling e2e-staging-saas keeps its push-only if-guard so the shared-harness PR-mode early-exit never affects it. SSOT required-contexts.txt updated in the same PR (lint-no-coe-on-required order). Sound, convention-following. LGTM.

Reviewed against the in-file gating exemplar (e2e-staging-concierge-creates-workspace): REQUIRE_LIVE=${{ pull_request && '0' || '1' }} matches (PRs lack staging creds → bash-clean self-check; real boot runs post-merge with =1, no longer masked by continue-on-error). Confirmed the platform-boot step invokes the patched tests/e2e/test_staging_full_saas.sh, and the sibling e2e-staging-saas keeps its push-only if-guard so the shared-harness PR-mode early-exit never affects it. SSOT required-contexts.txt updated in the same PR (lint-no-coe-on-required order). Sound, convention-following. LGTM.

core-security approved these changes 2026-06-21 09:32:34 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Security: PRs get no staging creds (REQUIRE_LIVE=0 self-check only) — no secret exposure on the PR lane; real run is push/dispatch/cron. continue-on-error removal makes a real boot regression fail loud post-merge (was silently masked). No new secret surfaces. LGTM.

Security: PRs get no staging creds (REQUIRE_LIVE=0 self-check only) — no secret exposure on the PR lane; real run is push/dispatch/cron. continue-on-error removal makes a real boot regression fail loud post-merge (was silently masked). No new secret surfaces. LGTM.

core-devops scheduled this pull request to auto merge when all checks succeed 2026-06-21 09:32:36 +00:00

core-devops merged commit 7677264eb8 into main 2026-06-21 09:32:37 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

molecule-code-reviewer

core-security

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3116