test(e2e): live staging e2e — reconciler heals a terminated EC2 (core#2261) #2270

Merged

hongming merged 1 commits from feat/core2261-reconciler-live-e2e into main 2026-06-05 01:11:52 +00:00

Conversation 0 Commits 1 Files Changed 2

+689

hongming commented 2026-06-05 01:09:45 +00:00

Owner

Copy Link

Copy Source

What

A live staging E2E that proves the core#2261 instance-state reconciler
(workspace-server/internal/registry/cp_instance_reconciler.go) actually heals
a terminated EC2 against real infra — the real-infra complement to the
deterministic unit tests, which only pin the reconcile logic against fakes.

tests/e2e/test_reconciler_heals_terminated_instance.sh:

1. Provisions a fresh staging org + ONE workspace (same default
   runtime/model + provisioning/token conventions as
   test_staging_full_saas.sh), polls the tenant API until status=online,
   and captures its instance_id.
2. Kills it — aws ec2 terminate-instances on that exact captured
   instance_id (falls back to a slug-tag describe via
   lib/aws_leak_check.sh if the id wasn't surfaced).
3. Asserts the reconciler heals it:
   • PRIMARY (gate, ~180s): the workspace status leaves online — the
     reconciler detected the dead instance via IsRunning and flipped it.
     This is the core#2247 regression guard: a dead instance must NOT keep
     reading online. PRIMARY failing exits 1.
   • SECONDARY (best-effort, ~600s): it auto-reprovisions — status
     returns to online on an instance_id that differs from the
     terminated one (the onOffline → RestartByID existing-volume heal). If
     the reprovision doesn't finish in the bound it's logged clearly but
     does not fail — PRIMARY stands as the gate. A future tightening to a
     hard fail is deliberately one edit away (noted inline).
4. Teardown always — an up-front EXIT/INT/TERM trap deletes the tenant
   and leak-sweeps slug-tagged EC2, so a mid-test failure never orphans a box.

Workflow

.gitea/workflows/e2e-staging-reconciler.yml, modeled on
e2e-staging-saas.yml (same CP_STAGING_ADMIN_API_TOKEN + AWS secrets,
E2E_AWS_TERMINATE_LEAKS=1, "Verify required secrets present" preflight,
belt-and-braces teardown). Triggers: workflow_dispatch + a paths filter on
the reconciler source, the new script, and the libs (so it runs when the
reconciler changes) + a daily schedule.

NON-required initially (continue-on-error: true) — a brand-new live E2E
that provisions/terminates real EC2 should not hard-gate every merge until it
has a green track record. A header note documents the promotion to
branch-required.

Validation

• shellcheck --severity=warning (CI-exact) clean; default-severity clean.
• bash -n parse-clean.
• Bulk shellcheck across all tests/e2e/*.sh clean (no sibling broken).
• lint_cleanup_traps.sh clean; workflow-YAML linter + continue-on-error
  tracker linter clean (job-level continue-on-error references mc#1982).
• The script was NOT executed against staging — it provisions/terminates
  real EC2 and costs money. It runs against staging only in CI.

Refs core#2261, core#2247.

🤖 Generated with Claude Code

## What A **live staging E2E** that proves the core#2261 instance-state reconciler (`workspace-server/internal/registry/cp_instance_reconciler.go`) actually heals a terminated EC2 against **real infra** — the real-infra complement to the deterministic unit tests, which only pin the reconcile logic against fakes. `tests/e2e/test_reconciler_heals_terminated_instance.sh`: 1. Provisions a fresh staging org + ONE workspace (same default runtime/model + provisioning/token conventions as `test_staging_full_saas.sh`), polls the tenant API until `status=online`, and captures its `instance_id`. 2. **Kills it** — `aws ec2 terminate-instances` on that exact captured `instance_id` (falls back to a slug-tag describe via `lib/aws_leak_check.sh` if the id wasn't surfaced). 3. **Asserts the reconciler heals it:** - **PRIMARY (gate, ~180s):** the workspace `status` leaves `online` — the reconciler detected the dead instance via `IsRunning` and flipped it. This is the core#2247 regression guard: a dead instance must NOT keep reading `online`. PRIMARY failing exits 1. - **SECONDARY (best-effort, ~600s):** it auto-reprovisions — `status` returns to `online` on an `instance_id` that **differs** from the terminated one (the `onOffline → RestartByID` existing-volume heal). If the reprovision doesn't finish in the bound it's logged clearly but **does not fail** — PRIMARY stands as the gate. A future tightening to a hard fail is deliberately one edit away (noted inline). 4. **Teardown always** — an up-front `EXIT/INT/TERM` trap deletes the tenant and leak-sweeps slug-tagged EC2, so a mid-test failure never orphans a box. ## Workflow `.gitea/workflows/e2e-staging-reconciler.yml`, modeled on `e2e-staging-saas.yml` (same `CP_STAGING_ADMIN_API_TOKEN` + AWS secrets, `E2E_AWS_TERMINATE_LEAKS=1`, "Verify required secrets present" preflight, belt-and-braces teardown). Triggers: `workflow_dispatch` + a paths filter on the reconciler source, the new script, and the libs (so it runs when the reconciler changes) + a daily `schedule`. **NON-required initially** (`continue-on-error: true`) — a brand-new live E2E that provisions/terminates real EC2 should not hard-gate every merge until it has a green track record. A header note documents the promotion to branch-required. ## Validation - `shellcheck --severity=warning` (CI-exact) clean; default-severity clean. - `bash -n` parse-clean. - Bulk shellcheck across all `tests/e2e/*.sh` clean (no sibling broken). - `lint_cleanup_traps.sh` clean; workflow-YAML linter + continue-on-error tracker linter clean (job-level `continue-on-error` references mc#1982). - The script was **NOT executed against staging** — it provisions/terminates real EC2 and costs money. It runs against staging only in CI. Refs core#2261, core#2247. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hongming added 1 commit 2026-06-05 01:09:45 +00:00

test(e2e): live staging e2e — reconciler heals a terminated EC2 (core#2261) ... 53ec08cbdb

Provisions a real staging workspace, terminates its EC2 out-of-band, and
asserts the core#2261 instance-state reconciler heals it against real infra.

PRIMARY assertion (gate): within ~180s the workspace status leaves 'online'
— the reconciler detected the dead instance via CPProvisioner.IsRunning and
flipped it. A terminated EC2 masquerading as 'online' is exactly the
core#2247 regression this guards.

SECONDARY assertion (best-effort, ~600s): the onOffline -> RestartByID
existing-volume heal brings it back to 'online' on a NEW instance_id. Logged
but non-fatal — PRIMARY is the gate; a future tightening to a hard fail is
one edit away (noted in the script).

Kill primitive: aws ec2 terminate-instances on the captured instance_id
(falls back to slug-tag describe). Teardown is guaranteed by an up-front
EXIT/INT/TERM trap that deletes the tenant + leak-sweeps slug-tagged EC2
(reuses lib/aws_leak_check.sh), so a mid-test failure never orphans a box.

Real-infra complement to the deterministic unit tests
(cp_instance_reconciler.go). New workflow e2e-staging-reconciler.yml fires on
reconciler/script/lib changes + a daily schedule. NON-required initially
(continue-on-error: true) — promote to branch-required once green on main for
a de-flake window.

Refs core#2261, core#2247.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

hongming added the tier:low label 2026-06-05 01:11:28 +00:00

core-security approved these changes 2026-06-05 01:11:30 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Security (core#2261). Real-infra e2e; guaranteed teardown prevents EC2 leaks; AWS-creds preflight; slug-tagged for the orphan sweeper. No prod-runtime change. Approve.

Security (core#2261). Real-infra e2e; guaranteed teardown prevents EC2 leaks; AWS-creds preflight; slug-tagged for the orphan sweeper. No prod-runtime change. Approve.

core-qa approved these changes 2026-06-05 01:11:51 +00:00

core-qa left a comment

Member

Copy Link

Copy Source

QA approve (core#2261 live reconciler e2e).

QA approve (core#2261 live reconciler e2e).

hongming merged commit 71010e618a into main 2026-06-05 01:11:52 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

core-security

core-qa

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label tier:low

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2270