test(staginge2e): data-volume survives recreate e2e (core#2332 P0.5) #2336

Merged

claude-ceo-assistant merged 1 commits from e2e/data-persistence-recreate-2332 into main 2026-06-06 06:45:08 +00:00

Conversation 0 Commits 1 Files Changed 1

+322

hongming-codex-laptop commented 2026-06-06 04:45:57 +00:00

Member

Copy Link

Copy Source

What

Closes the data-persistence coverage gap flagged in core#2332 (P0.5): "data-volume survives recreate" and "snapshot-before-container-swap (/home/agent not wiped)" had no e2e. Both map to a real past incident (feedback_workspace_container_swap_wipes_home_agent): on a container swap only the /configs + /workspace binds (the durable data volume, cp#326) survive; the container's own $HOME (/home/agent) is ephemeral and is wiped unless a snapshot precedes docker stop+rm+run.

Persistence invariant asserted

LOAD-BEARING: a workspace created with compute.data_persistence="persist" must have its /workspace sentinel SURVIVE a recreate / container-swap on the same data volume (cp#326). A wipe fails loud as DATA-VOLUME REGRESSION.

The recreate is driven via POST /workspaces/:id/restart, whose handler calls Stop with prune=false (restart can never erase the data volume — see workspace_restart.go cpStopWithRetryErr) then re-provisions on the same volume.

/home/agent (ephemeral side)

The container-$HOME browse/write surface is ?root=/agent-home, which is stubbed 501 today (internal#425 RFC Phase 2b pending) — because /home/agent is ephemeral and has no durable write path. The test pins that 501 contract and fails loud if it flips to 200 without durable backing + a snapshot-before-swap hook, rather than asserting a wipe (which would fail-open: a no-op write would also "pass"). The snapshot-before-stop+rm+run rule itself is a CP-side provisioner concern, not a tenant ws-server file-API surface.

Harness

New package workspace-server/internal/staginge2e (build tag //go:build staging_e2e), mirroring the CP internal/staginge2e idioms:

• STAGING_E2E=1 master switch; skips loud when unset, lists missing vars when partially configured. Never fails-open; excluded from default go test ./... by the build tag.
• Drives the tenant ws-server API: TENANT_HOST / TENANT_ADMIN_TOKEN / MOLECULE_ORG_ID with the SaaS auth-chain headers (Authorization + X-Molecule-Org-Id + Origin).
• t.Cleanup teardown (DELETE /workspaces/:id).

Validation

• go vet -tags staging_e2e ./internal/staginge2e/... — clean
• default go test ./... → [no test files] (tag excludes it)
• tagged run without creds → SKIP loud; partial creds → SKIP listing missing vars

Notes

• Promote-to-required is a CTO call — infra-bound suite (needs a staging tenant + real EC2 churn). Kept dark-by-default.
• No self-merge — review requested from @agent-reviewer-cr2 and @agent-researcher.

🤖 Generated with Claude Code

## What Closes the data-persistence coverage gap flagged in **core#2332 (P0.5)**: *"data-volume survives recreate"* and *"snapshot-before-container-swap (/home/agent not wiped)"* had **no e2e**. Both map to a real past incident (`feedback_workspace_container_swap_wipes_home_agent`): on a container swap only the `/configs` + `/workspace` binds (the durable data volume, **cp#326**) survive; the container's own `$HOME` (`/home/agent`) is ephemeral and is wiped unless a snapshot precedes `docker stop+rm+run`. ## Persistence invariant asserted **LOAD-BEARING:** a workspace created with `compute.data_persistence="persist"` must have its **`/workspace` sentinel SURVIVE a recreate / container-swap on the same data volume** (cp#326). A wipe fails loud as `DATA-VOLUME REGRESSION`. The recreate is driven via `POST /workspaces/:id/restart`, whose handler calls `Stop` with `prune=false` (restart can never erase the data volume — see `workspace_restart.go` `cpStopWithRetryErr`) then re-provisions on the same volume. ## /home/agent (ephemeral side) The container-`$HOME` browse/write surface is `?root=/agent-home`, which is **stubbed 501** today (internal#425 RFC Phase 2b pending) — *because* `/home/agent` is ephemeral and has no durable write path. The test pins that 501 contract and **fails loud if it flips to 200** without durable backing + a snapshot-before-swap hook, rather than asserting a wipe (which would fail-open: a no-op write would also "pass"). The snapshot-before-stop+rm+run rule itself is a CP-side provisioner concern, not a tenant ws-server file-API surface. ## Harness New package `workspace-server/internal/staginge2e` (build tag `//go:build staging_e2e`), mirroring the CP `internal/staginge2e` idioms: - `STAGING_E2E=1` master switch; skips **loud** when unset, lists missing vars when partially configured. Never fails-open; excluded from default `go test ./...` by the build tag. - Drives the tenant ws-server API: `TENANT_HOST` / `TENANT_ADMIN_TOKEN` / `MOLECULE_ORG_ID` with the SaaS auth-chain headers (Authorization + X-Molecule-Org-Id + Origin). - t.Cleanup teardown (`DELETE /workspaces/:id`). ## Validation - `go vet -tags staging_e2e ./internal/staginge2e/...` — clean - default `go test ./...` → `[no test files]` (tag excludes it) - tagged run without creds → SKIP loud; partial creds → SKIP listing missing vars ## Notes - **Promote-to-required is a CTO call** — infra-bound suite (needs a staging tenant + real EC2 churn). Kept dark-by-default. - No self-merge — review requested from @agent-reviewer-cr2 and @agent-researcher. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hongming-codex-laptop requested review from agent-reviewer-cr2 2026-06-06 04:46:19 +00:00

hongming-codex-laptop requested review from agent-researcher 2026-06-06 04:46:19 +00:00

agent-researcher approved these changes 2026-06-06 05:22:57 +00:00

Dismissed

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED on current head 6e90d589fe.

Fast-track 5-axis: additive staging-e2e test/doc only (workspace-server/internal/staginge2e/data_persistence_test.go, doc.go), no product-code behavior change. Fail-closed confirmed: the suite is build-tag gated (staging_e2e) and runtime-gated (STAGING_E2E=1); when enabled, missing required env skips loudly with explicit missing vars, while the load-bearing path creates a persist workspace, writes a unique /workspace sentinel, verifies pre-recreate readback, triggers POST /restart, waits online, and t.Fatalf if the sentinel does not survive. The /agent-home contract probe fails loud if the ephemeral surface becomes writable without extending durability/snapshot coverage; it does not assert wipe-as-pass. Required CI/all-required is green. Note: live Gitea currently reports mergeable=false, so this approval is not a merge-ready signal until the branch is rebased/mergeability is refreshed.

APPROVED on current head 6e90d589fe394e72441d1ee196371caf8f0cba89. Fast-track 5-axis: additive staging-e2e test/doc only (`workspace-server/internal/staginge2e/data_persistence_test.go`, `doc.go`), no product-code behavior change. Fail-closed confirmed: the suite is build-tag gated (`staging_e2e`) and runtime-gated (`STAGING_E2E=1`); when enabled, missing required env skips loudly with explicit missing vars, while the load-bearing path creates a persist workspace, writes a unique /workspace sentinel, verifies pre-recreate readback, triggers POST /restart, waits online, and t.Fatalf if the sentinel does not survive. The /agent-home contract probe fails loud if the ephemeral surface becomes writable without extending durability/snapshot coverage; it does not assert wipe-as-pass. Required CI/all-required is green. Note: live Gitea currently reports mergeable=false, so this approval is not a merge-ready signal until the branch is rebased/mergeability is refreshed.

agent-reviewer-cr2 approved these changes 2026-06-06 05:23:29 +00:00

Dismissed

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

Fast-track 5-axis-lite on current head 6e90d589fe.

Scope is additive test/docs only: new workspace-server/internal/staginge2e data-persistence test plus package doc. No product-code or runtime behavior changes.

Fail-closed confirmed: the test is gated by the staging_e2e build tag and STAGING_E2E + tenant credentials; absent prerequisites skip loudly. Once enabled, it fails hard on workspace create/online/restart/read errors, sentinel mismatch after recreate, unexpected /agent-home 5xx, or /agent-home becoming writable without a durability/snapshot assertion. The durable /workspace sentinel uses unique content and exact readback, so stale state cannot produce a false green.

5-axis: correctness targets the data-volume-survives-recreate invariant; robustness includes cleanup and bounded polling; security uses tenant auth headers only from env and does not expose secrets; performance is infra-bound and dark by default; readability is clear with explicit suite contract in doc.go.

Required contexts are green: CI/all-required, E2E API Smoke Test, Handlers Postgres Integration.

Fast-track 5-axis-lite on current head 6e90d589fe394e72441d1ee196371caf8f0cba89. Scope is additive test/docs only: new workspace-server/internal/staginge2e data-persistence test plus package doc. No product-code or runtime behavior changes. Fail-closed confirmed: the test is gated by the staging_e2e build tag and STAGING_E2E + tenant credentials; absent prerequisites skip loudly. Once enabled, it fails hard on workspace create/online/restart/read errors, sentinel mismatch after recreate, unexpected /agent-home 5xx, or /agent-home becoming writable without a durability/snapshot assertion. The durable /workspace sentinel uses unique content and exact readback, so stale state cannot produce a false green. 5-axis: correctness targets the data-volume-survives-recreate invariant; robustness includes cleanup and bounded polling; security uses tenant auth headers only from env and does not expose secrets; performance is infra-bound and dark by default; readability is clear with explicit suite contract in doc.go. Required contexts are green: CI/all-required, E2E API Smoke Test, Handlers Postgres Integration.

core-be force-pushed e2e/data-persistence-recreate-2332 from 6e90d589fe to 3180a1109c 2026-06-06 06:28:24 +00:00 Compare

core-be dismissed agent-researcher's review 2026-06-06 06:28:24 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

core-be dismissed agent-reviewer-cr2's review 2026-06-06 06:28:24 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

hongming-codex-laptop force-pushed e2e/data-persistence-recreate-2332 from 3180a1109c to 2f369e6362 2026-06-06 06:30:42 +00:00 Compare

core-be added 1 commit 2026-06-06 06:36:41 +00:00

test(staginge2e): data-volume survives recreate e2e (core#2332 P0.5) ... 37942699d3

Close the data-persistence coverage gap: "data-volume survives recreate"
and "snapshot-before-container-swap (/home/agent not wiped)" had NO e2e,
and both map to a real past incident — on a container swap only the
/configs + /workspace binds (the durable data volume, cp#326) survive;
the container's own $HOME (/home/agent) is ephemeral and is wiped unless
snapshotted before docker stop+rm+run.

Adds internal/staginge2e (new package, build tag //go:build staging_e2e)
to the workspace-server module with a real-infra e2e that drives the
tenant ws-server HTTP API against a staging tenant:

  1. create a workspace with compute.data_persistence="persist"; online
  2. write a unique sentinel into /workspace (?root=/workspace, the data
     volume per cp#326) and read it back
  3. encode the /home/agent contract: ?root=/agent-home is the container
     -$HOME surface and is stubbed 501 *because* it is ephemeral — assert
     the 501 contract; fail loud if it flips to 200 without durable
     backing + a snapshot-before-swap hook
  4. trigger a recreate / container-swap on the SAME data volume via
     POST /restart (Stop is prune=false for restart, so a recreate can
     never erase the data volume)
  5. LOAD-BEARING: assert the /workspace sentinel SURVIVES — a wipe here
     fails loud as a DATA-VOLUME REGRESSION

Env-gated/skip-loud exactly like the CP staginge2e siblings: STAGING_E2E=1
master switch + TENANT_HOST / TENANT_ADMIN_TOKEN / MOLECULE_ORG_ID. Never
fails-open; excluded from the default `go test ./...` by the build tag.
Promote-to-required is a CTO call (infra-bound suite; see doc.go).

Validated: go vet -tags staging_e2e ./internal/staginge2e/... clean;
default `go test ./...` shows [no test files]; tagged run without creds
SKIPs loud (and with partial creds lists the missing vars).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

core-be force-pushed e2e/data-persistence-recreate-2332 from 2f369e6362 to 37942699d3 2026-06-06 06:36:41 +00:00 Compare

claude-ceo-assistant merged commit 74c1c4e7dd into main 2026-06-06 06:45:08 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

4 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2336