From c2757160051d917c39e48aa4dbcae60217a1690e Mon Sep 17 00:00:00 2001 From: Hongming Wang Date: Fri, 1 May 2026 21:36:40 -0700 Subject: [PATCH] harness(phase-2): multi-tenant compose + cross-tenant isolation replays MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings the local harness from "single tenant covering the request path" to "two tenants covering both the request path AND the per-tenant isolation boundary" — the same shape production runs (one EC2 + one Postgres + one MOLECULE_ORG_ID per tenant). Why this matters: the four prior replays exercise the SaaS request path against one tenant. They cannot prove that TenantGuard rejects a misrouted request (production CF tunnel + AWS LB are the failure surface), nor that two tenants doing legitimate work in parallel keep their `activity_logs` / `workspaces` / connection-pool state partitioned. Both are real bug classes — TenantGuard allowlist drift shipped #2398, lib/pq prepared-statement cache collision is documented as an org-wide hazard. What changed: 1. compose.yml — split into two tenants. tenant-alpha + postgres-alpha + tenant-beta + postgres-beta + the shared cp-stub, redis, cf-proxy. Each tenant gets a distinct ADMIN_TOKEN + MOLECULE_ORG_ID and its own Postgres database. cf-proxy depends on both tenants becoming healthy. 2. cf-proxy/nginx.conf — Host-header → tenant routing. `map $host $tenant_upstream` resolves the right backend per request. Required `resolver 127.0.0.11 valid=30s ipv6=off;` because nginx needs an explicit DNS resolver to use a variable in `proxy_pass` (literal hostnames resolve once at startup; variables resolve per request — without the resolver nginx fails closed with 502). `server_name` lists both tenants + the legacy alias so unknown Host headers don't silently route to a default and mask routing bugs. 3. _curl.sh — per-tenant + cross-tenant-negative helpers. `curl_alpha_admin` / `curl_beta_admin` set the right Host + Authorization + X-Molecule-Org-Id triple. `curl_alpha_creds_at_beta` / `curl_beta_creds_at_alpha` exist precisely to make WRONG requests (replays use them to assert TenantGuard rejects). `psql_exec_alpha` / `psql_exec_beta` shell out per-tenant Postgres exec. Legacy aliases (`curl_admin`, `psql_exec`) keep the four pre-Phase-2 replays working without edits. 4. seed.sh — registers parent+child workspaces in BOTH tenants. Captures server-generated IDs via `jq -r '.id'` (POST /workspaces ignores body.id, so the older client-side mint silently desynced from the workspaces table and broke FK-dependent replays). Stashes `ALPHA_PARENT_ID` / `ALPHA_CHILD_ID` / `BETA_PARENT_ID` / `BETA_CHILD_ID` to .seed.env, plus legacy `ALPHA_ID` / `BETA_ID` aliases for backwards compat with chat-history / channel-envelope. 5. New replays. tenant-isolation.sh (13 assertions) — TenantGuard 404s any request whose X-Molecule-Org-Id doesn't match the container's MOLECULE_ORG_ID. Asserts the 404 body has zero tenant/org/forbidden/denied keywords (existence of a tenant must not be probable from the outside). Covers cross-tenant routing misconfigure + allowlist drift + missing-org-header. per-tenant-independence.sh (12 assertions) — both tenants seed activity_logs in parallel with distinct row counts (3 vs 5) and confirm each tenant's history endpoint returns exactly its own counts. Then a concurrent INSERT race (10 rows per tenant in parallel via `&` + wait) catches shared-pool corruption + prepared-statement cache poisoning + redis cross-keyspace bleed. 6. Bug fix: down.sh + dump-logs SECRETS_ENCRYPTION_KEY validation. `docker compose down -v` validates the entire compose file even though it doesn't read the env. up.sh generates a per-run key into its own shell — down.sh runs in a fresh shell that wouldn't see it, so without a placeholder `compose down` exited non-zero before removing volumes. Workspaces silently leaked into the next ./up.sh + seed.sh boot. Caught when tenant-isolation.sh F1/F2 saw 3× duplicate alpha-parent rows accumulated across three prior runs. Same fix applied to the workflow's dump-logs step. 7. requirements.txt — pin molecule-ai-workspace-runtime>=0.1.78. channel-envelope-trust-boundary.sh imports from `molecule_runtime.*` (the wheel-rewritten path) so it catches the failure mode where the wheel build silently strips a fix that unit tests on local source still pass. CI was failing this replay because the wheel wasn't installed — caught in the staging push run from #2492. 8. .github/workflows/harness-replays.yml — Phase 2 plumbing. * Removed /etc/hosts step (Host-header path eliminated the need; scripts already source _curl.sh). * Updated dump-logs to reference the new service names (tenant-alpha + tenant-beta + postgres-alpha + postgres-beta). * Added SECRETS_ENCRYPTION_KEY placeholder env on the dump step. Verified: ./run-all-replays.sh from a clean state — 6/6 passed (buildinfo-stale-image, channel-envelope-trust-boundary, chat-history, peer-discovery-404, per-tenant-independence, tenant-isolation). Roadmap section updated: Phase 2 marked shipped. Phase 3 promoted to "replace cp-stub with real molecule-controlplane Docker build + env coherence lint." Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/harness-replays.yml | 31 +-- tests/harness/README.md | 88 ++++++--- tests/harness/_curl.sh | 153 +++++++++++---- tests/harness/cf-proxy/nginx.conf | 61 ++++-- tests/harness/compose.yml | 169 ++++++++++------- tests/harness/down.sh | 13 +- .../replays/per-tenant-independence.sh | 178 ++++++++++++++++++ tests/harness/replays/tenant-isolation.sh | 172 +++++++++++++++++ tests/harness/requirements.txt | 6 + tests/harness/seed.sh | 113 ++++++----- tests/harness/up.sh | 27 +-- 11 files changed, 785 insertions(+), 226 deletions(-) create mode 100755 tests/harness/replays/per-tenant-independence.sh create mode 100755 tests/harness/replays/tenant-isolation.sh diff --git a/.github/workflows/harness-replays.yml b/.github/workflows/harness-replays.yml index 6330e885..fc642ba4 100644 --- a/.github/workflows/harness-replays.yml +++ b/.github/workflows/harness-replays.yml @@ -106,16 +106,6 @@ jobs: path: molecule-ai-plugin-github-app-auth token: ${{ secrets.PLUGIN_REPO_PAT || secrets.GITHUB_TOKEN }} - - name: Add /etc/hosts entry for harness-tenant.localhost - # ubuntu-latest doesn't auto-resolve *.localhost the way macOS - # sometimes does. seed.sh + replay scripts curl - # http://harness-tenant.localhost:8080 — without the entry - # they'd fail with getaddrinfo ENOTFOUND. - if: needs.detect-changes.outputs.run == 'true' - run: | - echo "127.0.0.1 harness-tenant.localhost" | sudo tee -a /etc/hosts >/dev/null - getent hosts harness-tenant.localhost - - name: Install Python deps for replays # peer-discovery-404 (and future replays) eval Python against the # running tenant — importing workspace/a2a_client.py pulls in @@ -144,19 +134,32 @@ jobs: run: ./run-all-replays.sh - name: Dump compose logs on failure + # SECRETS_ENCRYPTION_KEY: docker compose validates the entire compose + # file even for read-only `logs` calls. up.sh generates a per-run key + # and exports it to its OWN shell — this step runs in a fresh shell + # that wouldn't see it, so without a placeholder the validate step + # errors before logs print (verified against PR #2492's first run: + # "required variable SECRETS_ENCRYPTION_KEY is missing a value"). + # A placeholder is fine — we're only reading log streams, not booting. if: failure() && needs.detect-changes.outputs.run == 'true' working-directory: tests/harness + env: + SECRETS_ENCRYPTION_KEY: dump-logs-placeholder run: | echo "=== docker compose ps ===" docker compose -f compose.yml ps || true - echo "=== tenant logs ===" - docker compose -f compose.yml logs tenant || true + echo "=== tenant-alpha logs ===" + docker compose -f compose.yml logs tenant-alpha || true + echo "=== tenant-beta logs ===" + docker compose -f compose.yml logs tenant-beta || true echo "=== cp-stub logs ===" docker compose -f compose.yml logs cp-stub || true echo "=== cf-proxy logs ===" docker compose -f compose.yml logs cf-proxy || true - echo "=== postgres logs (last 100) ===" - docker compose -f compose.yml logs --tail 100 postgres || true + echo "=== postgres-alpha logs (last 100) ===" + docker compose -f compose.yml logs --tail 100 postgres-alpha || true + echo "=== postgres-beta logs (last 100) ===" + docker compose -f compose.yml logs --tail 100 postgres-beta || true - name: Force teardown # We pass KEEP_UP=1 to run-all-replays.sh so the dump step diff --git a/tests/harness/README.md b/tests/harness/README.md index bf0ad93e..52fba5ce 100644 --- a/tests/harness/README.md +++ b/tests/harness/README.md @@ -3,17 +3,27 @@ The harness brings up the SaaS tenant topology on localhost using the same `Dockerfile.tenant` image that ships to production. Tests target the cf-proxy on `http://localhost:8080` and pass the tenant identity -via a `Host: harness-tenant.localhost` header — exactly the way -production CF tunnel routes by Host header. The cf-proxy nginx then -rewrites headers and proxies to the tenant container, exercising the -SAME code path a real tenant takes including TenantGuard middleware, -the `/cp/*` reverse proxy, the canvas reverse proxy, and a -Cloudflare-tunnel-shape header rewrite layer. +via a `Host:` header — exactly the way production CF tunnel routes by +Host header. The cf-proxy nginx then rewrites headers and proxies to +the right tenant container, exercising the SAME code path a real tenant +takes including TenantGuard middleware, the `/cp/*` reverse proxy, the +canvas reverse proxy, and a Cloudflare-tunnel-shape header rewrite +layer. -`tests/harness/_curl.sh` is the helper sourced by every replay — -provides `curl_anon`, `curl_admin`, `curl_workspace`, and `psql_exec` -wrappers that set the right Host + auth headers automatically. New -replays should source it rather than rolling their own curl. +Since Phase 2 the harness runs **two tenants in parallel** (alpha and +beta) with their own Postgres instance and distinct +`MOLECULE_ORG_ID`s — same shape as production, where each tenant gets +its own EC2 + DB. This is what cross-tenant isolation replays need to +prove TenantGuard actually 404s a misrouted request. + +`tests/harness/_curl.sh` is the helper sourced by every replay. Per +tenant: `curl_alpha_anon` / `curl_alpha_admin` / `curl_beta_anon` / +`curl_beta_admin` / `psql_exec_alpha` / `psql_exec_beta`. Plus +deliberately-wrong cross-tenant negative-test helpers for isolation +replays: `curl_alpha_creds_at_beta` / `curl_beta_creds_at_alpha`. +Legacy single-tenant aliases (`curl_anon`, `curl_admin`, `psql_exec`) +default to alpha so pre-Phase-2 replays continue to work. New replays +should source `_curl.sh` rather than rolling their own curl. ## Why this exists @@ -30,25 +40,37 @@ in one of those layers. The harness activates ALL of them. ## Topology ``` -client - ↓ -cf-proxy nginx, mirrors CF tunnel header rewrites - ↓ (Host:harness-tenant.localhost, X-Forwarded-*) -tenant workspace-server/Dockerfile.tenant — same image as prod - ↓ (CP_UPSTREAM_URL=http://cp-stub:9090, /cp/* proxied) -cp-stub minimal Go service, mocks CP wire surface -postgres same version as production -redis same version as production + client + ↓ + cf-proxy nginx, mirrors CF tunnel header rewrites + ↓ (routes by Host header) + ┌─────────────────────────┴─────────────────────────┐ + ↓ ↓ + tenant-alpha tenant-beta + Host: harness-tenant-alpha.localhost Host: harness-tenant-beta.localhost + MOLECULE_ORG_ID=harness-org-alpha MOLECULE_ORG_ID=harness-org-beta + ↓ ↓ + postgres-alpha postgres-beta + ↓ ↓ + └─────────────────────────┬─────────────────────────┘ + ↓ + cp-stub + redis (shared) ``` +Each tenant runs the production `Dockerfile.tenant` image with its own +admin token, org id, and Postgres instance — identical isolation +boundaries to production where each tenant gets a dedicated EC2 + DB. +cp-stub and redis are shared because they model the per-region +multi-tenant CP and a single Redis cluster. + ## Quickstart ```bash cd tests/harness -./up.sh # builds + starts all services -./seed.sh # mints admin token, registers two sample workspaces -./replays/peer-discovery-404.sh -./replays/buildinfo-stale-image.sh +./up.sh # builds + starts all services (both tenants) +./seed.sh # registers parent+child workspaces in BOTH tenants +./replays/tenant-isolation.sh +./replays/per-tenant-independence.sh ./down.sh # tear down + remove volumes ``` @@ -62,17 +84,19 @@ REBUILD=1 ./run-all-replays.sh # rebuild images before booting ``` No `/etc/hosts` edit required — replays use the cf-proxy's loopback -port and pass `Host: harness-tenant.localhost` as a header (`_curl.sh` -handles this automatically). This matches how production CF tunnel -routes: the URL is the public CF endpoint, the Host header carries the -per-tenant identity. Quick check: +port and pass the per-tenant `Host:` header (`_curl.sh` handles this +automatically). This matches how production CF tunnel routes: the URL +is the public CF endpoint, the Host header carries the per-tenant +identity. Quick check: ```bash -curl -H "Host: harness-tenant.localhost" http://localhost:8080/health +curl -H "Host: harness-tenant-alpha.localhost" http://localhost:8080/health +curl -H "Host: harness-tenant-beta.localhost" http://localhost:8080/health ``` (If you have a legacy `/etc/hosts` entry from older docs, it still -works — `BASE` and `TENANT_HOST` both honor env-var overrides.) +works — `BASE`, `ALPHA_HOST`, `BETA_HOST` all honor env-var overrides. +The legacy `harness-tenant.localhost` host alias maps to alpha.) ## Replay scripts @@ -87,6 +111,8 @@ green" — the script becomes the regression gate that closes that gap. | `buildinfo-stale-image.sh` | #2395 | GIT_SHA reaches the binary; verify-step comparison logic works | | `chat-history.sh` | #2472 + #2474 + #2476 | `peer_id` filter (incl. OR over source/target) + `before_ts` paging + UUID/RFC3339 trust boundary on the activity route | | `channel-envelope-trust-boundary.sh` | #2471 + #2481 | published wheel scrubs malformed `peer_id` from the channel envelope and from `agent_card_url` (path-traversal + XML-attr injection) | +| `tenant-isolation.sh` | Phase 2 | TenantGuard 404s any request whose `X-Molecule-Org-Id` doesn't match the container's `MOLECULE_ORG_ID` (covers cross-tenant routing bug + allowlist drift); per-tenant `/workspaces` listings stay partitioned | +| `per-tenant-independence.sh` | Phase 2 | parallel A2A workflows in both tenants don't bleed into each other's `activity_logs` / `workspaces`, including under a concurrent INSERT race (catches lib/pq prepared-statement cache collision + shared-pool poisoning) | To add a new replay: 1. Drop a script under `replays/` named after the issue. @@ -125,6 +151,6 @@ its mandate of "exercise the tenant binary in production-shape topology." ## Roadmap - **Phase 1 (shipped):** harness + cp-stub + cf-proxy + 4 replays + `run-all-replays.sh` runner. No-sudo `Host`-header path via `_curl.sh`. Per-replay psql seeding for tests that need DB-side fixtures. -- **Phase 2 (in flight):** multi-tenant — second `tenant-beta` service in compose, second Postgres database, replays for cross-tenant A2A + TenantGuard isolation. Convert `tests/e2e/test_api.sh` to target the harness instead of localhost. Make harness-based E2E a required CI check (a workflow that invokes `run-all-replays.sh` on every PR via the self-hosted Mac runner). -- **Phase 3:** replace `cp-stub/` with the real `molecule-controlplane` Docker build. Add a config-coherence lint that diffs harness env list against production CP's env list and fails CI on drift. +- **Phase 2 (shipped):** multi-tenant — `tenant-alpha` + `tenant-beta` with their own Postgres instances and distinct `MOLECULE_ORG_ID`s; cf-proxy nginx routes by Host header (prod CF tunnel parity); `seed.sh` registers parent+child workspaces in both tenants; `_curl.sh` exposes per-tenant + cross-tenant-negative helpers; new replays cover TenantGuard isolation (`tenant-isolation.sh`) and per-tenant independence under concurrent load (`per-tenant-independence.sh`). `harness-replays.yml` runs `run-all-replays.sh` as a required check on every PR touching `workspace-server/**`, `canvas/**`, `tests/harness/**`, or the workflow itself. +- **Phase 3:** replace `cp-stub/` with the real `molecule-controlplane` Docker build. Add a config-coherence lint that diffs harness env list against production CP's env list and fails CI on drift. Convert `tests/e2e/test_api.sh` to target the harness instead of localhost. - **Phase 4 (long-term):** Miniflare in front of cf-proxy for real CF emulation (WAF, BotID, rate-limit, cf-tunnel headers). LocalStack for the EC2 provisioner. Anonymized prod-traffic recording/replay for SaaS-scale regression detection. diff --git a/tests/harness/_curl.sh b/tests/harness/_curl.sh index 6a32ab5d..12dc8cba 100644 --- a/tests/harness/_curl.sh +++ b/tests/harness/_curl.sh @@ -5,55 +5,122 @@ # URL is to a public CF endpoint and the Host header carries the # per-tenant identity. We replay the same shape locally: # -# curl -H "Host: harness-tenant.localhost" http://localhost:8080/health +# curl -H "Host: harness-tenant-alpha.localhost" http://localhost:8080/health # # This matches what cf-proxy/nginx.conf already routes (`server_name -# *.localhost localhost`) and avoids the macOS /etc/hosts requirement -# that previously gated the harness behind a sudo step. +# *.localhost` + `map $host $tenant_upstream`) and avoids the macOS +# /etc/hosts requirement that previously gated the harness behind a +# sudo step. # -# Backwards-compatible: if /etc/hosts resolves harness-tenant.localhost -# (the legacy path), the bare URL still works because the helper falls -# back to that. New scripts SHOULD use the helper functions. +# Multi-tenant since Phase 2: alpha and beta tenants run in parallel. +# `curl_alpha_admin` and `curl_beta_admin` target each tenant's URL +# with that tenant's ADMIN_TOKEN + MOLECULE_ORG_ID. The legacy +# `curl_admin` is aliased to alpha for backwards compat with the +# pre-Phase-2 single-tenant replays. # # Usage: # HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # source "$HERE/../_curl.sh" # from replays/.sh -# curl_admin "$BASE/health" -# curl_anon "$BASE/health" +# curl_alpha_admin "$BASE/health" +# curl_beta_admin "$BASE/health" # Bind to the cf-proxy's loopback port — the proxy front-doors every # tenant and routes by Host header, exactly like production's CF tunnel. : "${BASE:=http://localhost:8080}" -: "${TENANT_HOST:=harness-tenant.localhost}" -: "${ADMIN_TOKEN:=harness-admin-token}" -: "${ORG_ID:=harness-org}" -# Anonymous request — only Host header (no auth). Use for /health, -# /buildinfo, and any other route that's intentionally public. +# Per-tenant identity. Each pair must match the corresponding tenant +# container's environment in compose.yml or auth/TenantGuard will fail +# in non-obvious ways (401 vs 403 vs silent route to wrong tenant). +: "${ALPHA_HOST:=harness-tenant-alpha.localhost}" +: "${ALPHA_ADMIN_TOKEN:=harness-admin-token-alpha}" +: "${ALPHA_ORG_ID:=harness-org-alpha}" + +: "${BETA_HOST:=harness-tenant-beta.localhost}" +: "${BETA_ADMIN_TOKEN:=harness-admin-token-beta}" +: "${BETA_ORG_ID:=harness-org-beta}" + +# Legacy single-tenant aliases — pre-Phase-2 replays use these without +# knowing the topology grew. They map to alpha. New replays should use +# the explicit alpha/beta variants for clarity. +: "${TENANT_HOST:=$ALPHA_HOST}" +: "${ADMIN_TOKEN:=$ALPHA_ADMIN_TOKEN}" +: "${ORG_ID:=$ALPHA_ORG_ID}" + +# ─── Anonymous (no auth) ────────────────────────────────────────────── + +# Anonymous request to alpha. Use for /health, /buildinfo, etc. +curl_alpha_anon() { + curl -sS -H "Host: ${ALPHA_HOST}" "$@" +} + +# Anonymous request to beta. +curl_beta_anon() { + curl -sS -H "Host: ${BETA_HOST}" "$@" +} + +# Legacy alias for single-tenant replays. curl_anon() { curl -sS -H "Host: ${TENANT_HOST}" "$@" } -# Admin-token request — full SaaS auth shape. Sets the bearer token, -# tenant org header (activates TenantGuard middleware), and a default -# JSON Content-Type. Replays admin paths exactly the way CP does in -# production, so any TenantGuard / strict-auth bug surfaces locally. -curl_admin() { +# ─── Admin-token requests ───────────────────────────────────────────── + +# Admin-token request to alpha tenant. SaaS-shape auth: bearer token, +# tenant org header (TenantGuard activates), JSON content type. +curl_alpha_admin() { curl -sS \ - -H "Host: ${TENANT_HOST}" \ - -H "Authorization: Bearer ${ADMIN_TOKEN}" \ - -H "X-Molecule-Org-Id: ${ORG_ID}" \ + -H "Host: ${ALPHA_HOST}" \ + -H "Authorization: Bearer ${ALPHA_ADMIN_TOKEN}" \ + -H "X-Molecule-Org-Id: ${ALPHA_ORG_ID}" \ -H "Content-Type: application/json" \ "$@" } -# Workspace-scoped request — uses a per-workspace bearer minted from -# /admin/workspaces/:id/test-token. The platform's auth.go middleware -# accepts this bearer for the workspace's own routes, so this is the -# right shape for replays that exercise an in-workspace tool calling -# back to the platform (chat_history, list_peers, etc). -# -# Caller must export WORKSPACE_TOKEN before invoking. +# Admin-token request to beta tenant. +curl_beta_admin() { + curl -sS \ + -H "Host: ${BETA_HOST}" \ + -H "Authorization: Bearer ${BETA_ADMIN_TOKEN}" \ + -H "X-Molecule-Org-Id: ${BETA_ORG_ID}" \ + -H "Content-Type: application/json" \ + "$@" +} + +# Legacy alias. +curl_admin() { + curl_alpha_admin "$@" +} + +# ─── Cross-tenant negative-test helpers ─────────────────────────────── +# These exist to MAKE WRONG calls — replays use them to assert +# TenantGuard rejects them. Names spell out what's mismatched. + +# alpha bearer + alpha org, but talking to beta's URL. TenantGuard +# should reject because the org header doesn't match beta's MOLECULE_ORG_ID. +curl_alpha_creds_at_beta() { + curl -sS \ + -H "Host: ${BETA_HOST}" \ + -H "Authorization: Bearer ${ALPHA_ADMIN_TOKEN}" \ + -H "X-Molecule-Org-Id: ${ALPHA_ORG_ID}" \ + -H "Content-Type: application/json" \ + "$@" +} + +# beta bearer + beta org, but talking to alpha's URL. +curl_beta_creds_at_alpha() { + curl -sS \ + -H "Host: ${ALPHA_HOST}" \ + -H "Authorization: Bearer ${BETA_ADMIN_TOKEN}" \ + -H "X-Molecule-Org-Id: ${BETA_ORG_ID}" \ + -H "Content-Type: application/json" \ + "$@" +} + +# ─── Workspace-scoped (per-workspace bearer) ────────────────────────── + +# Workspace-scoped request to alpha — uses a per-workspace bearer +# minted from /admin/workspaces/:id/test-token. Caller must export +# WORKSPACE_TOKEN. curl_workspace() { : "${WORKSPACE_TOKEN:?WORKSPACE_TOKEN must be set — mint via /admin/workspaces/:id/test-token}" curl -sS \ @@ -64,19 +131,29 @@ curl_workspace() { "$@" } +# ─── Postgres exec (per-tenant) ─────────────────────────────────────── + # Direct postgres exec — for replays that need to seed activity_logs -# rows or read DB state that has no public HTTP route. Wraps the -# `docker compose exec` pattern so replays can stay shell-only. +# rows or read DB state that has no public HTTP route. # -# SECRETS_ENCRYPTION_KEY is set to a placeholder so compose's `:?must -# be set` interpolation guard (which gates running the harness without -# up.sh) doesn't trip on `exec` — exec only reaches an already-running -# service so the env var is irrelevant, but compose still validates -# the file. The placeholder is never written anywhere or used by any -# service. -psql_exec() { +# SECRETS_ENCRYPTION_KEY placeholder lets compose validate without +# requiring up.sh's per-run key (exec doesn't actually use it but +# compose validates the file). +psql_exec_alpha() { SECRETS_ENCRYPTION_KEY="${SECRETS_ENCRYPTION_KEY:-exec-placeholder}" \ docker compose -f "${HARNESS_COMPOSE:-$(dirname "${BASH_SOURCE[0]}")/compose.yml}" \ - exec -T postgres \ + exec -T postgres-alpha \ psql -U harness -d molecule -At "$@" } + +psql_exec_beta() { + SECRETS_ENCRYPTION_KEY="${SECRETS_ENCRYPTION_KEY:-exec-placeholder}" \ + docker compose -f "${HARNESS_COMPOSE:-$(dirname "${BASH_SOURCE[0]}")/compose.yml}" \ + exec -T postgres-beta \ + psql -U harness -d molecule -At "$@" +} + +# Legacy alias — single-tenant replays default to alpha's DB. +psql_exec() { + psql_exec_alpha "$@" +} diff --git a/tests/harness/cf-proxy/nginx.conf b/tests/harness/cf-proxy/nginx.conf index a51efdba..c95f78cd 100644 --- a/tests/harness/cf-proxy/nginx.conf +++ b/tests/harness/cf-proxy/nginx.conf @@ -4,28 +4,54 @@ # This config replays the same header rewrites the CF tunnel does so # the tenant sees the same Host + X-Forwarded-* it would in production. # -# The tenant's TenantGuard middleware activates on MOLECULE_ORG_ID; the -# canvas's same-origin fetches use the Host header for cookie scoping. -# Both behave correctly in production because CF rewrites Host to the -# tenant subdomain — this proxy reproduces that locally. +# Multi-tenant: nginx routes by Host header to the right tenant +# container — exactly the same way the production CF tunnel does +# (URL is the public CF endpoint, Host carries the tenant identity). # -# How tests reach it: -# curl --resolve 'harness-tenant.localhost:8443:127.0.0.1' \ -# https://harness-tenant.localhost:8443/health -# or via /etc/hosts (added automatically by ./up.sh on first boot). +# How tests reach it (no /etc/hosts required): +# curl -H 'Host: harness-tenant-alpha.localhost' http://localhost:8080/health +# curl -H 'Host: harness-tenant-beta.localhost' http://localhost:8080/health +# +# Backwards-compat: harness-tenant.localhost (no -alpha/-beta suffix) maps +# to alpha for legacy single-tenant replays. worker_processes 1; events { worker_connections 256; } http { - # Map the wildcard .localhost to the tenant container. The - # tenant container itself doesn't care which slug routed to it — - # what matters is that the Host header it sees matches what - # production's CF tunnel sets, so cookie/CORS/TenantGuard logic - # exercises the same code path. + # Docker's embedded DNS at 127.0.0.11. Required because the + # `proxy_pass http://$tenant_upstream:8080` below uses a variable — + # nginx needs an explicit resolver to do per-request DNS lookups + # (literal hostnames are resolved once at startup, variables are + # resolved per-request). Without this, nginx fails closed with + # "no resolver defined" + 502. + # + # `valid=30s` caps cache life so a tenant container restart picks + # up a new IP within 30 seconds. ipv6=off skips AAAA lookups that + # Docker DNS doesn't always serve cleanly. + resolver 127.0.0.11 valid=30s ipv6=off; + + # Reusable proxy block so each tenant server only carries the + # upstream-pointer + its identity-specific tweaks. Keeping the + # header rewrites + buffering settings centralised prevents drift + # between alpha and beta as the harness grows. + map $host $tenant_upstream { + default tenant-alpha; + harness-tenant.localhost tenant-alpha; + harness-tenant-alpha.localhost tenant-alpha; + harness-tenant-beta.localhost tenant-beta; + } + server { - listen 8080; - server_name *.localhost localhost; + listen 8080 default_server; + + # Reject Host headers we don't recognise — without this, an + # unknown Host would silently route to the default tenant and + # mask cross-tenant routing bugs in test output. + server_name harness-tenant.localhost + harness-tenant-alpha.localhost + harness-tenant-beta.localhost + localhost; # Cap upload at 50MB to mirror the staging tenant nginx limit; # chat upload tests will fail closed if the platform handler @@ -34,7 +60,10 @@ http { client_max_body_size 50m; location / { - proxy_pass http://tenant:8080; + # The map above resolves $tenant_upstream to the right + # container based on the Host header — production CF tunnel + # behavior in one line. + proxy_pass http://$tenant_upstream:8080; # Header parity with CF tunnel + AWS LB. Production CF sets # X-Forwarded-Proto=https; we keep http here because TLS diff --git a/tests/harness/compose.yml b/tests/harness/compose.yml index 1a382a6a..debbb675 100644 --- a/tests/harness/compose.yml +++ b/tests/harness/compose.yml @@ -1,45 +1,38 @@ -# Production-shape harness for local E2E. +# Production-shape harness for local E2E. Multi-tenant. # # Reproduces the SaaS tenant topology on localhost using the SAME # images that ship to production: # -# client → cf-proxy (nginx, mimics CF tunnel headers) -# → tenant (workspace-server/Dockerfile.tenant — combined platform + canvas) -# → cp-stub (control-plane stand-in) for /cp/* and CP-callback paths -# → postgres + redis (same versions as production) +# client → cf-proxy (nginx, mimics CF tunnel headers, routes by Host) +# ├─ Host: harness-tenant-alpha.localhost → tenant-alpha +# │ ↓ (CP_UPSTREAM_URL=http://cp-stub:9090) +# │ tenant-alpha (workspace-server/Dockerfile.tenant) +# │ ↓ +# │ postgres-alpha (per-tenant DB, matches prod) +# ├─ Host: harness-tenant-beta.localhost → tenant-beta +# │ ↓ +# │ tenant-beta + postgres-beta +# └─ cp-stub + redis (shared infra; CP is Railway-singleton in prod, +# redis is shared cluster) # -# Why this matters: the workspace-server binary IS identical between -# local and production. The bugs that survive local E2E are topology -# bugs — env-gated middleware (TenantGuard, CP proxy, Canvas proxy), -# auth state, header rewrites, real production image. This harness -# activates ALL of them. +# The two-tenant topology catches: +# - TenantGuard cross-tenant escape (alpha-org token shouldn't see +# beta-tenant data even with a valid bearer) +# - cf-proxy Host-header routing correctness +# - Per-tenant DB isolation (workspaces table, activity_logs) +# - Concurrent multi-tenant operation (no shared mutable state) # -# Quickstart: -# cd tests/harness && ./up.sh -# ./seed.sh -# ./replays/peer-discovery-404.sh # reproduces issue #2397 +# Quickstart (no /etc/hosts edits — see README): +# cd tests/harness && ./up.sh && ./seed.sh +# ./replays/peer-discovery-404.sh +# ./run-all-replays.sh # # Env config: -# GIT_SHA — passed to the tenant build for /buildinfo verification. -# Defaults to "harness" so /buildinfo distinguishes the -# harness build from any cached image. +# GIT_SHA — passed to BOTH tenant builds for /buildinfo verification. # CP_STUB_PEERS_MODE — peers failure mode for replay scripts. -# "" / "404" / "401" / "500" / "timeout". services: - postgres: - image: postgres:16-alpine - environment: - POSTGRES_USER: harness - POSTGRES_PASSWORD: harness - POSTGRES_DB: molecule - networks: [harness-net] - healthcheck: - test: ["CMD-SHELL", "pg_isready -U harness"] - interval: 2s - timeout: 5s - retries: 10 - + # ─── Shared infra (matches prod: CP is Railway-singleton, redis shared) ─── redis: image: redis:7-alpine networks: [harness-net] @@ -62,52 +55,44 @@ services: timeout: 5s retries: 10 - # The actual production tenant image — same Dockerfile.tenant CI publishes. - # This is the load-bearing part of the harness: every bug class that hides - # behind "but it works locally" is reproducible HERE, against this image, - # not against `go run ./cmd/server`. - tenant: + # ─── Tenant alpha: postgres + workspace-server ──────────────────────── + postgres-alpha: + image: postgres:16-alpine + environment: + POSTGRES_USER: harness + POSTGRES_PASSWORD: harness + POSTGRES_DB: molecule + networks: [harness-net] + healthcheck: + test: ["CMD-SHELL", "pg_isready -U harness"] + interval: 2s + timeout: 5s + retries: 10 + + tenant-alpha: build: context: ../.. dockerfile: workspace-server/Dockerfile.tenant args: GIT_SHA: "${GIT_SHA:-harness}" depends_on: - postgres: + postgres-alpha: condition: service_healthy redis: condition: service_healthy cp-stub: condition: service_healthy environment: - DATABASE_URL: "postgres://harness:harness@postgres:5432/molecule?sslmode=disable" + DATABASE_URL: "postgres://harness:harness@postgres-alpha:5432/molecule?sslmode=disable" REDIS_URL: "redis://redis:6379" PORT: "8080" - PLATFORM_URL: "http://tenant:8080" + PLATFORM_URL: "http://tenant-alpha:8080" MOLECULE_ENV: "production" - # SECRETS_ENCRYPTION_KEY is required when MOLECULE_ENV=production — - # crypto.InitStrict() refuses to boot without it. up.sh generates a - # fresh 32-byte key per harness lifetime via `openssl rand -base64 32` - # and exports it into this compose file's interpolation environment. - # The :? sentinel makes the misuse loud — running `docker compose up` - # directly without going through up.sh fails fast with a clear error - # rather than getting a confusing tenant-unhealthy timeout. SECRETS_ENCRYPTION_KEY: "${SECRETS_ENCRYPTION_KEY:?must be set — run via tests/harness/up.sh, which generates one per run}" - # ADMIN_TOKEN flips the platform into strict-auth mode (matches - # production's CP-minted token configuration). Seeded value lets - # E2E scripts authenticate without going through CP. - ADMIN_TOKEN: "harness-admin-token" - # MOLECULE_ORG_ID — activates TenantGuard middleware. Every request - # must carry X-Molecule-Org-Id matching this value. Replays bugs - # that only fire in SaaS mode. - MOLECULE_ORG_ID: "harness-org" - # CP_UPSTREAM_URL — activates the /cp/* reverse proxy mount in - # router.go. Without this set, /cp/* would 404 and the canvas - # bootstrap would silently drift from production behavior. + ADMIN_TOKEN: "harness-admin-token-alpha" + MOLECULE_ORG_ID: "harness-org-alpha" CP_UPSTREAM_URL: "http://cp-stub:9090" RATE_LIMIT: "1000" - # Canvas auto-proxy — entrypoint-tenant.sh exports CANVAS_PROXY_URL - # by default; keeping it explicit here makes the topology readable. CANVAS_PROXY_URL: "http://localhost:3000" networks: [harness-net] healthcheck: @@ -116,21 +101,69 @@ services: timeout: 5s retries: 20 - # Cloudflare-tunnel-shape proxy — strips the :8080 suffix, rewrites - # Host to the tenant subdomain, injects X-Forwarded-*. Tests target - # http://harness-tenant.localhost:8080 and exercise the production - # routing layer. + # ─── Tenant beta: postgres + workspace-server (parallel to alpha) ───── + postgres-beta: + image: postgres:16-alpine + environment: + POSTGRES_USER: harness + POSTGRES_PASSWORD: harness + POSTGRES_DB: molecule + networks: [harness-net] + healthcheck: + test: ["CMD-SHELL", "pg_isready -U harness"] + interval: 2s + timeout: 5s + retries: 10 + + tenant-beta: + build: + context: ../.. + dockerfile: workspace-server/Dockerfile.tenant + args: + GIT_SHA: "${GIT_SHA:-harness}" + depends_on: + postgres-beta: + condition: service_healthy + redis: + condition: service_healthy + cp-stub: + condition: service_healthy + environment: + DATABASE_URL: "postgres://harness:harness@postgres-beta:5432/molecule?sslmode=disable" + REDIS_URL: "redis://redis:6379" + PORT: "8080" + PLATFORM_URL: "http://tenant-beta:8080" + MOLECULE_ENV: "production" + SECRETS_ENCRYPTION_KEY: "${SECRETS_ENCRYPTION_KEY:?must be set — run via tests/harness/up.sh, which generates one per run}" + # Distinct ADMIN_TOKEN — replays use this to verify TenantGuard + # blocks alpha-token presented at beta's URL. + ADMIN_TOKEN: "harness-admin-token-beta" + MOLECULE_ORG_ID: "harness-org-beta" + CP_UPSTREAM_URL: "http://cp-stub:9090" + RATE_LIMIT: "1000" + CANVAS_PROXY_URL: "http://localhost:3000" + networks: [harness-net] + healthcheck: + test: ["CMD-SHELL", "wget -q -O- http://localhost:8080/health || exit 1"] + interval: 5s + timeout: 5s + retries: 20 + + # ─── cf-proxy: routes by Host to the right tenant container ─────────── + # Production shape: same single CF tunnel front-doors every tenant + # subdomain — the Host header carries the tenant identity, not the + # routing destination. Local cf-proxy mirrors this exactly. cf-proxy: image: nginx:1.27-alpine depends_on: - tenant: + tenant-alpha: + condition: service_healthy + tenant-beta: condition: service_healthy volumes: - ./cf-proxy/nginx.conf:/etc/nginx/nginx.conf:ro - # Bind to 127.0.0.1 only — the harness uses a hardcoded ADMIN_TOKEN - # ("harness-admin-token") so binding 0.0.0.0 (compose's default) - # would expose admin access to anyone on the local network or VPN. - # Loopback-only is safe for E2E and prevents a known-token leak. + # Bind to 127.0.0.1 only — hardcoded ADMIN_TOKENs make 0.0.0.0 + # exposure unsafe even on a local network. ports: - "127.0.0.1:8080:8080" networks: [harness-net] diff --git a/tests/harness/down.sh b/tests/harness/down.sh index 683c4dae..fb1b305f 100755 --- a/tests/harness/down.sh +++ b/tests/harness/down.sh @@ -1,6 +1,17 @@ #!/usr/bin/env bash +# Tear down the harness and wipe per-tenant volumes. +# +# SECRETS_ENCRYPTION_KEY placeholder: docker compose validates the entire +# compose file even for `down -v` (a destructive read-only operation that +# doesn't read the env). up.sh generates a per-run key into its own +# shell — this script runs in a fresh shell that wouldn't see it. Without +# the placeholder, `compose down` exits non-zero before removing volumes, +# silently leaking workspaces+activity_logs into the next ./up.sh + seed.sh +# (verified 2026-05-02: tenant-isolation.sh F1/F2 saw 3× duplicate +# alpha-parent + alpha-child rows accumulated across three prior boots). set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$HERE" -docker compose -f compose.yml down -v --remove-orphans +SECRETS_ENCRYPTION_KEY="${SECRETS_ENCRYPTION_KEY:-down-placeholder}" \ + docker compose -f compose.yml down -v --remove-orphans echo "[harness] down + volumes removed." diff --git a/tests/harness/replays/per-tenant-independence.sh b/tests/harness/replays/per-tenant-independence.sh new file mode 100755 index 00000000..86e8759a --- /dev/null +++ b/tests/harness/replays/per-tenant-independence.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Replay for per-tenant independence — each tenant runs the same +# workflow concurrently with no cross-bleed in workspaces table or +# activity_logs. +# +# What this proves that tenant-isolation.sh doesn't: +# tenant-isolation.sh proves that REQUESTS get rejected at the +# middleware layer when they target the wrong tenant. THIS replay +# proves that even when both tenants are doing legitimate work +# simultaneously, the back-end state stays partitioned: no row in +# alpha's activity_logs ever shows up in beta's, no FK-resolution +# ever crosses tenants, etc. +# +# Test shape: seed activity_logs in BOTH tenants in parallel using +# distinct row counts (3 vs 5) so we can distinguish them. Then +# fetch each tenant's history and assert the count + content match +# the seed exactly — proves no leak in either direction. +# +# Phases: +# A. Seed alpha tenant: 3 a2a_receive rows (parent ← child). +# B. Seed beta tenant: 5 a2a_receive rows (parent ← child). +# C. GET alpha history → exactly 3 rows, all alpha-summary. +# D. GET beta history → exactly 5 rows, all beta-summary. +# E. Direct DB sanity — alpha PG has only alpha rows, beta PG only beta. +# F. Concurrent write race — both tenants take turns INSERTing +# simultaneously; each tenant's count after the race matches what +# it INSERTed. Catches "shared cache poison" / "shared connection +# pool" failure modes that don't show up in single-tenant tests. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +HARNESS_ROOT="$(dirname "$HERE")" +cd "$HARNESS_ROOT" + +if [ ! -f .seed.env ]; then + echo "[replay] no .seed.env — running ./seed.sh first..." + ./seed.sh +fi +# shellcheck source=/dev/null +source .seed.env +# shellcheck source=../_curl.sh +source "$HARNESS_ROOT/_curl.sh" + +PASS=0 +FAIL=0 + +assert() { + local desc="$1" expected="$2" actual="$3" + if [ "$expected" = "$actual" ]; then + printf " PASS %s\n" "$desc" + PASS=$((PASS + 1)) + else + printf " FAIL %s\n expected: %s\n got : %s\n" "$desc" "$expected" "$actual" >&2 + FAIL=$((FAIL + 1)) + fi +} + +# ─── Cleanup (idempotent) ────────────────────────────────────────────── +psql_exec_alpha >/dev/null </dev/null </dev/null </dev/null </dev/null </dev/null </dev/null </dev/null <&2 + FAIL=$((FAIL + 1)) + fi +} + +# ─── Phase A: positive controls ──────────────────────────────────────── +echo "[replay] A. positive controls — each tenant accepts its own valid creds" + +ALPHA_OWN=$(curl_alpha_admin -o /dev/null -w '%{http_code}' "$BASE/workspaces") +assert_status "A1: alpha creds at alpha returns 200" "200" "$ALPHA_OWN" + +BETA_OWN=$(curl_beta_admin -o /dev/null -w '%{http_code}' "$BASE/workspaces") +assert_status "A2: beta creds at beta returns 200" "200" "$BETA_OWN" + +# ─── Phase B: alpha creds at beta's URL → 404 ────────────────────────── +echo "" +echo "[replay] B. alpha-org header at beta's URL — TenantGuard must 404" + +CROSS_AB=$(curl_alpha_creds_at_beta -o /tmp/iso-ab.json -w '%{http_code}' "$BASE/workspaces") +assert_status "B1: alpha-org header at beta URL → 404" "404" "$CROSS_AB" + +# Body must be a generic 404 — never reveal that beta exists or that +# the org check fired (TenantGuard is intentionally indistinguishable +# from "no such route" to an outside scanner). +B_BODY=$(cat /tmp/iso-ab.json) +if echo "$B_BODY" | grep -qiE "tenant|org|forbidden|denied"; then + printf " FAIL B2: 404 body leaks tenant/org/auth keywords (info disclosure)\n body: %s\n" "$B_BODY" >&2 + FAIL=$((FAIL + 1)) +else + printf " PASS B2: 404 body has no tenant/org leak\n" + PASS=$((PASS + 1)) +fi + +# ─── Phase C: beta creds at alpha's URL → 404 ────────────────────────── +echo "" +echo "[replay] C. beta-org header at alpha's URL — TenantGuard must 404" + +CROSS_BA=$(curl_beta_creds_at_alpha -o /tmp/iso-ba.json -w '%{http_code}' "$BASE/workspaces") +assert_status "C1: beta-org header at alpha URL → 404" "404" "$CROSS_BA" + +# ─── Phase D: right URL, garbage org header ──────────────────────────── +echo "" +echo "[replay] D. right URL, garbage org header → 404" + +GARBAGE=$(curl -sS -o /dev/null -w '%{http_code}' \ + -H "Host: ${ALPHA_HOST}" \ + -H "Authorization: Bearer ${ALPHA_ADMIN_TOKEN}" \ + -H "X-Molecule-Org-Id: not-the-right-org" \ + "$BASE/workspaces") +assert_status "D1: garbage org id at alpha URL → 404" "404" "$GARBAGE" + +# ─── Phase E: bearer present but no org header at all → 404 ──────────── +echo "" +echo "[replay] E. valid bearer but missing X-Molecule-Org-Id → 404" + +NO_ORG=$(curl -sS -o /dev/null -w '%{http_code}' \ + -H "Host: ${ALPHA_HOST}" \ + -H "Authorization: Bearer ${ALPHA_ADMIN_TOKEN}" \ + "$BASE/workspaces") +assert_status "E1: missing X-Molecule-Org-Id → 404" "404" "$NO_ORG" + +# ─── Phase F: per-tenant DB isolation via list_workspaces ────────────── +echo "" +echo "[replay] F. per-tenant DB isolation via /workspaces listing" + +ALPHA_LIST=$(curl_alpha_admin "$BASE/workspaces") +ALPHA_NAMES=$(echo "$ALPHA_LIST" | jq -r '.[].name' | sort | tr '\n' ',' | sed 's/,$//') +echo "[replay] alpha tenant sees: $ALPHA_NAMES" + +if [ "$ALPHA_NAMES" = "alpha-child,alpha-parent" ]; then + printf " PASS F1: alpha enumerates only alpha workspaces\n" + PASS=$((PASS + 1)) +else + printf " FAIL F1: alpha enumerated unexpected workspaces\n expected: alpha-child,alpha-parent\n got : %s\n" "$ALPHA_NAMES" >&2 + FAIL=$((FAIL + 1)) +fi + +BETA_LIST=$(curl_beta_admin "$BASE/workspaces") +BETA_NAMES=$(echo "$BETA_LIST" | jq -r '.[].name' | sort | tr '\n' ',' | sed 's/,$//') +echo "[replay] beta tenant sees: $BETA_NAMES" + +if [ "$BETA_NAMES" = "beta-child,beta-parent" ]; then + printf " PASS F2: beta enumerates only beta workspaces\n" + PASS=$((PASS + 1)) +else + printf " FAIL F2: beta enumerated unexpected workspaces\n expected: beta-child,beta-parent\n got : %s\n" "$BETA_NAMES" >&2 + FAIL=$((FAIL + 1)) +fi + +# Cross-check: neither tenant's list contains the other's workspace ids. +LEAKED_INTO_ALPHA=$(echo "$ALPHA_LIST" | jq -r --arg b1 "$BETA_PARENT_ID" --arg b2 "$BETA_CHILD_ID" \ + '[.[] | select(.id == $b1 or .id == $b2)] | length') +assert_status "F3: alpha list contains zero beta workspace ids" "0" "$LEAKED_INTO_ALPHA" + +LEAKED_INTO_BETA=$(echo "$BETA_LIST" | jq -r --arg a1 "$ALPHA_PARENT_ID" --arg a2 "$ALPHA_CHILD_ID" \ + '[.[] | select(.id == $a1 or .id == $a2)] | length') +assert_status "F4: beta list contains zero alpha workspace ids" "0" "$LEAKED_INTO_BETA" + +# ─── Phase G: /health is allowlisted (sanity) ────────────────────────── +echo "" +echo "[replay] G. /health stays public on both tenants (TenantGuard allowlist sanity)" + +ALPHA_HEALTH=$(curl -sS -o /dev/null -w '%{http_code}' -H "Host: ${ALPHA_HOST}" "$BASE/health") +assert_status "G1: alpha /health public → 200" "200" "$ALPHA_HEALTH" + +BETA_HEALTH=$(curl -sS -o /dev/null -w '%{http_code}' -H "Host: ${BETA_HOST}" "$BASE/health") +assert_status "G2: beta /health public → 200" "200" "$BETA_HEALTH" + +echo "" +if [ "$FAIL" -gt 0 ]; then + echo "[replay] FAIL: $PASS pass, $FAIL fail" + exit 1 +fi +echo "[replay] PASS: $PASS/$PASS — TenantGuard isolation + per-tenant DB partitioning hold" diff --git a/tests/harness/requirements.txt b/tests/harness/requirements.txt index 75a30722..14210ca8 100644 --- a/tests/harness/requirements.txt +++ b/tests/harness/requirements.txt @@ -12,3 +12,9 @@ # when a new replay introduces a new Python import. httpx>=0.28.1 + +# channel-envelope-trust-boundary.sh imports from `molecule_runtime.*` (the +# wheel-rewritten path) so it catches the failure mode where the wheel +# build silently strips a fix that unit tests on local source still pass. +# >= 0.1.78 ships PR #2481's peer_id trust-boundary guard. +molecule-ai-workspace-runtime>=0.1.78 diff --git a/tests/harness/seed.sh b/tests/harness/seed.sh index 2532cbe6..fdcbd672 100755 --- a/tests/harness/seed.sh +++ b/tests/harness/seed.sh @@ -1,13 +1,20 @@ #!/usr/bin/env bash -# Seed the harness with two registered workspaces so peer-discovery -# replay scripts have something to discover. +# Seed BOTH tenants with parent + child workspaces so peer-discovery +# and cross-tenant replays have something to discover. # -# - "alpha" parent (tier 0) -# - "beta" child of alpha (tier 1) +# Tenant alpha: +# - alpha-parent (tier 0) +# - alpha-child (tier 1, child of alpha-parent) +# Tenant beta: +# - beta-parent (tier 0) +# - beta-child (tier 1, child of beta-parent) # -# Both register via the platform's /workspaces endpoint, which is what -# CP does at provision time. The platform then has them in its DB; -# tool_list_peers from inside alpha can resolve beta as a peer. +# IDs are server-generated (POST /workspaces ignores body.id) — we +# capture the returned id rather than minting client-side. Older +# versions silently desynced from the workspaces table, breaking +# FK-dependent replays. +# +# All four IDs persist to .seed.env so replays can target any of them. set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -16,51 +23,67 @@ cd "$HERE" # shellcheck source=_curl.sh source "$HERE/_curl.sh" -echo "[seed] confirming tenant is reachable via cf-proxy..." -HEALTH=$(curl_anon "$BASE/health" || echo "") -if [ -z "$HEALTH" ]; then - echo "[seed] FAILED: $BASE/health unreachable. Did ./up.sh complete?" +create_workspace() { + local tenant="$1" name="$2" tier="$3" parent="${4:-}" + local body + if [ -n "$parent" ]; then + body="{\"name\":\"$name\",\"tier\":$tier,\"parent_id\":\"$parent\",\"runtime\":\"langgraph\"}" + else + body="{\"name\":\"$name\",\"tier\":$tier,\"runtime\":\"langgraph\"}" + fi + local id + if [ "$tenant" = "alpha" ]; then + id=$(curl_alpha_admin -X POST "$BASE/workspaces" -d "$body" | jq -r '.id') + else + id=$(curl_beta_admin -X POST "$BASE/workspaces" -d "$body" | jq -r '.id') + fi + if [ -z "$id" ] || [ "$id" = "null" ]; then + echo "[seed] FAIL: $tenant/$name workspace creation returned no id" >&2 + return 1 + fi + echo "$id" +} + +echo "[seed] confirming both tenants reachable..." +ALPHA_HEALTH=$(curl_alpha_anon "$BASE/health" || echo "") +BETA_HEALTH=$(curl_beta_anon "$BASE/health" || echo "") +if [ -z "$ALPHA_HEALTH" ] || [ -z "$BETA_HEALTH" ]; then + echo "[seed] FAIL: tenant unreachable. alpha='$ALPHA_HEALTH' beta='$BETA_HEALTH'" + echo " Did ./up.sh complete cleanly?" exit 1 fi -echo "[seed] $HEALTH" +echo "[seed] alpha: $ALPHA_HEALTH" +echo "[seed] beta : $BETA_HEALTH" -echo "[seed] confirming /buildinfo returns the harness GIT_SHA..." -BUILD=$(curl_anon "$BASE/buildinfo" || echo "") -echo "[seed] $BUILD" +echo "" +echo "[seed] tenant alpha — creating alpha-parent + alpha-child ..." +ALPHA_PARENT_ID=$(create_workspace alpha alpha-parent 0) +echo "[seed] alpha-parent id=$ALPHA_PARENT_ID" +ALPHA_CHILD_ID=$(create_workspace alpha alpha-child 1 "$ALPHA_PARENT_ID") +echo "[seed] alpha-child id=$ALPHA_CHILD_ID" -# Create alpha (parent) and beta (child of alpha). The handler always -# generates the workspace id server-side and ignores any id in the -# request body, so we capture the returned id rather than minting one -# locally — older versions of this script minted client-side and would -# silently desync from the workspaces table, breaking FK-dependent -# replays (chat-history seeds activity_logs which has a FK to workspaces). -echo "[seed] creating workspace 'alpha' (parent)..." -ALPHA_ID=$(curl_admin -X POST "$BASE/workspaces" \ - -d '{"name":"alpha","tier":0,"runtime":"langgraph"}' \ - | jq -r '.id') -if [ -z "$ALPHA_ID" ] || [ "$ALPHA_ID" = "null" ]; then - echo "[seed] FAIL: alpha workspace creation returned no id" - exit 1 -fi -echo "[seed] alpha id=$ALPHA_ID" +echo "" +echo "[seed] tenant beta — creating beta-parent + beta-child ..." +BETA_PARENT_ID=$(create_workspace beta beta-parent 0) +echo "[seed] beta-parent id=$BETA_PARENT_ID" +BETA_CHILD_ID=$(create_workspace beta beta-child 1 "$BETA_PARENT_ID") +echo "[seed] beta-child id=$BETA_CHILD_ID" -echo "[seed] creating workspace 'beta' (child of alpha)..." -BETA_ID=$(curl_admin -X POST "$BASE/workspaces" \ - -d "{\"name\":\"beta\",\"tier\":1,\"parent_id\":\"$ALPHA_ID\",\"runtime\":\"langgraph\"}" \ - | jq -r '.id') -if [ -z "$BETA_ID" ] || [ "$BETA_ID" = "null" ]; then - echo "[seed] FAIL: beta workspace creation returned no id" - exit 1 -fi -echo "[seed] beta id=$BETA_ID" - -# Stash IDs so replay scripts pick them up. +# Stash IDs for replay scripts. +# +# Backwards-compat: ALPHA_ID + BETA_ID aliases keep pre-Phase-2 replays +# working (they used these names for the alpha tenant's parent + child). { - echo "ALPHA_ID=$ALPHA_ID" - echo "BETA_ID=$BETA_ID" + echo "ALPHA_PARENT_ID=$ALPHA_PARENT_ID" + echo "ALPHA_CHILD_ID=$ALPHA_CHILD_ID" + echo "BETA_PARENT_ID=$BETA_PARENT_ID" + echo "BETA_CHILD_ID=$BETA_CHILD_ID" + echo "# legacy aliases — pre-Phase-2 replays expect these names" + echo "ALPHA_ID=$ALPHA_PARENT_ID" + echo "BETA_ID=$ALPHA_CHILD_ID" } > "$HERE/.seed.env" echo "" echo "[seed] done. IDs persisted to tests/harness/.seed.env" -echo "[seed] ALPHA_ID=$ALPHA_ID" -echo "[seed] BETA_ID=$BETA_ID" +echo "[seed] alpha: parent=$ALPHA_PARENT_ID child=$ALPHA_CHILD_ID" +echo "[seed] beta : parent=$BETA_PARENT_ID child=$BETA_CHILD_ID" diff --git a/tests/harness/up.sh b/tests/harness/up.sh index 87a6cf91..1dad2272 100755 --- a/tests/harness/up.sh +++ b/tests/harness/up.sh @@ -38,21 +38,22 @@ if [ "$REBUILD" = true ]; then docker compose -f compose.yml build --no-cache tenant cp-stub fi -echo "[harness] starting cp-stub + postgres + redis + tenant + cf-proxy ..." +echo "[harness] starting redis + cp-stub + tenant-alpha + tenant-beta + cf-proxy ..." docker compose -f compose.yml up -d --wait -# Sudo-free reachability: cf-proxy/nginx routes by Host header (matches -# production CF tunnel), so replays target loopback :8080 with a Host -# header rather than depending on /etc/hosts resolution. _curl.sh -# centralises this. Legacy /etc/hosts users still work — the BASE env -# var override accepts either shape. +# Sudo-free reachability: cf-proxy/nginx routes by Host header to the +# right tenant container (matches production CF tunnel: same URL, +# different Host = different tenant). Replays target loopback :8080 +# with a per-tenant Host header. _curl.sh centralises the helper +# functions (curl_alpha_admin, curl_beta_admin, etc.). echo "" -echo "[harness] up." -echo " Tenant via cf-proxy: http://localhost:8080/health" -echo " (Host: harness-tenant.localhost)" -echo " cp-stub: internal-only via compose net" +echo "[harness] up. Multi-tenant topology:" +echo " tenant-alpha: Host: harness-tenant-alpha.localhost" +echo " tenant-beta: Host: harness-tenant-beta.localhost" +echo " legacy alias: Host: harness-tenant.localhost → alpha" echo "" -echo " Quick check:" -echo " curl -H 'Host: harness-tenant.localhost' http://localhost:8080/health" +echo " Quick check (no /etc/hosts needed):" +echo " curl -H 'Host: harness-tenant-alpha.localhost' http://localhost:8080/health" +echo " curl -H 'Host: harness-tenant-beta.localhost' http://localhost:8080/health" echo "" -echo "Next: ./seed.sh # mint admin token + register sample workspaces" +echo "Next: ./seed.sh # register parent+child workspaces in BOTH tenants"