c275716005
Brings the local harness from "single tenant covering the request path" to "two tenants covering both the request path AND the per-tenant isolation boundary" — the same shape production runs (one EC2 + one Postgres + one MOLECULE_ORG_ID per tenant). Why this matters: the four prior replays exercise the SaaS request path against one tenant. They cannot prove that TenantGuard rejects a misrouted request (production CF tunnel + AWS LB are the failure surface), nor that two tenants doing legitimate work in parallel keep their `activity_logs` / `workspaces` / connection-pool state partitioned. Both are real bug classes — TenantGuard allowlist drift shipped #2398, lib/pq prepared-statement cache collision is documented as an org-wide hazard. What changed: 1. compose.yml — split into two tenants. tenant-alpha + postgres-alpha + tenant-beta + postgres-beta + the shared cp-stub, redis, cf-proxy. Each tenant gets a distinct ADMIN_TOKEN + MOLECULE_ORG_ID and its own Postgres database. cf-proxy depends on both tenants becoming healthy. 2. cf-proxy/nginx.conf — Host-header → tenant routing. `map $host $tenant_upstream` resolves the right backend per request. Required `resolver 127.0.0.11 valid=30s ipv6=off;` because nginx needs an explicit DNS resolver to use a variable in `proxy_pass` (literal hostnames resolve once at startup; variables resolve per request — without the resolver nginx fails closed with 502). `server_name` lists both tenants + the legacy alias so unknown Host headers don't silently route to a default and mask routing bugs. 3. _curl.sh — per-tenant + cross-tenant-negative helpers. `curl_alpha_admin` / `curl_beta_admin` set the right Host + Authorization + X-Molecule-Org-Id triple. `curl_alpha_creds_at_beta` / `curl_beta_creds_at_alpha` exist precisely to make WRONG requests (replays use them to assert TenantGuard rejects). `psql_exec_alpha` / `psql_exec_beta` shell out per-tenant Postgres exec. Legacy aliases (`curl_admin`, `psql_exec`) keep the four pre-Phase-2 replays working without edits. 4. seed.sh — registers parent+child workspaces in BOTH tenants. Captures server-generated IDs via `jq -r '.id'` (POST /workspaces ignores body.id, so the older client-side mint silently desynced from the workspaces table and broke FK-dependent replays). Stashes `ALPHA_PARENT_ID` / `ALPHA_CHILD_ID` / `BETA_PARENT_ID` / `BETA_CHILD_ID` to .seed.env, plus legacy `ALPHA_ID` / `BETA_ID` aliases for backwards compat with chat-history / channel-envelope. 5. New replays. tenant-isolation.sh (13 assertions) — TenantGuard 404s any request whose X-Molecule-Org-Id doesn't match the container's MOLECULE_ORG_ID. Asserts the 404 body has zero tenant/org/forbidden/denied keywords (existence of a tenant must not be probable from the outside). Covers cross-tenant routing misconfigure + allowlist drift + missing-org-header. per-tenant-independence.sh (12 assertions) — both tenants seed activity_logs in parallel with distinct row counts (3 vs 5) and confirm each tenant's history endpoint returns exactly its own counts. Then a concurrent INSERT race (10 rows per tenant in parallel via `&` + wait) catches shared-pool corruption + prepared-statement cache poisoning + redis cross-keyspace bleed. 6. Bug fix: down.sh + dump-logs SECRETS_ENCRYPTION_KEY validation. `docker compose down -v` validates the entire compose file even though it doesn't read the env. up.sh generates a per-run key into its own shell — down.sh runs in a fresh shell that wouldn't see it, so without a placeholder `compose down` exited non-zero before removing volumes. Workspaces silently leaked into the next ./up.sh + seed.sh boot. Caught when tenant-isolation.sh F1/F2 saw 3× duplicate alpha-parent rows accumulated across three prior runs. Same fix applied to the workflow's dump-logs step. 7. requirements.txt — pin molecule-ai-workspace-runtime>=0.1.78. channel-envelope-trust-boundary.sh imports from `molecule_runtime.*` (the wheel-rewritten path) so it catches the failure mode where the wheel build silently strips a fix that unit tests on local source still pass. CI was failing this replay because the wheel wasn't installed — caught in the staging push run from #2492. 8. .github/workflows/harness-replays.yml — Phase 2 plumbing. * Removed /etc/hosts step (Host-header path eliminated the need; scripts already source _curl.sh). * Updated dump-logs to reference the new service names (tenant-alpha + tenant-beta + postgres-alpha + postgres-beta). * Added SECRETS_ENCRYPTION_KEY placeholder env on the dump step. Verified: ./run-all-replays.sh from a clean state — 6/6 passed (buildinfo-stale-image, channel-envelope-trust-boundary, chat-history, peer-discovery-404, per-tenant-independence, tenant-isolation). Roadmap section updated: Phase 2 marked shipped. Phase 3 promoted to "replace cp-stub with real molecule-controlplane Docker build + env coherence lint." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
90 lines
3.2 KiB
Bash
Executable File
90 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Seed BOTH tenants with parent + child workspaces so peer-discovery
|
|
# and cross-tenant replays have something to discover.
|
|
#
|
|
# Tenant alpha:
|
|
# - alpha-parent (tier 0)
|
|
# - alpha-child (tier 1, child of alpha-parent)
|
|
# Tenant beta:
|
|
# - beta-parent (tier 0)
|
|
# - beta-child (tier 1, child of beta-parent)
|
|
#
|
|
# IDs are server-generated (POST /workspaces ignores body.id) — we
|
|
# capture the returned id rather than minting client-side. Older
|
|
# versions silently desynced from the workspaces table, breaking
|
|
# FK-dependent replays.
|
|
#
|
|
# All four IDs persist to .seed.env so replays can target any of them.
|
|
|
|
set -euo pipefail
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
cd "$HERE"
|
|
|
|
# shellcheck source=_curl.sh
|
|
source "$HERE/_curl.sh"
|
|
|
|
create_workspace() {
|
|
local tenant="$1" name="$2" tier="$3" parent="${4:-}"
|
|
local body
|
|
if [ -n "$parent" ]; then
|
|
body="{\"name\":\"$name\",\"tier\":$tier,\"parent_id\":\"$parent\",\"runtime\":\"langgraph\"}"
|
|
else
|
|
body="{\"name\":\"$name\",\"tier\":$tier,\"runtime\":\"langgraph\"}"
|
|
fi
|
|
local id
|
|
if [ "$tenant" = "alpha" ]; then
|
|
id=$(curl_alpha_admin -X POST "$BASE/workspaces" -d "$body" | jq -r '.id')
|
|
else
|
|
id=$(curl_beta_admin -X POST "$BASE/workspaces" -d "$body" | jq -r '.id')
|
|
fi
|
|
if [ -z "$id" ] || [ "$id" = "null" ]; then
|
|
echo "[seed] FAIL: $tenant/$name workspace creation returned no id" >&2
|
|
return 1
|
|
fi
|
|
echo "$id"
|
|
}
|
|
|
|
echo "[seed] confirming both tenants reachable..."
|
|
ALPHA_HEALTH=$(curl_alpha_anon "$BASE/health" || echo "")
|
|
BETA_HEALTH=$(curl_beta_anon "$BASE/health" || echo "")
|
|
if [ -z "$ALPHA_HEALTH" ] || [ -z "$BETA_HEALTH" ]; then
|
|
echo "[seed] FAIL: tenant unreachable. alpha='$ALPHA_HEALTH' beta='$BETA_HEALTH'"
|
|
echo " Did ./up.sh complete cleanly?"
|
|
exit 1
|
|
fi
|
|
echo "[seed] alpha: $ALPHA_HEALTH"
|
|
echo "[seed] beta : $BETA_HEALTH"
|
|
|
|
echo ""
|
|
echo "[seed] tenant alpha — creating alpha-parent + alpha-child ..."
|
|
ALPHA_PARENT_ID=$(create_workspace alpha alpha-parent 0)
|
|
echo "[seed] alpha-parent id=$ALPHA_PARENT_ID"
|
|
ALPHA_CHILD_ID=$(create_workspace alpha alpha-child 1 "$ALPHA_PARENT_ID")
|
|
echo "[seed] alpha-child id=$ALPHA_CHILD_ID"
|
|
|
|
echo ""
|
|
echo "[seed] tenant beta — creating beta-parent + beta-child ..."
|
|
BETA_PARENT_ID=$(create_workspace beta beta-parent 0)
|
|
echo "[seed] beta-parent id=$BETA_PARENT_ID"
|
|
BETA_CHILD_ID=$(create_workspace beta beta-child 1 "$BETA_PARENT_ID")
|
|
echo "[seed] beta-child id=$BETA_CHILD_ID"
|
|
|
|
# Stash IDs for replay scripts.
|
|
#
|
|
# Backwards-compat: ALPHA_ID + BETA_ID aliases keep pre-Phase-2 replays
|
|
# working (they used these names for the alpha tenant's parent + child).
|
|
{
|
|
echo "ALPHA_PARENT_ID=$ALPHA_PARENT_ID"
|
|
echo "ALPHA_CHILD_ID=$ALPHA_CHILD_ID"
|
|
echo "BETA_PARENT_ID=$BETA_PARENT_ID"
|
|
echo "BETA_CHILD_ID=$BETA_CHILD_ID"
|
|
echo "# legacy aliases — pre-Phase-2 replays expect these names"
|
|
echo "ALPHA_ID=$ALPHA_PARENT_ID"
|
|
echo "BETA_ID=$ALPHA_CHILD_ID"
|
|
} > "$HERE/.seed.env"
|
|
|
|
echo ""
|
|
echo "[seed] done. IDs persisted to tests/harness/.seed.env"
|
|
echo "[seed] alpha: parent=$ALPHA_PARENT_ID child=$ALPHA_CHILD_ID"
|
|
echo "[seed] beta : parent=$BETA_PARENT_ID child=$BETA_CHILD_ID"
|