History

Hongming Wang 5ebe6ccb33 test: regression guards for 2026-04-23 hermes + CP bug wave Three complementary regression tests for the chain of P0s fixed today. Each targets a specific bug class that reached production, and will fire loud if any of them regress. ## 1. E2E A2A assertion enhancements (tests/e2e/test_staging_full_saas.sh) The existing A2A check looked for "error\|exception" in the response text, which was too broad and missed the actual error patterns we hit. Now matches each known error class individually with a diagnostic fail message pointing at the exact bug: - "[hermes-agent error 401]" → hermes #12 (API_SERVER_KEY) - "hermes-agent unreachable" → gateway process died - "model_not_found" → hermes #13 (model prefix) - "Encrypted content is not supported" → hermes #14 (api_mode) - "Unknown provider" → bridge PROVIDER misconfig Also asserts the response contains the PONG token the prompt asked for — catches silent-truncation/echo regressions. ## 2. Hermes install.sh bridge shell harness (tools/test-hermes-bridge.sh) 4 scenarios × 16 assertions, all offline (no docker, no network): - openai-bridge-happy: OPENAI_API_KEY + openai/gpt-4o → provider=custom, model="gpt-4o" (prefix stripped), api_mode=chat_completions - operator-custom-wins: explicit HERMES_CUSTOM_* → bridge skipped - openrouter-not-touched: OPENROUTER_API_KEY → provider=openrouter, slug kept - non-prefixed-model: bare "gpt-4o" → prefix-strip is a no-op Runs in <1s, can be wired into template-hermes CI. Pins the exact config.yaml shape — any drift in derive-provider.sh or the bridge if-block breaks a test. ## 3. Canvas ConfigTab hermes tests (ConfigTab.hermes.test.tsx) 5 vitest cases covering the #1894 bugs: - Runtime loads from workspace metadata when config.yaml missing - "No config.yaml found" red error hidden for hermes - Hermes info banner shown instead - Langgraph workspace still sees the red error (regression-guard the other way) - config.yaml runtime wins over workspace metadata when present ## Running bash tools/test-hermes-bridge.sh # 16 assertions cd canvas && npx vitest run src/components/tabs/__tests__/ConfigTab.hermes.test.tsx # 5 cases # E2E enhancements ride on the existing staging E2E workflow ## Not yet covered (tracked in #1900) CP admin delete-tenant EC2 cascade, cp-provisioner instance_id lookup (#1738), purge audit SQL mismatch (#241), and pq prepared- statement cache collision (#242). These are in-controlplane-repo concerns — separate PR with CP-side sqlmock + integration tests. Closes items in #1900. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-23 17:45:13 -07:00
..
e2e	test: regression guards for 2026-04-23 hermes + CP bug wave	2026-04-23 17:45:13 -07:00
README.md	chore: final open-source cleanup — binary, stale paths, private refs	2026-04-18 00:38:55 -07:00

Hongming Wang 5ebe6ccb33 test: regression guards for 2026-04-23 hermes + CP bug wave

Three complementary regression tests for the chain of P0s fixed today.
Each targets a specific bug class that reached production, and will
fire loud if any of them regress.

## 1. E2E A2A assertion enhancements (tests/e2e/test_staging_full_saas.sh)

The existing A2A check looked for "error|exception" in the response text,
which was too broad and missed the actual error patterns we hit. Now
matches each known error class individually with a diagnostic fail
message pointing at the exact bug:

  - "[hermes-agent error 401]"        → hermes #12 (API_SERVER_KEY)
  - "hermes-agent unreachable"        → gateway process died
  - "model_not_found"                 → hermes #13 (model prefix)
  - "Encrypted content is not supported" → hermes #14 (api_mode)
  - "Unknown provider"                → bridge PROVIDER misconfig

Also asserts the response contains the PONG token the prompt asked for —
catches silent-truncation/echo regressions.

## 2. Hermes install.sh bridge shell harness (tools/test-hermes-bridge.sh)

4 scenarios × 16 assertions, all offline (no docker, no network):

  - openai-bridge-happy: OPENAI_API_KEY + openai/gpt-4o →
    provider=custom, model="gpt-4o" (prefix stripped),
    api_mode=chat_completions
  - operator-custom-wins: explicit HERMES_CUSTOM_* → bridge skipped
  - openrouter-not-touched: OPENROUTER_API_KEY → provider=openrouter,
    slug kept
  - non-prefixed-model: bare "gpt-4o" → prefix-strip is a no-op

Runs in <1s, can be wired into template-hermes CI. Pins the exact
config.yaml shape — any drift in derive-provider.sh or the bridge
if-block breaks a test.

## 3. Canvas ConfigTab hermes tests (ConfigTab.hermes.test.tsx)

5 vitest cases covering the #1894 bugs:

  - Runtime loads from workspace metadata when config.yaml missing
  - "No config.yaml found" red error hidden for hermes
  - Hermes info banner shown instead
  - Langgraph workspace still sees the red error (regression-guard the
    other way)
  - config.yaml runtime wins over workspace metadata when present

## Running

  bash tools/test-hermes-bridge.sh                # 16 assertions
  cd canvas && npx vitest run src/components/tabs/__tests__/ConfigTab.hermes.test.tsx  # 5 cases
  # E2E enhancements ride on the existing staging E2E workflow

## Not yet covered (tracked in #1900)

CP admin delete-tenant EC2 cascade, cp-provisioner instance_id
lookup (#1738), purge audit SQL mismatch (#241), and pq prepared-
statement cache collision (#242). These are in-controlplane-repo
concerns — separate PR with CP-side sqlmock + integration tests.

Closes items in #1900.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-23 17:45:13 -07:00

e2e

test: regression guards for 2026-04-23 hermes + CP bug wave

2026-04-23 17:45:13 -07:00

README.md

chore: final open-source cleanup — binary, stale paths, private refs

2026-04-18 00:38:55 -07:00

README.md

Tests

This repo uses the standard monorepo testing convention: unit tests live with their package, cross-component E2E tests live here.

Where to find tests

Scope	Location
Go unit + integration (platform, CLI, handlers)	`workspace-server/*/_test.go` — run with `cd workspace-server && go test -race ./...`
TypeScript unit (canvas components, hooks, store)	`canvas/src/**/__tests__/` — run with `cd canvas && npm test -- --run`
TypeScript unit (MCP server handlers)	`mcp-server/src/__tests__/` — run with `cd mcp-server && npx jest`
Python unit (workspace runtime, adapters)	`workspace/tests/` — run with `cd workspace && python3 -m pytest`
Python unit (SDK: plugin + remote agent)	`sdk/python/tests/` — run with `cd sdk/python && python3 -m pytest`
Cross-component E2E (spans platform + runtime + HTTP)	`tests/e2e/` ← you are here

Why split this way

Go requires co-located _test.go files to access unexported symbols.
Per-package test commands keep the inner loop fast — changing canvas doesn't re-run Go tests.
tests/e2e/ covers scenarios that no single package owns: a full workspace lifecycle, A2A across two provisioned agents, delegation chains, bundle round-trips.

Running E2E

Every E2E script here assumes the platform is running at localhost:8080 and (where noted) provisioned agents are online. See the header comment of each .sh for specifics.

Cleaning up rogue test workspaces

If an E2E run aborts before its teardown runs (Ctrl-C, crash, CI timeout), the platform can be left with workspaces whose config volume is stale or empty — Docker's unless-stopped restart policy then spins those containers in a FileNotFoundError loop. The platform's pre-flight check (#17) marks such workspaces failed on the next restart, but a manual cleanup is useful:

bash scripts/cleanup-rogue-workspaces.sh               # deletes ws with id/name starting aaaaaaaa-, bbbbbbbb-, cccccccc-, test-ws-
MOLECULE_URL=http://host:8080 bash scripts/cleanup-rogue-workspaces.sh

The script DELETEs each matching workspace via the API and force-removes the ws-<id[:12]> container as a belt-and-suspenders fallback.