2 Commits

Author	SHA1	Message	Date
Hongming Wang	5ebe6ccb33	test: regression guards for 2026-04-23 hermes + CP bug wave ... Three complementary regression tests for the chain of P0s fixed today. Each targets a specific bug class that reached production, and will fire loud if any of them regress. ## 1. E2E A2A assertion enhancements (tests/e2e/test_staging_full_saas.sh) The existing A2A check looked for "error|exception" in the response text, which was too broad and missed the actual error patterns we hit. Now matches each known error class individually with a diagnostic fail message pointing at the exact bug: - "[hermes-agent error 401]" → hermes #12 (API_SERVER_KEY) - "hermes-agent unreachable" → gateway process died - "model_not_found" → hermes #13 (model prefix) - "Encrypted content is not supported" → hermes #14 (api_mode) - "Unknown provider" → bridge PROVIDER misconfig Also asserts the response contains the PONG token the prompt asked for — catches silent-truncation/echo regressions. ## 2. Hermes install.sh bridge shell harness (tools/test-hermes-bridge.sh) 4 scenarios × 16 assertions, all offline (no docker, no network): - openai-bridge-happy: OPENAI_API_KEY + openai/gpt-4o → provider=custom, model="gpt-4o" (prefix stripped), api_mode=chat_completions - operator-custom-wins: explicit HERMES_CUSTOM_* → bridge skipped - openrouter-not-touched: OPENROUTER_API_KEY → provider=openrouter, slug kept - non-prefixed-model: bare "gpt-4o" → prefix-strip is a no-op Runs in <1s, can be wired into template-hermes CI. Pins the exact config.yaml shape — any drift in derive-provider.sh or the bridge if-block breaks a test. ## 3. Canvas ConfigTab hermes tests (ConfigTab.hermes.test.tsx) 5 vitest cases covering the #1894 bugs: - Runtime loads from workspace metadata when config.yaml missing - "No config.yaml found" red error hidden for hermes - Hermes info banner shown instead - Langgraph workspace still sees the red error (regression-guard the other way) - config.yaml runtime wins over workspace metadata when present ## Running bash tools/test-hermes-bridge.sh # 16 assertions cd canvas && npx vitest run src/components/tabs/__tests__/ConfigTab.hermes.test.tsx # 5 cases # E2E enhancements ride on the existing staging E2E workflow ## Not yet covered (tracked in #1900) CP admin delete-tenant EC2 cascade, cp-provisioner instance_id lookup (#1738), purge audit SQL mismatch (#241), and pq prepared- statement cache collision (#242). These are in-controlplane-repo concerns — separate PR with CP-side sqlmock + integration tests. Closes items in #1900. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-23 17:45:13 -07:00
Hongming Wang	a56b765b2d	docs: testing strategy + PR hygiene + backend parity matrix + boot-event postmortem (#1824) ... Bundles the documentation and lightweight tooling landed during the 2026-04-23 ops/triage session. Pure additions — no behavior changes. ## Added ### docs/architecture/backends.md Parity matrix for Docker vs EC2 (SaaS) workspace backends. 18 features tabulated with current status; 6 ranked drift risks; enforcement hooks (parity-lint + contract tests). Living document — owners are workspace-server + controlplane teams. ### docs/engineering/testing-strategy.md Tiered test-coverage floors instead of a blanket 100% target. Seven tiers by code class (auth/crypto → generated DTOs). Per-package current-state snapshot + targets. Tracks the 3 biggest coverage gaps (tokens.go 0%, workspace_provision.go 0%, wsauth ~48%) against their tier-1/2 floors. ### docs/engineering/pr-hygiene.md Captures the patterns that keep diffs reviewable. Motivated by the 2026-04-23 backlog audit where 8 of 23 open PRs had 70-380-file bloat from stale branch drift. Covers: small-PR sizing, rebase-not-merge, cherry-pick-onto-fresh-base for recovery, targeting staging first, describing why-not-what. ### docs/engineering/postmortem-2026-04-23-boot-event-401.md Postmortem for the /cp/tenants/boot-event 401 race. Root cause (DB INSERT ordered AFTER readiness check), detection path (E2E + manual log inspection), lessons (write-before-read pattern, integration tests needed, E2E alerting gap, invariants-as-comments). ### tools/check-template-parity.sh CI lint for template repos — diffs the `${VAR:+VAR=${VAR}}` provider- key forwarders between install.sh (bare-host / EC2 path) and start.sh (Docker path). Catches the #5 drift risk from backends.md before it ships. ### workspace-server/internal/provisioner/backend_contract_test.go Shared behavioral contract scaffold for Provisioner + CPProvisioner. Compile-time assertions catch method-signature drift today; scenario- level runs are t.Skip'd pending backend nil-hardening (drift risk #6, see backends.md). ## Updated ### README.md Links the new engineering docs + backends parity matrix into the Documentation Map so agents and humans can actually find them. ## Related issues - #1814 — unblock workspace_provision_test.go (broadcaster interface) - #1813 — nil-client panic hardening (drift risk #6) - #1815 — Canvas vitest coverage instrumentation - #1816 — tokens.go 0% → 85% - #1817 — 5 sqlmock column-drift failures - #1818 — Python pytest-cov setup - #1819 — wsauth middleware coverage gap - #1821 — tiered coverage policy (meta) - #1822 — backend parity drift tracker Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>	2026-04-23 19:59:38 +00:00