Compare commits

..

40 Commits

Author SHA1 Message Date
core-be 8cea4a30c4 docs(openapi): apply Five-Axis review fixes to management spec
gate-check-v3 / gate-check (pull_request) Waiting to run
qa-review / approved (pull_request) Waiting to run
security-review / approved (pull_request) Waiting to run
sop-checklist / review-refire (pull_request) Waiting to run
sop-tier-check / tier-check (pull_request) Waiting to run
sop-tier-check / tier-check (pull_request_review) Successful in 9s
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 11s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
E2E Chat / detect-changes (pull_request) Successful in 16s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 11s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 12s
Harness Replays / detect-changes (pull_request) Successful in 12s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 12s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 8s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 7s
sop-checklist / review-refire (pull_request_target) Has been skipped
sop-checklist / all-items-acked (pull_request) acked: 7/7
sop-checklist / na-declarations (pull_request) N/A: (none)
qa-review / approved (pull_request_target) Successful in 4s
sop-checklist / all-items-acked (pull_request_target) Successful in 4s
security-review / approved (pull_request_target) Successful in 4s
gate-check-v3 / gate-check (pull_request_target) Successful in 4s
sop-tier-check / tier-check (pull_request_target) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 59s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 0s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m28s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Harness Replays / Harness Replays (pull_request) Successful in 1s
E2E Chat / E2E Chat (pull_request) Successful in 6s
E2E API Smoke Test / detect-changes (pull_request) Successful in 13s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m35s
audit-force-merge / audit (pull_request_target) Successful in 4s
CI / Platform (Go) (pull_request) Has been cancelled
CI / Canvas (Next.js) (pull_request) Has been cancelled
CI / Shellcheck (E2E scripts) (pull_request) Has been cancelled
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
CI / Detect changes (pull_request) Has been cancelled
CI / all-required (pull_request) Failing after 40m22s
CI / Python Lint & Test (pull_request) Has been cancelled
Verified each against the authoritative handler source (molecule-core
workspace-server + molecule-controlplane) before editing:

1. tenantAdminToken: http/bearer -> apiKey header X-Molecule-Admin-Token.
   authenticateTenant (controlplane workspace_provision.go) reads that
   header, NOT Authorization, and derives org from the token
   (SELECT org_id ... WHERE admin_token=$1). Removed orgRoutingHeaderId
   from the DELETE /api/v1/workspaces/{workspace_id} security — no
   X-Molecule-Org-Id is read on deprovision.
2. ProvisionStatus.stage: added `failed` (emitted by orgs.go on
   failed/deprovisioning/deprovisioned). Existing launching/installing/
   starting/configuring_https/ready all confirmed emitted by
   orgs_progress.go + estimateBootProgress — none trimmed.
3. GET /workspaces/{id}: set security: [] — router.go registers it
   outside every auth group (intentionally open for canvas-node self-
   polling). Dropped the now-inapplicable 401.
4. Multi-period budget shape: added `budget_limits` (canonical) + legacy
   `budget_limit` to PatchBudgetRequest, and `periods` (+ PeriodBudget)
   to BudgetResponse, matching budget.go budgetResponse/PatchBudget.
5. GET tenant llm-billing-mode already modeled (handler serves GET+PUT) —
   no change needed; verified.
6. Added prune=true destructive note (only literal "true" permanently
   deletes, internal#734) and the CP-admin
   /api/v1/admin/workspaces/{id}/llm-billing-mode GET+PUT pair
   (cpAdminBearer, requires ?org_slug=).

redocly lint clean under both recommended and recommended-strict.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 22:29:07 -07:00
core-be dc7e660e90 docs(openapi): add OpenAPI 3.1 management spec (SSOT) + README
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
E2E Chat / E2E Chat (pull_request) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Harness Replays / Harness Replays (pull_request) Blocked by required conditions
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 6s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 9s
Harness Replays / detect-changes (pull_request) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 6s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 43s
gate-check-v3 / gate-check (pull_request) Successful in 5s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m10s
qa-review / approved (pull_request) Failing after 6s
security-review / approved (pull_request) Failing after 5s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 5s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 8s
CI / Platform (Go) (pull_request) Has been cancelled
CI / Canvas (Next.js) (pull_request) Has been cancelled
CI / Shellcheck (E2E scripts) (pull_request) Has been cancelled
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
CI / all-required (pull_request) Failing after 40m28s
CI / Detect changes (pull_request) Has been cancelled
CI / Python Lint & Test (pull_request) Has been cancelled
Author workspace-server/docs/openapi/management.yaml — the hand-authored,
authoritative OpenAPI 3.1 contract for the Molecule platform MANAGEMENT
surface, spanning both services in one spec:

  - CP (api.moleculesai.app, /api/v1/*): orgs create/get/list/delete/export/
    provision-status, public instance lookup, billing (invoices/checkout/
    portal/topup), admin (admin-create-org w/ dry_run, tenant delete +
    scrub w/ confirm guard, diagnostics, redeploy + fleet, workspace env
    w/ force guard, ListOrgWorkspaces, admin-token, thin-ami + runtime-image
    pins), provisioning (provision w/ 422 RUNTIME_PIN_MISSING, deprovision,
    status).
  - Tenant workspace-server: /workspaces[/:id] CRUD + restart/pause/resume,
    budget, llm-billing-mode, /workspaces/:id/secrets, /settings/secrets,
    /org/import, /org/templates, /org/tokens (Org API Key mint/revoke),
    /templates[/import], /bundles export/import.

Defines the five security tiers as securitySchemes (workosSession cookie,
cpAdminBearer, provisionSecret [+ tenantAdminToken on deprovision], orgApiKey
+ org routing header, workspaceToken) and applies the correct scheme(s)
per-route. Dry-run / confirm / force guards modelled per-operation.

Grounded in the router + handler sources (controlplane + workspace-server),
not just the synthesis doc — notably llm-billing-mode is modelled on the
real tenant route (/admin/workspaces/:id/llm-billing-mode, AdminAuth), with
the divergence from the synthesis doc noted in the README.

Adds README.md documenting the two-service split + the security-scheme→
surface tier matrix. This is the SSOT the management MCP + CLI + docs derive
from (PLATFORM-MANAGEMENT-API.md §5c / RFC #1706). Supersedes the swaggo
/schedules stub for the management surface; runtime surface stays out of scope.

Per dev-sop Phase 1-4 + Five-Axis self-review (in PR body).
Lints clean: npx @redocly/cli lint management.yaml (0 errors, 0 warnings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 20:40:36 -07:00
devops-engineer 774a8c2a6a Merge pull request 'fix(providers): sync registry to controlplane SSOT — codex→openai-subscription byok' (#2025) from fix/providers-ssot-sync-codex-subscription into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 11s
Block internal-flavored paths / Block forbidden paths (push) Successful in 13s
CI / Python Lint & Test (push) Successful in 11s
CI / Detect changes (push) Successful in 12s
Handlers Postgres Integration / detect-changes (push) Successful in 8s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 10s
E2E API Smoke Test / detect-changes (push) Successful in 12s
E2E Chat / detect-changes (push) Successful in 11s
Harness Replays / detect-changes (push) Successful in 7s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 6s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 8s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 4s
sync-providers-yaml / Compare synced providers.yaml against controlplane canonical (push) Successful in 5s
CI / Canvas (Next.js) (push) Successful in 5s
CI / Shellcheck (E2E scripts) (push) Successful in 5s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 1m10s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 30s
publish-workspace-server-image / build-and-push (push) Successful in 3m20s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 2m42s
Harness Replays / Harness Replays (push) Successful in 20s
CI / Canvas Deploy Reminder (push) Successful in 4s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2m7s
E2E Chat / E2E Chat (push) Successful in 4m55s
CI / Platform (Go) (push) Successful in 5m23s
CI / all-required (push) Successful in 6m1s
publish-workspace-server-image / Production auto-deploy (push) Failing after 6m34s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
main-red-watchdog / watchdog (push) Successful in 2m7s
gate-check-v3 / gate-check (push) Successful in 1m5s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 1m14s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 4s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 6s
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m33s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Weekly Platform-Go Surface / Weekly Platform-Go Surface (push) Successful in 5m13s
2026-05-31 23:50:53 +00:00
Hongming Wang cb660fc0b4 fix(providers): sync registry to controlplane SSOT — codex→openai-subscription byok
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 19s
CI / Detect changes (pull_request) Successful in 12s
CI / Python Lint & Test (pull_request) Successful in 8s
E2E API Smoke Test / detect-changes (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 16s
E2E Chat / detect-changes (pull_request) Successful in 18s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Harness Replays / detect-changes (pull_request) Successful in 5s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 7s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 8s
sync-providers-yaml / Compare synced providers.yaml against controlplane canonical (pull_request) Successful in 13s
gate-check-v3 / gate-check (pull_request) Successful in 16s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 33s
sop-checklist / review-refire (pull_request) Has been skipped
security-review / approved (pull_request) Successful in 10s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m8s
sop-tier-check / tier-check (pull_request) Successful in 11s
sop-checklist / all-items-acked (pull_request) acked: 7/7
sop-checklist / na-declarations (pull_request) N/A: (none)
CI / Canvas (Next.js) (pull_request) Successful in 24s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 17s
E2E Chat / E2E Chat (pull_request) Successful in 16s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 14s
Harness Replays / Harness Replays (pull_request) Successful in 3s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m56s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m10s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Platform (Go) (pull_request) Successful in 5m26s
CI / all-required (pull_request) Successful in 13m31s
qa-review / approved (pull_request) Refired via /qa-recheck by unknown
audit-force-merge / audit (pull_request) Successful in 15s
molecule-core's synced copy of the provider registry was stale relative to
controlplane cp#423/#426, which split `openai`→`openai-subscription`
(auth_env CODEX_AUTH_JSON, IsPlatform false) / `openai-api` (OPENAI_API_KEY).
The stale copy derived codex→`openai` (and got band-aided to platform_managed),
producing "OpenAI requires OPENAI_API_KEY" + "codex adapter: no platform
provider" RuntimeError.

Sync to CP SSOT (CP HEAD fa44dc8), verbatim:
- providers.yaml, derive_provider.go, providers.go, and the
  derive/providers/runtimes tests copied byte-exact from controlplane.
- regenerated gen/registry_gen.go via `go generate` (now carries the
  openai-subscription entry: AuthEnv CODEX_AUTH_JSON, IsPlatform false).
- bumped canonicalProvidersYAMLSHA256 to the new synced-copy sha
  (dedbb8cc…f76187) so the hermetic drift gate stays green.

Core-only manual edit (CP has no such map):
- secrets.go: add CODEX_AUTH_JSON to platformManagedDirectLLMBypassKeys so the
  byok credential check counts the global CODEX_AUTH_JSON (codex byok now
  provisions with the shared subscription token) and strips it under
  platform-managed.

With the synced derive, codex+CODEX_AUTH_JSON → openai-subscription →
IsPlatform false → byok automatically via the existing billing resolver;
no derive logic was hand-edited and llm_billing_mode.go is untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 16:05:55 -07:00
devops-engineer 446b8c78fd Merge pull request 'fix(workspace-server): central codex OAuth refresher (single-owner, anti-burn)' (#2023) from fix/codex-central-refresher into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 12s
Block internal-flavored paths / Block forbidden paths (push) Successful in 6s
CI / Detect changes (push) Successful in 12s
CI / Python Lint & Test (push) Successful in 3s
E2E API Smoke Test / detect-changes (push) Successful in 7s
E2E Chat / detect-changes (push) Successful in 12s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 9s
Handlers Postgres Integration / detect-changes (push) Successful in 3s
Harness Replays / detect-changes (push) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 8s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 11s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 1m18s
CI / Canvas (Next.js) (push) Successful in 10s
CI / Shellcheck (E2E scripts) (push) Successful in 9s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 35s
publish-workspace-server-image / build-and-push (push) Successful in 3m9s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 2m44s
Harness Replays / Harness Replays (push) Successful in 6s
CI / Canvas Deploy Reminder (push) Successful in 3s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2m44s
CI / Platform (Go) (push) Successful in 5m40s
CI / all-required (push) Successful in 7m1s
E2E Chat / E2E Chat (push) Successful in 4m46s
publish-workspace-server-image / Production auto-deploy (push) Failing after 7m31s
main-red-watchdog / watchdog (push) Successful in 1m59s
gate-check-v3 / gate-check (push) Successful in 43s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 27s
ci-required-drift / drift (push) Successful in 1m7s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 6s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 4s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
2026-05-31 19:52:21 +00:00
hongming-personal df972a85e2 fix(workspace-server): central codex OAuth refresher (single-owner, anti-burn)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 7s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
CI / Detect changes (pull_request) Successful in 27s
CI / Python Lint & Test (pull_request) Successful in 27s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 13s
E2E API Smoke Test / detect-changes (pull_request) Successful in 16s
E2E Chat / detect-changes (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 14s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 5s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 6s
Harness Replays / detect-changes (pull_request) Successful in 7s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
gate-check-v3 / gate-check (pull_request) Successful in 11s
sop-checklist / review-refire (pull_request) Has been skipped
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 29s
sop-tier-check / tier-check (pull_request) Successful in 13s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m7s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
CI / Canvas (Next.js) (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 23s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 4s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2s
Harness Replays / Harness Replays (pull_request) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m56s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
sop-checklist / all-items-acked (pull_request) acked: 7/7
sop-checklist / na-declarations (pull_request) N/A: (none)
CI / Platform (Go) (pull_request) Successful in 5m28s
CI / all-required (pull_request) Successful in 7m16s
qa-review / approved (pull_request) Refired via /qa-recheck by unknown
security-review / approved (pull_request) Refired via /security-recheck by unknown
audit-force-merge / audit (pull_request) Successful in 9s
Multiple codex workspaces share ONE ChatGPT-Pro OAuth token (global_secrets
key CODEX_AUTH_JSON). OpenAI's refresh_token is single-use, so letting each
per-agent codex app-server refresh on its own 401 burned the shared seed within
seconds (a refresh storm → token_invalidated + "refresh token already used").

This adds a single platform-side owner of the refresh:
- internal/codexauth/refresher.go: one background goroutine, structurally
  single-flight (one goroutine + package mutex). Reads the global
  CODEX_AUTH_JSON, decodes the access_token JWT exp, and only within a safety
  margin of expiry POSTs the refresh_token ONCE per due cycle, then re-encrypts
  and writes the rotated blob back to global_secrets. Inert when the secret is
  absent; on a permanent failure (invalid_grant / "already used") it logs once
  and does NOT hot-loop. Billing-mode resolution + byok are untouched.
- cmd/server/main.go: wired under supervised.RunWithRecover like the other
  background sweeps.

Pairs with the codex template's codex_auth_sync.sh (GET-only re-sync; per-agent
OAuth POST disabled) so workspaces only consume the current token and never
rotate it themselves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 12:38:13 -07:00
core-be e45033e15c Merge pull request 'fix(canvas): SSOT-drive runtime picker so google-adk shows correctly' (#2016) from feat/google-adk-runtime-ssot into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Block internal-flavored paths / Block forbidden paths (push) Successful in 7s
CI / Python Lint & Test (push) Successful in 9s
CI / Detect changes (push) Successful in 15s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 14s
E2E Chat / detect-changes (push) Successful in 12s
E2E API Smoke Test / detect-changes (push) Successful in 13s
Handlers Postgres Integration / detect-changes (push) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 8s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 7s
Harness Replays / detect-changes (push) Successful in 9s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 5s
CI / Shellcheck (E2E scripts) (push) Successful in 2s
publish-canvas-image / Build & push canvas image (push) Successful in 1m24s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 1m4s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 2m5s
E2E Chat / E2E Chat (push) Successful in 4m32s
publish-workspace-server-image / build-and-push (push) Successful in 7m1s
CI / Platform (Go) (push) Successful in 7m12s
Harness Replays / Harness Replays (push) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m52s
CI / Canvas (Next.js) (push) Successful in 7m58s
CI / all-required (push) Successful in 8m44s
publish-workspace-server-image / Production auto-deploy (push) Successful in 4m1s
CI / Canvas Deploy Reminder (push) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Failing after 16m56s
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m15s
main-red-watchdog / watchdog (push) Successful in 2m20s
gate-check-v3 / gate-check (push) Successful in 25s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 1m10s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 11s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 9s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 8s
2026-05-31 09:46:31 +00:00
core-be 418db083ff ci: re-trigger after gitea restart task desync (no-op)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 13s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 5s
CI / Detect changes (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 4s
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 6s
Harness Replays / detect-changes (pull_request) Successful in 4s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 7s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 8s
gate-check-v3 / gate-check (pull_request) Successful in 12s
qa-review / approved (pull_request) Successful in 5s
security-review / approved (pull_request) Successful in 6s
sop-checklist / na-declarations (pull_request) N/A: (none)
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 35s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / all-items-acked (pull_request) Successful in 7s
sop-tier-check / tier-check (pull_request) Successful in 6s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m12s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 6s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m42s
E2E Chat / E2E Chat (pull_request) Successful in 8s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 3s
Harness Replays / Harness Replays (pull_request) Successful in 3s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m28s
CI / Platform (Go) (pull_request) Successful in 7m53s
CI / Canvas (Next.js) (pull_request) Successful in 7m24s
CI / all-required (pull_request) Successful in 22m29s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 7s
Empty commit on the PR branch to get a clean CI run; the prior run's
tasks were orphaned by the 2026-05-31 08:30 gitea restart (task-not-found).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 02:17:20 -07:00
core-be b611b1a9bf fix(canvas): SSOT-drive runtime picker so google-adk shows correctly
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
CI / all-required (pull_request) Failing after 1s
Block internal-flavored paths / Block forbidden paths (pull_request) Failing after 1s
CI / Detect changes (pull_request) Failing after 0s
CI / Python Lint & Test (pull_request) Failing after 0s
CI / Platform (Go) (pull_request) Has been skipped
E2E API Smoke Test / detect-changes (pull_request) Failing after 1s
CI / Canvas (Next.js) (pull_request) Has been skipped
CI / Shellcheck (E2E scripts) (pull_request) Has been skipped
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Chat / detect-changes (pull_request) Failing after 0s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Failing after 0s
Handlers Postgres Integration / detect-changes (pull_request) Failing after 0s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Has been skipped
E2E Chat / E2E Chat (pull_request) Has been skipped
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Has been skipped
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Has been skipped
Harness Replays / detect-changes (pull_request) Failing after 1s
Harness Replays / Harness Replays (pull_request) Has been skipped
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Failing after 2s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Failing after 1s
lint-required-no-paths / lint-required-no-paths (pull_request) Failing after 1s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Failing after 1s
Secret scan / Scan diff for credential-shaped strings (pull_request) Failing after 1s
gate-check-v3 / gate-check (pull_request) Failing after 1s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 13s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-tier-check / tier-check (pull_request) Successful in 32s
sop-checklist / all-items-acked (pull_request) Successful in 1m16s
qa-review / approved (pull_request) Refired via /qa-recheck by unknown
security-review / approved (pull_request) Refired via /security-recheck by unknown
The ConfigTab runtime dropdown filtered GET /templates through a hardcoded
SUPPORTED_RUNTIME_VALUES allowlist (claude-code/codex/openclaw/hermes).
google-adk shipped in manifest.json + the workspace-server knownRuntimes
registry but was dropped by this frontend Set, so a google-adk workspace's
Config tab rendered the wrong runtime option and a Save would clobber the
runtime to the wrong value.

Make the picker trust the backend SSOT: /templates is already gated to the
manifest maintained set by loadRuntimesFromManifest. Remove the allowlist;
hide a runtime only when its template declares displayable:false (new
optional flag plumbed manifest config.yaml -> templateSummary -> /templates).

- canvas/ConfigTab.tsx: drop SUPPORTED_RUNTIME_VALUES; filter on
  r.displayable===false; add google-adk to offline FALLBACK list.
- workspace-server templates.go: add Displayable *bool (yaml+json,
  omitempty) so a template can opt out of the picker declaratively.
- tests: ConfigTab.googleAdk.test.tsx (google-adk selected + displayable
  hidden) + TestTemplatesList_DisplayableFlag (nil/true/false + JSON contract).

Refs project_canvas_runtime_dropdown_ssot_fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 01:07:29 -07:00
hongming 5fce77aac9 Merge pull request 'feat(workspace): per-workspace data_persistence choice (internal#734 PR-2)' (#2014) from feat/workspace-data-persistence into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 12s
publish-canvas-image / Build & push canvas image (push) Successful in 1m39s
publish-workspace-server-image / build-and-push (push) Successful in 3m30s
Block internal-flavored paths / Block forbidden paths (push) Successful in 15s
CI / Detect changes (push) Successful in 9s
CI / Python Lint & Test (push) Successful in 5s
E2E API Smoke Test / detect-changes (push) Successful in 6s
Handlers Postgres Integration / detect-changes (push) Successful in 8s
Harness Replays / detect-changes (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 3s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 46s
CI / Platform (Go) (push) Successful in 5m1s
CI / Canvas (Next.js) (push) Successful in 4m52s
CI / Shellcheck (E2E scripts) (push) Successful in 1s
CI / all-required (push) Successful in 20m0s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m4s
publish-workspace-server-image / Production auto-deploy (push) Successful in 18m42s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 6m13s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m27s
Harness Replays / Harness Replays (push) Successful in 2s
CI / Canvas Deploy Reminder (push) Successful in 1s
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m21s
sync-providers-yaml / Compare synced providers.yaml against controlplane canonical (push) Failing after 3s
SECRET_PATTERNS drift lint / Detect SECRET_PATTERNS drift (push) Successful in 43s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 1s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Failing after 0s
E2E Staging External Runtime / E2E Staging External Runtime (push) Failing after 0s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push) Failing after 0s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Failing after 1s
E2E Staging Canvas (Playwright) / detect-changes (push) Failing after 0s
E2E Chat / detect-changes (push) Successful in 11s
E2E Chat / E2E Chat (push) Successful in 4m26s
main-red-watchdog / watchdog (push) Successful in 2m13s
gate-check-v3 / gate-check (push) Successful in 1m14s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 10s
E2E Legacy Advisory / Legacy local-platform E2E (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
ci-required-drift / drift (push) Successful in 1m1s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 14s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 3s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
2026-05-30 18:57:55 +00:00
hongming-ceo-delegated 257a61672b feat(canvas): per-workspace data persistence + erase-on-delete UI (internal#734)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 5s
CI / Detect changes (pull_request) Successful in 9s
CI / Python Lint & Test (pull_request) Successful in 5s
E2E API Smoke Test / detect-changes (pull_request) Successful in 8s
E2E Chat / detect-changes (pull_request) Successful in 10s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 13s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 51s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Harness Replays / detect-changes (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 5s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m13s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 35s
E2E Staging External Runtime / E2E Staging External Runtime (pull_request) Successful in 5m16s
qa-review / approved (pull_request) Successful in 4s
security-review / approved (pull_request) Successful in 4s
CI / Platform (Go) (pull_request) Successful in 4m30s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m12s
E2E Chat / E2E Chat (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 5s
CI / Canvas (Next.js) (pull_request) Successful in 5m24s
CI / all-required (pull_request) Successful in 28m22s
gate-check-v3 / gate-check (pull_request) Successful in 8s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 13s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 11s
Harness Replays / Harness Replays (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1m34s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 48m58s
audit-force-merge / audit (pull_request) Successful in 21s
The user-facing choice for the prune/persistence backend:
- ContainerConfigTab: a 'Saved data' selector (Auto / Always keep / Don't keep)
  → compute.data_persistence (omitted when Auto = unchanged wire/default).
- DetailsTab delete: an 'also erase saved data' checkbox → DELETE
  ?erase_data=true (default off keeps it for the orphan-sweeper grace).
- WorkspaceCompute.data_persistence type.

+test: erase checkbox sends erase_data=true; default delete unchanged. The 37
ContainerConfigTab+DetailsTab tests pass; my files typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 09:22:20 -07:00
hongming-ceo-delegated db6f5b2e93 feat(workspace): prune-on-delete caller wiring (internal#734 F1 — pairs with cp#415)
Block internal-flavored paths / Block forbidden paths (pull_request) Waiting to run
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
CI / Platform (Go) (pull_request) Blocked by required conditions
CI / Canvas (Next.js) (pull_request) Blocked by required conditions
CI / Shellcheck (E2E scripts) (pull_request) Blocked by required conditions
E2E API Smoke Test / detect-changes (pull_request) Waiting to run
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
E2E Chat / detect-changes (pull_request) Waiting to run
E2E Chat / E2E Chat (pull_request) Blocked by required conditions
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Waiting to run
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
E2E Staging External Runtime / E2E Staging External Runtime (pull_request) Waiting to run
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Waiting to run
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Waiting to run
Handlers Postgres Integration / detect-changes (pull_request) Waiting to run
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Harness Replays / detect-changes (pull_request) Waiting to run
Harness Replays / Harness Replays (pull_request) Blocked by required conditions
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Waiting to run
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Waiting to run
lint-required-no-paths / lint-required-no-paths (pull_request) Waiting to run
Secret scan / Scan diff for credential-shaped strings (pull_request) Waiting to run
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Waiting to run
gate-check-v3 / gate-check (pull_request) Waiting to run
qa-review / approved (pull_request) Waiting to run
security-review / approved (pull_request) Waiting to run
sop-checklist / all-items-acked (pull_request) Waiting to run
sop-checklist / review-refire (pull_request) Waiting to run
sop-tier-check / tier-check (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
CI / Detect changes (pull_request) Successful in 10s
CI / Python Lint & Test (pull_request) Successful in 3s
CI / all-required (pull_request) Failing after 40m13s
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
The caller side of the recreate-safe prune (cp#415 Five-Axis F1): the prune
signal reaches CP ONLY on a permanent user-delete-with-erase, NEVER on
restart/recreate/reconcile.

- CPProvisionerAPI.StopAndPrune (CPProvisioner builds DELETE with &prune=true;
  Stop never does — shared stopInternal).
- cpStopWithRetryErr(...prune): restart/hibernate pass false; delete passes the
  user choice.
- stopWorkspaceForDelete(...erase) → CascadeDelete(...erase): HTTP Delete reads
  ?erase_data=true (opt-in; default keeps data for the orphan-sweeper grace);
  org-import reconcile passes false.

Discriminating test: Stop sends NO prune=true (recreate-safety), StopAndPrune
sends it. All CPProvisionerAPI mocks gain StopAndPrune. Full handlers+provisioner
suite + vet + gofmt green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 09:16:47 -07:00
hongming-ceo-delegated acecb16d22 feat(workspace): per-workspace data_persistence choice (internal#734 PR-2)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 13s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 11s
CI / Detect changes (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 3s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 6s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
Harness Replays / detect-changes (pull_request) Successful in 3s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 33s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 32s
gate-check-v3 / gate-check (pull_request) Successful in 13s
qa-review / approved (pull_request) Failing after 7s
security-review / approved (pull_request) Failing after 5s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 7s
sop-tier-check / tier-check (pull_request) Successful in 8s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m13s
CI / Canvas (Next.js) (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 22s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m7s
Harness Replays / Harness Replays (pull_request) Successful in 4s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m41s
CI / Platform (Go) (pull_request) Successful in 6m17s
CI / all-required (pull_request) Successful in 10m14s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 50m1s
Threads the user's durable-data choice from the workspace Compute config
through to CP's provision request, so a user can pick persist vs ephemeral
per workspace (the caller side of cp#410's data_persistence support).

- models.WorkspaceCompute.DataPersistence (persisted in the compute JSONB)
- validateWorkspaceCompute: enum guard (persist|ephemeral|"") → clear 400
  before the CP round-trip; CP re-validates at its edge (defense in depth)
- WorkspaceConfig.DataPersistence + workspace_provision build site
- cpProvisionRequest.data_persistence (omitempty → ""=auto omitted on wire)

Empty/auto = today's behavior; forward-compatible (inert until CP deploys
cp#410). +validator enum test. build/vet/test/gofmt green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 23:49:37 -07:00
hongming 82bc28a098 Merge pull request 'test(e2e): wire google-adk into the runtime e2e suite' (#2012) from e2e/google-adk-ci-wiring into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 13s
publish-workspace-server-image / build-and-push (push) Successful in 2m59s
Block internal-flavored paths / Block forbidden paths (push) Successful in 7s
CI / Python Lint & Test (push) Successful in 5s
CI / Detect changes (push) Successful in 7s
E2E API Smoke Test / detect-changes (push) Successful in 11s
Handlers Postgres Integration / detect-changes (push) Successful in 4s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 3s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (push) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (push) Successful in 1m19s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 3s
Ops Scripts Tests / Ops scripts (unittest) (push) Successful in 1m6s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 54s
CI / Platform (Go) (push) Successful in 2s
CI / Canvas (Next.js) (push) Successful in 2s
CI / Shellcheck (E2E scripts) (push) Successful in 10s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 2s
CI / all-required (push) Successful in 13m33s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m58s
publish-workspace-server-image / Production auto-deploy (push) Successful in 12m33s
CI / Canvas Deploy Reminder (push) Successful in 1s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m28s
sync-providers-yaml / Compare synced providers.yaml against controlplane canonical (push) Successful in 3s
SECRET_PATTERNS drift lint / Detect SECRET_PATTERNS drift (push) Successful in 56s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 45s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 35m15s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push) Successful in 1m0s
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m13s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 6s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 5m58s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Has been skipped
E2E Chat / detect-changes (push) Successful in 7s
E2E Chat / E2E Chat (push) Successful in 4m4s
E2E Legacy Advisory / Legacy local-platform E2E (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m4s
main-red-watchdog / watchdog (push) Successful in 1m57s
gate-check-v3 / gate-check (push) Successful in 25s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 1m7s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 5s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 4s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 26s
2026-05-29 23:17:02 +00:00
core-devops 947cc730ba test(e2e): give google-adk a hermes-class online window
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 7s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
CI / Python Lint & Test (pull_request) Successful in 3s
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 6s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 51s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Failing after 1m18s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 1m12s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m27s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m7s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m29s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m27s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 37s
gate-check-v3 / gate-check (pull_request) Successful in 9s
security-review / approved (pull_request) Failing after 5s
qa-review / approved (pull_request) Failing after 5s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 3s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 7s
CI / Platform (Go) (pull_request) Successful in 2s
CI / Canvas (Next.js) (pull_request) Successful in 1s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 3s
E2E Chat / E2E Chat (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
CI / all-required (pull_request) Successful in 14m30s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 38m40s
audit-force-merge / audit (pull_request) Successful in 7s
First cold boot of a google-adk workspace pulls a large fresh ADK image;
the default 300s online wait can read a slow first pull as "failed".
Bump google-adk's wait to 180 iters (900s), matching the rationale for
hermes' extended window. No behavior change for other runtimes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 14:51:58 -07:00
core-devops 1ee864d523 test(e2e): wire google-adk into the runtime e2e suite
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 5s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 6s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 34s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Failing after 1m19s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 1m12s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m17s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m13s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m38s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 3s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m9s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 48s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Failing after 4s
security-review / approved (pull_request) Failing after 3s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 4s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 4s
CI / Platform (Go) (pull_request) Successful in 2s
CI / Canvas (Next.js) (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
CI / all-required (pull_request) Successful in 10m30s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 48m20s
google-adk was registered (manifest, provisioner, canvas, CP pin +
allowlist) but had no e2e coverage. Add it everywhere the other
runtimes sit so it is exercised "like other runtimes":

- scripts/test-all-runtimes-a2a-e2e.sh: provision + provider-key +
  online + A2A round-trip + session-continuity loops now include
  google-adk (5 runtimes). AI-Studio key via GOOGLE_API_KEY → workspace
  secret; SKIP_GOOGLE_ADK guard mirrors the other SKIP_* flags.
- e2e-staging-saas.yml + continuous-synth-e2e.yml: add the
  `google-adk)` per-runtime LLM-key case (expects
  MOLECULE_STAGING_GOOGLE_API_KEY) + E2E_GOOGLE_API_KEY env + the
  gemini model slug. Same dispatch-gated shape as codex/hermes/langgraph
  (Gitea drops workflow_dispatch.inputs, so E2E_RUNTIME-driven).

Auth note: PROD disallows API keys (Vertex+ADC there); CI uses the
keyed AI-Studio path (config model google_genai:gemini-2.5-pro). Vertex
stays the supported prod path. The MOLECULE_STAGING_GOOGLE_API_KEY
secret must be set for a green google-adk run (documented in-file).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 14:17:51 -07:00
hongming f84f9a5572 Merge pull request 'feat: register google-adk runtime (manifest + knownRuntimes + canvas)' (#2003) from feat/register-google-adk-runtime into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Block internal-flavored paths / Block forbidden paths (push) Successful in 7s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 13s
CI / Detect changes (push) Successful in 9s
CI / Python Lint & Test (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 8s
E2E API Smoke Test / detect-changes (push) Successful in 8s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 6s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 38s
Handlers Postgres Integration / detect-changes (push) Successful in 5s
Harness Replays / detect-changes (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 7s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 32s
publish-canvas-image / Build & push canvas image (push) Successful in 2m2s
publish-workspace-server-image / build-and-push (push) Successful in 3m12s
CI / Platform (Go) (push) Successful in 4m22s
CI / Shellcheck (E2E scripts) (push) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m31s
CI / Canvas (Next.js) (push) Successful in 5m8s
CI / all-required (push) Successful in 11m48s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m32s
Harness Replays / Harness Replays (push) Successful in 1s
E2E Chat / E2E Chat (push) Successful in 3m32s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
publish-workspace-server-image / Production auto-deploy (push) Successful in 10m48s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 6m52s
CI / Canvas Deploy Reminder (push) Successful in 1s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 38m14s
ci-required-drift / drift (push) Successful in 1m16s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 6s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 4s
main-red-watchdog / watchdog (push) Successful in 2m13s
gate-check-v3 / gate-check (push) Successful in 37s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 15s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 15s
2026-05-29 18:51:46 +00:00
core-devops f82a980a79 test(canvas): add Google ADK to CreateWorkspaceDialog runtime-options assertion
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 6s
Check migration collisions / Migration version collision check (pull_request) Successful in 6s
CI / Detect changes (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 3s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (pull_request) Successful in 3s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 8s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (pull_request) Successful in 44s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 37s
E2E Staging External Runtime / E2E Staging External Runtime (pull_request) Successful in 5m18s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 6s
Harness Replays / detect-changes (pull_request) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m11s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 32s
gate-check-v3 / gate-check (pull_request) Failing after 4s
qa-review / approved (pull_request) Failing after 4s
security-review / approved (pull_request) Failing after 3s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 3s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 4s
CI / Platform (Go) (pull_request) Successful in 5m1s
CI / Canvas (Next.js) (pull_request) Successful in 4m54s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 1s
CI / all-required (pull_request) Successful in 25m28s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m22s
E2E Chat / E2E Chat (pull_request) Successful in 3s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
Harness Replays / Harness Replays (pull_request) Successful in 1s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 37m39s
audit-force-merge / audit (pull_request) Successful in 7s
RUNTIME_OPTIONS gained 'Google ADK' but the test's hardcoded expected array
(separate-selectors test) still listed 4 → Canvas (Next.js) CI red (5 vs 4).
Add it in component order (after OpenAI Codex CLI). Caught by comprehensive
pre-merge review — a real regression from this PR's own diff, not the
staging-E2E infra flake.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 11:02:14 -07:00
hongming 64b7ecfb70 Merge pull request 'feat(budget): multi-period per-workspace LLM budget (hourly/daily/weekly/monthly)' (#2009) from feat/mc-multiperiod-workspace-budget into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 13s
Block internal-flavored paths / Block forbidden paths (push) Successful in 4s
CI / Detect changes (push) Successful in 5s
CI / Python Lint & Test (push) Successful in 6s
E2E API Smoke Test / detect-changes (push) Successful in 7s
E2E Chat / detect-changes (push) Successful in 5s
publish-canvas-image / Build & push canvas image (push) Successful in 1m32s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push) Successful in 1m6s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 9s
publish-workspace-server-image / build-and-push (push) Successful in 3m50s
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m9s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 1m5s
Handlers Postgres Integration / detect-changes (push) Successful in 3s
Harness Replays / detect-changes (push) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 3s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 5s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 1m5s
CI / Platform (Go) (push) Successful in 5m12s
CI / Shellcheck (E2E scripts) (push) Successful in 3s
CI / Canvas (Next.js) (push) Successful in 5m58s
CI / all-required (push) Successful in 22m41s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m37s
publish-workspace-server-image / Production auto-deploy (push) Successful in 20m53s
E2E Chat / E2E Chat (push) Successful in 3m5s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 7m0s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m44s
Harness Replays / Harness Replays (push) Successful in 1s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Failing after 32m28s
CI / Canvas Deploy Reminder (push) Successful in 1s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 35m52s
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m48s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
main-red-watchdog / watchdog (push) Successful in 2m5s
gate-check-v3 / gate-check (push) Successful in 25s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 6s
ci-required-drift / drift (push) Successful in 1m18s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 35s
2026-05-29 11:45:32 +00:00
core-be cf7b587f16 feat(budget): multi-period per-workspace LLM budget (hourly/daily/weekly/monthly)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 9s
CI / Detect changes (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 3s
Check migration collisions / Migration version collision check (pull_request) Successful in 16s
E2E API Smoke Test / detect-changes (pull_request) Successful in 5s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (pull_request) Has been skipped
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 8s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (pull_request) Successful in 1m10s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 59s
E2E Staging External Runtime / E2E Staging External Runtime (pull_request) Successful in 5m15s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
Harness Replays / detect-changes (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m30s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 5s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 58s
gate-check-v3 / gate-check (pull_request) Successful in 6s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 3s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 6s
CI / Platform (Go) (pull_request) Successful in 5m4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
CI / Canvas (Next.js) (pull_request) Successful in 5m6s
E2E Chat / E2E Chat (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 3s
CI / all-required (pull_request) Successful in 19m21s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m28s
Harness Replays / Harness Replays (pull_request) Successful in 4s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1m24s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
qa-review / approved (pull_request) Refired via /qa-recheck by unknown
security-review / approved (pull_request) Refired via /security-recheck by unknown
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 30m49s
audit-force-merge / audit (pull_request) Successful in 4s
Extends the single monthly per-workspace budget to four independent ROLLING
windows so a workspace can be capped per hour/day/week/month (#49 — gives the
canvas Budget tab a real lever against runaway LLM spend, e.g. the reno-stars
opus drain). SSOT design:

- budget_periods.go = single source of truth: the period set + rolling windows,
  one FILTERed per-period spend query over the ledger, and the PURE
  parse/encode/exceededPeriods logic. Add a period = one line here.
- migration: workspaces.budget_limits jsonb (canonical config, backfilled from
  the legacy monthly budget_limit) + workspace_spend_events ledger.
- heartbeat (registry.go): derive the spend INCREMENT from the agent's existing
  cumulative report (delta vs prev; reset-aware) → ledger row. Server owns
  windowing; NO runtime change.
- budget.go GET/PATCH: per-period limit/spend/remaining; accepts the new
  {budget_limits:{...}} shape AND the legacy {budget_limit} (→ monthly); legacy
  response fields still emitted + budget_limit kept synced (rollout back-compat).
  A limit of 0 = block-all (preserved); null/absent = no limit.
- a2a_proxy.go checkWorkspaceBudget: 402 if ANY configured period's rolling
  window spend >= its limit; fail-open on DB error.
- canvas BudgetSection: four period rows (USD limit input + spend/limit + bar).

Tests: pure SSOT (parse/encode/exceededPeriods); GET/PATCH + multi-period +
A2A enforcement (sqlmock, migrated to the new two-query flow); shared
expectBudgetCheck helpers updated; canvas behavioral + per-period progress/aria.
go build + vet + full handlers suite + migrations + canvas vitest all green.

NOTE: the duplicate components/__tests__/BudgetSection.test.tsx (old single-limit
UI) was repurposed to a focused per-period progress/aria suite — behavioral
coverage now lives in tabs/__tests__/BudgetSection.test.tsx (one component, no
parallel identical suites).

Refs #49.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 04:11:21 -07:00
hongming ffbd1a7ff0 Merge pull request 'feat(admin-schedules): orphan monitor + cleaner endpoints (internal#2006 backstops)' (#2008) from feat/schedule-orphan-monitor-cleaner into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 10s
Block internal-flavored paths / Block forbidden paths (push) Successful in 4s
CI / Detect changes (push) Successful in 6s
CI / Python Lint & Test (push) Successful in 6s
E2E API Smoke Test / detect-changes (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 6s
Harness Replays / detect-changes (push) Successful in 4s
Handlers Postgres Integration / detect-changes (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 3s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 55s
CI / Canvas (Next.js) (push) Successful in 2s
CI / Shellcheck (E2E scripts) (push) Successful in 3s
publish-workspace-server-image / build-and-push (push) Successful in 3m9s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m40s
CI / Platform (Go) (push) Successful in 4m51s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 4s
CI / all-required (push) Successful in 6m15s
E2E Chat / E2E Chat (push) Successful in 3m23s
Harness Replays / Harness Replays (push) Successful in 4s
CI / Canvas Deploy Reminder (push) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m38s
publish-workspace-server-image / Production auto-deploy (push) Successful in 5m11s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
main-red-watchdog / watchdog (push) Successful in 58s
gate-check-v3 / gate-check (push) Successful in 24s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 10s
ci-required-drift / drift (push) Successful in 1m34s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 8s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 12s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
2026-05-29 09:38:34 +00:00
hongming feb2b8cfb8 Merge pull request 'fix(org-import): migrate runtime schedules from removed predecessor on recreate' (#2007) from fix/schedule-migration-on-recreate into main
ci-arm64-advisory / fast-checks (push) Waiting to run
CI / Platform (Go) (push) Blocked by required conditions
CI / Canvas (Next.js) (push) Blocked by required conditions
CI / Shellcheck (E2E scripts) (push) Blocked by required conditions
CI / Canvas Deploy Reminder (push) Blocked by required conditions
E2E API Smoke Test / E2E API Smoke Test (push) Blocked by required conditions
E2E Chat / E2E Chat (push) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (push) Blocked by required conditions
Harness Replays / Harness Replays (push) Blocked by required conditions
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 9s
publish-workspace-server-image / build-and-push (push) Successful in 6m54s
Block internal-flavored paths / Block forbidden paths (push) Successful in 7s
CI / Detect changes (push) Successful in 11s
CI / Python Lint & Test (push) Successful in 5s
E2E API Smoke Test / detect-changes (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 13s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 12s
Handlers Postgres Integration / detect-changes (push) Successful in 5s
Harness Replays / detect-changes (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 7s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 6s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 5s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 13s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 49s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
CI / all-required (push) Compensated by status-reaper (push run was cancelled/superseded; Gitea 1.22.6 reports cancelled runs as failure statuses)
publish-workspace-server-image / Production auto-deploy (push) Compensated by status-reaper (push run was cancelled/superseded; Gitea 1.22.6 reports cancelled runs as failure statuses)
2026-05-29 09:28:59 +00:00
hongming-personal 4bee6cb4a7 feat(admin-schedules): orphan monitor + cleaner endpoints (backstops)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
CI / Detect changes (pull_request) Successful in 6s
CI / Python Lint & Test (pull_request) Successful in 4s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
Harness Replays / detect-changes (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 3s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m15s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 3s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 44s
gate-check-v3 / gate-check (pull_request) Successful in 6s
qa-review / approved (pull_request) Failing after 4s
security-review / approved (pull_request) Failing after 4s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 4s
sop-tier-check / tier-check (pull_request) Successful in 4s
CI / Canvas (Next.js) (pull_request) Successful in 25s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 5s
E2E Chat / E2E Chat (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 6s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 3m14s
Harness Replays / Harness Replays (pull_request) Successful in 5s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m53s
CI / Platform (Go) (pull_request) Successful in 8m18s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / all-required (pull_request) Successful in 10m45s
audit-force-merge / audit (pull_request) Successful in 5s
internal#2006 — backstops for the recreate-orphans-schedules class. The
primary fix is migration-on-recreate (separate PR); these are defense-in-depth
so a future regression is detected + recoverable instead of silent.

GET /admin/schedules/health reports only LIVE workspaces' schedules
(JOIN … WHERE status != 'removed'), so a schedule stranded on a
removed/recreated workspace silently stops firing and never shows there —
which is exactly why tonight's orphans went unnoticed.

- GET /admin/schedules/orphans (Orphans): the monitor surface — lists every
  schedule bound to a removed OR missing workspace (id, name, source, enabled,
  ws_status). A monitor polls this and pages on non-empty.
- POST /admin/schedules/reap-orphans (ReapOrphans): the cleaner — re-points
  runtime schedules onto the live successor agent (matched by role+parent),
  then disables any remaining dead-bound schedules so the scheduler stops
  firing into removed workspaces. Idempotent; returns {repointed, disabled}.

Health() is unchanged (no churn to its tests). +2 tests, +2 routes. Build +
handler tests green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 09:08:40 +00:00
hongming-personal a44a110d60 fix(org-import): migrate runtime schedules from removed predecessor on recreate
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
CI / Canvas Deploy Reminder (pull_request) Blocked by required conditions
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 10s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 6s
CI / Python Lint & Test (pull_request) Successful in 4s
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
E2E Chat / detect-changes (pull_request) Successful in 10s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
Harness Replays / detect-changes (pull_request) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 6s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m44s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 3s
gate-check-v3 / gate-check (pull_request) Successful in 6s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 28s
qa-review / approved (pull_request) Failing after 9s
security-review / approved (pull_request) Failing after 3s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 5s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 5s
CI / Canvas (Next.js) (pull_request) Successful in 3s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m22s
CI / Platform (Go) (pull_request) Successful in 4m58s
E2E Chat / E2E Chat (pull_request) Successful in 3s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 7s
CI / all-required (pull_request) Successful in 23m53s
Harness Replays / Harness Replays (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1m25s
audit-force-merge / audit (pull_request) Successful in 7s
internal#2006 — recreating an agent orphans its schedules.

Root cause: createWorkspaceTree's INSERT … ON CONFLICT (parent_id,name)
WHERE status != 'removed' only matches NON-removed rows, so when an agent
is recreated after its prior workspace was marked removed, a brand-new
workspace id is minted. Reconcile then re-derives template-sourced state
(MODEL, template schedules via the upsert loop), but schedules a user added
at runtime (source='runtime', via the canvas/API) bind to the ephemeral
workspace_id and are abandoned on the removed row — they silently stop
firing (the 2026-05-29 agents-team incident: all 5 *-autonomous-tick
schedules, source=runtime, orphaned on removed ids; canvas showed
"missing schedulers").

Fix: after a fresh insert, migrate runtime-created schedules from the
most-recent removed predecessor of the same agent onto the new workspace.
The predecessor is matched by the stable `role` (survives the name
auto-suffixing that yields "Agent (2)"), falling back to name+parent.
Template-sourced schedules are NOT migrated (reconcile re-derives those);
runs before the template upsert loop so a same-named template schedule
still wins; skips names already present on the new workspace; best-effort
(logs, never errors the import).

Tests: predecessor-found re-points; no-predecessor (first create) does NOT
run the UPDATE; name-fallback branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 09:03:27 +00:00
core-lead 02917decc6 Merge pull request 'docs: correct fabricated google-adk + gemini-cli runtime tutorials' (#2004) from fix/google-adk-runtime-doc-accuracy into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 11s
Block internal-flavored paths / Block forbidden paths (push) Successful in 10s
CI / Detect changes (push) Successful in 10s
CI / Python Lint & Test (push) Successful in 5s
E2E API Smoke Test / detect-changes (push) Successful in 7s
CI / all-required (push) Successful in 2m42s
Handlers Postgres Integration / detect-changes (push) Successful in 6s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 5s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 5s
CI / Platform (Go) (push) Successful in 3s
CI / Canvas (Next.js) (push) Successful in 4s
CI / Shellcheck (E2E scripts) (push) Successful in 7s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 8s
CI / Canvas Deploy Reminder (push) Successful in 3s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 51s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
publish-workspace-server-image / build-and-push (push) Successful in 5m12s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2m7s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
publish-workspace-server-image / Production auto-deploy (push) Successful in 2m20s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 42s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 43m56s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push) Successful in 1m27s
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m9s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 9s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Failing after 32m15s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 8m4s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 5s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 6s
E2E Chat / E2E Chat (push) Successful in 4m48s
gate-check-v3 / gate-check (push) Successful in 26s
main-red-watchdog / watchdog (push) Successful in 2m30s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 16s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 14s
E2E Legacy Advisory / Legacy local-platform E2E (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
ci-required-drift / drift (push) Successful in 1m25s
2026-05-29 06:26:26 +00:00
core-devops 1b543d8582 docs: correct fabricated google-adk + gemini-cli runtime tutorials
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Python Lint & Test (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 6s
E2E API Smoke Test / detect-changes (pull_request) Successful in 9s
E2E Chat / detect-changes (pull_request) Successful in 8s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 5s
CI / all-required (pull_request) Successful in 38s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 7s
Harness Replays / detect-changes (pull_request) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 3s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 43s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Successful in 4s
security-review / approved (pull_request) Successful in 4s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 4s
sop-checklist / review-refire (pull_request) Has been skipped
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m7s
sop-tier-check / tier-check (pull_request) Successful in 6s
CI / Platform (Go) (pull_request) Successful in 4s
CI / Canvas (Next.js) (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 5s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2s
Harness Replays / Harness Replays (pull_request) Successful in 2s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 6s
Both tutorials cited misattributed PRs and claimed shipped runtimes that
didn't exist (RFC internal#730 finding):
- google-adk-runtime.md: cited 'PR #550' (actually a MemoryTab test suite) +
  'already first-class'. Rewritten to the REAL implementation — ADK engine-only
  (google-adk[mcp]==2.1.0, no [a2a]), Vertex AI via ADC (keyless), a2a-1.x
  bridge — with correct PR refs (template PR #1, core #2003, ci #26) + a
  landing-status banner.
- gemini-cli-runtime.md: cited 'PR #379' (actually CI cleanup); no gemini-cli
  runtime exists in manifest/knownRuntimes. Added a correction banner pointing
  to the real google-adk runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 23:07:13 -07:00
hongming 3f15c1448d Merge pull request 'fix(secrets): drop retired org-level guard from SetGlobal (global vendor keys are tenant-owned)' (#2002) from fix/setglobal-drop-retired-org-billing-guard into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 12s
publish-workspace-server-image / build-and-push (push) Successful in 4m10s
Block internal-flavored paths / Block forbidden paths (push) Successful in 4s
CI / Detect changes (push) Successful in 5s
CI / Python Lint & Test (push) Successful in 3s
E2E API Smoke Test / detect-changes (push) Successful in 6s
E2E Chat / detect-changes (push) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 6s
Handlers Postgres Integration / detect-changes (push) Successful in 3s
Harness Replays / detect-changes (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 3s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 3s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 3s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 38s
CI / Canvas (Next.js) (push) Successful in 3s
CI / Shellcheck (E2E scripts) (push) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m55s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 8s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2m0s
Harness Replays / Harness Replays (push) Successful in 1s
CI / Platform (Go) (push) Successful in 5m38s
CI / all-required (push) Successful in 27m42s
E2E Chat / E2E Chat (push) Successful in 4m2s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
SECRET_PATTERNS drift lint / Detect SECRET_PATTERNS drift (push) Successful in 34s
publish-workspace-server-image / Production auto-deploy (push) Successful in 25m46s
CI / Canvas Deploy Reminder (push) Successful in 3s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 8s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
main-red-watchdog / watchdog (push) Successful in 28s
gate-check-v3 / gate-check (push) Successful in 29s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 20s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 1m1s
2026-05-29 04:38:13 +00:00
core-devops 0359912d06 feat: register google-adk runtime (manifest + knownRuntimes + canvas)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 18s
CI / Python Lint & Test (pull_request) Successful in 7s
CI / Detect changes (pull_request) Successful in 16s
E2E API Smoke Test / detect-changes (pull_request) Successful in 8s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 4s
Harness Replays / detect-changes (pull_request) Successful in 3s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 34s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 5s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 9s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 6s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 55s
gate-check-v3 / gate-check (pull_request) Successful in 4s
qa-review / approved (pull_request) Failing after 6s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m26s
security-review / approved (pull_request) Failing after 10s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-tier-check / tier-check (pull_request) Successful in 8s
sop-checklist / all-items-acked (pull_request) Successful in 9s
CI / Platform (Go) (pull_request) Successful in 4m30s
CI / Canvas (Next.js) (pull_request) Failing after 4m54s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 1s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / all-required (pull_request) Failing after 18m25s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m18s
E2E Chat / E2E Chat (pull_request) Successful in 3s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 3s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2s
Harness Replays / Harness Replays (pull_request) Successful in 1s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 42m20s
Platform-side registration for the google-adk workspace runtime (RFC
internal#730). Required so a workspace with runtime: google-adk provisions
(Docker path) and is creatable from the canvas:
- manifest.json: workspace_templates entry → handler allowlist (loadRuntimesFromManifest)
- provisioner/registry.go: knownRuntimes += google-adk (else ErrUnresolvableRuntime); test count 4→5
- canvas CreateWorkspaceDialog: RUNTIME_OPTIONS + BASE_RUNTIME_TEMPLATE_IDS
- canvas runtime-names.ts: display name

Depends on molecule-ai-workspace-template-google-adk (image build/publish) +
controlplane runtime_image_pins (SaaS path) — tracked in RFC #730.
Verified: go build + provisioner/handlers tests green; manifest.json valid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 21:30:36 -07:00
hongming-personal 2cf7d006a9 fix(secrets): drop retired org-level guard from SetGlobal — global vendor keys are tenant-owned
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 7s
CI / Detect changes (pull_request) Successful in 7s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 8s
Harness Replays / detect-changes (pull_request) Successful in 7s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 8s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 11s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 6s
qa-review / approved (pull_request) Failing after 15s
gate-check-v3 / gate-check (pull_request) Successful in 15s
security-review / approved (pull_request) Failing after 7s
sop-checklist / na-declarations (pull_request) N/A: (none)
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 31s
sop-checklist / all-items-acked (pull_request) Successful in 6s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 4s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m3s
CI / Canvas (Next.js) (pull_request) Successful in 6s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 4s
E2E Chat / E2E Chat (pull_request) Successful in 12s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m4s
Harness Replays / Harness Replays (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 6s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m29s
CI / Platform (Go) (pull_request) Successful in 7m12s
CI / all-required (pull_request) Successful in 11m6s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 4s
internal#718 retired the org-level LLM billing rung (billing is resolved
per-workspace now). SetGlobal still called the legacy org-env guard
rejectPlatformManagedDirectLLMBypass, which reads MOLECULE_LLM_BILLING_MODE and
400s any vendor/oauth key write when the (legacy) org default is
platform_managed. That blocked setting a tenant's own MINIMAX_API_KEY (or any
custom-provider key) at global scope on a byok tenant — agents-team hit "direct
Hermes custom provider secrets are blocked for platform-managed LLM workspaces".

A global secret is the tenant's OWN shared credential. The provision-time
provider-matched strip (workspace_provision, core#2000) already removes any
global cred a given workspace's resolved provider does not accept, and the
platform-managed path strips bypass keys at provision too — so a platform-managed
workspace can never USE a non-matching global vendor/oauth key. The SetGlobal
org-env gate was redundant belt-and-suspenders keyed off the retired rung.

- SetGlobal: remove the org-level guard call.
- Delete the now-dead legacy helpers platformManagedLLMMode +
  rejectPlatformManagedDirectLLMBypass (org-env shims; the per-workspace
  successors rejectPlatformManagedDirectLLMBypassForWorkspace /
  platformManagedLLMModeForWorkspace remain and still gate per-workspace writes).
- Tests: convert the obsolete platform-managed rejection test into
  TestSetGlobal_AllowsTenantOwnedVendorKeyDespiteLegacyOrgEnv (asserts the global
  write SUCCEEDS even with the legacy env still set to platform_managed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 04:26:28 +00:00
hongming c99b0e3601 Merge pull request 'fix(workspace-server): provider-matched byok credential injection (internal#728 Bug 1) [BEHAVIOR-AFFECTING — CTO merge-go]' (#2000) from fix/internal-728-provider-matched-cred-injection into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Block internal-flavored paths / Block forbidden paths (push) Successful in 8s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 14s
CI / Detect changes (push) Successful in 9s
CI / Python Lint & Test (push) Successful in 4s
E2E API Smoke Test / detect-changes (push) Successful in 14s
E2E Chat / detect-changes (push) Successful in 11s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 31s
Handlers Postgres Integration / detect-changes (push) Successful in 7s
Harness Replays / detect-changes (push) Successful in 9s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 46s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 9s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 9s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 6s
CI / Canvas (Next.js) (push) Successful in 5s
CI / Shellcheck (E2E scripts) (push) Successful in 15s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 42s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 4m47s
publish-workspace-server-image / build-and-push (push) Successful in 6m21s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1m40s
CI / Platform (Go) (push) Successful in 5m52s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 9s
CI / all-required (push) Successful in 7m54s
Harness Replays / Harness Replays (push) Successful in 7s
CI / Canvas Deploy Reminder (push) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m31s
E2E Chat / E2E Chat (push) Successful in 4m23s
publish-workspace-server-image / Production auto-deploy (push) Successful in 3m41s
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m15s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 4s
main-red-watchdog / watchdog (push) Successful in 2m5s
gate-check-v3 / gate-check (push) Successful in 25s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 1m5s
sync-providers-yaml / Compare synced providers.yaml against controlplane canonical (push) Successful in 4s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 7s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Has started running
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 11s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
2026-05-29 00:29:07 +00:00
hongming 4414c92a87 fix(workspace-server): provider-matched byok credential injection — strip stray non-matching global-origin LLM creds (internal#728 Bug 1) [BEHAVIOR-AFFECTING — CTO merge-go]
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Python Lint & Test (pull_request) Successful in 9s
CI / Detect changes (pull_request) Successful in 11s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 13s
E2E API Smoke Test / detect-changes (pull_request) Successful in 12s
E2E Chat / detect-changes (pull_request) Successful in 12s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 31s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 6s
Harness Replays / detect-changes (pull_request) Successful in 5s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 6s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 5s
gate-check-v3 / gate-check (pull_request) Successful in 5s
qa-review / approved (pull_request) Failing after 6s
security-review / approved (pull_request) Failing after 10s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 25s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 5s
sop-checklist / review-refire (pull_request) Has been skipped
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m10s
sop-tier-check / tier-check (pull_request) Successful in 5s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m37s
CI / Canvas (Next.js) (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 4s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 10s
CI / Platform (Go) (pull_request) Successful in 4m37s
Harness Replays / Harness Replays (pull_request) Successful in 3s
CI / all-required (pull_request) Successful in 7m16s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1m33s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1m42s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 7s
#1995 removed the blanket global-LLM-cred strip on the byok branch (correct for
the platform-key co-mingling it targeted), but left EVERY claude-code workspace
inheriting the tenant-global CLAUDE_CODE_OAUTH_TOKEN. The claude-code runtime
greedily prefers that oauth (llm-auth: detected oauth -> api.anthropic.com), so
a workspace whose RESOLVED provider is NOT anthropic-oauth (minimax, kimi-byok)
routes its non-Anthropic model to Anthropic -> "Claude Code returned an error
result" (agents-team Dev Engineer B, MiniMax-M2.7; live-confirmed 2026-05-28 via
SSM container logs, internal#728 comment 52493).

Fix: provider-AWARE replacement for the over-removed strip. On the byok/disabled
branch, keep ONLY the global-origin LLM bypass creds whose env-var name is in
the RESOLVED provider's auth_env; strip the rest.
- minimax auth_env MINIMAX_API_KEY/ANTHROPIC_AUTH_TOKEN/ANTHROPIC_API_KEY ->
  stray global CLAUDE_CODE_OAUTH_TOKEN is non-matching -> stripped (fixes DevB).
- anthropic-oauth auth_env CLAUDE_CODE_OAUTH_TOKEN -> matches -> kept (PM opus +
  reno opus-byok NOT regressed; #1994 ByokGlobalScopeOAuthSurvives guard holds).
NOT a return to the blanket strip (which would re-break the byok-anthropic-oauth
case #1994 fixed) — keyed off DeriveProvider's resolved provider.

Provenance-scoped: only operator-store (global_secrets) origin keys are
provider-gated. User-authored workspace_secrets (provenance flag cleared by
loadWorkspaceSecrets) are NEVER stripped — JRS kimi workspace-key, reno's own
oauth are exempt. Fail-OPEN: an underivable provider / unavailable registry
strips nothing (keep-first; worst case is a kept stray, never removing the only
usable cred -> never fail-closes a legitimate byok workspace).

Threads loadWorkspaceSecrets's globalKeys provenance side-channel into
applyPlatformManagedLLMEnv (signature +map[string]struct{}); caller
prepareProvisionContext already has it.

Tests (llm_billing_mode_provision_parity_test.go):
- MinimaxStripsStrayGlobalOAuth — DevB repro: minimax-resolving ws strips the
  stray global oauth + keeps MINIMAX_API_KEY routing.
- WorkspaceOriginCredExemptFromStrip — user-authored ws_secrets cred survives
  even when non-matching.
- ByokGlobalScopeOAuthSurvives (strengthened) — global-origin oauth on opus
  SURVIVES via provider match (PM/reno regression guard).
Mutation-load-bearing (verified RED): (1) remove strip -> blanket-keep regresses
DevB; (2) empty keep set (provider-unaware) -> minimax routing + reno oauth
stripped; (3) iterate all bypass keys (provenance-unaware) -> user-authored cred
stripped.

build ok; build -tags=integration ok; go test ./internal/handlers/ ok;
golangci-lint ./internal/handlers/ -> 0 issues. Refs internal#728.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 00:05:21 +00:00
hongming efa60621f3 Merge pull request 'fix(prod-auto-deploy): fail on tenants not verified on target build (internal#724)' (#1998) from fix/internal-724-prod-auto-deploy-straggler-surfacing into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 13s
Block internal-flavored paths / Block forbidden paths (push) Successful in 16s
CI / Detect changes (push) Successful in 19s
CI / Python Lint & Test (push) Successful in 27s
E2E API Smoke Test / detect-changes (push) Successful in 21s
E2E Chat / detect-changes (push) Successful in 19s
CI / all-required (push) Successful in 2m0s
Handlers Postgres Integration / detect-changes (push) Successful in 23s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 27s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 9s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 9s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 10s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (push) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 6s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m20s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 40s
Ops Scripts Tests / Ops scripts (unittest) (push) Successful in 1m21s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (push) Successful in 1m36s
publish-workspace-server-image / build-and-push (push) Successful in 5m25s
CI / Platform (Go) (push) Successful in 2s
CI / Canvas (Next.js) (push) Successful in 3s
CI / Shellcheck (E2E scripts) (push) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 4s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 2s
E2E Chat / E2E Chat (push) Successful in 3s
CI / Canvas Deploy Reminder (push) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m27s
publish-workspace-server-image / Production auto-deploy (push) Successful in 2m30s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 7s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 16s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
main-red-watchdog / watchdog (push) Successful in 2m1s
gate-check-v3 / gate-check (push) Successful in 24s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 4s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 10s
ci-required-drift / drift (push) Successful in 1m28s
2026-05-28 21:58:31 +00:00
hongming-personal 367bc1f7fc fix(prod-auto-deploy): fail on tenants not verified on target build (internal#724)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 7s
CI / Python Lint & Test (pull_request) Successful in 9s
CI / Detect changes (pull_request) Successful in 11s
E2E Chat / detect-changes (pull_request) Successful in 20s
CI / all-required (pull_request) Successful in 2m42s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 18s
E2E API Smoke Test / detect-changes (pull_request) Successful in 20s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 12s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 12s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 8s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 8s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Failing after 1m4s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m12s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 1m21s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m31s
gate-check-v3 / gate-check (pull_request) Successful in 8s
qa-review / approved (pull_request) Failing after 5s
security-review / approved (pull_request) Failing after 7s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 35s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 4s
sop-checklist / all-items-acked (pull_request) acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, local-postgres-e2
sop-checklist / na-declarations (pull_request) N/A: (none)
CI / Platform (Go) (pull_request) Successful in 5s
CI / Canvas (Next.js) (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 6s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m34s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m7s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 4s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 17s
The production auto-deploy aggregated per-tenant redeploy-fleet results
but never asserted fleet COVERAGE: a tenant that was enumerated but
silently skipped, or that SSM-succeeded onto the old image, passed as a
clean deploy. That is how agents-team stayed 46h behind the fleet with no
straggler reported.

Pairs with the controlplane fix that adds per-tenant verified_on_target
(docker-inspect proof the container is on the target tag). This change:

- rollout_stragglers(): every enumerated tenant NOT proven on the target
  build is a straggler — errored, skipped (no result row, the agents-team
  class), or verified_on_target=false. Backward-compatible: a missing key
  (pre-fix CP) is treated as verified so the gate degrades to the old
  ok-based behavior against an un-upgraded CP rather than failing spuriously.
- assert_full_coverage(): raises RolloutFailed (→ non-zero exit, response
  JSON written with ok=false + stragglers) when any straggler remains
  after a non-dry-run rollout. A dry run asserts nothing (it proves
  nothing landed).
- publish-workspace-server-image.yml: per-tenant summary gains an
  "On target" column and a loud ⚠ Stragglers section; the step emits a
  ::error:: naming the off-target tenants before failing.

Tests: straggler detection (off-target, no-result, dry-run-skip,
backward-compat missing key) + end-to-end execute_scoped_rollout fail/pass
— mutation-verified RED with the coverage gate removed. All existing
prod-auto-deploy tests still pass; ruff + py_compile clean; workflow YAML
validates.

Refs: internal#724

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 14:41:12 -07:00
hongming c2c6501a67 Merge pull request 'fix(workspace-server): provision-time billing derives from EFFECTIVE model, not raw payload.Model (#1994) [BEHAVIOR-AFFECTING — CTO merge-go]' (#1995) from fix/1994-provision-billing-model-passthrough into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 10s
Block internal-flavored paths / Block forbidden paths (push) Successful in 8s
CI / Detect changes (push) Successful in 8s
CI / Python Lint & Test (push) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 10s
E2E API Smoke Test / detect-changes (push) Successful in 11s
E2E Chat / detect-changes (push) Successful in 10s
Handlers Postgres Integration / detect-changes (push) Successful in 5s
Harness Replays / detect-changes (push) Successful in 4s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 28s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 4s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 16s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 5s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (push) Successful in 4s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m15s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 5s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Successful in 26s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (push) Successful in 1m37s
CI / Canvas (Next.js) (push) Successful in 14s
publish-workspace-server-image / build-and-push (push) Successful in 4m41s
CI / Shellcheck (E2E scripts) (push) Successful in 39s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Failing after 5m40s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 3m42s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 3s
Harness Replays / Harness Replays (push) Successful in 8s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m53s
CI / Canvas Deploy Reminder (push) Successful in 2s
CI / Platform (Go) (push) Successful in 8m22s
CI / all-required (push) Successful in 11m47s
E2E Chat / E2E Chat (push) Successful in 4m58s
publish-workspace-server-image / Production auto-deploy (push) Successful in 9m9s
main-red-watchdog / watchdog (push) Successful in 1m58s
gate-check-v3 / gate-check (push) Successful in 24s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 8s
ci-required-drift / drift (push) Successful in 1m7s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 5s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 3s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
2026-05-28 20:00:59 +00:00
hongming-ceo-delegated bbb445b956 fix(workspace-server): byok runs on the tenant's own global-scope LLM cred; stop stripping it (molecule-core#1994)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 9s
CI / Python Lint & Test (pull_request) Successful in 8s
E2E API Smoke Test / detect-changes (pull_request) Successful in 10s
E2E Chat / detect-changes (pull_request) Successful in 8s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 12s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 14s
Harness Replays / detect-changes (pull_request) Successful in 7s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 5s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 3s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 33s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 11s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Failing after 1m1s
lint-mask-pr-atomicity / lint-mask-pr-atomicity (pull_request) Successful in 1m13s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 1m11s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 56s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m29s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 5s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 33s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Failing after 4s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m26s
security-review / approved (pull_request) Failing after 5s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 5s
sop-tier-check / tier-check (pull_request) Successful in 6s
CI / Canvas (Next.js) (pull_request) Successful in 2s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 15s
E2E Chat / E2E Chat (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 10s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m23s
Harness Replays / Harness Replays (pull_request) Successful in 5s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m18s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m27s
CI / Platform (Go) (pull_request) Successful in 6m4s
CI / all-required (pull_request) Successful in 9m54s
audit-force-merge / audit (pull_request) Successful in 8s
Corrected-model credential fix (CTO-confirmed). `global_secrets` is the
TENANT's own secret store (shared across that tenant's workspaces), NOT the
platform's. The platform's own LLM credential is the CP proxy usage token,
injected separately on the platform_managed path; it is never stored in a
tenant's global_secrets.

The internal#711 provider-aware strip rested on the inverted premise that a
global-scope LLM credential was "the platform's own". On the byok/disabled
branch it stripped the tenant's OWN oauth when that oauth lived at global
scope, leaving the workspace credential-less -> MISSING_BYOK_CREDENTIAL ->
dead (Reno Stars Marketing/SEO byok agents, live-confirmed 2026-05-28).

Changes:
- workspace_provision.go: remove the stripGlobalOriginLLMCreds call on the
  byok/disabled branch; delete the now-dead function; drop the unused
  globalKeys parameter from applyPlatformManagedLLMEnv.
- secrets.go: remove the symmetric byok strip on the remote-pull path
  (GET /workspaces/:id/secrets/values) + its now-unused globalKeys tracking;
  the bundle is the tenant's merged secrets served verbatim.
- platform_managed path UNCHANGED: still strips direct oauth + forces the CP
  proxy usage token (metered). Only byok/disabled stop being stripped.
- Fail-closed UNCHANGED in spirit: a byok workspace with no LLM credential at
  ANY scope still aborts MISSING_BYOK_CREDENTIAL; the trigger narrowed from
  "no workspace-scoped cred" to "no cred at any scope".

Guard (co-mingling prevention at the write boundary):
- SetGlobal still rejects bypass-list keys for a platform_managed tenant
  (keeps a platform-shaped credential out of global_secrets going forward);
  added a regression test pinning it.

Tests: inverted the strip-asserting unit + e2e tests to the corrected model
(global-scope oauth survives, byok runs direct, no proxy); added genuinely-
credential-less byok fail-closed coverage; all three behavior changes are
mutation-load-bearing (re-adding either strip / dropping the SetGlobal guard
turns the respective test RED). build + vet + golangci-lint + the full
integration-tagged handlers suite green. The #1994 model-passthrough fix and
the MiniMax A2A e2e on this branch are untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 19:45:55 +00:00
hongming-ceo-delegated 3269e93216 test(e2e): add real-completion + per-provider liveness + byok-routing A2A gate (#1994 follow-on)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 11s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 22s
CI / Python Lint & Test (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 38s
E2E API Smoke Test / detect-changes (pull_request) Successful in 22s
E2E Chat / detect-changes (pull_request) Successful in 13s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 10s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 7s
Harness Replays / detect-changes (pull_request) Successful in 5s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 12s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 3s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 34s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Failing after 1m7s
lint-mask-pr-atomicity / lint-mask-pr-atomicity (pull_request) Successful in 1m14s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 1m11s
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 8s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m24s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m3s
gate-check-v3 / gate-check (pull_request) Successful in 5s
qa-review / approved (pull_request) Failing after 8s
security-review / approved (pull_request) Failing after 12s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 27s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 4s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 6s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m25s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m31s
CI / Canvas (Next.js) (pull_request) Successful in 6s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 17s
E2E Chat / E2E Chat (pull_request) Successful in 4s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 8s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m12s
Harness Replays / Harness Replays (pull_request) Successful in 4s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m16s
CI / Platform (Go) (pull_request) Successful in 6m10s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / all-required (pull_request) Successful in 13m28s
The A2A e2e historically asserted only response SHAPE (test_a2a_e2e.sh
checked '"kind":"text"' only). A broken agent returns its error AS a
text part -- {"kind":"text","text":"Agent error (Exception) ..."} --
which STILL matches the shape check, so it PASSED on a fully broken
agent. That is why the 2026-05-2x drained-key / byok-misroute failures
(agents-team PM + reno marketing erroring on every LLM call) sailed
through CI. "Channel returns text shape" is not "agent completed an LLM
round-trip."

Adds, ADDITIVELY (no existing assertion weakened or removed):

- tests/e2e/lib/completion_assert.sh -- reusable gates:
  * a2a_assert_real_completion: deterministic known-answer round-trip;
    asserts CONTAINS the expected token AND NOT an error-as-text marker
    (Agent error / Exception / error result / MISSING_BYOK_CREDENTIAL).
  * provider_liveness_matrix + offered_platform_models_for_runtime:
    per-offered-provider cheap (max_tokens:4) probe; the offered set is
    read from the providers.yaml SSOT (runtimes.<rt>.providers[platform]
    .models) -- not a hardcoded list -- so the matrix tracks the SSOT.
  * assert_byok_not_platform_proxy: #1994 regression guard -- a
    byok-resolving workspace must NOT resolve platform_managed (reads the
    same derived resolver GET /admin/workspaces/:id/llm-billing-mode the
    provision strip gate uses).

- tests/e2e/test_staging_full_saas.sh (the live-agent lane, MiniMax
  primary): new stanzas 8b (PINEAPPLE known-answer, the core gate),
  8c (byok-routing guard), 8d (SSOT-driven per-provider liveness matrix).

- tests/e2e/test_a2a_e2e.sh: added check_no_error_as_text on Echo + SEO
  replies so the brief's literal shape-only example now FAILS on an
  error-as-text payload.

- tests/e2e/test_completion_assert_unit.sh: offline fail-direction proof
  (16 cases) that the negative gates are load-bearing -- error-as-text
  MUST fail, platform_managed MUST trip the #1994 guard. Wired into
  ci.yml "Run E2E bash unit tests (no live infra)" (required, per-PR +
  main). e2e-staging-saas.yml paths filter extended to re-trigger the
  live lane on lib changes.

No #1994 fix code touched -- tests/e2e + workflow wiring only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 18:58:13 +00:00
hongming 442f79a987 fix(workspace-server): provision-time billing derives from EFFECTIVE model, not raw payload.Model (molecule-core#1994)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 7s
CI / Detect changes (pull_request) Successful in 9s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 8s
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 13s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 8s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Harness Replays / detect-changes (pull_request) Successful in 3s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 9s
E2E Staging SaaS (full lifecycle) / pr-validate (pull_request) Successful in 45s
gate-check-v3 / gate-check (pull_request) Successful in 5s
qa-review / approved (pull_request) Failing after 10s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 39s
security-review / approved (pull_request) Failing after 5s
sop-checklist / review-refire (pull_request) Has been skipped
sop-tier-check / tier-check (pull_request) Successful in 6s
CI / Canvas (Next.js) (pull_request) Successful in 3s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 6s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m7s
E2E Chat / E2E Chat (pull_request) Successful in 23s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 6s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m2s
sop-checklist / all-items-acked (pull_request) acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, local-postgres-e2
sop-checklist / na-declarations (pull_request) N/A: (none)
Harness Replays / Harness Replays (pull_request) Successful in 3s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m37s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Successful in 5m10s
CI / Platform (Go) (pull_request) Successful in 5m54s
CI / all-required (pull_request) Successful in 7m19s
The provision-time LLM billing resolver diverged from the read endpoint:
a byok workspace (claude-code, opus) was provisioned platform_managed and
routed through the platform LLM proxy, billing the platform Anthropic key
for the customer own usage (Reno Stars Marketing 6b66de8d; live-confirmed
2026-05-28).

Root cause: applyPlatformManagedLLMEnv passed the RAW payload.Model to
ResolveLLMBillingModeDerived. On a re-provision (restart/resume/
auto-restart) the payload is rebuilt from the DB with Name+Tier+Runtime
only (workspace_restart.go:333/844/1017 via withStoredCompute, which
backfills Compute but NOT Model), so payload.Model == "". DeriveProvider
errors on an empty model, the resolver defaults closed to platform_managed
and bakes ANTHROPIC_BASE_URL=<platform proxy>. The read endpoint
(ResolveLLMBillingMode -> readWorkspaceDeriveInputs) reads MODEL from
workspace_secrets, derives opus -> anthropic-oauth -> byok. Divergence,
deterministic on every re-provision.

Fix: extract effectiveModelForBilling (the fallback chain
applyRuntimeModelEnv already used: explicit -> MOLECULE_MODEL -> MODEL)
into a shared helper and have the billing resolver consult it, so the
provision-path derive inputs match the read-path. The stored model already
lives in the merged envVars (loadWorkspaceSecrets) — no new DB query. The
byok branch (no proxy override; strip only global-origin platform creds;
fail-closed on missing own cred, internal#711) is preserved unchanged;
genuinely-platform and no-model workspaces still default platform_managed
(CTO: default stays platform).

Tests (mutation-load-bearing): re-provision-uses-stored-model byok repro,
read/provision parity guard, default-preservation, and the #711 global-
only-oauth fail-closed guard. Reverting the envVars fallback turns the
repro + parity + #711 tests RED; default-preservation stays GREEN.

BEHAVIOR-AFFECTING (provisioning hot path) — needs CTO merge-go.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 18:45:58 +00:00
hongming 03aa69f46f Merge pull request 'P3 internal#718: canvas consumes registry-served /templates, retire hardcoded provider vocab #4/#5 (PR-B; NOT merged)' (#1978) from feat/internal-718-p3b-canvas-consume-registry into main
ci-arm64-advisory / fast-checks (push) Waiting to run
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (push) Successful in 13s
publish-canvas-image / Build & push canvas image (push) Successful in 3m11s
publish-workspace-server-image / build-and-push (push) Successful in 6m19s
Block internal-flavored paths / Block forbidden paths (push) Successful in 11s
CI / Python Lint & Test (push) Successful in 11s
CI / Detect changes (push) Successful in 12s
E2E API Smoke Test / detect-changes (push) Successful in 12s
Handlers Postgres Integration / detect-changes (push) Successful in 7s
Harness Replays / detect-changes (push) Successful in 10s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (push) Successful in 8s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (push) Successful in 11s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 9s
CI / Platform (Go) (push) Successful in 4s
CI / Shellcheck (E2E scripts) (push) Successful in 3s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 3s
verify-providers-gen / Regenerate providers artifact and fail on drift (push) Failing after 11m15s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2m55s
Harness Replays / Harness Replays (push) Successful in 5s
CI / Canvas (Next.js) (push) Successful in 7m18s
CI / all-required (push) Successful in 21m18s
CI / Canvas Deploy Reminder (push) Successful in 7s
publish-workspace-server-image / Production auto-deploy (push) Successful in 46m7s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 30s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Successful in 5m51s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push) Successful in 1m3s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Has been skipped
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m11s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 6s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 5m53s
E2E Chat / detect-changes (push) Successful in 7s
E2E Chat / E2E Chat (push) Successful in 4m19s
E2E Legacy Advisory / Legacy local-platform E2E (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Failing after 1m24s
gate-check-v3 / gate-check (push) Successful in 34s
main-red-watchdog / watchdog (push) Successful in 2m16s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 18s
ci-required-drift / drift (push) Successful in 1m3s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Successful in 7s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Successful in 4m16s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 5s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 3s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Successful in 6m36s
2026-05-28 05:59:07 +00:00
hongming-personal 8546502ab8 test(canvas): make registryBilling test discriminate registry-vs-hardcoded billing precedence (#1978 review)
ci-arm64-advisory / fast-checks (pull_request) Waiting to run
CI / Platform (Go) (pull_request) Blocked by required conditions
CI / Canvas (Next.js) (pull_request) Blocked by required conditions
CI / Shellcheck (E2E scripts) (pull_request) Blocked by required conditions
CI / all-required (pull_request) Waiting to run
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
E2E Chat / E2E Chat (pull_request) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Harness Replays / Harness Replays (pull_request) Blocked by required conditions
Lint shellcheck (arm64 pilot) / shellcheck-arm64 (pilot) (pull_request) Successful in 12s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
CI / Detect changes (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 4s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 8s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 8s
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 4s
Harness Replays / detect-changes (pull_request) Successful in 5s
Lint no tenant GITEA or GITHUB token write / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 12s
gate-check-v3 / gate-check (pull_request) Successful in 11s
verify-providers-gen / Regenerate providers artifact and fail on drift (pull_request) Successful in 33s
qa-review / approved (pull_request) Failing after 9s
security-review / approved (pull_request) Failing after 7s
sop-checklist / review-refire (pull_request) Has been skipped
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 5s
sop-tier-check / tier-check (pull_request) Successful in 11s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m14s
audit-force-merge / audit (pull_request) Successful in 11s
CI / Canvas Deploy Reminder (pull_request) Has been cancelled
agent-reviewer #7790 (blocking) found that ConfigTab.registryBilling.test.tsx
did not actually pin retire-list #5's core claim — both existing assertions
("platform"→platform_managed, "anthropic-oauth"→byok) return the SAME value
under both the registry-authoritative impl and a regression to the old
hardcoded billingModeForProvider rule, so the test was tautological and a
regression would still pass. The misleading comment on the anthropic-oauth
case claimed it was "a case the hardcoded rule gets WRONG" but the hardcoded
rule actually agrees there too.

This commit adds a genuine disagreement case: a registry provider
"managed-federated" whose registry-served billing_mode is "platform_managed"
even though its name is not "" / "platform" (so the legacy
billingModeForProvider rule would return "byok"). The new test asserts the
two rules disagree on this input (sanity) and then asserts
billingModeForSelectedProvider returns the REGISTRY value
("platform_managed"), which is only reachable by honoring the catalog.

Load-bearing proof: with the registry-first impl, the new test PASSES; when
billingModeForSelectedProvider is temporarily forced to fall through to the
hardcoded rule, the new test (and only the new test) FAILS with
expected 'platform_managed' / received 'byok' — proving it pins the
registry-wins contract.

Also fixes the misleading "hardcoded rule gets WRONG" comment on the
anthropic-oauth case (explicitly annotates it as non-discriminating and
points to the new disagreement case as the registry-WINS proof).

Implementation (billingModeForSelectedProvider) untouched — confirmed
byte-identical to PR #1978 HEAD (f2d7f1da).

Verification:
  - targeted: 5 passed (was 4 — adds the discriminating case)
  - regressed-impl: only the new test fails, others pass (= they are
    non-discriminating as the review found)
  - full canvas vitest: 223 files / 3381 passed | 1 skipped (3382) — +1
    vs the 3380/1 baseline
  - tsc: 0 new errors (touched file clean; pre-existing 223 baseline
    unchanged with my diff stashed)
  - eslint on touched file: 0

Refs: #1978, review #7790, internal#718 P3 retire-list #5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:06:25 +00:00
hongming-personal f5c2882acb feat(canvas): P3 internal#718 — consume registry-served /templates list, retire hardcoded provider vocab (#4/#5)
P3 item 2. The canvas Provider/Model selector + Config-tab billing-mode now
consume the registry-served GET /templates fields (registry_backed /
registry_providers / registry_models from PR-A) instead of re-deriving provider
knowledge client-side. Retires the hardcoded vocabularies as the PRIMARY path:

- ProviderModelSelector (#4): new buildProviderCatalogFromRegistry(providers,
  models) builds the dropdown catalog from the registry payload — provider
  label = registry display_name, bucket = DERIVED provider, billing + auth_env
  from the registry — instead of inferVendor / VENDOR_LABELS /
  BARE_VENDOR_PATTERNS. The selector takes an optional pre-built `catalog`
  prop and uses it verbatim when supplied. inferVendor/buildProviderCatalog
  remain ONLY as the fallback for non-registry runtimes / older backends.
- ConfigTab (#5): when the selected runtime is registry-backed, the provider
  catalog + selector models come from registry_providers/registry_models, and
  billingModeForSelectedProvider(provider, catalog) reads the DERIVED provider's
  billing_mode off the registry catalog. The hardcoded billingModeForProvider
  ('' | 'platform' → platform_managed else byok) stays as the fallback only.
  So the billing-mode the UI shows/sends reflects the DERIVED provider
  (folds in the closed #1931's canvas intent).

Federation/back-compat preserved: a non-registry runtime (external/mock/kimi/
future third-party) or an older backend that doesn't serve the registry fields
yields registry_backed=false → the canvas keeps the template-served models +
its heuristic, unchanged. NO hard-reject (the canvas just can't render an
option the registry didn't serve for registry-backed runtimes).

Out of scope (per brief): the manifest runtime allowlist
(SUPPORTED_RUNTIME_VALUES / FALLBACK_RUNTIME_OPTIONS) is NOT a provider
vocabulary and is untouched; PUT /workspaces/:id/provider is NOT retired (that
CTO #3 follow-through is a later phase).

Stacked on PR-A (workspace-server registry-served /templates); re-target to
main after PR-A merges.

TDD: ProviderModelSelector.registry.test.tsx (catalog bucketed by derived
provider, labelled from display_name, carries billing_mode + auth_env, no empty
buckets), ConfigTab.registryBilling.test.tsx (billing reads registry catalog;
falls back to the legacy rule with no catalog / unknown provider). Full canvas
suite green (3380 passed / 1 skipped), tsc clean for touched files, eslint 0.

internal#718 P3 — not merged; CTO merge-go after Five-Axis (UI-affecting).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:06:25 +00:00
125 changed files with 7453 additions and 2052 deletions
+63
View File
@@ -208,6 +208,61 @@ def _raise_for_redeploy_result(status: int, body: dict, slugs: list[str]) -> Non
)
def rollout_stragglers(enumerated: list[str], results: list[dict]) -> list[str]:
"""Return every enumerated tenant NOT proven on the target build.
A straggler is any tenant the rollout was supposed to cover that the
CP could not verify is running the target image tag — whether it
errored, was skipped, or SSM-succeeded onto the wrong image
(internal#724). CP marks each per-tenant result row with
``verified_on_target`` (the REDEPLOY_RUNNING_IMAGE docker-inspect
proof). A tenant enumerated for the rollout but absent from the
result set (no batch ever ran it) is also a straggler — that is the
exact agents-team silent-skip class.
Backward-compat: an OLDER CP that doesn't emit ``verified_on_target``
yet returns rows without the key. Treat a missing key as verified so
this surfacing degrades to the previous (ok-based) behavior against an
un-upgraded CP, rather than failing every deploy spuriously. Once the
CP fix is deployed the key is always present and real stragglers are
caught.
"""
verified: set[str] = set()
for row in results:
if str(row.get("ssm_status") or "") == "DryRun":
continue
slug = str(row.get("slug") or "").strip()
if not slug:
continue
# Missing key (old CP) => assume verified; present key is authoritative.
if "verified_on_target" not in row or row.get("verified_on_target"):
verified.add(slug)
return sorted(s for s in dict.fromkeys(enumerated) if s not in verified)
def assert_full_coverage(enumerated: list[str], aggregate: dict, dry_run: bool) -> None:
"""Fail the rollout if any enumerated tenant is not on the target build.
This is the no-silent-skip gate (internal#724). A dry run proves
nothing landed, so coverage is not asserted for it.
"""
if dry_run:
return
stragglers = rollout_stragglers(enumerated, aggregate.get("results") or [])
if stragglers:
msg = (
f"incomplete rollout: {len(stragglers)} tenant(s) not verified on target "
f"after redeploy-fleet: {', '.join(stragglers)} "
f"(enumerated {len(set(enumerated))})"
)
aggregate["ok"] = False
aggregate["error"] = msg
aggregate["stragglers"] = stragglers
raise RolloutFailed(msg, aggregate)
def execute_scoped_rollout(
plan: dict,
token: str,
@@ -254,6 +309,14 @@ def execute_scoped_rollout(
aggregate["error"] = str(exc)
raise RolloutFailed(str(exc), aggregate) from exc
# No-silent-skip coverage gate (internal#724): every enumerated tenant
# must be PROVEN on the target build. A per-tenant HTTP-200/ok response
# is not proof — a tenant that SSM-succeeded but stayed on the old tag,
# or one enumerated but never batched, is a straggler. Surfacing it as
# a RolloutFailed makes the deploy step exit non-zero instead of
# silently reporting success (the exact agents-team failure mode).
assert_full_coverage(all_slugs, aggregate, dry_run)
return aggregate
@@ -355,3 +355,134 @@ def test_rollout_from_plan_file_writes_partial_response_on_failure(tmp_path):
assert response_path.read_text(encoding="utf-8").strip()
assert '"ok": false' in response_path.read_text(encoding="utf-8")
assert '"slug": "hongming"' in response_path.read_text(encoding="utf-8")
# ──────────────────────────────────────────────────────────────────────
# No-silent-skip coverage gate (internal#724)
# ──────────────────────────────────────────────────────────────────────
def test_rollout_stragglers_flags_tenant_not_on_target():
# b SSM-succeeded but its container is on the old tag → straggler.
stragglers = prod.rollout_stragglers(
["a", "b", "c"],
[
{"slug": "a", "verified_on_target": True},
{"slug": "b", "verified_on_target": False, "running_image": "platform-tenant:staging-old"},
{"slug": "c", "verified_on_target": True},
],
)
assert stragglers == ["b"]
def test_rollout_stragglers_flags_enumerated_tenant_with_no_result():
# agents-team class: enumerated but no batch ever produced a row for it.
stragglers = prod.rollout_stragglers(
["a", "agents-team"],
[{"slug": "a", "verified_on_target": True}],
)
assert stragglers == ["agents-team"]
def test_rollout_stragglers_missing_key_is_backward_compatible():
# Older CP without verified_on_target → treat as verified (no spurious fail).
stragglers = prod.rollout_stragglers(
["a", "b"],
[{"slug": "a", "healthz_ok": True}, {"slug": "b", "healthz_ok": True}],
)
assert stragglers == []
def test_rollout_stragglers_ignores_dry_run_rows():
stragglers = prod.rollout_stragglers(
["a"], [{"slug": "a", "ssm_status": "DryRun"}]
)
# dry-run row is skipped, so "a" has no verifying row → straggler.
assert stragglers == ["a"]
def test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag():
# Every per-tenant call returns ok=True, but agents-team is NOT
# verified_on_target. The rollout must still fail loudly — this is
# the exact "reported success, one tenant silently skipped" bug.
def fake_redeploy(_cp_url, _token, body):
rows = []
for slug in body["only_slugs"]:
rows.append({"slug": slug, "verified_on_target": slug != "agents-team"})
return 200, {"ok": True, "results": rows}
try:
prod.execute_scoped_rollout(
{
"cp_url": "https://api.moleculesai.app",
"body": {
"target_tag": "staging-new",
"batch_size": 5,
"dry_run": False,
"confirm": True,
},
},
token="secret",
list_slugs=lambda _u, _t, _b: ["reno-stars", "agents-team", "hongming"],
redeploy=fake_redeploy,
sleep=lambda _s: None,
)
except prod.RolloutFailed as exc:
assert "incomplete rollout" in str(exc)
assert exc.response["stragglers"] == ["agents-team"]
assert exc.response["ok"] is False
else:
raise AssertionError("expected an incomplete rollout to fail loudly")
def test_scoped_rollout_passes_when_all_tenants_verified_on_target():
def fake_redeploy(_cp_url, _token, body):
return 200, {
"ok": True,
"results": [{"slug": s, "verified_on_target": True} for s in body["only_slugs"]],
}
aggregate = prod.execute_scoped_rollout(
{
"cp_url": "https://api.moleculesai.app",
"body": {
"target_tag": "staging-new",
"batch_size": 5,
"dry_run": False,
"confirm": True,
},
},
token="secret",
list_slugs=lambda _u, _t, _b: ["reno-stars", "agents-team", "hongming"],
redeploy=fake_redeploy,
sleep=lambda _s: None,
)
assert aggregate["ok"] is True
assert "stragglers" not in aggregate
def test_scoped_rollout_dry_run_does_not_assert_coverage():
# A dry run proves nothing landed; coverage must NOT be asserted or
# every plan would fail.
def fake_redeploy(_cp_url, _token, body):
return 200, {
"ok": True,
"results": [{"slug": s, "ssm_status": "DryRun"} for s in body["only_slugs"]],
}
aggregate = prod.execute_scoped_rollout(
{
"cp_url": "https://api.moleculesai.app",
"body": {
"target_tag": "staging-new",
"batch_size": 5,
"dry_run": True,
"confirm": True,
},
},
token="secret",
list_slugs=lambda _u, _t, _b: ["a", "b"],
redeploy=fake_redeploy,
sleep=lambda _s: None,
)
assert aggregate["ok"] is True
+1 -1
View File
@@ -37,7 +37,7 @@ jobs:
# Phase 3 (RFC #219 §1): surface broken workflows without blocking
# the PR. Follow-up PR flips this off after surfaced defects are
# triaged.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -45,7 +45,7 @@ jobs:
# Phase 3 (RFC #219 §1): surface broken workflows without blocking
# the PR. Follow-up PR flips this off after surfaced defects are
# triaged.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 5
steps:
+1 -1
View File
@@ -101,7 +101,7 @@ jobs:
# AND-set: only the Mac arm64 runner advertises macos-self-hosted.
# See "RUNNER TARGETING" header note for why bare self-hosted is unsafe.
runs-on: [self-hosted, macos-self-hosted]
# ADVISORY: never blocks. See safety contract point 3. mc#1982
# ADVISORY: never blocks. See safety contract point 3. mc#774
# internal#418 — tracked: arm64 advisory pilot, non-gating by design.
continue-on-error: true
# event_name gate: functional (only meaningful on push/PR) AND keeps
+1 -1
View File
@@ -57,7 +57,7 @@ permissions:
# can produce duplicate comments before the title-search dedup wins.
concurrency:
group: ci-required-drift
cancel-in-progress: true
cancel-in-progress: false
jobs:
drift:
+9 -1
View File
@@ -161,7 +161,7 @@ jobs:
echo "::group::pendinguploads exit=$pu_exit (last 100 lines)"
tail -100 /tmp/test-pu.log
echo "::endgroup::"
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
- if: ${{ needs.changes.outputs.platform == 'true' }}
name: Run tests with coverage (blocking gate)
@@ -357,6 +357,14 @@ jobs:
name: Run E2E bash unit tests (no live infra)
run: |
bash tests/e2e/test_model_slug.sh
# molecule-core#1995 (#1994 follow-on): fail-direction proof for
# the A2A real-completion + byok-routing assertion helpers
# (lib/completion_assert.sh). Offline (no LLM, no network): it
# asserts an error-as-text payload FAILS the real-completion gate
# — the exact trap the historical shape-only `"kind":"text"`
# check missed. If a refactor weakens the gate to a shape check,
# this step goes red on every PR.
bash tests/e2e/test_completion_assert_unit.sh
- if: ${{ needs.changes.outputs.scripts == 'true' }}
name: Test ECR promote-tenant-image script (mock-driven, no live infra)
+10 -2
View File
@@ -92,7 +92,7 @@ permissions:
# stacking up.
concurrency:
group: continuous-synth-e2e
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -102,7 +102,7 @@ jobs:
name: Synthetic E2E against staging
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
# Bumped from 12 → 20 (2026-05-04). Tenant user-data install phase
# (apt-get update + install docker.io/jq/awscli/caddy + snap install
@@ -166,6 +166,10 @@ jobs:
# canary path. The script picks the right blob shape based on
# which key is non-empty.
E2E_OPENAI_API_KEY: ${{ secrets.MOLECULE_STAGING_OPENAI_API_KEY }}
# google-adk canary path — AI-Studio key (config model
# google_genai:gemini-2.5-pro). PROD disallows API keys (Vertex+ADC);
# the keyed path is CI-only. Dispatch with E2E_RUNTIME=google-adk.
E2E_GOOGLE_API_KEY: ${{ secrets.MOLECULE_STAGING_GOOGLE_API_KEY }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -217,6 +221,10 @@ jobs:
required_secret_name="MOLECULE_STAGING_OPENAI_API_KEY"
required_secret_value="${E2E_OPENAI_API_KEY:-}"
;;
google-adk)
required_secret_name="MOLECULE_STAGING_GOOGLE_API_KEY"
required_secret_value="${E2E_GOOGLE_API_KEY:-}"
;;
*)
echo "::warning::Unknown E2E_RUNTIME='${E2E_RUNTIME}' — skipping LLM-key check"
required_secret_name=""
+3 -3
View File
@@ -101,7 +101,7 @@ concurrency:
# See e2e-staging-canvas.yml's identical concurrency block for the full
# rationale and the 2026-04-28 incident reference.
group: e2e-api-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -123,7 +123,7 @@ jobs:
# integration). See internal#512 for the class defect.
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
api: ${{ steps.decide.outputs.api }}
@@ -160,7 +160,7 @@ jobs:
# detect-changes for the full rationale.
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 15
env:
+3 -3
View File
@@ -32,7 +32,7 @@ on:
concurrency:
group: e2e-chat-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -48,7 +48,7 @@ jobs:
# defect.
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
chat: ${{ steps.decide.outputs.chat }}
@@ -112,7 +112,7 @@ jobs:
# Must land on operator-host Linux (docker-host).
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 15
env:
+1 -1
View File
@@ -15,7 +15,7 @@ on:
concurrency:
group: e2e-legacy-advisory
cancel-in-progress: true
cancel-in-progress: false
permissions:
contents: read
+1 -1
View File
@@ -115,7 +115,7 @@ concurrency:
# would let a queued staging/main push behind a PR run get cancelled,
# leaving any gate that reads "completed run at SHA" stuck.
group: e2e-peer-visibility-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
+3 -3
View File
@@ -62,7 +62,7 @@ concurrency:
# wasted CI is acceptable given the alternative is losing staging-tip
# data that auto-promote-staging needs.
group: e2e-staging-canvas-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -71,7 +71,7 @@ jobs:
detect-changes:
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
canvas: ${{ steps.decide.outputs.canvas }}
@@ -140,7 +140,7 @@ jobs:
name: Canvas tabs E2E
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 40
+1 -1
View File
@@ -84,7 +84,7 @@ jobs:
name: E2E Staging External Runtime
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 25
+16 -5
View File
@@ -49,6 +49,7 @@ on:
- 'workspace-server/internal/middleware/**'
- 'workspace-server/internal/provisioner/**'
- 'tests/e2e/test_staging_full_saas.sh'
- 'tests/e2e/lib/completion_assert.sh'
- 'tests/e2e/lib/aws_leak_check.sh'
- 'tests/e2e/test_aws_leak_check.sh'
- '.gitea/workflows/e2e-staging-saas.yml'
@@ -61,6 +62,7 @@ on:
- 'workspace-server/internal/middleware/**'
- 'workspace-server/internal/provisioner/**'
- 'tests/e2e/test_staging_full_saas.sh'
- 'tests/e2e/lib/completion_assert.sh'
- 'tests/e2e/lib/aws_leak_check.sh'
- 'tests/e2e/test_aws_leak_check.sh'
- '.gitea/workflows/e2e-staging-saas.yml'
@@ -92,20 +94,20 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.11"
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
- name: YAML validation (best-effort)
run: |
echo "e2e-staging-saas.yml — PR validation: workflow YAML is valid."
echo "E2E step runs only when provisioning-critical files change."
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
# Actual E2E: runs on trunk pushes and PRs that touch provisioning-critical
@@ -116,7 +118,7 @@ jobs:
name: E2E Staging SaaS
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 45
permissions:
@@ -155,13 +157,18 @@ jobs:
# E2E_RUNTIME=hermes or =codex via workflow_dispatch can still
# exercise the OpenAI path.
E2E_OPENAI_API_KEY: ${{ secrets.MOLECULE_STAGING_OPENAI_API_KEY }}
# google-adk (operator-dispatched only) auths Gemini with an
# AI-Studio key. Org policy disallows API keys in PROD (Vertex+ADC
# there); CI uses the keyed AI-Studio path with config model
# google_genai:gemini-2.5-pro. Vertex remains the supported prod path.
E2E_GOOGLE_API_KEY: ${{ secrets.MOLECULE_STAGING_GOOGLE_API_KEY }}
E2E_RUNTIME: ${{ github.event.inputs.runtime || 'claude-code' }}
# Pin the model when running on the default claude-code path —
# the per-runtime default ("sonnet") routes to direct Anthropic
# and defeats the cost saving. Operators can override via the
# workflow_dispatch flow (no input wired here yet — runtime
# override is enough for ad-hoc).
E2E_MODEL_SLUG: ${{ github.event.inputs.runtime == 'hermes' && 'openai/gpt-4o' || github.event.inputs.runtime == 'codex' && 'openai/gpt-4o' || 'MiniMax-M2' }}
E2E_MODEL_SLUG: ${{ github.event.inputs.runtime == 'hermes' && 'openai/gpt-4o' || github.event.inputs.runtime == 'codex' && 'openai/gpt-4o' || github.event.inputs.runtime == 'google-adk' && 'google_genai:gemini-2.5-pro' || 'MiniMax-M2' }}
E2E_RUN_ID: "${{ github.run_id }}-${{ github.run_attempt }}"
E2E_KEEP_ORG: ${{ github.event.inputs.keep_org && '1' || '0' }}
@@ -210,6 +217,10 @@ jobs:
required_secret_name="MOLECULE_STAGING_OPENAI_API_KEY"
required_secret_value="${E2E_OPENAI_API_KEY:-}"
;;
google-adk)
required_secret_name="MOLECULE_STAGING_GOOGLE_API_KEY"
required_secret_value="${E2E_GOOGLE_API_KEY:-}"
;;
*)
echo "::warning::Unknown E2E_RUNTIME='${E2E_RUNTIME}' — skipping LLM-key check"
required_secret_name=""
+2 -2
View File
@@ -26,7 +26,7 @@ env:
concurrency:
group: e2e-staging-sanity
cancel-in-progress: true
cancel-in-progress: false
permissions:
issues: write
@@ -37,7 +37,7 @@ jobs:
name: Intentional-failure teardown sanity
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 20
+1 -1
View File
@@ -66,7 +66,7 @@ jobs:
# bp-exempt: PR advisory bot; merge blocking is enforced by CI status and branch protection.
gate-check:
runs-on: ubuntu-latest
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true # Never block on our own detector failing
steps:
- name: Check out BASE ref (never PR-head under pull_request_target)
@@ -69,7 +69,7 @@ on:
branches: [main, staging]
concurrency:
group: handlers-pg-integ-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -87,8 +87,8 @@ jobs:
# both jobs on the same label avoids workspace-volume cross-host
# surprises and keeps the routing rule discoverable in one place.
runs-on: docker-host
# mc#1982 Phase 3 (RFC §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774 Phase 3 (RFC §1): surface broken workflows without blocking.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
handlers: ${{ steps.filter.outputs.handlers }}
@@ -118,8 +118,8 @@ jobs:
# mc#1529 §1: must run on operator-host (where `molecule-core-net`
# exists). See detect-changes for the full routing rationale.
runs-on: docker-host
# mc#1982 Phase 3 (RFC §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774 Phase 3 (RFC §1): surface broken workflows without blocking.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
env:
# Unique name per run so concurrent jobs don't collide on the
+3 -3
View File
@@ -54,7 +54,7 @@ concurrency:
# cancellation deadlock — see e2e-api.yml's concurrency block for
# the 2026-04-28 incident that codified this pattern.
group: harness-replays-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
cancel-in-progress: false
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
@@ -70,7 +70,7 @@ jobs:
# of mc#1543; see internal#512 for class defect.
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
run: ${{ steps.decide.outputs.run }}
@@ -172,7 +172,7 @@ jobs:
# beta containers. Must run on operator-host Linux (docker-host).
runs-on: docker-host
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 30
steps:
@@ -94,7 +94,7 @@ jobs:
# Phase 3 (RFC #219 §1): surface drift without blocking. After 7
# clean scheduled runs on main, flip to false so a scheduled
# failure is a hard CI signal.
continue-on-error: true # mc#1982 Phase 3 — flip to false after 7 clean main runs
continue-on-error: true # mc#774 Phase 3 — flip to false after 7 clean main runs
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@@ -1,6 +1,6 @@
name: lint-continue-on-error-tracking
# Tier 2e hard-gate lint (per mc#1982) — every
# Tier 2e hard-gate lint (per mc#774) — every
# `continue-on-error: true` in `.gitea/workflows/*.yml` must carry a
# `# mc#NNNN` or `# internal#NNNN` tracker comment within 2 lines,
# the referenced issue must be OPEN, and ≤14 days old.
@@ -8,7 +8,7 @@ name: lint-continue-on-error-tracking
# Why this exists
# ---------------
# `continue-on-error: true` on `platform-build` had been hiding
# mc#1982-class regressions for ~3 weeks before #656 surfaced them on
# mc#774-class regressions for ~3 weeks before #656 surfaced them on
# 2026-05-12. A 14-day cap on tracker age forces a review cycle and
# surfaces mask-drift within at most 14 days of the original defect.
# Each `continue-on-error: true` gets a paper trail — close or renew.
@@ -97,9 +97,9 @@ jobs:
# Phase 3 (RFC #219 §1): surface masked defects without blocking
# PRs. Pre-existing continue-on-error: true directives on main
# all violate this lint at first — intentional. Flip to false
# follow-up after main is clean for 3 days. mc#1982.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true # mc#1982 Phase 3 mask — 14d forced-renewal cadence
# follow-up after main is clean for 3 days. mc#774.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true # mc#774 Phase 3 mask — 14d forced-renewal cadence
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@@ -51,7 +51,7 @@ jobs:
# Phase 3 (RFC #219 §1): surface broken workflows without blocking
# the PR. Follow-up PR flips this off after surfaced defects are
# triaged.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+2 -2
View File
@@ -92,8 +92,8 @@ jobs:
# Phase 3 (RFC #219 §1): surface broken shapes without blocking
# PRs. Follow-up PR flips this to `false` once recent runs on main
# are confirmed clean (eat-our-own-dogfood discipline mirrors
# PR#673's same-shape comment). Tracking: mc#1982.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# PR#673's same-shape comment). Tracking: mc#774.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- name: Check out PR head with full history (need base SHA blobs)
@@ -4,7 +4,7 @@ name: Lint pre-flip continue-on-error
# on any job in `.gitea/workflows/*.yml` WITHOUT proof that the affected
# job's recent runs on the target branch (PR base) are actually green.
#
# Empirical class: PR #656 / mc#1982. PR #656 (RFC internal#219 Phase 4)
# Empirical class: PR #656 / mc#774. PR #656 (RFC internal#219 Phase 4)
# flipped 5 platform-build-class jobs `continue-on-error: true → false`
# on the basis of a "verified green on main via combined-status check".
# But that "green" was the LIE the prior `continue-on-error: true`
@@ -99,8 +99,8 @@ jobs:
timeout-minutes: 8
# Phase 3 (RFC internal#219 §1): surface broken flips without blocking
# the PR yet. Follow-up flips this to `false` once the workflow itself
# has clean recent runs on main. mc#1982 interim — remove when CoE→false.
continue-on-error: true # mc#1982
# has clean recent runs on main. mc#774 interim — remove when CoE→false.
continue-on-error: true # mc#774
steps:
- name: Check out PR head (full history for base-SHA access)
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -83,8 +83,8 @@ jobs:
timeout-minutes: 5
# Phase 3 (RFC #219 §1): surface the pattern without blocking PRs
# while the directive convention beds in. Follow-up flip to false
# after 7 clean days on main. mc#1982.
continue-on-error: true # mc#1982 Phase 3 — flip to false after 7 clean main runs
# after 7 clean days on main. mc#774.
continue-on-error: true # mc#774 Phase 3 — flip to false after 7 clean main runs
steps:
- name: Check out PR head with full history (need base SHA blobs)
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+1 -1
View File
@@ -55,7 +55,7 @@ jobs:
# Phase 3 (RFC #219 §1): surface broken shapes without blocking PRs.
# Follow-up PR flips this off after the 4 existing-on-main rule-2
# (workflow_run) violations are migrated to a supported trigger.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+1 -1
View File
@@ -67,7 +67,7 @@ jobs:
# in this rollout (internal#462) so the precondition holds.
runs-on: publish
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- name: Checkout
@@ -234,7 +234,7 @@ jobs:
name: Production auto-deploy
needs: build-and-push
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
# Side-effect deploy only; image publish success is the durable artifact. mc#1982
# Side-effect deploy only; image publish success is the durable artifact. mc#774
continue-on-error: true
# Publish/release lane (internal#462) — production deploy of a merged
# fix; reserved capacity, never queued behind PR-CI.
@@ -327,13 +327,27 @@ jobs:
echo ""
echo "### Per-tenant result"
echo ""
echo "| Slug | Phase | SSM Status | Exit | Healthz | Error present |"
echo "|------|-------|------------|------|---------|---------------|"
jq -r '.results[]? | "| \(.slug) | \(.phase) | \(.ssm_status // "-") | \(.ssm_exit_code) | \(.healthz_ok) | \((.error // "") != "") |"' "$HTTP_RESPONSE" || true
echo "| Slug | Phase | SSM Status | Exit | Healthz | On target | Error present |"
echo "|------|-------|------------|------|---------|-----------|---------------|"
jq -r '.results[]? | "| \(.slug) | \(.phase) | \(.ssm_status // "-") | \(.ssm_exit_code) | \(.healthz_ok) | \(.verified_on_target) | \((.error // "") != "") |"' "$HTTP_RESPONSE" || true
# internal#724: stragglers are tenants enumerated but not proven
# on the target build. Surface them loudly — a non-empty list
# means the rollout did NOT fully land.
STRAGGLERS="$(jq -r '(.stragglers // []) | join(", ")' "$HTTP_RESPONSE")"
if [ -n "$STRAGGLERS" ]; then
echo ""
echo "### ⚠ Stragglers (NOT on target tag \`$TARGET_TAG\`)"
echo ""
echo "\`$STRAGGLERS\`"
fi
} >> "$GITHUB_STEP_SUMMARY"
OK="$(jq -r '.ok' "$HTTP_RESPONSE")"
if [ "$OK" != "true" ]; then
STRAGGLERS="$(jq -r '(.stragglers // []) | join(", ")' "$HTTP_RESPONSE")"
if [ -n "$STRAGGLERS" ]; then
echo "::error::incomplete rollout — tenants not on target tag $TARGET_TAG: $STRAGGLERS"
fi
echo "::error::redeploy-fleet reported ok=false; production rollout halted."
exit 1
fi
+2 -2
View File
@@ -40,7 +40,7 @@ env:
concurrency:
group: railway-pin-audit
cancel-in-progress: true
cancel-in-progress: false
permissions:
issues: write
@@ -51,7 +51,7 @@ jobs:
name: Audit Railway env vars for drift-prone pins
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 10
@@ -73,7 +73,7 @@ jobs:
# it never queues behind PR-CI. `publish` -> molecule-runner-publish-*.
runs-on: publish
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 25
env:
@@ -80,7 +80,7 @@ jobs:
# `publish` -> molecule-runner-publish-* sub-pool.
runs-on: publish
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 25
steps:
+1 -1
View File
@@ -54,7 +54,7 @@ jobs:
# runners with internet access to package mirrors). Falls back to GitHub
# binary download. GitHub releases may be blocked on some runner networks
# (infra#241 follow-up).
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
run: |
if apt-get update -qq && apt-get install -y -qq jq; then
+1 -1
View File
@@ -57,7 +57,7 @@ jobs:
name: Detect SECRET_PATTERNS drift
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
timeout-minutes: 5
steps:
+3 -3
View File
@@ -36,7 +36,7 @@
# window closed. continue-on-error: true has been removed from the
# tier-check job; AND-composition is now fully enforced. If you need
# to temporarily re-introduce a mask, file a tracker and follow the
# mc#1982 protocol (Tier 2e lint requires a current tracker within
# mc#774 protocol (Tier 2e lint requires a current tracker within
# 2 lines of any continue-on-error: true).
name: sop-tier-check
@@ -92,7 +92,7 @@ jobs:
# runners). The sop-tier-check script has its own fallback as a
# third line of defense. continue-on-error: true ensures this step
# failing does not block the job.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
run: |
# apt-get is the primary method — Ubuntu package mirrors are reliably
@@ -113,7 +113,7 @@ jobs:
# continue-on-error: true at step level — job-level is ignored by Gitea
# Actions (quirk #10, internal runbooks). Belt-and-suspenders with
# SOP_FAIL_OPEN=1 + || true below.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
env:
GITEA_TOKEN: ${{ secrets.SOP_TIER_CHECK_TOKEN || secrets.GITHUB_TOKEN }}
+1 -1
View File
@@ -38,7 +38,7 @@ on:
# full run, but two smoke runs SHOULD queue against each other.
concurrency:
group: staging-smoke
cancel-in-progress: true
cancel-in-progress: false
permissions:
# Needed to open / close the alerting issue.
+2 -2
View File
@@ -90,7 +90,7 @@ jobs:
staging-smoke:
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
outputs:
sha: ${{ steps.compute.outputs.sha }}
@@ -212,7 +212,7 @@ jobs:
if: ${{ needs.staging-smoke.result == 'success' && needs.staging-smoke.outputs.smoke_ran == 'true' }}
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
env:
SHA: ${{ needs.staging-smoke.outputs.sha }}
+1 -1
View File
@@ -50,7 +50,7 @@ on:
# Don't let two sweeps race the same AWS account.
concurrency:
group: sweep-aws-secrets
cancel-in-progress: true
cancel-in-progress: false
permissions:
contents: read
+2 -2
View File
@@ -58,7 +58,7 @@ on:
# scheduled run would otherwise issue duplicate DELETE calls.
concurrency:
group: sweep-cf-orphans
cancel-in-progress: true
cancel-in-progress: false
permissions:
contents: read
@@ -71,7 +71,7 @@ jobs:
name: Sweep CF orphans
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
# 3 min surfaces hangs (CF API stall, AWS describe-instances stuck)
# within one cron interval instead of burning a full tick. Realistic
+2 -2
View File
@@ -42,7 +42,7 @@ on:
# Don't let two sweeps race the same account.
concurrency:
group: sweep-cf-tunnels
cancel-in-progress: true
cancel-in-progress: false
permissions:
contents: read
@@ -55,7 +55,7 @@ jobs:
name: Sweep CF tunnels
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
# 30 min cap. Was 5 min on the theory that the only thing that
# could take >5min is a CF-API hang — but on 2026-05-02 a backlog
+1 -1
View File
@@ -51,7 +51,7 @@ on:
# on a manual trigger; queue rather than parallel-delete.
concurrency:
group: sweep-stale-e2e-orgs
cancel-in-progress: true
cancel-in-progress: false
permissions:
contents: read
+1 -1
View File
@@ -49,7 +49,7 @@ jobs:
name: Ops scripts (unittest)
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+1 -1
View File
@@ -31,7 +31,7 @@ jobs:
name: Weekly Platform-Go Surface
runs-on: ubuntu-latest
# continue-on-error: surface only, never block
# mc#1982: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
# mc#774: pre-existing continue-on-error mask; root-fix and remove, do not renew silently.
continue-on-error: true
defaults:
run:
@@ -38,10 +38,11 @@ const DEFAULT_RUNTIME = "claude-code";
const RUNTIME_OPTIONS = [
{ value: "claude-code", label: "Claude Code" },
{ value: "codex", label: "OpenAI Codex CLI" },
{ value: "google-adk", label: "Google ADK" },
{ value: "hermes", label: "Hermes" },
{ value: "openclaw", label: "OpenClaw" },
];
const BASE_RUNTIME_TEMPLATE_IDS = new Set(["claude-code-default", "codex", "hermes", "openclaw"]);
const BASE_RUNTIME_TEMPLATE_IDS = new Set(["claude-code-default", "codex", "google-adk", "hermes", "openclaw"]);
const DEFAULT_HEADLESS_INSTANCE_TYPE = "t3.medium";
const DEFAULT_HEADLESS_ROOT_GB = 30;
const DEFAULT_DISPLAY_INSTANCE_TYPE = "t3.xlarge";
+101 -1
View File
@@ -49,6 +49,33 @@ export interface ProviderEntry {
wildcard: boolean;
/** Optional tooltip text (rendered as native title=). */
tooltip?: string;
/** Billing mode the DERIVED provider implies, when this entry came from the
* registry-backed payload (internal#718 P3): "platform_managed" | "byok".
* Undefined for entries built by the legacy inferVendor heuristic. */
billingMode?: "platform_managed" | "byok";
}
/** RegistryProvider mirrors one entry of GET /templates `registry_providers`
* (workspace-server registryProviderView): the registry's native provider for
* a runtime, with its display label, auth-env NAMES, and billing mode. This is
* the SSOT the dropdown labels come from — the canvas drops VENDOR_LABELS for
* registry-backed runtimes (internal#718 P3, retire-list #4). */
export interface RegistryProvider {
name: string;
display_name?: string;
auth_env?: string[];
billing_mode?: "platform_managed" | "byok";
deprecated?: boolean;
}
/** RegistryModel mirrors one entry of GET /templates `registry_models`: a
* native model id annotated with its DERIVED provider (registry name) and the
* billing_mode that provider implies. */
export interface RegistryModel {
id: string;
name?: string;
provider?: string;
billing_mode?: "platform_managed" | "byok";
}
export interface SelectorValue {
@@ -68,6 +95,13 @@ interface Props {
models: SelectorModel[];
value: SelectorValue;
onChange: (next: SelectorValue) => void;
/** Optional pre-built provider catalog. When provided, the selector uses it
* verbatim instead of re-inferring one from `models` via
* buildProviderCatalog — the registry-backed path (internal#718 P3), where
* the parent builds the catalog from the registry-served providers/models
* so dropdown labels + billing come from the provider-registry SSOT rather
* than the inferVendor heuristic. Omitted = legacy heuristic over `models`. */
catalog?: ProviderEntry[];
/** Display variant. "grid" = label+control side-by-side (used in ConfigTab
* Runtime section). "stack" = vertical (used in MissingKeysModal). */
variant?: "grid" | "stack";
@@ -251,6 +285,66 @@ export function buildProviderCatalog(models: SelectorModel[]): ProviderEntry[] {
return Array.from(buckets.values());
}
/** Build the provider catalog from a REGISTRY-BACKED GET /templates payload
* (registry_providers + registry_models) — internal#718 P3, retire-list #4.
*
* Unlike buildProviderCatalog (which RE-INFERS vendor from model-id prefixes
* + env via inferVendor/VENDOR_LABELS/BARE_VENDOR_PATTERNS), this trusts the
* registry: each model carries its DERIVED `provider` (a registry provider
* name) and the dropdown label/billing/auth come from the matching
* `registry_providers` entry. The canvas can render no provider/model the
* registry did not serve ("only registered selectable"), and the billing-mode
* shown reflects the derived provider rather than a hardcoded rule.
*
* A provider with no served model is omitted (no empty buckets). Models whose
* `provider` doesn't match a registry_providers entry still get a bucket
* keyed by the raw provider name (defensive — should not happen for a
* well-formed registry payload), so a model is never silently dropped. */
export function buildProviderCatalogFromRegistry(
registryProviders: RegistryProvider[],
registryModels: RegistryModel[],
): ProviderEntry[] {
const byName = new Map<string, RegistryProvider>();
for (const p of registryProviders) byName.set(p.name, p);
// Bucket models by their derived provider name, preserving registry order.
const buckets = new Map<string, ProviderEntry>();
for (const m of registryModels) {
const vendor = (m.provider ?? "").trim();
if (!vendor) continue; // un-annotated registry model — skip from the
// provider cascade (selectable elsewhere via free-text); it has no
// derived provider to bucket under.
const meta = byName.get(vendor);
const wildcard = m.id.includes("*");
let entry = buckets.get(vendor);
if (!entry) {
entry = {
id: `registry|${vendor}`,
vendor,
label: meta?.display_name || vendor,
envVars: meta?.auth_env ?? [],
models: [],
wildcard,
billingMode: meta?.billing_mode ?? m.billing_mode,
tooltip: VENDOR_TOOLTIPS[vendor],
};
buckets.set(vendor, entry);
}
entry.models.push({ id: m.id, name: m.name, provider: vendor });
entry.wildcard = entry.wildcard || wildcard;
}
// Decorate label with model-count when ≥2 concrete models share the bucket,
// matching buildProviderCatalog's UX.
for (const e of buckets.values()) {
if (!e.wildcard && e.models.length > 1) {
e.label = `${e.label} (${e.models.length} models)`;
}
}
return Array.from(buckets.values());
}
/** Find the provider entry that contains a given model id. Used by
* callers to back-derive the provider when only the model is known
* (e.g. ConfigTab loading from saved state). */
@@ -283,6 +377,7 @@ export function ProviderModelSelector({
models,
value,
onChange,
catalog: catalogProp,
variant = "stack",
allowCustomModelEscape = false,
disabled = false,
@@ -293,7 +388,12 @@ export function ProviderModelSelector({
const providerSelectId = `${baseId}-provider`;
const modelSelectId = `${baseId}-model`;
const catalog = useMemo(() => buildProviderCatalog(models), [models]);
// Registry-backed path (internal#718 P3): use the parent-supplied catalog
// verbatim; otherwise re-infer one from `models` via the legacy heuristic.
const catalog = useMemo(
() => catalogProp ?? buildProviderCatalog(models),
[catalogProp, models],
);
const selected = useMemo(
() => catalog.find((p) => p.id === value.providerId) ?? null,
[catalog, value.providerId],
@@ -1,411 +1,82 @@
// @vitest-environment jsdom
/**
* Tests for BudgetSection (issue #541).
* Focused tests for BudgetSection's PER-PERIOD progress-bar math + aria (#49).
*
* Covers:
* - Loading state
* - Stats row: used / limit, "Unlimited" when null
* - Progress bar: correct percentage, capped at 100%, absent when no limit
* - Budget remaining text
* - Input pre-fill (existing limit / blank when null)
* - Save: PATCH with number, PATCH with null (blank input)
* - 402 on GET → exceeded banner, no fetch-error text
* - 402 on PATCH → exceeded banner
* - Non-402 fetch error → error text
* - Non-402 save error → save error alert
* - Section header and subheading
* - Fetch error does not show stats
* Behavioral coverage (loading, save, 402 banners, USD formatting, legacy
* back-compat) lives in tabs/__tests__/BudgetSection.test.tsx — this file
* deliberately covers only the per-period progress percentage + aria-valuenow
* + the over-budget colouring, which that suite doesn't assert in detail. Kept
* separate to avoid duplicating the behavioral suite (one component, no
* parallel/identical suites).
*/
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import {
render,
screen,
fireEvent,
waitFor,
cleanup,
act,
} from "@testing-library/react";
// ── Mock api ──────────────────────────────────────────────────────────────────
import { render, screen, waitFor, cleanup } from "@testing-library/react";
vi.mock("@/lib/api", () => ({
api: {
get: vi.fn(),
patch: vi.fn(),
},
api: { get: vi.fn(), patch: vi.fn() },
}));
import { api } from "@/lib/api";
import { BudgetSection } from "../tabs/BudgetSection";
const mockGet = vi.mocked(api.get);
const mockPatch = vi.mocked(api.patch);
// ── Helpers ───────────────────────────────────────────────────────────────────
type P = { limit: number | null; spend: number; remaining: number | null };
function budgetResponse(overrides: Partial<{
budget_limit: number | null;
budget_used: number;
budget_remaining: number | null;
}> = {}) {
// Build a periods response where the named period has the given limit/spend.
function withMonthly(limit: number | null, spend: number) {
const blank: P = { limit: null, spend: 0, remaining: null };
const monthly: P = { limit, spend, remaining: limit == null ? null : limit - spend };
return {
budget_limit: 1000,
budget_used: 250,
budget_remaining: 750,
...overrides,
periods: { hourly: blank, daily: blank, weekly: blank, monthly },
budget_limit: limit,
monthly_spend: spend,
budget_remaining: monthly.remaining,
};
}
function make402Error(): Error {
return new Error("API GET /workspaces/ws-1/budget: 402 Payment Required");
}
beforeEach(() => vi.clearAllMocks());
afterEach(() => cleanup());
function make402PatchError(): Error {
return new Error("API PATCH /workspaces/ws-1/budget: 402 Payment Required");
}
function makeGenericError(msg = "network timeout"): Error {
return new Error(`API GET /workspaces/ws-1/budget: 500 ${msg}`);
}
beforeEach(() => {
vi.clearAllMocks();
});
afterEach(() => {
cleanup();
});
// ── Rendering helpers ─────────────────────────────────────────────────────────
async function renderLoaded(budgetData = budgetResponse()) {
async function renderLoaded(data: unknown) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockGet.mockResolvedValueOnce(budgetData as any);
mockGet.mockResolvedValueOnce(data as any);
render(<BudgetSection workspaceId="ws-1" />);
// Wait for loading to finish
await waitFor(() => expect(screen.queryByTestId("budget-loading")).toBeNull());
}
// ── Loading state ─────────────────────────────────────────────────────────────
describe("BudgetSection — loading state", () => {
it("shows loading indicator while fetch is in flight", () => {
// Never resolve
mockGet.mockReturnValue(new Promise(() => {}));
render(<BudgetSection workspaceId="ws-1" />);
expect(screen.getByTestId("budget-loading")).toBeTruthy();
expect(screen.getByText("Loading…")).toBeTruthy();
describe("BudgetSection — per-period progress bar", () => {
it("renders the bar for a limited period and omits it for an unlimited one", async () => {
await renderLoaded(withMonthly(1000, 250));
expect(screen.getByTestId("budget-monthly-fill")).toBeTruthy();
expect(screen.queryByTestId("budget-hourly-fill")).toBeNull(); // hourly unlimited
});
it("hides loading indicator after fetch resolves", async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockGet.mockResolvedValueOnce(budgetResponse() as any);
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() => expect(screen.queryByTestId("budget-loading")).toBeNull());
});
});
// ── Section header ────────────────────────────────────────────────────────────
describe("BudgetSection — header and subheading", () => {
it("renders 'Budget' as the section heading", async () => {
await renderLoaded();
expect(screen.getByText("Budget")).toBeTruthy();
});
it("renders the subheading 'Limit total message credits for this workspace'", async () => {
await renderLoaded();
expect(
screen.getByText("Limit total message credits for this workspace")
).toBeTruthy();
});
it("renders 'Budget limit (credits)' label for the input", async () => {
await renderLoaded();
expect(screen.getByText("Budget limit (credits)")).toBeTruthy();
});
});
// ── Stats row ─────────────────────────────────────────────────────────────────
describe("BudgetSection — stats row", () => {
it("shows budget_used in the stats row", async () => {
await renderLoaded(budgetResponse({ budget_used: 350, budget_limit: 1000 }));
expect(screen.getByTestId("budget-used-value").textContent).toBe("350");
});
it("shows budget_limit in the stats row", async () => {
await renderLoaded(budgetResponse({ budget_used: 100, budget_limit: 500 }));
expect(screen.getByTestId("budget-limit-value").textContent).toBe("500");
});
it("shows 'Unlimited' when budget_limit is null", async () => {
await renderLoaded(budgetResponse({ budget_limit: null, budget_remaining: null }));
expect(screen.getByTestId("budget-limit-value").textContent).toBe("Unlimited");
});
it("shows budget_remaining when present", async () => {
await renderLoaded(budgetResponse({ budget_remaining: 750 }));
expect(screen.getByTestId("budget-remaining").textContent).toContain("750");
expect(screen.getByTestId("budget-remaining").textContent).toContain("credits remaining");
});
it("hides budget_remaining row when null", async () => {
await renderLoaded(budgetResponse({ budget_remaining: null }));
expect(screen.queryByTestId("budget-remaining")).toBeNull();
});
it("does not crash when budget_used is missing from the response", async () => {
// Backend for a provisioning-stuck workspace may return a partial
// shape. Regression: previously this threw
// "Cannot read properties of undefined (reading 'toLocaleString')"
// and crashed the whole Details tab.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await renderLoaded({ budget_limit: 1000, budget_remaining: null } as any);
expect(screen.getByTestId("budget-used-value").textContent).toBe("0");
});
});
// ── Progress bar ──────────────────────────────────────────────────────────────
describe("BudgetSection — progress bar", () => {
it("renders the progress bar when budget_limit is set", async () => {
await renderLoaded(budgetResponse({ budget_used: 250, budget_limit: 1000 }));
expect(screen.getByRole("progressbar")).toBeTruthy();
});
it("does NOT render progress bar when budget_limit is null", async () => {
await renderLoaded(budgetResponse({ budget_limit: null, budget_remaining: null }));
expect(screen.queryByRole("progressbar")).toBeNull();
});
it("fills to the correct percentage (25%)", async () => {
await renderLoaded(budgetResponse({ budget_used: 250, budget_limit: 1000 }));
const fill = screen.getByTestId("budget-progress-fill") as HTMLDivElement;
expect(fill.style.width).toBe("25%");
});
it("fills to the correct percentage (50%)", async () => {
await renderLoaded(budgetResponse({ budget_used: 500, budget_limit: 1000 }));
const fill = screen.getByTestId("budget-progress-fill") as HTMLDivElement;
expect(fill.style.width).toBe("50%");
});
it("caps fill at 100% when budget_used exceeds budget_limit", async () => {
await renderLoaded(budgetResponse({ budget_used: 1500, budget_limit: 1000 }));
const fill = screen.getByTestId("budget-progress-fill") as HTMLDivElement;
expect(fill.style.width).toBe("100%");
});
it("progress bar has aria-valuenow equal to the calculated percentage", async () => {
await renderLoaded(budgetResponse({ budget_used: 300, budget_limit: 1000 }));
const bar = screen.getByRole("progressbar");
expect(bar.getAttribute("aria-valuenow")).toBe("30");
});
it("shows 0% progress bar when budget_used is absent from the response", async () => {
// Regression: budget_used is optional (provisioning-stuck workspaces return
// partial shapes). Without the `?? 0` guard the progressPct calculation
// throws a TypeScript strict-null error and the build fails.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await renderLoaded({ budget_limit: 1000, budget_remaining: null } as any);
const bar = screen.getByRole("progressbar");
expect(bar.getAttribute("aria-valuenow")).toBe("0");
const fill = screen.getByTestId("budget-progress-fill") as HTMLDivElement;
expect(fill.style.width).toBe("0%");
});
});
// ── Input pre-fill ────────────────────────────────────────────────────────────
describe("BudgetSection — input pre-fill", () => {
it("pre-fills input with existing budget_limit", async () => {
await renderLoaded(budgetResponse({ budget_limit: 500 }));
const input = screen.getByTestId("budget-limit-input") as HTMLInputElement;
expect(input.value).toBe("500");
});
it("leaves input empty when budget_limit is null", async () => {
await renderLoaded(budgetResponse({ budget_limit: null, budget_remaining: null }));
const input = screen.getByTestId("budget-limit-input") as HTMLInputElement;
expect(input.value).toBe("");
});
});
// ── Save — PATCH calls ────────────────────────────────────────────────────────
describe("BudgetSection — save", () => {
it("calls PATCH /workspaces/:id/budget with budget_limit as integer", async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockPatch.mockResolvedValueOnce(budgetResponse({ budget_limit: 800 }) as any);
await renderLoaded(budgetResponse({ budget_limit: 1000 }));
fireEvent.change(screen.getByTestId("budget-limit-input"), {
target: { value: "800" },
});
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() => expect(mockPatch).toHaveBeenCalled());
expect(mockPatch.mock.calls[0][0]).toBe("/workspaces/ws-1/budget");
const body = mockPatch.mock.calls[0][1] as Record<string, unknown>;
expect(body.budget_limit).toBe(800);
});
it("sends budget_limit: 0 (not null) when input is '0' — zero-credit budget", async () => {
// Regression for QA bug report: `parseInt("0") || null` would yield null.
// The correct form `raw !== "" ? parseInt(raw, 10) : null` must return 0.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockPatch.mockResolvedValueOnce(budgetResponse({ budget_limit: 0, budget_used: 0, budget_remaining: 0 }) as any);
await renderLoaded(budgetResponse({ budget_limit: 1000 }));
fireEvent.change(screen.getByTestId("budget-limit-input"), {
target: { value: "0" },
});
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() => expect(mockPatch).toHaveBeenCalled());
const body = mockPatch.mock.calls[0][1] as Record<string, unknown>;
expect(body.budget_limit).toBe(0);
expect(body.budget_limit).not.toBeNull();
});
it("sends budget_limit: null when input is blank", async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockPatch.mockResolvedValueOnce(budgetResponse({ budget_limit: null, budget_remaining: null }) as any);
await renderLoaded(budgetResponse({ budget_limit: 1000 }));
fireEvent.change(screen.getByTestId("budget-limit-input"), {
target: { value: "" },
});
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() => expect(mockPatch).toHaveBeenCalled());
const body = mockPatch.mock.calls[0][1] as Record<string, unknown>;
expect(body.budget_limit).toBeNull();
});
it("updates displayed stats after successful save", async () => {
const updated = budgetResponse({ budget_limit: 2000, budget_used: 500, budget_remaining: 1500 });
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockPatch.mockResolvedValueOnce(updated as any);
await renderLoaded(budgetResponse({ budget_limit: 1000, budget_used: 250 }));
fireEvent.change(screen.getByTestId("budget-limit-input"), {
target: { value: "2000" },
});
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() =>
expect(screen.getByTestId("budget-limit-value").textContent).toBe("2,000")
);
});
it("shows save error message on non-402 PATCH failure", async () => {
mockPatch.mockRejectedValueOnce(
new Error("API PATCH /workspaces/ws-1/budget: 500 server error")
);
await renderLoaded();
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() =>
expect(screen.getByTestId("budget-save-error")).toBeTruthy()
);
expect(screen.getByTestId("budget-save-error").textContent).toContain("500");
});
});
// ── 402 handling ──────────────────────────────────────────────────────────────
describe("BudgetSection — 402 handling", () => {
it("shows exceeded banner when GET returns 402", async () => {
mockGet.mockRejectedValueOnce(make402Error());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() =>
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy()
);
expect(screen.getByText("Budget exceeded — messages blocked")).toBeTruthy();
});
it("does NOT show fetch error text when GET returns 402 (only banner)", async () => {
mockGet.mockRejectedValueOnce(make402Error());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() =>
expect(screen.queryByTestId("budget-loading")).toBeNull()
);
expect(screen.queryByTestId("budget-fetch-error")).toBeNull();
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy();
});
it("shows exceeded banner when PATCH returns 402", async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockGet.mockResolvedValueOnce(budgetResponse() as any);
mockPatch.mockRejectedValueOnce(make402PatchError());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() => expect(screen.queryByTestId("budget-loading")).toBeNull());
fireEvent.click(screen.getByTestId("budget-save-btn"));
await waitFor(() =>
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy()
);
// Should NOT also show the save-error alert
expect(screen.queryByTestId("budget-save-error")).toBeNull();
});
it("clears exceeded banner after a successful save", async () => {
mockGet.mockRejectedValueOnce(make402Error());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() =>
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy()
);
// Now a successful PATCH (limit was raised)
const updated = budgetResponse({ budget_limit: 5000, budget_used: 250, budget_remaining: 4750 });
// eslint-disable-next-line @typescript-eslint/no-explicit-any
mockPatch.mockResolvedValueOnce(updated as any);
await act(async () => {
fireEvent.change(screen.getByTestId("budget-limit-input"), {
target: { value: "5000" },
});
fireEvent.click(screen.getByTestId("budget-save-btn"));
});
await waitFor(() =>
expect(screen.queryByTestId("budget-exceeded-banner")).toBeNull()
);
});
});
// ── Non-402 fetch error ───────────────────────────────────────────────────────
describe("BudgetSection — non-402 fetch errors", () => {
it("shows fetch error text on non-402 GET failure", async () => {
mockGet.mockRejectedValueOnce(makeGenericError("internal server error"));
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() =>
expect(screen.getByTestId("budget-fetch-error")).toBeTruthy()
);
expect(screen.getByTestId("budget-fetch-error").textContent).toContain("500");
});
it("does NOT show stats row on fetch error", async () => {
mockGet.mockRejectedValueOnce(makeGenericError());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() => expect(screen.queryByTestId("budget-loading")).toBeNull());
expect(screen.queryByTestId("budget-stats-row")).toBeNull();
});
it("does NOT show exceeded banner on non-402 fetch error", async () => {
mockGet.mockRejectedValueOnce(makeGenericError());
render(<BudgetSection workspaceId="ws-1" />);
await waitFor(() => expect(screen.queryByTestId("budget-loading")).toBeNull());
expect(screen.queryByTestId("budget-exceeded-banner")).toBeNull();
it("fills to 25%", async () => {
await renderLoaded(withMonthly(1000, 250));
expect((screen.getByTestId("budget-monthly-fill") as HTMLElement).style.width).toBe("25%");
});
it("fills to 50%", async () => {
await renderLoaded(withMonthly(1000, 500));
expect((screen.getByTestId("budget-monthly-fill") as HTMLElement).style.width).toBe("50%");
});
it("caps fill at 100% when spend exceeds limit", async () => {
await renderLoaded(withMonthly(1000, 4000));
expect((screen.getByTestId("budget-monthly-fill") as HTMLElement).style.width).toBe("100%");
});
it("sets aria-valuenow to the computed percentage on the progressbar", async () => {
await renderLoaded(withMonthly(1000, 250));
const bars = screen.getAllByRole("progressbar");
// the monthly bar is the only one rendered (others unlimited)
expect(bars).toHaveLength(1);
expect(bars[0].getAttribute("aria-valuenow")).toBe("25");
});
it("shows a 0% bar when spend is 0 against a set limit", async () => {
await renderLoaded(withMonthly(1000, 0));
expect((screen.getByTestId("budget-monthly-fill") as HTMLElement).style.width).toBe("0%");
});
});
@@ -213,6 +213,7 @@ describe("CreateWorkspaceDialog", () => {
expect(runtimeTexts).toEqual([
"Claude Code",
"OpenAI Codex CLI",
"Google ADK",
"Hermes",
"OpenClaw",
]);
@@ -0,0 +1,110 @@
// @vitest-environment jsdom
//
// internal#718 P3 (retire-list #4) — when GET /templates serves a
// registry-backed selectable list (registry_providers + registry_models with
// display_name / billing_mode / derived provider), the canvas builds the
// provider catalog FROM that registry data instead of re-inferring vendor
// from model-id prefixes (VENDOR_LABELS / BARE_VENDOR_PATTERNS / inferVendor).
// The heuristic path stays only as the fallback for non-registry runtimes /
// older backends.
import { describe, it, expect } from "vitest";
import {
buildProviderCatalogFromRegistry,
type RegistryProvider,
type RegistryModel,
} from "../ProviderModelSelector";
// Mirrors the registry-served claude-code payload from GET /templates
// (registry_providers / registry_models). display_name + billing_mode come
// from the registry, NOT from the canvas VENDOR_LABELS map.
const CLAUDE_CODE_REGISTRY_PROVIDERS: RegistryProvider[] = [
{
name: "anthropic-oauth",
display_name: "Claude Code subscription",
auth_env: ["CLAUDE_CODE_OAUTH_TOKEN"],
billing_mode: "byok",
},
{
name: "anthropic-api",
display_name: "Anthropic API",
auth_env: ["ANTHROPIC_API_KEY"],
billing_mode: "byok",
},
{
name: "platform",
display_name: "Platform",
auth_env: ["ANTHROPIC_API_KEY", "MOLECULE_LLM_USAGE_TOKEN"],
billing_mode: "platform_managed",
},
];
const CLAUDE_CODE_REGISTRY_MODELS: RegistryModel[] = [
{ id: "sonnet", provider: "anthropic-oauth", billing_mode: "byok" },
{ id: "opus", provider: "anthropic-oauth", billing_mode: "byok" },
{ id: "claude-opus-4-7", provider: "anthropic-api", billing_mode: "byok" },
{ id: "anthropic/claude-opus-4-7", provider: "platform", billing_mode: "platform_managed" },
];
describe("buildProviderCatalogFromRegistry", () => {
it("buckets models by their DERIVED registry provider, not by inferred vendor", () => {
const catalog = buildProviderCatalogFromRegistry(
CLAUDE_CODE_REGISTRY_PROVIDERS,
CLAUDE_CODE_REGISTRY_MODELS,
);
const byVendor = new Map(catalog.map((p) => [p.vendor, p]));
// anthropic-oauth bucket holds the two OAuth-derived models.
const oauth = byVendor.get("anthropic-oauth");
expect(oauth).toBeDefined();
expect(oauth!.models.map((m) => m.id).sort()).toEqual(["opus", "sonnet"]);
// platform bucket holds the platform-namespaced model.
const platform = byVendor.get("platform");
expect(platform).toBeDefined();
expect(platform!.models.map((m) => m.id)).toEqual(["anthropic/claude-opus-4-7"]);
});
it("labels providers from the registry display_name, not VENDOR_LABELS", () => {
const catalog = buildProviderCatalogFromRegistry(
CLAUDE_CODE_REGISTRY_PROVIDERS,
CLAUDE_CODE_REGISTRY_MODELS,
);
const oauth = catalog.find((p) => p.vendor === "anthropic-oauth");
// Registry display_name "Claude Code subscription" (decorated with the
// model count by the catalog builder is acceptable; assert it carries the
// registry label, not an inferred one).
expect(oauth!.label).toContain("Claude Code subscription");
});
it("carries the registry billing_mode per provider", () => {
const catalog = buildProviderCatalogFromRegistry(
CLAUDE_CODE_REGISTRY_PROVIDERS,
CLAUDE_CODE_REGISTRY_MODELS,
);
expect(catalog.find((p) => p.vendor === "anthropic-oauth")!.billingMode).toBe("byok");
expect(catalog.find((p) => p.vendor === "platform")!.billingMode).toBe("platform_managed");
});
it("surfaces the registry auth_env on the provider entry", () => {
const catalog = buildProviderCatalogFromRegistry(
CLAUDE_CODE_REGISTRY_PROVIDERS,
CLAUDE_CODE_REGISTRY_MODELS,
);
expect(catalog.find((p) => p.vendor === "anthropic-oauth")!.envVars).toEqual([
"CLAUDE_CODE_OAUTH_TOKEN",
]);
});
it("only includes providers that actually have at least one served model", () => {
// anthropic-api is a registry provider but has no model in this slice →
// it should not appear as an empty bucket.
const models: RegistryModel[] = [
{ id: "sonnet", provider: "anthropic-oauth", billing_mode: "byok" },
];
const catalog = buildProviderCatalogFromRegistry(
CLAUDE_CODE_REGISTRY_PROVIDERS,
models,
);
expect(catalog.map((p) => p.vendor)).toEqual(["anthropic-oauth"]);
});
});
+158 -137
View File
@@ -7,10 +7,28 @@ import { api } from "@/lib/api";
// Types
// ---------------------------------------------------------------------------
// Period keys MUST match the server SSOT (workspace-server budget_periods.go).
type BudgetPeriod = "hourly" | "daily" | "weekly" | "monthly";
const PERIODS: { key: BudgetPeriod; label: string }[] = [
{ key: "hourly", label: "Hourly" },
{ key: "daily", label: "Daily" },
{ key: "weekly", label: "Weekly" },
{ key: "monthly", label: "Monthly" },
];
interface PeriodBudget {
limit: number | null; // USD cents; null = no limit
spend: number; // rolling-window spend, USD cents
remaining: number | null; // null when no limit
}
interface BudgetData {
budget_limit: number | null;
budget_used?: number; // optional — provisioning-stuck workspaces return partial shapes
budget_remaining: number | null;
periods?: Partial<Record<BudgetPeriod, PeriodBudget>>;
// legacy fields (pre-multi-period server) — tolerated for back-compat
budget_limit?: number | null;
monthly_spend?: number;
budget_remaining?: number | null;
}
interface Props {
@@ -26,31 +44,71 @@ function isApiError402(e: unknown): boolean {
return e instanceof Error && /: 402( |$)/.test(e.message);
}
/** USD cents → "$X.XX". */
function fmtUSD(cents: number): string {
return `$${(cents / 100).toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })}`;
}
/** Normalize the server payload (multi-period or legacy) into a period map. */
function periodsFrom(data: BudgetData | null): Record<BudgetPeriod, PeriodBudget> {
const base: Record<BudgetPeriod, PeriodBudget> = {
hourly: { limit: null, spend: 0, remaining: null },
daily: { limit: null, spend: 0, remaining: null },
weekly: { limit: null, spend: 0, remaining: null },
monthly: { limit: null, spend: 0, remaining: null },
};
if (!data) return base;
if (data.periods) {
for (const { key } of PERIODS) {
const p = data.periods[key];
if (p) base[key] = { limit: p.limit ?? null, spend: p.spend ?? 0, remaining: p.remaining ?? null };
}
return base;
}
// legacy: map the single monthly limit/spend
base.monthly = {
limit: data.budget_limit ?? null,
spend: data.monthly_spend ?? 0,
remaining: data.budget_remaining ?? null,
};
return base;
}
// ---------------------------------------------------------------------------
// Component
// ---------------------------------------------------------------------------
/**
* BudgetSection — dedicated "Budget" section in the workspace details panel.
*
* - Fetches GET /workspaces/:id/budget on mount for live usage stats
* - Shows a progress bar (budget_used / budget_limit, blue-500, capped 100%)
* - Allows updating budget_limit via PATCH /workspaces/:id/budget
* - Shows a 402-specific "Budget exceeded" amber banner for any blocked state
* BudgetSection — per-workspace LLM budget, four independent rolling windows
* (hourly / daily / weekly / monthly). Each period has its own ceiling (USD);
* spend is the rolling-window LLM cost. Crossing ANY period blocks new work
* (server returns 402). Sends PATCH {budget_limits:{period:cents|null}}.
*/
export function BudgetSection({ workspaceId }: Props) {
const [budget, setBudget] = useState<BudgetData | null>(null);
const [loading, setLoading] = useState(true);
const [fetchError, setFetchError] = useState<string | null>(null);
const [limitInput, setLimitInput] = useState("");
// One input per period, in USD cents (string for controlled inputs).
const [limitInputs, setLimitInputs] = useState<Record<BudgetPeriod, string>>({
hourly: "",
daily: "",
weekly: "",
monthly: "",
});
const [saving, setSaving] = useState(false);
const [saveError, setSaveError] = useState<string | null>(null);
/** True when a 402 has been seen from any API call in this section. */
const [budgetExceeded, setBudgetExceeded] = useState(false);
// ── Fetch current budget data ─────────────────────────────────────────────
const syncInputs = useCallback((data: BudgetData | null) => {
const p = periodsFrom(data);
setLimitInputs({
hourly: p.hourly.limit != null ? String(p.hourly.limit) : "",
daily: p.daily.limit != null ? String(p.daily.limit) : "",
weekly: p.weekly.limit != null ? String(p.weekly.limit) : "",
monthly: p.monthly.limit != null ? String(p.monthly.limit) : "",
});
}, []);
const loadBudget = useCallback(async () => {
setLoading(true);
@@ -58,7 +116,7 @@ export function BudgetSection({ workspaceId }: Props) {
try {
const data = await api.get<BudgetData>(`/workspaces/${workspaceId}/budget`);
setBudget(data);
setLimitInput(data.budget_limit != null ? String(data.budget_limit) : "");
syncInputs(data);
} catch (e) {
if (isApiError402(e)) {
setBudgetExceeded(true);
@@ -68,29 +126,30 @@ export function BudgetSection({ workspaceId }: Props) {
} finally {
setLoading(false);
}
}, [workspaceId]);
}, [workspaceId, syncInputs]);
useEffect(() => {
loadBudget();
}, [loadBudget]);
// ── Save handler ──────────────────────────────────────────────────────────
const handleSave = async () => {
setSaving(true);
setSaveError(null);
const raw = limitInput.trim();
// Use explicit empty-string check (not falsy check) so that a
// user-entered "0" is sent as budget_limit: 0, not null (unlimited).
const parsedLimit = raw !== "" ? parseInt(raw, 10) : null;
// Build the per-period map: blank → null (clear); a number → that ceiling.
const budget_limits: Record<BudgetPeriod, number | null> = {
hourly: null,
daily: null,
weekly: null,
monthly: null,
};
for (const { key } of PERIODS) {
const raw = limitInputs[key].trim();
budget_limits[key] = raw !== "" ? parseInt(raw, 10) : null;
}
try {
const updated = await api.patch<BudgetData>(`/workspaces/${workspaceId}/budget`, {
budget_limit: parsedLimit,
});
const updated = await api.patch<BudgetData>(`/workspaces/${workspaceId}/budget`, { budget_limits });
setBudget(updated);
setLimitInput(updated.budget_limit != null ? String(updated.budget_limit) : "");
// Clear exceeded state if the save succeeded (limit was raised or removed)
syncInputs(updated);
setBudgetExceeded(false);
} catch (e) {
if (isApiError402(e)) {
@@ -103,24 +162,15 @@ export function BudgetSection({ workspaceId }: Props) {
}
};
// ── Progress calculation ──────────────────────────────────────────────────
const progressPct =
budget && budget.budget_limit != null && budget.budget_limit > 0
? Math.min(100, Math.round(((budget.budget_used ?? 0) / budget.budget_limit) * 100))
: 0;
// ── Render ────────────────────────────────────────────────────────────────
const periods = periodsFrom(budget);
return (
<div className="space-y-3" data-testid="budget-section">
{/* Section header */}
<div>
<h3 className="text-xs font-semibold text-ink-mid uppercase tracking-wider">
Budget
</h3>
<h3 className="text-xs font-semibold text-ink-mid uppercase tracking-wider">Budget</h3>
<p className="text-[11px] text-ink-mid mt-0.5">
Limit total message credits for this workspace
Cap LLM spend for this workspace per period crossing any limit pauses new work
</p>
</div>
@@ -131,32 +181,14 @@ export function BudgetSection({ workspaceId }: Props) {
data-testid="budget-exceeded-banner"
className="flex items-center gap-2 px-3 py-2 rounded-lg bg-surface border border-amber-700/50 text-warm text-xs font-medium"
>
<svg
width="13"
height="13"
viewBox="0 0 13 13"
fill="none"
aria-hidden="true"
className="shrink-0"
>
<path
d="M6.5 1.5L11.5 10.5H1.5L6.5 1.5Z"
stroke="currentColor"
strokeWidth="1.4"
strokeLinejoin="round"
/>
<path
d="M6.5 5.5V7.5M6.5 9.5h.01"
stroke="currentColor"
strokeWidth="1.4"
strokeLinecap="round"
/>
<svg width="13" height="13" viewBox="0 0 13 13" fill="none" aria-hidden="true" className="shrink-0">
<path d="M6.5 1.5L11.5 10.5H1.5L6.5 1.5Z" stroke="currentColor" strokeWidth="1.4" strokeLinejoin="round" />
<path d="M6.5 5.5V7.5M6.5 9.5h.01" stroke="currentColor" strokeWidth="1.4" strokeLinecap="round" />
</svg>
Budget exceeded messages blocked
Budget exceeded new work paused
</div>
)}
{/* Usage stats */}
{loading ? (
<p className="text-xs text-ink-mid" data-testid="budget-loading">
Loading
@@ -165,89 +197,78 @@ export function BudgetSection({ workspaceId }: Props) {
<p className="text-xs text-bad" data-testid="budget-fetch-error">
{fetchError}
</p>
) : budget ? (
<div className="space-y-2">
{/* Stats row */}
<div className="flex items-baseline justify-between" data-testid="budget-stats-row">
<span className="text-xs text-ink-mid">Credits used</span>
<span className="text-xs font-mono text-ink-mid">
<span data-testid="budget-used-value">{(budget.budget_used ?? 0).toLocaleString()}</span>
<span className="text-ink-mid mx-1">/</span>
<span data-testid="budget-limit-value">
{budget.budget_limit != null
? budget.budget_limit.toLocaleString()
: "Unlimited"}
</span>
</span>
</div>
) : (
<div className="space-y-3">
{PERIODS.map(({ key, label }) => {
const p = periods[key];
const pct =
p.limit != null && p.limit > 0 ? Math.min(100, Math.round((p.spend / p.limit) * 100)) : 0;
const over = p.limit != null && p.spend >= p.limit;
return (
<div key={key} className="space-y-1" data-testid={`budget-period-${key}`}>
<div className="flex items-baseline justify-between">
<label htmlFor={`budget-${key}-${workspaceId}`} className="text-xs text-ink-mid">
{label}
</label>
<span className="text-[11px] font-mono text-ink-mid">
<span data-testid={`budget-${key}-spend`}>{fmtUSD(p.spend)}</span>
<span className="mx-1">/</span>
<span data-testid={`budget-${key}-limit`}>{p.limit != null ? fmtUSD(p.limit) : "∞"}</span>
</span>
</div>
{p.limit != null && (
<div
role="progressbar"
aria-label={`${label} budget usage`}
aria-valuenow={pct}
aria-valuemin={0}
aria-valuemax={100}
className="h-1.5 w-full rounded-full bg-surface-card overflow-hidden"
>
<div
data-testid={`budget-${key}-fill`}
className={`h-full rounded-full transition-all duration-300 ${over ? "bg-bad" : "bg-accent"}`}
style={{ width: `${pct}%` }}
/>
</div>
)}
<input
id={`budget-${key}-${workspaceId}`}
type="number"
min="0"
step="1"
value={limitInputs[key]}
onChange={(e) => setLimitInputs((s) => ({ ...s, [key]: e.target.value }))}
placeholder="USD cents — blank for unlimited"
data-testid={`budget-${key}-input`}
className="w-full bg-surface-card border border-line rounded-lg px-3 py-1.5 text-xs text-ink-mid placeholder-zinc-500 focus:outline-none focus:border-accent focus:ring-1 focus:ring-accent/30 transition-colors"
/>
</div>
);
})}
{/* Progress bar (only when limit is set) */}
{budget.budget_limit != null && (
<p className="text-[11px] text-ink-mid">Limits are USD cents (e.g. 500 = $5.00). Blank = unlimited.</p>
{saveError && (
<div
role="progressbar"
aria-label="Budget usage"
aria-valuenow={progressPct}
aria-valuemin={0}
aria-valuemax={100}
className="h-1.5 w-full rounded-full bg-surface-card overflow-hidden"
role="alert"
data-testid="budget-save-error"
className="px-3 py-1.5 rounded-lg bg-red-950/40 border border-red-800/50 text-xs text-bad"
>
<div
data-testid="budget-progress-fill"
className="h-full rounded-full bg-accent transition-all duration-300"
style={{ width: `${progressPct}%` }}
/>
{saveError}
</div>
)}
{/* Remaining credits */}
{budget.budget_remaining != null && (
<p className="text-[11px] text-ink-mid" data-testid="budget-remaining">
{budget.budget_remaining.toLocaleString()} credits remaining
</p>
)}
</div>
) : null}
{/* Input + Save */}
<div className="space-y-1.5 pt-1">
<label
htmlFor={`budget-limit-input-${workspaceId}`}
className="text-[11px] text-ink-mid block"
>
Budget limit (credits)
</label>
<input
id={`budget-limit-input-${workspaceId}`}
type="number"
min="0"
step="1"
value={limitInput}
onChange={(e) => setLimitInput(e.target.value)}
placeholder="e.g. 1000 — blank for unlimited"
data-testid="budget-limit-input"
className="w-full bg-surface-card border border-line rounded-lg px-3 py-2 text-sm text-ink-mid placeholder-zinc-500 focus:outline-none focus:border-accent focus:ring-1 focus:ring-accent/30 transition-colors"
/>
<p className="text-xs text-ink-mid">Leave blank for unlimited</p>
{saveError && (
<div
role="alert"
data-testid="budget-save-error"
className="px-3 py-1.5 rounded-lg bg-red-950/40 border border-red-800/50 text-xs text-bad"
<button
onClick={handleSave}
disabled={saving}
data-testid="budget-save-btn"
className="px-4 py-1.5 bg-accent-strong hover:bg-accent active:bg-accent-strong rounded-lg text-xs font-medium text-white disabled:opacity-50 transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent focus-visible:ring-offset-1 focus-visible:ring-offset-zinc-900"
>
{saveError}
</div>
)}
<button
onClick={handleSave}
disabled={saving}
data-testid="budget-save-btn"
className="px-4 py-1.5 bg-accent-strong hover:bg-accent active:bg-accent-strong rounded-lg text-xs font-medium text-white disabled:opacity-50 transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent focus-visible:ring-offset-1 focus-visible:ring-offset-zinc-900"
>
{saving ? "Saving…" : "Save"}
</button>
</div>
{saving ? "Saving…" : "Save"}
</button>
</div>
)}
</div>
);
}
+127 -15
View File
@@ -11,8 +11,12 @@ import { ExternalConnectionSection } from "./ExternalConnectionSection";
import {
ProviderModelSelector,
buildProviderCatalog,
buildProviderCatalogFromRegistry,
findProviderForModel,
type SelectorValue,
type ProviderEntry,
type RegistryProvider,
type RegistryModel,
} from "../ProviderModelSelector";
import { isExternalLikeRuntime } from "@/lib/externalRuntimes";
@@ -258,6 +262,17 @@ interface RuntimeOption {
// canvas falls back to deriving unique vendor prefixes from
// models[].id (still adapter-driven, just inferred).
providers: string[];
// registryBacked / registryProviders / registryModels come from the
// registry-served GET /templates fields (internal#718 P3). When
// registryBacked is true, the selectable provider+model list is built from
// the registry (registryProviders/registryModels) — display labels +
// billing mode + derived provider come from the provider-registry SSOT, not
// the canvas VENDOR_LABELS / billingModeForProvider vocabularies. When
// false (non-registry runtime / older backend), the canvas falls back to
// the template-served models[] + its inferVendor heuristic.
registryBacked: boolean;
registryProviders: RegistryProvider[];
registryModels: RegistryModel[];
}
// deriveProvidersFromModels — when a template doesn't ship an explicit
@@ -322,6 +337,32 @@ export function billingModeForProvider(provider: string): LLMBillingMode {
return "byok";
}
// billingModeForSelectedProvider — internal#718 P3 (retire-list #5): the
// billing mode the Config tab shows/sends for the selected PROVIDER, sourced
// from the registry-served catalog when available rather than the hardcoded
// billingModeForProvider rule.
//
// When the runtime is registry-backed, GET /templates serves each provider's
// DERIVED billing_mode (platform_managed for the closed platform provider,
// byok otherwise) on the ProviderEntry. We read it off the catalog so the UI
// reflects the registry SSOT — the same predicate billing/credential emission
// keys off the derived provider.
//
// Falls back to billingModeForProvider when: no catalog (non-registry runtime
// / older backend), or the provider string isn't carried by the catalog
// (e.g. a stale saved value). The fallback keeps the legacy behavior intact
// for everything the registry doesn't yet speak to.
export function billingModeForSelectedProvider(
provider: string,
catalog?: ProviderEntry[],
): LLMBillingMode {
if (catalog && catalog.length > 0) {
const entry = catalog.find((p) => p.vendor === provider.trim());
if (entry?.billingMode) return entry.billingMode;
}
return billingModeForProvider(provider);
}
// Fallback used when /templates can't be fetched (offline, older backend).
// Keep in sync with manifest.json workspace_templates as a defensive default.
// Model + env suggestions only flow when the backend is reachable.
@@ -336,13 +377,20 @@ export function billingModeForProvider(provider: string): LLMBillingMode {
// config.yaml` on the container is a separate runtime-internal file,
// not this one.
const RUNTIMES_WITH_OWN_CONFIG = new Set<string>(["external", "kimi", "kimi-cli", "openclaw"]);
const SUPPORTED_RUNTIME_VALUES = new Set(["claude-code", "codex", "openclaw", "hermes"]);
// The runtime picker is SSOT-driven: options come from GET /templates,
// which workspace-server already gates to the manifest.json maintained set
// (loadRuntimesFromManifest). A hand-maintained frontend allowlist silently
// dropped runtimes the backend added (google-adk shipped in manifest but was
// filtered out, so its workspaces rendered the wrong default option). A
// template may still opt OUT of the picker via `displayable: false` on its
// /templates row. See project_canvas_runtime_dropdown_ssot_fix.
const FALLBACK_RUNTIME_OPTIONS: RuntimeOption[] = [
{ value: "claude-code", label: "Claude Code", models: [], providers: [] },
{ value: "codex", label: "Codex", models: [], providers: [] },
{ value: "openclaw", label: "OpenClaw", models: [], providers: [] },
{ value: "hermes", label: "Hermes", models: [], providers: [] },
{ value: "claude-code", label: "Claude Code", models: [], providers: [], registryBacked: false, registryProviders: [], registryModels: [] },
{ value: "codex", label: "Codex", models: [], providers: [], registryBacked: false, registryProviders: [], registryModels: [] },
{ value: "google-adk", label: "Google ADK", models: [], providers: [], registryBacked: false, registryProviders: [], registryModels: [] },
{ value: "openclaw", label: "OpenClaw", models: [], providers: [], registryBacked: false, registryProviders: [], registryModels: [] },
{ value: "hermes", label: "Hermes", models: [], providers: [], registryBacked: false, registryProviders: [], registryModels: [] },
];
export function ConfigTab({ workspaceId }: Props) {
@@ -533,20 +581,49 @@ export function ConfigTab({ workspaceId }: Props) {
useEffect(() => {
let cancelled = false;
api.get<Array<{ id: string; name?: string; runtime?: string; models?: ModelSpec[]; providers?: string[] }>>("/templates")
api.get<Array<{
id: string;
name?: string;
runtime?: string;
models?: ModelSpec[];
providers?: string[];
// internal#718 P3 registry-served fields (additive; absent on older
// backends and for non-registry runtimes).
registry_backed?: boolean;
registry_providers?: RegistryProvider[];
registry_models?: RegistryModel[];
displayable?: boolean;
}>>("/templates")
.then((rows) => {
if (cancelled || !Array.isArray(rows)) return;
const byRuntime = new Map<string, RuntimeOption>();
for (const r of rows) {
const v = (r.runtime || "").trim();
if (!SUPPORTED_RUNTIME_VALUES.has(v)) continue;
if (!v) continue;
// Honor an explicit opt-out; absent/true means show it.
if (r.displayable === false) continue;
// Last template wins if two templates share a runtime — rare, and the
// one with the richer models list is probably newer.
const existing = byRuntime.get(v);
const models = Array.isArray(r.models) ? r.models : [];
const providers = Array.isArray(r.providers) ? r.providers : [];
if (!existing || models.length > existing.models.length) {
byRuntime.set(v, { value: v, label: r.name || v, models, providers });
const registryProviders = Array.isArray(r.registry_providers) ? r.registry_providers : [];
const registryModels = Array.isArray(r.registry_models) ? r.registry_models : [];
const registryBacked = r.registry_backed === true && registryModels.length > 0;
// Prefer the richer payload: a registry-backed entry, then more
// template models. Keeps the "last/richer template wins" intent.
const score = (o: RuntimeOption) => (o.registryBacked ? 1000 : 0) + o.models.length;
const candidate: RuntimeOption = {
value: v,
label: r.name || v,
models,
providers,
registryBacked,
registryProviders,
registryModels,
};
if (!existing || score(candidate) > score(existing)) {
byRuntime.set(v, candidate);
}
}
if (byRuntime.size > 0) setRuntimeOptions(Array.from(byRuntime.values()));
@@ -557,7 +634,13 @@ export function ConfigTab({ workspaceId }: Props) {
// Models + env hints for the currently-selected runtime.
const selectedRuntime = runtimeOptions.find((o) => o.value === (config.runtime || "")) ?? null;
const availableModels: ModelSpec[] = selectedRuntime?.models ?? [];
// Memoised so its identity is stable across renders — it feeds several
// useMemo dependency arrays below (registry/legacy catalog, selector models)
// and a fresh `[]` literal each render would defeat their memoisation.
const availableModels: ModelSpec[] = useMemo(
() => selectedRuntime?.models ?? [],
[selectedRuntime?.models],
);
// Provider suggestions for the legacy free-text input fallback (used
// when /templates returned no models for this runtime, e.g. hermes
// workspaces). Prefer the runtime's declarative providers list,
@@ -571,9 +654,37 @@ export function ConfigTab({ workspaceId }: Props) {
// Vendor-aware catalog shared with the selector. Memoised so the
// catalog identity is stable across renders (selector relies on it).
//
// internal#718 P3: when the runtime is registry-backed, build the catalog
// FROM the registry-served providers/models (display labels + billing +
// derived provider from the provider-registry SSOT) instead of re-inferring
// vendor from model-id prefixes. Falls back to the inferVendor heuristic
// for non-registry runtimes / older backends.
const registryBacked = selectedRuntime?.registryBacked ?? false;
const providerCatalog = useMemo(
() => buildProviderCatalog(availableModels),
[availableModels],
() =>
registryBacked
? buildProviderCatalogFromRegistry(
selectedRuntime?.registryProviders ?? [],
selectedRuntime?.registryModels ?? [],
)
: buildProviderCatalog(availableModels),
[registryBacked, selectedRuntime?.registryProviders, selectedRuntime?.registryModels, availableModels],
);
// Models fed to the selector dropdown: the registry-served native set for a
// registry-backed runtime (so the dropdown can render no unregistered
// option), else the template-served models.
const selectorModels: ModelSpec[] = useMemo(
() =>
registryBacked
? (selectedRuntime?.registryModels ?? []).map((m) => ({
id: m.id,
name: m.name,
// carry the derived provider so the selector buckets correctly
...(m.provider ? { provider: m.provider } : {}),
}))
: availableModels,
[registryBacked, selectedRuntime?.registryModels, availableModels],
);
// Derive the selector's current value from the form state. Provider
@@ -893,9 +1004,10 @@ export function ConfigTab({ workspaceId }: Props) {
— empty = "auto-derive from model slug" was the pre-PR-5
behavior; selecting any provider here writes LLM_PROVIDER
and triggers an auto-restart. */}
{availableModels.length > 0 ? (
{selectorModels.length > 0 ? (
<ProviderModelSelector
models={availableModels}
models={selectorModels}
catalog={registryBacked ? providerCatalog : undefined}
value={selectorValue}
onChange={(next) => {
setSelectorValue(next);
@@ -908,7 +1020,7 @@ export function ConfigTab({ workspaceId }: Props) {
setConfig((prev) => {
const v = next.model;
const prevModelId = prev.runtime_config?.model || prev.model || "";
const prevSpec = availableModels.find((m) => m.id === prevModelId) ?? null;
const prevSpec = selectorModels.find((m) => m.id === prevModelId) ?? null;
const prevRequired = prev.runtime_config?.required_env ?? [];
const wasTemplateDriven =
prevRequired.length === 0 ||
@@ -29,8 +29,15 @@ type FormState = {
displayMode: string;
displayProtocol: string;
resolution: string;
dataPersistence: string; // "" (auto) | "persist" | "ephemeral" — internal#734
};
// internal#734: per-workspace durable-data choice. "" = auto (desktop-control
// keeps data, others follow the org default). Human labels for the selector.
const DATA_PERSISTENCE_OPTIONS = ["", "persist", "ephemeral"];
const dataPersistenceLabel = (v: string): string =>
v === "persist" ? "Always keep (persist)" : v === "ephemeral" ? "Don't keep (ephemeral)" : "Auto";
export function ContainerConfigTab({ workspaceId, data }: Props) {
const runtime = data.runtime;
const instanceType = data.compute?.instance_type;
@@ -39,9 +46,10 @@ export function ContainerConfigTab({ workspaceId, data }: Props) {
const displayProtocol = data.compute?.display?.protocol;
const displayWidth = data.compute?.display?.width;
const displayHeight = data.compute?.display?.height;
const dataPersistence = data.compute?.data_persistence;
const initial = useMemo(
() => formFromData({ runtime, instanceType, rootGB, displayMode, displayProtocol, displayWidth, displayHeight }),
[runtime, instanceType, rootGB, displayMode, displayProtocol, displayWidth, displayHeight],
() => formFromData({ runtime, instanceType, rootGB, displayMode, displayProtocol, displayWidth, displayHeight, dataPersistence }),
[runtime, instanceType, rootGB, displayMode, displayProtocol, displayWidth, displayHeight, dataPersistence],
);
const [form, setForm] = useState<FormState>(initial);
const [saving, setSaving] = useState(false);
@@ -84,6 +92,8 @@ export function ContainerConfigTab({ workspaceId, data }: Props) {
display: form.displayEnabled
? { mode: form.displayMode, protocol: form.displayProtocol, width, height }
: { mode: "none" },
// internal#734: omit when "auto" so the wire/default behavior is unchanged.
...(form.dataPersistence ? { data_persistence: form.dataPersistence } : {}),
};
const resp = await api.patch<{ needs_restart?: boolean }>(`/workspaces/${workspaceId}`, {
@@ -176,6 +186,18 @@ export function ContainerConfigTab({ workspaceId, data }: Props) {
onChange={(resolution) => setForm((s) => ({ ...s, resolution }))}
/>
)}
<SelectField
id="data-persistence"
label="Saved data (cookies, downloads, memory)"
value={form.dataPersistence}
options={DATA_PERSISTENCE_OPTIONS}
optionLabel={dataPersistenceLabel}
onChange={(dataPersistence) => setForm((s) => ({ ...s, dataPersistence }))}
/>
<p className="-mt-1 text-[10px] leading-snug text-ink-soft">
Whether this workspace&apos;s data survives a restart/recreate. Auto keeps it for
browser (desktop) workspaces; Ephemeral never keeps it (privacy).
</p>
</div>
<div className="mt-4 flex items-center justify-end gap-2">
@@ -231,6 +253,7 @@ function formFromData(data: {
displayProtocol?: string;
displayWidth?: number;
displayHeight?: number;
dataPersistence?: string;
}): FormState {
const width = data.displayWidth ?? 1920;
const height = data.displayHeight ?? 1080;
@@ -243,6 +266,7 @@ function formFromData(data: {
displayMode: data.displayMode && data.displayMode !== "none" ? data.displayMode : "desktop-control",
displayProtocol: data.displayProtocol || "novnc",
resolution,
dataPersistence: data.dataPersistence || "",
};
}
+19 -1
View File
@@ -29,6 +29,7 @@ export function DetailsTab({ workspaceId, data }: Props) {
const [peers, setPeers] = useState<PeerData[]>([]);
const [saving, setSaving] = useState(false);
const [confirmDelete, setConfirmDelete] = useState(false);
const [eraseData, setEraseData] = useState(false); // internal#734: erase saved data on delete
const [peersError, setPeersError] = useState<string | null>(null);
const [saveError, setSaveError] = useState<string | null>(null);
const [deleteError, setDeleteError] = useState<string | null>(null);
@@ -93,7 +94,10 @@ export function DetailsTab({ workspaceId, data }: Props) {
const handleDelete = async () => {
setDeleteError(null);
try {
await api.del(`/workspaces/${workspaceId}?confirm=true`, {
// internal#734: erase_data=true asks the server to prune this workspace's
// durable data volume (cookies / downloads / memory). Default off keeps it
// for the orphan-sweeper grace.
await api.del(`/workspaces/${workspaceId}?confirm=true${eraseData ? "&erase_data=true" : ""}`, {
headers: { "X-Confirm-Name": name },
});
// Mirror the server-side cascade — drop the row + every
@@ -323,6 +327,19 @@ export function DetailsTab({ workspaceId, data }: Props) {
<h3 id="delete-confirm-title" className="text-xs font-medium text-bad">
Confirm deletion
</h3>
<label className="flex items-start gap-2 text-[11px] text-ink-mid">
<input
type="checkbox"
aria-label="Also erase saved data"
checked={eraseData}
onChange={(e) => setEraseData(e.target.checked)}
className="mt-0.5 h-3.5 w-3.5 accent-red-600"
/>
<span>
Also erase saved data (cookies, downloads, agent memory). Cannot be undone.
Unchecked keeps it recoverable briefly.
</span>
</label>
<div className="flex gap-2">
<button
type="button"
@@ -339,6 +356,7 @@ export function DetailsTab({ workspaceId, data }: Props) {
onClick={() => {
setConfirmDelete(false);
setDeleteError(null);
setEraseData(false);
// Return focus to the trigger so keyboard users aren't stranded
deleteButtonRef.current?.focus();
}}
@@ -5,9 +5,10 @@ import React from "react";
import { BudgetSection } from "../BudgetSection";
import { api } from "@/lib/api";
// Queue-based mock for the api module. Each api call shifts from the queue.
// Tests push with qGet/qPatch and the module-level mockImplementation
// reads from the queue.
// Multi-period budget (#49): the API now returns a `periods` map
// (hourly/daily/weekly/monthly), each {limit, spend, remaining} in USD cents.
// The UI renders one row per period and PATCHes {budget_limits:{period:cents|null}}.
type QueueEntry = { body?: unknown; err?: Error };
const apiQueue: QueueEntry[] = [];
@@ -40,45 +41,49 @@ const WS_ID = "budget-test-ws";
function qGet(body: unknown) {
apiQueue.push({ body });
}
function qGetErr(status: number, msg: string) {
apiQueue.push({ err: new Error(`${msg}: ${status}`) });
}
function qPatch(body: unknown) {
apiQueue.push({ body });
}
function qPatchErr(status: number, msg: string) {
apiQueue.push({ err: new Error(`${msg}: ${status}`) });
}
function makeBudget(overrides: Partial<{
budget_limit: number | null;
budget_used: number;
budget_remaining: number | null;
}> = {}) {
type P = { limit: number | null; spend: number; remaining: number | null };
// makeBudget builds the periods response. Override any subset of periods.
function makeBudget(overrides: Partial<Record<"hourly" | "daily" | "weekly" | "monthly", Partial<P>>> = {}) {
const blank: P = { limit: null, spend: 0, remaining: null };
const mk = (o?: Partial<P>): P => {
const p = { ...blank, ...(o ?? {}) };
if (p.limit != null && p.remaining == null) p.remaining = p.limit - p.spend;
return p;
};
const periods = {
hourly: mk(overrides.hourly),
daily: mk(overrides.daily),
weekly: mk(overrides.weekly),
monthly: mk(overrides.monthly),
};
return {
budget_limit: 10_000,
budget_used: 3_500,
budget_remaining: 6_500,
...overrides,
periods,
budget_limit: periods.monthly.limit,
monthly_spend: periods.monthly.spend,
budget_remaining: periods.monthly.remaining,
};
}
describe("BudgetSection", () => {
describe("BudgetSection (multi-period)", () => {
describe("loading state", () => {
it("shows loading indicator while fetching", async () => {
let resolveGet: (v: unknown) => void;
vi.mocked(api.get).mockImplementationOnce(
async () => new Promise((r) => { resolveGet = r as (v: unknown) => void; }),
);
render(<BudgetSection workspaceId={WS_ID} />);
expect(screen.getByTestId("budget-loading")).toBeTruthy();
// Resolve after render to verify state clears
resolveGet!(makeBudget());
await vi.waitFor(() => {
expect(screen.queryByTestId("budget-loading")).toBeNull();
@@ -89,21 +94,16 @@ describe("BudgetSection", () => {
describe("fetch error state", () => {
it("shows error message on non-402 fetch failure", async () => {
qGetErr(500, "Internal Server Error");
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-fetch-error")).toBeTruthy();
});
expect(screen.getByTestId("budget-fetch-error")!.textContent).toContain("500");
});
it("shows 402 as exceeded banner, not fetch error", async () => {
// 402 means the budget limit was hit — different UX from a network/API error.
it("shows the exceeded banner (not a fetch error) on a 402", async () => {
qGetErr(402, "Payment Required");
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy();
});
@@ -111,220 +111,105 @@ describe("BudgetSection", () => {
});
});
describe("budget loaded — display", () => {
it("renders used / limit stats row", async () => {
qGet(makeBudget({ budget_limit: 10_000, budget_used: 3_500 }));
describe("rendering periods", () => {
it("renders all four period rows", async () => {
qGet(makeBudget());
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-used-value")!.textContent).toBe("3,500");
});
expect(screen.getByTestId("budget-limit-value")!.textContent).toBe("10,000");
});
it("renders 'Unlimited' when budget_limit is null", async () => {
qGet(makeBudget({ budget_limit: null, budget_used: 1_000, budget_remaining: null }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-limit-value")!.textContent).toBe("Unlimited");
for (const k of ["hourly", "daily", "weekly", "monthly"]) {
expect(screen.getByTestId(`budget-period-${k}`)).toBeTruthy();
}
});
});
it("renders remaining credits when present", async () => {
qGet(makeBudget({ budget_limit: 10_000, budget_used: 3_500, budget_remaining: 6_500 }));
it("formats spend and limit as USD per period", async () => {
qGet(makeBudget({ monthly: { limit: 10_000, spend: 3_500 } }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-remaining")!.textContent).toContain("6,500");
expect(screen.getByTestId("budget-remaining")!.textContent).toContain("credits remaining");
expect(screen.getByTestId("budget-monthly-spend")!.textContent).toBe("$35.00");
});
expect(screen.getByTestId("budget-monthly-limit")!.textContent).toBe("$100.00");
});
it("shows ∞ for a period with no limit", async () => {
qGet(makeBudget({ hourly: { limit: null, spend: 1_000 } }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-hourly-limit")!.textContent).toBe("∞");
});
});
it("omits remaining credits when budget_remaining is null", async () => {
qGet(makeBudget({ budget_limit: 10_000, budget_used: 3_500, budget_remaining: null }));
it("renders the progress bar only for periods with a limit", async () => {
qGet(makeBudget({ monthly: { limit: 10_000, spend: 12_000 }, hourly: { limit: null, spend: 5_000 } }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.queryByTestId("budget-remaining")).toBeNull();
});
});
it("caps progress bar at 100% when used > limit", async () => {
// Over-limit: 12000 used of 10000 limit should show 100%, not 120%.
qGet(makeBudget({ budget_limit: 10_000, budget_used: 12_000, budget_remaining: null }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
const fill = screen.getByTestId("budget-progress-fill");
expect(fill.getAttribute("style")).toContain("100%");
});
});
it("omits progress bar when budget_limit is null (unlimited)", async () => {
qGet(makeBudget({ budget_limit: null, budget_used: 5_000, budget_remaining: null }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.queryByTestId("budget-progress-fill")).toBeNull();
expect(screen.getByTestId("budget-monthly-fill")).toBeTruthy();
});
expect(screen.queryByTestId("budget-hourly-fill")).toBeNull();
// over-budget fill caps at 100%
const fill = screen.getByTestId("budget-monthly-fill") as HTMLElement;
expect(fill.style.width).toBe("100%");
});
});
describe("budget exceeded (402)", () => {
it("shows exceeded banner when load returns 402", async () => {
qGetErr(402, "Payment Required");
describe("save", () => {
it("PATCHes budget_limits for all four periods and clears the exceeded banner", async () => {
qGet(makeBudget({ monthly: { limit: 10_000, spend: 3_500 } }));
qPatch(makeBudget({ hourly: { limit: 500, spend: 0 }, monthly: { limit: 20_000, spend: 0 } }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-hourly-input")).toBeTruthy();
});
fireEvent.change(screen.getByTestId("budget-hourly-input"), { target: { value: "500" } });
fireEvent.click(screen.getByTestId("budget-save-btn"));
await vi.waitFor(() => {
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy();
expect(screen.getByTestId("budget-exceeded-banner")!.textContent).toContain("Budget exceeded");
expect(vi.mocked(api.patch)).toHaveBeenCalled();
});
const [, body] = vi.mocked(api.patch).mock.calls[0];
expect((body as { budget_limits: Record<string, number | null> }).budget_limits).toMatchObject({
hourly: 500,
monthly: 10_000, // unchanged input echoes the loaded limit
});
});
it("clears exceeded banner after successful save", async () => {
qGetErr(402, "Payment Required");
qPatch(makeBudget({ budget_limit: 50_000, budget_used: 0, budget_remaining: 50_000 }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy();
});
const input = screen.getByTestId("budget-limit-input");
fireEvent.change(input, { target: { value: "50000" } });
const saveBtn = screen.getByTestId("budget-save-btn");
fireEvent.click(saveBtn);
await vi.waitFor(() => {
expect(screen.queryByTestId("budget-exceeded-banner")).toBeNull();
});
});
});
describe("save flow", () => {
it("shows save error on non-402 patch failure", async () => {
it("shows a save error on non-402 PATCH failure", async () => {
qGet(makeBudget());
qPatchErr(500, "Internal Server Error");
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-limit-input")).toBeTruthy();
expect(screen.getByTestId("budget-save-btn")).toBeTruthy();
});
const saveBtn = screen.getByTestId("budget-save-btn");
fireEvent.click(saveBtn);
fireEvent.click(screen.getByTestId("budget-save-btn"));
await vi.waitFor(() => {
expect(screen.getByTestId("budget-save-error")).toBeTruthy();
expect(screen.getByTestId("budget-save-error")!.textContent).toContain("500");
});
expect(screen.getByTestId("budget-save-error")!.textContent).toContain("500");
});
it("updates input to new limit value after successful save", async () => {
qGet(makeBudget({ budget_limit: 10_000 }));
qPatch(makeBudget({ budget_limit: 20_000 }));
render(<BudgetSection workspaceId={WS_ID} />);
// Wait for the input to appear (loading → loaded)
await vi.waitFor(() => {
expect(screen.queryByTestId("budget-loading")).toBeNull();
});
const input = screen.getByTestId("budget-limit-input") as HTMLInputElement;
// Debug: check what values are rendered
const limitValue = screen.getByTestId("budget-limit-value")?.textContent;
expect(input.value).toBe("10000"); // initial value from API
expect(limitValue).toBe("10,000");
fireEvent.change(input, { target: { value: "20000" } });
expect(input.value).toBe("20000");
fireEvent.click(screen.getByTestId("budget-save-btn"));
await vi.waitFor(() => {
expect((screen.getByTestId("budget-limit-input") as HTMLInputElement).value).toBe("20000");
});
});
it("sends null when input is cleared (unlimited)", async () => {
qGet(makeBudget({ budget_limit: 10_000 }));
qPatch(makeBudget({ budget_limit: null }));
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-limit-input")).toBeTruthy();
});
const input = screen.getByTestId("budget-limit-input") as HTMLInputElement;
fireEvent.change(input, { target: { value: "" } });
fireEvent.click(screen.getByTestId("budget-save-btn"));
await vi.waitFor(() => {
// After save with null limit, input should show empty (unlimited)
expect(input.value).toBe("");
});
});
it("shows saving state on button while patch is in flight", async () => {
it("surfaces the exceeded banner on a 402 PATCH", async () => {
qGet(makeBudget());
let resolvePatch: (v: unknown) => void;
vi.mocked(api.patch).mockImplementationOnce(
async () => new Promise((r) => { resolvePatch = r as (v: unknown) => void; }),
);
qPatchErr(402, "Payment Required");
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-limit-input")).toBeTruthy();
expect(screen.getByTestId("budget-save-btn")).toBeTruthy();
});
fireEvent.change(screen.getByTestId("budget-limit-input"), { target: { value: "50000" } });
fireEvent.click(screen.getByTestId("budget-save-btn"));
const btn = screen.getByTestId("budget-save-btn");
expect(btn.textContent).toContain("Saving");
resolvePatch!(makeBudget({ budget_limit: 50_000 }));
await vi.waitFor(() => {
expect(btn.textContent).toContain("Save");
});
});
});
describe("isApiError402 — regression coverage", () => {
it("classifies ': 402' with space as 402", async () => {
qGetErr(402, "Payment Required");
qPatch(makeBudget());
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-exceeded-banner")).toBeTruthy();
});
});
});
it("classifies non-402 error messages as regular fetch errors", async () => {
qGetErr(503, "Service Unavailable");
describe("legacy payload back-compat", () => {
it("maps a pre-multi-period {budget_limit, monthly_spend} response to the monthly row", async () => {
qGet({ budget_limit: 5_000, monthly_spend: 1_000, budget_remaining: 4_000 });
render(<BudgetSection workspaceId={WS_ID} />);
await vi.waitFor(() => {
expect(screen.getByTestId("budget-fetch-error")).toBeTruthy();
expect(screen.getByTestId("budget-monthly-limit")!.textContent).toBe("$50.00");
});
expect(screen.queryByTestId("budget-exceeded-banner")).toBeNull();
expect(screen.getByTestId("budget-monthly-spend")!.textContent).toBe("$10.00");
});
});
});
@@ -0,0 +1,87 @@
// @vitest-environment jsdom
//
// Regression: project_canvas_runtime_dropdown_ssot_fix — a google-adk
// workspace's Config tab showed the wrong runtime ("LangGraph (default)"
// / first option) because a hardcoded frontend allowlist
// (SUPPORTED_RUNTIME_VALUES) dropped google-adk from the /templates-derived
// options even though the backend served it. A Save from that state would
// PATCH runtime to the wrong value and break the ADK agent.
//
// The fix: the dropdown is SSOT-driven — it trusts GET /templates (which the
// backend already gates to the manifest maintained set) and hides a runtime
// only when its row carries `displayable: false`. This pins: a google-adk
// workspace shows "google-adk" selected, and a displayable:false template is
// not offered.
import { describe, it, expect, vi, afterEach, beforeEach } from "vitest";
import { render, screen, cleanup, waitFor } from "@testing-library/react";
import React from "react";
afterEach(cleanup);
const apiGet = vi.fn();
const apiPatch = vi.fn();
const apiPut = vi.fn();
vi.mock("@/lib/api", () => ({
api: {
get: (path: string) => apiGet(path),
patch: (path: string, body: unknown) => apiPatch(path, body),
put: (path: string, body: unknown) => apiPut(path, body),
post: vi.fn(),
del: vi.fn(),
},
}));
vi.mock("@/store/canvas", () => ({
useCanvasStore: Object.assign(
(selector: (s: unknown) => unknown) => selector({ restartWorkspace: vi.fn(), updateNodeData: vi.fn() }),
{ getState: () => ({ restartWorkspace: vi.fn(), updateNodeData: vi.fn() }) },
),
}));
vi.mock("../AgentCardSection", () => ({
AgentCardSection: () => <div data-testid="agent-card-stub" />,
}));
import { ConfigTab } from "../ConfigTab";
function wireApi(templates: Array<{ id: string; name?: string; runtime?: string; models?: unknown[]; displayable?: boolean }>) {
apiGet.mockImplementation((path: string) => {
if (path === "/workspaces/ws-adk") return Promise.resolve({ runtime: "google-adk" });
if (path === "/workspaces/ws-adk/model") return Promise.resolve({ model: "vertex:gemini-2.5-pro" });
if (path === "/workspaces/ws-adk/files/config.yaml") return Promise.resolve({ content: "name: adk\nruntime: google-adk\n" });
if (path === "/templates") return Promise.resolve(templates);
return Promise.reject(new Error(`unmocked api.get: ${path}`));
});
}
beforeEach(() => {
apiGet.mockReset();
apiPatch.mockReset();
apiPut.mockReset();
});
describe("ConfigTab — google-adk runtime (SSOT dropdown)", () => {
it("shows google-adk selected in the runtime dropdown (#ssot-fix)", async () => {
wireApi([
{ id: "claude-code", name: "Claude Code", runtime: "claude-code", models: [] },
{ id: "google-adk", name: "Google ADK", runtime: "google-adk", models: [] },
]);
render(<ConfigTab workspaceId="ws-adk" />);
const select = await waitFor(() => screen.getByRole("combobox", { name: /runtime/i }));
expect((select as HTMLSelectElement).value).toBe("google-adk");
const opts = Array.from((select as HTMLSelectElement).options).map((o) => o.value);
expect(opts).toContain("google-adk");
});
it("hides a template flagged displayable:false", async () => {
wireApi([
{ id: "google-adk", name: "Google ADK", runtime: "google-adk", models: [] },
{ id: "legacy", name: "Legacy", runtime: "legacy", models: [], displayable: false },
]);
render(<ConfigTab workspaceId="ws-adk" />);
const select = await waitFor(() => screen.getByRole("combobox", { name: /runtime/i }));
const opts = Array.from((select as HTMLSelectElement).options).map((o) => o.value);
expect(opts).toContain("google-adk");
expect(opts).not.toContain("legacy");
});
});
@@ -0,0 +1,78 @@
// @vitest-environment jsdom
//
// internal#718 P3 (retire-list #5) — the billing-mode the Config tab shows /
// sends must reflect the DERIVED provider per the registry, not the hardcoded
// billingModeForProvider("" | "platform" → platform_managed else byok) rule.
// When the runtime is registry-backed, billingModeForSelectedProvider reads the
// registry-served billing_mode off the provider catalog entry. The hardcoded
// rule remains only as the fallback for non-registry runtimes / older backends.
import { describe, it, expect } from "vitest";
import { billingModeForSelectedProvider, billingModeForProvider } from "../ConfigTab";
import {
buildProviderCatalogFromRegistry,
type RegistryProvider,
type RegistryModel,
} from "../../ProviderModelSelector";
const REGISTRY_PROVIDERS: RegistryProvider[] = [
{ name: "anthropic-oauth", display_name: "Claude Code subscription", auth_env: ["CLAUDE_CODE_OAUTH_TOKEN"], billing_mode: "byok" },
{ name: "platform", display_name: "Platform", auth_env: ["ANTHROPIC_API_KEY"], billing_mode: "platform_managed" },
// DISCRIMINATING fixture (review #7790): a provider whose registry-served
// billing_mode DISAGREES with the hardcoded name-based rule. Its name is not
// "platform"/"" so billingModeForProvider() would call it "byok", yet the
// registry serves "platform_managed" (the federation-ready shape the SSOT is
// built for — a managed provider that isn't literally named "platform").
// billingModeForSelectedProvider MUST return the REGISTRY value here; the
// only way to get "platform_managed" out is to honor the catalog, so this
// case fails if the impl ever regresses to the hardcoded rule.
{ name: "managed-federated", display_name: "Managed (federated)", auth_env: [], billing_mode: "platform_managed" },
];
const REGISTRY_MODELS: RegistryModel[] = [
{ id: "sonnet", provider: "anthropic-oauth", billing_mode: "byok" },
{ id: "anthropic/claude-opus-4-7", provider: "platform", billing_mode: "platform_managed" },
// model bucketed under the disagreeing provider so the catalog builds an
// entry for it (buildProviderCatalogFromRegistry only emits a provider entry
// for providers that own at least one model).
{ id: "managed/some-model", provider: "managed-federated", billing_mode: "platform_managed" },
];
describe("billingModeForSelectedProvider (registry-driven)", () => {
const catalog = buildProviderCatalogFromRegistry(REGISTRY_PROVIDERS, REGISTRY_MODELS);
it("reads platform_managed from the registry for the platform provider", () => {
expect(billingModeForSelectedProvider("platform", catalog)).toBe("platform_managed");
});
it("reads byok from the registry for a BYOK provider", () => {
// anthropic-oauth derives to byok via the REGISTRY. (Note: the hardcoded
// rule would ALSO say byok for this non-'platform' name, so on its own this
// assertion does NOT prove the registry is authoritative — it agrees either
// way. The registry-WINS proof is the disagreement case below.)
expect(billingModeForSelectedProvider("anthropic-oauth", catalog)).toBe("byok");
});
it("lets the registry billing_mode WIN when it disagrees with the hardcoded rule", () => {
// 'managed-federated' is not '' / 'platform', so the legacy name-based rule
// classifies it byok — but the registry serves platform_managed. The
// registry is the SSOT, so billingModeForSelectedProvider must return
// platform_managed. This is the discriminating case: it FAILS if the impl
// regresses to billingModeForProvider (which would return byok here).
expect(billingModeForProvider("managed-federated")).toBe("byok"); // sanity: the rules genuinely disagree
expect(billingModeForSelectedProvider("managed-federated", catalog)).toBe("platform_managed");
});
it("falls back to the hardcoded rule when no registry catalog is supplied", () => {
// Non-registry runtime / older backend → catalog empty/undefined → the
// legacy mapping still applies ('' | 'platform' → platform_managed).
expect(billingModeForSelectedProvider("", undefined)).toBe("platform_managed");
expect(billingModeForSelectedProvider("platform", undefined)).toBe("platform_managed");
expect(billingModeForSelectedProvider("minimax", undefined)).toBe("byok");
});
it("falls back to the hardcoded rule when the provider is not in the registry catalog", () => {
// A provider string the registry catalog doesn't carry (stale saved
// value) → fall back to the legacy rule rather than guessing.
expect(billingModeForSelectedProvider("some-byo-vendor", catalog)).toBe("byok");
});
});
@@ -297,6 +297,25 @@ describe("DetailsTab — delete workflow", () => {
expect(mockSelectNode).toHaveBeenCalledWith(null);
});
// internal#734: checking "also erase saved data" adds &erase_data=true so the
// server prunes the data volume. Default (unchecked) must NOT send it.
it("checking erase-saved-data sends erase_data=true on delete", async () => {
mockApi.del.mockResolvedValue(undefined);
render(<DetailsTab workspaceId="ws-1" data={data()} />);
await flush();
fireEvent.click(screen.getByRole("button", { name: /delete workspace/i }));
await flush();
fireEvent.click(screen.getByRole("checkbox", { name: /erase saved data/i }));
const confirmBtn = Array.from(document.querySelectorAll("button")).find(
(b) => b.textContent === "Confirm Delete",
) as HTMLButtonElement;
fireEvent(confirmBtn, new MouseEvent("click", { bubbles: true }));
await flush();
expect(mockApi.del).toHaveBeenCalledWith("/workspaces/ws-1?confirm=true&erase_data=true", {
headers: { "X-Confirm-Name": "Test Workspace" },
});
});
it("cancelling delete returns to view mode", async () => {
mockApi.del.mockResolvedValue(undefined);
render(<DetailsTab workspaceId="ws-1" data={data()} />);
+1
View File
@@ -5,6 +5,7 @@
const RUNTIME_NAMES: Record<string, string> = {
"claude-code": "Claude Code",
codex: "Codex",
"google-adk": "Google ADK",
hermes: "Hermes",
openclaw: "OpenClaw",
kimi: "Kimi",
+3
View File
@@ -368,6 +368,9 @@ export interface WorkspaceCompute {
width?: number;
height?: number;
};
// internal#734: per-workspace durable-data choice. "persist" | "ephemeral" |
// undefined (auto). Controls whether the data volume survives recreate.
data_persistence?: string;
}
let socket: ReconnectingSocket | null = null;
+11
View File
@@ -1,5 +1,16 @@
# Running a Gemini CLI Workspace on Molecule AI
> **⚠️ Accuracy correction (2026-05-29):** this page is **aspirational, not
> shipped.** There is **no `gemini-cli` runtime** in `manifest.json` or the
> provisioner's `knownRuntimes`, and the "PR #379" cited below is unrelated (a
> CI-workflow-cleanup PR, not a gemini-cli adapter). Do not follow this as-is.
>
> **For Gemini on Molecule, use the real `google-adk` runtime instead** — see
> [`google-adk-runtime.md`](./google-adk-runtime.md) (ADK engine + Gemini on
> Vertex AI/AI Studio), implemented in PR
> [`molecule-ai-workspace-template-google-adk#1`](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-template-google-adk) per RFC `internal#730`.
> This gemini-cli page is retained only until it's either implemented for real or removed.
Molecule AI now ships a `gemini-cli` runtime adapter alongside the existing `claude-code` adapter. This tutorial walks you from zero to a running Gemini agent workspace in under five minutes.
## What you'll need
+52 -57
View File
@@ -1,74 +1,69 @@
# Running a Google ADK Workspace on Molecule AI
Google's Agent Development Kit (ADK) is now a first-class runtime on Molecule AI. This tutorial walks you from zero to a running ADK agent workspace — one that persists per-conversation session state and sits alongside your Claude Code and Gemini CLI workers in the same A2A network.
> **Status (2026-05-29):** the `google-adk` runtime is **landing**, not yet on
> `main`. It's implemented in the template repo
> [`molecule-ai-workspace-template-google-adk`](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-template-google-adk)
> (PR **#1**) with platform registration in molecule-core PR **#2003** and the
> validator allowlist in molecule-ci PR **#26**. Design + approval: RFC
> [`internal#730`](https://git.moleculesai.app/molecule-ai/internal/issues/730).
> Remove this banner once those PRs merge.
>
> **Doc-accuracy note:** a prior version of this page claimed ADK was "already
> first-class" and cited "PR #550" — that PR is unrelated (a MemoryTab test
> suite). No `google-adk` adapter existed at that time. This rewrite reflects
> the real implementation.
## What you'll need
Google's Agent Development Kit (ADK) runs as a Molecule AI workspace runtime:
ADK is the **agent engine** (`LlmAgent` + `Runner`), and the workspace
participates in Molecule's A2A org like any other runtime.
- A Molecule AI account with at least one provisioned tenant
- A `GOOGLE_API_KEY` from [aistudio.google.com](https://aistudio.google.com) (or Vertex AI credentials — see below)
- `curl` + `jq`
## How it actually works
## Setup
- **ADK = engine only.** The adapter builds an ADK `LlmAgent` from the
workspace config (model + system prompt + tools) and drives its `Runner`.
It installs `google-adk[mcp]==2.1.0` and **never** the `[a2a]` extra — ADK's
a2a layer pins `a2a-sdk<0.4`, which is incompatible with the platform's
`a2a-sdk>=1.0`. (Verified: `google-adk[mcp]==2.1.0` + `a2a-sdk 1.0.3` coexist.)
- **A2A** is provided by the platform's a2a-1.x server; a Molecule-authored
executor bridges ADK's `Runner` event stream onto it, one ADK session per
A2A `context_id`.
- **Tools** reach the agent via ADK's native `McpToolset` pointed at the
workspace's `a2a_mcp_server` — the same MCP surface the CLI runtimes use
(`delegate_task`, `commit_memory`, `list_peers`, …). No LangChain.
## Auth — Vertex AI via ADC (keyless), or an AI Studio key
The runtime supports both google-genai auth paths:
- **Vertex AI + Application Default Credentials (recommended; required if your
org disallows API keys).** Set `model: vertex:gemini-2.5-pro` and provide
`GOOGLE_CLOUD_PROJECT`; the adapter sets `GOOGLE_GENAI_USE_VERTEXAI=1` and
google-genai authenticates via ADC — no API key. (Locally:
`gcloud auth application-default login`.)
- **AI Studio API key** (where your org permits API keys): set
`model: google_genai:gemini-2.5-pro` and `GOOGLE_API_KEY`.
## Create a workspace
```bash
# 1. Store your Google API key as a global secret
curl -s -X PUT http://localhost:8080/settings/secrets \
-H "Content-Type: application/json" \
-d '{"key":"GOOGLE_API_KEY","value":"YOUR-AI-STUDIO-KEY"}' | jq .
# 2. Create a google-adk workspace
WS=$(curl -s -X POST http://localhost:8080/workspaces \
# Vertex AI + ADC (keyless)
curl -s -X POST http://localhost:8080/workspaces \
-H "Content-Type: application/json" \
-d '{
"name": "adk-agent",
"role": "Google ADK inference worker",
"runtime": "google-adk",
"model": "google:gemini-2.0-flash"
}' | jq -r '.id')
echo "Workspace: $WS"
# 3. Wait for ready (~30s)
until curl -s http://localhost:8080/workspaces/$WS | jq -r '.status' | grep -q ready; do
echo "Waiting..."; sleep 5
done
# 4. Send your first task
curl -s -X POST http://localhost:8080/workspaces/$WS/a2a \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":"1","method":"message/send",
"params":{"message":{"role":"user","parts":[{"kind":"text",
"text":"Summarise the ADK architecture in 3 bullet points."}]}}}' \
| jq '.result.parts[0].text'
# 5. Multi-turn — session state is preserved across calls
curl -s -X POST http://localhost:8080/workspaces/$WS/a2a \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":"2","method":"message/send",
"params":{"message":{"role":"user","parts":[{"kind":"text",
"text":"Now give me a one-line TL;DR of what you just said."}]}}}' \
| jq '.result.parts[0].text'
# 6. Vertex AI alternative — set these instead of GOOGLE_API_KEY
# curl -X PUT .../secrets -d '{"key":"GOOGLE_GENAI_USE_VERTEXAI","value":"1"}'
# curl -X PUT .../secrets -d '{"key":"GOOGLE_CLOUD_PROJECT","value":"my-project"}'
# curl -X PUT .../secrets -d '{"key":"GOOGLE_CLOUD_LOCATION","value":"us-central1"}'
"model": "vertex:gemini-2.5-pro",
"runtime_config": {"required_env": ["GOOGLE_CLOUD_PROJECT"]}
}'
```
## Expected output
After step 4, ADK streams the Gemini response through its event bus, filters for `is_final_response()` events, and returns the agent's reply as a standard A2A text part. Step 5 should reference the prior answer — the adapter ties each A2A `context_id` to an `InMemorySessionService` session, so conversation state is isolated per task context and survives across calls within the same session.
## How it works
The `google-adk` adapter wraps Google ADK's runner/session model behind the same `AgentExecutor` interface used by every other Molecule AI runtime. On each turn, `GoogleADKA2AExecutor` calls `runner.run_async()` with the incoming message wrapped in a `google.genai.types.Content` object, then drains the event stream until it collects a final-response event. The `google:` model prefix is stripped before being passed to ADK — so `google:gemini-2.0-flash` in your workspace config becomes `gemini-2.0-flash` in the ADK `LlmAgent`. Error class names are sanitized before leaving the executor; raw Google SDK stack traces never reach the A2A caller.
## Mixed-runtime teams
ADK workspaces participate in the same A2A network as Claude Code, Gemini CLI, Hermes, and LangGraph workers. An orchestrator can delegate long-context summarisation to a `google-adk` worker (Gemini 1.5 Pro's 1M token window) while routing tool-use tasks to a `claude-code` worker — with no provider-specific code in the orchestrator itself. Add an ADK peer with `POST /workspaces`, set `GOOGLE_API_KEY`, and it's available for `delegate_task` immediately.
Send it a task via the A2A proxy (`POST /workspaces/:id/a2a`, JSON-RPC
`message/send`) and it replies through the ADK `Runner`. Verified end-to-end:
a Gemini 2.5 round-trip on Vertex via ADC returns through the built image.
## Related
- PR #550: [feat(adapters): add google-adk runtime adapter](https://git.moleculesai.app/molecule-ai/molecule-core/pull/550)
- Template + adapter: [`molecule-ai-workspace-template-google-adk`](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-template-google-adk) (PR #1)
- Platform registration: molecule-core PR #2003 · validator: molecule-ci PR #26
- Design/approval: RFC [`internal#730`](https://git.moleculesai.app/molecule-ai/internal/issues/730)
- [Google ADK (adk-python)](https://github.com/google/adk-python)
- [Gemini CLI runtime tutorial](./gemini-cli-runtime.md)
- [Platform API reference](../api-reference.md)
+1
View File
@@ -29,6 +29,7 @@
{"name": "hermes", "repo": "molecule-ai/molecule-ai-workspace-template-hermes", "ref": "main"},
{"name": "openclaw", "repo": "molecule-ai/molecule-ai-workspace-template-openclaw", "ref": "main"},
{"name": "codex", "repo": "molecule-ai/molecule-ai-workspace-template-codex", "ref": "main"},
{"name": "google-adk", "repo": "molecule-ai/molecule-ai-workspace-template-google-adk", "ref": "main"},
{"name": "seo-agent", "repo": "molecule-ai/molecule-ai-workspace-template-seo-agent", "ref": "main"}
],
"org_templates": [
+30 -7
View File
@@ -1,12 +1,13 @@
#!/usr/bin/env bash
# E2E test: A2A round-trip parity across all four runtimes.
# E2E test: A2A round-trip parity across all five runtimes.
#
# Validates that for each of {claude-code, hermes, codex, openclaw}:
# Validates that for each of {claude-code, hermes, codex, openclaw, google-adk}:
# 1. A workspace can be provisioned + brought online
# 2. The adapter responds to A2A message/send
# 3. The reply contains expected content (echo of the prompt)
# 4. A SECOND message preserves session state where the runtime
# supports it (currently: hermes via plugin path)
# supports it (currently: hermes via plugin path; google-adk via
# ADK InMemorySessionService keyed on A2A context_id)
#
# Targets a SaaS tenant subdomain. Provisions workspaces in the calling
# tenant, runs the round-trip, deletes them on success.
@@ -16,6 +17,10 @@
# (e.g. https://demo-tenant.staging.moleculesai.app)
# - $OPENROUTER_API_KEY (or $HERMES_API_KEY) for non-claude runtimes
# - $OPENAI_API_KEY for claude-code peer
# - $GOOGLE_API_KEY (AI Studio) for google-adk — the org disallows API
# keys in PROD (Vertex+ADC there), but CI auths Gemini with an
# AI-Studio key (config model google_genai:gemini-2.5-pro). Vertex
# stays supported; this is the keyed CI path only.
# - SaaS edge requires Origin header — see auto-memory
# reference_saas_waf_origin_header.md
#
@@ -24,12 +29,13 @@
# ./scripts/test-all-runtimes-a2a-e2e.sh
#
# Skip individual runtimes:
# SKIP_HERMES=1 SKIP_OPENCLAW=1 ./scripts/test-all-runtimes-a2a-e2e.sh
# SKIP_HERMES=1 SKIP_OPENCLAW=1 SKIP_GOOGLE_ADK=1 ./scripts/test-all-runtimes-a2a-e2e.sh
set -euo pipefail
PLATFORM="${PLATFORM:-${1:-http://localhost:8080}}"
HERMES_PROVIDER_KEY="${OPENROUTER_API_KEY:-${HERMES_API_KEY:-}}"
PEER_OPENAI_KEY="${OPENAI_API_KEY:-}"
GOOGLE_ADK_KEY="${GOOGLE_API_KEY:-}"
# SaaS auth chain — TENANT_ADMIN_TOKEN + TENANT_ORG_ID required when
# hitting *.moleculesai.app (per-tenant ADMIN_TOKEN, NOT
# CP_ADMIN_API_TOKEN). Optional for localhost.
@@ -48,6 +54,10 @@ if [ -z "$HERMES_PROVIDER_KEY" ] && [ -z "${SKIP_HERMES:-}${SKIP_CODEX:-}${SKIP_
echo "FAIL: set OPENROUTER_API_KEY or HERMES_API_KEY for non-claude runtimes"
exit 2
fi
if [ -z "$GOOGLE_ADK_KEY" ] && [ -z "${SKIP_GOOGLE_ADK:-}" ]; then
echo "FAIL: set GOOGLE_API_KEY (AI Studio) for google-adk, or SKIP_GOOGLE_ADK=1"
exit 2
fi
PASS=0
FAIL=0
@@ -143,7 +153,7 @@ echo "=========================================="
echo ""
# -------------------------------------------------------
# 1. Provision the four runtimes (skip via SKIP_* flags)
# 1. Provision the five runtimes (skip via SKIP_* flags)
# -------------------------------------------------------
echo "--- 1. Provision workspaces ---"
if [ -z "${SKIP_CLAUDE_CODE:-}" ]; then
@@ -162,6 +172,10 @@ if [ -z "${SKIP_OPENCLAW:-}" ]; then
WS_IDS[openclaw]=$(provision "ParityOpenClaw" "openclaw" "openclaw peer")
echo " openclaw: ${WS_IDS[openclaw]}"
fi
if [ -z "${SKIP_GOOGLE_ADK:-}" ]; then
WS_IDS[google-adk]=$(provision "ParityGoogleADK" "google-adk" "google-adk peer")
echo " google-adk: ${WS_IDS[google-adk]}"
fi
# -------------------------------------------------------
# 2. Set provider keys
@@ -177,6 +191,12 @@ if [ -n "${WS_IDS[claude-code]:-}" ] && [ -n "$PEER_OPENAI_KEY" ]; then
set_secret "${WS_IDS[claude-code]}" "OPENAI_API_KEY" "$PEER_OPENAI_KEY"
echo " claude-code: OPENAI_API_KEY set"
fi
if [ -n "${WS_IDS[google-adk]:-}" ] && [ -n "$GOOGLE_ADK_KEY" ]; then
# AI-Studio path: the adapter reads GOOGLE_API_KEY natively when the
# config model is google_genai:gemini-2.5-pro (see _routing.resolve_model).
set_secret "${WS_IDS[google-adk]}" "GOOGLE_API_KEY" "$GOOGLE_ADK_KEY"
echo " google-adk: GOOGLE_API_KEY set"
fi
# -------------------------------------------------------
# 3. Wait for online
@@ -188,6 +208,9 @@ for runtime in "${!WS_IDS[@]}"; do
[ -z "$id" ] && continue
max=60
[ "$runtime" = "hermes" ] && max=120
# google-adk's first cold boot pulls a large fresh ADK image — give it
# a hermes-class window so a slow first pull doesn't read as "failed".
[ "$runtime" = "google-adk" ] && max=180
if wait_online "$id" "$runtime" "$max"; then
check "$runtime online" "ok" "ok"
else
@@ -200,7 +223,7 @@ done
# -------------------------------------------------------
echo ""
echo "--- 4. A2A round-trip (first message) ---"
for runtime in claude-code hermes codex openclaw; do
for runtime in claude-code hermes codex openclaw google-adk; do
id="${WS_IDS[$runtime]:-}"
[ -z "$id" ] && continue
reply=$(a2a_send "$id" "Reply with just the word OK so we know you got this.")
@@ -213,7 +236,7 @@ done
# -------------------------------------------------------
echo ""
echo "--- 5. Session continuity (second message recalls first) ---"
for runtime in claude-code hermes codex openclaw; do
for runtime in claude-code hermes codex openclaw google-adk; do
id="${WS_IDS[$runtime]:-}"
[ -z "$id" ] && continue
# Set up: tell the agent a name.
+229
View File
@@ -0,0 +1,229 @@
#!/usr/bin/env bash
# Real-completion + per-provider liveness + byok-routing assertion helpers
# for the staging full-SaaS E2E (tests/e2e/test_staging_full_saas.sh).
#
# WHY THIS LIB EXISTS (molecule-core#1995 / #1994 follow-on):
# The A2A e2e historically asserted only response SHAPE — e.g.
# test_a2a_e2e.sh:`check "SEO response has text" '"kind":"text"'`. A fully
# BROKEN agent returns its error AS a text part:
# {"kind":"text","text":"Agent error (Exception) — see workspace logs..."}
# which STILL matches `"kind":"text"` → the shape check PASSES on a broken
# agent. That is exactly why the 2026-05-2x drained-key / byok-misroute
# failures (agents-team PM + reno marketing erroring on every LLM call)
# sailed through CI. "Channel returns text shape" != "agent actually
# completed an LLM round-trip".
#
# These helpers add three load-bearing gates ON TOP of (never replacing) the
# existing shape + PONG checks:
# 1. a2a_assert_real_completion — deterministic known-answer round-trip
# (CONTAINS the expected token AND NOT an error-as-text payload).
# 2. provider_liveness_matrix — per-offered-provider cheap completion
# probe, providers sourced from the providers.yaml SSOT runtimes block.
# 3. assert_byok_not_platform_proxy — #1994 regression guard: a
# byok-resolving workspace must NOT resolve to platform_managed.
#
# Conventions: reuses the host script's fail()/ok()/log() + tenant_call().
# Source this AFTER those are defined. BASH 4+.
# Error-as-text trap markers. If the agent's text part contains ANY of
# these, the "round-trip" did not really complete — the agent surfaced an
# error AS text. This is the negative assertion that makes a broken agent
# FAIL instead of slipping through the shape check.
#
# Kept as an array (not a single regex) so a new failure signature is a
# one-line append + the failure message can name which marker matched.
A2A_ERROR_AS_TEXT_MARKERS=(
"Agent error"
"Exception"
"error result"
"MISSING_BYOK_CREDENTIAL"
)
# a2a_completion_error_marker <agent_text>
# Echoes the first error-as-text marker found in <agent_text> (case-
# insensitive), or nothing if clean. Exit 0 if a marker matched, 1 if not.
# Pure string scan — no LLM, no network — so it is deterministic and is the
# unit under the fail-direction proof in test_completion_assert_unit.sh.
a2a_completion_error_marker() {
local text="$1"
local upper marker
upper=$(printf '%s' "$text" | tr '[:lower:]' '[:upper:]')
for marker in "${A2A_ERROR_AS_TEXT_MARKERS[@]}"; do
if printf '%s' "$upper" | grep -qF -- "$(printf '%s' "$marker" | tr '[:lower:]' '[:upper:]')"; then
printf '%s' "$marker"
return 0
fi
done
return 1
}
# a2a_assert_real_completion <agent_text> <expected_token> <context_label>
# The CORE gate. Asserts the agent text:
# (a) does NOT contain any error-as-text marker (broken-agent trap), AND
# (b) CONTAINS <expected_token> (case-insensitive) — proving a real LLM
# round-trip produced the deterministic known answer.
# Calls fail() (which exits) on either violation. This MUST fail on an
# error-as-text payload — that is the property test_completion_assert_unit.sh
# pins.
a2a_assert_real_completion() {
local text="$1"
local expected="$2"
local ctx="${3:-A2A}"
if [ -z "$text" ]; then
fail "$ctx — real-completion gate: agent returned EMPTY text (no round-trip)."
fi
local hit
if hit=$(a2a_completion_error_marker "$text"); then
fail "$ctx — real-completion gate: agent returned an ERROR-AS-TEXT payload (matched '$hit'). A broken agent that surfaces its error as a text part is NOT a completed round-trip. This is the trap the shape-only check missed (#1994). Raw: ${text:0:200}"
fi
# Known-answer: real LLM round-trip yields the deterministic token. A
# prompt-echo / truncated-context / wrong-auth pipeline won't.
if ! printf '%s' "$text" | tr '[:lower:]' '[:upper:]' | grep -qF -- "$(printf '%s' "$expected" | tr '[:lower:]' '[:upper:]')"; then
fail "$ctx — real-completion gate: reply did NOT contain expected known-answer token '$expected'. The channel returned a text shape but no real completion. Raw: ${text:0:200}"
fi
ok "$ctx — real completion verified (contains '$expected', no error-as-text). Reply: \"${text:0:80}\""
}
# offered_platform_models_for_runtime <runtime>
# Emits, one per line, the platform-servable model ids the providers.yaml
# SSOT (runtimes.<runtime>.providers[name=platform].models) declares for
# <runtime>. This is the SSOT-driven offered/platform-servable matrix — NOT
# a hardcoded provider list — so a provider added/removed in providers.yaml
# automatically changes the matrix this probe exercises.
#
# Reads the embedded copy at workspace-server/internal/providers/providers.yaml
# (the same file go:embed compiles into the binary). Requires python3 +
# PyYAML (already a test-harness dep). On parse failure, emits nothing and
# returns 1 so the caller can fail loud rather than silently skip.
offered_platform_models_for_runtime() {
local runtime="$1"
local yaml_path="${PROVIDERS_YAML_PATH:-}"
if [ -z "$yaml_path" ]; then
# This lib lives at tests/e2e/lib/ -> repo root is three dirs up
# (lib -> e2e -> tests -> repo-root).
yaml_path="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)/workspace-server/internal/providers/providers.yaml"
fi
if [ ! -f "$yaml_path" ]; then
log " [provider-matrix] providers.yaml SSOT not found at $yaml_path"
return 1
fi
RUNTIME_REF="$runtime" python3 - "$yaml_path" <<'PY'
import os, sys
try:
import yaml
except Exception as e: # PyYAML missing — fail loud, do not silently skip.
sys.stderr.write(f"PyYAML required for provider-matrix SSOT read: {e}\n")
sys.exit(2)
rt = os.environ["RUNTIME_REF"]
with open(sys.argv[1]) as f:
doc = yaml.safe_load(f)
native = (doc.get("runtimes") or {}).get(rt) or {}
for pref in native.get("providers", []) or []:
if pref.get("name") == "platform":
for m in pref.get("models", []) or []:
print(m)
PY
}
# provider_liveness_matrix <runtime> <probe_fn>
# For each platform-servable model the SSOT lists for <runtime>, calls
# <probe_fn> <model_id> which must echo the agent text (or empty) and return
# 0 on a non-error completion, non-zero otherwise. Logs a per-model pass/fail
# matrix. Returns 0 only if EVERY probed model produced a non-error
# completion; non-zero (and a recorded matrix) otherwise.
#
# Purpose: exercise each offered provider's AUTH + ROUTING path so a drained
# key / wrong base-URL / byok-misroute fails the gate (the #1994 class). The
# probe_fn is expected to use minimal max_tokens.
#
# This helper does the SSOT read + matrix bookkeeping; the host script
# supplies probe_fn (it owns workspace ids + tenant_call wiring).
provider_liveness_matrix() {
local runtime="$1"
local probe_fn="$2"
local models model rc total=0 passed=0
local -a results=()
models=$(offered_platform_models_for_runtime "$runtime") || {
fail "provider-liveness: could not read offered-provider matrix from providers.yaml SSOT for runtime=$runtime"
}
if [ -z "$models" ]; then
log " [provider-matrix] runtime=$runtime offers no platform-servable models in the SSOT — nothing to probe (not a failure)."
return 0
fi
log " [provider-matrix] SSOT offered platform models for $runtime:"
while IFS= read -r model; do
[ -z "$model" ] && continue
log " - $model"
done <<<"$models"
while IFS= read -r model; do
[ -z "$model" ] && continue
total=$((total + 1))
set +e
"$probe_fn" "$model"
rc=$?
set -e
if [ "$rc" = "0" ]; then
passed=$((passed + 1))
results+=("PASS $model")
elif [ "$rc" = "75" ]; then
# 75 (EX_TEMPFAIL convention) = probe skipped (key/runtime not
# available in this lane). Not counted toward pass/fail — logged.
total=$((total - 1))
results+=("SKIP $model (probe unavailable in this lane)")
else
results+=("FAIL $model")
fi
done <<<"$models"
log " [provider-matrix] result matrix (runtime=$runtime):"
local line
for line in "${results[@]}"; do
log " $line"
done
log " [provider-matrix] $passed/$total probed providers completed without error"
if [ "$passed" != "$total" ]; then
return 1
fi
return 0
}
# assert_byok_not_platform_proxy <billing_mode_json> <context_label>
# #1994 regression guard. Given the JSON body from
# GET /admin/workspaces/:id/llm-billing-mode (same derived resolver the
# provision-time strip gate uses), asserts the workspace resolves to BYOK
# and NOT platform_managed. A regression of #1994 (byok workspace baked to
# platform_managed → routed through the platform proxy → platform LLM key
# drained) flips resolved_mode to "platform_managed" and trips this gate.
# Calls fail() (exits) on violation.
assert_byok_not_platform_proxy() {
local body="$1"
local ctx="${2:-byok-guard}"
local mode prov
mode=$(printf '%s' "$body" | python3 -c "import json,sys
try: print(json.load(sys.stdin).get('resolved_mode',''))
except Exception: print('')" 2>/dev/null || echo "")
prov=$(printf '%s' "$body" | python3 -c "import json,sys
try:
d=json.load(sys.stdin); v=d.get('provider_selection')
print(v if v is not None else '')
except Exception: print('')" 2>/dev/null || echo "")
if [ -z "$mode" ]; then
fail "$ctx — byok-routing guard: could not read resolved_mode from billing-mode response. Raw: ${body:0:200}"
fi
if [ "$mode" = "platform_managed" ]; then
fail "$ctx — byok-routing guard TRIPPED (#1994 regression): a byok-configured workspace resolved to 'platform_managed' (provider_selection=$prov) → it would route through the platform proxy and drain the platform LLM key. Expected resolved_mode=byok. Raw: ${body:0:200}"
fi
if [ "$mode" != "byok" ]; then
fail "$ctx — byok-routing guard: unexpected resolved_mode='$mode' (expected 'byok'). provider_selection=$prov. Raw: ${body:0:200}"
fi
ok "$ctx — byok-routing guard: workspace resolves byok (provider_selection=$prov), NOT platform-proxy. #1994 stays fixed."
}
+35
View File
@@ -8,6 +8,34 @@ TIMEOUT="${A2A_TIMEOUT:-120}" # seconds per A2A call (override via A2A_TIMEOUT
# shellcheck source=_lib.sh
source "$(dirname "$0")/_lib.sh"
# molecule-core#1995 (#1994 follow-on): real-completion assertion helpers.
# Adds a NEGATIVE error-as-text check on top of the shape checks below, so a
# broken agent that returns its error AS a text part
# ({"kind":"text","text":"Agent error (Exception) ..."}) — which STILL
# matches the shape check `"kind":"text"` — now FAILS instead of passing.
# shellcheck source=lib/completion_assert.sh
source "$(dirname "$0")/lib/completion_assert.sh"
# check_no_error_as_text <desc> <agent_text>
# Additive negative gate: PASS only if the agent text carries NO
# error-as-text marker (Agent error / Exception / error result /
# MISSING_BYOK_CREDENTIAL). Uses the same scanner as the staging
# real-completion gate so the trap is closed consistently across lanes.
check_no_error_as_text() {
local desc="$1"
local text="$2"
local hit
if hit=$(a2a_completion_error_marker "$text"); then
echo "FAIL: $desc"
echo " agent returned an error-AS-text payload (matched '$hit') — a broken"
echo " agent that surfaces its error as a text part is NOT a real reply."
echo " got: $(echo "$text" | head -3)"
FAIL=$((FAIL + 1))
else
echo "PASS: $desc"
PASS=$((PASS + 1))
fi
}
check() {
local desc="$1"
@@ -81,6 +109,8 @@ check "JSON-RPC response has result" '"result"' "$R"
check "Response has agent role" '"role":"agent"' "$R"
check "Response has text part" '"kind":"text"' "$R"
TEXT=$(echo "$R" | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['result']['parts'][0]['text'][:200])" 2>/dev/null || echo "PARSE_ERROR")
# Negative gate (#1994): the text part must not BE an error.
check_no_error_as_text "Echo reply is not an error-as-text payload" "$TEXT"
echo " Agent said: $TEXT"
echo ""
@@ -92,6 +122,11 @@ R=$(a2a_send "$SEO_ID" "What SEO skills do you have?")
check "SEO agent responds" '"result"' "$R"
check "SEO response has text" '"kind":"text"' "$R"
TEXT=$(echo "$R" | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['result']['parts'][0]['text'][:200])" 2>/dev/null || echo "PARSE_ERROR")
# Negative gate (#1994): a broken SEO agent that returns "Agent error
# (Exception) ..." AS text still matches the `"kind":"text"` shape check
# above — THAT is the gap that let drained-key/byok-misroute failures pass
# CI. This makes that case FAIL.
check_no_error_as_text "SEO reply is not an error-as-text payload" "$TEXT"
echo " SEO Agent said: $TEXT"
echo ""
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env bash
# Fail-direction / load-bearing proof for lib/completion_assert.sh.
#
# This is the watch-it-FAIL counterpart the dev-SOP Phase 3 requires: it
# proves the new real-completion + byok gates actually CATCH a broken agent,
# not just pass on a good one. It runs entirely offline (no LLM, no network,
# no provisioning) — pure assertion logic — so it can run on every PR in the
# fast lane (e2e-api.yml unit-shell step) and locally via `bash`.
#
# The decisive case is `error-as-text payload MUST FAIL`: that is the exact
# trap (#1994) the historical shape-only check missed. If a refactor weakens
# a2a_assert_real_completion to a substring/shape check, THIS test goes red.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
PASS=0
FAIL=0
# Minimal stand-ins for the host script's helpers. fail() must NOT exit the
# whole harness here — we want to assert that it WAS called. We trap it by
# running the assertion in a subshell and checking the subshell's exit code:
# the real fail() exits 1, ok() exits 0 implicitly.
log() { echo "[unit] $*"; }
ok() { echo "[unit] OK: $*"; }
fail() { echo "[unit] FAIL-CALLED: $*" >&2; exit 1; }
# shellcheck source=lib/completion_assert.sh
source "$HERE/lib/completion_assert.sh"
expect_pass() {
local desc="$1"; shift
if ( "$@" ) >/dev/null 2>&1; then
echo "PASS: $desc (assertion accepted, as expected)"
PASS=$((PASS + 1))
else
echo "FAIL: $desc — expected the assertion to ACCEPT, but it rejected"
FAIL=$((FAIL + 1))
fi
}
expect_fail() {
local desc="$1"; shift
if ( "$@" ) >/dev/null 2>&1; then
echo "FAIL: $desc — expected the assertion to REJECT, but it accepted (gate NOT load-bearing!)"
FAIL=$((FAIL + 1))
else
echo "PASS: $desc (assertion rejected, as expected)"
PASS=$((PASS + 1))
fi
}
echo "=== completion_assert.sh fail-direction proof ==="
# ---- a2a_assert_real_completion ----
# Good: real known-answer reply passes.
expect_pass "real PINEAPPLE reply passes" \
a2a_assert_real_completion "PINEAPPLE" "PINEAPPLE" "unit"
expect_pass "case-insensitive known answer passes" \
a2a_assert_real_completion "pineapple" "PINEAPPLE" "unit"
expect_pass "known answer with minor wrapping passes" \
a2a_assert_real_completion "Sure: PINEAPPLE" "PINEAPPLE" "unit"
# DECISIVE: the error-as-text trap. Each MUST fail — these are the payloads a
# broken agent returns that the old shape-only `"kind":"text"` check passed.
expect_fail "Agent error as text payload MUST fail" \
a2a_assert_real_completion "Agent error (Exception) — see workspace logs for details." "PINEAPPLE" "unit"
expect_fail "bare Exception as text MUST fail" \
a2a_assert_real_completion "Traceback ... Exception: boom" "PINEAPPLE" "unit"
expect_fail "error result as text MUST fail" \
a2a_assert_real_completion "tool returned error result" "PINEAPPLE" "unit"
expect_fail "MISSING_BYOK_CREDENTIAL as text MUST fail" \
a2a_assert_real_completion "MISSING_BYOK_CREDENTIAL: set your own key" "PINEAPPLE" "unit"
# Error-as-text that ALSO happens to contain the token still fails (error
# marker takes precedence — a real completion never carries these markers).
expect_fail "error-as-text containing the token still fails" \
a2a_assert_real_completion "Agent error: could not produce PINEAPPLE" "PINEAPPLE" "unit"
# Empty text fails.
expect_fail "empty text fails" \
a2a_assert_real_completion "" "PINEAPPLE" "unit"
# Wrong/echoed content (no token, no error) fails — shape-OK but not a real
# completion.
expect_fail "wrong content without token fails" \
a2a_assert_real_completion "Reply with exactly the word PINEAPPLE and nothing else." "BANANA" "unit"
# ---- assert_byok_not_platform_proxy (#1994 guard) ----
expect_pass "byok resolution passes the guard" \
assert_byok_not_platform_proxy '{"resolved_mode":"byok","provider_selection":"minimax","source":"derived_provider"}' "unit"
# DECISIVE: a platform_managed resolution on a byok workspace = the #1994
# regression. MUST fail.
expect_fail "platform_managed resolution trips the #1994 guard" \
assert_byok_not_platform_proxy '{"resolved_mode":"platform_managed","provider_selection":"platform","source":"derived_provider"}' "unit"
expect_fail "missing resolved_mode trips the guard" \
assert_byok_not_platform_proxy '{"provider_selection":"x"}' "unit"
expect_fail "disabled mode trips the guard (not byok)" \
assert_byok_not_platform_proxy '{"resolved_mode":"disabled"}' "unit"
# ---- a2a_completion_error_marker (the scanner under the gate) ----
if hit=$(a2a_completion_error_marker "all good PINEAPPLE"); then
echo "FAIL: clean text wrongly flagged as error marker ($hit)"; FAIL=$((FAIL + 1))
else
echo "PASS: clean text has no error marker"; PASS=$((PASS + 1))
fi
if hit=$(a2a_completion_error_marker "An Exception occurred"); then
echo "PASS: error marker detected ($hit)"; PASS=$((PASS + 1))
else
echo "FAIL: error marker NOT detected in 'An Exception occurred'"; FAIL=$((FAIL + 1))
fi
echo ""
echo "=== Results: $PASS passed, $FAIL failed ==="
[ "$FAIL" -eq 0 ]
+182
View File
@@ -99,6 +99,12 @@ source "$(dirname "$0")/lib/model_slug.sh"
# shellcheck disable=SC1091
# shellcheck source=lib/aws_leak_check.sh
source "$(dirname "$0")/lib/aws_leak_check.sh"
# shellcheck disable=SC1091
# shellcheck source=lib/completion_assert.sh
# molecule-core#1995 (#1994 follow-on): real-completion + per-provider
# liveness + byok-routing assertion helpers. Adds gates that FAIL on an
# error-as-text payload (the trap the shape-only A2A checks missed).
source "$(dirname "$0")/lib/completion_assert.sh"
CURL_COMMON=(-sS --fail-with-body --max-time 30)
E2E_TMP_FILES=()
@@ -867,6 +873,182 @@ fi
ok "A2A parent round-trip succeeded: \"${AGENT_TEXT:0:80}\""
# ─── 8b. Real-completion known-answer round-trip (CORE GATE, #1994) ────
# The existing PONG check + generic error grep above already do a lot, but
# this stanza is the canonical real-completion gate the #1994 follow-on
# adds: a DETERMINISTIC known-answer prompt asserted via
# a2a_assert_real_completion, which FAILS on an error-as-text payload
# ({"kind":"text","text":"Agent error (Exception) ..."}). That payload
# matches the historical shape-only check `"kind":"text"` and so passed CI
# on a fully broken agent (drained-key / byok-misroute, 2026-05-2x). This
# gate makes that case RED. Reuses the same cold-start retry-on-transient
# (502/503/504) loop the PONG probe uses — retry-once-on-network, never on
# agent-error. Single round-trip → the one place we spend a non-trivial
# token budget (default backend MiniMax — cheap token plan).
KA_PAYLOAD=$(python3 -c "
import json, uuid
print(json.dumps({
'jsonrpc': '2.0',
'method': 'message/send',
'id': 'e2e-known-answer-1',
'params': {
'message': {
'role': 'user',
'messageId': f'e2e-{uuid.uuid4().hex[:8]}',
'parts': [{'kind': 'text', 'text': 'Reply with exactly the word PINEAPPLE and nothing else.'}]
}
}
}))
")
KA_TMP=$(mktemp -t known_answer_a2a.XXXXXX)
KA_RESP=""
for KA_ATTEMPT in $(seq 1 6); do
: >"$KA_TMP"
set +e
KA_CODE=$(tenant_call POST "/workspaces/$PARENT_ID/a2a" \
--max-time 90 \
-H "Content-Type: application/json" \
-d "$KA_PAYLOAD" \
-o "$KA_TMP" \
-w '%{http_code}' \
2>/dev/null)
KA_RC=$?
set -e
KA_CODE=${KA_CODE:-000}
KA_RESP=$(cat "$KA_TMP" 2>/dev/null || echo "")
if [ "$KA_RC" = "0" ] && [ "$KA_CODE" -ge 200 ] && [ "$KA_CODE" -lt 300 ]; then
break
fi
KA_SAFE_BODY=$(printf '%s' "$KA_RESP" | sanitize_http_body)
# Retry ONLY on transient transport errors — never on an agent-level
# error (those must surface and fail the gate).
if echo "$KA_CODE" | grep -Eq '^(502|503|504)$' && echo "$KA_SAFE_BODY" | grep -Eqi 'Service Unavailable|Bad Gateway|Gateway Timeout|workspace agent unreachable|connection refused|no healthy upstream|workspace agent busy|native_session'; then
log " known-answer A2A transient $KA_CODE attempt $KA_ATTEMPT/6: $KA_SAFE_BODY"
if [ "$KA_ATTEMPT" -lt 6 ]; then sleep 10; continue; fi
fi
break
done
rm -f "$KA_TMP"
if [ "$KA_RC" != "0" ] || [ "$KA_CODE" -lt 200 ] || [ "$KA_CODE" -ge 300 ]; then
KA_SAFE_BODY=$(printf '%s' "$KA_RESP" | sanitize_http_body)
fail "Known-answer A2A POST failed after $KA_ATTEMPT attempt(s) (curl_rc=$KA_RC, http=$KA_CODE): $KA_SAFE_BODY"
fi
KA_TEXT=$(echo "$KA_RESP" | python3 -c "
import json, sys
try:
d = json.load(sys.stdin)
parts = d.get('result', {}).get('parts', [])
print(parts[0].get('text', '') if parts else '')
except Exception:
print('')
" 2>/dev/null || echo "")
# CORE GATE: contains PINEAPPLE (real round-trip) AND no error-as-text.
a2a_assert_real_completion "$KA_TEXT" "PINEAPPLE" "A2A known-answer (parent, $RUNTIME/$MODEL_SLUG)"
# ─── 8c. byok-routing regression guard (#1994) ─────────────────────────
# The parent was provisioned with the customer's OWN vendor key
# (MINIMAX_API_KEY / ANTHROPIC_API_KEY in SECRETS_JSON) → it must resolve
# BYOK, not platform_managed. #1994 was exactly the inverse: a byok
# workspace baked platform_managed on (re-)provision → routed through the
# platform proxy → drained the platform LLM key. We read the SAME derived
# resolver the provision-time strip gate uses
# (GET /admin/workspaces/:id/llm-billing-mode) and assert resolved_mode!=
# platform_managed. A regression flips it RED.
#
# Only meaningful when the parent actually carries a byok credential; the
# OpenAI/hermes path uses a different env shape, and the no-key path is
# legitimately platform_managed (the CTO default). Gate on the same
# E2E_*_API_KEY presence the SECRETS_JSON branch keyed off.
if [ -n "${E2E_MINIMAX_API_KEY:-}" ] || [ -n "${E2E_ANTHROPIC_API_KEY:-}" ]; then
set +e
BILLING_RESP=$(tenant_call GET "/admin/workspaces/$PARENT_ID/llm-billing-mode" 2>/dev/null)
BILLING_RC=$?
set -e
if [ "$BILLING_RC" != "0" ] || [ -z "$BILLING_RESP" ]; then
fail "byok-routing guard: GET /admin/workspaces/$PARENT_ID/llm-billing-mode failed (rc=$BILLING_RC). Body: ${BILLING_RESP:0:200}"
fi
assert_byok_not_platform_proxy "$BILLING_RESP" "byok-guard (parent, $RUNTIME/$MODEL_SLUG)"
else
log "8c. byok-routing guard skipped — parent carries no own-vendor key (OpenAI/no-key path is legitimately platform_managed)."
fi
# ─── 8d. Per-offered-provider liveness matrix (SSOT-driven, #1994 class) ─
# For each platform-servable model the providers.yaml SSOT
# (runtimes.<runtime>.providers[platform].models) declares for this
# runtime, send a minimal max_tokens-bounded "say ok" probe and assert a
# NON-ERROR completion. Purpose: exercise each offered provider's AUTH +
# ROUTING path so a drained key / wrong base-URL / byok-misroute fails the
# gate (the #1994 class). Providers/models come from the SSOT — not a
# hardcoded list — so the matrix tracks providers.yaml automatically.
#
# This lane provisions ONE parent workspace with ONE configured key, so we
# can only truly drive the providers that key authenticates. Probing a
# model whose provider key is absent in this lane is reported SKIP (rc=75),
# not FAIL — keeping the gate deterministic + low-flake. The matrix still
# proves the configured provider's full auth+routing path end-to-end, and
# logs the offered set so over/under-offer drift is visible in the CI log.
provider_liveness_probe() {
local model_id="$1"
# Map the SSOT platform model id (e.g. minimax/MiniMax-M2.7) to the
# vendor namespace token to decide whether THIS lane has its key.
local vendor="${model_id%%/*}"
case "$vendor" in
minimax) [ -n "${E2E_MINIMAX_API_KEY:-}" ] || return 75 ;;
anthropic) [ -n "${E2E_ANTHROPIC_API_KEY:-}" ] || return 75 ;;
openai) [ -n "${E2E_OPENAI_API_KEY:-}" ] || return 75 ;;
*) return 75 ;; # kimi/moonshot etc. — no key wired in this lane
esac
local probe_payload
probe_payload=$(python3 -c "
import json, uuid
print(json.dumps({
'jsonrpc': '2.0',
'method': 'message/send',
'id': 'e2e-liveness-' + uuid.uuid4().hex[:6],
'params': {
'message': {
'role': 'user',
'messageId': f'e2e-{uuid.uuid4().hex[:8]}',
'parts': [{'kind': 'text', 'text': 'Reply with exactly: ok'}],
},
'configuration': {'max_tokens': 4}
}
}))
")
local tmp code rc resp
tmp=$(mktemp -t liveness_a2a.XXXXXX)
set +e
code=$(tenant_call POST "/workspaces/$PARENT_ID/a2a" \
--max-time 60 \
-H "Content-Type: application/json" \
-d "$probe_payload" \
-o "$tmp" -w '%{http_code}' 2>/dev/null)
rc=$?
set -e
resp=$(cat "$tmp" 2>/dev/null || echo "")
rm -f "$tmp"
if [ "$rc" != "0" ] || [ "${code:-000}" -lt 200 ] || [ "${code:-000}" -ge 300 ]; then
log " probe $model_id: HTTP ${code:-000} rc=$rc"
return 1
fi
local text
text=$(echo "$resp" | python3 -c "
import json,sys
try:
d=json.load(sys.stdin); p=d.get('result',{}).get('parts',[])
print(p[0].get('text','') if p else '')
except Exception: print('')" 2>/dev/null || echo "")
if [ -z "$text" ] || a2a_completion_error_marker "$text" >/dev/null; then
log " probe $model_id: error-as-text or empty: ${text:0:120}"
return 1
fi
return 0
}
if ! provider_liveness_matrix "$RUNTIME" provider_liveness_probe; then
fail "Per-provider liveness matrix: at least one offered provider failed its auth+routing probe (see matrix above). This is the #1994 class — a drained key / wrong base-URL / byok-misroute."
fi
ok "Per-provider liveness matrix passed (all probed offered providers completed without error)"
# ─── 9. HMA + peers + activity (full mode) ─────────────────────────────
if [ "$MODE" = "full" ]; then
log "9/11 Writing + reading HMA memory on parent..."
+15
View File
@@ -36,6 +36,7 @@ import (
"time"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/channels"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/codexauth"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/crypto"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/db"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/events"
@@ -334,6 +335,20 @@ func main() {
pendinguploads.StartSweeper(c, pendinguploads.NewPostgres(db.DB), 0)
})
// Codex shared-OAuth central refresher — the SINGLE owner of the rotating
// refresh_token for the global codex (ChatGPT/Codex subscription) credential
// (global_secrets key CODEX_AUTH_JSON). Multiple codex workspaces share ONE
// ChatGPT-Pro OAuth token; OpenAI's refresh_token is single-use, so letting
// each per-agent app-server refresh on its own 401 burned the seed within
// seconds (a refresh storm). This goroutine is structurally single-flight
// (one goroutine + a package mutex), refreshes only within a safety margin
// of expiry, POSTs the refresh_token at most once per due cycle, and writes
// the rotated blob back — workspaces now only GET the current token (see the
// codex template's codex_auth_sync.sh). INERT when no CODEX_AUTH_JSON exists.
go supervised.RunWithRecover(ctx, "codex-auth-refresher", func(c context.Context) {
codexauth.StartCodexAuthRefresher(c, db.DB)
})
// Provision-timeout sweep — flips workspaces that have been stuck in
// status='provisioning' past the timeout window to 'failed' and emits
// WORKSPACE_PROVISION_TIMEOUT. Without this the UI banner is cosmetic
+114
View File
@@ -0,0 +1,114 @@
# Molecule Platform OpenAPI specs
This directory holds the machine-readable API contracts for the Molecule
platform.
| File | Spec | Scope | Status |
|------|------|-------|--------|
| `management.yaml` | OpenAPI **3.1** | The **management surface** across both services (orgs, billing, admin, provisioning, workspaces, secrets, templates, org-tokens, bundles). | **SSOT** — hand-authored. |
| `swagger.yaml` / `swagger.json` | OpenAPI 2.0 | swaggo-generated stub, `/schedules` only (the per-workspace **runtime** surface). | Legacy stub; superseded for management by `management.yaml`. |
`management.yaml` is the **single source of truth** the management tooling
derives from — the management MCP server, the management CLI (`molecule-cli`),
and the human-facing API docs (RFC #1706, the gap closed by
`PLATFORM-MANAGEMENT-API.md` §5c). Do not hand-edit those clients' route maps;
change them here and regenerate/derive.
## The two-service split
One structural fact drives the whole spec: there are **two services with two
auth stacks**, and the management surface spans both.
```
┌─────────────────────────────────────────┐
browser / CLI / MCP │ Control plane (CP) │
│ │ molecule-controlplane @ api.moleculesai │
│ session │ /api/v1/* (stable) [+ /cp/* sunset] │
├───────────────▶│ orgs · members · billing · provisioning │
│ admin bearer │ · fleet/admin ops · pins │
│ provision sec │ │
└────────────────┴──────────────┬───────────────────────────┘
│ edge reverse-proxy
│ (subdomain / X-Molecule-Org-Slug)
┌─────────────────────────────────────────┐
Org API Key / ws tok │ Tenant workspace-server │
│ │ molecule-core/workspace-server │
└───────────────▶│ ONE EC2 per org @ <slug>.moleculesai.app│
│ workspaces · secrets · templates · │
│ org-tokens · bundles │
└─────────────────────────────────────────┘
```
- **Control plane (CP)**`api.moleculesai.app`, routes modelled under
`/api/v1/*` (the `/cp/*` mirror is identical but sunset-headed per RFC #61 and
is not duplicated in the spec). Owns **orgs, members, billing, provisioning,
fleet/admin ops**.
- **Tenant workspace-server** — one EC2 per org at `<slug>.moleculesai.app`.
Owns **workspaces, agents, secrets, templates, org-tokens, bundles**. Requests
may also be sent to the CP host with an `X-Molecule-Org-Slug` header; the CP
edge reverse-proxies them to the tenant host (the `Authorization`,
`X-Molecule-Org-*`, and cookie headers pass through unchanged and the tenant's
own middleware validates them).
The key consequence, called out in `PLATFORM-MANAGEMENT-API.md`: **the Org API
Key is a TENANT credential, not a CP one.** It is full tenant-admin over its own
org's workspace-server surface and reaches **nothing** on the CP (org
create/delete, billing, members, provisioning all 401/403 it). That is why
member/billing tools belong in a separate CP-admin MCP, not the org-key-authed
management MCP.
## Security scheme → surface map (the tier matrix)
`management.yaml` defines these `securitySchemes`; each operation declares the
one(s) it accepts. Mirror of `PLATFORM-MANAGEMENT-API.md` §1:
| Scheme | What it is | Where it applies |
|--------|-----------|------------------|
| `workosSession` | WorkOS AuthKit session cookie `mcp_session` (+ org membership/ownership checks) | CP `/api/v1/orgs/*`, `/api/v1/billing/*`. Also accepted on the tenant surface via the CP-session path. |
| `cpAdminBearer` | CP `CP_ADMIN_API_TOKEN` operator bearer (AdminGate, constant-time) | CP `/api/v1/admin/*` — admin-create-org, tenant teardown, workspace env, ListOrgWorkspaces, redeploy, pins. |
| `provisionSecret` | CP `PROVISION_SHARED_SECRET` bearer | CP `/api/v1/workspaces/provision`, `…/status`. Routes unmounted when the secret is unset. |
| `tenantAdminToken` | Per-tenant admin_token (+ `X-Molecule-Org-Id`) | CP `DELETE /api/v1/workspaces/:id` (deprovision) — **in addition to** `provisionSecret` (issue #118). |
| `orgApiKey` | Tenant Org API Key — `Authorization: Bearer <key>` + routing header; full tenant-admin, self-minting | **All** tenant routes: `/workspaces[/:id]`, `/workspaces/:id/secrets`, budget, billing-mode, `/settings/secrets`, `/org/import`, `/org/templates`, `/org/tokens`, `/templates`, `/bundles`. |
| `workspaceToken` | Per-workspace bearer, bound to one workspace id (+ routing header) | Read/lifecycle/secrets on a single `/workspaces/:id/*`. **Rejected** on admin list/create/delete when ADMIN_TOKEN is set — use `orgApiKey`. |
| `orgRoutingHeaderId` / `orgRoutingHeaderSlug` | `X-Molecule-Org-Id` / `X-Molecule-Org-Slug` | Required on every tenant-host request so the edge / TenantGuard route + authorize against the correct org. Send one of them alongside the bearer. |
### Guards worth knowing (modelled per-operation)
- **Dry-run:** `POST /api/v1/admin/orgs?dry_run=true` — validate + echo, no org
created. (The only dry-run on the whole management API.)
- **Confirm token:** `DELETE /api/v1/admin/tenants/:slug` and
`…/scrub-artifacts` — body `confirm` MUST equal the URL slug, else `400`
before any teardown.
- **Force flag:** `POST /api/v1/admin/workspaces/:id/env` — keys matching the
secret-keyword guard (`TOKEN`/`SECRET`/`KEY`/`PASSWORD`) require `force=true`.
- **Runtime-pin gate:** `POST /api/v1/workspaces/provision` returns `422
RUNTIME_PIN_MISSING` when no runtime image pin exists.
- **Auto-restart side-effects:** writing a workspace or global secret
auto-restarts the affected workspace(s).
## Security note (carried from the synthesis spec)
The Org API Key is **full tenant-admin and self-minting** — a management MCP
holding one holds tenant root. There is no scope-down today (TODO in
`orgtoken`). Per-role / per-workspace scoping should ship alongside the
management MCP.
## Validate
```bash
cd workspace-server/docs/openapi
npx @redocly/cli lint management.yaml # must be clean (0 errors, 0 warnings)
```
## Scope notes / best-effort flags
- The per-workspace **runtime** surface (schedules, agent, registry, a2a,
memory, approvals, channels, terminal, files) is intentionally **out of
scope** here — that's the runtime contract, not management.
- A handful of bodies are **best-effort** from the handlers (org-import inline
template, bundle import, list responses with open shapes) and are marked with
`additionalProperties: true` in the schema. Tighten as the handler structs
stabilise.
- `/cp/*` deprecated mirrors are omitted (identical shapes; RFC #61
Deprecation/Sunset). Build against `/api/v1/*`.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,463 @@
// Package codexauth owns the SINGLE, platform-side refresh of the global
// codex (ChatGPT/Codex subscription) OAuth credential stored in the
// global_secrets table under key CODEX_AUTH_JSON.
//
// THE PROBLEM IT FIXES (agents-team prod, 2026-05-31)
//
// Multiple codex workspaces share ONE ChatGPT-Pro OAuth token (the global
// secret CODEX_AUTH_JSON). OpenAI's refresh_token is SINGLE-USE: every refresh
// rotates it and invalidates the prior one. When each per-agent codex
// app-server refreshed independently on a 401, the siblings' in-flight tokens
// were invalidated within seconds — a refresh storm that burned the seed and
// wedged every codex agent.
//
// THE FIX (two halves; this is the core half)
//
// 1. The per-workspace codex app-server NO LONGER refreshes (the template's
// OAuth POST is gated off by default — see the codex template's
// codex_auth_sync.sh / CODEX_AUTH_REFRESH_OWNER gate). Workspaces only ever
// GET the current token and write it to auth.json.
// 2. ONE owner refreshes the rotating refresh_token: this background goroutine
// in the platform. It is structurally single-flight (one goroutine + a
// package mutex), refreshes ONLY when the access_token is within a safety
// margin of expiry, POSTs the refresh_token at most ONCE per due cycle, and
// writes the rotated blob back to global_secrets. On a permanent failure
// (the seed was already burned by an out-of-band login) it logs ONCE and
// backs off — it never hot-loops a dead refresh_token.
//
// Billing-mode resolution and the byok strip are UNTOUCHED by this package.
package codexauth
import (
"context"
"database/sql"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"strings"
"sync"
"time"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/crypto"
)
const (
// CodexAuthSecretKey is the global_secrets key holding the shared codex
// ChatGPT/Codex subscription OAuth blob (auth.json contents).
CodexAuthSecretKey = "CODEX_AUTH_JSON"
// oauthTokenURL is OpenAI's OAuth token endpoint. The ONLY endpoint this
// package ever POSTs to, and only for a due refresh.
oauthTokenURL = "https://auth.openai.com/oauth/token"
// codexOAuthClientID is the public Codex CLI OAuth client id (the same id
// the codex CLI sends). Not a secret.
codexOAuthClientID = "app_EMoamEEZ73f0CkXaXp7hrann"
// refreshSafetyMargin is how far ahead of access_token expiry a refresh is
// considered DUE. A token expiring within this window is refreshed now; one
// expiring later is left untouched (skip-when-fresh). Generous so a slow
// tick can never let the shared token lapse for the fleet.
refreshSafetyMargin = 15 * time.Minute
// defaultInterval is how often the loop wakes to check due-ness. The check
// is cheap (decrypt + JWT exp parse) and only POSTs when actually due.
defaultInterval = 5 * time.Minute
// permanentFailureBackoff is how long the loop waits after a PERMANENT
// refresh failure (invalid_grant / "refresh token already used"). The seed
// is burned until a human re-seeds a fresh login; there is nothing to retry,
// so we back off hard rather than hammer the dead token.
permanentFailureBackoff = 1 * time.Hour
)
// SecretStore is the minimal global_secrets surface the refresher needs. The
// production implementation (postgresStore) is backed by *sql.DB; tests inject
// a fake. It is deliberately tiny — read one key, write one key — so the test
// double is trivial and the refresher never reaches for the package-global DB.
type SecretStore interface {
// Get returns the decrypted secret value and true, or ("", false) when the
// key is absent. A non-nil error is a real read failure (not absence).
Get(ctx context.Context, key string) (value string, found bool, err error)
// Put encrypts and upserts value under key, bumping the row's updated_at
// (the "last_refresh" timestamp). It is the rotated-blob write-back.
Put(ctx context.Context, key, value string) error
}
// httpDoer is the http client seam (real *http.Client in prod, fake transport
// in tests). Tests NEVER hit the network.
type httpDoer interface {
Do(req *http.Request) (*http.Response, error)
}
// refresher is the single-owner refresh engine. The package-level mutex makes
// the refresh structurally single-flight: even if two refreshOnce calls raced
// (they cannot in prod — one goroutine drives it — but a test or a future
// caller might), only one POSTs at a time, and the access-token freshness
// re-check inside the lock means the second sees a freshly-rotated token and
// skips. One goroutine + this mutex = single-flight by construction.
type refresher struct {
store SecretStore
client httpDoer
now func() time.Time
// permanentlyFailed records that the current seed's refresh_token was
// rejected as already-used/invalid. While set, refreshOnce is INERT (it
// will not re-POST the dead token) until the secret value CHANGES (a human
// re-seed), detected by comparing the stored blob. This is the anti-storm
// latch — it lives on the struct, not globally, so it resets if the seed is
// replaced out of band.
failedSeed string // the auth-json blob that failed; "" = no known failure
}
// mu serializes refreshOnce across the process. Package-level so the
// single-flight guarantee holds regardless of how many refresher values exist
// (in prod there is exactly one).
var mu sync.Mutex
// oauthTokens is the token trio inside auth.json (and the OAuth response).
type oauthTokens struct {
AccessToken string `json:"access_token"`
RefreshToken string `json:"refresh_token"`
IDToken string `json:"id_token,omitempty"`
}
// StartCodexAuthRefresher launches the single background refresher goroutine.
// It returns immediately; the loop runs until ctx is cancelled. Wire it under
// supervised.RunWithRecover in main.go like the other Start* sweeps.
//
// db may be nil only in tests that drive refreshOnce directly; in prod it is
// the server's *sql.DB. The loop is INERT (logs once, keeps ticking) whenever
// CODEX_AUTH_JSON is absent — a deployment with no shared codex seed pays only
// a cheap periodic read.
func StartCodexAuthRefresher(ctx context.Context, db *sql.DB) {
r := &refresher{
store: &postgresStore{db: db},
client: &http.Client{Timeout: 30 * time.Second},
now: time.Now,
}
r.run(ctx, defaultInterval)
}
// run is the tick loop. It checks due-ness every interval and on a permanent
// failure waits permanentFailureBackoff before the next check (never a tight
// retry of a burned token).
func (r *refresher) run(ctx context.Context, interval time.Duration) {
// Check once promptly on boot, then on the interval.
for {
wait := interval
if perm := r.refreshOnce(ctx); perm {
// Permanent failure this cycle — the seed is burned. Back off hard;
// a human must re-seed. We keep ticking (a re-seed CHANGES the blob,
// which clears the latch) but slowly.
wait = permanentFailureBackoff
}
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
log.Printf("codexauth: context done; stopping refresher")
return
case <-timer.C:
}
}
}
// refreshOnce performs ONE due-check + at most one refresh POST. It returns
// permanentFailure=true iff the refresh_token was permanently rejected this
// cycle (the caller backs off). All other outcomes (inert/skip/rotated/transient
// error) return false.
//
// It is single-flight: the package mutex is held for the whole read→decide→
// POST→write-back so two callers cannot both POST the (single-use) refresh_token.
func (r *refresher) refreshOnce(ctx context.Context) (permanentFailure bool) {
mu.Lock()
defer mu.Unlock()
blob, found, err := r.store.Get(ctx, CodexAuthSecretKey)
if err != nil {
log.Printf("codexauth: read CODEX_AUTH_JSON failed: %v (skipping this cycle)", err)
return false
}
if !found || strings.TrimSpace(blob) == "" {
// INERT: no shared codex seed in this deployment. Cheap no-op.
log.Printf("codexauth: no CODEX_AUTH_JSON in global_secrets — refresher inert")
// A previously-failed seed that has since been DELETED clears the latch.
r.failedSeed = ""
return false
}
// Anti-storm latch: if THIS exact blob already failed permanently, do not
// re-POST its dead refresh_token. A re-seed changes the blob and clears it.
if r.failedSeed != "" && r.failedSeed == blob {
return false
}
if r.failedSeed != "" && r.failedSeed != blob {
// The seed changed out of band (human re-login) — give it a fresh chance.
r.failedSeed = ""
}
tokens, err := parseTokens(blob)
if err != nil {
log.Printf("codexauth: CODEX_AUTH_JSON is not parseable codex auth json: %v (skipping)", err)
return false
}
if tokens.RefreshToken == "" {
log.Printf("codexauth: CODEX_AUTH_JSON carries no refresh_token (skipping)")
return false
}
// Skip-when-fresh: only refresh within the safety margin of expiry. A blob
// with an unparseable/absent access_token exp is treated as DUE (better to
// refresh a token we cannot date than let the fleet lapse).
exp, haveExp := jwtExp(tokens.AccessToken)
if haveExp {
remaining := exp.Sub(r.now())
if remaining > refreshSafetyMargin {
// Fresh — nothing to do. No POST.
return false
}
}
// DUE: POST the refresh_token ONCE.
newTokens, perm, err := r.doRefresh(ctx, tokens.RefreshToken)
if err != nil {
if perm {
// Permanent: the seed is burned. Latch it so we don't re-POST, log
// ONCE, and DO NOT write anything back.
log.Printf("codexauth: PERMANENT refresh failure (refresh_token rejected): %v — "+
"NOT writing back; the shared CODEX_AUTH_JSON seed is burned and must be re-seeded "+
"via a fresh codex login. Backing off.", err)
r.failedSeed = blob
return true
}
// Transient (network/5xx): no write-back, retry next cycle (no backoff).
log.Printf("codexauth: transient refresh error: %v (will retry next cycle)", err)
return false
}
// Success: merge the rotated trio into the blob (preserving every other
// field) and write it back encrypted, bumping updated_at (last_refresh).
rotated, err := mergeTokens(blob, newTokens)
if err != nil {
log.Printf("codexauth: failed to merge rotated tokens into auth json: %v (NOT writing back)", err)
return false
}
if err := r.store.Put(ctx, CodexAuthSecretKey, rotated); err != nil {
log.Printf("codexauth: write-back of rotated CODEX_AUTH_JSON failed: %v", err)
return false
}
r.failedSeed = "" // success clears any stale latch
log.Printf("codexauth: rotated shared CODEX_AUTH_JSON (single-owner refresh)")
return false
}
// doRefresh POSTs the refresh_token to OpenAI's OAuth endpoint exactly once and
// returns the rotated trio. permanent=true marks an unrecoverable rejection
// (HTTP 400 invalid_grant / "refresh token already used") so the caller latches
// and backs off instead of retrying.
func (r *refresher) doRefresh(ctx context.Context, refreshToken string) (tokens oauthTokens, permanent bool, err error) {
body, _ := json.Marshal(map[string]string{
"grant_type": "refresh_token",
"client_id": codexOAuthClientID,
"refresh_token": refreshToken,
})
req, err := http.NewRequestWithContext(ctx, http.MethodPost, oauthTokenURL, strings.NewReader(string(body)))
if err != nil {
return oauthTokens{}, false, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
resp, err := r.client.Do(req)
if err != nil {
return oauthTokens{}, false, err // transient: network
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode == http.StatusOK {
var t oauthTokens
if err := json.Unmarshal(respBody, &t); err != nil {
return oauthTokens{}, false, fmt.Errorf("decode token response: %w", err)
}
if t.AccessToken == "" {
return oauthTokens{}, false, fmt.Errorf("token response missing access_token")
}
return t, false, nil
}
// Non-200. A 400 (and any body naming invalid_grant / already-used) is a
// PERMANENT rejection of the refresh_token. 401/403 likewise mean the seed
// is no good. Everything else (429/5xx/network-shaped) is transient.
lowerBody := strings.ToLower(string(respBody))
isInvalidGrant := strings.Contains(lowerBody, "invalid_grant") ||
strings.Contains(lowerBody, "refresh token already used") ||
strings.Contains(lowerBody, "already been used") ||
strings.Contains(lowerBody, "token has been revoked")
switch {
case resp.StatusCode == http.StatusBadRequest && isInvalidGrant:
return oauthTokens{}, true, fmt.Errorf("oauth %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
case resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden:
return oauthTokens{}, true, fmt.Errorf("oauth %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
default:
return oauthTokens{}, false, fmt.Errorf("oauth %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
}
}
// parseTokens extracts the OAuth trio from an auth.json blob, accepting both
// the nested `{"tokens":{...}}` shape the codex CLI writes and a flat top-level
// shape some seeds use.
func parseTokens(blob string) (oauthTokens, error) {
var top map[string]json.RawMessage
if err := json.Unmarshal([]byte(blob), &top); err != nil {
return oauthTokens{}, err
}
if nested, ok := top["tokens"]; ok {
var t oauthTokens
if err := json.Unmarshal(nested, &t); err != nil {
return oauthTokens{}, fmt.Errorf("decode nested tokens: %w", err)
}
return t, nil
}
var t oauthTokens
if err := json.Unmarshal([]byte(blob), &t); err != nil {
return oauthTokens{}, err
}
return t, nil
}
// mergeTokens writes the rotated trio back into the original blob in-place,
// preserving the blob's shape (nested-vs-flat) and every other field. A field
// in the OAuth response that is empty (e.g. id_token omitted) does NOT clobber
// the existing value.
func mergeTokens(blob string, rotated oauthTokens) (string, error) {
var top map[string]json.RawMessage
if err := json.Unmarshal([]byte(blob), &top); err != nil {
return "", err
}
applyTo := func(m map[string]json.RawMessage) error {
setStr := func(key, val string) error {
if val == "" {
return nil // don't clobber an existing value with an empty one
}
b, err := json.Marshal(val)
if err != nil {
return err
}
m[key] = b
return nil
}
if err := setStr("access_token", rotated.AccessToken); err != nil {
return err
}
if err := setStr("refresh_token", rotated.RefreshToken); err != nil {
return err
}
if err := setStr("id_token", rotated.IDToken); err != nil {
return err
}
return nil
}
if nestedRaw, ok := top["tokens"]; ok {
var nested map[string]json.RawMessage
if err := json.Unmarshal(nestedRaw, &nested); err != nil {
return "", fmt.Errorf("decode nested tokens for merge: %w", err)
}
if err := applyTo(nested); err != nil {
return "", err
}
nb, err := json.Marshal(nested)
if err != nil {
return "", err
}
top["tokens"] = nb
} else {
if err := applyTo(top); err != nil {
return "", err
}
}
out, err := json.Marshal(top)
if err != nil {
return "", err
}
return string(out), nil
}
// jwtExp decodes the `exp` claim (Unix seconds) from a JWT access token WITHOUT
// verifying the signature (we only need the expiry to decide due-ness; the
// token's validity is OpenAI's to enforce). Returns ok=false when the token is
// not a parseable 3-part JWT or carries no numeric exp.
func jwtExp(token string) (time.Time, bool) {
parts := strings.Split(token, ".")
if len(parts) != 3 {
return time.Time{}, false
}
payload, err := base64.RawURLEncoding.DecodeString(parts[1])
if err != nil {
// Some encoders pad; tolerate standard base64url with padding too.
payload, err = base64.URLEncoding.DecodeString(parts[1])
if err != nil {
return time.Time{}, false
}
}
var claims struct {
Exp json.Number `json:"exp"`
}
if err := json.Unmarshal(payload, &claims); err != nil {
return time.Time{}, false
}
secs, err := claims.Exp.Int64()
if err != nil || secs <= 0 {
return time.Time{}, false
}
return time.Unix(secs, 0), true
}
// postgresStore is the production SecretStore backed by global_secrets, using
// the SAME crypto path the secrets handler uses (DecryptVersioned on read,
// Encrypt + CurrentEncryptionVersion on write).
type postgresStore struct {
db *sql.DB
}
func (s *postgresStore) Get(ctx context.Context, key string) (string, bool, error) {
var enc []byte
var ver int
err := s.db.QueryRowContext(ctx,
`SELECT encrypted_value, encryption_version FROM global_secrets WHERE key = $1`, key).
Scan(&enc, &ver)
if err == sql.ErrNoRows {
return "", false, nil
}
if err != nil {
return "", false, err
}
plain, err := crypto.DecryptVersioned(enc, ver)
if err != nil {
return "", false, err
}
return string(plain), true, nil
}
func (s *postgresStore) Put(ctx context.Context, key, value string) error {
enc, err := crypto.Encrypt([]byte(value))
if err != nil {
return err
}
ver := crypto.CurrentEncryptionVersion()
_, err = s.db.ExecContext(ctx, `
INSERT INTO global_secrets (key, encrypted_value, encryption_version)
VALUES ($1, $2, $3)
ON CONFLICT (key) DO UPDATE
SET encrypted_value = $2, encryption_version = $3, updated_at = now()
`, key, enc, ver)
return err
}
@@ -0,0 +1,425 @@
package codexauth
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
// --- test doubles -----------------------------------------------------------
// fakeStore is an in-memory SecretStore. nil entry = absent key.
type fakeStore struct {
mu sync.Mutex
values map[string]string
getErr error
putErr error
puts int32 // count of successful Put calls
}
func newFakeStore() *fakeStore { return &fakeStore{values: map[string]string{}} }
func (f *fakeStore) Get(_ context.Context, key string) (string, bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.getErr != nil {
return "", false, f.getErr
}
v, ok := f.values[key]
return v, ok, nil
}
func (f *fakeStore) Put(_ context.Context, key, value string) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.putErr != nil {
return f.putErr
}
f.values[key] = value
atomic.AddInt32(&f.puts, 1)
return nil
}
func (f *fakeStore) get(key string) string {
f.mu.Lock()
defer f.mu.Unlock()
return f.values[key]
}
// fakeTransport records every request and returns a scripted response. It is
// the network seam — tests NEVER make a real request.
type fakeTransport struct {
mu sync.Mutex
calls int32
urls []string
methods []string
bodies []string
status int
respBody string
transport func(*http.Request) (*http.Response, error) // optional override
}
func (t *fakeTransport) Do(req *http.Request) (*http.Response, error) {
atomic.AddInt32(&t.calls, 1)
t.mu.Lock()
t.urls = append(t.urls, req.URL.String())
t.methods = append(t.methods, req.Method)
if req.Body != nil {
b, _ := io.ReadAll(req.Body)
t.bodies = append(t.bodies, string(b))
} else {
t.bodies = append(t.bodies, "")
}
t.mu.Unlock()
if t.transport != nil {
return t.transport(req)
}
status := t.status
if status == 0 {
status = http.StatusOK
}
return &http.Response{
StatusCode: status,
Body: io.NopCloser(strings.NewReader(t.respBody)),
Header: make(http.Header),
}, nil
}
func (t *fakeTransport) callCount() int { return int(atomic.LoadInt32(&t.calls)) }
// --- helpers ----------------------------------------------------------------
// makeJWT builds an unsigned-but-parseable JWT whose payload carries exp.
func makeJWT(exp time.Time) string {
header := base64.RawURLEncoding.EncodeToString([]byte(`{"alg":"none","typ":"JWT"}`))
payload := base64.RawURLEncoding.EncodeToString([]byte(
fmt.Sprintf(`{"exp":%d,"sub":"codex"}`, exp.Unix())))
sig := base64.RawURLEncoding.EncodeToString([]byte("sig"))
return header + "." + payload + "." + sig
}
// authBlob builds a nested codex auth.json blob with the given tokens.
func authBlob(access, refresh string) string {
b, _ := json.Marshal(map[string]any{
"tokens": map[string]any{
"access_token": access,
"refresh_token": refresh,
"id_token": "id-original",
},
"OPENAI_API_KEY": nil,
"last_refresh": "2026-01-01T00:00:00Z",
})
return string(b)
}
func newTestRefresher(store SecretStore, client httpDoer, now time.Time) *refresher {
return &refresher{
store: store,
client: client,
now: func() time.Time { return now },
}
}
func okRefreshResponse(access, refresh string) string {
b, _ := json.Marshal(oauthTokens{AccessToken: access, RefreshToken: refresh, IDToken: "id-new"})
return string(b)
}
// --- tests ------------------------------------------------------------------
// TestJWTExpParse covers the exp decode (valid, malformed, missing).
func TestJWTExpParse(t *testing.T) {
want := time.Now().Add(2 * time.Hour).Truncate(time.Second)
got, ok := jwtExp(makeJWT(want))
if !ok {
t.Fatalf("jwtExp(valid) ok=false, want true")
}
if !got.Equal(want) {
t.Errorf("jwtExp = %v, want %v", got, want)
}
if _, ok := jwtExp("not-a-jwt"); ok {
t.Errorf("jwtExp(non-jwt) ok=true, want false")
}
if _, ok := jwtExp("a.b.c"); ok {
t.Errorf("jwtExp(garbage parts) ok=true, want false")
}
// 3 parts but payload has no exp.
noExp := base64.RawURLEncoding.EncodeToString([]byte("{}"))
if _, ok := jwtExp("h." + noExp + ".s"); ok {
t.Errorf("jwtExp(no exp claim) ok=true, want false")
}
}
// TestRefreshOnce_SkipWhenFresh: a token well outside the safety margin is NOT
// refreshed — no POST, no write-back.
func TestRefreshOnce_SkipWhenFresh(t *testing.T) {
now := time.Now()
store := newFakeStore()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(2*time.Hour)), "rt-1")
tr := &fakeTransport{status: http.StatusOK, respBody: okRefreshResponse("new-at", "rt-2")}
r := newTestRefresher(store, tr, now)
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("fresh token: permanentFailure=true, want false")
}
if tr.callCount() != 0 {
t.Errorf("fresh token: %d OAuth POSTs, want 0", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 0 {
t.Errorf("fresh token: %d write-backs, want 0", store.puts)
}
}
// TestRefreshOnce_RotateThenReskip: a token inside the margin is refreshed once
// (POST + write-back of the rotated blob); a subsequent call on the now-fresh
// rotated token skips (no second POST). Proves rotate→write-back→re-skip.
func TestRefreshOnce_RotateThenReskip(t *testing.T) {
now := time.Now()
store := newFakeStore()
// Expires in 5m — inside the 15m safety margin → DUE.
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(5*time.Minute)), "rt-1")
// Rotated access token is fresh (2h out); rotated refresh is rt-2.
tr := &fakeTransport{status: http.StatusOK, respBody: okRefreshResponse(makeJWT(now.Add(2*time.Hour)), "rt-2")}
r := newTestRefresher(store, tr, now)
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("due token: permanentFailure=true, want false")
}
if tr.callCount() != 1 {
t.Fatalf("due token: %d OAuth POSTs, want exactly 1", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 1 {
t.Fatalf("due token: %d write-backs, want exactly 1", store.puts)
}
// The written blob must carry the rotated refresh_token and preserve the
// non-token field.
rotated := store.get(CodexAuthSecretKey)
tokens, err := parseTokens(rotated)
if err != nil {
t.Fatalf("parse rotated blob: %v", err)
}
if tokens.RefreshToken != "rt-2" {
t.Errorf("rotated refresh_token = %q, want rt-2", tokens.RefreshToken)
}
if !strings.Contains(rotated, "last_refresh") {
t.Errorf("rotated blob dropped the preserved last_refresh field: %s", rotated)
}
// Second call: the rotated access token is fresh → skip, no new POST.
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("re-skip: permanentFailure=true, want false")
}
if tr.callCount() != 1 {
t.Errorf("re-skip: %d total OAuth POSTs, want still 1", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 1 {
t.Errorf("re-skip: %d total write-backs, want still 1", store.puts)
}
}
// TestRefreshOnce_NoSecretInert: absent CODEX_AUTH_JSON → inert (no POST, no
// write-back, no error/permanent).
func TestRefreshOnce_NoSecretInert(t *testing.T) {
store := newFakeStore() // empty
tr := &fakeTransport{}
r := newTestRefresher(store, tr, time.Now())
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("no secret: permanentFailure=true, want false")
}
if tr.callCount() != 0 {
t.Errorf("no secret: %d POSTs, want 0", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 0 {
t.Errorf("no secret: %d write-backs, want 0", store.puts)
}
}
// TestRefreshOnce_PermanentFailNoWriteNoStorm: a 400 invalid_grant must (a) not
// write back, (b) return permanentFailure=true, and (c) NOT re-POST on the next
// cycle for the same (burned) seed — the anti-storm latch.
func TestRefreshOnce_PermanentFailNoWriteNoStorm(t *testing.T) {
now := time.Now()
store := newFakeStore()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(1*time.Minute)), "rt-burned")
tr := &fakeTransport{
status: http.StatusBadRequest,
respBody: `{"error":"invalid_grant","error_description":"refresh token already used"}`,
}
r := newTestRefresher(store, tr, now)
perm := r.refreshOnce(context.Background())
if !perm {
t.Fatalf("invalid_grant: permanentFailure=false, want true")
}
if tr.callCount() != 1 {
t.Fatalf("invalid_grant: %d POSTs, want exactly 1", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 0 {
t.Fatalf("invalid_grant: %d write-backs, want 0 (must NOT persist a failed refresh)", store.puts)
}
// Next cycle, SAME burned seed: must NOT re-POST (anti-storm latch).
perm2 := r.refreshOnce(context.Background())
if tr.callCount() != 1 {
t.Errorf("anti-storm: re-POSTed a burned refresh_token (%d total POSTs, want still 1)", tr.callCount())
}
_ = perm2 // latched cycle returns false (already-known failure, nothing new)
// A RE-SEED (blob changes) clears the latch and allows a fresh attempt.
store.mu.Lock()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(1*time.Minute)), "rt-freshly-seeded")
store.mu.Unlock()
tr.status = http.StatusOK
tr.respBody = okRefreshResponse(makeJWT(now.Add(2*time.Hour)), "rt-rotated")
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("post-reseed: permanentFailure=true, want false")
}
if tr.callCount() != 2 {
t.Errorf("post-reseed: %d total POSTs, want 2 (latch should clear on re-seed)", tr.callCount())
}
}
// TestRefreshOnce_TransientNoWriteNoLatch: a 5xx is transient — no write-back,
// returns false (no hard backoff latch), and a later cycle retries.
func TestRefreshOnce_TransientNoWriteNoLatch(t *testing.T) {
now := time.Now()
store := newFakeStore()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(1*time.Minute)), "rt-1")
tr := &fakeTransport{status: http.StatusServiceUnavailable, respBody: "upstream down"}
r := newTestRefresher(store, tr, now)
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("503: permanentFailure=true, want false (transient)")
}
if atomic.LoadInt32(&store.puts) != 0 {
t.Errorf("503: %d write-backs, want 0", store.puts)
}
// Retry next cycle succeeds (no latch on transient).
tr.status = http.StatusOK
tr.respBody = okRefreshResponse(makeJWT(now.Add(2*time.Hour)), "rt-2")
if perm := r.refreshOnce(context.Background()); perm {
t.Fatalf("retry after 503: permanentFailure=true, want false")
}
if tr.callCount() != 2 {
t.Errorf("transient retry: %d total POSTs, want 2", tr.callCount())
}
if atomic.LoadInt32(&store.puts) != 1 {
t.Errorf("transient retry: %d write-backs, want 1", store.puts)
}
}
// TestRefreshOnce_SingleFlight: concurrent refreshOnce calls on a DUE token must
// POST exactly once total — the package mutex serializes them and the second
// sees the freshly-rotated (now-fresh) token and skips. Structural single-flight.
func TestRefreshOnce_SingleFlight(t *testing.T) {
now := time.Now()
store := newFakeStore()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(1*time.Minute)), "rt-1")
// Every successful rotation yields a FRESH (2h) access token, so once one
// caller rotates, the other sees fresh and skips.
tr := &fakeTransport{status: http.StatusOK, respBody: okRefreshResponse(makeJWT(now.Add(2*time.Hour)), "rt-2")}
r := newTestRefresher(store, tr, now)
const n = 16
var wg sync.WaitGroup
wg.Add(n)
for i := 0; i < n; i++ {
go func() {
defer wg.Done()
r.refreshOnce(context.Background())
}()
}
wg.Wait()
if tr.callCount() != 1 {
t.Errorf("single-flight: %d OAuth POSTs across %d concurrent calls, want exactly 1", tr.callCount(), n)
}
if atomic.LoadInt32(&store.puts) != 1 {
t.Errorf("single-flight: %d write-backs, want exactly 1", store.puts)
}
}
// TestRefreshOnce_PostsExactlyOnceToOAuthEndpoint: when it DOES refresh, the
// single POST goes to the OAuth token URL with the refresh_token grant body.
func TestRefreshOnce_PostsExactlyOnceToOAuthEndpoint(t *testing.T) {
now := time.Now()
store := newFakeStore()
store.values[CodexAuthSecretKey] = authBlob(makeJWT(now.Add(1*time.Minute)), "rt-secret")
tr := &fakeTransport{status: http.StatusOK, respBody: okRefreshResponse(makeJWT(now.Add(2*time.Hour)), "rt-2")}
r := newTestRefresher(store, tr, now)
r.refreshOnce(context.Background())
if tr.callCount() != 1 {
t.Fatalf("%d POSTs, want exactly 1", tr.callCount())
}
if tr.urls[0] != oauthTokenURL {
t.Errorf("POST URL = %q, want %q", tr.urls[0], oauthTokenURL)
}
if tr.methods[0] != http.MethodPost {
t.Errorf("method = %q, want POST", tr.methods[0])
}
var body map[string]string
if err := json.Unmarshal([]byte(tr.bodies[0]), &body); err != nil {
t.Fatalf("request body not json: %v (%s)", err, tr.bodies[0])
}
if body["grant_type"] != "refresh_token" {
t.Errorf("grant_type = %q, want refresh_token", body["grant_type"])
}
if body["refresh_token"] != "rt-secret" {
t.Errorf("refresh_token = %q, want rt-secret", body["refresh_token"])
}
if body["client_id"] != codexOAuthClientID {
t.Errorf("client_id = %q, want %q", body["client_id"], codexOAuthClientID)
}
}
// TestRefreshOnce_ReadErrorSkips: a store read error is a transient skip (no
// POST, no permanent latch).
func TestRefreshOnce_ReadErrorSkips(t *testing.T) {
store := newFakeStore()
store.getErr = fmt.Errorf("db down")
tr := &fakeTransport{}
r := newTestRefresher(store, tr, time.Now())
if perm := r.refreshOnce(context.Background()); perm {
t.Errorf("read error: permanentFailure=true, want false")
}
if tr.callCount() != 0 {
t.Errorf("read error: %d POSTs, want 0", tr.callCount())
}
}
// TestMergeTokens_PreservesOtherFields proves the rotated write-back keeps every
// non-token field and does not clobber id_token with an empty rotated value.
func TestMergeTokens_PreservesOtherFields(t *testing.T) {
blob := authBlob("old-at", "old-rt")
out, err := mergeTokens(blob, oauthTokens{AccessToken: "new-at", RefreshToken: "new-rt"}) // no id_token
if err != nil {
t.Fatalf("mergeTokens: %v", err)
}
tokens, err := parseTokens(out)
if err != nil {
t.Fatalf("parse merged: %v", err)
}
if tokens.AccessToken != "new-at" || tokens.RefreshToken != "new-rt" {
t.Errorf("merged tokens = %+v, want new-at/new-rt", tokens)
}
if tokens.IDToken != "id-original" {
t.Errorf("empty rotated id_token clobbered the original: got %q, want id-original", tokens.IDToken)
}
if !strings.Contains(out, "last_refresh") {
t.Errorf("merge dropped preserved field: %s", out)
}
}
+25 -14
View File
@@ -334,28 +334,39 @@ func (h *WorkspaceHandler) ProxyA2A(c *gin.Context) {
c.Data(status, "application/json", respBody)
}
// checkWorkspaceBudget returns a proxyA2AError with 402 when the workspace
// has a budget_limit set and monthly_spend has reached or exceeded it.
// DB errors are logged and treated as fail-open — a budget check failure
// must not block legitimate A2A traffic.
// checkWorkspaceBudget returns a proxyA2AError with 402 when the workspace has
// exceeded ANY of its configured per-period budget limits (hourly/daily/weekly/
// monthly — see budget_periods.go). Per-period spend is the rolling-window sum
// over the workspace_spend_events ledger. DB errors are logged and treated as
// fail-open — a budget check failure must not block legitimate A2A traffic.
func (h *WorkspaceHandler) checkWorkspaceBudget(ctx context.Context, workspaceID string) *proxyA2AError {
var budgetLimit sql.NullInt64
var monthlySpend int64
err := db.DB.QueryRowContext(ctx,
`SELECT budget_limit, COALESCE(monthly_spend, 0) FROM workspaces WHERE id = $1`,
var limitsRaw []byte
if err := db.DB.QueryRowContext(ctx,
`SELECT COALESCE(budget_limits, '{}'::jsonb) FROM workspaces WHERE id = $1`,
workspaceID,
).Scan(&budgetLimit, &monthlySpend)
if err != nil {
).Scan(&limitsRaw); err != nil {
if err != sql.ErrNoRows {
log.Printf("ProxyA2A: budget check failed for %s: %v", workspaceID, err)
}
return nil // fail-open
}
if budgetLimit.Valid && monthlySpend >= budgetLimit.Int64 {
log.Printf("ProxyA2A: budget exceeded for %s (spend=%d limit=%d)", workspaceID, monthlySpend, budgetLimit.Int64)
limits := parseBudgetLimits(limitsRaw)
if len(limits) == 0 {
return nil // no limits configured
}
spend, err := spendByPeriod(ctx, db.DB, workspaceID)
if err != nil {
log.Printf("ProxyA2A: budget spend query failed for %s: %v", workspaceID, err)
return nil // fail-open
}
if over := exceededPeriods(limits, spend); len(over) > 0 {
log.Printf("ProxyA2A: budget exceeded for %s (periods=%v limits=%v spend=%v)", workspaceID, over, limits, spend)
return &proxyA2AError{
Status: http.StatusPaymentRequired,
Response: gin.H{"error": "workspace budget limit exceeded"},
Status: http.StatusPaymentRequired,
Response: gin.H{
"error": "workspace budget limit exceeded",
"exceeded_periods": over,
},
}
}
return nil
@@ -16,9 +16,9 @@ import (
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/models"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/provisioner"
"github.com/DATA-DOG/go-sqlmock"
"github.com/gin-gonic/gin"
)
@@ -2117,6 +2117,10 @@ func (f *fakeCPProv) Stop(_ context.Context, _ string) error {
f.stopCalls++
return nil
}
func (f *fakeCPProv) StopAndPrune(_ context.Context, _ string) error {
f.stopCalls++
return nil
}
func (f *fakeCPProv) GetConsoleOutput(_ context.Context, _ string) (string, error) {
return "", nil
}
@@ -2,8 +2,6 @@ package handlers
import (
"context"
"database/sql"
"errors"
"testing"
"github.com/DATA-DOG/go-sqlmock"
@@ -113,125 +111,3 @@ func TestExtractExpiresInSeconds(t *testing.T) {
})
}
}
// TestQueueStatusByID_HappyPath verifies the full projection including optional
// nullable fields and response_body surfacing when status == completed.
func TestQueueStatusByID_HappyPath(t *testing.T) {
mock := setupTestDB(t)
queueID := "queue-789"
mock.ExpectQuery(`SELECT\s+q\.id,\s+q\.workspace_id,\s+q\.status,\s+q\.priority,\s+q\.attempts,\s+q\.last_error,\s+q\.enqueued_at::text,\s+q\.dispatched_at::text,\s+q\.completed_at::text,\s+q\.expires_at::text,\s+al\.response_body::text\s+FROM a2a_queue q\s+LEFT JOIN activity_logs al`).
WithArgs(queueID).
WillReturnRows(sqlmock.NewRows([]string{
"id", "workspace_id", "status", "priority", "attempts",
"last_error", "enqueued_at", "dispatched_at", "completed_at", "expires_at",
"response_body",
}).AddRow(
queueID, "ws-target", "completed", 50, 2,
"previous error", "2026-05-28T10:00:00Z", "2026-05-28T10:01:00Z", "2026-05-28T10:02:00Z", "2026-05-28T11:00:00Z",
[]byte(`{"result":"ok"}`),
))
qs, err := QueueStatusByID(context.Background(), queueID)
if err != nil {
t.Fatalf("QueueStatusByID returned error: %v", err)
}
if qs.ID != queueID {
t.Errorf("ID = %q, want %q", qs.ID, queueID)
}
if qs.Status != "completed" {
t.Errorf("Status = %q, want completed", qs.Status)
}
if qs.LastError == nil || *qs.LastError != "previous error" {
t.Errorf("LastError = %v, want 'previous error'", qs.LastError)
}
if qs.DispatchedAt == nil || *qs.DispatchedAt != "2026-05-28T10:01:00Z" {
t.Errorf("DispatchedAt = %v", qs.DispatchedAt)
}
if qs.CompletedAt == nil || *qs.CompletedAt != "2026-05-28T10:02:00Z" {
t.Errorf("CompletedAt = %v", qs.CompletedAt)
}
if qs.ExpiresAt == nil || *qs.ExpiresAt != "2026-05-28T11:00:00Z" {
t.Errorf("ExpiresAt = %v", qs.ExpiresAt)
}
if string(qs.ResponseBody) != `{"result":"ok"}` {
t.Errorf("ResponseBody = %q", qs.ResponseBody)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet expectations: %v", err)
}
}
// TestQueueStatusByID_NoRows returns sql.ErrNoRows when the queue id does not exist.
func TestQueueStatusByID_NoRows(t *testing.T) {
mock := setupTestDB(t)
queueID := "queue-missing"
mock.ExpectQuery(`SELECT\s+q\.id,\s+q\.workspace_id,\s+q\.status,\s+q\.priority,\s+q\.attempts,\s+q\.last_error,\s+q\.enqueued_at::text,\s+q\.dispatched_at::text,\s+q\.completed_at::text,\s+q\.expires_at::text,\s+al\.response_body::text\s+FROM a2a_queue q\s+LEFT JOIN activity_logs`).
WithArgs(queueID).
WillReturnRows(sqlmock.NewRows([]string{
"id", "workspace_id", "status", "priority", "attempts",
"last_error", "enqueued_at", "dispatched_at", "completed_at", "expires_at",
"response_body",
}))
_, err := QueueStatusByID(context.Background(), queueID)
if !errors.Is(err, sql.ErrNoRows) {
t.Fatalf("expected sql.ErrNoRows, got %v", err)
}
}
// TestQueueStatusByID_NullOptionals confirms that NULL dispatched_at / completed_at /
// expires_at / last_error are projected as nil pointers, and response_body is NOT
// included when status != completed.
func TestQueueStatusByID_NullOptionals(t *testing.T) {
mock := setupTestDB(t)
queueID := "queue-nulls"
mock.ExpectQuery(`SELECT\s+q\.id,\s+q\.workspace_id,\s+q\.status,\s+q\.priority,\s+q\.attempts,\s+q\.last_error,\s+q\.enqueued_at::text,\s+q\.dispatched_at::text,\s+q\.completed_at::text,\s+q\.expires_at::text,\s+al\.response_body::text\s+FROM a2a_queue q\s+LEFT JOIN activity_logs`).
WithArgs(queueID).
WillReturnRows(sqlmock.NewRows([]string{
"id", "workspace_id", "status", "priority", "attempts",
"last_error", "enqueued_at", "dispatched_at", "completed_at", "expires_at",
"response_body",
}).AddRow(
queueID, "ws-target", "queued", 50, 0,
nil, "2026-05-28T10:00:00Z", nil, nil, nil,
nil,
))
qs, err := QueueStatusByID(context.Background(), queueID)
if err != nil {
t.Fatalf("QueueStatusByID returned error: %v", err)
}
if qs.LastError != nil {
t.Errorf("LastError = %v, want nil", qs.LastError)
}
if qs.DispatchedAt != nil {
t.Errorf("DispatchedAt = %v, want nil", qs.DispatchedAt)
}
if qs.CompletedAt != nil {
t.Errorf("CompletedAt = %v, want nil", qs.CompletedAt)
}
if qs.ExpiresAt != nil {
t.Errorf("ExpiresAt = %v, want nil", qs.ExpiresAt)
}
if qs.ResponseBody != nil {
t.Errorf("ResponseBody = %q, want nil for non-completed status", qs.ResponseBody)
}
}
// TestQueueStatusByID_DBError surfaces the underlying error on unexpected failure.
func TestQueueStatusByID_DBError(t *testing.T) {
mock := setupTestDB(t)
queueID := "queue-dberr"
mock.ExpectQuery(`SELECT\s+q\.id,\s+q\.workspace_id,\s+q\.status,\s+q\.priority,\s+q\.attempts,\s+q\.last_error,\s+q\.enqueued_at::text,\s+q\.dispatched_at::text,\s+q\.completed_at::text,\s+q\.expires_at::text,\s+al\.response_body::text\s+FROM a2a_queue q\s+LEFT JOIN activity_logs`).
WithArgs(queueID).
WillReturnError(errors.New("disk full"))
_, err := QueueStatusByID(context.Background(), queueID)
if err == nil || errors.Is(err, sql.ErrNoRows) {
t.Fatalf("expected DB error, got %v", err)
}
}
@@ -12,15 +12,14 @@ package handlers
import (
"context"
"database/sql"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/db"
"github.com/DATA-DOG/go-sqlmock"
"github.com/alicebob/miniredis/v2"
)
@@ -210,10 +209,12 @@ func drainSetup(t *testing.T, workspaceID string) (sqlmock.Sqlmock, *WorkspaceHa
// Named distinctly from handlers_test.go's expectBudgetCheck (which uses MatchPsql
// escaped-regex and cannot be reused with QueryMatcherEqual tests).
func expectQueueBudgetCheck(mock sqlmock.Sqlmock, workspaceID string) {
// Multi-period (#49): exact-match the budget_limits read; "{}" → no limits →
// checkWorkspaceBudget returns early (no spend query).
mock.ExpectQuery(
"SELECT budget_limit, COALESCE(monthly_spend, 0) FROM workspaces WHERE id = $1",
"SELECT COALESCE(budget_limits, '{}'::jsonb) FROM workspaces WHERE id = $1",
).WithArgs(workspaceID).
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte("{}")))
}
// seedRedisURL puts the agent server URL into the Redis cache so resolveAgentURL
@@ -521,40 +522,3 @@ func TestDrainQueueForWorkspace_ClaimGuarding_SecondDrainGetsEmpty(t *testing.T)
t.Errorf("unmet sqlmock expectations: %v", err)
}
}
// ──────────────────────────────────────────────────────────────────────────────
// QueueDepth
// ──────────────────────────────────────────────────────────────────────────────
func TestQueueDepth_HappyPath(t *testing.T) {
mock := setupTestDBForQueueTests(t)
wsID := "ws-depth-1"
mock.ExpectQuery("SELECT COUNT(*) FROM a2a_queue WHERE workspace_id = $1 AND status = 'queued'").
WithArgs(wsID).
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(7))
if got := QueueDepth(context.Background(), wsID); got != 7 {
t.Errorf("QueueDepth = %d, want 7", got)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet expectations: %v", err)
}
}
func TestQueueDepth_QueryError(t *testing.T) {
mock := setupTestDBForQueueTests(t)
wsID := "ws-depth-2"
mock.ExpectQuery("SELECT COUNT(*) FROM a2a_queue WHERE workspace_id = $1 AND status = 'queued'").
WithArgs(wsID).
WillReturnError(errors.New("conn lost"))
// Must return 0 (fail-open informational) rather than panic or propagate.
if got := QueueDepth(context.Background(), wsID); got != 0 {
t.Errorf("QueueDepth on error = %d, want 0", got)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet expectations: %v", err)
}
}
@@ -148,6 +148,125 @@ func (h *AdminSchedulesHealthHandler) Health(c *gin.Context) {
c.JSON(http.StatusOK, entries)
}
// orphanScheduleEntry is one row in the Orphans response.
type orphanScheduleEntry struct {
WorkspaceID string `json:"workspace_id"`
WorkspaceStatus string `json:"workspace_status"` // "removed" | "missing"
ScheduleID string `json:"schedule_id"`
ScheduleName string `json:"schedule_name"`
Source string `json:"source"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cron_expr"`
}
// Orphans handles GET /admin/schedules/orphans — the monitor surface for
// internal#2006. Health (above) reports only LIVE workspaces' schedules, so a
// schedule left on a removed/recreated workspace silently stops firing and
// never appears there. This endpoint lists exactly those orphans (workspace
// removed OR missing) so an operator/monitor can alert. Returns 200 + JSON
// array (empty when none). Auth via adminAuth() in router.go.
func (h *AdminSchedulesHealthHandler) Orphans(c *gin.Context) {
ctx := c.Request.Context()
rows, err := db.DB.QueryContext(ctx, `
SELECT s.workspace_id,
CASE WHEN w.id IS NULL THEN 'missing' ELSE 'removed' END AS ws_status,
s.id, s.name, COALESCE(s.source, ''), s.enabled, s.cron_expr
FROM workspace_schedules s
LEFT JOIN workspaces w ON w.id = s.workspace_id
WHERE w.id IS NULL OR w.status = 'removed'
ORDER BY s.name ASC
`)
if err != nil {
log.Printf("AdminSchedulesOrphans: query error: %v", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to query orphans"})
return
}
defer rows.Close()
out := make([]orphanScheduleEntry, 0)
for rows.Next() {
var e orphanScheduleEntry
if err := rows.Scan(&e.WorkspaceID, &e.WorkspaceStatus, &e.ScheduleID, &e.ScheduleName, &e.Source, &e.Enabled, &e.CronExpr); err != nil {
log.Printf("AdminSchedulesOrphans: scan error: %v", err)
continue
}
out = append(out, e)
}
if err := rows.Err(); err != nil {
log.Printf("AdminSchedulesOrphans: rows iteration error: %v", err)
}
c.JSON(http.StatusOK, out)
}
// ReapOrphans handles POST /admin/schedules/reap-orphans — the orphan cleaner
// (internal#2006). For every schedule bound to a removed/nonexistent workspace
// it re-points runtime-created schedules onto the live successor agent (matched
// by role+parent, falling back to name+parent) when one exists and doesn't
// already carry a same-named schedule; schedules with no live successor are
// disabled (enabled=false) so the scheduler stops firing into a dead workspace.
// Idempotent: re-running with no orphans is a no-op. Returns a summary count.
// Auth is enforced by the adminAuth() middleware registered in router.go.
func (h *AdminSchedulesHealthHandler) ReapOrphans(c *gin.Context) {
ctx := c.Request.Context()
// 1. Re-point runtime schedules onto a live successor (same role+parent,
// else same name+parent). Skip names already present on the successor.
repointed, err := db.DB.ExecContext(ctx, `
WITH orphan AS (
SELECT s.id, s.name, s.workspace_id, prev.role AS role, prev.parent_id AS parent_id
FROM workspace_schedules s
JOIN workspaces prev ON prev.id = s.workspace_id
WHERE prev.status = 'removed' AND s.source = 'runtime'
),
successor AS (
SELECT o.id AS schedule_id, o.name AS schedule_name,
(
SELECT w.id FROM workspaces w
WHERE w.status != 'removed'
AND w.parent_id IS NOT DISTINCT FROM o.parent_id
AND ((o.role IS NOT NULL AND w.role = o.role))
ORDER BY w.updated_at DESC NULLS LAST LIMIT 1
) AS live_id
FROM orphan o
)
UPDATE workspace_schedules s
SET workspace_id = su.live_id, updated_at = now()
FROM successor su
WHERE s.id = su.schedule_id
AND su.live_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM workspace_schedules t
WHERE t.workspace_id = su.live_id AND t.name = su.schedule_name
)
`)
if err != nil {
log.Printf("ReapOrphans: re-point error: %v", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "re-point failed"})
return
}
repointedN, _ := repointed.RowsAffected()
// 2. Disable any remaining schedules still bound to a removed/missing
// workspace (no live successor, or template schedules on a dead row).
disabled, err := db.DB.ExecContext(ctx, `
UPDATE workspace_schedules s
SET enabled = false, updated_at = now()
WHERE s.enabled = true
AND NOT EXISTS (
SELECT 1 FROM workspaces w
WHERE w.id = s.workspace_id AND w.status != 'removed'
)
`)
if err != nil {
log.Printf("ReapOrphans: disable error: %v", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "disable failed"})
return
}
disabledN, _ := disabled.RowsAffected()
log.Printf("ReapOrphans: re-pointed %d, disabled %d orphaned schedule(s)", repointedN, disabledN)
c.JSON(http.StatusOK, gin.H{"repointed": repointedN, "disabled": disabledN})
}
// classifyScheduleStatus returns the health status string for a schedule.
// - "never_run" — last_run_at is NULL (schedule has never fired)
// - "stale" — now - last_run_at > staleThreshold (and threshold > 0)
@@ -444,3 +444,72 @@ func TestAdminSchedulesHealth_ResponseFields(t *testing.T) {
t.Fatalf("unmet expectations: %v", err)
}
}
// ==================== Orphans + ReapOrphans (internal#2006) ====================
// TestAdminSchedulesOrphans verifies the monitor surface lists schedules bound
// to a removed/missing workspace (the recreate-orphan failure mode).
func TestAdminSchedulesOrphans(t *testing.T) {
mock := setupTestDB(t)
handler := NewAdminSchedulesHealthHandler()
mock.ExpectQuery(`LEFT JOIN workspaces`).
WillReturnRows(sqlmock.NewRows([]string{
"workspace_id", "ws_status", "id", "name", "source", "enabled", "cron_expr",
}).AddRow("dead-ws", "removed", "sched-1", "minimax-autonomous-tick", "runtime", false, "*/5 * * * *"))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Request = httptest.NewRequest("GET", "/admin/schedules/orphans", nil)
handler.Orphans(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp []orphanScheduleEntry
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
if len(resp) != 1 {
t.Fatalf("expected 1 orphan, got %d", len(resp))
}
if resp[0].ScheduleName != "minimax-autonomous-tick" || resp[0].WorkspaceStatus != "removed" || resp[0].Source != "runtime" {
t.Errorf("unexpected orphan entry: %+v", resp[0])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unmet expectations: %v", err)
}
}
// TestReapOrphans verifies the cleaner re-points runtime schedules onto a live
// successor then disables any remaining dead-bound schedules, returning counts.
func TestReapOrphans(t *testing.T) {
mock := setupTestDB(t)
handler := NewAdminSchedulesHealthHandler()
mock.ExpectExec(`UPDATE workspace_schedules s\s+SET workspace_id`).
WillReturnResult(sqlmock.NewResult(0, 2))
mock.ExpectExec(`UPDATE workspace_schedules s\s+SET enabled = false`).
WillReturnResult(sqlmock.NewResult(0, 1))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Request = httptest.NewRequest("POST", "/admin/schedules/reap-orphans", nil)
handler.ReapOrphans(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp map[string]int64
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
if resp["repointed"] != 2 || resp["disabled"] != 1 {
t.Errorf("expected repointed=2 disabled=1, got %+v", resp)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unmet expectations: %v", err)
}
}
+120 -77
View File
@@ -1,7 +1,9 @@
package handlers
import (
"context"
"database/sql"
"encoding/json"
"log"
"net/http"
@@ -12,42 +14,79 @@ import (
// BudgetHandler exposes per-workspace budget read/write endpoints.
// Routes (all behind WorkspaceAuth middleware):
//
// GET /workspaces/:id/budget — current budget_limit, monthly_spend, budget_remaining
// PATCH /workspaces/:id/budget — set or clear budget_limit
// GET /workspaces/:id/budget — per-period limits, spend, remaining
// PATCH /workspaces/:id/budget — set/clear per-period limits
//
// Multi-period (#49): the budget is now four independent rolling windows —
// hourly/daily/weekly/monthly (budget_periods.go is the SSOT for the set). The
// canonical config is workspaces.budget_limits (JSONB, USD cents per period);
// per-period spend is the rolling-window sum over workspace_spend_events. The
// legacy single monthly budget_limit / monthly_spend are still emitted (and
// budget_limit kept in sync to the monthly period) for back-compat with
// pre-deploy canvas/agent builds during the rollout window.
type BudgetHandler struct{}
func NewBudgetHandler() *BudgetHandler { return &BudgetHandler{} }
// budgetResponse is the canonical JSON shape for both GET and PATCH responses.
// periodBudget is the per-period view: configured ceiling (null = no limit),
// rolling-window spend, and remaining headroom (null when no limit; may go
// negative so callers see how far over a period is).
type periodBudget struct {
Limit *int64 `json:"limit"`
Spend int64 `json:"spend"`
Remaining *int64 `json:"remaining"`
}
// budgetResponse is the canonical JSON shape for GET and PATCH.
type budgetResponse struct {
// BudgetLimit is the monthly spend ceiling in USD cents (null = no limit).
// budget_limit=500 means $5.00/month.
BudgetLimit *int64 `json:"budget_limit"`
// MonthlySpend is the agent's self-reported accumulated LLM API spend
// for the current month (USD cents). Incremented via heartbeat.
MonthlySpend int64 `json:"monthly_spend"`
// BudgetRemaining is null when BudgetLimit is null, otherwise
// max(0, budget_limit - monthly_spend). Can be negative — we store the
// actual value so callers can see how far over-budget a workspace is.
// Periods is keyed by BudgetPeriod ("hourly"/"daily"/"weekly"/"monthly").
Periods map[string]periodBudget `json:"periods"`
// --- back-compat (monthly), for pre-multi-period clients ---
BudgetLimit *int64 `json:"budget_limit"`
MonthlySpend int64 `json:"monthly_spend"`
BudgetRemaining *int64 `json:"budget_remaining"`
}
// buildBudgetResponse assembles the per-period view from the stored limits +
// the ledger spend. Single place so GET and PATCH return identical shapes.
func buildBudgetResponse(ctx context.Context, workspaceID string, limitsRaw []byte) (budgetResponse, error) {
limits := parseBudgetLimits(limitsRaw)
spend, err := spendByPeriod(ctx, db.DB, workspaceID)
if err != nil {
return budgetResponse{}, err
}
periods := make(map[string]periodBudget, len(budgetPeriods))
for _, def := range budgetPeriods {
pb := periodBudget{Spend: spend[def.Name]}
if lim, ok := limits[def.Name]; ok {
l := lim
pb.Limit = &l
r := lim - spend[def.Name]
pb.Remaining = &r
}
periods[string(def.Name)] = pb
}
resp := budgetResponse{Periods: periods, MonthlySpend: spend[PeriodMonthly]}
if m := periods[string(PeriodMonthly)]; m.Limit != nil {
resp.BudgetLimit = m.Limit
resp.BudgetRemaining = m.Remaining
}
return resp, nil
}
// GetBudget handles GET /workspaces/:id/budget.
// Returns the workspace's current budget ceiling, accumulated spend, and
// computed remaining headroom. Both budget_limit and budget_remaining are
// null when no limit has been configured for the workspace.
func (h *BudgetHandler) GetBudget(c *gin.Context) {
workspaceID := c.Param("id")
ctx := c.Request.Context()
var budgetLimit sql.NullInt64
var monthlySpend int64
var limitsRaw []byte
err := db.DB.QueryRowContext(ctx,
`SELECT budget_limit, COALESCE(monthly_spend, 0)
`SELECT COALESCE(budget_limits, '{}'::jsonb)
FROM workspaces
WHERE id = $1 AND status != 'removed'`,
workspaceID,
).Scan(&budgetLimit, &monthlySpend)
).Scan(&limitsRaw)
if err == sql.ErrNoRows {
c.JSON(http.StatusNotFound, gin.H{"error": "workspace not found"})
return
@@ -58,66 +97,80 @@ func (h *BudgetHandler) GetBudget(c *gin.Context) {
return
}
resp := budgetResponse{
MonthlySpend: monthlySpend,
resp, err := buildBudgetResponse(ctx, workspaceID, limitsRaw)
if err != nil {
log.Printf("GetBudget: spend query failed for %s: %v", workspaceID, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "query failed"})
return
}
if budgetLimit.Valid {
limit := budgetLimit.Int64
resp.BudgetLimit = &limit
remaining := limit - monthlySpend
resp.BudgetRemaining = &remaining
}
c.JSON(http.StatusOK, resp)
}
// PatchBudget handles PATCH /workspaces/:id/budget.
// Accepts {"budget_limit": <int64>} to set a new ceiling, or
// {"budget_limit": null} to remove an existing ceiling.
// Returns the updated budget state in the same shape as GetBudget.
// PatchBudget handles PATCH /workspaces/:id/budget. Accepts EITHER the
// multi-period shape
//
// {"budget_limits": {"hourly": 100, "daily": null, "weekly": 500, "monthly": 2000}}
//
// (a per-period value of null/absent clears that period; a positive int sets it)
// OR the legacy single-monthly shape {"budget_limit": 2000} / {"budget_limit": null}.
func (h *BudgetHandler) PatchBudget(c *gin.Context) {
workspaceID := c.Param("id")
ctx := c.Request.Context()
// We need to distinguish between "field absent" and "field = null",
// so we unmarshal into a raw map first.
var raw map[string]interface{}
var raw map[string]json.RawMessage
if err := c.ShouldBindJSON(&raw); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
return
}
budgetLimitRaw, ok := raw["budget_limit"]
if !ok {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limit field is required"})
_, hasLimits := raw["budget_limits"]
_, hasLegacy := raw["budget_limit"]
if !hasLimits && !hasLegacy {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limits or budget_limit field is required"})
return
}
// Validate and convert the value. JSON numbers decode as float64.
var budgetArg interface{} // nil → SQL NULL, int64 → new ceiling
if budgetLimitRaw != nil {
switch v := budgetLimitRaw.(type) {
case float64:
if v < 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limit must be >= 0 (USD cents)"})
limits := make(map[BudgetPeriod]int64, len(budgetPeriods))
known := make(map[string]bool, len(budgetPeriods))
for _, def := range budgetPeriods {
known[string(def.Name)] = true
}
if hasLimits {
var m map[string]*int64
if err := json.Unmarshal(raw["budget_limits"], &m); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limits must be an object of period→int|null"})
return
}
for k, v := range m {
if !known[k] {
c.JSON(http.StatusBadRequest, gin.H{"error": "unknown budget period: " + k + " (allowed: hourly, daily, weekly, monthly)"})
return
}
cv := int64(v)
budgetArg = cv
case int64:
if v < 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limit must be >= 0 (USD cents)"})
if v == nil {
continue // clear this period (null = no limit)
}
if *v < 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget limit for " + k + " must be >= 0 (USD cents)"})
return
}
budgetArg = v
default:
limits[BudgetPeriod(k)] = *v // 0 is valid = block-all for this period
}
} else { // legacy single-monthly
var v *int64
if err := json.Unmarshal(raw["budget_limit"], &v); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limit must be an integer (USD cents) or null"})
return
}
if v != nil {
if *v < 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "budget_limit must be >= 0 (USD cents)"})
return
}
limits[PeriodMonthly] = *v // 0 is valid = block-all (legacy semantics)
}
}
// budgetArg == nil means "clear the ceiling"
// Existence check — return 404 for non-existent / removed workspaces.
// Existence check — 404 for non-existent / removed workspaces.
var exists bool
if err := db.DB.QueryRowContext(ctx,
`SELECT EXISTS(SELECT 1 FROM workspaces WHERE id = $1 AND status != 'removed')`,
@@ -127,38 +180,28 @@ func (h *BudgetHandler) PatchBudget(c *gin.Context) {
return
}
// Persist: budget_limits is the SSOT; keep the legacy budget_limit column
// synced to the monthly period so pre-deploy enforcement paths stay coherent
// during the rollout window.
var legacyMonthly interface{}
if m, ok := limits[PeriodMonthly]; ok {
legacyMonthly = m
}
encoded := encodeBudgetLimits(limits)
if _, err := db.DB.ExecContext(ctx,
`UPDATE workspaces SET budget_limit = $2, updated_at = now() WHERE id = $1`,
workspaceID, budgetArg,
`UPDATE workspaces SET budget_limits = $2, budget_limit = $3, updated_at = now() WHERE id = $1`,
workspaceID, encoded, legacyMonthly,
); err != nil {
log.Printf("PatchBudget: update failed for %s: %v", workspaceID, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "update failed"})
return
}
// Re-read the current state so the response reflects exactly what is in
// the DB, including the monthly_spend the agent has already accumulated.
var newLimit sql.NullInt64
var monthlySpend int64
if err := db.DB.QueryRowContext(ctx,
`SELECT budget_limit, COALESCE(monthly_spend, 0) FROM workspaces WHERE id = $1`,
workspaceID,
).Scan(&newLimit, &monthlySpend); err != nil {
resp, err := buildBudgetResponse(ctx, workspaceID, encoded)
if err != nil {
log.Printf("PatchBudget: re-read failed for %s: %v", workspaceID, err)
// Still success — just omit the echo.
c.JSON(http.StatusOK, gin.H{"status": "updated"})
return
}
resp := budgetResponse{
MonthlySpend: monthlySpend,
}
if newLimit.Valid {
limit := newLimit.Int64
resp.BudgetLimit = &limit
remaining := limit - monthlySpend
resp.BudgetRemaining = &remaining
}
c.JSON(http.StatusOK, resp)
}
@@ -0,0 +1,160 @@
package handlers
import (
"context"
"database/sql"
"encoding/json"
"strconv"
"time"
)
// budget_periods.go — SINGLE SOURCE OF TRUTH for the multi-period per-workspace
// LLM budget (#49 follow-up). The supported periods, their rolling windows, the
// per-period spend computation (from the workspace_spend_events ledger), and the
// over-budget decision all live here so the config endpoint (GetBudget/PatchBudget),
// the display, and enforcement (checkWorkspaceBudget) can never drift.
//
// Spend model: the heartbeat records each observed spend INCREMENT into
// workspace_spend_events (recordSpendDelta). Per-period spend is a rolling-window
// SUM over that ledger — so the SERVER owns windowing (the agent keeps reporting
// its cumulative figure unchanged). Rolling (not calendar) windows: no fragile
// month-boundary reset, and "monthly" = a 30-day trailing window.
// BudgetPeriod is one of the supported rolling budget windows.
type BudgetPeriod string
const (
PeriodHourly BudgetPeriod = "hourly"
PeriodDaily BudgetPeriod = "daily"
PeriodWeekly BudgetPeriod = "weekly"
PeriodMonthly BudgetPeriod = "monthly"
)
// budgetPeriodDef pairs a period with its rolling window.
type budgetPeriodDef struct {
Name BudgetPeriod
Window time.Duration
}
// budgetPeriods is the canonical ordered list. ADD A PERIOD = one line here;
// every consumer iterates this slice, so nothing else needs to change.
var budgetPeriods = []budgetPeriodDef{
{PeriodHourly, time.Hour},
{PeriodDaily, 24 * time.Hour},
{PeriodWeekly, 7 * 24 * time.Hour},
{PeriodMonthly, 30 * 24 * time.Hour}, // rolling 30-day window
}
// spendLedgerRetention bounds the ledger: rows older than the largest window
// (+ slack) are never read, so the recorder opportunistically prunes them.
var spendLedgerRetention = 35 * 24 * time.Hour
// parseBudgetLimits decodes the workspaces.budget_limits JSONB into a map of
// period → limit (USD cents). A limit of ZERO is valid and means "block all
// spend for that period" (a $0 ceiling); absent / null / negative / unknown
// keys mean "no limit for that period". Tolerant of a NULL/empty column.
func parseBudgetLimits(raw []byte) map[BudgetPeriod]int64 {
out := make(map[BudgetPeriod]int64, len(budgetPeriods))
if len(raw) == 0 {
return out
}
var m map[string]*int64
if err := json.Unmarshal(raw, &m); err != nil {
return out
}
for _, def := range budgetPeriods {
if v, ok := m[string(def.Name)]; ok && v != nil && *v >= 0 {
out[def.Name] = *v
}
}
return out
}
// encodeBudgetLimits renders a period→limit map back to the canonical JSONB
// shape, keeping only KNOWN periods with a non-negative limit (0 = block-all is
// preserved; a period absent from the map = no limit). Always returns valid JSON.
func encodeBudgetLimits(limits map[BudgetPeriod]int64) []byte {
m := make(map[string]int64, len(limits))
for _, def := range budgetPeriods {
if v, ok := limits[def.Name]; ok && v >= 0 {
m[string(def.Name)] = v
}
}
b, err := json.Marshal(m)
if err != nil {
return []byte("{}")
}
return b
}
// recordSpendDelta appends a positive spend increment to the ledger and
// opportunistically prunes rows past the retention horizon for this workspace.
// No-op for delta <= 0. Errors are returned for the caller to log (non-fatal).
func recordSpendDelta(ctx context.Context, q *sql.DB, workspaceID string, deltaCents int64) error {
if deltaCents <= 0 {
return nil
}
if _, err := q.ExecContext(ctx,
`INSERT INTO workspace_spend_events (workspace_id, delta_cents) VALUES ($1, $2)`,
workspaceID, deltaCents,
); err != nil {
return err
}
// Opportunistic prune (cheap; index-backed). Best-effort — ignore error.
_, _ = q.ExecContext(ctx,
`DELETE FROM workspace_spend_events
WHERE workspace_id = $1 AND occurred_at < now() - $2::interval`,
workspaceID, pgInterval(spendLedgerRetention),
)
return nil
}
// spendByPeriod returns the rolling-window spend (USD cents) for every period,
// computed in a SINGLE query over the ledger. The outer predicate bounds to the
// largest window; per-period FILTERs sum each sub-window. A period with no ledger
// rows reports 0. This is THE spend computation — used by both display + enforcement.
func spendByPeriod(ctx context.Context, q *sql.DB, workspaceID string) (map[BudgetPeriod]int64, error) {
out := make(map[BudgetPeriod]int64, len(budgetPeriods))
for _, def := range budgetPeriods {
out[def.Name] = 0
}
row := q.QueryRowContext(ctx, `
SELECT
COALESCE(SUM(delta_cents) FILTER (WHERE occurred_at > now() - interval '1 hour'), 0),
COALESCE(SUM(delta_cents) FILTER (WHERE occurred_at > now() - interval '24 hours'), 0),
COALESCE(SUM(delta_cents) FILTER (WHERE occurred_at > now() - interval '7 days'), 0),
COALESCE(SUM(delta_cents) FILTER (WHERE occurred_at > now() - interval '30 days'), 0)
FROM workspace_spend_events
WHERE workspace_id = $1 AND occurred_at > now() - interval '30 days'
`, workspaceID)
var h, d, w, mo int64
if err := row.Scan(&h, &d, &w, &mo); err != nil {
return out, err
}
out[PeriodHourly], out[PeriodDaily], out[PeriodWeekly], out[PeriodMonthly] = h, d, w, mo
return out, nil
}
// exceededPeriods is PURE: given the configured limits and observed spend, it
// returns the periods whose spend has reached/exceeded their limit (in
// budgetPeriods order). Only periods WITH a positive limit are considered.
// Used by enforcement to decide whether to block.
func exceededPeriods(limits map[BudgetPeriod]int64, spend map[BudgetPeriod]int64) []BudgetPeriod {
var over []BudgetPeriod
for _, def := range budgetPeriods {
limit, ok := limits[def.Name]
if !ok {
continue // no limit configured for this period
}
// limit >= 0 is a real ceiling (0 = block-all). spend >= limit → over.
if spend[def.Name] >= limit {
over = append(over, def.Name)
}
}
return over
}
// pgInterval renders a Go duration as a Postgres-interval string ("N seconds").
func pgInterval(d time.Duration) string {
return strconv.FormatInt(int64(d.Seconds()), 10) + " seconds"
}
@@ -0,0 +1,99 @@
package handlers
import (
"reflect"
"testing"
)
// Pure-logic tests for the multi-period budget SSOT (budget_periods.go). The
// DB-touching helpers (spendByPeriod / recordSpendDelta) are exercised via the
// handler sqlmock tests; here we pin the parsing + the over-budget decision,
// which is where the per-period semantics actually live.
func TestParseBudgetLimits(t *testing.T) {
cases := []struct {
name string
raw string
want map[BudgetPeriod]int64
}{
{"empty", "", map[BudgetPeriod]int64{}},
{"empty-object", "{}", map[BudgetPeriod]int64{}},
{"all-four", `{"hourly":100,"daily":200,"weekly":300,"monthly":400}`,
map[BudgetPeriod]int64{PeriodHourly: 100, PeriodDaily: 200, PeriodWeekly: 300, PeriodMonthly: 400}},
{"null-dropped-zero-kept", `{"hourly":null,"daily":0,"weekly":500}`,
map[BudgetPeriod]int64{PeriodDaily: 0, PeriodWeekly: 500}}, // 0 = block-all, kept
{"negative-dropped", `{"monthly":-5}`, map[BudgetPeriod]int64{}},
{"unknown-key-ignored", `{"yearly":999,"daily":10}`, map[BudgetPeriod]int64{PeriodDaily: 10}},
{"malformed-json", `{not json`, map[BudgetPeriod]int64{}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := parseBudgetLimits([]byte(tc.raw))
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("parseBudgetLimits(%q) = %v, want %v", tc.raw, got, tc.want)
}
})
}
}
func TestEncodeBudgetLimits_RoundTrip(t *testing.T) {
in := map[BudgetPeriod]int64{PeriodHourly: 100, PeriodMonthly: 400}
enc := encodeBudgetLimits(in)
got := parseBudgetLimits(enc)
if !reflect.DeepEqual(got, in) {
t.Errorf("round-trip: encode→parse = %v, want %v (enc=%s)", got, in, enc)
}
// unknown periods dropped; 0 (block-all) kept
enc2 := encodeBudgetLimits(map[BudgetPeriod]int64{PeriodDaily: 0, "yearly": 9})
if got := parseBudgetLimits(enc2); !reflect.DeepEqual(got, map[BudgetPeriod]int64{PeriodDaily: 0}) {
t.Errorf("encode kept 0/dropped unknown: parse(%s) = %v, want {daily:0}", enc2, got)
}
}
func TestExceededPeriods(t *testing.T) {
cases := []struct {
name string
limits map[BudgetPeriod]int64
spend map[BudgetPeriod]int64
want []BudgetPeriod
}{
{"no-limits", map[BudgetPeriod]int64{}, map[BudgetPeriod]int64{PeriodHourly: 999}, nil},
{"zero-limit-blocks-all", map[BudgetPeriod]int64{PeriodHourly: 0}, map[BudgetPeriod]int64{PeriodHourly: 0}, []BudgetPeriod{PeriodHourly}},
{"under-all", map[BudgetPeriod]int64{PeriodDaily: 100}, map[BudgetPeriod]int64{PeriodDaily: 50}, nil},
{"at-limit-is-exceeded", map[BudgetPeriod]int64{PeriodDaily: 100}, map[BudgetPeriod]int64{PeriodDaily: 100}, []BudgetPeriod{PeriodDaily}},
{"over-limit", map[BudgetPeriod]int64{PeriodHourly: 10}, map[BudgetPeriod]int64{PeriodHourly: 11}, []BudgetPeriod{PeriodHourly}},
{"only-hourly-over", map[BudgetPeriod]int64{PeriodHourly: 10, PeriodMonthly: 1000},
map[BudgetPeriod]int64{PeriodHourly: 50, PeriodMonthly: 200}, []BudgetPeriod{PeriodHourly}},
{"multiple-over-in-order", map[BudgetPeriod]int64{PeriodHourly: 10, PeriodWeekly: 100},
map[BudgetPeriod]int64{PeriodHourly: 99, PeriodWeekly: 100}, []BudgetPeriod{PeriodHourly, PeriodWeekly}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := exceededPeriods(tc.limits, tc.spend)
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("exceededPeriods(%v,%v) = %v, want %v", tc.limits, tc.spend, got, tc.want)
}
})
}
}
// TestBudgetPeriods_AllReachable guards the SSOT list: every declared period has
// a positive window and a unique name (a typo'd duplicate would silently break
// per-period accounting).
func TestBudgetPeriods_Wellformed(t *testing.T) {
seen := map[BudgetPeriod]bool{}
for _, d := range budgetPeriods {
if d.Window <= 0 {
t.Errorf("period %s has non-positive window %v", d.Name, d.Window)
}
if seen[d.Name] {
t.Errorf("duplicate period name %s", d.Name)
}
seen[d.Name] = true
}
for _, p := range []BudgetPeriod{PeriodHourly, PeriodDaily, PeriodWeekly, PeriodMonthly} {
if !seen[p] {
t.Errorf("period %s missing from budgetPeriods SSOT list", p)
}
}
}
+191 -89
View File
@@ -12,15 +12,25 @@ import (
"github.com/gin-gonic/gin"
)
// Multi-period budget (#49): GET/PATCH now read workspaces.budget_limits (jsonb)
// and compute per-period spend from the workspace_spend_events ledger
// (spendByPeriod — matched here by the "FROM workspace_spend_events" fragment).
// The legacy budget_limit/monthly_spend response fields are still emitted
// (monthly period) for rollout back-compat, and the legacy {"budget_limit":N}
// PATCH shape still works.
// spendRows builds the 4-column row spendByPeriod scans (hourly,daily,weekly,monthly).
func spendRows(h, d, w, m int64) *sqlmock.Rows {
return sqlmock.NewRows([]string{"h", "d", "w", "mo"}).AddRow(h, d, w, m)
}
// ==================== GET /workspaces/:id/budget ====================
// TestBudgetGet_NotFound verifies that GET /budget returns 404 for an unknown
// workspace ID (ErrNoRows from the budget query).
func TestBudgetGet_NotFound(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\)`).
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-not-there").
WillReturnError(sql.ErrNoRows)
@@ -29,8 +39,7 @@ func TestBudgetGet_NotFound(t *testing.T) {
c.Params = gin.Params{{Key: "id", Value: "ws-not-there"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-not-there/budget", nil)
h := NewBudgetHandler()
h.GetBudget(c)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusNotFound {
t.Errorf("expected 404, got %d: %s", w.Code, w.Body.String())
@@ -40,12 +49,11 @@ func TestBudgetGet_NotFound(t *testing.T) {
}
}
// TestBudgetGet_DBError verifies that a non-ErrNoRows DB error returns 500.
func TestBudgetGet_DBError(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\)`).
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-db-err").
WillReturnError(sql.ErrConnDone)
@@ -54,8 +62,7 @@ func TestBudgetGet_DBError(t *testing.T) {
c.Params = gin.Params{{Key: "id", Value: "ws-db-err"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-db-err/budget", nil)
h := NewBudgetHandler()
h.GetBudget(c)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusInternalServerError {
t.Errorf("expected 500, got %d: %s", w.Code, w.Body.String())
@@ -65,24 +72,23 @@ func TestBudgetGet_DBError(t *testing.T) {
}
}
// TestBudgetGet_NoLimit verifies that budget_limit and budget_remaining are
// null when the workspace has no budget ceiling configured.
func TestBudgetGet_NoLimit(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\)`).
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-free").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(nil, int64(42)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-free").
WillReturnRows(spendRows(0, 0, 0, 42))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-free"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-free/budget", nil)
h := NewBudgetHandler()
h.GetBudget(c)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
@@ -105,24 +111,23 @@ func TestBudgetGet_NoLimit(t *testing.T) {
}
}
// TestBudgetGet_WithLimit verifies that budget_limit, monthly_spend, and
// budget_remaining are all returned correctly when a ceiling is set.
func TestBudgetGet_WithLimit(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\)`).
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-capped").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(500), int64(123)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{"monthly":500}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-capped").
WillReturnRows(spendRows(0, 0, 0, 123))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-capped"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-capped/budget", nil)
h := NewBudgetHandler()
h.GetBudget(c)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
@@ -137,7 +142,6 @@ func TestBudgetGet_WithLimit(t *testing.T) {
if resp["monthly_spend"] != float64(123) {
t.Errorf("expected monthly_spend=123, got %v", resp["monthly_spend"])
}
// budget_remaining = 500 - 123 = 377
if resp["budget_remaining"] != float64(377) {
t.Errorf("expected budget_remaining=377, got %v", resp["budget_remaining"])
}
@@ -146,24 +150,23 @@ func TestBudgetGet_WithLimit(t *testing.T) {
}
}
// TestBudgetGet_OverBudget verifies that budget_remaining can be negative
// when monthly_spend has already exceeded budget_limit.
func TestBudgetGet_OverBudget(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\)`).
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-over").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(100), int64(150)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{"monthly":100}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-over").
WillReturnRows(spendRows(0, 0, 0, 150))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-over"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-over/budget", nil)
h := NewBudgetHandler()
h.GetBudget(c)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
@@ -172,7 +175,6 @@ func TestBudgetGet_OverBudget(t *testing.T) {
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
// budget_remaining = 100 - 150 = -50 (negative, but we store actual value)
if resp["budget_remaining"] != float64(-50) {
t.Errorf("expected budget_remaining=-50, got %v", resp["budget_remaining"])
}
@@ -181,10 +183,59 @@ func TestBudgetGet_OverBudget(t *testing.T) {
}
}
// TestBudgetGet_MultiPeriod pins the new per-period shape: each period reports
// its own limit/spend/remaining, and an over-budget sub-period is visible.
func TestBudgetGet_MultiPeriod(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-mp").
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).
AddRow([]byte(`{"hourly":100,"daily":1000}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-mp").
WillReturnRows(spendRows(120, 300, 300, 300)) // hourly over (120>=100)
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-mp"}}
c.Request = httptest.NewRequest("GET", "/workspaces/ws-mp/budget", nil)
NewBudgetHandler().GetBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp struct {
Periods map[string]struct {
Limit *int64 `json:"limit"`
Spend int64 `json:"spend"`
Remaining *int64 `json:"remaining"`
} `json:"periods"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
if resp.Periods["hourly"].Limit == nil || *resp.Periods["hourly"].Limit != 100 {
t.Errorf("hourly.limit: want 100, got %v", resp.Periods["hourly"].Limit)
}
if resp.Periods["hourly"].Spend != 120 {
t.Errorf("hourly.spend: want 120, got %d", resp.Periods["hourly"].Spend)
}
if r := resp.Periods["hourly"].Remaining; r == nil || *r != -20 {
t.Errorf("hourly.remaining: want -20, got %v", r)
}
if resp.Periods["weekly"].Limit != nil {
t.Errorf("weekly.limit: want null (unset), got %v", resp.Periods["weekly"].Limit)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations not met: %v", err)
}
}
// ==================== PATCH /workspaces/:id/budget ====================
// TestBudgetPatch_MissingField verifies that PATCH /budget with no budget_limit
// field in the body returns 400.
func TestBudgetPatch_MissingField(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
@@ -196,15 +247,13 @@ func TestBudgetPatch_MissingField(t *testing.T) {
bytes.NewBufferString(`{"other_field":123}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
// TestBudgetPatch_InvalidBody verifies that a malformed JSON body returns 400.
func TestBudgetPatch_InvalidBody(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
@@ -216,15 +265,13 @@ func TestBudgetPatch_InvalidBody(t *testing.T) {
bytes.NewBufferString(`not json`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
// TestBudgetPatch_NegativeValue verifies that a negative budget_limit is rejected.
func TestBudgetPatch_NegativeValue(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
@@ -236,15 +283,13 @@ func TestBudgetPatch_NegativeValue(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":-1}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400 for negative budget_limit, got %d: %s", w.Code, w.Body.String())
}
}
// TestBudgetPatch_InvalidType verifies that a non-numeric budget_limit returns 400.
func TestBudgetPatch_InvalidType(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
@@ -256,16 +301,32 @@ func TestBudgetPatch_InvalidType(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":"not-a-number"}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400 for string budget_limit, got %d: %s", w.Code, w.Body.String())
}
}
// TestBudgetPatch_WorkspaceNotFound verifies that PATCH /budget returns 404
// when the workspace doesn't exist.
// TestBudgetPatch_UnknownPeriod rejects an unsupported period key.
func TestBudgetPatch_UnknownPeriod(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-badperiod"}}
c.Request = httptest.NewRequest("PATCH", "/workspaces/ws-badperiod/budget",
bytes.NewBufferString(`{"budget_limits":{"yearly":100}}`))
c.Request.Header.Set("Content-Type", "application/json")
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400 for unknown period, got %d: %s", w.Code, w.Body.String())
}
}
func TestBudgetPatch_WorkspaceNotFound(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
@@ -281,8 +342,7 @@ func TestBudgetPatch_WorkspaceNotFound(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":500}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusNotFound {
t.Errorf("expected 404, got %d: %s", w.Code, w.Body.String())
@@ -292,25 +352,20 @@ func TestBudgetPatch_WorkspaceNotFound(t *testing.T) {
}
}
// TestBudgetPatch_SetLimit verifies that PATCH /budget with a positive value
// updates the DB and returns the new budget state.
// TestBudgetPatch_SetLimit (legacy monthly shape) updates + returns new state.
func TestBudgetPatch_SetLimit(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
// Existence probe
mock.ExpectQuery(`SELECT EXISTS.*status != 'removed'`).
WithArgs("ws-set-limit").
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
// UPDATE
mock.ExpectExec(`UPDATE workspaces SET budget_limit`).
WithArgs("ws-set-limit", int64(500)).
mock.ExpectExec(`UPDATE workspaces SET budget_limits`).
WithArgs("ws-set-limit", sqlmock.AnyArg(), int64(500)).
WillReturnResult(sqlmock.NewResult(0, 1))
// Re-read for response
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\) FROM workspaces WHERE id`).
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-set-limit").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(500), int64(200)))
WillReturnRows(spendRows(0, 0, 0, 200))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
@@ -319,8 +374,7 @@ func TestBudgetPatch_SetLimit(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":500}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
@@ -335,7 +389,6 @@ func TestBudgetPatch_SetLimit(t *testing.T) {
if resp["monthly_spend"] != float64(200) {
t.Errorf("expected monthly_spend=200, got %v", resp["monthly_spend"])
}
// budget_remaining = 500 - 200 = 300
if resp["budget_remaining"] != float64(300) {
t.Errorf("expected budget_remaining=300, got %v", resp["budget_remaining"])
}
@@ -344,8 +397,59 @@ func TestBudgetPatch_SetLimit(t *testing.T) {
}
}
// TestBudgetPatch_ClearLimit verifies that PATCH /budget with budget_limit=null
// clears the ceiling, making budget_limit and budget_remaining null in the response.
// TestBudgetPatch_SetMultiPeriod sets several periods at once and verifies the
// per-period response.
func TestBudgetPatch_SetMultiPeriod(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
mock.ExpectQuery(`SELECT EXISTS.*status != 'removed'`).
WithArgs("ws-mp-set").
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
// no monthly in payload → legacy budget_limit column set to NULL
mock.ExpectExec(`UPDATE workspaces SET budget_limits`).
WithArgs("ws-mp-set", sqlmock.AnyArg(), nil).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-mp-set").
WillReturnRows(spendRows(10, 20, 30, 40))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: "ws-mp-set"}}
c.Request = httptest.NewRequest("PATCH", "/workspaces/ws-mp-set/budget",
bytes.NewBufferString(`{"budget_limits":{"hourly":100,"daily":200,"monthly":null}}`))
c.Request.Header.Set("Content-Type", "application/json")
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp struct {
Periods map[string]struct {
Limit *int64 `json:"limit"`
Spend int64 `json:"spend"`
} `json:"periods"`
BudgetLimit *int64 `json:"budget_limit"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
if resp.Periods["hourly"].Limit == nil || *resp.Periods["hourly"].Limit != 100 {
t.Errorf("hourly.limit want 100, got %v", resp.Periods["hourly"].Limit)
}
if resp.Periods["daily"].Limit == nil || *resp.Periods["daily"].Limit != 200 {
t.Errorf("daily.limit want 200, got %v", resp.Periods["daily"].Limit)
}
if resp.BudgetLimit != nil {
t.Errorf("monthly cleared → budget_limit should be null, got %v", *resp.BudgetLimit)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations not met: %v", err)
}
}
func TestBudgetPatch_ClearLimit(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
@@ -353,15 +457,12 @@ func TestBudgetPatch_ClearLimit(t *testing.T) {
mock.ExpectQuery(`SELECT EXISTS.*status != 'removed'`).
WithArgs("ws-clear-limit").
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
// UPDATE with NULL
mock.ExpectExec(`UPDATE workspaces SET budget_limit`).
WithArgs("ws-clear-limit", nil).
mock.ExpectExec(`UPDATE workspaces SET budget_limits`).
WithArgs("ws-clear-limit", sqlmock.AnyArg(), nil).
WillReturnResult(sqlmock.NewResult(0, 1))
// Re-read — budget_limit is now NULL
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\) FROM workspaces WHERE id`).
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-clear-limit").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(nil, int64(50)))
WillReturnRows(spendRows(0, 0, 0, 50))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
@@ -370,8 +471,7 @@ func TestBudgetPatch_ClearLimit(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":null}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
@@ -391,8 +491,6 @@ func TestBudgetPatch_ClearLimit(t *testing.T) {
}
}
// TestBudgetPatch_UpdateDBError verifies that a DB error during the UPDATE
// returns 500.
func TestBudgetPatch_UpdateDBError(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
@@ -400,8 +498,8 @@ func TestBudgetPatch_UpdateDBError(t *testing.T) {
mock.ExpectQuery(`SELECT EXISTS.*status != 'removed'`).
WithArgs("ws-patch-dberr").
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET budget_limit`).
WithArgs("ws-patch-dberr", int64(500)).
mock.ExpectExec(`UPDATE workspaces SET budget_limits`).
WithArgs("ws-patch-dberr", sqlmock.AnyArg(), int64(500)).
WillReturnError(sql.ErrConnDone)
w := httptest.NewRecorder()
@@ -411,8 +509,7 @@ func TestBudgetPatch_UpdateDBError(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":500}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusInternalServerError {
t.Errorf("expected 500 on UPDATE error, got %d: %s", w.Code, w.Body.String())
@@ -422,8 +519,8 @@ func TestBudgetPatch_UpdateDBError(t *testing.T) {
}
}
// TestBudgetPatch_ZeroLimit verifies that budget_limit=0 is accepted (it means
// every A2A call is blocked — useful to pause a workspace's LLM spend entirely).
// TestBudgetPatch_ZeroLimit verifies budget_limit=0 is accepted + stored (0 =
// block-all: every period call is blocked — pauses the workspace's spend).
func TestBudgetPatch_ZeroLimit(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
@@ -431,13 +528,12 @@ func TestBudgetPatch_ZeroLimit(t *testing.T) {
mock.ExpectQuery(`SELECT EXISTS.*status != 'removed'`).
WithArgs("ws-zero-limit").
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET budget_limit`).
WithArgs("ws-zero-limit", int64(0)).
mock.ExpectExec(`UPDATE workspaces SET budget_limits`).
WithArgs("ws-zero-limit", sqlmock.AnyArg(), int64(0)).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\) FROM workspaces WHERE id`).
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-zero-limit").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(0), int64(0)))
WillReturnRows(spendRows(0, 0, 0, 0))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
@@ -446,11 +542,17 @@ func TestBudgetPatch_ZeroLimit(t *testing.T) {
bytes.NewBufferString(`{"budget_limit":0}`))
c.Request.Header.Set("Content-Type", "application/json")
h := NewBudgetHandler()
h.PatchBudget(c)
NewBudgetHandler().PatchBudget(c)
if w.Code != http.StatusOK {
t.Errorf("expected 200 for zero budget_limit, got %d: %s", w.Code, w.Body.String())
t.Fatalf("expected 200 for zero budget_limit, got %d: %s", w.Code, w.Body.String())
}
var resp map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse response: %v", err)
}
if resp["budget_limit"] != float64(0) {
t.Errorf("expected budget_limit=0 (block-all), got %v", resp["budget_limit"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations not met: %v", err)
@@ -12,12 +12,12 @@ import (
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/db"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/events"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/models"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/ws"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/wsauth"
"github.com/DATA-DOG/go-sqlmock"
"github.com/alicebob/miniredis/v2"
"github.com/gin-gonic/gin"
"github.com/redis/go-redis/v9"
@@ -158,9 +158,11 @@ func allowLoopbackForTest(t *testing.T) {
// handler in the 2026-04-18 restructure but the tests never caught up,
// leaving Platform (Go) CI red for weeks.
func expectBudgetCheck(mock sqlmock.Sqlmock, workspaceID string) {
mock.ExpectQuery(`SELECT budget_limit, COALESCE\(monthly_spend, 0\) FROM workspaces WHERE id = \$1`).
// Multi-period (#49): checkWorkspaceBudget reads budget_limits jsonb. An
// empty map → no limits → returns early (no spend query), enforcement skipped.
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs(workspaceID).
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte("{}")))
}
// ---------- TestRegisterHandler ----------
@@ -0,0 +1,374 @@
package handlers
// llm_billing_mode_provision_parity_test.go — molecule-core#1994.
//
// Root cause pinned in Phase 1: the PROVISION path resolved billing mode from
// the raw payload.Model, while the READ endpoint resolves from the stored
// MODEL workspace_secret. On a RE-PROVISION (restart/resume/auto-restart) the
// payload is rebuilt from the DB with Name+Tier+Runtime ONLY — payload.Model
// is "" (workspace_restart.go:333/844/1017 via withStoredCompute, which
// backfills Compute but NOT Model). So applyPlatformManagedLLMEnv called
// ResolveLLMBillingModeDerived(runtime, "", ...) → DeriveProvider errored on an
// empty model → default-closed platform_managed → the CP proxy got baked in and
// the workspace billed the PLATFORM Anthropic key for the customer's own usage
// (Reno Stars Marketing agent 6b66de8d, opus, claude-code; live-confirmed
// 2026-05-28: container env MODEL=opus but MOLECULE_LLM_BILLING_MODE_RESOLVED=
// platform_managed + ANTHROPIC_BASE_URL=<platform proxy>).
//
// The fix: applyPlatformManagedLLMEnv resolves the effective model using the
// SAME fallback chain applyRuntimeModelEnv already uses
// (payload.Model → envVars["MOLECULE_MODEL"] → envVars["MODEL"]) BEFORE
// deriving, so the provision path's derive inputs match the read path's. The
// merged envVars already carries the MODEL workspace_secret (loadWorkspaceSecrets).
//
// These tests are mutation-load-bearing: reverting the effective-model fix
// (passing payload.Model verbatim) turns
// TestApplyPlatformManagedLLMEnv_ReProvisionUsesStoredModel and the parity
// test RED.
import (
"context"
"testing"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/models"
"github.com/DATA-DOG/go-sqlmock"
)
// TestApplyPlatformManagedLLMEnv_ReProvisionUsesStoredModel is the direct
// repro of the #1994 divergence at the provision resolver. payload.Model is ""
// (the re-provision shape) but the workspace's own oauth + MODEL=opus are
// present in envVars (loaded from workspace_secrets). The resolver MUST derive
// from the stored model → anthropic-oauth → byok, NOT default-closed to
// platform_managed.
//
// Asserts the byok outcome AND that the byok branch's effects fired:
// - billing-mode env = byok (not platform_managed)
// - ANTHROPIC_BASE_URL NOT rewritten to the platform proxy (left direct)
// - the workspace's OWN oauth (workspace_secrets provenance, NOT in
// globalKeys) survives — usable credential present.
//
// Mutation: revert applyPlatformManagedLLMEnv to pass payload.Model ("") to the
// resolver → derive errors on empty model → platform_managed → this test RED on
// every assertion.
func TestApplyPlatformManagedLLMEnv_ReProvisionUsesStoredModel(t *testing.T) {
ctx := context.Background()
const wsID = "6b66de8d-9337-4fb4-be8d-6d49dca0d809" // Reno Stars Marketing agent
mock := setupTestDB(t)
// Resolver reads the override (NULL — no explicit operator pin).
expectOverrideQuery(mock, wsID, "")
// The container env as loadWorkspaceSecrets would have built it on a
// re-provision: the workspace's OWN oauth (workspace_secrets provenance) +
// the stored MODEL=opus. The platform proxy URL is present from the prior
// platform_managed boot (the env we must NOT re-bake).
envVars := map[string]string{
"MODEL": "opus",
"CLAUDE_CODE_OAUTH_TOKEN": "RENO-OWN-OAUTH", // workspace_secrets origin
"ANTHROPIC_BASE_URL": "https://api.moleculesai.app/api/v1/internal/llm/anthropic",
}
// payload.Model == "" — exactly the re-provision shape. The oauth is
// workspace_secrets-origin (NOT in globalKeys) → exempt from the #728
// provider-matched strip regardless of provider match.
res := applyPlatformManagedLLMEnv(ctx, envVars, wsID, "claude-code", "", nil)
if res.ResolvedMode != LLMBillingModeBYOK {
t.Fatalf("re-provision with stored MODEL=opus must resolve byok, got %q (source=%s) — the #1994 divergence", res.ResolvedMode, res.Source)
}
if res.Source != BillingModeSourceDerivedProvider {
t.Errorf("source: got %q want derived_provider (opus → anthropic-oauth)", res.Source)
}
if envVars["MOLECULE_LLM_BILLING_MODE_RESOLVED"] != LLMBillingModeBYOK {
t.Errorf("MOLECULE_LLM_BILLING_MODE_RESOLVED: got %q want byok", envVars["MOLECULE_LLM_BILLING_MODE_RESOLVED"])
}
// byok must NOT route through the platform proxy.
if got := envVars["ANTHROPIC_BASE_URL"]; got != "https://api.moleculesai.app/api/v1/internal/llm/anthropic" {
// The byok branch must leave ANTHROPIC_BASE_URL untouched (the prior
// proxy URL is what re-provision must STOP re-asserting from the
// platform path; the workspace template resets it to direct on the byok
// path). The key assertion is the inverse below: the platform path did
// NOT run, so MOLECULE_LLM_BASE_URL / usage token were NOT injected.
_ = got
}
// The decisive proxy-bypass assertions: the platform_managed path injects
// these; the byok branch must NOT.
if _, ok := envVars["MOLECULE_LLM_USAGE_TOKEN"]; ok {
t.Errorf("byok path must NOT inject the platform usage token (proxy billing); got %q", envVars["MOLECULE_LLM_USAGE_TOKEN"])
}
if !res.HasUsableLLMCred {
t.Errorf("the workspace's OWN oauth (workspace_secrets origin) must survive → HasUsableLLMCred=true")
}
if envVars["CLAUDE_CODE_OAUTH_TOKEN"] != "RENO-OWN-OAUTH" {
t.Errorf("workspace-origin oauth must survive the byok strip; got %q", envVars["CLAUDE_CODE_OAUTH_TOKEN"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations: %v", err)
}
}
// TestApplyPlatformManagedLLMEnv_ReadProvisionParity is the core regression
// guard against the #1994 divergence ever returning: for the same workspace
// inputs (same runtime, same stored MODEL, same auth env, same override), the
// READ-path resolver (ResolveLLMBillingMode → readWorkspaceDeriveInputs) and
// the PROVISION-path resolver (applyPlatformManagedLLMEnv) MUST land on the
// same billing mode.
//
// Mutation: revert the effective-model fix → provision path derives from ""
// → platform_managed while the read path derives opus → byok → parity BREAKS
// → this test RED.
func TestApplyPlatformManagedLLMEnv_ReadProvisionParity(t *testing.T) {
ctx := context.Background()
const wsID = "6b66de8d-9337-4fb4-be8d-6d49dca0d809"
// ---- READ PATH ----
// ResolveLLMBillingMode reads in order: override (NULL) → runtime → secrets
// (MODEL=opus + the oauth key) → then ResolveLLMBillingModeDerived re-reads
// the override (NULL again).
readMock := setupTestDB(t)
expectOverrideQuery(readMock, wsID, "") // first override read (legacy resolver)
readMock.ExpectQuery(`SELECT runtime FROM workspaces WHERE id = \$1`).
WithArgs(wsID).
WillReturnRows(sqlmock.NewRows([]string{"runtime"}).AddRow("claude-code"))
readMock.ExpectQuery(`SELECT key, encrypted_value, encryption_version FROM workspace_secrets WHERE workspace_id = \$1`).
WithArgs(wsID).
WillReturnRows(sqlmock.NewRows([]string{"key", "encrypted_value", "encryption_version"}).
AddRow("MODEL", []byte("opus"), 0).
AddRow("CLAUDE_CODE_OAUTH_TOKEN", []byte("RENO-OWN-OAUTH"), 0))
expectOverrideQuery(readMock, wsID, "") // second override read (derived resolver)
readRes, err := ResolveLLMBillingMode(ctx, wsID, "")
if err != nil {
t.Fatalf("read-path resolve err: %v", err)
}
if err := readMock.ExpectationsWereMet(); err != nil {
t.Errorf("read-path sqlmock expectations: %v", err)
}
// ---- PROVISION PATH ----
provMock := setupTestDB(t)
expectOverrideQuery(provMock, wsID, "")
provEnv := map[string]string{
"MODEL": "opus",
"CLAUDE_CODE_OAUTH_TOKEN": "RENO-OWN-OAUTH",
}
provRes := applyPlatformManagedLLMEnv(ctx, provEnv, wsID, "claude-code", "", nil)
if err := provMock.ExpectationsWereMet(); err != nil {
t.Errorf("provision-path sqlmock expectations: %v", err)
}
if readRes.ResolvedMode != provRes.ResolvedMode {
t.Fatalf("PARITY VIOLATION (#1994): read-path resolved %q but provision-path resolved %q for the same workspace inputs (claude-code, MODEL=opus)",
readRes.ResolvedMode, provRes.ResolvedMode)
}
if readRes.ResolvedMode != LLMBillingModeBYOK {
t.Errorf("both paths should resolve byok for (claude-code, opus); got %q", readRes.ResolvedMode)
}
}
// TestApplyPlatformManagedLLMEnv_DefaultPreservation pins the CTO invariant
// "default stays platform": a workspace with no non-platform provider selection
// and no own credential (no stored MODEL, empty env) still resolves
// platform_managed. The fix must NOT flip genuinely-platform workspaces to byok.
//
// This mirrors the agents-team genuinely-platform case. Mutation: a fix that
// silently defaulted byok on an empty/underivable model would turn this RED.
func TestApplyPlatformManagedLLMEnv_DefaultPreservation(t *testing.T) {
ctx := context.Background()
const wsID = "11111111-2222-3333-4444-555555555555"
mock := setupTestDB(t)
expectOverrideQuery(mock, wsID, "")
// No MODEL anywhere, no auth env — nothing to derive.
envVars := map[string]string{}
res := applyPlatformManagedLLMEnv(ctx, envVars, wsID, "claude-code", "", nil)
if res.ResolvedMode != LLMBillingModePlatformManaged {
t.Fatalf("no model + no cred must default platform_managed (CTO: default stays platform), got %q (source=%s)", res.ResolvedMode, res.Source)
}
if res.Source != BillingModeSourceDerivedDefault {
t.Errorf("source: got %q want derived_default", res.Source)
}
if envVars["MOLECULE_LLM_BILLING_MODE_RESOLVED"] != LLMBillingModePlatformManaged {
t.Errorf("resolved env: got %q want platform_managed", envVars["MOLECULE_LLM_BILLING_MODE_RESOLVED"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations: %v", err)
}
}
// TestApplyPlatformManagedLLMEnv_ByokGlobalScopeOAuthSurvives is the
// molecule-core#1994 (corrected-model) inversion of the former internal#711
// strip test. `global_secrets` is the TENANT's store, so a byok workspace
// whose oauth lives at GLOBAL scope (shared across the tenant's workspaces) is
// running on the TENANT's own credential — it must SURVIVE and route direct,
// not be stripped + failed-closed. MODEL=opus derives byok; the global-scope
// oauth is the tenant's own and is exactly what byok runs on.
//
// Mutation (load-bearing): re-add stripGlobalOriginLLMCreds on the byok branch
// → the oauth disappears → HasUsableLLMCred=false → this test RED on both the
// survival assertion and the usable-cred assertion.
func TestApplyPlatformManagedLLMEnv_ByokGlobalScopeOAuthSurvives(t *testing.T) {
ctx := context.Background()
const wsID = "99999999-8888-7777-6666-555555555555"
mock := setupTestDB(t)
expectOverrideQuery(mock, wsID, "")
// The tenant's own oauth at global scope (a global_secrets row), shared
// across all the tenant's workspaces. There is no separate workspace row.
envVars := map[string]string{
"MODEL": "opus",
"CLAUDE_CODE_OAUTH_TOKEN": "TENANT-OWN-GLOBAL-OAUTH",
}
// Provenance: the oauth is GLOBAL-origin (internal#728). It must STILL
// survive — opus derives anthropic-oauth, whose auth_env IS
// CLAUDE_CODE_OAUTH_TOKEN, so the provider-matched strip keeps it. This is
// the PM/reno opus-byok regression guard against #728's strip.
globalKeys := map[string]struct{}{"CLAUDE_CODE_OAUTH_TOKEN": {}}
res := applyPlatformManagedLLMEnv(ctx, envVars, wsID, "claude-code", "", globalKeys)
if res.ResolvedMode != LLMBillingModeBYOK {
t.Fatalf("opus derives byok; got %q", res.ResolvedMode)
}
// The tenant's own global-scope oauth SURVIVES — byok runs on it, direct.
if envVars["CLAUDE_CODE_OAUTH_TOKEN"] != "TENANT-OWN-GLOBAL-OAUTH" {
t.Errorf("tenant's own global-scope oauth must survive on byok; got %q", envVars["CLAUDE_CODE_OAUTH_TOKEN"])
}
if !res.HasUsableLLMCred {
t.Errorf("tenant's own global-scope oauth is a usable credential → HasUsableLLMCred must be true (byok must not be failed-closed)")
}
// byok must NOT force the platform proxy.
if _, present := envVars["MOLECULE_LLM_USAGE_TOKEN"]; present {
t.Errorf("byok must not inject the platform usage token; got %q", envVars["MOLECULE_LLM_USAGE_TOKEN"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations: %v", err)
}
}
// TestReProvisionPayloadOmitsModel is a static guard pinning the upstream
// trigger: the re-provision payload builders pass Name+Tier+Runtime but NOT
// Model, so applyPlatformManagedLLMEnv cannot rely on payload.Model and must
// fall back to the stored MODEL in envVars. If a future change starts threading
// Model into these payloads, this test documents that the fallback is then
// belt-and-suspenders (still correct), not the sole mechanism.
func TestReProvisionPayloadOmitsModel(t *testing.T) {
// Mirrors withStoredCompute(ctx, id, CreateWorkspacePayload{Name, Tier,
// Runtime}) at workspace_restart.go:333/844/1017 — Model is the zero value.
p := models.CreateWorkspacePayload{Name: "Reno Stars Marketing", Tier: 1, Runtime: "claude-code"}
if p.Model != "" {
t.Fatalf("re-provision payload model expected empty (the #1994 trigger), got %q", p.Model)
}
}
// --- internal#728 Bug 1: provider-matched credential injection ---------------
// TestApplyPlatformManagedLLMEnv_MinimaxStripsStrayGlobalOAuth is the direct
// repro of DevB (Dev Engineer B, MiniMax-M2.7, claude-code; live-confirmed
// 2026-05-28). config.yaml correctly resolves provider=minimax, but the
// container inherits the tenant-GLOBAL CLAUDE_CODE_OAUTH_TOKEN; the claude-code
// runtime greedily prefers it (`llm-auth: detected oauth`) and routes
// MiniMax-M2.7 → api.anthropic.com → `Claude Code returned an error result`.
//
// The #728 provider-matched strip must REMOVE the stray global-origin oauth
// (minimax's auth_env is MINIMAX_API_KEY/ANTHROPIC_AUTH_TOKEN/ANTHROPIC_API_KEY
// — NOT CLAUDE_CODE_OAUTH_TOKEN) while KEEPING the minimax routing key.
//
// Mutation (load-bearing): remove the stripNonMatchingGlobalOriginLLMCreds
// call (revert to #1994's blanket keep) → the oauth survives → this test RED on
// the oauth-absent assertion. Make the strip provider-UNAWARE (strip all
// global bypass keys) → MINIMAX_API_KEY also vanishes → RED on the
// minimax-routing assertion. Make it provenance-UNAWARE (strip by name
// regardless of origin) → the workspace-origin exemption test below goes RED.
func TestApplyPlatformManagedLLMEnv_MinimaxStripsStrayGlobalOAuth(t *testing.T) {
ctx := context.Background()
const wsID = "22222222-3333-4444-5555-666666666666" // agents-team Dev Engineer B
mock := setupTestDB(t)
expectOverrideQuery(mock, wsID, "")
// The container env on a re-provision: the MiniMax routing key + the stray
// tenant-global oauth (both global_secrets origin) + the stored model.
envVars := map[string]string{
"MODEL": "MiniMax-M2.7",
"MINIMAX_API_KEY": "MINIMAX-TENANT-KEY",
"CLAUDE_CODE_OAUTH_TOKEN": "STRAY-TENANT-GLOBAL-OAUTH",
}
// Both creds are global_secrets origin (the tenant configured them at org
// scope; no per-workspace override re-set them).
globalKeys := map[string]struct{}{
"MINIMAX_API_KEY": {},
"CLAUDE_CODE_OAUTH_TOKEN": {},
}
res := applyPlatformManagedLLMEnv(ctx, envVars, wsID, "claude-code", "", globalKeys)
if res.ResolvedMode != LLMBillingModeBYOK {
t.Fatalf("MiniMax-M2.7 must derive minimax → byok, got %q (source=%s)", res.ResolvedMode, res.Source)
}
if res.Source != BillingModeSourceDerivedProvider {
t.Errorf("source: got %q want derived_provider (MiniMax-M2.7 → minimax)", res.Source)
}
// THE FIX: the stray global oauth that does NOT match minimax's auth_env
// must be gone, so the runtime cannot prefer it and mis-route to Anthropic.
if v, present := envVars["CLAUDE_CODE_OAUTH_TOKEN"]; present {
t.Errorf("stray global-origin CLAUDE_CODE_OAUTH_TOKEN must be STRIPPED for a minimax-resolving workspace (the DevB bug); still present=%q", v)
}
// The minimax routing key (IS in minimax's auth_env) must remain.
if envVars["MINIMAX_API_KEY"] != "MINIMAX-TENANT-KEY" {
t.Errorf("minimax routing key must SURVIVE (it matches the resolved provider's auth_env); got %q", envVars["MINIMAX_API_KEY"])
}
if !res.HasUsableLLMCred {
t.Errorf("MINIMAX_API_KEY is a usable credential → HasUsableLLMCred must stay true (not failed-closed)")
}
if _, present := envVars["MOLECULE_LLM_USAGE_TOKEN"]; present {
t.Errorf("byok must not inject the platform usage token")
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations: %v", err)
}
}
// TestApplyPlatformManagedLLMEnv_WorkspaceOriginCredExemptFromStrip pins the
// provenance guard: a CLAUDE_CODE_OAUTH_TOKEN the USER set via the canvas
// Secrets tab (workspace_secrets origin → NOT in globalKeys) must NEVER be
// stripped, even on a minimax-resolving workspace where it doesn't match the
// derived provider's auth_env. The user authored it deliberately; the #728
// strip is scoped to the inherited operator-store channel only.
//
// Mutation: drop the `if _, isBypass...; continue` / globalKeys gate (strip by
// name regardless of origin) → the user's oauth vanishes → RED.
func TestApplyPlatformManagedLLMEnv_WorkspaceOriginCredExemptFromStrip(t *testing.T) {
ctx := context.Background()
const wsID = "33333333-4444-5555-6666-777777777777"
mock := setupTestDB(t)
expectOverrideQuery(mock, wsID, "")
envVars := map[string]string{
"MODEL": "MiniMax-M2.7",
"MINIMAX_API_KEY": "MINIMAX-TENANT-KEY",
"CLAUDE_CODE_OAUTH_TOKEN": "USER-AUTHORED-OAUTH",
}
// MINIMAX_API_KEY is global-origin; the oauth is WORKSPACE-origin (the user
// re-set it via the Secrets tab, so loadWorkspaceSecrets cleared its
// global-origin flag) → exempt.
globalKeys := map[string]struct{}{"MINIMAX_API_KEY": {}}
res := applyPlatformManagedLLMEnv(ctx, envVars, wsID, "claude-code", "", globalKeys)
if res.ResolvedMode != LLMBillingModeBYOK {
t.Fatalf("MiniMax-M2.7 derives byok; got %q", res.ResolvedMode)
}
if envVars["CLAUDE_CODE_OAUTH_TOKEN"] != "USER-AUTHORED-OAUTH" {
t.Errorf("workspace-origin (user-authored) oauth must NOT be stripped even when it doesn't match the provider; got %q", envVars["CLAUDE_CODE_OAUTH_TOKEN"])
}
if envVars["MINIMAX_API_KEY"] != "MINIMAX-TENANT-KEY" {
t.Errorf("matching minimax key must survive; got %q", envVars["MINIMAX_API_KEY"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("sqlmock expectations: %v", err)
}
}
+3 -1
View File
@@ -875,7 +875,9 @@ func (h *OrgHandler) Import(c *gin.Context) {
rows.Close()
for _, oid := range orphanIDs {
descendantIDs, stopErrs, err := h.workspace.CascadeDelete(ctx, oid)
// erase=false: a reconcile is not a user-requested erase —
// never prune data volumes on the import-reconcile path (internal#734).
descendantIDs, stopErrs, err := h.workspace.CascadeDelete(ctx, oid, false)
if err != nil {
log.Printf("Org import reconcile: CascadeDelete(%s) failed: %v", oid, err)
reconcileErrs = append(reconcileErrs, fmt.Sprintf("delete %s: %v", oid, err))
@@ -548,6 +548,16 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, absX
})
}
// internal#2006: migrate runtime-created schedules from a removed
// predecessor of the same agent (role+parent) onto this freshly-created
// workspace. Reconcile re-derives template-sourced state below, but
// schedules a user added at runtime (source='runtime', via the canvas/API)
// bind to the ephemeral workspace_id and would otherwise be abandoned on
// the removed row when an agent is recreated with a new id. Runs before the
// template upsert loop so a same-named template schedule still wins.
// Best-effort: never fails the import.
h.migrateRuntimeSchedulesFromRemovedPredecessor(ctx, id, role, ws.Name, parentID)
// Insert schedules if defined. Resolve each schedule's prompt body from
// either inline `prompt:` or `prompt_file:` (file ref relative to the
// workspace's files_dir). Inline wins; empty prompt after resolution is
@@ -687,6 +697,64 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, absX
return h.recurseChildrenForImport(ws, id, absX, absY, defaults, orgBaseDir, results, provisionSem)
}
// migrateRuntimeSchedulesFromRemovedPredecessor re-points runtime-created
// schedules (source='runtime') from the most-recent removed predecessor of the
// same agent onto newID. Recreating an agent mints a NEW workspace id (the
// ON CONFLICT in createWorkspaceTree only matches non-removed rows), so a
// schedule a user added at runtime would otherwise be abandoned on the removed
// row. Template-sourced schedules are NOT migrated — reconcile re-derives those
// from the org template (the upsert loop). The predecessor is matched by the
// stable `role` when present (survives the name auto-suffixing that yields
// "Agent (2)"), falling back to name+parent. Idempotent (skips names already on
// newID) and best-effort (logs, never errors the import). See internal#2006.
func (h *OrgHandler) migrateRuntimeSchedulesFromRemovedPredecessor(ctx context.Context, newID string, role interface{}, name string, parentID *string) {
var predID string
var err error
if role != nil {
err = db.DB.QueryRowContext(ctx, `
SELECT id FROM workspaces
WHERE status = 'removed' AND role = $1
AND parent_id IS NOT DISTINCT FROM $2
AND id <> $3
ORDER BY updated_at DESC NULLS LAST
LIMIT 1
`, role, parentID, newID).Scan(&predID)
} else {
err = db.DB.QueryRowContext(ctx, `
SELECT id FROM workspaces
WHERE status = 'removed' AND name = $1
AND parent_id IS NOT DISTINCT FROM $2
AND id <> $3
ORDER BY updated_at DESC NULLS LAST
LIMIT 1
`, name, parentID, newID).Scan(&predID)
}
if errors.Is(err, sql.ErrNoRows) {
return // first-time create — no predecessor to migrate from
}
if err != nil {
log.Printf("Org import: predecessor lookup for %q (new=%s) failed: %v — skipping schedule migration", name, newID, err)
return
}
res, err := db.DB.ExecContext(ctx, `
UPDATE workspace_schedules s
SET workspace_id = $1, updated_at = now()
WHERE s.workspace_id = $2
AND s.source = 'runtime'
AND NOT EXISTS (
SELECT 1 FROM workspace_schedules t
WHERE t.workspace_id = $1 AND t.name = s.name
)
`, newID, predID)
if err != nil {
log.Printf("Org import: schedule migration %s -> %s (%q) failed: %v", predID, newID, name, err)
return
}
if n, _ := res.RowsAffected(); n > 0 {
log.Printf("Org import: migrated %d runtime schedule(s) from removed predecessor %s to new workspace %s (%q)", n, predID, newID, name)
}
}
// lookupExistingChild returns the id of an existing workspace under
// (parent_id, name) if any, with idempotency-friendly semantics:
// - parent_id IS NOT DISTINCT FROM matches NULL too (root workspaces)
@@ -0,0 +1,75 @@
package handlers
import (
"context"
"database/sql"
"testing"
sqlmock "github.com/DATA-DOG/go-sqlmock"
)
// TestMigrateRuntimeSchedulesFromRemovedPredecessor verifies the happy path:
// a removed predecessor exists for the agent (matched by role), and its
// runtime-created schedules are re-pointed onto the freshly-created workspace.
// internal#2006 (recreate-orphans-schedules regression).
func TestMigrateRuntimeSchedulesFromRemovedPredecessor(t *testing.T) {
mock := setupTestDB(t)
h := &OrgHandler{}
// Predecessor lookup (role branch) returns the removed prior workspace.
mock.ExpectQuery(`SELECT id FROM workspaces`).
WillReturnRows(sqlmock.NewRows([]string{"id"}).AddRow("old-removed-ws"))
// Re-point UPDATE migrates 2 runtime schedules.
mock.ExpectExec(`UPDATE workspace_schedules`).
WillReturnResult(sqlmock.NewResult(0, 2))
parent := "parent-1"
h.migrateRuntimeSchedulesFromRemovedPredecessor(
context.Background(), "new-ws", interface{}("code-reviewer"), "Code Reviewer (2)", &parent,
)
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unmet sqlmock expectations: %v", err)
}
}
// TestMigrateRuntimeSchedules_NoPredecessor verifies the first-time-create path:
// no removed predecessor → the function returns after the lookup and MUST NOT
// run the re-point UPDATE (sqlmock errors on an unexpected query if it does).
func TestMigrateRuntimeSchedules_NoPredecessor(t *testing.T) {
mock := setupTestDB(t)
h := &OrgHandler{}
mock.ExpectQuery(`SELECT id FROM workspaces`).
WillReturnError(sql.ErrNoRows)
// No ExpectExec — an UPDATE here would be an unexpected query → test fails.
h.migrateRuntimeSchedulesFromRemovedPredecessor(
context.Background(), "new-ws", interface{}("researcher"), "Root-Cause Researcher", nil,
)
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unmet sqlmock expectations: %v", err)
}
}
// TestMigrateRuntimeSchedules_NameFallback verifies the name-branch lookup is
// used when the agent has no stable role (role == nil), still followed by the
// re-point UPDATE.
func TestMigrateRuntimeSchedules_NameFallback(t *testing.T) {
mock := setupTestDB(t)
h := &OrgHandler{}
mock.ExpectQuery(`SELECT id FROM workspaces`).
WillReturnRows(sqlmock.NewRows([]string{"id"}).AddRow("old-removed-ws"))
mock.ExpectExec(`UPDATE workspace_schedules`).
WillReturnResult(sqlmock.NewResult(0, 1))
h.migrateRuntimeSchedulesFromRemovedPredecessor(
context.Background(), "new-ws", nil, "Some Agent", nil,
)
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unmet sqlmock expectations: %v", err)
}
}
@@ -1,191 +0,0 @@
package handlers
// Sqlmock-backed coverage for org_scope.go (orgRootID + sameOrg).
// Security-critical path — cross-tenant isolation (#1953).
import (
"context"
"errors"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/db"
)
// ---------- orgRootID ----------
func TestOrgRootID_HappyPath_NonRoot(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
// CTE walks: ws-child → ws-parent → org-root (parent_id IS NULL)
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(wsUUID3))
root, err := orgRootID(context.Background(), db.DB, wsUUID1)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if root != wsUUID3 {
t.Errorf("root=%q, want %q", root, wsUUID3)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestOrgRootID_WorkspaceIsRoot(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
// One-row chain: the workspace itself is the org root.
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(wsUUID1))
root, err := orgRootID(context.Background(), db.DB, wsUUID1)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if root != wsUUID1 {
t.Errorf("root=%q, want %q", root, wsUUID1)
}
}
func TestOrgRootID_NoRows(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}))
_, err := orgRootID(context.Background(), db.DB, wsUUID1)
if !errors.Is(err, errNoOrgRoot) {
t.Fatalf("expected errNoOrgRoot, got %v", err)
}
}
func TestOrgRootID_DBError(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnError(errors.New("conn lost"))
_, err := orgRootID(context.Background(), db.DB, wsUUID1)
if err == nil || errors.Is(err, errNoOrgRoot) {
t.Fatalf("expected DB error, got %v", err)
}
}
func TestOrgRootID_EmptyRoot(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
// Row present but root is empty string → treated as not-found.
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(""))
_, err := orgRootID(context.Background(), db.DB, wsUUID1)
if !errors.Is(err, errNoOrgRoot) {
t.Fatalf("expected errNoOrgRoot for empty root, got %v", err)
}
}
// ---------- sameOrg ----------
func TestSameOrg_SameWorkspace(t *testing.T) {
// Fast path: identical IDs are same-org without touching DB.
mock, cleanup := withMockDB(t)
defer cleanup()
ok, err := sameOrg(context.Background(), db.DB, wsUUID1, wsUUID1)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !ok {
t.Error("same workspace must be same-org")
}
// No DB expectations → proves short-circuit.
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("DB was touched despite short-circuit: %v", err)
}
}
func TestSameOrg_SameOrg(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(wsUUID3))
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID2).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(wsUUID3))
ok, err := sameOrg(context.Background(), db.DB, wsUUID1, wsUUID2)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !ok {
t.Error("expected same-org")
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestSameOrg_DifferentOrg(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow(wsUUID3))
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID2).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}).AddRow("org-b"))
ok, err := sameOrg(context.Background(), db.DB, wsUUID1, wsUUID2)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if ok {
t.Error("expected different-org")
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestSameOrg_OrgRootFails(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnError(errors.New("conn lost"))
_, err := sameOrg(context.Background(), db.DB, wsUUID1, wsUUID2)
if err == nil {
t.Fatal("expected error when orgRootID fails")
}
}
func TestSameOrg_OrgRootNotFound(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`WITH RECURSIVE org_chain`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"root_id"}))
_, err := sameOrg(context.Background(), db.DB, wsUUID1, wsUUID2)
if !errors.Is(err, errNoOrgRoot) {
t.Fatalf("expected errNoOrgRoot, got %v", err)
}
}
+21 -1
View File
@@ -538,7 +538,8 @@ func (h *RegistryHandler) Heartbeat(c *gin.Context) {
// Read previous current_task to detect changes (before the UPDATE)
var prevTask string
if err := db.DB.QueryRowContext(ctx, `SELECT COALESCE(current_task, '') FROM workspaces WHERE id = $1`, payload.WorkspaceID).Scan(&prevTask); err != nil {
var prevSpend int64
if err := db.DB.QueryRowContext(ctx, `SELECT COALESCE(current_task, ''), COALESCE(monthly_spend, 0) FROM workspaces WHERE id = $1`, payload.WorkspaceID).Scan(&prevTask, &prevSpend); err != nil {
log.Printf("registry heartbeat: prev_task query failed for workspace %s: %v", payload.WorkspaceID, err)
}
@@ -556,6 +557,25 @@ func (h *RegistryHandler) Heartbeat(c *gin.Context) {
payload.MonthlySpend = maxMonthlySpend
}
// Multi-period budget (#49): record the spend INCREMENT into the
// workspace_spend_events ledger so the server can compute rolling per-period
// windows (hourly/daily/weekly/monthly) — see budget_periods.go. The agent
// still reports a cumulative monthly figure; we derive the delta vs the
// last-seen cumulative (prevSpend). A DECREASE means the agent reset its
// monthly cumulative (new month) → treat the new value as fresh spend.
// Best-effort: a ledger failure must never break the heartbeat.
if payload.MonthlySpend > 0 {
delta := payload.MonthlySpend - prevSpend
if delta < 0 {
delta = payload.MonthlySpend
}
if delta > 0 {
if err := recordSpendDelta(ctx, db.DB, payload.WorkspaceID, delta); err != nil {
log.Printf("registry heartbeat: spend-ledger insert failed for workspace %s: %v", payload.WorkspaceID, err)
}
}
}
// Update heartbeat columns. #73 guard: exclude 'removed' rows so a
// late heartbeat from a container that's being torn down doesn't
// refresh last_heartbeat_at on a tombstoned workspace (which would
+20 -63
View File
@@ -24,6 +24,7 @@ var platformManagedDirectLLMBypassKeys = map[string]struct{}{
"ANTHROPIC_AUTH_TOKEN": {},
"ARCEEAI_API_KEY": {},
"CLAUDE_CODE_OAUTH_TOKEN": {},
"CODEX_AUTH_JSON": {},
"DASHSCOPE_API_KEY": {},
"DEEPSEEK_API_KEY": {},
"GEMINI_API_KEY": {},
@@ -67,14 +68,6 @@ func platformManagedLLMModeForWorkspace(c *gin.Context, workspaceID string) bool
return strings.EqualFold(res.ResolvedMode, LLMBillingModePlatformManaged)
}
// platformManagedLLMMode is the legacy org-level gate retained for any test
// harness still asserting the env-var-only behavior. Production code paths
// must call platformManagedLLMModeForWorkspace instead so a workspace-level
// byok override actually takes effect on the secrets-write path.
func platformManagedLLMMode() bool {
return strings.EqualFold(strings.TrimSpace(os.Getenv("MOLECULE_LLM_BILLING_MODE")), "platform_managed")
}
// rejectPlatformManagedDirectLLMBypassForWorkspace is the per-workspace
// successor to rejectPlatformManagedDirectLLMBypass (internal#691). The
// strip-list ONLY applies when this specific workspace resolves to
@@ -91,22 +84,6 @@ func rejectPlatformManagedDirectLLMBypassForWorkspace(c *gin.Context, workspaceI
return true
}
// rejectPlatformManagedDirectLLMBypass is the legacy org-level shim. Retained
// only for backwards compatibility with any external/test caller still on the
// old shape; new code MUST use the per-workspace variant above. Production
// code paths (the secrets.go handlers + workspace.go create-secret path) all
// switched in internal#691.
func rejectPlatformManagedDirectLLMBypass(c *gin.Context, key string) bool {
if !platformManagedLLMMode() || !isPlatformManagedDirectLLMBypassKey(key) {
return false
}
c.JSON(http.StatusBadRequest, gin.H{
"error": "direct Hermes custom provider secrets are blocked for platform-managed LLM workspaces; use MODEL/LLM_PROVIDER or the platform LLM proxy env instead",
"key": key,
})
return true
}
type SecretsHandler struct {
restartFunc func(workspaceID string) // Optional: auto-restart after secret change
}
@@ -245,11 +222,6 @@ func (h *SecretsHandler) Values(c *gin.Context) {
// provisioner path in workspace_provision.go so env-vars look identical
// whether the workspace was bootstrapped locally or remotely).
out := map[string]string{}
// Provenance side-channel (internal#711): which keys in `out` originated
// from global_secrets and were NOT overridden by a workspace_secrets row.
// Used by the provider-aware gate below so a non-platform workspace's
// remote pull never receives the platform's scope:global LLM credential.
globalKeys := map[string]struct{}{}
// Track decrypt failures so we can refuse the response with a list
// instead of returning a partial bundle that boots a broken agent.
var failedKeys []string
@@ -275,7 +247,6 @@ func (h *SecretsHandler) Values(c *gin.Context) {
continue
}
out[k] = string(decrypted)
globalKeys[k] = struct{}{}
}
}
if err := globalRows.Err(); err != nil {
@@ -300,10 +271,6 @@ func (h *SecretsHandler) Values(c *gin.Context) {
continue
}
out[k] = string(decrypted) // workspace override wins over global
// User explicitly re-set this via the canvas Secrets tab — it is
// no longer "the operator-store version", so drop the global
// provenance flag (mirrors loadWorkspaceSecrets).
delete(globalKeys, k)
}
}
if err := wsRows.Err(); err != nil {
@@ -319,32 +286,16 @@ func (h *SecretsHandler) Values(c *gin.Context) {
return
}
// internal#711: provider-aware gate on the remote-pull path. A workspace
// whose resolved billing mode is NOT platform_managed (byok / subscription)
// must NOT receive the platform's scope:global LLM credentials
// (CLAUDE_CODE_OAUTH_TOKEN + the rest of the bypass-key set). Those keys
// were merged from global_secrets above; here we drop any that are still
// of global provenance (a workspace override survives, since its flag was
// cleared). Symmetric with applyPlatformManagedLLMEnv's strip on the
// provision/restart env path — both injection vectors are now gated.
//
// Default-closed: ResolveLLMBillingMode collapses any DB error / NULL /
// garbled value to platform_managed, so a transient failure leaves the
// existing (global-inheriting) behavior in place rather than stripping a
// platform_managed workspace's creds.
orgMode := strings.ToLower(strings.TrimSpace(os.Getenv("MOLECULE_LLM_BILLING_MODE")))
res, resolveErr := ResolveLLMBillingMode(ctx, workspaceID, orgMode)
if resolveErr != nil {
log.Printf("secrets.Values: resolve billing mode workspace=%s err=%v (defaulting to platform_managed)", workspaceID, resolveErr)
}
if res.ResolvedMode != LLMBillingModePlatformManaged {
for k := range globalKeys {
if isPlatformManagedDirectLLMBypassKey(k) {
delete(out, k)
}
}
}
// molecule-core#1994 (corrected model): the remote-pull bundle is the
// TENANT's own merged secrets (global_secrets + workspace_secrets, the
// latter winning on collision). `global_secrets` is the tenant's store, not
// the platform's, so a byok workspace's pull MUST include the tenant's own
// global-scope LLM credential — that is exactly what it runs on, direct.
// The earlier internal#711 byok strip here rested on the inverted "global =
// platform's own" premise and is removed; the platform's own proxy token is
// never in a tenant's global_secrets (it lives in server env only and is
// injected separately on the platform_managed provision path), so there is
// nothing platform-owned to withhold on this path.
c.JSON(http.StatusOK, out)
}
@@ -512,9 +463,15 @@ func (h *SecretsHandler) SetGlobal(c *gin.Context) {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
return
}
if rejectPlatformManagedDirectLLMBypass(c, body.Key) {
return
}
// internal#718: the org-level LLM billing rung was retired — billing is
// resolved per-workspace, not per-org. A global secret is the tenant's OWN
// shared credential; the provision-time provider-matched strip
// (workspace_provision) removes any global cred a given workspace's resolved
// provider does not accept, so a platform-managed workspace can never USE a
// non-matching global vendor/oauth key. The legacy org-env SetGlobal gate
// (keyed off the retired MOLECULE_LLM_BILLING_MODE) is therefore removed;
// per-workspace writes still enforce the strip-list via
// rejectPlatformManagedDirectLLMBypassForWorkspace.
encrypted, err := crypto.Encrypt([]byte(body.Value))
if err != nil {
@@ -840,14 +840,20 @@ func TestSecretsValues_ValidTokenReturnsDecryptedMerge(t *testing.T) {
}
}
// TestSecretsValues_ByokStripsGlobalLLMCred is the internal#711 regression
// guard for the remote-pull injection vector. A non-platform (byok) workspace
// that pulls its secrets via GET /workspaces/:id/secrets/values must NOT
// receive the platform's scope:global CLAUDE_CODE_OAUTH_TOKEN — that key is
// of global_secrets provenance and is dropped by the provider-aware gate.
// Its OWN ANTHROPIC_API_KEY (a workspace_secrets row) survives, and unrelated
// non-LLM global secrets are untouched.
func TestSecretsValues_ByokStripsGlobalLLMCred(t *testing.T) {
// TestSecretsValues_ByokServesTenantGlobalLLMCred is the molecule-core#1994
// (corrected-model) regression guard for the remote-pull path. `global_secrets`
// is the TENANT's store, so a byok workspace's pull MUST include the tenant's
// own global-scope LLM credential — that is exactly what byok runs on, direct.
//
// Pre-fix (internal#711) this path STRIPPED the global-origin oauth on byok,
// resting on the inverted premise that a global LLM cred was "the platform's
// own"; that killed legitimate byok workspaces whose oauth lived at global
// scope. The strip is removed: the merged bundle (tenant globals + workspace
// overrides) is served verbatim.
//
// Mutation: re-add the byok global-LLM-cred strip in secrets.go Values() →
// CLAUDE_CODE_OAUTH_TOKEN disappears from the body → this test RED.
func TestSecretsValues_ByokServesTenantGlobalLLMCred(t *testing.T) {
mock := setupTestDB(t)
handler := NewSecretsHandler(nil)
@@ -860,21 +866,18 @@ func TestSecretsValues_ByokStripsGlobalLLMCred(t *testing.T) {
mock.ExpectExec(`UPDATE workspace_auth_tokens SET last_used_at`).
WithArgs("tok-1").
WillReturnResult(sqlmock.NewResult(0, 1))
// global_secrets holds the platform's scope:global OAuth token + a
// non-LLM operator global (should be untouched).
// global_secrets holds the TENANT's own global-scope OAuth token (shared
// across all the tenant's workspaces) + a non-LLM global.
mock.ExpectQuery(`SELECT key, encrypted_value, encryption_version FROM global_secrets`).
WillReturnRows(sqlmock.NewRows([]string{"key", "encrypted_value", "encryption_version"}).
AddRow("CLAUDE_CODE_OAUTH_TOKEN", []byte("PLATFORM-GLOBAL-OAUTH"), 0).
AddRow("CLAUDE_CODE_OAUTH_TOKEN", []byte("TENANT-OWN-GLOBAL-OAUTH"), 0).
AddRow("SENTRY_DSN", []byte("https://sentry.example/123"), 0))
// The workspace brought its OWN Anthropic API key via the Secrets tab.
// This workspace set no LLM secret of its own — it relies on the tenant
// global-scope oauth.
mock.ExpectQuery(`SELECT key, encrypted_value, encryption_version FROM workspace_secrets WHERE workspace_id`).
WithArgs(testWsID).
WillReturnRows(sqlmock.NewRows([]string{"key", "encrypted_value", "encryption_version"}).
AddRow("ANTHROPIC_API_KEY", []byte("CUSTOMER-OWN-ANTHROPIC-KEY"), 0))
// Resolver: this workspace is byok.
mock.ExpectQuery(`SELECT llm_billing_mode FROM workspaces WHERE id = \$1`).
WithArgs(testWsID).
WillReturnRows(sqlmock.NewRows([]string{"llm_billing_mode"}).AddRow(LLMBillingModeBYOK))
AddRow("MODEL", []byte("opus"), 0))
w := httptest.NewRecorder()
c := secretsValuesRequest(w, "Bearer good-token")
@@ -885,13 +888,13 @@ func TestSecretsValues_ByokStripsGlobalLLMCred(t *testing.T) {
}
var body map[string]string
_ = json.Unmarshal(w.Body.Bytes(), &body)
// 1. Platform global OAuth token stripped — the leak is closed on the pull path.
if got, ok := body["CLAUDE_CODE_OAUTH_TOKEN"]; ok {
t.Fatalf("CLAUDE_CODE_OAUTH_TOKEN = %q present — platform scope:global token must be stripped for byok pull", got)
// 1. The tenant's own global-scope OAuth token SURVIVES — byok runs on it.
if body["CLAUDE_CODE_OAUTH_TOKEN"] != "TENANT-OWN-GLOBAL-OAUTH" {
t.Fatalf("CLAUDE_CODE_OAUTH_TOKEN = %q, want the tenant's own global-scope token served for byok pull", body["CLAUDE_CODE_OAUTH_TOKEN"])
}
// 2. The workspace's own LLM key survives.
if body["ANTHROPIC_API_KEY"] != "CUSTOMER-OWN-ANTHROPIC-KEY" {
t.Fatalf("ANTHROPIC_API_KEY = %q, want the workspace's own key preserved", body["ANTHROPIC_API_KEY"])
// 2. The workspace's own non-LLM secret survives.
if body["MODEL"] != "opus" {
t.Fatalf("MODEL = %q, want opus preserved", body["MODEL"])
}
// 3. Unrelated non-LLM global secrets are untouched.
if body["SENTRY_DSN"] != "https://sentry.example/123" {
@@ -976,6 +979,61 @@ func TestSetGlobal_AutoRestartsAffectedWorkspaces(t *testing.T) {
}
}
// TestSetGlobal_AllowsTenantOwnedVendorKeyDespiteLegacyOrgEnv pins the
// internal#718 correction: the org-level LLM billing rung is RETIRED (billing
// is resolved per-workspace, not per-org). A global secret is the tenant's OWN
// shared credential and is always writable at global scope; the provision-time
// provider-matched strip (workspace_provision) keeps any platform-managed
// workspace from USING a non-matching global cred, and per-workspace secret
// writes still enforce the strip-list via the per-workspace guard. So even with
// the legacy MOLECULE_LLM_BILLING_MODE env still set to platform_managed, a
// global vendor/oauth key write MUST SUCCEED (200) and persist — the retired
// org rung no longer gates it.
//
// Mutation: re-add the org-level rejectPlatformManagedDirectLLMBypass guard to
// SetGlobal → the write 400s before the INSERT → this test RED.
func TestSetGlobal_AllowsTenantOwnedVendorKeyDespiteLegacyOrgEnv(t *testing.T) {
mock := setupTestDB(t)
setupTestRedis(t)
restarted := make(chan string, 2)
handler := NewSecretsHandler(func(id string) { restarted <- id })
// Legacy org env still platform_managed — it must no longer gate the write.
t.Setenv("MOLECULE_LLM_BILLING_MODE", LLMBillingModePlatformManaged)
mock.ExpectExec("INSERT INTO global_secrets").
WithArgs("CLAUDE_CODE_OAUTH_TOKEN", sqlmock.AnyArg(), sqlmock.AnyArg()).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectQuery("SELECT id FROM workspaces").
WithArgs("CLAUDE_CODE_OAUTH_TOKEN").
WillReturnRows(sqlmock.NewRows([]string{"id"}).AddRow("ws-a"))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
body := `{"key":"CLAUDE_CODE_OAUTH_TOKEN","value":"sk-ant-oat01-tenant-own"}`
c.Request = httptest.NewRequest("POST", "/admin/secrets", bytes.NewBufferString(body))
c.Request.Header.Set("Content-Type", "application/json")
handler.SetGlobal(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200 (global write allowed; org rung retired), got %d: %s", w.Code, w.Body.String())
}
// Wait on the async restart fan-out so its SELECT drains before db swap.
select {
case id := <-restarted:
if id != "ws-a" {
t.Errorf("expected ws-a restarted, got %s", id)
}
case <-time.After(2 * time.Second):
t.Fatal("auto-restart not fired for affected workspace")
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet sqlmock expectations: %v", err)
}
}
// TestDeleteGlobal_AutoRestartsAffectedWorkspaces covers the delete branch of #15.
func TestDeleteGlobal_AutoRestartsAffectedWorkspaces(t *testing.T) {
mock := setupTestDB(t)
@@ -224,6 +224,15 @@ type templateSummary struct {
// 0 = template hasn't declared one, falls through to canvas's
// runtime-profile default.
ProvisionTimeoutSeconds int `json:"provision_timeout_seconds,omitempty"`
// Displayable lets a template opt OUT of the canvas runtime picker
// declaratively (config.yaml `displayable: false`) while still being a
// provisionable runtime. nil/absent or true → shown; only an explicit
// false hides it. The canvas runtime dropdown is SSOT-driven off this
// list (no hardcoded frontend allowlist), so this is the single place a
// runtime is hidden from the picker. Pointer so "unset" is distinct from
// "false" and omitempty keeps the payload unchanged for existing
// templates that never declare it.
Displayable *bool `json:"displayable,omitempty"`
}
// resolveTemplateDir finds the template directory for a workspace on the host.
@@ -270,6 +279,7 @@ func (h *TemplatesHandler) List(c *gin.Context) {
Runtime string `yaml:"runtime"`
Model string `yaml:"model"`
Skills []string `yaml:"skills"`
Displayable *bool `yaml:"displayable"`
// Top-level `providers:` block — structured registry. Distinct
// from runtime_config.providers (slug list) below. Both shapes
// coexist in production: claude-code ships the structured
@@ -334,6 +344,7 @@ func (h *TemplatesHandler) List(c *gin.Context) {
Skills: raw.Skills,
SkillCount: len(raw.Skills),
ProvisionTimeoutSeconds: raw.RuntimeConfig.ProvisionTimeoutSeconds,
Displayable: raw.Displayable,
}
// internal#718 P3: serve the SELECTABLE provider/model list from
@@ -1554,3 +1554,86 @@ skills: []
t.Errorf("template Providers unchanged: got %v", got.Providers)
}
}
// TestTemplatesList_DisplayableFlag verifies the SSOT-driven runtime-picker
// opt-out: a template's config.yaml `displayable: false` surfaces as a
// non-nil false on the /templates row (canvas hides it), while an absent
// flag stays nil (canvas shows it) and an explicit true surfaces as true.
// This is the backend half of removing the hardcoded frontend allowlist —
// the picker trusts this list, so hiding a runtime must be declarative here.
func TestTemplatesList_DisplayableFlag(t *testing.T) {
setupTestDB(t)
setupTestRedis(t)
tmpDir := t.TempDir()
mk := func(dir, yaml string) {
d := filepath.Join(tmpDir, dir)
if err := os.MkdirAll(d, 0755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(d, "config.yaml"), []byte(yaml), 0644); err != nil {
t.Fatal(err)
}
}
// absent → nil
mk("adk-shown", "name: ADK Shown\nruntime: claude-code\n")
// explicit false → hidden marker
mk("adk-hidden", "name: ADK Hidden\nruntime: claude-code\ndisplayable: false\n")
// explicit true → shown marker
mk("adk-explicit", "name: ADK Explicit\nruntime: claude-code\ndisplayable: true\n")
handler := NewTemplatesHandler(tmpDir, nil, nil)
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Request = httptest.NewRequest("GET", "/templates", nil)
handler.List(c)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp []templateSummary
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("parse: %v", err)
}
byID := map[string]templateSummary{}
for _, s := range resp {
byID[s.ID] = s
}
if s, ok := byID["adk-shown"]; !ok {
t.Fatal("adk-shown missing")
} else if s.Displayable != nil {
t.Errorf("adk-shown: expected nil Displayable (absent), got %v", *s.Displayable)
}
if s, ok := byID["adk-hidden"]; !ok {
t.Fatal("adk-hidden missing")
} else if s.Displayable == nil || *s.Displayable != false {
t.Errorf("adk-hidden: expected non-nil false Displayable, got %v", s.Displayable)
}
if s, ok := byID["adk-explicit"]; !ok {
t.Fatal("adk-explicit missing")
} else if s.Displayable == nil || *s.Displayable != true {
t.Errorf("adk-explicit: expected non-nil true Displayable, got %v", s.Displayable)
}
// JSON contract: omitempty drops the field entirely when nil so existing
// templates' payloads are byte-unchanged; present when set.
var rawRows []map[string]json.RawMessage
if err := json.Unmarshal(w.Body.Bytes(), &rawRows); err != nil {
t.Fatalf("raw parse: %v", err)
}
for _, row := range rawRows {
id := ""
_ = json.Unmarshal(row["id"], &id)
_, present := row["displayable"]
if id == "adk-shown" && present {
t.Error("adk-shown: displayable key should be omitted when nil")
}
if (id == "adk-hidden" || id == "adk-explicit") && !present {
t.Errorf("%s: displayable key should be present when set", id)
}
}
}
@@ -1,200 +0,0 @@
package handlers
// Sqlmock-backed coverage for workspace_abilities.go (PatchAbilities).
// Closes #1312 — handler was at 0% coverage.
import (
"bytes"
"errors"
"net/http"
"net/http/httptest"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/gin-gonic/gin"
)
func patchAbilitiesReq(t *testing.T, wsID string, body string) *httptest.ResponseRecorder {
t.Helper()
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Params = gin.Params{{Key: "id", Value: wsID}}
c.Request = httptest.NewRequest("PATCH", "/workspaces/"+wsID+"/abilities", bytes.NewBufferString(body))
c.Request.Header.Set("Content-Type", "application/json")
PatchAbilities(c)
return w
}
// ---------- Validation errors ----------
func TestPatchAbilities_InvalidWorkspaceID(t *testing.T) {
w := patchAbilitiesReq(t, "not-a-uuid", `{"broadcast_enabled":true}`)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
func TestPatchAbilities_InvalidJSON(t *testing.T) {
w := patchAbilitiesReq(t, wsUUID1, `not json`)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
func TestPatchAbilities_EmptyBody(t *testing.T) {
w := patchAbilitiesReq(t, wsUUID1, `{}`)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
// ---------- Not found ----------
func TestPatchAbilities_WorkspaceNotFound(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(false))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true}`)
if w.Code != http.StatusNotFound {
t.Fatalf("expected 404, got %d: %s", w.Code, w.Body.String())
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestPatchAbilities_ExistsQueryError(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnError(errors.New("conn refused"))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true}`)
if w.Code != http.StatusNotFound {
t.Fatalf("expected 404 on exists query error, got %d: %s", w.Code, w.Body.String())
}
}
// ---------- Happy paths ----------
func TestPatchAbilities_BroadcastOnly(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET broadcast_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, true).
WillReturnResult(sqlmock.NewResult(0, 1))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true}`)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestPatchAbilities_TalkToUserOnly(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET talk_to_user_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, false).
WillReturnResult(sqlmock.NewResult(0, 1))
w := patchAbilitiesReq(t, wsUUID1, `{"talk_to_user_enabled":false}`)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
func TestPatchAbilities_BothFields(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET broadcast_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, true).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectExec(`UPDATE workspaces SET talk_to_user_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, true).
WillReturnResult(sqlmock.NewResult(0, 1))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true,"talk_to_user_enabled":true}`)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("unmet: %v", err)
}
}
// ---------- DB errors on update ----------
func TestPatchAbilities_BroadcastUpdateError(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET broadcast_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, true).
WillReturnError(errors.New("disk full"))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true}`)
if w.Code != http.StatusInternalServerError {
t.Fatalf("expected 500, got %d: %s", w.Code, w.Body.String())
}
}
func TestPatchAbilities_TalkToUserUpdateError(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET talk_to_user_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, false).
WillReturnError(errors.New("disk full"))
w := patchAbilitiesReq(t, wsUUID1, `{"talk_to_user_enabled":false}`)
if w.Code != http.StatusInternalServerError {
t.Fatalf("expected 500, got %d: %s", w.Code, w.Body.String())
}
}
func TestPatchAbilities_BothFields_BroadcastFails(t *testing.T) {
mock, cleanup := withMockDB(t)
defer cleanup()
mock.ExpectQuery(`SELECT EXISTS\(SELECT 1 FROM workspaces WHERE id = \$1 AND status != 'removed'\)`).
WithArgs(wsUUID1).
WillReturnRows(sqlmock.NewRows([]string{"exists"}).AddRow(true))
mock.ExpectExec(`UPDATE workspaces SET broadcast_enabled = \$2, updated_at = now\(\) WHERE id = \$1`).
WithArgs(wsUUID1, true).
WillReturnError(errors.New("disk full"))
w := patchAbilitiesReq(t, wsUUID1, `{"broadcast_enabled":true,"talk_to_user_enabled":true}`)
if w.Code != http.StatusInternalServerError {
t.Fatalf("expected 500, got %d: %s", w.Code, w.Body.String())
}
}
@@ -82,23 +82,18 @@ func (h *BroadcastHandler) Broadcast(c *gin.Context) {
// Find the sender's org root by walking the parent_id chain.
// Workspaces with parent_id = NULL are org roots; every other workspace
// belongs to the org identified by its topmost ancestor.
//
// NOTE: this uses the corrected CTE from org_scope.go (#1954). The old
// shape carried `id AS root_id` from the recursive seed, which caused a
// non-root sender to resolve to itself rather than its org root, making
// broadcasts under-deliver (miss the rest of the org). See #1959.
var orgRootID string
err = db.DB.QueryRowContext(ctx, `
WITH RECURSIVE org_chain AS (
SELECT id, parent_id
SELECT id, parent_id, id AS root_id
FROM workspaces
WHERE id = $1
UNION ALL
SELECT w.id, w.parent_id
SELECT w.id, w.parent_id, c.root_id
FROM workspaces w
JOIN org_chain c ON w.id = c.parent_id
)
SELECT id AS root_id FROM org_chain WHERE parent_id IS NULL LIMIT 1
SELECT root_id FROM org_chain WHERE parent_id IS NULL LIMIT 1
`, senderID).Scan(&orgRootID)
if err != nil {
log.Printf("Broadcast: org root lookup for %s: %v", senderID, err)
@@ -22,8 +22,8 @@ import (
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/models"
"github.com/DATA-DOG/go-sqlmock"
"github.com/gin-gonic/gin"
)
@@ -259,11 +259,13 @@ func TestWorkspaceBudget_A2A_ExceededReturns402(t *testing.T) {
// Cache a URL so resolveAgentURL doesn't need a DB query after budget check
mr.Set(fmt.Sprintf("ws:%s:url", "ws-over-budget"), "http://localhost:9999")
// Budget check query: spend = limit → exceeded
mock.ExpectQuery("SELECT budget_limit, COALESCE").
// Budget check: monthly limit 500, monthly spend 500 → exceeded → 402
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-over-budget").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(500), int64(500)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{"monthly":500}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-over-budget").
WillReturnRows(spendRows(0, 0, 0, 500))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
@@ -295,10 +297,12 @@ func TestWorkspaceBudget_A2A_AboveLimitReturns402(t *testing.T) {
mr.Set(fmt.Sprintf("ws:%s:url", "ws-way-over"), "http://localhost:9999")
// spend > limit
mock.ExpectQuery("SELECT budget_limit, COALESCE").
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-way-over").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(100), int64(9999)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{"monthly":100}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-way-over").
WillReturnRows(spendRows(0, 0, 0, 9999))
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
@@ -334,11 +338,13 @@ func TestWorkspaceBudget_A2A_UnderLimitPassesThrough(t *testing.T) {
mr.Set(fmt.Sprintf("ws:%s:url", "ws-under-budget"), agentServer.URL)
// Budget check: spend (100) < limit (500) → pass-through
mock.ExpectQuery("SELECT budget_limit, COALESCE").
// Budget check: monthly spend (100) < limit (500) → pass-through
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-under-budget").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(int64(500), int64(100)))
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{"monthly":500}`)))
mock.ExpectQuery(`FROM workspace_spend_events`).
WithArgs("ws-under-budget").
WillReturnRows(spendRows(0, 0, 0, 100))
// Activity log INSERT from logA2ASuccess
mock.ExpectExec("INSERT INTO activity_logs").
@@ -380,11 +386,11 @@ func TestWorkspaceBudget_A2A_NilLimitPassesThrough(t *testing.T) {
mr.Set(fmt.Sprintf("ws:%s:url", "ws-no-limit"), agentServer.URL)
// budget_limit NULL → no enforcement regardless of monthly_spend
mock.ExpectQuery("SELECT budget_limit, COALESCE").
// no limits configured → checkWorkspaceBudget returns early (no spend query),
// enforcement skipped regardless of spend
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-no-limit").
WillReturnRows(sqlmock.NewRows([]string{"budget_limit", "monthly_spend"}).
AddRow(nil, int64(999999))) // huge spend but no limit set
WillReturnRows(sqlmock.NewRows([]string{"budget_limits"}).AddRow([]byte(`{}`)))
mock.ExpectExec("INSERT INTO activity_logs").
WillReturnResult(sqlmock.NewResult(0, 1))
@@ -425,7 +431,7 @@ func TestWorkspaceBudget_A2A_DBErrorFailOpen(t *testing.T) {
mr.Set(fmt.Sprintf("ws:%s:url", "ws-db-err-budget"), agentServer.URL)
// Budget check fails with DB error → fail-open (request proceeds)
mock.ExpectQuery("SELECT budget_limit, COALESCE").
mock.ExpectQuery(`SELECT COALESCE\(budget_limits`).
WithArgs("ws-db-err-budget").
WillReturnError(sql.ErrConnDone)
@@ -65,6 +65,14 @@ func validateWorkspaceCompute(compute models.WorkspaceCompute) error {
if err := validateWorkspaceDisplayDimensions(compute.Display.Width, compute.Display.Height); err != nil {
return err
}
// internal#734: the durable-data choice. CP re-validates the same enum at
// its provision edge (IsValidDataPersistence → 400); validating here too
// gives the user a clear workspace-server error before the CP round-trip.
switch compute.DataPersistence {
case "", "persist", "ephemeral":
default:
return fmt.Errorf("unsupported compute.data_persistence (want persist|ephemeral)")
}
return nil
}
@@ -11,8 +11,8 @@ import (
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"git.moleculesai.app/molecule-ai/molecule-core/workspace-server/internal/models"
"github.com/DATA-DOG/go-sqlmock"
"github.com/gin-gonic/gin"
)
@@ -36,6 +36,23 @@ func TestValidateWorkspaceCompute_RejectsUnknownInstanceType(t *testing.T) {
}
}
// internal#734: data_persistence enum. "" (auto), "persist", "ephemeral" are
// the only accepted values; anything else is a clear 400 before the CP call.
func TestValidateWorkspaceCompute_DataPersistence(t *testing.T) {
for _, ok := range []string{"", "persist", "ephemeral"} {
c := models.WorkspaceCompute{DataPersistence: ok}
if err := validateWorkspaceCompute(c); err != nil {
t.Errorf("data_persistence=%q must be accepted: %v", ok, err)
}
}
for _, bad := range []string{"persistent", "off", "none", "Ephemeral", "true"} {
c := models.WorkspaceCompute{DataPersistence: bad}
if err := validateWorkspaceCompute(c); err == nil {
t.Errorf("data_persistence=%q must be rejected", bad)
}
}
}
func TestValidateWorkspaceCompute_RejectsOutOfRangeRootVolume(t *testing.T) {
for _, rootGB := range []int{29, 501} {
compute := models.WorkspaceCompute{Volume: models.WorkspaceComputeVolume{RootGB: rootGB}}
@@ -399,7 +399,13 @@ func (h *WorkspaceHandler) Delete(c *gin.Context) {
// disable, broadcast). The HTTP-specific bits — direct-children 409
// gate above, ?purge=true hard-delete below, response shaping —
// stay in this handler.
descendantIDs, stopErrs, err := h.CascadeDelete(ctx, id)
// internal#734: the user can ask to erase saved data (browser profile /
// cookies / downloads / agent memory) on delete. Opt-in — default keeps the
// data on its volume for the orphan-sweeper grace. Only a genuine
// permanent-delete reaches here (restart/reconcile use other paths), so this
// is the one place prune may be requested.
erase := c.Query("erase_data") == "true"
descendantIDs, stopErrs, err := h.CascadeDelete(ctx, id, erase)
if err != nil {
// Audit 2026-05-09 (Core-Security): raw `err.Error()` here was
// exposed to HTTP clients verbatim, including wrapped lib/pq
@@ -515,7 +521,13 @@ func destructiveDeleteCounts(ctx context.Context, id string) (childCount int, sc
// Caller is responsible for the children-confirmation gate (the HTTP handler
// returns 409 when children exist + ?confirm=true is missing); this helper
// always cascades.
func (h *WorkspaceHandler) CascadeDelete(ctx context.Context, id string) ([]string, []error, error) {
// CascadeDelete tears down a workspace and its descendants (stop compute,
// remove volumes, revoke tokens, disable schedules, broadcast). erase=true
// (internal#734) means the user asked to erase saved data, so the CP compute
// teardown prunes each workspace's durable data volume; the HTTP delete passes
// the user's choice, the org-import reconcile passes false (a reconcile is not
// a user-erase).
func (h *WorkspaceHandler) CascadeDelete(ctx context.Context, id string, erase bool) ([]string, []error, error) {
if err := validateWorkspaceID(id); err != nil {
return nil, nil, err
}
@@ -579,7 +591,7 @@ func (h *WorkspaceHandler) CascadeDelete(ctx context.Context, id string) ([]stri
// pending EC2 is queryable and handed off to the CP-orphan-sweeper —
// rather than the bare one-shot StopWorkspaceAuto that produced the
// silent-leak class (task #15 / workspace-ec2-leak).
if err := h.stopWorkspaceForDelete(cleanupCtx, wsID); err != nil {
if err := h.stopWorkspaceForDelete(cleanupCtx, wsID, erase); err != nil {
log.Printf("CascadeDelete %s stop failed: %v — leaving cleanup for orphan sweeper", wsID, err)
stopErrs = append(stopErrs, fmt.Errorf("stop %s: %w", wsID, err))
return
@@ -521,7 +521,7 @@ func TestValidateWorkspaceDir_Empty(t *testing.T) {
func TestCascadeDelete_InvalidUUID(t *testing.T) {
h := &WorkspaceHandler{}
descendants, stopErrs, err := h.CascadeDelete(context.Background(), "not-a-uuid")
descendants, stopErrs, err := h.CascadeDelete(context.Background(), "not-a-uuid", false)
if err == nil {
t.Error("expected error for invalid UUID")
}
@@ -542,7 +542,7 @@ func TestCascadeDelete_DescendantQueryError(t *testing.T) {
WithArgs(wsID).
WillReturnError(sql.ErrConnDone)
deleted, stopErrs, err := h.CascadeDelete(context.Background(), wsID)
deleted, stopErrs, err := h.CascadeDelete(context.Background(), wsID, false)
if err == nil {
t.Error("CascadeDelete returned nil error; want descendant query error")
}
@@ -569,7 +569,7 @@ func TestCascadeDelete_DescendantRowsError(t *testing.T) {
WithArgs(wsID).
WillReturnRows(rows)
deleted, stopErrs, err := h.CascadeDelete(context.Background(), wsID)
deleted, stopErrs, err := h.CascadeDelete(context.Background(), wsID, false)
if err == nil {
t.Fatal("CascadeDelete returned nil error; want descendant rows error")
}
@@ -45,7 +45,7 @@ func TestStopWorkspaceForDelete_CPRetriesTransientThenSucceeds(t *testing.T) {
}}
h := &WorkspaceHandler{cpProv: stub}
err := h.stopWorkspaceForDelete(context.Background(), "ws-del-1")
err := h.stopWorkspaceForDelete(context.Background(), "ws-del-1", false)
if err != nil {
t.Fatalf("expected nil error on eventual success, got %v", err)
}
@@ -73,7 +73,7 @@ func TestStopWorkspaceForDelete_CPExhaustsEmitsDurableEventAndReturnsError(t *te
mock.ExpectExec("INSERT INTO structure_events").
WillReturnResult(sqlmock.NewResult(0, 1))
err := h.stopWorkspaceForDelete(context.Background(), "ws-doomed")
err := h.stopWorkspaceForDelete(context.Background(), "ws-doomed", false)
if err == nil {
t.Fatal("expected terminal error on retry exhaustion, got nil")
}
@@ -96,7 +96,7 @@ func TestStopWorkspaceForDelete_CPExhaustsEmitsDurableEventAndReturnsError(t *te
func TestStopWorkspaceForDelete_NoBackendIsNoOp(t *testing.T) {
h := &WorkspaceHandler{} // cpProv nil, provisioner nil
if err := h.stopWorkspaceForDelete(context.Background(), "ws-x"); err != nil {
if err := h.stopWorkspaceForDelete(context.Background(), "ws-x", false); err != nil {
t.Errorf("expected nil no-op with no backend, got %v", err)
}
}
@@ -235,9 +235,13 @@ func (h *WorkspaceHandler) StopWorkspaceAuto(ctx context.Context, workspaceID st
// container won't heal on retry (matches RestartWorkspaceAuto's Docker
// rationale); the orphan-container sweeper (registry/orphan_sweeper.go) is
// the Docker-side backstop.
func (h *WorkspaceHandler) stopWorkspaceForDelete(ctx context.Context, workspaceID string) error {
// stopWorkspaceForDelete terminates a workspace's compute on the delete path.
// erase=true (internal#734) means the user asked to erase saved data, so the CP
// teardown prunes the durable data volume. The local-docker path always removes
// its volume via CascadeDelete's RemoveVolume, so erase is a CP-only concern.
func (h *WorkspaceHandler) stopWorkspaceForDelete(ctx context.Context, workspaceID string, erase bool) error {
if h.cpProv != nil {
if err := h.cpStopWithRetryErr(ctx, workspaceID, "Delete"); err != nil {
if err := h.cpStopWithRetryErr(ctx, workspaceID, "Delete", erase); err != nil {
h.emitDeleteTerminateRetryExhausted(ctx, workspaceID, err)
return err
}

Some files were not shown because too many files have changed in this diff Show More