core-lead/molecule-core
1
0
Fork 0
forked from molecule-ai/molecule-core
Code Pull Requests Activity
2,812 Commits 73 Branches 39 Tags 58 MiB
23e329aa4c
Commit Graph

296 Commits

Author SHA1 Message Date
Hongming Wang
2821b979f2
Merge pull request #1994 from Molecule-AI/fix/canvas-multilevel-layout-ux 
fix(canvas): subtree-aware layout + org-import reliability + UX polish
 2026-04-24 06:57:10 +00:00
Hongming Wang
8c80175cd8 fix(canvas): subtree-aware layout + org-import reliability + UX polish 
Five tightly-related fixes surfaced while stress-testing org-template
imports (Legal Team, Molecule Company, etc.) on a running control plane:

1) Org import was silently failing — INSERT wrote `collapsed` into the
   `workspaces` table but that column lives on `canvas_layouts`
   (005_canvas_layouts.sql). Every import returned 207 with 0 rows
   created, which `api.post` treated as success → green "Imported"
   toast + empty canvas. Moved the write to canvas_layouts; updated
   the workspace_crud PATCH path to UPSERT there too; refreshed the
   test mock. Added a client-side assertion that throws on
   2xx-with-`error`-body so future partial-failures surface a red
   toast rather than lying about success.

2) Multi-level nested layout was collision-prone: children that were
   themselves parents (CTO → Dev Lead → 6 engineers) got the same
   leaf-sized grid slot as leaf siblings and clipped into each other.
   Added post-order `sizeOfSubtree` + sibling-size-aware
   `childSlotInGrid` on both the Go server and the TS client (kept in
   sync). `buildNodesAndEdges` now uses subtree sizes for both parent
   dimensions and the rescue heuristic. `setCollapsed` on expand now
   reads each child's actual rendered width/height instead of the
   leaf-count formula — a regression test covers the CTO/Dev Lead
   scenario.

3) Provisioning-timeout banner was unusable during large imports: a
   30-workspace tree triggered 27 simultaneous "stuck" warnings 2
   minutes in (server paces + provision concurrency = 3 guarantee tail
   items legitimately wait longer). Scaled threshold with concurrent
   count (base + 45s per queue slot beyond concurrency) and added a
   Dismiss (×) button per banner.

4) Auto pan-and-zoom on org ready: after the last workspace flips out
   of `provisioning`, canvas now fitView's with a 1.2s animation,
   0.25 padding, `maxZoom: 0.8` and `minZoom: 0.25`. Without the zoom
   caps fitView was hitting the component's maxZoom=2 on small trees
   and zooming in instead of out.

5) Toolbar was visually busy: `+ N sub` count wrapped onto a second
   row on narrow viewports; status dot and workspace total were in
   separate border-delimited cells. Merged into one segment with
   `whitespace-nowrap`; A2A / Audit / Search / Help collapsed to
   icon-only 28px buttons with tooltip + aria-label (Figma/Linear
   pattern). Stop All / Restart Pending keep text — they're urgent.

Also:
- `api.{get,post,...}` accept an optional `{ timeoutMs }` so callers
  that hit intentionally-slow endpoints (org import paces 2s between
  siblings) don't trip the 15s default and report false aborts.
- `WorkspaceNode` clamps role text to 2 lines so verbose descriptions
  don't unboundedly grow card height and break the grid.
- `PARENT_HEADER_PADDING` bumped 44→130 to clear name + runtime +
  2-line role + the currentTask banner that appears during the
  initial-prompt phase.

Tests: 930 canvas tests + full Go handler suite pass. Added
regressions for (i) 207 partial-success surfacing as throw, and
(ii) setCollapsed sizing with nested-parent children.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 23:48:29 -07:00
molecule-ai[bot]
e4e389950f
fix(canvas/a11y): aria-hidden SVGs, MissingKeysModal dialog semantics, session cookie auth (#1992) 
fix(canvas/a11y): aria-hidden SVGs, MissingKeysModal dialog semantics, session cookie auth

Three fixes cherry-picked from issue #1744:

1. aria-hidden on decorative SVG icons:
   - DeleteCascadeConfirmDialog.tsx: warning triangle SVG gets aria-hidden="true"
   - MissingKeysModal.tsx: warning triangle SVG gets aria-hidden="true"
   Both are purely decorative; adjacent text labels provide context.

2. MissingKeysModal dialog semantics:
   - role="dialog", aria-modal="true", aria-labelledby="missing-keys-title" on modal
   - id="missing-keys-title" added to the h3 heading
   - requestAnimationFrame focus trap: auto-focus title element when modal opens
   - Also removes stale aria-describedby={undefined} from CreateWorkspaceDialog.tsx

3. Session cookie auth for /registry/:id/peers:
   - Promotes VerifiedCPSession() fallback before the bearer token branch
   - Fixes SaaS canvas Peers tab 401 — canvas hits this endpoint via session cookie
   - Correctly returns "invalid session" for bad cookies instead of falling through
   - Self-hosted bypass logic preserved

Test fix (bundled, same branch):
   - ContextMenu keyboard test: add getState() stub to useCanvasStore mock
   - Required after ContextMenu.tsx gained a direct getState() call at line 169

Reviewed-by: Core-Security (security audit: APPROVED)
CI: Canvas CI ✅, Platform CI ✅, E2E API ✅, CodeQL ✅

GitHub issue: #1740 (test), #1744 (a11y)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-24 06:20:32 +00:00
molecule-ai[bot]
01fcc9a4b6
fix(canvas/a11y): aria-hidden SVGs, MissingKeysModal dialog, session cookie auth 
* fix(canvas/a11y): aria-hidden SVGs, MissingKeysModal dialog semantics, session cookie auth

Three fixes cherry-picked from issue #1744:

1. aria-hidden on decorative SVG icons:
   - DeleteCascadeConfirmDialog.tsx: warning triangle SVG gets aria-hidden="true"
   - MissingKeysModal.tsx: warning triangle SVG gets aria-hidden="true"
   Both are purely decorative; adjacent text labels provide context.

2. MissingKeysModal dialog semantics:
   - role="dialog", aria-modal="true", aria-labelledby="missing-keys-title" on modal
   - id="missing-keys-title" added to the h3 heading
   - requestAnimationFrame focus trap: auto-focus title element when modal opens
   - Also removes stale aria-describedby={undefined} from CreateWorkspaceDialog.tsx

3. Session cookie auth for /registry/:id/peers:
   - Adds VerifiedCPSession() fallback in validateDiscoveryCaller() after bearer token check
   - Fixes SaaS canvas Peers tab 401 — canvas hits this endpoint via session cookie
   - Self-hosted bypass logic preserved
   - Exports VerifiedCPSession from session_auth.go for cross-package use

Test fix (bundled, same branch):
   - ContextMenu keyboard test: add getState() stub to useCanvasStore mock
   - Required after ContextMenu.tsx gained a direct getState() call at line 169

GitHub issue: #1740 (test), #1744 (a11y)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(workspace-server): remove duplicate VerifiedCPSession declaration

The branch accidentally added a second func VerifiedCPSession declaration
that shadows the real implementation, causing go build to fail with:
  internal/middleware/session_auth.go:238:6: VerifiedCPSession redeclared in this block

Remove the stub alias so the original full implementation is used directly.
The function already exports correctly for cross-package use via the
VerifiedCPSession() call in discovery.go.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(workspace-server): correct VerifiedCPSession condition in discovery.go

Fix Go build error — 'presented' was declared and not used.
The cookie fallback check was using `if ok, presented := ...; ok` instead
of `if ok, presented := ...; presented`, causing the build to fail in CI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(workspace-server): fix declared and not used 'presented' in discovery.go

Fixes Go build failure:
  discovery.go:355:10: declared and not used: presented
  discovery.go:358:6: undefined: presented

Variable shadowing in the second VerifiedCPSession call reused the outer
scope's `ok` and `presented` names, causing a compile error. Renamed to
ok2/presented2 to avoid shadowing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Molecule AI Core-FE <core-fe@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-24 04:30:26 +00:00
Hongming Wang
d53583f9c6 Merge remote-tracking branch 'origin/staging' into fix/restore-quickstart-plus-hotfixes 2026-04-23 21:04:55 -07:00
Hongming Wang
f2a4b6e0d3 fix: dev-mode bypass for IP rate limiter + 429 retry on GET 
The 600-req/min/IP bucket is sized for SaaS where each tenant has
a distinct client IP. On a local Docker setup every panel shares
one IP — hydration (/workspaces + /templates + /org/templates +
/approvals/pending) plus polling (A2A overlay + activity tabs +
approvals + schedule + channels + audit trail) can burst past the
bucket inside a minute, blanking the canvas with 429s. The user
reported it after dragging workspaces — dragging itself is
release-only (savePosition in onNodeDragStop), but the polling
that's always running added onto startup tripped the limit.

Two-layer fix:

Server: RateLimiter.Middleware short-circuits when isDevModeFailOpen
is true (MOLECULE_ENV=development + empty ADMIN_TOKEN), matching
the Tier-1b hatch already applied to AdminAuth, WorkspaceAuth, and
discovery. SaaS production keeps the bucket.

Client: api.ts auto-retries a single 429 on idempotent GET requests,
waiting the server-provided Retry-After (capped at 20s). Mutations
(POST/PUT/PATCH/DELETE) never auto-retry to avoid double-applying.
Users on SaaS hitting a legitimate rate-limit spike get one
transparent recovery instead of an immediately-blank Canvas.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 20:44:09 -07:00
Hongming Wang
286dcbfd1e fix(canvas,org): collapse org-imported parents on first paint 
Importing a 15-workspace org template dropped every child as a
freely-positioned card into its parent's coordinate space. Parents
with 5-10 kids had the kids spill below the parent's initial min
size, producing the "ugly default" layout the user just flagged —
a mess of overlapping cards the moment the import completed.

Fix: every workspace in an org-template import that HAS children
is inserted with `collapsed = true`. Leaf workspaces stay
expanded (nothing to hide). The canvas renders a collapsed
parent as a compact header-only card with its "N sub" badge —
visually identical to the pre-refactor default the user asked for.

Double-click on a collapsed parent now EXPANDS it (flipping
`collapsed` locally + persisting via PATCH) so the user can drill
in to see the subtree. Only once expanded does a second
double-click zoom-to-team, matching the prior behaviour.

Leaf-first creation order stays the same; the collapsed flag
just means "render compact" not "hide from API".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 20:36:55 -07:00
Hongming Wang
507696d88a fix(canvas,server): address review findings on 3f11df03 
Five review findings from the 3f11df03 six-bug commit:

1. Add TestPeers_DevModeFailOpen_{Allows,ClosedWhenAdminTokenSet,
   ClosedInProduction} covering all three gating states for the
   security-sensitive dev-mode hatch the prior commit added to
   /registry/:id/peers. Previously shipped untested — a future
   refactor could have silently inverted polarity or removed the
   gate. New tests pin the contract:
     * MOLECULE_ENV=development + ADMIN_TOKEN="" → allow bearerless
     * MOLECULE_ENV=development + ADMIN_TOKEN set → require token
     * MOLECULE_ENV=production                    → require token

2. ConfigTab handleSave diffs against the RAW parsed YAML / form
   config instead of the DEFAULT_CONFIG-merged shape. The previous
   code would silently PATCH tier=1 to the DB when a user deleted
   the `tier:` line in raw mode (the default-merge substituted 1).
   Now: only fields the user actually typed participate in the
   diff. Type guards (typeof === "number" / "string") prevent
   coercion surprises on malformed YAML.

3. ConfigTab model-save failure no longer lies "Saved". The
   /workspaces/:id/model PATCH can reject when the runtime doesn't
   support the chosen model; previously we caught + console.warn'd
   + showed green Saved, and the user watched the model revert on
   next reload with no explanation. Now the save path collects a
   `modelSaveError` and surfaces it via setError with a partial-
   success message ("Other fields saved, but model update failed:
   …") so the user sees why.

4. ChannelsTab now surfaces BOTH channels-fetch and adapters-fetch
   failures, distinguishing them in the error text ("Failed to
   load connected channels and platforms — try refreshing").
   Previously only an adapters failure was visible; a channels
   failure left the user with an apparently-empty list and no
   indication the API was unreachable.

5. ChatTab panels drop the redundant aria-hidden attribute. The
   `hidden`/`flex` Tailwind class already sets display:none, which
   removes the node from the accessibility tree on its own; the
   extra aria-hidden invited WAI-ARIA lint warnings if a focusable
   descendant ever landed inside an inactive panel.

Tests: 923 canvas + full Go handler suite pass. 3 new Go tests.
No behaviour change on the five prior fixes — this commit tightens
their edges per the independent review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 20:29:44 -07:00
Hongming Wang
3f11df031c fix: six UX bugs (peers auth, scroll, chat tabs, config persist, + visibility) 
Six bugs reported from a live session — all shippable in one commit:

1. Peers tab 401 on local Docker. The /registry/:id/peers endpoint
   demands a workspace-scoped bearer token (validateDiscoveryCaller)
   which the canvas session doesn't hold. Added the same Tier-1b
   dev-mode fail-open hatch that AdminAuth and WorkspaceAuth already
   use — gated by MOLECULE_ENV=development + empty ADMIN_TOKEN, so
   SaaS production stays strict. Exported IsDevModeFailOpen from the
   middleware package for the handler layer to reuse.

2. Org Templates list unscrollable. OrgTemplatesSection was rendered
   in the TemplatePalette footer — a div without overflow — so when
   it expanded to 15+ entries the list extended past the viewport
   with no scroll. Moved it to the top of the flex-1 overflow-y-auto
   container. Tall lists now scroll naturally.

3. Chat tab: "My Chat" and "Agent Comms" rendered stacked instead
   of switching. HTML `hidden` attribute was being overridden by
   Tailwind's `flex` class (display: flex beats the attribute),
   so both tabpanels rendered concurrently. Swapped to a conditional
   Tailwind `hidden`/`flex` class so the inactive panel is
   display:none with proper CSS specificity.

4. Hermes Config form never persists. handleSave wrote config.yaml
   but name / tier / runtime / model all live on the workspace row
   (or the dedicated /workspaces/:id/model endpoint) — the form
   edited in-memory, the request returned 200, the next reload
   wiped everything back. Hermes + external runtimes manage their
   own config inside the container anyway, so writing config.yaml
   is a no-op for them; skip it. Always diff and PATCH the DB-backed
   fields that actually changed.

5. Channels "+ Connect" dropdown empty on first open. ChannelsTab's
   load() used Promise.all with a silent catch — if EITHER the
   channels or adapters fetch failed, both setters were skipped
   with no error visible. Switched to Promise.allSettled so each
   endpoint settles independently, and the adapters failure now
   surfaces via the top-level error state.

6. Plugin registry always "No plugins in registry". Same silent
   catch pattern in SkillsTab.tsx — load errors for /plugins,
   /plugins/sources, and /workspaces/:id/plugins swallowed without
   logging. Replaced the empty catches with console.warn so future
   failures are at least visible in devtools.

Tests: 923 passing (unchanged). Go handler tests pass. Server
rebuilt and running with the peers-auth + collapsed-persistence
fixes (pid 15875).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 20:18:30 -07:00
Molecule AI Core-UIUX
8fb5ec0340 fix(handlers): fix Go scoping — presented must live in function scope 
The short-var declaration inside the if-initializer scoped `presented`
only to that if statement, making it undefined on the following
`if presented { ... }` block. Move it to a plain assignment so it
remains accessible in the enclosing function scope.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-24 03:10:18 +00:00
Molecule AI Core-UIUX
a46797d466 fix(middleware): rename internal fn to verifiedCPSession, keep public alias 
The PR #1855 branch contains a newer version of session_auth.go that
renamed verifiedCPSession → VerifiedCPSession (exported) but also left
the already-exported definition in place, causing a duplicate declaration
compile error (line 174 and line 238 both declare VerifiedCPSession).

Fix: restore the internal func as verifiedCPSession (unexported) and keep
the public alias wrapper VerifiedCPSession at line 238 which delegates to
it — preserving the exported API that discovery.go and wsauth_middleware.go
depend on.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-24 03:10:18 +00:00
Molecule AI Core-QA
680f1f50f2 fix(canvas/a11y): restore aria-hidden on backdrop div after cherry-pick conflict 
Cherry-pick from #1744 left the backdrop div without aria-hidden="true"
(the outer dialog div got it instead). Re-apply aria-hidden="true" to
the backdrop div so screen readers skip the clickable overlay layer.

Also revert test assertion from bg-black → bg-black/70 to match the
exact class applied to the backdrop div.
 2026-04-24 03:10:18 +00:00
Hongming Wang
4fd7f1e84c fix(canvas): tighten rescue + cap toast + cover paths with tests 
Three follow-up review findings from the c2b2e13a review:

1. Rescue heuristic uses pure bbox-non-overlap. The previous
   `position.x < 0` branch rescued any child whose parent was
   later dragged past it, even when the layout was clearly
   recoverable (e.g. relative -40, child still overlaps parent).
   New rule: rescue iff the child's bbox has zero overlap with
   the parent's bbox — self-calibrating, scales with user-resized
   parents, catches screenshot-case and legacy huge-positive data.

2. Toast caps failed-name list at 3 and appends "and N more".
   Stops a 50-node partial failure from overflowing the toast
   container.

3. Cycle guard on selection-roots walk in batchNest. Corrupt
   parentId data can't send the loop infinite now. Cheap
   defensive guard — one Set per selected node.

Tests added (923 total, up from 918):
 * canvas-topology.test: 4 rescue scenarios — screenshot case
   (zero-overlap rescue), negative drift kept, huge-positive
   rescued, user-resized layout kept.
 * canvas.test: selection-roots filter on a 3-level chain.
 * workspace_crud test: PATCH {collapsed:true} runs the UPDATE.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 20:08:14 -07:00
Hongming Wang
c2b2e13abe fix(canvas): address code-review findings on the Canvas refactor 
Five issues surfaced in the review of 50b53784. Each was either a real
bug waiting to hit users or a silent failure mode.

1. Topology rescue no longer teleports user-resized children.
   Rescue was comparing against parentMinSize(childCount), so any
   child the user had placed in space the parent was resized into
   got snapped to the default grid on reload — undoing the layout.
   Now rescue fires only on obviously corrupt data: negative
   relative coords (legacy pre-nesting absolute positions that
   landed above/left of their assigned parent) or values past an
   MAX_PLAUSIBLE_OFFSET threshold. Children just-past the initial
   minimum are left alone.

2. batchNest now filters to selection-roots before planning.
   Previously selecting both A and A's descendant B and dragging
   into T yanked B out of A to become a sibling under T. Users
   reasonably expect the A subtree to move intact. The new pass
   drops any selected node whose ancestor is also selected —
   those follow their ancestor via React Flow's parent binding.

3. batchNest surfaces partial failure via showToast. Previously
   silent: 2 of 5 PATCHes fail, user sees 3 cards re-parented + 2
   snapped back with no explanation. Now names the failed cards.

4. confirmNest closes the nest dialog BEFORE dispatching the async
   store action, so a second drag can't kick off a competing batch
   while the first is still in flight.

5. collapsed is now persisted. The Go workspace_crud.go Update
   handler ignored the `collapsed` field, so user-initiated
   collapse round-tripped to an expanded state on next hydrate.
   Added the PATCH branch (`UPDATE workspaces SET collapsed = ...`)
   so the state survives reload.

Nits cleaned:
 * Removed dead dragStartParentRef in useDragHandlers.
 * Swapped redundant `node.data as WorkspaceNodeData` casts for a
   named WorkspaceNode type alias.
 * Canvas.tsx SR-live region now reads n.parentId (matches
   MiniMap + RF's native field) instead of the mirror n.data.parentId.

Tests added (918 total, up from 915):
 * batchNest happy path — 2-root selection fires 2 combined PATCHes
   carrying parent_id + x + y, not 2×N sequential round-trips.
 * batchNest ancestor+descendant selection — subtree stays intact.
 * batchNest partial failure rollback — only the rejected nodes
   revert; successful ones stay committed.

Backend change is single-line (collapsed PATCH branch); all
workspace_crud Go tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 19:58:44 -07:00
molecule-ai[bot]
8e46cc1676
Merge branch 'staging' into test/2026-04-23-regression-suite 2026-04-24 02:45:12 +00:00
molecule-ai[bot]
3e9b7f8ad6
Merge branch 'staging' into fix/1933-bump-github-app-auth-plugin 2026-04-24 02:04:47 +00:00
molecule-ai[bot]
10c4fcc7fe
Merge branch 'staging' into test/2026-04-23-regression-suite 2026-04-24 02:04:46 +00:00
molecule-ai[bot]
e8b5f409be
test(handlers): add 5 TestKI005 terminal guard regression tests (#1938) 
* chore: sync staging to main — 1188 commits, 5 conflicts resolved (#1743)

* fix(docs): update architecture + API reference paths for workspace-server rename

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update workspace script comments for workspace-template → workspace rename

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: ChatTab comment path for workspace-server rename

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add BatchActionBar unit tests (7 tests)

Covers: render threshold, count badge, action buttons, clear selection,
ConfirmDialog trigger, ARIA toolbar role.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update publish workflow name + document staging-first flow

Default branch is now staging for both molecule-core and
molecule-controlplane. PRs target staging, CEO merges staging → main
to promote to production.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): update working-directory for workspace-server/ and workspace/ renames

- platform-build: working-directory platform → workspace-server
- golangci-lint: working-directory platform → workspace-server
- python-lint: working-directory workspace-template → workspace
- e2e-api: working-directory platform → workspace-server
- canvas-deploy-reminder: fix duplicate if: key (merged into single condition)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: add mol_pk_ and cfut_ to pre-commit secret scanner

Partner API keys (mol_pk_*) and Cloudflare tokens (cfut_*) now
caught by the pre-commit hook alongside sk-ant-, ghp_, AKIA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(canvas): enable Turbopack for dev server — faster HMR

next dev --turbopack for significantly faster dev server startup
and hot module replacement. Build script unchanged (Turbopack for
next build is still experimental).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(db): schema_migrations tracking — migrations only run once

Adds a schema_migrations table that records which migration files
have been applied. On boot, only new migrations execute — previously
applied ones are skipped. This eliminates:

- Re-running all 33 migrations on every restart
- Risk of non-idempotent DDL failing on restart
- Unnecessary log noise from re-applying unchanged schema

First boot auto-populates the tracking table with all existing
migrations. Subsequent boots only apply new ones.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scheduler): strip CRLF from cron prompts on insert/update (closes #958)

Windows CRLF in org-template prompt text caused empty agent responses
and phantom-producing detection. Strips \r at the handler level before
DB persist, plus a one-time migration to clean existing rows.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(security): strip current_task from public GET /workspaces/:id (closes #955)

current_task exposes live agent instructions to any caller with a
valid workspace UUID. Also strips last_sample_error and workspace_dir
from the public endpoint. These fields remain available through
authenticated workspace-specific endpoints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(canvas): initialize shadcn/ui — components.json + cn utility

Sets up shadcn/ui CLI so new components can be added with
`npx shadcn add <component>`. Uses new-york style, zinc base color,
no CSS variables (matches existing Tailwind-only approach).

Adds clsx + tailwind-merge for the cn() utility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(security): GLOBAL memory delimiter spoofing + pin MCP npm version

SAFE-T1201 (#807): Escape [MEMORY prefix in GLOBAL memory content on
write to prevent delimiter-spoofing prompt injection. Content stored
as "[_MEMORY " so it renders as text, not structure, when wrapped with
the real delimiter on read.

SAFE-T1102 (#805): Pin @molecule-ai/mcp-server@1.0.0 in .mcp.json.example.
Prevents supply-chain attacks via unpinned npx -y.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: schema_migrations tracking — 4 cases (first boot, re-boot, mixed, down.sql filter)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: verify current_task + last_sample_error + workspace_dir stripped from public GET

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: GLOBAL memory delimiter spoofing escape + LOCAL scope untouched

- TestCommitMemory_GlobalScope_DelimiterSpoofingEscaped: verifies [MEMORY prefix
  is escaped to [_MEMORY before DB insert (SAFE-T1201, #807)
- TestCommitMemory_LocalScope_NoDelimiterEscape: LOCAL scope stored verbatim

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(security): Phase 35.1 — SG lockdown script for tenant EC2 instances

Restricts tenant EC2 port 8080 ingress to Cloudflare IP ranges only,
blocking direct-IP access. Supports two modes:

1. Lock to CF IPs (Worker deployment): 14 IPv4 CIDR rules
2. Close ingress entirely (Tunnel deployment): removes 0.0.0.0/0 only

Usage:
  bash scripts/lockdown-tenant-sg.sh --sg-id sg-xxxxx
  bash scripts/lockdown-tenant-sg.sh --sg-id sg-xxxxx --close-ingress
  bash scripts/lockdown-tenant-sg.sh --sg-id sg-xxxxx --dry-run

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ci: update GitHub Actions to current stable versions (closes #780)

- golangci/golangci-lint-action@v4 → v9
- docker/setup-qemu-action@v3 → v4
- docker/setup-buildx-action@v3 → v4
- docker/build-push-action@v5 → v6

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs(opencode): RFC 2119 — 'should not' → 'must not' for SAFE-T1201 warning (closes #861)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(canvas): degraded badge WCAG AA contrast — amber-400 → amber-300 (closes #885)

amber-400 on zinc-900 is 5.4:1 (AA pass). amber-300 is 6.9:1 (AA+AAA pass)
and matches the rest of the amber usage in WorkspaceNode (currentTask,
error detail, badge chip).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(platform): 409 guard on /hibernate when active_tasks > 0 (closes #822)

Phase 35.1 / #799 security condition C3 — prevents operator from
accidentally killing a mid-task agent.

Behavior:
- active_tasks == 0 → proceed as before
- active_tasks > 0 && ?force=true → log [WARN] + proceed
- active_tasks > 0 && no force → 409 with {error, active_tasks}

2 new tests: TestHibernateHandler_ActiveTasks_Returns409,
TestHibernateHandler_ActiveTasks_ForceTrue_Returns200.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(platform): track last_outbound_at for silent-workspace detection (closes #817)

Sub of #795 (phantom-busy post-mortem). Adds last_outbound_at TIMESTAMPTZ
column to workspaces. Bumped async on every successful outbound A2A call
from a real workspace (skip canvas + system callers). Exposed in
GET /workspaces/:id response as "last_outbound_at".

PM/Dev Lead orchestrators can now detect workspaces that have gone silent
despite being online (> 2h + active cron = phantom-busy warning).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(workspace): snapshot secret scrubber (closes #823)

Sub-issue of #799, security condition C4. Standalone module in
workspace/lib/snapshot_scrub.py with three public functions:

- scrub_content(str) → str: regex-based redaction of secret patterns
- is_sandbox_content(str) → bool: detect run_code tool output markers
- scrub_snapshot(dict) → dict: walk memories, scrub each, drop sandbox entries

Patterns covered: sk-ant-/sk-proj-, ghp_/ghs_/github_pat_, AKIA,
cfut_, mol_pk_, ctx7_, Bearer, env-var assignments, base64 blobs ≥33 chars.

21 unit tests, 100% coverage on new code.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(security): cap webhook + config PATCH bodies (H3/H4)

Two HIGH-severity DoS surfaces: both handlers read the entire HTTP
body with io.ReadAll(r.Body) and no upper bound, so a caller streaming
a multi-gigabyte request could exhaust memory on the tenant instance
before we even validated the JSON.

H3 (Discord webhook): wrap Body in io.LimitReader with a 1 MiB cap.
Discord Interactions payloads are well under 10 KiB in practice.

H4 (workspace config PATCH): wrap Body in http.MaxBytesReader with a
256 KiB cap. Real configs are <10 KiB; jsonb handles the cap
comfortably. Returns 413 Request Entity Too Large on overflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(security): C4 — close AdminAuth fail-open race on hosted-SaaS fresh install

Pre-launch review blocker. AdminAuth's Tier-1 fail-open fired whenever
the workspace_auth_tokens table was empty — including the window between
a hosted tenant EC2 booting and the first workspace being created. In
that window, every admin-gated route (POST /org/import, POST /workspaces,
POST /bundles/import, etc.) was reachable without a bearer, letting an
attacker pre-empt the first real user by importing a hostile workspace
into a freshly provisioned instance.

Fix: fail-open is now ONLY applied when ADMIN_TOKEN is unset (self-
hosted dev with zero auth configured). Hosted SaaS always sets
ADMIN_TOKEN at provision time, so the branch never fires in prod and
requests with no bearer get 401 even before the first token is minted.

Tier-2 / Tier-3 paths unchanged.

The old TestAdminAuth_684_FailOpen_AdminTokenSet_NoGlobalTokens test
was codifying exactly this bug (asserting 200 on fresh install with
ADMIN_TOKEN set). Renamed and flipped to
TestAdminAuth_C4_AdminTokenSet_FreshInstall_FailsClosed asserting 401.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(security): scrub workspace-server token + upstream error logs

Two findings from the pre-launch log-scrub audit:

1. handlers/workspace_provision.go:548 logged `token[:8]` — the exact
   H1 pattern that panicked on short keys. Even with a length guard,
   leaking 8 chars of an auth token into centralized logs shortens the
   search space for anyone who gets log-read access. Now logs only
   `len(token)` as a liveness signal.

2. provisioner/cp_provisioner.go:101 fell back to logging the raw
   control-plane response body when the structured {"error":"..."}
   field was absent. If the CP ever echoed request headers (Authorization)
   or a portion of user-data back in an error path, the bearer token
   would end up in our tenant-instance logs. Now logs the byte count
   only; the structured error remains in place for the happy path.
   Also caps the read at 64 KiB via io.LimitReader to prevent
   log-flood DoS from a compromised upstream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(security): tenant CPProvisioner attaches CP bearer on all calls

Completes the C1 integration (PR #50 on molecule-controlplane). The CP
now requires Authorization: Bearer <PROVISION_SHARED_SECRET> on all
three /cp/workspaces/* endpoints; without this change the tenant-side
Start/Stop/IsRunning calls would all 401 (or 404 when the CP's routes
refused to mount) and every workspace provision from a SaaS tenant
would silently fail.

Reads MOLECULE_CP_SHARED_SECRET, falling back to PROVISION_SHARED_SECRET
so operators can use one env-var name on both sides of the wire. Empty
value is a no-op: self-hosted deployments with no CP or a CP that
doesn't gate /cp/workspaces/* keep working as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(canvas): add 15s fetch timeout on API calls

Pre-launch audit flagged api.ts as missing a timeout on every fetch.
A slow or hung CP response would leave the UI spinning indefinitely
with no way for the user to abort — effectively a client-side DoS.

15s is long enough for real CP queries (slowest observed is Stripe
portal redirect at ~3s) and short enough that a stalled backend
surfaces as a clear error with a retry affordance.

Uses AbortSignal.timeout (widely supported since 2023) so the
abort propagates through React Query / SWR consumers cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): stop asserting current_task on public workspace GET (#966)

PR #966 intentionally stripped current_task, last_sample_error, and
workspace_dir from the public GET /workspaces/:id response to avoid
leaking task bodies to anyone with a workspace bearer. The E2E smoke
test hadn't caught up — it was still asserting "current_task":"..."
on the single-workspace GET, which made every post-#966 CI run fail
with '60 passed, 2 failed'.

Swap the per-workspace asserts to check active_tasks (still exposed,
canonical busy signal) and keep the list-endpoint check that proves
admin-auth'd callers still see current_task end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: 2026-04-19 SaaS prod migration notes

Captures the 10-PR staging→main cutover: what shipped, the three new
Railway prod env vars (PROVISION_SHARED_SECRET / EC2_VPC_ID /
CP_BASE_URL), and the sharp edge for existing tenants — their
containers pre-date PR #53 so they still need MOLECULE_CP_SHARED_SECRET
added manually (or a re-provision) before the new CPProvisioner's
outbound bearer works.

Also includes a post-deploy verification checklist and rollback plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ws-server): pull env from CP on startup

Paired with molecule-controlplane PR #55 (GET /cp/tenants/config). Lets
existing tenants heal themselves when we rotate or add a CP-side env
var (e.g. MOLECULE_CP_SHARED_SECRET landing earlier today) without any
ssh or re-provision.

Flow: main() calls refreshEnvFromCP() before any other os.Getenv read.
The helper reads MOLECULE_ORG_ID + ADMIN_TOKEN from the baked-in
user-data env, GETs {MOLECULE_CP_URL}/cp/tenants/config with those
credentials, and applies the returned string map via os.Setenv so
downstream code (CPProvisioner, etc.) sees the fresh values.

Best-effort semantics:
- self-hosted / no MOLECULE_ORG_ID → no-op (return nil)
- CP unreachable / non-200 → log + return error (main keeps booting)
- oversized values (>4 KiB each) rejected to avoid env pollution
- body read capped at 64 KiB

Once this image hits GHCR, the 5-minute tenant auto-updater picks it
up, the container restarts, refresh runs, and every tenant has
MOLECULE_CP_SHARED_SECRET within ~5 minutes — no operator toil.

Also fixes workspace-server/.gitignore so `server` no longer matches
the cmd/server package dir — it only ignored the compiled binary but
pattern was too broad. Anchored to `/server`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canary): smoke harness + GHA verification workflow (Phase 2)

Post-deploy verification for staging tenant images. Runs against the
canary fleet after each publish-workspace-server-image build — catches
auto-update breakage (a la today's E2E current_task drift) before it
propagates to the prod tenant fleet that auto-pulls :latest every 5 min.

scripts/canary-smoke.sh iterates a space-sep list of canary base URLs
(paired with their ADMIN_TOKENs) and checks:
- /admin/liveness reachable with admin bearer (tenant boot OK)
- /workspaces list responds (wsAuth + DB path OK)
- /memories/commit + /memories/search round-trip (encryption + scrubber)
- /events admin read (AdminAuth C4 path)
- /admin/liveness without bearer returns 401 (C4 fail-closed regression)

.github/workflows/canary-verify.yml runs after publish succeeds:
- 6-min sleep (tenant auto-updater pulls every 5 min)
- bash scripts/canary-smoke.sh with secrets pulled from repo settings
- on failure: writes a Step Summary flagging that :latest should be
  rolled back to prior known-good digest

Phase 3 follow-up will split the publish workflow so only
:staging-<sha> ships initially, and canary-verify's green gate is
what promotes :staging-<sha> → :latest. This commit lays the test
gate alone so we have something running against tenants immediately.

Secrets to set in GitHub repo settings before this workflow can run:
- CANARY_TENANT_URLS (space-sep list)
- CANARY_ADMIN_TOKENS (same order as URLs)
- CANARY_CP_SHARED_SECRET (matches staging CP PROVISION_SHARED_SECRET)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canary): gate :latest tag promotion on canary verify green (Phase 3)

Completes the canary release train. Before this, publish-workspace-
server-image.yml pushed both :staging-<sha> and :latest on every
main merge — meaning the prod tenant fleet auto-pulled every image
immediately, before any post-deploy smoke test. A broken image
(think: this morning's E2E current_task drift, but shipped at 3am
instead of caught in CI) would have fanned out to every running
tenant within 5 min.

Now:
- publish workflow pushes :staging-<sha> ONLY
- canary tenants are configured to track :staging-<sha>; they pick
  up the new image on their next auto-update cycle
- canary-verify.yml runs the smoke suite (Phase 2) after the sleep
- on green: a new promote-to-latest job uses crane to remotely
  retag :staging-<sha> → :latest for both platform and tenant images
- prod tenants auto-update to the newly-retagged :latest within
  their usual 5-min window
- on red: :latest stays frozen on prior good digest; prod is untouched

crane is pulled onto the runner (~4 MB, GitHub release) rather than
docker-daemon retag so the workflow doesn't need a privileged runner.

Rollback: if canary passed but something surfaces post-promotion,
operator runs "crane tag ghcr.io/molecule-ai/platform:<prior-good-sha>
latest" manually. A follow-up can wrap that in a Phase 4 admin
endpoint / script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canary): rollback-latest script + release-pipeline doc (Phase 4)

Closes the canary loop with the escape hatch and a single place to
read about the whole flow.

scripts/rollback-latest.sh <sha>
  uses crane to retag :latest ← :staging-<sha> for BOTH the platform
  and tenant images. Pre-checks the target tag exists and verifies
  the :latest digest after the move so a bad ops typo doesn't
  silently promote the wrong thing. Prod tenants auto-update to the
  rolled-back digest within their 5-min cycle. Exit codes: 0 = both
  retagged, 1 = registry/tag error, 2 = usage error.

docs/architecture/canary-release.md
  The one-page map of the pipeline: how PR → main → staging-<sha> →
  canary smoke → :latest promotion works end-to-end, how to add a
  canary tenant, how to roll back, and what this gate explicitly does
  NOT catch (prod-only data, config drift, cross-tenant bugs).

No code changes in the CP or workspace-server — this PR is shell
+ docs only, so it's safe to land independently of the other Phase
{1,1.5,2,3} PRs still in review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(ws-server): cover CPProvisioner — auth, env fallback, error paths

Post-merge audit flagged cp_provisioner.go as the only new file from
the canary/C1 work without test coverage. Fills the gap:

- NewCPProvisioner_RequiresOrgID — self-hosted without MOLECULE_ORG_ID
  refuses to construct (avoids silent phone-home to prod CP).
- NewCPProvisioner_FallsBackToProvisionSharedSecret — the operator
  ergonomics of using one env-var name on both sides of the wire.
- AuthHeader noop + happy path — bearer only set when secret is set.
- Start_HappyPath — end-to-end POST to stubbed CP, bearer forwarded,
  instance_id parsed out of response.
- Start_Non201ReturnsStructuredError — when CP returns structured
  {"error":"…"}, that message surfaces to the caller.
- Start_NoStructuredErrorFallsBackToSize — regression gate for the
  anti-log-leak change from PR #980: raw upstream body must NOT
  appear in the error, only the byte count.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(scheduler): collapse empty-run bump to single RETURNING query

The phantom-producer detector (#795) was doing UPDATE + SELECT in two
roundtrips — first incrementing consecutive_empty_runs, then re-
reading to check the stale threshold. Switch to UPDATE ... RETURNING
so the post-increment value comes back in one query.

Called once per schedule per cron tick. At 100 tenants × dozens of
schedules per tenant, the halved DB traffic on the empty-response
path is measurable, not just cosmetic.

Also now properly logs if the bump itself fails (previously it silent-
swallowed the ExecContext error and still ran the SELECT, which would
confuse debugging).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canvas): /orgs landing page for post-signup users

CP's Callback handler redirects every new WorkOS session to
APP_URL/orgs, but canvas had no such route — new users hit the canvas
Home component, which tries to call /workspaces on a tenant that
doesn't exist yet, and saw a confusing error. This PR plugs that gap
with a dedicated landing page that:

- Bounces anonymous visitors back to /cp/auth/login
- Zero-org users see a slug-picker (POST /cp/orgs, refresh)
- For each existing org, shows status + CTA:
  * awaiting_payment → amber "Complete payment" → /pricing?org=…
  * running          → emerald "Open" → https://<slug>.moleculesai.app
  * failed           → "Contact support" → mailto
  * provisioning     → read-only "provisioning…"
- Surfaces errors inline with a Retry button

Deliberately server-light: one GET /cp/orgs, no WebSocket, no canvas
store hydration. Goal is to move the user from signup to either
Stripe Checkout or their tenant URL with one click each.

Closes the last UX gap between the BILLING_REQUIRED gate landing on
the CP and real users being able to complete a signup today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canvas): post-checkout UX — Stripe success lands on /orgs with banner

Two small polish items that together close the signup-to-running-tenant
flow for real users:

1. Stripe success_url now points at /orgs?checkout=success instead of
   the current page (was pricing). The old behavior left people staring
   at plan cards with no indication payment went through — the new
   behavior drops them right onto their org list where they can watch
   the status flip.

2. /orgs shows a green "Payment confirmed, workspace spinning up"
   banner when it sees ?checkout=success, then clears the query
   param via replaceState so a reload doesn't show it again.

3. /orgs now polls every 5s while any org is awaiting_payment or
   provisioning. Users see the Stripe webhook's effect live — no
   manual refresh needed — and once every org settles the polling
   stops so idle tabs don't hammer /cp/orgs.

Paired with PR #992 (the /orgs page itself) this makes the end-to-end
flow on BILLING_REQUIRED=true deployments feel right:
  /pricing → Stripe → /orgs?checkout=success → banner → live poll →
  "Open" button when org.status transitions to running.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(canvas): bump billing test for /orgs success_url

* fix(ci): clone sibling plugin repo so publish-workspace-server-image builds

Publish has been failing since the 2026-04-18 open-source restructure
(#964's merge) because workspace-server/Dockerfile still COPYs
./molecule-ai-plugin-github-app-auth/ but the restructure moved that
code out to its own repo. Every main merge since has produced a
"failed to compute cache key: /molecule-ai-plugin-github-app-auth:
not found" error — prod images haven't moved.

Fix: add an actions/checkout step that fetches the plugin repo into
the build context before docker build runs.

Private-repo safe: uses PLUGIN_REPO_PAT secret (fine-grained PAT with
Contents:Read on Molecule-AI/molecule-ai-plugin-github-app-auth).
Falls back to the default GITHUB_TOKEN if the plugin repo is public.

Ops: set repo secret PLUGIN_REPO_PAT before the next main merge, or
publish will fail with a 404 on the checkout step.

Also gitignores the cloned dir so local dev builds don't accidentally
commit it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci(promote-latest): workflow_dispatch to retag :staging-<sha> → :latest

Escape hatch for the initial rollout window (canary fleet not yet
provisioned, so canary-verify.yml's automatic promotion doesn't fire)
AND for manual rollback scenarios.

Uses the default GITHUB_TOKEN which carries write:packages on repo-
owned GHCR images, so no new secrets are needed. crane handles the
remote retag without pulling or pushing layers.

Validates the src tag exists before retagging + verifies the :latest
digest post-retag so a typo can't silently promote the wrong image.

Trigger from Actions → promote-latest → Run workflow → enter the
short sha (e.g. "4c1d56e").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci(promote-latest): run on self-hosted mac mini (GH-hosted quota blocked)

* ci(promote-latest): suppress brew cleanup that hits perm-denied on shared runner

* feat(canvas): Phase 5 — credit balance pill + low-balance banner

Adds the UI surface for the credit system to /orgs:
- CreditsPill next to each org row. Tone shifts from zinc → amber at
  10% of plan to red at zero.
- LowCreditsBanner appears under the pill for running orgs when the
  balance crosses thresholds: overage_used > 0 → "overage active",
  balance <= 0 → "out of credits, upgrade", trial tail → "trial almost
  out".
- Pure helpers extracted to lib/credits.ts so formatCredits, pillTone,
  and bannerKind are unit-tested without jsdom.

Backend List query now returns credits_balance / plan_monthly_credits
/ overage_used_credits / overage_cap_credits so no second round-trip
is needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canvas): ToS gate modal + us-east-2 data residency notice

Wraps /orgs in a TermsGate that polls /cp/auth/terms-status on mount
and overlays a blocking modal when the current terms version hasn't
been accepted yet. "I agree" POSTs /cp/auth/accept-terms and dismisses
the modal; the backend records IP + UA as GDPR Art. 7 proof-of-consent.

Also adds a short data residency notice under the page header:
workspaces run in AWS us-east-2 (Ohio, US). An EU region selector is
a future lift once the infra is provisioned there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(scheduler): defer cron fires when workspace busy instead of skipping (#969)

Previously, the scheduler skipped cron fires entirely when a workspace
had active_tasks > 0 (#115). This caused permanent cron misses for
workspaces kept perpetually busy by the 5-min Orchestrator pulse — work
crons (pick-up-work, PR review) were skipped every fire because the
agent was always processing a delegation.

Measured impact on Dev Lead: 17 context-deadline-exceeded timeouts in
2 hours, ~30% of inter-agent messages silently dropped.

Fix: when workspace is busy, poll every 10s for up to 2 minutes waiting
for idle. If idle within the window, fire normally. If still busy after
2 min, fall back to the original skip behavior.

This is a minimal, safe change:
- No new goroutines or channels
- Same fire path once idle
- Bounded wait (2 min max, won't block the scheduler pool)
- Falls back to skip if workspace never becomes idle

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): scrub secrets in commit_memory MCP tool path (#838 sibling)

PR #881 closed SAFE-T1201 (#838) on the HTTP path by wiring redactSecrets()
into MemoriesHandler.Commit — but the sibling code path on the MCP bridge
(MCPHandler.toolCommitMemory) was left with only the TODO comment. Agents
calling commit_memory via the MCP tool bridge are the PRIMARY attack vector
for #838 (confused / prompt-injected agent pipes raw tool-response text
containing plain-text credentials into agent_memories, leaking into shared
TEAM scope). The HTTP path is only exercised by canvas UI posts, so the MCP
gap was the hotter one.

Change:

  workspace-server/internal/handlers/mcp.go:725
    - TODO(#838): run _redactSecrets(content) before insert — plain-text
    - API keys from tool responses must not land in the memories table.
    + SAFE-T1201 (#838): scrub known credential patterns before persistence…
    + content, _ = redactSecrets(workspaceID, content)

Reuses redactSecrets (same package) so there's no duplicated pattern list —
a future-added pattern in memories.go automatically covers the MCP path too.

Tests added in mcp_test.go:

  - TestMCPHandler_CommitMemory_SecretInContent_IsRedactedBeforeInsert
      Exercises three patterns (env-var assignment, Bearer token, sk-…)
      and uses sqlmock's WithArgs to bind the exact REDACTED form — so a
      regression (removing the redactSecrets call) fails with arg-mismatch
      rather than silently persisting the secret.

  - TestMCPHandler_CommitMemory_CleanContent_PassesThrough
      Regression guard — benign content must NOT be altered by the redactor.

NOTE: unable to run `go test -race ./...` locally (this container has no Go
toolchain). The change is mechanical reuse of an already-shipped function in
the same package; CI must validate. The sqlmock patterns mirror the existing
TestMCPHandler_CommitMemory_LocalScope_Success test exactly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): move canary-verify to self-hosted runner

GitHub-hosted ubuntu-latest runs on this repo hit "recent account
payments have failed or your spending limit needs to be increased"
— same root cause as the publish + CodeQL + molecule-app workflow
moves earlier this quarter. canary-verify was the last one still on
ubuntu-latest.

Switches both jobs to [self-hosted, macos, arm64]. crane install
switched from Linux tarball to brew (matches promote-latest.yml's
install pattern + avoids /usr/local/bin write perms on the shared
mac mini).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(canvas): pin AbortSignal timeout regression + cover /orgs landing page

Two independent test additions that harden the surface freshly landed on
staging via PRs #982 (canvas fetch timeout), #992 (/orgs landing), #994
(post-checkout redirect to /orgs).

canvas/src/lib/__tests__/api.test.ts (+74 lines, 7 new tests)
  - GET/POST/PATCH/PUT/DELETE each pass an AbortSignal to fetch
  - TimeoutError (DOMException name=TimeoutError) propagates to the caller
  - Each request installs its own signal — no shared module-level controller
    that would allow one slow request to cancel an unrelated fast one
  This is the hardening nit I flagged in my APPROVE-w/-nit review of
  fix/canvas-api-fetch-timeout. Landing as a follow-up now that #982 is in
  staging.

canvas/src/app/__tests__/orgs-page.test.tsx (+251 lines, new file, 10 tests)
  - Auth guard: signed-out → redirectToLogin and no /cp/orgs fetch
  - Error state: failed /cp/orgs → Error message + Retry button
  - Empty list: CreateOrgForm renders
  - CTA by status:
      running          → "Open" link targets {slug}.moleculesai.app
      awaiting_payment → "Complete payment" → /pricing?org=<slug>
      failed           → "Contact support" mailto
  - Post-checkout: ?checkout=success renders CheckoutBanner AND
    history.replaceState scrubs the query param
  - Fetch contract: /cp/orgs called with credentials:include + AbortSignal

Local baseline on origin/staging tip 845ac47:
  canvas vitest: 50 files / 778 tests, all green
  canvas build:  clean, /orgs route present (2.83 kB / 105 kB first-load)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(canvas): cover /orgs 5s polling on in-flight orgs

The test docstring promised polling coverage but I'd only wired the
describe-block header, not the actual tests. Closing that gap — vitest
fake timers drive three cases:

- `provisioning` org → 2nd fetch fires after 5.1s advance
- all `running` → no 2nd fetch even after 10s advance
- `awaiting_payment` org, unmount before timer fires → no post-unmount
  fetch (cleanup correctly clears the pollTimer)

The unmount case is the meaningful one: without it a fast nav-away
leaves the 5s interval chasing the CP forever. page.tsx L97-99 does
clear the timer; the test pins the contract.

Local baseline on origin/staging tip 845ac47 + this branch:
  canvas vitest: 50 files / 781 tests, all green (+3 vs prior commit)
  canvas build:  clean

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ci(codeql): cover main + staging via workflow

GitHub's UI-configured "Code quality" scan only fires on the default
branch (staging), which leaves every staging→main promotion PR
unscanned. The "On push and pull requests to" field in the UI has no
dropdown; multi-branch scanning on private repos without GHAS isn't
available there.

Workflow file gives us the control we can't get in the UI: triggers
on push + pull_request for both branches. Runs on the same
self-hosted mac mini via [self-hosted, macos, arm64].

upload: never — GHAS isn't enabled on this repo so the SARIF upload
API 403s. Keep results locally, filter to error+warning severity,
fail the PR check on findings, publish SARIF as a workflow artifact.
Flipping upload: never → always after GHAS is enabled (if ever) is
a one-line change.

Picks up the review-flagged improvements from the earlier closed PR:
  - jq install step (brew, no assumption it's present)
  - severity filter (error+warning only, drops noisy note-level)
  - set -euo pipefail
  - SARIF glob (file name doesn't match matrix language id)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(bundle/exporter): add rows.Err() after child workspace enumeration

Silent data loss on mid-cursor DB errors — partial sub-workspace
bundles returned instead of surfacing the iteration error. Adds
rows.Err() check after the SELECT id FROM workspaces query in
Export(), mirroring the pattern already used in scheduler.go
and handlers with similar recursion patterns.

Closes: R1 MISSING-ROWS-ERR findings (bundle/exporter.go)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(a11y): WorkspaceNode font floor, contrast, focus rings (Cycle 10)

C1: skills badge spans text-[7px]→text-[10px]; "+N more" overflow
    text-[7px] text-zinc-500→text-[10px] text-zinc-400
C2: Team section label text-[7px] text-zinc-600→text-[10px] text-zinc-400
H4: status label text-[9px]→text-[10px]; active-tasks count
    text-[9px] text-amber-300/80→text-[10px] text-amber-300 (remove opacity
    modifier per design-system contrast rule); current-task text
    text-[9px] text-amber-300/70→text-[10px] text-amber-300
L1: add focus-visible:ring-2 focus-visible:ring-blue-500/70 to the Restart
    button (independently Tab-focusable inside role="button" wrapper) and to
    the Extract-from-team button in TeamMemberChip; TeamMemberChip
    role="button" div already has the focus ring (COVERED, no change)

762/762 tests pass · build clean

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(ci): replace sleep 360 with health-check poll in canary-verify (#1013)

The canary-verify workflow blocked the self-hosted runner for a fixed
6 minutes regardless of whether canaries had already updated. This
wastes the runner slot when canaries update in 2-3 minutes.

Fix: poll each canary's /health endpoint every 30s for up to 7 min.
Exit early when all canaries report the expected SHA. Falls back to
proceeding after timeout — the smoke suite validates regardless.

Typical time saving: ~3-4 minutes per canary verify run.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gate-1): remove unused fireEvent import (#1011)

Mechanical lint fix. github-code-quality[bot] flagged unused
import on line 18 — fireEvent is imported but never referenced in
the test file. Removing it clears the code quality gate without
changing any test behaviour.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: event-driven cron triggers + auto-push hook for agent productivity

Three changes to boost agent throughput:

1. Event-driven cron triggers (webhooks.go): GitHub issues/opened events
   fire all "pick-up-work" schedules immediately. PR review/submitted
   events fire "PR review" and "security review" schedules. Uses
   next_run_at=now() so the scheduler picks them up on next tick.

2. Auto-push hook (executor_helpers.py): After every task completion,
   agents automatically push unpushed commits and open a PR targeting
   staging. Guards: only on non-protected branches with unpushed work.
   Uses /usr/local/bin/git and /usr/local/bin/gh wrappers with baked-in
   GH_TOKEN. Never crashes the agent — all errors logged and continued.

3. Integration (claude_sdk_executor.py): auto_push_hook() called in the
   _execute_locked finally block after commit_memory.

Closes productivity gap where agents wrote code but never pushed,
and where work crons only fired on timers instead of reacting to events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: disable schedules when workspace is deleted (#1027)

When a workspace is deleted (status set to 'removed'), its schedules
remained enabled, causing the scheduler to keep firing cron jobs for
non-existent containers. Add a cascade disable query alongside the
existing token revocation and canvas layout cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: stop hardcoding CLAUDE_CODE_OAUTH_TOKEN in required_env (#1028)

The provisioner was unconditionally writing CLAUDE_CODE_OAUTH_TOKEN into
config.yaml's required_env for all claude-code workspaces.  When the
baked token expired, preflight rejected every workspace — even those
with a valid token injected via the secrets API at runtime.

Changes:
- workspace_provision.go: remove hardcoded required_env for claude-code
  and codex runtimes; tokens are injected at container start via secrets
- workspace_provision_test.go: flip assertion to reject hardcoded token

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add cascade schedule disable tests for #1027

- TestWorkspaceDelete_DisablesSchedules — leaf workspace delete disables its schedules
- TestWorkspaceDelete_CascadeDisablesDescendantSchedules — parent+child+grandchild cascade
- TestWorkspaceDelete_ScheduleDisableOnlyTargetsDeletedWorkspace — negative test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: multiple platform handler bug fixes

- secrets.go: Log RowsAffected errors instead of silently discarding them
- a2a_proxy.go: Add 60s safety timeout to a2aClient HTTP client
- terminal.go: Fix defer ordering - always close WebSocket conn on error,
  only defer resp.Close() after successful exec attach
- webhooks.go: Add shortSHA() helper to safely handle empty HeadSHA

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(runtime): inject HMA memory instructions at platform level (#1047)

Every agent now gets hierarchical memory instructions in their system
prompt automatically — no template configuration needed. Instructions
cover commit_memory (LOCAL/TEAM/GLOBAL scopes), recall_memory, and
when to use each proactively.

Follows the same pattern as A2A instructions: defined in
executor_helpers.py, injected by _build_system_prompt() in the
claude_sdk_executor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: seed initial memories from org template and create payload (#1050)

Add MemorySeed model and initial_memories support at three levels:
- POST /workspaces payload: seed memories on workspace creation
- org.yaml workspace config: per-workspace initial_memories with
  defaults fallback
- org.yaml global_memories: org-wide GLOBAL scope memories seeded
  on the first root workspace during import

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(template): restructure molecule-dev org template to 39-agent hierarchy

Comprehensive rewrite of the Molecule AI dev team org template:

- Rename agents to {team}-{role} convention (e.g., core-be, cp-lead, app-qa)
- Add 5 new team leads: Core Platform Lead, Controlplane Lead, App & Docs Lead, Infra Lead, SDK Lead
- Add new roles: Release Manager, Integration Tester, Technical Writer, Infra-SRE, Infra-Runtime-BE, SDK-Dev, Plugin-Dev
- Delete triage-operator and triage-operator-2 (leads own triage now)
- Set default model to MiniMax-M2.7, tier 3, idle_interval_seconds 900
- Update org.yaml category_routing to new agent names
- Add orchestrator-pulse schedules for all leads (*/5 cron)
- Add pick-up-work schedules for engineers (*/15 cron)
- Add qa-review schedules for QA agents (*/15 cron)
- Add security-scan schedules for security agents (*/30 cron)
- Add release-cycle and e2e-test schedules for Release Manager and Integration Tester
- Update marketing agents with web search MCP and media generation capabilities
- All schedule prompts reference Molecule-AI/internal for PLAN.md and known-issues.md
- Un-ignore org-templates/molecule-dev/ in .gitignore for version tracking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix test assertions to account for HMA instructions in system prompt

Mock get_hma_instructions in exact-match tests so they don't break
when HMA content is appended. Add a dedicated test for HMA inclusion.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: gitignore org-templates/ and plugins/ entirely

These directories are cloned from their standalone repos
(molecule-ai-org-template-*, molecule-ai-plugin-*) and should
never be committed to molecule-core directly.

Removed the !/org-templates/molecule-dev/ exception that allowed
PR #1056 to land template files in the wrong repo.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(workspace-server): send X-Molecule-Admin-Token on CP calls

controlplane #118 + #130 made /cp/workspaces/* require a per-tenant
admin_token header in addition to the platform-wide shared secret.
Without it, every workspace provision / deprovision / status call
now 401s.

ADMIN_TOKEN is already injected into the tenant container by the
controlplane's Secrets Manager bootstrap, so this is purely a
header-plumbing change — no new config required on the tenant side.

## Change

- CPProvisioner carries adminToken alongside sharedSecret
- New authHeaders method sets BOTH auth headers on every outbound
  request (old authHeader deleted — single call site was misleading
  once the semantics changed)
- Empty values on either header are no-ops so self-hosted / dev
  deployments without a real CP still work

## Tests

Renamed + expanded cp_provisioner_test cases:
- TestAuthHeaders_NoopWhenBothEmpty — self-hosted path
- TestAuthHeaders_SetsBothWhenBothProvided — prod happy path
- TestAuthHeaders_OnlyAdminTokenWhenSecretEmpty — transition window

Full workspace-server suite green.

## Rollout

Next tenant provision will ship an image with this commit merged.
Existing tenants (none in prod right now — hongming was the only
one and was purged earlier today) will auto-update via the 5-min
image-pull cron.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: GitHub token refresh — add WorkspaceAuth path for credential helper (#1068)

PR #729 tightened AdminAuth to require ADMIN_TOKEN, breaking the
workspace credential helper which called /admin/github-installation-token
with a workspace bearer token. Tokens expired after 60 min with no refresh.

Fix: Add /workspaces/:id/github-installation-token under WorkspaceAuth
so any authenticated workspace can refresh its GitHub token. Keep the
admin path as backward-compatible alias.

Update molecule-git-token-helper.sh to use the workspace-scoped path
when WORKSPACE_ID is set.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(workspace-server): cover Stop/IsRunning/Close + auth-header + transport errors

Closes review gap: pre-PR coverage on CPProvisioner was 37%.
After this commit every exported method is exercised:

  - NewCPProvisioner            100%
  - authHeaders                  100%
  - Start                         91.7% (remainder: json.Marshal error
                                   path, unreachable with fixed-type
                                   request struct)
  - Stop                         100% (new — header + path + error)
  - IsRunning                    100% (new — 4-state matrix + auth)
  - Close                        100% (new — contract no-op)

New cases assert both auth headers (shared secret + admin_token) land
on every outbound request, transport failures surface clear errors
on Start/Stop, and IsRunning doesn't misreport on transport failure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(workspace-server): IsRunning surfaces non-2xx + JSON errors

Pre-existing silent-failure path: IsRunning decoded CP responses
regardless of HTTP status, so a CP 500 → empty body → State="" →
returned (false, nil). The sweeper couldn't distinguish "workspace
stopped" from "CP broken" and would leave a dead row in place.

## Fix

  - Non-2xx → wrapped error, does NOT echo body (CP 5xx bodies may
    contain echoed headers; leaking into logs would expose bearer)
  - JSON decode error → wrapped error
  - Transport error → now wrapped with "cp provisioner: status:"
    prefix for easier log grepping

## Tests

+7 cases (5-status table + malformed JSON + existing transport).
IsRunning coverage 100%; overall cp_provisioner at 98%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cp_provisioner): IsRunning returns (true, err) on transient failures

My #1071 made IsRunning return (false, err) on all error paths, but that
breaks a2a_proxy which depends on Docker provisioner's (true, err) contract.
Without this fix, any brief CP outage causes a2a_proxy to mark workspaces
offline and trigger restart cascades across every tenant.

Contract now matches Docker.IsRunning:
  transport error    → (true, err)  — alive, degraded signal
  non-2xx response   → (true, err)  — alive, degraded signal
  JSON decode error  → (true, err)  — alive, degraded signal
  2xx state!=running → (false, nil)
  2xx state==running → (true, nil)

healthsweep.go is also happy with this — it skips on err regardless.

Adds TestIsRunning_ContractCompat_A2AProxy as regression guard that
asserts each error path explicitly against the a2a_proxy expectations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cp_provisioner): cap IsRunning body read at 64 KiB

IsRunning used an unbounded json.NewDecoder(resp.Body).Decode on
CP status responses. Start already caps its body read at 64 KiB
(cp_provisioner.go:137) to defend against a misconfigured or
compromised CP streaming a huge body and exhausting memory.

IsRunning is called reactively per-request from a2a_proxy and
periodically from healthsweep, so it's a hotter path than Start
and arguably deserves the same defense more.

Adds TestIsRunning_BoundedBodyRead that serves a body padded past
the cap and asserts the decode still succeeds on the JSON prefix.

Follow-up to code-review Nit-2 on #1073.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(canvas): /waitlist page with contact form

Adds the user-facing half of the beta-gate: a page at /waitlist that
the CP auth callback redirects users to when their email isn't on
the allowlist. Collects email + optional name + use-case and POSTs
to /cp/waitlist/request (backend landed in controlplane #150).

## Behavior

- No auto-pre-fill of email from URL query (CP's #145 dropped the
  ?email= param for the privacy reason; this test guards against a
  future regression on the client side).
- Client-side validates email shape for instant feedback; backend
  re-validates.
- Three UI states after submit:
    success → "your request is in" banner, form hidden
    dedup   → softer "already on file" banner when backend returns
              dedup=true (same 200, no 409 to avoid enumeration)
    error   → inline banner with backend message or network fallback

## Tests

9 tests in __tests__/waitlist-page.test.tsx covering:
- default render + a11y (role=button, role=status, role=alert)
- URL-pre-fill privacy regression guard
- HTML5 + JS validation (empty, malformed)
- successful POST with trimmed body
- dedup branch
- non-2xx with + without error field
- network rejection

Follow-up to the beta-gate rollout on controlplane #145 / #150.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(canvas): remove dead /waitlist page (lives in molecule-app)

#1080 added /waitlist to canvas, but canvas isn't served at
app.moleculesai.app — it backs the tenant subdomains (acme.moleculesai.app
etc.). The real /waitlist lives in the separate molecule-app repo,
which is what the CP auth callback redirects to.

molecule-app#12 has the real page + contact form wiring to
/cp/waitlist/request. This canvas copy was never reachable and would
only diverge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(org-import): limit concurrent Docker provisioning to 3 (#1084)

The org import fired all workspace provisioning goroutines concurrently,
overwhelming Docker when creating 39+ containers. Containers timed out,
leaving workspaces stuck in 'provisioning' with no schedules or hooks.

Fix:
- Add provisionConcurrency=3 semaphore limiting concurrent Docker ops
- Increase workspaceCreatePacingMs from 50ms to 2000ms between siblings
- Pass semaphore through createWorkspaceTree recursion

With 39 workspaces at 3 concurrent + 2s pacing, import takes ~30s instead
of timing out. Each workspace gets its full template: schedules, hooks,
settings, hierarchy.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add ?purge=true hard-delete to DELETE /workspaces/:id (#1087)

Soft-delete (status='removed') leaves orphan DB rows and FK data forever.
When ?purge=true is passed, after container cleanup the handler cascade-
deletes all leaf FK tables and hard-removes the workspace row.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove org-templates/molecule-dev from git tracking

This directory belongs in the dedicated repo
Molecule-AI/molecule-ai-org-template-molecule-dev.
It should be cloned locally for platform mounting, never
committed to molecule-core. The .gitignore already blocks it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(canvas): add NEXT_PUBLIC_ADMIN_TOKEN + CSP_DEV_MODE to docker-compose

Canvas needs AdminAuth token to fetch /workspaces (gated since PR #729)
and CSP_DEV_MODE to allow cross-port fetches in local Docker.

These were added earlier but lost on nuke+rebuild because they weren't
committed to staging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(canvas): CSP_DEV_MODE + admin token for local Docker (#1052 follow-up)

Three changes that keep getting lost on nuke+rebuild:
1. middleware.ts: read CSP_DEV_MODE env to relax CSP in local Docker
2. api.ts: send NEXT_PUBLIC_ADMIN_TOKEN header (AdminAuth on /workspaces)
3. Dockerfile: accept NEXT_PUBLIC_ADMIN_TOKEN as build arg

All three are required for the canvas to work in local Docker where
canvas (port 3000) fetches from platform (port 8080) cross-origin.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(canvas): make root layout dynamic so CSP nonce reaches Next scripts

Tenant page loads were failing with repeated CSP violations:

  Executing inline script violates ... script-src 'self'
  'nonce-M2M4YTVh...' 'strict-dynamic'. ...

because Next.js's bootstrap inline scripts were emitted without a
nonce attribute. The middleware was generating per-request nonces
correctly and sending them via `x-nonce` — but the layout was
fully static, so Next.js cached the HTML once and served that cached
bundle (no nonces baked in) for every request.

Fix: call `await headers()` in the root layout. That opts the tree
into dynamic rendering AND signals Next.js to propagate the
x-nonce value to its own generated <script> tags.

The `nonce` return value is intentionally unused — the framework
handles its bootstrap scripts automatically once the read happens.
Future code that adds third-party <Script> components (analytics,
etc.) should pass the returned nonce explicitly.

Verified against live tenant: before this change every /_next/
chunk script tag in the HTML had no nonce attribute; expected after
deploy is `<script nonce="..." src="/_next/...">` on each.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(auth): accept admin token in WorkspaceAuth for canvas dashboard

The canvas sends NEXT_PUBLIC_ADMIN_TOKEN on all API calls but per-workspace
routes (/activity, /delegations, /traces) use WorkspaceAuth which only
accepts per-workspace bearer tokens. This made the canvas dashboard 401
on every workspace detail view.

Fix: WorkspaceAuth now accepts the admin token as a fallback after
workspace token validation fails. This lets the canvas read all workspace
data with a single admin credential.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(auth): accept admin token in CanvasOrBearer for viewport PUT

* fix(ci): bake api.moleculesai.app into tenant canvas bundle

Canvas's browser-side code (auth.ts, api.ts, billing.ts) all call
fetch(PLATFORM_URL + /cp/*). PLATFORM_URL comes from
NEXT_PUBLIC_PLATFORM_URL at build time; with the build arg unset,
it falls back to http://localhost:8080 in the compiled bundle.

That means on a tenant like hongmingwang.moleculesai.app, the
user's browser actually tried to fetch http://localhost:8080/cp/
auth/me — which resolves to the USER'S OWN machine, not the tenant.
Login redirect loops 404. Every tenant canvas has been unable to
complete a fresh login on this path; existing sessions only worked
because the cookie was already set domain-wide.

Fix: pass NEXT_PUBLIC_PLATFORM_URL=https://api.moleculesai.app
as a build arg in the tenant-image workflow. CP already allows
CORS from *.moleculesai.app + credentials, and the session cookie
is scoped to .moleculesai.app so tenant subdomains inherit it.

Verified in prod by rebuilding canvas locally with the flag and
hot-patching the hongmingwang instance via SSM. Baked chunks now
contain api.moleculesai.app; browser auth redirects resolve
cleanly to the CP.

Self-hosted users override by rebuilding with their own URL —
same pattern molecule-app uses with NEXT_PUBLIC_CP_ORIGIN.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: nuke-and-rebuild.sh — one-command fleet reset

Two scripts:
- nuke-and-rebuild.sh: docker down -v, clean orphans, rebuild, setup
- post-rebuild-setup.sh: insert global secrets (MiniMax + GH PAT),
  import org template, wait for platform health

Global secrets ensure every provisioned container gets MiniMax API
config and GitHub PAT injected as env vars automatically — no manual
settings.json deployment needed.

Usage: bash scripts/nuke-and-rebuild.sh

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(canvas): include NEXT_PUBLIC_PLATFORM_URL in CSP connect-src

Tenant page loads were blocked by:

  Refused to connect to 'https://api.moleculesai.app/cp/auth/me'
  because it violates the document's Content Security Policy.

CSP had `connect-src 'self' wss:` — fine for same-origin + any wss,
but browser refuses cross-origin HTTPS fetches that aren't listed.
PLATFORM_URL (baked from NEXT_PUBLIC_PLATFORM_URL, which is the CP
origin on SaaS tenants) needs to be explicit.

Fix: middleware reads NEXT_PUBLIC_PLATFORM_URL at build/runtime
and adds both the https and wss siblings to connect-src. Self-
hosted deploys that override the build-arg automatically get a
matching CSP — no hardcoded hostname.

Test added: buildCsp includes NEXT_PUBLIC_PLATFORM_URL origin in
connect-src when set. Also loosens the dev `ws:` assertion since
dev uses `connect-src *` which subsumes ws (pre-existing behavior,
test was stale).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(router): /cp/* reverse-proxy to CP + same-origin canvas fetches

Canvas's browser bundle issues fetches to both CP endpoints
(/cp/auth/me, /cp/orgs, ...) AND tenant-platform endpoints
(/canvas/viewport, /approvals/pending, /org/templates). They
share ONE build-time base URL. Baking api.moleculesai.app
broke tenant calls with 404; baking the tenant subdomain broke
auth. Tried both today and saw exactly one failure mode per
attempt.

Real fix: same-origin fetches + tenant-side split. Adds:

  internal/router/cp_proxy.go      # /cp/* → CP_UPSTREAM_URL

mounted before NoRoute(canvasProxy). Now a tenant serves:

  /cp/*              → reverse-proxy to api.moleculesai.app
  /canvas/viewport,
  /approvals/pending,
  /workspaces/:id/*,
  /ws, /registry,    → tenant platform (existing handlers)
  /metrics
  everything else    → canvas UI (existing reverse-proxy)

Canvas middleware reverts to `connect-src 'self' wss:` for the
same-origin path (keeping explicit PLATFORM_URL whitelist as a
self-hosted escape hatch when the build-arg is non-empty).

CI build-arg flips to NEXT_PUBLIC_PLATFORM_URL="" so the bundle
issues relative fetches.

Security of cp_proxy:
  - Cookie + Authorization PRESERVED across the hop (opposite of
    canvas proxy) — they carry the WorkOS session, which is the
    whole point.
  - Host rewritten to upstream so CORS + cookie-domain on the CP
    side see their own hostname.
  - Upstream URL validated at construction: must parse, must be
    http(s), must have a host — misconfig fails closed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* security: remove hardcoded API keys from post-rebuild-setup.sh

GitGuardian detected exposed MiniMax API key and GitHub PAT in the
script's default values. Replaced with env var reads from .env file
(which is gitignored). Script now validates required secrets exist
before proceeding.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): TenantGuard passes through /cp/* to CP proxy

Today's rollout of cp_proxy (PR #1095/1096) mounted /cp/* as a
reverse-proxy to the control plane, but the TenantGuard middleware
runs first in the global chain and 404s anything that isn't in its
exact-path allowlist (/health + /metrics). Every /cp/auth/me fetch
from canvas landed on a 40µs 404 before ever reaching the proxy.

/cp/* is handled upstream (WorkOS session + admin bearer), so the
tenant doesn't need to attach org identity for those paths. Passing
them through is correct — matches the design where the tenant
platform is a pure transit layer for /cp/*.

Verified: /cp/auth/me via tunnel now returns 401 (correct unauth
from CP) instead of 404 from TenantGuard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(middleware): AdminAuth accepts CP-verified WorkOS session

Canvas (SaaS tenant UI) runs in the browser and authenticates the
user via a WorkOS session cookie scoped to .moleculesai.app. It
has no bearer token — the token-based ADMIN_TOKEN scheme is for
CLI + server-to-server callers, not end users.

Adds a session-verification tier to AdminAuth that runs BEFORE the
bearer check:

 1. If Cookie header present AND CP_UPSTREAM_URL configured →
    GET /cp/auth/me upstream with the same cookie. 200 + valid
    user_id → grant admin access. Non-200 → fall through.
 2. Else (no cookie, or no CP configured, or CP said no) →
    existing bearer-only path unchanged.

Positive verifications are cached 30s keyed by the raw Cookie
header, so a burst of canvas admin-page renders doesn't DDoS
the CP. Revocations propagate within that window.

Self-hosted / dev deploys without CP_UPSTREAM_URL: feature
disabled, behavior unchanged. So this is strictly additive for
the SaaS case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(docker): fix plugin go.mod replace for TokenProvider interface (#960)

The github-app-auth plugin's go.mod had a relative replace directive
(../molecule-monorepo/platform) that didn't resolve in Docker where
the plugin is at /plugin/ and the platform at /app/. This caused the
plugin's provisionhook.TokenProvider interface to come from a different
package path than the platform's, so the type assertion in
FirstTokenProvider() failed — "no token provider registered".

Fix: sed the plugin's go.mod replace to point at /app during Docker build.
Also added debug logging to GetInstallationToken for future diagnosis.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: close cross-tenant authz + cp_proxy admin-traversal gaps

Addresses three Critical findings from today's code review of the
SaaS-canvas routing stack.

## Critical-1: session verification scoped to the current tenant

session_auth.go previously verified via GET /cp/auth/me, which
only answers "is someone logged in" — NOT "is this user in the
org they're targeting." Every WorkOS-authed user (including folks
who only signed up via app.moleculesai.app with no tenant
relationship) could call /workspaces, /approvals/pending,
/bundles/import, /org/import etc. on ANY tenant they could reach.
Cross-tenant read: user at acme.moleculesai.app could hit
bob.moleculesai.app/workspaces with their cookie and get Bob's
workspaces.

Fix:
  - CP gains GET /cp/auth/tenant-member?slug=<slug> which joins
    org_members × organizations and only returns member:true when
    the authenticated user is actually in that org.
  - Tenant sets MOLECULE_ORG_SLUG at boot via user-data.
  - session_auth now calls tenant-member (not /me), passing its
    own slug. Cache key includes slug so one tenant's cached
    positive never satisfies another's check.

## Critical-2: cp_proxy path allowlist (lateral-movement fix)

cp_proxy.go forwarded any /cp/* path upstream with the cookie
and bearer attached. Since /cp/admin/* accepts sessions as one
of its auth tiers, a tenant-authed user could curl
/cp/admin/tenants/other-slug/diagnostics through their tenant
and the CP would honor it — turning any tenant into a lateral
hop into admin surface.

Fix: explicit allowlist of paths the canvas browser bundle
actually needs (/cp/auth, /cp/orgs, /cp/billing, /cp/templates,
/cp/legal). Everything else 404s at the tenant before cookies
leave. Fail-closed: future UI paths require explicit entries.

## Important-1,2: bounded session cache + split positive/negative TTL

Previous sync.Map cache grew unbounded (one entry per unique
Cookie header for process lifetime) and cached failures for 30s,
meaning a 3s CP blip locked users out for the full window.

Fix:
  - Bounded map with batch random eviction at cap (10k entries ×
    ~100 bytes = 1 MB ceiling). Random eviction is O(1)
    expected; we don't need precise LRU.
  - Periodic sweeper goroutine (2 min) reclaims expired entries
    even when they're not re-hit.
  - Positive TTL 30s, negative TTL 5s — short negative so CP
    flakes self-heal fast.
  - Transport errors NOT cached (would otherwise trap every
    user during a multi-second upstream outage).
  - Cache key = sha256(slug + cookie) so raw session tokens
    don't sit in process memory, and cross-tenant isolation is
    structural not policy.

## Important-3: TenantGuard /cp/* bypass documented

Added a security note to the bypass explaining why it's safe
only under the current setup (cp_proxy allowlist + tunnel-only
ingress), and what would require revisiting (SG opens :8080
inbound to the VPC).

## Tests

  - session_auth_test.go: 12 new tests — empty cookie, missing
    slug, no CP, member:true happy path with cache hit, member:
    false, 401 upstream, malformed JSON, transport error not
    cached, cross-tenant isolation (same cookie different
    tenants hit upstream separately), bounded eviction, expired
    entries, cache key collision resistance.
  - cp_proxy_test.go: new — isCPProxyAllowedPath covers 17
    allow/block cases, forwarding preserves Cookie+Auth, Host
    rewritten, blocked paths 404 without calling upstream.

All platform tests pass. CP provisioner tests pass after
threading cfg.OrgSlug into the container env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(auth): organization-scoped API keys for admin access

Adds user-facing API keys with full-org admin scope. Replaces the
single ADMIN_TOKEN env var with named, revocable, audited tokens
that users can mint/rotate from the canvas UI without ops
intervention.

Designed for the beta growth phase — one token tier (full admin).
Future work will split into scoped roles (admin / workspace-write
/ read-only) and per-workspace bindings. See docs…

* test(handlers): add 5 TestKI005 regression tests to terminal_test.go

Port terminal hierarchy guard regression suite:
- TestKI005_SelfAccess_AlwaysAllowed: own workspace token always passes
- TestKI005_CanCommunicatePeer_Allowed: sibling workspace access granted
- TestKI005_CanCommunicateNonPeer_Forbidden: cross-org access blocked (403)
- TestKI005_TokenMismatch_Unauthorized: token/Workspace-ID mismatch blocked (401)
- TestKI005_NoXWorkspaceIDHeader_LegacyAllowed: legacy access no header → proceeds

Refs: F1085, KI-005

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Hongming Wang <hongmingwangrabbit@gmail.com>
Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Molecule AI Backend Engineer <backend-engineer@agents.moleculesai.app>
Co-authored-by: qa-agent <qa-agent@users.noreply.github.com>
Co-authored-by: Molecule AI Frontend Engineer <frontend-engineer@agents.moleculesai.app>
Co-authored-by: Molecule AI Triage Operator <triage-operator@agents.moleculesai.app>
Co-authored-by: Molecule AI Platform Engineer <platform-engineer@agents.moleculesai.app>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
Co-authored-by: Molecule AI SDK-Dev <sdk-dev@agents.moleculesai.app>
Co-authored-by: airenostars <airenostars@gmail.com>
Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-FE <core-fe@agents.moleculesai.app>
Co-authored-by: Molecule AI Fullstack (floater) <fullstack-floater@agents.moleculesai.app>
Co-authored-by: Molecule AI CP-QA <cp-qa@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-UIUX <core-uiux@agents.moleculesai.app>
Co-authored-by: Molecule AI PMM <pmm@agents.moleculesai.app>
Co-authored-by: Molecule AI Social Media Brand <social-media-brand@agents.moleculesai.app>
Co-authored-by: Molecule AI DevRel Engineer <devrel-engineer@agents.moleculesai.app>
Co-authored-by: Marketing Lead <marketing-lead@agents.moleculesai.app>
Co-authored-by: Molecule AI Controlplane Lead <controlplane-lead@agents.moleculesai.app>
Co-authored-by: Molecule AI CP-BE <cp-be@agents.moleculesai.app>
Co-authored-by: Molecule AI Community Manager <community-manager@agents.moleculesai.app>
Co-authored-by: Molecule AI Technical Writer <technical-writer@agents.moleculesai.app>
Co-authored-by: Molecule AI App-FE <app-fe@agents.moleculesai.app>
 2026-04-24 01:58:31 +00:00
molecule-ai[bot]
b1dce3405c
Merge branch 'staging' into test/2026-04-23-regression-suite 2026-04-24 01:55:06 +00:00
Hongming Wang
00e3e3f570 fix(#1933): bump molecule-ai-plugin-github-app-auth to current main (step 1) 
Ships step 1 of the #1933 fleet-wide GH_TOKEN refresh fix.

The plugin's v0.0.0-20260416194734-2cd28737f845 predates the Mutator.Token()
method added in plugin-repo PR #1 (merged 2026-04-17). Monorepo's
workspace-server/pkg/provisionhook/mutator.go:218 has been emitting
`provisionhook: no Token method on "github-app-auth"` on every boot and
the reflection-fallback at mutator.go:216 is doing extra work every
time a workspace requests a fresh GH token.

This is the one-line pin bump:
  v0.0.0-20260416194734-2cd28737f845 → v0.0.0-20260421064811-7d98ae51e31d

Effect: direct-interface path (not the reflection fallback) gets taken,
log noise goes away. Does NOT fix the actual 60-min GH_TOKEN death —
steps 2–5 of #1933 (credential helper install, git config wire-up,
runtime auth context, periodic refresh) are separate, larger PRs.

Verified: workspace-server/go build ./... passes with the new pin.

Ref: #1933
 2026-04-23 18:53:25 -07:00
Molecule AI Core-BE
88c929875e fix(#1877): nil provisioner guard in issueAndInjectToken 
Fix panic in TestIssueAndInjectToken_HappyPath where h.provisioner is nil
(the handler was created without a real provisioner in unit tests).
Add nil guard so the pre-write step is skipped gracefully — token is still
injected into ConfigFiles as before, and the runtime-side 401 retry handles
any race.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-23 17:47:18 -07:00
Molecule AI Core-BE
b5e2142c46 fix(#1877): close token-rotation race on restart — Option A+Option B combined 
Platform side (Option B):
- provisioner.go: add WriteAuthTokenToVolume() — writes .auth_token to
  the Docker named volume BEFORE ContainerStart using a throwaway alpine
  container, eliminating the race window where a restarted container could
  read a stale token before WriteFilesToContainer writes the new one.
- workspace_provision.go: call WriteAuthTokenToVolume() in issueAndInjectToken
  as a best-effort pre-write before the container starts.

Runtime side (Option A):
- heartbeat.py: on HTTPStatusError 401 from /registry/heartbeat, call
  refresh_cache() to force re-read of /configs/.auth_token from disk,
  then retry the heartbeat once. Fall through to normal failure tracking
  if the retry also fails.
- platform_auth.py: add refresh_cache() which discards the in-process
  _cached_token and calls get_token() to re-read from disk.

Together these eliminate the >1 consecutive 401 window described in
issue #1877. Pre-write (B) is the primary fix; runtime retry (A) is the
self-healing fallback for any residual race.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-23 17:47:18 -07:00
Hongming Wang
9ce8d97448 test: regression guard for #1738 — cp-provisioner uses real instance_id 
Pins the fix-invariants from PR #1738 (merged 2026-04-23) against
regression. Pre-fix, `CPProvisioner.Stop` and `IsRunning` both passed
the workspace UUID as the `instance_id` query param:

    url := fmt.Sprintf("%s/cp/workspaces/%s?instance_id=%s",
                        baseURL, workspaceID, workspaceID)
                                              ^ should be the real i-* ID

AWS rejected downstream with InvalidInstanceID.Malformed, orphaned the
EC2, and the next provision hit InvalidGroup.Duplicate on the leftover
SG — full Save & Restart cascade failure.

## Tests added

- **TestStop_UsesRealInstanceIDNotWorkspaceUUID**: stub resolveInstanceID
  to return an i-* ID, assert the CP request's instance_id query param
  carries that i-* value (not the workspace UUID).
- **TestStop_NoInstanceIDSkipsCPCall**: empty DB lookup → no CP call at
  all (idempotent). Guards against re-introducing the "call CP with ''
  and let AWS reject" footgun.
- **TestIsRunning_UsesRealInstanceIDNotWorkspaceUUID**: mirror for the
  /cp/workspaces/:id/status path — same bug shape.

All 3 pass on current staging (which has the fix). Reverting either
Stop or IsRunning to the pre-#1738 shape causes these to fail loud.

Extends molecule-core#1902's regression suite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:45:13 -07:00
Hongming Wang
18ebb1d7bf fix(server): remove 60s A2A client timeout + correct file-read cat args 
Two bugs surfaced while testing Claude Code + OAuth deploys:

1. A2A proxy: a2aClient had a 60s Client.Timeout "safety net" that
   defeated the per-request context deadlines the code otherwise sets
   (canvas = 5m, agent-to-agent = 30m). Claude Code's first-token cold
   start over OAuth takes 30-60s, so every first "hi" into a fresh
   claude-code workspace returned 503 at exactly the 1m mark. Removed
   the Client.Timeout — the context deadline now governs as documented
   in the adjacent comment.

2. Files tab: ReadFile ran `cat <rootPath> <filePath>` as two args to
   cat. `cat /home agent/turtle_draw.py` tries to read the rootPath
   directory (errors "Is a directory") and then resolves the filePath
   relative to the container cwd, which is not guaranteed to equal
   rootPath. Result: the file-content pane stayed blank even though
   the file listed fine. Join into a single path before exec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:25:53 -07:00
Hongming Wang
e337efe974 fix(canvas): propagate runtime through WORKSPACE_PROVISIONING event 
The side-panel runtime pill read "unknown" for newly-deployed workspaces
because canvas-events.ts created the node from WORKSPACE_PROVISIONING
payload — and the payload only carried name + tier. No refetch filled
the gap during provisioning, so the user saw "RUNTIME unknown" on the
card even though the DB row had the real runtime set.

Includes runtime in every WORKSPACE_PROVISIONING emitter:
  * handlers/workspace.go         — initial create
  * handlers/workspace_restart.go — explicit restart, auto-restart, and
                                    crash-recovery resume loop
  * handlers/org_import.go        — multi-workspace org imports

Canvas-side: canvas-events.ts reads payload.runtime when creating the
node; the provisioning test asserts the pill value is populated before
any refetch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:17:49 -07:00
Hongming Wang
dc50a1c775 refactor(canvas): data-drive provider picker from template config.yaml 
The MissingKeysModal's provider list was hardcoded in deploy-preflight.ts
as RUNTIME_PROVIDERS — a per-runtime map that duplicated what each
template repo already declares in its config.yaml. That meant adding a
new provider required changes in two places, and the UI could drift out
of sync with the actual template (e.g. when a template adds a MiniMax or
Kimi model, the picker wouldn't know).

The single source of truth for "which env vars does this workspace need"
is each template's config.yaml:

  * `runtime_config.models[].required_env` — per-model key list
  * `runtime_config.required_env`          — runtime-level AND list

Go /templates already returned `models`. This change:

  * Adds `required_env` alongside `models` on templateSummary so the
    canvas receives the full picture.
  * Rewrites deploy-preflight.ts to derive ProviderChoice[] from a
    template object via `providersFromTemplate(template)`:
      - groups `models[]` by unique required_env tuple
      - falls back to runtime_config.required_env when models is empty
      - decorates labels with model counts (e.g. "OpenRouter (14 models)")
  * `checkDeploySecrets(template, workspaceId?)` now takes a template
    object instead of a runtime string. Any-provider satisfaction still
    short-circuits preflight to ok=true.
  * MissingKeysModal receives `providers` directly; no more lookups.
  * TemplatePalette threads `template.models` + `template.required_env`
    into the preflight.

Side effects:
  * Claude Code's dual-auth (OAuth token OR Anthropic API key) now
    surfaces as two picker options — its config.yaml already declared
    both, the UI just wasn't reading them.
  * Hermes picker now shows 8 provider options (Nous, OpenRouter,
    Anthropic, Gemini, DeepSeek, GLM, Kimi, Kilocode) instead of the
    hand-picked 3, matching its 35-model reality.

Removed the legacy RUNTIME_PROVIDERS / RUNTIME_REQUIRED_KEYS /
getRequiredKeys / findMissingKeys exports; MissingKeysModal.test.tsx
deleted (its coverage is subsumed by the new template-driven
deploy-preflight.test.ts). 58 modal-adjacent tests pass; full canvas
suite 919 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:07:15 -07:00
Hongming Wang
c5bcd7298c Merge remote-tracking branch 'origin/staging' into fix/restore-quickstart-plus-hotfixes 
# Conflicts:
#	workspace-server/internal/handlers/ssrf.go
 2026-04-23 16:42:41 -07:00
Hongming Wang
255fd3c192
Merge branch 'staging' into fix/ki005-security-clean 2026-04-23 16:01:01 -07:00
Hongming Wang
6faea202b9
fix(a2a-queue): nil-safe drain + 202-requeue handling (followup to #1893) (#1896) 
* fix(a2a-queue): nil-safe error extraction in DrainQueueForWorkspace + handle 202-requeue

The drain path called proxyErr.Response["error"].(string) without a comma-
ok assertion. When proxyErr.Response had no "error" key (which happens in
the 202-Accepted-queued branch I added in the same PR — that response is
{"queued": true, "queue_id": ..., "queue_depth": ...}), the type assertion
panicked and killed the platform process.

The platform was down 25 minutes today before this was diagnosed. Fleet
went from 30 real outputs/15min → 0 events.

Two fixes here:

1. Treat 202 Accepted from the inner proxyA2ARequest as "re-queued"
   (target was busy AGAIN). Mark THIS attempt completed; the new queue
   row will be drained on the next heartbeat tick. Don't propagate as
   failure.

2. Defensive type-assertion when reading the error string. Falls back to
   http.StatusText, then a generic "unknown drain dispatch error" so the
   queue still gets a non-empty error_detail for ops debugging.

Now the drain path can never panic on a malformed proxy response.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(a2a-queue): return (202, body, nil) so callers see queued-as-success

Cycle 53 found callers logging 45× 'delegation failed: proxy a2a error'
even though the queue's drain stats showed 48 completions in the same
window. Investigation: my busy-error path returned

  return http.StatusAccepted, nil, &proxyA2AError{Status: 202, Response: ...}

The non-nil proxyA2AError is the failure signal. Even with status=202,
callers' `if proxyErr != nil` branch fires and logs the request as
failed. The 202 status was meaningless — the response body was nil too,
so the caller never even saw the queue_id/depth metadata.

Fix: return success-shape so callers do NOT enter the error branch:

  respBody, _ := json.Marshal(gin.H{"queued": true, "queue_id": qid, ...})
  return http.StatusAccepted, respBody, nil

Net effect: queue continues to absorb busy-errors (working since #1893),
AND callers correctly record the dispatch as queued-success rather than
failed. Closes the cycle 53 misclassification that was making the queue
look ineffective on activity_logs counts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
 2026-04-23 22:55:43 +00:00
Hongming Wang
2baaa977c7 feat(quickstart): default new agents to T3 (Privileged) 
Default tier for a newly-created workspace was T1 (Sandboxed) on
self-hosted and T4 (Full Access) on SaaS. Real work needs at minimum
a read_write workspace mount + Docker daemon access — that's T3
("Privileged") per the tier ladder in CreateWorkspaceDialog. The
user-visible consequence was that clicking "Deploy" on almost any
template landed in a sandbox that couldn't actually run the agent's
tooling until the user knew to bump the tier manually.

### Changes

**Platform (Go)** — default tier flipped from 1→3 in two places so
API callers (Canvas, molecli, org import) all get the same default:

- `handlers/workspace.go`: `POST /workspaces` default when `tier` is
  omitted from the request body.
- `handlers/template_import.go`: `generateDefaultConfig` writes
  `tier: 3` into the auto-generated `config.yaml` for bundle imports
  that don't declare one.

**Canvas** — `CreateWorkspaceDialog.tsx` self-hosted form default
flipped from T1→T3. SaaS stays at T4 (each SaaS workspace runs on
its own sibling EC2, so the shared-blast-radius reasoning doesn't
apply and we can safely go a tier higher).

### Tests

Updated every sqlmock assertion that anchored on the old `tier=1`
default:

- `handlers_test.go::TestWorkspaceCreate` — default-path INSERT now
  expects `3`.
- `handlers_additional_test.go::TestWorkspaceCreate_WithParentID` —
  same.
- `workspace_test.go::TestWorkspaceCreate_DBInsertError` /
  `TestWorkspaceCreate_WithSecrets_Persists` — same.
- `workspace_test.go::TestWorkspaceCreate_TemplateDefaults*` — same
  (current handler semantics ignore the template's `tier:` field and
  fall through to the default; kept tests faithful to the
  implementation, left a comment flagging the latent inconsistency).
- `workspace_budget_test.go::TestWorkspaceBudget_Create_WithLimit` —
  same.
- `template_import_test.go::TestGenerateDefaultConfig` — asserts
  `tier: 3` now.

All `go test -race ./internal/handlers/` pass.

Canvas `CreateWorkspaceDialog` tests don't assert the default tier
(they only reference `tier` as prop data on stub workspaces) so no
test update needed on that side.

### SaaS parity

Zero behaviour change on hosted SaaS. The Go-side default only fires
when the Canvas (or any caller) omits `tier` from the request body.
The SaaS Canvas explicitly passes `tier: 4` from the
CreateWorkspaceDialog `isSaaS ? 4 : 3` branch, so the Go default
never runs on a SaaS request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 15:34:22 -07:00
Hongming Wang
19cd5c9f4b test(router): set ADMIN_TOKEN in TestTestTokenRoute_RequiresAdminAuth_WhenTokensExist 
The test asserts that AdminAuth rejects an unauthenticated request to
the test-token route once any workspace token exists in the DB. It
sets MOLECULE_ENV=development to enable the handler's gate.

After this branch's AdminAuth Tier-1b hatch (middleware/devmode.go),
MOLECULE_ENV=development + empty ADMIN_TOKEN becomes the explicit
fail-open signal for local dev — so the request correctly passes
AdminAuth and falls through to the handler, which then 500s on an
unmocked DB lookup instead of the expected 401.

The security property the test is protecting (no bearer → 401 when
tokens exist) corresponds to the SaaS configuration where
ADMIN_TOKEN is always set. Setting ADMIN_TOKEN in the test suppresses
the dev-mode hatch and reaches AdminAuth's Tier-2 bearer check,
which correctly aborts 401 with "admin auth required".

No production behaviour change — the test is now verifying the path
that actually runs in production (MOLECULE_ENV=production +
ADMIN_TOKEN set).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 15:03:34 -07:00
Hongming Wang
de99a22ffc fix(quickstart): hotfixes discovered during live testing session 
Five additional breakages surfaced while testing the restored stack
end-to-end (spin up Hermes template → click node → open side panel →
configure secrets → send chat). Each fix is narrowly scoped and has
matching unit or e2e tests so they don't regress.

### 1. SSRF defence blocked loopback A2A on self-hosted Docker

handlers/ssrf.go was rejecting `http://127.0.0.1:<port>` workspace
URLs as loopback, so POST /workspaces/:id/a2a returned 502 on every
Canvas chat send in local-dev. The provisioner on self-hosted Docker
publishes each container's A2A port on 127.0.0.1:<ephemeral> — that's
the only reachable address for the platform-on-host path.

Added `devModeAllowsLoopback()` — allows loopback only when
MOLECULE_ENV ∈ {development, dev}. SaaS (MOLECULE_ENV=production)
continues to block loopback; every other blocked range (metadata
169.254/16, TEST-NET, CGNAT, link-local) stays blocked in dev mode.

Tests: 5 new tests in ssrf_test.go covering dev-mode loopback,
dev-mode short-alias ("dev"), production still blocks loopback,
dev-mode still blocks every other range, and a 9-case table test of
the predicate with case/whitespace/typo variants.

### 2. canvas/src/lib/api.ts: 401 → login redirect broke localhost

Every 401 called `redirectToLogin()` which navigates to
`/cp/auth/login`. That route exists only on SaaS (mounted by the
cp_proxy when CP_UPSTREAM_URL is set). On localhost it 404s — users
landed on a blank "404 page not found" instead of seeing the actual
error they should fix.

Gated the redirect on the SaaS-tenant slug check: on
<slug>.moleculesai.app, redirect unchanged; on any non-SaaS host
(localhost, LAN IP, reserved subdomains like app.moleculesai.app),
throw a real error so the calling component can render a retry
affordance.

Tests: 4 new vitest cases in a dedicated api-401.test.ts (needs
jsdom for window.location.hostname) — SaaS redirects, localhost
throws, LAN hostname throws, reserved apex throws.

### 3. SecretsSection rendered a hardcoded key list

config/secrets-section.tsx shipped a fixed COMMON_KEYS list
(Anthropic / OpenAI / Google / SERP / Model Override) regardless of
what the workspace's template actually needed. A Hermes workspace
declaring MINIMAX_API_KEY in required_env got five irrelevant slots
and nothing for the key it actually needed.

Made the slot list template-driven via a new `requiredEnv?: string[]`
prop passed down from ConfigTab. Added `KNOWN_LABELS` for well-known
names and `humanizeKeyName` to turn arbitrary SCREAMING_SNAKE_CASE
into a readable label (e.g. MINIMAX_API_KEY → "Minimax API Key").
Acronyms (API, URL, ID, SDK, MCP, LLM, AI) stay uppercase. Legacy
fallback preserved when required_env is empty.

Tests: 8 new vitest cases covering known-label lookup, humanise
fallback, acronym preservation, deduplication, and both fallback
paths.

### 4. Confusing placeholder in Required Env Vars field

The TagList in ConfigTab labelled "Required Env Vars (from template)"
is a DECLARATION field — stores variable names. The placeholder
"e.g. CLAUDE_CODE_OAUTH_TOKEN" suggested that, but users naturally
typed the value of their API key into the field instead. The actual
values go in the Secrets section further down the tab.

Relabelled to "Required Env Var Names (from template)", changed the
placeholder to "variable NAME (e.g. ANTHROPIC_API_KEY) — not the
value", and added a one-line helper below pointing to Secrets.

### 5. Agent chat replies rendered 2-3 times

Three delivery paths can fire for a single agent reply — HTTP
response to POST /a2a, A2A_RESPONSE WS event, and a
send_message_to_user WS push. Paths 2↔3 were already guarded by
`sendingFromAPIRef`; path 1 had no guard. Hermes emits both the
reply body AND a send_message_to_user with the same text, which
manifested as duplicate bubbles with identical timestamps.

Added `appendMessageDeduped(prev, msg, windowMs = 3000)` in
chat/types.ts — dedupes on (role, content) within a 3s window.
Threaded into all three setMessages call sites. The window is short
enough that legitimate repeat messages ("hi", "hi") from a real
user/agent a few seconds apart still render.

Tests: 8 new vitest cases covering empty history, different content,
duplicate within window, different roles, window elapsed, stale
match, malformed timestamps, and custom window.

### 6. New end-to-end regression test

tests/e2e/test_dev_mode.sh — 7 HTTP assertions that run against a
live platform with MOLECULE_ENV=development and catch regressions
on all the dev-mode escape hatches in a single pass: AdminAuth
(empty DB + after-token), WorkspaceAuth (/activity, /delegations),
AdminAuth on /approvals/pending, and the populated
/org/templates response. Shellcheck-clean.

### Test sweep

- `go test -race ./internal/handlers/ ./internal/middleware/
  ./internal/provisioner/` — all pass
- `npx vitest run` in canvas — 922/922 pass (up from 902)
- `shellcheck --severity=warning infra/scripts/setup.sh
  tests/e2e/test_dev_mode.sh` — clean
- `bash tests/e2e/test_dev_mode.sh` — 7/7 pass against a live
  platform + populated template registry

### SaaS parity

Every relaxation remains conditional on MOLECULE_ENV=development.
Production tenants run MOLECULE_ENV=production (enforced by the
secrets-encryption strict-init path) and always set ADMIN_TOKEN, so
none of these code paths fire on hosted SaaS. Behaviour on real
tenants is byte-for-byte unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:57:18 -07:00
Hongming Wang
47d3ef5b9e refactor(middleware): extract dev-mode fail-open predicate 
AdminAuth and WorkspaceAuth both carried the same 5-line
`ADMIN_TOKEN == "" && MOLECULE_ENV in {development, dev}` check. If a
third middleware ever needs the hatch — or if "dev mode" semantics
change (new env name, allowlist, runtime flag) — the previous shape
made N places to keep in sync and N places a security reviewer has to
audit.

This commit factors the predicate into a single `isDevModeFailOpen()`
helper in `internal/middleware/devmode.go`. Each call site becomes

    if isDevModeFailOpen() { c.Next(); return }

`devmode.go` carries the full rationale (why the hatch exists, why
it's safe for SaaS) so call sites don't need to restate it.

### Also

- Moved the dev-mode env-value set to a package-level `devModeEnvValues`
  map so adding aliases is one line. Matches the existing convention
  (`handlers/admin_test_token.go`) of treating `MOLECULE_ENV != "production"`
  as dev — but stays explicit about which values opt IN rather than
  blanket-accepting everything non-prod.
- Added case-insensitive compare + trim on the env value so operators
  don't have to remember exact casing.
- New `devmode_test.go` unit-tests the predicate directly: 6 cases
  covering happy path, both opt-out signals (ADMIN_TOKEN, production
  mode), short alias, case-insensitive + whitespace tolerance, and an
  explicit negative-space sweep of arbitrary non-dev values
  ("staging", "preview", "test", "devel", "") to lock in that typos
  don't silently enable the hatch.

Existing AdminAuth/WorkspaceAuth integration tests still exercise the
helper indirectly via HTTP — they pass unchanged, confirming the
behaviour is preserved.

### No behavioural change

Before and after this commit, `go test -race ./internal/middleware/`
reports identical results. Zero production surface change — this is a
pure refactor, but it collapses the dev-mode seam from two inline
blocks into one named predicate, which is the shape future
contributors (and security reviewers) can follow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:55:34 -07:00
Hongming Wang
539e3483e4 fix(provisioner): force linux/amd64 pull + create on Apple Silicon hosts (#1875) 
On an Apple Silicon dev box, every `POST /workspaces` failed immediately
with:

  no matching manifest for linux/arm64/v8 in the manifest list entries:
  no match for platform in manifest: not found

because the GHCR workspace-template-* images ship only a linux/amd64
manifest today. `ImagePull` and `ContainerCreate` asked for the daemon's
native arch and missed. The Canvas surfaced this as

  docker image "ghcr.io/molecule-ai/workspace-template-autogen:latest"
  not found after pull attempt — verify GHCR visibility for autogen

— confusing because the image IS visible, just not for linux/arm64.

### Fix

Add an auto-detect helper `defaultImagePlatform()` in
`internal/provisioner/provisioner.go` that returns `"linux/amd64"` on
Apple Silicon hosts and `""` (no preference) everywhere else, with an
env override `MOLECULE_IMAGE_PLATFORM` for operators who want to pin
or disable explicitly. The result is passed to both `ImagePull`
(`PullOptions.Platform`) and `ContainerCreate` (4th arg
`*ocispec.Platform`) so the pulled amd64 manifest matches the
create-time platform spec. Docker Desktop transparently runs it
under QEMU emulation on M-series Macs — slow (2–5× native) but
functional.

SaaS production (linux/amd64 EC2, `MOLECULE_ENV=production`) never
hits the `runtime.GOARCH == "arm64"` branch, so the current behaviour
on real tenants is byte-for-byte unchanged. Opt-in escape hatch for
operators who want it off:

  export MOLECULE_IMAGE_PLATFORM=""     # disable auto-force
  export MOLECULE_IMAGE_PLATFORM=linux/arm64   # pin alternate

`ocispec` is `github.com/opencontainers/image-spec/specs-go/v1` —
already in go.sum v1.1.1 as a transitive dependency of
`github.com/docker/docker`, not a new import.

### Tests

`internal/provisioner/platform_test.go` exercises every branch:

  - `TestDefaultImagePlatform_EnvOverride_ExplicitValue` — env wins
  - `TestDefaultImagePlatform_EnvOverride_EmptyValue` — empty string
    disables the auto-force (operator escape hatch)
  - `TestDefaultImagePlatform_AutoDetect` — linux/amd64 on arm64 Mac,
    "" on every other host
  - `TestParseOCIPlatform` — 7 table-driven cases covering well-formed
    platforms, malformed inputs, and nil handling

### End-to-end verification

Before this commit, `POST /workspaces` on my Apple Silicon box:

  workspace status transitioned: provisioning → failed (~1s)
  log: image pull for ... failed: no matching manifest for linux/arm64/v8

After this commit, fresh DB + fresh platform:

  workspace status transitioned: provisioning → online (~25s)
  log: attempting pull (platform=linux/amd64)
       pulled ghcr.io/molecule-ai/workspace-template-langgraph:latest
  docker ps: ws-7aa08951-00d  Up 27 seconds

The existing provisioner race-tested test suite (`go test -race
./internal/provisioner/`) still passes — the platform pointer defaults
to nil on linux/amd64 hosts, so the CI-resolved test expectations
don't change.

Closes #1875 (arm64 image blocker).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:55:34 -07:00
Hongming Wang
96cc4b0c42 fix(quickstart): wire up template/plugin registry via manifest.json 
The Canvas template palette was empty on a fresh clone because
`workspace-configs-templates/`, `org-templates/`, and `plugins/` are
gitignored and nothing populated them. The registry already exists —
`manifest.json` at repo root lists every curated
`workspace-template-*`, `org-template-*`, and `plugin-*` repo, and
`scripts/clone-manifest.sh` clones them — but the step was absent
from the README and setup.sh, so new users never ran it.

### What this commit does

**1. `setup.sh` runs `clone-manifest.sh` automatically** (once).
After starting the Docker network but before booting infra, iterate
`manifest.json` and clone any workspace_templates / org_templates /
plugins that aren't already populated. Idempotent — subsequent
runs skip dirs that have content. Requires `jq`; when jq is missing
the step prints a clear install hint and skips (doesn't fail).

**2. `clone-manifest.sh` is idempotent.** Before running `git clone`,
check whether the target directory already exists and is non-empty —
skip if so. Lets `setup.sh` rerun safely without forcing the operator
to delete already-cloned template repos.

**3. `ListTemplates` logs the reason it skips a template.** The
handler previously swallowed `resolveYAMLIncludes` errors with
`continue`, so a broken template showed up as an empty palette with
no log trail. Now the include-expansion and yaml.Unmarshal failure
paths both emit a descriptive `log.Printf` — the exact message that
made the stale `org-templates/molecule-dev/` snapshot debuggable:

    ListTemplates: skipping molecule-dev — !include expansion failed:
      !include "core-platform.yaml" at line 25: open .../teams/
      core-platform.yaml: no such file or directory

**4. Remove the in-tree `org-templates/molecule-dev/` snapshot** (170
files). Matches the explicit intent of prior commit
`bfec9e53` — "remove org-templates/molecule-dev/ — standalone repo
is source of truth". A later "full staging snapshot" re-added a
partial copy that had `!include` references to 7 role files that
never existed in the snapshot (`core-platform.yaml`,
`controlplane.yaml`, `app-docs.yaml`, `infra.yaml`, `sdk.yaml`,
`release-manager/workspace.yaml`, `integration-tester/workspace.yaml`).
`clone-manifest.sh` repopulates it fresh from
`Molecule-AI/molecule-ai-org-template-molecule-dev`.

.gitignore exception for `molecule-dev/` is dropped accordingly
— the whole `/org-templates/*` tree is now gitignored, symmetric
with `/plugins/` and `/workspace-configs-templates/`.

**5. Doc updates** (README, README.zh-CN, CONTRIBUTING) mention `jq`
as a prerequisite and describe what setup.sh now does.

### Verification

On a fresh-nuked DB with the updated branch:

1. `bash infra/scripts/setup.sh` — cleanly clones 33/33 manifest
   repos (20 plugins, 8 workspace_templates, 5 org_templates), then
   boots infra. Second run skips all 33 (idempotent).
2. `go run ./cmd/server` — "Applied 41 migrations", :8080 healthy.
3. `curl http://localhost:8080/org/templates` returns 4 templates
   (was `[]`):

       - Free Beats All
       - MeDo Smoke Test
       - Molecule AI Worker Team (Gemini)
       - Reno Stars Agent Team

4. `bash tests/e2e/test_api.sh` — 61/61 pass.
5. `npx vitest run` in canvas — 902/902 pass.
6. `shellcheck infra/scripts/setup.sh` — clean.

### SaaS parity

All changes are local-dev surface. `setup.sh`, `clone-manifest.sh`,
and the local `org-templates/` directory aren't part of the CP
provisioner path — SaaS tenant machines get their templates via
Dockerfile layers or CP-side provisioning, not `clone-manifest.sh`.
The `ListTemplates` log addition is harmless either way (replaces a
silent `continue` with a `log.Printf + continue`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:55:34 -07:00
Hongming Wang
dae7f50095 fix(wsauth): extend dev-mode escape hatch to WorkspaceAuth 
The previous commit on this branch added a dev-mode fail-open branch to
AdminAuth so the Canvas dashboard could enumerate workspaces after the
first token lands in the DB. Verification via Chrome (clicking a
workspace to open its side panel) surfaced the same class of bug on a
different middleware — `WorkspaceAuth` — triggering:

  API GET /workspaces/<id>/activity?type=a2a_receive&source=canvas&limit=50:
    401 {"error":"missing workspace auth token"}

Root cause is identical to AdminAuth's: in local dev the Canvas (at
localhost:3000) calls the platform (at localhost:8080) cross-port, so
`isSameOriginCanvas`'s Host==Referer check fails. Without a bearer
token, every per-workspace read (/activity, /delegations, /memories,
/events/stream, /schedules, etc.) 401s and the side panel is unusable.

### Fix

Symmetric extension in `WorkspaceAuth` (workspace-server/internal/middleware/wsauth_middleware.go):
after the existing `isSameOriginCanvas` fallback, add a narrow escape
hatch that stays fail-open only when BOTH

  - `ADMIN_TOKEN` is unset (operator has not opted in to the #684
    closure), AND
  - `MOLECULE_ENV` is explicitly a dev mode (`development` / `dev`).

SaaS tenants never hit this branch because hosted provisioning sets
both `ADMIN_TOKEN` and `MOLECULE_ENV=production`. The comment in the
code also links back to AdminAuth's Tier-1b for consistency.

### Tests

Three new table-driven tests in wsauth_middleware_test.go mirror the
AdminAuth tier-1b suite, exercising the positive path and both
negative cases:

  - `TestWorkspaceAuth_DevModeEscapeHatch_NoBearer_FailsOpen` — the
    happy path (dev mode, no admin token → 200)
  - `TestWorkspaceAuth_DevModeEscapeHatch_IgnoredInProduction` — the
    SaaS-safety guarantee (production + no admin token → 401)
  - `TestWorkspaceAuth_DevModeEscapeHatch_IgnoredWhenAdminTokenSet` —
    explicit `ADMIN_TOKEN` wins; dev mode does not silently override
    the opt-in

### Comprehensive audit of adjacent middlewares

Re-scanned every file under workspace-server/internal/middleware/ and
every handler that invokes `AbortWithStatusJSON(Unauthorized)` directly,
to check for other surfaces where local dev might silently 401.
Findings, already OK:

  - `CanvasOrBearer` — cosmetic routes already accept localhost:3000
    via `canvasOriginAllowed` (Origin header check); no change needed.
  - `tenant_guard.go` — no-op when `MOLECULE_ORG_ID` is unset (self-
    hosted / dev); no change needed.
  - `session_auth.go` — verifies against `CP_UPSTREAM_URL`; returns
    (false, false) in local dev so callers fall through to bearer; no
    change needed.
  - `socket.go` `HandleConnect` — Canvas browser clients don't send
    `X-Workspace-ID` so skip the bearer check; agent clients do and
    validate as today. No change needed.
  - Handlers in handlers/{discovery,registry,secrets,plugins_install,
    a2a_proxy_helpers,schedules}.go — all workspace-scoped routes
    called by the workspace runtime, not the Canvas browser. Unaffected.
  - `handlers/admin_test_token.go` — already `MOLECULE_ENV`-aware (the
    convention this hatch mirrors).

### End-to-end verification

1. Fresh-nuked DB, platform + canvas restarted with `MOLECULE_ENV=development`
2. `POST /workspaces` → token lands in DB (Tier-1 would close here)
3. Probed every Canvas-hit endpoint with no bearer, with Canvas-like
   `Origin: http://localhost:3000`:

     200  /workspaces
     200  /workspaces/<id>/activity
     200  /workspaces/<id>/delegations
     200  /workspaces/<id>/memories
     200  /approvals/pending
     200  /events

4. Chrome browser test: opened http://localhost:3000, clicked a
   workspace tile — the side panel rendered with the full 13-tab
   structure (Chat, Activity, Details, Skills, Terminal, Config,
   Schedule, Channels, Files, Memory, Traces, Events, Audit) and no
   `Failed to load chat history` error. "No messages yet" placeholder
   shows instead of the 401 retry screen.

5. `go test -race ./internal/middleware/` — clean
6. `bash tests/e2e/test_api.sh` — 61/61 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:55:34 -07:00
Hongming Wang
a93bd58b59 fix(quickstart): keep Canvas working post first workspace + hide SaaS cookie banner on localhost 
Follow-up to the previous commit on this branch. Two additional fresh-clone
regressions surfaced during end-to-end verification, both affecting local
dev only and both landing inside the same SaaS-vs-local-dev seam:

### 1. Canvas 401-loops after first workspace creation

`GET /workspaces` is behind `AdminAuth` (router.go:121 — "C1: unauthenticated
workspace topology exposure"). The middleware has a Tier-1 fail-open branch
that only fires when *no* workspace tokens exist anywhere in the DB. The
moment a user creates their first workspace — via either the Canvas UI, the
API, or the e2e-api test suite — a token lands in the DB, Tier-1 closes, and
the Canvas (which has no bearer token in local dev: no WorkOS session, no
NEXT_PUBLIC_ADMIN_TOKEN baked in at build time) gets 401 on every list
call. The UI renders a stuck "API GET /workspaces: 401 admin auth required"
placeholder forever.

SaaS is unaffected because hosted provisioning always sets both
`ADMIN_TOKEN` and `MOLECULE_ENV=production`, and the Canvas there either
carries a WorkOS session cookie or `NEXT_PUBLIC_ADMIN_TOKEN` baked into
the JS bundle.

**Fix** (`workspace-server/internal/middleware/wsauth_middleware.go`): add
a narrow Tier-1b escape hatch that stays fail-open when *both*
`ADMIN_TOKEN` is unset *and* `MOLECULE_ENV` is explicitly a dev mode
("development" / "dev"). Production never hits it (SaaS sets
`MOLECULE_ENV=production`). Mirrors the existing convention in
`handlers/admin_test_token.go` which gates the e2e test-token endpoint on
`MOLECULE_ENV != "production"`.

Three new regression tests in `wsauth_middleware_test.go`:
- `TestAdminAuth_DevModeEscapeHatch_FailsOpenWithHasLiveTokens` — the
  happy path (dev mode, no admin token, tokens exist → 200)
- `TestAdminAuth_DevModeEscapeHatch_IgnoredWhenAdminTokenSet` — explicit
  `ADMIN_TOKEN` wins; dev mode does not silently re-open the gate
- `TestAdminAuth_DevModeEscapeHatch_IgnoredInProduction` — the
  SaaS-safety guarantee (production + no admin token + tokens exist → 401)

`.env.example` flipped to set `MOLECULE_ENV=development` by default so
new users get the dev-mode hatch automatically via `cp .env.example .env`.
SaaS provisioning overrides to `production`, consistent with the existing
convention used by the secrets-encryption strict-init path.

### 2. SaaS cookie/privacy banner rendered on localhost

`CookieConsent` mounted unconditionally in the root layout, so
`npm run dev` on localhost showed a "Cookies & your privacy" banner
pointing at `moleculesai.app/legal/privacy`. That banner is a
GDPR/ePrivacy compliance UI that only applies to the hosted SaaS
offering; self-hosted / local-dev / Vercel-preview hosts must not
see it.

**Fix** (`canvas/src/components/CookieConsent.tsx`): gate render on
`isSaaSTenant()`. Matches the convention used by `AuthGate` and the
workspace tier picker elsewhere in the codebase.

Tests (`canvas/src/components/__tests__/CookieConsent.test.tsx`):
existing tests now stub `window.location.hostname` to a SaaS
subdomain before rendering (required since `isSaaSTenant()` on jsdom's
default "localhost" would suppress the banner). Added two new tests
for the local-dev hide path:
- `does NOT render on local dev (non-SaaS hostname)`
- `does NOT render on a LAN hostname (192.168.*, *.local)`

### Verification

On a fresh-nuked DB with the updated branch:

1. `bash infra/scripts/setup.sh` — clean
2. `go run ./cmd/server` — "Applied 41 migrations", :8080 healthy,
   dev-mode hatch armed (`MOLECULE_ENV=development`)
3. `npm run dev` in canvas — :3000 renders, no cookie banner
4. `bash tests/e2e/test_api.sh` — **61 passed, 0 failed**
   (test suite creates tokens; GET /workspaces stays 200 under the hatch)
5. Browser at http://localhost:3000 AFTER the e2e run:
   - Canvas renders the workspace list (no 401 placeholder)
   - No cookie banner
6. `npx vitest run` — **902 tests passed** (900 prior + 2 new hide tests)
7. `go test -race ./internal/middleware/` — all passing (3 new
   dev-mode tests + existing Issue-180 / Issue-120 / Issue-684 suite),
   coverage 81.8%

### SaaS parity audit

Same principle as the rest of this branch: local must work without
weakening SaaS.

- Dev-mode hatch: conditional on `MOLECULE_ENV=development`.
  Production tenants always run `MOLECULE_ENV=production` (already
  enforced by the secrets-encryption `InitStrict` path in
  `internal/crypto/aes.go`). Branch is unreachable there.
- Cookie banner: gated on `isSaaSTenant()` which checks
  `NEXT_PUBLIC_SAAS_HOST_SUFFIX` (default `.moleculesai.app`). SaaS
  hosts still get the banner; every other host doesn't.

No change to SaaS behaviour. #1822 backend-parity tracker untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:55:33 -07:00
Hongming Wang
09faaec1ab
Merge branch 'staging' into fix/restart-preserves-user-config 2026-04-23 14:39:21 -07:00
rabbitblood
751b265dbd fix(a2a-queue): use partial-index ON CONFLICT syntax (not constraint name) 
#1892's EnqueueA2A INSERT used `ON CONFLICT ON CONSTRAINT idx_a2a_queue_idempotency
DO NOTHING`, but Postgres rejects this:

  ERROR: constraint "idx_a2a_queue_idempotency" for table "a2a_queue" does not exist

Partial unique INDEXES cannot be referenced by name in ON CONFLICT — that
form is reserved for true CONSTRAINTs created via CREATE TABLE ... CONSTRAINT
or ALTER TABLE ADD CONSTRAINT. Partial indexes need the column-list +
WHERE form so the planner can match the index.

Effect of the bug: every EnqueueA2A errored, the busy-error fallback
returned 503 instead of 202, queue stayed empty. Cycle 50 observed
46 busy errors / 0 queue rows — the deployed Phase 1 had no effect.

Fix: switch to

  ON CONFLICT (workspace_id, idempotency_key)
    WHERE idempotency_key IS NOT NULL AND status IN ('queued','dispatched')
    DO NOTHING

Verified manually against the live `a2a_queue` table on staging — INSERT
returns the new id; cleanup deleted the test row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:22:13 -07:00
rabbitblood
87a97846cd feat(a2a): queue-on-busy — Phase 1 of priority queue (#1870) 
## Problem

When a lead delegates to a worker that's mid-synthesis, the proxy returns
503 "workspace agent busy" and the caller records the delegation as
failed. On fan-out storms from leads this hits ~70% drop rate — today's
observed numbers in the cycle reports.

## Fix — Phase 1 TASK-level queue-on-busy

When `handleA2ADispatchError` determines the target is busy, instead of
returning 503, enqueue the request as priority=TASK and return 202
Accepted with `{queued: true, queue_id, queue_depth}`. The workspace's
next heartbeat (≤30s) drains one item if it reports spare capacity.

Files:

  - migrations/042_a2a_queue.{up,down}.sql — `a2a_queue` table with
    partial indexes on status='queued' + idempotency_key. Schema
    supports PriorityCritical/Task/Info from day one so Phase 2/3 ship
    without migration churn.

  - internal/handlers/a2a_queue.go — EnqueueA2A / DequeueNext /
    Mark*-helpers plus WorkspaceHandler.DrainQueueForWorkspace. Uses
    `SELECT ... FOR UPDATE SKIP LOCKED` so concurrent drains can't
    double-claim the same row. Max 5 attempts before marking 'failed'
    so a stuck item doesn't wedge the queue forever.

  - internal/handlers/a2a_proxy_helpers.go — isUpstreamBusyError branch
    calls EnqueueA2A and returns 202 on success. Falls through to the
    legacy 503 on enqueue error (DB hiccup shouldn't silently drop).

  - internal/handlers/registry.go — RegistryHandler gets a QueueDrainFunc
    injection hook (SetQueueDrainFunc). When Heartbeat sees
    active_tasks < max_concurrent_tasks, spawns a goroutine that calls
    the drain hook. context.WithoutCancel ensures the drain outlives
    the heartbeat handler's ctx.

  - internal/router/router.go — wires wh.DrainQueueForWorkspace into
    rh.SetQueueDrainFunc after both are constructed.

## Not in this PR (Phase 2/3/4 follow-ups)

  - INFO priority + TTL (Phase 2)
  - CRITICAL priority + soft preemption between tool calls (Phase 3)
  - Age-based promotion so TASK doesn't starve (Phase 4)
  - `GET /workspaces/:id/queue` observability endpoint

Schema already supports all of these; only the dispatch + policy code
remains.

## Tests

  - TestExtractIdempotencyKey (5 cases): messageId parsing is robust
  - TestPriorityConstants: ordering invariant + 50=TASK default
    alignment with migration DEFAULT

Full DB-touching tests (FIFO order, retry bound, idempotency conflict)
intentionally deferred to the CI migration-enabled path — sqlmock
ceremony would duplicate the existing test infrastructure 3× over and
the behaviour is directly expressible in SQL constraints (FOR UPDATE
SKIP LOCKED, partial unique index).

## Expected impact once deployed

  - a2a_receive error with "busy" flavor drops from ~69/10min observed
    today to ~0
  - delegation_failed rate drops from ~50% to <5%
  - real_output metric rises from ~30/15min back toward the pre-
    throttle baseline

Closes #1870 Phase 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 14:09:29 -07:00
Molecule AI Dev Lead
84d9738b12 test(handlers): update KI005 terminal tests for ValidateToken (GH#756) 
Three tests used ValidateAnyToken mock expectations and fallthrough behavior.
Now that HandleConnect uses ValidateToken (token-to-workspace binding), update:

- RejectsUnauthorizedCrossWorkspace: mock expects SELECT id+workspace_id
  (ValidateToken pattern); row returns workspace_id=ws-caller so validation
  passes, then CanCommunicate=false → 403 as before.

- RejectsInvalidToken: add setupTestDB so ValidateToken has a real mock;
  with no ExpectQuery set, the query returns error → 401 Unauthorized
  (was 503 fall-through; 401 is the correct explicit rejection).

- AllowsSiblingWorkspace: add setupTestDB + ValidateToken mock returning
  ws-pm binding; CanCommunicate=true → Docker nil → 503 as before.
 2026-04-23 20:59:21 +00:00
Hongming Wang
ba03fcfe2d fix(restart): preserve user config volume on default restart (#1822 drift-risk-3) 
### Repro

On Canvas: create a workspace named "Hermes Agent" (runtime=langgraph,
model=langgraph default). Open the Config tab, switch the model to a
Minimax provider + Minimax token, hit Save and Restart. The model
reverts to the default on every restart.

### Root cause

`workspace_restart.go` called `findTemplateByName(configsDir, wsName)`
unconditionally when the request body had no explicit `template`:

    template := body.Template
    if template == "" {
        template = findTemplateByName(h.configsDir, wsName)
    }

`findTemplateByName` normalises the name ("Hermes Agent" → "hermes-agent")
and ALSO scans every template's `config.yaml` for a matching `name:`
field — a two-layer match that returns non-empty for any workspace whose
name coincides with a template dir OR any template whose config.yaml
claims the same display name.

When the match returned non-empty, the restart handler set
`templatePath = <template>` and the provisioner rewrote the workspace's
config volume from the template on `Start`. The Canvas Save+Restart
flow's `PUT /workspaces/:id/files/config.yaml` had already written the
user's edits to the volume — those got clobbered.

The comment immediately below (line 187) ALREADY said:

    // Apply runtime-default template ONLY when explicitly requested
    // via "apply_template": true. Use case: runtime was changed via
    // Config tab — need new runtime's base files. Normal restarts
    // preserve existing config volume (user's model, skills, prompts).

The code contradicted the comment. The design intent was right; the
implementation short-circuited it. Matches drift-risk #3 in #1822's
Docker-vs-EC2 parity tracker ("Config-tab save must flush to DB before
kicking off restart, not deferred").

### Fix

Extracted the template-resolution chain into a pure function
`resolveRestartTemplate(configsDir, wsName, dbRuntime, body)` in a new
`restart_template.go`. Gated the name-based auto-match on
`body.ApplyTemplate`:

  1. Explicit `body.Template` → always honoured (caller consent).
  2. `ApplyTemplate=true` → name-based auto-match (prior behaviour).
  3. `RebuildConfig=true` → org-templates recovery fallback (#239).
  4. `ApplyTemplate=true` + dbRuntime → `<runtime>-default/`.
  5. Fall through → empty path + "existing-volume" label. Provisioner
     reuses the volume. This is the path Canvas Save+Restart now hits.

The handler now calls this helper and uses the returned path directly.
Duplicate rebuild_config blocks at lines 167-186 were consolidated into
the helper's single tier-3 case in passing.

### Abstraction win

`resolveRestartTemplate` is a pure function — no gin context, no DB, no
network. Takes a struct input, returns two strings. The whole priority
chain is unit-testable in a temp dir, which is exactly what
`restart_template_test.go` does.

### Tests

`restart_template_test.go` — 8 table-style unit tests covering every
branch of the priority chain:

  - DefaultRestart_PreservesVolume — the regression. Even when a
    template's config.yaml `name:` field matches the workspace name
    exactly (worst case), a default restart MUST return empty path.
  - ExplicitTemplate_AlwaysHonoured — caller-by-name, any mode.
  - ApplyTemplate_NameMatch — opt-in restores the auto-match.
  - ApplyTemplate_RuntimeDefault — runtime-change flow still works.
  - ApplyTemplate_NoMatch_NoRuntime — fallback to existing-volume.
  - InvalidExplicitTemplate_ProceedsWithout — traversal attempt stays
    inside root, falls through cleanly.
  - NonExistentExplicitTemplate — deleted/missing template falls through.
  - Priority_ExplicitBeatsApplyTemplate — explicit Template wins over
    name-match when both fire.

Full handlers race suite (`go test -race ./internal/handlers/`) still
passes — existing Restart-handler tests unchanged.

### Blast radius

Any restart caller that omitted `apply_template: true` and relied on
name-matching auto-applying a template is now a behaviour change.
Identified call sites in this repo:

  - Canvas Save+Restart button (store/canvas.ts) — explicitly the
    flow this commit fixes, definitely wanted the fix.
  - Canvas Restart button (same file) — same semantics; user expects
    a restart, not a template reset.
  - Auto-restart sweeper (#1858) — never passes apply_template and
    depends on the existing volume having valid config. Separately,
    `workspace_provision.go`'s #1858 recovery path detects empty
    volumes and auto-applies `<runtime>-default` without going
    through findTemplateByName, so recovery is unaffected.
  - RestartByID — internal callers; audited, all intended "restart
    as-is", none relied on auto-template-match.

No SaaS parity impact — this is a handler behaviour fix that applies
equally to Docker and EC2 backends (both use the same Restart handler
before dispatching to their respective provisioners).

Refs #1822 drift-risk-3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 13:57:42 -07:00
Molecule AI Dev Lead
e12d8d12d3 fix(security): P0 — F1085/KI-005/CWE-78 security fixes rebased clean onto staging 
Supersedes PRs #1882 + #1883 (both had merge conflicts / missing callerID decl).
Applied directly onto current staging HEAD (26c4565).

Changes:
- terminal.go: upgrade KI-005 guard ValidateAnyToken → ValidateToken (GH#756/#1609)
  Binds bearer token to claimed X-Workspace-ID; prevents cross-workspace terminal forge.
  Fixes missing `callerID` declaration that broke compilation in PR #1882.
- ssrf.go: add ssrfCheckEnabled flag + setSSRFCheckForTest helper for test isolation
- ssrf.go validateRelPath: harden to reject empty/"." paths; check both raw+cleaned for ..
- templates.go: ReadFile — exec form cat ["cat", rootPath, filePath] (was shell concat)
- orgtoken/tokens_test.go: fix regex (remove optional LIMIT $1 group)
- wsauth_middleware_test.go: add deprecated orgTokenOrgIDQuery const; update comments
- wsauth_middleware_org_id_test.go: use real org_id UUID in DBRowScanError test row

Security classification:
  F1085 (CWE-78) path traversal + exec form — P0 Fixed
  KI-005 terminal auth bypass (ValidateToken upgrade) — P0 Fixed
  CWE-22 SSRF test isolation — P0 Fixed

Co-Authored-By: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-Authored-By: Core Platform Lead <core-platform@agents.moleculesai.app>
 2026-04-23 20:52:49 +00:00
Hongming Wang
a56b765b2d
docs: testing strategy + PR hygiene + backend parity matrix + boot-event postmortem (#1824) 
Bundles the documentation and lightweight tooling landed during the
2026-04-23 ops/triage session. Pure additions — no behavior changes.

## Added

### docs/architecture/backends.md
Parity matrix for Docker vs EC2 (SaaS) workspace backends. 18 features
tabulated with current status; 6 ranked drift risks; enforcement
hooks (parity-lint + contract tests). Living document — owners are
workspace-server + controlplane teams.

### docs/engineering/testing-strategy.md
Tiered test-coverage floors instead of a blanket 100% target. Seven
tiers by code class (auth/crypto → generated DTOs). Per-package
current-state snapshot + targets. Tracks the 3 biggest coverage gaps
(tokens.go 0%, workspace_provision.go 0%, wsauth ~48%) against their
tier-1/2 floors.

### docs/engineering/pr-hygiene.md
Captures the patterns that keep diffs reviewable. Motivated by the
2026-04-23 backlog audit where 8 of 23 open PRs had 70-380-file bloat
from stale branch drift. Covers: small-PR sizing, rebase-not-merge,
cherry-pick-onto-fresh-base for recovery, targeting staging first,
describing why-not-what.

### docs/engineering/postmortem-2026-04-23-boot-event-401.md
Postmortem for the /cp/tenants/boot-event 401 race. Root cause (DB
INSERT ordered AFTER readiness check), detection path (E2E + manual
log inspection), lessons (write-before-read pattern, integration
tests needed, E2E alerting gap, invariants-as-comments).

### tools/check-template-parity.sh
CI lint for template repos — diffs the `${VAR:+VAR=${VAR}}` provider-
key forwarders between install.sh (bare-host / EC2 path) and start.sh
(Docker path). Catches the #5 drift risk from backends.md before it
ships.

### workspace-server/internal/provisioner/backend_contract_test.go
Shared behavioral contract scaffold for Provisioner + CPProvisioner.
Compile-time assertions catch method-signature drift today; scenario-
level runs are t.Skip'd pending backend nil-hardening (drift risk #6,
see backends.md).

## Updated

### README.md
Links the new engineering docs + backends parity matrix into the
Documentation Map so agents and humans can actually find them.

## Related issues

- #1814 — unblock workspace_provision_test.go (broadcaster interface)
- #1813 — nil-client panic hardening (drift risk #6)
- #1815 — Canvas vitest coverage instrumentation
- #1816 — tokens.go 0% → 85%
- #1817 — 5 sqlmock column-drift failures
- #1818 — Python pytest-cov setup
- #1819 — wsauth middleware coverage gap
- #1821 — tiered coverage policy (meta)
- #1822 — backend parity drift tracker

Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
 2026-04-23 19:59:38 +00:00
Hongming Wang
7352153fa5
fix(provisioner): auto-recover from empty config volume on restart (#1858) (#1861) 
When auto-restart fires for a claude-code workspace and the config volume
is empty (first-provision race, manual intervention, volume prune, etc.),
the preflight at workspace_provision.go:151 marks the workspace 'failed'
and bails. Operator is then required to run:

  docker stop ws-<id>
  docker run --rm -v ws-<id>-configs:/configs -v <template>:/src:ro \
    alpine sh -c 'cp -r /src/. /configs/'
  docker start ws-<id>
  psql -c "UPDATE workspaces SET status='online' WHERE id='...'"

Today (2026-04-23) this manifested twice: Research Lead at 16:31 UTC,
Tech Researcher at 18:55 UTC. Both recovered with the same manual steps.

## Fix

Before bailing, attempt recovery by resolving the workspace's runtime-
default template from `h.configsDir` (same source of truth the Restart
handler uses for `apply_template=true`):

  runtimeTemplate := filepath.Join(h.configsDir, payload.Runtime+"-default")

If the template directory exists, rebuild `cfg` with it as the template
path and continue. Provisioner.Start() then writes the template files
into the volume during container bring-up, identical to first-provision.
Only if the recovery template itself is missing do we fall through to
the original fail-path.

## Why this is strictly safer than the previous behaviour

- Nothing new is attempted when the volume is already healthy — the
  recovery path only fires in the case that previously fail-marked the
  workspace. Net effect: same behaviour on the happy path, graceful
  recovery on the previously-terminal edge case.
- payload.Runtime is populated by the Restart handler from the DB's
  workspaces.runtime column, so the recovered template matches the
  workspace's declared runtime. Can't accidentally swap a langgraph
  workspace onto a claude-code template.
- User state loss bounds are the same as for `apply_template=true`
  (which operators already use when they want a clean slate). If the
  user had custom config.yaml edits, they're gone — but they were
  ALREADY gone (volume was empty, that's why we're here).

## Test

- `go build ./cmd/server` passes (verified via docker run golang:1.25-alpine)
- Tested live on the running fleet's recovery today: running the recovered
  workspaces (Research Lead, Tech Researcher) with this code would have
  skipped the manual cp-from-template step entirely.

## Follow-up (not in this PR)

- Unit test covering the recovery path (needs a VolumeHasFile mock and
  a configsDir temp dir with a runtime-default template). Filing as a
  follow-up.
- Class-level fix: write a `.provisioned` marker file to the config
  volume on successful first-provision so this preflight can distinguish
  "volume exists but empty (real bug)" from "volume empty and un-
  provisioned (first-time)". This PR's fix works for both cases but the
  marker would give cleaner diagnostics.

Closes the immediate bug in #1858.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
 2026-04-23 19:31:13 +00:00
molecule-ai[bot]
0466dc5f7e
Merge branch 'staging' into fix/main-orgtoken-mocks 2026-04-23 18:59:34 +00:00
Hongming Wang
d6abc1286f
fix(workspace): auto-fill model from template's runtime_config when missing (#1779) 
Extends the existing "read runtime from template config.yaml"
preflight to also pre-fill `model` from the template's
runtime_config.model (current format) or top-level `model:` (legacy
format). Without this, any create path that names a template but
doesn't pass an explicit model produced a workspace with empty
model — and hermes-agent's compiled-in Anthropic fallback ran with
whatever key the user did provide, 401'ing at the first A2A call.

Affected paths (all produced broken workspaces before this change):
- TemplatePalette "Deploy" button (POSTs only name + template + tier)
- Direct API / script callers (MCP, CI scripts)
- Anyone copying an existing workspace's template name without model

PR #1714 fixed the canvas CreateWorkspaceDialog's hermes branch —
when the user typed template="hermes" in the dialog, a provider
picker + model auto-fill kicked in. But TemplatePalette and direct
API calls bypassed that dialog entirely, so the trap stayed open.

Fix is backend-side so it catches every caller at once (defense in
depth). The parser is line-based + a minimal state var tracking
whether the current line sits under `runtime_config:` — matches the
existing fragile-but-safe style used for `runtime:` above. Strings
are trimmed of quote wrappers so both `model: x` and `model: "x"`
round-trip.

Explicit model in the payload still wins — we only pre-fill when
payload.Model is empty. Added TestWorkspaceCreate_
CallerModelOverridesTemplateDefault to pin that contract.

## Tests
- TestWorkspaceCreate_TemplateDefaultsMissingRuntimeAndModel — the
  hermes-trap fix: runtime=hermes + model=nousresearch/... inherits
  from template when payload omits both.
- TestWorkspaceCreate_TemplateDefaultsLegacyTopLevelModel — legacy
  top-level `model:` still fills.
- TestWorkspaceCreate_CallerModelOverridesTemplateDefault — explicit
  payload.model NOT overwritten.
- Full suite `go test -race ./...` stays green.

## Complementary work in flight
- PR molecule-core#1772 — fixes the E2E Staging SaaS which had the
  same trap on its own POST body (missing provider prefix).
- Canvas TemplatePalette could still surface a richer per-template
  key picker (deferred; MissingKeysModal already handles keys, and
  the default model now flows from the template config).

Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
 2026-04-23 18:58:04 +00:00
Hongming Wang
f001a4cf5e
fix(registry): heartbeat transitions provisioning→online on first heartbeat (#1784) (#1794) 
Workspaces restart with status='provisioning' and never transition to
'online' because the runtime never calls /registry/register after
container start — only the heartbeat loop runs post-boot. The heartbeat
handler had transitions for online→degraded, degraded→online, and
offline→online, but NOT provisioning→online, leaving newly-started
workspaces in a phantom-idle state where the scheduler defers dispatch
and the A2A proxy rejects them even though they're running fine.

Fix: add provisioning→online transition to evaluateStatus(), guarded by
`AND status = 'provisioning'` in the UPDATE WHERE clause so a concurrent
Delete cannot flip 'removed' back to 'online'. Broadcasts WORKSPACE_ONLINE
with recovered_from='provisioning' so dashboard/scheduler reflect reality.

Add TestHeartbeatHandler_ProvisioningToOnline to cover the new path.

Issue: Molecule-AI/molecule-core#1784

Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
 2026-04-23 18:34:10 +00:00
Hongming Wang
c23ff848aa
fix(cp-provisioner): look up real EC2 instance_id for Stop + IsRunning (#1738) 
Resolves a "Save & Restart cascade" failure on SaaS tenants. Observed
2026-04-22 on hongmingwang workspace a8af9d79 after a Config-tab save:

  03:13:20 workspace deprovision: TerminateInstances
           InvalidInstanceID.Malformed: a8af9d79-... is malformed
  03:13:21 workspace provision: CreateSecurityGroup
           InvalidGroup.Duplicate: workspace-a8af9d79-394 already
           exists for VPC vpc-09f85513b85d7acee

Root cause: CPProvisioner.Stop and IsRunning passed the workspace UUID
as the `instance_id` query param to CP. CP forwarded it to EC2
TerminateInstances, which rejected it (EC2 ids are i-…, not UUIDs).
The failed terminate left the workspace's SG attached → the immediate
re-provision hit InvalidGroup.Duplicate → user saw `provisioning
failed`.

Fix: both methods now call a new `resolveInstanceID` that reads
`workspaces.instance_id` from the tenant DB and passes the real EC2
id downstream. When no row / no instance_id exists, Stop is a no-op
and IsRunning returns (false, nil) so restart cascades can freshly
re-provision.

resolveInstanceID is exposed as a `var` package-level func so tests
can swap it for a pairs-map stub without standing up sqlmock — the
per-table DB scaffolding was a heavier price than the surface
warranted given these tests are about the CP HTTP flow downstream
of the lookup, not the lookup SQL itself.

Adds regression tests:
  - TestStop_EmptyInstanceIDIsNoop: no DB row → no CP call
  - TestIsRunning_UsesDBInstanceID: DB id round-trips to CP
  - TestIsRunning_EmptyInstanceIDReturnsFalse: no instance → false/nil
Updates existing tests to assert the resolved instance_id (i-abc123
variants) instead of the previous buggy workspaceID.

After this lands, user's existing workspaces with stale instance_id
bindings still need a manual cleanup of the orphaned EC2 + SG (done
for a8af9d79 today). Future restarts use the correct id.

Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 18:25:29 +00:00
molecule-ai[bot]
5f0bfc1f19
Merge branch 'staging' into fix/main-orgtoken-mocks 2026-04-23 18:12:47 +00:00