Four findings from the security audit on PR #641:
FIX 1 (MEDIUM): import_url scheme validation
- Reject non-HTTPS import URLs with 400 before forwarding to CF API.
Prevents SSRF via http://, git://, ssh://, file:// etc.
FIX 2 (MEDIUM): CF 5xx error leakage
- Add cfErrMessage() helper: returns "upstream service error" for CF 5xx
responses and non-CF errors, passes through 4xx messages.
- Applied at all four CF-error response sites (Create, Get, Fork, Token).
FIX 3 (LOW): repo name validation
- Add package-level repoNameRE = ^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$
- Validate in Create and Fork handlers when caller supplies an explicit name.
Auto-generated names ("molecule-ws-<id>") are always safe and skip validation.
FIX 4 (LOW): response body size limit in CF client
- Wrap resp.Body with io.LimitReader(1 MB) before json.NewDecoder in do().
Prevents memory exhaustion from a runaway/malicious CF response.
Tests: 16 new tests covering all four fixes (cfErrMessage 4xx/5xx/non-API,
import_url non-HTTPS cases, invalid repo names in Create and Fork).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a minimal but complete integration with the Cloudflare Artifacts API
(private beta Apr 2026, public beta May 2026) — "Git for agents" versioned
workspace-snapshot storage.
## What's included
**`platform/internal/artifacts/client.go`** — typed Go HTTP client for the
CF Artifacts REST API:
- CreateRepo, GetRepo, ForkRepo, ImportRepo, DeleteRepo
- CreateToken, RevokeToken
- CF v4 response-envelope decoding; *APIError with StatusCode + Message
**`platform/internal/handlers/artifacts.go`** — four workspace-scoped
Gin handlers (all behind WorkspaceAuth middleware):
- POST /workspaces/:id/artifacts — attach or import a CF Artifacts repo
- GET /workspaces/:id/artifacts — get linked repo info (DB + live CF)
- POST /workspaces/:id/artifacts/fork — fork the workspace's repo
- POST /workspaces/:id/artifacts/token — mint a short-lived git credential
**`platform/migrations/028_workspace_artifacts.up.sql`** — `workspace_artifacts`
table: one-to-one link between a workspace and its CF Artifacts repo.
Credentials are never stored; only the credential-stripped remote URL.
**`platform/internal/router/router.go`** — wire the four routes into the
existing wsAuth group.
## Configuration
Two env vars gate the feature (returns 503 when either is absent):
- CF_ARTIFACTS_API_TOKEN — Cloudflare API token with Artifacts write perms
- CF_ARTIFACTS_NAMESPACE — Cloudflare Artifacts namespace name
## Tests
- 10 client-level tests (httptest.Server + CF v4 envelope mocks)
- 14 handler-level tests (sqlmock DB + mock CF server)
- Helper unit tests for stripCredentials, cfErrToHTTP
All 21 packages pass (go test ./...).
Closes#595
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both conflicts were comment-only — identical logic on both sides:
- registry.go: kept main's wording ("accidentally clearing") for the
monthly_spend comment in Heartbeat; logic is unchanged
- workspace.go: kept HEAD's comment (describes PR #634's clamping
behaviour: [0, maxMonthlySpend]); logic is unchanged
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merge gate passed (all 7 gates). Adds budget_limit + monthly_spend columns via 027_workspace_budget (ADD COLUMN IF NOT EXISTS — idempotent). A2A budget enforcement is fail-open on DB errors. WorkspaceAuth on all budget routes. Schema migration — CEO explicit authorization in chat. Merging before #634 which writes to monthly_spend.
Rebase on origin/fix/issue-631-migration-gap which inserts token_usage
(025) and org_plugin_allowlist (026); bump workspace_budget from 025 to
027 so the sequential runner applies all three in the correct order.
Update workspace_budget_test.go and workspace_test.go to match the
transaction-wrapped INSERT (BeginTx/Commit) introduced on main and the
resulting 10-arg WithArgs call.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Security Auditor findings on PR #611:
Fix 1 (BLOCKING): Remove budget_limit handling from Update() entirely.
PATCH /workspaces/:id uses ValidateAnyToken — any enrolled workspace bearer
could self-clear its own spending ceiling. The dedicated AdminAuth-gated
PATCH /workspaces/:id/budget is the only authorised write path.
Fix 2 (MEDIUM): Strip budget_limit and monthly_spend from Get() response
before c.JSON(). GET /workspaces/:id is on the open router — any caller
with a valid UUID must not read billing data.
Also updates four existing tests in workspace_budget_test.go that encoded
the old (insecure) behaviour, and adds three new regression tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Budget limit exceeded on A2A proxy now returns HTTP 402 PaymentRequired
instead of 429 TooManyRequests, matching the issue spec and the FE amber
banner check. Updates a2a_proxy.go, workspace_budget_test.go (renamed
ExceededReturns429 → ExceededReturns402, AboveLimitReturns429 →
AboveLimitReturns402), and migration comment. All go test ./... pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New BudgetHandler with GetBudget and PatchBudget methods
- GET returns budget_limit (null or int64 USD cents), monthly_spend,
and computed budget_remaining (null when no limit, can be negative
when over-budget so callers can see the magnitude of the overage)
- PATCH accepts {budget_limit: int64|null}; null clears the ceiling;
validates non-negative values; re-reads DB to echo final state
- Both handlers are wired in router.go under the WorkspaceAuth group
- 14 unit tests covering happy paths, 404, 400 validation, DB errors,
over-budget state, zero limit, and clear-limit round-trip
- All 20 packages pass go test ./... and go build ./... is clean
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Migration 025: ADD COLUMN budget_limit BIGINT DEFAULT NULL and
monthly_spend BIGINT NOT NULL DEFAULT 0 to workspaces table
- Models: BudgetLimit *int64 in CreateWorkspacePayload;
MonthlySpend int64 in HeartbeatPayload
- workspace.go: scanWorkspaceRow, workspaceListQuery, Get, Create, and
Update all handle budget_limit/monthly_spend; budget_limit is gated
as a sensitiveUpdateField
- registry.go: heartbeat conditionally writes monthly_spend only when
payload.MonthlySpend > 0 (avoids overwriting with zero)
- a2a_proxy.go: checkWorkspaceBudget() returns 429 when
monthly_spend >= budget_limit (NULL = no limit; fail-open on DB error)
- Tests: 8 new workspace_budget_test.go tests + patched existing tests
for the 20-column scanWorkspaceRow and 10-param CREATE INSERT
Field type: BIGINT (int64), units: USD cents (budget_limit=500 = $5.00/month)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A malicious or buggy agent could report MonthlySpend = math.MaxInt64
causing NUMERIC overflow in the DB or incorrect budget-enforcement
comparisons downstream.
Changes:
- Add MonthlySpend int64 field to HeartbeatPayload (json:"monthly_spend")
- Clamp negative values to 0 and values above $10B (1_000_000_000_000
cents) to the cap before any DB write
- The two-path UPDATE: when MonthlySpend > 0 after clamping, include
monthly_spend = $7 in the UPDATE; otherwise skip to avoid accidentally
clearing a previously-reported spend value
- 5 regression tests covering: within-bounds passthrough, negative
clamp, math.MaxInt64 overflow clamp, exact-cap boundary, and
zero/omitted no-update path
Note: this branch introduces MonthlySpend to HeartbeatPayload; it will
need trivial conflict resolution when feat/issue-541-budget-limit-backend
merges, as that branch also adds the field (without the cap). Keep this
branch's clamping logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adversarial or buggy agents can report INT64_MAX token counts via A2A
responses. Without clamping, upsertTokenUsage would pass these directly to
Postgres NUMERIC(12,6), causing a silent upsert failure that corrupts the
workspace's cost accounting.
Fix: clamp input_tokens/output_tokens to [0, 10_000_000] before any
arithmetic or DB write. 10M tokens/call is well above any real LLM API
response; clamped values still produce valid cost rows.
Adds 4 regression tests:
- TestUpsertTokenUsage_615_CapsInt64Max — INT64_MAX → maxTokensPerCall
- TestUpsertTokenUsage_615_CapsNegative — negative → 0 (no DB call)
- TestUpsertTokenUsage_615_NormalValuesUnchanged — passthrough for normal counts
- TestUpsertTokenUsage_615_ExactlyAtCap — at-cap value accepted unchanged
Closes#615
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add an org-scoped allowlist table so org admins can restrict which plugins
workspace agents are allowed to install. An empty allowlist means
allow-all (backward-compatible with existing deployments).
• migrations/027_org_plugin_allowlist.{up,down}.sql — new table + unique
index on (org_id, plugin_name)
• handlers/org_plugin_allowlist.go — resolveOrgID, checkOrgPluginAllowlist
(fail-open on DB errors), GetAllowlist, PutAllowlist (atomic tx replace)
• handlers/org_plugin_allowlist_test.go — 23 unit tests covering all
handler paths, resolveOrgID, and all checkOrgPluginAllowlist branches
• handlers/plugins_install.go — allowlist gate between resolveAndStage and
deliverToContainer; returns 403 if plugin is blocked
• router/router.go — GET/PUT /orgs/:id/plugins/allowlist under AdminAuth
All tests pass; go build ./... clean; gosec Issues: 0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migration 026 adds workspace_token_usage table (uuid pk, workspace_id FK with
CASCADE, period_start TIMESTAMPTZ, input_tokens, output_tokens, call_count,
estimated_cost_usd NUMERIC(12,6), updated_at) with a UNIQUE index on
(workspace_id, period_start) for day-granularity upserts.
A2A proxy (proxyA2ARequest) now spawns a detached goroutine after each
successful call to extractAndUpsertTokenUsage, which:
1. Parses usage.input_tokens / usage.output_tokens from result.usage
(JSON-RPC wrapper) with fallback to top-level usage (direct Anthropic).
2. Calls upsertTokenUsage — INSERT ... ON CONFLICT DO UPDATE so multi-
call days accumulate correctly. Estimated cost = input×$0.000003 +
output×$0.000015 (Claude Sonnet default; adjustable in a later phase).
Token tracking never blocks the critical A2A path.
New endpoint: GET /workspaces/:id/metrics (wsAuth — WorkspaceAuth bearer
bound to :id). Returns:
{"input_tokens":N,"output_tokens":N,"total_calls":N,
"estimated_cost_usd":"0.000000","period_start":"...","period_end":"..."}
404 if workspace missing. Period is current UTC day.
11 new tests (4 handler + 7 parse-unit); 19/19 packages pass.
Closes#593
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add in-process SSE subscription mechanism to Broadcaster (SubscribeSSE,
deliverToSSE) so both RecordAndBroadcast *and* BroadcastOnly fan out to
SSE subscribers — critical because BroadcastOnly skips Redis pub/sub and
would be invisible to a Redis-only subscriber (AGENT_MESSAGE, A2A_RESPONSE,
TASK_UPDATED are all BroadcastOnly events).
- Add handlers/sse.go: SSEHandler.StreamEvents sets text/event-stream headers,
checks workspace existence (404 if missing), subscribes via broadcaster, and
wraps each WSMessage in an AG-UI envelope:
data: {"type":"<event>","timestamp":<unix_ms>,"data":{...}}\n\n
- Register wsAuth.GET("/workspaces/:id/events/stream") behind existing
WorkspaceAuth middleware — bearer token bound to :id.
- Add 6 tests: Content-Type, initial ping, AG-UI format, workspace filter
(cross-workspace events not leaked), 404 on missing workspace, multiple
sequential events.
All 19 packages pass. Build clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: the github-app-auth plugin injects GH_TOKEN + GITHUB_TOKEN
into each workspace container's env at provision time (EnvMutator). Those
are GitHub App installation tokens with a fixed ~60 min TTL. The plugin
has an in-process cache that proactively refreshes 5 min before expiry —
but the workspace env is set once at container start and never updated.
Any workspace alive >60 min ends up with an expired token.
Fix (Option B — on-demand endpoint):
pkg/provisionhook:
- Add TokenProvider interface: Token(ctx) (token, expiresAt, error)
Lives in pkg/ (public) so the github-app-auth plugin can implement it.
- Add Registry.FirstTokenProvider() — discovers the first mutator that
also satisfies TokenProvider via interface assertion. Safe under
concurrent reads (existing RWMutex).
platform/internal/handlers/github_token.go:
- New GitHubTokenHandler serving GET /admin/github-installation-token
- Delegates to the registered TokenProvider (plugin cache — always fresh)
- 404 if no GitHub App configured, 500 + [github] prefix log on error
- Never logs the token itself
platform/internal/handlers/workspace.go:
- Add TokenRegistry() getter so the router can wire the handler without
coupling to WorkspaceHandler internals
platform/internal/router/router.go:
- Register GET /admin/github-installation-token under AdminAuth
workspace-template/:
- scripts/molecule-git-token-helper.sh — git credential helper; calls
the platform endpoint on every push/fetch; falls through to next
helper (operator PAT) if platform unreachable
- entrypoint.sh — configure the credential helper at startup
Why Option B over Option A (background goroutine):
- The plugin already has its own cache refresh; nothing to refresh here.
- Pushing env updates into running containers requires docker exec, which
the architecture explicitly rejects (issue #547 "Alternatives").
- Pull-based is stateless, trivially testable, zero extra goroutines.
Closes#547
Co-authored-by: Molecule AI DevOps Engineer <devops-engineer@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
When a workspace delegated a task to itself, it would acquire
_run_lock twice on the same goroutine mutex, blocking permanently.
Add an early-return guard in `DelegationHandler.Delegate` that
returns HTTP 400 {"error": "self-delegation not permitted"} as soon
as sourceID == body.TargetID, before any DB or A2A work is done.
Adds TestDelegate_SelfDelegation_Rejected to delegation_test.go.
Closes#548
Co-authored-by: Molecule AI Backend Engineer <backend-engineer@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
`CreateWorkspacePayload` was missing a `Secrets` field, so any
`secrets: { KEY: value }` included in a POST /workspaces body was
silently dropped by ShouldBindJSON.
Changes:
- Add `Secrets map[string]string` field to `CreateWorkspacePayload`
- Wrap workspace INSERT in a DB transaction; iterate over secrets,
encrypt each value via `crypto.Encrypt`, and upsert into
`workspace_secrets` within the same tx — rollback both on any failure
- Add `mock.ExpectBegin()`/`mock.ExpectCommit()`/`mock.ExpectRollback()`
to all existing Create tests that were missing transaction expectations
- Add 3 new tests: WithSecrets_Persists, SecretPersistFails_RollsBack,
EmptySecrets_OK
Closes#545
Co-authored-by: Molecule AI Backend Engineer <backend-engineer@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
No env vars to configure. The platform auto-detects the backend:
MOLECULE_ORG_ID set → SaaS tenant → control plane provisioner
MOLECULE_ORG_ID empty → self-hosted → Docker provisioner
The control plane URL defaults to https://api.moleculesai.app (override
with CP_PROVISION_URL for testing). No FLY_API_TOKEN on the tenant.
Removed: direct Fly provisioner (FlyProvisioner) — all SaaS workspace
provisioning goes through the control plane which holds the Fly token
and manages billing, quotas, and cleanup.
Two backends: CPProvisioner (SaaS) and Docker Provisioner (self-hosted).
Closes#494
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PATCH /workspaces/:id field-level auth for parent_id/tier/runtime
required a bearer token, blocking canvas nesting (drag-to-nest).
Added IsSameOriginCanvas check so the tenant canvas can update
sensitive fields without a bearer.
Exported IsSameOriginCanvas from middleware package so workspace.go
can call it for the field-level auth path.
DELETE /workspaces/:id is behind AdminAuth which already has the
same-origin check — if delete still fails, it's a different issue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When CONTAINER_BACKEND=flyio, workspaces are provisioned as Fly Machines
instead of local Docker containers. This enables workspace deployment
on SaaS tenants where no Docker daemon is available.
New files:
- provisioner/fly_provisioner.go: FlyProvisioner with Start/Stop/
IsRunning/Restart/Close via Fly Machines API (api.machines.dev/v1)
- FlyRuntimeImages maps runtimes to GHCR image tags
Changes:
- main.go: select Docker vs Fly based on CONTAINER_BACKEND env var
- workspace.go: SetFlyProvisioner() setter, Create checks flyProv first
- workspace_provision.go: provisionWorkspaceFly() loads secrets, calls
FlyProvisioner.Start, issues auth token for the new machine
Env vars for Fly backend:
- CONTAINER_BACKEND=flyio (activates Fly provisioner)
- FLY_API_TOKEN (Fly deploy token)
- FLY_WORKSPACE_APP (Fly app name for workspace machines)
- FLY_REGION (default: ord)
Closes#494
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Settings panel: wire TokensTab into "API Tokens" tab (was imported
but not rendered). Rename "API Keys" → "Secrets", add "API Tokens"
tab. Fix docs link → doc.moleculesai.app/docs/tokens.
2. Referer match hardening: require exact host match or trailing slash
to prevent evil.com subdomain bypass. Cache CANVAS_PROXY_URL at
init time instead of per-request os.Getenv.
3. Extract shared deriveWsBaseUrl() to lib/ws-url.ts — eliminates
duplicate 12-line derivation in socket.ts and TerminalTab.tsx.
4. Token list pagination: add ?limit= and ?offset= params (default
50, max 200) to GET /workspaces/:id/tokens.
507/507 canvas tests pass, Go build + vet clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two adjacent fixes that surfaced trying to bring the molecule-dev org
template back up against the new standalone workspace-template-* repos.
1) handlers/org.go — expand ${VAR} in workspace_dir before validation.
The molecule-dev pm/workspace.yaml (and any operator's per-host
binding) ships `workspace_dir: ${WORKSPACE_DIR}` so each operator
can pick the host path PM bind-mounts. Without expansion the literal
"${WORKSPACE_DIR}" string reaches validateWorkspaceDir and fails with
"must be an absolute path", aborting the whole org import.
Other fields (channel config, prompts) already go through expandWithEnv;
workspace_dir was the last hold-out.
2) provisioner/provisioner.go — inject PYTHONPATH=/app for every
workspace container. Standalone template Dockerfiles COPY adapter.py
to /app and set ENV ADAPTER_MODULE=adapter, but molecule-runtime is
a pip console_script entry point so cwd isn't on sys.path
automatically. Setting PYTHONPATH here fixes every adapter image at
once instead of needing 8 PRs against template repos. Operator
override still wins (workspace EnvVars are appended after, so Docker
takes the later duplicate).
Note: this unblocks the import path but does NOT make claude-code /
hermes / etc. boot. The runtime itself has a separate top-level
`from adapters import` that breaks against modular templates —
tracked at workspace-runtime#1.
Tests: TestBuildContainerEnv_InjectsPYTHONPATH +
TestBuildContainerEnv_WorkspaceEnvVarsCanOverridePYTHONPATH lock the
default + operator-override invariants. expandWithEnv is already covered
by TestExpandWithEnv_* — the workspace_dir use site is a one-line call
to that primitive.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `provisionhook.EnvMutator` extension point so out-of-tree plugins
(e.g. github-app-auth, vault-secrets) can inject or override env vars
right before container Start, without forking core or piling more
provider-specific code into the handlers package.
WorkspaceHandler gains an optional `envMutators *provisionhook.Registry`
wired in via SetEnvMutators during boot. The hook fires after built-in
secret loads + per-agent git identity, so plugins can both read what's
already there and override anything they own (GIT_AUTHOR_*, GITHUB_TOKEN).
A nil registry is a no-op via Registry.Run's nil-receiver branch — keeps
the hot path a single nil compare and means existing flows stay green
even with zero plugins registered.
Mutator failure aborts provisioning and marks the workspace failed with
the wrapped error in last_sample_error. Failing fast surfaces the cause
to the operator instead of letting an agent boot into opaque "git push
401" loops it can never recover from on its own.
Tests cover ordered execution, chained env visibility, first-error abort,
nil-receiver no-op, nil-mutator drop, registration order, and concurrent
register-vs-run safety (-race clean).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#460, #461.
**#460 — YAML injection via unquoted skill/prompt filenames**
`generateDefaultConfig` extracted skill directory names and prompt file
names from user-supplied `body.Files` keys and wrote them directly into
YAML list items without quoting:
cfg.WriteString(" - " + s + "\n")
`validateRelPath` only blocks path traversal (`../`); it does NOT block
YAML control characters including newlines. On Linux, filenames can
contain newlines, so an attacker with any live workspace bearer token
could submit:
{"files": {"skills/legit\nruntime: malicious/SKILL.md": "# skill"}}
The generated config.yaml would then contain `runtime: malicious` as a
top-level YAML key, overriding the runtime for workspaces provisioned
from the template.
Fix: extract `yamlEscape` as a reusable local from the same
`strings.NewReplacer` already used for the `name` field (#221) and apply
it to both the `skills:` and `prompt_files:` list items, wrapping each
in double-quotes.
**#461 — Docker error details in ReplaceFiles 500 responses**
`ReplaceFiles` returned `fmt.Sprintf("failed to write files: %v", err)`
in two 500 paths, where `err` comes from Docker API calls and may include
internal container names, volume names, and daemon error messages.
Fix: log the full error server-side and return a static opaque string to
the caller.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Container rebuild or volume wipe caused workspaces to lose /configs/.auth_token.
On re-registration the platform returned no auth_token (HasAnyLiveToken==true →
no re-issue), leaving the workspace unable to authenticate any subsequent API call.
Fix: provisionWorkspaceOpts now calls issueAndInjectToken before Start(). This
revokes any existing live tokens (plaintext is irrecoverable from the stored hash,
so rotation is the only safe path) and issues a fresh token that is written into
cfg.ConfigFiles[".auth_token"]. WriteFilesToContainer delivers it to /configs
immediately after ContainerStart, racing safely ahead of the Python adapter's
1-2s startup time.
Failure modes are soft: revoke or issue errors skip injection with a warning;
provisioning continues and the workspace recovers on the next restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add an optional channel_budget (INTEGER, nullable) to workspace_channels
via migration 024. When channel_budget IS NOT NULL and message_count has
reached the budget, the Send handler returns 429 {"error":"channel budget
exceeded"} and aborts before calling SendOutbound.
Implementation details:
- Single SELECT query reads both message_count and channel_budget in one
round-trip (avoids TOCTOU window between read and write)
- Fail-open on DB error: transient failures log but don't block sends
- Early-return on budget hit is before SendOutbound so message_count
cannot be incremented past the limit by a concurrent send that slips
through the window (best-effort; atomic enforcement requires DB-level CAS)
- NULL channel_budget = unlimited (default, backward-compatible)
Migration is idempotent (ADD COLUMN IF NOT EXISTS). Down migration drops
the column cleanly.
Four sqlmock tests cover: at-limit → 429, above-limit → 429, NULL budget
passes through, under-limit passes through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce `memoryRecallMaxLimit = 50` constant and honour the `?limit=N`
query parameter in Search. Values above 50 are silently clamped to 50;
absent or invalid values default to 50. The LIMIT clause is now a
parameterised argument (nextArg pattern) instead of a hardcoded literal.
Three sqlmock tests verify the cap, the explicit limit, and the default.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Register handler was serialising the raw Go error into the HTTP response:
c.JSON(500, gin.H{"error": fmt.Sprintf("failed to register: %v", err)})
PostgreSQL errors wrapped by lib/pq contain table names, constraint names, and
driver-version strings — enough for a caller to fingerprint the schema and craft
targeted attacks. The error is already logged at full detail with Printf before
this line, so callers only need the generic message.
Fix: replace the Sprintf with a static "registration failed" string (same pattern
the heartbeat and update-card handlers already used).
New test: TestRegister_DBErrorResponseIsOpaque verifies the response body is the
opaque string and that "sql:", "pq:", and "connection" substrings are absent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a workspace container AND its /configs Docker volume are both destroyed,
the restart handler previously had no recovery path — findTemplateByName searched
only the top-level configsDir, which holds workspace-instance dirs (ws-{id[:12]}/),
not the role-named org-template source directories.
Fix: add `rebuild_config: true` to the POST /workspaces/:id/restart body struct.
When set, the handler falls back to searching configsDir/org-templates/ via the
existing findTemplateByName logic (which already handles name normalisation and
config.yaml name-field matching). The workspace can then self-recover with its own
bearer token — no admin intervention required.
New helper: resolveOrgTemplate(configsDir, wsName) — pure function, independently
tested (4 cases: hit-by-dir, hit-by-config-yaml, no org-templates dir, no match).
Usage:
curl -X POST -H "Authorization: Bearer $(cat /configs/.auth_token)" \
-d '{"rebuild_config": true}' \
http://platform:8080/workspaces/$WORKSPACE_ID/restart
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Part 4 of 4 — terminal step of the org.yaml scalability refactor. Each
role in the molecule-dev template now owns its own workspace.yaml file,
colocated with the existing system-prompt.md / initial-prompt.md /
idle-prompt.md / schedules/*.md. Team files shrink to a leader's own
definition plus a list of !include refs.
## Platform change
`resolveYAMLIncludes` now uses a TWO-ROOT model:
- Path resolution is relative to the INCLUDING file's directory
(natural sibling + cousin refs, C-include / Sass @import convention).
- Security bound is the ORIGINAL org root (`rootDir`), preserved across
all recursion depths. Sibling-dir refs like `../my-role/workspace.yaml`
from a team file are now allowed (they stay inside the org template);
refs that escape the root still error.
Regression coverage: new `TestResolveYAMLIncludes_SiblingDirAccess`
reproduces the Phase 4 pattern (team file at `teams/x.yaml` referencing
`../<role>/workspace.yaml`) — fails without the fix, passes with.
## Template change
Atomized 15 child workspaces across 3 team files:
- `teams/research.yaml`: 58 → 30 lines; 3 children now !include refs
- `teams/dev.yaml`: 222 → 38 lines; 6 children now !include refs
- `teams/marketing.yaml`: 143 → 28 lines; 6 children now !include refs
Each role now has `<role>/workspace.yaml` colocated with its prompts.
Example `frontend-engineer/` directory:
frontend-engineer/
├── workspace.yaml (24 lines — name/role/tier/canvas/plugins/...)
├── system-prompt.md (from earlier phases)
├── initial-prompt.md
├── idle-prompt.md
└── (no schedules for this role — but if added, schedules/<slug>.md)
## File-size progression across all 4 phases
| State | org.yaml | total `.yaml` in tree |
|---|---:|---:|
| Before (main) | 1801 lines / 108 KB | 1801 / 108 KB (one file) |
| After Phase 1 (#389) | 1687 | 1687 / 101 KB |
| After Phase 2 (#390) | 676 | 676 / 35 KB |
| After Phase 3 (#393) | 114 | 683 (1 + 6 teams) / 33 KB |
| **After this PR** | **114** | **~698** (1 + 6 + 15 workspace) / 35 KB |
Aggregate size is flat — the decrease came from prompt externalization
in Phases 1/2; Phases 3/4 reorganize structure without adding content.
The win is readability and ownership:
- Every individual file fits on 1-2 screens.
- Adding a new role is now: create `<role>/` dir, add `workspace.yaml`
+ `system-prompt.md` + prompts, add ONE `!include` line to the team
file. No touching of aggregated mega-YAML.
- Team files can be reviewed + merged independently.
## Tests
All 10 `TestResolveYAMLIncludes_*` tests pass, including the real-template
integration test (`TestResolveYAMLIncludes_RealMoleculeDev`) which now
walks org.yaml → teams/pm.yaml → teams/research.yaml → ../market-analyst/
workspace.yaml and validates the full 21-role tree unmarshals cleanly.
Plus all existing `TestResolvePromptRef` + `TestOrgYAML` + `TestInitialPrompt`
suites stay green.
## Ops followup
After merging all 4 phases and deploying, the `POST /org/import`
endpoint should produce a workspace tree byte-identical to the
pre-refactor state. Verify with:
diff <(curl POST /org/import before) <(curl POST /org/import after)
or by spot-checking:
- `/configs/config.yaml` bodies across all 21 workspaces
- `workspace_schedules.prompt` row values
The externalization is lossless — YAML literal to file and back
recovers the same string modulo trailing-whitespace normalization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes the silent-overwrite hole where two agents racing a read-modify-
write on the same memory key left only one agent's update. Relevant for
orchestrators (PM, Dev Lead, Marketing Lead) keeping structured running
state (delegation-result ledgers, task queues) in memory, and for the
``research-backlog:*`` keys that multiple idle loops write in parallel.
## Semantics
### Back-compat path (no if_match_version)
Unchanged: ``INSERT ... ON CONFLICT UPDATE`` last-write-wins. Every
existing agent tool, every existing ``commit_memory`` call, every
existing cron that writes memory — all continue to work with no edit.
### Optimistic-lock path (if_match_version set)
1. Client calls ``GET /memory/:key`` → ``{value, version: V}``
2. Client modifies value locally
3. Client ``POST /memory {key, value, if_match_version: V}``
4. Server: ``UPDATE ... WHERE version = V`` + RETURNING new version
5. On match → 200 + ``{version: V+1}``
6. On mismatch → 409 + ``{expected_version: V, current_version: <actual>}``
7. Client reads the actual version and retries.
### Create-only marker
``if_match_version: 0`` means "create iff the key doesn't exist yet".
Two agents simultaneously seeding a shared key will see exactly one
success + one 409 — no silent collision, no duplicate-init work.
### Schema
Migration 023 adds ``version BIGINT NOT NULL DEFAULT 1``. Existing rows
baseline at 1. New rows start at 1. Every successful write (both paths)
increments: ``version = version + 1`` on update, ``1`` on insert.
## Why version, not updated_at
``updated_at`` has second-granularity and can collide between concurrent
writers on a fast clock. A monotonic counter is collision-free and more
readable in the 409 response body ("expected 5, current is 7 — you
missed 2 writes" tells an agent exactly what to re-read).
## Why ``if_match_version`` and not an ETag header
JSON field keeps it in the request body, visible alongside the value
payload. Agents assembling requests programmatically don't have to
remember to thread a header through their HTTP client wrapper; the
existing ``commit_memory`` tool can grow one optional kwarg and match
the existing signature shape.
## Tests
11 memory-handler cases covering every path:
- GET list / get (with version in response shape)
- Set with no version (back-compat upsert, returns new version)
- Set with if_match_version match (happy path, increment)
- Set with if_match_version mismatch (409 + expected/current fields)
- Set with if_match_version=0 on absent key (create-only success)
- Set with if_match_version=N on absent key (409 — caller's mental
model is wrong)
- Bad inputs (missing key, malformed JSON)
- Delete happy + error path
Full ``go test ./internal/handlers/`` green.
## Follow-up (not in this PR)
- Workspace-template tool update: ``commit_memory(content, *,
if_match_version=None)`` surfaces the new option + on 409 surfaces
the current_version so agents can retry without manual re-read.
- Named checkpoints table (``workspace_checkpoints``) for durable
orchestrator state snapshots. Different concern than per-key locking;
separate PR.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of the 2026-04-16 09:10 UTC six-container restart cascade.
## Timeline
09:10:26 — PM sent a batch delegation to 15+ agents (Dev Lead coordinating).
09:10:26-27 — 4 leaders/auditors (Security, RL, BE, DevOps) simultaneously
hit "workspace agent unreachable — container restart triggered"
even though their containers were running fine. Another 2
(DL, UIUX) tripped in the next few seconds.
09:10:27 — Provisioner stopped + recreated 6 containers in parallel. A2A
callers got EOFs, PM's batch coordination stalled.
## Root cause
`provisioner.IsRunning` collapsed every ContainerInspect error into
`(false, nil)`, including transient Docker daemon hiccups:
func IsRunning(...) (bool, error) {
info, err := p.cli.ContainerInspect(ctx, name)
if err != nil {
return false, nil // Container doesn't exist ← MISREAD
}
return info.State.Running, nil
}
The comment said "Container doesn't exist" but the error was actually
any of: daemon timeout, socket EOF, context deadline, connection
refused. Under load (batch delegation fan-out → 15 concurrent HTTP
inbound → 15 concurrent Claude Code subprocesses → Docker daemon CPU
pressure), ContainerInspect calls started failing transiently. All 6
calls returned `(false, nil)`. Caller `maybeMarkContainerDead` treated
`running=false` as "container is dead, restart it" → six parallel
restarts. This was exactly the destructive-on-error pattern we keep
trying to kill (see #160 SDK-stderr-probe, #318 fail-open classes).
## Fix
`IsRunning` now distinguishes NotFound from transient errors:
- Legitimately missing container (caller deleted, Docker pruned) →
`(false, nil)` — safe to act on; caller marks dead + restarts.
- Any other error (daemon timeout, socket issue, context deadline) →
`(true, err)` — caller stays on the alive path. The transient error
is preserved so metrics + logging still see it, but it does NOT
trigger the destructive restart branch.
`isContainerNotFound` matches on error-message substring — same
approach docker/cli uses internally — to avoid pulling in errdefs as a
direct dep. Truth table tests in `isrunning_test.go` cover 8 cases:
NotFound variants (real + generic), nil, empty, and the 4 transient-
error shapes we've actually observed (deadline, EOF, connection-refused,
i/o timeout).
## Caller update
`maybeMarkContainerDead` in a2a_proxy.go now logs the transient inspect
error (was silently discarded via `_`). Visibility without
destructiveness. If this error becomes persistent, we'll see it in
platform logs rather than diagnosing after another restart cascade.
## Expected impact
- Zero restart cascades from the current class of transient inspect
errors (EOF, timeout, connection refused).
- Dead containers still detected within the A2A layer because an actual
stopped container returns NotFound on inspect, and the TTL monitor
(180s post #386) catches anything that slips through.
- New visibility in platform logs when inspect has trouble — previously
silent.
Combined with the TTL fix in #386, the defense-in-depth on spurious
restart is now:
1. IsRunning only returns false for real NotFound
2. Liveness TTL is 180s, surviving 5+ missed heartbeats
3. A2A proxy 503-Busy path retries with backoff before touching
restart logic at all
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Part 3 of 4 in the scalability refactor. Adds YAML `!include` support
to the org importer and splits molecule-dev/org.yaml (676 lines post-
Phase 2) into 6 team / role files; top-level org.yaml drops to 114 lines
of pure scaffolding.
## Platform changes
New `platform/internal/handlers/org_include.go`:
- `resolveYAMLIncludes(data, baseDir)` — pre-processes a YAML document,
expanding any scalar tagged `!include <path>` with the parsed content
of the referenced file.
- Path resolution via `resolveInsideRoot` so a crafted `!include
../../etc/passwd` can't escape the org template directory (same
defense the existing `files_dir` copy uses).
- Nested includes supported: each included file carries its own search
root (its directory), so `teams/pm.yaml` with `!include research.yaml`
resolves to `teams/research.yaml` — matching the convention of
C-include / Sass @import / most package systems.
- Cycle detection via visited-set keyed on absolute path; belt-and-
braces `maxIncludeDepth = 16` cap in case symlinks or path
normalization defeats the set.
- Inline-template mode (POST /org/import with raw JSON body, no `dir`)
errors cleanly when a file ref is used — can't resolve without a
base.
Wired into both `ListTemplates` (so /org/templates shows an accurate
workspace count after the split) and `Import` (expansion happens before
unmarshal into OrgTemplate).
## Template changes
molecule-dev/org.yaml now contains only:
- name + description
- defaults (runtime, plugins, category_routing, initial_prompt text)
- `workspaces: [!include teams/pm.yaml, !include teams/marketing.yaml]`
New files:
- `teams/pm.yaml` — PM top-level, children are !include refs
- `teams/research.yaml` — Research Lead + Market Analyst + Technical
Researcher + Competitive Intelligence (inline children)
- `teams/dev.yaml` — Dev Lead + FE/BE/DevOps/Security/QA/UIUX (inline)
- `teams/marketing.yaml` — Marketing Lead + DevRel/PMM/Content/
Community/SEO/Social (inline)
- `teams/documentation-specialist.yaml` — leaf
- `teams/triage-operator.yaml` — leaf
## File-size impact
| State | org.yaml lines | total config size |
|---|---:|---:|
| Before (main) | 1801 | 108 KB |
| After Phase 1 (#389) | 1687 | 101 KB |
| After Phase 2 (#390) | 676 | 35 KB |
| After this PR | **114** | **4 KB** (org.yaml only) |
With the 6 team files (total ~570 lines of structural yaml), every file
is now under 230 lines and individually readable without scrolling past
a single team's boundaries.
## Tests
`platform/internal/handlers/org_include_test.go` — 9 cases:
- Flat include (single file, single workspace)
- Nested include (file → file → file)
- Traversal rejection (`../secret.yaml`, `../../secret.yaml`)
- Cycle detection (a↔b)
- Empty path error
- Missing file error
- Inline-template error (baseDir empty)
- No-op when YAML has no includes (safety: we always run the preprocessor)
- **Integration**: load the real `org-templates/molecule-dev/org.yaml`,
resolve includes, unmarshal into OrgTemplate, verify PM + Marketing
Lead are top-level and PM has ≥4 children after expansion.
All 9 pass + existing `TestResolvePromptRef` + `TestOrgYAML` suites stay
green.
## Ownership implication
Each team file can now be owned + reviewed independently. When the
marketing team adds a 7th role, the diff is in `teams/marketing.yaml`
alone — no merge conflicts against PM or research changes in the same
review window. Same for the eventual engineer team, security team, etc.
## What's next
- **Phase 4 (queued):** per-workspace atomization. Each role gets
`<role>/workspace.yaml`; team files shrink to a list of !include
refs. Terminal step in the scalability arc — at that point adding a
new role is one new file under `org-templates/molecule-dev/<role>/`
plus one line in the team's manifest.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rebased cleanly onto current main (resolves the add/add conflicts that
blocked CI on PR #374 — the original branch diverged from a pre-repo-bootstrap
commit that predated most files).
Changes:
- schedules.go: add scheduleHealthResponse struct + Health handler
(mirrors A2A proxy auth pattern: X-Workspace-ID + CanCommunicate gate)
- router.go: register GET /workspaces/:id/schedules/health on r (not wsAuth)
so peer agents can query without holding the target workspace's bearer token
- schedules_test.go: 7 new tests (missing caller 401, self-call OK, legacy
peer grandfathered, non-peer 403, system caller bypass, no prompt exposure,
DB error 500)
isSystemCaller/validateCallerToken reused from a2a_proxy.go (same package).
registry.CanCommunicate import added to schedules.go.
Closes#249
Supersedes PR #374 (which could not get CI due to merge conflict)
Co-authored-by: PM (Molecule AI) <pm@molecule-ai.internal>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Every workspace now commits under its own name. Step 3 of the three-
step agent-separation plan (platform-level git identity today;
GitHub App migration follows as Option 1).
## Problem
All 20+ agents in the molecule-dev template (PM, Dev Lead, Research
Lead, FE, BE, DevOps, Security, QA, UIUX, Marketing roles, etc.) share
a single GITHUB_TOKEN — specifically the CEO's personal PAT. So every
commit, PR, and issue across the live repos ends up attributed to
HongmingWang-Rabbit. `git log` can't distinguish "which agent wrote
this code" from "did the CEO write it"; neither can the authority-
verification rule in triage-operator/philosophy.md (rule #3).
## Fix
When the provisioner starts a workspace container, it now sets:
GIT_AUTHOR_NAME = "Molecule AI <Workspace Name>"
GIT_AUTHOR_EMAIL = <slug>@agents.moleculesai.app
GIT_COMMITTER_NAME = (same)
GIT_COMMITTER_EMAIL = (same)
Git prefers these env vars over `git config user.name` / `user.email`,
so no per-container git-config step is needed; every commit automatically
carries the right authorship.
Examples (20 agents, 20 distinct identities):
Frontend Engineer → frontend-engineer@agents.moleculesai.app
Backend Engineer → backend-engineer@agents.moleculesai.app
Product Marketing Manager → product-marketing-manager@agents.moleculesai.app
UIUX Designer → uiux-designer@agents.moleculesai.app
Domain `agents.moleculesai.app` is deliberate: marks the email as a
bot address without resembling a real inbox.
## Operator override preserved
`applyAgentGitIdentity` runs AFTER the secret-load loops in
`provisionWorkspaceOpts`, but uses `setIfEmpty` so any workspace_secret
with the same key wins. Teams that want custom authorship (shared org
signing identity, a person-on-the-loop owner) can still set
`GIT_AUTHOR_NAME` via /workspaces/:id/secrets and get their value
through to git.
## What this does NOT solve (yet)
- PR / issue authorship is still whoever owns GITHUB_TOKEN (the shared
PAT). That needs the GitHub App migration (Option 1, next PR). The
commit-level split shipped here is the prerequisite: the App path
will keep these env vars and just swap the PAT for a short-lived
installation token.
- Existing containers continue with their pre-fix env (git env vars
are baked in at container-create time). Applying is one plain
`POST /workspaces/:id/restart` per agent after this merges +
deploys — the restart goes through provisionWorkspace which picks
up the new injection.
## Tests
`agent_git_identity_test.go` — 4 behavior tests + a 10-row slug test:
- fills all 4 env vars from a workspace name
- operator override via pre-set env is preserved (setIfEmpty semantics)
- empty / whitespace workspace name is a no-op (no `unknown@...` emails)
- nil map doesn't panic (defensive)
- slugify handles spaces / punctuation / edge hyphens / em-dashes
All 15 cases pass; platform build clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
