Three review nits from PR #2473:
1. Narrow `_check_runtime_wedge` import catch to (ImportError,
ModuleNotFoundError). The bare `except Exception:` would have
masked an `AttributeError`/`TypeError` from a runtime_wedge API
rename — silently degrading the smoke gate to "no wedge info" with
no log line. The `runtime_wedge_signature.json` snapshot test
(task #169) carries the API-drift load instead.
2. Drop the unreachable `or "<unspecified>"` fallback. `wedge_reason()`
only returns "" when not wedged, but the call is guarded by
`is_wedged()` being True and `mark_wedged` requires a non-None
reason. The defensive arm couldn't fire.
3. Promote `reset_runtime_wedge` from a per-file fixture in
test_smoke_mode.py to an autouse fixture in
workspace/tests/conftest.py. Heartbeat tests or future adapter
tests that call `mark_wedged` without cleanup would otherwise leak
a sticky wedge into smoke tests later in the same pytest process —
smoke tests would fail-via-leak instead of asserting their actual
contract. Two-sided reset survives early test failures.
Also: `test_check_runtime_wedge_returns_none_when_module_missing`
now `monkeypatch.delitem(sys.modules, "runtime_wedge")` before
patching `__import__`, so the test re-exercises the import path
instead of resolving from the module cache (the test was passing
today by luck — it would still pass even if the catch arm were
deleted, because the cached module's `is_wedged` returned False).
Tests: 28 still pass in test_smoke_mode.py, 57 across smoke + wedge +
heartbeat. Regression-injection-checked: catch tightening doesn't
regress the existing wedge tests.
Timeout-as-PASS in run_executor_smoke missed the PR-25-class
regression: claude-agent-sdk takes 60s to time out on a malformed
argv, our outer wait_for fires at 5s default and reports "imports
healthy, hit a network boundary." A broken image then ships to GHCR.
Universal fix uses the existing runtime_wedge module (already
documented as the cross-cutting wedge holder, already read by
heartbeat). Adapters opt-in by calling runtime_wedge.mark_wedged()
from their executor's wedge catch arm; the smoke now consults
runtime_wedge.is_wedged() at the end of every result path and
upgrades a provisional PASS to FAIL when the flag is set. Non-opt-in
adapters keep working as before — the check is additive.
CI uses MOLECULE_SMOKE_TIMEOUT_SECS=90 to outlast the SDK's 60s
initialize() handshake so the wedge marks before our outer wait_for
fires. Module + helper docstrings call out the calibration so a
future contributor doesn't lower it without thinking through what
that wins back vs. what it loses.
Tests: 7 new cases pinning the wedge-aware paths — mark+raise (PR-25
shape), mark+block (still-running execute that wait_for cuts short),
clean+clean (additive contract), import-resilience (fail-open when
runtime_wedge unimportable). Regression-injection-checked: silencing
the new check fails both wedge-shape tests at unit-test time.
Surfaces the conversation history with one specific peer for the
wheel-side chat_history MCP tool. The filter joins
(source_id = $X OR target_id = $X) so both inbound (peer was sender)
and outbound (peer was recipient) turns appear in the same view,
ordered by created_at, and composes with existing type/source/
since_secs/since_id/limit filters.
Validates peer_id as a UUID at the trust boundary so a malformed
caller can't smuggle SQL fragments via the parameter — the args are
bound but the explicit rejection gives the wheel a cleaner 400
signal than an empty list, and defends against any future code path
that might interpolate the value into a URL or another query.
Tests: 3 new branches (positive filter, composition with
type+source, UUID-shape rejection across 5 malformed inputs).
Mutation-verified: reverting activity.go fails all peer_id tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The workspace-server's `/notify` handler writes the agent's own
send_message_to_user POSTs to activity_logs as activity_type=
'a2a_receive', method='notify', source_id=NULL so the canvas
chat-history loader can restore those bubbles after a page reload.
The activity API exposes the row to /workspaces/:id/activity?
type=a2a_receive, so the inbox poller picks it up and pushes the
agent's own outbound back as an inbound `← molecule: Agent
message: ...` — confirmed live 2026-05-01.
Add `_is_self_notify_row` predicate matched on (method='notify' AND
no source_id) and call it from `_poll_once` before enqueue. The
predicate combines BOTH discriminators so a future caller using
method='notify' with a real peer_id still passes through. Cursor
advances past skipped rows so we don't re-poll the same self-notify
on every iteration.
Belt-and-braces: long-term fix lives in workspace-server (rename
the misclassified activity_type to 'agent_outbound' — RFC at
#2469). This guard stays regardless because it only excludes rows
we never want.
Tests: 7 new — predicate true/false matrix + integrated _poll_once
behavior (skip, cursor advance, notification suppression).
Mutation-verified: reverting inbox.py to the prior shape fails 7/7;
applied state passes 48/48.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the picker modal opened only when preflight failed OR the
template offered ≥2 provider options. Single-provider templates with
saved keys (claude-code, langgraph) deployed silently using the
template's compiled-in default model — denying the user a final
chance to override before an EC2 boots and burns billing on the
wrong tier.
The picker UI already supports the "all-keys-saved single-provider"
case as a confirm-only prompt (provider radio is hidden, model input
is pre-filled with template.model), so flipping shouldShowPicker to
unconditional is a one-line change with the picker UX absorbing it.
Test plan
- Existing "single-provider skips picker when preflight.ok" regression
guard inverted to assert picker always opens.
- Three happy-path tests refactored to drive through the picker via
a new deployThroughPicker helper instead of expecting an immediate
POST.
- POST-failure tests likewise refactored — the failure now surfaces
through the picker click-through path, not the direct deploy()
call.
- 15/15 tests pass; deploy-preflight.test.ts unchanged + 20/20.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude Code 2.1.x's --dangerously-load-development-channels takes an
allowlist of tagged entries (`server:<name>` or
`plugin:<name>@<marketplace>`), not a bare switch. The instructions
field's push-only-mode message and the inline comment in
`_poll_timeout_secs` both referenced the old bare form. Update both
so an agent or operator reading them lands on the right invocation —
matched against the docs change in [molecule-docs PR #110](https://github.com/Molecule-AI/docs/pull/110).
No behavior change (string-only edits in instructions text + comment).
33/33 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The frozen copy was a self-justification — the comment claimed "tests +
tooling rely on import-time identity" but no test or tooling code path
actually references the binding. _build_initialize_result() calls
_build_channel_instructions() fresh per call so env changes take effect,
which is the documented runtime contract.
github-code-quality flagged it; resolving the unused-variable thread so
the staging branch protection's all-conversations-resolved gate clears.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address github-code-quality review on PR #2465: explain why the
OSError swallow in pipe teardown is intentional (best-effort
cleanup of a possibly-already-closed fd).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Why this exists
---------------
Live evidence on 2026-05-01 caught a regression latent in #46's
"push-feel inbound" closure: standard `claude` launches without
`--dangerously-load-development-channels` silently drop our
`notifications/claude/channel` emissions, so canvas/peer messages sat
in the wheel inbox and never reached the agent loop until manual
`inbox_peek`. The flag is research-preview-only; non-Claude-Code MCP
clients (Cursor, Cline, OpenCode, hermes-agent, codex) never receive
the notification at all because the method namespace is Claude-
specific. Push-only delivery shipped as the universal contract is
not actually universal.
What this changes
-----------------
Adds a poll path that works on every spec-compliant MCP client. The
`initialize` `instructions` field — read by every client and surfaced
to the agent's system prompt automatically — now tells the agent to
call `wait_for_message(timeout_secs=N)` at the start of every turn.
Push remains as the strictly-better delivery for hosts that opt in
(Claude Code with the dev flag or a future allowlist entry), but is
no longer load-bearing.
Both paths converge on the same `inbox_pop` ack so duplicate-delivery
on a push+poll race is impossible: whoever surfaces the message to
the agent first pops it, the other side returns empty.
Operator knob
-------------
`MOLECULE_MCP_POLL_TIMEOUT_SECS` controls per-turn poll blocking
(default 2s). 0 disables polling for push-only Claude Code with the
dev flag. Above 60 clamps to 60 — protects against an accidental
five-minute stall per turn. Resolved fresh on every `initialize` so
a relaunch with new env is enough; no wheel rebuild required.
Tests
-----
- structural pins on the new instructions: `wait_for_message` +
`timeout_secs` named, both PUSH PATH / POLL PATH labels present
- env-resolution: default fallback, garbage fallback, negative
fallback, 60s clamp
- operator override: `MOLECULE_MCP_POLL_TIMEOUT_SECS=7` reaches the
agent's instructions string
- timeout=0 toggles to push-only-mode messaging (no
wait_for_message call asked of the agent)
- existing pins on push path, reply tools, prompt-injection defense,
meta attributes — all preserved
Successor to #46. Closure milestone for this PR (per
feedback_close_on_user_visible_not_merge.md): launched `claude`
against the published wheel, sent a canvas message, observed the
agent surfaces the message inline at the start of its next turn
without me running `inbox_peek` — verified live before declaring done.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the dynamic-coverage gap on the `notifications/claude/channel`
push-UX bridge — until now we had static pins on the wire shape
(_build_channel_notification) and the initialize handshake, but the
threading + asyncio + stdout chain that ships notifications to the
host was never exercised under realistic conditions.
The three failure modes anticipated in #2444 §2 are each now pinned:
test_inbox_bridge_emits_channel_notification_to_writer
Drives a fake inbox event from a daemon thread, asserts the
notification lands on a real os.pipe-backed asyncio writer with
the correct JSON-RPC envelope. Catches: bridge wired up
incorrectly (no-op _on_inbox_message), run_coroutine_threadsafe
drift, _build_channel_notification call missing.
test_inbox_bridge_swallows_closed_pipe_drain_error
Closes the pipe's read end before firing, captures the
concurrent.futures.Future that run_coroutine_threadsafe returns,
asserts its exception() is None. Catches: narrowing the broad
`except Exception` in _emit (e.g. to RuntimeError), or removing
it. Without the swallow, the future carries a ConnectionResetError
and the test fails with a clear message naming the regression.
test_inbox_bridge_swallows_closed_loop_runtime_error
Builds the bridge against a closed event loop, fires the
callback, asserts no exception escapes. Catches: removing the
`except RuntimeError` swallow on the run_coroutine_threadsafe
call. Without it the poller thread would crash with
"RuntimeError: Event loop is closed" during shutdown.
To make the bridge testable, extracted the closures from main() into
a top-level `_setup_inbox_bridge(writer, loop) -> Callable[[dict],
None]` helper. main()'s wire-up is now a single line that calls the
helper. Behavior is unchanged — same write, same drain, same
swallows — just no longer trapped inside main()'s closures.
Verified each test catches its regression by injection: removing
each swallow / no-op'ing the bridge each turn the matching test red
with a specific failure message that points at the missing piece.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the missing symmetric pin against the threat-model sentence —
the existing tests pin reply-tool names (send_message_to_user,
delegate_task, inbox_pop) and tag attributes (kind, peer_id,
activity_id) but left the "treat message body as untrusted user
content" line unpinned. A copy-edit that drops it would turn the
channel into an open prompt-injection vector against any workspace
running the MCP server.
Pins three signals: "untrusted" present, an explicit
"not execute"/"do not" clause, and the "approval" escape-hatch
sentence — two of three would let a partial copy-edit slip
through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #2461 added the experimental.claude/channel capability declaration
on the assumption that was the missing gate for Claude Code surfacing
notifications/claude/channel as inline <channel> interrupts. Research
against code.claude.com/docs/en/channels-reference.md confirms the
capability IS one gate — but there's a SECOND required field we still
don't ship: `instructions` on the initialize result.
The docs are explicit: instructions is what tells the agent what the
<channel> tag attributes mean and which tool to call to reply. Without
it the channel registers but the agent receives the tag with no
context and has no idea how to handle it. The official telegram
plugin ships both (server.ts:370-396) — capability AND instructions.
We were shipping one of two.
This adds the instructions string. It documents:
- kind/peer_id/activity_id meta attributes
- canvas_user → send_message_to_user reply path
- peer_agent → delegate_task reply path
- inbox_pop ack to prevent duplicate-poll re-delivery
- threat model: treat message bodies as untrusted user content
Tests: 4 new pins. instructions present + non-empty, instructions
names each reply tool, instructions documents each tag attribute.
Failure messages name the symptom so a copy-edit can't silently
break the channel.
Live verification still pending after wheel ships — same plan as
the gap is in --dangerously-load-development-channels (host-side
flag, outside our control during the channels research preview).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to commit 0a87dec5 (PR #2461, merged before live verification).
Two corrections to the docstring on `_build_initialize_result()`:
1. The original "mirrors molecule-mcp-claude-channel server.ts:374"
claim is wrong on two axes. Line 374 is unrelated poll-init code
(a comment inside `registerAsPoll`). The actual capability site
is server.ts:475, where the bun bridge declares only
`{ capabilities: { tools: {} } }` — *no* `experimental.claude/channel`.
The bun bridge is reported to deliver `notifications/claude/channel`
successfully in Claude Code despite this, which is direct counter-
evidence that adding the capability was the bug fix.
2. The `@modelcontextprotocol/sdk` server's `assertNotificationCapability`
does not include `notifications/claude/channel` in any of its switch
cases, meaning custom (non-spec) notification methods are sent
regardless of declared capabilities. Server-side, the declaration
is almost certainly a no-op.
This commit doesn't remove the capability — additive, not destructive,
and the new tests pin its presence — but downgrades the docstring's
certainty so the next person debugging "channel notification didn't
fire" doesn't trust a stale claim and pursues the more likely root
causes:
- writer.drain() swallowing exceptions on a closed pipe
- inbox-thread → asyncio.run_coroutine_threadsafe race during init
- MCP transport not yet attached when the first inbox event fires
Live verification per #2444 §2 (fresh Claude Code session on this wheel
with a peer A2A message, observe whether the interrupt fires) remains
the open hard-gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without this capability declaration in the initialize handshake,
Claude Code's MCP client receives our notifications/claude/channel
emissions but silently drops them — they never become inline
<channel> tags in the conversation. The push-UX bridge added in
PR #2433 ships, fires, and is invisible.
This was anticipated as a failure mode in #2444 §2 ("Notification
arrives but Claude Code doesn't surface it — host doesn't recognize
the method"), and confirmed live in this session: a canvas chat
"hi" landed in the inbox queue (inbox_peek returned it) but never
woke the agent until inbox_peek was called by hand.
The contract matches molecule-mcp-claude-channel/server.ts:374
where the bun bridge declares the same experimental flag.
Refactor: extracted _build_initialize_result() so the handshake
shape is unit-testable. Pure function, no behavioral change beyond
adding the experimental capability to the result.
Tests: 3 new pins on the initialize result (capability presence,
tools-still-there, protocolVersion stable). Closes the live-
verification gap §2 of #2444.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review of #2460 found two issues:
1. Critical: Override button in ProviderPickerModal called
/settings/secrets when no workspaceId, overwriting the GLOBAL
secret used by every workspace. The only consumers of this
modal today (TemplatePalette, EmptyState via useTemplateDeploy)
never pass workspaceId, so Override was always destructive.
Removed entirely — the picker still solves the user-reported
bug (always-ask + reuse saved keys); per-workspace key override
can be a separate PR that plumbs secrets through POST /workspaces.
2. Optional: /settings/secrets was being fetched twice — once
inside checkDeploySecrets (silently) and again in the hook to
populate configuredKeys. Surfaced configuredKeys on
PreflightResult so the hook re-uses the existing fetch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clicking a hermes template tile silently deployed when global env
covered the API key, producing "No LLM provider configured" 500
because the workspace booted with no explicit model slug — the
adapter fell back to its compiled-in default which 401s on the
user's actual provider key.
Fix: in useTemplateDeploy, open the picker whenever the template
declares ≥2 provider options, even when preflight.ok=true. The
modal renders pre-saved keys as Saved (with an Override link) and
adds a model input pre-filled from the template's default. Single-
provider templates (claude-code, langgraph) still skip the picker
since there's nothing to choose.
POST /workspaces now includes the picker's model slug so hermes-
style routing reads the prefix at install time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wheel-build smoke gate detected `configs_dir` missing from
scripts/build_runtime_package.py:TOP_LEVEL_MODULES. Without it the
build would ship `import configs_dir` un-rewritten and every
external-runtime install would die on `ModuleNotFoundError` at first
import.
Two callers used `import configs_dir as _configs_dir` to belt-and-
suspenders against an imagined name collision, but the rewriter
rejects `import X as Y` because the rewrite would produce
`import molecule_runtime.X as X as Y` (invalid syntax). No actual
collision exists (only docstring/comment references). Switched to
plain `import configs_dir`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runtime persists per-workspace state (`.auth_token`,
`.platform_inbound_secret`, `.mcp_inbox_cursor`) under `/configs` —
the workspace-EC2 mount path. Inside a container that's writable,
agent-owned. Outside a container, `/configs` either doesn't exist or
isn't writable by an unprivileged user.
The default broke the external-runtime path (`pip install
molecule-ai-workspace-runtime` + `molecule-mcp` on a Mac/Linux
laptop). First heartbeat tries to persist `.platform_inbound_secret`
and crashes:
[Errno 30] Read-only file system: '/configs'
The heartbeat thread logs and dies. Workspace flips offline within
a minute. Operator sees no actionable error.
Adds workspace/configs_dir.py — single resolution point with a tiered
fallback:
1. CONFIGS_DIR env var, if set — explicit operator override
(preserves existing tests + custom deployments verbatim).
2. /configs — if it exists AND is writable. In-container default;
unchanged behavior for every prod workspace.
3. ~/.molecule-workspace — created with mode 0700 so per-file 0600
perms aren't undermined by a world-readable parent.
Migrates the four readers (platform_auth, platform_inbound_auth,
mcp_cli, inbox) to call configs_dir.resolve() instead of
inlining `Path(os.environ.get("CONFIGS_DIR", "/configs"))`.
Existing tests that assert the old `/configs`-as-default contract
updated to assert the new contract: when CONFIGS_DIR is unset, path
resolves to a writable location — `/configs` if present, fallback
otherwise. Tests skip the fallback branch on hosts that DO have a
writable `/configs` (CI containers).
Verified the original repro is fixed: with no CONFIGS_DIR set on
macOS, configs_dir.resolve() returns ~/.molecule-workspace, the dir
exists, and writes succeed.
Test suite: 1454 passed, 3 skipped, 2 xfailed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the data-driven pattern PR #2454 set in ConfigTab: read
runtime_config.providers from /templates and filter the modal's
provider <select> to that subset. Same source of truth, three fewer
hardcoded copies of the provider list.
Behavior:
- Template declares providers → dropdown shows only those.
- Template ships no providers field → fall back to full HERMES_PROVIDERS
catalog (back-compat for older templates / self-hosted setups).
- Declared list has no overlap with our static metadata → fall back to
full catalog so the form can't lock the operator out.
- hermesProvider snaps back to the first available pick when its
current value falls out of the filtered list.
Tests: 3 new pinning the filter, no-providers-field fallback, and
the unknown-providers fallback. All 27 CreateWorkspaceDialog tests
pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Demo-day preparation bundle for the funding demo (~2026-05-06). Adds:
- scripts/demo-freeze.sh — captures current ghcr.io
workspace-template-* :latest digests for all 8 runtimes, then
disables both cascade vectors that could re-tag :latest mid-demo:
publish-runtime.yml in molecule-core (PATH 1 — staging push to
workspace/** auto-bumps the wheel and fans out to 8 templates) and
publish-image.yml in each of the 8 template repos (PATH 2 — direct
template repo merge re-tags :latest). Defaults to dry-run; requires
--execute to apply. Writes both digest + workflow receipts to
scripts/demo-freeze-snapshots/.
- scripts/demo-thaw.sh — re-enables every workflow demo-freeze.sh
disabled, keyed off the receipt timestamp. Defaults to executing
(the inverse safety polarity from freeze, where the destructive
default is dry-run). --dry-run prints without applying.
- scripts/demo-day-runbook.md — operator runbook indexing the six
rollback levers (platform image rollback, template image rollback,
tenant redeploy, workspace delete, Railway rollback, Vercel
rollback) plus pre-warm timing and post-demo cleanup. Also covers
read-only diagnostics for "is this working?" moments and the
CP_ADMIN_API_TOKEN rotation step that must follow demo (the token
gets copy-pasted into shells during incident response).
- scripts/demo-freeze-snapshots/.gitignore — generated freeze
receipts are operational state, not source. Tracked .gitkeep so
the directory exists when the script writes to it.
Both scripts dry-run-tested locally. Did not exercise --execute since
that would actually disable production workflows mid-development.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production incident on hongming.moleculesai.app 2026-05-01T18:30Z —
fresh-tenant signup chat upload returned 500 with the body
{"error":"failed to prepare uploads dir"}. Diagnosis required SSM
access to the workspace stderr to recover errno + actual path.
The root-cause fix lives in claude-code template entrypoint
(molecule-ai-workspace-template-claude-code#23 — pre-create the
.molecule subtree as root before gosu drops to agent). This change
is the diagnostic improvement: when mkdir fails for any reason in
the future (EACCES, ENOSPC, EROFS, etc.), the response carries
the errno + offending path so the operator inspecting browser
devtools sees the real cause without needing SSM.
Backwards compatible — top-level "error" key is unchanged so
existing canvas / external alert rules continue to match. New
fields are additive: path, errno, detail.
Test pins the diagnostic shape so a future struct refactor can't
silently drop these fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Option B PR-5. Canvas Config tab now exposes a Provider override input
that's adapter-driven from each runtime's template — no hardcoded
provider list in the canvas. PUT /workspaces/:id/provider on Save
when dirty; auto-restart suppression to avoid double-restart with
the model handler's own restart.
The dropdown's suggestion list comes from /templates →
runtime_config.providers (the field added in
molecule-ai-workspace-template-hermes PR #31). For templates that
haven't migrated to the explicit providers list yet, suggestions
derive from model[].id slug prefixes — still adapter-driven, just
inferred. This keeps existing templates working while platform team
migrates them one at a time.
workspace-server changes:
- Add Providers []string field to templateSummary JSON
- Parse runtime_config.providers in /templates handler
- 2 new tests pin the surfacing + omitempty behavior
canvas changes:
- Remove hardcoded PROVIDER_SUGGESTIONS constant
- Add provider/originalProvider state + PUT-on-save logic
- Add deriveProvidersFromModels() fallback helper
- Wire RuntimeOption.providers from /templates response
- 8 new tests pin the behavior end-to-end
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror of PUT /model. Stores the provider slug as the LLM_PROVIDER
workspace secret so the canvas can update model + provider
independently — a user might keep the same model alias and switch
providers (route through a different gateway), or vice versa.
Forcing both into one endpoint imposes a single Save+Restart per
change; two endpoints let canvas update each as the user picks.
Plumbs through the existing chain: secret-load → envVars → CP
req.Env → user-data env exports → /configs/config.yaml (after
controlplane PR #364 lands the heredoc append).
Tests: 5 new cases mirroring SetModel/GetModel exactly — default
empty response, DB error, upsert with restart trigger, empty-clears,
invalid-UUID rejection.
Part of: Option B PR-2 (#196) — workspace-server plumbs LLM_PROVIDER
Stack: PR-1 schema (#2441 merged)
PR-2 (this) ws-server endpoint
PR-3 (#364 open) CP user-data persistence
PR-4 (pending) hermes adapter consume
PR-5 (pending) canvas Provider dropdown
#2429 review finding. The 410-Gone path issues a follow-up
`SELECT updated_at` after detecting status='removed'. If that query
fails (workspace row deleted between the two queries, transient DB
error, etc.), `removedAt` stays as Go's zero time and the JSON body
emits `"removed_at": "0001-01-01T00:00:00Z"` — a misleading timestamp
the client has to know to ignore.
Now we branch on `removedAt.IsZero()` and emit `null` for the failed
path. The actionable signal (the 410 + hint) is unchanged; only the
timestamp shape gets cleaner.
Pinned by `TestWorkspaceGet_RemovedReturns410WithNullRemovedAtOnTimestampFetchFailure`,
which simulates the row vanishing via `sqlmock`'s `WillReturnError(sql.ErrNoRows)`.
The original `_RemovedReturns410` test now also asserts that the
happy-path timestamp is a non-null value (was just checking the key
existed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up A to PR #2449 — that PR taught the platform to return 410
Gone for status='removed' workspaces; this PR teaches get_workspace_info
to consume that signal.
Before: every non-200 collapsed into {"error": "not found"}, which
made the 2026-04-30 incident impossible to diagnose — the operator
KNEW the workspace_id existed (they'd just registered it), but the
runtime kept reporting "not found" for a deleted-but-not-purged row.
After: 410 produces a distinct {"error": "removed", "id", "removed_at",
"hint"} dict so callers (heartbeat-loop, channel bridge, dashboard
tools) can surface "your workspace was deleted, re-onboard" instead
of "not found". Falls back to a default hint if the platform body
isn't parseable so the actionable signal doesn't depend on body
shape parity.
Two new tests:
- TestGetWorkspaceInfo.test_410_returns_removed_with_hint
- TestGetWorkspaceInfo.test_410_with_unparseable_body_falls_back_to_default_hint
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hermes-style declarative block grouping cadence + verbosity knobs into
one place. Schema-only in this PR — wiring into heartbeat.py and main.py
lands in PR-3 of the #119 stack.
Two fields with live consumers waiting:
- heartbeat_interval_seconds (default 30, clamped to [5, 300])
→ heartbeat.py:134 currently has hard-coded HEARTBEAT_INTERVAL = 30
- log_level (default "INFO", uppercased at parse)
→ main.py:465 currently has hard-coded log_level="info"
Clamp band [5, 300] is intentional: sub-5s flooded the platform during
IR-2026-03-11; >5min lets crashed workspaces look healthy long enough
to mask failure. Coerce at parse so adapters and heartbeat.py can read
the value without re-validating.
Tests pin defaults, explicit YAML override, partial override, and
parametrized clamp behavior (10 cases including garbage strings + None).
Part of: task #119 (adopt hermes-style architecture)
Stack: PR-1 schema → PR-2 event_log → PR-3 wire consumers → PR-4 skill compat
Defense-in-depth at the endpoint level. Previously, GET /workspaces/:id
returned 200 OK with `status:"removed"` in the body for deleted
workspaces — silent-fail UX hit on the hongmingwang tenant 2026-04-30:
the channel bridge / molecule-mcp wheel had a dead workspace_id + token
in .env, get_workspace_info returned 200 → caller assumed everything
was fine, then every subsequent /registry/* call 401d because tokens
were revoked, and operators had no idea their workspace was gone.
#2425 fixed the steady-state heartbeat path (escalate to ERROR after
3 consecutive 401s). This change is the startup-time defense — fail
loud when the operator first probes the workspace instead of waiting
for the heartbeat to sour.
The 410 body includes:
{error: "workspace removed", id, removed_at, hint: "Regenerate ..."}
Audit-trail consumers that need the body shape of a removed workspace
(admin views, "show me deleted workspaces" tooling) opt into the
legacy 200 + body via ?include_removed=true. Without this opt-in path
the audit trail becomes invisible at the API layer.
Two new tests pinned:
- TestWorkspaceGet_RemovedReturns410
- TestWorkspaceGet_RemovedWithIncludeQueryReturns200
Follow-ups in separate PRs:
- Update workspace/a2a_client.py get_workspace_info to surface
"removed" specifically rather than collapsing into "not found"
- Update channel bridge getWorkspaceInfo (server.ts) to detect 410
→ log clear "workspace was deleted, re-onboard" error
- Audit canvas/* + admin tooling consumers that may rely on the
legacy 200 + status:"removed" shape; switch them to the
?include_removed=true opt-in if needed
- Update docs (runtime-mcp.mdx Troubleshooting + external-agents.mdx
lifecycle table)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups from the #2275 Phase 1 self-review:
1. `_SMOKE_TIMEOUT_SECS = float(os.environ.get(...))` was evaluated at
module load. main.py imports smoke_mode unconditionally — before
the is_smoke_mode() check — so a malformed
MOLECULE_SMOKE_TIMEOUT_SECS env value would SystemExit every
workspace boot, not just smoke runs. Wrapped in try/except with a
5.0 fallback. Probability of a typo'd env var hitting production
is low (it's a CI-only knob), but the footgun is removed entirely.
Regression test reloads the module under a malformed env value.
2. `_real_a2a_sdk_available()` caught (ImportError, AttributeError).
`from X import Y` raises ImportError when Y is missing on X — never
AttributeError. Dropped the unreachable branch.
No behavior change for the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
