@ -11,66 +11,66 @@ Entries are published daily at 23:50 UTC.
### ✨ New features
- **SaaS Federation v2 tutorial**: a clean, self-contained walkthrough for platform operators who want to run multi-tenant workspaces from a single control plane. Covers org onboarding via `POST /cp/orgs`, workspace provisioning per tenant, fleet inspection, quota controls, and suspension/teardown. (`molecule-core` [#1700](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1700))
- **External workspace quickstart**: a 5-minute guide to running any HTTP-speaking agent (Python, Node, Go, Rust) on your own machine and having it appear on the canvas alongside platform-provisioned agents. Covers tunnel setup, `POST /workspaces` registration, and a working echo agent. (`molecule-core` [#1760](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1760))
- **SaaS Federation v2 tutorial**: a clean, self-contained walkthrough for platform operators who want to run multi-tenant workspaces from a single control plane. Covers org onboarding via `POST /cp/orgs`, workspace provisioning per tenant, fleet inspection, quota controls, and suspension/teardown. (`molecule-core` [#1700](https://github.com/Molecule-AI/molecule-core/pull/1700))
- **External workspace quickstart**: a 5-minute guide to running any HTTP-speaking agent (Python, Node, Go, Rust) on your own machine and having it appear on the canvas alongside platform-provisioned agents. Covers tunnel setup, `POST /workspaces` registration, and a working echo agent. (`molecule-core` [#1760](https://github.com/Molecule-AI/molecule-core/pull/1760))
### 🔧 Fixes
- **SSRF guard in SaaS mode**: previously the SSRF protection was blocking all RFC-1918 private IP ranges (`10/8`, `172.16/12`, `192.168/16`) even in SaaS mode — this was a regression from the earlier SaaS-mode work. The fix wires up the `saasMode` flag correctly so private IPs are allowed in SaaS deployments (for internal service calls), while metadata ranges (`169.254/16`), CGNAT, loopback, and link-local remain blocked in every mode. IPv6 ULA (`fd00::/8`) handling is also now correct. (`molecule-core` [#1692](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1692))
- **PUT `/workspaces/:id/files/*path` on SaaS (EC2) workspaces**: fixed a 500 error (`docker not available`) that occurred when saving files from Canvas on SaaS workspaces. The handler now detects non-Docker workspaces via `workspaces.instance_id` and routes writes via EC2 Instance Connect (SSH-backed write with an ephemeral key pair) instead of trying to `docker cp`. (`molecule-core` [#1702](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1702))
- **SSRF guard in SaaS mode**: previously the SSRF protection was blocking all RFC-1918 private IP ranges (`10/8`, `172.16/12`, `192.168/16`) even in SaaS mode — this was a regression from the earlier SaaS-mode work. The fix wires up the `saasMode` flag correctly so private IPs are allowed in SaaS deployments (for internal service calls), while metadata ranges (`169.254/16`), CGNAT, loopback, and link-local remain blocked in every mode. IPv6 ULA (`fd00::/8`) handling is also now correct. (`molecule-core` [#1692](https://github.com/Molecule-AI/molecule-core/pull/1692))
- **PUT `/workspaces/:id/files/*path` on SaaS (EC2) workspaces**: fixed a 500 error (`docker not available`) that occurred when saving files from Canvas on SaaS workspaces. The handler now detects non-Docker workspaces via `workspaces.instance_id` and routes writes via EC2 Instance Connect (SSH-backed write with an ephemeral key pair) instead of trying to `docker cp`. (`molecule-core` [#1702](https://github.com/Molecule-AI/molecule-core/pull/1702))
### 📚 Docs
- **molecli shell completion**: tab completion for `molecule` CLI in bash, zsh, fish, and PowerShell — covers all subcommands and flags. (`docs` [#79](https://git.moleculesai.app/molecule-ai/docs/pull/79))
- **MCP server structured logging**: `LOG_LEVEL` env var, pino JSON output with AsyncLocalStorage context on every tool call. (`docs` [#78](https://git.moleculesai.app/molecule-ai/docs/pull/78))
- **molecli shell completion**: tab completion for `molecule` CLI in bash, zsh, fish, and PowerShell — covers all subcommands and flags. (`docs` [#79](https://github.com/Molecule-AI/docs/pull/79))
- **MCP server structured logging**: `LOG_LEVEL` env var, pino JSON output with AsyncLocalStorage context on every tool call. (`docs` [#78](https://github.com/Molecule-AI/docs/pull/78))
### 🧹 Internal
- SaaS Federation v2 tutorial published — clean rewrite of #1613, now with correct HTTP status codes, fleet metrics endpoint, and security model table (`molecule-core` [#1700](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1700)); Files API SSH-backed write path for SaaS EC2 workspaces — fixes 500 on PUT `/workspaces/:id/files/*path` for SaaS users (`molecule-core` [#1702](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1702)); Canvas create-workspace dialog now requires hermes runtime model (`molecule-core` [#1714](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1714)).
- EC2 Instance Connect SSH tutorial published (`molecule-core` [#1617](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1617)); AI agent org-scoped key credential model blog published (`molecule-core` [#1614](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1614)); Phase 30 Day 2 social package ready (`molecule-core` [#1662](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1662)).
- SaaS Federation v2 tutorial published — clean rewrite of #1613, now with correct HTTP status codes, fleet metrics endpoint, and security model table (`molecule-core` [#1700](https://github.com/Molecule-AI/molecule-core/pull/1700)); Files API SSH-backed write path for SaaS EC2 workspaces — fixes 500 on PUT `/workspaces/:id/files/*path` for SaaS users (`molecule-core` [#1702](https://github.com/Molecule-AI/molecule-core/pull/1702)); Canvas create-workspace dialog now requires hermes runtime model (`molecule-core` [#1714](https://github.com/Molecule-AI/molecule-core/pull/1714)).
- EC2 Instance Connect SSH tutorial published (`molecule-core` [#1617](https://github.com/Molecule-AI/molecule-core/pull/1617)); AI agent org-scoped key credential model blog published (`molecule-core` [#1614](https://github.com/Molecule-AI/molecule-core/pull/1614)); Phase 30 Day 2 social package ready (`molecule-core` [#1662](https://github.com/Molecule-AI/molecule-core/pull/1662)).
### 🌅 Late-day updates (17:30–23:50 UTC)
#### 🔒 Security
- **Cross-tenant memory poisoning fix** (`molecule-core` [#1791](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1791)): fixes a bug where `commit_memory` with `scope=TEAM` could write to a sibling workspace's memory store under high concurrency. `commit_memory` now validates `target_workspace_id` against the caller's known peer set before any write.
- **CWE-78 shell injection hardening** (`molecule-core` [#1885](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1885)): `shellQuote` now uses `strconv.Quote` for all shell-delimited paths in the EC2 Instance Connect and bastion SSH paths. Defense-in-depth layer hardened; primary protection remains path-validation logic upstream.
- **Cross-tenant memory poisoning fix** (`molecule-core` [#1791](https://github.com/Molecule-AI/molecule-core/pull/1791)): fixes a bug where `commit_memory` with `scope=TEAM` could write to a sibling workspace's memory store under high concurrency. `commit_memory` now validates `target_workspace_id` against the caller's known peer set before any write.
- **CWE-78 shell injection hardening** (`molecule-core` [#1885](https://github.com/Molecule-AI/molecule-core/pull/1885)): `shellQuote` now uses `strconv.Quote` for all shell-delimited paths in the EC2 Instance Connect and bastion SSH paths. Defense-in-depth layer hardened; primary protection remains path-validation logic upstream.
#### ✨ New features
- **A2A priority queue — Phase 1** (`molecule-core` [#1892](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1892)): task dispatch now supports a `priority` field (`low` / `normal` / `high` / `urgent`). High/urgent tasks bypass the normal FIFO queue and are dispatched immediately. Phase 2 (priority inversion deadlock prevention) on the roadmap.
- **A2A priority queue — Phase 1** (`molecule-core` [#1892](https://github.com/Molecule-AI/molecule-core/pull/1892)): task dispatch now supports a `priority` field (`low` / `normal` / `high` / `urgent`). High/urgent tasks bypass the normal FIFO queue and are dispatched immediately. Phase 2 (priority inversion deadlock prevention) on the roadmap.
#### 🔧 Fixes
- **A2A queue nil-safe drain** (`molecule-core` [#1893](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1893), [#1896](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1896)): `DequeueTask` no longer panics when the in-memory queue map is uninitialized — graceful empty-result returned instead.
- **Workspaces stuck in `provisioning` after失败** (`molecule-core` [#1794](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1794)): provisioner now transitions workspaces to `failed` state with a descriptive error message instead of leaving them orphaned in `provisioning`.
- **Dedup settings hooks double-fire** (`molecule-core` [#1797](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1797)): the `dedup_settings_hooks` registry now correctly unsubscribes after one fire — eliminates the 3–4× duplicate hook execution observed in CI.
- **Semantic memory search returning stale results** (`molecule-core` [#1778](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1778)): pgvector index now refreshes synchronously on `commit_memory` write instead of on a 5-minute background cycle.
- **pgvector migration race in E2E CI** (`molecule-core` [#1777](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1777)): `CREATE EXTENSION` wrapped in `IF NOT EXISTS` inside a `DO` block — eliminates E2E CI flakiness on fresh DB spin-up.
- **EC2 Instance Connect endpoint not found in us-west-2** (`molecule-core` [#1779](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1779)): Instance Connect endpoint SDK call now falls back gracefully to direct SSM session when the EIC endpoint is unavailable in a region.
- **Canvas topology overlay edge labels clipped** (`molecule-core` [#1802](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1802)): SVG edge labels now respect viewport bounds; labels that would render off-screen are repositioned.
- **Audit trail panel not loading for large workspaces** (`molecule-core` [#1854](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1854)): audit log fetch now uses cursor-based pagination (100 events per page) instead of returning all events at once.
- **Hermes `response_format` not forwarded to MiniMax** (`molecule-core` [#1861](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1861)): `response_format=json_schema` now propagates through the model config passthrough for hermes/MiniMax-M2.7-highspeed workspaces.
- **Memory Inspector panel memory leak** (`molecule-core` [#1871](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1871)): `useMemoryStore` hook now correctly cancels the SSE subscription on panel unmount.
- **Token revocation cache stale-read window** (`molecule-core` [#1888](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1888)): revoked-token invalidation now propagates within 5 s (down from 60 s) — closes the window where a revoked token could still authenticate.
- **TenantGuard same-origin bypass (regression)** (`molecule-core` [#1898](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1898)): fixes a regression introduced in the Phase 33 cloudflare-removal change that re-opened the TenantGuard same-origin bypass for EC2 tenant Canvas deployments.
- **A2A queue nil-safe drain** (`molecule-core` [#1893](https://github.com/Molecule-AI/molecule-core/pull/1893), [#1896](https://github.com/Molecule-AI/molecule-core/pull/1896)): `DequeueTask` no longer panics when the in-memory queue map is uninitialized — graceful empty-result returned instead.
- **Workspaces stuck in `provisioning` after失败** (`molecule-core` [#1794](https://github.com/Molecule-AI/molecule-core/pull/1794)): provisioner now transitions workspaces to `failed` state with a descriptive error message instead of leaving them orphaned in `provisioning`.
- **Dedup settings hooks double-fire** (`molecule-core` [#1797](https://github.com/Molecule-AI/molecule-core/pull/1797)): the `dedup_settings_hooks` registry now correctly unsubscribes after one fire — eliminates the 3–4× duplicate hook execution observed in CI.
- **Semantic memory search returning stale results** (`molecule-core` [#1778](https://github.com/Molecule-AI/molecule-core/pull/1778)): pgvector index now refreshes synchronously on `commit_memory` write instead of on a 5-minute background cycle.
- **pgvector migration race in E2E CI** (`molecule-core` [#1777](https://github.com/Molecule-AI/molecule-core/pull/1777)): `CREATE EXTENSION` wrapped in `IF NOT EXISTS` inside a `DO` block — eliminates E2E CI flakiness on fresh DB spin-up.
- **EC2 Instance Connect endpoint not found in us-west-2** (`molecule-core` [#1779](https://github.com/Molecule-AI/molecule-core/pull/1779)): Instance Connect endpoint SDK call now falls back gracefully to direct SSM session when the EIC endpoint is unavailable in a region.
- **Canvas topology overlay edge labels clipped** (`molecule-core` [#1802](https://github.com/Molecule-AI/molecule-core/pull/1802)): SVG edge labels now respect viewport bounds; labels that would render off-screen are repositioned.
- **Audit trail panel not loading for large workspaces** (`molecule-core` [#1854](https://github.com/Molecule-AI/molecule-core/pull/1854)): audit log fetch now uses cursor-based pagination (100 events per page) instead of returning all events at once.
- **Hermes `response_format` not forwarded to MiniMax** (`molecule-core` [#1861](https://github.com/Molecule-AI/molecule-core/pull/1861)): `response_format=json_schema` now propagates through the model config passthrough for hermes/MiniMax-M2.7-highspeed workspaces.
- **Memory Inspector panel memory leak** (`molecule-core` [#1871](https://github.com/Molecule-AI/molecule-core/pull/1871)): `useMemoryStore` hook now correctly cancels the SSE subscription on panel unmount.
- **Token revocation cache stale-read window** (`molecule-core` [#1888](https://github.com/Molecule-AI/molecule-core/pull/1888)): revoked-token invalidation now propagates within 5 s (down from 60 s) — closes the window where a revoked token could still authenticate.
- **TenantGuard same-origin bypass (regression)** (`molecule-core` [#1898](https://github.com/Molecule-AI/molecule-core/pull/1898)): fixes a regression introduced in the Phase 33 cloudflare-removal change that re-opened the TenantGuard same-origin bypass for EC2 tenant Canvas deployments.
#### 📚 Docs
- **Chrome DevTools MCP tutorial** (`docs` [#1798](https://git.moleculesai.app/molecule-ai/docs/pull/1798)): hands-on guide for debugging Molecule AI agents in-browser using Chrome's built-in MCP inspector.
- **Phase 34 launch page** (`docs` [#1799](https://git.moleculesai.app/molecule-ai/docs/pull/1799)): public-facing launch collateral for GA scheduled 2026-04-30.
- **Tool Trace demo environment** (`docs` [#1844](https://git.moleculesai.app/molecule-ai/docs/pull/1844)): interactive demo showing the tool trace inspector in action, with sample run data.
- **Enterprise battlecard** (`docs` [#1864](https://git.moleculesai.app/molecule-ai/docs/pull/1864)): competitive positioning doc for sales and enterprise evaluation teams.
- **Chrome DevTools MCP tutorial** (`docs` [#1798](https://github.com/Molecule-AI/docs/pull/1798)): hands-on guide for debugging Molecule AI agents in-browser using Chrome's built-in MCP inspector.
- **Phase 34 launch page** (`docs` [#1799](https://github.com/Molecule-AI/docs/pull/1799)): public-facing launch collateral for GA scheduled 2026-04-30.
- **Tool Trace demo environment** (`docs` [#1844](https://github.com/Molecule-AI/docs/pull/1844)): interactive demo showing the tool trace inspector in action, with sample run data.
- **Enterprise battlecard** (`docs` [#1864](https://github.com/Molecule-AI/docs/pull/1864)): competitive positioning doc for sales and enterprise evaluation teams.
#### 🧹 Internal
- `a2a-sdk` hot-pinned to `0.3.x` across all workspace template repos (`molecule-core` [#1890](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1890)); SDK upgrade path documented in `KI-009` (`internal` [#1631](https://git.moleculesai.app/molecule-ai/internal/issues/1631)).
- `a2a-sdk` hot-pinned to `0.3.x` across all workspace template repos (`molecule-core` [#1890](https://github.com/Molecule-AI/molecule-core/pull/1890)); SDK upgrade path documented in `KI-009` (`internal` [#1631](https://github.com/Molecule-AI/internal/issues/1631)).
- Phase 34 CI matrix expanded to cover Node 22 and Go 1.24 (`molecule-ci`).
#### 🔧 Runtime fixes
- **Heartbeat 401 retry** (`molecule-ai-workspace-runtime` [#40](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-runtime/pull/40)): heartbeat worker now retries with fresh token on 401 before declaring the workspace unreachable — eliminates false `disconnected` status during token rotation.
- **LLM token auto-detect** (`molecule-ai-workspace-runtime` [#38](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-runtime/pull/38)): hermes runtime now auto-detects `max_tokens` from model context window and request timeout when not explicitly configured.
- **Heartbeat 401 retry** (`molecule-ai-workspace-runtime` [#40](https://github.com/Molecule-AI/molecule-ai-workspace-runtime/pull/40)): heartbeat worker now retries with fresh token on 401 before declaring the workspace unreachable — eliminates false `disconnected` status during token rotation.
- **LLM token auto-detect** (`molecule-ai-workspace-runtime` [#38](https://github.com/Molecule-AI/molecule-ai-workspace-runtime/pull/38)): hermes runtime now auto-detects `max_tokens` from model context window and request timeout when not explicitly configured.
---
@ -84,7 +84,7 @@ Customer selects `model=minimax/MiniMax-M2.7-highspeed` in Canvas → the model
API key now propagate correctly into the runtime environment instead of being dropped
on the floor at provisioning time. Works for hermes workspaces in both hosted SaaS
- **F1085 deleteViaEphemeral**: `rm` scope restricted to `/configs` volume only —
prevents deletion of application code or workspace files if the exec form is
exploited. Applied to both `main` and `staging`. (`molecule-core` [#1682](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1682), [#1616](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1616))
exploited. Applied to both `main` and `staging`. (`molecule-core` [#1682](https://github.com/Molecule-AI/molecule-core/pull/1682), [#1616](https://github.com/Molecule-AI/molecule-core/pull/1616))
### 🔧 Fixes
- Canvas now fetches the runtime and model dropdown from the `/templates` registry
at load time — runtime list stays current without code deploys. (`molecule-core` [#1666](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1666))
at load time — runtime list stays current without code deploys. (`molecule-core` [#1666](https://github.com/Molecule-AI/molecule-core/pull/1666))
- Canvas accessibility: `aria-hidden` correctly applied to decorative SVGs;
`MissingKeysModal` now uses correct dialog semantics and manages focus. (`molecule-core` [#1594](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1594))
`MissingKeysModal` now uses correct dialog semantics and manages focus. (`molecule-core` [#1594](https://github.com/Molecule-AI/molecule-core/pull/1594))
- Provisioner pulls workspace template images from GHCR instead of Docker Hub
for faster cold starts and reduced third-party dependency. (`molecule-core` [#1624](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1624))
for faster cold starts and reduced third-party dependency. (`molecule-core` [#1624](https://github.com/Molecule-AI/molecule-core/pull/1624))
- Shared runtime heartbeat no longer leaves workspaces in a phantom-busy state after
@ -158,7 +158,7 @@ The `id` field is your workspace ID — remember it.
|---|---|
| "Failed to send message — agent may be unreachable" | The tenant couldn't POST to your URL. Verify `curl https://<your-tunnel>/health` returns 200 from another machine. |
| Response takes > 30s | Canvas times out around 30s. Keep initial implementations simple. For long-running work, return a placeholder and use [polling mode](#next-step-polling-mode-preview) (once available). |
| Agent duplicated in chat | Known canvas bug where WebSocket + HTTP responses both render. Fixed in [molecule-core #1517](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1517). |
| Agent duplicated in chat | Known canvas bug where WebSocket + HTTP responses both render. Fixed in [molecule-core #1517](https://github.com/Molecule-AI/molecule-core/pull/1517). |
| Agent replies but canvas shows "Agent unreachable" | Check the tenant can reach your URL. Cloudflare quick tunnels rotate — the URL in your canvas may point at a dead tunnel after restart. |
| Getting 404 when POSTing to tenant | Add `X-Molecule-Org-Id` header. The tenant's security layer 404s unmatched origin requests by design. |
@ -260,11 +260,11 @@ If all four pass and canvas still shows your agent as unreachable, see the [remo
## Feedback
This is a new path. Tell us what broke:
- Open an issue: https://git.moleculesai.app/molecule-ai/molecule-core/issues/new?labels=external-workspace
- Open an issue: https://github.com/Molecule-AI/molecule-core/issues/new?labels=external-workspace
- Submit a PR improving this doc if something tripped you up — the faster we can make the quickstart, the more developers we bring in
@ -105,9 +105,9 @@ In **SaaS mode** (`saasMode()` returns true), cross-EC2 traffic to RFC-1918 addr
### Regression (2026-04-21)
PR [#1363](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1363) (handler refactor) moved `isPrivateOrMetadataIP` into `a2a_proxy_helpers.go` but kept a **pre-SaaS version** that unconditionally blocked RFC-1918 addresses, breaking cross-EC2 communication in SaaS. The old version also **returned `false` for all IPv6 inputs**, fully bypassing SSRF protection for IPv6 targets.
PR [#1363](https://github.com/Molecule-AI/molecule-core/pull/1363) (handler refactor) moved `isPrivateOrMetadataIP` into `a2a_proxy_helpers.go` but kept a **pre-SaaS version** that unconditionally blocked RFC-1918 addresses, breaking cross-EC2 communication in SaaS. The old version also **returned `false` for all IPv6 inputs**, fully bypassing SSRF protection for IPv6 targets.
PR [#1430](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1430) restores the correct SaaS-gated logic and adds proper IPv6 coverage to the A2A proxy path.
PR [#1430](https://github.com/Molecule-AI/molecule-core/pull/1430) restores the correct SaaS-gated logic and adds proper IPv6 coverage to the A2A proxy path.
### User-facing summary
@ -118,7 +118,7 @@ Platform outbound requests from workspaces (MCP tool calls, A2A proxy routing) v
## 2026-04-21 — Audit Ledger HMAC Chain Guard
**Severity:** Low (denial-of-service / data integrity)
**PRs:** [#1339](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1339), [#1352](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1352), [#1354](https://git.moleculesai.app/molecule-ai/molecule-core/pull/1354) (backport to `main`)
**PRs:** [#1339](https://github.com/Molecule-AI/molecule-core/pull/1339), [#1352](https://github.com/Molecule-AI/molecule-core/pull/1352), [#1354](https://github.com/Molecule-AI/molecule-core/pull/1354) (backport to `main`)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.