---
title: Changelog
description: Customer-facing release notes for Molecule AI — updated daily.
---

All notable changes to the Molecule AI platform are documented here.
Entries are published daily at 23:50 UTC.

---
## 2026-04-23

### ✨ New features

- **SaaS Federation v2 tutorial**: a clean, self-contained walkthrough for platform operators who want to run multi-tenant workspaces from a single control plane. Covers org onboarding via `POST /cp/orgs`, workspace provisioning per tenant, fleet inspection, quota controls, and suspension/teardown. (`molecule-core` [#1700](https://github.com/Molecule-AI/molecule-core/pull/1700))
- **External workspace quickstart**: a 5-minute guide to running any HTTP-speaking agent (Python, Node, Go, Rust) on your own machine and having it appear on the canvas alongside platform-provisioned agents. Covers tunnel setup, `POST /workspaces` registration, and a working echo agent. (`molecule-core` [#1760](https://github.com/Molecule-AI/molecule-core/pull/1760))

### 🔧 Fixes

- **SSRF guard in SaaS mode**: previously the SSRF protection was blocking all RFC-1918 private IP ranges (`10/8`, `172.16/12`, `192.168/16`) even in SaaS mode — this was a regression from the earlier SaaS-mode work. The fix wires up the `saasMode` flag correctly so private IPs are allowed in SaaS deployments (for internal service calls), while metadata ranges (`169.254/16`), CGNAT, loopback, and link-local remain blocked in every mode. IPv6 ULA (`fd00::/8`) handling is also now correct. (`molecule-core` [#1692](https://github.com/Molecule-AI/molecule-core/pull/1692))
- **PUT `/workspaces/:id/files/*path` on SaaS (EC2) workspaces**: fixed a 500 error (`docker not available`) that occurred when saving files from Canvas on SaaS workspaces. The handler now detects non-Docker workspaces via `workspaces.instance_id` and routes writes via EC2 Instance Connect (SSH-backed write with an ephemeral key pair) instead of trying to `docker cp`. (`molecule-core` [#1702](https://github.com/Molecule-AI/molecule-core/pull/1702))

### 📚 Docs

- **molecli shell completion**: tab completion for `molecule` CLI in bash, zsh, fish, and PowerShell — covers all subcommands and flags. (`docs` [#79](https://github.com/Molecule-AI/docs/pull/79))
- **MCP server structured logging**: `LOG_LEVEL` env var, pino JSON output with AsyncLocalStorage context on every tool call. (`docs` [#78](https://github.com/Molecule-AI/docs/pull/78))

### 🧹 Internal

- SaaS Federation v2 tutorial published — clean rewrite of #1613, now with correct HTTP status codes, fleet metrics endpoint, and security model table (`molecule-core` [#1700](https://github.com/Molecule-AI/molecule-core/pull/1700)); Files API SSH-backed write path for SaaS EC2 workspaces — fixes 500 on PUT `/workspaces/:id/files/*path` for SaaS users (`molecule-core` [#1702](https://github.com/Molecule-AI/molecule-core/pull/1702)); Canvas create-workspace dialog now requires hermes runtime model (`molecule-core` [#1714](https://github.com/Molecule-AI/molecule-core/pull/1714)).
- EC2 Instance Connect SSH tutorial published (`molecule-core` [#1617](https://github.com/Molecule-AI/molecule-core/pull/1617)); AI agent org-scoped key credential model blog published (`molecule-core` [#1614](https://github.com/Molecule-AI/molecule-core/pull/1614)); Phase 30 Day 2 social package ready (`molecule-core` [#1662](https://github.com/Molecule-AI/molecule-core/pull/1662)).

### 🌅 Late-day updates (17:30–23:50 UTC)

#### 🔒 Security

- **Cross-tenant memory poisoning fix** (`molecule-core` [#1791](https://github.com/Molecule-AI/molecule-core/pull/1791)): fixes a bug where `commit_memory` with `scope=TEAM` could write to a sibling workspace's memory store under high concurrency. `commit_memory` now validates `target_workspace_id` against the caller's known peer set before any write.
- **CWE-78 shell injection hardening** (`molecule-core` [#1885](https://github.com/Molecule-AI/molecule-core/pull/1885)): `shellQuote` now uses `strconv.Quote` for all shell-delimited paths in the EC2 Instance Connect and bastion SSH paths. Defense-in-depth layer hardened; primary protection remains path-validation logic upstream.

#### ✨ New features

- **A2A priority queue — Phase 1** (`molecule-core` [#1892](https://github.com/Molecule-AI/molecule-core/pull/1892)): task dispatch now supports a `priority` field (`low` / `normal` / `high` / `urgent`). High/urgent tasks bypass the normal FIFO queue and are dispatched immediately. Phase 2 (priority inversion deadlock prevention) on the roadmap.

#### 🔧 Fixes

- **A2A queue nil-safe drain** (`molecule-core` [#1893](https://github.com/Molecule-AI/molecule-core/pull/1893), [#1896](https://github.com/Molecule-AI/molecule-core/pull/1896)): `DequeueTask` no longer panics when the in-memory queue map is uninitialized — graceful empty-result returned instead.
- **Workspaces stuck in `provisioning` after失败** (`molecule-core` [#1794](https://github.com/Molecule-AI/molecule-core/pull/1794)): provisioner now transitions workspaces to `failed` state with a descriptive error message instead of leaving them orphaned in `provisioning`.
- **Dedup settings hooks double-fire** (`molecule-core` [#1797](https://github.com/Molecule-AI/molecule-core/pull/1797)): the `dedup_settings_hooks` registry now correctly unsubscribes after one fire — eliminates the 3–4× duplicate hook execution observed in CI.
- **Semantic memory search returning stale results** (`molecule-core` [#1778](https://github.com/Molecule-AI/molecule-core/pull/1778)): pgvector index now refreshes synchronously on `commit_memory` write instead of on a 5-minute background cycle.
- **pgvector migration race in E2E CI** (`molecule-core` [#1777](https://github.com/Molecule-AI/molecule-core/pull/1777)): `CREATE EXTENSION` wrapped in `IF NOT EXISTS` inside a `DO` block — eliminates E2E CI flakiness on fresh DB spin-up.
- **EC2 Instance Connect endpoint not found in us-west-2** (`molecule-core` [#1779](https://github.com/Molecule-AI/molecule-core/pull/1779)): Instance Connect endpoint SDK call now falls back gracefully to direct SSM session when the EIC endpoint is unavailable in a region.
- **Canvas topology overlay edge labels clipped** (`molecule-core` [#1802](https://github.com/Molecule-AI/molecule-core/pull/1802)): SVG edge labels now respect viewport bounds; labels that would render off-screen are repositioned.
- **Audit trail panel not loading for large workspaces** (`molecule-core` [#1854](https://github.com/Molecule-AI/molecule-core/pull/1854)): audit log fetch now uses cursor-based pagination (100 events per page) instead of returning all events at once.
- **Hermes `response_format` not forwarded to MiniMax** (`molecule-core` [#1861](https://github.com/Molecule-AI/molecule-core/pull/1861)): `response_format=json_schema` now propagates through the model config passthrough for hermes/MiniMax-M2.7-highspeed workspaces.
- **Memory Inspector panel memory leak** (`molecule-core` [#1871](https://github.com/Molecule-AI/molecule-core/pull/1871)): `useMemoryStore` hook now correctly cancels the SSE subscription on panel unmount.
- **Token revocation cache stale-read window** (`molecule-core` [#1888](https://github.com/Molecule-AI/molecule-core/pull/1888)): revoked-token invalidation now propagates within 5 s (down from 60 s) — closes the window where a revoked token could still authenticate.
- **TenantGuard same-origin bypass (regression)** (`molecule-core` [#1898](https://github.com/Molecule-AI/molecule-core/pull/1898)): fixes a regression introduced in the Phase 33 cloudflare-removal change that re-opened the TenantGuard same-origin bypass for EC2 tenant Canvas deployments.

#### 📚 Docs

- **Chrome DevTools MCP tutorial** (`docs` [#1798](https://github.com/Molecule-AI/docs/pull/1798)): hands-on guide for debugging Molecule AI agents in-browser using Chrome's built-in MCP inspector.
- **Phase 34 launch page** (`docs` [#1799](https://github.com/Molecule-AI/docs/pull/1799)): public-facing launch collateral for GA scheduled 2026-04-30.
- **Tool Trace demo environment** (`docs` [#1844](https://github.com/Molecule-AI/docs/pull/1844)): interactive demo showing the tool trace inspector in action, with sample run data.
- **Enterprise battlecard** (`docs` [#1864](https://github.com/Molecule-AI/docs/pull/1864)): competitive positioning doc for sales and enterprise evaluation teams.

#### 🧹 Internal

- `a2a-sdk` hot-pinned to `0.3.x` across all workspace template repos (`molecule-core` [#1890](https://github.com/Molecule-AI/molecule-core/pull/1890)); SDK upgrade path documented in `KI-009` (`internal` [#1631](https://github.com/Molecule-AI/internal/issues/1631)).
- Phase 34 CI matrix expanded to cover Node 22 and Go 1.24 (`molecule-ci`).

#### 🔧 Runtime fixes

- **Heartbeat 401 retry** (`molecule-ai-workspace-runtime` [#40](https://github.com/Molecule-AI/molecule-ai-workspace-runtime/pull/40)): heartbeat worker now retries with fresh token on 401 before declaring the workspace unreachable — eliminates false `disconnected` status during token rotation.
- **LLM token auto-detect** (`molecule-ai-workspace-runtime` [#38](https://github.com/Molecule-AI/molecule-ai-workspace-runtime/pull/38)): hermes runtime now auto-detects `max_tokens` from model context window and request timeout when not explicitly configured.

---


## 2026-04-22

### ✨ New features

#### Workspace model propagation — hermes MiniMax flow
Customer selects `model=minimax/MiniMax-M2.7-highspeed` in Canvas → the model and
API key now propagate correctly into the runtime environment instead of being dropped
on the floor at provisioning time. Works for hermes workspaces in both hosted SaaS
and self-hosted EC2 deployments.
(`molecule-core` [#1685](https://github.com/Molecule-AI/molecule-core/pull/1685))

#### EC2 Instance Connect Endpoint — one-click shell from Canvas
Canvas Terminal tab now uses AWS EC2 Instance Connect Endpoint to open a PTY inside
any workspace EC2 instance — no SSH keys to manage, no IP to copy, no security group
rules to configure. IAM policy gates access, STS pushes a short-lived key that
auto-expires, and every tunnel open is recorded in CloudTrail.
See the [EC2 Instance Connect guide](/docs/infra/workspace-terminal).
(`molecule-core` [#1554](https://github.com/Molecule-AI/molecule-core/pull/1554))

#### Phase 33 — Cloudflare Tunnel replaced with direct-connect public IPs
Cloud-hosted workspaces no longer route through `cloudflared`. Each workspace gets
its own public IP from the VPC subnet and connects directly to the platform over
TLS on port 443. Reduces latency by ~20–40 ms (region-dependent), removes the
Cloudflare egress cost dependency, and enables direct `curl` debugging without
the tunnel path.
See the [migration blog post](/blog/cloudflare-tunnel-migration).
(`molecule-core` [#1612](https://github.com/Molecule-AI/molecule-core/pull/1612))

### 🔒 Security

- **F1085 deleteViaEphemeral**: `rm` scope restricted to `/configs` volume only —
  prevents deletion of application code or workspace files if the exec form is
  exploited. Applied to both `main` and `staging`. (`molecule-core` [#1682](https://github.com/Molecule-AI/molecule-core/pull/1682), [#1616](https://github.com/Molecule-AI/molecule-core/pull/1616))

### 🔧 Fixes

- Canvas now fetches the runtime and model dropdown from the `/templates` registry
  at load time — runtime list stays current without code deploys. (`molecule-core` [#1666](https://github.com/Molecule-AI/molecule-core/pull/1666))
- Canvas accessibility: `aria-hidden` correctly applied to decorative SVGs;
  `MissingKeysModal` now uses correct dialog semantics and manages focus. (`molecule-core` [#1594](https://github.com/Molecule-AI/molecule-core/pull/1594))
- Provisioner pulls workspace template images from GHCR instead of Docker Hub
  for faster cold starts and reduced third-party dependency. (`molecule-core` [#1624](https://github.com/Molecule-AI/molecule-core/pull/1624))
- Shared runtime heartbeat no longer leaves workspaces in a phantom-busy state after
  task completion. (`molecule-ai-workspace-runtime` [#37](https://github.com/Molecule-AI/molecule-ai-workspace-runtime/pull/37))

### 📚 Docs

- **MCP server structured logging**: `LOG_LEVEL` env var (`trace`/`debug`/`info`/`warn`/`error`/`fatal`),
  pino JSON output in production, pretty-print in development, AsyncLocalStorage
  context on every log entry (tool name, request ID, workspace ID). (`docs` [#78](https://github.com/Molecule-AI/docs/pull/78))
- **molecli shell completion**: tab completion for `molecule` CLI in bash, zsh, fish,
  and PowerShell — covers all subcommands and flags. (`docs` [#79](https://github.com/Molecule-AI/docs/pull/79))

### 🧹 Internal

- 34 internal changes across `molecule-core`, `molecule-ci`, and template repos:
  CI workflow migration to `ubuntu-latest`, security patch backports (CWE-22/CWE-78),
  Go build fixes, canvas Dockerfile GID fix, Go linter upgrades, duplicate-symbol
  resolution, and reusable `publish-template-image` workflow for all workspace template
  repos. (`molecule-core`, `molecule-ci`)

---



## 2026-04-17

A high-velocity day: 80+ PRs merged across platform, canvas, runtimes, security, and channels.

### ✨ New features

#### opencode Integration — MCP bridge for AI coding agents
Connect [opencode](https://opencode.ai) to any Molecule AI workspace over a
standard `Authorization: Bearer` remote MCP connection. opencode gains the full
A2A tool surface (`delegate_task`, `list_peers`, `recall_memory`, and more)
via two transports: Streamable HTTP (`POST /workspaces/:id/mcp`) and SSE
(backwards-compat `GET /workspaces/:id/mcp/stream`). Rate-limited to 120 req/min
per token. See the [opencode Integration guide](/docs/opencode).
(#840, #842)

#### Slack — per-agent identity with Bot Token mode
The Slack channel adapter now supports dual-mode outbound: **Bot Token** (new,
recommended) and Incoming Webhook (legacy, unchanged). With a `bot_token` each
workspace posts under its own display name and icon via `chat:write.customize`.
Markdown is automatically converted to Slack `mrkdwn` format.
See [Channels](/docs/channels).
(#844, #851)

#### AG-UI compatible SSE endpoint
New `GET /workspaces/:id/events` endpoint streams agent events as AG-UI
compatible Server-Sent Events. Enables AG-UI frontend integrations to subscribe
to live workspace activity without polling.
(#601)

#### A2A topology overlay on the canvas
The canvas now renders a live A2A topology overlay — every workspace as a node,
every in-flight delegation as an animated directed edge. Zoom to team, click any
edge to inspect the task payload.
(#751)

#### Audit trail visualisation panel
A new audit trail panel in the canvas surfaces the HMAC-SHA256 immutable event
log per workspace — every task received, LLM call, and completion in
chronological order with chain-of-custody verification.
(#651, #759)

#### Workspace hibernation — auto-pause idle workspaces
Workspaces that receive no tasks for `HIBERNATION_IDLE_MINUTES` (default: 30)
are automatically hibernated (containers paused, resources freed). They
auto-wake on the next inbound task with full state restored. Manage via
`POST /workspaces/:id/hibernate` and `POST /workspaces/:id/wake`.
See [API Reference](/docs/api-reference).
(#724)

#### Temporal workflow checkpoints — step-level persistence
Workspace templates now persist intermediate workflow steps to the database.
On container restart (crash, deploy, hibernate/wake) the workspace resumes from
the last completed step rather than restarting the whole task. Step endpoints
documented in the [API Reference](/docs/api-reference).
(#797, #803)

#### Semantic memory search
Agent memory is now vector-indexed via pgvector. `recall_memory` accepts an
optional `?q=` parameter for semantic (embedding) search in addition to exact
keyword match. Nearest-neighbour results are ranked by cosine similarity and
colour-coded in the canvas Memory Inspector.
(#784, #787)

#### Memory Inspector panel
A new canvas panel lets you browse, search, and inspect all `LOCAL` and `TEAM`
memory keys for any workspace — live, without leaving the canvas.
(#738)

#### Hermes — stacked system messages
The Hermes runtime now accepts a `system_blocks` list: each block (persona,
tools, reasoning policy) is merged in order rather than overwriting the previous
system prompt. Enables persona stacking for complex multi-role workflows.
See [API Reference](/docs/api-reference) → Runtimes section.
(#655, #798)

#### Hermes — native `tools` parameter
Hermes passes tools to the model via the native `tools=[]` API parameter instead
of text-in-prompt injection. Structured tool definitions, better token efficiency,
and full compatibility with Nous/Hermes-3 tool call format.
(#644)

#### Hermes — structured output (`response_format`)
`response_format=json_schema` is now wired through to the model. Hermes
workspaces can request strict JSON output against a defined schema.
(#645)

#### AGENTS.md auto-generation
Platform workspaces now auto-generate an `AGENTS.md` file in the workspace
container at boot. The file lists all peer workspaces visible to this workspace,
their roles, and their capabilities — giving LLMs automatic context about the
org topology without manual prompt engineering.
(#763)

#### Discord channel adapter
A new Discord adapter joins Telegram, Slack, and Lark. Configure with a
`bot_token` and `channel_id` to send and receive messages on Discord.
(#656)

#### Per-workspace budget limits
Set a `budget_limit` (USD) on any workspace. The A2A executor enforces the limit
at task dispatch — tasks that would exceed the monthly cap are rejected with a
`429 Budget Exceeded` error. Configure via `PATCH /workspaces/:id`.
(#611, #606)

#### Per-workspace token metrics
`GET /workspaces/:id/metrics` returns token counts (input, output, cache read/write)
aggregated over rolling 1-hour and 30-day windows. Live usage is displayed in the
canvas WorkspaceUsage panel.
(#602, #627)

#### Claude Opus 4.7 — effort levels and task budget
Workspace config now exposes `effort` (`low` / `medium` / `high` / `xhigh` /
`max`) and `task_budget` (token ceiling) for Anthropic Claude workspaces.
`xhigh` and `max` activate extended thinking (Opus 4.7+ only). Configure in the
Canvas Config tab or via `PATCH /workspaces/:id`.
(#639, #654, #669)

#### Plugin supply-chain hardening
All plugin refs must now be pinned (no `latest`, no floating branches). Unpinned
refs are blocked at load time unless `PLUGIN_ALLOW_UNPINNED=true`. SHA-256
integrity checking available for plugin archives.
(#775)

#### Org-level plugin governance registry
A new per-org allowlist controls which plugins workspaces in that org are
permitted to load. Managed via `POST/DELETE /admin/orgs/:orgId/plugins/allowlist`.
(#610)

#### Schedule health endpoint
`GET /admin/schedules/health` returns cross-workspace cron health: last-fired,
next-scheduled, consecutive-empty count, and phantom detection status for every
schedule in the org.
(#671, #796)

#### Fly Machines provisioner
The platform now supports `PROVISIONER=flyio` — workspaces are provisioned as
Fly Machines instead of Docker containers or EC2 instances. See the
[self-hosting guide](/docs/self-hosting).
(#578 — docs PR #7)

### 🔒 Security

- **Auth hardening** — PATCH `/workspaces/:id` now requires ownership
  validation; UUID fields are validated before DB queries; input lengths bounded
  across all handlers. (#692, #701)
- **Admin token isolation** — `AdminAuth` middleware correctly rejects workspace
  bearer tokens when `ADMIN_TOKEN` is set, preventing privilege escalation from
  workspace token → admin. (#684, #729)
- **Metrics route auth** — `GET /workspaces/:id/metrics` now requires workspace
  bearer token; previously it was unauthenticated. (#696)
- **X-Workspace-ID forgery** — Requests spoofing the `system-caller/` prefix in
  `X-Workspace-ID` headers are rejected. (#766)
- **GLOBAL memory injection safeguards** — `commit_memory` with `scope: GLOBAL`
  now validates content for prompt injection patterns before persisting. (#769)
- **Security headers** — `X-Content-Type-Options: nosniff` and
  `X-Frame-Options: DENY` added to all API responses. (#629)
- **Token revocation hardening** — Revoked tokens are purged from the in-memory
  cache within 60s; previously the cache could serve revoked tokens until TTL
  expiry. (#696)
- **MCP server** — npm version pinned; `-y` flag removed from install commands.
  (SAFE-MCP NEW-003, #808 — docs PR #18)
- **Canvas test-token endpoint** — gated behind `AdminAuth` and removed from
  general router. (#612, #708)

### 🔧 Fixes

- Fixed `POST /workspaces` not persisting the secrets envelope on create. (#568)
- Fixed self-delegation deadlock when a workspace delegates to itself. (#570)
- Fixed GitHub installation token expiry — tokens now refresh automatically before
  expiry rather than failing mid-operation. (#567)
- Fixed `TenantGuard` same-origin bypass for EC2 tenant Canvas. (#584)
- Fixed pgvector migration to wrap in `DO` block, eliminating E2E CI failures
  from duplicate extension install. (#843, #670, #636)
- Fixed scheduler dropping schedules with `NULL next_run_at` permanently. (#728)
- Fixed `ValidateToken` not checking `removed` workspace status, allowing tokens
  for deleted workspaces to authenticate. (#719)
- Fixed canvas hydration error UI, radio keyboard nav, and zoom-to-team
  shortcut. (#565)
- Fixed canvas UX: error handling, accessibility, loading state. (#587)
- Fixed canvas deploy preflight to require env keys for Hermes and Gemini CLI
  runtimes. (#588)
- Fixed budget/spend counters capping before DB upsert to prevent NUMERIC
  overflow. (#630, #634)
- Fixed pgvector TEXT→UUID FK type mismatch in migrations 028 and 031 that
  blocked all E2E runs. (#646, #670, #843)
- Fixed duplicate hook firings (3–4×) in `dedup_settings_hooks`. (#551, #597)
- Accessibility fixes: keyboard access on `TeamMemberChip`, `role=alert` on
  status banners, close button label, `ProvisioningTimeout` modal. (#841)

### 📚 Docs

- Google ADK runtime — added hands-on Quickstart section. (docs PR #8)
- Hermes — full runtime reference page. (docs PR #9)
- AGENTS.md — auto-generation documented in concepts. (docs PR #10)
- Semantic memory search — `?q=` param documented in API reference. (docs PR #11)
- Canvas A2A topology overlay + audit trail panel. (docs PR #12)
- molecule-medo plugin — opt-in platform plugin page. (docs PR #13)
- Workspace hibernation — status lifecycle, endpoints, auto-wake behaviour. (docs PR #14)
- molecule-audit-ledger — HMAC chain, `/audit` endpoint, `LedgerHooks`, CLI. (docs PR #15)
- Hermes stacked system messages — `system_blocks` kwarg. (docs PR #16)
- Plugin supply chain security — pinned refs required, SHA-256 integrity. (docs PR #17)
- SAFE-MCP audit report 2026-04-17. (docs PR #18)
- Temporal workflow checkpoints — step endpoints, auto-resume behaviour. (docs PR #19)

---

_Changelog entries are compiled by the [Documentation Specialist](https://github.com/Molecule-AI) from all merged pull requests for the day. Times are UTC._
