Closes#790. Depends on feat/issue-583-1-checkpoint-persistence (PR #788).
Platform (Go) — checkpoints_integration_test.go (5 new tests):
1. ThreeStepPersistence: POST task_receive/llm_call/task_complete → GET returns
all 3 in step_index DESC order with correct names and payloads.
2. CrashResume_HighestStepIsResumptionPoint: POST steps 0+1 only (crash before
step 2) → GET shows step_index=1 as the resume point; task_complete absent.
3. UpsertIdempotency_LatestPayloadWins: POST same (wf_id, step_name) twice with
different payloads → List returns only the second payload (ON CONFLICT DO UPDATE).
4. PostCascadeDelete_Returns404: simulate post ON-DELETE-CASCADE state (empty
rows) → List returns 404 as expected after workspace deletion.
5. AuthGate_NoToken_Returns401: router-level test with WorkspaceAuth middleware;
POST/GET/DELETE all return 401 without a bearer token (no DB calls made).
workspace-template — _save_checkpoint + 4 Python tests:
- Add async _save_checkpoint() to temporal_workflow.py: POST to the platform
checkpoint endpoint after each activity stage; fully non-fatal (try/except
inside the function, plus defence-in-depth try/except at every call site).
- 4 new pytest cases (test_temporal_workflow.py):
- nonfatal_on_http_error: _save_checkpoint raises HTTPStatusError (500) →
task_receive_activity still returns {"status":"received"}.
- nonfatal_on_network_error: _save_checkpoint raises ConnectError →
llm_call_activity still returns success LLMResult.
- success_path: _save_checkpoint no-op → activity returns correctly;
checkpoint called with correct args.
- standalone_http_error_is_swallowed: real _save_checkpoint function swallows
HTTP 500 from a mocked httpx.AsyncClient; returns None.
All 36 temporal workflow Python tests pass.
Go tests: Go binary not in this container; test file verified for syntax and
against the sqlmock patterns used throughout the handlers package.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closes#790. Depends on feat/issue-583-1-checkpoint-persistence (PR #788).
Platform (Go) — checkpoints_integration_test.go (5 new tests):
1. ThreeStepPersistence: POST task_receive/llm_call/task_complete → GET returns
all 3 in step_index DESC order with correct names and payloads.
2. CrashResume_HighestStepIsResumptionPoint: POST steps 0+1 only (crash before
step 2) → GET shows step_index=1 as the resume point; task_complete absent.
3. UpsertIdempotency_LatestPayloadWins: POST same (wf_id, step_name) twice with
different payloads → List returns only the second payload (ON CONFLICT DO UPDATE).
4. PostCascadeDelete_Returns404: simulate post ON-DELETE-CASCADE state (empty
rows) → List returns 404 as expected after workspace deletion.
5. AuthGate_NoToken_Returns401: router-level test with WorkspaceAuth middleware;
POST/GET/DELETE all return 401 without a bearer token (no DB calls made).
workspace-template — _save_checkpoint + 4 Python tests:
- Add async _save_checkpoint() to temporal_workflow.py: POST to the platform
checkpoint endpoint after each activity stage; fully non-fatal (try/except
inside the function, plus defence-in-depth try/except at every call site).
- 4 new pytest cases (test_temporal_workflow.py):
- nonfatal_on_http_error: _save_checkpoint raises HTTPStatusError (500) →
task_receive_activity still returns {"status":"received"}.
- nonfatal_on_network_error: _save_checkpoint raises ConnectError →
llm_call_activity still returns success LLMResult.
- success_path: _save_checkpoint no-op → activity returns correctly;
checkpoint called with correct args.
- standalone_http_error_is_swallowed: real _save_checkpoint function swallows
HTTP 500 from a mocked httpx.AsyncClient; returns None.
All 36 temporal workflow Python tests pass.
Go tests: Go binary not in this container; test file verified for syntax and
against the sqlmock patterns used throughout the handlers package.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents three upgrade strategies for keeping tenant EC2 instances
current with platform-tenant:latest:
- Option A: Rolling restart via CP admin endpoint (coordinated)
- Option B: Sidecar auto-updater cron (implemented, 5 min interval)
- Option C: Blue-green via Worker (zero downtime, future)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents three upgrade strategies for keeping tenant EC2 instances
current with platform-tenant:latest:
- Option A: Rolling restart via CP admin endpoint (coordinated)
- Option B: Sidecar auto-updater cron (implemented, 5 min interval)
- Option C: Blue-green via Worker (zero downtime, future)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Badge was always text-zinc-500; apply blue-500 (>=0.8), zinc-400 (0.5–0.8),
zinc-600 (<0.5) per spec. Add 3 vitest tests for each color tier (725 total).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Badge was always text-zinc-500; apply blue-500 (>=0.8), zinc-400 (0.5–0.8),
zinc-600 (<0.5) per spec. Add 3 vitest tests for each color tier (725 total).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds step-level checkpoint storage so workflows can resume from the
last completed step after a crash or restart without replaying prior work.
- Migration: `workflow_checkpoints` table — workspace_id (FK + CASCADE),
workflow_id, step_name, step_index, completed_at, payload JSONB.
UNIQUE(workspace_id, workflow_id, step_name) + covering index on
(workspace_id, workflow_id, completed_at DESC).
- Handlers (platform/internal/handlers/checkpoints.go):
POST /workspaces/:id/checkpoints — upsert via ON CONFLICT DO UPDATE
GET /workspaces/:id/checkpoints/:wfid — list steps ordered step_index DESC
DELETE /workspaces/:id/checkpoints/:wfid — clear on clean shutdown (404 if none)
- Router: all three routes on the wsAuth group (WorkspaceAuth middleware);
workspace A's token cannot reach workspace B's checkpoints.
- Tests (11 cases, sqlmock + race-safe): upsert-insert, upsert-update,
payload forwarding, list-ordered, list-not-found, rows.Err() → 500,
delete-success, delete-not-found, callerMismatch 403 on all 3 endpoints.
Closes#788. Parent: #583-1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds step-level checkpoint storage so workflows can resume from the
last completed step after a crash or restart without replaying prior work.
- Migration: `workflow_checkpoints` table — workspace_id (FK + CASCADE),
workflow_id, step_name, step_index, completed_at, payload JSONB.
UNIQUE(workspace_id, workflow_id, step_name) + covering index on
(workspace_id, workflow_id, completed_at DESC).
- Handlers (platform/internal/handlers/checkpoints.go):
POST /workspaces/:id/checkpoints — upsert via ON CONFLICT DO UPDATE
GET /workspaces/:id/checkpoints/:wfid — list steps ordered step_index DESC
DELETE /workspaces/:id/checkpoints/:wfid — clear on clean shutdown (404 if none)
- Router: all three routes on the wsAuth group (WorkspaceAuth middleware);
workspace A's token cannot reach workspace B's checkpoints.
- Tests (11 cases, sqlmock + race-safe): upsert-insert, upsert-update,
payload forwarding, list-ordered, list-not-found, rows.Err() → 500,
delete-success, delete-not-found, callerMismatch 403 on all 3 endpoints.
Closes#788. Parent: #583-1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Post-mortem fix: UIUX Designer ran 22 cron fires over 23 hours with
every single response being empty or '(no response generated)'. The
scheduler reported status=ok because the HTTP call succeeded — nobody
caught it until the CEO asked.
Changes:
- Migration 032: adds consecutive_empty_runs INT to workspace_schedules
- scheduler.go: captures response body from ProxyA2ARequest (was _),
checks for empty/sentinel markers via isEmptyResponse(), increments
consecutive_empty_runs on empty ok responses, resets on non-empty.
When consecutive_empty_runs >= 3, sets last_status='stale' with a
descriptive error message.
The 'stale' status is surfaced via:
- GET /admin/schedules/health (merged in #671)
- PM's silence detector (companion fix in org-template PR)
- Maintenance loop response-body sampling (operator-side fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Post-mortem fix: UIUX Designer ran 22 cron fires over 23 hours with
every single response being empty or '(no response generated)'. The
scheduler reported status=ok because the HTTP call succeeded — nobody
caught it until the CEO asked.
Changes:
- Migration 032: adds consecutive_empty_runs INT to workspace_schedules
- scheduler.go: captures response body from ProxyA2ARequest (was _),
checks for empty/sentinel markers via isEmptyResponse(), increments
consecutive_empty_runs on empty ok responses, resets on non-empty.
When consecutive_empty_runs >= 3, sets last_status='stale' with a
descriptive error message.
The 'stale' status is surfaced via:
- GET /admin/schedules/health (merged in #671)
- PM's silence detector (companion fix in org-template PR)
- Maintenance loop response-body sampling (operator-side fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rebase of feat/issue-576-pgvector-semantic-memory onto current main,
preserving the #767 security layer (globalMemoryDelimiter + GLOBAL audit
log) that predates this branch.
Changes layered on top of main:
- Migration 031: embedding vector(1536) column + ivfflat cosine-ops index
(renumbered from 029 — 029/030 were taken by workspace-hibernation and
audit-events)
- Commit: embed-on-write after INSERT, non-fatal on embedding failure
- Search: semantic cosine-distance path when EmbeddingFunc is wired up;
falls back to FTS/ILIKE; GLOBAL delimiter wrapping applies on both paths
- EmbeddingFunc injection pattern; WithEmbedding chainable builder
All security invariants preserved:
- globalMemoryDelimiter wrapping on GLOBAL scope in both semantic + FTS
- GLOBAL write audit log (SHA-256 forensic trail) in Commit
- TestRecallMemory_GlobalScope_HasDelimiter passes
- TestMemoriesCommit_Global_AsRoot passes
- 3 new pgvector tests pass
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
Rebase of feat/issue-576-pgvector-semantic-memory onto current main,
preserving the #767 security layer (globalMemoryDelimiter + GLOBAL audit
log) that predates this branch.
Changes layered on top of main:
- Migration 031: embedding vector(1536) column + ivfflat cosine-ops index
(renumbered from 029 — 029/030 were taken by workspace-hibernation and
audit-events)
- Commit: embed-on-write after INSERT, non-fatal on embedding failure
- Search: semantic cosine-distance path when EmbeddingFunc is wired up;
falls back to FTS/ILIKE; GLOBAL delimiter wrapping applies on both paths
- EmbeddingFunc injection pattern; WithEmbedding chainable builder
All security invariants preserved:
- globalMemoryDelimiter wrapping on GLOBAL scope in both semantic + FTS
- GLOBAL write audit log (SHA-256 forensic trail) in Commit
- TestRecallMemory_GlobalScope_HasDelimiter passes
- TestMemoriesCommit_Global_AsRoot passes
- 3 new pgvector tests pass
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
Addresses all 4 review points from PR #786:
1. Worker resilience: 3-tier cache (in-memory → KV → CP API) with stale
fallback so CP outages are invisible to tenants
2. WebSocket proxying: documented upgradeHeader handling, fallback to
keep Caddy for WS-only if Workers WS is unreliable
3. SG automation: note to auto-update Cloudflare IP ranges, don't hardcode
4. Trusted proxy: X-Forwarded-For / CF-Connecting-IP trust chain documented
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addresses all 4 review points from PR #786:
1. Worker resilience: 3-tier cache (in-memory → KV → CP API) with stale
fallback so CP outages are invisible to tenants
2. WebSocket proxying: documented upgradeHeader handling, fallback to
keep Caddy for WS-only if Workers WS is unreliable
3. SG automation: note to auto-update Cloudflare IP ranges, don't hardcode
4. Trusted proxy: X-Forwarded-For / CF-Connecting-IP trust chain documented
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add org-templates/molecule-dev/system-prompt.md as a canonical org-level
shared-context template for all molecule-dev org agents. The Communication
section explains that /workspace/AGENTS.md is auto-generated at startup from
config.yaml (via agents_md.py / PR #763), describes the AAIF format it
follows, explains the GET /workspace/AGENTS.md peer-discovery contract, and
tells agents to keep their config.yaml name/role/description accurate as the
sole source of truth.
Also restructure the /org-templates/ gitignore rule from a hard directory-ignore
to a content-glob pattern so this specific reference template can be tracked
while all other cloned standalone-repo content remains ignored.
Co-authored-by: Molecule AI Documentation Specialist <documentation-specialist@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Add org-templates/molecule-dev/system-prompt.md as a canonical org-level
shared-context template for all molecule-dev org agents. The Communication
section explains that /workspace/AGENTS.md is auto-generated at startup from
config.yaml (via agents_md.py / PR #763), describes the AAIF format it
follows, explains the GET /workspace/AGENTS.md peer-discovery contract, and
tells agents to keep their config.yaml name/role/description accurate as the
sole source of truth.
Also restructure the /org-templates/ gitignore rule from a hard directory-ignore
to a content-glob pattern so this specific reference template can be tracked
while all other cloned standalone-repo content remains ignored.
Co-authored-by: Molecule AI Documentation Specialist <documentation-specialist@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds path-filter table so developers and agents know which files
trigger which CI jobs, and that docs-only PRs skip everything.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds path-filter table so developers and agents know which files
trigger which CI jobs, and that docs-only PRs skip everything.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI now detects which paths changed and skips irrelevant jobs:
- Platform (Go): only runs when platform/** changes
- Canvas (Next.js): only runs when canvas/** changes
- Python Lint: only runs when workspace-template/** changes
- Shellcheck: only runs when tests/e2e/** or scripts/** change
- E2E API: only runs when platform/** or tests/e2e/** change
Docs-only PRs (*.md, docs/**) skip all 5 jobs, saving ~15 min of
runner time per PR. Uses dorny/paths-filter for the CI workflow and
native paths: filter for the E2E workflow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI now detects which paths changed and skips irrelevant jobs:
- Platform (Go): only runs when platform/** changes
- Canvas (Next.js): only runs when canvas/** changes
- Python Lint: only runs when workspace-template/** changes
- Shellcheck: only runs when tests/e2e/** or scripts/** change
- E2E API: only runs when platform/** or tests/e2e/** change
Docs-only PRs (*.md, docs/**) skip all 5 jobs, saving ~15 min of
runner time per PR. Uses dorny/paths-filter for the CI workflow and
native paths: filter for the E2E workflow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(eco-watch): add BeeAI ACP + Claw Code — 2026-04-17
BeeAI ACP (i-am-bee/acp, IBM) — REST/OpenAPI agent comm protocol, direct
A2A alternative; Copilot CLI ACP support already in preview. GH #777 filed
for TR comparison vs A2A.
Claw Code (ultraworkers/claw-code) — 100k+★ Rust+Python clean-room rewrite
of Claude Code architecture; architectural reference + competitive signal for
molecule-ai-workspace-template-claude-code.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(eco-watch): mark BeeAI ACP as archived — A2A won consolidation
IBM archived i-am-bee/acp on Aug 27, 2025; contributed to AAIF/A2A
working group. No bridge or shim needed — Molecule's A2A bet vindicated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Molecule AI Research Lead <research-lead@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(eco-watch): add BeeAI ACP + Claw Code — 2026-04-17
BeeAI ACP (i-am-bee/acp, IBM) — REST/OpenAPI agent comm protocol, direct
A2A alternative; Copilot CLI ACP support already in preview. GH #777 filed
for TR comparison vs A2A.
Claw Code (ultraworkers/claw-code) — 100k+★ Rust+Python clean-room rewrite
of Claude Code architecture; architectural reference + competitive signal for
molecule-ai-workspace-template-claude-code.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(eco-watch): mark BeeAI ACP as archived — A2A won consolidation
IBM archived i-am-bee/acp on Aug 27, 2025; contributed to AAIF/A2A
working group. No bridge or shim needed — Molecule's A2A bet vindicated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Molecule AI Research Lead <research-lead@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds Phase 33 plan and architecture doc for replacing per-tenant DNS
records with a wildcard DNS + Cloudflare Worker proxy pattern.
Eliminates: DNS propagation delays, NXDOMAIN caching, per-instance
Let's Encrypt, Caddy on EC2. Same pattern used by Vercel, Railway,
Fly.io, WordPress, n8n.
4-phase migration: deploy Worker → stop creating DNS records →
remove Caddy from EC2 → cleanup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
