From 475bfa067feaee54a500e83bc6722dd99f8bc6e9 Mon Sep 17 00:00:00 2001 From: "molecule-ai[bot]" <276602405+molecule-ai[bot]@users.noreply.github.com> Date: Tue, 21 Apr 2026 07:50:11 +0000 Subject: [PATCH] docs: add docs/architecture/molecule-technical-doc.md --- .../architecture/molecule-technical-doc.md | 1161 +++++++++++++++++ 1 file changed, 1161 insertions(+) create mode 100644 content/docs/architecture/molecule-technical-doc.md diff --git a/content/docs/architecture/molecule-technical-doc.md b/content/docs/architecture/molecule-technical-doc.md new file mode 100644 index 0000000..dd38d4a --- /dev/null +++ b/content/docs/architecture/molecule-technical-doc.md @@ -0,0 +1,1161 @@ +# Molecule AI — Comprehensive Technical Documentation + +> Definitive technical reference for the Molecule AI Agent Team platform. +> Based on a full non-invasive scan of the [molecule-monorepo](https://github.com/Molecule-AI/molecule-monorepo) repository. + +--- + +## Table of Contents + +1. [Executive Summary](#1-executive-summary) +2. [Product Positioning](#2-product-positioning) +3. [System Architecture](#3-system-architecture) +4. [Database Schema](#4-database-schema) +5. [Workspace Lifecycle](#5-workspace-lifecycle) +6. [Communication Rules](#6-communication-rules) +7. [Platform API Routes](#7-platform-api-routes) +8. [A2A Protocol](#8-a2a-protocol) +9. [Hierarchical Memory Architecture](#9-hierarchical-memory-architecture) +10. [Runtime Tier System](#10-runtime-tier-system) +11. [Provisioning & Container Lifecycle](#11-provisioning--container-lifecycle) +12. [Workspace Runtime](#12-workspace-runtime) +13. [Skills System](#13-skills-system) +14. [Bundle System](#14-bundle-system) +15. [Canvas UI](#15-canvas-ui) +16. [Tools & Capabilities](#16-tools--capabilities) +17. [Coordinator Pattern](#17-coordinator-pattern) +18. [Codebase Structure](#18-codebase-structure) +19. [Key Design Patterns](#19-key-design-patterns) +20. [Docker Compose Orchestration](#20-docker-compose-orchestration) +21. [Environment Variables](#21-environment-variables) +22. [Recent Feature Highlights](#22-recent-feature-highlights) +23. [Known Gaps & Backlog](#23-known-gaps--backlog) +24. [Licensing & Commercialization Path](#24-licensing--commercialization-path) +25. [OSS Growth Research](#25-oss-growth-research) +26. [Technical Debt & Constraints](#26-technical-debt--constraints) +27. [Production Deployment](#27-production-deployment) +28. [MCP Server & Integrations](#28-mcp-server--integrations) +29. [Summary Statistics](#29-summary-statistics) +30. [Vision: From Agent Teams to Robot Teams](#30-vision-from-agent-teams-to-robot-teams) + +--- + +## 1. Executive Summary + +Molecule AI is an **org-native control plane for heterogeneous AI agent teams**. It is not a workflow builder or a replacement for agent frameworks; rather, it is the operational and organizational layer that sits above multiple runtime frameworks and provides: + +- **Workspace-as-role abstraction** (not task nodes) +- **Hierarchy-driven topology** (org chart = communication paths) +- **Hierarchical Memory Architecture (HMA)** with LOCAL/TEAM/GLOBAL scopes +- **A2A (Agent-to-Agent)** direct inter-workspace communication via JSON-RPC 2.0 +- **Canvas-based visual team building** with drag-to-nest hierarchy +- **Comprehensive control plane operations** — registry, heartbeats, lifecycle, approvals, secrets, traces, bundles + +Six runtime adapters ship production-ready on `main`: LangGraph, DeepAgents, Claude Code, CrewAI, AutoGen, OpenClaw. + +--- + +## 2. Product Positioning + +### Core Narrative + +**One-liner**: *"Molecule AI is the org-native control plane for heterogeneous AI agent teams."* + +### Five Key Differentiators + +| # | Principle | Implication | +|---|-----------|-------------| +| 1 | **Workspace = role, not task** | Internal AI model can swap, but organizational identity persists across model/framework changes | +| 2 | **Org chart = topology** | Hierarchy determines communication boundaries — no manual edge wiring needed | +| 3 | **Heterogeneous runtime support** | 6 adapters shipped; teams choose freely without forced standardization | +| 4 | **Memory follows org boundaries** | HMA prevents over-sharing, aligns data isolation with organizational structure | +| 5 | **Skill evolution loop** | memory → signal → skill → hot-reload → operational improvement (self-improving flywheel) | + +### What Molecule AI Is NOT + +| Category | Examples | How Molecule AI Differs | +|----------|---------|---------------------| +| Workflow builders | n8n, Windmill, Temporal | Molecule AI models **roles**, not tasks | +| Agent frameworks | LangGraph, CrewAI, AutoGen | Molecule AI sits **above** frameworks as runtime adapters | +| Coding agents | Claude Code, Cursor, Codex | Molecule AI runs coding agents as **workspace roles** alongside other types | +| Chat UIs | ChatGPT, Claude.ai | Molecule AI is **operational infrastructure**, not a conversation interface | + +--- + +## 3. System Architecture + +### System Boundary Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Canvas (Next.js 15 · port 3000) │ +│ React Flow + Zustand + WebSocket │ +│ Visual drag-to-nest org chart · 10-tab ops panel │ +└──────────────────┬──────────────────────────────────────────┘ + │ HTTP + WebSocket +┌──────────────────▼──────────────────────────────────────────┐ +│ Platform (Go / Gin · port 8080) │ +│ Control plane: workspace CRUD, registry, discovery, │ +│ A2A proxy, activity, memory APIs, secrets, approvals │ +└─────────┬────────────────────────────────────┬──────────────┘ + │ │ + Postgres 16 Redis 7 + (internal: 5432) (internal: 6379) + +┌─────────────────────────────────────────────────────────────┐ +│ Workspace Runtime (Python 3.11+ Docker image) │ +│ Pluggable adapters: LangGraph, DeepAgents, Claude Code, │ +│ CrewAI, AutoGen, OpenClaw │ +│ A2A protocol server · heartbeat · skills · HMA memory │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ Langfuse (self-hosted · ClickHouse + Postgres backend) │ +│ OpenTelemetry traces for every LLM call │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Network Model + +| Path | Protocol | Purpose | +|------|----------|---------| +| Canvas ↔ Platform | HTTP REST + WebSocket | UI operations + real-time event fanout | +| Platform ↔ Postgres | TCP | Source of truth for all durable state | +| Platform ↔ Redis | TCP | Ephemeral state (liveness TTL), caching, pub/sub | +| Workspace ↔ Workspace | HTTP (A2A JSON-RPC 2.0) | Direct peer-to-peer, **platform not in data path** | +| Workspace → Langfuse | HTTP | Automatic OpenTelemetry tracing | +| Docker Network | `molecule-monorepo-net` | Internal-only by default, no exposed DB/Redis ports | + +### Core Components + +**1. Canvas (Next.js 15)** +- React Flow for visual workspace graph +- Zustand for state management +- WebSocket for real-time updates +- 10-tab side panel: Chat, Activity, Details, Skills, Terminal, Config, Files, Memory, Traces, Events +- Drag-to-nest team building +- Bundle import/export via drag-and-drop +- Empty state with template palette + onboarding wizard + +**2. Platform (Go 1.25+ / Gin)** +- Gin-based REST API + WebSocket hub +- Workspace lifecycle management (CRUD + pause/resume/restart) +- Registry and heartbeat system (30s default) +- Hierarchy-aware access control (`CanCommunicate()`) +- A2A proxy for browser-safe inter-workspace communication +- Event broadcasting (Redis pub/sub → WebSocket fanout) +- Docker provisioner with T1–T4 tier enforcement +- Activity logging with configurable retention (default 7 days) +- Secrets management (AES-256-GCM encryption) +- File, terminal, bundle, template, traces APIs +- Langfuse integration +- Prometheus metrics endpoint + +**3. Workspace Runtime (Python 3.11+)** +- Unified `workspace/` Docker image +- Adapter-driven execution (6 adapters) +- A2A server via Uvicorn +- Heartbeat loop (30s default) +- Skill hot-reload system (~3 second propagation) +- Memory tools with HMA scope support +- Approval/human-in-the-loop integration +- Activity reporting +- Awareness namespace integration (optional) +- Plugin-mounted shared rules and skills + +**4. Infrastructure** +- **Postgres 16**: Source of truth (workspaces, events, activity, secrets, memories) +- **Redis 7**: Ephemeral state (liveness TTL 60s), URL caching, pub/sub +- **Langfuse 2.x**: LLM tracing and observability (self-hosted, ClickHouse backend) +- **Docker**: Workspace provisioning with T1–T4 tier system +- **LiteLLM proxy** (optional): Unified API for multiple model providers +- **Ollama** (optional): Local LLM models + +--- + +## 4. Database Schema + +11 migration files in `workspace-server/migrations/`. + +### Core Tables + +| Table | Purpose | Key Columns | +|-------|---------|-------------| +| `workspaces` | Current state registry | `id`, `name`, `role`, `tier` (1-4), `status`, `parent_id`, `agent_card` (JSONB), `url`, `forwarded_to`, `last_heartbeat_at`, `last_error_rate`, `active_tasks`, `uptime_seconds`, `current_task`, `runtime` | +| `agents` | Agent assignment history | `workspace_id`, `model`, `status`, `removed_at`, `removal_reason` | +| `workspace_secrets` | Encrypted credentials | `workspace_id`, `key`, `encrypted_value` (BYTEA, AES-256-GCM) | +| `agent_memories` | HMA-scoped memory | `workspace_id`, `content`, `scope` (LOCAL/TEAM/GLOBAL) | +| `structure_events` | **Immutable** event log (APPEND-ONLY, never UPDATE/DELETE) | `event_type`, `workspace_id`, `agent_id`, `target_id`, `payload` (JSONB) | +| `activity_logs` | Operational activity with retention | `workspace_id`, `activity_type`, `source_id`, `target_id`, `method`, `request_body`, `response_body`, `duration_ms`, `status`, `error_detail` | +| `canvas_layouts` | Node visual positions | `workspace_id`, `x`, `y`, `collapsed` | +| `canvas_viewport` | Canvas pan/zoom state | Single row, upserted | + +### Redis Key Patterns + +| Key Pattern | Value | TTL | Purpose | +|-------------|-------|-----|---------| +| `ws:{id}` | `"online"` | 60s | Liveness detection (heartbeat refreshes) | +| `ws:{id}:url` | Host-mapped URL | 5min | URL cache for external discovery | +| `ws:{id}:internal_url` | Docker-internal URL | — | Container-to-container discovery | +| `events:broadcast` | pub/sub channel | — | Event fanout to WebSocket hub | + +--- + +## 5. Workspace Lifecycle + +### State Machine + +``` +provisioning → online ↔ degraded + ↓ ↓ ↓ + failed offline offline + ↓ ↓ + retry (auto-restart) + +↓ (any state) +paused → (user resumes) → provisioning + +↓ (any state) +removed +``` + +### Status Definitions + +| Status | Meaning | Canvas Indicator | +|--------|---------|-----------------| +| `provisioning` | Waiting for first heartbeat | Spinner | +| `online` | Heartbeat received, reachable | Green dot | +| `degraded` | Online but `error_rate ≥ 0.5` | Yellow node with warning | +| `offline` | Heartbeat TTL expired, unreachable | Gray node | +| `paused` | User paused, container stopped, config preserved | Indigo badge | +| `failed` | Provisioning timeout or launch error | Red node + retry button | +| `removed` | Deleted, kept for event log history | Node removed from Canvas | + +### Health Detection (Three Layers) + +| Layer | Mechanism | Interval | Trigger | +|-------|-----------|----------|---------| +| **Passive** | Redis TTL expiry | 60s heartbeat key | Liveness monitor callback | +| **Proactive** | Docker API poll | Every 15s | Health sweep goroutine | +| **Reactive** | A2A proxy connection error | On-demand | `provisioner.IsRunning()` check | + +All three layers call `onWorkspaceOffline()` → broadcast `WORKSPACE_OFFLINE` + auto-restart. + +### Cascade Behavior + +- **Pause**: Pausing a parent cascades to all children. Children of a paused parent cannot be individually resumed. +- **Delete**: Removes container, cleans memory (DB rows, Redis keys). Structure events and Agent Card history are **never** deleted. + +--- + +## 6. Communication Rules + +### Hierarchy = Topology + +The `CanCommunicate()` function is the single source of truth for all access control. + +| Direction | Allowed | Example | +|-----------|---------|---------| +| Sibling ↔ Sibling | **YES** | Marketing Agent ↔ Developer PM | +| Parent → Child | **YES** | Developer PM → Frontend Agent | +| Child → Parent | **YES** | Frontend Agent → Developer PM | +| Skip levels | **NO** | Frontend Agent → Business Core (403) | +| Cross-team | **NO** | Frontend Agent → Operations Team (403) | + +### Access Check Algorithm + +``` +IF caller.parent_id == target.parent_id → ALLOW (siblings) +ELIF both parent_id IS NULL → ALLOW (root-level siblings) +ELIF caller.ID == target.parent_id → ALLOW (parent→child) +ELIF target.ID == caller.parent_id → ALLOW (child→parent) +ELSE → DENY (403 Forbidden) +``` + +This same logic governs: A2A delegation, memory scope enforcement, activity visibility, approval routing, and WebSocket event fanout. + +--- + +## 7. Platform API Routes + +### Workspace Lifecycle (8 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `POST` | `/workspaces` | Create and provision new workspace | +| `GET` | `/workspaces` | List all with canvas layout data | +| `GET` | `/workspaces/:id` | Get single workspace | +| `PATCH` | `/workspaces/:id` | Update name, role, tier, runtime, parent | +| `DELETE` | `/workspaces/:id` | Remove workspace | +| `POST` | `/workspaces/:id/restart` | Restart container | +| `POST` | `/workspaces/:id/pause` | Pause with cascade to children | +| `POST` | `/workspaces/:id/resume` | Resume from pause | + +### Registry & Discovery (5 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `POST` | `/registry/register` | Workspace self-registration on startup | +| `POST` | `/registry/heartbeat` | Liveness + current task + error rate (30s interval) | +| `POST` | `/registry/update-card` | Push Agent Card updates (skills, capabilities) | +| `GET` | `/registry/discover/:id` | Resolve workspace URL (hierarchy-validated) | +| `GET` | `/registry/:id/peers` | List reachable peers for workspace | + +### Memory (6 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `POST` | `/workspaces/:id/memories` | Commit HMA-scoped memory (LOCAL/TEAM/GLOBAL) | +| `GET` | `/workspaces/:id/memories` | Search scoped memories with query | +| `DELETE` | `/workspaces/:id/memories/:memoryId` | Delete specific memory entry | +| `GET` | `/workspaces/:id/memory` | List key/value workspace memory | +| `POST` | `/workspaces/:id/memory` | Upsert key/value pair (optional TTL) | +| `DELETE` | `/workspaces/:id/memory/:key` | Delete key/value entry | + +### Secrets & Config (5 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `GET` | `/workspaces/:id/secrets` | Get merged workspace + global secrets | +| `POST` | `/workspaces/:id/secrets` | Upsert workspace secret (**triggers auto-restart**) | +| `DELETE` | `/workspaces/:id/secrets/:key` | Delete secret (**triggers auto-restart**) | +| `GET` | `/workspaces/:id/config` | Get workspace config.yaml | +| `PATCH` | `/workspaces/:id/config` | Update workspace config | + +### A2A & Activity (5 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `POST` | `/workspaces/:id/a2a` | Proxy A2A request to target workspace | +| `GET` | `/workspaces/:id/activity` | List activity rows (filterable) | +| `POST` | `/workspaces/:id/activity` | Report activity from workspace | +| `POST` | `/workspaces/:id/notify` | Emit user-facing notification | +| `GET` | `/workspaces/:id/session-search` | Search recent activity + memory for contextual recall | + +### Team & Hierarchy (2 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `POST` | `/workspaces/:id/expand` | Expand workspace into team (become coordinator) | +| `POST` | `/workspaces/:id/collapse` | Collapse team back to single workspace | + +### Files, Terminal, Templates, Bundles (8 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `GET` | `/workspaces/:id/files[/*path]` | List or read files | +| `PUT` | `/workspaces/:id/files/*path` | Write file | +| `DELETE` | `/workspaces/:id/files/*path` | Delete file | +| `WS` | `/workspaces/:id/terminal` | WebSocket terminal session into container | +| `GET` | `/bundles/export/:id` | Export workspace as `.bundle.json` | +| `POST` | `/bundles/import` | Import workspace bundle (recursive) | +| `GET` | `/templates` | List available workspace templates | +| `POST` | `/templates/import` | Import custom template folder | + +### Observability & Real-Time (5 endpoints) + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| `GET` | `/health` | Health check | +| `GET` | `/metrics` | Prometheus metrics (v0.0.4 format) | +| `GET` | `/workspaces/:id/traces` | Langfuse trace links | +| `GET` | `/events[/:workspaceId]` | Event stream (SSE) | +| `WS` | `/ws` | WebSocket hub for real-time event fanout | + +--- + +## 8. A2A Protocol + +### Message Format (JSON-RPC 2.0 over HTTP) + +```json +{ + "jsonrpc": "2.0", + "id": "task-123", + "method": "message/send", + "params": { + "message": { + "role": "user", + "parts": [{"kind": "text", "text": "Build the login feature"}], + "messageId": "msg-456" + } + } +} +``` + +### Two Call Modes + +| Mode | Method | Use Case | +|------|--------|----------| +| Synchronous | `message/send` | Short tasks, immediate response | +| Streaming | `message/sendSubscribe` | Long tasks, SSE progress updates | + +### Discovery Flow + +1. Caller queries `GET /registry/discover/:targetId` with `X-Workspace-ID` header +2. Platform validates `CanCommunicate(caller, target)` — returns 403 if denied +3. Returns Docker-internal URL (workspace caller) or host-mapped URL (Canvas/external caller) +4. Caller sends A2A message **directly** to target (peer-to-peer) +5. Target processes task and responds + +### Task State Machine + +``` +submitted → working → completed + → failed + → canceled + → input-required → working (caller provides input) +``` + +### Authentication + +- **MVP (current)**: Discovery-time validation only. Direct A2A calls are unauthenticated (acceptable for self-hosted Docker network isolation). +- **Post-MVP**: Platform-issued short-lived signed tokens scoped to caller/target pair. + +--- + +## 9. Hierarchical Memory Architecture + +### Three Scopes + +| Scope | Visibility | Write Access | Use Case | +|-------|-----------|-------------|----------| +| **LOCAL** | This workspace only | Self | Private scratch facts, reasoning, working state | +| **TEAM** | Parent + children + siblings | Self | Handoffs, coordination, team-level knowledge | +| **GLOBAL** | Readable by all workspaces | Root only | Org-wide policies, standards, institutional knowledge | + +### Four Memory Surfaces + +| Surface | Storage | Endpoint | Purpose | +|---------|---------|----------|---------| +| **Scoped agent memory** | `agent_memories` table | `POST /workspaces/:id/memories` | HMA-backed distributed memory with scope enforcement | +| **Key/value workspace memory** | `workspace_memory` table | `POST /workspaces/:id/memory` | Simple structured state, UI-visible, optional TTL | +| **Activity recall** | `activity_logs` + `agent_memories` | `GET /workspaces/:id/session-search` | "What just happened?" contextual recall | +| **Awareness-backed** | External service | Same tool interface | When `AWARENESS_URL` + `AWARENESS_NAMESPACE` configured | + +### Memory → Skill Compounding Flywheel + +``` +Task execution + → Durable insight captured in LOCAL/TEAM memory + → Repeated success patterns detected (repetition signal) + → Memory row promoted → SKILL.md package created + → Hot-reload (~3 seconds) → skill injected into live runtime + → Agent Card updated → broadcast to peers via WebSocket + → Future tasks use promoted skill → faster + more reliable + → Organization becomes more capable over time +``` + +Key property: promotion events are **visible** in activity logs. Skills are **inspectable** in Canvas Skills tab. This is not hidden prompt inflation. + +--- + +## 10. Runtime Tier System + +| Tier | Name | Container Flags | Use Case | +|------|------|-----------------|----------| +| **T1** | Sandboxed | Read-only rootfs, tmpfs /tmp, 512 MiB, no `/workspace` mount | Untrusted code, text-only analysis | +| **T2** | Standard (default) | Read-write, 512 MiB, 1 CPU, `/workspace` mount | Most agent workloads | +| **T3** | Privileged | `--privileged`, `--pid=host`, Docker network access | Internal tooling, elevated operations | +| **T4** | Full Access | T3 + `--network=host` + Docker socket mount | System-level orchestration, DevOps | + +Unknown tier values default to T2 for safety. Applied via `provisioner.ApplyTierConfig()` during container creation. + +--- + +## 11. Provisioning & Container Lifecycle + +### Docker Networking + +- All containers join `molecule-monorepo-net` private network +- Container naming: `ws-{workspace_id[:12]}` +- Ephemeral host port binding: `127.0.0.1:0→8000/tcp` + +### URL Resolution + +| Caller | URL Type | Example | +|--------|----------|---------| +| Workspace (container) | Docker-internal | `http://ws-{id}:8000` | +| Canvas (browser) | Host-mapped | `http://127.0.0.1:{ephemeral_port}` | + +### Container Cleanup on Delete + +1. Docker container stopped and removed +2. Memory cleaned (DB rows, Redis keys) +3. Status set to `removed` +4. `WORKSPACE_REMOVED` event written to structure_events +5. Structure events and Agent Card history **never** deleted (audit trail) + +--- + +## 12. Workspace Runtime + +### Entry Point: `workspace/main.py` + +**Startup Sequence** (10 steps): + +1. Initialize telemetry (OpenTelemetry, no-op if packages absent) +2. Load `config.yaml` into `WorkspaceConfig` dataclass +3. Run preflight validation (model availability, skills, configs) +4. Create `HeartbeatLoop` for background task tracking +5. Resolve adapter from `runtime` field in config +6. Run adapter `setup()` and `create_executor()` +7. Build Agent Card from loaded skills + runtime capabilities +8. Register: `POST /registry/register` with workspace ID + Agent Card +9. Start heartbeat loop (30s interval) + skill hot-reload watcher +10. Serve A2A over Uvicorn on configured port + +### Runtime Configuration Schema (`config.yaml`) + +```yaml +name: "Workspace Name" +description: "" +version: "1.0.0" +tier: 2 # 1=sandboxed, 2=standard, 3=privileged, 4=full-host +model: "anthropic:claude-sonnet-4-6" # provider:model syntax +runtime: "langgraph" # langgraph | deepagents | claude-code | crewai | autogen | openclaw +runtime_config: # Runtime-specific settings + command: "claude" # For CLI runtimes + args: [] + auth_token_file: ".auth-token" + timeout: 0 + model: "" # Override model just for this runtime +skills: ["skill1", "skill2"] # Folder names under skills/ +tools: ["web_search", "filesystem"] # Built-in tool names +prompt_files: ["system-prompt.md"] # Additional prompt text files +shared_context: [] # Files from parent workspace + +a2a: + port: 8000 + streaming: true + push_notifications: true + +delegation: + retry_attempts: 3 + retry_delay: 5.0 + timeout: 120.0 + escalate: true + +sandbox: + backend: "subprocess" # subprocess | docker + memory_limit: "256m" + timeout: 30 + +rbac: + roles: ["operator"] + allowed_actions: {} + +hitl: + channels: + - type: "dashboard" + default_timeout: 300 + bypass_roles: [] + +governance: + enabled: false + policy_mode: "audit" # audit | permissive | strict + policy_file: "" + +security_scan: + mode: "warn" # warn | block | off + +compliance: + mode: "owasp_agentic" + prompt_injection: "detect" # detect | block + max_tool_calls_per_task: 50 + max_task_duration_seconds: 300 +``` + +### Six Runtime Adapters + +| Adapter | Core Strength | Image Tag | +|---------|--------------|-----------| +| **LangGraph** | Graph-based state machine, tool use, streaming | `workspace-template:langgraph` | +| **DeepAgents** | Deep planning, multi-step task decomposition | `workspace-template:deepagents` | +| **Claude Code** | Native coding workflows, CLI continuity, OAuth auth | `workspace-template:claude-code` | +| **CrewAI** | Role-based crews, structured task orchestration | `workspace-template:crewai` | +| **AutoGen** | Multi-agent conversations, explicit strategies | `workspace-template:autogen` | +| **OpenClaw** | CLI-native runtime, own session model | `workspace-template:openclaw` | + +**Branch-level WIP**: NemoClaw (NVIDIA T4 + Docker socket) on `feat/nemoclaw-t4-docker`. + +Each adapter implements `setup()` + `create_executor()`. The base adapter provides shared infrastructure: system prompt assembly, skill loading, tool registration, coordinator detection, plugin injection. + +--- + +## 13. Skills System + +### Three Capability Sources + +1. **Workspace-local skills**: `skills//SKILL.md` + `tools/` directory +2. **Plugin-mounted rules**: `/plugins` volume (read-only), shared across all workspaces +3. **Built-in tools**: delegation, approval, memory, sandbox, telemetry, audit, compliance, governance + +### Skill Directory Structure + +``` +skills/generate-seo-page/ +├── SKILL.md # YAML frontmatter + natural language instructions +├── tools/ +│ ├── write_page.py # @tool-decorated Python functions +│ └── check_gsc.py +├── examples/ # Few-shot examples +├── templates/ # Reference files (HTML, etc.) +└── links.yaml # External resource URLs +``` + +### SKILL.md Frontmatter + +```yaml +--- +name: "Generate SEO Landing Page" +description: "Create SEO-optimized bilingual landing pages" +version: "1.0.0" +tags: ["seo", "content", "bilingual"] +examples: ["Create a Vancouver renovation page in EN/ZH"] +requires: + env: ["GSC_CREDENTIALS"] + bins: ["jq"] +--- + +# Agent instructions follow in natural language... +``` + +### Hot-Reload Pipeline + +| Step | Action | Timing | +|------|--------|--------| +| 1 | File watcher detects change in `skills/` | 2s debounce | +| 2 | Reload skill metadata + tool Python modules | Immediate | +| 3 | Rebuild agent tools and Agent Card | ~100ms | +| 4 | Broadcast updated card via WebSocket | ~50ms | +| 5 | Peer system prompts rebuilt with new awareness | ~500ms | +| **Total** | End-to-end propagation | **~3 seconds** | + +--- + +## 14. Bundle System + +### Bundle Format (`.bundle.json`) + +```json +{ + "schema": "1.0", + "id": "seo-agent-vancouver", + "name": "Vancouver SEO Agent", + "tier": 1, + "model": "anthropic:claude-sonnet-4-6", + "system_prompt": "...full prompt text...", + "skills": [{ + "id": "generate-seo-page", + "name": "Generate SEO Landing Page", + "files": { + "SKILL.md": "---\nname: ...", + "tools/write_page.py": "def write_page(...):\n ..." + } + }], + "tools": [{"id": "web_search", "config": {}}], + "prompts": {"prompts/page-generation.md": "..."}, + "sub_workspaces": [], + "agent_card": {"...": "A2A card snapshot"}, + "author": "molecule", + "version": "1.2.0" +} +``` + +### Inclusion/Exclusion Rules + +| Included | Excluded | +|----------|----------| +| Full system prompt text | API keys / secrets | +| All skill files (inlined) | Memory / conversation history | +| Prompt templates + assets | Database data | +| Tool configurations | Runtime state | +| Sub-workspace bundles (recursive) | | +| Agent Card snapshot | | + +### Workflow + +- **Export**: Right-click workspace → "Export as bundle" → `.bundle.json` download +- **Import**: Drag `.bundle.json` onto Canvas → `POST /bundles/import` → recursive provisioning → new IDs → `source_bundle_id` traces lineage + +--- + +## 15. Canvas UI + +### Tech Stack + +| Layer | Technology | +|-------|-----------| +| Framework | Next.js 15 (App Router) | +| Graph | React Flow v12 (`@xyflow/react`) | +| State | Zustand | +| Styling | TailwindCSS v4 | +| Real-time | Native WebSocket API | + +### Core Interactions + +- **Drag-to-Nest**: Drag workspace over another → overlap detection → highlight → drop → update `parent_id` +- **Right-Click Menu**: Open Details/Chat/Terminal, Restart, Duplicate, Export Bundle, Expand/Collapse Team, Extract from Team, Delete +- **Template Palette**: Empty state shows up to 6 templates + "Create blank workspace" +- **Onboarding Wizard**: 4-step guided setup tracked in localStorage + +### 10-Tab Operations Panel + +| # | Tab | Function | +|---|-----|----------| +| 1 | **Chat** | A2A conversational interface with session history (last 20 messages) | +| 2 | **Activity** | Rich operation log — A2A messages, task updates, logs, skill promotions (filterable) | +| 3 | **Details** | Workspace metadata, runtime summary, status, Agent Card, restart/pause controls, peer list | +| 4 | **Skills** | Live skill display from Agent Card — metadata, tags, examples | +| 5 | **Terminal** | WebSocket shell into workspace container | +| 6 | **Config** | Structured YAML editor — runtime, skills, tools, A2A, delegation, sandbox settings | +| 7 | **Files** | File browser + editor for /configs, /workspace, /home, /plugins | +| 8 | **Memory** | Scoped memory view (LOCAL/TEAM/GLOBAL) + key/value workspace memory with TTL | +| 9 | **Traces** | Langfuse trace viewer — every LLM call with input/output/tokens/cost | +| 10 | **Events** | Structure event stream — real-time workspace change log | + +### Real-Time Architecture + +| Phase | Mechanism | +|-------|-----------| +| Initial load | `GET /workspaces` → Zustand store hydration | +| Live updates | WebSocket events → `applyEvent()` → instant Canvas re-render | +| Position persistence | `onNodeDragStop` → `PATCH /workspaces/:id` with x, y | +| Error recovery | Error boundary with reload button + hydration retry banner | + +--- + +## 16. Tools & Capabilities + +### Workspace Tools (`workspace/builtin_tools/`) + +| Tool File | Purpose | RBAC | +|-----------|---------|------| +| `memory.py` | HMA memory `commit_memory()` / `search_memory()` | memory.write, memory.read | +| `delegation.py` | A2A delegation to peer workspaces with retry + tracing | delegate permission | +| `approval.py` | Human-in-the-loop approval flow with polling/WebSocket | approve permission | +| `audit.py` | RBAC enforcement + audit trail logging | audit enforcement | +| `compliance.py` | OWASP Agentic compliance checks | compliance check | +| `governance.py` | Microsoft Agent Governance Toolkit integration | policy evaluation | +| `hitl.py` | Multi-channel HITL (dashboard, Slack, email) | hitl.bypass_roles | +| `sandbox.py` | Code execution (subprocess or Docker backend) | sandbox access | +| `telemetry.py` | OpenTelemetry span creation and tracing | trace emission | +| `awareness_client.py` | Awareness namespace memory wrapper | memory scope | +| `security_scan.py` | CVE and security scanning (pip-audit/Snyk) | security audit | +| `temporal_workflow.py` | Temporal.io workflow integration | workflow engine | +| `a2a_tools.py` | A2A delegation helpers and route resolution | delegate/receive | + +### Built-In MCP Tools (from `.mcp.json`) + +| Server | Purpose | +|--------|---------| +| `molecule` | 20+ platform management tools (workspace CRUD, chat, memory, teams, secrets, files, approvals) | +| `awareness-memory` | Persistent cross-session memory via Awareness SDK | + +--- + +## 17. Coordinator Pattern + +### How Team Expansion Works + +1. Workspace "expands into team" → becomes **coordinator** (team lead) +2. Coordinator fetches children's Agent Cards to understand capabilities +3. For each incoming task: analyzes → selects best-suited child → delegates via A2A +4. Aggregates responses when tasks span multiple children +5. Falls back to self-handling only if no child suitable + +### Key Properties + +- **Enforcement**: Coordinators **cannot** do direct work — all execution delegated to children +- **Recursive**: Child workspaces can themselves expand into teams (unlimited depth) +- **Transparent**: Upstream parent doesn't need to know if child is single agent or team of fifty +- **Detectable**: `coordinator.py` checks `get_children()` — if children exist, coordinator mode activates + +--- + +## 18. Codebase Structure + +### Python Runtime (95 files) + +``` +workspace/ +├── main.py # Entry point (startup sequence) +├── config.py # Config parsing → dataclasses (120+ lines) +├── heartbeat.py # 30s heartbeat loop +├── preflight.py # Startup validation +├── plugins.py # Plugin rule/skill injection +├── coordinator.py # Team lead routing +├── prompt.py # System prompt builder +├── adapters/ +│ ├── __init__.py # Adapter registry +│ ├── base.py # BaseAdapter interface +│ ├── shared_runtime.py # Shared execution logic +│ ├── langgraph/adapter.py +│ ├── deepagents/adapter.py +│ ├── claude_code/adapter.py +│ ├── crewai/adapter.py +│ ├── autogen/adapter.py +│ └── openclaw/adapter.py +├── tools/ # 14 tool files +├── skills/ +│ ├── loader.py # SKILL.md parser + tool loader +│ └── watcher.py # Hot-reload file watcher +└── tests/ # 148 pytest tests +``` + +### Go Platform (94 files) + +``` +workspace-server/ +├── cmd/ +│ ├── server/main.go # Entry point + dependency injection +│ └── cli/ # molecli TUI dashboard +├── internal/ +│ ├── handlers/ # 26 HTTP handler files (26k+ lines) +│ ├── registry/ # 6 files — workspace registry + access control +│ ├── provisioner/ # 8 files — Docker provisioning + tier enforcement +│ ├── ws/ # 4 files — WebSocket hub + fanout +│ ├── events/ # 3 files — event broadcasting + Postgres persistence +│ ├── router/ # 2 files — route definitions + middleware +│ ├── db/ # 6 files — Postgres + Redis drivers, migrations +│ └── crypto/ # 2 files — AES-256-GCM secrets encryption +└── migrations/ # 11 SQL migration files +``` + +### Canvas Frontend (62 TypeScript files) + +``` +canvas/ +├── src/ +│ ├── store/ # Zustand store (workspaces, viewport, chat, activity) +│ ├── components/ # React Flow nodes, side panel tabs, context menus, modals +│ ├── hooks/ # Custom hooks (WebSocket, resize, etc.) +│ └── lib/ # API client, utilities +└── tests/ # 188 Vitest tests +``` + +--- + +## 19. Key Design Patterns + +### 1. Import Cycle Prevention (Go) + +Function injection avoids circular imports between packages: +```go +hub := ws.NewHub(registry.CanCommunicate) +broadcaster := events.NewBroadcaster(hub) +registry.StartLivenessMonitor(ctx, onWorkspaceOffline) +``` + +### 2. JSONB Handling + +Go `[]byte` must convert to `string()` before JSONB insert with `::jsonb` cast. `lib/pq` treats `[]byte` as bytea, not JSONB. + +### 3. Event Sourcing + +`structure_events` table is append-only — **never UPDATE, never DELETE**. Provides complete audit trail and event replay capability. + +### 4. Template Resolution + +On workspace create: (1) check template folder → (2) try `{runtime}-default` → (3) generate minimal config via `ensureDefaultConfig()`. + +### 5. Hierarchy-Driven Everything + +`CanCommunicate()` is the single source of truth. All operational concerns (communication, memory, access, approvals, event visibility) derive from the same hierarchy. + +--- + +## 20. Docker Compose Orchestration + +### Full Stack (`docker-compose.yml`) + +| Service | Image | Port | Purpose | +|---------|-------|------|---------| +| `postgres` | postgres:16 | 5432 (internal) | Primary database (`wal_level=logical`) | +| `redis` | redis:7 | 6379 (internal) | Cache + pub/sub (`notify-keyspace-events=KEA`) | +| `langfuse-clickhouse` | clickhouse/clickhouse-server | internal | Analytics backend | +| `langfuse-web` | langfuse/langfuse | 3100 | Observability UI | +| `platform` | Built from Go | 8080 | Control plane | +| `canvas` | Built from Next.js | 3000 | Frontend | + +### Optional Profiles + +| Profile | Service | Purpose | +|---------|---------|---------| +| `multi-provider` | LiteLLM proxy | Unified API for OpenAI, Anthropic, Google, etc. | +| `local-models` | Ollama | Local LLM inference | + +### Infrastructure-Only (`docker-compose.infra.yml`) + +Postgres + Redis + Langfuse only (for local development without containerized workspace-server/canvas). + +--- + +## 21. Environment Variables + +### Platform (Go) + +| Variable | Default | Purpose | +|----------|---------|---------| +| `DATABASE_URL` | `postgres://dev:dev@localhost:5432/molecule?sslmode=prefer` | Postgres connection | +| `REDIS_URL` | `redis://localhost:6379` | Redis connection | +| `PORT` | `8080` | Platform listen port | +| `PLATFORM_URL` | `http://host.docker.internal:8080` | Injected to workspace containers | +| `SECRETS_ENCRYPTION_KEY` | Optional | AES-256 key (32 bytes) for secret encryption | +| `CONFIGS_DIR` | `/configs` | Workspace config template directory | +| `PLUGINS_DIR` | `/plugins` | Shared plugin directory | +| `ACTIVITY_RETENTION_DAYS` | `7` | Activity log retention | +| `ACTIVITY_CLEANUP_INTERVAL_HOURS` | `6` | Cleanup frequency | +| `CORS_ORIGINS` | `http://localhost:3000,...` | CORS whitelist | +| `RATE_LIMIT` | `600` | Requests per minute | +| `WORKSPACE_DIR` | Optional | Shared workspace volume | +| `AWARENESS_URL` | Optional | Awareness service URL | + +### Canvas (Next.js) + +| Variable | Default | Purpose | +|----------|---------|---------| +| `NEXT_PUBLIC_PLATFORM_URL` | `http://localhost:8080` | Platform backend URL | +| `NEXT_PUBLIC_WS_URL` | `ws://localhost:8080/ws` | WebSocket URL | +| `PORT` | `3000` | Canvas listen port | + +### Workspace Runtime (Python) + +| Variable | Default | Purpose | +|----------|---------|---------| +| `WORKSPACE_ID` | `workspace-default` | Unique workspace identifier | +| `WORKSPACE_CONFIG_PATH` | `/configs` | Config directory mount | +| `PLATFORM_URL` | `http://platform:8080` | Platform connection | +| `PARENT_ID` | Empty | Parent workspace ID (set if nested) | +| `AWARENESS_URL` | Optional | Awareness service | +| `AWARENESS_NAMESPACE` | Optional | Scoped namespace for awareness memory | +| `LANGFUSE_HOST` | `http://langfuse-web:3000` | Langfuse endpoint | +| `LANGFUSE_PUBLIC_KEY` | Optional | Langfuse auth | +| `LANGFUSE_SECRET_KEY` | Optional | Langfuse auth | +| `DEPLOYMENT_RETRY_ATTEMPTS` | `3` | Delegation retry count | +| `DELEGATION_TIMEOUT` | `120` | Delegation timeout (seconds) | +| `APPROVAL_TIMEOUT` | `300` | Approval wait timeout (seconds) | +| `AUDIT_LOG_PATH` | `/var/log/molecule/audit.jsonl` | Audit log file path | + +--- + +## 22. Recent Feature Highlights + +| Feature | Description | +|---------|-------------| +| **A2A streaming response** | Real-time task result delivery via SSE (`message/sendSubscribe`) | +| **Onboarding wizard** | 4-step guided first-run experience in Canvas | +| **Global API keys** | Platform-wide secrets with per-workspace override + AES-256 encryption | +| **Coordinator enforcement** | Team leads cannot do work, only route and aggregate | +| **Cascade pause/resume** | Pausing a parent cascades to all children; paused children can't be individually resumed | +| **Graceful A2A errors** | `[A2A_ERROR]` sentinel + retry with exponential backoff + fallback | +| **Canvas error boundary** | React class component catches render errors, shows retry button | +| **Hydration retry** | Banner with "Retry" button + `PLATFORM_URL` hint on WebSocket stale state | +| **Activity log retention** | Configurable cleanup (default 7 days, `ACTIVITY_RETENTION_DAYS`) | +| **Security hardening** | Hub double-close race fix (`sync.Once`), A2A proxy timeout (5min canvas, ∞ workspace), Python JSON decode guards | + +--- + +## 23. Known Gaps & Backlog + +### Test Coverage + +18 of 26 Go handler files have **zero unit tests**: a2a_proxy, workspace, templates, registry, discovery, secrets, etc. Current: 278 tests with 25% baseline enforced. + +### Silent Failures + +6+ locations with fire-and-forget `ExecContext` DB writes need proper error handling (activity log inserts, event broadcasts). + +### Python Tool Error Handling + +Tools call `resp.json()` without catching JSON decode errors. Should wrap in try/except for malformed responses. + +### Branch-Level Work + +| Branch | Feature | Status | +|--------|---------|--------| +| `feat/nemoclaw-t4-docker` | NemoClaw adapter (NVIDIA T4 support) | WIP | +| Backlog | Firecracker backend (faster cold starts) | Planned | +| Backlog | E2B backend (cloud-hosted code sandbox) | Planned | +| Backlog | pgvector semantic memory search | Planned | +| Backlog | Canvas search, batch operations, keyboard shortcuts | Planned | + +--- + +## 24. Licensing & Commercialization Path + +### Open Source (Current) + +- **License**: MIT +- **Strategy**: Maximize adoption, zero friction +- **Model**: Follows n8n Community Edition approach + +### SaaS Path (Future `molecule-cloud` repo) + +| Feature | Technology | +|---------|-----------| +| Authentication | Clerk or Auth.js | +| Multi-tenancy | `org_id` column added to schema | +| Billing | Stripe integration | +| Managed infrastructure | ECS + Neon + Upstash | +| White-labeling | Custom Canvas branding | + +**Key principle**: No changes to core open-source repo. SaaS layer is purely additive. + +--- + +## 25. OSS Growth Research + +Analysis of 8 OSS agent projects (from `oss-agent-growth-research.md`): + +### Winning Launch Formula + +``` +[Viral Demo] + [HN Front Page] + [One Major Amplifier] + [Zero-Friction Install] + ↓ ↓ ↓ ↓ + 60s video 400+ upvotes Karpathy / Altman / docker compose up + screen rec top comment Major YouTuber 3 commands max +``` + +Every Tier 1 launch (Open Interpreter, CrewAI) had all four elements. + +### Documentation Best Practice (Diataxis Model) + +| Type | Purpose | Example | +|------|---------|---------| +| Tutorials | Learning-oriented | "Build your first agent team in 5 minutes" | +| How-to guides | Task-oriented | "How to configure RBAC for production" | +| Explanation | Understanding-oriented | "Why memory follows org boundaries" | +| Reference | Information-oriented | API route tables, config schema | + +--- + +## 26. Technical Debt & Constraints + +### Hard Design Constraints + +1. **Platform never routes agent messages** — A2A is strictly peer-to-peer +2. **Postgres is fact source, Redis is cache** — Redis loss is fully recoverable +3. **`structure_events` is append-only** — Never UPDATE, never DELETE +4. **`workspace-template` has no business logic** — Logic lives in `workspace-configs-templates/` +5. **Bundles never include secrets** — API keys forbidden from serialization +6. **Hierarchy = topology** — No manual edge wiring; all communication derived from `parent_id` + +--- + +## 27. Production Deployment + +### Multi-Host Configuration + +- Docker-internal URLs (`http://ws-{id}:8000`) work directly between containers +- Nginx on host handles TLS termination +- For external HTTPS: proxy requests to host-mapped URLs + +### Volume Management + +| Mode | Configuration | Behavior | +|------|--------------|----------| +| **Default** | No `WORKSPACE_DIR` | Each workspace gets isolated Docker volume `ws-{id}-workspace` | +| **Shared** | `WORKSPACE_DIR=/path` | All agents mount same host directory (read/write) | + +--- + +## 28. MCP Server & Integrations + +### Molecule AI MCP Server (`mcp-server/`) + +20+ tools for Claude Code, Cursor, Codex, or any MCP client: + +- Workspace CRUD (list, create, get, delete, restart) +- Agent communication (`chat_with_agent`) +- Memory operations (`commit_memory`, `search_memory`) +- Team management (`expand_team`, `collapse_team`) +- Secrets management (`set_secret`, `list_secrets`) +- File operations (`read_file`, `write_file`, `delete_file`) +- Approvals (`list_pending_approvals`, `decide_approval`) +- Config updates (`update_workspace`) +- Templates (`list_templates`) + +**Transport**: stdio (local CLI integration) + +```json +{ + "mcpServers": { + "molecule": { + "type": "stdio", + "command": "node", + "args": ["./mcp-server/dist/index.js"], + "env": {"MOLECULE_URL": "http://localhost:8080"} + } + } +} +``` + +### Awareness MCP Server + +For persistent cross-session memory: + +```json +{ + "awareness-memory": { + "type": "stdio", + "command": "npx", + "args": ["-y", "@awareness-sdk/local", "mcp"] + } +} +``` + +--- + +## 29. Summary Statistics + +| Metric | Value | +|--------|-------| +| Python runtime files | 95 | +| Go platform files | 94 | +| TypeScript/JS canvas files | 62 | +| Runtime adapter implementations | 6 | +| Go handler files | 26 | +| Postgres migrations | 11 | +| Core workspace tools | 14 | +| Platform API endpoints | 40+ | +| MCP tools | 20+ | +| Go tests | 278 (with `-race` flag) | +| Canvas Vitest tests | 188 | +| Python pytest tests | 148 | +| **Total tests** | **614** | +| Activity retention | 7 days (configurable) | +| Heartbeat interval | 30s (default) | +| Redis liveness TTL | 60s | +| Health sweep interval | 15s (proactive) | +| Skill hot-reload propagation | ~3 seconds | +| Coverage baseline (Go) | 25% enforced in CI | + +--- + +## 30. Vision: From Agent Teams to Robot Teams + +Molecule AI's workspace abstraction is **runtime-agnostic by design**. A workspace is a role with an A2A interface — not an LLM with a prompt. The same hierarchy, memory boundaries, approval chains, and governance that organize AI agents in containers today can organize any autonomous system that speaks A2A. + +| Phase | Era | Systems | Status | +|-------|-----|---------|--------| +| **NOW** | Software Agent Teams | LLM agents in Docker, 6 adapters, HMA, Langfuse, A2A | **LIVE on main** | +| **NEXT** | Terminal + Device Agents | Terminal bots, browser agents, IoT controllers, CI/CD agents | **BUILDING** | +| **HORIZON** | Embodied Robot Teams | Warehouse robots, autonomous vehicles, manufacturing cells, field inspection | **HORIZON** | + +> *The workspace is the role. The protocol is A2A. The boundary between digital and physical disappears — the organizational layer remains.* + +--- + +## Links + +- **GitHub**: https://github.com/Molecule-AI/molecule-monorepo +- **Architecture Docs**: https://github.com/Molecule-AI/molecule-monorepo/tree/main/docs/architecture +- **API Protocol**: https://github.com/Molecule-AI/molecule-monorepo/tree/main/docs/api-protocol +- **Agent Runtime**: https://github.com/Molecule-AI/molecule-monorepo/tree/main/docs/agent-runtime +- **Product Docs**: https://github.com/Molecule-AI/molecule-monorepo/tree/main/docs/product + +--- + +*© 2026 Molecule AI Technologies, Inc.*