Merge branch 'main' into fix/docs-runtime-model-observability-accuracy
This commit is contained in:
Hongming Wang
GitHub
commit
0894f6954e
@ -77,7 +77,8 @@ The Platform is the central control plane responsible for:
|
||||
| `REDIS_URL` | (required) | Redis connection string |
|
||||
| `PORT` | `8080` | Server listen port |
|
||||
| `PLATFORM_URL` | `http://host.docker.internal:PORT` | URL passed to agent containers |
|
||||
| `SECRETS_ENCRYPTION_KEY` | (optional) | AES-256 key, 32 bytes |
|
||||
| `SECRETS_ENCRYPTION_KEY` | (optional) | AES-256 key, 32 bytes — static-mode envelope encryption (dev/self-host). Set `KMS_KEY_ARN` instead for production AWS KMS envelopes. |
|
||||
| `KMS_KEY_ARN` | (optional) | AWS KMS CMK ARN — production envelope encryption with per-secret data keys |
|
||||
| `CORS_ORIGINS` | `http://localhost:3000,http://localhost:3001` | Allowed CORS origins |
|
||||
| `RATE_LIMIT` | `600` | Requests per minute |
|
||||
| `MOLECULE_ENV` | (optional) | Set `production` to hide test endpoints |
|
||||
|
||||
@ -80,7 +80,7 @@ CREATE TABLE workspace_secrets (
|
||||
);
|
||||
```
|
||||
|
||||
Stores API keys, credentials, and other secrets needed by workspace agents. Values are encrypted with AES-256 at the application layer. The encryption key comes from the `SECRETS_ENCRYPTION_KEY` environment variable on the platform — never stored in the database.
|
||||
Stores API keys, credentials, and other secrets needed by workspace agents. Values are sealed with envelope encryption — AWS KMS (per-secret data keys via `GenerateDataKey`, AES-256-GCM payload) when `KMS_KEY_ARN` is configured (production), or static-key AES-256-GCM under `SECRETS_ENCRYPTION_KEY` (dev / self-host). The static key, when used, is read from the platform environment and is never stored in the database. Both modes coexist during a KMS cutover, distinguished by a v2 prefix byte on KMS blobs. Implementation: `workspace-server/internal/crypto/envelope.go`.
|
||||
|
||||
The provisioner reads secrets from this table, decrypts them, and injects them as environment variables when spinning up workspace containers. Secrets are never included in bundles (see [Constraints — Rule 5](../development/constraints-and-rules.md)).
|
||||
|
||||
|
||||
@ -149,7 +149,7 @@ Six runtime adapters ship production-ready on `main`: LangGraph, DeepAgents, Cla
|
||||
- Event broadcasting (Redis pub/sub → WebSocket fanout)
|
||||
- Docker provisioner with T1–T4 tier enforcement
|
||||
- Activity logging with configurable retention (default 7 days)
|
||||
- Secrets management (AES-256-GCM encryption)
|
||||
- Secrets management (KMS-envelope encryption in prod; AES-256-GCM static-key mode for dev/self-host)
|
||||
- File, terminal, bundle, template, traces APIs
|
||||
- Langfuse integration
|
||||
- Prometheus metrics endpoint
|
||||
@ -186,7 +186,7 @@ Six runtime adapters ship production-ready on `main`: LangGraph, DeepAgents, Cla
|
||||
|-------|---------|-------------|
|
||||
| `workspaces` | Current state registry | `id`, `name`, `role`, `tier` (1-4), `status`, `parent_id`, `agent_card` (JSONB), `url`, `forwarded_to`, `last_heartbeat_at`, `last_error_rate`, `active_tasks`, `uptime_seconds`, `current_task`, `runtime` |
|
||||
| `agents` | Agent assignment history | `workspace_id`, `model`, `status`, `removed_at`, `removal_reason` |
|
||||
| `workspace_secrets` | Encrypted credentials | `workspace_id`, `key`, `encrypted_value` (BYTEA, AES-256-GCM) |
|
||||
| `workspace_secrets` | Encrypted credentials | `workspace_id`, `key`, `encrypted_value` (BYTEA — KMS envelope in prod, AES-256-GCM static-key blob in dev/self-host) |
|
||||
| `agent_memories` | HMA-scoped memory | `workspace_id`, `content`, `scope` (LOCAL/TEAM/GLOBAL) |
|
||||
| `structure_events` | **Immutable** event log (APPEND-ONLY, never UPDATE/DELETE) | `event_type`, `workspace_id`, `agent_id`, `target_id`, `payload` (JSONB) |
|
||||
| `activity_logs` | Operational activity with retention | `workspace_id`, `activity_type`, `source_id`, `target_id`, `method`, `request_body`, `response_body`, `duration_ms`, `status`, `error_detail` |
|
||||
@ -824,7 +824,7 @@ workspace-server/
|
||||
│ ├── events/ # 3 files — event broadcasting + Postgres persistence
|
||||
│ ├── router/ # 2 files — route definitions + middleware
|
||||
│ ├── db/ # 6 files — Postgres + Redis drivers, migrations
|
||||
│ └── crypto/ # 2 files — AES-256-GCM secrets encryption
|
||||
│ └── crypto/ # 2 files — envelope encryption (KMS or AES-256-GCM static key)
|
||||
└── migrations/ # 11 SQL migration files
|
||||
```
|
||||
|
||||
@ -907,7 +907,8 @@ Postgres + Redis + Langfuse only (for local development without containerized wo
|
||||
| `REDIS_URL` | `redis://localhost:6379` | Redis connection |
|
||||
| `PORT` | `8080` | Platform listen port |
|
||||
| `PLATFORM_URL` | `http://host.docker.internal:8080` | Injected to workspace containers |
|
||||
| `SECRETS_ENCRYPTION_KEY` | Optional | AES-256 key (32 bytes) for secret encryption |
|
||||
| `SECRETS_ENCRYPTION_KEY` | Optional | AES-256 key (32 bytes) for static-mode secret encryption — used when `KMS_KEY_ARN` is unset (dev/self-host) or to decrypt legacy blobs during a KMS cutover |
|
||||
| `KMS_KEY_ARN` | Optional | AWS KMS CMK ARN — when set, secrets use KMS envelope encryption (per-secret data keys via `GenerateDataKey`); production deployments use this path |
|
||||
| `CONFIGS_DIR` | `/configs` | Workspace config template directory |
|
||||
| `PLUGINS_DIR` | `/plugins` | Shared plugin directory |
|
||||
| `ACTIVITY_RETENTION_DAYS` | `7` | Activity log retention |
|
||||
@ -951,7 +952,7 @@ Postgres + Redis + Langfuse only (for local development without containerized wo
|
||||
|---------|-------------|
|
||||
| **A2A streaming response** | Real-time task result delivery via SSE (`message/sendSubscribe`) |
|
||||
| **Onboarding wizard** | 4-step guided first-run experience in Canvas |
|
||||
| **Global API keys** | Platform-wide secrets with per-workspace override + AES-256 encryption |
|
||||
| **Global API keys** | Platform-wide secrets with per-workspace override; KMS envelope encryption in prod (AES-256-GCM static-key mode in dev/self-host) |
|
||||
| **Coordinator enforcement** | Team leads cannot do work, only route and aggregate |
|
||||
| **Cascade pause/resume** | Pausing a parent cascades to all children; paused children can't be individually resumed |
|
||||
| **Graceful A2A errors** | `[A2A_ERROR]` sentinel + retry with exponential backoff + fallback |
|
||||
|
||||
@ -59,7 +59,7 @@ Direct A2A calls between workspaces are unauthenticated in MVP. Access control i
|
||||
|
||||
## 11. Secrets in Postgres, Encrypted
|
||||
|
||||
Workspace secrets (API keys, credentials) are stored in Postgres with AES-256 encryption at the application layer. The encryption key comes from the `SECRETS_ENCRYPTION_KEY` environment variable. Secrets are never included in bundles, never logged, never exposed via API responses.
|
||||
Workspace secrets (API keys, credentials) are stored in Postgres under envelope encryption. Production deployments use AWS KMS (`KMS_KEY_ARN`): each secret gets a fresh data key via `GenerateDataKey`, the payload is sealed with AES-256-GCM, and the KMS-encrypted DEK is stored alongside the ciphertext — rotating the CMK is a no-op for existing blobs. Dev and self-host deployments fall back to static-key AES-256-GCM under `SECRETS_ENCRYPTION_KEY`. Secrets are never included in bundles, never logged, never exposed via API responses.
|
||||
|
||||
## 12. Last-Write-Wins for MVP
|
||||
|
||||
|
||||
@ -59,9 +59,11 @@ documents — through tool calls, logs, or responses.
|
||||
|
||||
**Molecule AI controls:**
|
||||
|
||||
- **Encrypted secrets at rest:** Workspace secrets are encrypted with
|
||||
`SECRETS_ENCRYPTION_KEY` (AES-256) before storage. Plaintext never hits the
|
||||
database.
|
||||
- **Encrypted secrets at rest:** Workspace secrets are sealed with envelope
|
||||
encryption before storage — AWS KMS (per-secret data keys via `GenerateDataKey`,
|
||||
AES-256-GCM payload) when `KMS_KEY_ARN` is set, or static-key AES-256-GCM
|
||||
under `SECRETS_ENCRYPTION_KEY` for dev / self-host. Plaintext never hits the
|
||||
database, and the platform refuses to start with neither configured.
|
||||
- **Secrets scoped per-workspace:** A token scoped to workspace A cannot access
|
||||
workspace B's secrets.
|
||||
- **Memory access controls:** The MCP server's memory tools respect workspace
|
||||
|
||||
@ -98,7 +98,8 @@ docker compose up
|
||||
| `PORT` | `8080` | Platform HTTP port |
|
||||
| `PLATFORM_URL` | `http://host.docker.internal:PORT` | URL passed to agent containers to reach the platform |
|
||||
| `CORS_ORIGINS` | `http://localhost:3000,http://localhost:3001` | Comma-separated allowed origins |
|
||||
| `SECRETS_ENCRYPTION_KEY` | -- | AES-256 key (32 bytes) for encrypting workspace secrets |
|
||||
| `SECRETS_ENCRYPTION_KEY` | -- | AES-256 key (32 bytes) for static-mode envelope encryption of workspace secrets (dev/self-host path) |
|
||||
| `KMS_KEY_ARN` | -- | AWS KMS CMK ARN — when set, secrets use KMS envelope encryption (per-secret data keys); production SaaS deployments use this path |
|
||||
| `WORKSPACE_DIR` | -- | Global fallback host path for `/workspace` bind-mount |
|
||||
| `MOLECULE_ENV` | -- | Set to `production` to hide E2E helper endpoints |
|
||||
| `ACTIVITY_RETENTION_DAYS` | `7` | How long activity logs are retained |
|
||||
@ -154,7 +155,9 @@ This image serves both the API and the canvas frontend from a single container.
|
||||
|
||||
### Secrets Encryption
|
||||
|
||||
Set `SECRETS_ENCRYPTION_KEY` to a 32-byte AES-256 key to encrypt workspace secrets at rest. Without this variable, secrets are stored in plaintext.
|
||||
The platform supports two envelope-encryption modes for workspace secrets, picked at boot:
|
||||
|
||||
**Static mode (self-host / dev).** Set `SECRETS_ENCRYPTION_KEY` to a 32-byte AES-256 key. Each secret is sealed with AES-256-GCM under that single long-lived key. Without this variable (and without `KMS_KEY_ARN`), the platform refuses to start rather than silently storing plaintext.
|
||||
|
||||
```bash
|
||||
# Generate a key
|
||||
@ -163,6 +166,12 @@ openssl rand -hex 32
|
||||
|
||||
**Warning:** `SECRETS_ENCRYPTION_KEY` cannot be rotated without a data migration. Choose carefully before deploying to production.
|
||||
|
||||
**KMS mode (production SaaS).** Set `KMS_KEY_ARN` to an AWS KMS Customer Master Key ARN. Each `Encrypt()` call asks KMS for a fresh per-secret data encryption key (`GenerateDataKey`), seals the secret payload with AES-256-GCM under that DEK, and stores the KMS-encrypted DEK alongside the ciphertext (envelope encryption). Rotating the CMK is a no-op for existing blobs — KMS tracks key versions internally.
|
||||
|
||||
The two modes coexist during cutover: a v2 prefix byte tags KMS blobs; older static-mode blobs decrypt with `SECRETS_ENCRYPTION_KEY` until they're next written. Operators migrating to KMS can leave both env vars set during the transition.
|
||||
|
||||
Implementation: `workspace-server/internal/crypto/envelope.go`.
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
The `RATE_LIMIT` variable (default 600 requests/min) applies per client. Adjust based on your expected traffic.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user