core-lead/molecule-core
1
0
Fork 0
forked from molecule-ai/molecule-core
Code Pull Requests Activity

Merge pull request #2106 from Molecule-AI/docs/secrets-key-custody

Browse Source 
docs(security): document KMS-rooted custody chain for SECRETS_ENCRYPTION_KEY
This commit is contained in:
Hongming Wang 2026-04-26 18:51:16 +00:00 committed by GitHub
commit b0a33d9ebf
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 88 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/architecture/database-schema.md
View File

@ -77,7 +77,7 @@ CREATE TABLE workspace_secrets (
);
```
Stores API keys, credentials, and other secrets needed by workspace agents. Values are encrypted with AES-256 at the application layer. The encryption key comes from the `SECRETS_ENCRYPTION_KEY` environment variable on the platform — never stored in the database.
Stores API keys, credentials, and other secrets needed by workspace agents. Values are encrypted with AES-256-GCM at the application layer. The encryption key comes from the tenant's `SECRETS_ENCRYPTION_KEY` environment variable, provisioned at tenant boot by the control plane (which holds the master key in AWS KMS — see [secrets-key-custody.md](./secrets-key-custody.md)). The key is never stored in the database.
The provisioner reads secrets from this table, decrypts them, and injects them as environment variables when spinning up workspace containers. Secrets are never included in bundles (see [Constraints — Rule 5](../development/constraints-and-rules.md)).

2
docs/architecture/molecule-technical-doc.md
View File

@ -902,7 +902,7 @@ Postgres + Redis + Langfuse only (for local development without containerized wo
| `REDIS_URL` | `redis://localhost:6379` | Redis connection |
| `PORT` | `8080` | Platform listen port |
| `PLATFORM_URL` | `http://host.docker.internal:8080` | Injected to workspace containers |
| `SECRETS_ENCRYPTION_KEY` | Optional | AES-256 key (32 bytes) for secret encryption |
| `SECRETS_ENCRYPTION_KEY` | Optional | AES-256-GCM key (32 bytes) for tenant secret encryption. Provisioned at tenant boot by the control plane, which holds the master key in AWS KMS — see [secrets-key-custody.md](./secrets-key-custody.md). |
| `CONFIGS_DIR` | `/configs` | Workspace config template directory |
| `PLUGINS_DIR` | `/plugins` | Shared plugin directory |
| `ACTIVITY_RETENTION_DAYS` | `7` | Activity log retention |

85
docs/architecture/secrets-key-custody.md Normal file
View File

@ -0,0 +1,85 @@
# Secrets Key Custody
How the encryption keys that protect Molecule workspace secrets are managed, where each key lives, and what an attacker who compromises one layer can or cannot read.
This document exists because the platform repo (`workspace-server`) reads `SECRETS_ENCRYPTION_KEY` from its process env, which on its own looks like "encryption-at-rest theater." The full custody chain runs through the control plane (`molecule-controlplane`) where AWS KMS holds the key material at rest. Anyone reading only the platform repo sees half the picture.
## Two modes
The control plane's `internal/crypto.Envelope` ships in two modes, picked at boot from env:
| Mode | Trigger | At-rest format | Recommended for |
|------|---------|----------------|-----------------|
| **KMS envelope** | `KMS_KEY_ARN` set | Per-blob KMS-wrapped DEK + AES-256-GCM ciphertext | Production, multi-tenant SaaS |
| **Static key** | Only `SECRETS_ENCRYPTION_KEY` set | AES-256-GCM with one process-wide key | Dev, self-hosted single-tenant |
`Envelope.Decrypt` is dual-mode — it can read either format on the way out, so a deployment can flip from static-key to KMS envelope without re-encrypting historical rows. Code: `molecule-controlplane/internal/crypto/kms.go`.
## KMS envelope flow
When `KMS_KEY_ARN` is configured, every secret write looks like:
1. CP calls `kms.GenerateDataKey(KeyId=KMS_KEY_ARN, KeySpec=AES_256)` → returns `{Plaintext, CiphertextBlob}`.
2. CP encrypts the secret with AES-256-GCM using `Plaintext` as the key.
3. CP discards `Plaintext` from memory; persists the blob:
```
[0x02 prefix][uint16 BE: encrypted_dek_len][encrypted_dek][nonce(12)][ct+tag]
```
The `0x02` byte distinguishes v2 (KMS-wrapped) blobs from legacy static-key blobs.
4. To read: CP calls `kms.Decrypt(CiphertextBlob)` → recovers the AES key → unwraps the GCM ciphertext.
KMS calls cost ~$0.03 per 10k requests. We do not cache DEKs — provisioning rate is orders below steady-state reads, and not caching keeps key rotation reasoning simple.
## What lives where
| Layer | Key custody | Plaintext key in memory? |
|-------|-------------|--------------------------|
| AWS KMS | KMS-resident, never leaves the HSM | No (hardware) |
| `molecule-controlplane` process | KMS client + IAM role | Briefly per-secret-op only |
| CP database (`database_url_encrypted`, tenant secrets) | KMS-wrapped blobs | Never |
| Per-tenant `workspace-server` env (`SECRETS_ENCRYPTION_KEY`) | Provisioned at tenant boot by CP | Yes, for the tenant's process lifetime |
| Tenant Postgres (`workspace_secrets.value`) | AES-256-GCM with the tenant's key | Never |
The "plaintext in tenant memory" row is the standard envelope-encryption trade-off: a DEK has to be unwrapped somewhere to be used. The blast radius of compromising one tenant's process is one tenant's secrets — not the whole fleet.
## Threat model
| Attacker capability | Can they read tenant secrets? |
|---------------------|-------------------------------|
| Reads CP database backup | No — KMS unwrap requires IAM-scoped `kms:Decrypt` |
| Steals `KMS_KEY_ARN` value | No — ARN alone does nothing without IAM access |
| Compromises CP IAM role | Yes — can `kms:Decrypt` any wrapped DEK |
| Reads tenant Postgres (one tenant) | No — `SECRETS_ENCRYPTION_KEY` lives only in the tenant's own EC2 process env, not in DB |
| Compromises one tenant's EC2 | Yes for that tenant's secrets, no for any other tenant |
| Compromises CP host | Game over (CP can provision arbitrary tenants) |
The two boundaries the design protects:
- **DB-only compromise (incl. backups)** → secrets remain encrypted; attacker needs separate access to either KMS (prod) or CP env (dev).
- **One-tenant compromise** → blast radius limited to that tenant; no cross-tenant key reuse.
## Rotation
- **Tenant key rotation** (per-tenant `SECRETS_ENCRYPTION_KEY`): re-encrypt the tenant's `workspace_secrets` rows under a new key, then swap the env var. Static-key mode requires this for all rotation; KMS mode only requires it on suspected key compromise.
- **KMS CMK rotation**: AWS KMS handles annual automatic rotation of the customer master key. Re-wrapping data keys is unnecessary because each `Decrypt` call routes through the current CMK version automatically (KMS keeps prior versions for decrypt-only).
## Audit / compliance posture
For SOC2 / ISO 27001 / customer security questionnaires:
- **Key custody**: AWS KMS (FIPS 140-2 Level 3 HSM-backed)
- **Key isolation**: per-tenant DEK; no shared keys across tenants
- **Access control**: IAM-scoped `kms:Decrypt`, audited via CloudTrail
- **At-rest encryption**: AES-256-GCM (NIST-approved, authenticated)
- **In-transit encryption**: TLS 1.2+ for KMS, CP-to-tenant, tenant-to-DB
- **Rotation**: AWS-managed CMK rotation annually; manual DEK rotation on incident
## Pointers
- KMS envelope code: [`molecule-controlplane/internal/crypto/kms.go`](https://github.com/Molecule-AI/molecule-controlplane/blob/main/internal/crypto/kms.go)
- Static-key fallback: [`molecule-controlplane/internal/crypto/aes.go`](https://github.com/Molecule-AI/molecule-controlplane/blob/main/internal/crypto/aes.go)
- Tenant secrets handler: [`workspace-server/internal/crypto/aes.go`](../../workspace-server/internal/crypto/aes.go)
- Tenant secrets schema: [database-schema.md](./database-schema.md#workspace_secrets)

2
docs/development/constraints-and-rules.md
View File

@ -56,7 +56,7 @@ Direct A2A calls between workspaces are unauthenticated in MVP. Access control i
## 11. Secrets in Postgres, Encrypted
Workspace secrets (API keys, credentials) are stored in Postgres with AES-256 encryption at the application layer. The encryption key comes from the `SECRETS_ENCRYPTION_KEY` environment variable. Secrets are never included in bundles, never logged, never exposed via API responses.
Workspace secrets (API keys, credentials) are stored in Postgres with AES-256-GCM encryption at the application layer. The tenant's `SECRETS_ENCRYPTION_KEY` is provisioned at boot by the control plane, which holds the master key material in AWS KMS (envelope encryption, dual-mode with a static-key fallback for dev). Full custody chain in [secrets-key-custody.md](../architecture/secrets-key-custody.md). Secrets are never included in bundles, never logged, never exposed via API responses.
## 12. Last-Write-Wins for MVP