molecule-core/docs/runbooks/saas-secrets.md

SaaS secret rotation — runbook

Where each secret lives, why, and the full rotation procedure so a partial update doesn't silently break production.

Secret map

Secret	Location(s)	Purpose
FLY_API_TOKEN	(a) molecule-monorepo GitHub Actions secret (push image to registry.fly.io/molecule-tenant) + (b) fly secrets on molecule-cp app (control plane creates + deletes tenant Fly Machines)	Any Fly Machines API call
NEON_API_KEY	fly secrets on molecule-cp	Create + delete tenant Neon branches
DATABASE_URL	fly secrets on molecule-cp	Control-plane Postgres connection (Neon cool-sea-89357706)
TENANT_REDIS_URL	fly secrets on molecule-cp	Injected into every tenant container as REDIS_URL
SECRETS_ENCRYPTION_KEY	fly secrets on molecule-cp	AES-256 key wrapping tenant DB/Redis URLs in org_instances (provisioner + tenant use this)
GITHUB_TOKEN	Built-in GitHub Actions token	GHCR push; rotated automatically

Coupled secrets — MUST rotate together

FLY_API_TOKEN is the one secret duplicated across systems. Rotating only one will cause silent breakage:

• Rotating only (a) GHA → image publish workflow fails, but no alert; control plane keeps provisioning from the stale latest tag.
• Rotating only (b) Fly secrets → control plane's Fly API calls start erroring (401), tenant provisioning fails, but image publishes keep succeeding so everything looks fine on the build side.

Rotation procedure — FLY_API_TOKEN

1. Generate new token:

   flyctl tokens create deploy --name molecule-cp-rotation-$(date +%Y%m%d)
   

2. Update both locations (order matters — Fly secrets first, then GHA):

   # (b) Fly secrets — triggers zero-downtime redeploy
   flyctl secrets set --app molecule-cp FLY_API_TOKEN='FlyV1 fm2_...'
   
   # (a) GitHub Actions secret — next workflow run uses new token
   echo 'FlyV1 fm2_...' | gh secret set FLY_API_TOKEN --repo Molecule-AI/molecule-monorepo
   

3. Verify:

   # Control plane can reach Fly API:
   curl https://molecule-cp.fly.dev/health
   # Trigger image publish (dispatches workflow, pushes to both registries):
   gh workflow run publish-platform-image.yml --repo Molecule-AI/molecule-monorepo
   gh run list --repo Molecule-AI/molecule-monorepo --workflow publish-platform-image --limit 1
   

4. Revoke the old token:

   flyctl tokens list
   flyctl tokens revoke <id-of-old-token>
   

Rotation procedure — NEON_API_KEY

1. Create replacement key in Neon console → Account Settings → API Keys.
2. Update Fly secrets:

   flyctl secrets set --app molecule-cp NEON_API_KEY='napi_...'
   

3. Trigger a test provision (dry run — create + delete):

   curl -X POST https://molecule-cp.fly.dev/cp/orgs \
     -H 'Content-Type: application/json' \
     -d '{"slug":"keytest-'$(date +%s)'","name":"Rotation test"}'
   # Wait 60s, inspect logs:
   flyctl logs --app molecule-cp --no-tail | tail -30
   # Clean up the test org via DELETE once live
   

4. Revoke old key in Neon console.

Rotation procedure — SECRETS_ENCRYPTION_KEY

DANGEROUS: rotating this key will invalidate every encrypted row in org_instances.database_url_encrypted + redis_url_encrypted. Every tenant becomes unreachable until re-provisioned.

Mitigation: we intentionally defer real KMS + key-rotation to Phase H. Until then, do not rotate this key unless compromised. If compromise, procedure is:

1. Generate new key: openssl rand -hex 32
2. Set new key on molecule-cp.
3. For every row in org_instances: re-provision the tenant (creates fresh Neon branch + Fly machine). The old encrypted URLs are un-decryptable but irrelevant — we mint fresh ones.
4. Migration to rotate encrypted columns in-place (decrypt-with-old → encrypt- with-new) is Phase H work and requires envelope encryption with KMS.

Rotation procedure — DATABASE_URL (control plane)

The Neon molecule-cp project has a stable primary endpoint. Rotate only if:

• Neon forces a migration
• The connection-URI password is leaked

Procedure: regenerate URI via Neon API → flyctl secrets set DATABASE_URL=.... Zero-downtime (Fly applies secret via rolling restart).

Emergency contacts

• Fly: billing dashboard at fly.io → Support
• Neon: console.neon.tech → Support
• Upstash: upstash.com → Support
• GHCR: github.com/orgs/Molecule-AI (org admins)