core#242 PROD follow-up: tenant EC2 user-data must stage /etc/molecule-bootstrap/personas #128

New Issue

claude-ceo-assistant · 2026-05-08T16:53:24Z

2026-05-08 16:53:24 +00:00

Sub-issue of core#242 (CP provisioner persona injection)

The LOCAL surface shipped — docker-compose.yml bind-mounts ${HOME}/.molecule-ai/personas into the platform container at /etc/molecule-bootstrap/personas so org_import.go::loadPersonaEnvFile finds files locally.

The PROD surface remains: tenant EC2s don't have /etc/molecule-bootstrap/personas/ populated, so loadPersonaEnvFile silently no-ops on every workspace import. Per saved memory feedback_unified_credentials_file, the canonical pattern post-2026-05-06 is AWS Secrets Manager-fetched-at-boot rather than scp-from-operator. So:

Proposed approach

Mirror persona env files into AWS Secrets Manager. One secret per persona (or one consolidated secret for all 28). Sync from operator-host /etc/molecule-bootstrap/personas/ whenever the rotation cron fires.
Extend CP user-data (ec2.go provisioner) to fetch + stage at first boot. Read the secret(s) via the EC2 instance profile; write to /etc/molecule-bootstrap/personas/<role>/env on the EC2 host filesystem; the existing platform-service docker-run already mounts /etc into the container so the platform sees them.
Re-fetch on rotation. When persona tokens rotate (monthly cron), tenant EC2s need fresh values. Either: (a) trigger a per-tenant CP redeploy (heavy), or (b) add a per-tenant agent on the EC2 that polls Secrets Manager and overwrites the local files on change.

Option 3a is simpler and matches the existing CP redeploy pattern. Operator-host rotation cron also enqueues a CP-redeploy-fan-out job that re-pushes user-data to each tenant.

Acceptance criteria

AWS Secrets Manager has persona env content (one secret per persona or consolidated; design call)
ec2.go user-data fetches + stages persona files at first boot using instance-profile auth
Provisioning a fresh tenant + importing a workspace with role: dev-lead results in workspaces_secrets rows with GITEA_USER=dev-lead, etc.
Rotation cron (existing /opt/molecule/rotate-personas.py) extended to mirror to Secrets Manager + queue per-tenant redeploy

Out of scope

Per-EC2 polling daemon (option 3b above) — defer until rotation+redeploy cadence proves insufficient
Multi-region replication of persona secrets — current scale doesn't need it

Refs

core#242 (parent; LOCAL surface merged in PR — see commit history)
saved memory feedback_unified_credentials_file (AWS Secrets Manager is SSOT pattern post-suspension)
saved memory feedback_local_must_mimic_production (in-container path matches prod, established by LOCAL surface fix)
/opt/molecule/rotate-personas.py on operator host (where the rotation cron runs; needs the extension)

## Sub-issue of core#242 (CP provisioner persona injection) The LOCAL surface shipped — docker-compose.yml bind-mounts `${HOME}/.molecule-ai/personas` into the platform container at `/etc/molecule-bootstrap/personas` so `org_import.go::loadPersonaEnvFile` finds files locally. The PROD surface remains: tenant EC2s don't have `/etc/molecule-bootstrap/personas/` populated, so `loadPersonaEnvFile` silently no-ops on every workspace import. Per saved memory `feedback_unified_credentials_file`, the canonical pattern post-2026-05-06 is AWS Secrets Manager-fetched-at-boot rather than scp-from-operator. So: ## Proposed approach 1. **Mirror persona env files into AWS Secrets Manager.** One secret per persona (or one consolidated secret for all 28). Sync from operator-host `/etc/molecule-bootstrap/personas/` whenever the rotation cron fires. 2. **Extend CP user-data (`ec2.go` provisioner) to fetch + stage at first boot.** Read the secret(s) via the EC2 instance profile; write to `/etc/molecule-bootstrap/personas/<role>/env` on the EC2 host filesystem; the existing platform-service docker-run already mounts `/etc` into the container so the platform sees them. 3. **Re-fetch on rotation.** When persona tokens rotate (monthly cron), tenant EC2s need fresh values. Either: (a) trigger a per-tenant CP redeploy (heavy), or (b) add a per-tenant agent on the EC2 that polls Secrets Manager and overwrites the local files on change. Option 3a is simpler and matches the existing CP redeploy pattern. Operator-host rotation cron also enqueues a CP-redeploy-fan-out job that re-pushes user-data to each tenant. ## Acceptance criteria - AWS Secrets Manager has persona env content (one secret per persona or consolidated; design call) - `ec2.go` user-data fetches + stages persona files at first boot using instance-profile auth - Provisioning a fresh tenant + importing a workspace with `role: dev-lead` results in `workspaces_secrets` rows with `GITEA_USER=dev-lead`, etc. - Rotation cron (existing `/opt/molecule/rotate-personas.py`) extended to mirror to Secrets Manager + queue per-tenant redeploy ## Out of scope - Per-EC2 polling daemon (option 3b above) — defer until rotation+redeploy cadence proves insufficient - Multi-region replication of persona secrets — current scale doesn't need it ## Refs - core#242 (parent; LOCAL surface merged in PR — see commit history) - saved memory `feedback_unified_credentials_file` (AWS Secrets Manager is SSOT pattern post-suspension) - saved memory `feedback_local_must_mimic_production` (in-container path matches prod, established by LOCAL surface fix) - `/opt/molecule/rotate-personas.py` on operator host (where the rotation cron runs; needs the extension)

Sign in to join this conversation.

No Label

tier:high

tier:low

tier:medium

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#128