# Edit history — 2026-04-15

## tick-9: Phase 32 Phase B.2 image pipeline (PR #80) + tick-8 docs sync (PR #79)

Two merges:

### PR #79 — `docs: sync documentation with 2026-04-14 tick-8 merge (#78)`
Merge commit `d53a1287`. Tick-8 docs sync for the TenantGuard middleware.
Pure docs; CLAUDE.md test count + PLAN.md tick-8 block + edit-history entry.

### PR #80 — `feat(ci): publish-platform-image → ghcr.io/molecule-ai/platform (Phase B.2)`
Merge commit `c3cc8e87`. Noteworthy: ci-infra.

Adds `.github/workflows/publish-platform-image.yml`:
- Trigger: push to main touching `workspace-server/**`; also `workflow_dispatch`.
- Builds `workspace-server/Dockerfile` via `docker/build-push-action@v5`.
- Pushes two tags per run: `ghcr.io/molecule-ai/platform:latest` (floating)
  and `:sha-<short-commit>` (immutable, pin-friendly).
- GHA cache via `cache-from/cache-to: type=gha` for warm rebuilds.
- Permissions: `contents:read` + `packages:write`; authenticates to GHCR
  using the built-in `GITHUB_TOKEN`, no extra secrets.
- OCI labels propagate source URL + commit SHA for provenance.

Purpose: pairs with the private `molecule-controlplane` Fly + Neon
provisioner (PR #3 there, merged `2e85d5ad`) which reads
`TENANT_IMAGE=ghcr.io/molecule-ai/platform:<tag>` from env and spawns
each tenant Fly Machine from this image.

### Deployment state (informational — not in any repo)
- Fly apps (`molecule-cp`, `molecule-tenant`): **pending CEO** (`flyctl apps create`).
- Fly billing card: **pending CEO**.
- First real tenant provision: **blocked** on the two above.

### File deltas (public repo)
- `.github/workflows/publish-platform-image.yml` — new.
- `CLAUDE.md` — tick-9 block for the new CI workflow.
- `PLAN.md` — new "Recently launched (2026-04-15 tick-9)" entry.

---

## Overnight sweep (2026-04-15 16:30–19:10 UTC, ticks 17–30+)

One long session that started with a malware discovery, pivoted through a
half-day of security triage, landed ~27 PRs across both repos, and ended
with a self code-review cleanup round. Chronological order below, compressed
to the load-bearing details so future ticks can grep this file instead of
re-reading the JSONL cron-learnings stream.

### Security: malware cleanup + Fly credential rotation

Discovered `xmrig` cryptominer installed Dec 6 2025 via commodity
npm-dropper, running out of `/var/tmp/.X11-unix/xmrig-6.24.0/` as
`systemd-udevd` (camouflaged Linux daemon name on a Mac mini). Crontab
entry `*/10 * * * *` had been firing every 10 min for ~4 months until
tonight — ~17,500 launches. Wiped crontab, removed payload, rotated
`FLY_API_TOKEN` + `CLAUDE_CODE_OAUTH_TOKEN` + `GRAFANA_PROM_TOKEN`.
Mining-only payload (no backdoor confirmed): no SSH auth-keys, no
LaunchAgents, no extra shell hooks, no other xmrig copies. But personal
Fly token rotated via `flyctl auth login` invalidated the token still
in GitHub Actions secrets — surfaced much later as #199 publish
workflow 401. **Operator rule of thumb: always use `flyctl tokens create
deploy -a <app>` for CI, never a personal auth token.**

### Self-hosted CI runner migration

#186 switched every `ci.yml` job + `publish-platform-image.yml` from
`runs-on: ubuntu-latest` to `[self-hosted, macos, arm64]` (Apple-silicon
Mac mini `hongming-m1-mini`). Non-trivial adaptations:
- Replaced GH Actions `services: postgres/redis` (Linux-only) with
  inline `docker run` with `PG_CONTAINER` / `REDIS_CONTAINER` env vars
  and `docker rm -f` teardown in `if: always()`. Ports 15432/16379
  to avoid collision with host services.
- `ludeeus/action-shellcheck` (Docker action, Linux-only) → fallback
  to local `brew install shellcheck` + `find | xargs shellcheck`.
- `actions/setup-python@v5` hardcodes `/Users/runner/hostedtoolcache`
  (non-overridable — upstream limitation in the prebuilt setup.sh from
  `actions/python-versions`). Bypassed with a `Verify Python 3.11
  (Homebrew)` step that prepends `/opt/homebrew/opt/python@3.11/bin`
  to `$GITHUB_PATH`. One-time runner prep: `brew install python@3.11`.
- `publish-platform-image.yml` adds `docker/setup-qemu-action@v3`
  + `platforms: linux/amd64` explicit because the runner is arm64 and
  Fly tenant machines are amd64.

Controlplane PR #28 mirrored the same migration on its own single-job
ci.yml (1-line `runs-on` swap — no matrix adaptations needed).

Known runner rough edges tracked as follow-ups: #191 (persistent-state
docs), #199 (Fly registry 401 — resolved by minting a deploy token
scoped to `molecule-tenant`, tokens table previously empty).

### Security fixes — auth gating

Closed a cluster of unauthenticated-route findings surfaced by the
Security Auditor's hourly audit:

| PR | Issue | Fix |
|---|---|---|
| #94 | #C6 | RFC-1918 + link-local in registry URL validator |
| #99 | #104 | AdminAuth gate on GET /workspaces (topology leak) |
| #102 | — | ancestor↔descendant A2A for hierarchy routing |
| #106 | #103 HIGH | path-sanitize + admin-gate POST /org/import |
| #110 | — | revoke workspace_auth_tokens on workspace delete |
| #119 | — | IPv6 SSRF blocklist (fe80::/10, ::1/128, fc00::/7) + scheduler unit tests |
| #125/#162 | #138 | field-level authz on PATCH /workspaces/:id (cosmetic fields passthrough, sensitive fields bearer-required) |
| #155 | #151 | wire SecurityHeaders middleware |
| #167 | #164 CRIT #165 HIGH #166 MED | gate 6 unauth routes (bundles/export, bundles/import, events, events/:id, canvas/viewport PUT, admin/liveness) |
| #185 | #180 | AdminAuth on GET /approvals/pending |
| #200 | #190 HIGH | AdminAuth on POST /templates/import |
| #203 | #168 | CanvasOrBearer middleware on PUT /canvas/viewport only (route-split approach) |
| #209 | #169 C2 | source_id spoof defense in activity.Report |
| #233 | #226 MED | resolveInsideRoot on POST /workspaces template/runtime |

Rejected PR #194 (Origin-fallback approach) because it would have
re-opened #164 CRITICAL to curl-based spoofing. #168 correctly fixed
via the narrower route-split in #203.

Rejected PR #169 (large C1-C6 batch) because 4/7 findings were
duplicates of already-merged work and migration 022 numbering
collided with 022_workspace_schedules_source. Cherry-picked the one
genuinely new fix (C2 source_id spoof check) into #209 and closed
#169.

### Security fixes — data integrity

- **#212** CRITICAL migration-runner bug: `RunMigrations` globbed
  `*.sql` and sorted alphabetically, running `.down.sql` BEFORE
  `.up.sql` on every boot. Wiped `workspace_auth_tokens` + two other
  pairs on every platform restart, regressing AdminAuth to fail-open
  bootstrap mode. Filter to skip `.down.sql` + unit test in
  `postgres_migrate_test.go`.
- **#224** YAML injection in `generateDefaultConfig` — body.Name
  concatenated into YAML without escaping. Fixed by emitting as
  double-quoted YAML scalar with all control chars escaped. Structural
  test (parse + verify key count) instead of substring match.
- **#236** log-injection in the #209 security-event log line —
  attacker-controlled `source_id` echoed via `%s` allowed newline
  injection of fake log entries. Switched to `%q`.

### Infrastructure

- **AWS KMS envelope encryption** (controlplane PR #21). Per-secret DEK
  via `kms.GenerateDataKey`; blob layout `[0x02][dek_len][enc_dek][nonce][ct]`.
  Dual-mode: v2 blobs via KMS, legacy blobs via static `SECRETS_ENCRYPTION_KEY`.
  Auto-routes by leading byte; no rewrap migration needed.
- **Grafana Cloud remote-write** (controlplane PR #19 + #20). In-process
  counter registry + hand-rolled protobuf encoder. `cp_requests_total`
  emitted on every request. Push loop to
  `prometheus-prod-32-prod-ca-east-0.grafana.net/api/prom/push` with
  Basic auth. User 3116422, token via GRAFANA_PROM_TOKEN Fly secret.
- **/cp/status deep-probe** (controlplane PR #24) for Betterstack.
  Pings Postgres with 2s budget; returns 503 on DB miss. Distinct from
  `/health`.
- **Legal pages** (controlplane PR #26/#27). Public `/legal/{terms,
  privacy,dpa,acceptable}` served from embedded markdown. Dark-theme
  HTML shell, minimal markdown→HTML renderer (no dep), path-traversal
  safe via slug allowlist. Smoke covered.
- **Scheduler reliability**: #95 panic-recover in tick(), #149
  independent heartbeat goroutine so long fires don't look stale on
  /admin/liveness, #207 concurrency-aware skip when workspace
  active_tasks>0.

### Features

- **#205** idle-loop reflection pattern in workspace-template. Opt-in
  via `idle_prompt` + `idle_interval_seconds` in `config.yaml`.
  Self-sends the idle prompt via platform A2A proxy every interval
  while `heartbeat.active_tasks == 0`. Hermes/Letta shape.
- **#208** Hermes Phase 1 multi-provider. 15 providers via
  `adapters/hermes/providers.py` registry (Nous, OpenRouter, OpenAI,
  Anthropic, xAI, Gemini, Qwen, GLM, Kimi, MiniMax, DeepSeek, Groq,
  Together, Fireworks, Mistral). Back-compat with PR2 key resolution
  preserved. 26 tests.
- **#198** A2A protocol compliance batch closing #173/#174/#175:
  `cancel()` emits `TaskStatusUpdateEvent(canceled, final=True)`,
  `stateTransitionHistory=True` in AgentCapabilities. *Note:* wired
  `push_sender=PushNotificationSender()` and this crashed on startup
  because PushNotificationSender is an abstract base class — reverted
  in #210.
- **#186** self-hosted macOS runner migration (described above).

### Code-review self-audit

Ran /code-review on my own batch merges, surfaced 8 🟡 issues, split
follow-ups into two PRs:

- **#228** (Go side): CanvasOrBearer invalid-bearer fall-through fix,
  `short()` helper to replace unsafe `[:N]` slices in scheduler.go,
  security-event log on source_id spoof. 6 new tests:
  `TestShort_helper`, `TestRecordSkipped_writesSkippedStatus`,
  `TestRecordSkipped_shortWorkspaceIDNoPanic`,
  `TestActivityHandler_Report_SourceIDSpoofRejected`,
  `TestActivityHandler_Report_MatchingSourceIDAccepted`,
  `TestHistory_IncludesErrorDetail`.
- **#232** (Python/docs): idle-loop hardening
  (`asyncio.get_running_loop()`, `IDLE_FIRE_TIMEOUT_SECONDS` clamped,
  typed `HTTPError`/`URLError`/catch-all, `add_done_callback` for
  fire-and-forget error logging). `idle_prompt` documented in
  `org-templates/molecule-dev/org.yaml` defaults. New
  `docs/runbooks/admin-auth.md` documenting the three middleware
  variants (AdminAuth strict, CanvasOrBearer soft, WorkspaceAuth
  per-id) + the three-question test for adding routes to
  CanvasOrBearer.

### Other merged fixes

- #122 canvas grid origin offset (nodes spawn at 100,100 not 0,0)
- #123 dark-theme a11y (input contrast, search dialog, kbd hints)
- #131 WCAG critical (ARIA live toasts, dialog focus trap, keyboard nav)
- #139 code-review plugins for Dev Lead + QA Engineer
- #149 scheduler heartbeat pulse (#140)
- #150 ecosystem-watch daily sweep (Microsoft Agent Framework, Vercel Open Agents)
- #157 ecosystem-watch PM sweep
- #161 e2e test mock fix for #125 EXISTS probe
- #187 `SetTrustedProxies(nil)` closes #179 rate-limit bypass
- #188 e2e auth headers on `/events` + `/bundles/export` post-#167
- #189 revert Security Auditor cron to 2x/day (closes #178 token-budget regression)
- #192 test regression lock for #170 `DELETE /secrets/:key`
- #197 reapply user's a6cfc5f bypass-setup-python to main (dropped by #186 squash)
- #206 surface cron `error_detail` in schedule history (#152 problem B)
- #210 revert PushNotificationSender ABC crash (#204)
- #211 migration runner skips `.down.sql` (data loss regression)
- #216 enable idle-loop pilot on Technical Researcher
- #223 reno-stars default plugins to browser-automation
- #225 auth_headers() on /registry/register (#215)
- #227 unit tests for plugins_install_pipeline.go (37 cases, #217)
- #231 Claude SDK stderr probe for rate-limit error attribution (#160)
- #235 auth_headers() on initial_prompt + idle loop (#220)

### Issues closed (by merge or factual correction)

#85, #93, #100, #101, #103, #104, #105, #115, #126 epic parent, #127,
#128, #129, #132, #134, #135, #136, #138, #140, #141, #142, #143, #144,
#145, #146, #147, #148, #151, #152 prob B, #153, #154, #156, #160
(diagnosed, not fixed), #163, #164, #165, #166, #168, #170, #171, #172,
#173, #174, #175, #176, #177, #178, #180, #181, #183, #184, #190, #191
(accepted risk), #195, #199 (fixed Fly token rotation), #201, #202,
#204, #211, #213, #214, #215, #217, #218, #219, #220, #221, #226, #229,
#230, #234.

### Outstanding — needs user

- **#126** Slack adapter (Phase-H product decision)
- **#160** Claude Max OAuth quota (wait for reset / upgrade / API key switch)
- **#191** self-hosted runner persistent-state docs (P3)
- **#199** Fly registry token — **resolved this session** but re-run
  of `publish-platform-image` pending runner capacity
- Stripe Atlas application (launch blocker, 2-week lead)

### Test counts (post-session)

- Platform Go: **816 test functions** (+70 this session — scheduler, handlers, middleware, db, crypto tests added across #95/#99/#106/#110/#119/#151/#167/#185/#187/#192/#200/#203/#206/#207/#210/#211/#212/#227/#228/#232/#234)
- Canvas vitest: **453 tests** (+0 structure, +0 new tests this session — UI/a11y patches)
- Workspace-template pytest: **1180 tests** (+40 this session — Hermes providers, a2a cancel, idle loop implicit)
- MCP server jest: **97 tests** (unchanged)

### Infra notes (not in any repo)

- FLY_API_TOKEN GH Actions secret rotated to a deploy token scoped to
  `molecule-tenant` (1-year expiry). Docs runbook update needed.
- Mac mini runner env has `RUNNER_TOOL_CACHE` + `AGENT_TOOLSDIRECTORY`
  overrides. Python install via Homebrew is required one-time prep.
- `molecule-monorepo` still private; Actions billing workaround is
  the self-hosted runner rather than flipping public or raising the
  cap.