molecule-ai-status-cd

Continuous deployment for the Enter OS status page — https://status.enteros.ai, self-hosted on tc-2. Polls molecule-ai/molecule-ai-status main; when HEAD changes, rebuilds and rolls out on tc-2, then asserts the live page's DATA and rolls back if the assertion misses.

This repo is the SSOT for the pipeline. Nothing about the deploy lives only on the box: deploy/rollout.sh, deploy/assert-live.sh, deploy/docker-compose.yml and deploy/overlay/ are copied to tc-2 fresh on every run, so a hand-edit there cannot become what deploys.

Why this repo exists — the failure mode it is built against

The status page had been returning HTTP 200 continuously while showing probe data frozen on 2026-05-11. Its /data/history/* was proxied to main, but the 5-minute probe publishes to the uptime-probe-results branch. So the page rendered permanently green off a snapshot, structurally unable to go red, for 75 days, while the probes themselves ran flawlessly every 5 minutes. Every uptime check anyone could have pointed at it would have passed the whole time.

A status page that renders but reports stale data is worse than one that is honestly down. So this pipeline asserts the data, not the page: heartbeat age, per-monitor history age, byte-parity of the served monitor list against main on Gitea, and summary drift — all from outside, against the real public hostname.

The deployment

Host	tc-2 (100.64.0.5, tailnet), /home/molecule/molecule-ai-status/
Runtime	docker compose, one service app, container molecule-ai-status-app
Port	3400 on the host → 3105 in the container. Taken on tc-2: 3000 (JRS), 3100 (enteros landing), 5432/5433/6543/8000/8443/9000/9001 (Supabase + JRS Postgres/MinIO); 3200/3300 are claimed by the market/docs moves. The container port stays 3105 — the port the app's nginx config listens on — so the config is portable.
Image	molecule-ai-status:<sha12>, built on tc-2, pinned in .env as APP_IMAGE. :latest is banned — the deployed tag is the deploy state and a rollback is a tag change.
Edge	molecule-clients-k3s cloudflared tunnel (0502fb39-…, remotely-managed ingress) → http://100.64.0.5:3400
Public name	status.enteros.ai (canonical). status.moleculesai.app is retired and 308s here, in-app (see molecule-ai-status#52).
Data	Fetched by the browser from same-origin /data/*, reverse-proxied to Gitea raw: history/* → uptime-probe-results (what the probe writes), .upptimerc.yml → main (the reviewed monitor list).

What is NOT in this pipeline: the probe

The 5-minute Probe + commit workflow lives in the app repo, runs on the fleet, and publishes to uptime-probe-results. This deploy only ever reads that branch. There is no git push anywhere in deploy/ or in the workflow, and the contract (R12) fails the build if one appears — a deploy able to write to that branch could destroy monitoring history.

Moving the site therefore does not touch the probe at all: the page's origin changed, the data path did not.

Why the build happens on tc-2

Same reasoning as the co-located enteros-landing-cd: the target is docker-compose, not k3s, so there is no ctr import step to mirror, and shipping an image over the tailnet on every poll is pure cost. The invariant that matters — no CI on the prod box — is preserved: the job runs on the fleet runner (robot-1), and tc-2 is a client-serving box.

The post-rollout assertion — deploy/assert-live.sh

Run from the runner, against the real public hostname, so the Cloudflare + tunnel path is proven too. Freshness bound is MAX_AGE_S (default 1200 s = 20 min ≈ 4 missed probe runs; the page's own "Monitoring degraded" threshold is 15 min).

1. /build.json commit == the sha we just deployed — turns "a container restarted" into "this commit is live".
2. / is 200 and is the status page (title + meta description).
3. /app.js is 200, non-trivial, and really the dashboard script — an index.html fallback would 200 for it and render a blank page.
4. /data/.upptimerc.yml is byte-identical to main on Gitea — proves the monitor list shown is the reviewed one and that it is proxied live, not baked into the image.
5. The monitoring heartbeat is fresh. The single check that would have caught the 75-day dark period.
6. Every monitor in that config has a fresh history file — per-monitor, so one dead slug cannot hide behind six live ones. The monitor list is parsed from what the site served, so adding a monitor upstream needs no change here.
7. summary.json is well-formed and lists no monitor the config has dropped (the drift that had the live page advertising a retired docs URL).
8. status.moleculesai.app still 308s to the canonical host with the path preserved (molecule-ai-status#52).
9. No http:// subresources.

Any miss → the job repins the previous image (rollout.sh --pin) and exits non-zero.

rollout.sh additionally refuses to finish unless the container reports healthy (the compose healthcheck — the real signal, polled, with a never-reached 180 s safety cap), is serving the expected commit locally, and its /data proxy returns a fresh heartbeat from the box.

The no-op path still asserts

When main has not moved there is nothing to build — but the data can still have gone stale (the probe could have stopped; the results branch could have become unreachable). So the workflow re-runs the assertion anyway and goes red if the page has started lying. The CD is therefore also the alarm on the alarm.

Build overlay

deploy/overlay/ holds Dockerfile and nginx.docker.conf, byte-identical to molecule-ai-status#52, and they are injected only when the checkout does not carry them. They serve the public page today but existed only in an unpushed commit on a workstation, so the container serving status.enteros.ai had no source of truth. #52 puts them under version control; the moment it merges this overlay stops firing and nothing changes.

rollout.sh then fails the deploy loudly unless the effective nginx config proxies /data/history/* to uptime-probe-results and the config file to main — the anti-stale-data invariant, enforced before the image is even built rather than only after it is live.

Secrets (repo Actions secrets)

secret	what
TC2_DEPLOY_KEY_B64	base64 of a dedicated ed25519 private key; its public key is in tc-2's molecule authorized_keys as molecule-ai-status-cd. Rotate: regenerate, re-authorize on tc-2, update the secret.
SRC_GITEA_TOKEN	Not set, deliberately. molecule-ai/molecule-ai-status is public, so the HEAD-sha lookup needs no credential, and parking a broad token here to read a public sha would be scope for nothing. The workflow still sends the header when the secret exists, so making the app repo private later is a one-secret change.

No runtime secrets. The page is static and reads its data from Gitea at request time, so /home/molecule/molecule-ai-status/.env holds only APP_IMAGE. It is still operator state: the deploy rewrites exactly that one line and never creates or prints the file.

CI (.gitea/workflows/ci.yml)

No application code, so CI protects the deploy contract itself: workflow-yaml, shell (shellcheck over deploy/*.sh and every workflow run: block), deploy-contract (.gitea/scripts/deploy_contract.py — 12 rules), and all-required as the sentinel. tests/test_deploy_contract.py deletes each mechanism in turn and proves the matching rule actually goes red, so the contract cannot rot into a regex that passes on anything.

Operate

# deploy now (or wait <=5 min for the poll)
#   -> workflow_dispatch on this repo

# manual rollout / rollback on tc-2
ssh -i ~/.ssh/moleculesai_operator molecule@100.64.0.5
cd /home/molecule/molecule-ai-status
./.deploy/rollout.sh <40-hex-sha>                       # deploy a commit
./.deploy/rollout.sh --pin molecule-ai-status:<sha12>   # roll back to a built image
docker compose logs -f app

# assert the live page by hand (this is also how you check the DATA is real)
bash deploy/assert-live.sh https://status.enteros.ai [sha]

Edge helper (one-shot, kept for reproducibility & rollback)

deploy/tunnel-ingress.sh adds a hostname to the shared tunnel additively. It refuses to write unless every pre-existing rule survives and the http_status:404 catch-all is still last. That tunnel also fronts minoriskincare.com, jrsautocustoms.com and enteros.ai; a clobbered ingress list would take live client sites down.

CF_TUNNEL_TOKEN=... deploy/tunnel-ingress.sh snapshot before.json
CF_TUNNEL_TOKEN=... CF_DNS_TOKEN=... deploy/tunnel-ingress.sh \
    add status.enteros.ai http://100.64.0.5:3400 699a9a3e0d299fd10cb4b2c0f2ff65a3

Rollback of the move is one DNS record: repoint the status.enteros.ai CNAME back to cc9bf889-902a-40cf-877e-9c56abf471ec.cfargotunnel.com, whose ingress rule for that hostname is deliberately left in place on the prod box.

Follow-ups (deliberately not done here)

1. Merge molecule-ai-status#52, which upstreams Dockerfile + nginx.docker.conf into the app repo and retires status.moleculesai.app. The overlay here is byte-identical and becomes a no-op on merge; until then it is what puts those files under version control.
2. Security headers. Vercel used to add X-Frame-Options, X-Content-Type-Options, Referrer-Policy and Permissions-Policy (site/vercel.json); the nginx config does not, so they were silently lost when the page moved off Vercel. Fixing it belongs in the app repo's nginx.docker.conf (i.e. on top of #52), not in this overlay, which is kept byte-identical on purpose.
3. Retire the prod-box container. molecule-status on the home PC is still running and the old tunnel still has ingress rules for both status hostnames — kept as the rollback path. Remove once the tc-2 deployment has a few days of green.