Every publish-platform-image run since the 3ff40c4 self-hosted runner
migration has been failing with two runner-level issues that the
workflow now works around (keychain) or surfaces clearly (path):
1. "error storing credentials - err: exit status 1, out:
'User interaction is not allowed. (-25308)'"
docker/login-action tries to persist the GHCR + Fly tokens in the
macOS Keychain, but the Mac mini runner runs as a non-interactive
launchd service without an unlocked desktop session — keychain
access raises -25308. Fix: set DOCKER_CONFIG to a per-run temp dir
containing a plain config.json before the login step so credentials
land in a file, not the keychain. This is the same trick the
GitHub-hosted macos runners use in docker action examples.
2. "Unexpected error attempting to determine if executable file
exists '/usr/local/bin/docker': Error: EACCES: permission denied,
stat '/usr/local/bin/docker'"
Not a workflow bug — the runner literally can't read the Docker
binary path. Adds a diagnostic step before QEMU/buildx setup that
prints: PATH, `command -v docker`, `docker --version`, and
`ls -la` on both /usr/local/bin/docker and /opt/homebrew/bin/docker.
Surfacing these in the log means the next failure (if any) shows
the actual problem instead of hiding behind a cryptic buildx error.
Does NOT fix the root cause of #2 — that needs the user to SSH into
the Mac mini runner and reinstall / re-permission Docker Desktop
(or switch to Colima/OrbStack). The diagnostic output will tell us
exactly which path is broken.
The 20+ queued CI runs from `ci.yml` are unrelated to this PR — they
are stuck because the self-hosted runner has severely degraded queue
throughput (runs wait 2+ hours before being picked up). That's a
separate runner-health issue tracked as a user action in the triage
report.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
#186's squash-merge commit (3ff40c4b) took 15e15a21 (AGENT_TOOLSDIRECTORY
override) but missed a6cfc5f (bypass setup-python entirely) which was
pushed to the PR branch after the merge was initiated. The merge
commit still has the old setup-python@v5 job config.
Applies a6cfc5f's ci.yml verbatim via git checkout. Restores the
Homebrew-python3.11 bypass path that the user prototyped. No other
changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): migrate all jobs to self-hosted macOS arm64 runner
Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.
Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):
- e2e-api: `services: postgres/redis` replaced with inline `docker run`
steps. Ports remapped to 15432/16379 to avoid collision with anything
the host may already expose on the standard ports. Containers are named
(`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
`if: always()` step. Postgres readiness is still gated on pg_isready
via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
Replaced with a direct `shellcheck` invocation (pre-installed on the
runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
explicit `platforms: linux/amd64` on both `docker/build-push-action`
invocations. The runner is arm64 but Fly tenant machines pull amd64,
so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
behavior is unchanged.
Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job
* fix(ci): set AGENT_TOOLSDIRECTORY for python-lint on self-hosted runner
setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.
Fixes the only failing job in CI run 24469156329 on PR #186.
---------
Co-authored-by: Hongming Wang <HongmingWang-Rabbit@users.noreply.github.com>
Post-mortem on the failed publish-platform-image run on main (PR #82):
Fly's Docker registry requires username EXACTLY equal to "x". My
code-review "readability fix" changing it to "molecule-ai" caused
every push to return 401 Unauthorized. Verified locally:
echo $FLY_API_TOKEN | docker login registry.fly.io -u x --password-stdin
→ Login Succeeded
echo $FLY_API_TOKEN | docker login registry.fly.io -u molecule-ai --password-stdin
→ 401 Unauthorized
Lesson: don't second-guess docs that specify a literal value. Comment
now says "MUST be literal 'x'" with a 2026-04-15 verification note to
prevent future regressions.
Code-review process improvement: when reviewing a change against a
vendor API, prefer "preserve exact doc-specified values" over readability
suggestions. Logged as a cron-learning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addresses PR #82 code review: 🟡×3 + 🔵×5.
- Fly registry login username: 'x' → 'molecule-ai' + explanatory comment.
- Build & push split into two steps (GHCR / Fly registry) so a single-
registry outage can't fail the other. Second step uses 'if: always()'
to ensure Fly mirror runs even if GHCR push flakes.
- docs/runbooks/saas-secrets.md: full secret map + rotation procedures
for every SaaS credential, with danger-case callouts. Documents the
coupled FLY_API_TOKEN (lives in GHA secret AND fly secrets — must be
rotated in both).
- CLAUDE.md: new 'SaaS ops' section linking to the runbook.
Keeps ghcr.io/molecule-ai/platform private (per CEO direction — open-
source when full SaaS ships) while still letting the private control
plane's Fly provisioner boot tenant machines: Fly auto-authenticates
same-org machines against registry.fly.io, no per-tenant pull
credentials to wire.
Workflow now logs into both GHCR (using built-in GITHUB_TOKEN) and
Fly registry (using FLY_API_TOKEN secret) and pushes the same image to
four tags total:
- ghcr.io/molecule-ai/platform:latest
- ghcr.io/molecule-ai/platform:sha-<short>
- registry.fly.io/molecule-tenant:latest
- registry.fly.io/molecule-tenant:sha-<short>
Secret added via `gh secret set FLY_API_TOKEN` on the public repo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase B.2 companion to the private molecule-controlplane provisioner PR.
On every push to main that touches platform/**, builds platform/Dockerfile
and pushes to GHCR with two tags:
- :latest (floating, always main's tip)
- :sha-<short-commit> (immutable, pin-friendly)
Cache via GitHub Actions cache (cache-from: type=gha). Workflow_dispatch
trigger so we can re-publish after a docs-only merge if needed.
The private molecule-controlplane sets TENANT_IMAGE=ghcr.io/molecule-ai/platform:<tag>
and the provisioner creates each tenant Fly Machine from this image. Staying
on the same base image across tenants keeps upgrades atomic.
CLAUDE.md updated to document the new workflow in the CI pipeline section.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a `canvas-deploy-reminder` job to ci.yml that fires on every
push to main once `canvas-build` passes. It posts a commit comment via
the built-in GITHUB_TOKEN (no new secrets needed) reminding whoever
monitors CI to run:
cd /g/personal_programs/molecule-monorepo
git pull origin main
docker compose build canvas && docker compose up -d canvas
The comment includes the commit SHA and a direct link to the build log.
Rationale: 5 consecutive merge cycles (PRs #21, #25, #30, #32, #34)
went undeployed because there is no auto-deploy hook and the manual
step was silently forgotten. A commit comment on the merge commit is
the lowest-friction reminder that requires no external secrets or infra.
Does NOT run on PRs — only on direct pushes to main (i.e. post-merge).
Uses `needs: canvas-build` so the reminder only fires after build+tests
pass; a failing build produces no comment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _extract_token.py: narrow `except Exception` to
`except (json.JSONDecodeError, ValueError)`. Prevents swallowing
KeyboardInterrupt in edge cases and documents intent clearly.
- ci.yml shellcheck job: switch to ludeeus/action-shellcheck@master
(caches shellcheck binary across runs; saves the apt-get install).
Both changes verified locally: YAML parses, extract script still
extracts valid tokens and prints the stderr warning on malformed JSON.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 5 suggestions from the latest review pass.
## tests/e2e/_extract_token.py (new)
Extracted the 14-line python-in-bash heredoc from _lib.sh into a real
Python file. Easier to edit, fewer escaping traps, same behavior.
Shell helper now just shells out to it.
## tests/e2e/_lib.sh
- Replaced inline python with: python3 "$(dirname "${BASH_SOURCE[0]}")/_extract_token.py"
- Removed redundant sys.exit(0) as part of the extraction
## Shellcheck-clean scripts (new CI job enforces)
- Removed dead captures: BEFORE_COUNT (test_activity_e2e.sh), ORIG_SKILLS,
REIMPORT_SKILLS (test_api.sh), QA_TOKEN (test_comprehensive_e2e.sh)
- Renamed unused loop vars `i`, `j` -> `_` in 4 sites
- Added `# shellcheck disable=SC2046` on the two intentional word-splits
in test_claude_code_e2e.sh (docker stop/rm of multiple container IDs)
- Removed a useless re-register of QA mid-script (was done in Section 2)
## CI (.github/workflows/ci.yml)
- Replaced `sudo apt-get install postgresql-client` + psql with a direct
`docker exec` into the existing postgres:16 service container. Saves
~10-20s per CI run.
- Added new `shellcheck` job that lints tests/e2e/*.sh on every PR.
Local: shellcheck --severity=warning returns 0 across all 5 scripts.
## Verification
- go test -race ./internal/handlers/... : pass
- mcp-server: 96/96 jest
- canvas: 357/357 vitest + clean build
- tests/e2e/test_api.sh: 62/62
- tests/e2e/test_comprehensive_e2e.sh: 67/67
- shellcheck tests/e2e/*.sh : clean
- CI YAML: valid
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Last sweep of code-review items before merging PR #5.
## _lib.sh cleanup
- Removed unused e2e_register and e2e_heartbeat helpers (dead code —
no caller ever invoked them)
- Standardized on $BASE variable set via : "${BASE:=...}" so every
script uses one name (was mixed $BASE / $e2e_base)
- e2e_extract_token now writes stderr warnings on JSON parse failure
or missing auth_token, instead of silently returning empty. Previous
behavior made downstream "missing workspace auth token" 401s much
harder to diagnose
## Script cleanup
- test_api.sh, test_comprehensive_e2e.sh, test_activity_e2e.sh all
drop the redundant `e2e_base + BASE="$e2e_base"` aliasing; sourcing
_lib.sh sets BASE via : "${BASE:=...}" default
## CI hardening (.github/workflows/ci.yml)
- Postgres credentials now match .env.example (dev:dev — was
molecule:molecule, caused confusion for local repros)
- Added Go module cache via actions/setup-go cache:true +
cache-dependency-path: platform/go.sum. ~30s cold-run improvement
- New pre-E2E step asserts migrations actually ran by checking for
the 'workspaces' table. Catches future migration-author mistakes
before they surface as obscure E2E failures
## Follow-up issue
Filed Molecule-AI/molecule-monorepo#6 for the deterministic token-
mint admin endpoint. PR #5 uses an empirical "beat the container"
race (5/5 wins in benchmarks); issue #6 tracks the real fix for
any future CI load that invalidates the assumption.
## Verification
- bash tests/e2e/test_api.sh -> 62/62
- bash tests/e2e/test_comprehensive_e2e.sh -> 67/67
- python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" -> ok
## Operational note
Hourly PR-triage + issue-pickup cron scheduled this session (job id
0328bc8f, fires at :17 past each hour). Runtime reports it as
session-only despite durable:true — re-invoke via /loop or
CronCreate in a fresh session if needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow-up to the test_api.sh fix. Same Phase 30.1 + 30.6 staleness
existed in the other E2E scripts; same pattern applied.
## New tests/e2e/_lib.sh
Shared bash helpers so future scripts don't reimplement:
- e2e_extract_token — parse auth_token from register response
- e2e_register — register + echo token
- e2e_heartbeat — heartbeat with bearer auth
- e2e_cleanup_all_workspaces — pre-test state reset
## test_comprehensive_e2e.sh (14 fail -> 0 fail)
Root cause was deeper than test_api.sh: the script creates workspaces
at Section 2 but doesn't register them until Section 3. In between,
the platform provisioner spawns the Docker container, whose main.py
calls /registry/register first and claims the single-issue token.
The script's later register gets no auth_token back.
Fix: register each workspace immediately after POST /workspaces,
beating the container to the token. Empirically 5/5 wins in a tight
loop. PM/Dev/QA tokens captured at creation time; bearer auth threaded
through all heartbeat/update-card/discover/peers calls.
Removed the duplicate register calls in Section 3/4 that followed
(tokens already captured).
Result: 53/68 -> 67/67 (one duplicate check dropped).
## test_activity_e2e.sh
Same pattern applied on faith. Script still SKIPs cleanly when no
online agent is present; when an agent IS online, it now re-registers
it to mint a fresh bearer token and threads Authorization: Bearer on
the 3 heartbeat calls.
## test_api.sh refactor
Now sources _lib.sh and uses the shared helpers. No behavior change,
still 62/62.
## .github/workflows/ci.yml — new e2e-api job
Spins up Postgres 16 + Redis 7 as GitHub Actions services, builds the
platform binary, runs it in background with DATABASE_URL/REDIS_URL,
polls /health for 30s, then runs tests/e2e/test_api.sh. On failure
dumps platform.log for triage. 10-min job timeout.
This is the watchdog that would have caught Phase 30.1 auth drift
the day it landed. Picks test_api.sh not test_comprehensive_e2e.sh
because the latter depends on Docker-in-Docker for container
provisioning which is heavier than a PR gate should carry.
## Verification
- bash tests/e2e/test_api.sh -> 62/62
- bash tests/e2e/test_comprehensive_e2e.sh -> 67/67
- bash tests/e2e/test_activity_e2e.sh -> cleanly SKIPs (no agent)
- go build ./... -> clean
- .github/workflows/ci.yml -> valid YAML, new job added
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
