cancel-in-progress: false queues new runs so the single mac mini
runner doesn't fight itself when pushes stack during rebases or
cross-PR contention. Existing e2e-api.yml already has this pattern.
Fixes: 19 queued runs on single self-hosted runner (02:55 UTC snapshot)
Co-authored-by: Molecule AI Fullstack (floater) <fullstack-floater@agents.moleculesai.app>
Two changes to relieve macOS arm64 runner contention:
1. `changes` job: runs on `ubuntu-latest` instead of
`[self-hosted, macos, arm64]`. This job does a plain `git diff`
— it has zero macOS dependencies. Moving it off the runner frees
the slot immediately on every workflow trigger.
2. Add workflow-level concurrency to `ci.yml`:
`concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true`
Without this, every new push to a PR or main queues a full new
workflow run, each competing for the same single runner. With
`cancel-in-progress: true`, stale in-flight CI runs are cancelled
when a newer commit arrives — the runner always runs the latest
state, not a backlog of old ones.
Context: the self-hosted macOS arm64 runner is shared by ci.yml,
e2e-api.yml, canary-verify.yml, and publish-*.yml. The combination of
(1) the `changes` job holding the runner during `fetch-depth: 0`
checkout on every trigger, and (2) no workflow-level cancellation
caused 100+ queued runs with 0 in-progress.
Follow-up candidates (need verification before changing):
- platform-build: Go build may work on ubuntu-latest (no macOS deps)
- canvas-build: Next.js build may work on ubuntu-latest
- python-lint: needs `setup-python` instead of Homebrew Python
Co-authored-by: Molecule AI Infra-SRE <infra-sre@agents.moleculesai.app>
PRs targeting staging got no CI because the workflow only triggered
on main. Now runs on both main and staging pushes + PRs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove compiled workspace-server/server binary from git
- Fix .gitignore, .gitattributes, .githooks/pre-commit for renamed dirs
- Fix CI workflow path filters (workspace-template → workspace)
- Replace real EC2 IP and personal slug in test_saas_tenant.sh
- Scrub molecule-controlplane references in docs
- Fix stale workspace-template/ paths in provisioner, handlers, tests
- Clean tracked Python cache files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Replace hardcoded Cloudflare account/zone/KV IDs in wrangler.toml
with placeholders; add wrangler.toml to .gitignore, ship .example
- Replace real EC2 IPs in docs with <EC2_IP> placeholders
- Redact partial CF API token prefix in retrospective
- Parameterize Langfuse dev credentials in docker-compose.infra.yml
- Replace Neon project ID in runbook with <neon-project-id>
Community:
- Add CONTRIBUTING.md (build, test, branch conventions, CI info)
- Add CODE_OF_CONDUCT.md (Contributor Covenant 2.1)
Cleanup:
- Replace personal runner username/machine name in CI + PLAN.md
- Replace personal tenant URL in MCP setup guide
- Replace personal author field in bundle-system doc
- Replace personal login in webhook test fixture
- Rewrite cryptominer incident reference as generic security remediation
- Remove private repo commit hashes from PLAN.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HEAD~1 doesn't work for merge commits. Use github.event.before (the
previous main tip) for push events and github.event.pull_request.base.sha
for PRs. fetch-depth: 0 ensures both SHAs are available.
Fallback: if BASE is empty (new branch), run all jobs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dorny/paths-filter uses Docker internally which doesn't work on the
self-hosted macOS arm64 runner — every CI run since the path filter
change has failed with no jobs.
Replace with a simple git diff against HEAD~1 that checks path prefixes.
Same behavior, no Docker dependency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI now detects which paths changed and skips irrelevant jobs:
- Platform (Go): only runs when platform/** changes
- Canvas (Next.js): only runs when canvas/** changes
- Python Lint: only runs when workspace-template/** changes
- Shellcheck: only runs when tests/e2e/** or scripts/** change
- E2E API: only runs when platform/** or tests/e2e/** change
Docs-only PRs (*.md, docs/**) skip all 5 jobs, saving ~15 min of
runner time per PR. Uses dorny/paths-filter for the CI workflow and
native paths: filter for the E2E workflow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Job-level `concurrency.cancel-in-progress: false` only prevents sibling jobs
from killing each other — it does not protect the parent workflow run from
being cancelled when a new push arrives. Every PR push was cancelling the
in-progress E2E run, forcing manual `gh run rerun` across 7+ active PRs.
Fix: move e2e-api into `.github/workflows/e2e-api.yml` with a workflow-level
concurrency group (`e2e-api-${{ github.ref }}`, cancel-in-progress: false).
New pushes now queue behind the running E2E job instead of cancelling it.
Fast jobs (platform-build, canvas-build, shellcheck, python-lint) stay in
ci.yml and retain normal run-level cancellation for quick iteration feedback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove plugins/, workspace-configs-templates/, org-templates/ dirs (now
in standalone repos). Add manifest.json listing all 33 repos and
scripts/clone-manifest.sh to clone them. Both Dockerfiles now use the
manifest script instead of 33 hardcoded git-clone lines.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#399.
## Root cause
`publish-platform-image.yml` existed for the Go platform image but there
was no equivalent for the canvas. After every canvas PR merged, CI ran
`npm run build` and passed — but the live container at :3000 was never
updated. The `canvas-deploy-reminder` job only posted a comment asking
operators to manually rebuild, which was consistently missed.
## What this adds
- `.github/workflows/publish-canvas-image.yml`: triggers on `canvas/**`
changes to main (and `workflow_dispatch`). Mirrors the platform workflow:
macOS Keychain isolation, QEMU for linux/amd64, Buildx, GHCR push with
`:latest` + `:sha-<7>` tags.
- `NEXT_PUBLIC_PLATFORM_URL` / `NEXT_PUBLIC_WS_URL` resolve from
`workflow_dispatch` inputs → `CANVAS_PLATFORM_URL` / `CANVAS_WS_URL`
repo secrets → `localhost:8080` defaults (safe for self-hosted dev).
- Inputs are passed via env vars (not direct `${{ }}` interpolation) to
prevent shell injection from string inputs.
- `docker-compose.yml`: adds `image: ghcr.io/molecule-ai/canvas:latest`
to the canvas service so `docker compose pull canvas && docker compose
up -d canvas` applies the new image. `build:` is retained for local
development. Adds a comment clarifying that `NEXT_PUBLIC_*` runtime env
vars are ignored by the standalone bundle (build-time only).
- `ci.yml`: updates `canvas-deploy-reminder` commit comment to reference
`docker compose pull` as the fast path, with `docker compose build` as
the local-source fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now that the Molecule-AI org has two self-hosted Apple-silicon runners
(`hongming-m1-mini` + `hongming-m1-mini-2`) servicing the same label set,
two CI runs could execute the e2e-api job concurrently. Each run starts
fixed-name docker containers (`molecule-ci-postgres`, `molecule-ci-redis`)
bound to host ports 15432/16379 — a collision means the second run fails
with "container name already in use" or "port already in use".
Adds a workflow-level `concurrency: e2e-api` group to the job so GitHub
Actions serializes e2e-api executions globally regardless of which runner
picks them up. `cancel-in-progress: false` ensures later runs queue
rather than cancelling the in-flight one (we want every PR's e2e check
to actually execute, not get skipped by a newer push).
Tradeoff: e2e-api is now effectively single-threaded across the whole
org. Measured duration is ~1-2 min per run, so the added serialization
latency is small relative to total CI wall time. All other jobs still
parallelize across both runners.
#186's squash-merge commit (3ff40c4b) took 15e15a21 (AGENT_TOOLSDIRECTORY
override) but missed a6cfc5f (bypass setup-python entirely) which was
pushed to the PR branch after the merge was initiated. The merge
commit still has the old setup-python@v5 job config.
Applies a6cfc5f's ci.yml verbatim via git checkout. Restores the
Homebrew-python3.11 bypass path that the user prototyped. No other
changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): migrate all jobs to self-hosted macOS arm64 runner
Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.
Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):
- e2e-api: `services: postgres/redis` replaced with inline `docker run`
steps. Ports remapped to 15432/16379 to avoid collision with anything
the host may already expose on the standard ports. Containers are named
(`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
`if: always()` step. Postgres readiness is still gated on pg_isready
via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
Replaced with a direct `shellcheck` invocation (pre-installed on the
runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
explicit `platforms: linux/amd64` on both `docker/build-push-action`
invocations. The runner is arm64 but Fly tenant machines pull amd64,
so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
behavior is unchanged.
Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job
* fix(ci): set AGENT_TOOLSDIRECTORY for python-lint on self-hosted runner
setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.
Fixes the only failing job in CI run 24469156329 on PR #186.
---------
Co-authored-by: Hongming Wang <HongmingWang-Rabbit@users.noreply.github.com>
Adds a `canvas-deploy-reminder` job to ci.yml that fires on every
push to main once `canvas-build` passes. It posts a commit comment via
the built-in GITHUB_TOKEN (no new secrets needed) reminding whoever
monitors CI to run:
cd /g/personal_programs/molecule-monorepo
git pull origin main
docker compose build canvas && docker compose up -d canvas
The comment includes the commit SHA and a direct link to the build log.
Rationale: 5 consecutive merge cycles (PRs #21, #25, #30, #32, #34)
went undeployed because there is no auto-deploy hook and the manual
step was silently forgotten. A commit comment on the merge commit is
the lowest-friction reminder that requires no external secrets or infra.
Does NOT run on PRs — only on direct pushes to main (i.e. post-merge).
Uses `needs: canvas-build` so the reminder only fires after build+tests
pass; a failing build produces no comment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _extract_token.py: narrow `except Exception` to
`except (json.JSONDecodeError, ValueError)`. Prevents swallowing
KeyboardInterrupt in edge cases and documents intent clearly.
- ci.yml shellcheck job: switch to ludeeus/action-shellcheck@master
(caches shellcheck binary across runs; saves the apt-get install).
Both changes verified locally: YAML parses, extract script still
extracts valid tokens and prints the stderr warning on malformed JSON.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 5 suggestions from the latest review pass.
## tests/e2e/_extract_token.py (new)
Extracted the 14-line python-in-bash heredoc from _lib.sh into a real
Python file. Easier to edit, fewer escaping traps, same behavior.
Shell helper now just shells out to it.
## tests/e2e/_lib.sh
- Replaced inline python with: python3 "$(dirname "${BASH_SOURCE[0]}")/_extract_token.py"
- Removed redundant sys.exit(0) as part of the extraction
## Shellcheck-clean scripts (new CI job enforces)
- Removed dead captures: BEFORE_COUNT (test_activity_e2e.sh), ORIG_SKILLS,
REIMPORT_SKILLS (test_api.sh), QA_TOKEN (test_comprehensive_e2e.sh)
- Renamed unused loop vars `i`, `j` -> `_` in 4 sites
- Added `# shellcheck disable=SC2046` on the two intentional word-splits
in test_claude_code_e2e.sh (docker stop/rm of multiple container IDs)
- Removed a useless re-register of QA mid-script (was done in Section 2)
## CI (.github/workflows/ci.yml)
- Replaced `sudo apt-get install postgresql-client` + psql with a direct
`docker exec` into the existing postgres:16 service container. Saves
~10-20s per CI run.
- Added new `shellcheck` job that lints tests/e2e/*.sh on every PR.
Local: shellcheck --severity=warning returns 0 across all 5 scripts.
## Verification
- go test -race ./internal/handlers/... : pass
- mcp-server: 96/96 jest
- canvas: 357/357 vitest + clean build
- tests/e2e/test_api.sh: 62/62
- tests/e2e/test_comprehensive_e2e.sh: 67/67
- shellcheck tests/e2e/*.sh : clean
- CI YAML: valid
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Last sweep of code-review items before merging PR #5.
## _lib.sh cleanup
- Removed unused e2e_register and e2e_heartbeat helpers (dead code —
no caller ever invoked them)
- Standardized on $BASE variable set via : "${BASE:=...}" so every
script uses one name (was mixed $BASE / $e2e_base)
- e2e_extract_token now writes stderr warnings on JSON parse failure
or missing auth_token, instead of silently returning empty. Previous
behavior made downstream "missing workspace auth token" 401s much
harder to diagnose
## Script cleanup
- test_api.sh, test_comprehensive_e2e.sh, test_activity_e2e.sh all
drop the redundant `e2e_base + BASE="$e2e_base"` aliasing; sourcing
_lib.sh sets BASE via : "${BASE:=...}" default
## CI hardening (.github/workflows/ci.yml)
- Postgres credentials now match .env.example (dev:dev — was
molecule:molecule, caused confusion for local repros)
- Added Go module cache via actions/setup-go cache:true +
cache-dependency-path: platform/go.sum. ~30s cold-run improvement
- New pre-E2E step asserts migrations actually ran by checking for
the 'workspaces' table. Catches future migration-author mistakes
before they surface as obscure E2E failures
## Follow-up issue
Filed Molecule-AI/molecule-monorepo#6 for the deterministic token-
mint admin endpoint. PR #5 uses an empirical "beat the container"
race (5/5 wins in benchmarks); issue #6 tracks the real fix for
any future CI load that invalidates the assumption.
## Verification
- bash tests/e2e/test_api.sh -> 62/62
- bash tests/e2e/test_comprehensive_e2e.sh -> 67/67
- python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" -> ok
## Operational note
Hourly PR-triage + issue-pickup cron scheduled this session (job id
0328bc8f, fires at :17 past each hour). Runtime reports it as
session-only despite durable:true — re-invoke via /loop or
CronCreate in a fresh session if needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow-up to the test_api.sh fix. Same Phase 30.1 + 30.6 staleness
existed in the other E2E scripts; same pattern applied.
## New tests/e2e/_lib.sh
Shared bash helpers so future scripts don't reimplement:
- e2e_extract_token — parse auth_token from register response
- e2e_register — register + echo token
- e2e_heartbeat — heartbeat with bearer auth
- e2e_cleanup_all_workspaces — pre-test state reset
## test_comprehensive_e2e.sh (14 fail -> 0 fail)
Root cause was deeper than test_api.sh: the script creates workspaces
at Section 2 but doesn't register them until Section 3. In between,
the platform provisioner spawns the Docker container, whose main.py
calls /registry/register first and claims the single-issue token.
The script's later register gets no auth_token back.
Fix: register each workspace immediately after POST /workspaces,
beating the container to the token. Empirically 5/5 wins in a tight
loop. PM/Dev/QA tokens captured at creation time; bearer auth threaded
through all heartbeat/update-card/discover/peers calls.
Removed the duplicate register calls in Section 3/4 that followed
(tokens already captured).
Result: 53/68 -> 67/67 (one duplicate check dropped).
## test_activity_e2e.sh
Same pattern applied on faith. Script still SKIPs cleanly when no
online agent is present; when an agent IS online, it now re-registers
it to mint a fresh bearer token and threads Authorization: Bearer on
the 3 heartbeat calls.
## test_api.sh refactor
Now sources _lib.sh and uses the shared helpers. No behavior change,
still 62/62.
## .github/workflows/ci.yml — new e2e-api job
Spins up Postgres 16 + Redis 7 as GitHub Actions services, builds the
platform binary, runs it in background with DATABASE_URL/REDIS_URL,
polls /health for 30s, then runs tests/e2e/test_api.sh. On failure
dumps platform.log for triage. 10-min job timeout.
This is the watchdog that would have caught Phase 30.1 auth drift
the day it landed. Picks test_api.sh not test_comprehensive_e2e.sh
because the latter depends on Docker-in-Docker for container
provisioning which is heavier than a PR gate should carry.
## Verification
- bash tests/e2e/test_api.sh -> 62/62
- bash tests/e2e/test_comprehensive_e2e.sh -> 67/67
- bash tests/e2e/test_activity_e2e.sh -> cleanly SKIPs (no agent)
- go build ./... -> clean
- .github/workflows/ci.yml -> valid YAML, new job added
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
