* chore(ci): migrate all jobs to self-hosted macOS arm64 runner
Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.
Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):
- e2e-api: `services: postgres/redis` replaced with inline `docker run`
steps. Ports remapped to 15432/16379 to avoid collision with anything
the host may already expose on the standard ports. Containers are named
(`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
`if: always()` step. Postgres readiness is still gated on pg_isready
via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
Replaced with a direct `shellcheck` invocation (pre-installed on the
runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
explicit `platforms: linux/amd64` on both `docker/build-push-action`
invocations. The runner is arm64 but Fly tenant machines pull amd64,
so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
behavior is unchanged.
Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job
* fix(ci): set AGENT_TOOLSDIRECTORY for python-lint on self-hosted runner
setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.
Fixes the only failing job in CI run 24469156329 on PR #186.
---------
Co-authored-by: Hongming Wang <HongmingWang-Rabbit@users.noreply.github.com>
* chore(ci): migrate all jobs to self-hosted macOS arm64 runner
Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.
Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):
- e2e-api: `services: postgres/redis` replaced with inline `docker run`
steps. Ports remapped to 15432/16379 to avoid collision with anything
the host may already expose on the standard ports. Containers are named
(`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
`if: always()` step. Postgres readiness is still gated on pg_isready
via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
Replaced with a direct `shellcheck` invocation (pre-installed on the
runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
explicit `platforms: linux/amd64` on both `docker/build-push-action`
invocations. The runner is arm64 but Fly tenant machines pull amd64,
so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
behavior is unchanged.
Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job
* fix(ci): set AGENT_TOOLSDIRECTORY for python-lint on self-hosted runner
setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.
Fixes the only failing job in CI run 24469156329 on PR #186.
---------
Co-authored-by: Hongming Wang <HongmingWang-Rabbit@users.noreply.github.com>
The route wsAuth.DELETE("/secrets/:key", sech.Delete) was already moved
inside the WorkspaceAuth group in a prior commit, closing the CWE-306
unauthenticated-delete vector. This commit adds two regression tests to
lock that in:
- TestWorkspaceAuth_Issue170_SecretDelete_NoBearer_Returns401: workspace
with live tokens, no bearer header → 401 (blocks the attack).
- TestWorkspaceAuth_Issue170_SecretDelete_FailOpen_NoTokens: workspace
with no tokens (bootstrap/legacy) → 200 (fail-open preserved).
Mirrors the TestAdminAuth_Issue120_* and TestWorkspaceAuth_C4_C8_* patterns.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The route wsAuth.DELETE("/secrets/:key", sech.Delete) was already moved
inside the WorkspaceAuth group in a prior commit, closing the CWE-306
unauthenticated-delete vector. This commit adds two regression tests to
lock that in:
- TestWorkspaceAuth_Issue170_SecretDelete_NoBearer_Returns401: workspace
with live tokens, no bearer header → 401 (blocks the attack).
- TestWorkspaceAuth_Issue170_SecretDelete_FailOpen_NoTokens: workspace
with no tokens (bootstrap/legacy) → 200 (fail-open preserved).
Mirrors the TestAdminAuth_Issue120_* and TestWorkspaceAuth_C4_C8_* patterns.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every-10-min cadence introduced in PR #159 increased Security Auditor
from 2 runs/day to 144 runs/day (144x). Combined with PM, Research Lead,
Dev Lead, and other hourly evolution-lever crons, this is the likely
root cause of the P0 OAuth quota exhaustion (#160, resets Apr 17 23:00 UTC).
Restored: cron_expr 7 6,18 * * * (twice daily, 12-hour interval)
Schedule name updated to match new cadence.
Audit prompt content (DAST teardown, PM routing, PM deliverable) retained.
Every-10-min cadence introduced in PR #159 increased Security Auditor
from 2 runs/day to 144 runs/day (144x). Combined with PM, Research Lead,
Dev Lead, and other hourly evolution-lever crons, this is the likely
root cause of the P0 OAuth quota exhaustion (#160, resets Apr 17 23:00 UTC).
Restored: cron_expr 7 6,18 * * * (twice daily, 12-hour interval)
Schedule name updated to match new cadence.
Audit prompt content (DAST teardown, PM routing, PM deliverable) retained.
PR #167 gated /events and /bundles/export/:id behind AdminAuth. The e2e
script's 3 calls to these routes were unauthenticated and broke when the
runner picked them up for the first time on PR #186 (self-hosted runner
migration). Same admin-gate contract, same fix pattern as the #99/#110
e2e hotfixes.
POST /bundles/import is left unauthenticated because by that point in
the script both workspaces have been deleted and #110 revoked their
tokens, so HasAnyLiveTokenGlobal=0 and AdminAuth fails-open.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PR #167 gated /events and /bundles/export/:id behind AdminAuth. The e2e
script's 3 calls to these routes were unauthenticated and broke when the
runner picked them up for the first time on PR #186 (self-hosted runner
migration). Same admin-gate contract, same fix pattern as the #99/#110
e2e hotfixes.
POST /bundles/import is left unauthenticated because by that point in
the script both workspaces have been deleted and #110 revoked their
tokens, so HasAnyLiveTokenGlobal=0 and AdminAuth fails-open.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this call Gin's default trusts all X-Forwarded-For headers, letting
any caller rotate their effective IP and bypass per-IP rate limiting.
SetTrustedProxies(nil) forces c.ClientIP() to always return the real
TCP RemoteAddr.
Adds two regression tests: one documenting the pre-fix bypass, one
asserting the spoofed header is ignored after the fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without this call Gin's default trusts all X-Forwarded-For headers, letting
any caller rotate their effective IP and bypass per-IP rate limiting.
SetTrustedProxies(nil) forces c.ClientIP() to always return the real
TCP RemoteAddr.
Adds two regression tests: one documenting the pre-fix bypass, one
asserting the spoofed header is ignored after the fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GET /approvals/pending was registered on the open router with no
middleware, allowing any unauthenticated caller to enumerate all pending
approvals across every workspace on the platform.
Fix: add inline middleware.AdminAuth(db.DB) to the route registration,
matching the pattern used in PR #167 for bundles, events, and viewport.
The three workspace-scoped approvals routes (POST/GET /approvals,
POST /approvals/:id/decide) were already correctly behind WorkspaceAuth
inside the wsAuth group — no change needed there.
Tests: two new regression tests in wsauth_middleware_test.go —
TestAdminAuth_Issue180_ApprovalsListing_NoBearer_Returns401
TestAdminAuth_Issue180_ApprovalsListing_FailOpen_NoTokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GET /approvals/pending was registered on the open router with no
middleware, allowing any unauthenticated caller to enumerate all pending
approvals across every workspace on the platform.
Fix: add inline middleware.AdminAuth(db.DB) to the route registration,
matching the pattern used in PR #167 for bundles, events, and viewport.
The three workspace-scoped approvals routes (POST/GET /approvals,
POST /approvals/:id/decide) were already correctly behind WorkspaceAuth
inside the wsAuth group — no change needed there.
Tests: two new regression tests in wsauth_middleware_test.go —
TestAdminAuth_Issue180_ApprovalsListing_NoBearer_Returns401
TestAdminAuth_Issue180_ApprovalsListing_FailOpen_NoTokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CRITICAL (#164):
POST /bundles/import — anon callers could create arbitrary workspaces
with user-supplied system prompts, plugins, and secrets envelopes.
Fixed by gating behind AdminAuth (bundleAdmin group).
HIGH (#165):
GET /bundles/export/:id — anon UUID probe leaked full system prompts,
agent_card, plugins, memory for any workspace.
GET /events + GET /events/:workspaceId — anon read of the append-only
event log leaked org topology, workspace names, card fragments.
Both moved into the same bundleAdmin / eventsAdmin groups.
MEDIUM (#166):
PUT /canvas/viewport — anon callers could reset shared viewport state.
Gated via a scoped viewportAdmin group; GET stays open so canvas
bootstraps without a bearer.
GET /admin/liveness — operational-intel leak (scheduler cadence
reveals work pattern). Inline AdminAuth on the single handler.
All 6 routes use the same lazy-bootstrap admin auth the rest of the
platform uses: zero-token installs fail-open, once any token exists
every request must present a valid bearer.
Known follow-up: canvas uses session cookies not bearer tokens (same
pattern as #138). In multi-tenant production these canvas features —
Events tab, Export/Duplicate, viewport persist — will return 401 once
a workspace is token-enrolled. Needs cookie-accepting AdminAuth as a
follow-up (tracked as option B in #138 triage discussion); a new issue
will be filed for that scope. The security gain from closing #164
CRITICAL outweighs the canvas UX regression for tonight.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CRITICAL (#164):
POST /bundles/import — anon callers could create arbitrary workspaces
with user-supplied system prompts, plugins, and secrets envelopes.
Fixed by gating behind AdminAuth (bundleAdmin group).
HIGH (#165):
GET /bundles/export/:id — anon UUID probe leaked full system prompts,
agent_card, plugins, memory for any workspace.
GET /events + GET /events/:workspaceId — anon read of the append-only
event log leaked org topology, workspace names, card fragments.
Both moved into the same bundleAdmin / eventsAdmin groups.
MEDIUM (#166):
PUT /canvas/viewport — anon callers could reset shared viewport state.
Gated via a scoped viewportAdmin group; GET stays open so canvas
bootstraps without a bearer.
GET /admin/liveness — operational-intel leak (scheduler cadence
reveals work pattern). Inline AdminAuth on the single handler.
All 6 routes use the same lazy-bootstrap admin auth the rest of the
platform uses: zero-token installs fail-open, once any token exists
every request must present a valid bearer.
Known follow-up: canvas uses session cookies not bearer tokens (same
pattern as #138). In multi-tenant production these canvas features —
Events tab, Export/Duplicate, viewport persist — will return 401 once
a workspace is token-enrolled. Needs cookie-accepting AdminAuth as a
follow-up (tracked as option B in #138 triage discussion); a new issue
will be filed for that scope. The security gain from closing #164
CRITICAL outweighs the canvas UX regression for tonight.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#138. #125 moved PATCH /workspaces/:id into the wsAdmin AdminAuth
group to close the #120 unauth vulnerability, but broke canvas drag-
reposition and inline rename because canvas uses session cookies not
bearer tokens. Multi-tenant deployments with any live token would have
seen every canvas PATCH 401.
Option A per #138 triage: PATCH goes back on the open router, but
WorkspaceHandler.Update now enforces field-level authz:
Cosmetic (no bearer required):
name, role, x, y, canvas
Sensitive (bearer required when any live token exists):
tier — resource escalation
parent_id — A2A hierarchy manipulation
runtime — container image swap
workspace_dir — host bind-mount redirection
Fail-open bootstrap: HasAnyLiveTokenGlobal = 0 → pass-through
(fresh install, pre-Phase-30 upgrade path). Matches the same
lazy-bootstrap contract WorkspaceAuth and AdminAuth use elsewhere.
3 new tests cover all three branches of the matrix (cosmetic
no-bearer, sensitive no-bearer-rejected, sensitive fail-open).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#138. #125 moved PATCH /workspaces/:id into the wsAdmin AdminAuth
group to close the #120 unauth vulnerability, but broke canvas drag-
reposition and inline rename because canvas uses session cookies not
bearer tokens. Multi-tenant deployments with any live token would have
seen every canvas PATCH 401.
Option A per #138 triage: PATCH goes back on the open router, but
WorkspaceHandler.Update now enforces field-level authz:
Cosmetic (no bearer required):
name, role, x, y, canvas
Sensitive (bearer required when any live token exists):
tier — resource escalation
parent_id — A2A hierarchy manipulation
runtime — container image swap
workspace_dir — host bind-mount redirection
Fail-open bootstrap: HasAnyLiveTokenGlobal = 0 → pass-through
(fresh install, pre-Phase-30 upgrade path). Matches the same
lazy-bootstrap contract WorkspaceAuth and AdminAuth use elsewhere.
3 new tests cover all three branches of the matrix (cosmetic
no-bearer, sensitive no-bearer-rejected, sensitive fail-open).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
#125 added a SELECT EXISTS guard before WorkspaceHandler.Update applies
any UPDATE so nonexistent workspace IDs return 404 instead of silent
zero-row successes. The 4 existing WorkspaceUpdate_* sqlmock tests
didn't mock the probe, so they broke on main. This was not caught
because CI is blocked by the Actions billing cap.
Adds ExpectQuery for the EXISTS probe to:
- TestWorkspaceUpdate_ParentID
- TestWorkspaceUpdate_NameOnly
- TestWorkspaceUpdate_MultipleFields
- TestWorkspaceUpdate_RuntimeField
TestWorkspaceUpdate_BadJSON doesn't need the fix — it aborts on
c.ShouldBindJSON before reaching the guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
#125 added a SELECT EXISTS guard before WorkspaceHandler.Update applies
any UPDATE so nonexistent workspace IDs return 404 instead of silent
zero-row successes. The 4 existing WorkspaceUpdate_* sqlmock tests
didn't mock the probe, so they broke on main. This was not caught
because CI is blocked by the Actions billing cap.
Adds ExpectQuery for the EXISTS probe to:
- TestWorkspaceUpdate_ParentID
- TestWorkspaceUpdate_NameOnly
- TestWorkspaceUpdate_MultipleFields
- TestWorkspaceUpdate_RuntimeField
TestWorkspaceUpdate_BadJSON doesn't need the fix — it aborts on
c.ShouldBindJSON before reaching the guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Supersedes #158 (10-min uniform bump). That PR was too blunt — it treated
research/audit/orchestration crons the same when they have fundamentally
different cost/value/cadence profiles.
## The split
Three layers, three cadences, grounded in the survey of Hermes/Letta/
Trigger.dev/Inngest/AG2/Rivet/n8n/Composio/SWE-agent done this session.
Nobody in that survey runs while(true) per agent — they all combine
event-driven reactivity with short orchestration pulses on a coordinator.
This PR implements that split for our 12-workspace template.
| Layer | Roles | Cadence | Purpose |
|---|---|---|---|
| Orchestration | PM, Dev Lead, Research Lead | every 5 min | Check backlog, dispatch work, review completed tasks |
| Audit | Security Auditor | every 10 min | Focused security audit |
| Audit | UI/UX Designer | every 15 min | Vision-heavy, dial back from 10 |
| Deep-work | Research Lead (eco-watch) | every 30 min (8,38) | Was hourly |
| Deep-work | Dev Lead (template fitness) | every 30 min (15,45) | Was hourly |
| Deep-work | Technical Researcher (plugins) | hourly (unchanged) | Research-heavy, slow |
| Deep-work | DevOps (channels) | hourly (unchanged) | Research-heavy, slow |
| Reactive | BE, FE, DevOps, Docs | no cron | Execute A2A delegations |
## Orchestration pulse prompts
The three new schedules each carry a detailed orchestration_prompt:
- **PM** (5-min): scan all 12 workspaces, scan GH PRs/issues backlog
(external), scan memory backlog (internal), dispatch up to 3 tasks per
pulse, review completed work, write pulse summary to memory. Hard
rules: under 90s wall-clock, never dispatch to busy agents, write
"orchestrator-clean" and stop if genuinely nothing to do.
- **Dev Lead** (5-min, offset +1 from PM): same shape, scoped to
engineering team. Reviews open PRs from direct reports, matches idle
engineers to labeled GH issues (security/bug/feature), dispatches with
"fix/issue-N-slug" branch convention. Skips pulse if own template
fitness audit is in flight (:15, :45).
- **Research Lead** (5-min, offset +2 from PM): same shape, scoped to
research team. Matches Market Analyst / Technical Researcher /
Competitive Intelligence to research-labeled issues or memory-stashed
questions. Max 2 A2A per pulse (research is slow). Skips pulse if own
eco-watch is in flight (:8, :38).
## Cadence offset table
No two crons fire in the same minute:
:01,:11,:21,:31,:41,:51 — Security audit (Security Auditor)
:02,:07,:12,:17,:22,:27,:32,:37,:42,:47,:52,:57 — Dev Lead orchestrator
:04,:09,:14,:19,:24,:29,:34,:39,:44,:49,:54,:59 — Research Lead orchestrator
:01,:06,:11,:16,:21,:26,:31,:36,:41,:46,:51,:56 — PM orchestrator
:05,:20,:35,:50 — UI/UX audit (UIUX Designer)
:08,:38 — Ecosystem watch deep-work (Research Lead)
:15,:45 — Template fitness deep-work (Dev Lead)
:22 — Plugin curation (Technical Researcher)
:47 — Channel expansion (DevOps Engineer)
Note PM and Security Auditor share :01 — this is fine because they
target different workspaces so scheduler concurrency handles it.
## Cost estimate
- PM pulse: 12/hour × 24 × ~3k tokens = 864k tokens/day/org ~ $5/day
- Dev Lead pulse: same ~ $5/day
- Research Lead pulse: same ~ $5/day
- Audits (security 10min, UIUX 15min): ~$8/day/org combined
- Deep-work crons (unchanged from original): ~$4/day/org
**Total ~$27/day/org**. Comparable to #158's $25 but MUCH higher
utility because orchestration produces dispatches that keep workers
busy, whereas #158 just fired more audits against the same team.
Closes#158 (superseded — will close that PR with a pointer to this one).
## Related research
See docs/ecosystem-watch.md `### Hermes Agent` and today's research agent
output: event-driven + reflection-on-completion + short orchestration
pulses on leaders is the shape that delivers 24/7 activity without
runaway cost. This is the concrete implementation.
