Class F of #75 sweep. /commits/{sha}/statuses replaces unavailable workflow-runs API. 4 mapping buckets verified against synthetic+real Gitea data. Approved by security-auditor.
Two artifacts that unblock the parked follow-ups from #59:
1. scripts/edge-429-probe.sh (closes the "operator-blocked" status of
#62). An operator without CF/Vercel dashboard access can reproduce
a canvas-sized burst against a tenant subdomain and read each 429's
response shape — workspace-server bucket overflow (JSON body +
X-RateLimit-* headers) is distinguishable from CF (cf-ray) and
Vercel (x-vercel-id) by inspection of the report. Read-only,
parallel via background subshells (no GNU parallel dependency),
no credential use. Smoke-tested against example.com end-to-end.
2. docs/engineering/ratelimit-observability.md (closes the
"metric-blocked" status of #64). The existing
molecule_http_requests_total{path,status} counter + X-RateLimit-*
response headers already cover #64's acceptance criterion ("watch
metrics for two weeks"). The runbook collects the PromQL queries,
a decision tree for the re-tune (keep / per-tenant override /
change default), an alert rule template, and a hard "do not roll
ad-hoc per-bucket-key exposure" note (in-memory map includes
SHA-256 of bearer tokens — exposing it is a security review
surface, file a follow-up if needed).
Neither artifact changes runtime behaviour. Pure operational tooling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empty commit to trigger CI a second consecutive time per the SOP
'verify ≥1 representative workflow per class via workflow_dispatch
or push event ... ≥2 consecutive successful runs per class'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Part of the post-#66 sweep to remove `gh` CLI dependencies that fail
silently against Gitea. Class F covers `gh run list --workflow=X
--commit=SHA` shapes — querying whether a specific workflow ran (and
how it finished) for a specific SHA.
Why this is the only call site in class F:
`gh run list` hits GitHub's `/repos/.../actions/runs` REST endpoint.
Gitea exposes ZERO endpoints under `/repos/.../actions/runs` —
verified 2026-05-07 via swagger inspection: only secrets, variables,
and runner-registration tokens live under /actions/. There's no way
to query workflow run state via the Gitea v1 API directly.
However, every Gitea Actions job DOES emit a commit status with
`context = "<Workflow Name> / <Job Name> (<event>)"` (verified
2026-05-07 by reading /repos/.../commits/{sha}/statuses on a recent
main SHA). That surface is exactly what we need: each workflow run
leg is one status row, the aggregate state encodes the run outcome,
and Gitea exposes it under `/api/v1/repos/.../commits/{sha}/statuses`
which IS available.
Affected:
`auto-promote-on-e2e.yml` (lines 172-180):
Old: `gh run list --workflow e2e-staging-saas.yml --commit $SHA
--json status,conclusion --jq ...` returning a 5-bucket string
like `completed/success` | `in_progress/none` | `none/none` |
`completed/failure` | `completed/cancelled`.
New: `curl /api/v1/repos/.../commits/$SHA/statuses` + jq filter on
contexts whose name starts with
`"E2E Staging SaaS (full lifecycle) /"`. Mapping:
0 matched contexts → "none/none" (E2E paths-
filtered out
— same as
before)
any context = pending → "in_progress/none" (defer)
any context = error|failure → "completed/failure" (abort)
all contexts = success → "completed/success" (proceed)
The `completed/cancelled` arm of the case statement becomes
unreachable: Gitea status API doesn't expose a `cancelled` state
(it has success/failure/error/pending/warning), so per-SHA
concurrency cancellations now surface as `failure` and are handled
by the failure branch. Documented in-place; the cancelled arm is
kept as defense-in-depth for any future dual-host operation.
Verification:
- Live curl against the current main SHA returns `none/none` (E2E
was paths-filtered for that change set — expected).
- Synthetic-input jq tests verify all four mapping buckets:
no contexts → "none/none"
one context = pending → "in_progress/none"
success + success → "completed/success"
success + failure → "completed/failure"
- YAML syntax validates.
Token: continues to use act_runner's GITHUB_TOKEN (per-run, repo
read scope). The `/commits/{sha}/statuses` endpoint is repo-scoped,
no extra perms needed.
Closes part of #75. Master tracking issue at #75; companion PRs:
#80 (class A — `gh pr ...`), #81 (class D — `gh api ...`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hostile-self-review weakest-spot #2: if the devops-engineer persona
is ever renamed, the canary will go red even if everything else is
fine. Add an inline comment pointing the next editor at both files
that must update together (auto-sync-main-to-staging.yml's git
config + this canary's EXPECTED_PERSONA + the staging branch
protection's push_whitelist_usernames).
No behaviour change — comment-only.
While verifying Phase 4, found a real flaw in Probe 3 (`git ls-remote
refs/heads/staging`). On a public repo (which molecule-core is), Gitea
falls back to anonymous read on bad auth, so `ls-remote` succeeds even
with a junk token. The probe was therefore green-lighting rotated
tokens — false-green, the worst possible canary failure mode.
Rewritten to use `git push --dry-run` of the current staging SHA back
to `refs/heads/staging`:
- Push always authenticates (auth-gated on smart-protocol handshake,
before the dry-run can compute the empty-diff).
- NOP by construction: pushing the current tip back to itself is
"Everything up-to-date" with exit 0.
- Bad token → "Authentication failed", exit 128.
- Doesn't reach pre-receive (where branch-protection authz runs), so
scope is "auth only" — matches the design intent (failure mode B);
authz already covered daily by branch-protection-drift.yml.
Implementation note: `git push` requires a local repo. Spinning up a
fresh `git init` in a tempdir (~1KB, ~50ms) instead of pulling the
full repo via actions/checkout — actions/checkout would clone
~hundreds of MB for what amounts to "a place to run git from."
Local mutation tests pass:
- Real token: "Everything up-to-date" exit 0
- Junk token: "Authentication failed" exit 128 with actionable
::error:: messages pointing at the runbook
Header comment + runbook step-mapping updated to reflect new probe
shape. Refs: #72
Part of the post-#66 sweep to remove `gh` CLI dependencies that fail
silently against Gitea (which exposes /api/v1 only — no GraphQL → 405,
no /api/v3 → 404). Class A covers `gh pr list / view / diff / comment`
shapes.
Affected:
- `.github/workflows/auto-tag-runtime.yml`
Replaced `gh pr list --search SHA --json number,labels` with a curl
to `/api/v1/repos/.../pulls?state=closed&sort=newest&limit=50` +
jq filter on `merge_commit_sha == github.sha`. Same end-to-end
behaviour: locate the merged PR for this push, read its labels,
pick the bump kind. Defensive `?.name // empty` jq guard handles
unlabelled PRs without erroring. The 50-PR window is comfortably
larger than the volume of staging→main promotes that close in any
reasonable detection window.
- `scripts/check-stale-promote-pr.sh`
Rewrote `fetch_prs` and `post_comment` to call Gitea's REST API
directly. Gitea doesn't expose GitHub's compound `mergeStateStatus`
/ `reviewDecision` fields, so the new fetcher pulls
`/pulls?state=open&base=main` then for each PR pulls
`/pulls/{n}/reviews` and synthesizes the GitHub-shape JSON the rest
of the script (and the existing fixture-based unit tests) consume:
BLOCKED + REVIEW_REQUIRED ↔ mergeable=true AND 0 APPROVED reviews
DIRTY ↔ mergeable=false (alarm doesn't fire)
CLEAN + APPROVED ↔ mergeable=true AND ≥1 APPROVED review
Comment-posting moves to `POST /repos/.../issues/{n}/comments`
(Gitea treats PRs as issues for the comment surface, same as
GitHub's REST). All 23 fixture-driven unit tests still pass —
fixtures pass GitHub-shape JSON via PR_FIXTURE which short-circuits
the live fetch path.
- `scripts/ops/check_migration_collisions.py`
Replaced `gh pr list` + `gh pr diff` calls with stdlib `urllib`
against /api/v1. Helper `_gitea_get` centralizes auth + error
handling; uses GITEA_TOKEN env, falling back to GITHUB_TOKEN
(act_runner) and GH_TOKEN. Return shape from
`open_prs_with_migration_prefix` mimics the historical
`--json number,headRefName` so the call sites are unchanged. All 9
regex-classifier unit tests still pass; live integration test
against the production Gitea API returns 0 collisions for prefix=999
as expected.
curl invocation pattern is `curl --fail-with-body -sS` (NOT `-fsS` —
the two short-fail flags are mutually exclusive in modern curl;
caught by `curl: You must select either --fail or --fail-with-body,
not both` during local verification).
Token model: workflows pass act_runner's GITHUB_TOKEN (per-run, repo
read scope) — same surface used by the auto-sync fix in PR #66 plus
the surrounding workflows. No new repo secrets required.
Verification: bash unit tests (23/23 pass), python unittest (9/9 pass),
live curl call against production Gitea returns 200 with the expected
shape, YAML / shell / Python syntax all validate.
Closes part of #75. Other classes (D — `gh api`; F — `gh run list`)
land in follow-up PRs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two API probes used the unsafe shape rejected by
lint-curl-status-capture.yml (per feedback_curl_status_capture_pollution):
status=$(curl ... -w '%{http_code}' ... || echo "000")
When curl exits non-zero (transport error, --fail-with-body 4xx/5xx),
the `-w` already wrote a code; the `|| echo "000"` then APPENDS another
"000", yielding "000000" or "409000" — passes shape checks while looking
right.
Switch to the canonical safe shape (set +e + tempfile + cat):
set +e
curl ... -w '%{http_code}' >code_file 2>/dev/null
set -e
status=$(cat code_file 2>/dev/null || true)
[ -z "$status" ] && status="000"
Inline comment in both probe steps explains the lint constraint so
the next editor doesn't re-introduce the bad pattern.
Refs: #72, lint failure on PR #77 (1/22 red → 22/22 expected green)
Root cause: same as #65/PR-#66 — gh CLI calls Gitea GraphQL
(/api/graphql) which returns HTTP 405. Additionally, gh workflow
run calls /actions/workflows/{id}/dispatches which does not
exist on Gitea 1.22.6 (verified via swagger.v1.json).
Fix:
- Replace gh run list with Gitea REST combined-status endpoint
(GET /repos/{owner}/{repo}/commits/{ref}/status). Combined state
encodes the AND across every check context — simpler than the
per-workflow loop and immune to workflow-name collisions.
- Replace gh pr create / merge --auto with direct curl calls to
POST /pulls and POST /pulls/{N}/merge with merge_when_checks_succeed.
- Remove the post-merge polling tail entirely. The GitHub-era
GITHUB_TOKEN no-recursion rule does not apply on Gitea Actions
(verified empirically: PR #66 merge fired downstream pushes
naturally). Even if we wanted to dispatch, Gitea has no
workflow_dispatch REST endpoint.
Critical constraint: main has enable_push: false with no whitelist;
direct push is impossible for any persona. PR-mediated merge is the
only path. main has required_approvals: 1 — auto-merge waits for
Hongming's approval before landing, preserving the
feedback_prod_apply_needs_hongming_chat_go contract.
Identity: AUTO_SYNC_TOKEN (devops-engineer persona). Not founder PAT.
Per feedback_per_agent_gitea_identity_default. Same persona used by
auto-sync (PR #66) — keeps identity model coherent.
Header comment block fully rewritten with 4 failure-mode runbooks
(A: gates not green, B: PR-create non-201, C: merge schedule fails,
D: token rotated/scope wrong) per PR #66's pattern.
Refs: #65, #73, #195, PR #66 (canonical reference)
Closes#73
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 6h-cron synthetic check that fires the auth surface used by
auto-sync-main-to-staging.yml (PR #66) and emits a red workflow
status when AUTO_SYNC_TOKEN has drifted out of validity. Closes
hostile-self-review weakest-spot #3 from PR #66 (token-rotation
detection latency).
Read-only verification — no writes, no synthetic merge commits, no
canary branch noise. Three probes:
1. GET /api/v1/user → token authenticates as devops-engineer
2. GET /api/v1/repos/molecule-ai/molecule-core → read:repository scope
3. git ls-remote refs/heads/staging → exact HTTPS auth path used by
actions/checkout in the real auto-sync workflow
Hard-fail on missing AUTO_SYNC_TOKEN secret on both schedule and
workflow_dispatch — per feedback_schedule_vs_dispatch_secrets_hardening,
a silent soft-skip would make the canary itself drift-invisible (the
sweep-cf-orphans #2088 lesson). Operator runbook in workflow header.
Token reuse: same AUTO_SYNC_TOKEN as the workflow under monitor; no
new credential introduced. Read-only paths only.
Refs: #72, hostile-self-review #66
Stage 3 of #61 (final stage). Replaces the 5s setInterval poll with:
1. Initial bootstrap on mount + on filter-change + on workspaceId-
change (preserved from existing useEffect on loadActivities).
2. Manual Refresh button (preserved — still triggers loadActivities).
3. useSocketEvent subscription to ACTIVITY_LOGGED — every event
for THIS workspace prepends to the list, gated on the user's
autoRefresh toggle and current filter selection.
No interval poll. Steady-state HTTP traffic from this tab drops from
12 req/min (5s × 1 active workspace) to 0 outside of bootstraps and
manual refreshes. Live update latency drops from up to 5s to ~10ms.
The autoRefresh ("Live" / "Paused") toggle now gates LIVE updates
instead of polling cadence — semantically the same (paused = list
stays frozen), implementationally simpler.
The filter selection is honoured by the WS handler so a user
filtering to "Tasks" doesn't see live a2a_send rows trickle in. Same
shape the server-side `?type=<filter>` enforces on the bootstrap.
Test changes:
- 27 existing tests pass unchanged (filter / autoRefresh /
Refresh / loading / error / empty / count / row-content all
preserved)
- 7 new WS-subscription tests:
- WS push for matching workspace prepends with NO HTTP call
- WS push for different workspace ignored
- WS push respects active filter (non-matching ignored)
- WS push respects active filter (matching renders)
- WS push while autoRefresh paused ignored
- WS push for already-in-list row deduped (no double-render)
- NO 5s interval polling after mount
Mutation-tested:
- drop workspace_id filter → "different workspace" test fails
- drop autoRefresh gate → "paused" test fails
- drop filter gate → "non-matching activity_type" test fails
- drop dedup-by-id → "already in list deduped" test fails
Full canvas suite: 1396 passing, 0 failing. tsc clean.
No API or schema change. /workspaces/:id/activity HTTP endpoint
stays — used for bootstrap + manual refresh + filter-change reload.
ACTIVITY_LOGGED event shape unchanged.
Hostile self-review (three weakest spots):
1. Server-side activity_logs row UPDATES (status flips, etc.) are
not reflected post-#61 — the dedup-by-id check skips a re-fired
ACTIVITY_LOGGED for an existing row. Acceptable: activity_logs
is append-only by design (audit trail); status updates surface
as new task_update rows, not as in-place mutations. If a future
server change adds in-place updates, fire ACTIVITY_UPDATED as a
distinct event so this dedup logic stays simple.
2. WS handler is recreated on every render (filter / autoRefresh /
workspaceId state changes). useSocketEvent's ref-based pattern
keeps the bus subscription stable, but the handler closure
re-captures each render. Side effect: fine — handler call cost
is negligible.
3. The "error" filter matches activity_type === "error" (mirrors
server semantics). It does NOT match status === "error" rows
of other activity types — same as the polling version. Worth
re-evaluating in a separate PR if users expect the broader
semantic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage 2 of #61. Replaces the 60s setInterval poll that fanned out
across every visible workspace fetching `?type=delegation&limit=500`
with:
1. One bootstrap fan-out on mount (or on visible-ID-set change),
same shape as before — preserves the 60-min look-back history.
2. useSocketEvent subscription to ACTIVITY_LOGGED — every event
with activity_type=delegation + method=delegate from a visible
workspace appends to a local rolling buffer, edges are re-derived
via the existing buildA2AEdges helper.
3. showA2AEdges toggle off: clears edges + buffer.
No interval poll. The visibleIdsKey selector gate that fixed the
2026-05-04 render-loop incident is preserved — peer-discovery /
status-flip writes still don't trigger a wasteful re-bootstrap.
Steady-state HTTP traffic from this overlay drops from N req/min
(N visible workspaces × 1 cycle/min) to 0 outside of mount + visible-
ID-set-change bootstraps. Live update latency drops from up to 60s
to ~10ms.
Bootstrap race-aware: any WS arrivals that landed in the buffer
during the fetch await are preserved by id-dedup-with-fetched-first
ordering. No row is double-counted; no row is lost during in-flight
updates.
Test changes:
- 27 existing tests pass unchanged (buildA2AEdges purity preserved,
component visibility/visibleIdsKey/error-swallow behaviour
preserved).
- 6 new WS-subscription tests:
- NO 60s polling after bootstrap (clock advance fires nothing)
- WS push for delegation updates edges with NO HTTP call
- WS push for non-delegation activity_type ignored
- WS push for delegate_result ignored (mirrors buildA2AEdges
method filter)
- WS push from hidden workspace ignored
- WS push while showA2AEdges=false ignored
Mutation-tested:
- drop activity_type filter → "non-delegation" test fails
- drop method===delegate filter → "delegate_result" test fails
- drop visible-ws membership filter → "hidden workspace" test fails
Full canvas suite: 1395 passing, 0 failing. tsc clean.
No API or schema change. ACTIVITY_LOGGED event shape unchanged.
The /workspaces/:id/activity HTTP endpoint stays — used for bootstrap.
Hostile self-review (three weakest spots):
1. Bootstrap fetches up to 500 rows × N workspaces. Worst-case
buffer ~3000 entries before window-prune. Acceptable: window-
prune runs on every recomputeAndPush, buildA2AEdges aggregates
to at most N² edges. Real-world usage stays well under both.
2. WS handler re-arms on every bootstrap dependency change
(visibleIds change). useSocketEvent's ref-based pattern means
the bus subscription stays stable across renders, but the
handler closure re-captures bootstrap each time. Side effect:
fine — handler invocation just calls recomputeAndPush which is
idempotent.
3. delegate_result rows arriving over WS are silently dropped.
Acceptable: the existing buildA2AEdges already filters them out
at aggregation time (avoids double-counting); pre-filtering at
the WS handler is the correct mirror — keeps the bus path and
the bootstrap path consistent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OSS contributors who clone molecule-core and `go run ./workspace-server/cmd/server`
now get a working end-to-end provision without authenticating to GHCR or AWS ECR.
Pre-fix: with MOLECULE_IMAGE_REGISTRY unset, the provisioner attempted to pull
ghcr.io/molecule-ai/workspace-template-<runtime>:latest, which has been
returning 403 since the 2026-05-06 GitHub-org suspension.
Post-fix: when MOLECULE_IMAGE_REGISTRY is unset, the provisioner switches to
local-build mode — looks up the workspace-template-<runtime> repo's HEAD sha
on Gitea via a single API call, shallow-clones into ~/.cache/molecule/, and
runs `docker build --platform=linux/amd64`. SHA-pinned cache key skips the
clone+build entirely on subsequent provisions.
Production tenants are unaffected: every prod tenant sets the var to its
private ECR mirror, so the SaaS pull path is byte-for-byte identical.
SSOT for mode detection lives in Resolve() (registry_mode.go) returning a
discriminated RegistrySource{Mode, Prefix} so call sites that branch on
mode get a compile-time push instead of a string-equality footgun.
Coverage:
* registry_mode.go — new SSOT (Resolve, RegistryMode, IsKnownRuntime)
* registry_mode_test.go — 8 tests pinning mode-decision contract
* localbuild.go — clone+build pipeline (570 LOC, fully unit-tested)
* localbuild_test.go — 22 tests covering happy/sad paths, fail-closed
* provisioner.go — Start() inserts ensureLocalImageHook in local mode
* docs/adr/ADR-002 — design rationale + alternatives + security review
* docs/development/local-development.md — local-build flow + env overrides
Security:
* Allowlist-only runtime names (knownRuntimes) gate the clone path.
* Repo prefix hardcoded to git.moleculesai.app/molecule-ai/molecule-ai-workspace-template-;
forks via opt-in MOLECULE_LOCAL_TEMPLATE_REPO_PREFIX.
* MOLECULE_GITEA_TOKEN masked in every log line via maskTokenInURL/maskTokenInString.
* Fail-closed: Gitea unreachable / runtime not mirrored → clear error, never
silently fall back to GHCR/ECR.
* docker build invocation passes no --build-arg from external input.
* HTTP body cap 64KB on Gitea API responses (defence vs malicious upstream).
Closes#63 / Task #194.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
