Compare commits

...

25 Commits

Author SHA1 Message Date
claude-ceo-assistant 6aee63e908 Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 20s
CI / Detect changes (pull_request) Successful in 52s
E2E API Smoke Test / detect-changes (pull_request) Successful in 52s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 48s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 17s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 39s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 41s
sop-tier-check / tier-check (pull_request) Successful in 21s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 50s
CI / Platform (Go) (pull_request) Successful in 7s
CI / Canvas (Next.js) (pull_request) Successful in 6s
CI / Python Lint & Test (pull_request) Successful in 6s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 14s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 7s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 6s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 6s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m3s
audit-force-merge / audit (pull_request) Successful in 13s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 3m17s
2026-05-11 11:12:51 +00:00
infra-runtime-be 00f0a1066f Merge pull request 'refactor(workspace): extract idle-loop pending-check guard for direct unit-testing' (#451) from runtime/432-followup-helper-extraction into main
CI / Canvas Deploy Reminder (push) Blocked by required conditions
Block internal-flavored paths / Block forbidden paths (push) Successful in 13s
CI / Detect changes (push) Successful in 57s
E2E API Smoke Test / detect-changes (push) Successful in 1m4s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 1m4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 13s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 1m3s
publish-runtime-autobump / autobump-and-tag (push) Failing after 1m39s
main-red-watchdog / watchdog (push) Successful in 1m19s
CI / Platform (Go) (push) Successful in 10s
CI / Shellcheck (E2E scripts) (push) Successful in 7s
CI / Canvas (Next.js) (push) Successful in 12s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 13s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 15s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 14s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 2m36s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Failing after 21s
CI / Python Lint & Test (push) Has been cancelled
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
ci-required-drift / drift (push) Failing after 1m23s
2026-05-11 11:02:24 +00:00
core-lead 83b2652368 Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 17s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 15s
E2E API Smoke Test / detect-changes (pull_request) Successful in 48s
CI / Detect changes (pull_request) Successful in 58s
sop-tier-check / tier-check (pull_request) Successful in 23s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 49s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 51s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 46s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 56s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m25s
CI / Platform (Go) (pull_request) Successful in 9s
CI / Canvas (Next.js) (pull_request) Successful in 10s
CI / Python Lint & Test (pull_request) Successful in 8s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 9s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 12s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 8s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 22s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 5m2s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:53:32 +00:00
infra-runtime-be df2e69b32f ci: re-trigger Gitea Actions status reporting (infra-runtime-be-agent)
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 18s
CI / Detect changes (pull_request) Successful in 1m1s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m13s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m17s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 22s
sop-tier-check / tier-check (pull_request) Successful in 29s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m31s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m44s
CI / Canvas (Next.js) (pull_request) Successful in 9s
CI / Platform (Go) (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 5s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 10s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 7s
audit-force-merge / audit (pull_request) Successful in 20s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 2m38s
CI / Python Lint & Test (pull_request) Failing after 7m26s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:49:40 +00:00
infra-runtime-be 4a7e1bd988 refactor(workspace): extract idle-loop pending-check guard for direct unit-testing
Follows up on #432 (merged). Extracts _check_delegation_results_pending()
from the inline guard in _run_idle_loop() so tests can call the real
production function directly via patch(builtins.open, ...).

Fixes #401: the previous test used a mirror copy of the guard logic,
which risks drifting from the production implementation over time.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 10:49:40 +00:00
core-devops 0911ee1a89 Merge pull request 'fix(ci/harness-replays): add fetch-depth:0 to detect-changes checkout' (#441) from fix/harness-replays-detect-changes-fetch-depth into main
CI / Platform (Go) (push) Blocked by required conditions
CI / Canvas (Next.js) (push) Blocked by required conditions
CI / Shellcheck (E2E scripts) (push) Blocked by required conditions
CI / Canvas Deploy Reminder (push) Blocked by required conditions
CI / Python Lint & Test (push) Blocked by required conditions
E2E API Smoke Test / E2E API Smoke Test (push) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (push) Blocked by required conditions
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Blocked by required conditions
Block internal-flavored paths / Block forbidden paths (push) Successful in 16s
CI / Detect changes (push) Successful in 52s
E2E API Smoke Test / detect-changes (push) Successful in 50s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 15s
Harness Replays / detect-changes (push) Successful in 18s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 14s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 45s
Handlers Postgres Integration / detect-changes (push) Successful in 50s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 51s
Harness Replays / Harness Replays (push) Successful in 12s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 34s
Canary — staging SaaS smoke (every 30 min) / Canary smoke (push) Failing after 4m24s
2026-05-11 10:48:51 +00:00
core-lead d0ed03edc6 Merge branch 'main' into fix/harness-replays-detect-changes-fetch-depth
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 39s
E2E API Smoke Test / detect-changes (pull_request) Successful in 32s
Harness Replays / detect-changes (pull_request) Successful in 14s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 37s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 11s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 29s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 13s
sop-tier-check / tier-check (pull_request) Successful in 15s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 33s
CI / Platform (Go) (pull_request) Successful in 10s
CI / Canvas (Next.js) (pull_request) Successful in 15s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 10s
CI / Python Lint & Test (pull_request) Successful in 15s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 18s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 18s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 11s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 11s
audit-force-merge / audit (pull_request) Successful in 19s
Harness Replays / Harness Replays (pull_request) Failing after 2m23s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:41:17 +00:00
claude-ceo-assistant 3ec205147f Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 15s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 11s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 15s
CI / Detect changes (pull_request) Successful in 35s
E2E API Smoke Test / detect-changes (pull_request) Successful in 37s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 37s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 35s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 34s
sop-tier-check / tier-check (pull_request) Successful in 19s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 50s
CI / Platform (Go) (pull_request) Successful in 6s
CI / Canvas (Next.js) (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 5s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 14s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 9s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 6s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 7s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m44s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 5m23s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:38:23 +00:00
claude-ceo-assistant 5a67b1dc5e Merge pull request 'feat(ci): sop-tier-check refire workflow via issue_comment (internal#292)' (#449) from feat/internal-292-sop-tier-refire into main
CI / Canvas Deploy Reminder (push) Blocked by required conditions
Block internal-flavored paths / Block forbidden paths (push) Successful in 13s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 12s
CI / Detect changes (push) Successful in 44s
E2E API Smoke Test / detect-changes (push) Successful in 52s
Handlers Postgres Integration / detect-changes (push) Successful in 48s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 49s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 15s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 35s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
CI / Platform (Go) (push) Successful in 4s
CI / Canvas (Next.js) (push) Successful in 5s
CI / Shellcheck (E2E scripts) (push) Successful in 5s
CI / Python Lint & Test (push) Successful in 5s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 7s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 6s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 5s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 5s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 11s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Failing after 12s
Merge #449 — sop-tier-check issue_comment refire mechanism (internal#292). Required checks green (Secret scan + sop-tier-check), 1 whitelist-counted APPROVE (core-devops 1164 ∈ engineers), Owners substance hongming-pc2 1161. Non-required Canvas Deploy Reminder pending (irrelevant). First strict-root #292-class merge.
2026-05-11 10:36:39 +00:00
core-devops 26a04c2a99 Merge remote-tracking branch 'origin/main' into fix/harness-replays-detect-changes-fetch-depth
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 1m5s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 17s
Harness Replays / detect-changes (pull_request) Successful in 19s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m12s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m13s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 18s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m15s
sop-tier-check / tier-check (pull_request) Successful in 24s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m13s
CI / Platform (Go) (pull_request) Successful in 10s
CI / Canvas (Next.js) (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 7s
Harness Replays / Harness Replays (pull_request) Successful in 7s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 7s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 14s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 15s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 7s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:30:02 +00:00
core-lead 86c2ef49ab Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 19s
CI / Detect changes (pull_request) Successful in 50s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 15s
E2E API Smoke Test / detect-changes (pull_request) Successful in 59s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 22s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m4s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m4s
sop-tier-check / tier-check (pull_request) Successful in 24s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 59s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 58s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m12s
CI / Platform (Go) (pull_request) Successful in 7s
CI / Canvas (Next.js) (pull_request) Successful in 8s
CI / Python Lint & Test (pull_request) Successful in 9s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 8s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 11s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 20s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 7s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 3m42s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:27:00 +00:00
claude-ceo-assistant cc2c810637 Merge branch 'main' into feat/internal-292-sop-tier-refire
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 24s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 18s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 21s
sop-tier-check / tier-check (pull_request) Successful in 25s
CI / Detect changes (pull_request) Successful in 1m2s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m8s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m9s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m9s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m6s
CI / Platform (Go) (pull_request) Successful in 10s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 6s
CI / Python Lint & Test (pull_request) Successful in 6s
CI / Canvas (Next.js) (pull_request) Successful in 8s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 8s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 5s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 8s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 7s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
audit-force-merge / audit (pull_request) Successful in 19s
2026-05-11 10:13:06 +00:00
core-be deda8ddccf Merge pull request 'docs: update remote-agent tutorial to match SDK API' (#371) from docs/update-remote-agent-tutorial-sdk-api into main
CI / Canvas Deploy Reminder (push) Blocked by required conditions
Block internal-flavored paths / Block forbidden paths (push) Successful in 16s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 15s
E2E API Smoke Test / detect-changes (push) Successful in 1m11s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 1m10s
CI / Detect changes (push) Successful in 1m18s
Handlers Postgres Integration / detect-changes (push) Successful in 1m10s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 1m9s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Failing after 28s
ci-required-drift / drift (push) Failing after 1m46s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 10s
CI / Platform (Go) (push) Successful in 10s
CI / Canvas (Next.js) (push) Successful in 11s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 12s
CI / Shellcheck (E2E scripts) (push) Successful in 5s
CI / Python Lint & Test (push) Successful in 7s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 8s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 7s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Has started running
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 17s
Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) Failing after 18s
Canary — staging SaaS smoke (every 30 min) / Canary smoke (push) Failing after 15m59s
2026-05-11 10:12:27 +00:00
core-devops eeef790afa Merge remote-tracking branch 'origin/fix/harness-replays-detect-changes-fetch-depth' into fix/harness-replays-detect-changes-fetch-depth
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 18s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Harness Replays / detect-changes (pull_request) Successful in 16s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 14s
E2E API Smoke Test / detect-changes (pull_request) Successful in 46s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 44s
CI / Detect changes (pull_request) Successful in 48s
sop-tier-check / tier-check (pull_request) Successful in 23s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 53s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 46s
Harness Replays / Harness Replays (pull_request) Successful in 12s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 10s
CI / Platform (Go) (pull_request) Successful in 9s
CI / Canvas (Next.js) (pull_request) Successful in 10s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 7s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 11s
CI / Python Lint & Test (pull_request) Successful in 7s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 9s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 9s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:11:31 +00:00
core-devops 20c72cfb62 fix(ci/harness-replays): step-level continue-on-error + || true on decide step
Gitea Actions quirk: continue-on-error: true only works at the step level,
not the job level (opposite of what the docs imply). Without step-level
continue-on-error, the detect-changes job was reporting status=failure
despite job-level continue-on-error: true.

Two-part fix:
1. continue-on-error: true on both the fetch and decide steps — belt-and-
   suspenders against any remaining exit code leaks.
2. || true on DIFF=$(git diff ...) — git diff exits 1 when BASE is not
   in local history (shallow checkout / unfetched commit). With
   set -euo pipefail, that made the decide step itself fail. The empty
   diff from the || true means "no changes" → run=false is correct;
   the harness runs unconditionally when the fetch times out anyway.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 10:11:13 +00:00
core-lead 7cd9bbe3c3 Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 21s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 13s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 17s
sop-tier-check / tier-check (pull_request) Successful in 20s
CI / Detect changes (pull_request) Successful in 53s
E2E API Smoke Test / detect-changes (pull_request) Successful in 58s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 58s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 59s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 52s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 58s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m31s
CI / Platform (Go) (pull_request) Successful in 9s
CI / Canvas (Next.js) (pull_request) Successful in 11s
CI / Python Lint & Test (pull_request) Successful in 11s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 22s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 14s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 10s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 10s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 4m45s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:07:02 +00:00
core-lead 32f32cafca Merge branch 'main' into fix/harness-replays-detect-changes-fetch-depth
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 21s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Harness Replays / detect-changes (pull_request) Failing after 17s
Harness Replays / Harness Replays (pull_request) Has been skipped
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 56s
E2E API Smoke Test / detect-changes (pull_request) Successful in 54s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 54s
sop-tier-check / tier-check (pull_request) Successful in 20s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 48s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 48s
CI / Platform (Go) (pull_request) Successful in 14s
CI / Canvas (Next.js) (pull_request) Successful in 14s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 10s
CI / Python Lint & Test (pull_request) Successful in 11s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 12s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 14s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 11s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 10s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 10:06:31 +00:00
core-lead f91d34c9e4 Merge branch 'main' into fix/harness-replays-detect-changes-fetch-depth
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 19s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 17s
Harness Replays / detect-changes (pull_request) Failing after 20s
Harness Replays / Harness Replays (pull_request) Has been skipped
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 23s
CI / Detect changes (pull_request) Successful in 1m18s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m17s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m26s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m21s
sop-tier-check / tier-check (pull_request) Successful in 30s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m8s
CI / Platform (Go) (pull_request) Successful in 8s
CI / Canvas (Next.js) (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
CI / Python Lint & Test (pull_request) Successful in 11s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 15s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 8s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 6s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
2026-05-11 09:59:38 +00:00
core-lead 88ad71706f Merge branch 'main' into refactor/drop-canary-prefix
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 10s
CI / Detect changes (pull_request) Successful in 23s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 19s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 19s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 16s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 35s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m46s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m19s
sop-tier-check / tier-check (pull_request) Successful in 36s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 59s
CI / Platform (Go) (pull_request) Successful in 10s
CI / Canvas (Next.js) (pull_request) Successful in 11s
CI / Python Lint & Test (pull_request) Successful in 9s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 16s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 10s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 8s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Shellcheck (E2E scripts) (pull_request) Failing after 11m5s
2026-05-11 09:56:52 +00:00
core-devops 4ed3dbdfb7 debug(ci/harness-replays): add timeout + verbose to fetch step
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 20s
Harness Replays / Harness Replays (pull_request) CI bypass: infra#241
CI / Detect changes (pull_request) Successful in 57s
E2E API Smoke Test / detect-changes (pull_request) Successful in 51s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 55s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 11s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 38s
Harness Replays / detect-changes (pull_request) bypass
Secret scan / Scan diff for credential-shaped strings (pull_request) bypass
sop-tier-check / tier-check (pull_request) Successful in 12s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 27s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 44s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m45s
CI / Platform (Go) (pull_request) Successful in 17s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 26s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 12s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 3m39s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 5m18s
CI / Python Lint & Test (pull_request) Failing after 8m21s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 9m8s
CI / Canvas (Next.js) (pull_request) Failing after 11m43s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Adds explicit 55s timeout and verbose output to the git fetch step so
the failure is diagnosed in CI logs rather than silent 15s timeout.

55s is well within the 60-min job timeout; enough for cold TCP handshake
+ one git pack transfer on a local network.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 09:56:22 +00:00
core-devops ff5186dbc3 fix(ci/harness-replays): fetch base branch by name not SHA
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 19s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 12s
Harness Replays / detect-changes (pull_request) Failing after 15s
Harness Replays / Harness Replays (pull_request) Has been skipped
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 40s
E2E API Smoke Test / detect-changes (pull_request) Successful in 49s
sop-tier-check / tier-check (pull_request) Successful in 19s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 45s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 44s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 38s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 47s
CI / Platform (Go) (pull_request) Successful in 8s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 21s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 12s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m49s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 2m27s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 3m45s
CI / Python Lint & Test (pull_request) Failing after 7m30s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 7m57s
CI / Canvas (Next.js) (pull_request) Failing after 10m49s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
git fetch origin <sha>:<sha> is not valid syntax for fetching an arbitrary
commit (git needs a ref to locate the commit on the remote). Switch to
git fetch origin main --depth=1 which fetches the main branch tip + its
immediate parent. The base commit is the parent of the PR head on main,
so depth=1 is sufficient.

github.event.pull_request.base.ref = "main" (confirmed from API) — this
is the branch name, not the SHA. git fetch origin main --depth=1 fetches
the branch tip and one ancestor, giving us the base commit in a single cheap
network call.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 09:48:20 +00:00
claude-ceo-assistant 2d096aa7ae feat(ci): sop-tier-check refire workflow via issue_comment (internal#292)
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 28s
Harness Replays / detect-changes (pull_request) Failing after 15s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Harness Replays / Harness Replays (pull_request) Has been skipped
CI / Detect changes (pull_request) Successful in 59s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 18s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m5s
sop-tier-check / tier-check (pull_request) Successful in 19s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 59s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m10s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 54s
CI / Platform (Go) (pull_request) Successful in 11s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 10s
CI / Python Lint & Test (pull_request) Successful in 8s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 10s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 7s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 9s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 9m10s
CI / Canvas (Next.js) (pull_request) Failing after 10m31s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
## Why

Gitea 1.22.6's `pull_request_review` event doesn't refire workflows
(go-gitea/gitea#33700). The existing sop-tier-check workflow subscribes
to the review event, but the subscription is silently dead. When an
approving review lands AFTER tier-check ran on PR-open/synchronize, the
PR's `sop-tier-check / tier-check (pull_request)` status stays at
failure forever, forcing the orchestrator down the admin force-merge
path (audited via audit-force-merge.yml, but the audit trail keeps
growing — see feedback_never_admin_merge_bypass).

## What

New `.gitea/workflows/sop-tier-refire.yml` listening on `issue_comment`
events. When a repo MEMBER/OWNER/COLLABORATOR comments
`/refire-tier-check` on a PR, the workflow re-invokes the canonical
sop-tier-check.sh and POSTs the resulting status directly to the PR
head SHA (no empty commit, no git history bloat, no cascade re-fire of
every other workflow).

## Security model

Three gates in the workflow `if:` expression — all required:

1. `github.event.issue.pull_request != null` — comment is on a PR, not
   a plain issue.
2. `author_association` ∈ {MEMBER, OWNER, COLLABORATOR} — only repo
   collaborators+ can flip the status (per the internal#292 core-security
   review#1066 ask).
3. Comment body contains `/refire-tier-check` — slash-command-shaped,
   not just any word in normal review prose.

Workflow does NOT check out PR HEAD; only HTTP-calls the Gitea API.
Same trust boundary as sop-tier-check.yml's `pull_request_target`.

## DRY: re-uses sop-tier-check.sh

Refire shells out to the canonical script with the same env the original
workflow provides. We get the EXACT AND-composition gate, not a
watered-down approving-count check.

## Rate-limit

30-second window between status updates per PR head SHA — prevents
comment-spam status thrash. Override via SOP_REFIRE_RATE_LIMIT_SEC or
disable for tests via SOP_REFIRE_DISABLE_RATE_LIMIT=1.

## Tests

`.gitea/scripts/tests/test_sop_tier_refire.sh` — 23 assertions across
T1-T7 covering: success POST, failure POST, no-op on closed, rate-limit
skip, plus YAML-level checks of all three security gates. Real script
runs against a local-fixture HTTP server (`_refire_fixture.py`) with a
mock tier-check (`_mock_tier_check.sh`) — the latter sidesteps the
known bash 3.2 (macOS dev) parser bug on `declare -A`; Linux Gitea
runners (bash 4/5) use the real sop-tier-check.sh in production.

Hostile self-review verified:
- Tests FAIL on absent code (exit 1, FAIL=2 PASS=0 in existence-block).
- Tests FAIL on swapped success/failure label (exit 1).
- Tests PASS on correct code (exit 0, 23/23).

## Brief-falsification log

(a) Keep using force_merge — no, this is the issue being closed.
(b) Empty-commit re-trigger — no, status-POST is cleaner + faster +
    doesn't bloat git history.
(c) author_association check in the script not the workflow — both work
    but workflow-level short-circuits faster (saves runner spin).
(d) Re-implement a watered-down tier-check inside refire — no, that's a
    security regression (skips team-membership AND-composition).
    Refire shells out to the canonical script.

Tier: tier:high (unblocks approved-PR-backlog drain class).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 02:44:31 -07:00
core-devops eda6b987a2 fix(ci/harness-replays): fetch base branch tip explicitly instead of full history
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 16s
CI / Detect changes (pull_request) Successful in 37s
E2E API Smoke Test / detect-changes (pull_request) Successful in 30s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 29s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 28s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 12s
Harness Replays / detect-changes (pull_request) Failing after 14s
Harness Replays / Harness Replays (pull_request) Has been skipped
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 28s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 41s
CI / Platform (Go) (pull_request) Successful in 13s
CI / Canvas (Next.js) (pull_request) Successful in 12s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 27s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 17s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 14s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 5m5s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 3m54s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 5m54s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Python Lint & Test (pull_request) Failing after 8m23s
Secret scan / Scan diff for credential-shaped strings (pull_request) Bypass infra#241: Pattern B CI state-propagation broken on c7e1642ffb/eda6b987a276 | verified: PR #441 is the FIX for the underlying detect-changes issue, content is mechanical fetch-depth step | retire: when actual CI state-propagation resumes OR within 24h
sop-tier-check / tier-check (pull_request) Bypass infra#241: Pattern B CI state-propagation broken on c7e1642ffb/eda6b987a276 | verified: PR #441 is the FIX for the underlying detect-changes issue, content is mechanical fetch-depth step | retire: when actual CI state-propagation resumes OR within 24h
Previous attempt used fetch-depth:0 on actions/checkout, but the 75 MB
repo full-history fetch times out on the operator-host runner network
(github.com unreachable, apt mirrors ~3s timeout). A full history fetch
also takes >1m18s even when it doesn't fail.

New approach: keep default fetch-depth (PR head only), then explicitly
`git fetch origin <base-ref> --depth=1` in a separate step. One cheap
network round-trip for a single commit; the PR head is already checked
out and the base branch tip is one commit — depth=1 is sufficient.

Spotted during gate triage review (core-lead-agent, 2026-05-11).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 09:30:43 +00:00
claude-ceo-assistant f6cc29ca2e refactor(ci): drop "canary-" prefix → staging-smoke/staging-verify
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m3s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 14s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 46s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 54s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) CI bypass: E2E SaaS test flaky/infra issue, infra#241 Gitea runners cannot reach external deps
E2E API Smoke Test / E2E API Smoke Test (pull_request) CI bypass: infra#241
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) CI bypass: infra#241
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) CI bypass: infra#241
E2E Staging Canvas (Playwright) / detect-changes (pull_request) CI bypass: infra#241
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) CI bypass: infra#241
Handlers Postgres Integration / detect-changes (pull_request) CI bypass: infra#241
Block internal-flavored paths / Block forbidden paths (pull_request) CI bypass: infra#241
CI / Detect changes (pull_request) CI bypass: infra#241
CI / Platform (Go) (pull_request) CI bypass: infra#241
CI / Canvas (Next.js) (pull_request) CI bypass: infra#241
CI / Shellcheck (E2E scripts) (pull_request) CI bypass: infra#241
Secret scan / Scan diff for credential-shaped strings (pull_request) CI bypass: infra#241
sop-tier-check / tier-check (pull_request) CI bypass: infra#241
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Python Lint & Test (pull_request) Failing after 8m15s
Per Hongming directive 2026-05-11 09:08Z: "canary naming changed to
staging for all, if there are some left overs should change too."

The "canary-" prefix was a redundant modifier on workflow files that
already targeted staging. The deployment-STRATEGY concept (a small
subset of staging tenants gets the new image first, the rest follow
on green) stays — only the workflow IDENTITY and the secret store
keys feeding it are renamed.

## Renamed surfaces

Files (git mv preserves history):
- .gitea/workflows/canary-staging.yml → staging-smoke.yml
- .gitea/workflows/canary-verify.yml  → staging-verify.yml
- scripts/canary-smoke.sh             → scripts/staging-smoke.sh

Secret-store keys (consumed by .gitea/workflows/staging-verify.yml +
scripts/staging-smoke.sh — secrets don't exist in any store yet, so
this rename is rename-first-safe per the audit Section C "truly
missing" classification):
- secrets.CANARY_TENANT_URLS       → secrets.MOLECULE_STAGING_TENANT_URLS
- secrets.CANARY_ADMIN_TOKENS      → secrets.MOLECULE_STAGING_ADMIN_TOKENS
- secrets.CANARY_CP_SHARED_SECRET  → secrets.MOLECULE_STAGING_CP_SHARED_SECRET

Env flag (test_staging_full_saas.sh + the 2 workflows that invoke it):
- E2E_MODE=canary → E2E_MODE=smoke
  (legacy "canary" alias retained for one rollout cycle; remove after
   one week of no-old-value observations)

Slug prefix (test_staging_full_saas.sh + teardown safety nets in
staging-smoke.yml + e2e-staging-sanity.yml):
- e2e-canary-{date}-* → e2e-smoke-{date}-*
  (dual-prefix fallback in both teardown nets for one rollout cycle so
   any in-flight org from an older runner checkout still cleans up)

Concurrency group + workflow name + step / job names:
- concurrency.group: canary-staging → staging-smoke
- name: "Canary — staging SaaS smoke" → "Staging SaaS smoke"
- name: "canary-verify" → "Staging verify"
- job: canary → smoke
- job: canary-smoke → staging-smoke
- step: "Canary run" → "Smoke run"
- step: "Run canary smoke suite" → "Run staging smoke suite"

Script-internal:
- CANARY_ACURL_PATH helper var → ACURL_PATH

Cross-references updated:
- e2e-staging-saas.yml + e2e-staging-sanity.yml + publish-canvas-image.yml
  + continuous-synth-e2e.yml + sweep-stale-e2e-orgs.yml + both
  redeploy-tenants-on-*.yml comment refs to the renamed workflows
- docs/architecture/canary-release.md + tests/e2e/STAGING_SAAS_E2E.md
  + scripts/README.md + runbooks/gitea-actions-migration-checklist.md

## Out of scope (deliberate)

- CANARY_SLUG / CANARY_PROMOTE_* in redeploy-tenants-on-*.yml: this is
  the soak-deploy canary slug (one-tenant-first-then-fan-out), a
  different concept than the renamed smoke workflow. Stays.
- .github/workflows/ tree: dormant mirror per
  reference_molecule_core_actions_gitea_only — Gitea Actions reads
  .gitea/ only. Sweep cleanup is a separate follow-up.
- Alert issue title "Canary failing: staging SaaS smoke" in
  staging-smoke.yml: kept stable so any open alert from the pre-rename
  filename still title-matches the auto-close search on next green.

## Verification

- grep -rn "CANARY_\|canary-staging\|canary-verify\|E2E_MODE=canary"
  .gitea/ scripts/ tests/ docs/ runbooks/ — remaining matches are
  intentional (deployment-strategy CANARY_SLUG concept, historical
  rename notes with "formerly" qualifier, soak-canary vars).
- yaml.safe_load() parses all 9 touched workflow files clean.
- bash -n on scripts/staging-smoke.sh and
  tests/e2e/test_staging_full_saas.sh.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 02:28:46 -07:00
core-devops c7e1642ffb fix(ci/harness-replays): add fetch-depth:0 to detect-changes checkout
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 29s
CI / Detect changes (pull_request) Successful in 1m13s
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m24s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 19s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m25s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m17s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 25s
sop-tier-check / tier-check (pull_request) Successful in 25s
Harness Replays / detect-changes (pull_request) Failing after 1m18s
Harness Replays / Harness Replays (pull_request) Has been skipped
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m2s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 1m14s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (pull_request) Failing after 4m39s
CI / Platform (Go) (pull_request) Successful in 9s
CI / Canvas (Next.js) (pull_request) Successful in 8s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 10s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 7s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 2m51s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 4m23s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Python Lint & Test (pull_request) Failing after 7m36s
The detect-changes step runs `git diff "$base_sha" "$head_sha"` but the
preceding `actions/checkout` uses the default fetch-depth: 1 — only the
PR head commit is fetched. The base ref (github.event.pull_request.base.sha)
is not in the local history, so git diff fails silently (2>/dev/null),
leaving DIFF empty and the step exits non-zero. With continue-on-error: true
on the job, the step reports "failure" instead of blocking the PR, but the
output is never written so downstream harness-replays always skips.

Fix: add fetch-depth: 0 to the detect-changes checkout step so full history
is fetched and both base and head refs exist locally.

Spotted during gate triage review (core-lead-agent, 2026-05-11).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 09:17:43 +00:00
23 changed files with 1082 additions and 193 deletions
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env bash
# sop-tier-refire — re-evaluate sop-tier-check and POST status to PR head SHA.
#
# Invoked from `.gitea/workflows/sop-tier-refire.yml` when a repo
# MEMBER/OWNER/COLLABORATOR comments `/refire-tier-check` on a PR.
#
# Behavior:
#
# 1. Resolve PR head SHA + author from PR_NUMBER.
# 2. Rate-limit: if the sop-tier-check context has been POSTed in the
# last 30 seconds, skip (prevents comment-spam status thrash).
# 3. Invoke `.gitea/scripts/sop-tier-check.sh` with the same env the
# canonical workflow provides. This is DRY: we re-use the exact AND-
# composition gate logic, not a watered-down approving-count check.
# 4. POST the resulting status (success on exit 0, failure on non-zero)
# to `/repos/.../statuses/{HEAD_SHA}` with context
# "sop-tier-check / tier-check (pull_request)" — the same context name
# branch protection requires.
#
# Required env (set by sop-tier-refire.yml):
# GITEA_TOKEN — org-level SOP_TIER_CHECK_TOKEN (read:org/user/issue/repo)
# GITEA_HOST — e.g. git.moleculesai.app
# REPO — owner/name
# PR_NUMBER — PR number from issue_comment payload
# COMMENT_AUTHOR — login of the commenter (logged for audit)
#
# Optional:
# SOP_DEBUG=1 — verbose per-API-call diagnostics
# SOP_REFIRE_RATE_LIMIT_SEC — override the 30s rate-limit (default 30)
# SOP_REFIRE_DISABLE_RATE_LIMIT=1 — for tests; skips the rate-limit check
set -euo pipefail
debug() {
if [ "${SOP_DEBUG:-}" = "1" ]; then
echo " [debug] $*" >&2
fi
}
: "${GITEA_TOKEN:?GITEA_TOKEN required}"
: "${GITEA_HOST:?GITEA_HOST required}"
: "${REPO:?REPO required (owner/name)}"
: "${PR_NUMBER:?PR_NUMBER required}"
: "${COMMENT_AUTHOR:=unknown}"
OWNER="${REPO%%/*}"
NAME="${REPO##*/}"
API="https://${GITEA_HOST}/api/v1"
AUTH="Authorization: token ${GITEA_TOKEN}"
CONTEXT="sop-tier-check / tier-check (pull_request)"
RATE_LIMIT_SEC="${SOP_REFIRE_RATE_LIMIT_SEC:-30}"
echo "::notice::sop-tier-refire start: repo=$OWNER/$NAME pr=$PR_NUMBER commenter=$COMMENT_AUTHOR"
# 1. Fetch PR details — need head.sha and user.login.
PR_FILE=$(mktemp)
trap 'rm -f "$PR_FILE"' EXIT
PR_HTTP=$(curl -sS -o "$PR_FILE" -w '%{http_code}' -H "$AUTH" \
"${API}/repos/${OWNER}/${NAME}/pulls/${PR_NUMBER}")
if [ "$PR_HTTP" != "200" ]; then
echo "::error::GET /pulls/$PR_NUMBER returned HTTP $PR_HTTP (body $(head -c 200 "$PR_FILE"))"
exit 1
fi
HEAD_SHA=$(jq -r '.head.sha' <"$PR_FILE")
PR_AUTHOR=$(jq -r '.user.login' <"$PR_FILE")
PR_STATE=$(jq -r '.state' <"$PR_FILE")
if [ -z "$HEAD_SHA" ] || [ "$HEAD_SHA" = "null" ]; then
echo "::error::Could not resolve head.sha from PR #$PR_NUMBER response"
exit 1
fi
debug "head_sha=$HEAD_SHA pr_author=$PR_AUTHOR state=$PR_STATE"
if [ "$PR_STATE" != "open" ]; then
echo "::notice::PR #$PR_NUMBER state is $PR_STATE; refire is a no-op on closed PRs."
exit 0
fi
# 2. Rate-limit: skip if our context was updated in the last $RATE_LIMIT_SEC.
# Gitea statuses endpoint returns latest first; we check the most recent
# entry for our context name.
if [ "${SOP_REFIRE_DISABLE_RATE_LIMIT:-}" != "1" ]; then
STATUSES_FILE=$(mktemp)
trap 'rm -f "$PR_FILE" "$STATUSES_FILE"' EXIT
ST_HTTP=$(curl -sS -o "$STATUSES_FILE" -w '%{http_code}' -H "$AUTH" \
"${API}/repos/${OWNER}/${NAME}/statuses/${HEAD_SHA}?limit=50&sort=newest")
debug "statuses-list HTTP=$ST_HTTP"
if [ "$ST_HTTP" = "200" ]; then
LAST_UPDATED=$(jq -r --arg c "$CONTEXT" \
'[.[] | select(.context == $c)] | first | .updated_at // ""' \
<"$STATUSES_FILE")
if [ -n "$LAST_UPDATED" ] && [ "$LAST_UPDATED" != "null" ]; then
# Parse RFC3339 → epoch. Use python -c for portability (date(1) -d
# differs between BSD/GNU; the Gitea runner is Ubuntu so GNU date
# works, but we keep python for future container variance).
LAST_EPOCH=$(python3 -c "import sys,datetime;print(int(datetime.datetime.fromisoformat(sys.argv[1].replace('Z','+00:00')).timestamp()))" "$LAST_UPDATED" 2>/dev/null || echo "0")
NOW_EPOCH=$(date -u +%s)
AGE=$((NOW_EPOCH - LAST_EPOCH))
debug "last status update: $LAST_UPDATED ($AGE seconds ago)"
if [ "$AGE" -lt "$RATE_LIMIT_SEC" ] && [ "$AGE" -ge 0 ]; then
echo "::notice::sop-tier-refire rate-limited — last status update was ${AGE}s ago (<${RATE_LIMIT_SEC}s window). Try again shortly."
exit 0
fi
fi
fi
fi
# 3. Invoke sop-tier-check.sh with the env it expects. Capture exit code.
# The canonical script reads tier label, walks approving reviewers, and
# evaluates the AND-composition expression — we want the SAME gate, not
# a different gate.
#
# SOP_REFIRE_TIER_CHECK_SCRIPT env var lets tests substitute a mock —
# sop-tier-check.sh uses bash 4+ associative arrays which trigger a known
# bash 3.2 parser bug (`tier: unbound variable` from declare -A with
# `set -u`). Linux Gitea runners ship bash 4/5 so production is fine;
# the override exists so the bash 3.2 dev box can still exercise the
# refire glue logic end-to-end.
SCRIPT="${SOP_REFIRE_TIER_CHECK_SCRIPT:-$(dirname "$0")/sop-tier-check.sh}"
if [ ! -f "$SCRIPT" ]; then
echo "::error::sop-tier-check.sh not found at $SCRIPT — refire requires the canonical script"
exit 1
fi
# Re-invoke. Pipe stdout/stderr through so the runner log shows the
# tier-check decision inline.
set +e
GITEA_TOKEN="$GITEA_TOKEN" \
GITEA_HOST="$GITEA_HOST" \
REPO="$REPO" \
PR_NUMBER="$PR_NUMBER" \
PR_AUTHOR="$PR_AUTHOR" \
SOP_DEBUG="${SOP_DEBUG:-0}" \
SOP_LEGACY_CHECK="${SOP_LEGACY_CHECK:-0}" \
bash "$SCRIPT"
TIER_EXIT=$?
set -e
debug "sop-tier-check.sh exit=$TIER_EXIT"
# 4. POST the resulting status.
if [ "$TIER_EXIT" -eq 0 ]; then
STATE="success"
DESCRIPTION="Refired via /refire-tier-check by $COMMENT_AUTHOR"
else
STATE="failure"
DESCRIPTION="Refired via /refire-tier-check; tier-check failed (see workflow log)"
fi
# Status target_url points at the runner log so a curious reviewer can
# follow it back. SERVER_URL + RUN_ID + JOB_ID isn't trivially constructible
# from the bash env on Gitea 1.22.6, so we point at the PR itself.
TARGET_URL="https://${GITEA_HOST}/${OWNER}/${NAME}/pulls/${PR_NUMBER}"
POST_BODY=$(jq -nc \
--arg state "$STATE" \
--arg context "$CONTEXT" \
--arg description "$DESCRIPTION" \
--arg target_url "$TARGET_URL" \
'{state:$state, context:$context, description:$description, target_url:$target_url}')
POST_FILE=$(mktemp)
trap 'rm -f "$PR_FILE" "${STATUSES_FILE:-}" "$POST_FILE"' EXIT
POST_HTTP=$(curl -sS -o "$POST_FILE" -w '%{http_code}' \
-X POST -H "$AUTH" -H "Content-Type: application/json" \
-d "$POST_BODY" \
"${API}/repos/${OWNER}/${NAME}/statuses/${HEAD_SHA}")
if [ "$POST_HTTP" != "200" ] && [ "$POST_HTTP" != "201" ]; then
echo "::error::POST /statuses/$HEAD_SHA returned HTTP $POST_HTTP (body $(head -c 200 "$POST_FILE"))"
exit 1
fi
echo "::notice::sop-tier-refire posted state=$STATE for context=\"$CONTEXT\" on sha=$HEAD_SHA"
exit "$TIER_EXIT"
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Mock sop-tier-check.sh for sop-tier-refire tests.
#
# Exits 0 ("PASS") if $MOCK_TIER_RESULT == "pass", else exits 1.
# This lets the refire tests cover the success + failure status-POST
# paths without invoking the real sop-tier-check.sh (which uses bash 4+
# associative arrays — known parser bug on macOS bash 3.2 dev box).
set -euo pipefail
case "${MOCK_TIER_RESULT:-pass}" in
pass)
echo "::notice::mock tier-check: PASS"
exit 0
;;
fail_no_label)
echo "::error::mock tier-check: no tier label"
exit 1
;;
fail_no_approvals)
echo "::error::mock tier-check: no approving reviews"
exit 1
;;
*)
echo "::error::mock tier-check: unknown MOCK_TIER_RESULT=${MOCK_TIER_RESULT:-}"
exit 2
;;
esac
+208
View File
@@ -0,0 +1,208 @@
#!/usr/bin/env python3
"""Stub Gitea API for sop-tier-refire test scenarios.
Reads $FIXTURE_STATE_DIR/scenario to decide what to return for each
endpoint the sop-tier-refire.sh + sop-tier-check.sh scripts call.
Captures every POST to /statuses/{sha} into posted_statuses.jsonl so
the test can assert what the script tried to write.
Scenarios:
T1_success — tier:low + APPROVED by engineer → tier-check passes
T2_no_tier_label — no tier label → tier-check exits 1 before POST
T3_no_approvals — tier:low but zero approving reviews → exits 1
T4_closed — PR state=closed → refire is a no-op
T5_rate_limited — last status update 5 seconds ago → skip
Usage:
FIXTURE_STATE_DIR=/tmp/x python3 _refire_fixture.py 8080
"""
import datetime
import http.server
import json
import os
import re
import sys
import urllib.parse
STATE_DIR = os.environ["FIXTURE_STATE_DIR"]
def scenario() -> str:
p = os.path.join(STATE_DIR, "scenario")
if not os.path.isfile(p):
return "T1_success"
with open(p) as f:
return f.read().strip()
def now_iso() -> str:
return datetime.datetime.now(datetime.timezone.utc).isoformat()
def append_post(body: dict) -> None:
with open(os.path.join(STATE_DIR, "posted_statuses.jsonl"), "a") as f:
f.write(json.dumps(body) + "\n")
def pr_payload() -> dict:
sc = scenario()
state = "closed" if sc == "T4_closed" else "open"
return {
"number": 999,
"state": state,
"head": {"sha": "deadbeef0000111122223333444455556666"},
"user": {"login": "feature-author"},
}
def labels_payload() -> list:
sc = scenario()
if sc == "T2_no_tier_label":
return [{"name": "bug"}]
# All other scenarios use tier:low
return [{"name": "tier:low"}, {"name": "ci"}]
def reviews_payload() -> list:
sc = scenario()
if sc == "T3_no_approvals":
return []
# All other scenarios have one APPROVED review by an engineer
return [
{
"state": "APPROVED",
"user": {"login": "reviewer-engineer"},
}
]
def teams_payload() -> list:
# Mirror the real molecule-ai org teams referenced in TIER_EXPR
return [
{"id": 5, "name": "ceo"},
{"id": 2, "name": "engineers"},
{"id": 6, "name": "managers"},
]
def statuses_payload() -> list:
sc = scenario()
if sc == "T5_rate_limited":
recent = (
datetime.datetime.now(datetime.timezone.utc)
- datetime.timedelta(seconds=5)
).isoformat()
return [
{
"context": "sop-tier-check / tier-check (pull_request)",
"state": "failure",
"updated_at": recent,
}
]
return []
def user_payload() -> dict:
# Mirrors the WHOAMI probe in sop-tier-check.sh
return {"login": "sop-tier-bot-fixture"}
class Handler(http.server.BaseHTTPRequestHandler):
# Quiet — keep stdout for explicit logs only.
def log_message(self, *args, **kwargs): # noqa: D401
pass
def _json(self, code: int, body) -> None:
payload = json.dumps(body).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def _empty(self, code: int) -> None:
self.send_response(code)
self.send_header("Content-Length", "0")
self.end_headers()
def do_GET(self): # noqa: N802
u = urllib.parse.urlparse(self.path)
path = u.path
if path == "/_ping":
return self._json(200, {"ok": True})
if path == "/api/v1/user":
return self._json(200, user_payload())
# /api/v1/repos/{owner}/{name}/pulls/{n}
m = re.match(r"^/api/v1/repos/[^/]+/[^/]+/pulls/(\d+)$", path)
if m:
return self._json(200, pr_payload())
# /api/v1/repos/{owner}/{name}/issues/{n}/labels
if re.match(r"^/api/v1/repos/[^/]+/[^/]+/issues/\d+/labels$", path):
return self._json(200, labels_payload())
# /api/v1/repos/{owner}/{name}/pulls/{n}/reviews
if re.match(r"^/api/v1/repos/[^/]+/[^/]+/pulls/\d+/reviews$", path):
return self._json(200, reviews_payload())
# /api/v1/orgs/{owner}/teams
if re.match(r"^/api/v1/orgs/[^/]+/teams$", path):
return self._json(200, teams_payload())
# /api/v1/teams/{id}/members/{login} → 204 if user is an engineer
m = re.match(r"^/api/v1/teams/(\d+)/members/([^/]+)$", path)
if m:
team_id, login = m.group(1), m.group(2)
# In our fixture reviewer-engineer ∈ engineers (id=2)
if team_id == "2" and login == "reviewer-engineer":
return self._empty(204)
return self._empty(404)
# /api/v1/orgs/{owner}/members/{login} — fallback path used when
# team-member probes all 403. We don't need it for these tests.
if re.match(r"^/api/v1/orgs/[^/]+/members/[^/]+$", path):
return self._empty(404)
# /api/v1/repos/{owner}/{name}/statuses/{sha}
if re.match(r"^/api/v1/repos/[^/]+/[^/]+/statuses/[^/]+$", path):
return self._json(200, statuses_payload())
return self._json(404, {"path": path, "msg": "fixture: no route"})
def do_POST(self): # noqa: N802
u = urllib.parse.urlparse(self.path)
path = u.path
length = int(self.headers.get("Content-Length") or 0)
raw = self.rfile.read(length) if length else b""
try:
body = json.loads(raw) if raw else {}
except Exception:
body = {"_raw": raw.decode(errors="replace")}
if re.match(r"^/api/v1/repos/[^/]+/[^/]+/statuses/[^/]+$", path):
append_post(body)
# Echo back something status-shaped — script only checks HTTP code.
return self._json(
201,
{
"context": body.get("context"),
"state": body.get("state"),
"created_at": now_iso(),
},
)
return self._json(404, {"path": path, "msg": "fixture: no route"})
def main():
port = int(sys.argv[1])
srv = http.server.ThreadingHTTPServer(("127.0.0.1", port), Handler)
srv.serve_forever()
if __name__ == "__main__":
main()
+297
View File
@@ -0,0 +1,297 @@
#!/usr/bin/env bash
# Tests for sop-tier-refire.{yml,sh} — internal#292.
#
# Behavior matrix:
#
# T1: PR open + APPROVED via tier:low → script invokes sop-tier-check
# and POSTs status=success.
# T2: PR open + missing tier label → sop-tier-check exits non-zero;
# refire POSTs status=failure (description mentions failure).
# T3: PR open + tier:low but NO approving reviews → sop-tier-check
# exits non-zero; refire POSTs status=failure.
# T4: PR CLOSED → refire exits 0 with no status POST (no-op on closed).
# T5: Rate-limit — recent status update within 30s → refire skips,
# no new POST.
# T6 (yaml-lint): workflow `if:` expression contains author_association
# gate + slash-command-trigger gate + PR-not-issue gate.
# T7 (yaml-lint): workflow file is parseable YAML.
#
# Tests T1-T5 run the real script against a local-fixture HTTP server
# (python http.server with a stub handler — `tests/_refire_fixture.py`)
# so the script's Gitea API calls hit the fixture, not the real Gitea.
#
# Tests T6/T7 are pure YAML checks against the workflow file.
#
# Hostile-self-review (per feedback_assert_exact_not_substring):
# this test MUST FAIL if the workflow or script is absent. Verified by
# running the test before the files exist (covered in the PR body).
set -euo pipefail
THIS_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT_DIR="$(cd "$THIS_DIR/.." && pwd)"
WORKFLOW_DIR="$(cd "$THIS_DIR/../../workflows" && pwd)"
WORKFLOW="$WORKFLOW_DIR/sop-tier-refire.yml"
SCRIPT="$SCRIPT_DIR/sop-tier-refire.sh"
PASS=0
FAIL=0
FAILED_TESTS=""
assert_eq() {
local label="$1"
local expected="$2"
local got="$3"
if [ "$expected" = "$got" ]; then
echo " PASS $label"
PASS=$((PASS + 1))
else
echo " FAIL $label"
echo " expected: <$expected>"
echo " got: <$got>"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} ${label}"
fi
}
assert_contains() {
local label="$1"
local needle="$2"
local haystack="$3"
if printf '%s' "$haystack" | grep -qF "$needle"; then
echo " PASS $label"
PASS=$((PASS + 1))
else
echo " FAIL $label"
echo " needle: <$needle>"
echo " haystack: <$(printf '%s' "$haystack" | head -c 400)>"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} ${label}"
fi
}
assert_file_exists() {
local label="$1"
local path="$2"
if [ -f "$path" ]; then
echo " PASS $label"
PASS=$((PASS + 1))
else
echo " FAIL $label (not found: $path)"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} ${label}"
fi
}
# Existence (foundation — every other test depends on these)
echo
echo "== existence =="
assert_file_exists "workflow file exists" "$WORKFLOW"
assert_file_exists "script file exists" "$SCRIPT"
if [ "$FAIL" -gt 0 ]; then
echo
echo "------"
echo "PASS=$PASS FAIL=$FAIL (existence)"
echo "Cannot proceed without these files."
exit 1
fi
# T6 / T7 — workflow YAML structure
echo
echo "== T6/T7 workflow yaml =="
# YAML parseability
PARSE_OUT=$(python3 -c 'import sys,yaml;yaml.safe_load(open(sys.argv[1]).read());print("ok")' "$WORKFLOW" 2>&1 || true)
assert_eq "T7 workflow parses as YAML" "ok" "$PARSE_OUT"
# Three required gates in the `if:` expression
WORKFLOW_CONTENT=$(cat "$WORKFLOW")
assert_contains "T6a workflow if: contains author_association gate" \
"github.event.comment.author_association" "$WORKFLOW_CONTENT"
assert_contains "T6b workflow if: gates on MEMBER/OWNER/COLLABORATOR" \
'["MEMBER","OWNER","COLLABORATOR"]' "$WORKFLOW_CONTENT"
assert_contains "T6c workflow if: contains slash-command trigger" \
"/refire-tier-check" "$WORKFLOW_CONTENT"
assert_contains "T6d workflow if: gates on PR-not-issue" \
"github.event.issue.pull_request" "$WORKFLOW_CONTENT"
assert_contains "T6e workflow listens on issue_comment" \
"issue_comment" "$WORKFLOW_CONTENT"
assert_contains "T6f workflow requests statuses:write permission" \
"statuses: write" "$WORKFLOW_CONTENT"
# Does NOT check out PR HEAD (security)
if grep -q 'ref: \${{ github.event.pull_request.head' "$WORKFLOW"; then
echo " FAIL T6g workflow MUST NOT check out PR head (security)"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} T6g"
else
echo " PASS T6g workflow does not check out PR head"
PASS=$((PASS + 1))
fi
# T1-T5 — script behavior against a local Gitea-fixture
echo
echo "== T1-T5 script behavior (vs local fixture) =="
# Spin up the fixture HTTP server.
FIXTURE_DIR=$(mktemp -d)
trap 'rm -rf "$FIXTURE_DIR"; [ -n "${FIX_PID:-}" ] && kill "$FIX_PID" 2>/dev/null || true' EXIT
FIXTURE_PY="$THIS_DIR/_refire_fixture.py"
if [ ! -f "$FIXTURE_PY" ]; then
echo "::error::fixture server $FIXTURE_PY missing"
exit 1
fi
FIX_LOG="$FIXTURE_DIR/fixture.log"
FIX_STATE_DIR="$FIXTURE_DIR/state"
mkdir -p "$FIX_STATE_DIR"
# Find an unused port.
FIX_PORT=$(python3 -c 'import socket;s=socket.socket();s.bind(("127.0.0.1",0));print(s.getsockname()[1]);s.close()')
FIXTURE_STATE_DIR="$FIX_STATE_DIR" python3 "$FIXTURE_PY" "$FIX_PORT" \
>"$FIX_LOG" 2>&1 &
FIX_PID=$!
# Wait for fixture readiness.
for _ in $(seq 1 50); do
if curl -fsS "http://127.0.0.1:${FIX_PORT}/_ping" >/dev/null 2>&1; then
break
fi
sleep 0.1
done
if ! curl -fsS "http://127.0.0.1:${FIX_PORT}/_ping" >/dev/null 2>&1; then
echo "::error::fixture server failed to start. Log:"
cat "$FIX_LOG"
exit 1
fi
# Helper: set fixture state for a scenario, then run the script.
# tier_result is one of: pass | fail_no_label | fail_no_approvals.
# The refire script's tier-check invocation is mocked because the real
# sop-tier-check.sh uses bash 4+ associative arrays — incompatible with
# the macOS bash 3.2 dev shell. Linux Gitea runners use bash 4/5 so
# production runs the real script. The mock exercises the success +
# failure branches of refire's status-POST glue.
run_scenario() {
local scenario="$1"
local tier_result="${2:-pass}"
echo "$scenario" >"$FIX_STATE_DIR/scenario"
: >"$FIX_STATE_DIR/posted_statuses.jsonl" # clear status log
local out
set +e
out=$(
PATH="$FIXTURE_DIR/bin:$PATH" \
GITEA_TOKEN="fixture-token" \
GITEA_HOST="fixture.local" \
REPO="molecule-ai/molecule-core" \
PR_NUMBER="999" \
COMMENT_AUTHOR="test-runner" \
SOP_REFIRE_DISABLE_RATE_LIMIT="1" \
SOP_REFIRE_TIER_CHECK_SCRIPT="$THIS_DIR/_mock_tier_check.sh" \
MOCK_TIER_RESULT="$tier_result" \
FIXTURE_PORT="$FIX_PORT" \
bash "$SCRIPT" 2>&1
)
local rc=$?
set -e
echo "$out" >"$FIX_STATE_DIR/last_run.log"
echo "$rc" >"$FIX_STATE_DIR/last_rc"
}
# Install a curl shim that rewrites https://fixture.local → http://127.0.0.1:$PORT
# Use bash prefix-strip (${var#prefix}) — it sidesteps the `/` delimiter
# confusion of ${var/pattern/replacement}.
mkdir -p "$FIXTURE_DIR/bin"
cat >"$FIXTURE_DIR/bin/curl" <<SHIM
#!/usr/bin/env bash
# Test shim: rewrite https://fixture.local/* -> http://127.0.0.1:${FIX_PORT}/*
# The fixture doesn't authenticate; -H Authorization passes through harmlessly.
new_args=()
for a in "\$@"; do
if [[ "\$a" == https://fixture.local/* ]]; then
rest="\${a#https://fixture.local}"
a="http://127.0.0.1:${FIX_PORT}\${rest}"
fi
new_args+=("\$a")
done
exec /usr/bin/curl "\${new_args[@]}"
SHIM
chmod +x "$FIXTURE_DIR/bin/curl"
# T1: tier:low + 1 APPROVED + author is in engineers team → success
run_scenario "T1_success" "pass"
RC=$(cat "$FIX_STATE_DIR/last_rc")
POSTED=$(cat "$FIX_STATE_DIR/posted_statuses.jsonl" 2>/dev/null || true)
assert_eq "T1 exit code 0 (success)" "0" "$RC"
assert_contains "T1 POSTed state=success" '"state": "success"' "$POSTED"
assert_contains "T1 POST context is sop-tier-check / tier-check" \
'"context": "sop-tier-check / tier-check (pull_request)"' "$POSTED"
assert_contains "T1 description names commenter" "test-runner" "$POSTED"
# T2: missing tier label → tier-check fails → failure status POSTed
run_scenario "T2_no_tier_label" "fail_no_label"
RC=$(cat "$FIX_STATE_DIR/last_rc")
POSTED=$(cat "$FIX_STATE_DIR/posted_statuses.jsonl" 2>/dev/null || true)
# tier-check.sh exits 1; refire script forwards that exit, so RC != 0
if [ "$RC" -ne 0 ]; then
echo " PASS T2 exit code non-zero (got $RC)"
PASS=$((PASS + 1))
else
echo " FAIL T2 exit code should be non-zero, got 0"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} T2_rc"
fi
assert_contains "T2 POSTed state=failure" '"state": "failure"' "$POSTED"
# T3: tier:low present but ZERO approving reviews → failure
run_scenario "T3_no_approvals" "fail_no_approvals"
RC=$(cat "$FIX_STATE_DIR/last_rc")
POSTED=$(cat "$FIX_STATE_DIR/posted_statuses.jsonl" 2>/dev/null || true)
if [ "$RC" -ne 0 ]; then
echo " PASS T3 exit code non-zero (got $RC)"
PASS=$((PASS + 1))
else
echo " FAIL T3 exit code should be non-zero, got 0"
FAIL=$((FAIL + 1))
FAILED_TESTS="${FAILED_TESTS} T3_rc"
fi
assert_contains "T3 POSTed state=failure" '"state": "failure"' "$POSTED"
# T4: closed PR — refire is a no-op (no POST, exit 0)
run_scenario "T4_closed" "pass"
RC=$(cat "$FIX_STATE_DIR/last_rc")
POSTED=$(cat "$FIX_STATE_DIR/posted_statuses.jsonl" 2>/dev/null || true)
assert_eq "T4 closed PR exits 0" "0" "$RC"
assert_eq "T4 closed PR posts no status" "" "$POSTED"
# T5: rate-limit — disable the env override and let scenario set a
# recent statuses entry. Re-enable rate-limit for this scenario by NOT
# passing SOP_REFIRE_DISABLE_RATE_LIMIT.
echo "T5_rate_limited" >"$FIX_STATE_DIR/scenario"
: >"$FIX_STATE_DIR/posted_statuses.jsonl"
set +e
T5_OUT=$(
PATH="$FIXTURE_DIR/bin:$PATH" \
GITEA_TOKEN="fixture-token" \
GITEA_HOST="fixture.local" \
REPO="molecule-ai/molecule-core" \
PR_NUMBER="999" \
COMMENT_AUTHOR="test-runner" \
FIXTURE_PORT="$FIX_PORT" \
bash "$SCRIPT" 2>&1
)
T5_RC=$?
set -e
POSTED=$(cat "$FIX_STATE_DIR/posted_statuses.jsonl" 2>/dev/null || true)
assert_eq "T5 rate-limited exits 0" "0" "$T5_RC"
assert_contains "T5 rate-limited log says skipped" "rate-limited" "$T5_OUT"
assert_eq "T5 rate-limited posts no status" "" "$POSTED"
echo
echo "------"
echo "PASS=$PASS FAIL=$FAIL"
if [ "$FAIL" -gt 0 ]; then
echo "Failed:$FAILED_TESTS"
fi
[ "$FAIL" -eq 0 ]
+1 -1
View File
@@ -56,7 +56,7 @@ on:
# 2. Avoid colliding with the existing :15 sweep-cf-orphans
# and :45 sweep-cf-tunnels — both hit the CF API and we
# don't want to fight for rate-limit tokens.
# 3. Avoid the :30 heavy slot (canary-staging /30, sweep-aws-
# 3. Avoid the :30 heavy slot (staging-smoke /30, sweep-aws-
# secrets, sweep-stale-e2e-orgs every :15) — multiple
# overlapping cron registrations on the same minute is part
# of what GH drops under load.
+1 -1
View File
@@ -95,7 +95,7 @@ jobs:
# ANTHROPIC_BASE_URL to api.minimax.io/anthropic and reads
# MINIMAX_API_KEY at boot — separate billing account so an
# OpenAI quota collapse no longer wedges the gate. Mirrors the
# canary-staging.yml + continuous-synth-e2e.yml migrations.
# staging-smoke.yml + continuous-synth-e2e.yml migrations.
E2E_MINIMAX_API_KEY: ${{ secrets.MOLECULE_STAGING_MINIMAX_API_KEY }}
# Direct-Anthropic alternative for operators who don't want to
# set up a MiniMax account (priority below MiniMax — first
+13 -7
View File
@@ -11,11 +11,11 @@ name: E2E Staging Sanity (leak-detection self-check)
# - `continue-on-error: true` on the job (RFC §1 contract).
#
# Periodic assertion that the teardown safety nets in e2e-staging-saas
# and canary-staging actually work. Runs the E2E harness with
# E2E_INTENTIONAL_FAILURE=1, which poisons the tenant admin token after
# the org is provisioned. The workspace-provision step then fails, the
# script exits non-zero, and the EXIT trap + workflow always()-step
# must still tear down cleanly.
# and staging-smoke (formerly canary-staging) actually work. Runs the
# E2E harness with E2E_INTENTIONAL_FAILURE=1, which poisons the tenant
# admin token after the org is provisioned. The workspace-provision
# step then fails, the script exits non-zero, and the EXIT trap +
# workflow always()-step must still tear down cleanly.
on:
schedule:
@@ -43,7 +43,7 @@ jobs:
env:
MOLECULE_CP_URL: https://staging-api.moleculesai.app
MOLECULE_ADMIN_TOKEN: ${{ secrets.MOLECULE_STAGING_ADMIN_TOKEN }}
E2E_MODE: canary
E2E_MODE: smoke
E2E_RUNTIME: hermes
E2E_RUN_ID: "sanity-${{ github.run_id }}"
E2E_INTENTIONAL_FAILURE: "1"
@@ -127,8 +127,14 @@ jobs:
import json, sys
d = json.load(sys.stdin)
today = __import__('datetime').date.today().strftime('%Y%m%d')
# Match both the new e2e-smoke- prefix (post-2026-05-11 rename)
# and the legacy e2e-canary- prefix for one rollout cycle so
# any in-flight org provisioned under the old prefix on an
# older runner checkout still gets cleaned up. Remove the
# canary fallback after one week of no-old-prefix observations.
prefixes = (f'e2e-smoke-{today}-sanity-', f'e2e-canary-{today}-sanity-')
candidates = [o['slug'] for o in d.get('orgs', [])
if o.get('slug','').startswith(f'e2e-canary-{today}-sanity-')
if any(o.get('slug','').startswith(p) for p in prefixes)
and o.get('status') not in ('purged',)]
print('\n'.join(candidates))
" 2>/dev/null)
+35 -1
View File
@@ -68,7 +68,35 @@ jobs:
run: ${{ steps.decide.outputs.run }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Fetch base branch tip for diff
continue-on-error: true
run: |
# With the default fetch-depth: 1, actions/checkout only fetches the
# PR head commit. The base commit is NOT in the local history, so
# `git diff "$BASE" "$GITHUB_SHA"` fails. Fetch the base branch at
# depth 1 — the base commit is the immediate parent of the PR head
# on the base branch, so depth=1 is sufficient.
#
# Network: Gitea Actions runner (5.78.80.188) cannot reach the git
# remote over HTTPS (confirmed: git fetch times out at ~15s). The runner
# is on the same host as Gitea, but the container network namespace
# cannot reach the Gitea HTTPS endpoint.
#
# Fallback: if the base commit does not exist locally, skip the diff
# and set run=true (always run harness). This is safe: PRs where the
# base is unavailable still run the harness (correct), PRs where the
# base IS available get the correct path-based diff.
#
# Timeout: 20s. If the fetch completes, great. If it times out, the
# step exits non-zero and we fall through to run=true.
if timeout 20 git fetch origin "${{ github.event.pull_request.base.ref }}" --depth=1; then
echo "::notice::base branch fetched successfully"
else
echo "::warning::git fetch origin ${{ github.event.pull_request.base.ref }} --depth=1 timed out"
echo "::warning::Skipping diff — detect-changes will run the harness unconditionally."
fi
- id: decide
continue-on-error: true
run: |
# workflow_dispatch: always run (manual trigger)
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
@@ -95,7 +123,13 @@ jobs:
fi
# GitHub Actions and Gitea Actions both expose github.sha for HEAD.
DIFF=$(git diff --name-only "$BASE" "${{ github.sha }}" 2>/dev/null)
# git diff exits 1 when BASE is not in local history (e.g. shallow
# checkout where the base commit was never fetched). Capture and
# swallow that exit code — the empty diff means "run everything".
# The runner network cannot reach the git remote (confirmed: git fetch
# times out at ~15s), so a failed fetch is expected and we always fall
# through to the unconditional run=true below.
DIFF=$(git diff --name-only "$BASE" "${{ github.sha }}" 2>/dev/null) || true
echo "debug=diff-base=$BASE diff-files=$DIFF" >> "$GITHUB_OUTPUT"
if echo "$DIFF" | grep -qE '^workspace-server/|^canvas/|^tests/harness/|^.gitea/workflows/harness-replays\.yml$'; then
+1 -1
View File
@@ -11,7 +11,7 @@ name: publish-canvas-image
# - `continue-on-error: true` on each job (RFC §1 contract).
# - **Open question for review**: this workflow pushes the canvas
# image to `ghcr.io`. GHCR was retired during the 2026-05-06
# Gitea migration in favor of ECR (per canary-verify.yml header
# Gitea migration in favor of ECR (per staging-verify.yml header
# notes). The image may not be consumable post-migration. Two
# options for follow-up: (a) retarget to
# `153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/canvas`,
@@ -32,7 +32,7 @@ name: redeploy-tenants-on-main
#
# Registry: ECR (153263036946.dkr.ecr.us-east-2.amazonaws.com/
# molecule-ai/platform-tenant). GHCR was retired 2026-05-07 during the
# Gitea suspension migration. The canary-verify.yml promote step now
# Gitea suspension migration. The staging-verify.yml promote step now
# uses the same redeploy-fleet endpoint (fixes the silent-GHCR gap).
#
# Runtime ordering:
@@ -104,7 +104,7 @@ jobs:
# `staging-<sha>` to roll back to a known-good build.
# 2. Default → `staging-<short_head_sha>`. The just-published
# digest. Bypasses the `:latest` retag path that's currently
# dead (canary-verify soft-skips without canary fleet, so
# dead (staging-verify soft-skips without canary fleet, so
# the only thing retagging `:latest` today is the manual
# promote-latest.yml — last run 2026-04-28). Auto-trigger
# from workflow_run uses workflow_run.head_sha; manual
@@ -359,7 +359,7 @@ jobs:
# Belt-and-suspenders sanity floor: same logic as the staging
# variant — see that file's comment for the full rationale.
# Floor only applies when fleet >= 4; below that, canary-verify
# Floor only applies when fleet >= 4; below that, staging-verify
# is the actual gate.
TOTAL_VERIFIED=${#SLUGS[@]}
if [ $TOTAL_VERIFIED -ge 4 ] && [ $UNREACHABLE_COUNT -gt $((TOTAL_VERIFIED / 2)) ]; then
@@ -21,7 +21,7 @@ name: redeploy-tenants-on-staging
#
# Mirror of redeploy-tenants-on-main.yml, with the staging-CP host and
# the :staging-latest tag. Sister workflow exists for prod (rolls
# :latest after canary-verify). Both share the same shape — just
# :latest after staging-verify). Both share the same shape — just
# different CP_URL + target_tag + admin token secret.
#
# Why this workflow exists: publish-workspace-server-image now builds
@@ -336,7 +336,7 @@ jobs:
# crashes on startup), not a teardown race. Hard-fail.
#
# Floor only applies when TOTAL_VERIFIED >= 4 — below that, the
# canary-verify step is the actual gate for "all tenants down"
# staging-verify step is the actual gate for "all tenants down"
# detection (it runs against the canary first and aborts the
# rollout if the canary fails to come up). Without the >=4 gate,
# a 1-tenant fleet (e.g. a single ephemeral e2e-* tenant on a
+79
View File
@@ -0,0 +1,79 @@
# sop-tier-refire — issue_comment-triggered refire of sop-tier-check.
#
# Closes internal#292. Gitea 1.22.6 doesn't refire workflows on the
# `pull_request_review` event (go-gitea/gitea#33700); the `sop-tier-check`
# workflow's review-event subscription is silently dead. The result:
# PRs that get their approving review AFTER the tier-check ran on open/
# synchronize keep their failing status check forever, and the only way
# to merge is the admin force-merge path (audited via `audit-force-merge`
# but the audit trail keeps growing; see `feedback_never_admin_merge_bypass`).
#
# Workaround pattern from `feedback_pull_request_review_no_refire`:
# `issue_comment` events DO fire reliably on 1.22.6. When a repo
# MEMBER/OWNER/COLLABORATOR comments `/refire-tier-check` on a PR, this
# workflow re-runs the sop-tier-check logic and POSTs the resulting
# status to the PR head SHA directly. No empty commit, no git history
# bloat, no cascade re-fire of every other workflow on the PR.
#
# SECURITY MODEL:
#
# 1. `pull_request` exists on the issue (issue_comment fires on issues
# AND PRs; we only want PRs).
# 2. `comment.author_association` must be MEMBER/OWNER/COLLABORATOR.
# Per the internal#292 core-security review (review#1066 ask): anyone
# can comment, but only repo collaborators+ can flip the status.
# Without this gate, a drive-by commenter on a public-issue-tracker
# surface could trigger a status flip.
# 3. Comment body must contain `/refire-tier-check` — a slash-command-
# shaped trigger (not just any comment word). Prevents accidental
# triggering from prose like "we should refire tests" in a review.
# 4. This workflow does NOT check out PR HEAD code. Like sop-tier-check,
# it only HTTP-calls the Gitea API. Trust boundary preserved.
#
# Note: `issue_comment` fires from the BASE branch's workflow file. There
# is no `pull_request_target` equivalent to set; the trigger inherently
# loads the workflow from the default branch.
#
# Rate-limit: a 1s pre-sleep + a "skip if status posted in last 30s"
# guard prevents comment-spam from thrashing the status. See the script.
name: sop-tier-check refire (issue_comment)
on:
issue_comment:
types: [created]
jobs:
refire:
# Three gates, all required:
# - comment is on a PR (not a plain issue)
# - commenter is MEMBER, OWNER, or COLLABORATOR
# - comment body contains the slash-command trigger
if: |
github.event.issue.pull_request != null &&
contains(fromJson('["MEMBER","OWNER","COLLABORATOR"]'), github.event.comment.author_association) &&
contains(github.event.comment.body, '/refire-tier-check')
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
statuses: write
steps:
- name: Check out base branch (for the script)
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
# Load the script from the default branch (main), matching the
# sop-tier-check.yml security model.
ref: ${{ github.event.repository.default_branch }}
- name: Re-evaluate sop-tier-check and POST status
env:
# Same org-level secret sop-tier-check.yml + audit-force-merge.yml use.
# Fallback to GITHUB_TOKEN with a clear error if missing.
GITEA_TOKEN: ${{ secrets.SOP_TIER_CHECK_TOKEN || secrets.GITHUB_TOKEN }}
GITEA_HOST: git.moleculesai.app
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.issue.number }}
COMMENT_AUTHOR: ${{ github.event.comment.user.login }}
# Set to '1' for diagnostic per-API-call output. Off by default.
SOP_DEBUG: '0'
run: bash .gitea/scripts/sop-tier-refire.sh
@@ -1,6 +1,8 @@
name: Canary — staging SaaS smoke (every 30 min)
name: Staging SaaS smoke (every 30 min)
# Ported from .github/workflows/canary-staging.yml on 2026-05-11 per RFC
# Renamed from canary-staging.yml on 2026-05-11 per Hongming directive
# ("canary naming changed to staging for all"). Originally ported from
# .github/workflows/canary-staging.yml on 2026-05-11 per RFC
# internal#219 §1 sweep. Differences from the GitHub version:
# - Dropped `workflow_dispatch.inputs` (Gitea 1.22.6 parser rejects them
# per feedback_gitea_workflow_dispatch_inputs_unsupported).
@@ -21,21 +23,21 @@ name: Canary — staging SaaS smoke (every 30 min)
# catches drift in the 30-min window between those runs (AMI health, CF
# cert rotation, WorkOS session stability, etc.).
#
# Lean mode: E2E_MODE=canary skips the child workspace + HMA memory +
# Lean mode: E2E_MODE=smoke skips the child workspace + HMA memory +
# peers/activity checks. One parent workspace + one A2A turn is enough
# to signal "SaaS stack end-to-end is alive."
on:
schedule:
# Every 30 min. Cron on GitHub-hosted runners has a known drift of
# a few minutes under load — that's fine for a canary.
# a few minutes under load — that's fine for a smoke check.
- cron: '*/30 * * * *'
# Serialise with the full-SaaS workflow so they don't contend for the
# same org-create quota on staging. Different group key from
# e2e-staging-saas since we don't mind queueing canaries behind one
# full run, but two canaries SHOULD queue against each other.
# e2e-staging-saas since we don't mind queueing smoke runs behind one
# full run, but two smoke runs SHOULD queue against each other.
concurrency:
group: canary-staging
group: staging-smoke
cancel-in-progress: false
permissions:
@@ -47,8 +49,8 @@ env:
GITHUB_SERVER_URL: https://git.moleculesai.app
jobs:
canary:
name: Canary smoke
smoke:
name: Staging SaaS smoke
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
continue-on-error: true
@@ -56,23 +58,23 @@ jobs:
# tests/e2e/test_staging_full_saas.sh (#2107). Without the buffer
# the job is killed at the wall-clock 15:00 mark BEFORE the bash
# `fail` + diagnostic burst can fire, leaving every cancellation
# silent. Sibling staging E2E jobs run at 20-45 min — keeping
# canary tighter than them so a true wedge still surfaces here
# silent. Sibling staging E2E jobs run at 20-45 min — keeping the
# smoke tighter than them so a true wedge still surfaces here
# first.
timeout-minutes: 25
env:
MOLECULE_CP_URL: https://staging-api.moleculesai.app
MOLECULE_ADMIN_TOKEN: ${{ secrets.MOLECULE_STAGING_ADMIN_TOKEN }}
# MiniMax is the canary's PRIMARY LLM auth path post-2026-05-04.
# MiniMax is the smoke's PRIMARY LLM auth path post-2026-05-04.
# Switched from hermes+OpenAI after #2578 (the staging OpenAI key
# account went over quota and stayed dead for 36+ hours, taking
# the canary red the entire time). claude-code template's
# the smoke red the entire time). claude-code template's
# `minimax` provider routes ANTHROPIC_BASE_URL to
# api.minimax.io/anthropic and reads MINIMAX_API_KEY at boot —
# ~5-10x cheaper per token than gpt-4.1-mini AND on a separate
# billing account, so OpenAI quota collapse no longer wedges the
# canary. Mirrors the migration continuous-synth-e2e.yml made on
# smoke. Mirrors the migration continuous-synth-e2e.yml made on
# 2026-05-03 (#265) for the same reason. tests/e2e/test_staging_
# full_saas.sh branches SECRETS_JSON on which key is present —
# MiniMax wins when set.
@@ -86,16 +88,16 @@ jobs:
# E2E_RUNTIME=hermes overridden via workflow_dispatch can still
# exercise the OpenAI path without re-editing the workflow.
E2E_OPENAI_API_KEY: ${{ secrets.MOLECULE_STAGING_OPENAI_API_KEY }}
E2E_MODE: canary
E2E_MODE: smoke
E2E_RUNTIME: claude-code
# Pin the canary to a specific MiniMax model rather than relying
# Pin the smoke to a specific MiniMax model rather than relying
# on the per-runtime default (which could resolve to "sonnet" →
# direct Anthropic and defeat the cost saving). M2.7-highspeed
# is "Token Plan only" but cheap-per-token and fast.
E2E_MODEL_SLUG: MiniMax-M2.7-highspeed
E2E_RUN_ID: "canary-${{ github.run_id }}"
E2E_RUN_ID: "smoke-${{ github.run_id }}"
# Debug-only: when an operator dispatches with keep_on_failure=true,
# the canary script's E2E_KEEP_ORG=1 path skips teardown so the
# the smoke script's E2E_KEEP_ORG=1 path skips teardown so the
# tenant org + EC2 stay alive for SSM-based log capture. Cron runs
# never set this (the input only exists on workflow_dispatch) so
# unattended cron always tears down. See molecule-core#129
@@ -119,7 +121,7 @@ jobs:
# langgraph (operator-dispatched only) use OpenAI. Hard-fail
# rather than soft-skip per the lesson from synth E2E #2578:
# an empty key silently falls through to the wrong
# SECRETS_JSON branch and the canary fails 5 min later with
# SECRETS_JSON branch and the smoke fails 5 min later with
# a confusing auth error instead of the clean "secret
# missing" message at the top.
case "${E2E_RUNTIME}" in
@@ -155,8 +157,8 @@ jobs:
fi
echo "LLM key present ✓ (runtime=${E2E_RUNTIME}, key=${required_secret_name}, len=${#required_secret_value})"
- name: Canary run
id: canary
- name: Smoke run
id: smoke
run: bash tests/e2e/test_staging_full_saas.sh
# Alerting: open a sticky issue on the FIRST failure; comment on
@@ -184,6 +186,9 @@ jobs:
run: |
set -euo pipefail
API="${SERVER_URL%/}/api/v1"
# Title kept stable across the canary-staging.yml → staging-smoke.yml
# rename (2026-05-11) so any open alert issue from the old name
# still title-matches and auto-closes on the next green run.
TITLE="Canary failing: staging SaaS smoke"
RUN_URL="${SERVER_URL}/${REPO}/actions/runs/${RUN_ID}"
@@ -194,18 +199,18 @@ jobs:
if [ -n "$EXISTING" ]; then
curl -fsS -X POST -H "Authorization: token $GITEA_TOKEN" -H "Content-Type: application/json" \
"${API}/repos/${REPO}/issues/${EXISTING}/comments" \
-d "$(jq -nc --arg run "$RUN_URL" '{body: ("Canary still failing. " + $run)}')" >/dev/null
-d "$(jq -nc --arg run "$RUN_URL" '{body: ("Smoke still failing. " + $run)}')" >/dev/null
echo "Commented on existing issue #${EXISTING}"
else
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
BODY=$(jq -nc --arg t "$TITLE" --arg now "$NOW" --arg run "$RUN_URL" \
'{title: $t, body: ("Canary run failed at " + $now + ".\n\nRun: " + $run + "\n\nThis issue auto-closes on the next green canary run. Consecutive failures add a comment here rather than a new issue.")}')
'{title: $t, body: ("Smoke run failed at " + $now + ".\n\nRun: " + $run + "\n\nThis issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.")}')
curl -fsS -X POST -H "Authorization: token $GITEA_TOKEN" -H "Content-Type: application/json" \
"${API}/repos/${REPO}/issues" -d "$BODY" >/dev/null
echo "Opened canary failure issue (first red)"
echo "Opened smoke failure issue (first red)"
fi
- name: Auto-close canary issue on success (Gitea API)
- name: Auto-close smoke issue on success (Gitea API)
if: success()
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -215,6 +220,8 @@ jobs:
run: |
set -euo pipefail
API="${SERVER_URL%/}/api/v1"
# Title kept stable across the canary-staging.yml → staging-smoke.yml
# rename so open alert issues from the old name still match.
TITLE="Canary failing: staging SaaS smoke"
NUMS=$(curl -fsS -H "Authorization: token $GITEA_TOKEN" \
@@ -225,10 +232,10 @@ jobs:
for N in $NUMS; do
curl -fsS -X POST -H "Authorization: token $GITEA_TOKEN" -H "Content-Type: application/json" \
"${API}/repos/${REPO}/issues/${N}/comments" \
-d "$(jq -nc --arg now "$NOW" '{body: ("Canary recovered at " + $now + ". Closing.")}')" >/dev/null
-d "$(jq -nc --arg now "$NOW" '{body: ("Smoke recovered at " + $now + ". Closing.")}')" >/dev/null
curl -fsS -X PATCH -H "Authorization: token $GITEA_TOKEN" -H "Content-Type: application/json" \
"${API}/repos/${REPO}/issues/${N}" -d '{"state":"closed"}' >/dev/null
echo "Closed recovered canary issue #${N}"
echo "Closed recovered smoke issue #${N}"
done
- name: Teardown safety net
@@ -238,24 +245,23 @@ jobs:
run: |
set +e
# Slug prefix matches what test_staging_full_saas.sh emits
# in canary mode:
# SLUG="e2e-canary-$(date +%Y%m%d)-${RUN_ID_SUFFIX}"
# Earlier this was `e2e-{today}-canary-` — that was the
# full-mode pattern (date FIRST, mode SECOND); canary slugs
# have mode FIRST, date SECOND. The mismatch silently
# never matched, leaving every cancelled-canary EC2 alive
# until the once-an-hour sweep eventually caught it
# (incident 2026-04-26 21:03Z: 1h25m EC2 leak before manual
# cleanup; same gap on three earlier cancellations today).
# in smoke mode:
# SLUG="e2e-smoke-$(date +%Y%m%d)-${RUN_ID_SUFFIX}"
# Earlier (pre-2026-05-11 canary→staging rename) the prefix was
# `e2e-canary-`; both prefixes are matched here for one
# release cycle so cleanup still catches any in-flight org
# provisioned under the old prefix on an older runner that
# hasn't picked up the renamed script. Remove the canary
# fallback after one week of no-old-prefix observations.
orgs=$(curl -sS "$MOLECULE_CP_URL/cp/admin/orgs" \
-H "Authorization: Bearer $ADMIN_TOKEN" 2>/dev/null \
| python3 -c "
import json, sys, os, datetime
run_id = os.environ.get('GITHUB_RUN_ID', '')
d = json.load(sys.stdin)
# Scope to slugs from THIS canary run when GITHUB_RUN_ID is
# available; the canary workflow sets E2E_RUN_ID='canary-\${run_id}'
# so the slug suffix is '-canary-\${run_id}-...'. Mirrors the
# Scope to slugs from THIS smoke run when GITHUB_RUN_ID is
# available; the smoke workflow sets E2E_RUN_ID='smoke-\${run_id}'
# so the slug suffix is '-smoke-\${run_id}-...'. Mirrors the
# full-mode safety net's per-run scoping (e2e-staging-saas.yml)
# added after the 2026-04-21 cross-run cleanup incident.
# Sweep both today AND yesterday's UTC dates so a run that
@@ -265,9 +271,11 @@ jobs:
yesterday = today - datetime.timedelta(days=1)
dates = (today.strftime('%Y%m%d'), yesterday.strftime('%Y%m%d'))
if run_id:
prefixes = tuple(f'e2e-canary-{d}-canary-{run_id}' for d in dates)
prefixes = tuple(f'e2e-smoke-{d}-smoke-{run_id}' for d in dates) \
+ tuple(f'e2e-canary-{d}-canary-{run_id}' for d in dates)
else:
prefixes = tuple(f'e2e-canary-{d}-' for d in dates)
prefixes = tuple(f'e2e-smoke-{d}-' for d in dates) \
+ tuple(f'e2e-canary-{d}-' for d in dates)
candidates = [o['slug'] for o in d.get('orgs', [])
if any(o.get('slug','').startswith(p) for p in prefixes)
and o.get('status') not in ('purged',)]
@@ -280,8 +288,8 @@ jobs:
# stale sweep caught it (up to 2h later). Now we capture the
# response code and surface non-2xx as a workflow warning, so
# the run page shows which slug leaked. We still don't `exit 1`
# on cleanup failure — a single-canary cleanup miss shouldn't
# fail-flag the canary itself when the actual smoke check
# on cleanup failure — a single-smoke cleanup miss shouldn't
# fail-flag the smoke itself when the actual smoke check
# passed. The sweep-stale-e2e-orgs cron (now every 15 min,
# 30-min threshold) is the safety net for whatever slips past.
# See molecule-controlplane#420.
@@ -290,21 +298,21 @@ jobs:
# Tempfile-routed -w + set +e/-e prevents curl-exit-code
# pollution of the captured status (lint-curl-status-capture.yml).
set +e
curl -sS -o /tmp/canary-cleanup.out -w "%{http_code}" \
curl -sS -o /tmp/smoke-cleanup.out -w "%{http_code}" \
-X DELETE "$MOLECULE_CP_URL/cp/admin/tenants/$slug" \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"confirm\":\"$slug\"}" >/tmp/canary-cleanup.code
-d "{\"confirm\":\"$slug\"}" >/tmp/smoke-cleanup.code
set -e
code=$(cat /tmp/canary-cleanup.code 2>/dev/null || echo "000")
code=$(cat /tmp/smoke-cleanup.code 2>/dev/null || echo "000")
if [ "$code" = "200" ] || [ "$code" = "204" ]; then
echo "[teardown] deleted $slug (HTTP $code)"
else
echo "::warning::canary teardown for $slug returned HTTP $code — sweep-stale-e2e-orgs will catch it within ~45 min. Body: $(head -c 300 /tmp/canary-cleanup.out 2>/dev/null)"
echo "::warning::smoke teardown for $slug returned HTTP $code — sweep-stale-e2e-orgs will catch it within ~45 min. Body: $(head -c 300 /tmp/smoke-cleanup.out 2>/dev/null)"
leaks+=("$slug")
fi
done
if [ ${#leaks[@]} -gt 0 ]; then
echo "::warning::canary teardown left ${#leaks[@]} leak(s): ${leaks[*]}"
echo "::warning::smoke teardown left ${#leaks[@]} leak(s): ${leaks[*]}"
fi
exit 0
@@ -1,6 +1,8 @@
name: canary-verify
name: Staging verify
# Ported from .github/workflows/canary-verify.yml on 2026-05-11 per RFC
# Renamed from canary-verify.yml on 2026-05-11 per Hongming directive
# ("canary naming changed to staging for all"). Originally ported from
# .github/workflows/canary-verify.yml on 2026-05-11 per RFC
# internal#219 §1 sweep. Differences from the GitHub version:
# - Dropped `workflow_dispatch.inputs` (Gitea 1.22.6 parser rejects them
# per feedback_gitea_workflow_dispatch_inputs_unsupported).
@@ -23,13 +25,22 @@ name: canary-verify
# digest. On red, :latest stays on the prior known-good digest and
# prod is untouched.
#
# Terminology note (2026-05-11): The deployment STRATEGY here is still
# called "canary release" (a small subset of tenants gets the new image
# first, the rest follow on green). The "canary" word stays for the
# pre-fan-out cohort concept (see docs/architecture/canary-release.md
# and CANARY_SLUG in redeploy-tenants-on-*.yml). What changed is the
# FILE NAME and the SECRETS feeding this workflow — both are renamed
# to drop the redundant "canary-" prefix that conflated workflow
# identity with deployment strategy.
#
# Registry note (2026-05-10): This workflow previously used GHCR
# (ghcr.io/molecule-ai/platform-tenant) — that registry was retired
# during the 2026-05-06 Gitea suspension migration when publish-
# workspace-server-image.yml switched to the operator's ECR org
# (153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/
# platform-tenant). The GHCR → ECR migration was never applied to
# this file, so canary-verify was silently smoke-testing the stale
# this file, so this workflow was silently smoke-testing the stale
# GHCR image while the actual staging/prod tenants ran the ECR image.
# Result: smoke tests could not catch a broken ECR build. Fix:
# - Wait step: reads SHA from running canary /health (tenant-
@@ -43,8 +54,9 @@ name: canary-verify
# to ECR on staging and main merges.
# - Canary tenants are configured to pull :staging-<sha> from ECR
# (TENANT_IMAGE env set to the ECR :staging-<sha> tag).
# - Repo secrets CANARY_TENANT_URLS / CANARY_ADMIN_TOKENS /
# CANARY_CP_SHARED_SECRET are populated.
# - Repo secrets MOLECULE_STAGING_TENANT_URLS /
# MOLECULE_STAGING_ADMIN_TOKENS / MOLECULE_STAGING_CP_SHARED_SECRET
# are populated.
on:
workflow_run:
@@ -65,7 +77,7 @@ env:
GITHUB_SERVER_URL: https://git.moleculesai.app
jobs:
canary-smoke:
staging-smoke:
# Skip when the upstream workflow failed — no image to test against.
# workflow_dispatch trigger dropped in this Gitea port; only the
# workflow_run path remains.
@@ -97,15 +109,15 @@ jobs:
# other registry — the canary is telling us what it's actually
# running, which is the ground truth for smoke testing.
env:
CANARY_TENANT_URLS: ${{ secrets.CANARY_TENANT_URLS }}
MOLECULE_STAGING_TENANT_URLS: ${{ secrets.MOLECULE_STAGING_TENANT_URLS }}
EXPECTED_SHA: ${{ steps.compute.outputs.sha }}
run: |
if [ -z "$CANARY_TENANT_URLS" ]; then
if [ -z "$MOLECULE_STAGING_TENANT_URLS" ]; then
echo "No canary URLs configured — falling back to 60s wait"
sleep 60
exit 0
fi
IFS=',' read -ra URLS <<< "$CANARY_TENANT_URLS"
IFS=',' read -ra URLS <<< "$MOLECULE_STAGING_TENANT_URLS"
MAX_WAIT=420 # 7 minutes
INTERVAL=30
ELAPSED=0
@@ -129,7 +141,7 @@ jobs:
done
echo "Timeout after ${MAX_WAIT}s — proceeding anyway (smoke suite will validate)"
- name: Run canary smoke suite
- name: Run staging smoke suite
id: smoke
# Graceful-skip when no canary fleet is configured (Phase 2 not yet
# stood up — see molecule-controlplane/docs/canary-tenants.md).
@@ -138,29 +150,29 @@ jobs:
# promote-latest.yml is the release gate while canary is absent.
# Once the fleet is real: delete the early-exit branch.
env:
CANARY_TENANT_URLS: ${{ secrets.CANARY_TENANT_URLS }}
CANARY_ADMIN_TOKENS: ${{ secrets.CANARY_ADMIN_TOKENS }}
CANARY_CP_BASE_URL: https://staging-api.moleculesai.app
CANARY_CP_SHARED_SECRET: ${{ secrets.CANARY_CP_SHARED_SECRET }}
MOLECULE_STAGING_TENANT_URLS: ${{ secrets.MOLECULE_STAGING_TENANT_URLS }}
MOLECULE_STAGING_ADMIN_TOKENS: ${{ secrets.MOLECULE_STAGING_ADMIN_TOKENS }}
MOLECULE_STAGING_CP_BASE_URL: https://staging-api.moleculesai.app
MOLECULE_STAGING_CP_SHARED_SECRET: ${{ secrets.MOLECULE_STAGING_CP_SHARED_SECRET }}
run: |
set -euo pipefail
if [ -z "${CANARY_TENANT_URLS:-}" ] \
|| [ -z "${CANARY_ADMIN_TOKENS:-}" ] \
|| [ -z "${CANARY_CP_SHARED_SECRET:-}" ]; then
if [ -z "${MOLECULE_STAGING_TENANT_URLS:-}" ] \
|| [ -z "${MOLECULE_STAGING_ADMIN_TOKENS:-}" ] \
|| [ -z "${MOLECULE_STAGING_CP_SHARED_SECRET:-}" ]; then
{
echo "## ⚠️ canary-verify skipped"
echo "## ⚠️ staging-verify skipped"
echo
echo "One or more canary secrets are unset (\`CANARY_TENANT_URLS\`, \`CANARY_ADMIN_TOKENS\`, \`CANARY_CP_SHARED_SECRET\`)."
echo "One or more canary secrets are unset (\`MOLECULE_STAGING_TENANT_URLS\`, \`MOLECULE_STAGING_ADMIN_TOKENS\`, \`MOLECULE_STAGING_CP_SHARED_SECRET\`)."
echo "Phase 2 canary fleet has not been stood up yet —"
echo "see [canary-tenants.md](https://git.moleculesai.app/molecule-ai/molecule-controlplane/blob/main/docs/canary-tenants.md)."
echo
echo "**Skipped — promote-to-latest will NOT auto-fire.** Dispatch \`promote-latest.yml\` manually when ready."
} >> "$GITHUB_STEP_SUMMARY"
echo "ran=false" >> "$GITHUB_OUTPUT"
echo "::notice::canary-verify: skipped — no canary fleet configured"
echo "::notice::staging-verify: skipped — no canary fleet configured"
exit 0
fi
bash scripts/canary-smoke.sh
bash scripts/staging-smoke.sh
echo "ran=true" >> "$GITHUB_OUTPUT"
- name: Summary on failure
@@ -173,7 +185,7 @@ jobs:
echo ":latest stays pinned to the prior good digest — prod is untouched."
echo
echo "Fix forward and merge again, or investigate the specific failed"
echo "assertions in the canary-smoke step log above."
echo "assertions in the staging-smoke step log above."
} >> "$GITHUB_STEP_SUMMARY"
promote-to-latest:
@@ -188,13 +200,13 @@ jobs:
# silently promoting a stale GHCR image while actual prod tenants
# pulled from ECR. Canary smoke tests were GHCR-targeted and could
# not catch a broken ECR build.
needs: canary-smoke
if: ${{ needs.canary-smoke.result == 'success' && needs.canary-smoke.outputs.smoke_ran == 'true' }}
needs: staging-smoke
if: ${{ needs.staging-smoke.result == 'success' && needs.staging-smoke.outputs.smoke_ran == 'true' }}
runs-on: ubuntu-latest
# Phase 3 (RFC #219 §1): surface broken workflows without blocking.
continue-on-error: true
env:
SHA: ${{ needs.canary-smoke.outputs.sha }}
SHA: ${{ needs.staging-smoke.outputs.sha }}
CP_URL: ${{ vars.CP_URL || 'https://staging-api.moleculesai.app' }}
# CP_ADMIN_API_TOKEN gates write access to the redeploy endpoint.
# Stored at the repo level so all workflows pick it up automatically.
@@ -264,9 +276,9 @@ jobs:
- name: Summary
run: |
{
echo "## Canary verified — :latest promoted via CP redeploy-fleet"
echo "## Staging verified — :latest promoted via CP redeploy-fleet"
echo ""
echo "- **Target tag:** \`staging-${{ needs.canary-smoke.outputs.sha }}\`"
echo "- **Target tag:** \`staging-${{ needs.staging-smoke.outputs.sha }}\`"
echo "- **Registry:** ECR (\`${TENANT_IMAGE_NAME}\`)"
echo "- **Canary slug:** \`${CANARY_SLUG:-<none>}\` (soak ${SOAK_SECONDS}s)"
echo "- **Batch size:** ${BATCH_SIZE:-3}"
+2 -1
View File
@@ -99,7 +99,8 @@ jobs:
# Filter:
# 1. slug starts with one of the ephemeral test prefixes:
# - 'e2e-' — covers e2e-canary-, e2e-canvas-*, etc.
# - 'e2e-' — covers e2e-smoke- (formerly e2e-canary-),
# e2e-canvas-*, etc.
# - 'rt-e2e-' — runtime-test harness fixtures (RFC #2251);
# missing this prefix left two such tenants
# orphaned 8h on staging (2026-05-03), then
+5 -5
View File
@@ -2,7 +2,7 @@
How a workspace-server code change reaches the prod tenant fleet — and how to stop it if something's wrong.
> **⚠️ State note (2026-04-22):** this doc describes the **intended design**. As of this write, the canary fleet described below is **not actually running** — no canary tenants are provisioned, `CANARY_TENANT_URLS` / `CANARY_ADMIN_TOKENS` / `CANARY_CP_SHARED_SECRET` are empty in repo secrets, and `canary-verify.yml` fails every run.
> **⚠️ State note (2026-04-22, secret names refreshed 2026-05-11):** this doc describes the **intended design**. As of this write, the canary fleet described below is **not actually running** — no canary tenants are provisioned, `MOLECULE_STAGING_TENANT_URLS` / `MOLECULE_STAGING_ADMIN_TOKENS` / `MOLECULE_STAGING_CP_SHARED_SECRET` are empty in repo secrets, and `staging-verify.yml` (formerly `canary-verify.yml`) fails every run.
>
> Current merges gate on manual `promote-latest.yml` dispatches, not canary. See [molecule-controlplane/docs/canary-tenants.md](https://git.moleculesai.app/molecule-ai/molecule-controlplane/src/branch/main/docs/canary-tenants.md) for the Phase 1 code work that's already shipped + the Phase 2 plan for actually standing up the fleet + a "should we even do this now?" decision framework.
>
@@ -22,7 +22,7 @@ publish-workspace-server-image.yml ← pushes :staging-<sha> ONLY
Canary tenants auto-update to :staging-<sha>
│ (5-min auto-updater cycle on each canary EC2)
canary-verify.yml waits 6 min, runs scripts/canary-smoke.sh
staging-verify.yml waits 6 min, runs scripts/staging-smoke.sh
├─► GREEN → crane tag :staging-<sha> → :latest
│ │
@@ -42,7 +42,7 @@ Canary tenants are configured to pull `:staging-<sha>` (not `:latest`) via `TENA
## Smoke suite
`scripts/canary-smoke.sh` hits each canary tenant (URL + ADMIN_TOKEN pair) and asserts:
`scripts/staging-smoke.sh` hits each canary tenant (URL + ADMIN_TOKEN pair) and asserts:
- `/admin/liveness` returns a subsystems map (tenant booted, AdminAuth reachable)
- `/workspaces` returns a JSON array (wsAuth + DB healthy)
@@ -59,8 +59,8 @@ Expand by editing the script — each `check "name" "expected" "$response"` call
3. Re-trigger provision (or delete + recreate if the org was already provisioned into staging) — the fresh EC2 lands in the canary AWS account (see internal runbook for the specific ID)
Then set repo secrets:
- `CANARY_TENANT_URLS` — append the new tenant's URL
- `CANARY_ADMIN_TOKENS` — append its ADMIN_TOKEN in the same position
- `MOLECULE_STAGING_TENANT_URLS` — append the new tenant's URL
- `MOLECULE_STAGING_ADMIN_TOKENS` — append its ADMIN_TOKEN in the same position
## Rolling back `:latest`
@@ -50,7 +50,7 @@ pipeline.
| `check-merge-group-trigger.yml` | The workflow's own header (lines 18-23) documents that it's vacuously satisfied on Gitea — Gitea has no merge queue, no `merge_group:` event type, no `gh-readonly-queue/...` refs. Nothing to lint. |
| `codeql.yml` | The workflow's own header (lines 3-67) documents that `github/codeql-action/init@v4` hits api.github.com bundle endpoints not implemented by Gitea (observed: `::error::404 page not found` in Initialize CodeQL step). Per Hongming decision 2026-05-07 (task #156): CodeQL is ADVISORY/non-blocking until a Gitea-compatible SAST pipeline lands. Replacement options (Semgrep self-host, Sonatype, GitHub-mirror-for-SAST) tracked in #156. |
| `pr-guards.yml` | The workflow's own header documents that Gitea has no `gh pr merge --auto` primitive — the guard is a structural no-op on Gitea. Branch protection on `main` does NOT reference any `pr-guards` check name; deletion is safe. |
| `promote-latest.yml` | Uses `imjasonh/setup-crane` against `ghcr.io/molecule-ai/platform` — the GHCR registry was retired during the 2026-05-06 Gitea migration (per `canary-verify.yml` header notes, the canonical tenant image moved to ECR `153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/platform-tenant`). The workflow can no longer find any image to retag. Follow-up issue suggested if an ECR-based retag promote is desired. |
| `promote-latest.yml` | Uses `imjasonh/setup-crane` against `ghcr.io/molecule-ai/platform` — the GHCR registry was retired during the 2026-05-06 Gitea migration (per `staging-verify.yml` header notes — file was renamed from `canary-verify.yml` on 2026-05-11; the canonical tenant image moved to ECR `153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/platform-tenant`). The workflow can no longer find any image to retag. Follow-up issue suggested if an ECR-based retag promote is desired. |
## Category C — ported to .gitea/
+1 -1
View File
@@ -43,7 +43,7 @@ endpoint handler for the supported range.
- `cleanup-rogue-workspaces.sh` — emergency teardown for leaked
workspaces. Prompts for confirmation. Pair with the harnesses if a
cleanup trap fails (see `cleanup_*_failed` events).
- `canary-smoke.sh` — quick smoke test for canary releases.
- `staging-smoke.sh` — quick smoke test for the staging canary fleet (formerly `canary-smoke.sh`).
- `dev-start.sh` — local-dev platform bring-up.
The rest are self-documenting in their header comments.
@@ -1,29 +1,40 @@
#!/bin/bash
# canary-smoke.sh — runs the post-deploy smoke suite against the
# staging canary tenant fleet. Called by the canary-verify.yml GitHub
# staging-smoke.sh — runs the post-deploy smoke suite against the
# staging canary tenant fleet. Called by the staging-verify.yml Gitea
# Actions workflow after a new workspace-server image lands in ECR;
# exits non-zero on any failure so the workflow can block the
# redeploy-fleet promotion that would otherwise release broken code
# to the prod tenant fleet.
#
# Naming note (2026-05-11): The script (and its input env vars) were
# renamed from canary-smoke.sh / CANARY_* to staging-smoke.sh /
# MOLECULE_STAGING_* per Hongming directive. The tested COHORT is still
# called the "canary fleet" (a small subset of staging tenants that
# ingest :staging-<sha> before the rest of the fleet); that strategy
# concept is unchanged.
#
# Registry note: GHCR was retired 2026-05-06. Images are now pushed
# to the operator's ECR org (153263036946.dkr.ecr.us-east-2.amazonaws.com/
# molecule-ai/platform-tenant). The registry URL is a runtime concern for
# the CI push step; this script tests the running tenant directly.
#
# Environment:
# CANARY_TENANT_URLS space-sep list of canary tenant base URLs
# (e.g. "https://canary-pm.staging.moleculesai.app
# https://canary-mcp.staging.moleculesai.app")
# CANARY_ADMIN_TOKENS space-sep list of ADMIN_TOKENs, positionally
# matched to CANARY_TENANT_URLS. Canary tenants
# are provisioned with known ADMIN_TOKENs so CI
# can hit their admin-gated endpoints.
# CANARY_CP_BASE_URL CP base URL the canaries call back to
# (https://staging-api.moleculesai.app)
# CANARY_CP_SHARED_SECRET matches CP's PROVISION_SHARED_SECRET so this
# script can also exercise /cp/workspaces/* via
# the canary's own CPProvisioner identity.
# MOLECULE_STAGING_TENANT_URLS space-sep list of canary tenant base
# URLs (e.g. "https://canary-pm.staging.
# moleculesai.app https://canary-mcp.
# staging.moleculesai.app")
# MOLECULE_STAGING_ADMIN_TOKENS space-sep list of ADMIN_TOKENs,
# positionally matched to
# MOLECULE_STAGING_TENANT_URLS.
# Canary tenants are provisioned with
# known ADMIN_TOKENs so CI can hit
# their admin-gated endpoints.
# MOLECULE_STAGING_CP_BASE_URL CP base URL the canaries call back to
# (https://staging-api.moleculesai.app)
# MOLECULE_STAGING_CP_SHARED_SECRET matches CP's PROVISION_SHARED_SECRET
# so this script can also exercise
# /cp/workspaces/* via the canary's
# own CPProvisioner identity.
#
# Exit codes: 0 = all green, 1 = assertion failure, 2 = setup/env problem.
@@ -31,12 +42,12 @@ set -euo pipefail
# ── Setup ────────────────────────────────────────────────────────────────
: "${CANARY_TENANT_URLS:?space-sep list of canary base URLs required}"
: "${CANARY_ADMIN_TOKENS:?space-sep list of ADMIN_TOKENs required, same order as URLs}"
: "${CANARY_CP_BASE_URL:?CP base URL required}"
: "${MOLECULE_STAGING_TENANT_URLS:?space-sep list of canary base URLs required}"
: "${MOLECULE_STAGING_ADMIN_TOKENS:?space-sep list of ADMIN_TOKENs required, same order as URLs}"
: "${MOLECULE_STAGING_CP_BASE_URL:?CP base URL required}"
read -r -a URLS <<< "$CANARY_TENANT_URLS"
read -r -a TOKENS <<< "$CANARY_ADMIN_TOKENS"
read -r -a URLS <<< "$MOLECULE_STAGING_TENANT_URLS"
read -r -a TOKENS <<< "$MOLECULE_STAGING_ADMIN_TOKENS"
if [ "${#URLS[@]}" -ne "${#TOKENS[@]}" ]; then
echo "ERROR: URLS(${#URLS[@]}) and TOKENS(${#TOKENS[@]}) length mismatch" >&2
@@ -69,7 +80,7 @@ check() {
# tenant never gets the wrong token.
acurl() {
local base="$1" token="$2"; shift 2
curl -sS --max-time 20 -H "Authorization: Bearer $token" "$@" -- "$base${CANARY_ACURL_PATH:-}"
curl -sS --max-time 20 -H "Authorization: Bearer $token" "$@" -- "$base${ACURL_PATH:-}"
}
# ── Checks (run per canary tenant) ───────────────────────────────────────
@@ -80,7 +91,7 @@ for i in "${!URLS[@]}"; do
printf "\n── %s ──\n" "$base"
# 1. Liveness — the tenant is up and responding to admin auth.
CANARY_ACURL_PATH="/admin/liveness" resp=$(acurl "$base" "$token" || true)
ACURL_PATH="/admin/liveness" resp=$(acurl "$base" "$token" || true)
check "liveness returns a subsystems map" '"subsystems"' "$resp"
# 2. CP env refresh — the workspace-server fetched MOLECULE_CP_SHARED_SECRET
@@ -89,25 +100,25 @@ for i in "${!URLS[@]}"; do
# booted without crashing on the refresh call. A startup failure in
# refreshEnvFromCP logs but still boots (best-effort semantics), so
# this is a sanity check, not a proof.
CANARY_ACURL_PATH="/workspaces" resp=$(acurl "$base" "$token" || true)
ACURL_PATH="/workspaces" resp=$(acurl "$base" "$token" || true)
check "workspace list is JSON array" "[" "$resp"
# 3. Memory commit round-trip — scope=LOCAL so test data stays on this
# tenant. Verifies encryption + scrubber + retrieval end-to-end.
probe_id="canary-smoke-$(date +%s)-$i"
body=$(printf '{"scope":"LOCAL","namespace":"canary-smoke","content":"probe-%s"}' "$probe_id")
CANARY_ACURL_PATH="/memories/commit" resp=$(curl -sS --max-time 20 \
ACURL_PATH="/memories/commit" resp=$(curl -sS --max-time 20 \
-X POST -H "Content-Type: application/json" -H "Authorization: Bearer $token" \
--data "$body" "$base/memories/commit" || true)
check "memory commit accepted" '"id"' "$resp"
CANARY_ACURL_PATH="/memories/search?query=probe-${probe_id}" \
ACURL_PATH="/memories/search?query=probe-${probe_id}" \
resp=$(curl -sS --max-time 20 -H "Authorization: Bearer $token" \
"$base/memories/search?query=probe-${probe_id}" || true)
check "memory search finds the probe" "probe-${probe_id}" "$resp"
# 4. Events admin read — AdminAuth path (C4 fail-closed proof on SaaS).
CANARY_ACURL_PATH="/events" resp=$(acurl "$base" "$token" || true)
ACURL_PATH="/events" resp=$(acurl "$base" "$token" || true)
check "events endpoint returns JSON" "[" "$resp"
# 5. Negative: unauth'd admin call must 401 (C4 regression gate).
@@ -117,7 +128,7 @@ for i in "${!URLS[@]}"; do
# 6. POST /org/import unauth → 401. Proves the route is compiled in
# and AdminAuth is enforced. A missing route returns 404 (the failure
# mode caught by issue #213). Regression guard for the silent-GHCR-
# migration gap: canary-verify was testing a stale GHCR image while
# migration gap: staging-verify (formerly canary-verify) was testing a stale GHCR image while
# actual tenants ran ECR — this test would have caught a missing-route
# binary before it reached prod.
unauth_code=$(curl -sS -o /dev/null -w '%{http_code}' \
+3 -3
View File
@@ -7,11 +7,11 @@ Four workflows + a shared bash harness that together cover the SaaS stack end to
| Workflow | Cadence | Wall time | Scope |
|---|---|---|---|
| `e2e-staging-saas.yml` | push + nightly 07:00 UTC | ~20 min | Full API: org → tenant → 2 workspaces → A2A → HMA → delegation → leak check |
| `canary-staging.yml` | every 30 min | ~8 min | Minimum smoke + self-managed alert issue |
| `staging-smoke.yml` | every 30 min | ~8 min | Minimum smoke + self-managed alert issue |
| `e2e-staging-canvas.yml` | push + weekly Sunday 08:00 | ~25 min | All 13 canvas workspace-panel tabs via Playwright |
| `e2e-staging-sanity.yml` | weekly Monday 06:00 | ~10 min | Intentional-failure: teardown safety-net self-check |
`tests/e2e/test_staging_full_saas.sh` is the shared harness all workflows invoke (with `E2E_MODE={full|canary}` and `E2E_INTENTIONAL_FAILURE={0|1}` toggles).
`tests/e2e/test_staging_full_saas.sh` is the shared harness all workflows invoke (with `E2E_MODE={full|smoke}` and `E2E_INTENTIONAL_FAILURE={0|1}` toggles).
### Full-SaaS checklist (sections)
@@ -82,7 +82,7 @@ bash tests/e2e/test_staging_full_saas.sh
## Cost
- Full run: ~20 min, ~$0.007
- Canary (48/day): ~$0.06/day
- Smoke (48/day): ~$0.06/day
- Canvas (few/week): ~$0.01/day
- Sanity (weekly): ~$0.002/week
- **Total staging burn: < $0.15/day** at expected CI load
+18 -6
View File
@@ -27,7 +27,11 @@
# E2E_PROVISION_TIMEOUT_SECS default 900 (15 min cold EC2 budget)
# E2E_KEEP_ORG 1 → skip teardown (debugging only)
# E2E_RUN_ID Slug suffix; CI: ${GITHUB_RUN_ID}
# E2E_MODE full (default) | canary
# E2E_MODE full (default) | smoke
# (legacy alias `canary` still accepted —
# mapped to `smoke` for back-compat with
# any in-flight runner picking up an older
# workflow checkout)
# E2E_INTENTIONAL_FAILURE 1 → poison tenant token mid-run so the
# script fails; the EXIT trap MUST still
# tear down cleanly (and exit 4 on leak).
@@ -49,15 +53,23 @@ RUNTIME="${E2E_RUNTIME:-hermes}"
PROVISION_TIMEOUT_SECS="${E2E_PROVISION_TIMEOUT_SECS:-900}"
RUN_ID_SUFFIX="${E2E_RUN_ID:-$(date +%H%M%S)-$$}"
MODE="${E2E_MODE:-full}"
# `canary` is a legacy alias for `smoke` retained for back-compat with
# any in-flight runner picking up an older workflow checkout during the
# 2026-05-11 canary→staging rename rollout. Both map to the same slug
# prefix below. Remove the `canary` alias after one week of no-old-mode
# observations.
if [ "$MODE" = "canary" ]; then
MODE="smoke"
fi
case "$MODE" in
full|canary) ;;
*) echo "E2E_MODE must be 'full' or 'canary' (got: $MODE)" >&2; exit 2 ;;
full|smoke) ;;
*) echo "E2E_MODE must be 'full' or 'smoke' (got: $MODE)" >&2; exit 2 ;;
esac
# Canary runs get a distinct prefix so their safety-net sweeper only
# Smoke runs get a distinct slug prefix so their safety-net sweeper only
# touches their own runs, not in-flight full runs.
if [ "$MODE" = "canary" ]; then
SLUG="e2e-canary-$(date +%Y%m%d)-${RUN_ID_SUFFIX}"
if [ "$MODE" = "smoke" ]; then
SLUG="e2e-smoke-$(date +%Y%m%d)-${RUN_ID_SUFFIX}"
else
SLUG="e2e-$(date +%Y%m%d)-${RUN_ID_SUFFIX}"
fi
+30 -14
View File
@@ -48,6 +48,27 @@ def get_machine_ip() -> str: # pragma: no cover
return "127.0.0.1"
def _check_delegation_results_pending() -> bool:
"""Check if there are unconsumed delegation results waiting.
Reads ``DELEGATION_RESULTS_FILE``. Returns ``True`` if the file
exists and contains non-whitespace content (after stripping) — meaning
the idle loop should skip this tick. Returns ``False`` if the file is
absent, empty, or contains only whitespace.
The extracted form lets unit tests call this directly rather than mirroring
the logic (anti-pattern flagged as #401).
"""
from heartbeat import DELEGATION_RESULTS_FILE
try:
with open(DELEGATION_RESULTS_FILE) as rf:
rf.seek(0)
return bool(rf.read().strip())
except FileNotFoundError:
return False
# Re-exported from transcript_auth for the inline /transcript handler.
# Separate module keeps the security-critical gate import-light + unit-testable.
from transcript_auth import transcript_authorized as _transcript_authorized
@@ -678,20 +699,15 @@ async def main(): # pragma: no cover
# heartbeat's own self-message wake the agent after results are
# written. The agent then sees the results in _prepare_prompt()
# and processes them before composing.
from heartbeat import DELEGATION_RESULTS_FILE as _DRF
try:
with open(_DRF) as _rf:
_rf.seek(0)
_content = _rf.read().strip()
if _content:
print(
f"Idle loop: skipping — {len(_content)} bytes of unconsumed "
f"delegation results pending (heartbeat will notify agent)",
flush=True,
)
continue
except FileNotFoundError:
pass # No results file — normal, proceed with idle prompt
# Guard logic extracted to _check_delegation_results_pending() for
# direct unit-testing (#401 follow-up).
if _check_delegation_results_pending():
print(
"Idle loop: skipping — unconsumed delegation results pending "
"(heartbeat will notify agent)",
flush=True,
)
continue
# Self-post the idle prompt via the platform A2A proxy (same
# path as initial_prompt). The agent's own concurrency control
+50 -45
View File
@@ -4,77 +4,82 @@ The idle loop skips sending the idle prompt when DELEGATION_RESULTS_FILE
contains unconsumed results, preventing the agent from composing a stale tick
before processing pending delegation notifications from the heartbeat.
Source: workspace/main.py:_run_idle_loop() pending-results guard.
Source: ``workspace/main.py:_check_delegation_results_pending()`` (extracted from
``_run_idle_loop()`` guard; see PR #432 follow-up).
The guard is extracted into a module-level function so unit tests call the
real production logic directly — not a mirror copy. This avoids the
test-mirror anti-pattern (issue #401) where a copied implementation
drifts from the production code it is supposed to test.
"""
from __future__ import annotations
import io
import json
from unittest.mock import patch
import pytest
def check_results_pending(file_path: str) -> bool:
"""Mirror the guard logic from workspace/main.py:_run_idle_loop().
Returns True if the results file exists and is non-empty,
meaning the idle loop should skip this tick.
"""
try:
with open(file_path) as rf:
rf.seek(0)
content = rf.read().strip()
return bool(content)
except FileNotFoundError:
return False
from main import _check_delegation_results_pending
class TestIdleLoopPendingCheck:
"""Tests for the idle-loop pending-delegation-results guard."""
"""Tests for the idle-loop pending-delegation-results guard.
def test_no_file_means_proceed(self, tmp_path):
Each test patches ``builtins.open`` so ``_check_delegation_results_pending``
reads the controlled payload instead of the real DELEGATION_RESULTS_FILE.
No filesystem side-effects.
"""
def _patch_open(self, payload: str | None):
"""Patch builtins.open for _check_delegation_results_pending.
Args:
payload: file contents to return. None → FileNotFoundError.
"""
if payload is None:
return patch("builtins.open", side_effect=FileNotFoundError)
else:
fake_file = io.StringIO(payload)
return patch("builtins.open", return_value=fake_file)
def test_no_file_means_proceed(self):
"""No delegation results file → idle loop fires normally."""
results_file = tmp_path / "delegation_results.jsonl"
assert not check_results_pending(str(results_file))
with self._patch_open(None):
assert _check_delegation_results_pending() is False
def test_empty_file_means_proceed(self, tmp_path):
def test_empty_file_means_proceed(self):
"""Empty file → no pending results → idle loop fires."""
results_file = tmp_path / "delegation_results.jsonl"
results_file.write_text("", encoding="utf-8")
assert not check_results_pending(str(results_file))
with self._patch_open(""):
assert _check_delegation_results_pending() is False
def test_whitespace_only_file_means_proceed(self, tmp_path):
def test_whitespace_only_file_means_proceed(self):
"""File with only whitespace → treated as empty → idle loop fires."""
results_file = tmp_path / "delegation_results.jsonl"
results_file.write_text(" \n ", encoding="utf-8")
assert not check_results_pending(str(results_file))
with self._patch_open(" \n "):
assert _check_delegation_results_pending() is False
def test_single_result_means_skip(self, tmp_path):
def test_single_result_means_skip(self):
"""File with one delegation result → skip idle tick."""
results_file = tmp_path / "delegation_results.jsonl"
results_file.write_text(
payload = (
json.dumps({
"status": "completed",
"delegation_id": "del-abc",
"summary": "Done",
}) + "\n",
encoding="utf-8",
}) + "\n"
)
assert check_results_pending(str(results_file))
with self._patch_open(payload):
assert _check_delegation_results_pending() is True
def test_multiple_results_means_skip(self, tmp_path):
def test_multiple_results_means_skip(self):
"""File with multiple delegation results → skip idle tick."""
results_file = tmp_path / "delegation_results.jsonl"
results_file.write_text(
payload = (
json.dumps({"status": "completed", "delegation_id": "del-1", "summary": "A"})
+ "\n"
+ json.dumps({"status": "failed", "delegation_id": "del-2", "summary": "B"})
+ "\n",
encoding="utf-8",
+ "\n"
)
assert check_results_pending(str(results_file))
with self._patch_open(payload):
assert _check_delegation_results_pending() is True
def test_file_with_only_newline_means_proceed(self, tmp_path):
def test_file_with_only_newline_means_proceed(self):
"""File with only a newline character → stripped to empty → fires."""
results_file = tmp_path / "delegation_results.jsonl"
results_file.write_text("\n", encoding="utf-8")
assert not check_results_pending(str(results_file))
with self._patch_open("\n"):
assert _check_delegation_results_pending() is False