Compare commits

...

2 Commits

Author SHA1 Message Date
claude-ceo-assistant 4c54b59099 Merge pull request 'fix(ci)(interim): disable status-reaper + main-red-watchdog crons (machinery-down)' (#645) from infra/interim-disable-reaper-watchdog-crons into main
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 3s
Block internal-flavored paths / Block forbidden paths (push) Successful in 9s
CI / all-required (push) Successful in 1s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 9s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 9s
Handlers Postgres Integration / detect-changes (push) Successful in 17s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 17s
E2E API Smoke Test / detect-changes (push) Successful in 18s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 13s
CI / Detect changes (push) Successful in 19s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 3s
CI / Platform (Go) (push) Successful in 2s
CI / Canvas (Next.js) (push) Successful in 2s
CI / Shellcheck (E2E scripts) (push) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 2s
CI / Python Lint & Test (push) Successful in 2s
CI / Canvas Deploy Reminder (push) Has been skipped
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 2s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 3s
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 9s
qa-review / approved (pull_request) Failing after 12s
CI / Detect changes (pull_request) Successful in 14s
security-review / approved (pull_request) Failing after 12s
sop-tier-check / tier-check (pull_request) Successful in 14s
E2E API Smoke Test / detect-changes (pull_request) Successful in 15s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 15s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 17s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 17s
gate-check-v3 / gate-check (pull_request) Successful in 16s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 4s
CI / Platform (Go) (pull_request) Successful in 5s
CI / Python Lint & Test (pull_request) Successful in 5s
CI / Canvas (Next.js) (pull_request) Successful in 6s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 5s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 7s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 7s
CI / all-required (pull_request) Successful in 2s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
gate-check-v3 / gate-check (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
2026-05-12 02:45:52 +00:00
claude-ceo-assistant 6ee9ecdf0d fix(ci)(interim): disable status-reaper + main-red-watchdog crons
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 8s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 10s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 11s
qa-review / approved (pull_request) Failing after 12s
security-review / approved (pull_request) Failing after 10s
CI / Detect changes (pull_request) Successful in 17s
E2E API Smoke Test / detect-changes (pull_request) Successful in 19s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 19s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 20s
sop-tier-check / tier-check (pull_request) Successful in 11s
gate-check-v3 / gate-check (pull_request) Successful in 16s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 19s
CI / Platform (Go) (pull_request) Successful in 5s
CI / Canvas (Next.js) (pull_request) Successful in 5s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 4s
CI / Python Lint & Test (pull_request) Successful in 5s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 5s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 5s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 4s
CI / all-required (pull_request) Successful in 2s
audit-force-merge / audit (pull_request) Successful in 10s
RFC#420 Option-C machinery has been down ~2.5h:
- status-reaper rev2 (PR#633, merged 01:48Z): 0 'Compensated by status-reaper'
  status on the last 14 main commits. Schedule reds stranded on stale
  commits despite the rev2 sweep-last-10 design.
- main-red-watchdog: 'Failing after 10m56s' with timeout-minutes:5 — runner
  saturation queue-lag pushed it past its own timeout. No [main-red] issues
  filed during the outage despite 5 reds on HEAD e7965a0f at the high
  watermark.

Both workflows were themselves contributing to the red pileup on main +
queuing the ubuntu-latest pool. Cheap-and-safe interim: comment out the
schedule: blocks. workflow_dispatch: stays so they can be triggered
manually for debugging.

Re-enable after:
1. rev3 lands (likely scan_workflows() should LOG-and-skip rather than
   sys.exit on a malformed workflow; list_recent_commit_shas() should
   degrade gracefully)
2. Dedicated status-ops runner-label (route status-reaper + watchdog +
   ci-required-drift to it so they don't queue behind CI-merge-churn)

Per hongming-pc2 02:31Z directive: 'pick one: rev3+raise-timeout OR
temporarily disable the crons'. Choosing disable for safety while rev3
investigation proceeds.

Reviewed-by: hongming-pc2 (pre-APPROVE on sight 02:31Z)
Author: claude-ceo-assistant (orchestrator emergency; operator-host
unreachable 02:01-02:38Z blocked SSH-bridge to core-devops persona)

Cross-links: task #90 (rev2), task #75 (main-red sweep), RFC#420 Option-C
2026-05-11 19:39:43 -07:00
2 changed files with 17 additions and 11 deletions
+7 -4
View File
@@ -37,10 +37,13 @@ name: main-red-watchdog
# "unknown on type" when `workflow_dispatch.inputs.X` is present. Revisit
# when Gitea ≥ 1.23 is fleet-wide.
on:
schedule:
# Hourly at :05 — task spec calls for "off-zero" (`5 * * * *`),
# offset from :17 (ci-required-drift) and :00 (peak cron load).
- cron: '5 * * * *'
# SCHEDULE DISABLED 2026-05-12 — interim per RFC#420 Option-C machinery-down emergency
# Watchdog timing out behind runner saturation; rev3+dedicated-runner-label in flight
# Re-enable after rev3 lands + runner saturation root resolved
# schedule:
# # Hourly at :05 — task spec calls for "off-zero" (`5 * * * *`),
# # offset from :17 (ci-required-drift) and :00 (peak cron load).
# - cron: '5 * * * *'
workflow_dispatch:
# Read commit status + branch ref + issues; write issues (open/PATCH/close).
+10 -7
View File
@@ -53,13 +53,16 @@ name: status-reaper
# `inputs:` block here. Gitea 1.22.6 rejects the whole workflow as
# "unknown on type" when `workflow_dispatch.inputs.X` is present.
on:
schedule:
# Every 5 minutes. Off-zero alignment with sibling cron workflows:
# ci-required-drift (`:17`), main-red-watchdog (`:05`),
# railway-pin-audit (`:23`). 5-min cadence gives a tight enough
# close on schedule-triggered false-reds that main-red-watchdog
# (hourly :05) almost never files an issue on the false case.
- cron: '*/5 * * * *'
# SCHEDULE DISABLED 2026-05-12 — interim per RFC#420 Option-C machinery-down emergency
# Reaper rev2 not compensating + watchdog timeout-cascade; rev3 in flight
# Re-enable after rev3 lands + runner saturation root resolved
# schedule:
# # Every 5 minutes. Off-zero alignment with sibling cron workflows:
# # ci-required-drift (`:17`), main-red-watchdog (`:05`),
# # railway-pin-audit (`:23`). 5-min cadence gives a tight enough
# # close on schedule-triggered false-reds that main-red-watchdog
# # (hourly :05) almost never files an issue on the false case.
# - cron: '*/5 * * * *'
workflow_dispatch:
# Compensating-status POST needs write on repo statuses; no other