3 Commits

Author	SHA1	Message	Date
core-devops	fae62ac8c1	fix(ci): status-reaper rev3 widens window 10->30 + raises watchdog timeout + re-enables both crons ... Phase 1+2 evidence (rev2 PR#633, merged 01:48Z): 6/6 ticks post-merge with `compensated:0` despite ~25 known-stranded reds visible across those same 10 SHAs on direct probe ~30min later. Reaper run 17057 at 02:46Z explicitly logged: scanned 42 workflows; push-triggered=19, class-O candidates=23 status-reaper summary: {compensated:0, preserved_non_failure:185, scanned_shas:10, limit:10} Root cause: schedule workflows post `failure` to commit-status RETROACTIVELY 5-15 min after their merge. By the time reaper's next */5 tick lands, the stranded red is on a SHA that has already fallen OUTSIDE a 10-commit window during a burst-merge period. Reaper algorithm is correct; the lookback window is too narrow vs. the retroactive-failure-post lag. Three-in-one fix (atomic per hongming-pc2 GO 03:25Z): 1. `.gitea/scripts/status-reaper.py` DEFAULT_SWEEP_LIMIT 10 -> 30. Trades window-width-cheap for cadence-loady; kept `*/5` cron unchanged (avoiding `*/2` which would double runner load). 2. `.gitea/workflows/status-reaper.yml` Restore schedule cron block (revert mc#645 comment-out for THIS workflow only). Cron stays `*/5 * * * *`. 3. `.gitea/workflows/main-red-watchdog.yml` Restore schedule cron block (revert mc#645 comment-out) AND raise job-level `timeout-minutes: 5 -> 15`. Original 5min cap was producing cancels under runner-saturation latency, which fed the very `[main-red]` issues this workflow files (self-poisoning). 4. `tests/test_status_reaper.py` + test_default_sweep_limit_is_30 (contract pin) + test_reap_widened_window_catches_retroactive_failure: mocks 30 SHAs, plants the failing context on SHA[20] (depth strictly past rev2's window=10), asserts the compensation POST lands on that SHA. Existing tests retain explicit `limit=10` overrides and remain unchanged. Suite: 42/42 passed (was 40 + 2 new). Verification plan (post-merge, 10-15 min after merge / 2-3 cron ticks): - DB: SELECT id, status FROM action_run WHERE workflow_id= 'status-reaper.yml' ORDER BY id DESC LIMIT 5 -> all status=1 - Log via web UI: /molecule-ai/molecule-core/actions/runs/<index>/jobs/0/logs -> summary line should now show compensated > 0 with compensated_per_sha populated - Direct probe: pick a SHA in the last 30 main commits with class-O fails, GET /repos/molecule-ai/molecule-core/commits/{sha}/status -> compensated contexts now show state=success with description starting 'Compensated by status-reaper' If rev3 STILL shows compensated:0 after the window-widening, the diagnosis is wrong and a DIFFERENT bug needs to be uncovered (per hongming-pc2 caveat 03:25Z). Re-enabling the crons IS the diagnosis verification. Cross-links: - PR#618 (rev1, drop-concurrency, merge 4db64bcb) - PR#633 (rev2, sweep-recent-commits, merge e7965a0f) - PR#645 (interim disable, merge 4c54b590) — re-enable being reverted - task #90 (orch rev3 tracker) / task #46 (hongming-pc2 tracker) - feedback_brief_hypothesis_vs_evidence (empirical evidence above) - feedback_strict_root_only_after_class_a (3-in-one root fix vs. longer patching chain) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 20:29:06 -07:00
core-devops	98323734ea	feat(ci): status-reaper rev2 sweeps last 10 main commits (closes stranded-status gap) ... rev1 (PR #618, merged 4db64bcb) only inspected the CURRENT main HEAD per tick. Schedule workflows post `failure` to whatever SHA was HEAD when the run COMPLETED, which by the next */5 tick is usually a stale commit because main has already moved forward via merges. Result: rev1 was running successfully but with `compensated:0` on every tick across ~6 cycles (orchestrator + hongming-pc2 Phase 1+2 evidence 23:46Z / 23:59Z / 00:02Z); reds stranded on stale commits. rev2 sweeps the last 10 main commits per tick: - New `list_recent_commit_shas(branch, limit)` wraps GET /repos/{o}/{r}/commits?sha={branch}&limit={limit}. Vendor-truth probe 2026-05-11 confirms Gitea 1.22.6 returns a JSON list of commit objects with `sha` keys (per `feedback_smoke_test_vendor_truth_not_ shape_match`). - New `reap_branch()` orchestrates the sweep: - For each SHA: GET combined status with PER-SHA ERROR ISOLATION (refinement #7) — ApiError on one stale SHA logs `:⚠️:` and continues to the next. Different from the single-HEAD pre-rev2 path where fail-loud was correct; the sweep is best-effort across historical commits. - When `combined.state == "success"`: skip the per-context loop entirely (refinement #2, cost optimization, common case). - Otherwise delegate to the existing per-SHA `reap()` worker (logic UNCHANGED — `_has_push_trigger` / `parse_push_context` / `scan_workflows` not touched per refinement #6). - Aggregated counters preserve all rev1 fields PLUS: - `scanned_shas`: how many SHAs we actually iterated (always 10 in normal operation; less if commits API returns fewer) - `compensated_per_sha`: {<full_sha>: [<context>, ...]} for the SHAs that actually got at least one compensation - `reap()` now also returns `compensated_contexts` so `reap_branch()` can build `compensated_per_sha` without re-deriving it from the POST stream. Backwards-compatible — all existing test assertions check specific counter keys, none enforce a closed dict shape. - `main()` switches from `get_head_sha` + `get_combined_status` + `reap` to a single `reap_branch()` call. Adds `--limit` CLI flag for ops-driven sweep-width tuning (default 10). Design choices (refinements 1-4): - N=10: covers the burst-merge window between */5 ticks; older reds falling off acceptable (the schedule run that posted them has long since been overwritten by a real push trigger). - Skip combined=success early: most commits in the window will be green; short-circuit before the per-context loop saves work. - No de-dup needed (refinement #4): each workflow run posts to exactly one SHA, so two different SHAs in the sweep cannot have the same (context) pair eligible for compensation. Test suite: 37 + 3 = 40/40 cases pass. - New: test_reap_sweeps_n_shas_smoke (mock 3 SHAs, verify each GET'd) - New: test_reap_skips_combined_success_shas (verify the combined=success short-circuit; only the 1 failure SHA is iterated) - New: test_reap_continues_on_per_sha_apierror (per-SHA error isolation contract — ApiError on SHA[0] logged + skipped + SHA[1] processes) - All 37 existing rev1 tests pass unchanged (per-SHA worker logic + the helpers it consumes are untouched). Live dry-run smoke against git.moleculesai.app: scanned 41 workflows; push-triggered=18, class-O candidates=23 summary: {"branch":"main","compensated":0,"compensated_per_sha":{}, "dry_run":true,"limit":10,"preserved_non_failure":196, ...,"scanned_shas":10} Cross-link: - internal#327 (sibling publish-runtime-bot) - task #90 (orchestrator brief), task #46 (hongming-pc2 brief) - PR #618 (parent rev1, merge 4db64bcb) - `reference_post_suspension_pipeline` - `feedback_no_shared_persona_token_use` (commit author = core-devops, not hongming-pc2) - `feedback_strict_root_only_after_class_a` (root cause, not symptom) - `feedback_brief_hypothesis_vs_evidence` (evidence: compensated:0 across 6 cycles) Removal path: drop this workflow when Gitea >= 1.24 ships with a real fix for the hardcoded-suffix bug. Audit issue (filed alongside rev1) tracks the deletion as a follow-up sweep.	2026-05-11 18:41:39 -07:00
core-devops	afaf0a1e54	feat(ci): status-reaper compensates Gitea hardcoded-(push)-suffix on schedule-triggered operational workflow failures ... Root cause (verified via runs 14525 + 14526): Gitea 1.22.6 emits commit-status context as <workflow_name> / <job_name> (push) for ANY workflow run on the default-branch HEAD, REGARDLESS of the trigger event. Schedule- and workflow_dispatch-triggered runs therefore paint main red via a fake-push status. No upstream fix in 1.23-1.26.1 (sibling a6f20db1 research; internal#80 RFC). Design — Option B (b2 cron-based compensating-status POST): workflow_run is NOT supported on Gitea 1.22.6 (verified via modules/actions/workflows.go enumeration); cron is the only event-shaped option that fires reliably. Every 5min, .gitea/workflows/status-reaper.yml runs a stdlib + PyYAML scanner that: 1. Walks .gitea/workflows/*.yml. Resolves each workflow_id from top-level 'name:' (else filename stem). Fails LOUD on name-collision OR '/' in name (would break ' / ' context parsing downstream). Classifies each by 'push:' trigger presence (str / list / dict on: shapes all handled). 2. Reads main HEAD's combined commit status. 3. For each failure-state context ending ' (push)': - parses '<workflow_name> / <job_name> (push)'; - skips if workflow not in scan map (conservative); - preserves if workflow has push: trigger (real defect); - else POSTs state=success with the same context to /repos/{o}/{r}/statuses/{sha}, with a description that documents the workaround. Safety: - Only failure-state contexts whose suffix is ' (push)' are compensated. Branch_protections required checks on main (Secret scan, sop-tier-check) have ' (pull_request)' suffix — UNREACHABLE from this code path. Verified 2026-05-11 + test test_reap_required_check_pull_request_suffix_never_touched. - publish-workspace-server-image has a real push: trigger → PRESERVED. mc#576's docker-socket failure stays visible as intended. Explicit test fixture. - api() raises ApiError on non-2xx + JSON-decode failure per feedback_api_helper_must_raise_not_return_dict. Pre-fix 'soft-fail' would silently paint main green via omission. Persona: claude-status-reaper (Gitea uid 94, write:repository) — provisioned 2026-05-11 21:39Z by sub-agent aefaac1b. Token under secrets.STATUS_REAPER_TOKEN (no other write surface touched). Acceptance (post-merge verify, Step-5): Trigger one class-O workflow via workflow_dispatch (e.g. sweep-cf-tunnels). Observe reaper compensate the resulting (push)-suffix failure on the next 5-min tick. Real push-triggered failures (publish-workspace-server-image) MUST still red main. Removal path: Drop this workflow + script + tests when Gitea is upgraded to >= 1.24 with a fix for the hardcoded-suffix bug, OR when an upstream patch lands (internal#80 RFC). Tracked in post-merge audit issue. Cross-links: - sibling internal#327 (publish-runtime-bot) - sibling internal#328 (mc-drift-bot) - sibling internal#329 (Gitea dispatcher race) - sibling internal#330 (disk-GC cron Gitea-class bug) - upstream internal#80 (Gitea hardcoded-suffix RFC) - mc#576 (preserved by design — real push-trigger failure) - sub-agent aefaac1b (provisioning sibling) - sub-agent a6f20db1 (Option A research — no upstream fix) Tests: 37 pytest cases pass (incl. hongming-pc 22:08Z review's 3 design checks: name-collision fail-loud, '/' in name lint, name vs filename fallback).	2026-05-11 23:24:54 +00:00