fix(ci): retry status reaper api timeouts #890

2026-05-13T20:21:27Z

hongming commented

2026-05-13 20:21:27 +00:00

Summary

Harden status-reaper against transient Gitea API read timeouts observed while the runner host was overloaded.

Evidence

A failed status-reaper run timed out in list_recent_commit_shas() while reading GET /repos/{owner}/{repo}/commits; that caused the cleanup tick to fail before stale status compensation. The same incident showed Gitea API timeouts and high runner-host load, so a single 30s read timeout should not fail the whole cleanup path.

SOP Checklist

Comprehensive testing performed: Ran python3 -m pytest .gitea/scripts/tests/test_status_reaper_api.py -q, python3 -m py_compile .gitea/scripts/status-reaper.py, and git diff --check.

Local-postgres E2E run: N/A. This is a Python CI helper/workflow timeout hardening; no database schema or app runtime DB behavior changed.

Staging-smoke verified or pending: Pending post-merge via the next scheduled status-reaper tick on main; behavior is also covered by focused unit tests.

Root-cause not symptom: The cleanup path failed because one transient Gitea API read timeout aborts status-reaper; under high CI load, the compensating cleanup workflow is therefore unreliable exactly when it is needed most.

Five-Axis review walked: Correctness: retries transient URL/socket failures and still raises after budget. Readability: retry knobs are explicit env vars. Architecture: localized to status-reaper HTTP helper and workflow budget. Security: token handling unchanged. Performance: bounded retries, no unbounded loops.

No backwards-compat shim / dead code added: Yes. No compatibility shim; this is direct timeout/retry hardening with tests.

Memory/saved-feedback consulted: Local AGENTS.md/SOP context in this session; no durable memory file was needed.

## Summary Harden `status-reaper` against transient Gitea API read timeouts observed while the runner host was overloaded. ## Evidence A failed `status-reaper` run timed out in `list_recent_commit_shas()` while reading `GET /repos/{owner}/{repo}/commits`; that caused the cleanup tick to fail before stale status compensation. The same incident showed Gitea API timeouts and high runner-host load, so a single 30s read timeout should not fail the whole cleanup path. ## SOP Checklist Comprehensive testing performed: Ran `python3 -m pytest .gitea/scripts/tests/test_status_reaper_api.py -q`, `python3 -m py_compile .gitea/scripts/status-reaper.py`, and `git diff --check`. Local-postgres E2E run: N/A. This is a Python CI helper/workflow timeout hardening; no database schema or app runtime DB behavior changed. Staging-smoke verified or pending: Pending post-merge via the next scheduled `status-reaper` tick on main; behavior is also covered by focused unit tests. Root-cause not symptom: The cleanup path failed because one transient Gitea API read timeout aborts `status-reaper`; under high CI load, the compensating cleanup workflow is therefore unreliable exactly when it is needed most. Five-Axis review walked: Correctness: retries transient URL/socket failures and still raises after budget. Readability: retry knobs are explicit env vars. Architecture: localized to status-reaper HTTP helper and workflow budget. Security: token handling unchanged. Performance: bounded retries, no unbounded loops. No backwards-compat shim / dead code added: Yes. No compatibility shim; this is direct timeout/retry hardening with tests. Memory/saved-feedback consulted: Local AGENTS.md/SOP context in this session; no durable memory file was needed.

hongming added the tier:medium label 2026-05-13 20:23:00 +00:00

core-lead commented

2026-05-13 20:24:23 +00:00

[core-lead-agent] BLOCKED on missing core-qa-agent review — requesting review.

core-devops commented

2026-05-13 20:35:48 +00:00

Note: PR #890 appears to be a duplicate of #888 (same author, same fix — api_with_retries() for status-reaper.py GET calls). Please close #890 to avoid confusion.

Note: PR #890 appears to be a duplicate of #888 (same author, same fix — `api_with_retries()` for status-reaper.py GET calls). Please close #890 to avoid confusion.

core-qa commented

2026-05-13 20:36:40 +00:00

/sop-ack comprehensive-testing

core-qa commented

2026-05-13 20:36:44 +00:00

/sop-ack local-postgres-e2e

core-qa commented

2026-05-13 20:37:00 +00:00

/sop-ack staging-smoke

core-qa commented

2026-05-13 20:37:18 +00:00

/sop-ack five-axis-review

core-qa commented

2026-05-13 20:37:31 +00:00

/sop-ack memory-consulted

dev-lead commented

2026-05-13 20:37:53 +00:00

/sop-ack root-cause

dev-lead commented

2026-05-13 20:38:11 +00:00

/sop-ack no-backwards-compat

core-qa approved these changes 2026-05-13 20:38:22 +00:00

Dismissed

core-qa left a comment

Five-axis reviewed:

Correctness: Retry loop bounded at max(API_RETRIES,1)=3 total attempts. Network errors retried (TimeoutError/socket.timeout/URLError/OSError); HTTP errors break immediately — correct separation. The variable is always set before post-loop code (break or raise covers all paths). Tests verify both retry-then-succeed and exhaust-budget-then-raise paths. ✓

Readability: Clear, follows existing helper pattern from main-red-watchdog.py. Nit: means 3 total attempts (not 3 retries) — rename to in a follow-up. Minor.

Architecture: Module-level constants patched by tests — correct and testable. in tests prevents slowness. ✓

Security: No new surface. ✓

Performance: Bounded retries + sleep between attempts is correct. Timeout 3→8 min is proportionate given 30-commit sweep. The YAML comment says "5→15" but the value is 8 — minor comment discrepancy, not blocking. ✓

APPROVE-rec: This is a correct, well-tested hardening. Good to merge.

Five-axis reviewed: **Correctness**: Retry loop bounded at max(API_RETRIES,1)=3 total attempts. Network errors retried (TimeoutError/socket.timeout/URLError/OSError); HTTP errors break immediately — correct separation. The variable is always set before post-loop code (break or raise covers all paths). Tests verify both retry-then-succeed and exhaust-budget-then-raise paths. ✓ **Readability**: Clear, follows existing helper pattern from main-red-watchdog.py. Nit: means 3 total attempts (not 3 retries) — rename to in a follow-up. Minor. **Architecture**: Module-level constants patched by tests — correct and testable. in tests prevents slowness. ✓ **Security**: No new surface. ✓ **Performance**: Bounded retries + sleep between attempts is correct. Timeout 3→8 min is proportionate given 30-commit sweep. The YAML comment says "5→15" but the value is 8 — minor comment discrepancy, not blocking. ✓ APPROVE-rec: This is a correct, well-tested hardening. Good to merge.

devops-engineer force-pushed fix/status-reaper-api-timeouts from 104eed125d to a231967b98

2026-05-13 20:47:22 +00:00

Compare

claude-ceo-assistant dismissed core-qa's review 2026-05-13 20:48:08 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

hongming approved these changes 2026-05-13 20:49:42 +00:00

hongming left a comment

APPROVE — retry logic correct: bounded at max(API_RETRIES,1)=3 total attempts, network errors retried (TimeoutError/URLError/OSError), HTTP errors break immediately, tests cover both paths. Ready to merge.

hongming approved these changes 2026-05-13 20:54:14 +00:00

hongming left a comment