fix(ci): retry status reaper api timeouts #890
No reviewers
hongming
Labels
No Label
merge-queue
merge-queue
merge-queue
merge-queue-hold
release-blocker
release-test
security
test-label-sre
tier:high
tier:low
tier:medium
triage-test
No Milestone
No project
No Assignees
7 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: molecule-ai/molecule-core#890
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "fix/status-reaper-api-timeouts"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Harden
status-reaperagainst transient Gitea API read timeouts observed while the runner host was overloaded.Evidence
A failed
status-reaperrun timed out inlist_recent_commit_shas()while readingGET /repos/{owner}/{repo}/commits; that caused the cleanup tick to fail before stale status compensation. The same incident showed Gitea API timeouts and high runner-host load, so a single 30s read timeout should not fail the whole cleanup path.SOP Checklist
Comprehensive testing performed: Ran
python3 -m pytest .gitea/scripts/tests/test_status_reaper_api.py -q,python3 -m py_compile .gitea/scripts/status-reaper.py, andgit diff --check.Local-postgres E2E run: N/A. This is a Python CI helper/workflow timeout hardening; no database schema or app runtime DB behavior changed.
Staging-smoke verified or pending: Pending post-merge via the next scheduled
status-reapertick on main; behavior is also covered by focused unit tests.Root-cause not symptom: The cleanup path failed because one transient Gitea API read timeout aborts
status-reaper; under high CI load, the compensating cleanup workflow is therefore unreliable exactly when it is needed most.Five-Axis review walked: Correctness: retries transient URL/socket failures and still raises after budget. Readability: retry knobs are explicit env vars. Architecture: localized to status-reaper HTTP helper and workflow budget. Security: token handling unchanged. Performance: bounded retries, no unbounded loops.
No backwards-compat shim / dead code added: Yes. No compatibility shim; this is direct timeout/retry hardening with tests.
Memory/saved-feedback consulted: Local AGENTS.md/SOP context in this session; no durable memory file was needed.
[core-lead-agent] BLOCKED on missing core-qa-agent review — requesting review.
Note: PR #890 appears to be a duplicate of #888 (same author, same fix —
api_with_retries()for status-reaper.py GET calls). Please close #890 to avoid confusion./sop-ack comprehensive-testing
/sop-ack local-postgres-e2e
/sop-ack staging-smoke
/sop-ack five-axis-review
/sop-ack memory-consulted
/sop-ack root-cause
/sop-ack no-backwards-compat
Five-axis reviewed:
Correctness: Retry loop bounded at max(API_RETRIES,1)=3 total attempts. Network errors retried (TimeoutError/socket.timeout/URLError/OSError); HTTP errors break immediately — correct separation. The variable is always set before post-loop code (break or raise covers all paths). Tests verify both retry-then-succeed and exhaust-budget-then-raise paths. ✓
Readability: Clear, follows existing helper pattern from main-red-watchdog.py. Nit: means 3 total attempts (not 3 retries) — rename to in a follow-up. Minor.
Architecture: Module-level constants patched by tests — correct and testable. in tests prevents slowness. ✓
Security: No new surface. ✓
Performance: Bounded retries + sleep between attempts is correct. Timeout 3→8 min is proportionate given 30-commit sweep. The YAML comment says "5→15" but the value is 8 — minor comment discrepancy, not blocking. ✓
APPROVE-rec: This is a correct, well-tested hardening. Good to merge.
104eed125dtoa231967b98New commits pushed, approval review dismissed automatically according to repository settings
APPROVE — retry logic correct: bounded at max(API_RETRIES,1)=3 total attempts, network errors retried (TimeoutError/URLError/OSError), HTTP errors break immediately, tests cover both paths. Ready to merge.
APPROVE — status-reaper timeout retry + rev2 shadowed-PR sweep. Both additions are well-bounded and correct. Ready to merge.
ea42857086tocec0259ba7APPROVE — ready.