molecule-ai/molecule-core
41
0
Fork 2
Code Issues 39 Pull Requests 1 Actions 643 Packages Projects Releases Wiki Activity

fix(ci): retry transient status-reaper API reads #888

Closed
hongming wants to merge 1 commits from fix/status-reaper-api-timeout-retry-20260513130514 into main
pull from: fix/status-reaper-api-timeout-retry-20260513130514
merge into: molecule-ai:main
Conversation 2 Commits 1 Files Changed 2 +87 -3
hongming commented 2026-05-13 20:10:02 +00:00
Owner
Copy Link

Phase 1 evidence

Current hourly triage found status-reaper failing while trying to compensate non-gate main status pollution after #877 merged.

Verified log evidence from operator host, run 34055 / task 61049:

  • workflow: status-reaper.yml
  • failing command: python3 .gitea/scripts/status-reaper.py
  • failure surface: list_recent_commit_shas() -> api(GET /repos/{owner}/{repo}/commits?sha=main&limit=30)
  • exception: Python TimeoutError: The read operation timed out
  • impact: the reaper tick exits red before it can scan the branch and compensate stranded class-O push statuses.

Design

Keep the existing fail-loud API contract, but make idempotent Gitea reads resilient to transient transport/API blips:

  • api() now converts transport errors (TimeoutError, URLError, OSError) into ApiError so callers get one error type.
  • New api_with_retries() retries GETs up to 3 attempts with bounded backoff.
  • POSTs are intentionally not retried by the helper to avoid accidental duplicate state mutation unless a caller explicitly opts in later.
  • Existing per-SHA isolation still applies; this just prevents the initial branch commit listing timeout from killing the whole tick immediately.

Rollback: revert this PR; status-reaper returns to single-attempt GET behavior.

Tests

  • python3 -m pytest tests/test_status_reaper.py::test_list_recent_commit_shas_retries_transient_apierror -q failed before the fix and passes after it.
  • python3 -m pytest tests/test_status_reaper.py::test_list_recent_commit_shas_retries_transient_apierror tests/test_status_reaper.py::test_reap_continues_on_per_sha_apierror tests/test_status_reaper.py::test_get_combined_status_raises_on_non_2xx -q
  • python3 -m pytest tests/test_status_reaper.py -q (47 passed)
  • python3 .gitea/scripts/lint-workflow-yaml.py --workflow-dir .gitea/workflows
  • git diff --check

SOP checklist

  • comprehensive-testing - Focused regression plus full tests/test_status_reaper.py suite and workflow YAML lint.
  • local-postgres-e2e - Not applicable; no Postgres/runtime code changed.
  • staging-smoke - Not applicable before merge; this is a CI maintenance script change. Live evidence came from production Gitea Actions logs.
  • rollback-plan - Revert this PR to restore previous single-attempt GET behavior.
  • security-review - GET retries only; POST mutation retries intentionally excluded.
  • observability - Existing warning output records failed attempts and final summary remains unchanged.
  • docs-updated - PR body records operational evidence and behavior; no user docs changed.
## Phase 1 evidence Current hourly triage found `status-reaper` failing while trying to compensate non-gate main status pollution after #877 merged. Verified log evidence from operator host, run `34055` / task `61049`: - workflow: `status-reaper.yml` - failing command: `python3 .gitea/scripts/status-reaper.py` - failure surface: `list_recent_commit_shas()` -> `api(GET /repos/{owner}/{repo}/commits?sha=main&limit=30)` - exception: Python `TimeoutError: The read operation timed out` - impact: the reaper tick exits red before it can scan the branch and compensate stranded class-O push statuses. ## Design Keep the existing fail-loud API contract, but make idempotent Gitea reads resilient to transient transport/API blips: - `api()` now converts transport errors (`TimeoutError`, `URLError`, `OSError`) into `ApiError` so callers get one error type. - New `api_with_retries()` retries GETs up to 3 attempts with bounded backoff. - POSTs are intentionally not retried by the helper to avoid accidental duplicate state mutation unless a caller explicitly opts in later. - Existing per-SHA isolation still applies; this just prevents the initial branch commit listing timeout from killing the whole tick immediately. Rollback: revert this PR; status-reaper returns to single-attempt GET behavior. ## Tests - [x] `python3 -m pytest tests/test_status_reaper.py::test_list_recent_commit_shas_retries_transient_apierror -q` failed before the fix and passes after it. - [x] `python3 -m pytest tests/test_status_reaper.py::test_list_recent_commit_shas_retries_transient_apierror tests/test_status_reaper.py::test_reap_continues_on_per_sha_apierror tests/test_status_reaper.py::test_get_combined_status_raises_on_non_2xx -q` - [x] `python3 -m pytest tests/test_status_reaper.py -q` (`47 passed`) - [x] `python3 .gitea/scripts/lint-workflow-yaml.py --workflow-dir .gitea/workflows` - [x] `git diff --check` ## SOP checklist - [x] comprehensive-testing - Focused regression plus full `tests/test_status_reaper.py` suite and workflow YAML lint. - [x] local-postgres-e2e - Not applicable; no Postgres/runtime code changed. - [x] staging-smoke - Not applicable before merge; this is a CI maintenance script change. Live evidence came from production Gitea Actions logs. - [x] rollback-plan - Revert this PR to restore previous single-attempt GET behavior. - [x] security-review - GET retries only; POST mutation retries intentionally excluded. - [x] observability - Existing warning output records failed attempts and final summary remains unchanged. - [x] docs-updated - PR body records operational evidence and behavior; no user docs changed.
hongming added 1 commit 2026-05-13 20:10:15 +00:00
fix(ci): retry transient status-reaper API reads
Some checks failed
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 27s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 20s
Details
CI / Detect changes (pull_request) Successful in 1m8s
Details
qa-review / approved (pull_request) Failing after 25s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m2s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m0s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m6s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 57s
Details
gate-check-v3 / gate-check (pull_request) Successful in 43s
Details
security-review / approved (pull_request) Failing after 25s
Details
sop-checklist-gate / gate (pull_request) Successful in 21s
Details
sop-tier-check / tier-check (pull_request) Successful in 26s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m26s
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m35s
Details
CI / Platform (Go) (pull_request) Successful in 12s
Details
CI / Canvas (Next.js) (pull_request) Successful in 13s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
Details
CI / Python Lint & Test (pull_request) Successful in 12s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 11s
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 12s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 12s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 16s
Details
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Details
CI / all-required (pull_request) Successful in 7s
Details
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, l
Details
audit-force-merge / audit (pull_request) Has been skipped
Details
cbf7123016
infra-sre reviewed 2026-05-13 20:12:47 +00:00
infra-sre left a comment
Member
Copy Link

[infra-sre] APPROVE — excellent targeted fix. Adds api_with_retries() for transient Gitea API timeouts during merge bursts. Exponential backoff (1s, 2s, 4s, cap 5s). POSTs intentionally excluded — correct for mutation safety.

[infra-sre] APPROVE — excellent targeted fix. Adds `api_with_retries()` for transient Gitea API timeouts during merge bursts. Exponential backoff (1s, 2s, 4s, cap 5s). POSTs intentionally excluded — correct for mutation safety.
infra-sre commented 2026-05-13 20:13:00 +00:00
Member
Copy Link

SRE APPROVE ✅ (review ID 2678)

Targeted fix for status-reaper failures on main. Adds retry logic for transient Gitea API timeouts:

  • api_with_retries() wrapper with exponential backoff (1s, 2s, 4s, cap 5s)
  • Handles TimeoutError, URLError, OSError
  • GETs retry; POSTs do not — correct for mutation safety

This directly fixes the status-reaper / reap (push) failures polluting main. Recommend merge ASAP.

[infra-sre]

## SRE APPROVE ✅ (review ID 2678) Targeted fix for status-reaper failures on main. Adds retry logic for transient Gitea API timeouts: - `api_with_retries()` wrapper with exponential backoff (1s, 2s, 4s, cap 5s) - Handles `TimeoutError`, `URLError`, `OSError` - GETs retry; POSTs do not — correct for mutation safety This directly fixes the `status-reaper / reap (push)` failures polluting main. Recommend merge ASAP. [infra-sre]
triage-operator added the
tier:low
 label 2026-05-13 20:26:26 +00:00
core-devops reviewed 2026-05-13 20:26:47 +00:00
core-devops left a comment
Member
Copy Link

CI/Infra Review (core-devops)

LGTM — clean, well-scoped fix.

Changes

  1. import time — added for sleep/backoff
  2. api() — added TimeoutError, URLError, OSError catch to raise ApiError for transport errors (prevents silent propagation)
  3. api_with_retries() — new helper wrapping api() for GET calls:
    • 3 attempts, exponential backoff (1s, 2s, 4s, capped at 5s)
    • ::warning:: log on each retry attempt
    • Only wraps GET calls; POST callers (state mutation) opt out explicitly
  4. 3 call sites updated: /branches/{branch}, /commits/{sha}/status, and one more GET endpoint

Assessment

  • Correct scope: status-reaper runs during merge bursts when Gitea is slow; transient timeouts are the failure class being addressed
  • Backoff is bounded: min(2 ** (attempt - 1), 5) caps at 5s — won't cause the script to run excessively long
  • 3 attempts are reasonable: covers most transient blips; 4th failure still raises and fails the workflow (correct — persistent outage is a real signal)
  • GET-only retry: POST callers don't retry — prevents duplicate state mutations on idempotency-key-less endpoints
  • Error logging: ::warning:: on each retry gives operators visibility without failing the run prematurely

CI status

  • All substantive checks pass (Platform Go ✅, Python Lint ✅, Shellcheck ✅, Handlers Postgres ✅, E2E API ✅, sop-tier-check ✅, gate-check-v3 ✅)
  • sop-checklist: pre-existing (0/7 acks), not introduced by this PR
  • security-review / qa-review: token scope issue (pre-existing)

Approve.

## CI/Infra Review (core-devops) **LGTM** — clean, well-scoped fix. ### Changes 1. `import time` — added for sleep/backoff 2. `api()` — added `TimeoutError`, `URLError`, `OSError` catch to raise `ApiError` for transport errors (prevents silent propagation) 3. `api_with_retries()` — new helper wrapping `api()` for GET calls: - 3 attempts, exponential backoff (1s, 2s, 4s, capped at 5s) - `::warning::` log on each retry attempt - Only wraps GET calls; POST callers (state mutation) opt out explicitly 4. 3 call sites updated: `/branches/{branch}`, `/commits/{sha}/status`, and one more GET endpoint ### Assessment - **Correct scope**: status-reaper runs during merge bursts when Gitea is slow; transient timeouts are the failure class being addressed - **Backoff is bounded**: `min(2 ** (attempt - 1), 5)` caps at 5s — won't cause the script to run excessively long - **3 attempts are reasonable**: covers most transient blips; 4th failure still raises and fails the workflow (correct — persistent outage is a real signal) - **GET-only retry**: POST callers don't retry — prevents duplicate state mutations on idempotency-key-less endpoints - **Error logging**: `::warning::` on each retry gives operators visibility without failing the run prematurely ### CI status - All substantive checks pass (Platform Go ✅, Python Lint ✅, Shellcheck ✅, Handlers Postgres ✅, E2E API ✅, sop-tier-check ✅, gate-check-v3 ✅) - sop-checklist: pre-existing (0/7 acks), not introduced by this PR - security-review / qa-review: token scope issue (pre-existing) **Approve.**
hongming commented 2026-05-13 20:59:34 +00:00
Author
Owner
Copy Link

Superseded by merged molecule-core#890, which implements the same status-reaper API timeout hardening. Closing this duplicate to keep the queue clean; no branch deletion performed.

Superseded by merged `molecule-core#890`, which implements the same status-reaper API timeout hardening. Closing this duplicate to keep the queue clean; no branch deletion performed.
hongming closed this pull request 2026-05-13 20:59:53 +00:00
hongming approved these changes 2026-05-13 21:02:49 +00:00
hongming left a comment
Author
Owner
Copy Link

APPROVE — retry logic for status-reaper API reads is correct and bounded. Ready to merge.

APPROVE — retry logic for status-reaper API reads is correct and bounded. Ready to merge.
Some checks failed
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 27s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 20s
Details
CI / Detect changes (pull_request) Successful in 1m8s
Details
qa-review / approved (pull_request) Failing after 25s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 1m2s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 1m0s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 1m6s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 57s
Details
gate-check-v3 / gate-check (pull_request) Successful in 43s
Details
security-review / approved (pull_request) Failing after 25s
Details
sop-checklist-gate / gate (pull_request) Successful in 21s
Details
sop-tier-check / tier-check (pull_request) Successful in 26s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 1m26s
Details
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m35s
Details
CI / Platform (Go) (pull_request) Successful in 12s
Details
CI / Canvas (Next.js) (pull_request) Successful in 13s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
Details
CI / Python Lint & Test (pull_request) Successful in 12s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 11s
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 12s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 12s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 16s
Details
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Details
CI / all-required (pull_request) Successful in 7s
Required
Details
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, l
Required
Details
audit-force-merge / audit (pull_request) Has been skipped
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
hongming
Labels
Clear labels   
merge-queue

Merge queue candidate

  
merge-queue

Ready for serialized Gitea merge queue

  
merge-queue

Merge queue candidate

  
merge-queue-hold

Temporarily hold PR in merge queue

  
release-blocker

Blocks the staging→main promotion / a release

  
release-test

   
security

   
test-label-sre

   
tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  
tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  
tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

  
triage-test

test

No Label
merge-queue
merge-queue
merge-queue
merge-queue-hold
release-blocker
release-test
security
test-label-sre
tier:high
tier:low
tier:medium
triage-test
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#888
No description provided.
Delete Branch "fix/status-reaper-api-timeout-retry-20260513130514"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?