molecule-ai/molecule-core
51
0
Fork 2
Code Issues 165 Pull Requests 155 Actions 9 Packages Projects Releases Wiki Activity

CI gate RED on all core PRs: 4 conductor-snapshot tests use a frozen literal ts (10-min freshness time-bomb) -> /repos/// crash #2550

New Issue
Closed
opened 2026-06-10 18:41:58 +00:00 by core-be · 1 comment
core-be commented 2026-06-10 18:41:58 +00:00
Member
Copy Link
Copy Source

Summary

The required commit-status context Ops Scripts Tests / Ops scripts (unittest) (pull_request) is RED on every core PR, including identically on origin/main. This blocks the core merge gate for all PRs that touch scripts/** or .gitea/scripts/** (and the gate-check sees the red context regardless).

Reproduced on a fresh origin/main checkout:

4 failed, 379 passed, 2 skipped, 75 subtests passed
FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_list_candidate_issues_uses_snapshot_when_present
FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_list_queued_issues_uses_snapshot_label_filter
FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_get_combined_status_uses_snapshot_when_sha_matches
FAILED .gitea/scripts/tests/test_status_reaper_api.py::test_get_combined_status_uses_snapshot_when_sha_matches

Note on the green-workflow trap: the job carries continue-on-error: true (the mc#1982 mask), so the workflow shows green — but the per-step pytest failure still posts the required commit-status context as failure, and that is what the gate keys off. Don't be misled by the green check on the Actions tab.

Failing tests (exact)

  • test_gitea_merge_queue.py::test_list_candidate_issues_uses_snapshot_when_present
  • test_gitea_merge_queue.py::test_list_queued_issues_uses_snapshot_label_filter
  • test_gitea_merge_queue.py::test_get_combined_status_uses_snapshot_when_sha_matches
  • test_status_reaper_api.py::test_get_combined_status_uses_snapshot_when_sha_matches

All four crash with:

ValueError / ApiError: GET /repos///commits/<sha>/status ...

and captured stdout:

::notice::conductor snapshot stale (NNNNNs); self-fetching

Root cause (verified — this is a time-bomb test, not a flake)

The snapshot fixtures hardcode a frozen literal timestamp:

  • test_gitea_merge_queue.py: def _make_snapshot(prs, ts="2026-06-10T12:00:00Z")
  • test_status_reaper_api.py: two inline "ts": "2026-06-10T12:00:00Z" snapshot dicts

load_conductor_snapshot() (in .gitea/scripts/gitea-merge-queue.py lines ~247-264, mirrored in status-reaper.py ~185-202) only honors a snapshot within a 10-minute freshness window:

age_sec = (datetime.now(timezone.utc) - ts).total_seconds()
if age_sec > 600:  # 10 minutes
    print(f"::notice::conductor snapshot stale ({int(age_sec)}s); self-fetching")
    return None

These tests were authored on 2026-06-10 and only passed during the ~10-minute window after 2026-06-10T12:00:00Z. Once wall-clock passes 12:10:00Z, the loader treats the frozen snapshot as stale → returns None → the code falls through to a real API self-fetch → with empty GITEA_OWNER/GITEA_REPO env in the test harness this builds a malformed /repos///... URL → crash.

Mechanism is named, not "environmental": frozen-literal snapshot timestamp vs. a relative-age freshness check. The sibling test test_load_conductor_snapshot_ignores_stale_snapshot already does the correct thing — it builds ts as now - 15min — so the fix is to make the "uses_snapshot" fixtures equally relative.

Verified fix direction

Make _make_snapshot / the inline snapshot dicts default to a fresh now()-based timestamp (a _fresh_ts() helper), so the snapshot is always inside the freshness window when the suite runs. Tests that intentionally want a STALE snapshot pass ts= explicitly (the existing stale test is unchanged). Test-only change; no production-script change required.

Verified locally on a clean origin/main checkout: after patching both fixtures to a relative now() ts, the suite goes from 4 failed, 379 passed to 383 passed, 2 skipped with zero failures.

(Alternative considered and rejected: monkeypatch/inject the staleness check per-test — more invasive, and the fixture being time-relative is the more honest fix since it mirrors real conductor behavior. Bare-disabling the tests is explicitly NOT acceptable — it would re-mask the snapshot-consumption coverage.)

Impact

This required red context blocks the core merge gate for every affected PR, including the in-flight CI-hygiene stack:

  • #2548 — fix(merge-queue): silent skip for non-main base PRs (the durable skip-comment-flood fix)
  • #2539 — ci(lint): guard actions/setup-go cache on self-hosted fleet
  • #2541 — ci(lint): forbid continue-on-error on required-context jobs

Until this is fixed, those PRs cannot ride a clean gate.

A fix PR is being opened alongside this issue (test-only).

## Summary The required commit-status context **`Ops Scripts Tests / Ops scripts (unittest) (pull_request)`** is RED on **every** core PR, including identically on `origin/main`. This blocks the core merge gate for all PRs that touch `scripts/**` or `.gitea/scripts/**` (and the gate-check sees the red context regardless). Reproduced on a fresh `origin/main` checkout: ``` 4 failed, 379 passed, 2 skipped, 75 subtests passed FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_list_candidate_issues_uses_snapshot_when_present FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_list_queued_issues_uses_snapshot_label_filter FAILED .gitea/scripts/tests/test_gitea_merge_queue.py::test_get_combined_status_uses_snapshot_when_sha_matches FAILED .gitea/scripts/tests/test_status_reaper_api.py::test_get_combined_status_uses_snapshot_when_sha_matches ``` > Note on the green-workflow trap: the job carries `continue-on-error: true` (the mc#1982 mask), so the **workflow** shows green — but the per-step `pytest` failure still posts the **required commit-status context** as `failure`, and that is what the gate keys off. Don't be misled by the green check on the Actions tab. ## Failing tests (exact) - `test_gitea_merge_queue.py::test_list_candidate_issues_uses_snapshot_when_present` - `test_gitea_merge_queue.py::test_list_queued_issues_uses_snapshot_label_filter` - `test_gitea_merge_queue.py::test_get_combined_status_uses_snapshot_when_sha_matches` - `test_status_reaper_api.py::test_get_combined_status_uses_snapshot_when_sha_matches` All four crash with: ``` ValueError / ApiError: GET /repos///commits/<sha>/status ... ``` and captured stdout: ``` ::notice::conductor snapshot stale (NNNNNs); self-fetching ``` ## Root cause (verified — this is a time-bomb test, not a flake) The snapshot fixtures hardcode a **frozen literal** timestamp: - `test_gitea_merge_queue.py`: `def _make_snapshot(prs, ts="2026-06-10T12:00:00Z")` - `test_status_reaper_api.py`: two inline `"ts": "2026-06-10T12:00:00Z"` snapshot dicts `load_conductor_snapshot()` (in `.gitea/scripts/gitea-merge-queue.py` lines ~247-264, mirrored in `status-reaper.py` ~185-202) only honors a snapshot **within a 10-minute freshness window**: ```python age_sec = (datetime.now(timezone.utc) - ts).total_seconds() if age_sec > 600: # 10 minutes print(f"::notice::conductor snapshot stale ({int(age_sec)}s); self-fetching") return None ``` These tests were authored on 2026-06-10 and only passed during the ~10-minute window after `2026-06-10T12:00:00Z`. Once wall-clock passes `12:10:00Z`, the loader treats the frozen snapshot as stale → returns `None` → the code falls through to a **real API self-fetch** → with empty `GITEA_OWNER`/`GITEA_REPO` env in the test harness this builds a malformed `/repos///...` URL → crash. Mechanism is named, not "environmental": **frozen-literal snapshot timestamp vs. a relative-age freshness check.** The sibling test `test_load_conductor_snapshot_ignores_stale_snapshot` already does the correct thing — it builds `ts` as `now - 15min` — so the fix is to make the "uses_snapshot" fixtures equally relative. ## Verified fix direction Make `_make_snapshot` / the inline snapshot dicts default to a **fresh `now()`-based** timestamp (a `_fresh_ts()` helper), so the snapshot is always inside the freshness window when the suite runs. Tests that intentionally want a STALE snapshot pass `ts=` explicitly (the existing stale test is unchanged). Test-only change; no production-script change required. Verified locally on a clean `origin/main` checkout: after patching both fixtures to a relative `now()` ts, the suite goes from `4 failed, 379 passed` to **`383 passed, 2 skipped`** with zero failures. (Alternative considered and rejected: monkeypatch/inject the staleness check per-test — more invasive, and the fixture being time-relative is the more honest fix since it mirrors real conductor behavior. Bare-disabling the tests is explicitly NOT acceptable — it would re-mask the snapshot-consumption coverage.) ## Impact This required red context blocks the core merge gate for every affected PR, including the in-flight CI-hygiene stack: - **#2548** — fix(merge-queue): silent skip for non-main base PRs (the durable skip-comment-flood fix) - **#2539** — ci(lint): guard `actions/setup-go` cache on self-hosted fleet - **#2541** — ci(lint): forbid `continue-on-error` on required-context jobs Until this is fixed, those PRs cannot ride a clean gate. A fix PR is being opened alongside this issue (test-only).
core-be added the area/cikind/infrastructurerelease-blocker labels 2026-06-10 18:41:58 +00:00
core-be commented 2026-06-10 18:44:28 +00:00
Author
Member
Copy Link
Copy Source

Fix PR opened: #2551 (test-only, +24/-4). Verified in live CI on the PR head: Ops Scripts Tests / Ops scripts (unittest) is now success (was failure), gate-check-v3 green, qa-review + security-review approved, mergeable. Root cause confirmed: frozen-literal snapshot ts vs. the 10-minute freshness window in load_conductor_snapshot().

Fix PR opened: #2551 (test-only, +24/-4). Verified in live CI on the PR head: `Ops Scripts Tests / Ops scripts (unittest)` is now **success** (was failure), gate-check-v3 green, qa-review + security-review approved, mergeable. Root cause confirmed: frozen-literal snapshot ts vs. the 10-minute freshness window in `load_conductor_snapshot()`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label area/ci kind/infrastructure release-blocker
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2550
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?