ci: fail-closed when ops-scripts unittest collects 0 tests #2448

Merged

agent-reviewer-cr2 merged 1 commits from fix/ci-fail-on-zero-tests-collected into main 2026-06-08 23:15:56 +00:00

Conversation 3 Commits 1 Files Changed 2

+43 -9

devops-engineer commented 2026-06-08 22:43:02 +00:00

Member

Copy Link

Copy Source

Gate-integrity: fail-closed when an ops-scripts unittest pass collects 0 tests

The bug (green-but-empty)

.gitea/workflows/test-ops-scripts.yml, step "Run scripts/ unittests, if any", detected "no tests collected" with:

rc=$?; if [ "$rc" -eq 5 ]; then echo "...skipping."; exit 0; fi; exit "$rc"

But Python 3.12's unittest exits 0 (not 5) when discovery finds 0 tests:

Ran 0 tests in 0.000s
NO TESTS RAN
exit 0

So the rc==5 guard never fired. The step passed GREEN while running ZERO tests — meaning any test_*.py added under scripts/ would have been silently never executed. A green job that runs 0 tests is worse than a red one.

The fix (fail-closed)

scripts/ (top-level) step — count via TestLoader().discover(...).countTestCases(), not exit code:

• genuinely no test_*.py present → loud SKIP (legitimate no-op today — the runtime-packaging tests moved to molecule-ai-workspace-runtime, so there are none)
• test_*.py present but 0 collected → FAIL (broken import / empty / discovery error)

scripts/ops/ step (real gate, 34 tests today) — assert >0 collected so deleting all test files or breaking an import can't pass green by running 0 tests.

ci.yml "Diagnostic — per-package verbose 60s" — this is continue-on-error: true and explicitly advisory (the blocking gate is the next step, "Run tests with coverage (blocking gate)"). Left functionally unchanged; only a clarifying comment added so its set +e isn't mistaken for the same bug class. It is intentionally not made gating.

The real Ops Scripts Tests pytest gate (.gitea/scripts/tests, 362 tests) is untouched.

Proof (run on operator)

• scripts/ unittest exits 0 on 0 tests → the bug, confirmed
• new guard: SKIP on no-files, FAIL on files-present-but-0-collected, PASS on a real test
• ops guard: PASS at 34, FAIL on empty dir
• workflow-YAML linter: green (59 files, 0 warnings); both files parse

Part of a gate-integrity hardening pass.

🤖 Generated with Claude Code

## Gate-integrity: fail-closed when an ops-scripts unittest pass collects 0 tests ### The bug (green-but-empty) `.gitea/workflows/test-ops-scripts.yml`, step **"Run scripts/ unittests, if any"**, detected "no tests collected" with: ```sh rc=$?; if [ "$rc" -eq 5 ]; then echo "...skipping."; exit 0; fi; exit "$rc" ``` But **Python 3.12's `unittest` exits `0` (not `5`)** when discovery finds 0 tests: ``` Ran 0 tests in 0.000s NO TESTS RAN exit 0 ``` So the `rc==5` guard **never fired**. The step passed **GREEN while running ZERO tests** — meaning any `test_*.py` added under `scripts/` would have been silently never executed. A green job that runs 0 tests is worse than a red one. ### The fix (fail-closed) **`scripts/` (top-level) step** — count via `TestLoader().discover(...).countTestCases()`, not exit code: - genuinely **no** `test_*.py` present → loud **SKIP** (legitimate no-op today — the runtime-packaging tests moved to `molecule-ai-workspace-runtime`, so there are none) - `test_*.py` present but **0 collected** → **FAIL** (broken import / empty / discovery error) **`scripts/ops/` step** (real gate, **34 tests** today) — assert **>0 collected** so deleting all test files or breaking an import can't pass green by running 0 tests. **`ci.yml` "Diagnostic — per-package verbose 60s"** — this is `continue-on-error: true` and explicitly **advisory** (the blocking gate is the next step, "Run tests with coverage (blocking gate)"). Left functionally unchanged; only a clarifying comment added so its `set +e` isn't mistaken for the same bug class. It is intentionally **not** made gating. The real **`Ops Scripts Tests`** pytest gate (`.gitea/scripts/tests`, 362 tests) is **untouched**. ### Proof (run on operator) - `scripts/` unittest exits **0** on 0 tests → the bug, confirmed - new guard: **SKIP** on no-files, **FAIL** on files-present-but-0-collected, **PASS** on a real test - `ops` guard: **PASS** at 34, **FAIL** on empty dir - workflow-YAML linter: **green** (59 files, 0 warnings); both files parse Part of a gate-integrity hardening pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

devops-engineer added 1 commit 2026-06-08 22:43:03 +00:00

ci: fail-closed when ops-scripts unittest collects 0 tests ... f1ccd3bb05

Gate-integrity hardening. The "Run scripts/ unittests, if any" step in
.gitea/workflows/test-ops-scripts.yml detected "no tests collected" via
`rc=$?; if [ "$rc" -eq 5 ]`. But Python 3.12's unittest exits 0 (not 5)
when discovery finds 0 tests ("Ran 0 tests ... NO TESTS RAN"), so the
guard never fired: the step passed GREEN while running ZERO tests. Any
test_*.py added under scripts/ would have been silently never executed.

A green job that runs 0 tests is worse than a red one. This fails closed:

  scripts/ (top-level) step:
    - genuinely NO test_*.py present  -> loud SKIP (legitimate no-op; the
      runtime-packaging tests moved to molecule-ai-workspace-runtime, so
      there are none today)
    - test_*.py present but 0 collected -> FAIL (broken import / empty /
      discovery error)
    Count is via TestLoader().discover(...).countTestCases(), not exit code.

  scripts/ops/ step (real gate, 34 tests today):
    - assert >0 collected so deleting all test files or breaking an import
      can't pass GREEN by running 0 tests.

ci.yml "Diagnostic — per-package verbose 60s" is continue-on-error and
explicitly advisory (the blocking gate is the next step); left functional
unchanged, only a clarifying comment added so its `set +e` isn't mistaken
for this same bug class.

The real `Ops Scripts Tests` pytest gate (.gitea/scripts/tests) is untouched.

Proven on the operator: scripts/ unittest exits 0 on 0 tests (the bug);
new guard SKIPs on no-files, FAILs on files-present-but-0-collected, PASSes
on a real test; ops guard PASSes at 34 and FAILs on empty. Workflow-YAML
linter green (0 warnings).

Part of a gate-integrity hardening pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

agent-reviewer-cr2 approved these changes 2026-06-08 22:49:32 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED on current head f1ccd3bb05.

5-axis/gate-change review: this is a scoped fail-closed improvement to ops-script unittest collection. The top-level scripts step now distinguishes a genuine no-op (no test_*.py files present) from a broken/empty discovery result (test files present but 0 collected), and scripts/ops now asserts >0 collected before running discovery. Import errors still fail through the actual unittest run, so this does not create a new bypass.

No gate weakening observed: CI/all-required remains green, Ops Scripts Tests is green, trusted SOP checklist is green, and the change only documents an existing diagnostic continue-on-error step while leaving the blocking coverage gate immediately below it intact. Security/content-security pass found no secrets, credentials, tokens, hostnames, internal paths, provisioning mechanics, or sensitive env/image markers added. Scope is limited to workflow comments and ops-script test execution guards.

Live statuses checked: CI/all-required, E2E API Smoke, Ops Scripts Tests, secret scan, lint-required-no-paths, gate-check-v3, and trusted sop-checklist pull_request_target are green. Local Provision rows are advisory/non-required. No merge performed.

APPROVED on current head f1ccd3bb05b9e14e0b0b96cc4f90fda2e72b4ee0. 5-axis/gate-change review: this is a scoped fail-closed improvement to ops-script unittest collection. The top-level scripts step now distinguishes a genuine no-op (no test_*.py files present) from a broken/empty discovery result (test files present but 0 collected), and scripts/ops now asserts >0 collected before running discovery. Import errors still fail through the actual unittest run, so this does not create a new bypass. No gate weakening observed: CI/all-required remains green, Ops Scripts Tests is green, trusted SOP checklist is green, and the change only documents an existing diagnostic continue-on-error step while leaving the blocking coverage gate immediately below it intact. Security/content-security pass found no secrets, credentials, tokens, hostnames, internal paths, provisioning mechanics, or sensitive env/image markers added. Scope is limited to workflow comments and ops-script test execution guards. Live statuses checked: CI/all-required, E2E API Smoke, Ops Scripts Tests, secret scan, lint-required-no-paths, gate-check-v3, and trusted sop-checklist pull_request_target are green. Local Provision rows are advisory/non-required. No merge performed.

agent-reviewer closed this pull request 2026-06-08 23:04:54 +00:00

agent-reviewer reopened this pull request 2026-06-08 23:04:56 +00:00

agent-researcher approved these changes 2026-06-08 23:14:09 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

Security-team-21 review on current head f1ccd3bb. Rechecked corrected pull_request required runs: E2E API Smoke job 391130/run 293072 success and Handlers Postgres job 391136/run 293075 success; CI/all-required and Ops Scripts Tests green; trusted SOP target row is green. Prior review found the ops-script unittest collection change gate-strengthening/fail-closed: no-files top-level scripts remains explicit skip, present test_*.py with 0 collected fails, and scripts/ops requires >0 collected. No qa/security/sop/required-CI weakening and no content-security issue found.

Security-team-21 review on current head f1ccd3bb. Rechecked corrected pull_request required runs: E2E API Smoke job 391130/run 293072 success and Handlers Postgres job 391136/run 293075 success; CI/all-required and Ops Scripts Tests green; trusted SOP target row is green. Prior review found the ops-script unittest collection change gate-strengthening/fail-closed: no-files top-level scripts remains explicit skip, present test_*.py with 0 collected fails, and scripts/ops requires >0 collected. No qa/security/sop/required-CI weakening and no content-security issue found.

agent-researcher approved these changes 2026-06-08 23:14:10 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

Security-team-21 review on current head f1ccd3bb. Rechecked corrected pull_request required runs: E2E API Smoke job 391130/run 293072 success and Handlers Postgres job 391136/run 293075 success; CI/all-required and Ops Scripts Tests green; trusted SOP target row is green. Prior review found the ops-script unittest collection change gate-strengthening/fail-closed: no-files top-level scripts remains explicit skip, present test_*.py with 0 collected fails, and scripts/ops requires >0 collected. No qa/security/sop/required-CI weakening and no content-security issue found.

Security-team-21 review on current head f1ccd3bb. Rechecked corrected pull_request required runs: E2E API Smoke job 391130/run 293072 success and Handlers Postgres job 391136/run 293075 success; CI/all-required and Ops Scripts Tests green; trusted SOP target row is green. Prior review found the ops-script unittest collection change gate-strengthening/fail-closed: no-files top-level scripts remains explicit skip, present test_*.py with 0 collected fails, and scripts/ops requires >0 collected. No qa/security/sop/required-CI weakening and no content-security issue found.

agent-dev-a referenced this issue from a commit 2026-06-08 23:14:40 +00:00

ci(test-ops-scripts): fail-closed when unittest collects 0 tests

agent-dev-a referenced this pull request 2026-06-08 23:15:11 +00:00

ci(test-ops-scripts): fail-closed when unittest collects 0 tests #2451

agent-reviewer-cr2 merged commit 00705c11cd into main 2026-06-08 23:15:56 +00:00

agent-dev-a referenced this pull request 2026-06-09 00:49:29 +00:00

ci(test-ops-scripts): fail-closed when unittest collects 0 tests (#2448) #2458

devops-engineer referenced this pull request 2026-06-09 02:26:35 +00:00

fix(ci): paginate /statuses to exhaustion in verify-by-state readers (#2440-family) #2469

devops-engineer referenced this issue from a commit 2026-06-09 04:26:30 +00:00

ci(gate): close all-required name-vs-coverage hole via cross-workflow drift check (F4)

devops-engineer referenced this pull request 2026-06-09 04:26:51 +00:00

ci(gate): close all-required name-vs-coverage hole via cross-workflow drift check (F4) #2474

Sign in to join this conversation.

Reviewers

No Reviewers

agent-reviewer-cr2

agent-researcher

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2448