fix(ci): wait for platform /health on a migration-chain-proof budget (#2205) #2206

Merged

claude-ceo-assistant merged 1 commits from fix/e2e-api-health-wait-migration-chain into main 2026-06-04 05:32:29 +00:00

Conversation 1 Commits 1 Files Changed 4

+111 -19

core-devops commented 2026-06-04 04:48:25 +00:00

Member

Copy Link

Copy Source

What

Fixes the E2E API Smoke Test REQUIRED branch-protection gate going red on main (RCA in #2205), plus the three sibling local-platform E2E workflows that share the identical brittle pattern.

Root cause (#2205)

The platform is started in the background, then the workflow waited for /health with a fixed 30×1s loop. The platform binds /health only after applying the full migration chain on cold start — and that chain has grown past 30s (the run log reaches 20260523000000_schedule_consecutive_sdk_errors.up.sql before printing Platform starting on :PORT). The 30s budget expired before the server was reachable → downstream E2E assertions never ran → red. A fixed budget is brittle by construction: the migration chain keeps growing.

Fix — deterministic, not a bigger magic number

For each platform /health wait (e2e-api, e2e-chat, e2e-peer-visibility, e2e-legacy-advisory):

• Poll /health on a generous, clearly-commented 180s wall-clock budget that comfortably exceeds cold-start + full-migration time and stays robust as the chain grows. A 200 from /health is the real readiness signal (migrations done + server listening).
• Fast-fail + loud on a genuinely dead platform: if the backgrounded platform-server PID has exited (e.g. a broken migration crashed it), stop immediately and dump the platform log — never mask a real startup failure, never wait out the full budget for a process that's already gone.
• On true timeout, dump the platform log tail and fail with ::error::.

The unrelated Postgres-readiness seq 1 30 waits (not gated on the migration chain) are intentionally left unchanged.

Why it's robust + still fails loud

• Migrations take 45s → old 30s loop failed; new logic keeps polling, sees /health 200, proceeds. (Verified by simulation: server binding late → success.)
• Platform never binds (dead process) → PID-liveness check trips at once, dumps log, exits 1. (Verified: 0s fast-fail with log dump, not a 180s hang.)

Lints

• lint-curl-status-capture: passes — curl usage avoids the -w '%{http_code}' capture shape.
• lint-workflow-yaml: passes (56 files, 0 warnings). YAML parses; bash shellcheck-clean.

Closes #2205

🤖 Generated with Claude Code

## What Fixes the `E2E API Smoke Test` REQUIRED branch-protection gate going red on main (RCA in #2205), plus the three sibling local-platform E2E workflows that share the identical brittle pattern. ## Root cause (#2205) The platform is started in the background, then the workflow waited for `/health` with a fixed **30×1s** loop. The platform binds `/health` only **after applying the full migration chain** on cold start — and that chain has grown past 30s (the run log reaches `20260523000000_schedule_consecutive_sdk_errors.up.sql` before printing `Platform starting on :PORT`). The 30s budget expired before the server was reachable → downstream E2E assertions never ran → red. A fixed budget is brittle by construction: the migration chain keeps growing. ## Fix — deterministic, not a bigger magic number For each platform `/health` wait (`e2e-api`, `e2e-chat`, `e2e-peer-visibility`, `e2e-legacy-advisory`): - Poll `/health` on a **generous, clearly-commented 180s wall-clock budget** that comfortably exceeds cold-start + full-migration time and stays robust as the chain grows. A 200 from `/health` is the real readiness signal (migrations done + server listening). - **Fast-fail + loud on a genuinely dead platform**: if the backgrounded `platform-server` PID has exited (e.g. a broken migration crashed it), stop immediately and dump the platform log — never mask a real startup failure, never wait out the full budget for a process that's already gone. - On true timeout, dump the platform log tail and fail with `::error::`. The unrelated Postgres-readiness `seq 1 30` waits (not gated on the migration chain) are intentionally left unchanged. ## Why it's robust + still fails loud - Migrations take 45s → old 30s loop failed; new logic keeps polling, sees `/health` 200, proceeds. (Verified by simulation: server binding late → success.) - Platform never binds (dead process) → PID-liveness check trips at once, dumps log, exits 1. (Verified: 0s fast-fail with log dump, not a 180s hang.) ## Lints - `lint-curl-status-capture`: passes — curl usage avoids the `-w '%{http_code}'` capture shape. - `lint-workflow-yaml`: passes (56 files, 0 warnings). YAML parses; bash `shellcheck`-clean. Closes #2205 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-04 04:48:26 +00:00

fix(ci): wait for platform /health on a migration-chain-proof budget (#2205) ... 382a894f53

The `E2E API Smoke Test` REQUIRED gate (and the sibling local-platform E2E
workflows) started the platform in the background and waited for /health with
a fixed 30×1s loop (~30s). The platform binds /health only AFTER applying the
FULL migration chain on cold start; that chain now reaches past the 30s window
(the run log gets to 20260523000000_schedule_consecutive_sdk_errors.up.sql
before "Platform starting on :PORT"), so the health loop expired before the
server was reachable → downstream E2E never ran → main went red. A fixed budget
is brittle by construction because the migration chain grows every release.

Fix (deterministic, not a bigger magic number):
- Poll /health on a generous, clearly-commented wall-clock budget (180s) that
  comfortably exceeds cold-start + full-migration time and is robust to the
  chain continuing to grow. /health returning 200 is the real readiness signal
  (migrations done + server listening).
- Still fail fast + loud on a genuinely dead platform: if the backgrounded
  platform-server PID has exited (e.g. a broken migration crashed it), stop
  immediately and dump the platform log — we never mask a real startup failure,
  and we never wait out the full budget for a process that is already gone.
- On true timeout, dump the platform log tail and fail with ::error::.

Applied identically to the four workflows sharing the 30×1s platform-/health
pattern: e2e-api, e2e-chat, e2e-peer-visibility, e2e-legacy-advisory. The
unrelated Postgres-readiness `seq 1 30` waits (which are not gated on the
migration chain) are intentionally left unchanged.

curl usage avoids the -w '%{http_code}' status-capture shape, so
lint-curl-status-capture passes; lint-workflow-yaml passes on all 56 files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-ceo-assistant merged commit 7f25373309 into main 2026-06-04 05:32:29 +00:00

core-devops commented 2026-06-04 05:32:30 +00:00

Author

Member

Copy Link

Copy Source

Owner force-merged (honest bypass). Clears the E2E API Smoke required-gate flake (#2205): 30x1s health-wait was shorter than the growing migration chain; now a 180s readiness budget + kill -0 liveness (dead platform still fails fast+loud). Applied to e2e-api/e2e-chat/e2e-peer-visibility/e2e-legacy-advisory. Required CI green. Token revoked.

Owner force-merged (honest bypass). Clears the E2E API Smoke required-gate flake (#2205): 30x1s health-wait was shorter than the growing migration chain; now a 180s readiness budget + kill -0 liveness (dead platform still fails fast+loud). Applied to e2e-api/e2e-chat/e2e-peer-visibility/e2e-legacy-advisory. Required CI green. Token revoked.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2206