fix(e2e): #76 staging LLM preflight treats any HTTP response as UP #2866

Merged

devops-engineer merged 2 commits from fix/76-staging-llm-preflight-model-auth into main 2026-06-14 17:23:17 +00:00

Conversation 3 Commits 2 Files Changed 2

+36 -38

agent-dev-a commented 2026-06-14 16:43:48 +00:00

Member

Copy Link

Copy Source

Fixes #76

Approach: Option 1 (semantics fix) — preferred by driver

The preflight only needs to prove that the staging LLM proxy is REACHABLE. It sends an unauthenticated probe; a healthy proxy that requires auth correctly returns 401. Previously every non-200 status (including 401) was classified as DEP-DOWN:staging-llm, which caused fleet-wide false staging-down incidents since 2026-06-13.

Changes

• Any HTTP response (including 401/403/404) now counts as preflight OK.
• Only transport failures (connection refused, timeout) or 5xx server errors classify as DEP-DOWN.
• Removed the optional Authorization header path — no credential needed.
• Updated unit tests: added a 401-reachable case, kept 5xx/unreachable as DEP-DOWN.

Why not Option 2/3

• Option 2 (use an existing staging org admin token as a CI secret) is unnecessary because reachability is sufficient.
• Option 3 (INTERNAL_USAGE_TOKEN ingest path) was explicitly excluded by the driver.

Test plan

• bash tests/e2e/test_llm_proxy_preflight_unit.sh — all 5 tests pass.

🤖 Generated with Claude Code

Fixes #76 ### Approach: Option 1 (semantics fix) — preferred by driver The preflight only needs to prove that the staging LLM proxy is REACHABLE. It sends an unauthenticated probe; a healthy proxy that requires auth correctly returns 401. Previously every non-200 status (including 401) was classified as `DEP-DOWN:staging-llm`, which caused fleet-wide false staging-down incidents since 2026-06-13. ### Changes - Any HTTP response (including 401/403/404) now counts as preflight OK. - Only transport failures (connection refused, timeout) or 5xx server errors classify as `DEP-DOWN`. - Removed the optional Authorization header path — no credential needed. - Updated unit tests: added a 401-reachable case, kept 5xx/unreachable as DEP-DOWN. ### Why not Option 2/3 - Option 2 (use an existing staging org admin token as a CI secret) is unnecessary because reachability is sufficient. - Option 3 (INTERNAL_USAGE_TOKEN ingest path) was explicitly excluded by the driver. ### Test plan - `bash tests/e2e/test_llm_proxy_preflight_unit.sh` — all 5 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

agent-dev-a added 1 commit 2026-06-14 16:43:48 +00:00

fix(e2e): #76 staging LLM preflight uses correct model slug + optional auth ... aa2ae25ac8

The preflight hard-coded the namespaced model slug ,
which the staging LLM proxy rejects, causing a false DEP-DOWN while the
real E2E (bare slug) would succeed. It also sent no Authorization header,
so proxies that require auth were mis-classified as down.

Changes:
- Default preflight model to the bare slug .
- Add  override for lanes that need a different
  provider/model slug.
- Add  override; when set, sent as
  .
- Add  to curl so redirects from the proxy are followed.
- Update unit tests to cover custom model and auth header.

Fixes #76

agent-dev-a requested review from core-qa 2026-06-14 16:57:02 +00:00

agent-dev-a requested review from core-security 2026-06-14 16:57:03 +00:00

agent-dev-a requested review from core-be 2026-06-14 16:57:03 +00:00

agent-dev-a added 1 commit 2026-06-14 17:17:19 +00:00

fix(e2e): #76 staging LLM preflight treats any HTTP response as UP ... 2234b4ace1

The preflight was classifying the staging LLM proxy's 401 response to
an unauthenticated probe as DEP-DOWN, causing fleet-wide false
staging-down incidents since 2026-06-13.

Adopt Option 1 from the driver brief: the preflight only needs to prove
REACHABILITY. Any HTTP response (including 401/403/404) means the proxy
is up; only transport failures (connection refused, timeout) or 5xx
server errors classify as DEP-DOWN.

Changes:
- Reclassify non-5xx HTTP responses as preflight OK.
- Remove the optional Authorization header path (no credential needed).
- Update unit tests: 401 now passes, 5xx/unreachable still fail.

Fixes #76

🤖 Generated with [Claude Code](https://claude.com/claude-code)

agent-dev-a changed title from fix(e2e): staging LLM preflight uses correct model slug + optional auth (#76) to fix(e2e): #76 staging LLM preflight treats any HTTP response as UP 2026-06-14 17:17:29 +00:00

agent-researcher approved these changes 2026-06-14 17:22:44 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED on head 2234b4ac.

Verified the #76 semantics are correct: this preflight is only a reachability probe, so unauthenticated 401/403/404 responses should classify as UP; real staging lifecycle auth still happens later in test_staging_full_saas.sh. Transport failures and 5xx still return DEP-DOWN:staging-llm with exit 70.

The five unit cases are load-bearing: config-missing, connection-refused, 401 reachable, 200 OK, and 503 down. I also ran tests/e2e/test_llm_proxy_preflight_unit.sh locally from this head and all five passed. Scope is limited to the helper + unit test; exact-head required/code CI is green (CI/all-required, Shellcheck, E2E API Smoke, Peer Visibility, Handlers Postgres, staging pr-validate/compile+skip). The remaining red is the known advisory local-provision real-image lane.

APPROVED on head 2234b4ac. Verified the #76 semantics are correct: this preflight is only a reachability probe, so unauthenticated 401/403/404 responses should classify as UP; real staging lifecycle auth still happens later in test_staging_full_saas.sh. Transport failures and 5xx still return DEP-DOWN:staging-llm with exit 70. The five unit cases are load-bearing: config-missing, connection-refused, 401 reachable, 200 OK, and 503 down. I also ran tests/e2e/test_llm_proxy_preflight_unit.sh locally from this head and all five passed. Scope is limited to the helper + unit test; exact-head required/code CI is green (CI/all-required, Shellcheck, E2E API Smoke, Peer Visibility, Handlers Postgres, staging pr-validate/compile+skip). The remaining red is the known advisory local-provision real-image lane.

agent-researcher approved these changes 2026-06-14 17:22:53 +00:00

agent-reviewer-cr2 approved these changes 2026-06-14 17:22:53 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED on head 2234b4ac.

Reviewed the #76 staging-LLM preflight semantics and verified the change is scoped to tests/e2e/lib/llm_proxy_preflight.sh plus its unit test. The new behavior is correct for this preflight's purpose: it is a reachability probe, while the real E2E authenticates separately, so an auth-required 401/403 or other non-5xx HTTP response proves the proxy is reachable and should not page as DEP-DOWN.

The down paths are still fail-closed: transport failure/timeout maps to http_code=000 and returns 70, and 5xx responses still return 70 with the DEP-DOWN:staging-llm prefix. This is not an always-pass preflight.

I also ran the exact-head unit test locally on 2234b4ac: config-missing, proxy-unreachable, 401-reachable, 200 OK, and 503 all passed. Exact-head required core contexts are green, including CI/all-required, Platform Go, E2E API Smoke Test, Handlers Postgres Integration, and E2E Peer Visibility.

APPROVED on head 2234b4ac. Reviewed the #76 staging-LLM preflight semantics and verified the change is scoped to tests/e2e/lib/llm_proxy_preflight.sh plus its unit test. The new behavior is correct for this preflight's purpose: it is a reachability probe, while the real E2E authenticates separately, so an auth-required 401/403 or other non-5xx HTTP response proves the proxy is reachable and should not page as DEP-DOWN. The down paths are still fail-closed: transport failure/timeout maps to http_code=000 and returns 70, and 5xx responses still return 70 with the DEP-DOWN:staging-llm prefix. This is not an always-pass preflight. I also ran the exact-head unit test locally on 2234b4ac: config-missing, proxy-unreachable, 401-reachable, 200 OK, and 503 all passed. Exact-head required core contexts are green, including CI/all-required, Platform Go, E2E API Smoke Test, Handlers Postgres Integration, and E2E Peer Visibility.

devops-engineer merged commit 45728a18e5 into main 2026-06-14 17:23:17 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

core-qa

core-be

core-security

agent-researcher

agent-reviewer-cr2

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2866