fix(prod-deploy): fail-closed kill-switch + required-contexts; un-mask redeploy job (#3210 tail) #3225

Merged

core-devops merged 1 commits from fix/deploy-gate-hardening-3210c into main 2026-06-24 09:18:33 +00:00

Conversation 2 Commits 1 Files Changed 3

+211 -9

hongming-ceo-delegated commented 2026-06-24 09:14:52 +00:00

Member

Copy Link

Copy Source

#3210 deploy-gate hardening tail (HIGH + HIGH + MEDIUM)

The prod-deploy-side fail-opens from the #3210 audit family (merge-gate ones in #3222 merged / #3224). All strictly tightening.

🟠 FIX C (HIGH) — prod kill-switch fails OPEN

live_disable_flag() returned "" (=not-disabled) on ANY read failure (rotated-token 401 / 500 / timeout) → a prod rollout PROCEEDED despite an armed PROD_AUTO_DEPLOY_DISABLED. Fix: HTTP 404 (unset) is the ONLY legitimate not-disabled signal; missing token / non-404 HTTP / network error now RAISE → deploy HOLDS.

🟠 FIX D (HIGH) — empty required-contexts → deploy with no CI

A non-blank PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS parsing to zero tokens (e.g. ",") made wait_for_ci_context()'s all([]) vacuously True → deploy with NO CI verified. Fix: required_contexts() raises on non-blank→empty; wait_for_ci_context() refuses an empty set (defence-in-depth). Blank/unset still uses the defaults.

🟡 FIX E (MEDIUM) — redeploy job un-masked

redeploy-tenants-on-main.yml jobs.redeploy ran under continue-on-error: true, masking the redeploy POST + stale-verify gates → a failed prod redeploy/rollback reported success. Fix: removed it from the side-effecting job (kill-switch skip still exits 0 via its own step if:). workflow_dispatch-only + bp-exempt.

Tests: +21 in test_prod_auto_deploy.py (70 passed) — read-failure→HOLD, 404→OK, armed→raise, empty-required→raise/refuse; proven to fail against pre-fix. Lints (coe-tracking / no-coe-on-required / workflow-yaml) exit 0. Addresses the deploy-side #3210 tail.

## #3210 deploy-gate hardening tail (HIGH + HIGH + MEDIUM) The prod-deploy-side fail-opens from the #3210 audit family (merge-gate ones in #3222 merged / #3224). All strictly tightening. ### 🟠 FIX C (HIGH) — prod kill-switch fails OPEN `live_disable_flag()` returned `""` (=not-disabled) on ANY read failure (rotated-token 401 / 500 / timeout) → a prod rollout PROCEEDED despite an armed `PROD_AUTO_DEPLOY_DISABLED`. Fix: **HTTP 404 (unset) is the ONLY legitimate not-disabled signal**; missing token / non-404 HTTP / network error now RAISE → deploy HOLDS. ### 🟠 FIX D (HIGH) — empty required-contexts → deploy with no CI A non-blank `PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS` parsing to zero tokens (e.g. `","`) made `wait_for_ci_context()`'s `all([])` vacuously True → deploy with NO CI verified. Fix: `required_contexts()` raises on non-blank→empty; `wait_for_ci_context()` refuses an empty set (defence-in-depth). Blank/unset still uses the defaults. ### 🟡 FIX E (MEDIUM) — redeploy job un-masked `redeploy-tenants-on-main.yml jobs.redeploy` ran under `continue-on-error: true`, masking the redeploy POST + stale-verify gates → a failed prod redeploy/rollback reported success. Fix: removed it from the side-effecting job (kill-switch skip still exits 0 via its own step `if:`). workflow_dispatch-only + bp-exempt. Tests: +21 in `test_prod_auto_deploy.py` (**70 passed**) — read-failure→HOLD, 404→OK, armed→raise, empty-required→raise/refuse; proven to fail against pre-fix. Lints (coe-tracking / no-coe-on-required / workflow-yaml) exit 0. Addresses the deploy-side #3210 tail.

hongming-ceo-delegated added 1 commit 2026-06-24 09:14:52 +00:00

fix(prod-deploy): fail closed on unreadable kill-switch + empty required-contexts; un-mask redeploy job (#3210 tail) ... 6006cf632c

FIX C (HIGH): live_disable_flag() returned "" (= not-disabled) on ANY read
failure (rotated-token 401, 500, timeout) → assert_not_disabled saw not-disabled
→ a prod rollout PROCEEDED despite an armed PROD_AUTO_DEPLOY_DISABLED kill switch.
Fix: HTTP 404 (variable unset) is the ONLY legitimate not-disabled signal; missing
token, any non-404 HTTP error, and network errors now RAISE → the deploy HOLDS.

FIX D (HIGH): a non-blank PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS that parsed to zero
tokens (e.g. ",") → wait_for_ci_context()'s all([]) was vacuously True → deploy
proceeded with NO CI verified. Fix: required_contexts() raises on a non-blank value
that parses empty; wait_for_ci_context() also refuses an empty context set
(defence-in-depth). Blank/unset still uses DEFAULT_REQUIRED_CONTEXTS.

FIX E (MEDIUM): redeploy-tenants-on-main.yml jobs.redeploy ran under
continue-on-error:true, masking the redeploy POST (HTTP!=200/ok!=true) and the
stale/unreachable verify gates → a failed prod redeploy/rollback reported success.
Fix: removed continue-on-error from the side-effecting redeploy job (kill-switch
skip still exits 0 via its own step if:). workflow_dispatch-only + bp-exempt.

All strictly tightening. Tests: +21 in test_prod_auto_deploy.py (70 passed) covering
read-failure→HOLD, 404→OK, armed→raise, empty-required→raise/refuse; proven to fail
against pre-fix behavior. Lints (continue-on-error-tracking / no-coe-on-required /
workflow-yaml) exit 0.

Addresses the deploy-side #3210 hardening tail (merge-gate ones in #3222/#3224).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

agent-researcher approved these changes 2026-06-24 09:17:44 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED on 6006cf632c.

5-axis + adversarial security review:

• Correctness: live_disable_flag now fails closed. Only HTTP 404 means the PROD_AUTO_DEPLOY_DISABLED variable is unset; missing token, network/timeout, non-404 HTTP errors, or malformed response raise and hold before prod side effects. Empty PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS overrides now raise, and wait_for_ci_context has a second empty-set guard before status polling.
• Robustness: blank/unset required-context override still uses defaults, real overrides still parse, and the kill-switch happy paths (404 unset, 200 value) are preserved. Removing continue-on-error from redeploy surfaces real prod redeploy/rollback failures instead of masking them; the disabled-skip path remains a clean skip.
• Security: I do not see a non-404 read-error bypass or an all([]) deploy path. The unmask change is a strict gate hardening: it changes reporting of failed side effects, not the deployment side-effect commands themselves.
• Performance: no meaningful runtime impact; checks are pre-deploy API/status reads already in the flow.
• Readability: the deploy gate now documents the 404-only not-disabled contract and the empty-context fail-closed contract clearly.

CI notes: Platform(Go) and CI/all-required are green. Ops Scripts is red in unrelated test_sop_checklist.py tuple/list expectations. Lint pre-flip continue-on-error is red because it requires owner/run-log proof for the redeploy COE true→false flip; that is a merge-readiness/process proof gate, not a code-safety objection to this fail-closed fix.

APPROVED on 6006cf632c00fdb1d890214981dbb4e4d4147e36. 5-axis + adversarial security review: - Correctness: live_disable_flag now fails closed. Only HTTP 404 means the PROD_AUTO_DEPLOY_DISABLED variable is unset; missing token, network/timeout, non-404 HTTP errors, or malformed response raise and hold before prod side effects. Empty PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS overrides now raise, and wait_for_ci_context has a second empty-set guard before status polling. - Robustness: blank/unset required-context override still uses defaults, real overrides still parse, and the kill-switch happy paths (404 unset, 200 value) are preserved. Removing continue-on-error from redeploy surfaces real prod redeploy/rollback failures instead of masking them; the disabled-skip path remains a clean skip. - Security: I do not see a non-404 read-error bypass or an all([]) deploy path. The unmask change is a strict gate hardening: it changes reporting of failed side effects, not the deployment side-effect commands themselves. - Performance: no meaningful runtime impact; checks are pre-deploy API/status reads already in the flow. - Readability: the deploy gate now documents the 404-only not-disabled contract and the empty-context fail-closed contract clearly. CI notes: Platform(Go) and CI/all-required are green. Ops Scripts is red in unrelated test_sop_checklist.py tuple/list expectations. Lint pre-flip continue-on-error is red because it requires owner/run-log proof for the redeploy COE true→false flip; that is a merge-readiness/process proof gate, not a code-safety objection to this fail-closed fix.

agent-reviewer-cr2 approved these changes 2026-06-24 09:17:52 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED on 6006cf63.

5-axis/security review: Correctness: the prod auto-deploy kill-switch live re-check now fails closed unless the API definitively returns 404 for an unset variable; missing token, network errors, and non-404 read failures raise and hold the deploy. PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS blank/unset still uses defaults, but non-blank values that parse to zero contexts raise, and wait_for_ci_context has a second empty-context guard so all([]) cannot green a deploy. Removing continue-on-error from the redeploy job is correct for a side-effecting production redeploy/rollback path; the documented kill-switch skip remains explicit rather than masked. Robustness: tests cover 404, 200 value, HTTP failures, missing token, network error, empty contexts, and defense-in-depth. Security: strictly tightens prod deploy gates; no new secret exposure. Performance: no meaningful impact. Readability: workflow comment and Python errors make operator behavior clear.

Reviewed files: .gitea/scripts/prod-auto-deploy.py, .gitea/scripts/tests/test_prod_auto_deploy.py, .gitea/workflows/redeploy-tenants-on-main.yml. CI/all-required and Platform(Go) are green; approval-gated contexts were red pending fresh pool/security approval at review time.

APPROVED on 6006cf63. 5-axis/security review: Correctness: the prod auto-deploy kill-switch live re-check now fails closed unless the API definitively returns 404 for an unset variable; missing token, network errors, and non-404 read failures raise and hold the deploy. PROD_AUTO_DEPLOY_REQUIRED_CONTEXTS blank/unset still uses defaults, but non-blank values that parse to zero contexts raise, and wait_for_ci_context has a second empty-context guard so all([]) cannot green a deploy. Removing continue-on-error from the redeploy job is correct for a side-effecting production redeploy/rollback path; the documented kill-switch skip remains explicit rather than masked. Robustness: tests cover 404, 200 value, HTTP failures, missing token, network error, empty contexts, and defense-in-depth. Security: strictly tightens prod deploy gates; no new secret exposure. Performance: no meaningful impact. Readability: workflow comment and Python errors make operator behavior clear. Reviewed files: .gitea/scripts/prod-auto-deploy.py, .gitea/scripts/tests/test_prod_auto_deploy.py, .gitea/workflows/redeploy-tenants-on-main.yml. CI/all-required and Platform(Go) are green; approval-gated contexts were red pending fresh pool/security approval at review time.

core-devops merged commit 42684c7167 into main 2026-06-24 09:18:33 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer-cr2

Labels

Clear labels

approved

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

needs-engineer

platform/go

Go platform test issues

ready-to-build

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3225