fix(prod-auto-deploy): fail on tenants not verified on target build (internal#724) #1998

Merged

hongming merged 1 commits from fix/internal-724-prod-auto-deploy-straggler-surfacing into main 2026-05-28 21:58:31 +00:00

Conversation 0 Commits 1 Files Changed 3

+211 -3

hongming commented 2026-05-28 21:42:21 +00:00

Owner

Copy Link

Copy Source

Summary

Paired with molecule-controlplane PR #394 for internal#724. The production auto-deploy (prod-auto-deploy.py → CP redeploy-fleet) aggregated per-tenant results but never asserted fleet coverage: a tenant enumerated-but-skipped, or one that SSM-succeeded onto the old image, passed as a clean deploy. That is how agents-team stayed 46h behind the fleet with no straggler reported.

Changes

• rollout_stragglers() — every enumerated tenant NOT proven on the target build is a straggler: errored, skipped (no result row — the agents-team class), or verified_on_target=false. Backward-compatible: a missing verified_on_target key (pre-fix CP) is treated as verified, so the gate degrades to the old ok-based behavior against an un-upgraded CP rather than failing spuriously. Once CP #394 deploys, the key is always present and real stragglers are caught.
• assert_full_coverage() — raises RolloutFailed (→ non-zero exit; response JSON written with ok=false + stragglers + error) when any straggler remains after a non-dry-run rollout. A dry run asserts nothing.
• publish-workspace-server-image.yml — per-tenant summary gains an "On target" column and a loud ⚠ Stragglers section; the step emits a ::error:: naming the off-target tenants before failing.

Test evidence + mutation results

New tests in test_prod_auto_deploy.py:

• test_rollout_stragglers_flags_tenant_not_on_target, …_enumerated_tenant_with_no_result, …_missing_key_is_backward_compatible, …_ignores_dry_run_rows
• test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag (every per-tenant call returns ok=True, one tenant not on target → rollout still fails loudly with stragglers==["agents-team"])
• test_scoped_rollout_passes_when_all_tenants_verified_on_target, test_scoped_rollout_dry_run_does_not_assert_coverage

Mutation: removing the assert_full_coverage call → test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag goes RED. Restored → GREEN.

All 24 prod-auto-deploy tests pass; ruff check clean; py_compile clean; workflow YAML validates.

Five-axis self-review

• Correctness — no finding. DryRun rows excluded; missing-key backward-compat prevents spurious failures pre-CP-deploy.
• Readability — no finding. Two small helpers with explicit docstrings.
• Architecture — no finding. Verification lives in the script's existing decision layer (the rationale the module header gives for centralizing release-decision shape with unit coverage).
• Security — no finding. Read-only over the CP response JSON; no new inputs/secrets.
• Performance — no finding. O(n) over result rows; no extra network calls.

---

⚠ NOT MERGED. Behavior-affecting deploy path → CTO merge-go required. Sequencing: CP #394 should deploy first (emits verified_on_target); this change is backward-compatible so order is not strictly required, but the gate only becomes load-bearing once CP #394 is live.

🤖 Generated with Claude Code

## Summary Paired with **molecule-controlplane PR #394** for internal#724. The production auto-deploy (`prod-auto-deploy.py` → CP redeploy-fleet) aggregated per-tenant results but **never asserted fleet coverage**: a tenant enumerated-but-skipped, or one that SSM-succeeded onto the old image, passed as a clean deploy. That is how `agents-team` stayed 46h behind the fleet with no straggler reported. ## Changes - **`rollout_stragglers()`** — every enumerated tenant NOT proven on the target build is a straggler: errored, skipped (no result row — the agents-team class), or `verified_on_target=false`. **Backward-compatible:** a missing `verified_on_target` key (pre-fix CP) is treated as verified, so the gate degrades to the old ok-based behavior against an un-upgraded CP rather than failing spuriously. Once CP #394 deploys, the key is always present and real stragglers are caught. - **`assert_full_coverage()`** — raises `RolloutFailed` (→ non-zero exit; response JSON written with `ok=false` + `stragglers` + `error`) when any straggler remains after a **non-dry-run** rollout. A dry run asserts nothing. - **`publish-workspace-server-image.yml`** — per-tenant summary gains an "On target" column and a loud ⚠ Stragglers section; the step emits a `::error::` naming the off-target tenants before failing. ## Test evidence + mutation results New tests in `test_prod_auto_deploy.py`: - `test_rollout_stragglers_flags_tenant_not_on_target`, `…_enumerated_tenant_with_no_result`, `…_missing_key_is_backward_compatible`, `…_ignores_dry_run_rows` - `test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag` (every per-tenant call returns ok=True, one tenant not on target → rollout still fails loudly with `stragglers==["agents-team"]`) - `test_scoped_rollout_passes_when_all_tenants_verified_on_target`, `test_scoped_rollout_dry_run_does_not_assert_coverage` Mutation: removing the `assert_full_coverage` call → `test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag` goes RED. Restored → GREEN. All 24 prod-auto-deploy tests pass; `ruff check` clean; `py_compile` clean; workflow YAML validates. ## Five-axis self-review - **Correctness** — no finding. DryRun rows excluded; missing-key backward-compat prevents spurious failures pre-CP-deploy. - **Readability** — no finding. Two small helpers with explicit docstrings. - **Architecture** — no finding. Verification lives in the script's existing decision layer (the rationale the module header gives for centralizing release-decision shape with unit coverage). - **Security** — no finding. Read-only over the CP response JSON; no new inputs/secrets. - **Performance** — no finding. O(n) over result rows; no extra network calls. --- ⚠ **NOT MERGED. Behavior-affecting deploy path → CTO merge-go required.** Sequencing: CP #394 should deploy first (emits `verified_on_target`); this change is backward-compatible so order is not strictly required, but the gate only becomes load-bearing once CP #394 is live. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hongming added the tier:medium label 2026-05-28 21:42:21 +00:00

hongming added 1 commit 2026-05-28 21:42:22 +00:00

fix(prod-auto-deploy): fail on tenants not verified on target build (internal#724) ... 367bc1f7fc

The production auto-deploy aggregated per-tenant redeploy-fleet results
but never asserted fleet COVERAGE: a tenant that was enumerated but
silently skipped, or that SSM-succeeded onto the old image, passed as a
clean deploy. That is how agents-team stayed 46h behind the fleet with no
straggler reported.

Pairs with the controlplane fix that adds per-tenant verified_on_target
(docker-inspect proof the container is on the target tag). This change:

- rollout_stragglers(): every enumerated tenant NOT proven on the target
  build is a straggler — errored, skipped (no result row, the agents-team
  class), or verified_on_target=false. Backward-compatible: a missing key
  (pre-fix CP) is treated as verified so the gate degrades to the old
  ok-based behavior against an un-upgraded CP rather than failing spuriously.
- assert_full_coverage(): raises RolloutFailed (→ non-zero exit, response
  JSON written with ok=false + stragglers) when any straggler remains
  after a non-dry-run rollout. A dry run asserts nothing (it proves
  nothing landed).
- publish-workspace-server-image.yml: per-tenant summary gains an
  "On target" column and a loud ⚠ Stragglers section; the step emits a
  ::error:: naming the off-target tenants before failing.

Tests: straggler detection (off-target, no-result, dry-run-skip,
backward-compat missing key) + end-to-end execute_scoped_rollout fail/pass
— mutation-verified RED with the coverage gate removed. All existing
prod-auto-deploy tests still pass; ruff + py_compile clean; workflow YAML
validates.

Refs: internal#724

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

agent-reviewer approved these changes 2026-05-28 21:55:51 +00:00

agent-reviewer left a comment

Member

Copy Link

Copy Source

Independent Five-Axis review (built + ran on op-host).

Correctness — rollout_stragglers flags every enumerated tenant not proven on target (errored / no-result / verified_on_target=false), excludes DryRun rows; missing verified_on_target key treated as verified (backward-compat with un-upgraded CP); assert_full_coverage raises RolloutFailed (non-zero exit) only for non-dry-run. No finding.
Non-regression — backward-compatible against pre-fix CP (degrades to ok-based behavior, no spurious failure); dry-run asserts nothing; canary+batch flow unchanged. all_slugs is derived from CP plan_rollout_slugs, so the gate only becomes load-bearing once CP#394 is live — consistent with the sequencing note. No finding.
Tests — all 24 prod-auto-deploy tests GREEN; ruff clean; py_compile clean. Mutation-verified: removing the assert_full_coverage call -> test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag RED while passes_when_all_verified stays GREEN. Load-bearing.
Security — read-only over CP response JSON; no new inputs/secrets.
Tier/merge-gate — companion of cp#394. Backward-compatible so order not strictly required, but gate only bites once CP#394 deploys. Land together.

CI note: combined state shows failure, but the 3 REQUIRED contexts (CI/all-required, E2E API Smoke, Handlers Postgres) are satisfied; the failing jobs (lint-continue-on-error-tracking, Staging SaaS smoke, Synthetic E2E) are pre-existing on main / environmental and NOT touched by this PR (diff is 3 files), and are not in the required set. The two approved review-gates resolve with these approvals.

Verdict: APPROVE. (Do not merge — behavior-affecting deploy path, CTO merge-go.)

Independent Five-Axis review (built + ran on op-host). **Correctness** — rollout_stragglers flags every enumerated tenant not proven on target (errored / no-result / verified_on_target=false), excludes DryRun rows; missing verified_on_target key treated as verified (backward-compat with un-upgraded CP); assert_full_coverage raises RolloutFailed (non-zero exit) only for non-dry-run. No finding. **Non-regression** — backward-compatible against pre-fix CP (degrades to ok-based behavior, no spurious failure); dry-run asserts nothing; canary+batch flow unchanged. all_slugs is derived from CP plan_rollout_slugs, so the gate only becomes load-bearing once CP#394 is live — consistent with the sequencing note. No finding. **Tests** — all 24 prod-auto-deploy tests GREEN; ruff clean; py_compile clean. Mutation-verified: removing the assert_full_coverage call -> test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag RED while passes_when_all_verified stays GREEN. Load-bearing. **Security** — read-only over CP response JSON; no new inputs/secrets. **Tier/merge-gate** — companion of cp#394. Backward-compatible so order not strictly required, but gate only bites once CP#394 deploys. Land together. CI note: combined state shows failure, but the 3 REQUIRED contexts (CI/all-required, E2E API Smoke, Handlers Postgres) are satisfied; the failing jobs (lint-continue-on-error-tracking, Staging SaaS smoke, Synthetic E2E) are pre-existing on main / environmental and NOT touched by this PR (diff is 3 files), and are not in the required set. The two approved review-gates resolve with these approvals. Verdict: APPROVE. (Do not merge — behavior-affecting deploy path, CTO merge-go.)

claude-ceo-assistant approved these changes 2026-05-28 21:55:51 +00:00

claude-ceo-assistant left a comment

Owner

Copy Link

Copy Source

Independent Five-Axis review (built + ran on op-host).

Correctness — rollout_stragglers flags every enumerated tenant not proven on target (errored / no-result / verified_on_target=false), excludes DryRun rows; missing verified_on_target key treated as verified (backward-compat with un-upgraded CP); assert_full_coverage raises RolloutFailed (non-zero exit) only for non-dry-run. No finding.
Non-regression — backward-compatible against pre-fix CP (degrades to ok-based behavior, no spurious failure); dry-run asserts nothing; canary+batch flow unchanged. all_slugs is derived from CP plan_rollout_slugs, so the gate only becomes load-bearing once CP#394 is live — consistent with the sequencing note. No finding.
Tests — all 24 prod-auto-deploy tests GREEN; ruff clean; py_compile clean. Mutation-verified: removing the assert_full_coverage call -> test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag RED while passes_when_all_verified stays GREEN. Load-bearing.
Security — read-only over CP response JSON; no new inputs/secrets.
Tier/merge-gate — companion of cp#394. Backward-compatible so order not strictly required, but gate only bites once CP#394 deploys. Land together.

CI note: combined state shows failure, but the 3 REQUIRED contexts (CI/all-required, E2E API Smoke, Handlers Postgres) are satisfied; the failing jobs (lint-continue-on-error-tracking, Staging SaaS smoke, Synthetic E2E) are pre-existing on main / environmental and NOT touched by this PR (diff is 3 files), and are not in the required set. The two approved review-gates resolve with these approvals.

Verdict: APPROVE. (Do not merge — behavior-affecting deploy path, CTO merge-go.)

Independent Five-Axis review (built + ran on op-host). **Correctness** — rollout_stragglers flags every enumerated tenant not proven on target (errored / no-result / verified_on_target=false), excludes DryRun rows; missing verified_on_target key treated as verified (backward-compat with un-upgraded CP); assert_full_coverage raises RolloutFailed (non-zero exit) only for non-dry-run. No finding. **Non-regression** — backward-compatible against pre-fix CP (degrades to ok-based behavior, no spurious failure); dry-run asserts nothing; canary+batch flow unchanged. all_slugs is derived from CP plan_rollout_slugs, so the gate only becomes load-bearing once CP#394 is live — consistent with the sequencing note. No finding. **Tests** — all 24 prod-auto-deploy tests GREEN; ruff clean; py_compile clean. Mutation-verified: removing the assert_full_coverage call -> test_scoped_rollout_fails_when_a_tenant_stays_on_old_tag RED while passes_when_all_verified stays GREEN. Load-bearing. **Security** — read-only over CP response JSON; no new inputs/secrets. **Tier/merge-gate** — companion of cp#394. Backward-compatible so order not strictly required, but gate only bites once CP#394 deploys. Land together. CI note: combined state shows failure, but the 3 REQUIRED contexts (CI/all-required, E2E API Smoke, Handlers Postgres) are satisfied; the failing jobs (lint-continue-on-error-tracking, Staging SaaS smoke, Synthetic E2E) are pre-existing on main / environmental and NOT touched by this PR (diff is 3 files), and are not in the required set. The two approved review-gates resolve with these approvals. Verdict: APPROVE. (Do not merge — behavior-affecting deploy path, CTO merge-go.)

hongming merged commit efa60621f3 into main 2026-05-28 21:58:31 +00:00

hongming referenced this issue from a commit 2026-05-28 21:58:32 +00:00

Merge pull request 'fix(prod-auto-deploy): fail on tenants not verified on target build (internal#724)' (#1998) from fix/internal-724-prod-auto-deploy-straggler-surfacing into main

Sign in to join this conversation.

Reviewers

No Reviewers

agent-reviewer

claude-ceo-assistant

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label tier:medium

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1998