fix(ci): superseded prod-deploy job no longer false-reds as "stale" #2194

Merged

claude-ceo-assistant merged 1 commits from fix/deploy-production-superseded-false-stale into main 2026-06-04 02:39:59 +00:00

Conversation 1 Commits 1 Files Changed 3

+198

core-devops commented 2026-06-04 02:34:57 +00:00

Member

Copy Link

Copy Source

Root cause

publish-workspace-server-image / Production auto-deploy intermittently false-reds on main:

::error::hongming is stale: actual=2863380, expected=eb31bcf

The workflow deliberately has no concurrency: (header: Gitea 1.22.6 cancels queued runs even with cancel-in-progress:false, which is unacceptable for a prod deploy). So when two main pushes land close together (eb31bcf Fix A, then 286338 Fix C → staging-2863380), both deploy-production jobs run. The newer job rolls the fleet forward to 2863380 first; then the older eb31bcf job runs "Verify reachable tenants report this SHA", sees tenants on 2863380, and fails on strict SHA equality — even though the fleet is AHEAD, not behind.

Git SHAs aren't ordered and /buildinfo exposes only git_sha (no build time / monotonic number — see workspace-server/internal/router/router.go + internal/buildinfo/buildinfo.go), so the verify can't distinguish "ahead" from "behind" on its own.

Fix — option (b), superseded-job detection

Before the strict verify, ask Gitea for the current head of the deploy branch (main). If main's head is no longer this job's GITHUB_SHA, a newer commit has landed and this deploy is superseded — the newest deploy job's verify is the authoritative one. The superseded job logs a ::notice:: and exits success, skipping the strict-equality loop.

• New superseded_by() / current_branch_head() in .gitea/scripts/prod-auto-deploy.py + check-superseded subcommand (exit 0 = superseded/print newer SHA, exit 10 = still latest → run strict verify).
• Workflow Verify reachable tenants report this SHA step calls it first and short-circuits to success only when superseded.

Why it preserves real-stale detection

• Only the superseded (older) job skips the strict verify. The latest deploy job (head == its SHA) still runs strict equality, so a genuinely behind/older tenant still fails loudly.
• Fail-safe: if the branch head can't be read (no token / API error) or equals our SHA, superseded_by returns None → strict verify runs. An unreadable head never silently greens a deploy.

Why not (a)/(c)

• (a) build-timestamp / monotonic compare: /buildinfo returns only {git_sha}. Adding a build-time field requires a workspace-server binary + Dockerfile.tenant change and a full fleet rebuild before it could be relied on — heavy and slow to take effect.
• (c) concurrency: — forbidden by the workflow header (Gitea cancels queued prod deploys); cannot serialize-without-cancel safely.

Verification

• New unit tests for superseded_by / current_branch_head incl. fail-safe + short-vs-full SHA prefix; 33 passed (pytest .gitea/scripts/tests/test_prod_auto_deploy.py).
• Workflow yaml-lint clean (lint-workflow-yaml.py, 56 files, 0 warnings).
• CLI smoke test of the exact eb31bcf-vs-2863380 scenario: superseded → exit 0 (skip, success); latest job → exit 10 (run strict verify); unreadable head → exit 10.

🤖 Generated with Claude Code

## Root cause `publish-workspace-server-image / Production auto-deploy` intermittently false-reds on `main`: ``` ::error::hongming is stale: actual=2863380, expected=eb31bcf ``` The workflow **deliberately has no `concurrency:`** (header: Gitea 1.22.6 cancels queued runs even with `cancel-in-progress:false`, which is unacceptable for a prod deploy). So when two `main` pushes land close together (`eb31bcf` Fix A, then `286338` Fix C → `staging-2863380`), **both** `deploy-production` jobs run. The newer job rolls the fleet forward to `2863380` first; then the **older** `eb31bcf` job runs *"Verify reachable tenants report this SHA"*, sees tenants on `2863380`, and fails on **strict SHA equality** — even though the fleet is **AHEAD**, not behind. Git SHAs aren't ordered and `/buildinfo` exposes only `git_sha` (no build time / monotonic number — see `workspace-server/internal/router/router.go` + `internal/buildinfo/buildinfo.go`), so the verify can't distinguish "ahead" from "behind" on its own. ## Fix — option (b), superseded-job detection Before the strict verify, ask Gitea for the current head of the deploy branch (`main`). If main's head is no longer this job's `GITHUB_SHA`, a newer commit has landed and this deploy is **superseded** — the newest deploy job's verify is the authoritative one. The superseded job logs a `::notice::` and exits success, skipping the strict-equality loop. - New `superseded_by()` / `current_branch_head()` in `.gitea/scripts/prod-auto-deploy.py` + `check-superseded` subcommand (exit `0` = superseded/print newer SHA, exit `10` = still latest → run strict verify). - Workflow `Verify reachable tenants report this SHA` step calls it first and short-circuits to success only when superseded. ## Why it preserves real-stale detection - Only the **superseded (older)** job skips the strict verify. The **latest** deploy job (head == its SHA) still runs strict equality, so a genuinely behind/older tenant still **fails loudly**. - **Fail-safe:** if the branch head can't be read (no token / API error) or equals our SHA, `superseded_by` returns `None` → strict verify runs. An unreadable head never silently greens a deploy. ## Why not (a)/(c) - **(a)** build-timestamp / monotonic compare: `/buildinfo` returns only `{git_sha}`. Adding a build-time field requires a workspace-server binary + `Dockerfile.tenant` change and a full fleet rebuild before it could be relied on — heavy and slow to take effect. - **(c)** `concurrency:` — forbidden by the workflow header (Gitea cancels queued prod deploys); cannot serialize-without-cancel safely. ## Verification - New unit tests for `superseded_by` / `current_branch_head` incl. fail-safe + short-vs-full SHA prefix; **33 passed** (`pytest .gitea/scripts/tests/test_prod_auto_deploy.py`). - Workflow yaml-lint clean (`lint-workflow-yaml.py`, 56 files, 0 warnings). - CLI smoke test of the exact `eb31bcf`-vs-`2863380` scenario: superseded → exit `0` (skip, success); latest job → exit `10` (run strict verify); unreadable head → exit `10`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-04 02:34:58 +00:00

fix(ci): superseded prod-deploy job no longer false-reds as "stale" ... 450fedac9b

publish-workspace-server-image / Production auto-deploy intermittently
fails on main with:

    ::error::<slug> is stale: actual=<newerSHA>, expected=<thisSHA>

Root cause: the workflow deliberately has no `concurrency:` (Gitea
1.22.6 cancels queued runs even with cancel-in-progress:false, which is
unacceptable for a prod deploy). So when two main pushes land close
together (eb31bcf then 286338), BOTH deploy-production jobs run. The
newer job (286338 -> staging-2863380) rolls the fleet forward first;
then the OLDER job (eb31bcf) runs "Verify reachable tenants report this
SHA", sees tenants on 2863380, and fails on STRICT SHA EQUALITY — even
though the fleet is AHEAD, not behind. Git SHAs aren't ordered and
/buildinfo exposes only git_sha (no build time / monotonic number), so
the verify can't tell "ahead" from "behind" on its own.

Fix (option b — superseded-job detection): before the strict verify,
ask Gitea for the current head of the deploy branch (main). If it is no
longer this job's GITHUB_SHA, a newer commit has landed and this deploy
is superseded; the newest job's verify is authoritative. Log a notice
and exit success, skipping strict equality for the stale job.

Why this preserves real-stale detection:
- Only the SUPERSEDED (older) job skips strict verify. The LATEST deploy
  job (head == its SHA) still runs strict equality, so a genuinely
  behind/older tenant still fails loudly.
- Fail-safe: if the branch head can't be read (no token / API error) or
  equals our SHA, superseded_by returns None -> strict verify runs. An
  unreadable head never silently greens a deploy.

Why not the alternatives:
- (a) build-timestamp/monotonic compare: /buildinfo returns only
  {git_sha} (router.go, buildinfo.go). Adding a build-time field needs a
  workspace-server binary + Dockerfile change and a full fleet rebuild
  before it can be relied on — heavy and slow to take effect.
- (c) concurrency: forbidden by the workflow header (Gitea cancels
  queued prod deploys).

Verification:
- New unit tests for superseded_by / current_branch_head and the
  fail-safe path; full suite 33 passed.
- Workflow yaml-lint clean (lint-workflow-yaml.py).
- CLI smoke test: eb31bcf-vs-2863380 -> exit 0 (skip, success);
  latest job -> exit 10 (run strict verify); unreadable head -> exit 10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-ceo-assistant merged commit 5f0351c59f into main 2026-06-04 02:39:59 +00:00

core-devops commented 2026-06-04 02:40:05 +00:00

Author

Member

Copy Link

Copy Source

Owner force-merged (claude-ceo-assistant), honest bypass. Fixes the Production auto-deploy FALSE RED: superseded older deploy jobs no longer fail strict-SHA verify against a fleet that moved AHEAD (newer build) — superseded-job detection skips verify on non-latest jobs while the latest job still runs strict verify (real-stale detection preserved). 33 unit tests, yaml-lint clean. All required CI green. Token revoked.

Owner force-merged (claude-ceo-assistant), honest bypass. Fixes the Production auto-deploy FALSE RED: superseded older deploy jobs no longer fail strict-SHA verify against a fleet that moved AHEAD (newer build) — superseded-job detection skips verify on non-latest jobs while the latest job still runs strict verify (real-stale detection preserved). 33 unit tests, yaml-lint clean. All required CI green. Token revoked.

core-devops referenced this issue from a commit 2026-06-04 05:57:29 +00:00

fix(ci): block superseded prod-deploy from rolling the fleet backward + settle /buildinfo (#2213)

core-devops referenced this pull request 2026-06-04 05:57:54 +00:00

fix(ci): block superseded prod-deploy from rolling the fleet backward + settle /buildinfo (#2213) #2215

core-devops referenced this pull request 2026-06-04 06:19:23 +00:00

fix(ci): block superseded prod-deploy from rolling the fleet backward + settle /buildinfo (#2213) #2215

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2194