ci(publish-image): also tag :staging-latest so CP auto-picks up new builds
Root cause of the 2026-04-24 all-day E2E failure chain: Railway staging
CP had TENANT_IMAGE pinned to :staging-a14cf86 — a static SHA that had
silently drifted 10+ days stale. Every new tenant (including every E2E
run's fresh tenant) was spawned with that stale image, which predated
applyRuntimeModelEnv. Without applyRuntimeModelEnv, HERMES_DEFAULT_MODEL
never reached the workspace EC2 user-data, so install.sh fell back to
nousresearch/hermes-4-70b → openrouter → 401 "Missing Authentication
header" in every A2A reply.
Four correct fixes shipped today all got shadowed by this single stale
pin:
• template-hermes#19 (provider priority for openai/*)
• template-hermes#20 (decouple prefix-strip from bridge guard)
• molecule-controlplane#247 (force fresh /opt/adapter clone)
• molecule-core#1987 (E2E pins HERMES_CUSTOM_* as workaround)
Fix: publish each main build under both :staging-<sha> AND :staging-latest.
Change Railway staging CP's TENANT_IMAGE env to :staging-latest (done via
`railway variables --set` as part of this incident). Future main builds
then auto-propagate to new tenant provisions without any human in the
loop.
Safety: :staging-latest is the "most recent main build" — NOT a
canary-verified promotion. That distinction is preserved:
• Prod tenants still pull :latest (canary-verified, retagged by
canary-verify.yml only after the canary fleet green-lights a digest)
• Staging tenants now pull :staging-latest (every main build, pre-canary)
So staging becomes the canary: if a :staging-latest build regresses,
the staging canary fleet catches it before it can be promoted to :latest
for prod. This is what the canary design intended; the missing
:staging-latest tag was the hole.
Zero impact on image size / build time: Docker tags point at the same
digest, no duplicate push.
Follow-up: filed an issue tracking the need for CP's TENANT_IMAGE to
NEVER be pinned to a SHA in any environment — it must always float on a
named tag (:staging-latest for staging, :latest for prod).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a5a054e861
commit
24bfced630
@ -73,7 +73,20 @@ jobs:
|
||||
# - canary-verify.yml runs smoke tests against them
|
||||
# - On green → canary-verify retags :staging-<sha> → :latest
|
||||
# - On red → :latest stays on the prior good digest, prod is safe
|
||||
- name: Build & push platform image to GHCR (staging-<sha> only)
|
||||
# Every push of :staging-<sha> also retags the same digest as
|
||||
# :staging-latest so staging CP (which pins TENANT_IMAGE at
|
||||
# :staging-latest) picks up new builds automatically — no more manual
|
||||
# Railway env-var edits. Prod's :latest retag still happens in
|
||||
# canary-verify.yml after the canary fleet greenlights this digest;
|
||||
# :staging-latest is strictly the "most recent main build," not a
|
||||
# canary-verified promotion.
|
||||
#
|
||||
# Before this, TENANT_IMAGE on Railway staging was pinned to a static
|
||||
# :staging-<sha> and drifted months behind (2026-04-24 incident:
|
||||
# canary tenant ran :staging-a14cf86, 10 days stale, which lacked
|
||||
# applyRuntimeModelEnv and caused every E2E to route hermes+openai
|
||||
# through openrouter → 401). See issue filed with this PR.
|
||||
- name: Build & push platform image to GHCR (staging-<sha> + staging-latest)
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
@ -82,6 +95,7 @@ jobs:
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.IMAGE_NAME }}:staging-${{ steps.tags.outputs.sha }}
|
||||
${{ env.IMAGE_NAME }}:staging-latest
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
labels: |
|
||||
@ -89,7 +103,7 @@ jobs:
|
||||
org.opencontainers.image.revision=${{ github.sha }}
|
||||
org.opencontainers.image.description=Molecule AI platform (Go API server) — pending canary verify
|
||||
|
||||
- name: Build & push tenant image to GHCR (staging-<sha> only)
|
||||
- name: Build & push tenant image to GHCR (staging-<sha> + staging-latest)
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
@ -98,6 +112,7 @@ jobs:
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.TENANT_IMAGE_NAME }}:staging-${{ steps.tags.outputs.sha }}
|
||||
${{ env.TENANT_IMAGE_NAME }}:staging-latest
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
# Canvas uses same-origin fetches. The tenant Go platform
|
||||
|
||||
Loading…
Reference in New Issue
Block a user