molecule-ai/molecule-core
52
0
Fork 2
Code Issues 180 Pull Requests 21 Actions 8 Packages Projects Releases Wiki Activity

[main-red] production auto-deploy leaves hongming stale after redeploy-fleet #2213

New Issue
Closed
opened 2026-06-04 05:42:34 +00:00 by molecule-code-reviewer · 0 comments
molecule-code-reviewer commented 2026-06-04 05:42:34 +00:00
Member
Copy Link
Copy Source

MECHANISM: molecule-core main 7f25373309ec now has a distinct production auto-deploy red after image build. .gitea/workflows/publish-workspace-server-image.yml:251-302 calls production redeploy-fleet; .gitea/workflows/publish-workspace-server-image.yml:304-356 then verifies each returned tenant /buildinfo equals GITHUB_SHA. That verification failed for the canary/default tenant hongming: redeploy did not leave every reachable tenant on the target tag even though the rest of the returned tenants reported the new SHA.

EVIDENCE: run 206737, job 275427, context publish-workspace-server-image / Production auto-deploy (push), head 7f25373309ec. Log excerpt: hongming is stale: actual=7a72516, expected=7f25373. Same log shows agents-team, chloe-dong, go-to-market, jrs-auto, molecule-adk-demo, reno-stars, and x9-ai reporting 7f25373, so this is not a repo-wide image-build failure.

RECOMMENDED FIX SHAPE: investigate the production CP redeploy-fleet path and canary tenant hongming rollout handling, not molecule-core application code. Responsible surface is production deploy orchestration: .gitea/workflows/publish-workspace-server-image.yml plus the controlplane redeploy-fleet implementation/tenant selection. Minimal repro is re-run/query redeploy-fleet for target staging-7f25373, then compare https://hongming.moleculesai.app/buildinfo against another tenant that already reports 7f25373.

MECHANISM: molecule-core main `7f25373309ec` now has a distinct production auto-deploy red after image build. `.gitea/workflows/publish-workspace-server-image.yml:251-302` calls production `redeploy-fleet`; `.gitea/workflows/publish-workspace-server-image.yml:304-356` then verifies each returned tenant `/buildinfo` equals `GITHUB_SHA`. That verification failed for the canary/default tenant `hongming`: redeploy did not leave every reachable tenant on the target tag even though the rest of the returned tenants reported the new SHA. EVIDENCE: run `206737`, job `275427`, context `publish-workspace-server-image / Production auto-deploy (push)`, head `7f25373309ec`. Log excerpt: `hongming is stale: actual=7a72516, expected=7f25373`. Same log shows `agents-team`, `chloe-dong`, `go-to-market`, `jrs-auto`, `molecule-adk-demo`, `reno-stars`, and `x9-ai` reporting `7f25373`, so this is not a repo-wide image-build failure. RECOMMENDED FIX SHAPE: investigate the production CP redeploy-fleet path and canary tenant `hongming` rollout handling, not molecule-core application code. Responsible surface is production deploy orchestration: `.gitea/workflows/publish-workspace-server-image.yml` plus the controlplane `redeploy-fleet` implementation/tenant selection. Minimal repro is re-run/query redeploy-fleet for target `staging-7f25373`, then compare `https://hongming.moleculesai.app/buildinfo` against another tenant that already reports `7f25373`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2213
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?