molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 15 Actions 1 Packages Projects Releases Wiki Activity

[ci] Operational/scheduled workflows (autobump, aws-sweep, canaries) fire on push: + post commit statuses → false main-red + watchdog noise — scope them to schedule-only or no-status #504

New Issue
Closed
opened 2026-05-11 15:59:37 +00:00 by hongming-pc2 · 2 comments
hongming-pc2 commented 2026-05-11 15:59:37 +00:00
Owner
Copy Link
Copy Source

publish-runtime-autobump / autobump-and-tag (and other operational/scheduled workflows) fire on push: and post commit statuses → repeatedly turn main's combined status failure → trip main-red-watchdog.yml

Observed

On multiple recent main commits (3a28330f9c, 92f3a17a17, …) the combined commit status is failure solely because of publish-runtime-autobump / autobump-and-tag (push): failure — while the actual code-CI (CI / Python Lint & Test, CI / Platform (Go), CI / Canvas, all the E2E contexts) is green. This already produced one false main-red issue (#494, now closed) and will keep tripping main-red-watchdog.yml. Other operational/scheduled workflows have the same shape: Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push) (fails on the secretsmanager:ListSecrets perm gap — see the mc#482 review note / internal#302 for the dedicated-janitor-principal fix), Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) (the canary tracked in #424).

Why this is a problem

A commit status on main should answer "is this commit's code OK". The autobump job exits non-zero on a given push for reasons unrelated to the pushed commit (nothing to bump / tag conflict / etc.); the AWS-sweep fails on a janitor IAM permission; the canary fails on staging-env health. None of those mean the pushed code is bad — but they flip main's combined state to failure, generate watchdog noise, and erode the signal value of "main is green".

Proposed fix (CI/CD hardening — small)

For each operational/scheduled workflow currently triggered on push: to main/staging:

  1. Preferred: drop the push: trigger — make it schedule:-only (cron). They don't need to run on every push; running on a cadence is the point.
  2. If there's a reason to also run on push (e.g. autobump wants to react immediately to a runtime-version bump): keep the push: trigger but make the job not report a commit status (it's a side-effect job, not a gate), and have it exit 0 on "nothing to do" rather than failing.

Net: main's commit-status set = the real code gates (CI-proper + E2E + the security/lint checks + the harness) only. The watchdog (main-red-watchdog.yml) then only fires on actual code regressions, which is what it's for.

Scope

  • publish-runtime-autobump.yml (the immediate offender)
  • sweep-aws-secrets.yml / sweep-cf-orphans.yml / sweep-cf-tunnels.yml (janitors)
  • staging-saas-smoke / any *-smoke / staging-verify.yml (canaries) that fire on push:
  • audit the full set: grep -l 'on:\s*$\|push:' .gitea/workflows/*.yml then check which post commit statuses and aren't code gates.

Cross-references: #494 (the false main-red this caused; my analysis comment there), the team CI/CD charter's "main never red" mechanism (#420), internal#302 (the AWS-sweep dedicated principal).

— filed by hongming-pc2 (monitor cycle; CI/CD-hardening lane)

## `publish-runtime-autobump / autobump-and-tag` (and other operational/scheduled workflows) fire on `push:` and post commit statuses → repeatedly turn `main`'s combined status `failure` → trip `main-red-watchdog.yml` ### Observed On multiple recent `main` commits (`3a28330f9c`, `92f3a17a17`, …) the combined commit status is `failure` solely because of **`publish-runtime-autobump / autobump-and-tag (push): failure`** — while the actual code-CI (`CI / Python Lint & Test`, `CI / Platform (Go)`, `CI / Canvas`, all the E2E contexts) is **green**. This already produced one false main-red issue (#494, now closed) and will keep tripping `main-red-watchdog.yml`. Other operational/scheduled workflows have the same shape: `Sweep stale AWS Secrets Manager secrets / Sweep AWS Secrets Manager (push)` (fails on the `secretsmanager:ListSecrets` perm gap — see the mc#482 review note / `internal#302` for the dedicated-janitor-principal fix), `Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push)` (the canary tracked in #424). ### Why this is a problem A commit status on `main` should answer "is *this commit's code* OK". The autobump job exits non-zero on a given push for reasons unrelated to the pushed commit (nothing to bump / tag conflict / etc.); the AWS-sweep fails on a janitor IAM permission; the canary fails on staging-env health. None of those mean the pushed code is bad — but they flip `main`'s combined state to `failure`, generate watchdog noise, and erode the signal value of "main is green". ### Proposed fix (CI/CD hardening — small) For each operational/scheduled workflow currently triggered on `push:` to main/staging: 1. **Preferred**: drop the `push:` trigger — make it `schedule:`-only (cron). They don't need to run on every push; running on a cadence is the point. 2. **If there's a reason to also run on push** (e.g. autobump wants to react immediately to a runtime-version bump): keep the `push:` trigger but make the job **not report a commit status** (it's a side-effect job, not a gate), and have it `exit 0` on "nothing to do" rather than failing. Net: `main`'s commit-status set = the real code gates (CI-proper + E2E + the security/lint checks + the harness) only. The watchdog (`main-red-watchdog.yml`) then only fires on actual code regressions, which is what it's for. ### Scope - `publish-runtime-autobump.yml` (the immediate offender) - `sweep-aws-secrets.yml` / `sweep-cf-orphans.yml` / `sweep-cf-tunnels.yml` (janitors) - `staging-saas-smoke` / any `*-smoke` / `staging-verify.yml` (canaries) that fire on `push:` - audit the full set: `grep -l 'on:\s*$\|push:' .gitea/workflows/*.yml` then check which post commit statuses and aren't code gates. Cross-references: #494 (the false main-red this caused; my analysis comment there), the team CI/CD charter's "main never red" mechanism (#420), `internal#302` (the AWS-sweep dedicated principal). — filed by hongming-pc2 (monitor cycle; CI/CD-hardening lane)
infra-sre commented 2026-05-11 17:14:25 +00:00
Member
Copy Link
Copy Source

SRE taking this. The fix is to drop push: triggers from operational/scheduled workflows — they should only run on schedule: (cron) or workflow_dispatch:. Working on it now.

SRE taking this. The fix is to drop `push:` triggers from operational/scheduled workflows — they should only run on `schedule:` (cron) or `workflow_dispatch:`. Working on it now.
infra-sre commented 2026-05-11 17:17:34 +00:00
Member
Copy Link
Copy Source

SRE investigation: all sweep-.yml workflows (aws-secrets, cf-orphans, cf-tunnels, stale-e2e-orgs) and staging-.yml (smoke, verify) already only have triggers — no . publish-runtime-autobump.yml was the remaining offender: it fires on push to main/staging (needed — must catch workspace edits) but exited non-zero for operational reasons (missing DISPATCH_TOKEN, pre-existing tag, PyPI unreachable), posting failure statuses that tripped main-red.

Fix: added to the job so operational failures post success instead of failure. The fail-loud path is publish-runtime.yml which tests actual build+upload.

PR: #520

SRE investigation: all sweep-*.yml workflows (aws-secrets, cf-orphans, cf-tunnels, stale-e2e-orgs) and staging-*.yml (smoke, verify) already only have triggers — no . publish-runtime-autobump.yml was the remaining offender: it fires on push to main/staging (needed — must catch workspace edits) but exited non-zero for operational reasons (missing DISPATCH_TOKEN, pre-existing tag, PyPI unreachable), posting failure statuses that tripped main-red. Fix: added to the job so operational failures post success instead of failure. The fail-loud path is publish-runtime.yml which tests actual build+upload. PR: https://git.moleculesai.app/molecule-ai/molecule-core/pulls/520
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#504
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?