molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 20 Actions Packages Projects Releases Wiki Activity

CI/CD hardening WS2: fix CI fan-out — consolidate workflows + needs:-aggregator gate (make #2094 the standard) #2095

New Issue
Open
opened 2026-06-01 07:54:00 +00:00 by core-be · 1 comment
core-be commented 2026-06-01 07:54:00 +00:00
Member
Copy Link
Copy Source

Workstream 2 of the CI/CD + agent-fleet hardening RFC (molecule-ai/internal#749).

Problem (2026-06-01 incident)

~57 workflow files generating ~65 runs/PR overwhelm the Gitea Actions scheduler. The all-required poll-gate squats a runner slot up to 40 min/PR doing nothing but polling the statuses API — amplifying contention during the outage.

Proposed solution (make #2094 the org standard)

  • Consolidate workflows (collapse sub-second sibling lints into one run).
  • paths-gate / lazy-trigger heavy non-required advisory jobs.
  • needs:-aggregator required gate, NOT a poll-gate — plain needs: works on Gitea 1.22.6 / act_runner v0.6.1 (feedback_gitea_needs_works_only_ifalways_broken); if: always() aggregators land with the 1.26 upgrade (RFC ci-cd-elastic-runners-and-native-needs). Apply the #2094 pattern across every repo mirroring the sentinel.

Links

  • RFC PR: molecule-ai/internal#749
  • molecule-core #2094 — cut fan-out + stop the poll-gate slot-squat (this workstream starts here)
  • Sibling: internal WS1 #750 (runner-host split unblocks slot pressure), WS3 (alerting), WS4/WS5 (internal).
**Workstream 2 of the CI/CD + agent-fleet hardening RFC** (https://git.moleculesai.app/molecule-ai/internal/pulls/749). ## Problem (2026-06-01 incident) ~57 workflow files generating ~65 runs/PR overwhelm the Gitea Actions scheduler. The `all-required` **poll-gate squats a runner slot up to 40 min/PR** doing nothing but polling the statuses API — amplifying contention during the outage. ## Proposed solution (make #2094 the org standard) - **Consolidate workflows** (collapse sub-second sibling lints into one run). - **paths-gate / lazy-trigger** heavy non-required advisory jobs. - **`needs:`-aggregator required gate, NOT a poll-gate** — plain `needs:` works on Gitea 1.22.6 / act_runner v0.6.1 (`feedback_gitea_needs_works_only_ifalways_broken`); `if: always()` aggregators land with the 1.26 upgrade (RFC `ci-cd-elastic-runners-and-native-needs`). Apply the #2094 pattern across every repo mirroring the sentinel. ## Links - RFC PR: https://git.moleculesai.app/molecule-ai/internal/pulls/749 - molecule-core #2094 — cut fan-out + stop the poll-gate slot-squat (this workstream starts here) - Sibling: internal WS1 #750 (runner-host split unblocks slot pressure), WS3 (alerting), WS4/WS5 (internal).
core-be commented 2026-06-01 07:54:36 +00:00
Author
Member
Copy Link
Copy Source

CI/CD + agent-fleet hardening — workstream map

RFC PR: internal#749 — molecule-ai/internal#749

WS Title Issue Existing work
1 Separate CI execution plane from SCM/DB control plane internal#750 internal#91, operator-config#143, operator-config#144
2 CI fan-out architecture molecule-core#2095 molecule-core#2094
3 CI observability / alerting internal#751 —
4 Agent capability-tiering + tooling internal#752 —
5 Trustworthy merge-gate / governance internal#753 —

All five trace to the 2026-06-01 incident: one un-throttled backup on the single Gitea+Postgres+runners box took all CI red ~40 min with no alert (WS1–3), and the agent fleet failed review/RCA — cheap models 0/6 false, capable reviewers sandboxed without tooling (WS4–5).

### CI/CD + agent-fleet hardening — workstream map RFC PR: **internal#749** — https://git.moleculesai.app/molecule-ai/internal/pulls/749 | WS | Title | Issue | Existing work | |----|-------|-------|---------------| | 1 | Separate CI execution plane from SCM/DB control plane | internal#750 | internal#91, operator-config#143, operator-config#144 | | 2 | CI fan-out architecture | molecule-core#2095 | molecule-core#2094 | | 3 | CI observability / alerting | internal#751 | — | | 4 | Agent capability-tiering + tooling | internal#752 | — | | 5 | Trustworthy merge-gate / governance | internal#753 | — | All five trace to the **2026-06-01 incident**: one un-throttled backup on the single Gitea+Postgres+runners box took all CI red ~40 min with no alert (WS1–3), and the agent fleet failed review/RCA — cheap models 0/6 false, capable reviewers sandboxed without tooling (WS4–5).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2095
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?