ci: reduce scheduled runner load and prep prebaked browsers #1628

Merged

hongming merged 1 commits from fix/ci-cron-bots-prebake-1357 into main 2026-05-21 03:17:12 +00:00

Conversation 2 Commits 1 Files Changed 6

+42 -19

core-devops commented 2026-05-21 00:42:17 +00:00

Member

Copy Link

Copy Source

SOP: dev-sop.md evidence-based change for molecule-core#1357. Companion operator-config PR: molecule-ai/operator-config#117.

Problem

• CI is saturated by scheduled workflows and heavyweight runtime/browser setup.
• Last 24h investigation showed continuous-synth-e2e around 10 runner-hours/day, status-reaper around 3.6 runner-hours/day, and browser/E2E PR lanes with long waits.
• Canvas CI was deleting package-lock.json and running npm install, defeating deterministic install/cache behavior.

Change

• Remove Actions schedules from status-reaper and gitea-merge-queue; keep workflow_dispatch for manual fallback.
• Lower continuous synth E2E cadence from every 10 minutes to every 30 minutes.
• Make Playwright browser install steps prebake-aware: use /ms-playwright when present, otherwise fall back to install.
• Replace Canvas npm install with npm ci --include=optional --prefer-offline.

Verification

• python3 .gitea/scripts/lint-workflow-yaml.py --workflow-dir .gitea/workflows
• python3 -m pytest .gitea/scripts/tests/test_status_reaper_api.py .gitea/scripts/tests/test_gitea_merge_queue.py -q

Rollback

• Revert this PR to restore the original workflow schedules and install commands.
• If operator cron is already deployed, disable /etc/cron.d/molecule-core-* via the kill-switch files in operator-config#117 before restoring Actions schedules to avoid double-running bots.

Expected saving

• status-reaper: about 3.6 runner-hours/day removed from Actions queue once operator cron is deployed.
• continuous synth: roughly 6.7 runner-hours/day saved by moving 6 fires/hour to 2 fires/hour, based on the observed ~10 runner-hours/day 10-minute cadence.
• Playwright prebake: expected 2-8 minutes saved per browser E2E job on runners with /ms-playwright, with fallback preserving current behavior.
• Canvas npm ci: expected lower variance and usually 1-3 minutes saved versus deleting the lockfile and reinstalling.

SOP: dev-sop.md evidence-based change for molecule-core#1357. Companion operator-config PR: molecule-ai/operator-config#117. Problem - CI is saturated by scheduled workflows and heavyweight runtime/browser setup. - Last 24h investigation showed continuous-synth-e2e around 10 runner-hours/day, status-reaper around 3.6 runner-hours/day, and browser/E2E PR lanes with long waits. - Canvas CI was deleting package-lock.json and running npm install, defeating deterministic install/cache behavior. Change - Remove Actions schedules from status-reaper and gitea-merge-queue; keep workflow_dispatch for manual fallback. - Lower continuous synth E2E cadence from every 10 minutes to every 30 minutes. - Make Playwright browser install steps prebake-aware: use /ms-playwright when present, otherwise fall back to install. - Replace Canvas npm install with npm ci --include=optional --prefer-offline. Verification - python3 .gitea/scripts/lint-workflow-yaml.py --workflow-dir .gitea/workflows - python3 -m pytest .gitea/scripts/tests/test_status_reaper_api.py .gitea/scripts/tests/test_gitea_merge_queue.py -q Rollback - Revert this PR to restore the original workflow schedules and install commands. - If operator cron is already deployed, disable /etc/cron.d/molecule-core-* via the kill-switch files in operator-config#117 before restoring Actions schedules to avoid double-running bots. Expected saving - status-reaper: about 3.6 runner-hours/day removed from Actions queue once operator cron is deployed. - continuous synth: roughly 6.7 runner-hours/day saved by moving 6 fires/hour to 2 fires/hour, based on the observed ~10 runner-hours/day 10-minute cadence. - Playwright prebake: expected 2-8 minutes saved per browser E2E job on runners with /ms-playwright, with fallback preserving current behavior. - Canvas npm ci: expected lower variance and usually 1-3 minutes saved versus deleting the lockfile and reinstalling.

core-devops added 1 commit 2026-05-21 00:42:18 +00:00

ci: reduce scheduled runner load and prep prebaked browsers a9bc5e39d5

core-devops referenced this pull request 2026-05-21 00:42:37 +00:00

ci(scheduled-workflows): cancel_in_progress=false causing runner pool saturation, blocking all PRs #1357

hongming approved these changes 2026-05-21 03:17:02 +00:00

hongming left a comment

Owner

Copy Link

Copy Source

Approved via gitea-token per Hongming explicit bypass request.

Approved via gitea-token per Hongming explicit bypass request.

devops-engineer approved these changes 2026-05-21 03:17:03 +00:00

devops-engineer left a comment

Member

Copy Link

Copy Source

Approved via persona-devops-engineer-token per Hongming explicit bypass request.

Approved via persona-devops-engineer-token per Hongming explicit bypass request.

hongming merged commit c58ffd2828 into main 2026-05-21 03:17:12 +00:00

hongming referenced this issue from a commit 2026-05-21 03:17:14 +00:00

Merge pull request 'ci: reduce scheduled runner load and prep prebaked browsers' (#1628) from fix/ci-cron-bots-prebake-1357 into main

Sign in to join this conversation.

Reviewers

No Reviewers

hongming

devops-engineer

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1628