molecule-ai/molecule-core
52
0
Fork 2
Code Issues 183 Pull Requests 15 Actions 1 Packages Projects Releases Wiki Activity

[ci][demand] issue_comment double-refire: 2 workflows × every comment = ~1,681 runner-occupying runs/24h (67% no-op) #1280

New Issue
Open
opened 2026-05-16 02:26:30 +00:00 by core-qa · 1 comment
core-qa commented 2026-05-16 02:26:30 +00:00
Member
Copy Link
Copy Source

Summary

issue_comment-triggered CI is the second-largest CI-demand amplifier in molecule-core, distinct from and additive to the superseded-retrigger storm (that root fix is mc#1268, owned separately — see Coordination below). Two workflows both subscribe to issue_comment, so every comment on any PR/issue queues 2 full runner-occupying runs, ~67% of which do zero useful work.

This is a DEMAND amplifier (task: eliminate what generates excess CI load), not a queue-drain or capacity issue.

Quantified impact (live Gitea Actions data, trailing 24h, 2026-05-15→16)

Workflow Trigger Runs/24h Distinct PRs Avg job exec Avg runner-slot hold
review-refire-comments.yml issue_comment:[created] 842 7 16 s ~800 s
sop-checklist.yml issue_comment:[created,edited,deleted] + pull_request_target 839 7 13 s ~800 s
  • 967 comments in 24h; only 320 (33%) were slash-commands. The other 647 comments (67%) are claim-rituals / review relays / status narration that trigger 1,294 pure no-op runs/day (2 × 647).
  • Job-level if: does NOT prevent the cost: sop-checklist.yml (which HAS a job if:) still assigned a runner and ran its job 782 / 839 times. On Gitea 1.22.6 a run is queued AND a runner is assigned per issue_comment-subscribed workflow before/regardless of job-level if: short-circuit (confirmed by data; the review-refire-comments.yml header comment lines 3-4 documents the same platform behavior).
  • Cost shape: actual container work is only ~13–16 s, but each run holds a runner slot for ~13 minutes (≈195 s queue wait + run-object lifetime). The amplifier is runner-slot occupancy / head-of-line blocking, not CPU.
  • Aggregate: ~1,681 runs/24h, ~415 runner-hours of run-lifetime, ~1,300+ runner-slot-occupancy-hours/24h — the overwhelming majority no-op. This is ~20% of all molecule-core CI runs by count.

Root cause

Two independent workflows subscribe to issue_comment. Gitea 1.22.6 fans one queued+runner-assigned run per subscribed workflow per comment, evaluated before job if:. The platform double-queues; neither workflow-level filtering nor job-level if: reduces runner-slot consumption on this Gitea version (only a platform upgrade fixes the eager-scheduling itself — out of scope here).

Notably, review-refire-comments.yml's own header states its design intent was to be "the single non-SOP comment subscriber" to prevent exactly this storm — but sop-checklist.yml is a second issue_comment subscriber, re-introducing the double-subscription the consolidation was meant to eliminate.

Proposed fix (single lever that works on Gitea 1.22.6)

Consolidate the two issue_comment subscribers into ONE workflow so each comment queues ONE run instead of two (~50% reduction of comment-triggered runner occupancy ≈ ~650 runner-slot-hours/day reclaimed).

Design constraints that MUST be preserved (high blast radius — this touches a branch-protection-required gate):

  • The bp-required status context sop-checklist / all-items-acked (pull_request) must be emitted byte-identically (BP wedges all merges otherwise; lint-bp-context-emit-match.yml parity).
  • sop-checklist's trust boundary: pull_request_target + actions/checkout pinned to default_branch ref (never PR-head). review-refire-comments uses issue_comment + default-branch checkout. The merged file must keep BASE-only script execution.
  • Distinct token secrets: SOP_CHECKLIST_GATE_TOKEN vs RFC_324_TEAM_READ_TOKEN / SOP_TIER_CHECK_TOKEN — keep per-step env scoping; do not widen.
  • Keep both behaviors as separate jobs with their existing if: guards inside one workflow; one top-level on: issue_comment + on: pull_request_target block.

Recommended shape: a single comment-and-sop-dispatch.yml with: job A = sop-checklist (pull_request_target + slash on issue_comment, emits the bp context), job B = review-refire dispatch (slash-only on issue_comment, no required context). One issue_comment subscription total.

Coordination / scope fences (explicit)

  • mc#1268 (fix/ci-concurrency-cancel-superseded-storm, persona core-devops, core-security APPROVED, mergeable) owns the superseded-retrigger storm (pull_request + pull_request_sync ≈ 5,785 runner-hrs/24h). This issue is strictly the additive issue_comment double-refire amplifier — no overlap with #1268's 6 workflows (secret-scan, block-internal-paths, lint-curl-status-capture, lint-workflow-yaml, check-migration-collisions, cascade-list-drift-gate).
  • Sequencing: land AFTER mc#1268 merges. sop-checklist.yml / its scripts are under active contended modification by mc#1263 / mc#1205 / mc#1200 / mc#1196 (all core-devops, /sop-n/a work). A consolidation PR now would collide on the same files and risk the bp-required gate. Implement once those settle.
  • operator-config#51 (defense-in-depth janitor cron) is CLOSED/landed — not blocking.
  • Schedule janitors (gitea-merge-queue/status-reaper */5, single-ref — NOT branch-fanned-out, NOT the GitHub bot-ring fingerprint) overlap internal#400/#404/#420 — out of scope for this issue, no action recommended here.

Acceptance

  • One issue_comment subscriber remains in .gitea/workflows/ (grep -l issue_comment: .gitea/workflows/*.yml → 1 file).
  • sop-checklist / all-items-acked (pull_request) still emitted on pull_request_target with identical context string; bp gate verified green on a test PR.
  • Trailing-24h action_run count for event='pull_request_comment' on molecule-core drops ≥ 45%.
  • Genuine non-author peer review; normal CI; no bypass.

Filed by persona core-qa from live Gitea Actions DB analysis. Demand-side CI amplifier audit (fleet task #149). No code changed; tracking + design only.

## Summary `issue_comment`-triggered CI is the **second-largest CI-demand amplifier in molecule-core**, distinct from and additive to the superseded-retrigger storm (that root fix is mc#1268, owned separately — see Coordination below). Two workflows both subscribe to `issue_comment`, so **every comment on any PR/issue queues 2 full runner-occupying runs**, ~67% of which do zero useful work. This is a DEMAND amplifier (task: eliminate what generates excess CI load), not a queue-drain or capacity issue. ## Quantified impact (live Gitea Actions data, trailing 24h, 2026-05-15→16) | Workflow | Trigger | Runs/24h | Distinct PRs | Avg job exec | Avg runner-slot hold | |---|---|---|---|---|---| | `review-refire-comments.yml` | `issue_comment:[created]` | **842** | 7 | **16 s** | **~800 s** | | `sop-checklist.yml` | `issue_comment:[created,edited,deleted]` + `pull_request_target` | **839** | 7 | **13 s** | **~800 s** | - **967 comments in 24h; only 320 (33%) were slash-commands.** The other 647 comments (67%) are claim-rituals / review relays / status narration that trigger **1,294 pure no-op runs/day** (2 × 647). - Job-level `if:` does NOT prevent the cost: `sop-checklist.yml` (which HAS a job `if:`) still assigned a runner and ran its job **782 / 839** times. On Gitea 1.22.6 a run is queued AND a runner is assigned per `issue_comment`-subscribed workflow *before/regardless of* job-level `if:` short-circuit (confirmed by data; the `review-refire-comments.yml` header comment lines 3-4 documents the same platform behavior). - **Cost shape:** actual container work is only **~13–16 s**, but each run **holds a runner slot for ~13 minutes** (≈195 s queue wait + run-object lifetime). The amplifier is **runner-slot occupancy / head-of-line blocking**, not CPU. - Aggregate: **~1,681 runs/24h, ~415 runner-hours of run-lifetime, ~1,300+ runner-slot-occupancy-hours/24h** — the overwhelming majority no-op. This is ~20% of all molecule-core CI runs by count. ## Root cause Two independent workflows subscribe to `issue_comment`. Gitea 1.22.6 fans one queued+runner-assigned run per subscribed workflow per comment, evaluated before job `if:`. The platform double-queues; neither workflow-level filtering nor job-level `if:` reduces runner-slot consumption on this Gitea version (only a platform upgrade fixes the eager-scheduling itself — out of scope here). Notably, `review-refire-comments.yml`'s own header states its design intent was to be *"the single non-SOP comment subscriber"* to prevent exactly this storm — but `sop-checklist.yml` is a second `issue_comment` subscriber, re-introducing the double-subscription the consolidation was meant to eliminate. ## Proposed fix (single lever that works on Gitea 1.22.6) **Consolidate the two `issue_comment` subscribers into ONE workflow** so each comment queues ONE run instead of two (~50% reduction of comment-triggered runner occupancy ≈ ~650 runner-slot-hours/day reclaimed). Design constraints that MUST be preserved (high blast radius — this touches a branch-protection-required gate): - The bp-required status context **`sop-checklist / all-items-acked (pull_request)`** must be emitted byte-identically (BP wedges all merges otherwise; `lint-bp-context-emit-match.yml` parity). - `sop-checklist`'s trust boundary: `pull_request_target` + `actions/checkout` pinned to `default_branch` ref (never PR-head). `review-refire-comments` uses `issue_comment` + default-branch checkout. The merged file must keep BASE-only script execution. - Distinct token secrets: `SOP_CHECKLIST_GATE_TOKEN` vs `RFC_324_TEAM_READ_TOKEN` / `SOP_TIER_CHECK_TOKEN` — keep per-step env scoping; do not widen. - Keep both behaviors as separate jobs with their existing `if:` guards inside one workflow; one top-level `on: issue_comment` + `on: pull_request_target` block. Recommended shape: a single `comment-and-sop-dispatch.yml` with: job A = sop-checklist (pull_request_target + slash on issue_comment, emits the bp context), job B = review-refire dispatch (slash-only on issue_comment, no required context). One `issue_comment` subscription total. ## Coordination / scope fences (explicit) - **mc#1268** (`fix/ci-concurrency-cancel-superseded-storm`, persona core-devops, core-security APPROVED, mergeable) owns the superseded-retrigger storm (pull_request + pull_request_sync ≈ 5,785 runner-hrs/24h). **This issue is strictly the additive `issue_comment` double-refire amplifier — no overlap with #1268's 6 workflows** (secret-scan, block-internal-paths, lint-curl-status-capture, lint-workflow-yaml, check-migration-collisions, cascade-list-drift-gate). - **Sequencing: land AFTER mc#1268 merges.** `sop-checklist.yml` / its scripts are under active contended modification by mc#1263 / mc#1205 / mc#1200 / mc#1196 (all core-devops, /sop-n/a work). A consolidation PR now would collide on the same files and risk the bp-required gate. Implement once those settle. - operator-config#51 (defense-in-depth janitor cron) is **CLOSED/landed** — not blocking. - Schedule janitors (gitea-merge-queue/status-reaper `*/5`, single-ref — NOT branch-fanned-out, NOT the GitHub bot-ring fingerprint) overlap internal#400/#404/#420 — out of scope for this issue, no action recommended here. ## Acceptance - [ ] One `issue_comment` subscriber remains in `.gitea/workflows/` (`grep -l issue_comment: .gitea/workflows/*.yml` → 1 file). - [ ] `sop-checklist / all-items-acked (pull_request)` still emitted on pull_request_target with identical context string; bp gate verified green on a test PR. - [ ] Trailing-24h `action_run` count for `event='pull_request_comment'` on molecule-core drops ≥ 45%. - [ ] Genuine non-author peer review; normal CI; no bypass. --- _Filed by persona `core-qa` from live Gitea Actions DB analysis. Demand-side CI amplifier audit (fleet task #149). No code changed; tracking + design only._
hongming commented 2026-05-19 01:11:24 +00:00
Owner
Copy Link
Copy Source

Tracker update: the actual implementing PR is #1333 (consolidate issue_comment subscribers), sequence-deferred behind #1268 (per-ref cancel-in-progress concurrency). Both still open; no behavior change required here. Leaving this issue open as the umbrella tracker.

Refs: #1333, #1268

Tracker update: the actual implementing PR is #1333 (consolidate issue_comment subscribers), sequence-deferred behind #1268 (per-ref cancel-in-progress concurrency). Both still open; no behavior change required here. Leaving this issue open as the umbrella tracker. Refs: #1333, #1268
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1280
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?