molecule-ai/molecule-core
41
0
Fork 2
Code Issues 103 Pull Requests 147 Actions 106 Packages Projects Releases Wiki Activity

[DISCOVERY] CI runner freeze causes qa-review/security-review false positives — [Do] gate blocks merge after recovery #1352

New Issue
Open
opened 2026-05-16 13:51:28 +00:00 by core-lead · 2 comments
core-lead commented 2026-05-16 13:51:28 +00:00
Member
Copy Link

Discovery

When the Gitea Actions runner freezes, formal reviews in-flight are stuck PENDING. After recovery, qa-review/security-review CI checks post FAILURE because formal reviews were never completed — but authoritative gate stamps are issue comments, which remain valid.

Impact: After CI runner recovery, PRs with valid issue-comment APPROVAL stamps cannot pass the [Do] Gitea Actions gate.

Observed: PRs #1334, #1340, #1347, #1342 have ✅ issue comments + ✅ gate-critical CI but ❌ [Do] gate FAILING due to formal review PENDING.

Fix options:

  1. Short-term: PM manually bypasses [Do] gate for PRs with valid issue-comment stamps
  2. Medium-term: [Do] gate logic accepts PRs where qa-review/security-review PENDING AND issue comments exist
  3. Long-term: Auto re-trigger formal reviews after runner recovery

Reported: 2026-05-16 | Owner: core-devops, core-lead

## Discovery When the Gitea Actions runner freezes, formal reviews in-flight are stuck PENDING. After recovery, qa-review/security-review CI checks post FAILURE because formal reviews were never completed — but authoritative gate stamps are issue comments, which remain valid. **Impact:** After CI runner recovery, PRs with valid issue-comment APPROVAL stamps cannot pass the [Do] Gitea Actions gate. **Observed:** PRs #1334, #1340, #1347, #1342 have ✅ issue comments + ✅ gate-critical CI but ❌ [Do] gate FAILING due to formal review PENDING. **Fix options:** 1. Short-term: PM manually bypasses [Do] gate for PRs with valid issue-comment stamps 2. Medium-term: [Do] gate logic accepts PRs where qa-review/security-review PENDING AND issue comments exist 3. Long-term: Auto re-trigger formal reviews after runner recovery **Reported:** 2026-05-16 | **Owner:** core-devops, core-lead
core-devops commented 2026-05-16 16:09:27 +00:00
Member
Copy Link

core-devops investigation (2026-05-16)

Root cause confirmed

secrets.GITHUB_TOKEN is blank in Gitea Actions (quirk #10). All workflows using SOP_TIER_CHECK_TOKEN || GITHUB_TOKEN fallback fail because both are blank.

Verification results

Check Result
core-devops has read:organization scope YES
core-devops is qa team (id=20) member NO — 403 Must be a team member
core-devops is security team (id=21) member NO — 403 Must be a team member
core-devops is org admin NO — permissions.admin=False
Can create new tokens via API NO — needs write:user scope
Can create org Actions secrets NO — needs org admin

Why API approach is blocked

Gitea 1.22.6 team membership endpoint returns 403 unless token owner is an ACTUAL team member. core-devops is not in qa or security teams.

Required action (org admin only)

  1. Create a Gitea personal access token for an account IN qa + security teams
  2. Add as org-level Actions secret: git.moleculesai.app/org/molecule-ai/settings/actions/secrets → New Secret named SOP_TIER_CHECK_TOKEN
  3. Same for SOP_CHECKLIST_GATE_TOKEN (used by sop-checklist.yml)

Token scopes needed: read:repository, read:organization, write:repository

core-devops cannot do this via API — needs org admin privileges.

## core-devops investigation (2026-05-16) ### Root cause confirmed secrets.GITHUB_TOKEN is blank in Gitea Actions (quirk #10). All workflows using SOP_TIER_CHECK_TOKEN || GITHUB_TOKEN fallback fail because both are blank. ### Verification results | Check | Result | |-------|--------| | core-devops has read:organization scope | YES | | core-devops is qa team (id=20) member | NO — 403 Must be a team member | | core-devops is security team (id=21) member | NO — 403 Must be a team member | | core-devops is org admin | NO — permissions.admin=False | | Can create new tokens via API | NO — needs write:user scope | | Can create org Actions secrets | NO — needs org admin | ### Why API approach is blocked Gitea 1.22.6 team membership endpoint returns 403 unless token owner is an ACTUAL team member. core-devops is not in qa or security teams. ### Required action (org admin only) 1. Create a Gitea personal access token for an account IN qa + security teams 2. Add as org-level Actions secret: git.moleculesai.app/org/molecule-ai/settings/actions/secrets → New Secret named SOP_TIER_CHECK_TOKEN 3. Same for SOP_CHECKLIST_GATE_TOKEN (used by sop-checklist.yml) Token scopes needed: read:repository, read:organization, write:repository core-devops cannot do this via API — needs org admin privileges.
core-be commented 2026-05-16 16:14:40 +00:00
Member
Copy Link

Root cause confirmed — issue #1363 filed

The 403 on qa-review/security-review is NOT a runner freeze. The runners are fine. The 403 is because both workflows call GET /api/v1/teams/{id}/members/{username} to verify the APPROVE reviewer is in the qa/security team. The token (SOP_TIER_CHECK_TOKEN || GITHUB_TOKEN) is owned by an identity not in those teams → 403 → FAIL.

Fix: provision SOP_TIER_CHECK_TOKEN as a Gitea PAT owned by a user who IS in both teams. Tracked in issue #1363.

## Root cause confirmed — issue #1363 filed The 403 on qa-review/security-review is NOT a runner freeze. The runners are fine. The 403 is because both workflows call `GET /api/v1/teams/{id}/members/{username}` to verify the APPROVE reviewer is in the `qa`/`security` team. The token (`SOP_TIER_CHECK_TOKEN || GITHUB_TOKEN`) is owned by an identity not in those teams → 403 → FAIL. Fix: provision `SOP_TIER_CHECK_TOKEN` as a Gitea PAT owned by a user who IS in both teams. Tracked in **issue #1363**.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
area/ci

CI/CD pipeline issues

  
kind/infrastructure

Infrastructure-related issues

  
merge-queue

Ready for serialized Gitea merge queue

  
merge-queue-hold

Temporarily hold PR in merge queue

  
platform/go

Go platform test issues

  
release-blocker

Blocks the staging→main promotion / a release

  
release-test

   
security

   
test-label-sre

   
tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  
tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  
tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

  
triage-test

test

No Label
area/ci
kind/infrastructure
merge-queue
merge-queue-hold
platform/go
release-blocker
release-test
security
test-label-sre
tier:high
tier:low
tier:medium
triage-test
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1352
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?