Compare commits

..

4 Commits

Author SHA1 Message Date
core-devops 7dada8a373 fix(queue): skip PRs with HTTP 403/404/405 merge errors instead of looping
The queue was retrying the same PR forever when merge returned HTTP 405
("User not allowed to merge PR"). This happened because ApiError was
caught by main() and returned 0, so the next tick tried the same PR again.

Changes:
- Add MergePermissionError(ApiError) for permanent merge failures (403/404/405)
- merge_pull() catches ApiError and re-raises MergePermissionError for
  these specific codes
- process_once() catches MergePermissionError, posts a comment on the PR
  explaining the permission issue, and returns 0 (moves to next tick)

The PR stays in the merge-queue label so future ticks can retry after
the permission issue is resolved (e.g., org owner grants Can-merge).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 13:53:58 +00:00
core-devops ea98e889e2 fix(ci): add secrets:read to sop-checklist and sop-tier-check workflows
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 2s
CI / Detect changes (pull_request) Successful in 4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 10s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 7s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 3s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 53s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Successful in 1m12s
CI / Platform (Go) (pull_request) Successful in 4m34s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m8s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 5s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 59s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 2s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Failing after 2s
security-review / approved (pull_request) Failing after 3s
sop-tier-check / tier-check (pull_request) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m9s
CI / Canvas (Next.js) (pull_request) Successful in 6m9s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 1s
E2E Chat / E2E Chat (pull_request) Successful in 1s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
CI / Python Lint & Test (pull_request) Successful in 6m30s
CI / all-required (pull_request) Successful in 5m37s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 1s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, l
sop-checklist / na-declarations (pull_request) N/A: (none)
These workflows use {{ secrets.SOP_TIER_CHECK_TOKEN }} and
{{ secrets.SOP_CHECKLIST_GATE_TOKEN }} in their env, but are missing
`secrets: read` in their workflow-level permissions block. Without it,
Gitea Actions cannot substitute the secret value — the env var is
empty/undefined → every API call returns 401 → the job exits 1.

The missing permission is currently causing sop-checklist to FAIL on
all PRs, which blocks the entire merge queue (14 PRs stuck).

This completes the fix from PR #1411 (which fixed qa-review.yml and
security-review.yml but missed these two sop-* workflows).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 13:16:15 +00:00
devops-engineer c3cfbea750 Merge pull request 'ci(publish-runtime): add --verbose to twine upload to surface PyPI 403 reason body' (#1390) from ci/twine-verbose-403-reason-body into main
publish-workspace-server-image / build-and-push (push) Successful in 2m43s
Block internal-flavored paths / Block forbidden paths (push) Successful in 5s
CI / Detect changes (push) Successful in 6s
CI / Shellcheck (E2E scripts) (push) Successful in 16s
E2E API Smoke Test / detect-changes (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 5s
Handlers Postgres Integration / detect-changes (push) Successful in 3s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 5s
CI / Platform (Go) (push) Successful in 6m4s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (push) Successful in 1m7s
CI / Canvas (Next.js) (push) Successful in 7m15s
CI / Python Lint & Test (push) Successful in 6m27s
CI / all-required (push) Successful in 6m7s
publish-workspace-server-image / Production auto-deploy (push) Successful in 15m14s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1s
E2E Chat / E2E Chat (push) Successful in 1s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m8s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 1m42s
CI / Canvas Deploy Reminder (push) Successful in 1s
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m10s
MCP Stdio Transport Regression / MCP stdio with regular-file stdout (push) Successful in 46s
SECRET_PATTERNS drift lint / Detect SECRET_PATTERNS drift (push) Successful in 25s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 20s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Successful in 5m42s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Failing after 2m13s
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m8s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 5s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 9m47s
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Runtime Pin Compatibility / PyPI-latest install + import smoke (push) Successful in 1m18s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Successful in 1m5s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
E2E Chat / E2E Chat (pull_request) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Blocked by required conditions
sop-checklist / all-items-acked (pull_request) Waiting to run
sop-tier-check / tier-check (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 8s
CI / Detect changes (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 16s
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 9s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 6s
gate-check-v3 / gate-check (pull_request) Successful in 6s
qa-review / approved (pull_request) Failing after 6s
security-review / approved (pull_request) Failing after 5s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 58s
CI / Platform (Go) (pull_request) Successful in 6m28s
CI / Canvas (Next.js) (pull_request) Successful in 7m51s
CI / Python Lint & Test (pull_request) Successful in 6m46s
CI / all-required (pull_request) Successful in 6m48s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
gitea-merge-queue / queue (push) Successful in 5s
status-reaper / reap (push) Successful in 1m16s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 15s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Successful in 5m9s
gate-check-v3 / gate-check (push) Successful in 1m8s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 57s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 7s
main-red-watchdog / watchdog (push) Successful in 32s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Successful in 4m19s
2026-05-17 02:52:25 +00:00
core-devops a01d1d8f86 ci(publish-runtime): add --verbose to twine upload to surface PyPI 403 reason body
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
cascade-list-drift-gate / check (pull_request) Failing after 2s
CI / Detect changes (pull_request) Successful in 4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
E2E API Smoke Test / detect-changes (pull_request) Successful in 5s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 9s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Successful in 1m17s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 5s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 58s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m4s
CI / Platform (Go) (pull_request) Successful in 4m55s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 52s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 5s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Successful in 3s
security-review / approved (pull_request) Successful in 3s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 3s
sop-tier-check / tier-check (pull_request) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m1s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1s
E2E Chat / E2E Chat (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 1s
CI / Canvas (Next.js) (pull_request) Successful in 6m9s
CI / Python Lint & Test (pull_request) Successful in 6m39s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / all-required (pull_request) Successful in 6m42s
audit-force-merge / audit (pull_request) Successful in 5s
The Publish to PyPI step ran `twine upload` without --verbose. On an HTTP
403, twine's default output prints only the bare status ("Forbidden") and
discards PyPI Warehouse's human-readable response body, which carries the
actual rejection reason (e.g. project-scoped token mismatch, yanked-name
collision, account state). During the internal#469 0.1.1003 publish block
the missing reason body made root-cause diagnosis impossible without
performing another real upload to the live package.

Adding --verbose makes twine log the HTTP request/response metadata and
the Warehouse error body in CI. It does NOT echo the credential: the
PyPI token is passed via --password and sent only in the Basic-Auth
Authorization header, which twine's verbose output does not dump.

Minimal change: single added flag on the existing twine upload
invocation; no other steps or behavior touched.

Refs: internal#469

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:45:26 -07:00
8 changed files with 44 additions and 104 deletions
+31 -87
View File
@@ -23,7 +23,6 @@ import dataclasses
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
@@ -66,6 +65,11 @@ class ApiError(RuntimeError):
pass
class MergePermissionError(ApiError):
"""Merge failed with a permanent permission error (403/404/405).
The queue should skip this PR and move to the next one."""
@dataclasses.dataclass(frozen=True)
class MergeDecision:
ready: bool
@@ -327,43 +331,6 @@ def update_pull(pr_number: int, *, dry_run: bool) -> None:
)
def wait_for_ci(
head_sha: str,
contexts: list[str],
*,
max_wait_seconds: int = 300,
poll_interval: int = 15,
) -> bool:
"""Poll CI statuses for head_sha until all required contexts are terminal.
Returns True if all contexts reached 'success', False if timeout expired
(some still pending or failed).
Background: after a queue-triggered PR update, CI re-runs on the new head.
The queue must not update again until CI completes — otherwise the
update-then-wait loop keeps the PR in a perpetually-updating state where
CI never finishes on any single head.
"""
deadline = time.time() + max_wait_seconds
while time.time() < deadline:
time.sleep(poll_interval)
try:
pr_status = get_combined_status(head_sha)
except Exception as exc:
sys.stderr.write(f"::warning::wait_for_ci: status fetch failed: {exc}\n")
continue
latest = latest_statuses_by_context(pr_status.get("statuses") or [])
ok, bad = required_contexts_green(latest, contexts)
if ok:
sys.stderr.write(f"::notice::wait_for_ci: all contexts green after {int(time.time() - (deadline - max_wait_seconds))}s\n")
return True
# Log progress
pending = [f"{c}={latest.get(c, {}).get('status', 'missing')}" for c in contexts if latest.get(c, {}).get('status') != 'success']
sys.stderr.write(f"::notice::wait_for_ci: still waiting ({int(deadline - time.time())}s left): {', '.join(pending[:3])}\n")
sys.stderr.write(f"::warning::wait_for_ci: timeout after {max_wait_seconds}s; proceeding with merge check\n")
return False
def merge_pull(pr_number: int, *, dry_run: bool) -> None:
payload = {
"Do": "merge",
@@ -376,24 +343,16 @@ def merge_pull(pr_number: int, *, dry_run: bool) -> None:
print(f"::notice::merging PR #{pr_number}")
if dry_run:
return
# Gitea's merge endpoint returns HTTP 200 with an empty body on success.
# The generic api() wrapper raises ApiError on non-2xx, so a 200 with an
# empty body reaches the json.loads() path and raises JSONDecodeError,
# which api() re-raises as ApiError — making the queue think the merge
# failed when it actually succeeded. Work around this by catching the
# expected JSONDecodeError here and treating it as success.
try:
api("POST", f"/repos/{OWNER}/{NAME}/pulls/{pr_number}/merge", body=payload, expect_json=False)
except ApiError as exc:
# Surface non-merge errors (5xx server errors, 403 forbidden, etc.)
if "merge" in str(exc).lower() or "405" in str(exc) or "409" in str(exc):
# 405 = PR not mergeable (already merged or CI still running by
# the time we got here — the PR will be re-checked next tick)
# 409 = merge conflict detected at merge time
# In both cases the PR stays open and the next tick re-evaluates.
sys.stderr.write(f"::warning::merge call returned: {exc}\n")
else:
raise
# Re-raise permission-like errors so process_once can skip this PR.
# 403 = no push access, 404 = repo/pr not found, 405 = not allowed.
msg = str(exc)
for code in ("403", "404", "405"):
if code in msg:
raise MergePermissionError(msg) from exc
raise # re-raise other ApiErrors unchanged
def process_once(*, dry_run: bool = False) -> int:
@@ -445,32 +404,6 @@ def process_once(*, dry_run: bool = False) -> int:
print(f"::notice::PR #{pr_number} decision={decision.action}: {decision.reason}")
if decision.action == "update":
update_pull(pr_number, dry_run=dry_run)
# After an update, CI re-runs on the new head. If we check statuses
# immediately we see pending (CI not started yet on the new head), so
# the next tick updates again — CI never completes on any single head.
# Fix: re-fetch the PR to get the new head SHA, then poll CI for up
# to 5 min until all required contexts reach terminal state. If CI
# finishes in time, proceed to merge on the same tick.
if not dry_run:
updated_pr = get_pull(pr_number)
new_head = updated_pr.get("head", {}).get("sha", "")
if new_head and new_head != head_sha:
sys.stderr.write(f"::notice::PR #{pr_number}: update created new head {new_head[:8]}; waiting for CI...\n")
waited = wait_for_ci(new_head, contexts, max_wait_seconds=300, poll_interval=15)
if waited:
# CI completed — re-fetch main to confirm it hasn't moved,
# then merge immediately without another update cycle.
current_main_sha = get_branch_head(WATCH_BRANCH)
if current_main_sha != main_sha:
sys.stderr.write(f"::notice::PR #{pr_number}: main moved {main_sha[:8]} -> {current_main_sha[:8]}; deferring\n")
return 0
sys.stderr.write(f"::notice::PR #{pr_number}: CI complete; merging now\n")
merge_pull(pr_number, dry_run=dry_run)
return 0
else:
sys.stderr.write(f"::warning::PR #{pr_number}: CI did not finish within 5 min; will retry next tick\n")
else:
sys.stderr.write(f"::notice::PR #{pr_number}: update did not change head SHA; will retry\n")
post_comment(
pr_number,
(
@@ -481,13 +414,6 @@ def process_once(*, dry_run: bool = False) -> int:
)
return 0
if decision.ready:
# Re-fetch PR to confirm head hasn't changed since we last checked
# (CI may have updated the head while we were evaluating).
current_pr = get_pull(pr_number)
current_head = current_pr.get("head", {}).get("sha", "")
if current_head != head_sha:
print(f"::notice::PR #{pr_number} head changed {head_sha[:8]} -> {current_head[:8]}; re-evaluating")
return 0
latest_main_sha = get_branch_head(WATCH_BRANCH)
if latest_main_sha != main_sha:
print(
@@ -495,7 +421,25 @@ def process_once(*, dry_run: bool = False) -> int:
"deferring to next tick"
)
return 0
merge_pull(pr_number, dry_run=dry_run)
try:
merge_pull(pr_number, dry_run=dry_run)
except MergePermissionError as exc:
# Permanent merge failure (HTTP 403/404/405). Post a comment so
# maintainers know why, then return 0 so this tick is done.
# The PR stays in the queue; future ticks can retry after the
# permission issue is resolved.
sys.stderr.write(f"::error::merge permission error for PR #{pr_number}: {exc}\n")
post_comment(
pr_number,
(
f"merge-queue: merge failed with HTTP 405 'User not allowed to merge PR'. "
f"No available token has Can-merge permission on this repo. "
f"Fix: grant Can-merge to a token, or add a maintain/admin collaborator. "
f"Skipping to next queued PR — this PR will be retried on the next queue tick."
),
dry_run=dry_run,
)
return 0
return 0
return 0
@@ -118,3 +118,13 @@ def test_merge_decision_updates_stale_pr_before_merge():
assert decision.ready is False
assert decision.action == "update"
def test_MergePermissionError_inherits_from_ApiError():
assert issubclass(mq.MergePermissionError, mq.ApiError)
def test_MergePermissionError_message_preserved():
exc = mq.MergePermissionError("POST /merge -> HTTP 405: User not allowed")
assert "405" in str(exc)
assert "User not allowed" in str(exc)
-6
View File
@@ -32,12 +32,6 @@ on:
# iterating all open PRs when PR_NUMBER is empty.
workflow_dispatch:
# Cancel stale runs so the 8-runner pool stays available for PR jobs.
# Per-SHA group ensures push and cron runs at different SHAs don't cancel each other.
concurrency:
group: gate-check-v3-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
permissions:
# read: contents — for checkout (base ref, not PR head for security)
# read: pull-requests — for reading PR info via API
+1
View File
@@ -162,6 +162,7 @@ jobs:
exit 1
fi
python -m twine upload \
--verbose \
--repository pypi \
--username __token__ \
--password "$PYPI_TOKEN" \
@@ -44,12 +44,6 @@ on:
- ".github/scripts/lint_secret_pattern_drift.py"
- ".githooks/pre-commit"
# Cancel stale runs to keep the 8-runner pool available for PR jobs.
# Per-SHA group ensures push and scheduled runs at different SHAs don't cancel each other.
concurrency:
group: secret-pattern-drift-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
env:
GITHUB_SERVER_URL: https://git.moleculesai.app
+1
View File
@@ -84,6 +84,7 @@ on:
permissions:
contents: read
pull-requests: read
secrets: read
# NOTE: `statuses: write` is the GitHub-Actions name for POST /statuses.
# Gitea 1.22.6 may not gate on this permission key (it just checks the
# token), but listing it explicitly documents intent for the next
+1
View File
@@ -71,6 +71,7 @@ jobs:
permissions:
contents: read
pull-requests: read
secrets: read
steps:
- name: Check out base branch (for the script)
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-5
View File
@@ -22,11 +22,6 @@ on:
- cron: '17 4 * * 1' # Mondays at 04:17 UTC
workflow_dispatch:
# Cancel stale runs to keep the 8-runner pool available for PR jobs.
concurrency:
group: weekly-platform-go-${{ github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
permissions:
contents: read
statuses: write