Compare commits

...

4 Commits

Author SHA1 Message Date
core-devops 8ccf3a844c fix(queue): surface merge API errors instead of silent catch
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
CI / Detect changes (pull_request) Successful in 4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 8s
E2E API Smoke Test / detect-changes (pull_request) Successful in 7s
E2E Chat / detect-changes (pull_request) Successful in 5s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 5s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 3s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 56s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 5s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Failing after 3s
security-review / approved (pull_request) Failing after 3s
sop-tier-check / tier-check (pull_request) Successful in 3s
CI / Platform (Go) (pull_request) Successful in 4m11s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2s
E2E Chat / E2E Chat (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 1s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 2s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m2s
sop-checklist / all-items-acked (pull_request) acked: 0/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +4 — body-unfilled: comprehensive-testing, local-postgres-e2
sop-checklist / na-declarations (pull_request) N/A: (none)
CI / Canvas (Next.js) (pull_request) Successful in 5m33s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / Python Lint & Test (pull_request) Successful in 6m33s
CI / all-required (pull_request) Successful in 5m9s
audit-force-merge / audit (pull_request) Has been skipped
When the merge API returns a non-transient error (HTTP 405 permission
denied, HTTP 422 pre-receive hook block, etc.), the queue was catching
ApiError in the generic main-loop handler and exiting 0 — indistinguishable
from a successful-no-op tick.

Fix: catch ApiError specifically around merge_pull(), post a PR comment
with the error detail and a reference to SEV-1 internal#487, and return
exit code 2 so the workflow run is marked failed.

Exit codes:
  0 — success (merged, updated, or nothing to do)
  2 — merge API error (permission/hook issue, non-transient)

Fixes: SEV-1 internal#487 — queue silently failing to merge while
reporting success; merge permission error invisible without workflow
log inspection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 06:24:11 +00:00
core-devops 9ede993f3d fix(sop-checklist): probe() KeyError for gate names in compute_na_state
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
CI / Detect changes (pull_request) Successful in 4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 14s
E2E API Smoke Test / detect-changes (pull_request) Successful in 5s
E2E Chat / detect-changes (pull_request) Successful in 6s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 7s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 58s
gate-check-v3 / gate-check (pull_request) Successful in 4s
qa-review / approved (pull_request) Failing after 3s
security-review / approved (pull_request) Failing after 3s
Ops Scripts Tests / Ops scripts (unittest) (pull_request) Successful in 1m0s
sop-tier-check / tier-check (pull_request) Successful in 4s
CI / Platform (Go) (pull_request) Successful in 4m52s
CI / Canvas (Next.js) (pull_request) Successful in 6m35s
CI / Python Lint & Test (pull_request) Successful in 6m38s
CI / all-required (pull_request) Successful in 6m39s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1s
E2E Chat / E2E Chat (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 1s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 2s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
sop-checklist / all-items-acked (pull_request) [info tier:low] acked: 2/7 — missing: comprehensive-testing, local-postgres-e2e, staging-smoke, +2 — body-unfilled: comprehensive-testing, l
sop-checklist / na-declarations (pull_request) N/A: (none)
audit-force-merge / audit (pull_request) Has been skipped
compute_na_state() calls probe(gate_name, [user]) where gate_name is a gate
name like 'qa-review' or 'security-review' — these are not checklist item
slugs and are not in items_by_slug. probe() was doing:

    item = items_by_slug[slug]   # KeyError for 'qa-review'

This caused the sop-checklist workflow to crash on any PR that has N/A gates
configured (all 7 checklist items with /sop-n/a), producing a 30-minute
Failing status before Gitea kills the job.

Fix: add _required_teams_for() helper that falls back to na_gates lookup
when slug is not in items_by_slug. Gate names resolve to their
required_teams from the n/a_gates config section.

Adds TestProbeNaGateFallback regression test (58/58 passing).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 05:09:35 +00:00
devops-engineer c3cfbea750 Merge pull request 'ci(publish-runtime): add --verbose to twine upload to surface PyPI 403 reason body' (#1390) from ci/twine-verbose-403-reason-body into main
publish-workspace-server-image / build-and-push (push) Successful in 2m43s
Block internal-flavored paths / Block forbidden paths (push) Successful in 5s
CI / Detect changes (push) Successful in 6s
CI / Shellcheck (E2E scripts) (push) Successful in 16s
E2E API Smoke Test / detect-changes (push) Successful in 5s
E2E Chat / detect-changes (push) Successful in 5s
Handlers Postgres Integration / detect-changes (push) Successful in 3s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (push) Successful in 5s
CI / Platform (Go) (push) Successful in 6m4s
Runtime PR-Built Compatibility / detect-changes (push) Successful in 4s
Secret scan / Scan diff for credential-shaped strings (push) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (push) Successful in 1m7s
CI / Canvas (Next.js) (push) Successful in 7m15s
CI / Python Lint & Test (push) Successful in 6m27s
CI / all-required (push) Successful in 6m7s
publish-workspace-server-image / Production auto-deploy (push) Successful in 15m14s
E2E API Smoke Test / E2E API Smoke Test (push) Successful in 1s
E2E Chat / E2E Chat (push) Successful in 1s
Handlers Postgres Integration / Handlers Postgres Integration (push) Successful in 1m8s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) Successful in 1m42s
CI / Canvas Deploy Reminder (push) Successful in 1s
lint-bp-context-emit-match / lint-bp-context-emit-match (push) Successful in 1m10s
MCP Stdio Transport Regression / MCP stdio with regular-file stdout (push) Successful in 46s
SECRET_PATTERNS drift lint / Detect SECRET_PATTERNS drift (push) Successful in 25s
E2E Staging SaaS (full lifecycle) / pr-validate (push) Successful in 20s
E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) Successful in 5m42s
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push) Failing after 2m13s
E2E Staging External Runtime / E2E Staging External Runtime (push) Successful in 5m8s
E2E Staging Canvas (Playwright) / detect-changes (push) Successful in 5s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (push) Successful in 9m47s
Railway pin audit (drift detection) / Audit Railway env vars for drift-prone pins (push) Compensated by status-reaper (workflow has no push: trigger; Gitea 1.22.6 hardcoded-suffix bug — see .gitea/scripts/status-reaper.py)
Runtime Pin Compatibility / PyPI-latest install + import smoke (push) Successful in 1m18s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (push) Successful in 1m5s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Blocked by required conditions
E2E Chat / E2E Chat (pull_request) Blocked by required conditions
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Blocked by required conditions
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Blocked by required conditions
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Blocked by required conditions
sop-checklist / all-items-acked (pull_request) Waiting to run
sop-tier-check / tier-check (pull_request) Waiting to run
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 8s
CI / Detect changes (pull_request) Successful in 9s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 16s
E2E API Smoke Test / detect-changes (pull_request) Successful in 6s
E2E Chat / detect-changes (pull_request) Successful in 7s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 9s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 6s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 6s
gate-check-v3 / gate-check (pull_request) Successful in 6s
qa-review / approved (pull_request) Failing after 6s
security-review / approved (pull_request) Failing after 5s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 58s
CI / Platform (Go) (pull_request) Successful in 6m28s
CI / Canvas (Next.js) (pull_request) Successful in 7m51s
CI / Python Lint & Test (pull_request) Successful in 6m46s
CI / all-required (pull_request) Successful in 6m48s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
gitea-merge-queue / queue (push) Successful in 5s
status-reaper / reap (push) Successful in 1m16s
Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push) Successful in 15s
Continuous synthetic E2E (staging) / Synthetic E2E against staging (push) Successful in 5m9s
gate-check-v3 / gate-check (push) Successful in 1m8s
Sweep stale Cloudflare DNS records / Sweep CF orphans (push) Successful in 9s
ci-required-drift / drift (push) Successful in 57s
Sweep stale Cloudflare Tunnels / Sweep CF tunnels (push) Successful in 7s
main-red-watchdog / watchdog (push) Successful in 32s
Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push) Successful in 4m19s
2026-05-17 02:52:25 +00:00
core-devops a01d1d8f86 ci(publish-runtime): add --verbose to twine upload to surface PyPI 403 reason body
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 3s
cascade-list-drift-gate / check (pull_request) Failing after 2s
CI / Detect changes (pull_request) Successful in 4s
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
E2E API Smoke Test / detect-changes (pull_request) Successful in 5s
E2E Chat / detect-changes (pull_request) Successful in 6s
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 9s
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Successful in 1m17s
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 5s
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 58s
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 1m4s
CI / Platform (Go) (pull_request) Successful in 4m55s
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 52s
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 5s
gate-check-v3 / gate-check (pull_request) Successful in 3s
qa-review / approved (pull_request) Successful in 3s
security-review / approved (pull_request) Successful in 3s
sop-checklist / na-declarations (pull_request) N/A: (none)
sop-checklist / all-items-acked (pull_request) Successful in 3s
sop-tier-check / tier-check (pull_request) Successful in 3s
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 1m1s
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 1s
E2E Chat / E2E Chat (pull_request) Successful in 2s
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 2s
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 1s
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 1s
CI / Canvas (Next.js) (pull_request) Successful in 6m9s
CI / Python Lint & Test (pull_request) Successful in 6m39s
CI / Canvas Deploy Reminder (pull_request) Has been skipped
CI / all-required (pull_request) Successful in 6m42s
audit-force-merge / audit (pull_request) Successful in 5s
The Publish to PyPI step ran `twine upload` without --verbose. On an HTTP
403, twine's default output prints only the bare status ("Forbidden") and
discards PyPI Warehouse's human-readable response body, which carries the
actual rejection reason (e.g. project-scoped token mismatch, yanked-name
collision, account state). During the internal#469 0.1.1003 publish block
the missing reason body made root-cause diagnosis impossible without
performing another real upload to the live package.

Adding --verbose makes twine log the HTTP request/response metadata and
the Warehouse error body in CI. It does NOT echo the credential: the
PyPI token is passed via --password and sent only in the Basic-Auth
Authorization header, which twine's verbose output does not dump.

Minimal change: single added flag on the existing twine upload
invocation; no other steps or behavior touched.

Refs: internal#469

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:45:26 -07:00
5 changed files with 129 additions and 3 deletions
+17 -1
View File
@@ -407,7 +407,23 @@ def process_once(*, dry_run: bool = False) -> int:
"deferring to next tick"
)
return 0
merge_pull(pr_number, dry_run=dry_run)
try:
merge_pull(pr_number, dry_run=dry_run)
except ApiError as exc:
# Merge API errors (405 permission denied, 422 hook block, etc.)
# are NOT transient — retrying will not help. Surface the error
# on the PR immediately so it is visible without digging into
# workflow logs, and fail the workflow so it is distinguishable
# from a successful-no-op tick.
post_comment(
pr_number,
f"merge-queue: MERGE FAILED — {exc}. "
"This is a non-transient error (permission or hook issue). "
"See SEV-1 internal#487.",
dry_run=dry_run,
)
sys.stderr.write(f"::error::PR #{pr_number} merge failed: {exc}\n")
return 2 # distinct exit code so workflow run shows failure
return 0
return 0
+11 -2
View File
@@ -830,9 +830,18 @@ def main(argv: list[str] | None = None) -> int:
# one membership lookup per team.
team_member_cache: dict[tuple[str, int], bool | None] = {}
def _required_teams_for(slug: str) -> list[str] | None:
"""Look up required_teams for a slug from checklist items OR N/A gates."""
if slug in items_by_slug:
return items_by_slug[slug]["required_teams"]
if slug in na_gates:
return na_gates[slug].get("required_teams", [])
return None
def probe(slug: str, users: list[str]) -> list[str]:
item = items_by_slug[slug]
team_names: list[str] = item["required_teams"]
team_names = _required_teams_for(slug)
if team_names is None:
raise KeyError(f"slug '{slug}' not found in items or N/A gates")
# Resolve names → ids. NOTE: orgs/{org}/teams/search may not be
# available — fall back to the list endpoint.
team_ids: list[int] = []
@@ -1,6 +1,7 @@
import importlib.util
import sys
from pathlib import Path
from unittest.mock import patch
SCRIPT = Path(__file__).resolve().parents[1] / "gitea-merge-queue.py"
@@ -118,3 +119,54 @@ def test_merge_decision_updates_stale_pr_before_merge():
assert decision.ready is False
assert decision.action == "update"
def test_merge_failure_returns_nonzero_and_posts_comment(monkeypatch):
"""When merge_pull raises ApiError (e.g. HTTP 405 permission denied),
process_once returns exit code 2 (non-zero) and posts a comment on the PR.
This distinguishes merge-permission errors from successful-no-op ticks."""
captured_comment = {}
def fake_post_comment(pr_number, body, *, dry_run):
captured_comment["pr_number"] = pr_number
captured_comment["body"] = body
# Replace functions directly on the module object so process_once()
# (which looks them up by name at call time) picks up the fakes.
mq.list_queued_issues = lambda: [{
"number": 42,
"created_at": "2026-05-17T00:00:00Z",
"labels": [{"name": "merge-queue"}],
"pull_request": {},
}]
mq.get_pull = lambda n: {
"state": "open",
"base": {"ref": "main", "repo_id": 1},
"head": {"sha": "headsha", "repo_id": 1},
"merge_base": "abc123def",
}
mq.get_pull_commits = lambda n: [{"sha": "headsha"}]
mq.get_branch_head = lambda branch: "abc123def"
mq.get_combined_status = lambda sha: {
"state": "success",
"statuses": [{"context": "CI / all-required (push)", "status": "success"}],
}
mq.latest_statuses_by_context = lambda s: {
"CI / all-required (pull_request)": {"status": "success"},
"sop-checklist / all-items-acked (pull_request)": {"status": "success"},
}
mq.required_contexts_green = lambda statuses, contexts: (True, [])
mq.post_comment = fake_post_comment
# Simulate merge failing with HTTP 405 (permission denied).
# The ApiError raised by api() is caught inside process_once().
merge_error = mq.ApiError(
"POST /repos/x/y/pulls/42/merge -> HTTP 405: User not allowed to merge PR"
)
with patch.object(mq, "merge_pull", side_effect=merge_error):
exit_code = mq.process_once(dry_run=False)
assert exit_code == 2, f"Expected exit code 2, got {exit_code}"
assert captured_comment["pr_number"] == 42
assert "MERGE FAILED" in captured_comment["body"]
assert "405" in captured_comment["body"]
@@ -603,3 +603,51 @@ class TestComputeNaState(unittest.TestCase):
self.assertEqual(na_directives[0][0], "sop-n/a")
self.assertEqual(na_directives[0][1], "qa-review")
self.assertIn("no surface", na_directives[0][2])
class TestProbeNaGateFallback(unittest.TestCase):
"""Regression test: probe() must handle gate names (qa-review, security-review)
from N/A gates without raising KeyError.
mc#1389: compute_na_state calls probe(gate_name, [user]) where gate_name is
a gate name like 'qa-review' — NOT a checklist item slug. The probe must
resolve the gate's required_teams from na_gates, not raise KeyError from
items_by_slug lookup.
"""
def test_probe_resolves_gate_name_from_na_gates(self):
cfg = sop.load_config(CONFIG_PATH)
items = cfg["items"]
items_by_slug = {it["slug"]: it for it in items}
na_gates = cfg.get("n/a_gates", {})
# Reconstruct the _required_teams_for helper from sop-checklist.py
def _required_teams_for(slug):
if slug in items_by_slug:
return items_by_slug[slug]["required_teams"]
if slug in na_gates:
return na_gates[slug].get("required_teams", [])
return None
# Gate names should resolve from na_gates
self.assertEqual(
_required_teams_for("qa-review"),
["qa", "security", "engineers"],
)
self.assertEqual(
_required_teams_for("security-review"),
["security", "managers", "ceo"],
)
# Checklist item slugs should still resolve from items_by_slug
self.assertEqual(
_required_teams_for("comprehensive-testing"),
["qa", "engineers"],
)
self.assertEqual(
_required_teams_for("root-cause"),
["managers", "ceo"],
)
# Unknown slug should return None (not raise KeyError)
self.assertIsNone(_required_teams_for("nonexistent-slug"))
+1
View File
@@ -162,6 +162,7 @@ jobs:
exit 1
fi
python -m twine upload \
--verbose \
--repository pypi \
--username __token__ \
--password "$PYPI_TOKEN" \