111 Commits

Author	SHA1	Message	Date
Molecule AI Core-DevOps	ebeea0a9c1	fix(workflows): add mc#664 tracker to lint-pre-flip CoE directive ... lint-continue-on-error-tracking (Tier 2e) requires a tracker within ±2 lines of every `continue-on-error: true`. The inline comment was 3 lines above the directive, outside the scan window. Move mc#664 to an inline comment on the directive line so it is within ±2 lines (WINDOW=2 per lint_continue_on_error_tracking.py). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 07:38:13 +00:00
Molecule AI Core-DevOps	0970feef70	feat(ci)(hard-gate): lint-pre-flip catches continue-on-error true→false without run-log proof ... Empirical class — PR #656 / mc#664: PR #656 (RFC internal#219 Phase 4) flipped 5 platform-build-class jobs `continue-on-error: true → false` on the basis of a "verified green on main via combined-status check". But that "green" was the LIE the prior `continue-on-error: true` produced: Gitea Quirk #10 (internal#342 + dup #287) — a failed step inside a CoE:true job rolls up to a success job-level status. The precondition the PR claimed to verify was structurally fooled by the bug being flipped. mc#664 captured the surfaced defects (2 mutually-masked regressions): - Class 1: sqlmock helper drift since 2f36bb9a (24 days old) - Class 2: OFFSEC-001 contract collision since 7d1a189f (1 day old) Codified 04:35Z as hongming-pc2 charter §SOP-N rule (e) "run-log-grep-before-flip": pull the actual run log + grep for --- FAIL / FAIL\s BEFORE flipping; don't trust the masked combined-status. This commit structurally enforces that rule. What this PR adds: .gitea/workflows/lint-pre-flip-continue-on-error.yml — pre-merge pull_request gate, path-scoped to .gitea/workflows/**. Lands at continue-on-error:true (Phase 3 dogfood — flip to false in a follow-up only after this workflow has clean recent runs on main). .gitea/scripts/lint_pre_flip_continue_on_error.py — the lint: 1. Reads every .gitea/workflows/*.yml at the PR base SHA AND head SHA via git show <sha>:<path>. No checkout needed. 2. Parses both sides via PyYAML AST (per feedback_behavior_based_ast_gates — NOT grep, so comment churn and key-order changes don't false-positive). 3. For each flipped job (base=true, head=false), renders the commit-status context as "{workflow.name} / {job.name or job.key} (push)" and pulls combined commit-status for the last 5 commits on the PR base branch. 4. Fetches each matching run's log via the web-UI route {server_url}/{repo}/actions/runs/{run_id}/jobs/{job_idx}/logs (per reference_gitea_actions_log_fetch — Gitea 1.22.6 lacks REST /actions/runs/*; web-UI is the only working path, see reference_gitea_1_22_6_lacks_rest_rerun_endpoints). 5. Greps for --- FAIL / FAIL\s / ::error::. If status==success AND log shows fail markers, the job was masked. Emit ::error::file=... naming the failing test + offending run URL. .gitea/scripts/tests/test_lint_pre_flip_continue_on_error.py — 35 unittest cases covering the 5 acceptance tests from the spec + CoE coercion (truthy/falsy/quoted/absent) + context-name rendering + multi-flip aggregation + dry-run semantics + 3 graceful-degrade halt conditions (log-unavailable, zero-runs- history, zero-commits-on-branch). Live empirical confirmation: Ran the script against the PR#656 base→merge diff with RECENT_COMMITS_N=3 on main. Result: - platform-build flip BLOCKED — masked --- FAIL on TestExecuteDelegation_DeliveryConfirmedProxyError_TreatsAsSuccess + 4 more on action_run 13353. - canvas-build / shellcheck / python-lint flips PASS — no FAIL markers in their recent logs. Exactly the diagnosis hongming-pc2 charter §SOP-N rule (e) requires. Halt-condition graceful-degrade contract: - Log fetch 404 (act_runner pruned, transient outage): warn-not-block. - Zero recent runs of the flipped context (newly-added workflow): chicken-and-egg exemption — warn and allow. - YAML parse error in one workflow file: warn-not-block (the YAML lint workflows catch this separately). Cross-links: PR#656, mc#664, PR#665 (interim re-mask), Quirk #10 (internal#342 + dup #287), hongming-pc2 charter §SOP-N rule (e), feedback_strict_root_only_after_class_a, feedback_no_shared_persona_token_use. Refs: internal#342, internal#287, molecule-core#664, molecule-core#665	2026-05-12 07:27:19 +00:00
Molecule AI Core-DevOps	2ee7cb1493	fix(ci): replace workflow_run triggers with push+paths (Gitea 1.22.6) ... Three workflows used `workflow_run:` to trigger when `publish-workspace-server-image.yml` completed, but Gitea 1.22.6 does not support the `workflow_run` event (task #81). The workflows were silently dead — never firing despite `continue-on-error: true`. Replaced each with `push: branches: [X], paths: [.gitea/workflows/ publish-workspace-server-image.yml]` which fires on every commit to the publish workflow. This is functionally equivalent: only successful runs commit to the branch. Also: - `redeploy-tenants-on-staging.yml`: corrected branch from [main] to [staging] (was wrong in the original Gitea port). - `staging-verify.yml`: removed `if: workflow_run.conclusion==success` since push events lack this context; the smoke test itself is the safety net. - Added `workflow_dispatch` to all three for manual runs. This fixes the 3 Rule-2 violations reported by lint-workflow-yaml (lint from #671). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 07:19:50 +00:00
core-devops	0dae4b8eb0	feat(ci)(hard-gate): lint-continue-on-error-tracking (Tier 2e) ... Every `continue-on-error: true` in `.gitea/workflows/*.yml` must carry a `# mc#NNNN` or `# internal#NNNN` tracker comment within 2 lines, referencing an OPEN issue ≤14 days old. The class this prevents ----------------------- `continue-on-error: true` on platform-build had been hiding mc#664-class regressions for ~3 weeks before #656 surfaced them. A 14-day cap on tracker age forces a review cycle: close-or-renew. Implementation -------------- - `.gitea/scripts/lint_continue_on_error_tracking.py` — PyYAML line-tracking loader to find every job-level `continue-on-error: <truthy>`. Treats string `"true"` as truthy (Gitea evaluator coerces). For each, scans ±2 lines of the directive's source line for `# mc#NNN` / `# internal#NNN` (regex case-sensitive — `mc` and `internal` are conventional slugs). GETs each issue from the Gitea API; valid = exists + state=open + `age.days <= MAX_AGE_DAYS` (inclusive 14d boundary). Graceful-degrades on 403 (token-scope) per Tier 2a contract. - `.gitea/workflows/lint-continue-on-error-tracking.yml` — pull_request + push + daily 13:11Z schedule. Schedule run catches the age-expiry class (tracker was ≤14d when PR landed but is now 20d). Phase 3 (continue-on-error: true) per RFC #219 §1. - `tests/test_lint_continue_on_error_tracking.py` — 14 unit tests: coe=false ignored, open-recent mc#/internal# pass, no-comment fail, comment-too-far fail, closed-issue fail, too-old fail, 14d-boundary pass / 15d fail, 404 fail, 403 skip, multi-violation aggregation, comment-AFTER-directive pass, quoted "true" caught. Behaviour --------- Pre-existing continue-on-error: true directives on main violate this lint at first — intentional. They are the masked defects this lint exists to surface (see mc#664). Phase 3 contract means the lint runs surface-only; follow-up flip to continue-on-error: false after main is clean for 3 days. Auth uses DRIFT_BOT_TOKEN (same as ci-required-drift.yml) because `internal#NNN` references cross repositories — auto-GITHUB_TOKEN can't read molecule-ai/internal from molecule-core. Refs: #350	2026-05-12 07:05:07 +00:00
Molecule AI · core-devops	cc6fa8717d	Merge pull request 'feat(ci): sop-checklist-gate — peer-ack merge gate (RFC#351 Phase 2)' (#688) from feat/sop-checklist-gate-mvp into main	2026-05-12 07:03:49 +00:00
core-devops	76988c05cd	fix(ci): sop-checklist-gate exits 0 by default — POSTed status is the gate ... By default the gate script now exits 0 in non-dry-run mode regardless of ack state. The job-level pass/fail must NOT carry the gate signal — otherwise BP sees TWO failure signals (the job-auto-status + our POSTed status) and the user gets ambiguous error messages. The POSTed `sop-checklist / all-items-acked (pull_request)` status IS the gate. Job conclusion is informational. Added --exit-on-state for local debugging (restores the old non-zero-on-failure behavior). Default OFF — production behavior is exit 0 always. 51/51 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 06:13:58 +00:00
core-devops	72df12ecef	feat(ci): sop-checklist-gate — peer-ack merge gate (RFC#351 Phase 2) ... RFC#351 Step 2 of 6: implementation MVP of the SOP-checklist peer-ack merge gate. NOT yet wired to branch protection (Phase 4 needs separate authorization). What: - .gitea/sop-checklist-config.yaml — 7-item checklist with slug, numeric_alias (1..7), pr_section_marker, required_teams. Includes tier-aware failure-mode map: tier:high/medium=hard, tier:low=soft, default=hard (never silently lower the bar). - .gitea/scripts/sop-checklist-gate.py — parses PR body + comments, computes per-item ack state, posts commit-status "sop-checklist / all-items-acked (pull_request)". - .gitea/scripts/tests/test_sop_checklist_gate.py — 51 unit tests covering slug normalization, directive parsing, section-marker detection, ack-state computation (self-ack reject, revoke semantics, multi-user/multi-item, numeric aliases), tier-mode selection, and end-to-end happy path. - .gitea/workflows/sop-checklist-gate.yml — pull_request_target [opened/edited/synchronize/reopened] + issue_comment [created/edited/deleted]. Checks out BASE ref only (trust boundary per RFC#324 §A4). Mirrors qa-review/security-review patterns. Why: Hongming 2026-05-12T05:42Z asked for SOP-enforcing CI/CD that requires peer-ack on each checklist item before merge. Composes the existing patterns (scripts-lint PR-body parser + RFC#324 persona-whitelist commit-status + sop-tier-check tier-awareness) into one gate. Slash-command contract: /sop-ack <slug> [note] — register peer-ack (most-recent wins) /sop-revoke <slug> [reason] — invalidate own prior ack Slug normalization accepts kebab-case, snake_case, natural-spaces, or numeric 1..7 shorthand (all canonicalize to kebab-case via the config-driven alias table). Tests: 51/51 pass locally. Dry-run probe against PR#685 verified the full pipeline (PR fetch, comment fetch, ack computation, status description rendering inside the 140-char budget). Not yet: - Phase 3 (24h soak) - Phase 4 (BP PATCH to require this context — needs Hongming GO) - Phase 5 (cross-repo) - Phase 6 (dev-sop.md codification) - SOP_CHECKLIST_GATE_TOKEN secret provisioning (separate follow-up; fail-closed until provisioned, same as RFC_324_TEAM_READ_TOKEN pattern in qa-review.yml). Cross-links: - internal#351 (RFC body) - RFC#324 (qa-review/security-review — reused mechanism) - internal#346 (dev-sop.md SOP-14..SOP-20 — sibling rules) - feedback_pull_request_review_no_refire (why issue_comment trigger) - feedback_checkpointed_workflow_over_good_practice_doc (motivation) - feedback_fix_root_not_symptom (default-mode=hard rationale) ## What Add a SOP-checklist peer-ack merge gate: workflow + script + config + 51 unit tests. ## Why Hongming-requested mechanism to enforce SOP via CI/CD: each PR checklist item must be peer-acked before merge, with team-membership-verified ackers and tier-aware failure mode. ## Verification - 51/51 unit tests pass (slug normalization, parse_directives, section marker detection, ack-state including self-ack rejection + revoke semantics, tier-mode mapping, end-to-end happy path). - YAML lint clean (yaml.safe_load + lint-workflow-yaml.py on the new workflow — pre-existing fatals on unrelated files only). - Python syntax clean (py_compile). - Dry-run against live PR#685: PR fetch, comment enumeration, status description render all within 140-char budget — works end-to-end. ## Tier tier:medium — net-new CI workflow; no production impact; no BP change yet (Phase 4 separate auth). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 06:08:36 +00:00
core-devops	75af96586d	feat(ci)(hard-gate): lint-mask-pr-atomicity (Tier 2d) ... Blocks PRs that touch `.gitea/workflows/ci.yml` and modify ONLY ONE of {continue-on-error, all-required.sentinel.needs} without a `Paired: #NNN` reference in the PR body or a commit message. The split-pair class this prevents ---------------------------------- PR#665 (interim continue-on-error: true on platform-build) and PR#668 (sentinel-needs demotion of the same job) were designed as a pair but merged solo: #665 landed 04:47Z 2026-05-12, #668 still open at 05:07Z when watchdog #674 fired. ~20 min of main red + a cascade of false-positives. mc#664 was the surfaced incident. Implementation -------------- - `.gitea/scripts/lint_mask_pr_atomicity.py` — reads ci.yml at BASE_SHA and HEAD_SHA via `git show`, parses both via PyYAML AST (per feedback_behavior_based_ast_gates — NOT grep). Predicates: 1. any jobs.*.continue-on-error value diff 2. jobs.all-required.needs set diff (order-insensitive) Both → atomic, OK. Neither → no risk, OK. Exactly one → require `Paired: #NNN` in PR body or `git log base..head`. - `.gitea/workflows/lint-mask-pr-atomicity.yml` — pull_request trigger with paths filter on ci.yml + the lint files. Phase 3 (continue-on-error: true) per RFC #219 §1 ladder; follow-up flip after 3 clean days on main. - `tests/test_lint_mask_pr_atomicity.py` — 9 unit tests covering all prod branches per feedback_branch_count_before_approving: neither predicate, both atomic, coe-only/no-pair fail, needs-only/no-pair fail, coe-only/pair-in-body pass, needs-only/pair-in-commit pass, non-numeric pair rejection, ci.yml unchanged skip, newly-added ci.yml skip. Refs: #350	2026-05-11 23:06:18 -07:00
core-devops	d57ed520f0	feat(ci)(hard-gate): lint-workflow-yaml catches Gitea-1.22.6-hostile shapes ... Tier-2 hardening per RFC internal#219 §1 + charter §SOP-N rule (m). New CI lint that scans .gitea/workflows/*.yml for six structurally-hostile shapes that Gitea 1.22.6 silently rejects or ambiguously parses, BEFORE they reach main. Rules (4 fatal + 1 fatal cross-file + 1 heuristic-warn): 1. on.workflow_dispatch.inputs — Gitea 1.22.6 mis-parses inputs.X as sibling event types and rejects the entire workflow with [W] ignore invalid workflow ... unknown on type. Memory: feedback_gitea_workflow_dispatch_inputs_unsupported. Origin: 2026-05-11 publish-runtime-v1.0.0 silent freeze, ~24h PyPI lag. 2. on: workflow_run — not enumerated in Gitea 1.22.6 event types (verified via modules/actions/workflows.go; task #81). Workflow registers, fires for zero events. 3. workflow name: containing / — breaks the commit-status convention <workflow> / <job> (<event>) used by sop-tier-check + status-reaper to tokenize context strings. 4. cross-file name: collision — status-routing is by name; collision yields undefined commit-status updates (status-reaper rev1 class). 5. cross-repo uses: org/repo/subpath@ref — DEFAULT_ACTIONS_URL=github resolves to github.com/<org-suspended>/... and 404s. Memory: feedback_gitea_cross_repo_uses_blocked. Cross-link: task #109. 6. (WARN, heuristic) api.github.com refs without workflow-level env.GITHUB_SERVER_URL. Memory: feedback_act_runner_github_server_url. Per halt-condition 3: downgraded to warn-not-fail to avoid the 3 known benign hits on current main (OCI source label + jq-release pin) which use https://github.com/... not https://api.github.com/. Empirical history this hardens against: - status-reaper rev1 caught rule-4 (name-collision) class fail-loud - sop-tier-refire DOA-d on rule-2 (workflow_run partial) - #319 bootstrap-paradox (chained-defect class, related) - internal#329 dispatcher race (adjacent) - 2026-05-11 publish-runtime: rule-1, 24h PyPI freeze on runtime-v1.0.0 publish Triggers: - pull_request — pre-merge gate - push to main/staging — post-merge regression catch even if the PR gate is bypassed by branch-protection drift Per RFC #219 §1 contract: continue-on-error: true on the job during the surface-broken-shapes phase. Follow-up PR flips off after the 3 existing rule-2 violations on main are migrated to a supported trigger. Existing-on-main violations surfaced by this lint (3, informational, NOT auto-fixed per halt-condition 2): - .gitea/workflows/redeploy-tenants-on-main.yml — rule 2 - .gitea/workflows/redeploy-tenants-on-staging.yml — rule 2 - .gitea/workflows/staging-verify.yml — rule 2 All three have on: workflow_run: triggers that will fire for zero events. Fix path: replace with cron or with push+paths:[upstream-yml] gate. Tracked separately (do not block this PR). Tests: tests/test_lint_workflow_yaml.py — 15 pytest cases: - 6 × per-rule violation-detected (rules 1-3,5 + rule 4 cross-file + rule 6 heuristic-warn) - 6 × per-rule clean-passes - 1 × cross-file collision detected - 1 × all-violations-aggregated single file - 1 × empty workflow dir = exit 0 - 1 × vendor-truth: the exact 2026-05-11 publish-runtime YAML shape from feedback_gitea_workflow_dispatch_inputs_unsupported is caught (per feedback_smoke_test_vendor_truth_not_shape_match: fixtures mirror real Gitea 1.22.6 semantics, not yaml-parser quirks) 15/15 tests pass locally. Lint exits 1 against current .gitea/workflows/ because of the 3 existing rule-2 violations above; that is the gate working as intended (and continue-on-error keeps the PR-status soft until the violations are migrated).	2026-05-12 05:50:55 +00:00
core-devops	c0f594cd22	feat(ci)(hard-gate): lint-required-workflows-no-paths-filter (structural enforcement of feedback_path_filtered_workflow_cant_be_required) ... Add `.gitea/workflows/lint-required-no-paths.yml` + supporting script and tests that fail a PR if any workflow whose status-check context appears in `branch_protections/main.status_check_contexts` carries a `paths:` or `paths-ignore:` filter in its `on:` block. Why --- A required-check workflow with a paths filter silently degrades the merge gate. If a PR's diff doesn't match the filter, the workflow never fires; Gitea (1.22.6) treats the required context as `pending` (NOT `skipped == success`), so the PR cannot merge. A docs-only PR against `paths: ['**.go']` would be wedged forever — no human action produces a green. Previously this was prevented only by reviewer vigilance + the saved memory `feedback_path_filtered_workflow_cant_be_required`. This commit makes it a structural CI gate. Empirical baseline (verified 2026-05-11 against git.moleculesai.app/molecule-ai/molecule-core/branch_protections/main): status_check_contexts: - "Secret scan / Scan diff for credential-shaped strings (pull_request)" - "sop-tier-check / tier-check (pull_request)" - "CI / all-required (pull_request)" All three workflows (`secret-scan.yml`, `sop-tier-check.yml`, `ci.yml`) have NO paths/paths-ignore filter today. This lint locks that contract: a future PR adding `paths:` to any of them — or to any new required workflow per RFC#324 Step 2 (qa-review, security-review) — fails fast at PR time. How --- - Workflow runs on `pull_request: [opened, synchronize, reopened]` + `workflow_dispatch`. Deliberately NO `paths:` filter on itself — the workflow is self-evidently a meta-required-check. - Reads `branch_protections/main` via `DRIFT_BOT_TOKEN` (same secret ci-required-drift.yml uses — repo-admin scope required for the endpoint per Gitea 1.22.6). - Parses each context `<workflow_name> / <job_name> (<event>)`, walks `.gitea/workflows/*.yml` for a file whose `name:` matches, then YAML-AST-walks the `on:` block for `paths` / `paths-ignore` keys. Behavior-based gate per `feedback_behavior_based_ast_gates` — NOT grep-by-name, so reformatting / event moves still detect. - Token-scope fallback: if `branch_protections` returns 403/404, exits 0 with a loud `::error::` rather than red-X every PR. Token issues should be fixed at the token. Tests ----- 20 tests in `tests/test_lint_required_no_paths.py`, all green: - parse_context (3): standard, slash-in-job-name, malformed - resolve_workflow_file (2): match-by-name, missing - detect_paths_filters (8): clean, paths, paths-ignore, push.paths, both, on-string-shorthand, on-list-shorthand, on-event-null - run() end-to-end (7): empty contexts, clean workflow, paths fails, paths-ignore fails, unknown-context warns-not-fails, multi-required one-bad-one-good, protection-403 skip Live smoke (DRIFT_BOT_TOKEN against molecule-ai/molecule-core/main): all 3 required workflows clean — exit 0. Cross-links ----------- - `feedback_path_filtered_workflow_cant_be_required` (the rule now structurally enforced) - `feedback_behavior_based_ast_gates` (PyYAML AST walk, not grep) - ci-required-drift.yml (precedent for DRIFT_BOT_TOKEN reuse + branch_protections-read scope-fallback pattern) - Charter §SOP-N rule (f): required-checks must run unconditionally Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 05:48:22 +00:00
Molecule AI Core-Lead	9aa2b13934	fix(ci)(interim): re-add continue-on-error to platform-build (mc#664 fix-forward in flight) ... Phase-3-masked test failures in workspace-server/internal/handlers/ surfaced when #656 (RFC internal#219 Phase 4) flipped platform-build continue-on-error from true to false on 0e5152c3. The pre-#656 main was masking these: 4x delegation_test.go (lines 1110/1176/1228/1271): TestExecuteDelegation_DeliveryConfirmedProxyError_TreatsAsSuccess TestExecuteDelegation_ProxyErrorNon2xx_RemainsFailed TestExecuteDelegation_ProxyErrorEmptyBody_RemainsFailed TestExecuteDelegation_CleanProxyResponse_Unchanged Root cause: expectExecuteDelegationBase/Success/Failed helpers do not mock the DB queries production has issued since ~2026-04-21: - UPDATE workspaces SET last_outbound_at (commit 2f36bb9a, 2026-04-18, async goroutine fired from logA2ASuccess in a2a_proxy_helpers.go) - SELECT delivery_mode / SELECT runtime FROM workspaces (lookup* in a2a_proxy_helpers.go since file split in 64ccf8e1, 2026-04-21) - INSERT INTO activity_logs (a2a_receive) via LogActivity in logA2ASuccess/logA2AError (preexisting, not mocked) - recordLedgerStatus writes (RFC #2829 #318) Symptoms: sqlmock unexpected query → production short-circuits → trailing ExpectExec for completed/failed never fires → mock.ExpectationsWereMet() reports unmet remaining expectations. 8.11s uniform wall time is the delegationRetryDelay × 2 attempts after the first unexpected-query causes a transient retry path. Halt cond #3 applies (>7 days masked → broader sweep needed; many subsequent commits stacked on top). 1x mcp_test.go:433 (TestMCPHandler_CommitMemory_GlobalScope_Blocked): Commit 7d1a189f (2026-05-10) hardened mcp.go:427 to scrub err.Error() from JSON-RPC error.Message (OFFSEC-001 / #259) — returning the constant string "tool call failed" instead. The test asserts the message contains "GLOBAL". Production-vs-test contract collision; needs a design call (revert OFFSEC scrub for this code class, or update the test to assert a different oracle e.g. captured logs / specific error code). Halt cond #2 applies (alternate-class finding, not sqlmock-mismatch). Time-boxed Option A (90 min sqlmock update) does not fit either failure class within scope. Choosing Option B per brief: interim re-mask of platform-build only — the other 4 #656 flips (changes, canvas-build, shellcheck, python-lint) retain continue-on-error: false. This is a sequenced revert→fix→reflip per feedback_strict_root_only_after_class_a emergency clause, NOT a permanent re-mask. mc#664 stays open as the fix-then-reflip tracker. Process note for charter SOP-N (companion to vendor-truth-review-discipline): before flipping a job continue-on-error: true → false, do not trust the combined-status "success" signal alone — pull the actual run log and grep for --- FAIL / FAIL <package> to confirm the tests really pass. The masked green on 0e5152c3 came from continue-on-error suppressing the per-job status to neutral, which the combined-status aggregator counted as not-failure. Cross-links: - mc#664 (hongming-pc2 04:35Z Phase-3-masked defect filing) - mc#656 (the flip that surfaced this; 0e5152c3 first commit to actually run the Go tests against internal/handlers/* since the silent stack-up began) - feedback_strict_root_only_after_class_a (revert→fix→reflip discipline) - feedback_return_contract_change_audit_caller_tests (mcp case applies) - feedback_no_such_thing_as_flakes (these are real bugs, not flakes) Evidence (run 17810 / job 33895 / task 34532 on 0e5152c3): - 5x --- FAIL lines confirmed in actions_log/molecule-ai/molecule-core/e4/34532.log - delegation_test.go:1110/1176/1228/1271: "unmet sqlmock expectations" - mcp_test.go:433: "error message should mention GLOBAL, got: tool call failed" Gitea 1.22.6 quirk #10 confirmation: per the run, job-level continue-on-error DID still allow the combined commit-status to show neutral/success when the job logically failed — so the #656 PR check showed green even with these underlying failures masked. Reproduced. Co-Authored-By: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>	2026-05-12 04:40:32 +00:00
Molecule AI Core-DevOps	1719534bf3	fix(ci): RFC internal#219 Phase 4 — all-required sentinel enforced, stable jobs hard-fail ... Phase 4 of the force-merge protection fix (internal#219 §2). Changes: - audit-force-merge.yml REQUIRED_CHECKS: add CI / all-required (pull_request) — closes the audit gap; force-merge audit now checks ci/all-required. - ci.yml: flip continue-on-error: false on stable jobs (changes, platform-build, canvas-build, shellcheck, python-lint) — confirmed green on main 2026-05-12 combined-status check. The all-required sentinel (continue-on-error: true) will be flipped once branch protection PATCH lands (Owner-tier, delegated separately). NOT included in this PR (separate Owner-tier action required): - Branch protection PATCH: add ci/all-required as required check on main. Needed to make the sentinel actually block merges. Delegate to Core Platform Lead. Refs: molecule-core#622, molecule-core#623	2026-05-12 04:09:44 +00:00
core-devops	5674b0e067	fix(ci): status-reaper rev4 reads per-context "status" key not "state" (compensation was unreachable since rev1) ... Schema asymmetry in Gitea 1.22.6 combined-status response: - top-level `combined.state` → uses key "state" - per-entry `combined.statuses[i].*` → uses key "status", NOT "state" Pre-rev4 the per-entry loop in reap() (and the matching is_red() / render_body() in main-red-watchdog) read `s.get("state")` only, which returned None on every real Gitea response → state coerced to "" → `"" != "failure"` guard preserved every entry → compensation path unreachable since rev1. Empirical proof (orchestrator probe 2026-05-12 03:42Z): GET /repos/molecule-ai/molecule-core/commits/210da3b1/status → 29 per-entry items, ALL have key "status", ZERO have key "state". status value distribution: {success:18, failure:8, pending:3}. rev3 production run 17516 reported preserved_non_failure=585=30*19.5 (every context across all 30 SHAs preserved, none compensated) despite the same SHAs showing ~25 real failures via direct probe. Fix is one line per call site: s.get("state") → s.get("status") or s.get("state") The `state` fallback is defensive — keeps rev1-3 fixtures green and absorbs a hypothetical future Gitea version that emits both keys. Sibling-script audit: - main-red-watchdog.py: same bug at 3 sites (filter in is_red, display in render_body, debug dict in run_once). Bundled here because the fix is structurally identical and the failure mode matches. - ci-required-drift.py: no per-entry status iteration. Clean. Test gap (rev1-3 fixtures mirrored the bug): All 42 reaper fixtures + 26 watchdog fixtures used "state" per entry — same wrong key. That's why rev1-3 tests stayed green while the production code was no-op. Logged under `feedback_smoke_test_vendor_truth_not_shape_match`. New tests (8 total: 4 reaper + 4 watchdog) explicitly use the vendor-truth `status` per entry. Hostile self-review: temporarily reverted the reaper fix and re-ran — new tests FAILED at exactly the predicted assertion `assert counters["compensated"] == 1` → proves they're load-bearing, not tautological. Cross-links: task #90 (orchestrator), task #46 (hongming-pc2 paired investigation) PR #618 (rev1), PR #633 (rev2), PR #650 (rev3 widened window)	2026-05-11 20:44:20 -07:00
Molecule AI Core-DevOps	1c9255125e	fix(ci): make go vet hard-failing in weekly-platform-go	2026-05-12 03:37:52 +00:00
claude-ceo-assistant	f9214391fb	Merge branch 'main' into fix/audit-force-merge-pipefail	2026-05-12 03:34:13 +00:00
core-devops	fae62ac8c1	fix(ci): status-reaper rev3 widens window 10->30 + raises watchdog timeout + re-enables both crons ... Phase 1+2 evidence (rev2 PR#633, merged 01:48Z): 6/6 ticks post-merge with `compensated:0` despite ~25 known-stranded reds visible across those same 10 SHAs on direct probe ~30min later. Reaper run 17057 at 02:46Z explicitly logged: scanned 42 workflows; push-triggered=19, class-O candidates=23 status-reaper summary: {compensated:0, preserved_non_failure:185, scanned_shas:10, limit:10} Root cause: schedule workflows post `failure` to commit-status RETROACTIVELY 5-15 min after their merge. By the time reaper's next */5 tick lands, the stranded red is on a SHA that has already fallen OUTSIDE a 10-commit window during a burst-merge period. Reaper algorithm is correct; the lookback window is too narrow vs. the retroactive-failure-post lag. Three-in-one fix (atomic per hongming-pc2 GO 03:25Z): 1. `.gitea/scripts/status-reaper.py` DEFAULT_SWEEP_LIMIT 10 -> 30. Trades window-width-cheap for cadence-loady; kept `*/5` cron unchanged (avoiding `*/2` which would double runner load). 2. `.gitea/workflows/status-reaper.yml` Restore schedule cron block (revert mc#645 comment-out for THIS workflow only). Cron stays `*/5 * * * *`. 3. `.gitea/workflows/main-red-watchdog.yml` Restore schedule cron block (revert mc#645 comment-out) AND raise job-level `timeout-minutes: 5 -> 15`. Original 5min cap was producing cancels under runner-saturation latency, which fed the very `[main-red]` issues this workflow files (self-poisoning). 4. `tests/test_status_reaper.py` + test_default_sweep_limit_is_30 (contract pin) + test_reap_widened_window_catches_retroactive_failure: mocks 30 SHAs, plants the failing context on SHA[20] (depth strictly past rev2's window=10), asserts the compensation POST lands on that SHA. Existing tests retain explicit `limit=10` overrides and remain unchanged. Suite: 42/42 passed (was 40 + 2 new). Verification plan (post-merge, 10-15 min after merge / 2-3 cron ticks): - DB: SELECT id, status FROM action_run WHERE workflow_id= 'status-reaper.yml' ORDER BY id DESC LIMIT 5 -> all status=1 - Log via web UI: /molecule-ai/molecule-core/actions/runs/<index>/jobs/0/logs -> summary line should now show compensated > 0 with compensated_per_sha populated - Direct probe: pick a SHA in the last 30 main commits with class-O fails, GET /repos/molecule-ai/molecule-core/commits/{sha}/status -> compensated contexts now show state=success with description starting 'Compensated by status-reaper' If rev3 STILL shows compensated:0 after the window-widening, the diagnosis is wrong and a DIFFERENT bug needs to be uncovered (per hongming-pc2 caveat 03:25Z). Re-enabling the crons IS the diagnosis verification. Cross-links: - PR#618 (rev1, drop-concurrency, merge 4db64bcb) - PR#633 (rev2, sweep-recent-commits, merge e7965a0f) - PR#645 (interim disable, merge 4c54b590) — re-enable being reverted - task #90 (orch rev3 tracker) / task #46 (hongming-pc2 tracker) - feedback_brief_hypothesis_vs_evidence (empirical evidence above) - feedback_strict_root_only_after_class_a (3-in-one root fix vs. longer patching chain) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 20:29:06 -07:00
Molecule AI Infra-Runtime-BE	8c343e3ac4	fix(gitea): add || true guards to jq pipelines in audit-force-merge.sh ... Same root cause as sop-tier-check.sh (commit a1e8f46): when GITEA_TOKEN is empty or returns a non-JSON error page, the jq pipeline exits 1, triggering set -e and aborting before the SOP_FAIL_OPEN fallback can run. Added || true to all jq-piped variable assignments: - MERGE_SHA, MERGED_BY, TITLE, BASE_BRANCH, HEAD_SHA extractions (lines 52-56): guard against malformed/empty PR JSON - process-substitution in the status-check while loop (line 78): guard against empty/invalid STATUS response - FAILED_JSON construction (line 100): guard against empty FAILED_CHECKS array producing empty-pipeline jq failures Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 03:26:36 +00:00
Molecule AI Core-DevOps	df821c8258	fix(ci): sop-tier-check gracefully handles empty/invalid token ... SOP_FAIL_OPEN=1 was not preventing CI failures because three API calls with `set -euo pipefail` would abort the script before reaching the SOP_FAIL_OPEN exit block: 1. `WHOAMI=$(curl ... | jq -r ...)` — jq exits 1 on empty input, triggering set -e → script exits before SOP_FAIL_OPEN check. 2. `curl` for reviews — curl exits non-zero on 401 from empty token, triggering set -e → same problem. 3. `curl` for org teams list — same issue. Fix: add `|| true` to jq pipelines and `set +e` / `set -e` guards around curl calls that may fail with empty token. When SOP_FAIL_OPEN=1 and the token is invalid, the script now exits 0 instead of 1, preventing blocking CI failures on unconfigured runners. Refs: sop-tier-check failure on PRs #617, #621, #587, #562	2026-05-12 03:16:17 +00:00
Molecule AI Core-DevOps	7d011828e8	fix(ci): ci-required-drift handles 403/404 on protection endpoint gracefully ... Root cause: DRIFT_BOT_TOKEN lacks repo-admin scope → Gitea 1.22.6's `GET /repos/.../branch_protections/{branch}` returns 403/404 → ApiError → non-zero exit → workflow red. The token trail (internal#329) was never completed for mc-drift-bot on molecule-core. Fix (script): catch ApiError on the protection fetch; on 403/404 log a clear ::error:: diagnostic explaining the token-scope gap and return empty findings (skip this branch). The issue IS the alarm, not a red workflow. 5xx is still propagated (transient outage). Fix (workflow): remove stale transitional comment that claimed the all-required sentinel didn't exist yet (it landed in #553). Fixes: infra/ci-required-drift red on main (210da3b1→4db64bcb). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 03:13:37 +00:00
claude-ceo-assistant	6ee9ecdf0d	fix(ci)(interim): disable status-reaper + main-red-watchdog crons ... RFC#420 Option-C machinery has been down ~2.5h: - status-reaper rev2 (PR#633, merged 01:48Z): 0 'Compensated by status-reaper' status on the last 14 main commits. Schedule reds stranded on stale commits despite the rev2 sweep-last-10 design. - main-red-watchdog: 'Failing after 10m56s' with timeout-minutes:5 — runner saturation queue-lag pushed it past its own timeout. No [main-red] issues filed during the outage despite 5 reds on HEAD e7965a0f at the high watermark. Both workflows were themselves contributing to the red pileup on main + queuing the ubuntu-latest pool. Cheap-and-safe interim: comment out the schedule: blocks. workflow_dispatch: stays so they can be triggered manually for debugging. Re-enable after: 1. rev3 lands (likely scan_workflows() should LOG-and-skip rather than sys.exit on a malformed workflow; list_recent_commit_shas() should degrade gracefully) 2. Dedicated status-ops runner-label (route status-reaper + watchdog + ci-required-drift to it so they don't queue behind CI-merge-churn) Per hongming-pc2 02:31Z directive: 'pick one: rev3+raise-timeout OR temporarily disable the crons'. Choosing disable for safety while rev3 investigation proceeds. Reviewed-by: hongming-pc2 (pre-APPROVE on sight 02:31Z) Author: claude-ceo-assistant (orchestrator emergency; operator-host unreachable 02:01-02:38Z blocked SSH-bridge to core-devops persona) Cross-links: task #90 (rev2), task #75 (main-red sweep), RFC#420 Option-C	2026-05-11 19:39:43 -07:00
Molecule AI · core-lead	2ca0433a35	Merge branch 'main' into ci/review-check-tests-wire	2026-05-12 01:55:16 +00:00
core-devops	98323734ea	feat(ci): status-reaper rev2 sweeps last 10 main commits (closes stranded-status gap) ... rev1 (PR #618, merged 4db64bcb) only inspected the CURRENT main HEAD per tick. Schedule workflows post `failure` to whatever SHA was HEAD when the run COMPLETED, which by the next */5 tick is usually a stale commit because main has already moved forward via merges. Result: rev1 was running successfully but with `compensated:0` on every tick across ~6 cycles (orchestrator + hongming-pc2 Phase 1+2 evidence 23:46Z / 23:59Z / 00:02Z); reds stranded on stale commits. rev2 sweeps the last 10 main commits per tick: - New `list_recent_commit_shas(branch, limit)` wraps GET /repos/{o}/{r}/commits?sha={branch}&limit={limit}. Vendor-truth probe 2026-05-11 confirms Gitea 1.22.6 returns a JSON list of commit objects with `sha` keys (per `feedback_smoke_test_vendor_truth_not_ shape_match`). - New `reap_branch()` orchestrates the sweep: - For each SHA: GET combined status with PER-SHA ERROR ISOLATION (refinement #7) — ApiError on one stale SHA logs `:⚠️:` and continues to the next. Different from the single-HEAD pre-rev2 path where fail-loud was correct; the sweep is best-effort across historical commits. - When `combined.state == "success"`: skip the per-context loop entirely (refinement #2, cost optimization, common case). - Otherwise delegate to the existing per-SHA `reap()` worker (logic UNCHANGED — `_has_push_trigger` / `parse_push_context` / `scan_workflows` not touched per refinement #6). - Aggregated counters preserve all rev1 fields PLUS: - `scanned_shas`: how many SHAs we actually iterated (always 10 in normal operation; less if commits API returns fewer) - `compensated_per_sha`: {<full_sha>: [<context>, ...]} for the SHAs that actually got at least one compensation - `reap()` now also returns `compensated_contexts` so `reap_branch()` can build `compensated_per_sha` without re-deriving it from the POST stream. Backwards-compatible — all existing test assertions check specific counter keys, none enforce a closed dict shape. - `main()` switches from `get_head_sha` + `get_combined_status` + `reap` to a single `reap_branch()` call. Adds `--limit` CLI flag for ops-driven sweep-width tuning (default 10). Design choices (refinements 1-4): - N=10: covers the burst-merge window between */5 ticks; older reds falling off acceptable (the schedule run that posted them has long since been overwritten by a real push trigger). - Skip combined=success early: most commits in the window will be green; short-circuit before the per-context loop saves work. - No de-dup needed (refinement #4): each workflow run posts to exactly one SHA, so two different SHAs in the sweep cannot have the same (context) pair eligible for compensation. Test suite: 37 + 3 = 40/40 cases pass. - New: test_reap_sweeps_n_shas_smoke (mock 3 SHAs, verify each GET'd) - New: test_reap_skips_combined_success_shas (verify the combined=success short-circuit; only the 1 failure SHA is iterated) - New: test_reap_continues_on_per_sha_apierror (per-SHA error isolation contract — ApiError on SHA[0] logged + skipped + SHA[1] processes) - All 37 existing rev1 tests pass unchanged (per-SHA worker logic + the helpers it consumes are untouched). Live dry-run smoke against git.moleculesai.app: scanned 41 workflows; push-triggered=18, class-O candidates=23 summary: {"branch":"main","compensated":0,"compensated_per_sha":{}, "dry_run":true,"limit":10,"preserved_non_failure":196, ...,"scanned_shas":10} Cross-link: - internal#327 (sibling publish-runtime-bot) - task #90 (orchestrator brief), task #46 (hongming-pc2 brief) - PR #618 (parent rev1, merge 4db64bcb) - `reference_post_suspension_pipeline` - `feedback_no_shared_persona_token_use` (commit author = core-devops, not hongming-pc2) - `feedback_strict_root_only_after_class_a` (root cause, not symptom) - `feedback_brief_hypothesis_vs_evidence` (evidence: compensated:0 across 6 cycles) Removal path: drop this workflow when Gitea >= 1.24 ships with a real fix for the hardcoded-suffix bug. Audit issue (filed alongside rev1) tracks the deletion as a follow-up sweep.	2026-05-11 18:41:39 -07:00
Molecule AI Core-DevOps	c74c0a0283	fix(ci): add jq install to review-check-tests workflow + fix /tmp/jq hardcode ... Two fixes found during first CI run: 1. Workflow missing jq installation step — T12 jq-filter test needs jq which is not in the Gitea Actions ubuntu-latest runner image. Add the same install dance as sop-tier-check.yml (apt-get first, GitHub binary download fallback, infra#241 belt-and-suspenders). 2. test_review_check.sh hardcodes /tmp/jq in T12. In CI jq gets installed to /usr/bin/jq via apt-get. Fix: use `command -v jq` to resolve from PATH first, fall back to /tmp/jq for local dev. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 01:24:24 +00:00
Molecule AI Core-DevOps	a2a1e644ab	feat(ci): wire review-check.sh regression tests into CI (closes #540) ... New workflow .gitea/workflows/review-check-tests.yml triggers on every PR + push that touches review-check.sh or its test fixtures. Runs the existing 22-scenario regression suite (test_review_check.sh) which covers all issue #540 acceptance criteria. CONTRIBUTING.md updated with: - review-check-tests row in the CI job table - Local testing section with the smoke command Note: tests are bash-based (not bats) per existing test_review_check.sh design. Converting to bats would be refactoring rather than closing the gap. Bats dependency was never added to the runner-base image. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 01:24:24 +00:00
core-devops	9b10af08c9	fix(ci): status-reaper drops broken concurrency block (Gitea 1.22.6 cancel-cascade)	2026-05-12 00:41:36 +00:00
Molecule AI Core-BE	e05fb6911d	feat(ci): add per-package diagnostic step to platform-build job ... Adds a continue-on-error step that runs ./internal/handlers/... and ./internal/pendinguploads/... with -v -timeout 60s, tee-ing output to /tmp/ and emitting last-100-lines to step summary. Gitea Actions logs API returns 404 (gitea/gitea#22168), making the run-page step summary the only available signal when CI stalls. Step is stripped before merge. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 00:07:45 +00:00
Molecule AI Infra-SRE	3206966ee0	revert(ci): restore ubuntu-latest runner for publish workflows ... REVERT of #599 (infra/docker-runner-label) — urgent CI regression fix. The `docker` label is NOT registered on any act_runner. With runs-on: [ubuntu-latest, docker], publish-workflow jobs queue indefinitely with zero eligible runners — strictly worse than the pre-#599 coin-flip (50% success rate). Restore runs-on: ubuntu-latest so publish-workflow jobs can run again. The docker-label registration is the hard prerequisite that must be satisfied before re-applying #599. Fixes: publish-workspace-server-image + publish-canvas-image stuck in "Waiting to run" since #599 merged ~23:24Z. To re-apply: once `docker` label is registered on ≥2 runners, re-apply the runs-on: [ubuntu-latest, docker] change from #599 (branch infra/docker-runner-label). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 00:02:03 +00:00
Molecule AI Infra-Runtime-BE	a50cce0590	feat(ci): add weekly Platform-Go latent-error surface workflow ... Runs the full Platform-Go suite (build, vet, golangci-lint, tests with coverage thresholds) every Monday at 04:17 UTC regardless of whether workspace-server/ was touched by the last push. Background: ci.yml's platform-build gates real work on `needs.changes.outputs.platform == 'true'`. When no push touches workspace-server/, the suite never executes on main, so latent vet errors and test flakes can sit for weeks undetected. This workflow surfaces those errors in advance so the next workspace-server push doesn't trigger unexpected failures. Closes #567. Closes molecule-core#567. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 23:49:59 +00:00
Molecule AI Core-DevOps	0f63b7177a	fix(sre): add explicit 15s timeout to gate-check-v3 HTTP calls (closes #603) ... Adds DEFAULT_TIMEOUT=15 to gate_check.py and passes it to all urlopen() calls (api_get, comment POST, comment PATCH). Adds socket.setdefaulttimeout(15) to the inline Python in the workflow's cron step, catching the PR-polling loop too. Defence-in-depth: the real fix is provisioning SOP_TIER_CHECK_TOKEN in Gitea; this caps worst-case wall-clock at ~15 s per call when the token is missing or Gitea is unreachable. Fixes issue #603. Note: PR #603 (da1487ad) has the same changes but is missing `import socket` in the inline Python — that version would NameError at runtime. This branch carries the complete fix. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 23:36:21 +00:00
core-devops	afaf0a1e54	feat(ci): status-reaper compensates Gitea hardcoded-(push)-suffix on schedule-triggered operational workflow failures ... Root cause (verified via runs 14525 + 14526): Gitea 1.22.6 emits commit-status context as <workflow_name> / <job_name> (push) for ANY workflow run on the default-branch HEAD, REGARDLESS of the trigger event. Schedule- and workflow_dispatch-triggered runs therefore paint main red via a fake-push status. No upstream fix in 1.23-1.26.1 (sibling a6f20db1 research; internal#80 RFC). Design — Option B (b2 cron-based compensating-status POST): workflow_run is NOT supported on Gitea 1.22.6 (verified via modules/actions/workflows.go enumeration); cron is the only event-shaped option that fires reliably. Every 5min, .gitea/workflows/status-reaper.yml runs a stdlib + PyYAML scanner that: 1. Walks .gitea/workflows/*.yml. Resolves each workflow_id from top-level 'name:' (else filename stem). Fails LOUD on name-collision OR '/' in name (would break ' / ' context parsing downstream). Classifies each by 'push:' trigger presence (str / list / dict on: shapes all handled). 2. Reads main HEAD's combined commit status. 3. For each failure-state context ending ' (push)': - parses '<workflow_name> / <job_name> (push)'; - skips if workflow not in scan map (conservative); - preserves if workflow has push: trigger (real defect); - else POSTs state=success with the same context to /repos/{o}/{r}/statuses/{sha}, with a description that documents the workaround. Safety: - Only failure-state contexts whose suffix is ' (push)' are compensated. Branch_protections required checks on main (Secret scan, sop-tier-check) have ' (pull_request)' suffix — UNREACHABLE from this code path. Verified 2026-05-11 + test test_reap_required_check_pull_request_suffix_never_touched. - publish-workspace-server-image has a real push: trigger → PRESERVED. mc#576's docker-socket failure stays visible as intended. Explicit test fixture. - api() raises ApiError on non-2xx + JSON-decode failure per feedback_api_helper_must_raise_not_return_dict. Pre-fix 'soft-fail' would silently paint main green via omission. Persona: claude-status-reaper (Gitea uid 94, write:repository) — provisioned 2026-05-11 21:39Z by sub-agent aefaac1b. Token under secrets.STATUS_REAPER_TOKEN (no other write surface touched). Acceptance (post-merge verify, Step-5): Trigger one class-O workflow via workflow_dispatch (e.g. sweep-cf-tunnels). Observe reaper compensate the resulting (push)-suffix failure on the next 5-min tick. Real push-triggered failures (publish-workspace-server-image) MUST still red main. Removal path: Drop this workflow + script + tests when Gitea is upgraded to >= 1.24 with a fix for the hardcoded-suffix bug, OR when an upstream patch lands (internal#80 RFC). Tracked in post-merge audit issue. Cross-links: - sibling internal#327 (publish-runtime-bot) - sibling internal#328 (mc-drift-bot) - sibling internal#329 (Gitea dispatcher race) - sibling internal#330 (disk-GC cron Gitea-class bug) - upstream internal#80 (Gitea hardcoded-suffix RFC) - mc#576 (preserved by design — real push-trigger failure) - sub-agent aefaac1b (provisioning sibling) - sub-agent a6f20db1 (Option A research — no upstream fix) Tests: 37 pytest cases pass (incl. hongming-pc 22:08Z review's 3 design checks: name-collision fail-loud, '/' in name lint, name vs filename fallback).	2026-05-11 23:24:54 +00:00
Molecule AI Core-DevOps	e8c78d6a20	fix(ci): pin docker-capable runner label in both publish workflows (closes #576) ... Coin-flip failure: publish-workspace-server-image / build-and-push lands on runners without /var/run/docker.sock (molecule-runner-1 vs molecule-runner-4), failing the Docker daemon health check. Fix: - runs-on: ubuntu-latest → runs-on: [ubuntu-latest, docker] infra-sre registers a `docker` label on every act-runner that mounts /var/run/docker.sock (group=docker, perms 660+). Jobs without the `docker` label are never queued on socket-less runners. - Health check step now echoes the runner hostname in both the success path and the error path so failures are traceable to a specific host. Applied to: .gitea/workflows/publish-workspace-server-image.yml .gitea/workflows/publish-canvas-image.yml Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 23:19:53 +00:00
Molecule AI Infra-Runtime-BE	1688c1a991	fix(ci): strip JSON5 comments from manifest.json before clone-manifest.sh ... Integration Tester appends a trailing `// Triggered by ...` comment to manifest.json on each run. This is valid JSON5 but breaks `jq` which clone-manifest.sh uses to parse the file — causing publish-workspace-server-image and harness-replays to fail on every run. Fix: pipe manifest.json through `sed '/^[[:space:]]*\/\//d'` before passing to clone-manifest.sh, producing a clean JSON file for jq. harness-replays.yml: also downgrade the missing-token check from `exit 1` to a warning, consistent with publish-workspace-server-image.yml. All repos are public per the manifest.json OSS surface contract — token is only needed for private repos. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 22:19:55 +00:00
Molecule AI Core-DevOps	4b371918ec	fix(ci): all-required sentinel skips null-result Phase-3 jobs ... Fixes CI / all-required hard-failing on PRs during Phase 3 (RFC #219 S1). continue-on-error: true on all-required: prevents the sentinel from hard-blocking PRs while underlying build jobs use continue-on-error: true (Phase 3 surfacing contract). When Phase 3 ends, remove this so the sentinel again hard-fails on real failures. Assertion skips null results: toJSON(needs) returns result=null for Phase-3 suppressed jobs and in-flight jobs. The check excludes null from the bad-list rather than treating it as failure. Adds WARN: for in-flight null results so operators can see pending jobs without failing the gate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 22:02:02 +00:00
Molecule AI Infra-Runtime-BE	c58aef31e7	fix(ci): publish-workspace-server-image — remove mandatory AUTO_SYNC_TOKEN check ... The `Pre-clone manifest deps` step exits with error if AUTO_SYNC_TOKEN is not set. This was a safety belt added during initial development, but it is wrong: manifest.json explicitly records all listed repos as public on git.moleculesai.app (OSS surface contract). The token is only needed for private repos, which are handled at provision-time via the per-tenant credential resolver. Removing the hard exit lets the workflow succeed when: - AUTO_SYNC_TOKEN is absent (anonymous clone works for public repos) - AUTO_SYNC_TOKEN is set (authenticated clone still works) No functional change to the clone-manifest.sh call itself. Part of internal#327 / #561.	2026-05-11 21:30:37 +00:00
Molecule AI · infra-sre	b575ab8266	Merge branch 'main' into ci/540-review-check-bats-tests	2026-05-11 20:45:21 +00:00
Molecule AI Infra-Runtime-BE	8a7ca8ed33	fix(ci): publish-runtime-autobump bump-and-tag condition is always-skipped ... `if: github.event.pull_request.base.ref == ''` was meant to gate bump-and-tag to push events (not pull_request events which route to pr-validate). However, on a PR-merge push in Gitea Actions, the pull_request context is still attached with base.ref='main', so the condition always evaluated to false and bump-and-tag was permanently skipped. Fix: replace with `if: github.event_name == 'push'` which correctly fires only on branch pushes after the PR is merged. Also add `workflow_dispatch` trigger so the workflow can be manually dispatched when the Gitea Actions API (/actions/*) is unreachable (act_runner 404 on Gitea 1.22.6 — internal#327). Closes internal#327.	2026-05-11 20:41:57 +00:00
Molecule AI Core-DevOps	43cc27ade5	test(ci): add bats-style integration tests for review-check.sh (#540) ... Add 13 test cases (22 assertions) covering all key paths: - open/closed PR handling - non-author APPROVED review detection - dismissed review exclusion - team membership probe (204 member, 404 not-member, 403 fail-closed) - missing GITEA_TOKEN exits 1 - CURL_AUTH_FILE mode 600 and header format - jq filter correctness Uses a Python HTTP fixture server that reads scenario from a temp state dir, with a curl shim rewriting https://fixture.local/* to http://127.0.0.1:{port}/*. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 20:33:14 +00:00
Molecule AI Core-DevOps	4045fa4fec	feat(ci): add OCI labels + buildx to publish-workspace-server-image.yml (#554) ... Add all 4 OCI provenance labels (RFC internal#229 §X step 4 PR-1): - org.opencontainers.image.source — fixed from github.com → git.moleculesai.app - org.opencontainers.image.revision — GIT_SHA - org.opencontainers.image.created — ISO-8601 UTC timestamp - molecule.workflow.run_id — GITHUB_RUN_ID Switch docker build → docker buildx build + --push for both platform and tenant images. This enables future digest capture via `docker buildx imagetools inspect` in the CP atomic pin-update step. Uses pinned docker/setup-buildx-action@v4.0.0 (same version as publish-canvas-image.yml). docker buildx is pre-installed on Gitea Actions runners per workflow header. Part 1 of 2 for #554. Part 2 (atomic CP pin update via POST /cp/admin/runtime-image-pins) depends on the CP endpoint being available — tracked as PR-3 sub-issue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 20:04:19 +00:00
Molecule AI · core-devops	02aed70291	fix(ci): ci-required-drift uses scoped mc-drift-bot token (mirrors controlplane) ... Companion to molecule-controlplane PR#134. The `ci-required-drift` detector calls GET /repos/{owner}/{repo}/branch_protections/{branch}, which Gitea 1.22.6 gates behind the repo-ADMIN role. The previous fallback chain (`secrets.SOP_TIER_CHECK_TOKEN || secrets.GITHUB_TOKEN`) had only read or write — neither admin — so drift runs would 403. Switch to `secrets.DRIFT_BOT_TOKEN`, owned by the new least-privilege `mc-drift-bot` persona (team: drift-bot, permission: admin, scope: read:repository,write:issue,read:organization, repos: this + CP). Note: this repo's drift detector additionally requires the `all-required` sentinel job in ci.yml, which is being added in PR#553. After both PRs merge the drift workflow will be fully green. Audit trail in internal#329. Sibling pattern: internal#327 (publish-runtime-bot). Per feedback_per_agent_gitea_identity_default.	2026-05-11 12:47:51 -07:00
core-devops	22a1752eb3	feat(ci): add all-required sentinel job (RFC#219 Phase 4 / closes internal#286) ... Adds the `all-required` aggregator sentinel job to .gitea/workflows/ci.yml, mirroring the molecule-controlplane Phase 2a impl. The sentinel needs every non-event-gated job (changes, platform-build, canvas-build, shellcheck, python-lint) and asserts result==success per dep so skipped-as-green can't sneak through. Two immediate effects: 1. .gitea/workflows/ci-required-drift.yml stops hard-failing with exit 3 on the missing sentinel (see comment lines 26-31 of that workflow). 2. Branch protection can now (Step 5 follow-up, separate PR per feedback_never_admin_merge_bypass) point status_check_contexts at the single 'ci / all-required (pull_request)' name and CI churn underneath no longer requires protection edits. NOT in this PR (deferred Step 5 follow-up): - PATCH branch_protections/main to add 'ci / all-required (pull_request)' to status_check_contexts — Owners-tier change, separate PR. - Mirror the same context into audit-force-merge.yml REQUIRED_CHECKS env (RFC §6 — drift detector F3 will flag if the two diverge). Refs: - internal#219 (parent RFC, §2 Aggregator sentinel) - internal#286 (Phase 4 emergency bump — 2026-05-11 broken-merge evidence) - molecule-controlplane Phase 2a (reference impl, CP PR#112) - feedback_phantom_required_check_after_gitea_migration (incident class) - feedback_path_filtered_workflow_cant_be_required (sentinel has no paths: filter; fires on every push/PR per RFC §2) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 19:44:52 +00:00
Molecule AI Core-DevOps	f36052b0ff	fix(ci)(security): revert gate-check-v3 checkout to base SHA (internal#116 footgun) ... pull_request_target runs with the repo's secrets-context. Checking out github.event.pull_request.head.sha means a PR that modifies tools/gate-check-v3/gate_check.py executes that modified script with secrets. This is the canonical pull_request_target footgun. Fix: checkout base SHA instead of head SHA for pull_request_target events. Bug-1 (self-loop exclusion) and Bug-3 (403→exit0) from #547 are kept; only the checkout-ref regresses to the pre-#547 base-branch behavior. Refs: #551, internal#116, RFC#324 A4, feedback_pull_request_target_workflow_from_base Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 19:35:50 +00:00
Molecule AI Core-DevOps	c7d5089586	fix(ci)(security): stop token appearing in curl argv (#541) ... Token (especially long-lived RFC_324_TEAM_READ_TOKEN org-secret) passed via -H "Authorization: token ${TOKEN}" is visible in /proc/<pid>/cmdline and ps -ef on the runner host. Fix: write token to a mode-600 temp file and pass it to curl via -K (curl config file). The token never appears in the argv of any process; curl reads it from the fd-backed file. Affected: - .gitea/scripts/review-check.sh: CURL_AUTH_FILE + -K on all 3 curl calls - .gitea/workflows/qa-review.yml: privilege-check inline curl - .gitea/workflows/security-review.yml: privilege-check inline curl Fixes: #541 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 19:30:22 +00:00
Molecule AI Infra-SRE	2843d6214c	fix(ci): gate-check-v3 workflow uses PR branch (head) for script ... The gate-check job now checks out github.event.pull_request.head.sha instead of base.sha. This ensures that script fixes in PR branches (e.g. the self-loop exclusion in signal_6_ci) are actually used when evaluating that PR. Security note: this job only runs the read-only gate-check script (API reads + JSON stdout) and has continue-on-error: true, so running PR-branch code here carries minimal risk. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 19:26:23 +00:00
core-devops	ecbfa60f04	fix(ci): close fail-open in qa/security review checks (RFC#324 v1.3 §A1.1) + drop dead jq fallback ... Addresses hongming-pc review #1421 on PR #535. Blocker 1 (fail-open privilege gate): Original v1.2 design `if:`-gated the "Check out BASE" and "Evaluate" steps on the privilege-check step's `proceed` output. A non-collaborator commenting `/qa-recheck` produced proceed=false → both steps skipped → job conclusion = success → `qa-review / approved` context published as success with ZERO real APPROVE. Any visitor could green the gate. Fix per RFC#324 v1.3 §A1.1 option (b): drop privilege-gating of the eval entirely. The eval is read-only and idempotent (reads pulls/{N}/reviews + teams/{id}/members/{u}, both server-side state uninfluenced by who commented). Re-running on a non-collaborator's comment is harmless: if a real team-member APPROVE exists, the eval flips green; if not, it stays red. The privilege step is retained as a `::notice::` log line only (griefer-spotting), not a gate. Non-blocking nit 5 (dead jq fallback): `apt-get install jq` (no root) and `curl -o /usr/local/bin/jq` (no write perm on uid-1001 rootless runner) both can't succeed. Per feedback_ci_runner_install_needs_writable_path + #391/#402, jq is already baked into runner-base. Replace the install dance with a clear `exit 1` + diagnostic so a missing-jq runner fails loud rather than confusingly. Smoke-test (mocked Gitea API): no-approve → exit 1 (gate red) self-approve → exit 1 (gate red) dismissed-approve → exit 1 (gate red) non-team-approve → exit 1 (gate red) team-approve → exit 0 (gate green) Blocker 2 (A1-α event-suffix context-name verification) is the smoke-PR's job and is flagged in a follow-up comment on this PR — does not require workflow changes here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 11:45:59 -07:00
core-devops	e922351b78	feat(ci): add qa-review + security-review checks (RFC#324 Step 1 of 5) ... Adds the two job-conclusion-as-status review-gate workflows that will replace sop-tier-check (Step 3 of RFC#324). Both: - Trigger on pull_request_target (opened/synchronize/reopened) for the initial status, plus issue_comment for /qa-recheck and /security-recheck slash-command refire (Gitea 1.22.6 doesn't refire on pull_request_review per go-gitea/gitea#33700). - Use job name 'approved' so the published context is 'qa-review / approved' and 'security-review / approved' — NO POST /statuses, NO write:repository scope (RFC#324 v1.1 addendum A1-α). - Privilege-check slash-command commenters via /repos/.../collaborators/{u} (NOT github.event.comment.author_association — that field doesn't exist on Gitea 1.22.6, defect #1 from sop-tier-refire). - Run under pull_request_target's BASE-branch trust boundary; checkout pins to default_branch (never head.sha) and the workflows only HTTP-call the Gitea API; no PR-head code is executed (RFC#324 A4 + internal#116). Shared evaluator lives at .gitea/scripts/review-check.sh, parameterized by TEAM + TEAM_ID. Pass condition: at least one APPROVED, non-dismissed, non-author review whose user is a member of the named team. Branch-protection flip (Step 2) is intentionally NOT included in this PR. That is Owners-tier and blocked on (a) the first run of these workflows capturing the EXACT status-context names, and (b) RFC_324_TEAM_READ_TOKEN provisioning (filed as internal#325). Refs: internal#324, internal#325 (token follow-up). Closes: nothing yet — Steps 2 and 3 must land before #292/#319/#321 close. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 11:30:34 -07:00
Molecule AI Infra-SRE	48df991e6f	fix(ci): restore pull_request trigger + pr-validate to e2e-staging-saas ... PRs #516 and #530 removed the pull_request trigger from e2e-staging-saas to prevent double fires on provisioning-critical PR pushes. This caused a merge deadlock: branch protection requires status checks on every PR, but push-only workflows don't fire on PR branches, leaving required checks absent → Gitea blocks merge even though CI itself is green. Fix: restore pull_request trigger (branch protection needs status on every PR) and split the job into: - pr-validate: always posts success for pull_request paths (best-effort steps, continue-on-error: true — runner issues must not block merge) - e2e-staging-saas: guarded with `if: github.event.pull_request.base.ref == ''` so it only runs on trunk pushes, avoiding the double-fire that motivated the removal The gate-check-v3.yml workflow_dispatch.inputs removal from PRs #516/#530 is preserved unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 18:14:50 +00:00
Molecule AI Core-DevOps	bc30c3daa1	fix(ci): scope operational workflows to intended trigger windows (#504, #419) ... Issue #504: e2e-staging-saas.yml had BOTH push:[main] + pull_request:[main]. This caused the full 25-35 min staging provision+teardown cycle to fire on every PR push to main (in addition to the push trigger). The pull_request trigger is removed — branch protection ensures only merged code reaches main, so push:[main] is sufficient. Pre-merge E2E for provisioning paths is better served by local harness-replays.yml (which stays push+pull_request). Issue #419: gate-check-v3.yml had workflow_dispatch.inputs which Gitea 1.22.6 parser rejects with "unknown on type" (it mis-treats the inputs sub-keys as top-level on: event types). The entire workflow was silently ignored. Dropping the inputs block restores parsing. Manual dispatch from the Gitea UI works without the schema (github.event.inputs.X returns empty; the script iterates all open PRs when PR_NUMBER is empty). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 18:14:50 +00:00
Molecule AI Core-DevOps	783d5fb8d8	fix(ci): pass commits JSON via env block to avoid bash quoting break ... The detect-changes step's push path used `echo '${{ toJSON(github.event.commits) }}'` which broke on every main push because every main commit is a Gitea merge commit whose message contains single quotes (e.g. "Merge pull request 'fix: ...' from branch into main"). The embedded `'` ended the single-quoted bash string mid-JSON, and a subsequent `(` (e.g. in "#523)") was parsed as a subshell → "syntax error near unexpected token `('". This caused detect-changes to exit 2 → main-red. Fix: pass the JSON via an `env:` block (env values bypass shell quoting entirely) and pipe it to the script using `printf '%s' "$COMMITS_JSON"`. Closes #526. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 17:50:17 +00:00
Molecule AI Infra-SRE	6f90193382	fix(ci): add continue-on-error to publish-runtime-autobump (closes #504) ... publish-runtime-autobump fires on every push to main/staging that touches workspace/. It posts a commit status — and exits non-zero when there's nothing to bump, a DISPATCH_TOKEN is missing, or a tag already exists. None of those mean "the pushed code is broken," but they flip main's combined status to failure and trip the main-red-watchdog, generating false-positive issues (#494, #504). Fix: add `continue-on-error: true` to the autobump-and-tag job so operational failures (infra degradation, missing secrets, pre-existing tags) post success instead of failure. The fail-loud path remains in publish-runtime.yml which tests whether the runtime package actually builds and uploads. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-11 17:41:27 +00:00
Molecule AI Core-DevOps	9025e86cc7	fix(harness-replays): use github.event.commits for push event detect-changes (#499) ... Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app> Co-committed-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app>	2026-05-11 15:49:48 +00:00