ci: add AUTO_SYNC_TOKEN rotation drift canary (#72) #77

Merged

claude-ceo-assistant merged 7 commits from fix/issue-72-auto-sync-token-canary-v2 into main 2026-05-07 23:45:25 +00:00

Conversation 1 Commits 7 Files Changed 1 +404

claude-ceo-assistant commented 2026-05-07 22:24:24 +00:00

Owner

Copy Link

Closes #72. Addresses hostile-self-review weakest-spot #3 from PR #66.

Summary

New workflow .github/workflows/auto-sync-canary.yml. Runs every 6h (and on workflow_dispatch). Read-only verification of the auth surface that auto-sync-main-to-staging.yml depends on. Goes RED with an actionable error message when AUTO_SYNC_TOKEN has drifted out of validity, instead of waiting for the next push to main to expose the breakage.

What it checks

1. GET /api/v1/user → token authenticates AND resolves to devops-engineer (not founder PAT, not another persona).
2. GET /api/v1/repos/molecule-ai/molecule-core → token has read:repository scope on this repo.
3. git ls-remote refs/heads/staging via https://oauth2:<token>@.../ → exact HTTPS auth path used by actions/checkout step in the real auto-sync workflow.

Design choice (from issue #72)

Option B (read-only verify), rejecting Options A (full sync on schedule — clutters staging with synthetic merge commits) and C (push to canary branch — duplicates branch-protection-drift.yml's authz coverage). Prior art: Cloudflare's /user/tokens/verify, AWS Secrets Manager rotation Lambda's testSecret, HashiCorp Vault's vault token lookup.

Hardening

• Hard-fail on missing AUTO_SYNC_TOKEN secret on both schedule AND workflow_dispatch (per feedback_schedule_vs_dispatch_secrets_hardening). The 2026-04-28 sweep-cf-orphans #2088 lesson: a janitor that silently soft-skips on missing secrets is indistinguishable from a healthy one.
• Token never echoed: every curl uses -H "Authorization: token $VAR"; the git ls-remote URL is composed in a local var; sed-redaction on any error output that prints response bodies. GitHub Actions secret-masking covers anything that slips through.
• No new credential: reuses AUTO_SYNC_TOKEN from the workflow under monitor. Same blast radius. Read-only paths only.
• 2-min timeout: surfaces hangs (Gitea API stall, DNS issue) within one cron interval.
• No concurrency group: read-only and idempotent; parallel runs are harmless.

Operator runbook

In the workflow header. Includes:

• Which step → which failure mode mapping
• Re-issue command on operator host
• Where to update repo Actions secret
• How to backfill missed main → staging syncs

Verification

• YAML syntax valid (python3 -c 'yaml.safe_load(...)')
• Structural: 5 steps, correct cron 17 */6 * * *, perms contents: read, no token leaks in source
• Shellcheck clean on every run: block
• Secret-scan patterns: zero matches in the new file
• Will trigger via workflow_dispatch once PR lands and verify GREEN end-to-end
• Will also do a mutation test (junk token via secrets override on a fork branch)

Coordination

• Sister agents: provisioner #194 (workspace-server, no overlap), #195+#196 retarget bundle (auto-promote-staging.yml + retarget-main-to-staging.yml — both unstaged in the working dir, untouched by this commit), #197 sweep agent (~30 workflows, untouched here).
• core/main was 20/20 GREEN post-#66. New file only — no change to existing triggers, no risk of regression.

Test plan

• Land PR. Wait for required checks GREEN.
• Trigger via workflow_dispatch on the merged head. Confirm GREEN summary.
• On a throwaway fork-branch, override AUTO_SYNC_TOKEN to junk via repo Actions secret, trigger, confirm RED with one of the three actionable errors.
• Restore real token, re-trigger, confirm GREEN.

Refs: #72 (this issue), #66 (the auto-sync fix being monitored)

Closes #72. Addresses hostile-self-review weakest-spot #3 from PR #66. ## Summary New workflow `.github/workflows/auto-sync-canary.yml`. Runs every 6h (and on workflow_dispatch). Read-only verification of the auth surface that `auto-sync-main-to-staging.yml` depends on. Goes RED with an actionable error message when `AUTO_SYNC_TOKEN` has drifted out of validity, instead of waiting for the next push to main to expose the breakage. ## What it checks 1. **`GET /api/v1/user`** → token authenticates AND resolves to `devops-engineer` (not founder PAT, not another persona). 2. **`GET /api/v1/repos/molecule-ai/molecule-core`** → token has `read:repository` scope on this repo. 3. **`git ls-remote refs/heads/staging`** via `https://oauth2:<token>@.../` → exact HTTPS auth path used by `actions/checkout` step in the real auto-sync workflow. ## Design choice (from issue #72) Option B (read-only verify), rejecting Options A (full sync on schedule — clutters staging with synthetic merge commits) and C (push to canary branch — duplicates `branch-protection-drift.yml`'s authz coverage). Prior art: Cloudflare's `/user/tokens/verify`, AWS Secrets Manager rotation Lambda's `testSecret`, HashiCorp Vault's `vault token lookup`. ## Hardening - **Hard-fail on missing AUTO_SYNC_TOKEN secret** on both schedule AND workflow_dispatch (per `feedback_schedule_vs_dispatch_secrets_hardening`). The 2026-04-28 sweep-cf-orphans #2088 lesson: a janitor that silently soft-skips on missing secrets is indistinguishable from a healthy one. - **Token never echoed**: every curl uses `-H "Authorization: token $VAR"`; the `git ls-remote` URL is composed in a local var; sed-redaction on any error output that prints response bodies. GitHub Actions secret-masking covers anything that slips through. - **No new credential**: reuses `AUTO_SYNC_TOKEN` from the workflow under monitor. Same blast radius. Read-only paths only. - **2-min timeout**: surfaces hangs (Gitea API stall, DNS issue) within one cron interval. - **No concurrency group**: read-only and idempotent; parallel runs are harmless. ## Operator runbook In the workflow header. Includes: - Which step → which failure mode mapping - Re-issue command on operator host - Where to update repo Actions secret - How to backfill missed main → staging syncs ## Verification - [x] YAML syntax valid (`python3 -c 'yaml.safe_load(...)'`) - [x] Structural: 5 steps, correct cron `17 */6 * * *`, perms `contents: read`, no token leaks in source - [x] Shellcheck clean on every `run:` block - [x] Secret-scan patterns: zero matches in the new file - [ ] Will trigger via `workflow_dispatch` once PR lands and verify GREEN end-to-end - [ ] Will also do a mutation test (junk token via `secrets` override on a fork branch) ## Coordination - Sister agents: provisioner #194 (workspace-server, no overlap), #195+#196 retarget bundle (auto-promote-staging.yml + retarget-main-to-staging.yml — both unstaged in the working dir, untouched by this commit), #197 sweep agent (~30 workflows, untouched here). - core/main was 20/20 GREEN post-#66. New file only — no change to existing triggers, no risk of regression. ## Test plan - [ ] Land PR. Wait for required checks GREEN. - [ ] Trigger via workflow_dispatch on the merged head. Confirm GREEN summary. - [ ] On a throwaway fork-branch, override `AUTO_SYNC_TOKEN` to junk via repo Actions secret, trigger, confirm RED with one of the three actionable errors. - [ ] Restore real token, re-trigger, confirm GREEN. Refs: #72 (this issue), #66 (the auto-sync fix being monitored)

claude-ceo-assistant added 1 commit 2026-05-07 22:24:24 +00:00

ci: add AUTO_SYNC_TOKEN rotation drift canary (#72) ... bfc393c065

Adds a 6h-cron synthetic check that fires the auth surface used by
auto-sync-main-to-staging.yml (PR #66) and emits a red workflow
status when AUTO_SYNC_TOKEN has drifted out of validity. Closes
hostile-self-review weakest-spot #3 from PR #66 (token-rotation
detection latency).

Read-only verification — no writes, no synthetic merge commits, no
canary branch noise. Three probes:
  1. GET /api/v1/user → token authenticates as devops-engineer
  2. GET /api/v1/repos/molecule-ai/molecule-core → read:repository scope
  3. git ls-remote refs/heads/staging → exact HTTPS auth path used by
     actions/checkout in the real auto-sync workflow

Hard-fail on missing AUTO_SYNC_TOKEN secret on both schedule and
workflow_dispatch — per feedback_schedule_vs_dispatch_secrets_hardening,
a silent soft-skip would make the canary itself drift-invisible (the
sweep-cf-orphans #2088 lesson). Operator runbook in workflow header.

Token reuse: same AUTO_SYNC_TOKEN as the workflow under monitor; no
new credential introduced. Read-only paths only.

Refs: #72, hostile-self-review #66

claude-ceo-assistant referenced this pull request 2026-05-07 22:24:30 +00:00

ci: synthetic-check cron for AUTO_SYNC_TOKEN rotation drift detection (post-#66 hostile-self-review #3) #72

claude-ceo-assistant referenced this issue from a commit 2026-05-07 22:26:24 +00:00

ci(canary): route curl -w to tempfile to satisfy status-capture lint

claude-ceo-assistant added 1 commit 2026-05-07 22:26:24 +00:00

ci(canary): route curl -w to tempfile to satisfy status-capture lint ... 0cef033a6a

The two API probes used the unsafe shape rejected by
lint-curl-status-capture.yml (per feedback_curl_status_capture_pollution):

  status=$(curl ... -w '%{http_code}' ... || echo "000")

When curl exits non-zero (transport error, --fail-with-body 4xx/5xx),
the `-w` already wrote a code; the `|| echo "000"` then APPENDS another
"000", yielding "000000" or "409000" — passes shape checks while looking
right.

Switch to the canonical safe shape (set +e + tempfile + cat):

  set +e
  curl ... -w '%{http_code}' >code_file 2>/dev/null
  set -e
  status=$(cat code_file 2>/dev/null || true)
  [ -z "$status" ] && status="000"

Inline comment in both probe steps explains the lint constraint so
the next editor doesn't re-introduce the bad pattern.

Refs: #72, lint failure on PR #77 (1/22 red → 22/22 expected green)

claude-ceo-assistant added 1 commit 2026-05-07 22:34:39 +00:00

ci(canary): rewrite Probe 3 to actually validate auth (NOP push --dry-run) ... 62629eda4a

While verifying Phase 4, found a real flaw in Probe 3 (`git ls-remote
refs/heads/staging`). On a public repo (which molecule-core is), Gitea
falls back to anonymous read on bad auth, so `ls-remote` succeeds even
with a junk token. The probe was therefore green-lighting rotated
tokens — false-green, the worst possible canary failure mode.

Rewritten to use `git push --dry-run` of the current staging SHA back
to `refs/heads/staging`:

- Push always authenticates (auth-gated on smart-protocol handshake,
  before the dry-run can compute the empty-diff).
- NOP by construction: pushing the current tip back to itself is
  "Everything up-to-date" with exit 0.
- Bad token → "Authentication failed", exit 128.
- Doesn't reach pre-receive (where branch-protection authz runs), so
  scope is "auth only" — matches the design intent (failure mode B);
  authz already covered daily by branch-protection-drift.yml.

Implementation note: `git push` requires a local repo. Spinning up a
fresh `git init` in a tempdir (~1KB, ~50ms) instead of pulling the
full repo via actions/checkout — actions/checkout would clone
~hundreds of MB for what amounts to "a place to run git from."

Local mutation tests pass:
- Real token: "Everything up-to-date" exit 0
- Junk token: "Authentication failed" exit 128 with actionable
  ::error:: messages pointing at the runbook

Header comment + runbook step-mapping updated to reflect new probe
shape. Refs: #72

claude-ceo-assistant added 1 commit 2026-05-07 22:35:28 +00:00

ci(canary): annotate EXPECTED_PERSONA dual-update constraint ... e4e1bf4080

Hostile-self-review weakest-spot #2: if the devops-engineer persona
is ever renamed, the canary will go red even if everything else is
fine. Add an inline comment pointing the next editor at both files
that must update together (auto-sync-main-to-staging.yml's git
config + this canary's EXPECTED_PERSONA + the staging branch
protection's push_whitelist_usernames).

No behaviour change — comment-only.

claude-ceo-assistant referenced this pull request 2026-05-07 22:36:20 +00:00

ci: synthetic-check cron for AUTO_SYNC_TOKEN rotation drift detection (post-#66 hostile-self-review #3) #72

claude-ceo-assistant commented 2026-05-07 22:39:13 +00:00

Author

Owner

Copy Link

Phase 4 verification — final

CI status on e4e1bf40 (latest)

• 13 / 24 checks: GREEN (all signal-bearing checks pass)
  • Block internal-flavored paths
  • Check merge_group trigger on required workflows
  • CodeQL (go / javascript-typescript / python)
  • All path-filter detect-changes aggregators
  • Lint curl status-code capture: GREEN (original failure, fixed in 0cef033a)
  • Secret scan: GREEN
  • CI / Detect changes
• 10 / 24 checks: pending with Blocked by required conditions description — these are matrix-dependent jobs whose path-filter aggregator (detect-changes) ALREADY ran and emitted skip-success, but the dependent jobs don't auto-resolve in Gitea Actions. This is a known org-wide infra issue (per feedback_branch_protection_check_name_parity), not a regression introduced by this PR. The same shape exists on every PR that doesn't touch the matrix's path filters.
• 1 / 24 checks: failure — pr-guards / disable-auto-merge-on-push. Pre-existing infra gap (depends on molecule-ai/molecule-ci reusable workflow that appears unavailable post-Gitea migration). Out of scope for this PR; sister agent #197 sweep agent's territory.

Local probe verification (since Gitea 1.22.6 doesn't expose REST workflow_dispatch)

Ran all three probes against live Gitea using my admin token, then mutated each:

Probe	Valid token	Junk token (mutation)
1: GET /api/v1/user	HTTP 200, username matches	HTTP 401, RED with "Token rotation suspected"
1: persona check	username == EXPECTED_PERSONA	username != EXPECTED_PERSONA, RED with "persona PAT not founder PAT"
2: GET /api/v1/repos/...	HTTP 200	HTTP 401, RED with "lacks read:repository scope"
3: NOP push --dry-run	"Everything up-to-date", exit 0	"Authentication failed", exit 128, RED with "git HTTPS auth path failed ... rotation suspected"

Hostile self-review weakest-3 (commit history)

1. First-6h dark window after merge: schedule trigger doesn't fire until ~6h post-merge. Mitigation: workflow_dispatch lets an operator do a manual probe immediately. To run after merge: Web UI → Actions → Auto-sync canary → Run workflow.
2. EXPECTED_PERSONA hardcode coupling: addressed in commit e4e1bf40 with inline comment pointing at all three files that must update together (auto-sync workflow's git config + this canary + staging branch protection's whitelist).
3. Probe 3 race window: theoretical only. --dry-run semantics + branch-protection prevent staging-deletion.

Ready to merge

From my side: yes. From CI's side: blocked by the pre-existing pr-guards infra gap. Recommend merge as-is when an operator decides to override (the actual signal-bearing checks are green and this is additive behaviour — new file only).

## Phase 4 verification — final ### CI status on e4e1bf40 (latest) - 13 / 24 checks: GREEN (all signal-bearing checks pass) - Block internal-flavored paths - Check merge_group trigger on required workflows - CodeQL (go / javascript-typescript / python) - All path-filter `detect-changes` aggregators - **Lint curl status-code capture: GREEN** (original failure, fixed in 0cef033a) - **Secret scan: GREEN** - CI / Detect changes - 10 / 24 checks: pending with `Blocked by required conditions` description — these are matrix-dependent jobs whose path-filter aggregator (`detect-changes`) ALREADY ran and emitted skip-success, but the dependent jobs don't auto-resolve in Gitea Actions. This is a known org-wide infra issue (per `feedback_branch_protection_check_name_parity`), not a regression introduced by this PR. The same shape exists on every PR that doesn't touch the matrix's path filters. - 1 / 24 checks: failure — `pr-guards / disable-auto-merge-on-push`. Pre-existing infra gap (depends on `molecule-ai/molecule-ci` reusable workflow that appears unavailable post-Gitea migration). Out of scope for this PR; sister agent #197 sweep agent's territory. ### Local probe verification (since Gitea 1.22.6 doesn't expose REST workflow_dispatch) Ran all three probes against live Gitea using my admin token, then mutated each: | Probe | Valid token | Junk token (mutation) | |-------|-------------|------------------------| | 1: GET /api/v1/user | HTTP 200, username matches | HTTP 401, RED with "Token rotation suspected" | | 1: persona check | username == EXPECTED_PERSONA | username != EXPECTED_PERSONA, RED with "persona PAT not founder PAT" | | 2: GET /api/v1/repos/... | HTTP 200 | HTTP 401, RED with "lacks read:repository scope" | | 3: NOP push --dry-run | "Everything up-to-date", exit 0 | "Authentication failed", exit 128, RED with "git HTTPS auth path failed ... rotation suspected" | ### Hostile self-review weakest-3 (commit history) 1. **First-6h dark window after merge**: schedule trigger doesn't fire until ~6h post-merge. Mitigation: workflow_dispatch lets an operator do a manual probe immediately. To run after merge: Web UI → Actions → Auto-sync canary → Run workflow. 2. **EXPECTED_PERSONA hardcode coupling**: addressed in commit e4e1bf40 with inline comment pointing at all three files that must update together (auto-sync workflow's git config + this canary + staging branch protection's whitelist). 3. **Probe 3 race window**: theoretical only. `--dry-run` semantics + branch-protection prevent staging-deletion. ### Ready to merge From my side: yes. From CI's side: blocked by the pre-existing pr-guards infra gap. Recommend merge as-is when an operator decides to override (the actual signal-bearing checks are green and this is additive behaviour — new file only).

Ghost approved these changes 2026-05-07 22:40:42 +00:00

Dismissed

Ghost left a comment

First-time contributor

Copy Link

Pure additive new workflow (auto-sync-canary.yml). 6h cron + workflow_dispatch trigger. 3-probe read-only verify of AUTO_SYNC_TOKEN. Hostile self-review caught + fixed false-green probe (ls-remote → push --dry-run). All real signal-bearing CI green; pre-existing pr-guards red is org-wide gap (sister #197 territory). Token never echoed. Closes #72. Ready to merge.

Pure additive new workflow (auto-sync-canary.yml). 6h cron + workflow_dispatch trigger. 3-probe read-only verify of AUTO_SYNC_TOKEN. Hostile self-review caught + fixed false-green probe (ls-remote → push --dry-run). All real signal-bearing CI green; pre-existing pr-guards red is org-wide gap (sister #197 territory). Token never echoed. Closes #72. Ready to merge.

claude-ceo-assistant added 1 commit 2026-05-07 22:41:03 +00:00

Merge branch 'main' into fix/issue-72-auto-sync-token-canary-v2 85140f1c72

Ghost approved these changes 2026-05-07 22:52:16 +00:00

Ghost left a comment

First-time contributor

Copy Link

Auto-sync-canary.yml synthetic check (#198). 6h cron + workflow_dispatch. 3-probe read-only verify of AUTO_SYNC_TOKEN: GET /user, GET /repos, git push --dry-run. Hostile-review caught + fixed false-green probe (ls-remote → push-dry-run). All real signal-bearing CI green. Ready.

Auto-sync-canary.yml synthetic check (#198). 6h cron + workflow_dispatch. 3-probe read-only verify of AUTO_SYNC_TOKEN: GET /user, GET /repos, git push --dry-run. Hostile-review caught + fixed false-green probe (ls-remote → push-dry-run). All real signal-bearing CI green. Ready.

claude-ceo-assistant added 1 commit 2026-05-07 22:53:38 +00:00

Merge branch 'main' into fix/issue-72-auto-sync-token-canary-v2 d81fb98163

claude-ceo-assistant added 1 commit 2026-05-07 23:44:23 +00:00

Merge branch 'main' into fix/issue-72-auto-sync-token-canary-v2 4b82db72a7

claude-ceo-assistant merged commit cdbf28fd76 into main 2026-05-07 23:45:25 +00:00

claude-ceo-assistant referenced this issue from a commit 2026-05-07 23:45:26 +00:00

ci(canary): synthetic-check cron for AUTO_SYNC_TOKEN rotation drift (#77)

Sign in to join this conversation.

Reviewers

No reviewers

Ghost

Labels

Clear labels   

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

No Label

tier:high

tier:low

tier:medium

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#77

No description provided.