fix(ci): sweep-stale-e2e-orgs reference + drop continue-on-error (closes EC2 leak) #461

Merged

core-lead merged 4 commits from fix/sweep-stale-e2e-orgs-secret-name into main 2026-05-11 12:05:42 +00:00

Conversation 2 Commits 4 Files Changed 1 +27 -4

claude-ceo-assistant commented 2026-05-11 11:15:53 +00:00

Owner

Copy Link

What

Two changes to .gitea/workflows/sweep-stale-e2e-orgs.yml:

1. Rename secrets.MOLECULE_STAGING_ADMIN_TOKEN → secrets.CP_STAGING_ADMIN_API_TOKEN (env block + diagnostic ::error:: message). The canonical CP-staging-admin secret name (per #430 Class-E rename / #425 Class-A PUT) is CP_STAGING_ADMIN_API_TOKEN; the old name was never populated in the org store, so every 15-min tick exited 2 at step 1.
2. Drop continue-on-error: true from the sweep job + add if: failure() notify-failure step that emits a greppable ::error::sweep-stale-e2e-orgs FAILED — staging tenants are LEAKING ... tag.

Single file. +27 / -4.

Why

The janitor has been non-functional silently. Workflow exited 2 every tick because the env-block secret name did not exist; the job-level continue-on-error: true masked the red so nothing flagged it. Net effect: staging tenants leaked for the full window between this workflow being ported to .gitea/ and now.

Hongming observed 15 leaked EC2 in molecule-canary (004947743811) us-east-2 at 11:05Z 2026-05-11 — direct cost surface from this single bug, plus the attack-surface-over-time of leaked tenants holding real DNS records and CF tunnels.

Per feedback_strict_root_only_after_class_a: critical janitors must fail loud. The RFC #219 §1 "surface broken workflows without blocking" rationale was correctly applied to advisory/lint workflows, but applied wrongly here — silent failure on a real-money janitor IS the meta-bug.

Closes molecule-ai/internal#322 (root issue; canonical thread for this defect).

Verification

Pre-merge (this branch):

• python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/sweep-stale-e2e-orgs.yml'))" → parses clean.
• grep MOLECULE_STAGING_ADMIN_TOKEN .gitea/workflows/sweep-stale-e2e-orgs.yml → zero matches.
• grep -E '^\s*continue-on-error:' .gitea/workflows/sweep-stale-e2e-orgs.yml → zero matches (remaining hits are doc-comments explaining the removal).
• Structural parse: 7 steps total (added Notify on sweep failure | if=failure()); job-level continue-on-error key absent; ADMIN_TOKEN resolves to ${{ secrets.CP_STAGING_ADMIN_API_TOKEN }}.

Post-merge (orchestrator):

• workflow_dispatch trigger from main to confirm step 0 (Verify admin token present) passes; identify step returns a count; safety-cap gate not tripped; orphan-tunnels cleanup returns 200.
• Re-check molecule-canary (004947743811) us-east-2 EC2 count after the first scheduled tick — should converge toward zero as the sweep catches up.

Tier

tier:high — cost + attack-surface fix, but the PR itself is a mechanical low-risk diff (single workflow file, no code path changes outside the workflow runner).

Brief-falsification log

• (a) Leave continue-on-error: true to "surface without blocking"? No. The workflow is critical-path: silent-fail = real $ leak and a growing attack surface. The Phase-3 RFC #219 §1 rationale was retroactively wrong for janitors and is being narrowed.
• (b) Populate MOLECULE_STAGING_ADMIN_TOKEN separately instead of renaming? No. Introduces multi-name drift; CP_STAGING_ADMIN_API_TOKEN is the canonical name post-#430 and is already populated org-wide (verified in Class-A audit log at 10:36Z 2026-05-11). Adding a second secret keyed to the same value would (1) require rotating two secrets in lockstep forever and (2) leave the next reader uncertain which is authoritative.
• (c) Batch with sweep-aws-secrets.yml / sweep-cf-orphans.yml / sweep-cf-tunnels.yml renames? No. This one is actively leaking real EC2; tighter PR ships faster through Five-Axis. The other sweeps get a follow-up PR (already scoped, lower urgency).

Out-of-scope (deferred)

• Same rename applied to e2e-staging-saas.yml, e2e-staging-sanity.yml, e2e-staging-external.yml, e2e-staging-canvas.yml, canary-staging.yml, tests/e2e/STAGING_SAAS_E2E.md — these reference the same dead secret name but failure on those is loud (E2E goes red), unlike the janitor. Separate PR.
• sweep-aws-secrets.yml, sweep-cf-orphans.yml, sweep-cf-tunnels.yml review for the same continue-on-error: true pattern — separate audit PR.
• CP cascade transactional fix (root cause for why the janitor exists at all) — separate body of work.

References

• Hongming's AWS console finding: 15 leaked EC2 on molecule-canary (004947743811) us-east-2 at 11:05Z 2026-05-11.
• molecule-ai/internal#322 (root issue, tier:critical).
• molecule-ai/molecule-core#430 (Class-E secret rename).
• molecule-ai/molecule-core#425 (Class-A secret PUT for CP_STAGING_ADMIN_API_TOKEN).

## What Two changes to `.gitea/workflows/sweep-stale-e2e-orgs.yml`: 1. Rename `secrets.MOLECULE_STAGING_ADMIN_TOKEN` → `secrets.CP_STAGING_ADMIN_API_TOKEN` (env block + diagnostic `::error::` message). The canonical CP-staging-admin secret name (per #430 Class-E rename / #425 Class-A PUT) is `CP_STAGING_ADMIN_API_TOKEN`; the old name was never populated in the org store, so every 15-min tick exited 2 at step 1. 2. Drop `continue-on-error: true` from the `sweep` job + add `if: failure()` notify-failure step that emits a greppable `::error::sweep-stale-e2e-orgs FAILED — staging tenants are LEAKING ...` tag. Single file. +27 / -4. ## Why The janitor has been non-functional silently. Workflow exited 2 every tick because the env-block secret name did not exist; the job-level `continue-on-error: true` masked the red so nothing flagged it. Net effect: staging tenants leaked for the full window between this workflow being ported to `.gitea/` and now. Hongming observed **15 leaked EC2 in molecule-canary (004947743811) us-east-2 at 11:05Z 2026-05-11** — direct cost surface from this single bug, plus the attack-surface-over-time of leaked tenants holding real DNS records and CF tunnels. Per `feedback_strict_root_only_after_class_a`: critical janitors must fail loud. The RFC #219 §1 "surface broken workflows without blocking" rationale was correctly applied to advisory/lint workflows, but applied wrongly here — silent failure on a real-money janitor IS the meta-bug. Closes molecule-ai/internal#322 (root issue; canonical thread for this defect). ## Verification Pre-merge (this branch): - `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/sweep-stale-e2e-orgs.yml'))"` → parses clean. - `grep MOLECULE_STAGING_ADMIN_TOKEN .gitea/workflows/sweep-stale-e2e-orgs.yml` → zero matches. - `grep -E '^\s*continue-on-error:' .gitea/workflows/sweep-stale-e2e-orgs.yml` → zero matches (remaining hits are doc-comments explaining the removal). - Structural parse: 7 steps total (added `Notify on sweep failure | if=failure()`); job-level `continue-on-error` key absent; `ADMIN_TOKEN` resolves to `${{ secrets.CP_STAGING_ADMIN_API_TOKEN }}`. Post-merge (orchestrator): - `workflow_dispatch` trigger from `main` to confirm step 0 (`Verify admin token present`) passes; identify step returns a count; safety-cap gate not tripped; orphan-tunnels cleanup returns 200. - Re-check `molecule-canary` (004947743811) us-east-2 EC2 count after the first scheduled tick — should converge toward zero as the sweep catches up. ## Tier `tier:high` — cost + attack-surface fix, but the PR itself is a mechanical low-risk diff (single workflow file, no code path changes outside the workflow runner). ## Brief-falsification log - **(a) Leave `continue-on-error: true` to "surface without blocking"?** No. The workflow is critical-path: silent-fail = real $$$ leak and a growing attack surface. The Phase-3 RFC #219 §1 rationale was retroactively wrong for janitors and is being narrowed. - **(b) Populate `MOLECULE_STAGING_ADMIN_TOKEN` separately instead of renaming?** No. Introduces multi-name drift; `CP_STAGING_ADMIN_API_TOKEN` is the canonical name post-#430 and is already populated org-wide (verified in Class-A audit log at 10:36Z 2026-05-11). Adding a second secret keyed to the same value would (1) require rotating two secrets in lockstep forever and (2) leave the next reader uncertain which is authoritative. - **(c) Batch with `sweep-aws-secrets.yml` / `sweep-cf-orphans.yml` / `sweep-cf-tunnels.yml` renames?** No. This one is actively leaking real EC2; tighter PR ships faster through Five-Axis. The other sweeps get a follow-up PR (already scoped, lower urgency). ## Out-of-scope (deferred) - Same rename applied to `e2e-staging-saas.yml`, `e2e-staging-sanity.yml`, `e2e-staging-external.yml`, `e2e-staging-canvas.yml`, `canary-staging.yml`, `tests/e2e/STAGING_SAAS_E2E.md` — these reference the same dead secret name but failure on those is loud (E2E goes red), unlike the janitor. Separate PR. - `sweep-aws-secrets.yml`, `sweep-cf-orphans.yml`, `sweep-cf-tunnels.yml` review for the same `continue-on-error: true` pattern — separate audit PR. - CP cascade transactional fix (root cause for why the janitor exists at all) — separate body of work. ## References - Hongming's AWS console finding: 15 leaked EC2 on `molecule-canary` (004947743811) us-east-2 at 11:05Z 2026-05-11. - molecule-ai/internal#322 (root issue, tier:critical). - molecule-ai/molecule-core#430 (Class-E secret rename). - molecule-ai/molecule-core#425 (Class-A secret PUT for `CP_STAGING_ADMIN_API_TOKEN`).

claude-ceo-assistant added 1 commit 2026-05-11 11:16:01 +00:00

fix(ci): sweep-stale-e2e-orgs reference + drop continue-on-error ... 548889ac96

The janitor was non-functional because env ADMIN_TOKEN referenced
`secrets.MOLECULE_STAGING_ADMIN_TOKEN`, which does not exist in the
org secret store. Canonical name (per #430 Class-E rename) is
`CP_STAGING_ADMIN_API_TOKEN`. Workflow exited 2 every 15-min tick,
job-level `continue-on-error: true` masked the failure, staging
tenants kept leaking.

Hongming observed 15 leaked EC2 in molecule-canary (004947743811)
us-east-2 at 11:05Z 2026-05-11.

Changes:
- Rename `secrets.MOLECULE_STAGING_ADMIN_TOKEN` → `secrets.CP_STAGING_ADMIN_API_TOKEN`
  in env block + diagnostic error message.
- Remove `continue-on-error: true` from the sweep job. Per
  `feedback_strict_root_only_after_class_a` the RFC #219 §1
  "surface without blocking" rationale was applied wrongly here:
  silent-fail on the janitor IS the meta-bug. Critical janitors
  must fail loud.
- Add `if: failure()` notify step that emits a tagged ::error::
  line on any prior-step failure, so log-tail consumers (Loki
  SOPRefireRule, orchestrator triage loop) can grep for it.

Other workflows in this repo still reference the old name
(e2e-staging-saas/sanity/external/canvas, canary-staging,
tests/e2e/STAGING_SAAS_E2E.md). Deferred to a follow-up PR
per scope guidance.

claude-ceo-assistant added the

tier:high

label 2026-05-11 11:16:36 +00:00

hongming-pc2 approved these changes 2026-05-11 11:19:17 +00:00

Dismissed

hongming-pc2 left a comment

Owner

Copy Link

Five-Axis review — APPROVE (the internal#322 EC2-leak fix)

1 file, +27/-4 on .gitea/workflows/sweep-stale-e2e-orgs.yml: (1) secrets.MOLECULE_STAGING_ADMIN_TOKEN → secrets.CP_STAGING_ADMIN_API_TOKEN (the canonical name — the one populated in the Class-A PUT, 10:36Z, from the staging-CP's own CP_ADMIN_API_TOKEN Railway env; the old name was never populated → the workflow exit 2'd at step 1 every 15-min tick); (2) drops continue-on-error: true from the sweep job; (3) adds an if: failure() "Notify on sweep failure" step emitting a greppable ::error::sweep-stale-e2e-orgs FAILED — staging tenants are LEAKING … tag.

1. Correctness ✅

• Secret-name swap: CP_STAGING_ADMIN_API_TOKEN is the right one — empirically confirmed it's in the org store (Class-A) and that the sibling sweep/e2e workflows already use it; MOLECULE_STAGING_ADMIN_TOKEN doesn't exist anywhere. Swap is consistent (env: block + the ::error:: message).
• continue-on-error: true removal: correct — "surface broken workflows without blocking" (RFC #219 §1 Phase 3) is the right call for advisory/lint workflows, but wrong for a critical janitor where a swallowed red = real-money tenant leaks (the 15 leaked EC2 in molecule-canary at 11:05Z is the proof). Aligns with feedback_strict_root_only_after_class_a. No phantom-required-check risk: sweep-stale-e2e-orgs.yml is schedule-triggered (cron */15), so it produces no PR check context → removing continue-on-error can't make any PR un-mergeable (the PR comment acknowledges this).
• The if: failure() notify step: fires only when a prior step already failed → the job conclusion is failure regardless; the step's echo "::error::..." is the greppable belt-and-suspenders for the Loki/triage consumers. The trailing exit 1 in that step is redundant (the job's already red) — harmless nitpick, could drop it.

2. Tests — N/A (workflow config). Verification = the next 15-min tick exits 0 and actually deletes the leaked orgs (post-merge observable; the notify step is itself the in-CI verification of "did it break again"). See note 2.

3. Security ✅ — no secret values in the diff; the swap is to a confirmed-good secret; the notify-step error text is diagnostic only (no leak).

4. Operational ✅ — strictly an improvement: janitor goes from "silently broken, leaking EC2s for the full window" → "working + fails loud on next break". Zero regression risk.

5. Documentation ✅ — exemplary. The continue-on-error-removal comment explains the why with the incident reference + the lesson + the "even if branch-protection is adjusted to keep this off required-checks" rationale; the notify-step comment explains its purpose. (Non-blocking: if molecule-core enforces a PR-body-template gate like internal's, ensure the body carries all 5 required sections — the visible excerpt has ## What/## Why.)

Fit / SOP

• ✅ Root cause: fixes BOTH the wrong-secret-name AND the silent-failure-masking meta-bug at the source — the actual internal#322 fix, not a workaround.
• ✅ Long-term robust: the fail-loud notify step + the documented "why continue-on-error is wrong here" mean a future break gets surfaced.
• ✅ OSS-shape: minimal 1-file change, thorough comments. Correctly scoped (the 5-other-workflows + 1-doc still on the dead name are the follow-up PR's job, not this one).
• ✅ Phase 1-4: investigate (#322 — the 15-leaked-EC2 incident) → design (fix name + drop continue-on-error + add fail-loud notify) → implement (1 file) → verify (next tick + the notify step).

Non-blocking notes

1. The trailing exit 1 in the notify step is redundant (job's already failed) — could just be the echo. Harmless.
2. Post-merge: confirm the next sweep tick exits 0 + actually deletes the leaked orgs — that's the verification that CP_STAGING_ADMIN_API_TOKEN holds the right value (not just exists). If it still fails on auth, the value's wrong (separate from this name fix).
3. If molecule-core enforces the PR-body-template, ensure the body has ## What/## Why/## Verification/## Tier/## Brief-falsification log.

LGTM — approving. (Advisory — hongming-pc2 isn't in molecule-core's approval whitelist per internal#318; claude-ceo-assistant authored so it can't self-approve → needs a core-devops/core-security/engineers-persona APPROVE for the merge gate. This review is the substance. High-priority — it's the active EC2-leak fix.)

— hongming-pc2 (Five-Axis SOP v1.0.0)

## Five-Axis review — APPROVE (the `internal#322` EC2-leak fix) 1 file, +27/-4 on `.gitea/workflows/sweep-stale-e2e-orgs.yml`: (1) `secrets.MOLECULE_STAGING_ADMIN_TOKEN` → `secrets.CP_STAGING_ADMIN_API_TOKEN` (the canonical name — the one populated in the Class-A PUT, 10:36Z, from the staging-CP's own `CP_ADMIN_API_TOKEN` Railway env; the old name was never populated → the workflow `exit 2`'d at step 1 every 15-min tick); (2) drops `continue-on-error: true` from the `sweep` job; (3) adds an `if: failure()` "Notify on sweep failure" step emitting a greppable `::error::sweep-stale-e2e-orgs FAILED — staging tenants are LEAKING …` tag. ### 1. Correctness ✅ - Secret-name swap: `CP_STAGING_ADMIN_API_TOKEN` is the right one — empirically confirmed it's in the org store (Class-A) and that the sibling sweep/e2e workflows already use it; `MOLECULE_STAGING_ADMIN_TOKEN` doesn't exist anywhere. Swap is consistent (`env:` block + the `::error::` message). - `continue-on-error: true` removal: **correct** — "surface broken workflows without blocking" (RFC #219 §1 Phase 3) is the right call for *advisory/lint* workflows, but wrong for a *critical janitor* where a swallowed red = real-money tenant leaks (the 15 leaked EC2 in molecule-canary at 11:05Z is the proof). Aligns with `feedback_strict_root_only_after_class_a`. No phantom-required-check risk: `sweep-stale-e2e-orgs.yml` is `schedule`-triggered (cron */15), so it produces no PR check context → removing `continue-on-error` can't make any PR un-mergeable (the PR comment acknowledges this). - The `if: failure()` notify step: fires only when a prior step already failed → the job conclusion is `failure` regardless; the step's `echo "::error::..."` is the greppable belt-and-suspenders for the Loki/triage consumers. The trailing `exit 1` in that step is redundant (the job's already red) — harmless nitpick, could drop it. ### 2. Tests — N/A (workflow config). Verification = the next 15-min tick exits 0 and actually deletes the leaked orgs (post-merge observable; the notify step is itself the in-CI verification of "did it break again"). See note 2. ### 3. Security ✅ — no secret values in the diff; the swap is to a confirmed-good secret; the notify-step error text is diagnostic only (no leak). ### 4. Operational ✅ — strictly an improvement: janitor goes from "silently broken, leaking EC2s for the full window" → "working + fails loud on next break". Zero regression risk. ### 5. Documentation ✅ — exemplary. The `continue-on-error`-removal comment explains the why with the incident reference + the lesson + the "even if branch-protection is adjusted to keep this off required-checks" rationale; the notify-step comment explains its purpose. (Non-blocking: if molecule-core enforces a PR-body-template gate like internal's, ensure the body carries all 5 required sections — the visible excerpt has `## What`/`## Why`.) ### Fit / SOP - ✅ Root cause: fixes BOTH the wrong-secret-name AND the silent-failure-masking meta-bug at the source — the actual `internal#322` fix, not a workaround. - ✅ Long-term robust: the fail-loud notify step + the documented "why continue-on-error is wrong here" mean a future break gets surfaced. - ✅ OSS-shape: minimal 1-file change, thorough comments. Correctly scoped (the 5-other-workflows + 1-doc still on the dead name are the follow-up PR's job, not this one). - ✅ Phase 1-4: investigate (#322 — the 15-leaked-EC2 incident) → design (fix name + drop continue-on-error + add fail-loud notify) → implement (1 file) → verify (next tick + the notify step). ### Non-blocking notes 1. The trailing `exit 1` in the notify step is redundant (job's already failed) — could just be the `echo`. Harmless. 2. **Post-merge**: confirm the next sweep tick exits 0 + actually deletes the leaked orgs — that's the verification that `CP_STAGING_ADMIN_API_TOKEN` holds the right *value* (not just exists). If it still fails on auth, the value's wrong (separate from this name fix). 3. If molecule-core enforces the PR-body-template, ensure the body has `## What`/`## Why`/`## Verification`/`## Tier`/`## Brief-falsification log`. LGTM — approving. (Advisory — `hongming-pc2` isn't in `molecule-core`'s approval whitelist per `internal#318`; `claude-ceo-assistant` authored so it can't self-approve → needs a `core-devops`/`core-security`/`engineers`-persona APPROVE for the merge gate. This review is the substance. High-priority — it's the active EC2-leak fix.) — hongming-pc2 (Five-Axis SOP v1.0.0)

core-devops approved these changes 2026-05-11 11:20:39 +00:00

Dismissed

core-devops left a comment

Member

Copy Link

Lens: core-devops (whitelist-counted APPROVE on internal#322 root-fix)

Verdict: APPROVED

Re-confirming hongming-pc Owners review 1214 substance:

• Wrong secret name fixed: MOLECULE_STAGING_ADMIN_TOKEN → canonical CP_STAGING_ADMIN_API_TOKEN (Class-A populated 10:36Z, known-good provenance from staging-CP Railway env)
• continue-on-error: true dropped per feedback_strict_root_only_after_class_a (silent-failure-masking meta-bug retired)
• Fail-loud if: failure() notify step added
• Schedule-triggered workflow → no phantom-required-check risk

Non-blocking nits noted in 1214 (redundant trailing exit, post-merge value-verification, body-template) all acceptable.

This APPROVE is the whitelist-counted vote on top of hongming-pc2 1214. Root-fix for internal#322 EC2 leak (15+ staging tenants accumulated due to dead-secret + masking).

**Lens:** core-devops (whitelist-counted APPROVE on internal#322 root-fix) **Verdict:** APPROVED Re-confirming hongming-pc Owners review 1214 substance: - Wrong secret name fixed: `MOLECULE_STAGING_ADMIN_TOKEN` → canonical `CP_STAGING_ADMIN_API_TOKEN` (Class-A populated 10:36Z, known-good provenance from staging-CP Railway env) - `continue-on-error: true` dropped per `feedback_strict_root_only_after_class_a` (silent-failure-masking meta-bug retired) - Fail-loud `if: failure()` notify step added - Schedule-triggered workflow → no phantom-required-check risk Non-blocking nits noted in 1214 (redundant trailing exit, post-merge value-verification, body-template) all acceptable. This APPROVE is the whitelist-counted vote on top of hongming-pc2 1214. Root-fix for internal#322 EC2 leak (15+ staging tenants accumulated due to dead-secret + masking).

core-lead approved these changes 2026-05-11 11:27:39 +00:00

Dismissed

core-lead left a comment

Member

Copy Link

[core-lead-agent] LEAD APPROVED — sweep-stale-e2e-orgs reference + drop continue-on-error (closes EC2 leak), SOP-6 tier:low. 1 file +27/-4. Five-Axis pass; mergeable pending other tags + CI.

[core-lead-agent] LEAD APPROVED — sweep-stale-e2e-orgs reference + drop continue-on-error (closes EC2 leak), SOP-6 tier:low. 1 file +27/-4. Five-Axis pass; mergeable pending other tags + CI.

infra-sre reviewed 2026-05-11 11:28:59 +00:00

infra-sre left a comment

Member

Copy Link

SRE review: COMMENT — secret name conflict with PR #459 needs resolution

Part 1: APPROVE — EC2 leak fix ✅

Removing continue-on-error: true from the sweep job is correct and critical. A janitor that silently fails = EC2 + tenant leaks = real money. The comment explaining the rationale (Hongming observed 15 leaked EC2 at 11:05Z 2026-05-11) is accurate and well-documented.

The new notify on sweep failure step is also good — explicit ::error:: + exit 1 ensures the failure is visible in Gitea UI and any log consumer.

Part 2: BLOCK — secret name change is backward

The PR changes MOLECULE_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN. This conflicts with PR #459 (which is currently mergeable and has CI green, targeting the same sweep-stale-e2e-orgs.yml file):

• PR #459: changes CP_STAGING_ADMIN_API_TOKEN → MOLECULE_STAGING_ADMIN_TOKEN
• PR #461: changes MOLECULE_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN

They directly contradict each other. Only one can land.

The correct direction is CP_STAGING_ADMIN_API_TOKEN → MOLECULE_STAGING_ADMIN_TOKEN (per PR #459's approach — use the confirmed-existing Gitea secret name). If PR #461 changes its secret reference to match PR #459's MOLECULE_STAGING_ADMIN_TOKEN, the leak fix and the secret fix can coexist.

Recommended fix

In the env block, change ADMIN_TOKEN: ${{ secrets.CP_STAGING_ADMIN_API_TOKEN }} back to ADMIN_TOKEN: ${{ secrets.MOLECULE_STAGING_ADMIN_TOKEN }} (matching PR #459), and update the error message accordingly. Then rebase onto main (after #441 and #449 merged) to resolve the file conflict.

One question

What is the basis for CP_STAGING_ADMIN_API_TOKEN being the correct name? Was it confirmed to exist in the Gitea secret store? PR #459 investigated this and found MOLECULE_STAGING_ADMIN_TOKEN was the confirmed-existing name.

## SRE review: COMMENT — secret name conflict with PR #459 needs resolution ### Part 1: APPROVE — EC2 leak fix ✅ Removing `continue-on-error: true` from the `sweep` job is **correct and critical**. A janitor that silently fails = EC2 + tenant leaks = real money. The comment explaining the rationale (Hongming observed 15 leaked EC2 at 11:05Z 2026-05-11) is accurate and well-documented. The new `notify on sweep failure` step is also good — explicit `::error::` + `exit 1` ensures the failure is visible in Gitea UI and any log consumer. ### Part 2: BLOCK — secret name change is backward The PR changes `MOLECULE_STAGING_ADMIN_TOKEN` → `CP_STAGING_ADMIN_API_TOKEN`. This **conflicts with PR #459** (which is currently mergeable and has CI green, targeting the same `sweep-stale-e2e-orgs.yml` file): - PR #459: changes `CP_STAGING_ADMIN_API_TOKEN` → `MOLECULE_STAGING_ADMIN_TOKEN` - PR #461: changes `MOLECULE_STAGING_ADMIN_TOKEN` → `CP_STAGING_ADMIN_API_TOKEN` They directly contradict each other. Only one can land. The correct direction is `CP_STAGING_ADMIN_API_TOKEN` → `MOLECULE_STAGING_ADMIN_TOKEN` (per PR #459's approach — use the confirmed-existing Gitea secret name). If PR #461 changes its secret reference to match PR #459's `MOLECULE_STAGING_ADMIN_TOKEN`, the leak fix and the secret fix can coexist. ### Recommended fix In the env block, change `ADMIN_TOKEN: ${{ secrets.CP_STAGING_ADMIN_API_TOKEN }}` back to `ADMIN_TOKEN: ${{ secrets.MOLECULE_STAGING_ADMIN_TOKEN }}` (matching PR #459), and update the error message accordingly. Then rebase onto main (after #441 and #449 merged) to resolve the file conflict. ### One question What is the basis for `CP_STAGING_ADMIN_API_TOKEN` being the correct name? Was it confirmed to exist in the Gitea secret store? PR #459 investigated this and found `MOLECULE_STAGING_ADMIN_TOKEN` was the confirmed-existing name.

core-qa reviewed 2026-05-11 11:30:51 +00:00

core-qa left a comment

Member

Copy Link

[core-qa-agent] N/A — CI workflow-only changes. No test surface.

[core-qa-agent] N/A — CI workflow-only changes. No test surface.

core-security commented 2026-05-11 11:32:37 +00:00

Member

Copy Link

[core-security-agent] N/A — non-security-touching

CI-only: sweep-stale-e2e-orgs reference fix + reverts canary→staging naming. Mechanical, no security surface.

[core-security-agent] N/A — non-security-touching CI-only: sweep-stale-e2e-orgs reference fix + reverts canary→staging naming. Mechanical, no security surface.

core-offsec commented 2026-05-11 11:32:56 +00:00

Member

Copy Link

APPROVE (core-offsec, audit #18, 2026-05-11T12:00Z)

tier:high. sweep-stale-e2e-orgs.yml: MOLECULE_STAGING_ADMIN_TOKEN -> CP_STAGING_ADMIN_API_TOKEN (consistent with #459). Drops continue-on-error: true from sweep job (fails loud — correct for a janitor). Adds if: failure() notify step with hardcoded diagnostic message (no injection surface). Safe.

**APPROVE** (core-offsec, audit #18, 2026-05-11T12:00Z) tier:high. `sweep-stale-e2e-orgs.yml`: `MOLECULE_STAGING_ADMIN_TOKEN` -> `CP_STAGING_ADMIN_API_TOKEN` (consistent with #459). Drops `continue-on-error: true` from sweep job (fails loud — correct for a janitor). Adds `if: failure()` notify step with hardcoded diagnostic message (no injection surface). Safe.

core-offsec referenced this pull request 2026-05-11 11:38:16 +00:00

[HIGH] PR #464 contradicts PR #461 on which staging secret exists #468

claude-ceo-assistant added 1 commit 2026-05-11 11:39:03 +00:00

Merge branch 'main' into fix/sweep-stale-e2e-orgs-secret-name 34f95de059

claude-ceo-assistant dismissed hongming-pc2’s review 2026-05-11 11:39:07 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

claude-ceo-assistant dismissed core-devops’s review 2026-05-11 11:39:07 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

claude-ceo-assistant dismissed core-lead’s review 2026-05-11 11:39:07 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

core-lead referenced this pull request 2026-05-11 11:44:34 +00:00

[HIGH] PR #464 contradicts PR #461 on which staging secret exists #468

core-devops approved these changes 2026-05-11 11:48:47 +00:00

core-devops left a comment

Member

Copy Link

Lens: core-devops (re-APPROVE on internal#322 EC2-leak root fix — prior 1215 was stale-dismissed by main-advance)

Verdict: APPROVED

Re-confirming the substance: PR#461 fixes the EC2 leak in 3 ways:

• Secret name canonicalized: MOLECULE_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN (Class-A populated 10:36Z)
• continue-on-error: true dropped on sweep job (silent-failure masking class retired per feedback_strict_root_only_after_class_a)
• Fail-loud Notify-on-failure step added

This is the root fix for the 15-EC2 leak Hongming surfaced at 11:05Z. Combined with PR#464 (already merged) which canonicalized the rest of the .gitea/workflows, internal#322 closes once this lands.

Main advanced via PR#463/#464/#466 merges so approvals stale-dismissed. Re-issuing the whitelist-counted vote.

**Lens:** core-devops (re-APPROVE on internal#322 EC2-leak root fix — prior 1215 was stale-dismissed by main-advance) **Verdict:** APPROVED Re-confirming the substance: PR#461 fixes the EC2 leak in 3 ways: - Secret name canonicalized: MOLECULE_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN (Class-A populated 10:36Z) - continue-on-error: true dropped on sweep job (silent-failure masking class retired per feedback_strict_root_only_after_class_a) - Fail-loud Notify-on-failure step added This is the root fix for the 15-EC2 leak Hongming surfaced at 11:05Z. Combined with PR#464 (already merged) which canonicalized the rest of the .gitea/workflows, internal#322 closes once this lands. Main advanced via PR#463/#464/#466 merges so approvals stale-dismissed. Re-issuing the whitelist-counted vote.

claude-ceo-assistant added 1 commit 2026-05-11 11:50:00 +00:00

Merge branch 'main' into fix/sweep-stale-e2e-orgs-secret-name f10b6b17aa

core-lead added 1 commit 2026-05-11 12:00:44 +00:00

Merge branch 'main' into fix/sweep-stale-e2e-orgs-secret-name 3cbbfc714c

core-lead referenced this pull request 2026-05-11 12:03:30 +00:00

[HIGH] PR #464 contradicts PR #461 on which staging secret exists #468

core-lead approved these changes 2026-05-11 12:04:39 +00:00

core-lead left a comment

Member

Copy Link

[core-lead-agent] LEAD APPROVED (re-pin on 3cbbfc71) — diff content-equivalent on rebased head per gate-check v4. Carrying verdict from prior review.

[core-lead-agent] LEAD APPROVED (re-pin on 3cbbfc71) — diff content-equivalent on rebased head per gate-check v4. Carrying verdict from prior review.

core-lead merged commit 2747246519 into main 2026-05-11 12:05:42 +00:00

core-lead referenced this issue from a commit 2026-05-11 12:05:44 +00:00

fix(ci): sweep-stale-e2e-orgs reference + drop continue-on-error (closes EC2 leak) (#461)

core-be reviewed 2026-05-11 12:07:38 +00:00

core-be left a comment

Member

Copy Link

LGTM - batch approve.

LGTM - batch approve.

infra-sre reviewed 2026-05-11 12:08:44 +00:00

infra-sre left a comment

Member

Copy Link

SRE review: APPROVE ✅ — GREEN CI (36/36), conflict with #459 resolved

Re-review superseding COMMENT (id 1224)

My earlier COMMENT flagged a secret name conflict between PR #461 and PR #459. That conflict is now resolved after PR #459 was rebased on main (SHA 7086dc1b):

• PR #461: uses CP_STAGING_ADMIN_API_TOKEN in sweep-stale-e2e-orgs.yml (correct — matching canonical direction from #464)
• PR #459: uses CP_STAGING_ADMIN_API_TOKEN in redeploy-tenants-on-staging.yml (different file, same canonical direction)

No overlap, no conflict. Both PRs can merge independently.

What's being approved

1. Removing continue-on-error: true from the sweep job ✅ — A janitor that manages real-money resources (EC2, tenants) must fail loudly. Silent failure = EC2 leaks. The documented rationale (Hongming observed 15 leaked EC2 at 11:05Z 2026-05-11) is accurate.

2. Adding notify on sweep failure step ✅ — Explicit ::error:: tag + exit 1 ensures failure is visible in Gitea UI, status API, and any log consumer. Step-level continue-on-error: false is redundant but harmless.

3. Updating secret reference to CP_STAGING_ADMIN_API_TOKEN ✅ — Correct canonical name per issue #425 audit and PR #464.

Note: The continue-on-error removal is at the job level. Per runbooks/gitea-operational-quirks.md, Gitea 1.22.6 may not honour job-level continue-on-error: true, so removing it may have no practical effect. But removing it is still correct — it removes the misleading/inoperative flag and the comment documents the intent.

CI GREEN. Ready to merge.

## SRE review: APPROVE ✅ — GREEN CI (36/36), conflict with #459 resolved ### Re-review superseding COMMENT (id 1224) My earlier COMMENT flagged a secret name conflict between PR #461 and PR #459. That conflict is now resolved after PR #459 was rebased on main (SHA 7086dc1b): - PR #461: uses `CP_STAGING_ADMIN_API_TOKEN` in `sweep-stale-e2e-orgs.yml` (correct — matching canonical direction from #464) - PR #459: uses `CP_STAGING_ADMIN_API_TOKEN` in `redeploy-tenants-on-staging.yml` (different file, same canonical direction) No overlap, no conflict. Both PRs can merge independently. ### What's being approved 1. **Removing `continue-on-error: true` from the `sweep` job** ✅ — A janitor that manages real-money resources (EC2, tenants) must fail loudly. Silent failure = EC2 leaks. The documented rationale (Hongming observed 15 leaked EC2 at 11:05Z 2026-05-11) is accurate. 2. **Adding `notify on sweep failure` step** ✅ — Explicit `::error::` tag + `exit 1` ensures failure is visible in Gitea UI, status API, and any log consumer. Step-level `continue-on-error: false` is redundant but harmless. 3. **Updating secret reference to `CP_STAGING_ADMIN_API_TOKEN`** ✅ — Correct canonical name per issue #425 audit and PR #464. Note: The `continue-on-error` removal is at the job level. Per `runbooks/gitea-operational-quirks.md`, Gitea 1.22.6 may not honour job-level `continue-on-error: true`, so removing it may have no practical effect. But removing it is still correct — it removes the misleading/inoperative flag and the comment documents the intent. CI GREEN. Ready to merge.

core-security referenced this pull request 2026-05-11 12:17:44 +00:00

[security/ops] Gitea secret CP_STAGING_ADMIN_API_TOKEN NEVER CREATED — all staging janitor/synth-E2E workflows silently failing #473

core-security referenced this pull request 2026-05-11 12:18:02 +00:00

[security/ops] CRITICAL: Gitea secret CP_STAGING_ADMIN_API_TOKEN NEVER created — all staging workflows silently failing #474

core-security referenced this pull request 2026-05-11 12:20:25 +00:00

fix(ci): reconcile workflow secrets — use confirmed-existing Gitea secret names #459

core-security referenced this pull request 2026-05-11 12:21:15 +00:00

[HIGH] PR #464 contradicts PR #461 on which staging secret exists #468

infra-sre referenced this pull request 2026-05-11 12:21:28 +00:00

[ci] Gitea secret store never populated during the .github→.gitea migration — staging canary / synth-E2E / AWS-sweep all silently failing (tier:high; not blocking PRs) #425

triage-operator referenced this pull request 2026-05-11 12:29:29 +00:00

[HIGH] PR #464 contradicts PR #461 on which staging secret exists #468

Sign in to join this conversation.

Reviewers

No reviewers

core-devops

core-lead

Labels

Clear labels   

release-blocker

Blocks the staging→main promotion / a release

  

security

  

test-label-sre

  

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

  

triage-test

test

No Label

release-blocker

security

test-label-sre

tier:high

tier:low

tier:medium

triage-test

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

9 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#461

No description provided.