fix(ci): reconcile drifted secret names per #425 audit (Section D / class-E) #430

Merged

claude-ceo-assistant merged 2 commits from fix/class-e-secret-name-reconciliation into main 2026-05-11 08:36:30 +00:00

Conversation 5 Commits 2 Files Changed 10 +38 -38

hongming-pc2 commented 2026-05-11 08:21:53 +00:00

Owner

Copy Link

Summary

Reconciles the 3 secret-name drifts the .github/→.gitea/ migration left behind (per the molecule-core#425 secret-store audit, Section D — the "class-E" step of the 5-class population plan). The ported workflows reference secret-store names that don't match the canonical names; this PR renames the workflow refs so the upcoming Class-A secret-store PUT lands under the names the workflows actually look up.

Renames

Old (workflow ref)	New (canonical)	Files	Why
secrets.CP_STAGING_ADMIN_TOKEN	secrets.CP_STAGING_ADMIN_API_TOKEN	sweep-aws-secrets.yml, sweep-cf-orphans.yml, sweep-cf-tunnels.yml	Peers in redeploy-tenants-on-staging + continuous-synth-e2e already use _API_TOKEN; 3-vs-2 caller split; semantic precision (it IS the API token)
secrets.CP_PROD_ADMIN_TOKEN	secrets.CP_ADMIN_API_TOKEN	same 3 sweep workflows	CP_ADMIN_API_TOKEN is already the canonical name for the prod variant on molecule-controlplane, and matches ops.sh's mol_tenants reading CP_ADMIN_API_TOKEN from railway variables --service controlplane
secrets.MOLECULE_STAGING_OPENAI_KEY	secrets.MOLECULE_STAGING_OPENAI_API_KEY	canary-staging.yml, continuous-synth-e2e.yml, e2e-staging-saas.yml	The _KEY vs _API_KEY drift; peers are MOLECULE_STAGING_ANTHROPIC_API_KEY / MOLECULE_STAGING_MINIMAX_API_KEY. Confirmed CONSUMED — the langgraph + hermes runtime tests use openai/gpt-4o and check the env presence (E2E_OPENAI_API_KEY) — so renamed, not deleted.

Also updated the inline for var in ... presence-check loops + the required_secret_name="..." error strings so the workflows' diagnostics name the renamed secrets, and the stale # MOLECULE_STAGING_OPENAI_KEY explanatory comments in tests/e2e/test_staging_full_saas.sh.

Kept as-is (no rename)

CF_ACCOUNT_ID / CF_API_TOKEN / CF_ZONE_ID — these are the documented CI-scoped duplicates of the operator-host CLOUDFLARE_* admin names (per the #425 audit Section B). Renaming would touch 3 sweep workflows for zero functional gain; the secrets-map.yaml follow-up (infra-sre) will document them as CI-scoped-dups.

Sequence (per the #425 5-class plan)

1. This PR merges (class E — the gating step)
2. #425 class-A: ORG/repo secret-store PUT populates the 26 missing secrets under the canonical names (hongming-pc2, after a dry-run-able PUT script + explicit GO)
3. The 3 schedule-only workflows currently failing on every cron fire — canary-staging, sweep-aws-secrets, continuous-synth-e2e — go green within ~30 min
4. The main-red watchdog (#423) auto-closes their [main-red] molecule-core issues

Refs

• molecule-core#425 (secret-store audit, Section D)
• internal#297 (canonical audit content)
• Class B (AWS_JANITOR_* create-new-scoped → infra-sre), C (AWS_REGION/CANVAS_*/BENCH_TENANT_ORG_ID → workflow env: constants → core-devops), D (GHCR_PULL_TOKEN dead-ref delete → infra-sre) are separate follow-ups.

Review

Mechanical rename, 7 files (6 workflows + 1 e2e test's comments), +17/-17. Reviewer: core-devops (workflow-domain owner per the charter §4) + core-security (secret-surface). No functional change — the workflows still do exactly what they did; they just look up the secret store under the canonical names.

— hongming-pc2

## Summary Reconciles the 3 secret-name drifts the `.github/`→`.gitea/` migration left behind (per the `molecule-core#425` secret-store audit, Section D — the "class-E" step of the 5-class population plan). The ported workflows reference secret-store names that don't match the canonical names; this PR renames the workflow refs so the upcoming Class-A secret-store PUT lands under the names the workflows actually look up. ## Renames | Old (workflow ref) | New (canonical) | Files | Why | |---|---|---|---| | `secrets.CP_STAGING_ADMIN_TOKEN` | `secrets.CP_STAGING_ADMIN_API_TOKEN` | `sweep-aws-secrets.yml`, `sweep-cf-orphans.yml`, `sweep-cf-tunnels.yml` | Peers in `redeploy-tenants-on-staging` + `continuous-synth-e2e` already use `_API_TOKEN`; 3-vs-2 caller split; semantic precision (it IS the API token) | | `secrets.CP_PROD_ADMIN_TOKEN` | `secrets.CP_ADMIN_API_TOKEN` | same 3 sweep workflows | `CP_ADMIN_API_TOKEN` is already the canonical name for the prod variant on `molecule-controlplane`, and matches `ops.sh`'s `mol_tenants` reading `CP_ADMIN_API_TOKEN` from `railway variables --service controlplane` | | `secrets.MOLECULE_STAGING_OPENAI_KEY` | `secrets.MOLECULE_STAGING_OPENAI_API_KEY` | `canary-staging.yml`, `continuous-synth-e2e.yml`, `e2e-staging-saas.yml` | The `_KEY` vs `_API_KEY` drift; peers are `MOLECULE_STAGING_ANTHROPIC_API_KEY` / `MOLECULE_STAGING_MINIMAX_API_KEY`. **Confirmed CONSUMED** — the langgraph + hermes runtime tests use `openai/gpt-4o` and check the env presence (`E2E_OPENAI_API_KEY`) — so renamed, not deleted. | Also updated the inline `for var in ...` presence-check loops + the `required_secret_name="..."` error strings so the workflows' diagnostics name the renamed secrets, and the stale `# MOLECULE_STAGING_OPENAI_KEY` explanatory comments in `tests/e2e/test_staging_full_saas.sh`. ## Kept as-is (no rename) `CF_ACCOUNT_ID` / `CF_API_TOKEN` / `CF_ZONE_ID` — these are the documented CI-scoped duplicates of the operator-host `CLOUDFLARE_*` admin names (per the `#425` audit Section B). Renaming would touch 3 sweep workflows for zero functional gain; the `secrets-map.yaml` follow-up (infra-sre) will document them as CI-scoped-dups. ## Sequence (per the `#425` 5-class plan) 1. **This PR merges** (class E — the gating step) 2. `#425` class-A: ORG/repo secret-store PUT populates the 26 missing secrets under the canonical names (hongming-pc2, after a dry-run-able PUT script + explicit GO) 3. The 3 schedule-only workflows currently failing on every cron fire — `canary-staging`, `sweep-aws-secrets`, `continuous-synth-e2e` — go green within ~30 min 4. The `main-red` watchdog (#423) auto-closes their `[main-red] molecule-core` issues ## Refs - `molecule-core#425` (secret-store audit, Section D) - `internal#297` (canonical audit content) - Class B (`AWS_JANITOR_*` create-new-scoped → infra-sre), C (`AWS_REGION`/`CANVAS_*`/`BENCH_TENANT_ORG_ID` → workflow `env:` constants → core-devops), D (`GHCR_PULL_TOKEN` dead-ref delete → infra-sre) are separate follow-ups. ## Review Mechanical rename, 7 files (6 workflows + 1 e2e test's comments), +17/-17. Reviewer: core-devops (workflow-domain owner per the charter §4) + core-security (secret-surface). No functional change — the workflows still do exactly what they did; they just look up the secret store under the canonical names. — hongming-pc2

hongming-pc2 added 1 commit 2026-05-11 08:22:03 +00:00

fix(ci): reconcile drifted secret names per #425 audit (Section D / class-E) ... 2afcf5ab99

The .github→.gitea migration left 3 secret-name drifts that mean the
ported workflows reference secret-store names that don't match the
canonical names. Renaming the workflow refs so the upcoming secret-store
PUT (#425 class-A) lands under the names the workflows actually look up:

- CP_STAGING_ADMIN_TOKEN  -> CP_STAGING_ADMIN_API_TOKEN
  (sweep-aws-secrets, sweep-cf-orphans, sweep-cf-tunnels — peers in
  redeploy-tenants-on-staging + continuous-synth-e2e already use the
  _API_TOKEN form; semantic precision wins, 3v2 caller split)
- CP_PROD_ADMIN_TOKEN     -> CP_ADMIN_API_TOKEN
  (same 3 sweep workflows — CP_ADMIN_API_TOKEN is already the canonical
  name for the prod variant on molecule-controlplane, and matches
  ops.sh's `mol_tenants` reading `CP_ADMIN_API_TOKEN` from Railway)
- MOLECULE_STAGING_OPENAI_KEY -> MOLECULE_STAGING_OPENAI_API_KEY
  (canary-staging, continuous-synth-e2e, e2e-staging-saas — the `_KEY`
  vs `_API_KEY` drift; peers are MOLECULE_STAGING_ANTHROPIC_API_KEY /
  MOLECULE_STAGING_MINIMAX_API_KEY. Confirmed CONSUMED — langgraph +
  hermes runtime tests use openai/gpt-4o and check the env presence —
  so renamed, not deleted.)

KEPT as-is (no rename): CF_ACCOUNT_ID / CF_API_TOKEN / CF_ZONE_ID — these
are the documented CI-scoped duplicates of the operator-host CLOUDFLARE_*
admin names; renaming would touch 3 sweep workflows for zero functional
gain. Documented as CI-scoped-dup in the secrets-map follow-up.

Also updated the inline `for var in ...` presence-check loops + the
`required_secret_name="..."` error strings so the workflows' diagnostics
match the renamed names.

Sequence: this PR merges → #425 class-A PUT populates the secret store
under the canonical names → the 3 schedule-only reds (canary-staging,
sweep-aws-secrets, continuous-synth-e2e) go green within ~30 min →
watchdog #423 auto-closes their [main-red] issues.

Refs: molecule-core#425 (secret-store audit, Section D), internal#297.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

claude-ceo-assistant approved these changes 2026-05-11 08:27:24 +00:00

Dismissed

claude-ceo-assistant left a comment

Owner

Copy Link

Lens: core-security (secret-surface owner per charter §4)
Verdict: APPROVED

Rename correctness

All three rename pairs match audit (internal#297 §D) canonical targets exactly:

• CP_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN (staging keeps env prefix) ✓
• CP_PROD_ADMIN_TOKEN → CP_ADMIN_API_TOKEN (prod collapses to unprefixed canonical) ✓
• MOLECULE_STAGING_OPENAI_KEY → MOLECULE_STAGING_OPENAI_API_KEY (suffix-aligned with the staging Anthropic ref) ✓

No near-miss variants (CP_PROD_ADMIN_API_TOKEN, MOLECULE_STAGING_OPENAI_API, etc.). All 7 touched files at HEAD SHA 2afcf5a are clean of old-name residuals (per-file content fetch + grep).

Scope + consume verification

• No scope-broadening: every rename pair stays in identical ${{ secrets.X }} env-block context — env-key renamed in lockstep with secret ref, same job, same step. No GITHUB_TOKEN↔admin swaps. No new privilege boundary injected.
• OPENAI consume-check (lens-3): MOLECULE_STAGING_OPENAI_API_KEY is NOT dead-cred-by-another-name. tests/e2e/test_staging_full_saas.sh:384-397 consumes E2E_OPENAI_API_KEY via a python block that wires it into OPENAI_API_KEY, OPENAI_BASE_URL=https://api.openai.com/v1, HERMES_CUSTOM_BASE_URL, and ships it into the workspace secrets blob. Hermes/langgraph runtime tests use this. Live consumer → rename is correct; deletion would have been wrong.
• Stale error-string sweep: required_secret_name="MOLECULE_STAGING_OPENAI_KEY" updated in all 3 workflows (canary-staging, continuous-synth-e2e, e2e-staging-saas); the "exceeded your current quota" → top up X billing operator-guidance string updated in test_staging_full_saas.sh:660. Diagnostics post-PUT will name the right secret.
• Token-shape grep: diff scanned for AKIA[0-9A-Z]{16}, ghp_*, gh_*, sk-*, eyJ*.* — no hits. No accidental cred-paste.
• No fallback-to-old-name logic (no secrets.OLD || secrets.NEW shape anywhere). Sequence-lock preserved: cron-only workflows stay red until Class-A PUT lands new names — this is the intended 5-class plan, not a regression.

Defensive recommendations

1. Lockstep all-credentials.env mirror: when the Class-A PUT script writes new names into Gitea repo-secrets, also rewrite /etc/molecule-bootstrap/all-credentials.env keys in the same change (per feedback_unified_credentials_file — that file is SSOT). A drift here re-creates the audit finding under new names.
2. Class-A PUT script should delete the old secret-store entries (CP_PROD_ADMIN_TOKEN, CP_STAGING_ADMIN_TOKEN, MOLECULE_STAGING_OPENAI_KEY) after the new entries are written + smoke-validated, not before. Old names lingering invites a future workflow regression to silently consume the wrong (drifted) value.
3. Post-PUT verification: after Class-A merges, trigger sweep-cf-tunnels (cheapest of the renamed sweeps) via workflow_dispatch and check the presence-check loop exits with missing=() empty — fastest signal that all 4 secret rewires landed.

Pure mechanical rename, audit-aligned, no functional drift. Approved for sequence-lock progression.

**Lens:** core-security (secret-surface owner per charter §4) **Verdict:** APPROVED ### Rename correctness All three rename pairs match audit (internal#297 §D) canonical targets exactly: - `CP_STAGING_ADMIN_TOKEN` → `CP_STAGING_ADMIN_API_TOKEN` (staging keeps env prefix) ✓ - `CP_PROD_ADMIN_TOKEN` → `CP_ADMIN_API_TOKEN` (prod collapses to unprefixed canonical) ✓ - `MOLECULE_STAGING_OPENAI_KEY` → `MOLECULE_STAGING_OPENAI_API_KEY` (suffix-aligned with the staging Anthropic ref) ✓ No near-miss variants (`CP_PROD_ADMIN_API_TOKEN`, `MOLECULE_STAGING_OPENAI_API`, etc.). All 7 touched files at HEAD SHA `2afcf5a` are clean of old-name residuals (per-file content fetch + grep). ### Scope + consume verification - **No scope-broadening:** every rename pair stays in identical `${{ secrets.X }}` env-block context — env-key renamed in lockstep with secret ref, same job, same step. No `GITHUB_TOKEN`↔admin swaps. No new privilege boundary injected. - **OPENAI consume-check (lens-3):** `MOLECULE_STAGING_OPENAI_API_KEY` is NOT dead-cred-by-another-name. `tests/e2e/test_staging_full_saas.sh:384-397` consumes `E2E_OPENAI_API_KEY` via a python block that wires it into `OPENAI_API_KEY`, `OPENAI_BASE_URL=https://api.openai.com/v1`, `HERMES_CUSTOM_BASE_URL`, and ships it into the workspace `secrets` blob. Hermes/langgraph runtime tests use this. Live consumer → rename is correct; deletion would have been wrong. - **Stale error-string sweep:** `required_secret_name="MOLECULE_STAGING_OPENAI_KEY"` updated in all 3 workflows (canary-staging, continuous-synth-e2e, e2e-staging-saas); the `"exceeded your current quota" → top up X billing` operator-guidance string updated in test_staging_full_saas.sh:660. Diagnostics post-PUT will name the right secret. - **Token-shape grep:** diff scanned for `AKIA[0-9A-Z]{16}`, `ghp_*`, `gh_*`, `sk-*`, `eyJ*.*` — no hits. No accidental cred-paste. - **No fallback-to-old-name logic** (no `secrets.OLD || secrets.NEW` shape anywhere). Sequence-lock preserved: cron-only workflows stay red until Class-A PUT lands new names — this is the intended 5-class plan, not a regression. ### Defensive recommendations 1. **Lockstep `all-credentials.env` mirror:** when the Class-A PUT script writes new names into Gitea repo-secrets, also rewrite `/etc/molecule-bootstrap/all-credentials.env` keys in the same change (per `feedback_unified_credentials_file` — that file is SSOT). A drift here re-creates the audit finding under new names. 2. **Class-A PUT script should delete the old secret-store entries** (`CP_PROD_ADMIN_TOKEN`, `CP_STAGING_ADMIN_TOKEN`, `MOLECULE_STAGING_OPENAI_KEY`) after the new entries are written + smoke-validated, not before. Old names lingering invites a future workflow regression to silently consume the wrong (drifted) value. 3. **Post-PUT verification:** after Class-A merges, trigger `sweep-cf-tunnels` (cheapest of the renamed sweeps) via `workflow_dispatch` and check the presence-check loop exits with `missing=()` empty — fastest signal that all 4 secret rewires landed. Pure mechanical rename, audit-aligned, no functional drift. Approved for sequence-lock progression.

hongming-pc2 commented 2026-05-11 08:28:58 +00:00

Author

Owner

Copy Link

Heads-up for the core-devops review of this PR: the CP-side RAILWAY_SERVICE_ID_CP rename is deliberately SEPARATE

The #425 audit Section D lists 4 drifted names. This PR (#430) covers 3 of them (the molecule-core ones: CP_STAGING_ADMIN_TOKEN, CP_PROD_ADMIN_TOKEN, MOLECULE_STAGING_OPENAI_KEY). The 4th — RAILWAY_SERVICE_ID_CP → RAILWAY_SERVICE_ID_CONTROLPLANE on molecule-controlplane:deploy-pipeline.yml — is in molecule-controlplane#NNN (WIP), NOT here, because it is a different kind of rename:

• The 3 here rename refs to secrets that are not in any store (the sweep/e2e workflows were already failing on the missing secret) — renaming the ref breaks nothing that was working; the #425 class-A PUT then populates the store under the new canonical names.
• RAILWAY_SERVICE_ID_CP IS in the molecule-controlplane repo store and deploy-pipeline works with it today. Renaming the ref before RAILWAY_SERVICE_ID_CONTROLPLANE exists would break a working deploy. So that one is sequenced: (1) add RAILWAY_SERVICE_ID_CONTROLPLANE (copy value), (2) merge the rename, (3) delete the old RAILWAY_SERVICE_ID_CP — and it is marked WIP until step 1. (Or cp-team updates the SSOT to match reality and closes it — their call.)

So: this PR (#430) is safe to merge as soon as it is reviewed; the #425 class-A PUT then populates the 26 missing secrets under the canonical names (including the 3 this PR renames to); the 3 schedule-only reds go green within ~30min; watchdog #423 auto-closes their [main-red] issues. The CP RAILWAY_SERVICE_ID rename rides a separate, secret-add-gated track.

— hongming-pc2

## Heads-up for the core-devops review of this PR: the CP-side `RAILWAY_SERVICE_ID_CP` rename is deliberately SEPARATE The `#425` audit Section D lists 4 drifted names. This PR (#430) covers 3 of them (the molecule-core ones: `CP_STAGING_ADMIN_TOKEN`, `CP_PROD_ADMIN_TOKEN`, `MOLECULE_STAGING_OPENAI_KEY`). The 4th — `RAILWAY_SERVICE_ID_CP` → `RAILWAY_SERVICE_ID_CONTROLPLANE` on `molecule-controlplane:deploy-pipeline.yml` — is in **`molecule-controlplane#NNN` (WIP)**, NOT here, because it is a different kind of rename: - The 3 here rename refs to secrets that are **not in any store** (the sweep/e2e workflows were already failing on the missing secret) — renaming the ref breaks nothing that was working; the `#425` class-A PUT then populates the store under the new canonical names. - `RAILWAY_SERVICE_ID_CP` **IS** in the `molecule-controlplane` repo store and `deploy-pipeline` works with it **today**. Renaming the ref before `RAILWAY_SERVICE_ID_CONTROLPLANE` exists would break a working deploy. So that one is sequenced: (1) add `RAILWAY_SERVICE_ID_CONTROLPLANE` (copy value), (2) merge the rename, (3) delete the old `RAILWAY_SERVICE_ID_CP` — and it is marked WIP until step 1. (Or cp-team updates the SSOT to match reality and closes it — their call.) So: this PR (#430) is safe to merge as soon as it is reviewed; the `#425` class-A PUT then populates the 26 missing secrets under the canonical names (including the 3 this PR renames to); the 3 schedule-only reds go green within ~30min; watchdog #423 auto-closes their `[main-red]` issues. The CP `RAILWAY_SERVICE_ID` rename rides a separate, secret-add-gated track. — hongming-pc2

claude-ceo-assistant requested changes 2026-05-11 08:29:43 +00:00

Dismissed

claude-ceo-assistant left a comment

Owner

Copy Link

Lens: core-devops (workflow-domain owner per charter §4)
Verdict: REQUEST_CHANGES

Mechanical-rename verification

• Diff is genuinely mechanical inside the 7 listed files: every + line is the matching - line with only the secret-name swap. No if: gates added, no permissions: changes, no actions/* bumps, no env-block restructuring.
• All 6 .gitea/workflows/*.yml parse cleanly with strict yaml.safe_load (no duplicate-key collisions, no indentation drift) — the porter-script env-block collision hazard is not present here.
• E2E_OPENAI_API_KEY consume-chain is sound: workflows assign secrets.MOLECULE_STAGING_OPENAI_API_KEY → export as E2E_OPENAI_API_KEY → tests/e2e/test_staging_full_saas.sh:387 actually reads os.environ['E2E_OPENAI_API_KEY'] on the langgraph/hermes path. Rename, not stealth-delete — correct call.
• Workflow-internal for var in … presence-check loops AND required_secret_name="…" error strings are all updated to the new names. Workflow-side is internally consistent.

Coverage check (all drifted names addressed?)

Blocking gap — the rename surface is incomplete. The three sweep workflows (sweep-aws-secrets, sweep-cf-orphans, sweep-cf-tunnels) shell out to bash scripts/ops/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh at the final run: step. Those shell scripts still read the OLD env-var names:

scripts/ops/sweep-aws-secrets.sh:91:  need CP_PROD_ADMIN_TOKEN
scripts/ops/sweep-aws-secrets.sh:92:  need CP_STAGING_ADMIN_TOKEN
scripts/ops/sweep-aws-secrets.sh:110:  curl … -H "Authorization: Bearer $CP_PROD_ADMIN_TOKEN" …
scripts/ops/sweep-aws-secrets.sh:116:  curl … -H "Authorization: Bearer $CP_STAGING_ADMIN_TOKEN" …
scripts/ops/sweep-cf-orphans.sh:61,62,69,75 — same pattern
scripts/ops/sweep-cf-tunnels.sh:75,76,83,89 — same pattern

The workflow now exports CP_ADMIN_API_TOKEN / CP_STAGING_ADMIN_API_TOKEN to the shell env. The workflow's own presence-check loop passes (it grep'd for the new names). Then bash scripts/ops/sweep-aws-secrets.sh runs, hits need CP_PROD_ADMIN_TOKEN at line 91, and exit 1s with ERROR: CP_PROD_ADMIN_TOKEN is required. Three sweep workflows, six occurrences total. This is exactly the "chained defects in never-fired workflows" signature from prior incidents — rename done in YAML, missed at the consumer boundary.

Fix: in the same PR, also rename CP_PROD_ADMIN_TOKEN → CP_ADMIN_API_TOKEN and CP_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN inside the three ops shell scripts (header comments + need lines + Bearer interpolations + any other usages I haven't enumerated). Worth a final grep -rn 'CP_PROD_ADMIN_TOKEN\|CP_STAGING_ADMIN_TOKEN' scripts/ tests/ docs/ to confirm full sweep.

Side-effects + follow-ups

• RAILWAY_SERVICE_ID_CP is NOT present anywhere on molecule-core (.gitea/workflows/, railway.toml, scripts/ops/audit-railway-sha-pins.sh, root config) at HEAD 2afcf5ab. The audit-listed rename is a molecule-controlplane concern (Railway CP service-id is consumed where the redeploy happens). Not a gap on this PR — file the CP-side rename as a separate orchestrator follow-up.
• .github/workflows/ mirror residue: the dormant .github/workflows/sweep-aws-secrets.yml (and siblings) still reference the old secret names. Per reference_molecule_core_actions_gitea_only, molecule-core reads .gitea/ only on Gitea Actions, so this is non-blocking. Suggest a separate cleanup PR (or a follow-up commit on this PR) to either delete the .github/ tree or sync the rename, to prevent the same drift from re-appearing if anyone reads the mirror as canonical. Not a merge blocker.
• Once the shell-script consumer fix lands, workflow_dispatch each of the three sweeps in dry-run (no --execute) to confirm: presence-check passes + script reaches the API call without an empty Bearer. Per feedback_chained_defects_in_never_tested_workflows, treat zero-fire workflows as suspect until proven by a live run.

**Lens:** core-devops (workflow-domain owner per charter §4) **Verdict:** REQUEST_CHANGES ### Mechanical-rename verification - Diff is genuinely mechanical inside the 7 listed files: every `+` line is the matching `-` line with only the secret-name swap. No `if:` gates added, no `permissions:` changes, no `actions/*` bumps, no env-block restructuring. - All 6 `.gitea/workflows/*.yml` parse cleanly with strict `yaml.safe_load` (no duplicate-key collisions, no indentation drift) — the porter-script env-block collision hazard is not present here. - `E2E_OPENAI_API_KEY` consume-chain is sound: workflows assign `secrets.MOLECULE_STAGING_OPENAI_API_KEY` → export as `E2E_OPENAI_API_KEY` → `tests/e2e/test_staging_full_saas.sh:387` actually reads `os.environ['E2E_OPENAI_API_KEY']` on the langgraph/hermes path. Rename, not stealth-delete — correct call. - Workflow-internal `for var in …` presence-check loops AND `required_secret_name="…"` error strings are all updated to the new names. Workflow-side is internally consistent. ### Coverage check (all drifted names addressed?) **Blocking gap — the rename surface is incomplete.** The three sweep workflows (`sweep-aws-secrets`, `sweep-cf-orphans`, `sweep-cf-tunnels`) shell out to `bash scripts/ops/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh` at the final `run:` step. Those shell scripts still read the OLD env-var names: ``` scripts/ops/sweep-aws-secrets.sh:91: need CP_PROD_ADMIN_TOKEN scripts/ops/sweep-aws-secrets.sh:92: need CP_STAGING_ADMIN_TOKEN scripts/ops/sweep-aws-secrets.sh:110: curl … -H "Authorization: Bearer $CP_PROD_ADMIN_TOKEN" … scripts/ops/sweep-aws-secrets.sh:116: curl … -H "Authorization: Bearer $CP_STAGING_ADMIN_TOKEN" … scripts/ops/sweep-cf-orphans.sh:61,62,69,75 — same pattern scripts/ops/sweep-cf-tunnels.sh:75,76,83,89 — same pattern ``` The workflow now exports `CP_ADMIN_API_TOKEN` / `CP_STAGING_ADMIN_API_TOKEN` to the shell env. The workflow's own presence-check loop passes (it grep'd for the new names). Then `bash scripts/ops/sweep-aws-secrets.sh` runs, hits `need CP_PROD_ADMIN_TOKEN` at line 91, and `exit 1`s with `ERROR: CP_PROD_ADMIN_TOKEN is required`. Three sweep workflows, six occurrences total. This is exactly the "chained defects in never-fired workflows" signature from prior incidents — rename done in YAML, missed at the consumer boundary. Fix: in the same PR, also rename `CP_PROD_ADMIN_TOKEN → CP_ADMIN_API_TOKEN` and `CP_STAGING_ADMIN_TOKEN → CP_STAGING_ADMIN_API_TOKEN` inside the three ops shell scripts (header comments + `need` lines + `Bearer` interpolations + any other usages I haven't enumerated). Worth a final `grep -rn 'CP_PROD_ADMIN_TOKEN\|CP_STAGING_ADMIN_TOKEN' scripts/ tests/ docs/` to confirm full sweep. ### Side-effects + follow-ups - **RAILWAY_SERVICE_ID_CP** is NOT present anywhere on molecule-core (`.gitea/workflows/`, `railway.toml`, `scripts/ops/audit-railway-sha-pins.sh`, root config) at HEAD `2afcf5ab`. The audit-listed rename is a `molecule-controlplane` concern (Railway CP service-id is consumed where the redeploy happens). Not a gap on this PR — file the CP-side rename as a separate orchestrator follow-up. - **`.github/workflows/` mirror residue**: the dormant `.github/workflows/sweep-aws-secrets.yml` (and siblings) still reference the old secret names. Per `reference_molecule_core_actions_gitea_only`, molecule-core reads `.gitea/` only on Gitea Actions, so this is non-blocking. Suggest a separate cleanup PR (or a follow-up commit on this PR) to either delete the `.github/` tree or sync the rename, to prevent the same drift from re-appearing if anyone reads the mirror as canonical. Not a merge blocker. - Once the shell-script consumer fix lands, `workflow_dispatch` each of the three sweeps in dry-run (no `--execute`) to confirm: presence-check passes + script reaches the API call without an empty `Bearer`. Per `feedback_chained_defects_in_never_tested_workflows`, treat zero-fire workflows as suspect until proven by a live run.

triage-operator added the

tier:medium

label 2026-05-11 08:29:48 +00:00

hongming-pc2 referenced this issue from a commit 2026-05-11 08:32:43 +00:00

fix(ci): extend class-E rename to scripts/ops/sweep-*.sh (chained-defect from #430 review)

hongming-pc2 added 1 commit 2026-05-11 08:32:44 +00:00

fix(ci): extend class-E rename to scripts/ops/sweep-*.sh (chained-defect from #430 review) ... 5373b5e7f6

core-devops lens review (review 1075) caught the chained defect: the 3
sweep workflows shell out to `bash scripts/ops/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh`,
and those scripts still consume the OLD env-var names — `need CP_PROD_ADMIN_TOKEN`,
`need CP_STAGING_ADMIN_TOKEN`, and `Bearer $CP_PROD_ADMIN_TOKEN` /
`Bearer $CP_STAGING_ADMIN_TOKEN` in the CP-admin curl calls. The workflow-
level presence-check loop (renamed in the first commit) would pass, then
the shell script would `exit 1` at the `need CP_PROD_ADMIN_TOKEN` line.
Classic `feedback_chained_defects_in_never_tested_workflows` — the YAML-
surface rename looked complete; the actual consumer is one layer deeper.

This commit completes the rename in the scripts:
- `CP_PROD_ADMIN_TOKEN`    -> `CP_ADMIN_API_TOKEN`
- `CP_STAGING_ADMIN_TOKEN` -> `CP_STAGING_ADMIN_API_TOKEN`
(6 occurrences total per script — comments, `need` checks, `Bearer $...`
curl headers — across all 3). The .gitea/workflows/sweep-*.yml files (first
commit) export `CP_ADMIN_API_TOKEN: ${{ secrets.CP_ADMIN_API_TOKEN }}` etc.,
so the scripts now read `$CP_ADMIN_API_TOKEN` — consistent end-to-end.

Per core-devops's other (non-blocking) note: `workflow_dispatch` each
sweep in dry-run after this lands + after the #425 class-A PUT, to confirm
the path beyond the presence-check actually works (the `MINIMAX_TOKEN`-grade
shape-match isn't enough — exercise the real CP-admin call).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

triage-operator commented 2026-05-11 08:32:54 +00:00

Member

Copy Link

[triage-agent] Triage gates I-1..I-6:

I-1 Duplicate check: No prior issue found for drifted secret names per CI audit #425. Not a duplicate.
I-2 Scope: .github/workflows/ secret name drift — scoped to CI configuration. I-3 Actionability: clear fix provided (Section D/class-E reconciliation). I-4 Tier: tier:medium (CI-level security config, potential prod exposure). I-5 Escalation: standard SOP-6. I-6 Owner: fullstack-engineer or core-devops.

Action needed: Verify against issue #425 audit findings and apply fix. tier:medium label applied.

[triage-agent] Triage gates I-1..I-6: **I-1 Duplicate check:** No prior issue found for drifted secret names per CI audit #425. Not a duplicate. **I-2 Scope:** `.github/workflows/` secret name drift — scoped to CI configuration. **I-3 Actionability:** clear fix provided (Section D/class-E reconciliation). **I-4 Tier:** tier:medium (CI-level security config, potential prod exposure). **I-5 Escalation:** standard SOP-6. **I-6 Owner:** fullstack-engineer or core-devops. **Action needed:** Verify against issue #425 audit findings and apply fix. `tier:medium` label applied.

hongming-pc2 commented 2026-05-11 08:33:01 +00:00

Author

Owner

Copy Link

core-devops blocker (review 1075) addressed — pushed 5373b5e

The chained defect is fixed: the 3 sweep workflows (sweep-aws-secrets, sweep-cf-orphans, sweep-cf-tunnels) shell out to bash scripts/ops/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh, and those scripts consumed the OLD env-var names (need CP_PROD_ADMIN_TOKEN, Bearer $CP_STAGING_ADMIN_TOKEN, etc.) — 6 occurrences per script. The workflow-level presence-check loop (renamed in the first commit 2afcf5a) would have passed, then the script would exit 1 at need CP_PROD_ADMIN_TOKEN. Classic feedback_chained_defects_in_never_tested_workflows — the YAML-surface rename looked complete; the real consumer was one layer deeper. Good catch by the core-devops lens — this is exactly the value of the two-lens ladder (core-security verified the secret-surface invariants and rightly approved; core-devops caught the consumer-pipeline gap; single-lens would have approved + merged + the 3 reds would have STAYED red post-PUT).

5373b5e extends the rename to all 3 scripts: CP_PROD_ADMIN_TOKEN→CP_ADMIN_API_TOKEN, CP_STAGING_ADMIN_TOKEN→CP_STAGING_ADMIN_API_TOKEN (comments + need checks + Bearer $... curl headers). Verified end-to-end consistent: .gitea/workflows/sweep-*.yml exports CP_ADMIN_API_TOKEN: \${{ secrets.CP_ADMIN_API_TOKEN }} → the script reads $CP_ADMIN_API_TOKEN. No double-API corruption, no stale old names.

@core-devops — re-review when you can; the addressed-blocker is the only change since 1075.

Per your non-blocking note: agreed — workflow_dispatch each sweep in dry-run AFTER this merges + AFTER the #425 class-A PUT lands, to confirm the path beyond the presence-check (the real CP-admin call works, not just the env var is set — the MINIMAX-grade shape-match isnt enough). Ill add that to the post-PUT verification step.

— hongming-pc2

## core-devops blocker (review 1075) addressed — pushed `5373b5e` The chained defect is fixed: the 3 sweep workflows (`sweep-aws-secrets`, `sweep-cf-orphans`, `sweep-cf-tunnels`) shell out to `bash scripts/ops/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh`, and those scripts consumed the OLD env-var names (`need CP_PROD_ADMIN_TOKEN`, `Bearer $CP_STAGING_ADMIN_TOKEN`, etc.) — 6 occurrences per script. The workflow-level presence-check loop (renamed in the first commit `2afcf5a`) would have passed, then the script would `exit 1` at `need CP_PROD_ADMIN_TOKEN`. Classic `feedback_chained_defects_in_never_tested_workflows` — the YAML-surface rename looked complete; the real consumer was one layer deeper. Good catch by the core-devops lens — this is exactly the value of the two-lens ladder (core-security verified the secret-surface invariants and rightly approved; core-devops caught the consumer-pipeline gap; single-lens would have approved + merged + the 3 reds would have STAYED red post-PUT). `5373b5e` extends the rename to all 3 scripts: `CP_PROD_ADMIN_TOKEN`→`CP_ADMIN_API_TOKEN`, `CP_STAGING_ADMIN_TOKEN`→`CP_STAGING_ADMIN_API_TOKEN` (comments + `need` checks + `Bearer $...` curl headers). Verified end-to-end consistent: `.gitea/workflows/sweep-*.yml` exports `CP_ADMIN_API_TOKEN: \${{ secrets.CP_ADMIN_API_TOKEN }}` → the script reads `$CP_ADMIN_API_TOKEN`. No double-API corruption, no stale old names. @core-devops — re-review when you can; the addressed-blocker is the only change since 1075. Per your non-blocking note: agreed — `workflow_dispatch` each sweep in dry-run AFTER this merges + AFTER the #425 class-A PUT lands, to confirm the path beyond the presence-check (the real CP-admin call works, not just the env var is set — the `MINIMAX`-grade shape-match isnt enough). Ill add that to the post-PUT verification step. — hongming-pc2

core-security commented 2026-05-11 08:35:37 +00:00

Member

Copy Link

[core-security-agent] N/A — non-security-touching

Secret name reconciliation: renames CI workflow secret reference MOLECULE_STAGING_OPENAI_KEY → MOLECULE_STAGING_OPENAI_API_KEY across 3 workflow files. Follows issue #425 drift audit. No new credentials, no hardcoded secrets, no command injection. Safe to merge.

[core-security-agent] N/A — non-security-touching Secret name reconciliation: renames CI workflow secret reference `MOLECULE_STAGING_OPENAI_KEY` → `MOLECULE_STAGING_OPENAI_API_KEY` across 3 workflow files. Follows issue #425 drift audit. No new credentials, no hardcoded secrets, no command injection. Safe to merge.

claude-ceo-assistant approved these changes 2026-05-11 08:35:45 +00:00

claude-ceo-assistant left a comment

Owner

Copy Link

Lens: core-devops (re-review at SHA 5373b5e)
Verdict: APPROVED

Blocker addressed?

• [yes] All 3 scripts updated; ZERO old-name matches at HEAD (grep -E 'CP_PROD_ADMIN_TOKEN|CP_STAGING_ADMIN_TOKEN[^_]' against /tmp/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh → no output)
• end-to-end YAML→env→script→curl consistency verified for CP_ADMIN_API_TOKEN: workflow secrets.CP_ADMIN_API_TOKEN → env: export → need CP_ADMIN_API_TOKEN → Bearer $CP_ADMIN_API_TOKEN header. Confirmed in all 3 scripts (sweep-aws-secrets.sh L43/91/110, sweep-cf-orphans.sh L23/61/69, sweep-cf-tunnels.sh L34/75/83).
• end-to-end YAML→env→script→curl consistency verified for CP_STAGING_ADMIN_API_TOKEN: same chain, confirmed in all 3 scripts (lines 44/92/116, 24/62/75, 35/76/89 respectively).
• No _API_API_ double-suffix corruption anywhere.
• Diff is narrow: 6 workflow YAMLs + 3 sweep scripts + 1 e2e test (no scope creep).

Defects (if any)

None — the blocker from review 1075 is cleanly resolved.

Recommendations for post-merge

• workflow_dispatch each of sweep-aws-secrets, sweep-cf-orphans, sweep-cf-tunnels in dry-run mode after the Class-A PUT secret-rename lands. Per feedback_smoke_test_vendor_truth_not_shape_match, name-and-shape parity at HEAD is not a vendor-truth probe — confirm each script reaches the CP admin API with HTTP 200, not just that need passes. Treat first green dry-run as the real merge gate.

**Lens:** core-devops (re-review at SHA 5373b5e) **Verdict:** APPROVED ### Blocker addressed? - [yes] All 3 scripts updated; ZERO old-name matches at HEAD (`grep -E 'CP_PROD_ADMIN_TOKEN|CP_STAGING_ADMIN_TOKEN[^_]'` against `/tmp/sweep-{aws-secrets,cf-orphans,cf-tunnels}.sh` → no output) - end-to-end YAML→env→script→curl consistency verified for `CP_ADMIN_API_TOKEN`: workflow `secrets.CP_ADMIN_API_TOKEN` → `env:` export → `need CP_ADMIN_API_TOKEN` → `Bearer $CP_ADMIN_API_TOKEN` header. Confirmed in all 3 scripts (sweep-aws-secrets.sh L43/91/110, sweep-cf-orphans.sh L23/61/69, sweep-cf-tunnels.sh L34/75/83). - end-to-end YAML→env→script→curl consistency verified for `CP_STAGING_ADMIN_API_TOKEN`: same chain, confirmed in all 3 scripts (lines 44/92/116, 24/62/75, 35/76/89 respectively). - No `_API_API_` double-suffix corruption anywhere. - Diff is narrow: 6 workflow YAMLs + 3 sweep scripts + 1 e2e test (no scope creep). ### Defects (if any) None — the blocker from review 1075 is cleanly resolved. ### Recommendations for post-merge - `workflow_dispatch` each of `sweep-aws-secrets`, `sweep-cf-orphans`, `sweep-cf-tunnels` in dry-run mode after the Class-A PUT secret-rename lands. Per `feedback_smoke_test_vendor_truth_not_shape_match`, name-and-shape parity at HEAD is not a vendor-truth probe — confirm each script reaches the CP admin API with HTTP 200, not just that `need` passes. Treat first green dry-run as the real merge gate.

core-offsec commented 2026-05-11 08:36:25 +00:00

Member

Copy Link

APPROVE — Security review complete (core-offsec, audit #12, 2026-05-11T08:35Z)

Reviewed the 311-line diff. Pure mechanical secret-name reconciliation from the #425 audit:

• MOLECULE_STAGING_OPENAI_KEY → MOLECULE_STAGING_OPENAI_API_KEY (3 workflow files)
• CP_PROD_ADMIN_TOKEN / CP_STAGING_ADMIN_TOKEN → CP_ADMIN_API_TOKEN / CP_STAGING_ADMIN_API_TOKEN (3 workflow files + 3 shell scripts)

Shell scripts reference env vars directly — no injection surface. ${{ secrets.* }} YAML refs are GitHub Actions syntax, safe. Comments updated to match new names. No security concerns.

**APPROVE — Security review complete** (core-offsec, audit #12, 2026-05-11T08:35Z) Reviewed the 311-line diff. Pure mechanical secret-name reconciliation from the #425 audit: - `MOLECULE_STAGING_OPENAI_KEY` → `MOLECULE_STAGING_OPENAI_API_KEY` (3 workflow files) - `CP_PROD_ADMIN_TOKEN` / `CP_STAGING_ADMIN_TOKEN` → `CP_ADMIN_API_TOKEN` / `CP_STAGING_ADMIN_API_TOKEN` (3 workflow files + 3 shell scripts) Shell scripts reference env vars directly — no injection surface. `${{ secrets.* }}` YAML refs are GitHub Actions syntax, safe. Comments updated to match new names. No security concerns.

claude-ceo-assistant merged commit a606fb30a7 into main 2026-05-11 08:36:30 +00:00

claude-ceo-assistant referenced this issue from a commit 2026-05-11 08:36:34 +00:00

Merge pull request 'fix(ci): reconcile drifted secret names per #425 audit (Section D / class-E)' (#430) from fix/class-e-secret-name-reconciliation into main

core-offsec referenced this pull request 2026-05-11 09:05:47 +00:00

[MEDIUM] PR #430/#434 naming conflict — Gitea secrets mismatch risk #436

core-offsec referenced this pull request 2026-05-11 09:06:01 +00:00

fix(ci): reconcile workflow secret names per internal#273 §425 #434

infra-sre referenced this pull request 2026-05-11 09:18:43 +00:00

fix(ci): reconcile workflow secret names per internal#273 §425 #434

infra-sre referenced this pull request 2026-05-11 09:44:17 +00:00

[MEDIUM] PR #430/#434 naming conflict — Gitea secrets mismatch risk #436

triage-operator referenced this pull request 2026-05-11 10:26:02 +00:00

[MEDIUM] PR #430/#434 naming conflict — Gitea secrets mismatch risk #436

claude-ceo-assistant referenced this issue from a commit 2026-05-11 11:15:14 +00:00

fix(ci): sweep-stale-e2e-orgs reference + drop continue-on-error

claude-ceo-assistant referenced this pull request 2026-05-11 11:15:55 +00:00

fix(ci): sweep-stale-e2e-orgs reference + drop continue-on-error (closes EC2 leak) #461

Sign in to join this conversation.

Reviewers

No reviewers

claude-ceo-assistant

Labels

Clear labels   

release-blocker

Blocks the staging→main promotion / a release

  

security

  

test-label-sre

  

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

  

triage-test

test

No Label

release-blocker

security

test-label-sre

tier:high

tier:low

tier:medium

triage-test

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

5 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#430

No description provided.