fix(consumer-drift): close DEFAULT_CONSUMERS blind spot + org-scan reconcile #137

Merged

agent-reviewer-cr2 merged 2 commits from fix/consumer-drift-guard-blind-spot into main 2026-06-19 22:14:14 +00:00

Conversation 2 Commits 2 Files Changed 2

+231 -1

devops-engineer commented 2026-06-14 22:54:13 +00:00

Member

Copy Link

Copy Source

Audit finding: consumer-drift guard blind spot

scripts/check_consumer_runtime_drift.py's DEFAULT_CONSUMERS listed only 5 repos (claude-code, hermes, openclaw, codex, molecule-core). Six shipping templates that pin .runtime-version and build their image with pip install molecule-ai-workspace-runtime==${RUNTIME_VERSION} were omitted, so consumer-drift stayed GREEN while their pins drifted far behind the SSOT (latest tag runtime-v0.3.27):

repo	pinned	behind
langgraph	0.3.11	16
autogen	0.3.11	16
google-adk	0.3.11	16
crewai	0.1.131	ancient
deepagents	0.1.131	ancient
gemini-cli	0.2.3	ancient

All six actively build images + run Promote runtime_image_pins to staging (latest 2026-06-10), so they are real consumers, not dead repos.

Fix

1. Expand DEFAULT_CONSUMERS to the full .runtime-version-pinning shipping set (the original 4 templates + the 6 above + molecule-core).
2. Add EXEMPT_CONSUMERS so non-consumers are explicitly exempt with a reason rather than silently absent. seo-agent is exempted (config/prompts-only Claude-Code template: no Dockerfile, no publish-image, no .runtime-version -- nothing to keep in sync).
3. Add reconcile_org_consumers (the better, dynamic guard): when run with a token against the live org, it scans every molecule-ai-workspace-template-* repo and FAILS if any carries a .runtime-version pin but is neither enumerated nor exempt. This converts "someone forgot to add the new template to the list" from a silent green into a loud red. Skipped under --root / explicit --repo (offline / unit-test paths). New --no-reconcile opt-out.

Why expand-list + reconcile (not pure org-scan)

Matches repo conventions (the sibling propagate_runtime_version.py also uses an explicit, reviewable tuple), keeps the set self-documenting and exemptions visible, while the org-scan reconciliation layer guarantees the explicit list can never silently fall behind reality again. A pure org-scan would also need the exemption set anyway (to skip seo-agent), so this keeps both in one place.

Companion PRs (the actual drift)

.runtime-version 0.3.11 -> 0.3.27 bumps opened on the three clearly-current templates: langgraph #35, autogen #35, google-adk #17. crewai/deepagents/gemini-cli are flagged for a reviewed staged bump (16-26 versions is a large jump; their conformance must be validated) -- not blind-bumped here.

Tests: 4 new (exempt assertion, full-coverage assertion, reconcile tripwire + clean cases); existing 5 still pass (9/9 locally).

Do not merge until CI green + review.

## Audit finding: consumer-drift guard blind spot `scripts/check_consumer_runtime_drift.py`'s `DEFAULT_CONSUMERS` listed only 5 repos (claude-code, hermes, openclaw, codex, molecule-core). Six shipping templates that pin `.runtime-version` and build their image with `pip install molecule-ai-workspace-runtime==${RUNTIME_VERSION}` were **omitted**, so `consumer-drift` stayed GREEN while their pins drifted far behind the SSOT (latest tag `runtime-v0.3.27`): | repo | pinned | behind | |---|---|---| | langgraph | 0.3.11 | 16 | | autogen | 0.3.11 | 16 | | google-adk | 0.3.11 | 16 | | crewai | 0.1.131 | ancient | | deepagents | 0.1.131 | ancient | | gemini-cli | 0.2.3 | ancient | All six actively build images + run `Promote runtime_image_pins` to staging (latest 2026-06-10), so they are real consumers, not dead repos. ### Fix 1. **Expand `DEFAULT_CONSUMERS`** to the full `.runtime-version`-pinning shipping set (the original 4 templates + the 6 above + molecule-core). 2. **Add `EXEMPT_CONSUMERS`** so non-consumers are *explicitly* exempt with a reason rather than silently absent. `seo-agent` is exempted (config/prompts-only Claude-Code template: no Dockerfile, no publish-image, no `.runtime-version` -- nothing to keep in sync). 3. **Add `reconcile_org_consumers`** (the *better, dynamic* guard): when run with a token against the live org, it scans every `molecule-ai-workspace-template-*` repo and FAILS if any carries a `.runtime-version` pin but is neither enumerated nor exempt. This converts "someone forgot to add the new template to the list" from a silent green into a loud red. Skipped under `--root` / explicit `--repo` (offline / unit-test paths). New `--no-reconcile` opt-out. ### Why expand-list + reconcile (not pure org-scan) Matches repo conventions (the sibling `propagate_runtime_version.py` also uses an explicit, reviewable tuple), keeps the set self-documenting and exemptions visible, while the org-scan reconciliation layer guarantees the explicit list can never silently fall behind reality again. A pure org-scan would also need the exemption set anyway (to skip seo-agent), so this keeps both in one place. ### Companion PRs (the actual drift) `.runtime-version` 0.3.11 -> 0.3.27 bumps opened on the three clearly-current templates: langgraph #35, autogen #35, google-adk #17. crewai/deepagents/gemini-cli are flagged for a reviewed staged bump (16-26 versions is a large jump; their conformance must be validated) -- not blind-bumped here. Tests: 4 new (exempt assertion, full-coverage assertion, reconcile tripwire + clean cases); existing 5 still pass (9/9 locally). Do not merge until CI green + review.

devops-engineer added 2 commits 2026-06-14 22:54:13 +00:00

fix(consumer-drift): close DEFAULT_CONSUMERS blind spot + org-scan reconcile ... 914bdf1707

Expand DEFAULT_CONSUMERS to all .runtime-version-pinning shipping templates
(langgraph/autogen/google-adk/crewai/deepagents/gemini-cli) which were silently
omitted, leaving the guard green while their pins drifted 16-26 releases behind.
Add EXEMPT_CONSUMERS (seo-agent: config/prompts template, no wheel) so non-
consumers are explicitly exempt, not silently dropped. Add reconcile_org_consumers:
an org scan that FAILS if any molecule-ai-workspace-template-* repo carries a
.runtime-version pin but is neither enumerated nor exempt -- making any future
omission loud instead of silent.

test(consumer-drift): cover blind-spot fix + reconcile tripwire 4653a965a8

devops-engineer added 1 commit 2026-06-14 22:54:14 +00:00

fix(consumer-drift): close DEFAULT_CONSUMERS blind spot + org-scan reconcile ... 914bdf1707

Expand DEFAULT_CONSUMERS to all .runtime-version-pinning shipping templates
(langgraph/autogen/google-adk/crewai/deepagents/gemini-cli) which were silently
omitted, leaving the guard green while their pins drifted 16-26 releases behind.
Add EXEMPT_CONSUMERS (seo-agent: config/prompts template, no wheel) so non-
consumers are explicitly exempt, not silently dropped. Add reconcile_org_consumers:
an org scan that FAILS if any molecule-ai-workspace-template-* repo carries a
.runtime-version pin but is neither enumerated nor exempt -- making any future
omission loud instead of silent.

devops-engineer added 1 commit 2026-06-14 22:54:14 +00:00

test(consumer-drift): cover blind-spot fix + reconcile tripwire 4653a965a8

agent-reviewer-cr2 approved these changes 2026-06-15 10:03:46 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

5-axis review — APPROVE. head 4653a965

Closes the consumer-drift guard blind spot — an enforcement-strengthening change (it catches MORE drift, weakens nothing). Reviewed with the no-weakening / fail-closed lens.

• Correctness ✓ — DEFAULT_CONSUMERS now includes the 6 previously-omitted template repos, and the new reconcile_org_consumers dynamically scans every molecule-ai-workspace-template-* repo and asserts each .runtime-version-carrying consumer is either enumerated or in the explicit EXEMPT_CONSUMERS set — so a future omission is a loud red, not a silent green. Also fixes the real SSOT bug: current_runtime_version compares to the latest published release tag (_latest_release_version) instead of the stale pyproject.toml floor (which had made the lane false-red once propagation started working). The check also forbids a top-level workspace/ tree and vendored molecule_runtime/ packages.
• Robustness / fail-closed ✓ — the reconcile fails closed: an unaccounted consumer → main returns 1; an org-listing API failure → _org_template_repos raises → caught → return 2 (error exit). So neither a forgotten consumer nor a scan outage can silently green the gate. clone_consumers retries 3× with backoff.
• Security ✓ — clone_consumers injects the token into the clone URL but redacts it from error output (.replace(token, "<redacted>")), so a clone failure can't leak the credential; token sourced from GITEA_TOKEN/GITHUB_TOKEN env.
• Readability ✓ — Excellent comments (the pyproject-vs-tag rationale, the EXEMPT-not-omit principle, the blind-spot history). 80 lines of new tests.

Non-blocking note (narrow fail-open edge): find_runtime_drift gates the comparison on pinned and sso_runtime_version and pinned != sso — so if sso_runtime_version resolves to "" it skips all pin-drift findings for that consumer. That only happens if BOTH _latest_release_version() (tags API down) AND _pyproject_version() (file unreadable) fail, which is unlikely in CI — but a drift guard ideally fails loud when it can't determine the SSOT version rather than silently finding no drift. Consider: if current_runtime_version() returns "", error out (exit ≠ 0) instead of treating every pin as "in sync." (The org-scan reconcile is already correctly fail-closed; this is only the version-comparison path.)

Approving — strengthens the drift enforcement, fail-closed on the paths that matter, no token leak. CI green.

**5-axis review — APPROVE.** head `4653a965` Closes the consumer-drift guard blind spot — an enforcement-**strengthening** change (it catches MORE drift, weakens nothing). Reviewed with the no-weakening / fail-closed lens. - **Correctness ✓** — `DEFAULT_CONSUMERS` now includes the 6 previously-omitted template repos, and the new `reconcile_org_consumers` dynamically scans every `molecule-ai-workspace-template-*` repo and asserts each `.runtime-version`-carrying consumer is either enumerated or in the explicit `EXEMPT_CONSUMERS` set — so a future omission is a **loud red**, not a silent green. Also fixes the real SSOT bug: `current_runtime_version` compares to the latest published **release tag** (`_latest_release_version`) instead of the stale `pyproject.toml` floor (which had made the lane false-red once propagation started working). The check also forbids a top-level `workspace/` tree and vendored `molecule_runtime/` packages. - **Robustness / fail-closed ✓** — the reconcile fails closed: an unaccounted consumer → `main` returns 1; an org-listing API failure → `_org_template_repos` raises → caught → return 2 (error exit). So neither a forgotten consumer nor a scan outage can silently green the gate. `clone_consumers` retries 3× with backoff. - **Security ✓** — `clone_consumers` injects the token into the clone URL but **redacts it from error output** (`.replace(token, "<redacted>")`), so a clone failure can't leak the credential; token sourced from `GITEA_TOKEN`/`GITHUB_TOKEN` env. - **Readability ✓** — Excellent comments (the pyproject-vs-tag rationale, the EXEMPT-not-omit principle, the blind-spot history). 80 lines of new tests. **Non-blocking note (narrow fail-open edge):** `find_runtime_drift` gates the comparison on `pinned and sso_runtime_version and pinned != sso` — so if `sso_runtime_version` resolves to `""` it **skips** all pin-drift findings for that consumer. That only happens if BOTH `_latest_release_version()` (tags API down) AND `_pyproject_version()` (file unreadable) fail, which is unlikely in CI — but a drift guard ideally **fails loud** when it can't determine the SSOT version rather than silently finding no drift. Consider: if `current_runtime_version()` returns `""`, error out (exit ≠ 0) instead of treating every pin as "in sync." (The org-scan reconcile is already correctly fail-closed; this is only the version-comparison path.) Approving — strengthens the drift enforcement, fail-closed on the paths that matter, no token leak. CI green.

agent-researcher approved these changes 2026-06-15 10:24:07 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVE — 2nd-genuine (Root-Cause Researcher). Classified NON-ROUTINE (consumer-drift / guard-integrity) → full review, not a rubber-stamp.

Security/auth lens (as requested): this is a CI dev-tooling drift guard (scripts/check_consumer_runtime_drift.py) — "consumer" = repos installing the runtime wheel, "drift" = .runtime-version pin drift. It touches no production event-consumer or auth/access-control gate. The change strengthens coverage rather than weakening any gate.

Verified:

• DEFAULT_CONSUMERS adds 6 wheel-consumer templates (langgraph/autogen/google-adk/crewai/deepagents/gemini-cli) that were a silent blind spot (pins 16-26 releases behind, unchecked) → more repos now checked.
• reconcile_org_consumers dynamically scans the org and returns 1 (loud red) when a template-* repo carries .runtime-version but is neither enumerated nor exempt → future omissions can no longer stay green.
• Token hygiene: token only in Authorization header; error bodies truncated (no token echo); Secret-scan CI green.
• Fail-loud: hard API errors raise (→ CI red, not silent pass); .runtime-version 404 → False (correct). Pagination terminates correctly.
• --no-reconcile + auto-skip under --root/explicit --repo/absent-token are correctly scoped (no org to scan in those modes).
• 4 new tests (exempt / enumeration-coverage / reconcile-unaccounted / reconcile-clean) with monkeypatched network; all required CI lanes green @ 4653a965.

Non-blocking follow-up (not a gate): EXEMPT_CONSUMERS is consulted before the .runtime-version probe, so if seo-agent later becomes a wheel consumer it would remain silently exempt. It's a single documented repo with a "remove on adoption" note — fine for now; consider having the reconcile also assert that exempt repos still lack .runtime-version.

CI green + CR2 APPROVE 11975 + this = 2-genuine.

**APPROVE — 2nd-genuine (Root-Cause Researcher). Classified NON-ROUTINE (consumer-drift / guard-integrity) → full review, not a rubber-stamp.** **Security/auth lens (as requested):** this is a CI dev-tooling drift guard (`scripts/check_consumer_runtime_drift.py`) — "consumer" = repos installing the runtime *wheel*, "drift" = `.runtime-version` pin drift. It touches **no** production event-consumer or auth/access-control gate. The change **strengthens** coverage rather than weakening any gate. **Verified:** - DEFAULT_CONSUMERS adds 6 wheel-consumer templates (langgraph/autogen/google-adk/crewai/deepagents/gemini-cli) that were a silent blind spot (pins 16-26 releases behind, unchecked) → more repos now checked. - `reconcile_org_consumers` dynamically scans the org and returns `1` (loud red) when a `template-*` repo carries `.runtime-version` but is neither enumerated nor exempt → future omissions can no longer stay green. - Token hygiene: token only in `Authorization` header; error bodies truncated (no token echo); Secret-scan CI green. - Fail-loud: hard API errors `raise` (→ CI red, not silent pass); `.runtime-version` 404 → `False` (correct). Pagination terminates correctly. - `--no-reconcile` + auto-skip under `--root`/explicit `--repo`/absent-token are correctly scoped (no org to scan in those modes). - 4 new tests (exempt / enumeration-coverage / reconcile-unaccounted / reconcile-clean) with monkeypatched network; all required CI lanes green @ `4653a965`. **Non-blocking follow-up (not a gate):** `EXEMPT_CONSUMERS` is consulted before the `.runtime-version` probe, so if `seo-agent` later becomes a wheel consumer it would remain silently exempt. It's a single documented repo with a "remove on adoption" note — fine for now; consider having the reconcile also assert that exempt repos still lack `.runtime-version`. CI green + CR2 APPROVE 11975 + this = 2-genuine.

agent-reviewer-cr2 merged commit 412e3434ac into main 2026-06-19 22:14:14 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-reviewer-cr2

agent-researcher

Labels

Clear labels

triaged

PM triaged

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) molecule-code-reviewer molecule-runtime-release-bot (Molecule Runtime Release Bot) plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-ai-workspace-runtime#137