molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 2 Packages Projects Releases Wiki Activity

RFC: cross-template vendor-CLI wrapper schema-drift guards — apply codex lessons to claude-code, hermes, openclaw #1691

New Issue
Open
opened 2026-05-22 22:48:05 +00:00 by hongming · 0 comments
hongming commented 2026-05-22 22:48:05 +00:00
Owner
Copy Link
Copy Source

Why

2026-05-22 codex outage root cause: vendor CLI (openai-codex) changed wire protocol between 0.72 and 0.130 without bumping protocol version. Executor matched only the 0.72 method names. ~7h prod outage before identified.

The same risk exists for EVERY workspace template that wraps a vendor CLI as a subprocess:

  • claude-code template wraps claude CLI
  • hermes template wraps hermes CLI
  • openclaw template wraps openclaw CLI
  • (And future templates as we add runtimes.)

Each has its own subprocess-IO + notification-parsing layer that could silently break on vendor upgrade.

Proposal

Apply ALL four codex-side guards to EVERY vendor-CLI-wrapping template:

1. Pin vendor CLI version in Dockerfile (build-time grep -F check)

See codex template issue #34. Same pattern: ARG + npm install @version + grep verification.

2. Notification-schema contract test in CI

See codex template issue #31. Spawn the real vendor CLI, drive a real turn, assert each method name is recognized by the adapter's notification handlers. Plus an offline replay-fixture variant for credential-less CI.

3. Self-diagnosing wedge log

See codex template issue #32. Whenever an adapter's wedge-detector fires, include the actual notification methods seen + their counts. Never just say (deltas=0) without naming what WAS received.

4. CODEX_DEBUG=1-style instrumentation switch

See codex template issue #33. Bake stderr-trace hooks gated on an env var per template (CLAUDE_CODE_DEBUG, HERMES_DEBUG, OPENCLAW_DEBUG). Then prod debugging never requires docker cp + hot-patches.

Acceptance

  • Each vendor-CLI-wrapping template has all 4 guards merged
  • Cross-template audit: list every *_app_server*.py / equivalent + verify it has the patterns
  • README in each template links the troubleshooting flow (CLI_DEBUG=1 → check wedge log method tally → find vendor schema-drift early)

Touchpoints

  • molecule-ai-workspace-template-codex (already has the immediate fix via PR #30; the followups are #31-34)
  • molecule-ai-workspace-template-claude-code
  • molecule-ai-workspace-template-hermes
  • molecule-ai-workspace-template-openclaw
  • (audit other workspace-template-* repos and file per-template tickets as needed)

Owner

TBD. Could be parallelized 1 owner per template. ~1d each, ~1w total.

## Why 2026-05-22 codex outage root cause: vendor CLI (openai-codex) changed wire protocol between 0.72 and 0.130 without bumping protocol version. Executor matched only the 0.72 method names. ~7h prod outage before identified. The same risk exists for EVERY workspace template that wraps a vendor CLI as a subprocess: - claude-code template wraps `claude` CLI - hermes template wraps `hermes` CLI - openclaw template wraps `openclaw` CLI - (And future templates as we add runtimes.) Each has its own subprocess-IO + notification-parsing layer that could silently break on vendor upgrade. ## Proposal Apply ALL four codex-side guards to EVERY vendor-CLI-wrapping template: ### 1. Pin vendor CLI version in Dockerfile (build-time grep -F check) See codex template issue #34. Same pattern: ARG + npm install @version + grep verification. ### 2. Notification-schema contract test in CI See codex template issue #31. Spawn the real vendor CLI, drive a real turn, assert each method name is recognized by the adapter's notification handlers. Plus an offline replay-fixture variant for credential-less CI. ### 3. Self-diagnosing wedge log See codex template issue #32. Whenever an adapter's wedge-detector fires, include the actual notification methods seen + their counts. Never just say `(deltas=0)` without naming what WAS received. ### 4. CODEX_DEBUG=1-style instrumentation switch See codex template issue #33. Bake stderr-trace hooks gated on an env var per template (CLAUDE_CODE_DEBUG, HERMES_DEBUG, OPENCLAW_DEBUG). Then prod debugging never requires docker cp + hot-patches. ## Acceptance - Each vendor-CLI-wrapping template has all 4 guards merged - Cross-template audit: list every `*_app_server*.py` / equivalent + verify it has the patterns - README in each template links the troubleshooting flow (CLI_DEBUG=1 → check wedge log method tally → find vendor schema-drift early) ## Touchpoints - molecule-ai-workspace-template-codex (already has the immediate fix via PR #30; the followups are #31-34) - molecule-ai-workspace-template-claude-code - molecule-ai-workspace-template-hermes - molecule-ai-workspace-template-openclaw - (audit other workspace-template-* repos and file per-template tickets as needed) ## Owner TBD. Could be parallelized 1 owner per template. ~1d each, ~1w total.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1691
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?