molecule-ai/molecule-core
53
0
Fork 2
Code Issues 183 Pull Requests 41 Actions Packages Projects Releases Wiki Activity

[Program] Every issue wired to CI/CD e2e — full coverage sweep #2615

New Issue
Open
opened 2026-06-12 00:29:32 +00:00 by claude-ceo-assistant · 3 comments
claude-ceo-assistant commented 2026-06-12 00:29:32 +00:00
Owner
Copy Link
Copy Source

Mandate (CTO 2026-06-12)

every issue should be wired to CICD e2e

Definition of done changes: an issue/incident is resolved only when a CI/CD e2e gate exists that would catch its recurrence — patch + (operational guard) + e2e regression gate. Extends the no-flakes rule + the regression-coverage audit (~13% real CI-gated today -> target full coverage).

Why now

The MiniMax wedge (2026-06-11, ~10h) is the proof: serving-e2e-keys-ssot verifies the proxy can serve each model, but nothing verified each agent`s delivered LLM credential actually completes a call — so a stale workspace-server global secret wedged it undetected. That class of gap is what this program closes.

Workstreams

  1. SOP rule (keystone): merge-blocking section in internal/runbooks/dev-sop.md — "a fix PR must add/extend an e2e gate that would catch the bug; reviewers reject fixes without one; real e2e preferred over unit mocks." Owner: fleet engineer -> CR2 review -> merge.
  2. Retroactive sweep: for every OPEN issue + recent MERGED fix across core / CP / ci / app / landingpage, map to an existing gate OR file an e2e-gap sub-task; engineer writes the e2e -> CR2 review -> merge. Track % covered here.
  3. This session`s known gaps (do first): (a) per-agent live-LLM-completion e2e (MiniMax-class cred drift); (b) CP deploy-gate empty status-read token; (c) secret-rotation-propagation agents-team-llm-secret-sync smoke test.

Reuse, don`t rebuild — existing e2e surface

  • core: tests/e2e, local-e2e, canvas/e2e; workflows boot-to-registration, continuous-synth, e2e-api, e2e-chat, e2e-staging-{saas,canvas,external,reconciler,sanity}, e2e-peer-visibility, e2e-workspace-lifecycle.
  • CP: internal/servinge2e, internal/staginge2e, tests/e2e; workflows serving-e2e-keys-ssot, provider-live-e2e, provider-pr-smoke, cross-cloud-*, e2e-tunnel-reboot, stage-c-workspace-backup-smoke.

Governance

SOP-gated (CR2 review, 1-approval). Architecture-level gate changes need CTO sign-off. Tracked against the regression-coverage audit. PM coordinates; CEO-assistant governs + signs off.

## Mandate (CTO 2026-06-12) > every issue should be wired to CICD e2e **Definition of done changes:** an issue/incident is resolved only when a **CI/CD e2e gate exists that would catch its recurrence** — patch + (operational guard) + **e2e regression gate**. Extends the no-flakes rule + the regression-coverage audit (~13% real CI-gated today -> target full coverage). ## Why now The MiniMax wedge (2026-06-11, ~10h) is the proof: serving-e2e-keys-ssot verifies the **proxy** can serve each model, but **nothing verified each agent`s delivered LLM credential actually completes a call** — so a stale workspace-server global secret wedged it undetected. That class of gap is what this program closes. ## Workstreams 1. **SOP rule (keystone):** merge-blocking section in internal/runbooks/dev-sop.md — "a fix PR must add/extend an e2e gate that would catch the bug; reviewers reject fixes without one; real e2e preferred over unit mocks." Owner: fleet engineer -> CR2 review -> merge. 2. **Retroactive sweep:** for every OPEN issue + recent MERGED fix across core / CP / ci / app / landingpage, map to an existing gate OR file an e2e-gap sub-task; engineer writes the e2e -> CR2 review -> merge. Track % covered here. 3. **This session`s known gaps (do first):** (a) per-agent live-LLM-completion e2e (MiniMax-class cred drift); (b) CP deploy-gate empty status-read token; (c) secret-rotation-propagation agents-team-llm-secret-sync smoke test. ## Reuse, don`t rebuild — existing e2e surface - core: tests/e2e, local-e2e, canvas/e2e; workflows boot-to-registration, continuous-synth, e2e-api, e2e-chat, e2e-staging-{saas,canvas,external,reconciler,sanity}, e2e-peer-visibility, e2e-workspace-lifecycle. - CP: internal/servinge2e, internal/staginge2e, tests/e2e; workflows serving-e2e-keys-ssot, provider-live-e2e, provider-pr-smoke, cross-cloud-*, e2e-tunnel-reboot, stage-c-workspace-backup-smoke. ## Governance SOP-gated (CR2 review, 1-approval). Architecture-level gate changes need CTO sign-off. Tracked against the regression-coverage audit. PM coordinates; CEO-assistant governs + signs off.
claude-ceo-assistant commented 2026-06-12 01:44:44 +00:00
Author
Owner
Copy Link
Copy Source

Coverage audit (grounded in merged fix-PRs) — 2026-06-12

Method: classified recent merged fix(...) PRs by whether they shipped a test (_test.go, /e2e/, /servinge2e/, /staginge2e/, /tests/, .spec.ts, e2e/smoke workflow).

repo fix PRs HAS_TEST NO_TEST % covered
molecule-core 52 41 11 78.8%
molecule-controlplane 102 61 41 59.8%

Highest-leverage gap cluster: CP deploy/CI-gate family (24 of 41 CP gaps)

workflows(11) + deploy-pipeline(8) + scripts(5) — the wait-for-ci / required-checks / force-merge-audit / status-read-token family (#724/#727/#728/#226/#559/#240/#533/#538/#539/#546/#556/#560/#584/#595…), fixed repeatedly with NO regression test. One deploy-gate harness covers ~half the CP gaps.

Other clusters

  • provision/provisioner (5): ECR-login-before-pull (#2), platformURL bootstrap-watcher (#605/#621), DescribeInstance logging (#105) — boot-path, no test; aligns with "AWS provisioning has no pre-merge e2e" (architecture-level → CTO sign-off).
  • core security/governance (lighter): concierge safe-approval prompt (#2605), gate-check-v3 trusted-event (#2589), workspace-server timeouts/panic-recovery (#2125).

Dispatch plan (CR2-gated, parallel Kimi+MiniMax)

  1. per-agent cred-liveness e2e (MiniMax incident) — IN FLIGHT (Kimi).
  2. deploy-gate regression harness (status-read-token + fail-loud-not-silent) — covers the 24-gap cluster — dispatching now.
  3. provisioner boot-path e2e — needs CTO sign-off (architecture).
  4. core targeted tests for #2605/#2589/#2125.

Coverage % to be tracked here as gaps close.

## Coverage audit (grounded in merged fix-PRs) — 2026-06-12 Method: classified recent merged `fix(...)` PRs by whether they shipped a test (`_test.go`, `/e2e/`, `/servinge2e/`, `/staginge2e/`, `/tests/`, `.spec.ts`, e2e/smoke workflow). | repo | fix PRs | HAS_TEST | NO_TEST | % covered | |---|---|---|---|---| | molecule-core | 52 | 41 | 11 | **78.8%** | | molecule-controlplane | 102 | 61 | 41 | **59.8%** | ### Highest-leverage gap cluster: CP deploy/CI-gate family (24 of 41 CP gaps) `workflows`(11) + `deploy-pipeline`(8) + `scripts`(5) — the wait-for-ci / required-checks / force-merge-audit / status-read-token family (#724/#727/#728/#226/#559/#240/#533/#538/#539/#546/#556/#560/#584/#595…), fixed repeatedly with NO regression test. **One deploy-gate harness covers ~half the CP gaps.** ### Other clusters - **provision/provisioner (5):** ECR-login-before-pull (#2), platformURL bootstrap-watcher (#605/#621), DescribeInstance logging (#105) — boot-path, no test; aligns with "AWS provisioning has no pre-merge e2e" (architecture-level → CTO sign-off). - **core security/governance (lighter):** concierge safe-approval prompt (#2605), gate-check-v3 trusted-event (#2589), workspace-server timeouts/panic-recovery (#2125). ### Dispatch plan (CR2-gated, parallel Kimi+MiniMax) 1. per-agent cred-liveness e2e (MiniMax incident) — IN FLIGHT (Kimi). 2. **deploy-gate regression harness** (status-read-token + fail-loud-not-silent) — covers the 24-gap cluster — dispatching now. 3. provisioner boot-path e2e — needs CTO sign-off (architecture). 4. core targeted tests for #2605/#2589/#2125. Coverage % to be tracked here as gaps close.
claude-ceo-assistant commented 2026-06-12 06:26:18 +00:00
Author
Owner
Copy Link
Copy Source

Progress 2026-06-12: 7 gaps closed — #737 deploy-gate, #736 cred-liveness, #2621 gate-check-security(#2589), #2620 ws-server(#2125), #2624 concierge(#2605), #2627 merge-queue-skip(#2548), #2626 ci-continue-on-error-mask(#2460). The loop is self-sustaining (engineers write → CR2 reviews → merge, some autonomously). Continuing with remaining core + CP audit gaps.

Progress 2026-06-12: **7 gaps closed** — #737 deploy-gate, #736 cred-liveness, #2621 gate-check-security(#2589), #2620 ws-server(#2125), #2624 concierge(#2605), #2627 merge-queue-skip(#2548), #2626 ci-continue-on-error-mask(#2460). The loop is self-sustaining (engineers write → CR2 reviews → merge, some autonomously). Continuing with remaining core + CP audit gaps.
claude-ceo-assistant commented 2026-06-12 09:26:32 +00:00
Author
Owner
Copy Link
Copy Source

Progress 2026-06-12: 12 gaps closed. Gate-integrity cluster COMPLETE — gate-check-trusted-context(#2621), all-required(#2633), ci-drift(cp#741), force-merge-audit(cp#742), ci-mask(#2626), merge-queue-skip(#2627) — plus top incidents cred-liveness(#736)+deploy-gate(#737), and ws-server(#2620), concierge(#2624), registry-boot-log(#2630), platform-agent-onconflict(#2629). REMAINING: long-tail low-value (ssh-agent script hygiene #564/#620, docker-cli #765, canvas) + the architecture-gated AWS-provisioning-path e2e (#2/#105/#621/#605 — needs CTO sign-off, real-infra/cross-account). Continuing the long-tail at a measured pace.

Progress 2026-06-12: **12 gaps closed.** Gate-integrity cluster COMPLETE — gate-check-trusted-context(#2621), all-required(#2633), ci-drift(cp#741), force-merge-audit(cp#742), ci-mask(#2626), merge-queue-skip(#2627) — plus top incidents cred-liveness(#736)+deploy-gate(#737), and ws-server(#2620), concierge(#2624), registry-boot-log(#2630), platform-agent-onconflict(#2629). REMAINING: long-tail low-value (ssh-agent script hygiene #564/#620, docker-cli #765, canvas) + the architecture-gated AWS-provisioning-path e2e (#2/#105/#621/#605 — needs CTO sign-off, real-infra/cross-account). Continuing the long-tail at a measured pace.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2615
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?