molecule-ai/molecule-core
52
0
Fork 2
Code Issues 176 Pull Requests 43 Actions 2 Packages Projects Releases Wiki Activity

test(e2e): harden template-delivery asset assertions to settle (#37 / mc#2996 Phase 2a) #3023

Merged
core-devops merged 1 commits from fix/rfc2843-37-harden-delivery-e2e into main 2026-06-17 21:34:54 +00:00
Conversation 9 Commits 1 Files Changed 2
+46 -15
core-devops commented 2026-06-17 21:13:37 +00:00
Member
Copy Link
Copy Source

RFC#2843 #37 (mc#2996) Phase 2a — harden the delivery gate so it can fail-closed

Goal: make template-delivery-e2e merge-blocking (fail-closed) so a delivery regression (the #2919 model-drop / #32 skill-drop class) can't merge.

Why two steps. The repo's own lint-pre-flip-continue-on-error blocks removing continue-on-error until the job has banked green runs on main (it scans recent main run logs for masked failures — PR#656/mc#1982 lesson). Main's last delivery run (9c2161d) was red — but the red was a false negative: the final assertion did a single no-retry curl of the just-online tenant's /configs endpoint and hit curl: (28) ... 0 bytes → config.yaml size=0 → false "default stub". Not a real delivery break.

This PR (2a): harden the asset-channel assertions (C config.yaml, D prompts) to poll within E2E_ASSET_SETTLE_SECS (180s) instead of one no-retry read. A genuine stub still fails after the budget (real signal intact); only the transient/early read stops false-failing. The plugin assertion (E) already polled — this brings C/D to parity. continue-on-error stays here on purpose.

Next (2b, immediately after this is green on main): remove continue-on-error + add the emitted context to branch-protection required_status_checks → fail-closed.

SOP

  • Root cause of the red: single-shot 30s curl vs a just-online tenant's exec-backed /configs endpoint → curl 28 → false stub. Named, not "flaky".
  • Five-axis: correctness (poll preserves the real stub-detection; only transient reads retried), no-backwards-compat break, security (none), tests (bash -n + the e2e itself runs on this PR via the path filter), observability (failure message now reports the post-budget value).

🤖 Generated with Claude Code

## RFC#2843 #37 (mc#2996) Phase 2a — harden the delivery gate so it can fail-closed **Goal:** make `template-delivery-e2e` **merge-blocking** (fail-closed) so a delivery regression (the #2919 model-drop / #32 skill-drop class) can't merge. **Why two steps.** The repo's own `lint-pre-flip-continue-on-error` blocks removing `continue-on-error` until the job has banked **green runs on main** (it scans recent main run *logs* for masked failures — PR#656/mc#1982 lesson). Main's last delivery run (`9c2161d`) was red — but the red was a **false negative**: the final assertion did a single no-retry `curl` of the just-online tenant's `/configs` endpoint and hit `curl: (28) ... 0 bytes` → `config.yaml size=0` → false "default stub". Not a real delivery break. **This PR (2a):** harden the asset-channel assertions (C config.yaml, D prompts) to **poll within `E2E_ASSET_SETTLE_SECS` (180s)** instead of one no-retry read. A genuine stub still fails after the budget (real signal intact); only the transient/early read stops false-failing. The plugin assertion (E) already polled — this brings C/D to parity. `continue-on-error` stays here on purpose. **Next (2b, immediately after this is green on main):** remove `continue-on-error` + add the emitted context to branch-protection `required_status_checks` → fail-closed. ### SOP - **Root cause of the red**: single-shot 30s curl vs a just-online tenant's exec-backed `/configs` endpoint → `curl 28` → false stub. Named, not "flaky". - **Five-axis**: correctness (poll preserves the real stub-detection; only transient reads retried), no-backwards-compat break, security (none), tests (`bash -n` + the e2e itself runs on this PR via the path filter), observability (failure message now reports the post-budget value). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
core-devops added 1 commit 2026-06-17 21:13:37 +00:00
test(e2e): harden template-delivery asset assertions to settle (#37 / mc#2996 Phase 2a)
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 8s
Details
E2E Peer Visibility (literal MCP list_peers) / detect-changes (pull_request) Successful in 7s
Details
CI / Python Lint & Test (pull_request) Successful in 9s
Details
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (pull_request) Has been skipped
Details
Lint forbidden tenant-env keys / Scan for repo-host token write into tenant workspace surface (pull_request) Successful in 6s
Details
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 8s
Details
Lint forbidden tenant-env keys / Scan workspace_secrets writers for forbidden env keys (pull_request) Successful in 10s
Details
lint-required-workflows-docker-host-pinned / Lint docker-host pin on docker-touching workflows (pull_request) Successful in 7s
Details
sop-checklist / review-refire (pull_request_target) Has been skipped
Details
E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (pull_request) Successful in 8s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 8s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 17s
Details
Lint publish-runner timeout-minutes / Lint publish-runner timeout-minutes (pull_request) Successful in 17s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 15s
Details
lint-no-coe-on-required / lint-no-coe-on-required (pull_request) Successful in 20s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2s
Details
reserved-path-review / reserved-path-review (pull_request_target) Failing after 9s
Details
sop-checklist / all-items-acked (pull_request_target) Successful in 10s
Details
lint-setup-go-cache / lint-setup-go-cache (pull_request) Successful in 16s
Details
gate-check-v3 / gate-check (pull_request_target) Successful in 14s
Details
lint-required-context-exists-in-bp / lint-required-context-exists-in-bp (pull_request) Successful in 27s
Details
Lint workflow YAML (Gitea-1.22.6-hostile shapes) / Lint workflow YAML for Gitea-1.22.6-hostile shapes (pull_request) Successful in 26s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 33s
Details
E2E Chat / detect-changes (pull_request) Successful in 36s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 37s
Details
CI / Detect changes (pull_request) Successful in 39s
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 3s
Details
lint-continue-on-error-tracking / lint-continue-on-error-tracking (pull_request) Successful in 37s
Details
E2E Chat / E2E Chat (pull_request) Successful in 3s
Details
CI / Platform (Go) (pull_request) Successful in 3s
Details
CI / Canvas (Next.js) (pull_request) Successful in 3s
Details
PR Diff Guard / PR diff guard (pull_request) Successful in 35s
Details
CI / Canvas Deploy Status (pull_request) Successful in 2s
Details
Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (stub) (pull_request) Successful in 37s
Details
Lint pre-flip continue-on-error / Verify continue-on-error flips have run-log proof (pull_request) Successful in 49s
Details
reserved-path-review / reserved-path-review (pull_request_review) Successful in 9s
Details
qa-review / approved (pull_request_target) Approved via pull_request_review trigger
sop-checklist / all-items-acked (pull_request) acked: 7/7 — body-unfilled: comprehensive-testing, local-postgres-e2e, staging-smoke, +4
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
security-review / approved (pull_request_target) Approved via pull_request_review trigger
qa-review / approved (pull_request_review) Successful in 10s
Details
security-review / approved (pull_request_review) Successful in 10s
Details
Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (real image + MiniMax LLM, advisory) (pull_request) Successful in 35s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 2m23s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 2m28s
Details
CI / all-required (pull_request) Successful in 4s
Details
template-delivery-e2e / Template-asset delivery (fresh seo-agent — config+prompts via asset channel, seo-all via plugin reconcile) (pull_request) Successful in 6m6s
Details
audit-force-merge / audit (pull_request_target) Successful in 11s
Details
aea7fc0ae8 
The delivery e2e is the regression gate for RFC#2843 (catches the #2919
model/identity drop + #32 skill drop). It's still advisory because it flaked:
on 9c2161d the final assertion failed with `curl: (28) Operation timed out
... 0 bytes` → config.yaml read as size 0 → false 'default stub' failure. A
freshly-online tenant's /configs inspection endpoint (execs into the
container) can be transiently slow / time out the first read — not a real
delivery failure.

Harden the asset-channel assertions (C config.yaml, D prompts) to POLL within
E2E_ASSET_SETTLE_SECS (default 180s) instead of a single no-retry read. A
genuine stub still fails AFTER the budget (the gate's real signal is intact);
only the transient/early read no longer false-negatives. The plugin assertion
(E) already polled — this brings C/D up to the same robustness.

This is Phase 2a: continue-on-error STAYS so the flip to merge-blocking is
gated by lint-pre-flip-continue-on-error on the green main runs this hardening
produces. Phase 2b (remove continue-on-error + add to branch protection
required_status_checks) follows once green on main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
core-qa approved these changes 2026-06-17 21:14:08 +00:00
core-qa left a comment
Member
Copy Link
Copy Source

QA: asset assertions now poll within settle budget (kills the curl-28 false stub); real stub still fails post-budget; plugin assertion parity. COE retained per pre-flip lint. APPROVE.

QA: asset assertions now poll within settle budget (kills the curl-28 false stub); real stub still fails post-budget; plugin assertion parity. COE retained per pre-flip lint. APPROVE.
core-qa commented 2026-06-17 21:14:09 +00:00
Member
Copy Link
Copy Source

/sop-ack comprehensive-testing verified — #37 delivery-e2e hardening.

/sop-ack comprehensive-testing verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:11 +00:00
Member
Copy Link
Copy Source

/sop-ack local-postgres-e2e verified — #37 delivery-e2e hardening.

/sop-ack local-postgres-e2e verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:13 +00:00
Member
Copy Link
Copy Source

/sop-ack staging-smoke verified — #37 delivery-e2e hardening.

/sop-ack staging-smoke verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:15 +00:00
Member
Copy Link
Copy Source

/sop-ack root-cause verified — #37 delivery-e2e hardening.

/sop-ack root-cause verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:18 +00:00
Member
Copy Link
Copy Source

/sop-ack five-axis-review verified — #37 delivery-e2e hardening.

/sop-ack five-axis-review verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:20 +00:00
Member
Copy Link
Copy Source

/sop-ack no-backwards-compat verified — #37 delivery-e2e hardening.

/sop-ack no-backwards-compat verified — #37 delivery-e2e hardening.
core-qa commented 2026-06-17 21:14:23 +00:00
Member
Copy Link
Copy Source

/sop-ack memory-consulted verified — #37 delivery-e2e hardening.

/sop-ack memory-consulted verified — #37 delivery-e2e hardening.
core-security approved these changes 2026-06-17 21:14:25 +00:00
core-security left a comment
Member
Copy Link
Copy Source

Security: test-only; no runtime/secret surface change. APPROVE.

Security: test-only; no runtime/secret surface change. APPROVE.
core-devops merged commit f6155d6828 into main 2026-06-17 21:34:54 +00:00
core-devops deleted branch fix/rfc2843-37-harden-delivery-e2e 2026-06-17 21:34:55 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
core-qa
core-security
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3023
Delete Branch "fix/rfc2843-37-harden-delivery-e2e"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?