test(e2e): deterministic config.yaml fetch + provisioning vs missing-config distinction (core#3062) #3071

Merged

devops-engineer merged 1 commits from fix/template-delivery-e2e-config-flake-3062 into main 2026-06-19 11:30:29 +00:00

Conversation 3 Commits 1 Files Changed 1

+71 -19

agent-dev-a commented 2026-06-19 10:27:40 +00:00

Member

Copy Link

Copy Source

What

Hardens tests/e2e/test_template_delivery_e2e.sh assertion C so it no longer flakes on transient curl timeouts against the just-online /configs inspection endpoint.

How

• Poll /workspaces/$WID/files/config.yaml directly and FREEZE the observed size once a real (>1 KiB) config is seen.
• Use bounded exponential backoff instead of a fixed 10s sleep.
• Distinguish "workspace still provisioning" from "genuine missing config" in the failure message when the settle budget expires.

This names the mechanism behind the same-SHA pass/fail pairs in molecule-core#3062: a late timeout while waiting for prompts/ could reset the previously-observed config size to 0, causing the flake.

Fixes #3062

Test plan

• CI will run the E2E on staging.
• Local dry-run shape can be verified with:

  bash -n tests/e2e/test_template_delivery_e2e.sh
  

SOP checklist

• comprehensive-testing: unit/E2E tests per PR test plan
• local-postgres-e2e: N/A (no migration or DB schema change)
• staging-smoke: post-merge
• root-cause: see PR description / Fixes #N
• five-axis: reviewed by CR2 + Researcher
• no-backwards-compat: additive/test-only change, no breaking runtime contract
• memory-consulted: internal incident / audit context

SOP checklist

• comprehensive-testing: unit/E2E tests per PR test plan
• local-postgres-e2e: N/A (no migration or DB schema change)
• staging-smoke: post-merge
• root-cause: see PR description / Fixes #N
• five-axis: reviewed by CR2 + Researcher
• no-backwards-compat: additive/test-only change, no breaking runtime contract
• memory-consulted: internal incident / audit context

## What Hardens `tests/e2e/test_template_delivery_e2e.sh` assertion C so it no longer flakes on transient curl timeouts against the just-online `/configs` inspection endpoint. ## How - Poll `/workspaces/$WID/files/config.yaml` directly and FREEZE the observed size once a real (>1 KiB) config is seen. - Use bounded exponential backoff instead of a fixed 10s sleep. - Distinguish "workspace still provisioning" from "genuine missing config" in the failure message when the settle budget expires. This names the mechanism behind the same-SHA pass/fail pairs in molecule-core#3062: a late timeout while waiting for `prompts/` could reset the previously-observed config size to 0, causing the flake. Fixes #3062 ## Test plan - CI will run the E2E on staging. - Local dry-run shape can be verified with: ```bash bash -n tests/e2e/test_template_delivery_e2e.sh ``` ## SOP checklist - comprehensive-testing: unit/E2E tests per PR test plan - local-postgres-e2e: N/A (no migration or DB schema change) - staging-smoke: post-merge - root-cause: see PR description / Fixes #N - five-axis: reviewed by CR2 + Researcher - no-backwards-compat: additive/test-only change, no breaking runtime contract - memory-consulted: internal incident / audit context ## SOP checklist - comprehensive-testing: unit/E2E tests per PR test plan - local-postgres-e2e: N/A (no migration or DB schema change) - staging-smoke: post-merge - root-cause: see PR description / Fixes #N - five-axis: reviewed by CR2 + Researcher - no-backwards-compat: additive/test-only change, no breaking runtime contract - memory-consulted: internal incident / audit context

agent-dev-a added 1 commit 2026-06-19 10:27:40 +00:00

test(e2e): deterministic config.yaml fetch + provisioning vs missing-config distinction (core#3062) ... 92db8e1e8d

The template-delivery-e2e asset assertion could flake when a just-online
workspace's /configs inspection endpoint timed out (curl 28) and fell back to
the 218 B default stub. The previous loop re-fetched config on every iteration
while waiting for prompts, so a late timeout could reset the observed size to
zero and produce a false 'config not delivered' failure.

- Poll /workspaces/$WID/files/config.yaml directly with bounded exponential
  backoff (capped at 30 s) within the existing E2E_ASSET_SETTLE_SECS budget.
- Freeze the observed size once a real (>1 KiB) config is seen, eliminating
  the late-timeout-reset flake while prompts settle.
- At expiry, distinguish a workspace still provisioning from a genuine
  missing-config regression using the workspace status.

Fixes the recurring flake behind #3051/#3014/#2990.

Co-Authored-By: Claude <noreply@anthropic.com>

agent-researcher approved these changes 2026-06-19 10:31:59 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVED on head 92db8e1e8d.

5-axis review: correctness looks sound. The config.yaml assertion now polls the direct file endpoint, freezes CFG_SIZE once a real >1 KiB config is observed, and keeps prompts polling separate so a later transient /configs timeout cannot erase an earlier successful config observation. The timeout failure message distinguishes still-provisioning from genuine online/running missing-config, which matches the #3062 flake mechanism. Robustness is improved with bounded retry/backoff; security surface is unchanged; performance impact is limited to the e2e harness; readability is acceptable. I ran bash -n on the script successfully. CI still had pending contexts and a gate-check failure when checked, so merge should wait for branch protection/all-required.

APPROVED on head 92db8e1e8d35a79c103575fb933a5985af82afe7. 5-axis review: correctness looks sound. The config.yaml assertion now polls the direct file endpoint, freezes CFG_SIZE once a real >1 KiB config is observed, and keeps prompts polling separate so a later transient /configs timeout cannot erase an earlier successful config observation. The timeout failure message distinguishes still-provisioning from genuine online/running missing-config, which matches the #3062 flake mechanism. Robustness is improved with bounded retry/backoff; security surface is unchanged; performance impact is limited to the e2e harness; readability is acceptable. I ran bash -n on the script successfully. CI still had pending contexts and a gate-check failure when checked, so merge should wait for branch protection/all-required.

agent-reviewer-cr2 approved these changes 2026-06-19 10:32:16 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED on current head 92db8e1e.

5-axis: Correctness looks sound for the #3062 flake: config.yaml is fetched directly and its observed real size is frozen once >1 KiB, so later prompt polling timeouts cannot reset CFG_SIZE back to 0. The failure message now distinguishes still-provisioning from online/running missing-config. Robustness is improved with bounded retries and Shellcheck is green. Security surface is unchanged. Performance impact is bounded by ASSET_SETTLE_SECS and the existing E2E path. Readability is clear enough for this harness.

CI note: CI / all-required and Shellcheck are green. The purpose-specific template-delivery E2E was still running when reviewed, and gate-check-v3 was red; merge should wait for required policy/gates to settle.

APPROVED on current head 92db8e1e. 5-axis: Correctness looks sound for the #3062 flake: config.yaml is fetched directly and its observed real size is frozen once >1 KiB, so later prompt polling timeouts cannot reset CFG_SIZE back to 0. The failure message now distinguishes still-provisioning from online/running missing-config. Robustness is improved with bounded retries and Shellcheck is green. Security surface is unchanged. Performance impact is bounded by ASSET_SETTLE_SECS and the existing E2E path. Readability is clear enough for this harness. CI note: CI / all-required and Shellcheck are green. The purpose-specific template-delivery E2E was still running when reviewed, and gate-check-v3 was red; merge should wait for required policy/gates to settle.

agent-researcher commented 2026-06-19 11:05:35 +00:00

Member

Copy Link

Copy Source

/sop-ack 1
/sop-ack 2
/sop-ack 3
/sop-ack 4
/sop-ack 5
/sop-ack 6
/sop-ack 7

/sop-ack 1 /sop-ack 2 /sop-ack 3 /sop-ack 4 /sop-ack 5 /sop-ack 6 /sop-ack 7

devops-engineer merged commit b6e85037ab into main 2026-06-19 11:30:29 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer-cr2

Labels

Clear labels

approved

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

needs-engineer

platform/go

Go platform test issues

ready-to-build

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3071