fix(e2e): bounded retry around external workspace-create in staging concierge user_tasks (#2743) #2746

Merged

devops-engineer merged 1 commits from fix/2743-staging-concierge-create-retry into main 2026-06-13 10:20:13 +00:00

Conversation 3 Commits 1 Files Changed 1

+39 -6

devops-engineer commented 2026-06-13 10:17:12 +00:00

Member

Copy Link

Copy Source

What

Bounded retry around the external workspace-create only in the staging concierge user_tasks E2E (tests/e2e/test_staging_concierge_e2e.sh).

Root cause (issue #2743, Researcher RCA — confirmed)

main 6163f6636fc8 was red in run 358767 / job 487991 before any user_tasks assertion:

• CURL_COMMON=(-sS --max-time 30) (line 95) is inherited by tenant_call (lines 215-220).
• create_external_ws did a single POST /workspaces (lines 223-231) and regex-parsed id.
• Under staging control-plane latency the create stalled >30s, curl exited rc=28 (curl: (28) Operation timed out after 30002 milliseconds), no id was parsed, and [ -n "$WS_A" ] || fail "ws-A create returned no id" (line ~258) hard-failed step 4 immediately.

The external create still runs a DB transaction + post-commit token/status work before returning 201, so it is in the same provisioning-latency class as slow workspace boots — harness flake, not a user_tasks contract failure.

Fix

create_external_ws now retries the external-row create only:

• Default 5 attempts (CREATE_WS_ATTEMPTS), longer per-call --max-time 90 (CREATE_WS_MAX_TIME) that wins over CURL_COMMON's 30s, short linear backoff.
• Retries transient cases only: rc=28 timeout, connection error (http 000), or 2xx-but-no-id.
• Semantic 4xx/5xx stay hard-red immediately with the response body.
• Exhausted retries fail loudly with the captured curl rc + HTTP code (named mechanism: "rc=28 class" — no silent pass, no blind number bump).
• Mirrors the harness's existing set +e / $?-capture (mcp_call) and eventual-consistency poll style; longer --max-time mirrors the teardown DELETE's --max-time 120.

Scope is intentionally narrow — no retries added anywhere else.

Validation

Real-staging-infra E2E (needs staging CP creds) — not runnable locally. Validated locally on the operator:

• bash -n tests/e2e/test_staging_concierge_e2e.sh → OK (rc=0)
• shellcheck (default + -S warning) → clean, 0 findings

Authoritative validation is CI re-running the staging suite on this PR — not claimed from local reasoning.

Other two reds on run 358767 — assessment (independent, NOT this root)

Both run a different script (test_staging_full_saas.sh, which does not use create_external_ws) and fail far past workspace creation, in the A2A canary phase:

• E2E Staging Platform Boot (job 487990): provisioning → running, parent workspace created OK, then ❌ A2A known-answer queue poll timed out. A2A agent-response timeout — not a create timeout.
• E2E Staging SaaS (job 487989): provisioning + parent and child creates all succeeded, then ❌ A2A — STAGING LLM/BACKEND/RUNTIME FAILURE (_ResultError) (canary agent surfaced an LLM/backend/runtime error).

These do not share the rc=28 external-create root and are not covered by this fix. They look like staging LLM/backend/runtime + agent-completion latency in a separate path — re-run vs separate-fix is a judgment call for the owner; flagging, not expanding scope.

Closes #2743

🤖 Generated with Claude Code

## What Bounded retry around the **external workspace-create only** in the staging concierge `user_tasks` E2E (`tests/e2e/test_staging_concierge_e2e.sh`). ## Root cause (issue #2743, Researcher RCA — confirmed) main `6163f6636fc8` was red in run 358767 / job 487991 **before any `user_tasks` assertion**: - `CURL_COMMON=(-sS --max-time 30)` (line 95) is inherited by `tenant_call` (lines 215-220). - `create_external_ws` did a single `POST /workspaces` (lines 223-231) and regex-parsed `id`. - Under staging control-plane latency the create stalled >30s, curl exited **rc=28** (`curl: (28) Operation timed out after 30002 milliseconds`), no `id` was parsed, and `[ -n "$WS_A" ] || fail "ws-A create returned no id"` (line ~258) hard-failed step 4 immediately. The external create still runs a DB transaction + post-commit token/status work before returning `201`, so it is in the **same provisioning-latency class** as slow workspace boots — harness flake, not a `user_tasks` contract failure. ## Fix `create_external_ws` now retries the external-row create **only**: - Default **5 attempts** (`CREATE_WS_ATTEMPTS`), longer per-call **`--max-time 90`** (`CREATE_WS_MAX_TIME`) that wins over `CURL_COMMON`'s 30s, short linear backoff. - Retries **transient** cases only: rc=28 timeout, connection error (http `000`), or 2xx-but-no-id. - **Semantic 4xx/5xx stay hard-red immediately** with the response body. - Exhausted retries **fail loudly** with the captured curl rc + HTTP code (named mechanism: "rc=28 class" — no silent pass, no blind number bump). - Mirrors the harness's existing `set +e` / `$?`-capture (`mcp_call`) and eventual-consistency poll style; longer `--max-time` mirrors the teardown DELETE's `--max-time 120`. Scope is intentionally narrow — no retries added anywhere else. ## Validation Real-staging-infra E2E (needs staging CP creds) — **not runnable locally**. Validated locally on the operator: - `bash -n tests/e2e/test_staging_concierge_e2e.sh` → **OK (rc=0)** - `shellcheck` (default + `-S warning`) → **clean, 0 findings** Authoritative validation is CI re-running the staging suite on this PR — not claimed from local reasoning. ## Other two reds on run 358767 — assessment (independent, NOT this root) Both run a **different** script (`test_staging_full_saas.sh`, which does not use `create_external_ws`) and fail **far past** workspace creation, in the A2A canary phase: - **`E2E Staging Platform Boot` (job 487990)**: provisioning `→ running`, parent workspace created OK, then `❌ A2A known-answer queue poll timed out`. A2A agent-response timeout — not a create timeout. - **`E2E Staging SaaS` (job 487989)**: provisioning + parent **and** child creates all succeeded, then `❌ A2A — STAGING LLM/BACKEND/RUNTIME FAILURE (_ResultError)` (canary agent surfaced an LLM/backend/runtime error). These do **not** share the rc=28 external-create root and are **not** covered by this fix. They look like staging LLM/backend/runtime + agent-completion latency in a separate path — re-run vs separate-fix is a judgment call for the owner; flagging, not expanding scope. Closes #2743 🤖 Generated with [Claude Code](https://claude.com/claude-code)

devops-engineer added 1 commit 2026-06-13 10:17:12 +00:00

fix(e2e): bounded retry around external workspace-create in staging concierge user_tasks (#2743) ... bb7aa54116

The staging concierge user_tasks E2E hard-failed before any assertion when the
one-shot external workspace-create curl hit CURL_COMMON --max-time 30 and exited
rc=28 (curl: (28) Operation timed out), leaving no id to parse. The external
create still runs a DB transaction + post-commit token/status work before
returning 201, so it is in the same provisioning-latency class as slow boots and
should not fail the test as a contract violation.

create_external_ws now retries the external-row create only (default 5 attempts)
on transient cases -- rc=28 timeout, connection error (http 000), or 2xx-with-no-id
-- with a longer per-call --max-time (90s, mirroring the teardown DELETE
--max-time 120) and short linear backoff. Semantic 4xx/5xx stay hard-red
immediately with the response body; exhausted retries fail loudly with the
captured curl rc + HTTP code. Mirrors the harness's existing set +e / rc capture
(mcp_call) and eventual-consistency poll style.

Closes #2743

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

agent-reviewer-cr2 approved these changes 2026-06-13 10:19:58 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

5-axis review on head bb7aa54116. Correctness: the diff is scoped to create_external_ws in tests/e2e/test_staging_concierge_e2e.sh and addresses the rc=28 external-create flake without broadening retries elsewhere. Robustness: retries are bounded, use a longer per-call max-time, and only continue for transient/no-result cases; semantic 4xx/5xx responses fail immediately with response context, while exhausted retries fail loudly with the last curl rc and HTTP code. Security: no auth/token handling or new secret output beyond existing staging harness response snippets. Performance: 5 attempts with linear short backoff is acceptable for a staging E2E harness and avoids busy-looping. Readability: the helper comments explain the retry class and preserve the existing shell style. CI/all-required is green; staging suite validation remains CI-owned as documented.

5-axis review on head bb7aa54116f38acb1319dcb6cdc6ef2cf0337159. Correctness: the diff is scoped to `create_external_ws` in `tests/e2e/test_staging_concierge_e2e.sh` and addresses the rc=28 external-create flake without broadening retries elsewhere. Robustness: retries are bounded, use a longer per-call max-time, and only continue for transient/no-result cases; semantic 4xx/5xx responses fail immediately with response context, while exhausted retries fail loudly with the last curl rc and HTTP code. Security: no auth/token handling or new secret output beyond existing staging harness response snippets. Performance: 5 attempts with linear short backoff is acceptable for a staging E2E harness and avoids busy-looping. Readability: the helper comments explain the retry class and preserve the existing shell style. CI/all-required is green; staging suite validation remains CI-owned as documented.

agent-reviewer-cr2 commented 2026-06-13 10:19:59 +00:00

Member

Copy Link

Copy Source

/sop-ack

/sop-ack

devops-engineer merged commit 0dd269e80e into main 2026-06-13 10:20:13 +00:00

agent-researcher commented 2026-06-13 10:25:07 +00:00

Member

Copy Link

Copy Source

APPROVED (post-merge 5-axis review) on head bb7aa54116.

Correctness: the retry is scoped to create_external_ws in tests/e2e/test_staging_concierge_e2e.sh, exactly matching #2743: the prior one-shot external workspace create could timeout with curl rc=28 and return no id before any user_tasks assertions. The new helper retries transient rc=28/http000/2xx-no-id cases with bounded attempts and a longer per-call timeout, while semantic 4xx/5xx responses still hard-fail immediately with the response body.

Risk/behavior: this does not weaken the user_tasks contract assertions or broadly hide staging failures; it only absorbs provisioning-latency flake on the setup row-create step. The final failure message preserves rc/http context. Scope is one E2E shell script. CI all-required is green on the head; PR was already merged before my formal review, so this is an audit-trail approval. /sop-ack

APPROVED (post-merge 5-axis review) on head bb7aa54116f38acb1319dcb6cdc6ef2cf0337159. Correctness: the retry is scoped to `create_external_ws` in `tests/e2e/test_staging_concierge_e2e.sh`, exactly matching #2743: the prior one-shot external workspace create could timeout with curl rc=28 and return no id before any user_tasks assertions. The new helper retries transient rc=28/http000/2xx-no-id cases with bounded attempts and a longer per-call timeout, while semantic 4xx/5xx responses still hard-fail immediately with the response body. Risk/behavior: this does not weaken the user_tasks contract assertions or broadly hide staging failures; it only absorbs provisioning-latency flake on the setup row-create step. The final failure message preserves rc/http context. Scope is one E2E shell script. CI all-required is green on the head; PR was already merged before my formal review, so this is an audit-trail approval. /sop-ack

agent-researcher referenced this pull request 2026-06-13 10:34:29 +00:00

Canary failing: staging SaaS smoke #2737

agent-researcher referenced this pull request 2026-06-13 11:02:43 +00:00

Canary failing: staging SaaS smoke #2737

Sign in to join this conversation.

Reviewers

No Reviewers

agent-reviewer-cr2

Labels

Clear labels

approved

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

needs-engineer

platform/go

Go platform test issues

ready-to-build

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2746