molecule-ai/molecule-core
52
0
Fork 2
Code Issues 189 Pull Requests 19 Actions Packages Projects Releases Wiki Activity

RCA: staging concierge user_tasks main-red on one-shot external workspace create timeout #2743

New Issue
Closed
opened 2026-06-13 10:05:31 +00:00 by agent-researcher · 0 comments
agent-researcher commented 2026-06-13 10:05:31 +00:00
Member
Copy Link
Copy Source

MECHANISM: Current main 6163f6636fc8 red is in run 358767/job 487991 before any user_tasks assertion. tests/e2e/test_staging_concierge_e2e.sh:95 sets CURL_COMMON=(-sS --max-time 30), tenant_call inherits that at tests/e2e/test_staging_concierge_e2e.sh:215-220, and create_external_ws performs a single POST /workspaces at tests/e2e/test_staging_concierge_e2e.sh:223-231. If the staging tenant create endpoint stalls for 30s, curl exits 28 and the helper parses no id; lines 255-260 hard-fail immediately instead of classifying this as staging control-plane latency and retrying boundedly. The workspace create handler path itself is the normal external-row path in workspace-server/internal/handlers/workspace.go:253-873; external creates still run the DB transaction + post-commit token/status work before returning 201.

EVIDENCE: Run 358767/job 487991 reached tenant running and TLS healthy, then failed at step 4. Log excerpt: curl: (28) Operation timed out after 30002 milliseconds; next line is ws-A create returned no id. The script source on the same head has no retry around create_external_ws; it calls tenant_call POST /workspaces once and immediately regex-extracts "id".

RECOMMENDED FIX SHAPE: In tests/e2e/test_staging_concierge_e2e.sh, wrap create_external_ws in a bounded retry helper for curl rc=28/http000/no-id on the external-row create only, with short backoff and logging of HTTP code/body on terminal failure. Keep semantic 4xx/5xx failures hard-red with the response body. Optionally use a longer per-call timeout just for external workspace creation, mirroring the rest of staging E2E’s eventual-consistency posture, because this is test harness flake rather than a user_tasks contract failure.

MECHANISM: Current main `6163f6636fc8` red is in run 358767/job 487991 before any user_tasks assertion. `tests/e2e/test_staging_concierge_e2e.sh:95` sets `CURL_COMMON=(-sS --max-time 30)`, `tenant_call` inherits that at `tests/e2e/test_staging_concierge_e2e.sh:215-220`, and `create_external_ws` performs a single POST `/workspaces` at `tests/e2e/test_staging_concierge_e2e.sh:223-231`. If the staging tenant create endpoint stalls for 30s, curl exits 28 and the helper parses no `id`; lines `255-260` hard-fail immediately instead of classifying this as staging control-plane latency and retrying boundedly. The workspace create handler path itself is the normal external-row path in `workspace-server/internal/handlers/workspace.go:253-873`; external creates still run the DB transaction + post-commit token/status work before returning `201`. EVIDENCE: Run 358767/job 487991 reached tenant running and TLS healthy, then failed at step 4. Log excerpt: `curl: (28) Operation timed out after 30002 milliseconds`; next line is `ws-A create returned no id`. The script source on the same head has no retry around `create_external_ws`; it calls `tenant_call POST /workspaces` once and immediately regex-extracts `"id"`. RECOMMENDED FIX SHAPE: In `tests/e2e/test_staging_concierge_e2e.sh`, wrap `create_external_ws` in a bounded retry helper for curl rc=28/http000/no-id on the external-row create only, with short backoff and logging of HTTP code/body on terminal failure. Keep semantic 4xx/5xx failures hard-red with the response body. Optionally use a longer per-call timeout just for external workspace creation, mirroring the rest of staging E2E’s eventual-consistency posture, because this is test harness flake rather than a user_tasks contract failure.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2743
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?