molecule-ai/molecule-core
53
0
Fork 2
Code Issues 183 Pull Requests 41 Actions Packages Projects Releases Wiki Activity

Canary failing: staging SaaS smoke #2712

New Issue
Closed
opened 2026-06-13 06:06:43 +00:00 by gitea-actions · 8 comments
gitea-actions bot commented 2026-06-13 06:06:43 +00:00
Copy Link
Copy Source

Smoke run failed at 2026-06-13T06:06:43Z.

Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357348

This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.

Smoke run failed at 2026-06-13T06:06:43Z. Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357348 This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.
gitea-actions bot commented 2026-06-13 06:35:11 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357598

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357598
agent-researcher commented 2026-06-13 06:39:29 +00:00
Member
Copy Link
Copy Source

MECHANISM: The current staging-smoke red is not the old initial queued-response bug and not the #2713 duplicate-chat regression. On main 424f725e, tests/e2e/test_staging_full_saas.sh:1276-1279 sends through a2a_send_or_poll_queue, so queued 2xx responses are polled. This run reaches a healthy platform path first: staging key present, parent workspace provisioned, terminal reachable, config.yaml PUT 200, workspace online/routable. The failure is then the agent/runtime returning text Agent error (_ResultError), which the script catches at tests/e2e/test_staging_full_saas.sh:1352-1354 as a generic error-shaped response.

EVIDENCE: main staging-smoke run 357598, job 485802, head 424f725e. Log excerpt: A2A returned an error-shaped response. Immediately before that, the log shows online and routable and config.yaml PUT OK (HTTP 200), so CP provisioning/routing and queue polling are not the break. The job also shows MOLECULE_STAGING_MINIMAX_API_KEY present for runtime=claude-code, while OpenAI/Anthropic staging keys are empty, matching the MiniMax canary path.

RECOMMENDED FIX SHAPE: Treat this as a staging LLM/backend/runtime failure class, not a harness queue bug. In molecule-core, improve the smoke failure classification around tests/e2e/test_staging_full_saas.sh:1352-1354 so _ResultError gets a named backend/runtime diagnostic and, where possible, include recent workspace/agent log tail for the failed workspace id. Operationally, inspect the claude-code MiniMax canary workspace logs for the underlying _ResultError cause (auth/base-url/model/backend empty completion) before changing platform code.

MECHANISM: The current staging-smoke red is not the old initial queued-response bug and not the #2713 duplicate-chat regression. On main `424f725e`, `tests/e2e/test_staging_full_saas.sh:1276-1279` sends through `a2a_send_or_poll_queue`, so queued 2xx responses are polled. This run reaches a healthy platform path first: staging key present, parent workspace provisioned, terminal reachable, `config.yaml` PUT 200, workspace online/routable. The failure is then the agent/runtime returning text `Agent error (_ResultError)`, which the script catches at `tests/e2e/test_staging_full_saas.sh:1352-1354` as a generic error-shaped response. EVIDENCE: main staging-smoke run 357598, job 485802, head `424f725e`. Log excerpt: `A2A returned an error-shaped response`. Immediately before that, the log shows `online and routable` and `config.yaml PUT OK (HTTP 200)`, so CP provisioning/routing and queue polling are not the break. The job also shows `MOLECULE_STAGING_MINIMAX_API_KEY` present for `runtime=claude-code`, while OpenAI/Anthropic staging keys are empty, matching the MiniMax canary path. RECOMMENDED FIX SHAPE: Treat this as a staging LLM/backend/runtime failure class, not a harness queue bug. In molecule-core, improve the smoke failure classification around `tests/e2e/test_staging_full_saas.sh:1352-1354` so `_ResultError` gets a named backend/runtime diagnostic and, where possible, include recent workspace/agent log tail for the failed workspace id. Operationally, inspect the claude-code MiniMax canary workspace logs for the underlying `_ResultError` cause (auth/base-url/model/backend empty completion) before changing platform code.
agent-researcher commented 2026-06-13 06:49:38 +00:00
Member
Copy Link
Copy Source

MECHANISM: The continuous synthetic E2E red is the same staging runtime/backend _ResultError class as the staging-smoke red, not a provisioning, queue-polling, or artifact/file-path failure. On current main 424f725e, tests/e2e/test_staging_full_saas.sh:1276-1279 routes the parent A2A through a2a_send_or_poll_queue; the script then extracts result.parts[0].text at :1279-1284 and fails at the generic error catch :1352-1354 when the agent text contains _ResultError.

EVIDENCE: continuous-synth run 357604, job 485809, head 424f725e, failed after both parent and child were fully online/routable. The log shows tenant reachable, MODEL_SLUG=MiniMax-M2.7, both workspaces image upload/download OK, terminal reachable, config.yaml PUT OK (HTTP 200), and both workspaces re-online after config write. The failing excerpt is Agent error (_ResultError). That makes this downstream of CP/routing/files/queue handling and upstream in the claude-code MiniMax agent turn.

RECOMMENDED FIX SHAPE: Keep this tracked with the staging smoke alert, but split the generic _ResultError catch into a named staging-runtime/backend diagnostic in tests/e2e/test_staging_full_saas.sh, and attach/fetch the failing workspace agent log tail before teardown where possible. The next operator investigation should inspect the claude-code MiniMax canary workspace logs for the underlying result error (auth/base URL/model/backend response), not reopen #2708/#2713.

MECHANISM: The continuous synthetic E2E red is the same staging runtime/backend `_ResultError` class as the staging-smoke red, not a provisioning, queue-polling, or artifact/file-path failure. On current main `424f725e`, `tests/e2e/test_staging_full_saas.sh:1276-1279` routes the parent A2A through `a2a_send_or_poll_queue`; the script then extracts `result.parts[0].text` at `:1279-1284` and fails at the generic error catch `:1352-1354` when the agent text contains `_ResultError`. EVIDENCE: continuous-synth run 357604, job 485809, head `424f725e`, failed after both parent and child were fully online/routable. The log shows tenant reachable, `MODEL_SLUG=MiniMax-M2.7`, both workspaces image upload/download OK, terminal reachable, `config.yaml PUT OK (HTTP 200)`, and both workspaces re-online after config write. The failing excerpt is `Agent error (_ResultError)`. That makes this downstream of CP/routing/files/queue handling and upstream in the claude-code MiniMax agent turn. RECOMMENDED FIX SHAPE: Keep this tracked with the staging smoke alert, but split the generic `_ResultError` catch into a named staging-runtime/backend diagnostic in `tests/e2e/test_staging_full_saas.sh`, and attach/fetch the failing workspace agent log tail before teardown where possible. The next operator investigation should inspect the claude-code MiniMax canary workspace logs for the underlying result error (auth/base URL/model/backend response), not reopen #2708/#2713.
gitea-actions bot commented 2026-06-13 06:54:24 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357598

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357598
gitea-actions bot commented 2026-06-13 07:05:47 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357697

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357697
gitea-actions bot commented 2026-06-13 07:35:04 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357900

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/357900
gitea-actions bot commented 2026-06-13 08:06:55 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358088

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358088
gitea-actions bot commented 2026-06-13 08:35:22 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358286

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/358286
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2712
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?