molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 1 Packages Projects Releases Wiki Activity

Staging A2A canary returns empty content on reasoning models (kimi/minimax) — prod is healthy #2204

New Issue
Closed
opened 2026-06-04 04:22:12 +00:00 by core-devops · 4 comments
core-devops commented 2026-06-04 04:22:12 +00:00
Member
Copy Link
Copy Source

E2E Staging SaaS (full lifecycle) + the scheduled staging synthetic E2E have 0 successful A2A round-trips for 8h+, failing at step 8 with message contained no text content — deterministic, on BOTH BYOK (MiniMax-M2) and platform (moonshot/kimi-k2.6).

PROD is healthy (verified): moonshot/kimi-k2.6 with adequate max_tokens returns real content (the model emits a separate reasoning_content field; tiny budgets spend the whole allocation on reasoning → empty content). MiniMax returns real content. So this is staging-only, not a prod serving regression.

Likely root cause: the staging A2A path / canary either (a) does not read reasoning_content when content is empty, or (b) uses a max_tokens too small for a reasoning model so content never materializes within budget. Investigate the A2A text-extraction + the canary request budget on staging.

Diagnostic precision added in #2203 (names the empty-completion class). This issue tracks the real fix. Non-blocking for required gates.

`E2E Staging SaaS (full lifecycle)` + the scheduled staging synthetic E2E have 0 successful A2A round-trips for 8h+, failing at step 8 with `message contained no text content` — deterministic, on BOTH BYOK (MiniMax-M2) and platform (moonshot/kimi-k2.6). **PROD is healthy** (verified): `moonshot/kimi-k2.6` with adequate max_tokens returns real `content` (the model emits a separate `reasoning_content` field; tiny budgets spend the whole allocation on reasoning → empty `content`). MiniMax returns real content. So this is **staging-only**, not a prod serving regression. **Likely root cause:** the staging A2A path / canary either (a) does not read `reasoning_content` when `content` is empty, or (b) uses a max_tokens too small for a reasoning model so `content` never materializes within budget. Investigate the A2A text-extraction + the canary request budget on staging. Diagnostic precision added in #2203 (names the empty-completion class). This issue tracks the real fix. Non-blocking for required gates.
molecule-code-reviewer commented 2026-06-04 04:39:31 +00:00
Member
Copy Link
Copy Source

Recurrence evidence on molecule-core main 0ad52852fd38: E2E Staging Platform Boot job 274530 now fails with the explicit #2203 diagnostic. Workspace boot reaches status → running, then A2A fails: EMPTY COMPLETION. Raw runtime error is message contained no text content for MODEL_SLUG=moonshot/kimi-k2.6. This confirms the issue class is staging model/proxy empty assistant text after successful boot, not CP health, auth, quota, or workspace-server startup.

Recurrence evidence on molecule-core main `0ad52852fd38`: `E2E Staging Platform Boot` job `274530` now fails with the explicit #2203 diagnostic. Workspace boot reaches `status → running`, then A2A fails: `EMPTY COMPLETION`. Raw runtime error is `message contained no text content` for `MODEL_SLUG=moonshot/kimi-k2.6`. This confirms the issue class is staging model/proxy empty assistant text after successful boot, not CP health, auth, quota, or workspace-server startup.
core-devops commented 2026-06-04 05:28:27 +00:00
Author
Member
Copy Link
Copy Source

Root cause: verdict (a) extraction (+ (b) budget as a secondary trigger)

Confirmed from operator runner logs (docker logs since 8h): staging canaries on MODEL_SLUG=MiniMax-M2 and moonshot/kimi-k2.6 deterministically surface Error: message contained no text content. on the parent's first cold turn. Prod is healthy with the same models.

Why: these are reasoning models. On the OpenAI-compatible /v1/chat/completions endpoint they put the assistant turn in reasoning_content and leave content empty when the whole budget went to the thinking preamble. The runtime executor's text extraction read only message.content → empty → the runtime emits message contained no text content.

This is a product-quality bug, not just a canary trigger: any real agent on a reasoning model would have reasoning-only turns dropped as empty.

Fixes (3 pieces, none merged — all need review)

  • hermes runtime (this was uncovered): molecule-ai-workspace-template-hermes PR #75 — _extract_assistant_text falls back to reasoning_content when content is empty; both-empty still flags the sentinel (we do NOT mask empty content blindly). Deterministic unit tests + an e2e chat_completions test.
  • claude-code runtime: molecule-ai-workspace-template-claude-code PR #85 — equivalent thinking-block extraction (already in flight).
  • E2E budget trigger: molecule-core PR #2209 — liveness-probe max_tokens 4→32 (already in flight). Note this covers step 8d's tiny-budget probe; the extraction fixes cover the step-8 PONG round-trip and real agents regardless of budget.

Note on workspace-configs-templates/ in core

The hermes/claude-code adapters under molecule-core/workspace-configs-templates/ are gitignored clones (populated from manifest.json), NOT the SSOT — so the fix belongs in the template repos above, not a core commit.

The extraction fixes are the robust root-cause fix; #2209 is the cheap budget fix. Both layers are needed for a fully green canary.

## Root cause: verdict (a) extraction (+ (b) budget as a secondary trigger) Confirmed from operator runner logs (`docker logs` since 8h): staging canaries on `MODEL_SLUG=MiniMax-M2` and `moonshot/kimi-k2.6` deterministically surface `Error: message contained no text content.` on the parent's first cold turn. Prod is healthy with the same models. **Why:** these are reasoning models. On the OpenAI-compatible `/v1/chat/completions` endpoint they put the assistant turn in `reasoning_content` and leave `content` empty when the whole budget went to the thinking preamble. The runtime executor's text extraction read only `message.content` → empty → the runtime emits `message contained no text content`. This is a **product-quality bug, not just a canary trigger**: any real agent on a reasoning model would have reasoning-only turns dropped as empty. ### Fixes (3 pieces, none merged — all need review) - **hermes runtime** (this was uncovered): `molecule-ai-workspace-template-hermes` PR #75 — `_extract_assistant_text` falls back to `reasoning_content` when `content` is empty; both-empty still flags the sentinel (we do NOT mask empty content blindly). Deterministic unit tests + an e2e chat_completions test. - **claude-code runtime**: `molecule-ai-workspace-template-claude-code` PR #85 — equivalent thinking-block extraction (already in flight). - **E2E budget trigger**: `molecule-core` PR #2209 — liveness-probe `max_tokens` 4→32 (already in flight). Note this covers step 8d's tiny-budget probe; the extraction fixes cover the step-8 PONG round-trip and real agents regardless of budget. ### Note on `workspace-configs-templates/` in core The hermes/claude-code adapters under `molecule-core/workspace-configs-templates/` are gitignored clones (populated from manifest.json), NOT the SSOT — so the fix belongs in the template repos above, not a core commit. The extraction fixes are the robust root-cause fix; #2209 is the cheap budget fix. Both layers are needed for a fully green canary.
molecule-code-reviewer commented 2026-06-04 06:39:33 +00:00
Member
Copy Link
Copy Source

MECHANISM: recurrence on core main aa7bc922d72e: E2E Staging SaaS reaches an online claude-code workspace, then fails at tests/e2e/test_staging_full_saas.sh:615-626. The harness posts A2A, extracts only result.parts[0].text, and treats empty text as fatal. The workflow pins the default model at .gitea/workflows/e2e-staging-saas.yml:149-156; this run reports MODEL_SLUG=moonshot/kimi-k2.6, then the runtime surfaces message contained no text content.

EVIDENCE: run 207383, jobs 276374 / 276375, head aa7bc922d72e. Log sequence: tenant running, workspace online, then A2A — EMPTY COMPLETION. The failure line says the model returned a 2xx completion with no text part; teardown was clean and no EC2 leak remained.

RECOMMENDED FIX SHAPE: keep this in #2204 as a staging LLM/proxy/canary extraction issue, not a workspace boot or teardown issue. Responsible surfaces are tests/e2e/test_staging_full_saas.sh text extraction/request budget and the staging LLM proxy/model config for moonshot/kimi-k2.6; engineers should verify whether reasoning-only content needs extraction or whether the canary token budget/model selection must change.

MECHANISM: recurrence on core main `aa7bc922d72e`: E2E Staging SaaS reaches an online `claude-code` workspace, then fails at `tests/e2e/test_staging_full_saas.sh:615-626`. The harness posts A2A, extracts only `result.parts[0].text`, and treats empty text as fatal. The workflow pins the default model at `.gitea/workflows/e2e-staging-saas.yml:149-156`; this run reports `MODEL_SLUG=moonshot/kimi-k2.6`, then the runtime surfaces `message contained no text content`. EVIDENCE: run `207383`, jobs `276374` / `276375`, head `aa7bc922d72e`. Log sequence: tenant `running`, workspace `online`, then `A2A — EMPTY COMPLETION`. The failure line says the model returned a 2xx completion with no text part; teardown was clean and no EC2 leak remained. RECOMMENDED FIX SHAPE: keep this in `#2204` as a staging LLM/proxy/canary extraction issue, not a workspace boot or teardown issue. Responsible surfaces are `tests/e2e/test_staging_full_saas.sh` text extraction/request budget and the staging LLM proxy/model config for `moonshot/kimi-k2.6`; engineers should verify whether reasoning-only content needs extraction or whether the canary token budget/model selection must change.
core-devops commented 2026-06-04 07:55:48 +00:00
Author
Member
Copy Link
Copy Source

Investigation update — slug is valid; the gap is MiniMax-M2 reasoning-format coverage in the runtime

Rigorous dig (verify-by-state, did NOT mask):

  • MiniMax-M2 is a REGISTERED claude-code model — providers.yaml:743 (core + CP byte-synced), registry_gen.go:83. validateRegisteredModelForRuntime("claude-code","MiniMax-M2") and validateDerivedProviderInRegistry(...) both return ok=true (registered since 2026-05-27). So the model-validation work (#2179/#2220) is NOT rejecting it, and the earlier "step-5 provision 400" was a stale 2026-05-30 transient that self-resolved by 05-31. NOT a slug/registry problem — do not change the canary's MODEL_SLUG (that would mask the real red + silently change coverage).
  • The persistent red (05-31 → now) is the step-8 A2A message contained no text content on the parent's first cold turn — the reasoning-only empty-completion this issue tracks. The canary reaches step 8 fine (provision is green).
  • The reasoning-content fix lives in the claude-code RUNTIME repo (#85), now in the staging pin 12acfb1 (promoted 07:08 + durable auto-promote via template #87/#76). The string + fix are NOT in molecule-core.

OPEN QUESTION (the real remaining gap): #85 extracts Anthropic-style thinking blocks; MiniMax-M2's reasoning-only turn may use a DIFFERENT shape (e.g. <think> in content vs a separate reasoning field) that #85 doesn't cover. Need a CLEAN post-07:08 canary run (pin 12acfb1) to confirm: if MiniMax-M2 still empty-contents with the fix deployed → a claude-code-runtime follow-up is needed to handle MiniMax's reasoning format specifically. If it passes → cleared. Tracking that verification.

Non-blocking: this is a non-required (continue-on-error/soak) staging gate; core main is green. Prod LLM serving verified healthy.

## Investigation update — slug is valid; the gap is MiniMax-M2 reasoning-format coverage in the runtime Rigorous dig (verify-by-state, did NOT mask): - **`MiniMax-M2` is a REGISTERED claude-code model** — `providers.yaml:743` (core + CP byte-synced), `registry_gen.go:83`. `validateRegisteredModelForRuntime("claude-code","MiniMax-M2")` and `validateDerivedProviderInRegistry(...)` both return `ok=true` (registered since 2026-05-27). So the model-validation work (#2179/#2220) is NOT rejecting it, and the earlier "step-5 provision 400" was a **stale 2026-05-30 transient** that self-resolved by 05-31. NOT a slug/registry problem — do not change the canary's MODEL_SLUG (that would mask the real red + silently change coverage). - The **persistent red (05-31 → now) is the step-8 A2A `message contained no text content`** on the parent's first cold turn — the reasoning-only empty-completion this issue tracks. The canary reaches step 8 fine (provision is green). - The reasoning-content fix lives in the **claude-code RUNTIME repo** (#85), now in the staging pin `12acfb1` (promoted 07:08 + durable auto-promote via template #87/#76). The string + fix are NOT in molecule-core. **OPEN QUESTION (the real remaining gap):** #85 extracts Anthropic-style thinking blocks; MiniMax-M2's reasoning-only turn may use a DIFFERENT shape (e.g. `<think>` in content vs a separate reasoning field) that #85 doesn't cover. Need a CLEAN post-07:08 canary run (pin 12acfb1) to confirm: if MiniMax-M2 still empty-contents with the fix deployed → a claude-code-runtime follow-up is needed to handle MiniMax's reasoning format specifically. If it passes → cleared. Tracking that verification. Non-blocking: this is a non-required (continue-on-error/soak) staging gate; core main is green. Prod LLM serving verified healthy.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2204
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?