molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 15 Actions 1 Packages Projects Releases Wiki Activity

scheduler: workspace_schedules.last_status='ok' lies when adapter SDK throws internally (HTTP 200 != work done) #1696

New Issue
Closed
opened 2026-05-23 00:56:07 +00:00 by hongming · 0 comments
hongming commented 2026-05-23 00:56:07 +00:00
Owner
Copy Link
Copy Source

Problem

workspace_schedules.last_status is set to 'ok' whenever the scheduled A2A POST to the workspace returns HTTP 200. For runtimes whose adapter ALWAYS returns 200 even when the inner LLM call throws (claude-code-sdk: Exception: Claude Code returned an error result: success on Max-plan rate-limit / window saturation), this means the platform records a SUCCESS for what was actually a no-op.

Observed 2026-05-23 00:34Z – 00:50Z: pm-autonomous-tick (claude-code runtime on Max plan) fired 3 times. All 3 logged Scheduler: ... completed (HTTP 200), all 3 marked last_status='ok' in DB. ALL 3 actually surfaced Agent error (Exception) — see workspace logs for details in PM's chat tab. The schedule looked healthy in operator views; PM did zero work.

Why this matters

  • Operators can't see whether scheduled tasks are doing real work — last_status='ok' is a false-positive
  • consecutive_empty_runs only counts visibly-empty responses (per migration 032), not internal-errors-masked-as-200
  • Schedule-driven workloads on rate-limited runtimes silently no-op for hours until someone manually inspects logs

Proposed fix

Scheduler should inspect the response BODY, not just HTTP status. Adapter contract should include a top-level field like result_kind: 'ok'|'sdk_error'|'rate_limited'|'quota_exhausted' parseable by scheduler.go::scheduleFireOne. Specifically for claude-code-sdk adapter: when the SDK raises Exception: Claude Code returned an error result: success (subtype success with is_error=true), the adapter should emit result_kind: 'rate_limited' in the response body and the scheduler should map that to last_status='rate_limited' (or 'sdk_error') — NOT 'ok'.

Second-order: when a schedule hits rate_limited N times in a row, the scheduler should auto-disable + emit an activity_logs entry so it surfaces in the workspace chat (currently silent).

Workaround applied 2026-05-23 00:54Z

  • UPDATE workspace_schedules SET enabled=false WHERE name='pm-autonomous-tick' (PM no longer hit while CC Max plan window is saturated)
  • Kimi + MiniMax cron bumped to */5 * * * * (was */10 staggered) — they have independent token-plan capacity and absorb PM's scheduled workload

Related

  • reference_claude_code_prod_chat_blocked_oauth_org_not_allowed_not_image (Max plan window saturation = 429 surfacing as SDK success exception)
  • migration 032 consecutive_empty_runs counter — same family of issue, but only catches empty responses not SDK errors
## Problem `workspace_schedules.last_status` is set to `'ok'` whenever the scheduled A2A POST to the workspace returns HTTP 200. For runtimes whose adapter ALWAYS returns 200 even when the inner LLM call throws (claude-code-sdk: `Exception: Claude Code returned an error result: success` on Max-plan rate-limit / window saturation), this means the platform records a SUCCESS for what was actually a no-op. Observed 2026-05-23 00:34Z – 00:50Z: `pm-autonomous-tick` (claude-code runtime on Max plan) fired 3 times. All 3 logged `Scheduler: ... completed (HTTP 200)`, all 3 marked `last_status='ok'` in DB. ALL 3 actually surfaced `Agent error (Exception) — see workspace logs for details` in PM's chat tab. The schedule looked healthy in operator views; PM did zero work. ## Why this matters - Operators can't see whether scheduled tasks are doing real work — `last_status='ok'` is a false-positive - `consecutive_empty_runs` only counts visibly-empty responses (per migration 032), not internal-errors-masked-as-200 - Schedule-driven workloads on rate-limited runtimes silently no-op for hours until someone manually inspects logs ## Proposed fix Scheduler should inspect the response BODY, not just HTTP status. Adapter contract should include a top-level field like `result_kind: 'ok'|'sdk_error'|'rate_limited'|'quota_exhausted'` parseable by `scheduler.go::scheduleFireOne`. Specifically for claude-code-sdk adapter: when the SDK raises `Exception: Claude Code returned an error result: success` (subtype `success` with `is_error=true`), the adapter should emit `result_kind: 'rate_limited'` in the response body and the scheduler should map that to `last_status='rate_limited'` (or `'sdk_error'`) — NOT `'ok'`. Second-order: when a schedule hits `rate_limited` N times in a row, the scheduler should auto-disable + emit an activity_logs entry so it surfaces in the workspace chat (currently silent). ## Workaround applied 2026-05-23 00:54Z - `UPDATE workspace_schedules SET enabled=false WHERE name='pm-autonomous-tick'` (PM no longer hit while CC Max plan window is saturated) - Kimi + MiniMax cron bumped to `*/5 * * * *` (was `*/10` staggered) — they have independent token-plan capacity and absorb PM's scheduled workload ## Related - `reference_claude_code_prod_chat_blocked_oauth_org_not_allowed_not_image` (Max plan window saturation = 429 surfacing as SDK `success` exception) - migration 032 `consecutive_empty_runs` counter — same family of issue, but only catches empty responses not SDK errors
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1696
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?