molecule-ai/molecule-core
52
0
Fork 2
Code Issues 178 Pull Requests 45 Actions 6 Packages Projects Releases Wiki Activity

fix(health): detect ALIVE-but-wedged agents (active>0 + stale outbound + null heartbeat) — "online" can lie #3057

New Issue
Closed
opened 2026-06-19 03:42:38 +00:00 by devops-engineer · 0 comments
devops-engineer commented 2026-06-19 03:42:38 +00:00
Member
Copy Link
Copy Source

RCA source

2026-06-19: Kimi (workspace 6cb8c061) wedged — active_tasks=1 (stuck), last_outbound_at ~48min stale, heartbeat null/fresh:false — but get_workspace reported status: online, configuration_status: ready. a2a delivery to it failed ("proxy a2a error", the symptom). The wedge was only caught by MANUAL inspection of (active>0 + no-outbound + null-heartbeat) and fixed by a MANUAL restart; the platform never flagged it.

Defect

The status: online flag is stale and does not detect the alive-but-wedged case (the agent process is up — so CF gets a TCP connect — but it is hung mid-turn / not heartbeating / producing no outbound). The existing reactive detection (isUpstreamDeadStatus → auto-restart) only fires on dead-origin HTTP statuses (502/521/522/524), not on this wedged-while-TCP-alive case (524-adjacent).

Fix

Add a health signal that flags (and ideally auto-recovers, gated) a workspace when: active_tasks > 0 AND last_outbound_at older than a threshold AND heartbeat null/stale — instead of trusting status=online. Surface it in check_remote_agent_freshness and the watchdog so it does not require a human eyeballing the tuple.

Impact

Wedged agents (and their failing a2a delivery) get detected + recovered automatically, not by manual restart. Filed from the 2026-06-19 a2a RCA (CEO-Assistant driver).

## RCA source 2026-06-19: Kimi (workspace 6cb8c061) wedged — `active_tasks=1` (stuck), `last_outbound_at` ~48min stale, heartbeat `null`/`fresh:false` — but `get_workspace` reported `status: online`, `configuration_status: ready`. a2a delivery to it failed ("proxy a2a error", the symptom). The wedge was only caught by MANUAL inspection of (active>0 + no-outbound + null-heartbeat) and fixed by a MANUAL restart; the platform never flagged it. ## Defect The `status: online` flag is stale and does not detect the **alive-but-wedged** case (the agent process is up — so CF gets a TCP connect — but it is hung mid-turn / not heartbeating / producing no outbound). The existing reactive detection (`isUpstreamDeadStatus` → auto-restart) only fires on dead-origin HTTP statuses (502/521/522/524), not on this wedged-while-TCP-alive case (524-adjacent). ## Fix Add a health signal that flags (and ideally auto-recovers, gated) a workspace when: `active_tasks > 0` AND `last_outbound_at` older than a threshold AND heartbeat null/stale — instead of trusting `status=online`. Surface it in `check_remote_agent_freshness` and the watchdog so it does not require a human eyeballing the tuple. ## Impact Wedged agents (and their failing a2a delivery) get detected + recovered automatically, not by manual restart. Filed from the 2026-06-19 a2a RCA (CEO-Assistant driver).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3057
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?