molecule-ai/molecule-core
52
0
Fork 2
Code Issues 175 Pull Requests 17 Actions 4 Packages Projects Releases Wiki Activity

workspace with failed register stays 'online' via heartbeat-backfill while canvas chat delivery starves silently #2530

New Issue
Closed
opened 2026-06-10 11:36:32 +00:00 by core-devops · 1 comment
core-devops commented 2026-06-10 11:36:32 +00:00
Member
Copy Link
Copy Source

Live (agents-team concierge, 2026-06-10): after a container re-create lost the saved workspace auth token, every boot logged Register: HTTP 401 ... proceeding (heartbeat backfill is the recovery path). The workspace stayed online (heartbeats pass), the canvas showed a green dot — but every canvas chat message sat in queued for poll forever (3 user messages, 30+ min). The user-visible symptom: agent looks healthy, never replies.

Two asks:

  1. Status truth: a workspace whose register persistently 401s should surface as degraded (same posture as runtime_wedge), not online — the canvas should hint a restart/credential repair.
  2. Queued-chat delivery: poll-mode queued messages only flushed when a turn was already active (push-channel injection); an idle agent never drained its queue even when healthy. Either the runtime needs an idle inbox poll, or the tenant-side heartbeat drain needs to handle the platform-agent's delivery mode. Related observation: heartbeat drain repeatedly logged re-queued (target still busy) even with active_tasks=0.

Recovery used live: delete stale workspace_auth_tokens rows (register then re-bootstraps) or full reprovision.

🤖 Generated with Claude Code

**Live (agents-team concierge, 2026-06-10):** after a container re-create lost the saved workspace auth token, every boot logged `Register: HTTP 401 ... proceeding (heartbeat backfill is the recovery path)`. The workspace stayed **online** (heartbeats pass), the canvas showed a green dot — but every canvas chat message sat in `queued for poll` forever (3 user messages, 30+ min). The user-visible symptom: agent looks healthy, never replies. Two asks: 1. **Status truth**: a workspace whose register persistently 401s should surface as `degraded` (same posture as runtime_wedge), not online — the canvas should hint a restart/credential repair. 2. **Queued-chat delivery**: poll-mode queued messages only flushed when a turn was already active (push-channel injection); an idle agent never drained its queue even when healthy. Either the runtime needs an idle inbox poll, or the tenant-side heartbeat drain needs to handle the platform-agent's delivery mode. Related observation: heartbeat drain repeatedly logged `re-queued (target still busy)` even with active_tasks=0. Recovery used live: delete stale `workspace_auth_tokens` rows (register then re-bootstraps) or full reprovision. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
core-devops commented 2026-06-10 14:08:48 +00:00
Author
Member
Copy Link
Copy Source

Real root cause found (supersedes the poll-cadence theory): the concierge workspace row carried delivery_mode=poll — a stale leftover from the old bespoke-era registration — and resolveDeliveryMode rule 2 ("existing row wins") perpetuates it on every re-register, so the proxy queued every canvas message forever (an idle poll-mode workspace has NO drain path). Healthy claude-code workspaces are push.

Live fix applied: UPDATE workspaces SET delivery_mode='push' for the concierge — canvas chat now dispatches normally.

Durable asks: (1) installPlatformAgent / the #2508 seed-repair should reset delivery_mode along with status/runtime/tier when adopting a pre-existing row; (2) the original ask stands: an idle poll-mode workspace with queued messages needs SOME drain path (or poll-mode rows for runtimes that are push-capable should be self-healed at register, since rule 2 currently makes stale poll sticky forever).

**Real root cause found (supersedes the poll-cadence theory):** the concierge workspace row carried `delivery_mode=poll` — a stale leftover from the old bespoke-era registration — and `resolveDeliveryMode` rule 2 ("existing row wins") perpetuates it on every re-register, so the proxy queued every canvas message forever (an idle poll-mode workspace has NO drain path). Healthy claude-code workspaces are `push`. Live fix applied: `UPDATE workspaces SET delivery_mode='push'` for the concierge — canvas chat now dispatches normally. **Durable asks:** (1) `installPlatformAgent` / the #2508 seed-repair should reset `delivery_mode` along with status/runtime/tier when adopting a pre-existing row; (2) the original ask stands: an idle poll-mode workspace with queued messages needs SOME drain path (or poll-mode rows for runtimes that are push-capable should be self-healed at register, since rule 2 currently makes stale poll sticky forever).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2530
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?