molecule-ai/molecule-core
35
0
Fork 2
Code Issues 16 Pull Requests 2 Actions 49 Packages Projects Releases Wiki Activity

Merge pull request #2408 from Molecule-AI/auto/fix-healthsweep-gate-saas

Browse Source 
fix(boot): always start health-sweep goroutine — SaaS tenants need it for external-runtime liveness
This commit is contained in:
Hongming Wang 2026-04-30 20:12:28 +00:00 committed by GitHub
commit b5c7b349d8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 18 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
workspace-server/cmd/server/main.go
View File

@ -223,13 +223,24 @@ func main() {
registry.StartLivenessMonitor(c, onWorkspaceOffline)
})
// Proactive container health sweep — detects dead containers faster than Redis TTL.
// Checks all "online" workspaces against Docker every 15 seconds.
if prov != nil {
go supervised.RunWithRecover(ctx, "health-sweep", func(c context.Context) {
registry.StartHealthSweep(c, prov, 15*time.Second, onWorkspaceOffline)
})
}
// Proactive health sweep — two passes per tick:
// 1. Docker-side: checks "online" workspaces against the local Docker
// daemon (only runs when prov is non-nil, i.e. self-hosted mode).
// 2. Remote-side: scans runtime='external' rows whose last_heartbeat_at
// is past REMOTE_LIVENESS_STALE_AFTER and flips them to
// awaiting_agent. Runs regardless of provisioner mode — SaaS
// tenants need this even though they don't run Docker locally,
// because external-runtime workspaces are operator-managed and
// the platform-side liveness sweep is the only thing that
// transitions them off 'online' when the operator's CLI dies.
//
// Pre-2026-04-30 this goroutine was gated on prov != nil, which silently
// disabled the remote-side sweep on every SaaS tenant. The function in
// healthsweep.go has always handled nil checker correctly; only the
// orchestration was wrong. See #2392's CI failure for the trace.
go supervised.RunWithRecover(ctx, "health-sweep", func(c context.Context) {
registry.StartHealthSweep(c, prov, 15*time.Second, onWorkspaceOffline)
})
// Orphan-container reconcile sweep — finds running containers
// whose workspace row is already status='removed' and stops