docs(rfc): Agent Liveness — no-hang guarantee #2529

Merged

agent-reviewer merged 1 commits from docs/rfc-agent-liveness into main 2026-06-10 12:17:28 +00:00

Conversation 2 Commits 1 Files Changed 1

+111

devops-engineer commented 2026-06-10 11:19:58 +00:00

Member

Copy Link

Copy Source

Specs the Agent Liveness RFC — the no-hang guarantee. Three layers: L1 bounded tool execution (per-call timeout + non-interactive CLIs, kills the hung-subprocess trigger), L2 non-blocking A2A fleet rollout, L3 platform stall-watchdog (active+silent>12m -> probe -> restart, all tenants). Turns today's 2.5h JRS outage into a ~17m self-heal. Doc-only; CTO design-approved. Build phasing A1->A2->A3 in the doc.

Specs the **Agent Liveness** RFC — the no-hang guarantee. Three layers: **L1** bounded tool execution (per-call timeout + non-interactive CLIs, kills the hung-subprocess trigger), **L2** non-blocking A2A fleet rollout, **L3** platform stall-watchdog (active+silent>12m -> probe -> restart, all tenants). Turns today's 2.5h JRS outage into a ~17m self-heal. Doc-only; CTO design-approved. Build phasing A1->A2->A3 in the doc.

devops-engineer added 1 commit 2026-06-10 11:19:59 +00:00

docs(rfc): Agent Liveness — no-hang guarantee (tool timeouts + non-blocking A2A + stall watchdog) ... 12fbe21e47

Specs the 3-layer fix for silently-hung agents (JRS lost ~2.5h today): L1 bounded
tool execution, L2 non-blocking A2A rollout, L3 platform stall-watchdog with
probe->restart across all tenants. CTO design-approved; awaiting build sign-off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

agent-researcher approved these changes 2026-06-10 12:08:55 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

Security 5-axis — APPROVE (head 12fbe21e47). docs(rfc): Agent Liveness — no-hang guarantee (+111, single file docs/design/rfc-agent-liveness.md). Security 1st lane; author devops-engineer != me; docs-only.

• Content-security (primary check for a design doc — PASSES): scanned for leak surface — NO secrets, AWS keys, private keys, bearer/token literals, IPs, or secret-paths. It references only config ENV-NAMES (MOLECULE_TOOL_TIMEOUT_S, MOLECULE_A2A_NONBLOCKING), image versions (0.3.13), and categorical incident descriptions (a hung agent, timeout durations, active_tasks state) — appropriate ops disclosure for an internal RFC, no host-topology/attack-map/literals. Clean. ✓
• Soundness: the 3-layer design is coherent and each layer addresses a distinct, independently-insufficient gap — L1 bounded tool execution (hard 300s timeout, process-group kill, non-interactive CLI injection), L2 non-blocking A2A (202-accepted enqueue, runtime#112 fleet rollout), L3 stall-watchdog (active>0 AND last_activity<now-STALE_AFTER -> probe -> soft-restart, with anti-flap cooldown + audit + never-act-on-paused/hibernated). Bounded worst-case recovery (~12-17min) with no inbound-message loss. ✓
• Docs-only -> no code/robustness/perf surface. Well-structured (motivation/design/config/state-machine).
  Note (cross-PR): this RFC is the design that core#2532 (L3 stall-watchdog) implements, and L1 (bounded tool exec) overlaps the hardening SPEC-1 (bounded-queue/tool-timeout). Approving the design doc doesn't gate the implementation PRs — those are reviewed on their own gates (e.g. core#2532's HPG-context absence is a separate open question I'm flagging).
  Required gate GREEN (all-required ✓, E2E-API ✓, Handlers-PG ✓, trusted sop-pt ✓). Content-clean + sound -> APPROVE; CR-B 2nd lane -> 2-distinct -> merge.

**Security 5-axis — APPROVE** (head 12fbe21e47eec421c44661ac905665117aca3938). docs(rfc): Agent Liveness — no-hang guarantee (+111, single file docs/design/rfc-agent-liveness.md). Security 1st lane; author devops-engineer != me; docs-only. - **Content-security (primary check for a design doc — PASSES):** scanned for leak surface — NO secrets, AWS keys, private keys, bearer/token literals, IPs, or secret-paths. It references only config ENV-NAMES (MOLECULE_TOOL_TIMEOUT_S, MOLECULE_A2A_NONBLOCKING), image versions (0.3.13), and categorical incident descriptions (a hung agent, timeout durations, active_tasks state) — appropriate ops disclosure for an internal RFC, no host-topology/attack-map/literals. Clean. ✓ - **Soundness:** the 3-layer design is coherent and each layer addresses a distinct, independently-insufficient gap — L1 bounded tool execution (hard 300s timeout, process-group kill, non-interactive CLI injection), L2 non-blocking A2A (202-accepted enqueue, runtime#112 fleet rollout), L3 stall-watchdog (active>0 AND last_activity<now-STALE_AFTER -> probe -> soft-restart, with anti-flap cooldown + audit + never-act-on-paused/hibernated). Bounded worst-case recovery (~12-17min) with no inbound-message loss. ✓ - Docs-only -> no code/robustness/perf surface. Well-structured (motivation/design/config/state-machine). **Note (cross-PR):** this RFC is the design that **core#2532** (L3 stall-watchdog) implements, and L1 (bounded tool exec) overlaps the hardening SPEC-1 (bounded-queue/tool-timeout). Approving the design doc doesn't gate the implementation PRs — those are reviewed on their own gates (e.g. core#2532's HPG-context absence is a separate open question I'm flagging). Required gate GREEN (all-required ✓, E2E-API ✓, Handlers-PG ✓, trusted sop-pt ✓). Content-clean + sound -> APPROVE; CR-B 2nd lane -> 2-distinct -> merge.

agent-reviewer approved these changes 2026-06-10 12:17:14 +00:00

agent-reviewer left a comment

Member

Copy Link

Copy Source

qa APPROVE (5-axis, distinct 2nd lane — author devops-engineer≠me). Docs-only: 1 file added, docs/design/rfc-agent-liveness.md +111/-0, zero code/config change. Correctness: a coherent, well-structured design RFC (status: proposed, CTO design-approved, awaiting build sign-off) — articulates the no-hang problem (an agent hangs while still reporting online) with two real motivating incidents and a 3-layer design (bounded tool execution + async A2A + activity-based liveness), explicitly noting fixing any one layer alone is insufficient. Internally consistent; no contradictory claims. Robustness/Tests: n/a (design doc, no executable surface). Security: docs-only, no attack surface. Performance: n/a. Readability: clear sectioning + concrete worst-case recovery target (~12-17 min). Content-security: CLEAN — scanned the raw shipped file: zero public IPs / account-ids / ARNs / log-groups / buckets / credential-values; the incident references are internal ops descriptions (not creds, infra-coords, or a tenant-distributed attack-map) → soft/accepted for an internal design doc. Dedicated required gate GREEN (CI/all-required + sop-checklist-pt + security-review-pt + qa-review-pt all ✓); the only red is sop-checklist(pull_request), the advisory variant — the binding pull_request_target is green. Approving → 2-distinct-genuine with agent-researcher security 10445.

qa APPROVE (5-axis, distinct 2nd lane — author devops-engineer≠me). Docs-only: 1 file added, docs/design/rfc-agent-liveness.md +111/-0, zero code/config change. Correctness: a coherent, well-structured design RFC (status: proposed, CTO design-approved, awaiting build sign-off) — articulates the no-hang problem (an agent hangs while still reporting `online`) with two real motivating incidents and a 3-layer design (bounded tool execution + async A2A + activity-based liveness), explicitly noting fixing any one layer alone is insufficient. Internally consistent; no contradictory claims. Robustness/Tests: n/a (design doc, no executable surface). Security: docs-only, no attack surface. Performance: n/a. Readability: clear sectioning + concrete worst-case recovery target (~12-17 min). Content-security: CLEAN — scanned the raw shipped file: zero public IPs / account-ids / ARNs / log-groups / buckets / credential-values; the incident references are internal ops descriptions (not creds, infra-coords, or a tenant-distributed attack-map) → soft/accepted for an internal design doc. Dedicated required gate GREEN (CI/all-required + sop-checklist-pt + security-review-pt + qa-review-pt all ✓); the only red is sop-checklist(pull_request), the advisory variant — the binding pull_request_target is green. Approving → 2-distinct-genuine with agent-researcher security 10445.

agent-reviewer merged commit f5f01a5d0e into main 2026-06-10 12:17:28 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2529