fix(secrets): log rows.Scan error in restartAllAffectedByGlobalKey #2079

Closed
core-be wants to merge 40 commits from fix/secrets-scan-error-restart into main
Member
No description provided.
core-be added 40 commits 2026-06-01 04:59:15 +00:00
Sibling of #1347/internal#470 — the POLL-mode arm of the canvas
user-message data-loss bug Hongming reported ("i sometimes lose my own
message when i exit chat", 2026-05-16).

Hongming's tenant is entirely poll-mode (4 external workspaces, no URL —
verified empirically: every workspace returns the {delivery_mode:poll,
status:queued} short-circuit envelope), so #1347 (push-mode only,
persists AFTER the poll short-circuit) structurally cannot cover his
reported case. #1347's "poll-mode was never affected" framing is
overstated: logA2AReceiveQueued's durable activity_logs INSERT ran
inside h.goAsync(...) — a detached goroutine with no happens-before
barrier against the synthetic {status:queued} 200. The canvas sees the
send acknowledged while the row may still be racing; a workspace-server
restart / deploy / OOM / EC2 hibernation between the 200 and the
goroutine's commit loses the message permanently (chat-history reads
activity_logs; missing row = message gone on reopen). No fallback
either, unlike push-mode's legacy-INSERT path.

Fix: make the poll-mode ingest persist SYNCHRONOUS — committed before
the queued 200 — on a context.WithoutCancel context (parity with
persistUserMessageAtIngest). Best-effort preserved (LogActivity
logs+swallows INSERT errors, never blocks the send). Post-commit
broadcast still fires inside LogActivity (a missed WS event is not data
loss; the durable row is the truth chat-history re-reads on reopen).

TDD: a2a_poll_ingest_persist_test.go — deterministic RED (queued 200
returned in ~0.5ms, before the 150ms INSERT → DATA LOSS) → GREEN after
fix. Full internal/handlers + internal/messagestore suites green; vet
clean.

Refs: molecule-ai/internal#471 (tracking), molecule-ai/internal#470 (push-mode sibling, PR #1347)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sqlmock.ExpectationsWereMet() hangs indefinitely when the expected INSERT
mock never fires. If the production code ever regresses to goAsync
(pre-fix shape), the handler returns before the INSERT fires, the mock
never fires, and ExpectationsWereMet() blocks for the full test/-suite
timeout — wedging the CI run with no diagnostic.

Fix: check expectations in a goroutine with a 2s hard timeout. When
the mock has fired (synchronous production code), ExpectationsWereMet()
returns <1ms and the select fires the `case err := <-expectDone` arm.
When the mock has NOT fired (async regression), the 2s timeout fires and
the test fails with a clear message instead of hanging.

Also reduce insertDelay from 150ms → 50ms. 50ms is ~50× the normal INSERT
latency and sufficient to prove synchronous blocking; the larger value
was adding unnecessary suite-level wall-clock under -race detection,
where mock delays are amplified by the instrumenter's goroutine overhead.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
internal#212 (P0 from internal#211). When the embedded `claude` CLI emits a
terminal result message with is_error=true (e.g. 403 oauth_org_not_allowed
"Your organization has disabled Claude subscription access · Use an
Anthropic API key instead, or ask your admin to enable access"), the user
saw only `Agent error (Exception) — see workspace logs for details.` — a
dead end (no such logs UI) that discards the exact secret-safe, actionable
text the user needs.

Root cause was a multi-cut loss of the CLI's result/error/api_error_status:

  cut #2  sanitize_agent_error reduced every failure to type(exc).__name__.
          → add a `reason` passthrough: a pre-curated, user-actionable,
            secret-safe explanation is surfaced verbatim (still scrubbed for
            key/token/bearer as a second pass). reason wins over stderr;
            omitting it preserves the prior generic behavior exactly.

  cut #3a workspace-server dropped error_detail from the live
          ACTIVITY_LOGGED websocket broadcast (it was persisted to the DB
          column but never sent), so the canvas had nothing to render.
          → include error_detail in the broadcast payload (already capped
            at 4096 by the runtime's report_activity helper).

  cut #3b canvas useChatSocket hardcoded the opaque string, ignoring even
          the activity summary.
          → render error_detail (fallback: summary, then a generic retry
            hint). The dead "see workspace logs for details." phrase that
            pointed at nonexistent UI is removed (a full logs tab is a
            separate larger follow-up, not this PR — reason-first per CTO).

The runtime-side cut #1 (template-claude-code claude_sdk_executor._run_query
ignoring is_error and the SDK collapsing errors[] to the bare subtype
"success") is fixed in a stacked PR on
molecule-ai-workspace-template-claude-code (depends on this PR's
sanitize_agent_error `reason` kwarg, which ships via the
molecule-ai-workspace-runtime package).

Tests: 4 new sanitize_agent_error reason tests (verbatim surfacing, secret
scrub still applied, reason>stderr precedence, no-reason unchanged). Verified
fail-before / pass-after; full sanitize suite green; no new regressions (the
2 pre-existing test_get_a2a_instructions_mcp failures are unrelated).

Refs: internal#211, internal#212

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses internal#212 PR#1420 dual-review SECURITY finding (infra-sre /
infra-runtime-be): _sanitize_for_external missed three real credential
shapes because the legacy regex requires a `[ :=]+` separator after the
prefix:
- bare `sk-ant-api03-…` keys (real key uses `-`, not `[ :=]`)
- JSON-quoted "token"/"apiKey"/"secret"/"password" values
- `aws_secret_access_key=…`

Added three narrowly-scoped regexes (length thresholds tuned so curated
short examples like `sk-ant-EXAMPLE-SHORT` / `ghp_SHORT_TOKEN` and all
actionable auth/quota/HTTP guidance still pass through). Extended the unit
test with test_sanitize_agent_error_reason_scrubs_all_secret_formats
asserting redaction for all three new formats plus the original Bearer
regression. Full sanitize suite green; existing passthrough assertions
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two CI failures blocking PR #1420:
1. Secret scan: `workspace/tests/test_executor_helpers.py` contains two
   `sk-ant-DEADBEEF...` fixtures matching `sk-ant-[A-Za-z0-9_-]{40,}`.
   Replaced both with PLACEHOLDER_LONG_TOKEN_... (≥40 chars, no sk-ant-
   prefix — scrubber path still exercised).
2. Runtime PR-Built: `workspace/a2a_tools_identity.py` missing from
   TOP_LEVEL_MODULES in scripts/build_runtime_package.py, causing build
   failure with "TOP_LEVEL_MODULES drifted". Added it.

Both fixes verified locally:
- pytest affected tests: 3/3 PASSED
- build_runtime_package.py: builds cleanly

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The secret-scan workflow flags sk-ant-[A-Za-z0-9_-]{40,} patterns.
Two sk-ant-api03-* fixture tokens (47 and 62 chars) were present in
test_sanitize_agent_error_reason_scrubs_all_secret_formats. They were
not replaced by PR #1430 (which only fixed the sk-ant-DEADBEEF* tokens).

Replace with tokens that still exercise the same scrubber paths:

- BARE sk-* case (≥24 chars after "sk-"): use sk-FAKEPLACEHOLDER...
  (53 chars total; starts with "sk-" so the bare-pattern scrubber catches
  it, but lacks "sk-ant-" so the secret-scan pattern does not fire).

- JSON-quoted apiKey value (≥24 chars): use anon_fakefakefake...
  (45 chars; satisfies the JSON-quoted redaction path; does not match
  any secret-scan credential pattern).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CI / all-required sentinel job was never scheduled in the prior
ci.yml run (documented Gitea-1.22/act_runner skipped-sentinel quirk), so
it never posted its terminal status and the required context stayed
pending. Empty-tree commit is the sanctioned 1.22.6 rerun mechanism — it
makes the real sentinel job actually schedule and post its genuine
status. No source change.
On the mobile PWA, tapping into the chat input scaled the whole
viewport up. Root cause: iOS Safari/WebKit auto-zooms on focus when
the focused field's effective font-size is < 16px. The mobile chat
composer <textarea> (MobileChat.tsx) used fontSize: 14.5.

Fix is the root-cause one: raise the composer font to 16px. This
suppresses the focus-zoom WITHOUT a maximum-scale / user-scalable
viewport lock, so pinch-to-zoom accessibility is preserved. The app
has no viewport export (Next.js default width=device-width,
initial-scale=1) — intentionally left untouched.

Adds a regression test asserting the composer font-size is >= 16px.
Root cause: agent chat replies arrive ONLY via live AGENT_MESSAGE /
A2A_RESPONSE WebSocket events (store/socket.ts ReconnectingSocket).
The socket had NO visibilitychange/pageshow/online/focus recovery
anywhere — reconnect was driven solely by ws.onclose. iOS Safari /
Chrome-mobile freeze the page and silently tear the WS down WITHOUT
firing onclose when backgrounded / device-locked; on thaw nothing
re-arms (onclose never ran, interval timers were suspended), so the
socket is dead and every reply during suspension is lost.
store.rehydrate() only re-pulls /workspaces status, not chat, and
useChatHistory fetches DB history mount-only — so missed replies
never back-fill until remount (navigate away+back) or hard refresh.
Desktop never hits this (its onclose fires promptly). Not forked
code: MobileChat and desktop ChatTab share useChatSocket /
useChatHistory; the divergence is purely mobile browser lifecycle.

Fix (shared by desktop + mobile, restores the unified path):
- ReconnectingSocket installs visibilitychange/pageshow/online/focus
  listeners that force a reconnect when the page is foregrounded and
  the socket isn't OPEN/CONNECTING (no-ops on desktop where onclose
  already handled it; SSR-safe via typeof guards).
- On an open that follows a real loss, emit a socket-resume signal.
- useChatHistory subscribes to resume and re-runs loadInitial(), so
  chat threads back-fill the persisted messages missed while frozen —
  exactly what a remount does today, automatically.

Verification: code analysis + 9 new unit tests (socket.test.ts)
simulating background-suspend + visibility/pageshow/online/focus
transitions, asserting reconnect + resume emission + listener
teardown; full canvas suite green (3315 passed, 0 fail); npm run
build green. Mobile real-device confirmation still needed (cannot
be proven without a physical device) — see PR body.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
External / MCP-registered workspaces (delivery_mode=poll, e.g. the
local-Mac orchestrator) have no public URL: POST /workspaces/:id/a2a
short-circuits server-side (a2a_proxy.go:402) and returns a synthetic
{status:"queued", delivery_mode:"poll"} envelope IMMEDIATELY with no
reply. useChatSend treated that as a terminal response and called
releaseSendGuards() → `sending` went false the instant the POST
returned → the ChatTab thinking indicator vanished and the external
turn looked dead, even though the agent had not started. Internal
agents reply on the same synchronous POST so their indicator naturally
holds — that's the whole asymmetry; the transport is shared.

Fix (Tier A, minimal, client-only): detect the synthetic poll envelope
and KEEP `sending` true so the existing internal working-indicator
persists as a "received — agent is working" state. The eventual reply
already flows AgentMessageWriter → AGENT_MESSAGE WS push →
useChatSocket onAgentMessage/onSendComplete → releaseSendGuards, i.e.
the external reply now drives the indicator through the exact same path
an internal async reply uses — no parallel system. A generous 15-min
safety timer surfaces an honest, actionable error instead of an
infinite spinner if an offline poll agent never replies.

Tier B (lifecycle-driven interim progress) and Tier C (tool-call
parity — has an operator-privacy tradeoff, CTO decision required) are
designed in internal RFC external-workspace-canvas-progress-feedback,
not in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tests listRegistryFiltered and ListRegistry endpoint:
  - empty dir → empty list
  - files (non-dirs) → ignored
  - single plugin with plugin.yaml → appears in list
  - runtime filter: claude-code plugin matches, hermes plugin excluded
  - universal plugin (no runtimes field) → always included
  - nonexistent dir → empty list (ReadDir error is non-fatal)
  - HTTP endpoint: GET /plugins → 200 with JSON array

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
Co-committed-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
Co-authored-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
Co-committed-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-QA <core-qa@agents.moleculesai.app>
Co-committed-by: Molecule AI Core-QA <core-qa@agents.moleculesai.app>
Co-authored-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
Co-committed-by: Molecule AI Fullstack Engineer <fullstack-engineer@agents.moleculesai.app>
- useKeyboardShortcut (11 tests): enabled=false, meta/ctrl modifier
  guards, no-modifier guard, key mismatch, count increments, cleanup
  on unmount. Uses renderHook + spy on window.addEventListener +
  direct handler invocation with Object.defineProperty for modifier keys.
- useSocketEvent (3 tests): mount calls subscribeSocketEvents,
  unsubscribe on unmount, called only once on re-renders.
  Uses module-level mock with shared state to avoid vi.mock hoisting.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tests cssVar() for all 23 ColorToken values, verifies the
returned string is a valid CSS variable format, and confirms
inline style assignment works in the DOM.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tests theme-provider.tsx via renderHook so useEffect fires before
assertions. Covers: noopTheme fallback, initialTheme init, system
preference resolution, explicit theme override, setTheme state update,
and document.documentElement.dataset.theme sync on mount. matchMedia
stubbed via Object.defineProperty in beforeEach.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tests the workspace ID→name resolver for known IDs, null input,
nodes without name, empty name fallback, unknown ID fallback,
and useCallback memoization stability across re-renders. Uses
stable reference for mock nodes so deps are stable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tests the timestamp formatting helper exported from AuditTrailPanel:
just-now, minutes, hours, days boundaries, and exact boundary cases.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tests isPluginUnavailableError (MEMORY_PLUGIN_URL detection) and
formatTTL (null/undefined/expired/seconds/minutes/hours/days/invalid).
Uses vi.useFakeTimers + vi.setSystemTime for deterministic TTL tests.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes E2E Chat deterministically failing in CI with
"invalid character '/' after top-level value".

The Integration Tester appends "// Triggered by <job>" to manifest.json
after cloning. Go's json.Unmarshal rejects JSON5-style // comments.
Added stripJSON5Comments() that strips // comments while preserving
URLs with // in them (e.g. https://), then call it before json.Unmarshal
in loadRuntimesFromManifest.

Added 8 new tests covering:
- stripJSON5Comments: standalone comments, trailing comments, URLs
  preserved, inline comments, Integration Tester append, no-op
- loadRuntimesFromManifest: with trailing // comment, with inline //
  comment

Closes #1496

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirically root-caused: workspace/internal_chat_uploads.py:153 called
request.form(max_files=64, max_fields=32) without max_part_size, so
Starlette 1.0's 1 MiB default raised MultiPartException on every
single-part > 1 MiB. The Cloudflare-chunked-encoding hypothesis from
the issue body was source-level disproven (Starlette doesn't read
Content-Length/TE).

Three coupled changes per CTO directive:

1) Single source of truth across Go ws-server + Python workspace
   runtime. The Go-side const chatUploadMaxFileBytes /
   chatUploadMaxBytes are exported at provision time via env vars
   CHAT_UPLOAD_MAX_FILE_BYTES / CHAT_UPLOAD_MAX_TOTAL_BYTES
   (workspace_provision_shared.go::applyChatUploadLimits, defaulting
   layer — pre-set values win). Python module init reads the env;
   unset env keeps the legacy 25 MB / 50 MB defaults so an
   unprovisioned worker doesn't regress.

2) Raise the user-visible ceiling to 100 MB per file + 100 MB total.
   Issue #1520 asked for >= 100 MB; matching per-file = total avoids
   the "fits the total but 413'd on per-file" surprise.

3) Surface the MultiPartException string in the 400 body's `detail`
   field (per feedback_surface_actionable_failure_reason_to_user).
   MultiPartException messages describe shape, not content — no
   secrets — and they tell the user WHY (e.g. "Invalid boundary",
   "Part exceeded maximum size of …"). Bounded at 200 chars.

Tests:
- workspace/tests/test_internal_chat_uploads.py: pin 2 MiB part is now
  accepted (regression for #1520), parse-error 400 includes `detail`,
  total-cap 413 still fires above a per-file pass, env-driven SSOT
  override works, malformed env value falls back to default.
- workspace-server/internal/handlers/chat_upload_limits_test.go: pin
  the env-injection contract (both vars set to byte-stringified Go
  consts, pre-existing values preserved, 100 MB floor invariant).

All 28 Python tests in test_internal_chat_uploads.py pass; full
workspace-server/internal/handlers Go test package passes (14.2s).
The Python poll-mode A2A consumer in a2a_tools_delegation.py:184 reads
`terminal.get("error_detail")` from `/workspaces/:id/delegations` rows,
but the Go listDelegations handlers emitted only `error`. When a
delegation failed, polling silently lost the failure reason and fell
through to the generic "delegation failed" string — leaving the canvas
"FAILED TO DELIVER" surface with no actionable detail.

Both ledger-path (listDelegationsFromLedger) and activity_logs-path
(listDelegationsFromActivityLogs) now emit BOTH keys: `error_detail`
(canonical, what poll-mode reads) and `error` (kept for back-compat
with existing canvas UI consumers).

Regression tests pin both keys for both code paths.

Refs P1 #348, RFC #2829 PR-2 follow-up.
Per `feedback_surface_actionable_failure_reason_to_user`, an opaque
"workspace restart is the safe first move" prompt is the anti-pattern
when the underlying error is a still-in-flight long-running task —
which is exactly the canonical PM→Researcher codex coordination case
that surfaces "FAILED TO DELIVER" today.

Three improvements:

1. New `polling timeout after`/`check_task_status` pattern, ordered
   ABOVE the generic timeout bucket, routes to a hint that explicitly
   tells the user NOT to restart and to call check_task_status with
   the delegation_id to retrieve the in-flight result.

2. New optional `context.peerKind` parameter. When the caller knows
   the callee is a codex runtime, the empty-detail and generic-timeout
   hints specialise to call out that codex tasks can legitimately
   exceed the 600s sync-proxy budget.

3. The empty-detail branch (the most common "useless hint" path) no
   longer claims "restart is the safe first move" by default — it now
   tells the user to check the callee's Activity tab before restarting.

Existing single-arg callers continue to compile unchanged (context is
optional). Regression test count goes 9 → 17.

Refs P1 #348, feedback_surface_actionable_failure_reason_to_user.
Adds USER_MESSAGE WebSocket broadcast so a user's own outbound message
appears live on all their other sessions without requiring a manual
refresh.

**Server (Go)**
- New `EventUserMessage` constant in `events/types.go`
- New `extractCanvasUserMessage()` in `a2a_proxy_helpers.go` — parses
  A2A JSON-RPC request body, extracts text parts + file attachments
  from user-role message/send calls
- `logA2ASuccess()` now broadcasts `USER_MESSAGE` when canvas sends a
  message (callerID == "" + method == "message/send" + 2xx)

**Canvas (TypeScript)**
- `useChatSocket`: new `onUserMessage` callback + `USER_MESSAGE` handler
  in the socket event listener
- `ChatTab`: wires `onUserMessage` via `appendMessageDeduped`
- `MobileChat`: same wiring for mobile variant
- New test file: 9 tests covering text/file/text+file/wrong-workspace/
  empty/no-callback/other-events/replay/attachment-filtering cases

**Dedup contract**: the originating session's optimistic insert (useChatSend
step 2) and the incoming USER_MESSAGE broadcast share the same
`appendMessageDeduped` path — within the 3-second window the two collapse
into one bubble, so the user sees exactly one message bubble.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously a scan failure in the affected-workspace query was silently
dropped, meaning the workspace would not be auto-restarted and the
operator would never know. Now the error is logged so DB schema drift
or corrupted rows surface in the logs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
core-be closed this pull request 2026-06-01 05:03:14 +00:00

Pull request closed

Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2079