hermes-agent/tests/tools
Brian D. Evans e87a2100f6 fix(mcp): auto-reconnect + retry once when the transport session expires (#13383)
Streamable HTTP MCP servers may garbage-collect their server-side
session state while the OAuth token remains valid — idle TTL, server
restart, pod rotation, etc.  Before this fix, the tool-call handler
treated the resulting "Invalid or expired session" error as a plain
tool failure with no recovery path, so **every subsequent call on
the affected server failed until the gateway was manually
restarted**.  Reporter: #13383.

The OAuth-based recovery path (``_handle_auth_error_and_retry``)
already exists for 401s, but it only fires on auth errors.  Session
expiry slipped through because the access token is still valid —
nothing 401'd, so the existing recovery branch was skipped.

Fix
---
Add a sibling function ``_handle_session_expired_and_retry`` that
detects MCP session-expiry via ``_is_session_expired_error`` (a
narrow allow-list of known-stable substrings: ``"invalid or expired
session"``, ``"session expired"``, ``"session not found"``,
``"unknown session"``, etc.) and then uses the existing transport
reconnect mechanism:

* Sets ``MCPServerTask._reconnect_event`` — the server task's
  lifecycle loop already interprets this as "tear down the current
  ``streamablehttp_client`` + ``ClientSession`` and rebuild them,
  reusing the existing OAuth provider instance".
* Waits up to 15 s for the new session to come back ready.
* Retries the original call once.  If the retry succeeds, returns
  its result and resets the circuit-breaker error count.  If the
  retry raises, or if the reconnect doesn't ready in time, falls
  through to the caller's generic error path.

Unlike the 401 path, this does **not** call ``handle_401`` — the
access token is already valid and running an OAuth refresh would be
a pointless round-trip.

All 5 MCP handlers (``call_tool``, ``list_resources``, ``read_resource``,
``list_prompts``, ``get_prompt``) now consult both recovery paths
before falling through:

    recovered = _handle_auth_error_and_retry(...)          # 401 path
    if recovered is not None: return recovered
    recovered = _handle_session_expired_and_retry(...)     # new
    if recovered is not None: return recovered
    # generic error response

Narrow scope — explicitly not changed
-------------------------------------
* **Detection is string-based on a 5-entry allow-list.**  The MCP
  SDK wraps JSON-RPC errors in ``McpError`` whose exception type +
  attributes vary across SDK versions, so matching on message
  substrings is the durable path.  Kept narrow to avoid false
  positives — a regular ``RuntimeError("Tool failed")`` will NOT
  trigger spurious reconnects (pinned by
  ``test_is_session_expired_rejects_unrelated_errors``).
* **No change to the existing 401 recovery flow.**  The new path is
  consulted only after the auth path declines (returns ``None``).
* **Retry count stays at 1.**  If the reconnect-then-retry also
  fails, we don't loop — the error surfaces normally so the model
  sees a failed tool call rather than a hang.
* **``InterruptedError`` is explicitly excluded** from session-expired
  detection so user-cancel signals always short-circuit the same
  way they did before (pinned by
  ``test_is_session_expired_rejects_interrupted_error``).

Regression coverage
-------------------
``tests/tools/test_mcp_tool_session_expired.py`` (new, 16 cases):

Unit tests for ``_is_session_expired_error``:
* ``test_is_session_expired_detects_invalid_or_expired_session`` —
  reporter's exact wpcom-mcp text.
* ``test_is_session_expired_detects_expired_session_variant`` —
  "Session expired" / "expired session" variants.
* ``test_is_session_expired_detects_session_not_found`` — server GC
  variant ("session not found", "unknown session").
* ``test_is_session_expired_is_case_insensitive``.
* ``test_is_session_expired_rejects_unrelated_errors`` — narrow-scope
  canary: random RuntimeError / ValueError / 401 don't trigger.
* ``test_is_session_expired_rejects_interrupted_error`` — user cancel
  must never route through reconnect.
* ``test_is_session_expired_rejects_empty_message``.

Handler integration tests:
* ``test_call_tool_handler_reconnects_on_session_expired`` — reporter's
  full repro: first call raises "Invalid or expired session", handler
  signals ``_reconnect_event``, retries once, returns the retry's
  success result with no ``error`` key.
* ``test_call_tool_handler_non_session_expired_error_falls_through``
  — preserved-behaviour canary: random tool failures do NOT trigger
  reconnect.
* ``test_session_expired_handler_returns_none_without_loop`` —
  defensive: cold-start / shutdown race.
* ``test_session_expired_handler_returns_none_without_server_record``
  — torn-down server falls through cleanly.
* ``test_session_expired_handler_returns_none_when_retry_also_fails``
  — no retry loop on repeated failure.

Parametrised across all 4 non-``tools/call`` handlers:
* ``test_non_tool_handlers_also_reconnect_on_session_expired``
  [list_resources / read_resource / list_prompts / get_prompt].

**15 of 16 fail on clean ``origin/main`` (``6fb69229``)** with
``ImportError: cannot import name '_is_session_expired_error'``
— the fix's surface symbols don't exist there yet.  The 1 passing
test is an ordering artefact of pytest-xdist worker collection.

Validation
----------
``source venv/bin/activate && python -m pytest
tests/tools/test_mcp_tool_session_expired.py -q`` → **16 passed**.

Broader MCP suite (5 files:
``test_mcp_tool.py``, ``test_mcp_tool_401_handling.py``,
``test_mcp_tool_session_expired.py``, ``test_mcp_reconnect_signal.py``,
``test_mcp_oauth.py``) → **230 passed, 0 regressions**.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-24 05:28:45 -07:00
..
__init__.py
test_accretion_caps.py
test_ansi_strip.py
test_approval_heartbeat.py
test_approval.py test: cover absolute paths in project env/config approval regex 2026-04-23 14:05:36 -07:00
test_base_environment.py
test_browser_camofox_persistence.py
test_browser_camofox_state.py
test_browser_camofox.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_browser_cdp_override.py
test_browser_cdp_tool.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_browser_cleanup.py
test_browser_cloud_fallback.py
test_browser_console.py
test_browser_content_none_guard.py
test_browser_hardening.py
test_browser_homebrew_paths.py
test_browser_orphan_reaper.py
test_browser_secret_exfil.py
test_browser_ssrf_local.py
test_browser_supervisor.py feat(browser): CDP supervisor — dialog detection + response + cross-origin iframe eval (#14540) 2026-04-23 22:23:37 -07:00
test_budget_config.py
test_checkpoint_manager.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_clarify_tool.py
test_clipboard.py
test_code_execution_modes.py
test_code_execution.py
test_command_guards.py
test_config_null_guard.py
test_credential_files.py
test_cron_approval_mode.py
test_cron_prompt_injection.py
test_cronjob_tools.py
test_daytona_environment.py
test_debug_helpers.py
test_delegate_subagent_timeout_diagnostic.py feat(delegate): diagnostic dump when a subagent times out with 0 API calls (#15105) 2026-04-24 04:58:32 -07:00
test_delegate_toolset_scope.py
test_delegate.py fix(delegate): remove model-facing max_iterations override; config is authoritative (#14732) 2026-04-23 13:56:26 -07:00
test_discord_tool.py
test_docker_environment.py fix(docker): add SETUID/SETGID caps so gosu drop in entrypoint succeeds 2026-04-22 18:13:14 -07:00
test_docker_find.py
test_dockerfile_pid1_reaping.py fix(docker): reap orphaned subprocesses via tini as PID 1 (#15116) 2026-04-24 05:22:34 -07:00
test_env_passthrough.py fix(env_passthrough): reject Hermes provider credentials from skill passthrough (#13523) 2026-04-21 06:14:25 -07:00
test_feishu_tools.py
test_file_operations_edge_cases.py tools: normalize file tool pagination bounds 2026-04-22 06:11:41 -07:00
test_file_operations.py tools: normalize file tool pagination bounds 2026-04-22 06:11:41 -07:00
test_file_ops_cwd_tracking.py
test_file_read_guards.py
test_file_staleness.py fix(file_tools): resolve bookkeeping paths against live terminal cwd 2026-04-23 15:11:52 -07:00
test_file_state_registry.py feat(delegate): cross-agent file state coordination for concurrent subagents (#13718) 2026-04-21 16:41:26 -07:00
test_file_sync_back.py
test_file_sync_perf.py
test_file_sync.py
test_file_tools_container_config.py
test_file_tools_live.py
test_file_tools.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_file_write_safety.py
test_force_dangerous_override.py
test_fuzzy_match.py
test_hidden_dir_filter.py
test_homeassistant_tool.py
test_image_generation_env.py
test_image_generation_plugin_dispatch.py fix(image-gen): force-refresh plugin providers in long-lived sessions 2026-04-23 03:01:18 -07:00
test_image_generation.py feat(image-gen): add GPT Image 2 to FAL catalog (#13677) 2026-04-21 13:35:31 -07:00
test_interrupt.py
test_llm_content_none_guard.py
test_local_background_child_hang.py
test_local_env_blocklist.py
test_local_interrupt_cleanup.py
test_local_shell_init.py fix(terminal): auto-source ~/.profile and ~/.bash_profile so n/nvm PATH survives (#14534) 2026-04-23 05:15:37 -07:00
test_local_tempdir.py
test_managed_browserbase_and_modal.py
test_managed_media_gateways.py
test_managed_modal_environment.py
test_managed_server_tool_support.py
test_managed_tool_gateway.py
test_mcp_circuit_breaker.py
test_mcp_dynamic_discovery.py
test_mcp_oauth_bidirectional.py
test_mcp_oauth_cold_load_expiry.py
test_mcp_oauth_integration.py
test_mcp_oauth_manager.py
test_mcp_oauth.py
test_mcp_probe.py
test_mcp_reconnect_signal.py
test_mcp_stability.py fix(mcp): per-process PID isolation prevents cross-session crash on restart 2026-04-23 15:11:47 -07:00
test_mcp_structured_content.py
test_mcp_tool_401_handling.py
test_mcp_tool_issue_948.py
test_mcp_tool_session_expired.py fix(mcp): auto-reconnect + retry once when the transport session expires (#13383) 2026-04-24 05:28:45 -07:00
test_mcp_tool.py fix(mcp): seed protocol header before HTTP initialize 2026-04-23 22:01:24 -07:00
test_memory_tool_import_fallback.py
test_memory_tool.py
test_mixture_of_agents_tool.py chore(release): map devorun author + convert MoA defaults test to invariant 2026-04-23 15:14:11 -07:00
test_modal_bulk_upload.py
test_modal_sandbox_fixes.py
test_modal_snapshot_isolation.py
test_notify_on_complete.py
test_osv_check.py
test_parse_env_var.py guard terminal_tool import-time env parsing 2026-04-22 14:45:50 -07:00
test_patch_parser.py
test_process_registry.py
test_read_loop_detection.py
test_registry.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_resolve_path.py fix(file_tools): resolve bookkeeping paths against live terminal cwd 2026-04-23 15:11:52 -07:00
test_rl_training_tool.py
test_schema_sanitizer.py fix: sanitize tool schemas for llama.cpp backends; restore MCP in TUI (#15032) 2026-04-24 02:44:46 -07:00
test_search_hidden_dirs.py
test_send_message_missing_platforms.py
test_send_message_tool.py
test_session_search.py
test_signal_media.py
test_singularity_preflight.py
test_skill_env_passthrough.py
test_skill_improvements.py
test_skill_manager_tool.py feat(skills-guard): gate agent-created scanner on config.skills.guard_agent_created (default off) 2026-04-23 06:20:47 -07:00
test_skill_size_limits.py
test_skill_view_path_check.py
test_skill_view_traversal.py
test_skills_guard.py feat(skills-guard): gate agent-created scanner on config.skills.guard_agent_created (default off) 2026-04-23 06:20:47 -07:00
test_skills_hub_clawhub.py
test_skills_hub.py
test_skills_sync.py feat(skills_sync): surface collision with reset-hint 2026-04-23 05:09:08 -07:00
test_skills_tool.py fix(skills): follow symlinked category dirs consistently 2026-04-23 14:05:47 -07:00
test_spotify_client.py Add native Spotify tools with PKCE auth 2026-04-24 05:20:38 -07:00
test_ssh_bulk_upload.py
test_ssh_environment.py
test_symlink_prefix_confusion.py
test_sync_back_backends.py
test_terminal_compound_background.py
test_terminal_exit_semantics.py
test_terminal_foreground_timeout_cap.py
test_terminal_none_command_guard.py
test_terminal_output_transform_hook.py
test_terminal_requirements.py
test_terminal_timeout_output.py
test_terminal_tool_pty_fallback.py
test_terminal_tool_requirements.py
test_terminal_tool.py
test_threaded_process_handle.py
test_tirith_security.py
test_todo_tool.py
test_tool_backend_helpers.py
test_tool_call_parsers.py
test_tool_output_limits.py feat(skills): add design-md skill for Google's DESIGN.md spec (#14876) 2026-04-23 21:51:19 -07:00
test_tool_result_storage.py
test_transcription_tools.py fix(transcription): fall back to CPU when CUDA runtime libs are missing 2026-04-24 02:50:14 -07:00
test_transcription.py
test_tts_gemini.py
test_tts_kittentts.py
test_tts_max_text_length.py fix(tts): use per-provider input-character caps instead of global 4000 (#13743) 2026-04-21 17:49:39 -07:00
test_tts_mistral.py
test_tts_speed.py
test_url_safety.py feat(security): add global toggle to allow private/internal URL resolution 2026-04-22 14:38:59 -07:00
test_vision_tools.py
test_voice_cli_integration.py
test_voice_mode.py
test_watch_patterns.py
test_web_tools_config.py
test_web_tools_tavily.py
test_website_policy.py
test_windows_compat.py
test_write_deny.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00
test_yolo_mode.py
test_zombie_process_cleanup.py fix(tests): resolve 17 persistent CI test failures (#15084) 2026-04-24 03:46:46 -07:00