hermes-agent/tests
Siddharth Balyan 1fa76607c0
feat: trigram FTS5 index for CJK search, replace LIKE fallback (#16651)
* fix: bypass FTS5 for CJK queries in session_search

FTS5 default tokenizer splits CJK characters into individual tokens,
so multi-character queries like "大别山项目" become AND of single chars.
This produces few/no results compared to LIKE substring search.

For CJK queries, skip FTS5 entirely and use LIKE for accurate
phrase matching.

Fixes NousResearch/hermes-agent#15500

* fix: cache _contains_cjk, escape LIKE wildcards, add regression tests

On top of the CJK FTS5 bypass from #15509:

- Cache _contains_cjk() result in a local var to avoid redundant O(n)
  scans on every CJK query
- Escape %, _ in LIKE queries so literal wildcards in user input are
  not treated as SQL wildcards (consistent with other LIKE queries in
  hermes_state.py that use ESCAPE '\')
- Fix misleading comment ('or CJK fallback' → accurate description)
- Add 3 regression tests:
  - test_cjk_partial_fts5_results_supplemented_by_like (#15500 / #14829)
  - test_cjk_like_dedup_no_duplicates
  - test_cjk_like_escapes_wildcards (new wildcard escaping)

* feat: trigram FTS5 index for CJK search, replace LIKE fallback

Replace the LIKE '%query%' full-table-scan fallback for CJK queries with
a proper trigram FTS5 index (messages_fts_trigram).  The trigram tokenizer
creates overlapping 3-byte sequences so substring matching works natively
for any script — CJK, Thai, etc.

For queries with 3+ CJK characters: uses the trigram FTS5 table with
proper ranking, snippets, and indexed lookups.  For shorter queries
(1-2 CJK chars): falls back to LIKE since the trigram tokenizer needs
≥9 UTF-8 bytes (3 CJK chars) minimum.

Schema v10 migration creates the trigram table and backfills existing
messages.  Triggers keep the index in sync on INSERT/UPDATE/DELETE.

Builds on top of #16276 (bypass FTS5 for CJK, escape LIKE wildcards).

---------

Co-authored-by: vominh1919 <vominh1919@gmail.com>
2026-04-28 00:12:07 +05:30
..
acp fix(acp): include MCP toolsets in ACP sessions 2026-04-24 03:04:42 -07:00
agent fix(providers/gmi): post-salvage review fixes 2026-04-27 11:17:59 -07:00
cli fix(cli): pass session messages to shutdown_memory_provider (#15165 sibling) 2026-04-27 06:41:16 -07:00
cron fix(gateway,cron): close ephemeral agents + reap stale aux clients (salvage #13979) (#16598) 2026-04-27 07:41:42 -07:00
e2e test(discord): add guild to fake e2e messages 2026-04-25 18:25:56 -07:00
environments/benchmarks
fakes
gateway fix(gateway,cron): close ephemeral agents + reap stale aux clients (salvage #13979) (#16598) 2026-04-27 07:41:42 -07:00
hermes_cli fix(providers/gmi): post-salvage review fixes 2026-04-27 11:17:59 -07:00
hermes_state fix(resume): redirect --resume to the descendant that actually holds the messages 2026-04-24 03:04:42 -07:00
honcho_plugin feat(honcho): wizard cadence default 2, surface reasoning level, backwards-compat fallback 2026-04-18 22:50:55 -07:00
integration fix(discord): strip RTP padding before DAVE/Opus decode (#11267) 2026-04-16 16:50:15 -07:00
plugins feat(plugins): google_meet \u2014 join, transcribe, speak, follow up (#16364) 2026-04-27 06:22:25 -07:00
run_agent fix(copilot): mark native image requests as vision 2026-04-27 08:35:50 -07:00
skills fix(skills): honor scope query from Google OAuth redirect URL 2026-04-26 21:08:19 -07:00
tools fix(ci): resolve follow-up title edge case and flaky checks 2026-04-27 11:49:02 -05:00
tui_gateway Revert "feat(onboarding): port first-touch hints to the TUI (#16054)" (#16062) 2026-04-26 06:31:37 -07:00
website fix(website): auto-wrap ASCII-art code blocks in generated skill pages (#16497) 2026-04-27 03:38:39 -07:00
__init__.py
conftest.py feat(providers): add GMI Cloud as a first-class API-key provider (#11955) 2026-04-27 11:17:59 -07:00
run_interrupt_test.py
test_account_usage.py feat(account-usage): add per-provider account limits module 2026-04-21 01:56:35 -07:00
test_base_url_hostname.py security(runtime_provider): close OLLAMA_API_KEY substring-leak sweep miss (#13522) 2026-04-21 06:06:16 -07:00
test_batch_runner_checkpoint.py test: regression coverage for checkpoint dedup and inf/nan coercion 2026-04-24 14:32:21 -07:00
test_cli_file_drop.py fix(tui): improve macOS paste and shortcut parity 2026-04-21 08:00:00 -07:00
test_cli_skin_integration.py fix: align status bar skin tests with upstream main 2026-04-22 13:20:02 -07:00
test_ctx_halving_fix.py
test_empty_model_fallback.py
test_evidence_store.py
test_hermes_constants.py
test_hermes_logging.py fix(logging): attach gateway log after cli init 2026-04-26 19:01:26 -07:00
test_hermes_state.py feat: trigram FTS5 index for CJK search, replace LIKE fallback (#16651) 2026-04-28 00:12:07 +05:30
test_honcho_client_config.py
test_ipv4_preference.py
test_mcp_serve.py
test_mini_swe_runner.py fix(kimi): omit temperature entirely for Kimi/Moonshot models (#13157) 2026-04-20 12:23:05 -07:00
test_minimax_model_validation.py fix(models): validate MiniMax models against static catalog (#12611, #12460, #12399, #12547) 2026-04-19 22:44:47 -07:00
test_minisweagent_path.py
test_model_picker_scroll.py
test_model_tools_async_bridge.py fix(core): ensure non-blocking executor shutdown on async timeout 2026-04-22 14:42:32 -07:00
test_model_tools.py feat(hooks): add duration_ms to post_tool_call + transform_tool_result (#15429) 2026-04-25 22:13:12 -07:00
test_ollama_num_ctx.py
test_packaging_metadata.py
test_plugin_skills.py fix(tests): attach caplog to specific logger in 3 order-dependent tests (#11453) 2026-04-17 00:20:40 -07:00
test_project_metadata.py build(deps): add qrcode to dingtalk + feishu extras (parity with messaging) (#11627) 2026-04-17 13:31:53 -07:00
test_retry_utils.py
test_sql_injection.py
test_subprocess_home_isolation.py
test_timezone.py test: speed up slow tests (backoff + subprocess + IMDS network) (#11797) 2026-04-17 14:21:22 -07:00
test_toolset_distributions.py
test_toolsets.py feat(discord): split discord_server into discord + discord_admin tools 2026-04-25 04:50:14 -07:00
test_trajectory_compressor_async.py fix(kimi): omit temperature entirely for Kimi/Moonshot models (#13157) 2026-04-20 12:23:05 -07:00
test_trajectory_compressor.py fix(kimi): omit temperature entirely for Kimi/Moonshot models (#13157) 2026-04-20 12:23:05 -07:00
test_transform_tool_result_hook.py test: stop testing mutable data — convert change-detectors to invariants (#13363) 2026-04-20 23:20:33 -07:00
test_tui_gateway_server.py fix(tui): tighten fast-mode support validation 2026-04-27 13:00:11 -05:00
test_utils_truthy_values.py
test_yuanbao_integration.py yuanbao platform (#16298) 2026-04-26 18:50:49 -07:00
test_yuanbao_markdown.py yuanbao platform (#16298) 2026-04-26 18:50:49 -07:00
test_yuanbao_pipeline.py yuanbao platform (#16298) 2026-04-26 18:50:49 -07:00
test_yuanbao_proto.py yuanbao platform (#16298) 2026-04-26 18:50:49 -07:00