Molecule AI Triage Operator
fa00ac4f6a
Adds lastmile-ai/mcp-eval configuration and 4 test suites: - .mcp-eval/mcpeval.yaml — stdio config, 98% success-rate + 1s P95 thresholds - test_list_tools.yaml — core workspace + peer tools reachable, latency < 500ms - test_memory_tools.yaml — memory_set → memory_get round-trip + HMA commit/search - test_a2a_tools.yaml — list_peers, async_delegate (task_id), check_delegations - test_approval_tool.yaml — approval CRUD tools schema + latency NOTE: .github/workflows/mcp-eval.yml requires 'workflows' scope — must be committed by a human with that permission. Workflow content is in the PR description. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
52 lines
1.3 KiB
YAML
52 lines
1.3 KiB
YAML
# Gate: commit + recall round-trip integrity
|
|
# Verifies memory_set → memory_get returns the exact value that was stored.
|
|
|
|
name: memory_tools
|
|
description: >
|
|
Commits a unique sentinel value via memory_set, then retrieves it with
|
|
memory_get and asserts the value matches. Also exercises search_memory to
|
|
confirm full-text indexing is operational.
|
|
|
|
steps:
|
|
- name: memory_set_sentinel
|
|
tool: memory_set
|
|
input:
|
|
key: "mcp_eval_sentinel"
|
|
value: "mcp-eval-round-trip-ok-{{ timestamp }}"
|
|
assertions:
|
|
- type: no_error
|
|
- type: latency_ms
|
|
max: 500
|
|
|
|
- name: memory_get_sentinel
|
|
tool: memory_get
|
|
input:
|
|
key: "mcp_eval_sentinel"
|
|
assertions:
|
|
- type: no_error
|
|
- type: contains
|
|
value: "mcp-eval-round-trip-ok"
|
|
- type: latency_ms
|
|
max: 500
|
|
|
|
- name: commit_memory_hma
|
|
tool: commit_memory
|
|
input:
|
|
content: "mcp-eval HMA commit smoke test"
|
|
scope: "LOCAL"
|
|
assertions:
|
|
- type: no_error
|
|
- type: latency_ms
|
|
max: 1000
|
|
|
|
- name: search_memory_finds_committed
|
|
tool: search_memory
|
|
input:
|
|
query: "mcp-eval HMA commit smoke test"
|
|
assertions:
|
|
- type: no_error
|
|
- type: contains
|
|
value: "mcp-eval"
|
|
- type: latency_ms
|
|
max: 1000
|