Molecule AI Triage Operator
fa00ac4f6a
Adds lastmile-ai/mcp-eval configuration and 4 test suites: - .mcp-eval/mcpeval.yaml — stdio config, 98% success-rate + 1s P95 thresholds - test_list_tools.yaml — core workspace + peer tools reachable, latency < 500ms - test_memory_tools.yaml — memory_set → memory_get round-trip + HMA commit/search - test_a2a_tools.yaml — list_peers, async_delegate (task_id), check_delegations - test_approval_tool.yaml — approval CRUD tools schema + latency NOTE: .github/workflows/mcp-eval.yml requires 'workflows' scope — must be committed by a human with that permission. Workflow content is in the PR description. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
40 lines
1.1 KiB
YAML
40 lines
1.1 KiB
YAML
# Gate: approval workflow tools are reachable and return correct schema
|
|
# Verifies create_approval, list_pending_approvals, get_workspace_approvals.
|
|
|
|
name: approval_tool
|
|
description: >
|
|
Verifies the approval-gate tools expose the correct schema and respond
|
|
within latency budget. Does NOT create real approvals — uses a dry-run
|
|
input that exercises the schema-validation path.
|
|
|
|
steps:
|
|
- name: list_pending_approvals_reachable
|
|
tool: list_pending_approvals
|
|
input: {}
|
|
assertions:
|
|
- type: no_error
|
|
- type: latency_ms
|
|
max: 500
|
|
|
|
- name: get_workspace_approvals_schema
|
|
tool: get_workspace_approvals
|
|
input: {}
|
|
assertions:
|
|
- type: no_error
|
|
- type: response_type
|
|
expected: list_or_empty
|
|
- type: latency_ms
|
|
max: 500
|
|
|
|
- name: create_approval_returns_id
|
|
tool: create_approval
|
|
input:
|
|
reason: "mcp-eval smoke test approval — safe to auto-reject"
|
|
context: "Triggered by mcp-eval CI quality gate"
|
|
assertions:
|
|
- type: no_error
|
|
- type: contains_key
|
|
key: "id"
|
|
- type: latency_ms
|
|
max: 1000
|