2eddb36bff
Root cause of the 2026-04-23 E2E A2A regression: when a workspace's
model is openai/* and the tenant has an OPENROUTER_API_KEY set globally
(common on SaaS staging), derive-provider.sh was picking PROVIDER=openrouter
even when the WORKSPACE-level secret OPENAI_API_KEY was explicitly provided
for the direct-OpenAI path.
hermes then called OpenRouter with a key that was stale/empty for this
workspace, and OR returned `{"error": {"message": "Missing Authentication
header", "code": 401}}` — which surfaced in the A2A agent reply and
failed the E2E at step 8.
Fix: flip the priority. For openai/* model slugs, prefer `custom` (direct
OpenAI via install.sh's HERMES_CUSTOM_* bridge) when OPENAI_API_KEY is
present. Fall through to `openrouter` only when OPENAI_API_KEY is absent.
Operators who want OR for openai/* models can still:
- set HERMES_INFERENCE_PROVIDER=openrouter (wins via the explicit override at top of file), or
- use an openrouter/* model slug
Adds scripts/test-derive-provider.sh — 12 offline shell assertions
pinning the decision table including the exact #19 regression case.
Acceptance: E2E step 8 A2A returns a real PONG reply instead of OR's
401-shaped error from the agent response.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
113 lines
3.5 KiB
Bash
Executable File
113 lines
3.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# test-derive-provider.sh — offline unit tests for derive-provider.sh.
|
|
#
|
|
# derive-provider.sh has zero external deps (no network, no hermes install,
|
|
# no filesystem writes) — it's pure env-var → PROVIDER string. That makes
|
|
# it cheap to exercise in CI as a shell-level test: set env, source, check
|
|
# $PROVIDER. Runs in <1s.
|
|
#
|
|
# Covers regressions:
|
|
# #19 (2026-04-23 E2E) — openai/* + OPENAI_API_KEY must route to `custom`,
|
|
# NOT `openrouter`, even when a global OPENROUTER_API_KEY is present.
|
|
# Previously the wrong-priority rule hijacked operator intent and the
|
|
# A2A reply surfaced OpenRouter's `401 Missing Authentication header`.
|
|
|
|
set -u
|
|
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
SCRIPT="$HERE/derive-provider.sh"
|
|
|
|
if [ ! -f "$SCRIPT" ]; then
|
|
echo "FAIL: derive-provider.sh not found at $SCRIPT"
|
|
exit 2
|
|
fi
|
|
|
|
PASS=0
|
|
FAIL=0
|
|
|
|
assert_provider() {
|
|
local label="$1" expected="$2"
|
|
# Run in subshell so env mutations don't leak between cases.
|
|
local actual
|
|
actual=$(bash -c "
|
|
unset HERMES_INFERENCE_PROVIDER
|
|
$3
|
|
PROVIDER=''
|
|
. '$SCRIPT'
|
|
echo \$PROVIDER
|
|
")
|
|
if [ "$actual" = "$expected" ]; then
|
|
echo " PASS $label → $actual"
|
|
PASS=$((PASS+1))
|
|
else
|
|
echo " FAIL $label → got '$actual', expected '$expected'"
|
|
FAIL=$((FAIL+1))
|
|
fi
|
|
}
|
|
|
|
echo "== derive-provider.sh =="
|
|
|
|
# --- explicit override wins ---
|
|
assert_provider "HERMES_INFERENCE_PROVIDER=anthropic beats model slug" "anthropic" '
|
|
HERMES_INFERENCE_PROVIDER=anthropic
|
|
HERMES_DEFAULT_MODEL=openai/gpt-4o
|
|
'
|
|
|
|
# --- direct-SDK prefixes ---
|
|
assert_provider "minimax/M2 → minimax" "minimax" '
|
|
HERMES_DEFAULT_MODEL=minimax/MiniMax-M2
|
|
'
|
|
assert_provider "anthropic/claude → anthropic" "anthropic" '
|
|
HERMES_DEFAULT_MODEL=anthropic/claude-sonnet-4-6
|
|
'
|
|
|
|
# --- openai/* priority: REGRESSION TEST for #19 / 2026-04-23 E2E ---
|
|
# The scenario: operator provides OPENAI_API_KEY as a workspace secret,
|
|
# and the CP/tenant has OPENROUTER_API_KEY set globally. derive-provider
|
|
# must pick `custom` (direct OpenAI) to honor operator intent — NOT
|
|
# `openrouter` which would hit OR with a key that may be stale/empty.
|
|
assert_provider "openai/* + OPENAI_API_KEY + OPENROUTER_API_KEY → custom (#19 regression)" "custom" '
|
|
HERMES_DEFAULT_MODEL=openai/gpt-4o
|
|
export OPENAI_API_KEY=sk-test-openai
|
|
export OPENROUTER_API_KEY=sk-or-test
|
|
'
|
|
|
|
assert_provider "openai/* + only OPENAI_API_KEY → custom" "custom" '
|
|
HERMES_DEFAULT_MODEL=openai/gpt-4o
|
|
export OPENAI_API_KEY=sk-test-openai
|
|
'
|
|
|
|
assert_provider "openai/* + only OPENROUTER_API_KEY → openrouter" "openrouter" '
|
|
HERMES_DEFAULT_MODEL=openai/gpt-4o
|
|
export OPENROUTER_API_KEY=sk-or-test
|
|
'
|
|
|
|
assert_provider "openai/* + no keys → openrouter (fail-loud fallback)" "openrouter" '
|
|
HERMES_DEFAULT_MODEL=openai/gpt-4o
|
|
'
|
|
|
|
# --- nousresearch/* branches ---
|
|
assert_provider "nousresearch/* + HERMES_API_KEY → nous" "nous" '
|
|
HERMES_DEFAULT_MODEL=nousresearch/hermes-4-70b
|
|
export HERMES_API_KEY=h-test
|
|
'
|
|
assert_provider "nousresearch/* + only NOUS_API_KEY → nous" "nous" '
|
|
HERMES_DEFAULT_MODEL=nousresearch/hermes-4-70b
|
|
export NOUS_API_KEY=n-test
|
|
'
|
|
assert_provider "nousresearch/* + no nous keys → openrouter" "openrouter" '
|
|
HERMES_DEFAULT_MODEL=nousresearch/hermes-4-70b
|
|
'
|
|
|
|
# --- unknown prefix ---
|
|
assert_provider "unknown prefix → auto" "auto" '
|
|
HERMES_DEFAULT_MODEL=vendor-x/model-y
|
|
'
|
|
|
|
# --- no model at all ---
|
|
assert_provider "no HERMES_DEFAULT_MODEL → auto" "auto" ''
|
|
|
|
echo
|
|
echo "== results: $PASS passed, $FAIL failed =="
|
|
[ "$FAIL" -eq 0 ]
|