mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
3f5ed5a9c8
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
82 lines
3.1 KiB
Python
82 lines
3.1 KiB
Python
"""tool_search x Anthropic.
|
|
|
|
HTTP-probe row. Sends a single `/v1/messages` request whose `tools`
|
|
array includes a `tool_search_tool_regex_20251119` discovery tool, and
|
|
asserts the proxy round-trips it to the upstream without a 400. This
|
|
verifies LiteLLM's tool-search beta-header translation
|
|
(`advanced-tool-use-2025-11-20` for Anthropic-shape providers,
|
|
`tool-search-tool-2025-10-19` for Vertex/Bedrock) survives end-to-end.
|
|
|
|
The (feature, provider) for this cell is inferred from the file path by
|
|
`tests/e2e/claude_code/conftest.py`:
|
|
|
|
tests/e2e/claude_code/tool_search/test_anthropic.py
|
|
^^^^^^^^^^^ ^^^^^^^^^
|
|
feature_id provider
|
|
|
|
Why HTTP probe instead of CLI / MCP fan-out:
|
|
|
|
Real Claude Code activates tool_search by registering >N MCP tools and
|
|
relying on the model's internal heuristic to call the discovery tool
|
|
before any user tool. That setup requires standing up a stub MCP
|
|
server that exposes 50+ tool stubs and depends on Claude Code's
|
|
auto-deferral heuristic continuing to fire at today's tool count --
|
|
both of which break silently when Claude Code's threshold changes
|
|
between releases.
|
|
|
|
The bugs LiteLLM has actually shipped fixes for in this area
|
|
(2.1.117, 2.1.72, 2.1.70 in the Claude Code release notes) are
|
|
beta-header translation and proxy-side type recognition, not MCP
|
|
fan-out behavior. An HTTP probe hits exactly that surface: the
|
|
request goes out with a `tool_search_tool_regex_20251119` tool type,
|
|
the proxy is responsible for attaching the per-provider beta header
|
|
and forwarding, and the upstream either accepts or 400s. A red cell
|
|
here is always a proxy-side regression, not a flaky model-behavior
|
|
artifact.
|
|
|
|
Three Claude tiers are probed in sequence (count is too low to be
|
|
worth the parallelism overhead, and HTTP probes don't compete for
|
|
the proxy's `--num-workers` slots the way CLI subprocess runs do).
|
|
The matrix's "all three must pass" rule still applies via the
|
|
per-cell aggregator.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from claude_code._env import require_proxy
|
|
from claude_code.http_probe import (
|
|
assert_tool_search_shape,
|
|
probe_tool_search,
|
|
)
|
|
|
|
|
|
ANTHROPIC_MODELS = [
|
|
"claude-haiku-4-5",
|
|
"claude-sonnet-4-5",
|
|
"claude-opus-4-7",
|
|
]
|
|
|
|
|
|
@pytest.mark.covers("llm.messages.anthropic.tool_search.nonstream.works")
|
|
def test_tool_search_anthropic(compat_result):
|
|
"""Probe `/v1/messages` with a `tool_search_tool_regex_20251119`
|
|
tool and assert the proxy + upstream accept it for every Anthropic
|
|
tier."""
|
|
base_url, api_key = require_proxy(compat_result)
|
|
|
|
failures = []
|
|
for model in ANTHROPIC_MODELS:
|
|
result = probe_tool_search(base_url=base_url, api_key=api_key, model=model)
|
|
shape_error = assert_tool_search_shape(result)
|
|
if shape_error is not None:
|
|
error = f"[{model}] tool_search probe failed: {shape_error}"
|
|
compat_result.add({"status": "fail", "error": error})
|
|
failures.append(error)
|
|
continue
|
|
|
|
compat_result.add({"status": "pass"})
|
|
|
|
if failures:
|
|
pytest.fail("; ".join(failures), pytrace=False)
|