mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-19 00:01:29 +00:00
3f5ed5a9c8
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
133 lines
5 KiB
YAML
133 lines
5 KiB
YAML
# Proxy routing config for the Claude Code Compatibility Matrix PR gate.
|
|
#
|
|
# The tests under `tests/e2e/claude_code/` only know **alias** names (e.g.
|
|
# `claude-haiku-4-5-bedrock-invoke`). The proxy is the layer that maps
|
|
# each alias to a real upstream model id, region, and credentials.
|
|
#
|
|
# Adding a new (feature, provider) cell is a three-step change in the
|
|
# test repo (manifest + test file + alias here); changing which upstream
|
|
# model a cell exercises is a one-step change here, with no test edits.
|
|
#
|
|
# Aliases:
|
|
# - claude-{tier} → Anthropic API
|
|
# - claude-{tier}-bedrock-invoke → Bedrock InvokeModel API
|
|
# - claude-{tier}-bedrock-converse → Bedrock Converse API
|
|
# - claude-{tier}-vertex → GCP Vertex AI
|
|
# - claude-{tier}-azure → Microsoft Foundry (Anthropic deployments)
|
|
|
|
model_list:
|
|
# ---- Anthropic ----
|
|
- model_name: claude-haiku-4-5
|
|
litellm_params:
|
|
model: anthropic/claude-haiku-4-5
|
|
api_key: os.environ/ANTHROPIC_API_KEY
|
|
- model_name: claude-sonnet-4-5
|
|
litellm_params:
|
|
model: anthropic/claude-sonnet-4-5
|
|
api_key: os.environ/ANTHROPIC_API_KEY
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
- model_name: claude-opus-4-7
|
|
litellm_params:
|
|
model: anthropic/claude-opus-4-7
|
|
api_key: os.environ/ANTHROPIC_API_KEY
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
|
|
# ---- Bedrock (InvokeModel) ----
|
|
- model_name: claude-haiku-4-5-bedrock-invoke
|
|
litellm_params:
|
|
model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
|
|
aws_region_name: us-east-1
|
|
- model_name: claude-sonnet-4-5-bedrock-invoke
|
|
litellm_params:
|
|
model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
|
aws_region_name: us-east-1
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
- model_name: claude-opus-4-7-bedrock-invoke
|
|
litellm_params:
|
|
model: bedrock/us.anthropic.claude-opus-4-7
|
|
aws_region_name: us-east-1
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
|
|
# ---- Bedrock (Converse) ----
|
|
- model_name: claude-haiku-4-5-bedrock-converse
|
|
litellm_params:
|
|
model: bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
|
|
aws_region_name: us-east-1
|
|
- model_name: claude-sonnet-4-5-bedrock-converse
|
|
litellm_params:
|
|
model: bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
|
aws_region_name: us-east-1
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
- model_name: claude-opus-4-7-bedrock-converse
|
|
litellm_params:
|
|
model: bedrock/converse/us.anthropic.claude-opus-4-7
|
|
aws_region_name: us-east-1
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
|
|
# ---- Vertex AI ----
|
|
# `use_in_pass_through: true` registers each deployment's
|
|
# project/location/credentials with the /vertex_ai passthrough
|
|
# router, which the `passthrough` row needs to resolve
|
|
# .../models/{alias}:streamRawPredict URLs. That registration only
|
|
# reads the canonical `vertex_project`/`vertex_location` param names
|
|
# (not the `vertex_ai_*` aliases); the chat translation path accepts
|
|
# both.
|
|
- model_name: claude-haiku-4-5-vertex
|
|
litellm_params:
|
|
model: vertex_ai/claude-haiku-4-5
|
|
vertex_project: os.environ/VERTEXAI_PROJECT
|
|
vertex_location: global
|
|
use_in_pass_through: true
|
|
- model_name: claude-sonnet-4-5-vertex
|
|
litellm_params:
|
|
model: vertex_ai/claude-sonnet-4-5
|
|
vertex_project: os.environ/VERTEXAI_PROJECT
|
|
vertex_location: global
|
|
use_in_pass_through: true
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
- model_name: claude-opus-4-7-vertex
|
|
litellm_params:
|
|
model: vertex_ai/claude-opus-4-7
|
|
vertex_project: os.environ/VERTEXAI_PROJECT
|
|
vertex_location: global
|
|
use_in_pass_through: true
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
|
|
# ---- Microsoft Foundry (Anthropic deployments on Azure) ----
|
|
- model_name: claude-haiku-4-5-azure
|
|
litellm_params:
|
|
model: azure_ai/claude-haiku-4-5
|
|
api_base: os.environ/AZURE_AI_API_BASE
|
|
api_key: os.environ/AZURE_AI_API_KEY
|
|
- model_name: claude-sonnet-4-5-azure
|
|
litellm_params:
|
|
model: azure_ai/claude-sonnet-4-5
|
|
api_base: os.environ/AZURE_AI_API_BASE
|
|
api_key: os.environ/AZURE_AI_API_KEY
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
- model_name: claude-opus-4-7-azure
|
|
litellm_params:
|
|
model: azure_ai/claude-opus-4-7
|
|
api_base: os.environ/AZURE_AI_API_BASE
|
|
api_key: os.environ/AZURE_AI_API_KEY
|
|
extra_headers:
|
|
anthropic-beta: "context-1m-2025-08-07"
|
|
|
|
general_settings:
|
|
# Claude Code sends provider-specific headers (e.g. anthropic-beta) we
|
|
# want to forward verbatim to the upstream so the wire-shape under
|
|
# test matches what real customers send.
|
|
forward_client_headers_to_llm_api: true
|
|
|
|
litellm_settings:
|
|
drop_params: true
|
|
modify_params: true
|