mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
10462eddaf
|
test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634)
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests * test(e2e): harden remaining stage failures in harness Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps * test(e2e): drop claude_code and harness unit tests from this PR Keep management, router, budget, and shared conftest harness fixes only * test(e2e): restore E2E_RESULT pytest_runtest_makereport hook Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines * test(e2e): drop management control-plane write retries Transient 500 retries do not fix the underlying control plane failures * test(e2e): skip stage-red claude_code cells; fix multi-window budget latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s * test(e2e): require exactly one LLM-tier spend row for complexity router Keep alias membership for compose vs stage model names, but assert len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin LIT-4521 skip and align LIT-4522/23/24 skip reasons * test(e2e): harden router callable probe and multi-window budget exhaustion _router_is_callable treated any non-success chat whose body lacked "Invalid model name" as callable, so an unpropagated probe key (401), a generic 502, or a connection reset let the session proceed and hit real "Invalid model name" failures inside the tests. Require a Success outcome instead; the reload-race 400 and every infra/auth error now correctly read as not-callable. The multi-window budget test capped the tight window at 3e-6, which gpt-5.5 exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the 20-call loop, turning a reset test into a spurious "window never enforced" failure. Drop the tight cap to 1e-9 so the first billed call exhausts it regardless of model price; the roomy 1m window stays at 1.0 and never blocks. * test(e2e): use a tradeoff-decision prompt for the complexity router classifier "Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which made the test fail even though the classifier was running. The tier definitions key on what the request demands, not how hard the answer is, and a short direct question maps to SIMPLE regardless of subject. Swap in "Should I pay off my mortgage early or invest the extra money instead?". It carries none of the heuristic scorer's reasoning/technical/code keywords and stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier only has to avoid SIMPLE for the test to distinguish a real classifier run from the heuristic fallback. |
||
|
mubashir1osmani
|
3f5ed5a9c8
|
fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433)
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
|
||
|
mateo-berri
|
79bfe2e996 | docs(e2e/claude_code): document the known-red azure passthrough cell (anthropic-version dropped by the /azure fallback) | ||
|
mateo-berri
|
95fe8687c8 |
test(e2e/claude_code): add passthrough matrix row for the big-3 clouds and Anthropic API
Adds a 'passthrough' feature row to the Claude Code compat matrix that
drives the real claude CLI in each cloud's native mode against
LiteLLM's passthrough routes (the LLM-gateway setup from
code.claude.com/docs/en/gateway) instead of the /v1/messages
translation layer:
- anthropic: ANTHROPIC_BASE_URL={proxy}/anthropic, forwarded verbatim
to api.anthropic.com
- bedrock_invoke: CLAUDE_CODE_USE_BEDROCK=1 against {proxy}/bedrock;
the router resolves the alias in /model/{alias}/invoke-with-response-stream
- vertex_ai: CLAUDE_CODE_USE_VERTEX=1 against {proxy}/vertex_ai/v1;
alias, project, location and credentials resolve from the deployment,
which now sets use_in_pass_through: true (and the canonical
vertex_project/vertex_location param names) in test_config.yaml
- azure: CLAUDE_CODE_USE_FOUNDRY=1 against {proxy}/azure via the
AZURE_API_BASE/AZURE_API_KEY fallback (documented in the cron env
example)
- bedrock_converse: not_applicable; Claude Code has no Converse-wire
client
The shared cell body lives in _passthrough.py with injectable runner
and env (no monkeypatching), unit-covered in
_driver_unit_tests/test_passthrough.py including pins on the per-mode
CLI env contracts captured from a real claude CLI (2.1.210) run
against a request-logging sink.
|