mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
5d121fa697
|
fix(matrix-builder): greptile — surface pass over not_applicable when mixed
When a cell aggregates per-model results across three tiers (Haiku/Sonnet/Opus), a mix of (pass, not_applicable) used to short-circuit to not_applicable on the first NA match, discarding the passing tiers from the published matrix. Treat not_applicable like not_tested when mixed with pass: only return not_applicable when every observed row is NA. Otherwise any pass surfaces as pass, so the cell answers 'does this feature work on this provider?' truthfully when at least one tier passes. Add two regression tests pinning the new precedence: - mixed pass + NA → pass - all NA → not_applicable (with first reason) |
||
|
Cursor Agent
|
9928da27f3
|
fix(claude_code): bugbot — aggregate all fail errors + structural test every manifest feature
Addresses three Bugbot concerns flagged on PR #28027 that are real behavioral / coverage gaps: 1. matrix_builder._aggregate_cell now joins every failing tier's error in the published cell instead of silently dropping all but the first. When Haiku 429s and Opus times out on the same cell, both diagnostics land in the matrix JSON so docs-page triage can name both outliers. 2. _aggregate_cell treats 'not_tested' rows as absent data: they're dropped before computing the cell status. Previously a mixed (pass, not_tested) cell silently fell through to 'not_tested', discarding the passing tiers and hiding real coverage from the published matrix. A cell still aggregates to 'not_tested' when *every* row is 'not_tested' (or there are no rows at all). 3. test_v0_layout.py now structurally validates every feature declared in manifest.yaml (directory exists, __init__.py exists, every per-provider test_<provider>.py exists), not just the original six v0 rows. The EXPECTED_FEATURE_IDS / EXPECTED_PROVIDERS anchor constants still pin v0 positions; the new manifest-driven tests extend the same structural guarantees to every post-v0 row so a broken directory in 'count_tokens', 'tool_search', 'web_search', etc. fails CI instead of silently becoming a 'not_tested' cell. Three new builder tests pin the new aggregation behavior: - mixed pass + not_tested surfaces as pass - all-not_tested stays not_tested - multiple fail errors are joined with '; ' Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Cursor Agent
|
f41d3f91a3
|
fix(claude_code): harden parallel runner + de-dup basic_messaging cells
- run_claude_models_parallel: catch all exceptions in the per-model
worker and wrap unexpected ones into a ClaudeCLIError so the
documented 'errors as values' contract holds for OSError, ValueError,
etc., not just ClaudeCLIError. Without this, an unexpected raise in
any layer (rate limiter file I/O, infer_provider, etc.) abandons the
remaining models' results and crashes the calling test.
- test_run_claude_places_extra_args_before_prompt: drop the dead first
branch of the 'or' assertion — cmd[-3:] never matches that shape, so
the alternative was misleading dead code.
- basic_messaging_{non_streaming,streaming}/test_*.py: extract the
shared cell body into tests/claude_code/_basic_messaging.py.
Each per-provider file now declares its model list and calls
run_basic_messaging_cell(), eliminating ~700 lines of copy-paste
across 10 files. Updated _builder_unit_tests/test_v0_layout.py to
accept the helper-based pattern alongside direct run_claude() calls.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
mateo-berri
|
be65b4e23b |
feat(claude_code): rename thinking row + add 4 feature rows (15 total)
Matrix grows from 11 to 15 feature rows. All new tests collected + 180 unit tests still pass; smoke runs hit real LiteLLM bug surfaces on bedrock_invoke, bedrock_converse, and vertex_ai (cells correctly red in PR #142). Rename ------ `extended_thinking` -> `thinking` (directory, manifest id+name, 5 test fn names, 5 docstrings, builder unit-test fixtures, sample JSON, run_compat.sh). Existing test logic already covers both manual (`thinking.type=enabled`, Haiku 4.5) and adaptive (`thinking.type=adaptive`, Opus 4.7) shapes because Claude Code picks the shape per model from `--effort max`; the name change just stops the column from looking like a Claude 3.7 reference. New rows -------- - structured_outputs (5 files, CLI `--json-schema`). Claude Code synthesizes a single `StructuredOutput` tool from the schema and surfaces the tool_use input as `structured_output` on the trailing `result` event. Test ships its own `_validate_against_schema` so we don't take a jsonschema dep just for matrix surface. - count_tokens (5 files, HTTP probe). POSTs the proxy's `/v1/messages/count_tokens` directly and asserts the response is `{input_tokens: positive int}`. No CLI hook exists for this endpoint; the test goes through the new http_probe helper instead. - tool_search (5 files, HTTP probe). Sends `tools: [{type: tool_search_tool_regex_20251119, name: tool_search_tool_regex}]` and asserts the proxy doesn't 400. MCP fan-out via `--mcp-config` would also exercise the tool-search beta header path, but it's flaky w.r.t. Claude Code's internal tool-deferral threshold; the HTTP probe hits the actual bug surface (per-provider beta-header translation `advanced-tool-use-2025-11-20` vs `tool-search-tool-2025-10-19`). - long_context_1m (5 files, CLI `--betas context-1m-2025-08-07 --max-budget-usd 6`). A ~210k-token padded prompt over stdin exercises the 1M-context beta. Sonnet 4.6 + Opus 4.7 only -- Haiku 4.5's window is 200k, so it's excluded from MODELS (not marked not_applicable) to keep the per-cell aggregator semantics intact. Prompt uses a document-style preamble + 8 cycling pangrams rather than repeating identical chunks; without that, Opus 4.7 trips the safety filter mid-response with a Usage Policy refusal. `--max-budget-usd 6` is a runaway-loop guard, ~2x worst-case Opus per-cell spend. New helper ---------- `tests/claude_code/http_probe.py`: shared `ProbeResult` dataclass plus per-endpoint `probe_*` + `assert_*_shape` pairs for the HTTP-probe rows. Uses httpx with `anthropic-version: 2023-06-01` and a 30s timeout. |
||
|
mateo-berri
|
d05e45893e |
compat-matrix: parallel-fanout refactor + 5 new feature dirs + rate limiter
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
|
||
|
mateo-berri
|
a2ba4f850c |
RALPH: fix compat matrix - Azure now hosts Claude via Microsoft Foundry
Anthropic and Microsoft announced Claude Haiku 4.5, Sonnet 4.5/4.6, and
Opus 4.1/4.6/4.7 in Microsoft Foundry on 2025-11-18, so the matrix's
Azure column should exercise a real route through the LiteLLM proxy
rather than report not_applicable.
Foundry serves Claude on an Anthropic-shape /anthropic/v1/messages
endpoint (not the Azure OpenAI chat-completions route), and LiteLLM
already supports it via the azure_ai/claude-* provider prefix
(litellm/llms/azure_ai/anthropic/{handler,transformation,messages_transformation}.py).
- test_config.yaml: add 3 azure aliases pointing at azure_ai/claude-*
with AZURE_FOUNDRY_API_BASE / AZURE_FOUNDRY_API_KEY env
- 6x test_azure.py: replace not_applicable stubs with real run_claude
drivers, mirroring the existing test_vertex_ai.py shape exactly
- sample_compatibility-matrix.json: Azure cells flip to pass
- _builder_unit_tests: pin the new invariant (run_claude is used,
not_applicable is gone) and feed pass results across all 5 providers
in the 6x5 golden test
|
||
|
mateo-berri
|
2ab4e9cfd2 |
RALPH: compat matrix slice 5 - add remaining 5 v0 features (full 6x5 grid) (#26481, PRD #26476)
Slice 5 of the Claude Code Compatibility Matrix: extend the published
matrix from the 1x5 grid that landed in slice 2 to the full v0 6x5
grid described in the PRD's "Features in v0" section. After this
slice merges and the daily cron runs, the docs page reflects all six
v0 features against all five providers.
What landed:
- tests/claude_code/manifest.yaml
Five new entries appended in PRD row order:
basic_messaging_streaming, tool_use, prompt_caching_5m, vision,
extended_thinking. The manifest is the row-order source of truth
the matrix builder respects.
- tests/claude_code/<feature>/test_<provider>.py (25 new files)
For each of the five new features, five per-provider test files
modeled on slice 2's basic_messaging_non_streaming/. Each non-Azure
file parametrizes over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 and drives
the real `claude` CLI through the driver with feature-specific
options:
* basic_messaging_streaming — count-1-to-5 prompt; asserts the
stream-json wire actually emitted events plus a non-empty reply.
* tool_use — `--allowed-tools Bash` plus an `echo pong` prompt;
asserts a `tool_use` content block was emitted.
* prompt_caching_5m — same baseline prompt as non-streaming, but
asserts the upstream usage block reports
cache_creation_input_tokens or cache_read_input_tokens > 0
(Claude Code stamps cache_control on its system prompt by default,
so a single live call surfaces it).
* vision — decodes a checked-in 1x1 PNG (base64 const) into
`tmp_path` and attaches it via `--image`; asserts a non-empty
reply.
* extended_thinking — sets `MAX_THINKING_TOKENS=4096`; asserts a
`thinking` content block was emitted.
All five Azure files report `not_applicable` with the standard
reason: Azure OpenAI Service does not host Anthropic models.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 6x5 sample showing the realistic best-case outcome:
4 pass + 1 not_applicable (Azure) per row.
- tests/claude_code/_builder_unit_tests/test_v0_layout.py
New structural unit tests pinning the on-disk shape so future edits
can't silently flip the matrix shape:
* manifest lists all six v0 feature ids in PRD order
* manifest lists all five v0 provider columns in PRD order
* every (feature, provider) has a test file at the inferred path
* every test file references all three required Claude tiers
* every Azure test file is a `not_applicable` declaration
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Renamed the slice-2 1x5 golden test to
test_build_matrix_6x5_grid_matches_published_sample and rebuilt
its inputs to feed all six features. The golden file is now the
6x5 sample.
Key decisions:
- Per-feature per-provider test bodies are deliberately duplicated
(per the PRD: "Duplication across per-provider files is accepted").
Each file is self-contained so a contributor touching one cell
doesn't accidentally regress neighbors.
- Only Azure cells are marked `not_applicable` in this slice. Other
combinations that turn out to genuinely not apply on the live cron
run (e.g. a provider that doesn't support `thinking` for a tier)
will be tightened to `not_applicable` reasons in a follow-up; for
now they fail honestly, which the matrix renderer paints red.
- prompt_caching_5m's assertion (cache tokens > 0 in the usage block)
exercises the path Claude Code customers care about: that the proxy
preserves `cache_control` annotations end-to-end. It does not try
to differentiate cache_creation vs cache_read across runs.
- The vision PNG fixture is generated at test time from a base64
const rather than checked into git as a binary — keeps the diff
text-only and avoids needing PIL or any image-generation library.
Tests: 31 -> 142 unit tests passing (no proxy / no `claude` CLI
required). Test counts:
* 12 builder tests (was 11; +1 for 6x5 golden, the slice-2 1x5
test was renamed in place)
* 100 v0_layout structural tests (new)
* 10 driver tests (unchanged)
* 9 compat_result tests (unchanged)
* 14 publisher unit tests (unchanged)
* 8 PR-gate version-resolver tests (unchanged)
* 6 CircleCI structural tests (unchanged)
The 90 per-cell tests under tests/claude_code/<feature>/ continue to
require a running proxy + `claude` CLI; they only run inside the
CircleCI PR gate or the daily-cron VM (both established in slices
3 and 4).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
After slice 4's daily-cron lands the App credentials, the cron run
will replace the docs-side hand-authored JSON automatically; until
then the slice-2 1x5 sample remains in the docs repo.
Notes for next iteration:
- The exact `claude` CLI flags for tool-allowlist (`--allowed-tools`),
vision (`--image`), and extended thinking (`MAX_THINKING_TOKENS`)
are best-guess from the current Claude Code surface; if the live
PR-gate run reveals different flag names, tighten in place.
- Several non-Azure cells will likely need `not_applicable`
declarations once the cron VM produces real outcomes (e.g.
Bedrock Invoke + extended_thinking is uncertain). That refinement
is an iteration-2 follow-up driven by data, not a blocker for this
slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
mateo-berri
|
415cf3d6b1 |
RALPH: compat matrix slice 2 - add 4 provider columns for basic_messaging_non_streaming (#26478, PRD #26476)
Slice 2 of the Claude Code Compatibility Matrix: extend the tracer-bullet
cell from slice 1 across all four remaining provider columns for
basic_messaging_non_streaming. Proves the multi-provider, multi-model,
all-must-pass aggregation logic against a 1x5 grid that exercises every
status state.
What landed:
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py
- tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py
Per-provider files modeled on test_anthropic.py: each parametrizes
over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 (the three Claude tiers
required by the PRD), drives the real `claude` CLI through the
driver, and reports pass/fail via `compat_result`. Per-cell error
strings always include `[<model>]` so the docs tooltip can name the
failing model when a cell goes red.
- tests/claude_code/basic_messaging_non_streaming/test_azure.py
All three (Azure, Claude) cells report `not_applicable` with a
reason: Azure OpenAI Service does not host Anthropic models. The
test still parametrizes over the same three model ids so the test
count per cell is uniform across columns, and a future "Azure adds
Anthropic" announcement only requires flipping the body, not the
parametrization.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 1x5 sample updated to reflect the slice 2 outcome:
anthropic / bedrock_invoke / bedrock_converse / vertex_ai = pass,
azure = not_applicable.
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Two new golden-file tests:
1. 1x5 grid: feed the per-model results the four new test files
produce on a real run; assert the builder output equals the
hand-authored sample byte-for-byte.
2. fail-with-model-named: feed pass/fail/pass for one cell and assert
the cell aggregates to fail with the failing model id surfaced
in the error string (acceptance criterion: "the error string
identifies which model broke").
Key decisions:
- Duplication across the four per-provider files is accepted (per the
PRD) rather than extracted into a helper. Each file is self-contained
so a test author touching one provider doesn't accidentally regress
the others.
- Per-provider model alias names: `claude-<tier>-<provider-suffix>`
(e.g. `claude-haiku-4-5-bedrock-invoke`). These are the alias names
the proxy operator wires up in the routing config; the test only
knows the alias, the proxy knows the upstream model id and region.
- Azure is `not_applicable` rather than `not_tested` because the
cell will never apply, not "we haven't gotten to it yet" - the two
states are visually and semantically distinct in the rendered grid.
- Sample shows the realistic best-case outcome (4 pass + 1 NA). The
React renderer's coverage of the `fail` and `not_tested` states is
exercised by other cells in v1+, not the v0 sample.
Tests: 31 -> 34 passing (added 2 builder golden tests + 3 Azure
not_applicable parametrizations that pass without env vars).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
The hand-authored sample in this repo is the artifact the docs PR
copies; opening that doc PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
mateo-berri
|
6c573de426 |
RALPH: tracer-bullet for Claude Code compatibility matrix (#26477, PRD #26476)
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end path through every layer for a single (feature, provider) cell, so a future docs page can render a real green cell sourced from a real test. What landed in this repo: - tests/claude_code/manifest.yaml — feature manifest with one entry (basic_messaging_non_streaming) plus the v0 provider column order. - tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry point (run_claude); handles subprocess assembly, env overlay, stream-JSON parsing, and structured failure modes. `runner=` is a unit-test seam. - tests/claude_code/conftest.py — `compat_result` fixture (tagged-union recorder) + pytest_runtest_makereport hook that infers (feature, provider) from the file path and writes a structured compat-results.json artifact. - tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell model rule. - tests/claude_code/matrix_builder.py — pure-function builder from (manifest, results, run-metadata) to the v1 JSON schema. Aggregates per- model results into one cell (pass iff all pass). build_from_paths is the thin I/O wrapper for the publisher. - tests/claude_code/sample_compatibility-matrix.json — hand-authored sample of the v1 JSON; copied to the docs repo by hand as part of this slice. - Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests, 10 matrix-builder golden-file tests. 29/29 pass. Key decisions: - (feature, provider) is inferred from file path, not declared in metadata — mirrors the PRD's "no drift" goal. - Driver injects subprocess via a `runner` kwarg so unit tests don't need the real `claude` CLI; production callers leave it default. - Builder is a pure function on Mappings/Sequences; load/write live in a thin `build_from_paths` wrapper. Golden-file tests pin the schema. - `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_` so the conftest's path-inference hook skips them and they don't pollute the matrix artifact. - `compat-results.json` added to .gitignore (CI-only output). Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs): - The MDX page `docs/tutorials/claude-code-compatibility` and the `<CompatibilityMatrix />` React component. The hand-authored compatibility-matrix.json (`sample_compatibility-matrix.json` in this repo) is the artifact those docs files will consume; opening that doc PR is the next step in this slice. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |