mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
415cf3d6b1
Slice 2 of the Claude Code Compatibility Matrix: extend the tracer-bullet
cell from slice 1 across all four remaining provider columns for
basic_messaging_non_streaming. Proves the multi-provider, multi-model,
all-must-pass aggregation logic against a 1x5 grid that exercises every
status state.
What landed:
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py
- tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py
Per-provider files modeled on test_anthropic.py: each parametrizes
over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 (the three Claude tiers
required by the PRD), drives the real `claude` CLI through the
driver, and reports pass/fail via `compat_result`. Per-cell error
strings always include `[<model>]` so the docs tooltip can name the
failing model when a cell goes red.
- tests/claude_code/basic_messaging_non_streaming/test_azure.py
All three (Azure, Claude) cells report `not_applicable` with a
reason: Azure OpenAI Service does not host Anthropic models. The
test still parametrizes over the same three model ids so the test
count per cell is uniform across columns, and a future "Azure adds
Anthropic" announcement only requires flipping the body, not the
parametrization.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 1x5 sample updated to reflect the slice 2 outcome:
anthropic / bedrock_invoke / bedrock_converse / vertex_ai = pass,
azure = not_applicable.
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Two new golden-file tests:
1. 1x5 grid: feed the per-model results the four new test files
produce on a real run; assert the builder output equals the
hand-authored sample byte-for-byte.
2. fail-with-model-named: feed pass/fail/pass for one cell and assert
the cell aggregates to fail with the failing model id surfaced
in the error string (acceptance criterion: "the error string
identifies which model broke").
Key decisions:
- Duplication across the four per-provider files is accepted (per the
PRD) rather than extracted into a helper. Each file is self-contained
so a test author touching one provider doesn't accidentally regress
the others.
- Per-provider model alias names: `claude-<tier>-<provider-suffix>`
(e.g. `claude-haiku-4-5-bedrock-invoke`). These are the alias names
the proxy operator wires up in the routing config; the test only
knows the alias, the proxy knows the upstream model id and region.
- Azure is `not_applicable` rather than `not_tested` because the
cell will never apply, not "we haven't gotten to it yet" - the two
states are visually and semantically distinct in the rendered grid.
- Sample shows the realistic best-case outcome (4 pass + 1 NA). The
React renderer's coverage of the `fail` and `not_tested` states is
exercised by other cells in v1+, not the v0 sample.
Tests: 31 -> 34 passing (added 2 builder golden tests + 3 Azure
not_applicable parametrizations that pass without env vars).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
The hand-authored sample in this repo is the artifact the docs PR
copies; opening that doc PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| claude_code | ||
| code_coverage_tests | ||
| documentation_tests | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _flush_vcr_cache.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| eval_swe_bench.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| README.MD | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.