mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
2306816d40
A shadow eval whose judge_model is one of the router's tier models, the router's default model, or a reverse job's baseline_model was accepted with no warning. An LLM judge scores its own output higher than a rival's, so that tier's win rate measures the judge instead of the models, and the job's whole budget buys a result that has to be thrown away. start_shadow_eval now rejects it with a 400 naming the colliding arm. `judge_target` is the single answer to "where does a call to this name go for this caller, and what answers it", and the resolvability gate, the collision gate and the judge dispatch all read it. It has three outcomes and no others: the router serves the name, the SDK serves it, or nothing does. Splitting that question is what every bug here came from, so `router_resolves_model` and `answering_models` are gone rather than joined by a third. Two spellings of one model are one identity. A name is compared by what would answer it, resolved through every channel `get_model_list` composes and then put in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o` collides with a tier deployment serving `openai/gpt-4o`, and a judge given as `openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends are normalised because an admin writes them at different times. Answering is also per-caller. The shadow and judge calls carry the shadowed key's `user_api_key_team_id`, which is what the router selects deployments with, so the endpoint derives the job's teams once from the keys it already looks up and every check runs under them, and the judge dispatch picks its arm under the same team. A team's public model name resolves to nothing for everyone else and a team's own deployment resolves for nobody else, so a check that omits the team answers for a caller who does not exist. A collision under any one team fails the job, because every key's verdicts land in the same win rates. Three sites were separately re-deriving "the provider models this name resolves to", with unexplained divergence in whether they fell back to the literal name. `Router.resolved_litellm_models` is now the one owner; the routing-plugin candidate list and the stream-options check both delegate to it, and `_deployment_litellm_model` is gone. The router's arms come from `strategy_router_dependencies`, the same enumeration the health check reads. Only the roles that serve are arms: a classifier or embedding model picks the tier and never produces a response anyone judges. A semantic auto-router keeps its routes in an opaque config blob, so only its default model is enumerable and the guard is incomplete there by design, able to miss a collision but never to invent one The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one has to carry the base's drift to go green |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.