scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-19 00:01:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 88
litellm/tests/test_litellm/proxy
History
tin-berri 4e48d74455
feat(shadow_eval): measure both arms' cost so a job reports what the router would have saved (#38631) 
The attempt row now prices the real arm (the payload's response_cost plus its own
routing classifier when it routed) beside the shadow arm (completion plus the
classifier cost the routing decision writes back), and flags turns litellm's
response cache served. A per-leg funnel table counts the eligible requests that
produced no row (lost the sampling dice, unjudgeable shape, concurrency shed),
so results can weigh judged rows against the traffic they stand for. Job results
gain per-slice and overall arm spends plus the coverage counts, the budget gates
charge the shadow arm's classifier spend against max_budget, and the dashboard
shows the measured cost comparison beside the win rate

Resolves LIT-6358

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 15:13:19 -07:00
..
_experimental/mcp_server fix: enforce MCP toolsets attached to a team, org, or internal user (#38488) 2026-08-28 12:06:57 -07:00
a2a fix(a2a): reject malformed protocolVersion suffixes while keeping semver prereleases 2026-07-21 13:24:58 -07:00
agent_endpoints fix(a2a): merge fresh agent vectors into the live cache and drop entries of another dimension 2026-08-27 20:58:14 -07:00
analytics_endpoints feat(ui): add error-code drilldown for failed requests on caching page 2026-08-24 15:45:45 -07:00
anthropic_endpoints refactor(repositories): type prisma table access with one generic protocol 2026-08-25 12:14:17 +00:00
auth fix(proxy): let llm_api virtual keys read /model_group/info 2026-08-28 14:42:22 +00:00
batches_endpoints test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
client test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
common_utils feat(proxy): opt-in budget rollover carrying overage into the next window (#38514) 2026-08-27 12:46:09 -07:00
config_resolvers feat(proxy): add SAML 2.0 SSO for the admin UI (#31429) 2026-07-24 12:51:28 -07:00
credential_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
db feat(shadow_eval): measure both arms' cost so a job reports what the router would have saved (#38631) 2026-08-28 15:13:19 -07:00
discovery_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
enterprise_billing feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592) 2026-07-15 12:12:52 -07:00
experimental/mcp_server test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
fine_tuning_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
google_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
guardrails feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940) 2026-08-28 14:13:49 -07:00
health_endpoints feat(alerting): add native Microsoft Teams alerting destination (#38367) 2026-08-27 16:19:22 -07:00
hooks fix(interactions): poll queued background creates and drop the poll's deployment identity 2026-08-22 17:35:19 -07:00
image_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
list_api feat(proxy): add paginated GET /public/v1/model_hub (#38636) 2026-08-28 10:02:59 -07:00
logging_endpoints feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134) 2026-06-24 15:25:10 -07:00
management_endpoints feat(shadow_eval): measure both arms' cost so a job reports what the router would have saved (#38631) 2026-08-28 15:13:19 -07:00
management_helpers fix(key_management): allow /key/update to keep or shrink MCP server grants the key already holds (#38463) 2026-08-27 12:51:17 -07:00
memory fix(memory): return 404 when the memory row vanishes before delete 2026-08-25 16:20:17 -07:00
middleware feat(proxy): add Amazon Comprehend Medical passthrough provider 2026-08-17 15:44:06 -07:00
ocr_endpoints fix(ocr): validate body req_format in the proxy endpoint and run its tests in CI 2026-08-17 18:29:35 +00:00
openai_files_endpoint test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
pass_through_endpoints test(pass_through): pin stream pricing tests to injected divergent rate cards 2026-08-26 16:54:05 -07:00
policy_engine feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940) 2026-08-28 14:13:49 -07:00
prompts fix(prompts): propagate prompt deletes to every worker and pod 2026-08-26 18:01:08 -07:00
proxy_server fix(model_info): map supports_parallel_function_calling into ModelInfo 2026-08-28 12:09:48 -07:00
public_endpoints feat(proxy): add paginated GET /public/v1/model_hub (#38636) 2026-08-28 10:02:59 -07:00
rag_endpoints feat(proxy): enforce vector-store upload security controls on /v1/rag/ingest 2026-08-24 12:41:15 -07:00
realtime_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
rerank_endpoints fix(rerank): emit latency and cost headers on /rerank (#35419) 2026-08-25 15:54:25 -07:00
response_api_endpoints fix(completion_extras): forward reasoning_effort=max through the Responses API bridge 2026-08-25 09:40:23 -07:00
shutdown
spend_tracking fix(spend): remove the proxy-wide autorouter savings baseline override (#38700) 2026-08-28 14:53:27 -07:00
test_configs
types_utils fix(proxy): resolve router_settings.plugins dotted paths and load plugins from installed packages (#33644) 2026-07-17 11:23:18 -07:00
ui_crud_endpoints Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5_round2 2026-08-25 16:17:29 -07:00
utils feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940) 2026-08-28 14:13:49 -07:00
vector_store_endpoints refactor(repositories): type prisma table access with one generic protocol 2026-08-25 12:14:17 +00:00
vector_store_files_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
video_endpoints fix(videos): forward uploaded source file on /v1/videos/edits to the provider 2026-08-24 15:34:21 -07:00
__init__.py
conftest.py test(proxy): keep a leaked llm_router out of the next test in the worker 2026-08-20 23:19:17 -07:00
test__types.py test(proxy): pin the request-validation contracts in proxy/_types.py 2026-08-25 23:29:26 -07:00
test_aiohttp_cleanup_closed.py
test_aiohttp_session_recovery.py
test_api_key_masking_in_errors.py
test_audio_speech_prometheus_hooks.py
test_batch_expiry.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_metadata_none_fix.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_retrieve_bedrock.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_x_litellm_model_encoding.py test(proxy): give the x-litellm-model fallback test deterministic openai env creds 2026-08-17 12:42:57 -07:00
test_blocked_response_usage.py fix(responses_api): map bridged chat usage on guardrail-blocked replies 2026-08-14 17:04:27 -07:00
test_budget_reservation.py fix(router): forward the hold-back keepalive ping live and carry the withheld-output flag through the stream wrappers 2026-08-25 16:39:14 -07:00
test_caching_routes.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_chat_completion_metadata.py
test_claude_code_marketplace.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_common_request_processing.py fix(proxy): price the /v1/messages cost header by the deployment model, not the client alias 2026-08-28 12:09:25 -07:00
test_component_allowlists.py feat(proxy): authenticate to Azure Postgres with Microsoft Entra ID tokens 2026-08-20 11:50:16 -07:00
test_conftest.py test(proxy): stop monkeypatch.undo re-planting fixture-mocked prisma_client 2026-08-13 20:11:06 -07:00
test_cors_config.py
test_custom_proxy.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_dynamic_mcp_route.py fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845) 2026-08-14 17:04:32 -07:00
test_empty_model_list.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_enforce_user_param.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_fallback_management_endpoints.py
test_fastapi_offline_routes.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_filter_models_by_team_access_group.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_health_check_functions.py feat(health): opt-in model-group allowlist for background health checks and health-check routing (#38539) 2026-08-27 12:25:56 -07:00
test_health_check_max_tokens.py fix(proxy): derive auto-router health from its underlying models (#38174) 2026-08-26 16:41:54 -07:00
test_init_litellm_callbacks.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_langfuse_passthrough_security.py
test_lazy_openapi_snapshot.py refactor(proxy): type the snapshot fragments and wrap a long test line 2026-08-26 14:39:51 -07:00
test_litellm_pre_call_utils.py fix(router): scrub fallback stamp keys in place and strip them at the proxy boundary (#38690) 2026-08-28 14:18:56 -07:00
test_max_budget_env_var.py
test_mcp_asgi_response.py
test_model_based_routing_files_batches.py
test_model_deprecations_endpoint.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_model_dump_with_preserved_fields.py
test_model_id_header_propagation.py feat(proxy): add x-litellm-model-name response header with deployment model string (#33698) 2026-07-17 20:29:42 -07:00
test_model_info_default_limits.py
test_model_level_guardrails.py fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
test_model_list_healthy_only.py feat(proxy): hide unhealthy models from model listings, opt-in 2026-08-26 00:13:52 -07:00
test_modify_response_streaming_passthrough.py fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough (#30800) 2026-06-18 23:29:08 -07:00
test_openai_ws_passthrough_routes.py fix(proxy): close websocket cleanly when OpenAI credentials are missing 2026-08-16 14:40:37 -07:00
test_openapi_schema_validation.py
test_plugin_routes.py feat: litellm plugin architecture v2 (#30688) 2026-06-20 20:37:22 -07:00
test_pricing_field_strip.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prisma_engine_watchdog.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_prisma_migration.py fix(proxy): keep a failed prisma generate from failing the migration entrypoint (#37947) 2026-08-22 11:45:19 -07:00
test_prometheus_cleanup.py
test_provider_url_destination_guard.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_cli.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_logging_hook_detection.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_proxy_server.py Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fallback_access_group_check 2026-08-27 17:29:08 -07:00
test_proxy_types.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_utils.py fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging (#38102) 2026-08-25 12:44:16 -07:00
test_pyroscope.py
test_read_model_list.py feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135) 2026-06-23 19:16:34 -07:00
test_redis_auth_cache_flag.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_response_model_sanitization.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_route_a2a_models.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_route_llm_request.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sensitive_route_auth.py
test_shared_health_check.py fix(proxy): derive auto-router health from its underlying models (#38174) 2026-08-26 16:41:54 -07:00
test_spend_log_cleanup.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_swagger_chat_completions.py
test_team_member_update.py
test_team_org_move.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_tools_allowlist_enforcement.py fix(anthropic): close hybrid tool-name allowlist gap and keep native tools through guardrails 2026-08-26 20:37:11 -07:00
test_update_llm_router_resilience.py feat(complexity_router): custom classifier plugins via classifier_type 'custom' (#37249) 2026-08-18 14:09:19 -07:00