scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-08 22:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78
litellm/tests/test_litellm/proxy
History
Exact
Mateo Wang 75613bf22f
test: add regression coverage for twelve closed issues (#37974) 
* test: add regression coverage for twelve closed issues

Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:

- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges

Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.

* fix: stop the new regression tests polluting and tripping over shared global state

Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.

test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.

The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
2026-08-22 22:24:05 +00:00
..
_experimental/mcp_server test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
a2a fix(a2a): reject malformed protocolVersion suffixes while keeping semver prereleases 2026-07-21 13:24:58 -07:00
agent_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
analytics_endpoints feat(ui): split failed requests into their own series on the cache dashboard (#34862) 2026-07-29 09:48:17 -07:00
anthropic_endpoints fix(proxy): estimate failed-request input tokens on /v1/messages and count system prompts 2026-08-18 14:21:47 -07:00
auth fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958) 2026-08-22 14:25:55 -07:00
batches_endpoints test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
client test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
common_utils test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
config_resolvers feat(proxy): add SAML 2.0 SSO for the admin UI (#31429) 2026-07-24 12:51:28 -07:00
credential_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
db test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
discovery_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
enterprise_billing feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592) 2026-07-15 12:12:52 -07:00
experimental/mcp_server test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
fine_tuning_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
google_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
guardrails Merge pull request #36355 from harryzhou2000/fix/responses-bridge-preserve-reasoning-input-items 2026-08-22 15:07:02 -07:00
health_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
hooks test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
image_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
logging_endpoints
management_endpoints fix(proxy): make /team/member_delete's four cleanups atomic (#37959) 2026-08-22 14:24:57 -07:00
management_helpers test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
memory test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
middleware feat(proxy): add Amazon Comprehend Medical passthrough provider 2026-08-17 15:44:06 -07:00
ocr_endpoints fix(ocr): validate body req_format in the proxy endpoint and run its tests in CI 2026-08-17 18:29:35 +00:00
openai_files_endpoint test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
pass_through_endpoints test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
policy_engine test(callbacks): unwind the callbacks global the policy engine and realtime tests scaffold around (#37826) 2026-08-21 21:19:35 -07:00
prompts test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
proxy_server fix: keep accepting a loop ceiling that spells a whole number 2026-08-22 11:24:22 -07:00
public_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
rag_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
realtime_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
response_api_endpoints fix(responses_api): map bridged chat usage on guardrail-blocked replies 2026-08-14 17:04:27 -07:00
shutdown
spend_tracking test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_configs
types_utils fix(proxy): resolve router_settings.plugins dotted paths and load plugins from installed packages (#33644) 2026-07-17 11:23:18 -07:00
ui_crud_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
utils fix(guardrails): run policy pipelines when the caller sends its own metadata (/v1/messages, Claude Code) (#36889) 2026-08-21 13:04:04 -07:00
vector_store_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
vector_store_files_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
video_endpoints test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
__init__.py
conftest.py test(proxy): keep a leaked llm_router out of the next test in the worker 2026-08-20 23:19:17 -07:00
test_aiohttp_cleanup_closed.py
test_aiohttp_session_recovery.py
test_api_key_masking_in_errors.py
test_audio_speech_prometheus_hooks.py
test_batch_expiry.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_metadata_none_fix.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_retrieve_bedrock.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_batch_x_litellm_model_encoding.py test(proxy): give the x-litellm-model fallback test deterministic openai env creds 2026-08-17 12:42:57 -07:00
test_blocked_response_usage.py fix(responses_api): map bridged chat usage on guardrail-blocked replies 2026-08-14 17:04:27 -07:00
test_budget_reservation.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_caching_routes.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_chat_completion_metadata.py
test_claude_code_marketplace.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_common_request_processing.py fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958) 2026-08-22 14:25:55 -07:00
test_component_allowlists.py feat(proxy): authenticate to Azure Postgres with Microsoft Entra ID tokens 2026-08-20 11:50:16 -07:00
test_conftest.py test(proxy): stop monkeypatch.undo re-planting fixture-mocked prisma_client 2026-08-13 20:11:06 -07:00
test_cors_config.py
test_custom_proxy.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_dynamic_mcp_route.py fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845) 2026-08-14 17:04:32 -07:00
test_empty_model_list.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_enforce_user_param.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_fallback_management_endpoints.py
test_fastapi_offline_routes.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_filter_models_by_team_access_group.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_health_check_functions.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_health_check_max_tokens.py
test_init_litellm_callbacks.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_langfuse_passthrough_security.py
test_lazy_openapi_snapshot.py feat(proxy): add GET /v1/indexes to list vector store indexes (#36289) 2026-08-10 15:09:59 -07:00
test_litellm_pre_call_utils.py fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958) 2026-08-22 14:25:55 -07:00
test_max_budget_env_var.py
test_mcp_asgi_response.py
test_model_based_routing_files_batches.py
test_model_deprecations_endpoint.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_model_dump_with_preserved_fields.py
test_model_id_header_propagation.py feat(proxy): add x-litellm-model-name response header with deployment model string (#33698) 2026-07-17 20:29:42 -07:00
test_model_info_default_limits.py
test_model_level_guardrails.py fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
test_model_list_healthy_only.py
test_modify_response_streaming_passthrough.py
test_openai_ws_passthrough_routes.py fix(proxy): close websocket cleanly when OpenAI credentials are missing 2026-08-16 14:40:37 -07:00
test_openapi_schema_validation.py
test_plugin_routes.py
test_pricing_field_strip.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prisma_engine_watchdog.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_prisma_migration.py fix(proxy): keep a failed prisma generate from failing the migration entrypoint (#37947) 2026-08-22 11:45:19 -07:00
test_prometheus_cleanup.py
test_provider_url_destination_guard.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_cli.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_logging_hook_detection.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_proxy_server.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_types.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_proxy_utils.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_pyroscope.py
test_read_model_list.py
test_redis_auth_cache_flag.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_response_model_sanitization.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_route_a2a_models.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_route_llm_request.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sensitive_route_auth.py
test_shared_health_check.py
test_spend_log_cleanup.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_swagger_chat_completions.py
test_team_member_update.py
test_team_org_move.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_tools_allowlist_enforcement.py
test_update_llm_router_resilience.py feat(complexity_router): custom classifier plugins via classifier_type 'custom' (#37249) 2026-08-18 14:09:19 -07:00