mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
75613bf22f
* test: add regression coverage for twelve closed issues Adds targeted regression tests for behavior that was fixed but left ungated, so the fixes cannot silently regress: - #33772 openai cache_write_tokens cost - #34309 Responses API cache cost_breakdown - #35363 /v1/responses batch spend - #36619 auto-router api_base/api_key leak on a shared model name - #35359 batch fallbacks within the owning model group - #36523 passthrough streamed Responses spend log - #36646 passthrough embeddings spend log - #37147 non-object metadata on create_batch is a 400 - #35362 unscoped list files reads the managed-file store - #33221 gpt-5.6 bridges to Responses on function tools alone - #34487 LLM complexity classifier runs for every caller metadata shape - #35124 streamed /v1/messages emits success logging on both bridges Cost assertions read rates from litellm.model_cost rather than hardcoding dollar amounts, so they do not drift on repricing. * fix: stop the new regression tests polluting and tripping over shared global state Two shard failures, both from global state the new tests share with their neighbours rather than from the behaviour under test. test_main.py's local_cost_map pinned litellm.model_cost but left the get_model_info lru_cache warm, so completion_cost billed at whatever prices were cached earlier in the process while the assertions read the pinned map. Clear the cache on both sides of the fixture, matching the local_model_cost_map fixture in tests/test_litellm/conftest.py. The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush() on whatever queue happened to be around. A queue left non-empty by an earlier test is still bound to that test's loop, so join() either hangs or raises "bound to a different event loop". Rebind to the running loop before the call and wait for the captured payload instead of a fixed sleep. |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.