litellm/tests
yuneng-jiang 635a718ba1
ci: cut CircleCI wall time without loosening test isolation (#43347)
* ci: cut CircleCI wall time without loosening test isolation

* fix(ci): parse integration split files that follow --results

The CircleCI machine image ships Python 3.12.2, whose argparse leaves the
files positional empty when it follows an option and another positional, so
every extensions node exited with 'unrecognized arguments'. Reproduced on
3.12.2; parse_intermixed_args selects the files on 3.12.2, 3.12.13 and 3.13

* test(ci): resolve command references in the Rust toolchain guard

The Windows rustup install moved into the install_windows_toolchain command,
which the guard only recognized for install_rust. It now accepts any command
that installs a pinned rustup and reads the Windows toolchain pin from it

* ci: cache the Windows release cargo build from main

windows_release_wheel rebuilt every dependency with fat LTO on each run. It now
restores the release target and cargo registry saved by main's scheduled run,
drops the workspace crates' fingerprints so they always rebuild from the
checked-out source, and still runs the full LTO link

* ci: run the Windows release wheel build on windows.xlarge

The fat-LTO release build is the slowest job in the pipeline; more cores
speed up the dependency compile ahead of the final link

* ci: skip the Windows fingerprint cleanup when the cargo cache missed

On a cold cache the release fingerprint directory does not exist, and the
CircleCI PowerShell wrapper failed the step on the suppressed not-found error
2026-09-26 15:34:53 -07:00
..
_support fix(params): stop stream_chunk_size reaching provider request bodies (#42664) 2026-09-23 10:27:47 -07:00
agent_tests
audio_tests
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests ci: fail on new unbounded SQL IN lists and add a Prisma chunking helper (#42629) 2026-09-26 13:40:44 -07:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
guardrails_tests Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context" 2026-09-19 17:45:15 +00:00
image_gen_tests fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 2026-09-24 13:01:12 -07:00
integration ci: cut CircleCI wall time without loosening test isolation (#43347) 2026-09-26 15:34:53 -07:00
litellm_utils_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
llm_responses_api_testing test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
llm_translation test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
load_tests
local_testing test: remove substring guard test_default_api_base (#43355) 2026-09-26 22:28:28 +00:00
logging_callback_tests test(logging): drain the logging worker after each logging callback test so no later test inherits its events (#43344) 2026-09-26 14:08:26 -07:00
mcp_tests ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904) 2026-09-24 23:07:48 +00:00
multi_instance_e2e_tests
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests
otel_tests test: fix stale budget-status and bad-database-url assertions 2026-09-21 15:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
proxy_admin_ui_tests
proxy_behavior test(proxy_behavior): scope the management proxy fixture to its package so its spend monitor cannot race the spend tests (#43302) 2026-09-26 03:12:13 -07:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
router_unit_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) 2026-09-25 19:12:36 -07:00
spend_tracking_tests
store_model_in_db_tests test(integration): port langfuse callbacks-in-db coverage to the local harness (#43282) 2026-09-25 21:54:11 -07:00
test_litellm feat(proxy): add maximum_daily_tag_spend_retention_period cleanup setting (#39221) 2026-09-26 15:07:56 -07:00
test_litellm_rust test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
unified_google_tests test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520) 2026-09-22 13:07:37 -07:00
unit ci: cut CircleCI wall time without loosening test isolation (#43347) 2026-09-26 15:34:53 -07:00
vector_store_tests
windows_tests ci: cut CircleCI wall time without loosening test isolation (#43347) 2026-09-26 15:34:53 -07:00
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py fix(proxy): authorize key model aliases the same way as team aliases (#43049) 2026-09-24 19:51:02 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_rust_python_harness.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
test_service_logger_otel.py
test_spend_logs.py
test_team.py
test_team_logging.py
test_team_members.py
test_users.py
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.