scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-29 01:42:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 79
litellm/tests
History
shrey kharbanda bf60fa3960 test(e2e): cover sail completion_window pricing and service_tier rewrite 
Live proxy suites for the sail provider added in #42840: chat with
service_tier flex, balanced and absent bills each window's cost-map
rates read at run time and echoes no service_tier; /v1/responses with
extra_body.metadata.completion_window balanced bills the _balanced
keys; a flex-only sail model refuses the asap and balanced windows
with sail's own 400 and leaves only a zero-spend failure row; openai
keeps its own service_tier and bills the tier it reports back.

MUTATIONS.md records the five production mutations, each killed, and
the final green run. It is review evidence and not meant to merge.
2026-09-24 16:34:59 +00:00
..
_support fix(params): stop stream_chunk_size reaching provider request bodies (#42664) 2026-09-23 10:27:47 -07:00
agent_tests fix(a2a): keep the upstream status on card discovery failures and inject the card client in tests 2026-09-16 23:55:57 +00:00
audio_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests feat(batches): enrich batch cost rows with breakdown, identity, session, and org spend 2026-09-03 17:23:53 -04:00
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests fix(sail): bill by the completion window that reaches the wire and keep it through extra_body merges 2026-09-24 07:29:18 +00:00
documentation_tests fix(docs-test): restore line-anchored table regex in router settings check 2026-09-18 20:10:20 +00:00
e2e test(e2e): cover sail completion_window pricing and service_tier rewrite 2026-09-24 16:34:59 +00:00
enterprise fix(projects): persist explicit budget cap clears 2026-09-12 13:43:55 -07:00
guardrails_tests Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context" 2026-09-19 17:45:15 +00:00
image_gen_tests test: remove fully commented-out test files that collect no tests 2026-09-17 20:05:27 +00:00
integration test(straiker): deterministic integration audit of the v3 platform relay (#42781) 2026-09-23 15:57:51 -07:00
litellm-proxy-extras Merge remote-tracking branch 'origin/main' into litellm_v2_migration_startup 2026-09-18 20:55:05 -07:00
litellm_utils_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
llm_responses_api_testing test: fix five tests left stale by #41311, #41337, #39996 and #41310 2026-09-16 17:47:27 -07:00
llm_translation test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
load_tests feat(proxy): per-worker admission control that rejects excess requests with 503 (#39352) 2026-09-03 18:19:04 -07:00
local_testing fix(redis): authenticate sync clusters with IAM credential providers (#40204) 2026-09-23 17:23:55 -05:00
logging_callback_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
mcp_tests fix(mcp): apply post-call rewrites without stale structured output (#41530) 2026-09-22 13:44:25 -07:00
multi_instance_e2e_tests test: say whether a match= pattern is a regex or a literal (ruff RUF043) 2026-08-21 16:25:33 -07:00
ocr_tests test(ocr): replace per-provider OCR test classes with a declarative provider x auth x input matrix 2026-09-18 21:27:03 +00:00
openai_endpoints_tests test(responses): fix stale Anthropic smoke request 2026-09-11 13:56:27 -07:00
otel_tests test: fix stale budget-status and bad-database-url assertions 2026-09-21 15:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099) 2026-09-21 21:21:43 -07:00
proxy_admin_ui_tests refactor(tests): assign the streamed id and lock poll once instead of rebinding 2026-09-03 13:49:43 -07:00
proxy_behavior Merge pull request #42026 from BerriAI/litellm_user_jwt_savings 2026-09-21 16:41:16 -07:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests fix(proxy): keep the in-flight daily spend batch when shutdown cancels the flush (#42593) 2026-09-22 17:58:08 -07:00
router_unit_tests Merge pull request #42283 from BerriAI/litellm_mid_stream_fallback_walks_full_list 2026-09-21 15:14:04 -07:00
rust-python-harness test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
search_tests test: remove fully commented-out test files that collect no tests 2026-09-17 20:05:27 +00:00
spend_tracking_tests
store_model_in_db_tests test(proxy): expect the sanitized unknown-model message in the spend-log error test 2026-09-11 19:18:10 -07:00
test_gateway feat(proxy): offload spend tracking to a pod-local collector sidecar (#40545) 2026-09-10 17:14:13 -07:00
test_litellm test(llm_http_handler): give mocked provider configs the real merge_extra_body 2026-09-24 07:51:25 +00:00
test_litellm_rust feat(cache): add a guarded native response-cache resolver foundation (#42769) 2026-09-23 13:54:10 -07:00
unified_google_tests test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520) 2026-09-22 13:07:37 -07:00
unit fix(bedrock): honour stream_chunk_size in Invoke streaming (#42686) 2026-09-23 11:25:00 -07:00
vector_store_tests fix(vector-store): carry request metadata into the Router executor built from the router kwarg 2026-09-02 16:53:05 -07:00
windows_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
__init__.py
_fake_openai_endpoint_server.py test(timeout): time out against the local fake endpoint instead of api.openai.com 2026-09-03 09:53:30 -07:00
_flush_vcr_cache.py
_live_test_helpers.py refactor(test): validate cost-map entries into a typed model 2026-09-16 14:32:08 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
_vcr_redis_persister.py
_wait_helpers.py test: replace blind sleeps with deadline waits in callback and caching tests (#37660) 2026-08-20 18:48:43 +00:00
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
test_fallbacks.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py test: fix staging CI regressions from #38182, #38144, #38265, #37962, and #37969 2026-08-25 23:01:20 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py test: address review notes on the chronic-test repairs 2026-09-04 10:18:15 -07:00
test_new_vector_store_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_openai_endpoints.py test: fix five tests left stale by #41311, #41337, #39996 and #41310 2026-09-16 17:47:27 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_resource_cleanup.py
test_rust_python_harness.py refactor(rust): remove gateway, config, router, realtime, and trace-parity infrastructure 2026-09-16 16:00:07 +00:00
test_service_logger_otel.py
test_spend_logs.py
test_team.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_team_logging.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_team_members.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_users.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.