scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-26 01:12:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 99
litellm/tests
History
devin-ai-integration[bot] a286ebf42e
test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 
* test(integration): optional Anthropic tool properties stay optional on the OpenAI Responses wire (Pylon #6619)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Bedrock InvokeModel count-suffixed cache usage fields are reported and charged (Pylon #6708)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): anthropic messages honors the deployment request timeout (Pylon #6505)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): drop client_metadata before the Bedrock Converse body reaches the provider (Pylon #6645)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): repeat Bedrock requests under one session name assume the role once (Pylon #6681)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): messages stream keeps include_usage off the Responses wire with always_include_stream_usage (Pylon #6466)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): clamp sub-16 max_tokens to the Responses API floor instead of 400 (Pylon #6539)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): forwarded client x- headers reach the provider on /v1/responses (Pylon #6565)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Anthropic messages stop_sequences reach OpenAI-compatible providers as stop (Pylon #6536)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Codex namespace tools reach a chat upstream flattened and round-trip through /v1/responses (Pylon #6409)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): keep Claude 4.6 legacy thinking budget_tokens on /v1/messages (Pylon #6727)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): nvidia nim ranking keeps image passages and applies top_n without sending top_k (Pylon #6401)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): tpm-only model rejects priority traffic once recorded tokens reach the model tpm (Pylon #6344)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): reasoning-only chunks open an Anthropic thinking block at index zero on /v1/messages streams (Pylon #6337)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): file content streams to the client before the upstream finishes sending (Pylon #6315)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): agent whose card lives only at agentCard/v1.0 is reached with bearer auth (Pylon #6249)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): vertex batch create returns a batch when outputInfo is null (Pylon #6374)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): fireworks session id is sent as x-session-affinity and cached tokens land in spend log metadata (Pylon #6220)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): advisor sub-call failure does not cool down the executor deployment (Pylon #6212)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Gemini /v1/messages cache_control creates cachedContent with Anthropic ttl (Pylon #6221)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bedrock_mantle max_output_tokens below 16 is clamped before reaching Mantle (Pylon #6262)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): missing thinking signature 400 on /v1/messages retries without thinking blocks (Pylon #6222)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): format Gemini messages cache_control wire test (Pylon #6221)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): rebuilt shared aiohttp session keeps the configured keepalive timeout (Pylon #6387)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): sagemaker_chat signs the inference component header and sends hf_model_name as the body model (Pylon #6187)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Bedrock Converse DeepSeek drops Anthropic thinking and sends V3 reasoning_effort raw (Pylon #6149)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): concurrent team model TPM requests are reserved before the provider call (Pylon #6075)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): /v1/messages honors the configured timeout against a stalled upstream (Pylon #6025)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Codex additional_tools input items reach Bedrock Mantle as top-level tools (Pylon #6012)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): advisor api_base without api_key never sends the proxy Anthropic key to the caller host (Pylon #6226)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): rerank responses carry call id, latency and cost headers (Pylon #5981)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): parse the outbound Anthropic body with the typed JSON adapter (Pylon #6025)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Bedrock Knowledge Base search forwards userContext to the Retrieve body (Pylon #5991)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): Marengo 3.0 text embeddings reach Bedrock nested under inputType (Pylon #5949)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): sub-16 max_tokens over a responses deployment reaches OpenAI as 16 (Pylon #6008)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): midturn system correction reaches the OpenAI Responses wire via /v1/messages (Pylon #6449)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): concurrent requests over a key tpm limit are rejected before reaching the provider (Pylon #5737)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): chat to responses bridge keeps deployment AWS credentials for Bedrock Mantle SigV4 (Pylon #5870)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): vertex gemini stream split across many fragments completes without stalling the proxy (Pylon #5838)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bedrock mantle /v1/messages stream keeps stream true and relays SSE events (Pylon #5596)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): large chat payloads are released from worker memory after the request ends (Pylon #5920)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): streaming success logs v3 rate limit remaining values for callbacks (Pylon #5767)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): a database created search tool backs Anthropic web search interception (Pylon #5669)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): register july provider regression contracts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): make july provider regression tests deterministic under cache and worker sharing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): drop order-fragile worker memory probe pending a real retention regression check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): apply ruff import sorting and formatting

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): drop stale contract entry and pass question to advisor executor

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): use tiktoken-backed executor model in advisor tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 09:51:06 -07:00
..
agent_tests fix(a2a): keep the upstream status on card discovery failures and inject the card client in tests 2026-09-16 23:55:57 +00:00
audio_tests
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests feat(logger): dispatch Python logging through the Rust diagnostics processor (#42616) 2026-09-22 18:44:15 -07:00
documentation_tests fix(docs-test): restore line-anchored table regex in router settings check 2026-09-18 20:10:20 +00:00
e2e test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711) 2026-09-23 07:48:46 -07:00
enterprise
guardrails_tests Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context" 2026-09-19 17:45:15 +00:00
image_gen_tests test: remove fully commented-out test files that collect no tests 2026-09-17 20:05:27 +00:00
integration test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
litellm-proxy-extras Merge remote-tracking branch 'origin/main' into litellm_v2_migration_startup 2026-09-18 20:55:05 -07:00
litellm_utils_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
llm_responses_api_testing test: fix five tests left stale by #41311, #41337, #39996 and #41310 2026-09-16 17:47:27 -07:00
llm_translation test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
load_tests
local_testing fix: repair seven regressions caught by CircleCI on main (#42640) 2026-09-23 02:26:36 +00:00
logging_callback_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
mcp_tests fix(mcp): apply post-call rewrites without stale structured output (#41530) 2026-09-22 13:44:25 -07:00
multi_instance_e2e_tests
ocr_tests test(ocr): replace per-provider OCR test classes with a declarative provider x auth x input matrix 2026-09-18 21:27:03 +00:00
openai_endpoints_tests
otel_tests test: fix stale budget-status and bad-database-url assertions 2026-09-21 15:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099) 2026-09-21 21:21:43 -07:00
proxy_admin_ui_tests
proxy_behavior Merge pull request #42026 from BerriAI/litellm_user_jwt_savings 2026-09-21 16:41:16 -07:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests fix(proxy): keep the in-flight daily spend batch when shutdown cancels the flush (#42593) 2026-09-22 17:58:08 -07:00
router_unit_tests Merge pull request #42283 from BerriAI/litellm_mid_stream_fallback_walks_full_list 2026-09-21 15:14:04 -07:00
rust-python-harness test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
search_tests test: remove fully commented-out test files that collect no tests 2026-09-17 20:05:27 +00:00
spend_tracking_tests
store_model_in_db_tests
test_gateway
test_litellm feat(gemini): add gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts prices (#42752) 2026-09-23 09:28:09 -07:00
test_litellm_rust feat(secrets): route secret resolution through native Rust backends (#42619) 2026-09-23 08:24:57 -07:00
unified_google_tests test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520) 2026-09-22 13:07:37 -07:00
unit fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse (#42644) 2026-09-22 20:09:21 -07:00
vector_store_tests
windows_tests
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py refactor(test): validate cost-map entries into a typed model 2026-09-16 14:32:08 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py test: fix five tests left stale by #41311, #41337, #39996 and #41310 2026-09-16 17:47:27 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_rust_python_harness.py
test_service_logger_otel.py
test_spend_logs.py
test_team.py
test_team_logging.py
test_team_members.py
test_users.py

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.