litellm

mirror of https://github.com/BerriAI/litellm.git synced 2026-09-24 00:52:24 +00:00

History

devin-ai-integration[bot] 5c0b374f0a test(integration): regression tests for August provider translation and streaming bugs (#42621 ) * test(integration): Bedrock batch files upload completions and responses records as user messages (Pylon #6882) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): client Anthropic OAuth token never replaces Bedrock SigV4 authorization (Pylon #6888) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bridge /v1/messages and /v1/responses streams through empty-choices chunks (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prepend azure content-filter metadata chunk to the messages stream (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fireworks routers/ slug reaches the provider as accounts/fireworks/routers/<id> (Pylon #7030) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock hidden thinking tokens are not reported as text tokens (Pylon #7067) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai FLUX.2-flex image generation targets the flex provider path with the BFL body (Pylon #7092) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep Databricks streaming usage and cache reads in the client stream and spend log (Pylon #7094) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai-compatible image edits forward provider-specific form fields to the backend (Pylon #7122) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): replayed intercepted web search turn reaches Bedrock as text through /v1/messages (Pylon #7181) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed web search turn capped by max_agentic_loops ends the turn with snippets and ordered blocks (Pylon #7230) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock rerank keeps forwarded client headers out of the SigV4 signature (Pylon #7284) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai rerank authenticates with an Entra token when no api key is set (Pylon #7303) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): perplexity stream with cost breakdown object completes and bills total_cost (Pylon #7331) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai strips Anthropic message fields before the Foundry request (Pylon #7336) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): capped intercepted web search ends the turn without an internal tool_use block (Pylon #7378) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock passthrough converse-stream keeps event-stream content-type (Pylon #7482) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): v1/messages success exposes v3 priority rate limit headers (Pylon #7532) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): config deployment dropped by a stale boot cost map is restored after reload (Pylon #7564) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prefixed Opus 4.8 reasoning_effort reaches Bedrock as adaptive thinking (Pylon #7586) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): dashscope chat forwards reasoning_effort to the provider (Pylon #7606) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): failing stream logging callback still releases the max_parallel_requests slot (Pylon #7608) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gen 5 Claude Bedrock Invoke tool search sends the Bedrock beta field (Pylon #7642) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): databricks ai gateway api_base requests OAuth token from workspace origin (Pylon #7724) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): deepseek vision image content list reaches the provider unchanged (Pylon #7729) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle context overflow surfaces as 400 prompt is too long (Pylon #7732) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex history items reach Bedrock Mantle as supported Responses input types (Pylon #7783) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): chat over responses deployment returns finish_reason length when output tokens run out (Pylon #7784) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock_mantle rewrites Codex history items before the Responses wire (Pylon #7812) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor sub-call on /v1/messages uses the configured advisor deployment (Pylon #7828) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tencent thinking reaches the provider body instead of failing the request (Pylon #7834) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): xAI chat web search reaches /v1/responses with instructions and nested filters (Pylon #7835) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send Bedrock Converse config blocks once at top level (Pylon #7839) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock converse sends gpt-5 reasoning_effort as reasoning.effort (Pylon #7850) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock cohere.embed-english-v3 embeddings accept encoding_format and dimensions (Pylon #7963) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed chat completions emit SSE keepalive pings while the upstream is silent before its first token (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format the TTFT keepalive regression test (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai chat drops tool_choice when the request has no tools (Pylon #8022) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): stream whose first chunk has no choices falls back and bills the fallback (Pylon #8006) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583)" This reverts commit `864b65811f`. * test(integration): reconcile contracts manifest for bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): isolate proxy config writes in bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review feedback on keepalive, cost map reload and websearch order Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>		2026-09-22 20:52:00 -07:00
..
agent_tests	fix(a2a): keep the upstream status on card discovery failures and inject the card client in tests	2026-09-16 23:55:57 +00:00
audio_tests
base_sdk_tests	fix(mcp): explain missing public client dependencies	2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests
benchmarks	feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174 )	2026-09-22 04:41:11 +00:00
code_coverage_tests	feat(logger): dispatch Python logging through the Rust diagnostics processor (#42616 )	2026-09-22 18:44:15 -07:00
documentation_tests	fix(docs-test): restore line-anchored table regex in router settings check	2026-09-18 20:10:20 +00:00
e2e	test(e2e): tolerate provider-side flakes on five full-suite cells (#42628 )	2026-09-22 20:12:14 -07:00
enterprise
guardrails_tests	Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context"	2026-09-19 17:45:15 +00:00
image_gen_tests	test: remove fully commented-out test files that collect no tests	2026-09-17 20:05:27 +00:00
integration	test(integration): regression tests for August provider translation and streaming bugs (#42621 )	2026-09-22 20:52:00 -07:00
litellm-proxy-extras	Merge remote-tracking branch 'origin/main' into litellm_v2_migration_startup	2026-09-18 20:55:05 -07:00
litellm_utils_tests	test: point CircleCI-only suites at models still in the cost map (#42617 )	2026-09-22 17:28:34 -07:00
llm_responses_api_testing	test: fix five tests left stale by #41311 , #41337 , #39996 and #41310	2026-09-16 17:47:27 -07:00
llm_translation	test: point CircleCI-only suites at models still in the cost map (#42617 )	2026-09-22 17:28:34 -07:00
load_tests
local_testing	fix: repair seven regressions caught by CircleCI on main (#42640 )	2026-09-23 02:26:36 +00:00
logging_callback_tests	test: point CircleCI-only suites at models still in the cost map (#42617 )	2026-09-22 17:28:34 -07:00
mcp_tests	fix(mcp): apply post-call rewrites without stale structured output (#41530 )	2026-09-22 13:44:25 -07:00
multi_instance_e2e_tests
ocr_tests	test(ocr): replace per-provider OCR test classes with a declarative provider x auth x input matrix	2026-09-18 21:27:03 +00:00
openai_endpoints_tests
otel_tests	test: fix stale budget-status and bad-database-url assertions	2026-09-21 15:00:54 -07:00
pass_through_tests	fix(mcp): preserve legacy behavior on SDK2 and streamline verification	2026-09-18 22:28:31 -07:00
pass_through_unit_tests	feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099 )	2026-09-21 21:21:43 -07:00
proxy_admin_ui_tests
proxy_behavior	Merge pull request #42026 from BerriAI/litellm_user_jwt_savings	2026-09-21 16:41:16 -07:00
proxy_e2e_anthropic_messages_tests	fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1	2026-09-19 23:33:29 +00:00
proxy_migration_tests	test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570 )	2026-09-22 14:59:26 -07:00
proxy_security_tests	refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits	2026-09-19 18:53:14 -07:00
proxy_unit_tests	fix(proxy): keep the in-flight daily spend batch when shutdown cancels the flush (#42593 )	2026-09-22 17:58:08 -07:00
router_unit_tests	Merge pull request #42283 from BerriAI/litellm_mid_stream_fallback_walks_full_list	2026-09-21 15:14:04 -07:00
rust-python-harness	test: drop tests that pin provider-owned cost map values	2026-09-18 03:55:51 +00:00
search_tests	test: remove fully commented-out test files that collect no tests	2026-09-17 20:05:27 +00:00
spend_tracking_tests
store_model_in_db_tests
test_gateway
test_litellm	fix(pricing): drop the unpublished cached rate from the Gemini Live preview entries (#42651 )	2026-09-22 20:35:34 -07:00
test_litellm_rust	refactor(rust): align the cache crates with Python and wire every native backend (#42530 )	2026-09-22 13:13:02 -07:00
unified_google_tests	test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520 )	2026-09-22 13:07:37 -07:00
unit	fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse (#42644 )	2026-09-22 20:09:21 -07:00
vector_store_tests
windows_tests
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py	refactor(test): validate cost-map entries into a typed model	2026-09-16 14:32:08 -07:00
_openai_record_replay_proxy.py
_process_helpers.py	test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570 )	2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py	test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542 )	2026-09-22 14:20:18 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md	ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green	2026-09-20 07:05:42 +00:00
capturing_transport.py	test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542 )	2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py	test: fix five tests left stale by #41311 , #41337 , #39996 and #41310	2026-09-16 17:47:27 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_rust_python_harness.py
test_service_logger_otel.py
test_spend_logs.py
test_team.py
test_team_logging.py
test_team_members.py
test_users.py

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.