scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-06 07:48:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 112
litellm/tests/unit
History
quietjules 5a392b9e5f
fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 
* fix(anthropic): stop forcing the response_format json tool when Claude will think

A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.

AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.

The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.

* fix(bedrock): route native output and thinking token counts by whether Claude will think

Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.

* fix(anthropic): tell Claude to answer through the unforced response_format tool

Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output

* fix(anthropic): keep open-object schemas on the json tool when Claude will think

Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main

* fix(anthropic): hold streamed prose until the JSON tool decides the answer

In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.

* test(anthropic): build the stream test events without nested comprehensions

* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only

* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model

* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced

Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content

* fix(anthropic): drop the thinking False forcing case that completion never reaches

litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default

* test(bedrock): collect the Converse json_schema stream chunks in one assignment

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-05 04:07:11 +00:00
..
a2a_protocol fix(a2a)!: bill completion-bridge agents at their own prices on /a2a send and stream (#49150) 2026-10-30 19:43:45 -07:00
accounting feat(accounting): add opt-in SpendLogs ledger (#49856) 2026-11-04 17:49:15 -08:00
anthropic_interface fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) 2026-11-03 01:53:29 +00:00
batch_completion test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
batches fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed (#49824) 2026-11-04 20:22:06 +00:00
caching feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393) 2026-11-02 13:26:29 -08:00
completion_extras fix(responses): keep the system prompt in input for json_object on the chat to responses bridge (#49957) 2026-11-04 13:55:50 -08:00
compression test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
containers refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
cookbook feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
db_scripts fix(vector-stores): keep proxy context out of provider vector store metadata (#47021) 2026-10-17 12:51:47 -07:00
decisions fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) 2026-11-05 00:30:12 +00:00
endpoints feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
enterprise fix(managed_files): keep the upload the managed id encodes when two target models share a deployment (#49420) 2026-11-03 00:58:41 +00:00
expected_fine_tuning_api test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
expected_responses_api_request test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
experimental_mcp_client fix(mcp): load deprecated package aliases on demand (#48615) 2026-10-27 19:46:19 +00:00
extraction feat(extraction): add /v1/extraction with Cloudflare Clairic support (#47473) 2026-10-20 15:51:45 -07:00
files fix(hosted_vllm): accept the vLLM production-stack router's file and batch objects (#49068) 2026-10-30 23:11:34 -07:00
fine_tuning fix(fine_tuning): stop forwarding api_version to the OpenAI client (#44995) 2026-10-21 01:58:25 +00:00
fixtures test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
gateway fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping (#45039) 2026-10-29 10:25:14 -07:00
google_genai fix(google_genai): read one-object and null spellings of repeated fields in countTokens and generateContent (#48771) 2026-10-28 21:46:06 -07:00
harness fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) 2026-10-21 02:26:05 +00:00
images feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
integration_support test(mcp): cancel disconnected HTTP tool handlers (#49436) 2026-10-31 16:41:49 -07:00
integrations fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863) 2026-11-04 20:44:55 +00:00
interactions fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
litellm_core_utils fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
litellm_proxy_extras feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961) 2026-10-29 09:39:26 -07:00
llms fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
mcp_client feat(mcp): add an opt-in per-server setting to skip MCP tool output validation (#49961) 2026-11-03 20:42:10 -08:00
models fix(proxy): allow model instances in LiteLLM_EndUserTable validation (#41677) 2026-10-15 10:57:26 -07:00
ocr test(rust-bridge): relocate dispatch tests before routing changes (#46926) 2026-10-16 12:29:33 -07:00
passthrough fix(vllm): send the pass-through key as a bearer on both /vllm routes (#46861) 2026-10-30 08:57:48 -07:00
proxy feat(proxy): add model deprecation alerts with daily requests by key (#50022) 2026-11-05 04:03:42 +00:00
rag fix(rag)!: use team provider credentials and allow viewer and JWT team access (#45497) 2026-10-17 23:51:48 +00:00
realtime_api feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
repositories feat(accounting): add opt-in SpendLogs ledger (#49856) 2026-11-04 17:49:15 -08:00
rerank_api test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
responses fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
router_strategy fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) 2026-11-05 00:30:12 +00:00
router_utils fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863) 2026-11-04 20:44:55 +00:00
rust_bridge fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
sandbox fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
search fix(search): price Tavily advanced and Linkup deep searches at their own cost map entries (#47411) 2026-10-19 23:32:31 -07:00
secret_managers fix(proxy): let a config license unlock enterprise settings and spell "environment" right in license errors (#49745) 2026-11-03 09:56:08 +00:00
skills feat(skills): add a litellm-setup agent skill (#47731) 2026-10-28 15:44:43 -07:00
systemone fix(databricks): reuse base OAuth for decisions (#48395) (#48527) 2026-10-26 20:18:16 -07:00
telemetry feat(telemetry): add AggregatingSink, fixed-bucket histograms, Report and HttpExporter (#45487) 2026-10-13 18:33:01 -07:00
test_router fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) 2026-11-05 00:30:12 +00:00
text_completions feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393) 2026-11-02 13:26:29 -08:00
tracing fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344) 2026-10-16 00:53:52 +00:00
types fix(responses)!: authorize and decode comparison response IDs (#49402) 2026-11-03 03:40:46 +00:00
vector_stores fix(exceptions)!: honor status code when mapping OpenAI invalid_request_error exceptions (#32537) 2026-10-29 08:05:47 -07:00
videos feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785) 2026-11-02 21:28:46 -08:00
__init__.py test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
AGENTS.md feat(vector_stores): add OpenSearch k-NN search as a vector store provider (#47759) 2026-10-29 08:15:05 -07:00
conftest.py fix(anthropic): send an https PDF image_url to Claude as a document (#49708) 2026-11-03 07:05:15 +00:00
create_mock_standard_logging_payload.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_a2a_registry_lookup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_acompletion_session_reuse_e2e.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_add_deployment_no_master_key.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_aembedding_session_reuse_e2e.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_anthropic_beta_gated_fields.py fix(anthropic)!: drop beta-gated body fields under drop_params (#48793) 2026-10-29 06:25:31 -07:00
test_anthropic_beta_headers_filtering.py feat(anthropic): pick the beta allowlist row per deployment and surface input_transformations (#49689) 2026-11-03 01:55:26 +00:00
test_anthropic_skills_transformation.py feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
test_assert_ci_coverage.py docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI (#46796) 2026-10-16 14:57:49 -07:00
test_assert_secret_absent.py ci: run the integration groups on GitHub Actions (#48985) 2026-10-29 12:50:08 -07:00
test_assert_workflow_dir_hygiene.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_audio_transcription_rust_bridge.py fix(bedrock): omit system prompt from audio transcription and report its token usage (#41565) 2026-10-28 19:23:15 -07:00
test_azure_ad_token_credential_resolution.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_6_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_baseten_glm_5_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_batch_completion_models_all_responses.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_bedrock_marengo_embed_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_build_core_distribution.py test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
test_cerebras_llama_model_metadata.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_chat_ui_responses_session.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_licenses.py feat(proxy): embed enterprise LiteAdmin MCP in LiteLLM images (#44610) 2026-10-05 15:24:34 -07:00
test_check_mcp_operation_boundary.py refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
test_check_migrations_lock_timeout.py fix(proxy-extras): bound migration lock waits with SET LOCAL lock_timeout in the migration SQL (#46329) 2026-10-13 20:51:03 -07:00
test_check_migrations_no_data_rewrites.py fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948) 2026-10-01 14:12:22 -07:00
test_check_openapi_docs.py feat(skills): upload local skill ZIPs from the dashboard (#50008) 2026-11-04 00:56:40 -08:00
test_check_py310_typing_imports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_test_quality.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_check_type_discipline.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_check_unbounded_in_lists.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_check_windows_wheel_install.py test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
test_circleci_checkout.py ci(circleci): clone the public repo over HTTPS so main jobs stop failing at checkout (#49394) 2026-10-31 02:53:54 -07:00
test_circleci_path_filter.py refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
test_circleci_run_migration_tests.py test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
test_circleci_rust_toolchain.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_claude_fable_5_config.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_claude_haiku_5_5_config.py fix(model_prices): consolidate claude-haiku-5-5 over-100k pricing and capability flags (#45151) 2026-10-07 14:14:47 -07:00
test_claude_opus_4_6_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_4_8_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_5_config.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_claude_sonnet_5_config.py refactor(cost-map): remove openrouter rows, part 5 of 5 (#45669) 2026-10-09 22:48:55 +00:00
test_cloudflare_workers_ai_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_completion_timeout_resolution.py fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) 2026-10-17 22:08:02 +00:00
test_component_entrypoint.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_compression.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_conftest_isolation.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_constants.py fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054) 2026-10-29 20:32:04 -07:00
test_container_router.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculation_log_level.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculator.py fix(cost_calculator): keep regional and speed multipliers for per-call custom prices and log the billed per-call entry (#50063) 2026-11-04 15:47:57 -08:00
test_cost_map_guard.py fix(cost_map): mark fireworks_ai/minimax-m3 as vision and pin capabilities verified against live calls (#43390) 2026-10-29 00:23:11 -07:00
test_count_tokens_public_api.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_dashscope_image_generation.py feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
test_daybreak_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_deepseek_model_metadata.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_default_branch.py feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex (#48949) 2026-10-29 18:15:44 -07:00
test_detect_changes.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_docker_no_network_on_deploy.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_dockerfile_apk_repository.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_bedrock_realtime_extra.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_non_root.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_drop_params_env_var.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_e2e_egress_sentinel.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_eager_tiktoken_load.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_elevenlabs_tts_pricing.py fix(cost-map): rolling registry audit, add elevenlabs api rates, together minimax-m3.1-flash, mistral voxtral 3 and azure_ai phi-4 dates (#47623) 2026-10-23 14:57:35 -07:00
test_empty_mock_response.py fix: honor empty mock completion responses (#48162) (#48266) 2026-10-28 00:56:45 +00:00
test_env_key_doc_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_exports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_header_preservation.py fix(openai): keep the upstream error body on e.response for OpenAI and Azure errors (#49079) 2026-10-29 23:30:02 -07:00
test_exception_mapping_request_attribute.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_fake_openai_endpoint.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_filter_out_litellm_params.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
test_fireworks_serverless_model_costs.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gate_slot_lock.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_3_1_flash_lite_image_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_4_argon_model_metadata.py feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) 2026-10-15 22:54:23 +00:00
test_gemini_tts_native_audio_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_get_blog_posts.py chore(blog): refresh the bundled blog posts backup with the latest 50 posts from the docs RSS feed (#49971) 2026-11-03 23:41:15 +00:00
test_git_hooks.py chore(hooks): drop Conventional Branches pre-push hook in favor of litellm_ prefix (#44951) 2026-10-20 00:35:39 +00:00
test_gpt_5_4_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_5_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_image_cost_calculator.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_realtime_mode.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_groq_streaming_encoding.py fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
test_guardrail_exception_status_codes.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_guardrails.py Add guardrails.apply SDK helper (#49217) (#49469) 2026-10-31 23:37:45 +00:00
test_integration_run.py test(integration): isolate every multi-worker integration suite by default (#46027) 2026-10-12 03:46:06 -07:00
test_internal_context.py refactor(cache): remove experimental Redis and v2 caches (#50005) 2026-11-04 04:54:08 +00:00
test_lazy_imports.py fix(health)!: openai realtime health check reports healthy with a missing or invalid api key (#41233) 2026-10-29 00:40:02 -07:00
test_lint_base_counts.py revert(lint-gates): accept an empty base scan again, since zero violations is a legitimate count (#45319) 2026-10-08 08:40:08 +00:00
test_lint_workflow_diff_gates.py ci(unit): stop the full history clone on the db-and-spend shard, fetch the schema merge base by sha (#50120) 2026-11-04 22:40:38 +00:00
test_litellm_cli.py fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping (#45039) 2026-10-29 10:25:14 -07:00
test_litellm_overhead.py test(unit): wait on the work itself instead of fixed sleeps (#46261) 2026-10-13 12:41:06 -07:00
test_litellm_params_reserved_keys.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_logging.py feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708) 2026-10-29 03:47:04 -07:00
test_lowest_latency_zero_tokens.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_main.py fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
test_main_module_header.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_large_4_model_metadata.py fix(cost-map): price Mistral Large 4 at its list price now that the launch sale ended (#48825) 2026-10-28 18:12:55 +00:00
test_mistral_medium_3_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_small_4_0_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_zai_glm_5_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_block_unblock.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_model_cost_aliases.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_param_helper.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_model_prices_and_context_window.py fix(vertex-ai): sync tier prices and retirement floors (#49964) 2026-11-04 00:03:54 +00:00
test_model_response_normalization.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_1_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_3_model_metadata.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_mutation_report.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_nested_drop_params.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_no_top_level_test_invocations.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_non_chat_routes_open_llm_spans.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_openai_embedding_encoding_format_default.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_openai_record_replay_proxy.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_openai_service_tier_long_context_pricing.py fix(cost_calculator): bill ultrafast prompts above 272k at the ultrafast long-context rates (#43764) 2026-09-30 07:32:16 -07:00
test_openapi_docs_gate.py fix(ci): build base native extension for OpenAPI lint (#49136) 2026-10-29 16:56:44 -07:00
test_package_layout.py test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
test_package_metadata.py chore(deps): require anyio>=4.14.0 to rule out the lock-waiter deadlock (#47378) 2026-10-19 12:37:47 -07:00
test_pre_commit_lint.py ci: run the dashboard API types and Terraform endpoint checks on every change that can alter the spec (#49022) 2026-10-29 07:19:06 -07:00
test_prisma_generate_if_needed.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_private_usage_aliases.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_process_helpers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_alias_tracking.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_tags_pydantic.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_proxy_auth.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_pydantic_namespaces.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_rag_openai_ingestion.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_rate_limit_error_unification.py refactor(proxy): expose public names for private proxy helpers (#45170) 2026-10-07 18:59:27 -07:00
test_read_rc_version.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_redact_string_in_error_paths.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_redis.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_redis_credential_provider.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_register_model_custom_pricing.py fix(main)!: carry the full per-call price entry on the logging object instead of writing litellm.model_cost (#49532) 2026-11-03 16:03:34 -08:00
test_register_model_zero_cost_persistence.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_replicate_model_key_format.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_repo_agent_skills.py feat(skills): add a litellm-setup agent skill (#47731) 2026-10-28 15:44:43 -07:00
test_responses_api_bridge_non_stream.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_responses_id_security.py fix(responses)!: authorize and decode comparison response IDs (#49402) 2026-11-03 03:40:46 +00:00
test_responses_streaming_container_ownership.py fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703) 2026-10-15 11:00:14 -07:00
test_retrieve_batch_bedrock_dispatch.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_router_block_helpers.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_router_exception_redaction.py fix(streaming)!: map stream read timeouts to litellm.Timeout on chat, messages, and responses (#47198) 2026-10-18 13:58:34 +00:00
test_router_get_settings.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_router_google_genai.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_model_cost_isolation.py fix(router): keep a deployment's token limits off the shared backend cost map key (#49952) 2026-11-03 16:14:44 -08:00
test_router_order_fallback.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
test_router_per_deployment_num_retries.py fix(images): honor max_retries and prevent Router retry amplification (#48705) 2026-10-28 10:46:47 -07:00
test_router_redis_init.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_backoff_headers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_non_retryable_errors.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_policy_update.py fix(proxy): unset DB environment variables removed from the row on every worker by row revision (#46709) 2026-10-17 03:51:05 +00:00
test_router_silent_experiment.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_router_streaming_fallback_metadata.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_router_weighted_failover.py fix(router): stamp the failed deployment on every endpoint's attempt (#49941) 2026-11-04 11:08:21 -08:00
test_ruff_strict_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_sambanova_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_scaledown_model_prices.py feat(scaledown): add native text provider support (#44168) 2026-10-10 14:01:17 -07:00
test_scheduler.py fix(scheduler): read shared queue from Redis so expired dead-replica entries stop blocking waiters (#49387) 2026-10-31 11:43:46 -07:00
test_secret_redaction.py refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
test_seed_tracing_fixtures.py refactor(lens): connect LiteLLM to the independent Lens service (#45529) 2026-10-09 16:17:05 -07:00
test_select_ui_test_scope.py fix(ci): measure PR gates from the target branch's current tip (#49078) 2026-10-29 18:28:16 -07:00
test_service_logger.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_setup_wizard.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_shared_session_integration.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_socket_policy.py test(unit): block external sockets at import time and add a socket policy regression test 2026-09-20 08:07:50 +00:00
test_spend_logs_guard.py fix(ci): flag any schema.prisma change inside model LiteLLM_SpendLogs in the spend logs guard (LIT-10240) (#48905) 2026-10-28 15:52:15 -07:00
test_ssl_verify_unit.py test(unit): stop test modules from putting their own directory on sys.path (#43421) 2026-09-28 12:20:20 -07:00
test_stream_chunk_builder_annotations.py fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
test_stream_chunk_builder_citations.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_stream_chunk_builder_images.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_streaming_connection_cleanup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_system_message_format_bug.py fix(prompt_templates): do not mutate messages in-place in function_call_prompt (#41676) 2026-10-15 14:37:11 -07:00
test_test_quality_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_thinking_enabled.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_together_ai_model_metadata.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_tool_loop.py fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) 2026-10-29 00:56:02 -07:00
test_type_check_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_type_discipline_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_typesafe_model_metadata.py fix(cost): price typesafe/jev-1.14.0 (#46594) 2026-10-14 20:27:35 -07:00
test_unit_shard_missing_paths.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_unit_shard_per_test_timeout.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_utils.py fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
test_utils_get_model_info.py fix(bedrock): price claude_platform calls from their own cost map rows (#48150) 2026-10-24 13:54:18 -07:00
test_utils_get_optional_params.py fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761) 2026-10-29 04:46:29 -07:00
test_utils_module_docstring.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_uuid_helper.py fix: support Python 3.15 (#48359) 2026-10-26 16:00:11 +00:00
test_vcr_classification.py test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
test_vcr_conftest_common_banner.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_filters.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_leak_guard.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_redis_persister.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_vcr_safe_body_matcher.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
test_version.py feat(sdk): build core from an independent packaging manifest (#44340) 2026-10-08 18:40:14 -07:00
test_vertex_ai_xai_grok_prompt_caching_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_video_generation.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_with_dashboard_node.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_ws_vcr.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_xai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_xai_responses_auto_routing.py fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) 2026-10-17 21:39:14 +00:00