scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-05 07:37:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 361
litellm/tests/unit
History
tavren27 b9ad61d4f6
fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 
* fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call

* fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests

* fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500

* fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message

* fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails

A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever.

A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry.

On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed

* fix(agentic loop): bill retried attempts and clean up what failed or capped turns return

The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend

A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers

Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry

The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints

* fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns

A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call

Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try

* fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served

An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the
router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of
rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run.

The failure row bills every round the loop served before the failure, at any depth, and a later attempt
that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the
response cost headers adding up. The error response's x-litellm-response-cost reports the same amount
its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before.

A refused tool call now matches by id among the refused ids, or by name among refused calls that carry
no id, so a name-only refusal next to an id refusal is no longer kept.

* fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one

* fix(code_interpreter): delete only the session sandbox that failed, never its replacement

* test(code_interpreter): type the session sandbox test helpers

* fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors

* fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success

An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's
plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no
longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too.

A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row.

A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed
agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main.

A code interpreter sandbox that fails while running the code now says the code may have run but its
result is lost, instead of saying the code did not run.

* test(responses): bill a retried MCP request each attempt on its own row again

Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row

* fix(spend): name a failed follow-up attempt the success record already bills

With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off

* fix(spend): count a billed failed follow-up's tokens on the success row

* fix(spend): bill a failed follow-up on a fallback served from cache

* fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown

* fix(spend): keep a failed follow-up's billing state off the request-merged call details

* fix(spend): record the cost breakdown of a successful agentic follow-up

The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-04 02:00:33 +00:00
..
a2a_protocol fix(a2a)!: bill completion-bridge agents at their own prices on /a2a send and stream (#49150) 2026-10-30 19:43:45 -07:00
accounting feat(accounting): add accounting contracts, PricingSnapshot and ledger schema (LIT-10193) (#48862) 2026-10-28 14:39:25 -07:00
anthropic_interface fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) 2026-11-03 01:53:29 +00:00
batch_completion test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
batches chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
caching feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393) 2026-11-02 13:26:29 -08:00
completion_extras fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API (#49710) 2026-11-03 10:02:54 +00:00
compression test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
containers refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
cookbook feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
db_scripts fix(vector-stores): keep proxy context out of provider vector store metadata (#47021) 2026-10-17 12:51:47 -07:00
decisions fix(decisions)!: honor a deployment's own supports_audio_input and supports_video_input flags (#49207) 2026-10-31 01:09:46 -07:00
endpoints feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
enterprise fix(managed_files): keep the upload the managed id encodes when two target models share a deployment (#49420) 2026-11-03 00:58:41 +00:00
expected_fine_tuning_api test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
expected_responses_api_request test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
experimental_mcp_client fix(mcp): load deprecated package aliases on demand (#48615) 2026-10-27 19:46:19 +00:00
extraction feat(extraction): add /v1/extraction with Cloudflare Clairic support (#47473) 2026-10-20 15:51:45 -07:00
files fix(hosted_vllm): accept the vLLM production-stack router's file and batch objects (#49068) 2026-10-30 23:11:34 -07:00
fine_tuning fix(fine_tuning): stop forwarding api_version to the OpenAI client (#44995) 2026-10-21 01:58:25 +00:00
fixtures test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
gateway fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping (#45039) 2026-10-29 10:25:14 -07:00
google_genai fix(google_genai): read one-object and null spellings of repeated fields in countTokens and generateContent (#48771) 2026-10-28 21:46:06 -07:00
harness fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) 2026-10-21 02:26:05 +00:00
images feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
integration_support test(mcp): cancel disconnected HTTP tool handlers (#49436) 2026-10-31 16:41:49 -07:00
integrations fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
interactions fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
litellm_core_utils fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
litellm_proxy_extras feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961) 2026-10-29 09:39:26 -07:00
llms fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
mcp_client refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
models fix(proxy): allow model instances in LiteLLM_EndUserTable validation (#41677) 2026-10-15 10:57:26 -07:00
ocr test(rust-bridge): relocate dispatch tests before routing changes (#46926) 2026-10-16 12:29:33 -07:00
passthrough fix(vllm): send the pass-through key as a bearer on both /vllm routes (#46861) 2026-10-30 08:57:48 -07:00
proxy fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
rag fix(rag)!: use team provider credentials and allow viewer and JWT team access (#45497) 2026-10-17 23:51:48 +00:00
realtime_api feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
repositories refactor(proxy): inject gateway activity through a typed repository (#49116) 2026-10-30 16:25:11 -07:00
rerank_api test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
responses fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
router_strategy fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
router_utils fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783) 2026-11-03 12:56:22 +00:00
rust_bridge fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
sandbox fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
search fix(search): price Tavily advanced and Linkup deep searches at their own cost map entries (#47411) 2026-10-19 23:32:31 -07:00
secret_managers fix(proxy): let a config license unlock enterprise settings and spell "environment" right in license errors (#49745) 2026-11-03 09:56:08 +00:00
skills feat(skills): add a litellm-setup agent skill (#47731) 2026-10-28 15:44:43 -07:00
systemone fix(databricks): reuse base OAuth for decisions (#48395) (#48527) 2026-10-26 20:18:16 -07:00
telemetry feat(telemetry): add AggregatingSink, fixed-bucket histograms, Report and HttpExporter (#45487) 2026-10-13 18:33:01 -07:00
test_router fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) 2026-11-04 02:00:33 +00:00
text_completions feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393) 2026-11-02 13:26:29 -08:00
tracing fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344) 2026-10-16 00:53:52 +00:00
types fix(responses)!: authorize and decode comparison response IDs (#49402) 2026-11-03 03:40:46 +00:00
vector_stores fix(exceptions)!: honor status code when mapping OpenAI invalid_request_error exceptions (#32537) 2026-10-29 08:05:47 -07:00
videos feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785) 2026-11-02 21:28:46 -08:00
__init__.py test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
AGENTS.md feat(vector_stores): add OpenSearch k-NN search as a vector store provider (#47759) 2026-10-29 08:15:05 -07:00
conftest.py fix(anthropic): send an https PDF image_url to Claude as a document (#49708) 2026-11-03 07:05:15 +00:00
create_mock_standard_logging_payload.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_a2a_registry_lookup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_acompletion_session_reuse_e2e.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_add_deployment_no_master_key.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_aembedding_session_reuse_e2e.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_anthropic_beta_gated_fields.py fix(anthropic)!: drop beta-gated body fields under drop_params (#48793) 2026-10-29 06:25:31 -07:00
test_anthropic_beta_headers_filtering.py feat(anthropic): pick the beta allowlist row per deployment and surface input_transformations (#49689) 2026-11-03 01:55:26 +00:00
test_anthropic_skills_transformation.py feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
test_assert_ci_coverage.py docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI (#46796) 2026-10-16 14:57:49 -07:00
test_assert_secret_absent.py ci: run the integration groups on GitHub Actions (#48985) 2026-10-29 12:50:08 -07:00
test_assert_workflow_dir_hygiene.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_audio_transcription_rust_bridge.py fix(bedrock): omit system prompt from audio transcription and report its token usage (#41565) 2026-10-28 19:23:15 -07:00
test_azure_ad_token_credential_resolution.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_6_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_baseten_glm_5_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_batch_completion_models_all_responses.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_bedrock_marengo_embed_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_build_core_distribution.py test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
test_cerebras_llama_model_metadata.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_chat_ui_responses_session.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_licenses.py feat(proxy): embed enterprise LiteAdmin MCP in LiteLLM images (#44610) 2026-10-05 15:24:34 -07:00
test_check_mcp_operation_boundary.py refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
test_check_migrations_lock_timeout.py fix(proxy-extras): bound migration lock waits with SET LOCAL lock_timeout in the migration SQL (#46329) 2026-10-13 20:51:03 -07:00
test_check_migrations_no_data_rewrites.py fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948) 2026-10-01 14:12:22 -07:00
test_check_openapi_docs.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
test_check_py310_typing_imports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_test_quality.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_check_type_discipline.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_check_unbounded_in_lists.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_check_windows_wheel_install.py test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
test_circleci_checkout.py ci(circleci): clone the public repo over HTTPS so main jobs stop failing at checkout (#49394) 2026-10-31 02:53:54 -07:00
test_circleci_path_filter.py refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
test_circleci_run_migration_tests.py test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
test_circleci_rust_toolchain.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_claude_fable_5_config.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_claude_haiku_5_5_config.py fix(model_prices): consolidate claude-haiku-5-5 over-100k pricing and capability flags (#45151) 2026-10-07 14:14:47 -07:00
test_claude_opus_4_6_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_4_8_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_5_config.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_claude_sonnet_5_config.py refactor(cost-map): remove openrouter rows, part 5 of 5 (#45669) 2026-10-09 22:48:55 +00:00
test_cloudflare_workers_ai_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_completion_timeout_resolution.py fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) 2026-10-17 22:08:02 +00:00
test_component_entrypoint.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_compression.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_conftest_isolation.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_constants.py fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054) 2026-10-29 20:32:04 -07:00
test_container_router.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculation_log_level.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculator.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_cost_map_guard.py fix(cost_map): mark fireworks_ai/minimax-m3 as vision and pin capabilities verified against live calls (#43390) 2026-10-29 00:23:11 -07:00
test_count_tokens_public_api.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_dashscope_image_generation.py feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
test_daybreak_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_deepseek_model_metadata.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_default_branch.py feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex (#48949) 2026-10-29 18:15:44 -07:00
test_detect_changes.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_docker_no_network_on_deploy.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_dockerfile_apk_repository.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_bedrock_realtime_extra.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_non_root.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_drop_params_env_var.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_e2e_egress_sentinel.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_eager_tiktoken_load.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_elevenlabs_tts_pricing.py fix(cost-map): rolling registry audit, add elevenlabs api rates, together minimax-m3.1-flash, mistral voxtral 3 and azure_ai phi-4 dates (#47623) 2026-10-23 14:57:35 -07:00
test_empty_mock_response.py fix: honor empty mock completion responses (#48162) (#48266) 2026-10-28 00:56:45 +00:00
test_env_key_doc_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_exports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_header_preservation.py fix(openai): keep the upstream error body on e.response for OpenAI and Azure errors (#49079) 2026-10-29 23:30:02 -07:00
test_exception_mapping_request_attribute.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_fake_openai_endpoint.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_filter_out_litellm_params.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
test_fireworks_serverless_model_costs.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gate_slot_lock.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_3_1_flash_lite_image_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_4_argon_model_metadata.py feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) 2026-10-15 22:54:23 +00:00
test_gemini_tts_native_audio_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_get_blog_posts.py chore(blog): refresh the bundled blog posts backup with the latest 50 posts from the docs RSS feed (#49971) 2026-11-03 23:41:15 +00:00
test_git_hooks.py chore(hooks): drop Conventional Branches pre-push hook in favor of litellm_ prefix (#44951) 2026-10-20 00:35:39 +00:00
test_gpt_5_4_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_5_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_image_cost_calculator.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_realtime_mode.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_groq_streaming_encoding.py fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
test_guardrail_exception_status_codes.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_guardrails.py Add guardrails.apply SDK helper (#49217) (#49469) 2026-10-31 23:37:45 +00:00
test_integration_run.py test(integration): isolate every multi-worker integration suite by default (#46027) 2026-10-12 03:46:06 -07:00
test_internal_context.py feat(proxy)!: model family and access group entries in model_max_budget (#47722) 2026-10-28 15:40:57 -07:00
test_lazy_imports.py fix(health)!: openai realtime health check reports healthy with a missing or invalid api key (#41233) 2026-10-29 00:40:02 -07:00
test_lint_base_counts.py revert(lint-gates): accept an empty base scan again, since zero violations is a legitimate count (#45319) 2026-10-08 08:40:08 +00:00
test_lint_workflow_diff_gates.py fix(ci): measure PR gates from the target branch's current tip (#49078) 2026-10-29 18:28:16 -07:00
test_litellm_cli.py fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping (#45039) 2026-10-29 10:25:14 -07:00
test_litellm_overhead.py test(unit): wait on the work itself instead of fixed sleeps (#46261) 2026-10-13 12:41:06 -07:00
test_litellm_params_reserved_keys.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_logging.py feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708) 2026-10-29 03:47:04 -07:00
test_lowest_latency_zero_tokens.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_main.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
test_main_module_header.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_large_4_model_metadata.py fix(cost-map): price Mistral Large 4 at its list price now that the launch sale ended (#48825) 2026-10-28 18:12:55 +00:00
test_mistral_medium_3_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_small_4_0_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_zai_glm_5_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_block_unblock.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_model_cost_aliases.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_param_helper.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_model_prices_and_context_window.py fix(vertex-ai): sync tier prices and retirement floors (#49964) 2026-11-04 00:03:54 +00:00
test_model_response_normalization.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_1_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_3_model_metadata.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_mutation_report.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_nested_drop_params.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_no_top_level_test_invocations.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_non_chat_routes_open_llm_spans.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_openai_embedding_encoding_format_default.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_openai_record_replay_proxy.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_openai_service_tier_long_context_pricing.py fix(cost_calculator): bill ultrafast prompts above 272k at the ultrafast long-context rates (#43764) 2026-09-30 07:32:16 -07:00
test_openapi_docs_gate.py fix(ci): build base native extension for OpenAPI lint (#49136) 2026-10-29 16:56:44 -07:00
test_package_layout.py test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
test_package_metadata.py chore(deps): require anyio>=4.14.0 to rule out the lock-waiter deadlock (#47378) 2026-10-19 12:37:47 -07:00
test_pre_commit_lint.py ci: run the dashboard API types and Terraform endpoint checks on every change that can alter the spec (#49022) 2026-10-29 07:19:06 -07:00
test_prisma_generate_if_needed.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_private_usage_aliases.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_process_helpers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_alias_tracking.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_tags_pydantic.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_proxy_auth.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_pydantic_namespaces.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_rag_openai_ingestion.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_rate_limit_error_unification.py refactor(proxy): expose public names for private proxy helpers (#45170) 2026-10-07 18:59:27 -07:00
test_read_rc_version.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_redact_string_in_error_paths.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_redis.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_redis_credential_provider.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_register_model_custom_pricing.py fix(main)!: carry the full per-call price entry on the logging object instead of writing litellm.model_cost (#49532) 2026-11-03 16:03:34 -08:00
test_register_model_zero_cost_persistence.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_replicate_model_key_format.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_repo_agent_skills.py feat(skills): add a litellm-setup agent skill (#47731) 2026-10-28 15:44:43 -07:00
test_responses_api_bridge_non_stream.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_responses_id_security.py fix(responses)!: authorize and decode comparison response IDs (#49402) 2026-11-03 03:40:46 +00:00
test_responses_streaming_container_ownership.py fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703) 2026-10-15 11:00:14 -07:00
test_retrieve_batch_bedrock_dispatch.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_router_block_helpers.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_router_exception_redaction.py fix(streaming)!: map stream read timeouts to litellm.Timeout on chat, messages, and responses (#47198) 2026-10-18 13:58:34 +00:00
test_router_get_settings.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_router_google_genai.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_model_cost_isolation.py fix(router): keep a deployment's token limits off the shared backend cost map key (#49952) 2026-11-03 16:14:44 -08:00
test_router_order_fallback.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
test_router_per_deployment_num_retries.py fix(images): honor max_retries and prevent Router retry amplification (#48705) 2026-10-28 10:46:47 -07:00
test_router_redis_init.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_backoff_headers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_non_retryable_errors.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_policy_update.py fix(proxy): unset DB environment variables removed from the row on every worker by row revision (#46709) 2026-10-17 03:51:05 +00:00
test_router_silent_experiment.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_router_streaming_fallback_metadata.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_router_weighted_failover.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_ruff_strict_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_sambanova_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_scaledown_model_prices.py feat(scaledown): add native text provider support (#44168) 2026-10-10 14:01:17 -07:00
test_scheduler.py fix(scheduler): read shared queue from Redis so expired dead-replica entries stop blocking waiters (#49387) 2026-10-31 11:43:46 -07:00
test_secret_redaction.py refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
test_seed_tracing_fixtures.py refactor(lens): connect LiteLLM to the independent Lens service (#45529) 2026-10-09 16:17:05 -07:00
test_select_ui_test_scope.py fix(ci): measure PR gates from the target branch's current tip (#49078) 2026-10-29 18:28:16 -07:00
test_service_logger.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_setup_wizard.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_shared_session_integration.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_socket_policy.py test(unit): block external sockets at import time and add a socket policy regression test 2026-09-20 08:07:50 +00:00
test_spend_logs_guard.py fix(ci): flag any schema.prisma change inside model LiteLLM_SpendLogs in the spend logs guard (LIT-10240) (#48905) 2026-10-28 15:52:15 -07:00
test_ssl_verify_unit.py test(unit): stop test modules from putting their own directory on sys.path (#43421) 2026-09-28 12:20:20 -07:00
test_stream_chunk_builder_annotations.py fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
test_stream_chunk_builder_citations.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_stream_chunk_builder_images.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_streaming_connection_cleanup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_system_message_format_bug.py fix(prompt_templates): do not mutate messages in-place in function_call_prompt (#41676) 2026-10-15 14:37:11 -07:00
test_test_quality_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_thinking_enabled.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_together_ai_model_metadata.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_tool_loop.py fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) 2026-10-29 00:56:02 -07:00
test_type_check_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_type_discipline_gate.py ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
test_typesafe_model_metadata.py fix(cost): price typesafe/jev-1.14.0 (#46594) 2026-10-14 20:27:35 -07:00
test_unit_shard_missing_paths.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_unit_shard_per_test_timeout.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_utils.py chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) 2026-11-03 16:23:06 -08:00
test_utils_get_model_info.py fix(bedrock): price claude_platform calls from their own cost map rows (#48150) 2026-10-24 13:54:18 -07:00
test_utils_get_optional_params.py fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761) 2026-10-29 04:46:29 -07:00
test_utils_module_docstring.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_uuid_helper.py fix: support Python 3.15 (#48359) 2026-10-26 16:00:11 +00:00
test_vcr_classification.py test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
test_vcr_conftest_common_banner.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_filters.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_leak_guard.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_vcr_redis_persister.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_vcr_safe_body_matcher.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
test_version.py feat(sdk): build core from an independent packaging manifest (#44340) 2026-10-08 18:40:14 -07:00
test_vertex_ai_xai_grok_prompt_caching_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_video_generation.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_with_dashboard_node.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_ws_vcr.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_xai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_xai_responses_auto_routing.py fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) 2026-10-17 21:39:14 +00:00