litellm/tests
Mateo Wang be6d082bdc
fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166)
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts

* refactor(anthropic): hold the chat completions document change for a later commit

* fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai

* fix(snowflake): turn file parts into document blocks for Cortex Claude

* fix(prompt_security): sanitize chat file parts that carry file_data

* fix: answer 400 or drop PDFs for models that cannot read them

Models whose model_info or cost map entry sets supports_pdf_input to
false, and Ollama, now get the PDF dropped under drop_params and a 400
from litellm without it. GigaChat uploads file parts, SAP keeps sending
them as image_url data URLs, azure_ai passes dict content through, and
annotations other than url_citation move to provider_specific_fields
instead of failing the response.

* fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys

Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and
litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider
annotations to provider_specific_fields, and split them the same way when a stream is rebuilt.

* fix(vertex_ai): send PDF file parts to Vertex partner models as image_url

Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages
reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as
unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL.

* fix(cost-map): leave native DeepSeek PDF support unset

/v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there
and would only change chat completions, which this PR does not otherwise touch.

* fix(main): drop a user turn that held only a PDF under drop_params

Dropping the PDF from a PDF-only user turn left an empty content list, which
Foundry and OpenAI-shaped providers reject with a 400, so drop_params still
failed for that request. The emptied user turn is now removed.

* fix: read OCI PDFs, gate deployment base models, and stop empty drops

OCI generic chat sends file parts as image data URLs again, so OCI Gemini
keeps reading PDFs from /v1/messages and chat file parts.

The PDF gate now reads model_info.base_model, drops PDFs under drop_params
for azure_ai deployments that never declared PDF support (the old flattening
behavior), and answers 400 when dropping the PDF leaves no user message.

The cost map marks the Vertex partner text models that a live probe showed
answering without reading the PDF.

* test(integration): cover the PDF input gate, messages PDF file parts and provider annotations

Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed

* test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells

* fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload

* fix(main): PDF check reads the deployment's top-level base_model

* fix(main): keep roles alternating after drop_params removes a PDF-only turn

When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other

* test: type the PDF and annotation tests the PR added

* test: type the GigaChat, SAP and Snowflake PDF tests
2026-10-29 08:06:12 -07:00
..
_support fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) 2026-10-17 18:00:12 -07:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests build(model-catalog): ship the cost map inside the native extension (#48907) 2026-10-29 07:55:02 -07:00
benchmarks test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio (#43370) 2026-10-28 22:04:47 -07:00
code_coverage_tests feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) 2026-10-29 00:29:40 -07:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) 2026-10-29 00:29:40 -07:00
e2e_harness test(perf): ramp the load cases until the pod stops keeping up and floor sustained_rps (#47758) 2026-10-20 22:31:08 -07:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
rust-python-harness feat(rust): encode LiteLLM response ids on native responses (#46572) 2026-10-18 20:52:34 -07:00
test_litellm_rust fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) 2026-10-29 03:36:20 -07:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
windows_tests refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py
_openai_record_replay_proxy.py
_process_helpers.py
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.