litellm/tests/unit/llms
mika-lt 7ab37a1529
fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284)
* fix(count_tokens): answer 400, not 500, for a message without role or content

Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.

* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count

* refactor(bedrock): leave message validation to the Bedrock count handlers

Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy

* fix(count_tokens): name the chat message chat translation cannot send

A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back

* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content

Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it

* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse

The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate

* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse

A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again

* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given

* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks

* test(integration): expect the 400 for a count whose leading system message has no content

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:21:29 +00:00
..
a2a refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
aiml chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
aiohttp_openai fix(aiohttp_openai): forward service_tier in transform_response (#47263) 2026-10-28 13:14:44 -07:00
anthropic fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284) 2026-11-14 04:21:29 +00:00
apiserpent test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
aws_polly fix(aws_polly): read the engine from the bare model name (#47963) 2026-10-23 18:10:51 -07:00
azure fix(azure): strip GPT-5 routing marker from Responses deployments (#51438) 2026-11-12 11:14:52 -08:00
azure_ai fix(model_prices): add Azure DeepSeek V4.1 Flash 1017 and two Perplexity Agent API rows (#51444) 2026-11-12 14:00:06 -08:00
base_llm fix(router): prevent Responses fallback from replaying delivered tool calls (#51683) 2026-11-13 11:43:03 -08:00
baseten fix(aiand)!: show ai&'s name and logo, and sync ai&'s model list (#49158) 2026-10-30 10:55:26 -07:00
bedrock fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284) 2026-11-14 04:21:29 +00:00
bedrock_mantle fix(bedrock): forward Codex agent messages that carry encrypted content intact (#50457) 2026-11-06 17:47:43 -08:00
bigmodel feat(zai): support response_format and max_completion_tokens, add glm-5v-turbo pricing (#48335) 2026-10-29 20:30:50 -07:00
black_forest_labs refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
brave chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
bytez test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
cerebras test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
chat fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) 2026-10-14 14:19:25 -07:00
chatgpt fix(chatgpt): forward text and max_output_tokens to the Codex backend (#38086) 2026-11-06 06:14:32 -08:00
clarifai refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
claude_code refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
clf_ai_gateway refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
cloudflare fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
codestral fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
codex refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
cohere refactor(typing): clear 1141 basedpyright Any errors across 182 files (#50711) 2026-11-11 22:11:47 +00:00
cometapi chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
compactifai refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
crusoe fix(crusoe): update base URL to current endpoint (#33121) 2026-10-15 19:10:58 -07:00
custom_httpx feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) 2026-11-13 22:08:05 +00:00
dashscope feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
databricks chore(databricks): price claude-fable-5-2 from the proprietary pricing page (#51232) 2026-11-12 00:43:40 +00:00
dataforseo refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
datarobot test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
deepagents feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
deepgram fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
deepinfra chore(cost-map): add Soniox v6 and date Soniox, Bedrock Claude and Azure Whisper retirements (#51620) 2026-11-13 12:37:12 -08:00
deepseek fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
docker_model_runner test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
duckduckgo chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
e2b refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
edenai feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
elevenlabs fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
empiriolabs feat(decisions): add EmpirioLabs AI as a decisions provider (#49977) 2026-11-04 10:41:16 -08:00
exa_ai fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423) 2026-10-19 15:45:56 -07:00
fal_ai feat(fal_ai): bill fal video clips by duration and resolution (#51679) 2026-11-13 12:55:55 -08:00
fastcrw fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
featherless_ai refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
firecrawl fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
fireworks_ai fix(fireworks_ai): read fractional sizes like 1p5b in the size bucket fallback (#51451) 2026-11-13 17:49:49 +00:00
friendliai fix(friendliai): drop litellm-internal message fields before sending (#45092) 2026-10-20 01:22:07 +00:00
gdc test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
gemini feat(gemini): support gemini-decision-1 on /v1/decisions and /v1/systemone (#51220) 2026-11-11 18:33:22 -08:00
gigachat refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
github_copilot refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
google_pse chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
gradient_ai fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
groq fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602) 2026-10-31 00:29:10 -07:00
heroku fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
hosted_vllm fix(hosted_vllm): omit empty tools list from chat requests (#50812) 2026-11-09 19:49:13 -08:00
huggingface fix(huggingface): honor HUGGINGFACE_API_BASE and env endpoints for chat (#51449) 2026-11-13 10:35:34 -08:00
hyperbolic refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
inception refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
infinity chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
jina_ai fix(rerank): preserve token usage in spend logs (#49308) 2026-10-30 13:46:27 -07:00
lambda_ai test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
langflow fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
langgraph fix(streaming)!: fail a stream that ends without a finish reason (#47665) 2026-10-31 11:29:26 -07:00
laya feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
lemonade chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
linkup fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
litellm_proxy refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
llamafile refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
lm_studio refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
manus test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
meta fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939) 2026-10-16 15:24:40 +00:00
meta_llama refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
microsoft_365_copilot refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
milvus test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
minimax fix(minimax): forward reasoning_effort instead of dropping it (#49583) 2026-11-09 14:03:50 -08:00
mistral refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
modelscope fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
mongodb feat(bedrock): search knowledge bases by image and return image hits (#51063) 2026-11-12 00:19:34 +00:00
moonshot feat(cost-map): move -latest model aliases into the cost map (#50491) 2026-11-07 11:56:47 -08:00
morph feat(morph): support open models and cached input pricing (#48676) 2026-10-27 18:06:14 -07:00
nadir test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
nebius test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
neosantara refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
nimble test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
nlp_cloud refactor(typing): clear 1141 basedpyright Any errors across 182 files (#50711) 2026-11-11 22:11:47 +00:00
novita test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nscale test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nvidia_nim fix(embeddings): unwrap extra_body for providers with their own embedding handlers (#51335) 2026-11-12 15:41:50 -08:00
nvidia_riva fix(nvidia_riva): send the api key exactly once by calling riva over grpc directly (#50140) 2026-11-04 18:07:31 -08:00
oci fix(fips): hash MCP advisory-lock keys with SHA-256 in fips_mode and compute the OCI body digest as a security hash (#51165) 2026-11-11 16:22:14 -08:00
ocr test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
ollama fix(embeddings): unwrap extra_body for providers with their own embedding handlers (#51335) 2026-11-12 15:41:50 -08:00
oobabooga fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
openai feat(fal_ai): bill fal video clips by duration and resolution (#51679) 2026-11-13 12:55:55 -08:00
openai_like feat: update EmpirioLabs models in model_prices_and_context_window.json (#49980) 2026-11-04 20:11:52 +00:00
opencode fix(opencode): stream Go Messages-protocol models through chat completions (#51715) 2026-11-13 22:21:52 -05:00
openrouter fix: raise on failed OpenRouter non-streaming completions (#50880) 2026-11-14 03:02:00 +00:00
opensandbox chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
opensearch feat(opensearch): map vector store search filters to k-NN filters and add hybrid search (#51131) 2026-11-14 03:30:55 +00:00
ovhcloud fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
parallel_ai refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) 2026-11-11 01:57:54 +00:00
parasail refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
pass_through fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
perplexity fix(search): forward Perplexity search filters (#49344) 2026-10-30 17:08:32 -07:00
pg_vector fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
pi feat(harness): add pi harness for litellm.agent() (#45467) 2026-10-20 09:01:40 +00:00
publicai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
ragflow test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
recraft refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
reducto test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
replicate feat(replicate): add video generation, image generation and text to speech through the predictions API (#51360) 2026-11-12 08:44:50 -08:00
runwayml feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
s3_vectors refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
sagemaker fix(bedrock): send max_tokens and priority in Cohere embedding requests (#51435) 2026-11-12 23:44:08 +00:00
sail test(router_strategy): keep foreign asyncio task noise out of the budget limiter caplog asserts (#43375) 2026-10-29 06:22:06 -07:00
sambanova chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
sap refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
scaledown feat(scaledown): add native text provider support (#44168) 2026-10-10 14:01:17 -07:00
scaleway fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
searchapi chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
searxng chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
serper chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388) 2026-10-13 15:35:23 -07:00
snowflake fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
soniox chore(cost-map): add Soniox v6 and date Soniox, Bedrock Claude and Azure Whisper retirements (#51620) 2026-11-13 12:37:12 -08:00
stability refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
strands_decider feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN (#45678) 2026-10-09 22:51:19 -07:00
tavily fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search (#48684) 2026-10-27 18:52:06 -07:00
tencent fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933) 2026-10-30 08:48:43 -07:00
tinyfish test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
together_ai fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
tool_loop feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
topaz fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
triton fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
v0 test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
valkey test: migrate legacy provider tests to tests/unit (wave 2, phase 13) 2026-09-20 14:18:49 +00:00
vercel_ai_gateway refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
vertex_ai fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
vllm fix(vllm): list hosted_vllm wildcard models when the server has no api key (#40066) 2026-11-03 04:13:28 +00:00
volcengine feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
voyage fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
wandb test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
watsonx fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
webiq feat(search): add Microsoft Web IQ provider (#50883) 2026-11-10 22:43:36 +00:00
xai fix(xai): honor deployment image pixel pricing (#50490) 2026-11-07 02:17:16 +00:00
xinference test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
you_com test: migrate phase 14 wave 2 provider tests to tests/unit 2026-09-20 11:01:40 +00:00
zai feat(zai): support response_format and max_completion_tokens, add glm-5v-turbo pricing (#48335) 2026-10-29 20:30:50 -07:00
__init__.py test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
test_api_base_trailing_slash_routes.py fix(llms): join api_base and route with one shared helper (#47535) 2026-10-21 06:59:32 +00:00
test_cache_control_and_reasoning.py fix(minimax): forward reasoning_effort instead of dropping it (#49583) 2026-11-09 14:03:50 -08:00
test_custom_llm.py refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) 2026-11-11 01:57:54 +00:00
test_file_content_block.py fix(prompt_templates): keep moved private helpers importable from common_utils (#50721) 2026-11-09 20:52:52 +00:00
test_file_search_responses.py refactor(prompt_templates): move JSON schema helpers out of common_utils into json_schema_utils (#50241) 2026-11-05 17:25:53 -08:00
test_get_complete_url_trailing_slash.py feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
test_lifecycle_fix.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_oss_decision.py feat(decisions): add Laya and Bespoke providers to the Add Model form (#45260) 2026-10-14 08:11:11 -07:00
test_polling_url_origin_match.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_predibase_transformation.py fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
test_subscription_accounts.py feat(usage): show subscription-covered providers and account fees separately (#44772) 2026-10-30 03:58:36 -07:00