litellm/tests/unit/llms
Yogeshwaran Ravichandran 35b9df1f40
feat(mistral): add native upstream token counting support (#50652) (#50653)
* feat(mistral): add native upstream token counting support

* test(mistral): standardize on mistral-large-4 and add tool calls tests

* fix(mistral): unify credential resolution and guard Azure endpoint boundary

* fix(mistral): resolve api base from provider env vars and route responses count natively

* fix(test): use groq for bridged proxy tests and increase mistral coverage

Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.

* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter

* fix(mistral): suppress basedpyright warnings on untyped httpx handler

* fix(mistral): keep chat credential resolution and count with the key chat resolves

The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.

* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route

* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them

Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.

The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.

Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.

* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does

Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.

* fix(mistral): count what chat sends and keep the bundled Mistral model list

A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape

A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills

Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400

Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate

MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*

The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run

* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral

* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout

Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy

* fix(mistral): join count text blocks the way Mistral chat does

* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat

* fix(mistral): count the thinking chunks Mistral chat replays in assistant history

The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:19:57 -05:00
..
a2a refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
aiml
aiohttp_openai fix(aiohttp_openai): forward service_tier in transform_response (#47263) 2026-10-28 13:14:44 -07:00
anthropic fix(anthropic): restore sync messages streams for OpenAI and Azure (#51305) 2026-11-13 21:34:21 -08:00
apiserpent
aws_polly fix(aws_polly): read the engine from the bare model name (#47963) 2026-10-23 18:10:51 -07:00
azure fix(azure): strip GPT-5 routing marker from Responses deployments (#51438) 2026-11-12 11:14:52 -08:00
azure_ai fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content (#51747) 2026-11-14 02:08:11 -05:00
base_llm fix(router): prevent Responses fallback from replaying delivered tool calls (#51683) 2026-11-13 11:43:03 -08:00
baseten fix(aiand)!: show ai&'s name and logo, and sync ai&'s model list (#49158) 2026-10-30 10:55:26 -07:00
bedrock fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources (#50577) 2026-11-13 21:33:21 -08:00
bedrock_mantle fix(bedrock): forward Codex agent messages that carry encrypted content intact (#50457) 2026-11-06 17:47:43 -08:00
bigmodel feat(zai): support response_format and max_completion_tokens, add glm-5v-turbo pricing (#48335) 2026-10-29 20:30:50 -07:00
black_forest_labs
brave
bytez
cerebras
chat
chatgpt fix(chatgpt): forward text and max_output_tokens to the Codex backend (#38086) 2026-11-06 06:14:32 -08:00
clarifai
claude_code
clf_ai_gateway refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
cloudflare fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
codestral fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
codex
cohere refactor(typing): clear 1141 basedpyright Any errors across 182 files (#50711) 2026-11-11 22:11:47 +00:00
cometapi
compactifai refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
crusoe fix(crusoe): update base URL to current endpoint (#33121) 2026-10-15 19:10:58 -07:00
custom_httpx feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) 2026-11-13 22:08:05 +00:00
dashscope feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
databricks chore(databricks): price claude-fable-5-2 from the proprietary pricing page (#51232) 2026-11-12 00:43:40 +00:00
dataforseo refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
datarobot
deepagents
deepgram fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
deepinfra chore(cost-map): add Soniox v6 and date Soniox, Bedrock Claude and Azure Whisper retirements (#51620) 2026-11-13 12:37:12 -08:00
deepseek fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
docker_model_runner
duckduckgo
e2b
edenai feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
elevenlabs fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
empiriolabs feat(decisions): add EmpirioLabs AI as a decisions provider (#49977) 2026-11-04 10:41:16 -08:00
exa_ai fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423) 2026-10-19 15:45:56 -07:00
fal_ai feat(fal_ai): bill fal video clips by duration and resolution (#51679) 2026-11-13 12:55:55 -08:00
fastcrw fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
featherless_ai
firecrawl fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
fireworks_ai fix(fireworks_ai): read fractional sizes like 1p5b in the size bucket fallback (#51451) 2026-11-13 17:49:49 +00:00
friendliai fix(friendliai): drop litellm-internal message fields before sending (#45092) 2026-10-20 01:22:07 +00:00
gdc
gemini feat(gemini): support gemini-decision-1 on /v1/decisions and /v1/systemone (#51220) 2026-11-11 18:33:22 -08:00
gigachat refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
github_copilot refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
google_pse
gradient_ai fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
groq fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602) 2026-10-31 00:29:10 -07:00
heroku fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
hosted_vllm fix(hosted_vllm): omit empty tools list from chat requests (#50812) 2026-11-09 19:49:13 -08:00
huggingface fix(huggingface): honor HUGGINGFACE_API_BASE and env endpoints for chat (#51449) 2026-11-13 10:35:34 -08:00
hyperbolic
inception
infinity
jina_ai fix(rerank): preserve token usage in spend logs (#49308) 2026-10-30 13:46:27 -07:00
lambda_ai
langflow
langgraph fix(streaming)!: fail a stream that ends without a finish reason (#47665) 2026-10-31 11:29:26 -07:00
laya
lemonade
linkup fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
litellm_proxy refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
llamafile
lm_studio
manus
meta fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939) 2026-10-16 15:24:40 +00:00
meta_llama
microsoft_365_copilot refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
milvus
minimax fix(minimax): forward reasoning_effort instead of dropping it (#49583) 2026-11-09 14:03:50 -08:00
mistral feat(mistral): add native upstream token counting support (#50652) (#50653) 2026-11-14 03:19:57 -05:00
modelscope fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
mongodb feat(bedrock): search knowledge bases by image and return image hits (#51063) 2026-11-12 00:19:34 +00:00
moonshot feat(cost-map): move -latest model aliases into the cost map (#50491) 2026-11-07 11:56:47 -08:00
morph feat(morph): support open models and cached input pricing (#48676) 2026-10-27 18:06:14 -07:00
nadir
nebius
neosantara
nimble
nlp_cloud refactor(typing): clear 1141 basedpyright Any errors across 182 files (#50711) 2026-11-11 22:11:47 +00:00
novita
nscale
nvidia_nim fix(embeddings): unwrap extra_body for providers with their own embedding handlers (#51335) 2026-11-12 15:41:50 -08:00
nvidia_riva fix(nvidia_riva): send the api key exactly once by calling riva over grpc directly (#50140) 2026-11-04 18:07:31 -08:00
oci fix(fips): hash MCP advisory-lock keys with SHA-256 in fips_mode and compute the OCI body digest as a security hash (#51165) 2026-11-11 16:22:14 -08:00
ocr
ollama fix(embeddings): unwrap extra_body for providers with their own embedding handlers (#51335) 2026-11-12 15:41:50 -08:00
oobabooga fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
openai feat(fal_ai): bill fal video clips by duration and resolution (#51679) 2026-11-13 12:55:55 -08:00
openai_like feat: update EmpirioLabs models in model_prices_and_context_window.json (#49980) 2026-11-04 20:11:52 +00:00
opencode fix(opencode): stream Go Messages-protocol models through chat completions (#51715) 2026-11-13 22:21:52 -05:00
openrouter fix: raise on failed OpenRouter non-streaming completions (#50880) 2026-11-14 03:02:00 +00:00
opensandbox
opensearch feat(opensearch): map vector store search filters to k-NN filters and add hybrid search (#51131) 2026-11-14 03:30:55 +00:00
ovhcloud fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
parallel_ai refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) 2026-11-11 01:57:54 +00:00
parasail
pass_through fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
perplexity fix(search): forward Perplexity search filters (#49344) 2026-10-30 17:08:32 -07:00
pg_vector
pi feat(harness): add pi harness for litellm.agent() (#45467) 2026-10-20 09:01:40 +00:00
publicai
ragflow
recraft refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
reducto
replicate feat(replicate): add video generation, image generation and text to speech through the predictions API (#51360) 2026-11-12 08:44:50 -08:00
runwayml feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
s3_vectors refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
sagemaker fix(bedrock): send max_tokens and priority in Cohere embedding requests (#51435) 2026-11-12 23:44:08 +00:00
sail test(router_strategy): keep foreign asyncio task noise out of the budget limiter caplog asserts (#43375) 2026-10-29 06:22:06 -07:00
sambanova
sap refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
scaledown
scaleway fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
searchapi
searxng
serper
snowflake fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) 2026-10-29 08:06:12 -07:00
soniox chore(cost-map): add Soniox v6 and date Soniox, Bedrock Claude and Azure Whisper retirements (#51620) 2026-11-13 12:37:12 -08:00
stability
strands_decider
tavily fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search (#48684) 2026-10-27 18:52:06 -07:00
tencent fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933) 2026-10-30 08:48:43 -07:00
tinyfish
together_ai
tool_loop
topaz fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
triton fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
v0
valkey
vercel_ai_gateway
vertex_ai fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239) 2026-11-14 03:12:09 -05:00
vllm fix(vllm): list hosted_vllm wildcard models when the server has no api key (#40066) 2026-11-03 04:13:28 +00:00
volcengine feat(videos): add first-class OpenRouter video request fields (#50319) 2026-11-06 16:50:48 -08:00
voyage fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
wandb
watsonx fix(audio-transcription): keep transcripts out of transcription error messages (#50137) 2026-11-05 18:07:15 -08:00
webiq feat(search): add Microsoft Web IQ provider (#50883) 2026-11-10 22:43:36 +00:00
xai fix(xai): honor deployment image pixel pricing (#50490) 2026-11-07 02:17:16 +00:00
xinference
you_com
zai feat(zai): support response_format and max_completion_tokens, add glm-5v-turbo pricing (#48335) 2026-10-29 20:30:50 -07:00
__init__.py
test_api_base_trailing_slash_routes.py fix(llms): join api_base and route with one shared helper (#47535) 2026-10-21 06:59:32 +00:00
test_cache_control_and_reasoning.py fix(minimax): forward reasoning_effort instead of dropping it (#49583) 2026-11-09 14:03:50 -08:00
test_custom_llm.py refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) 2026-11-11 01:57:54 +00:00
test_file_content_block.py fix(prompt_templates): keep moved private helpers importable from common_utils (#50721) 2026-11-09 20:52:52 +00:00
test_file_search_responses.py refactor(prompt_templates): move JSON schema helpers out of common_utils into json_schema_utils (#50241) 2026-11-05 17:25:53 -08:00
test_get_complete_url_trailing_slash.py feat(dashscope): add image edits and text-to-speech (#48998) 2026-10-29 17:48:38 -07:00
test_lifecycle_fix.py
test_oss_decision.py
test_polling_url_origin_match.py
test_predibase_transformation.py fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) 2026-10-28 17:00:29 -07:00
test_subscription_accounts.py feat(usage): show subscription-covered providers and account fees separately (#44772) 2026-10-30 03:58:36 -07:00