litellm/tests
Yogeshwaran Ravichandran 35b9df1f40
feat(mistral): add native upstream token counting support (#50652) (#50653)
* feat(mistral): add native upstream token counting support

* test(mistral): standardize on mistral-large-4 and add tool calls tests

* fix(mistral): unify credential resolution and guard Azure endpoint boundary

* fix(mistral): resolve api base from provider env vars and route responses count natively

* fix(test): use groq for bridged proxy tests and increase mistral coverage

Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.

* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter

* fix(mistral): suppress basedpyright warnings on untyped httpx handler

* fix(mistral): keep chat credential resolution and count with the key chat resolves

The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.

* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route

* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them

Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.

The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.

Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.

* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does

Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.

* fix(mistral): count what chat sends and keep the bundled Mistral model list

A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape

A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills

Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400

Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate

MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*

The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run

* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral

* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout

Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy

* fix(mistral): join count text blocks the way Mistral chat does

* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat

* fix(mistral): count the thinking chunks Mistral chat replays in assistant history

The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:19:57 -05:00
..
_support feat(mcp): add an opt-in per-server setting to skip MCP tool output validation (#49961) 2026-11-03 20:42:10 -08:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
basic_proxy_startup_tests feat(docker)!: one shipped image and one entrypoint beside the Dockerfile (#43332) 2026-11-10 18:32:16 -08:00
benchmarks fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
code_coverage_tests feat(opensearch): map vector store search filters to k-NN filters and add hybrid search (#51131) 2026-11-14 03:30:55 +00:00
documentation_tests chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
e2e refactor(redis): delete the Python RedisBatch now that every request op is native (#51383) 2026-11-13 03:22:28 +00:00
e2e_harness test(e2e): add a Codex row where a sub-agent must read the task its parent hands it (#50494) 2026-11-07 21:16:39 +00:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration feat(mistral): add native upstream token counting support (#50652) (#50653) 2026-11-14 03:19:57 -05:00
rust-python-harness chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
test_litellm_rust perf(rust): stop re-serializing the Messages request body in Python logging callbacks (#51440) 2026-11-12 17:13:55 -08:00
typing chore: move config and docs off the repo root (#50966) 2026-11-11 18:29:35 -08:00
unit feat(mistral): add native upstream token counting support (#50652) (#50653) 2026-11-14 03:19:57 -05:00
windows_tests feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py
_vcr_conftest_common.py ci: remove CircleCI (#50950) 2026-11-11 15:38:27 -08:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.