mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-16 09:22:22 +00:00
* feat(mistral): add native upstream token counting support * test(mistral): standardize on mistral-large-4 and add tool calls tests * fix(mistral): unify credential resolution and guard Azure endpoint boundary * fix(mistral): resolve api base from provider env vars and route responses count natively * fix(test): use groq for bridged proxy tests and increase mistral coverage Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov. * fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter * fix(mistral): suppress basedpyright warnings on untyped httpx handler * fix(mistral): keep chat credential resolution and count with the key chat resolves The token counter now resolves its Mistral key and base through the same helper chat uses, so a count goes to the endpoint chat would call. A deployment that would route to Azure, or an Azure-only env key, falls back to the local estimate instead of sending that key to Mistral. Chat key and base precedence is unchanged from main. * fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route * fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge, which restores previous_response_id history. The count now takes the same bridge instead of forwarding previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone. The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them. Tests now go through the real handler with respx, and the integration Mistral peer serves the count route. * fix(mistral): leave empty assistant turns out of the count, as Mistral chat does Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400 through previous_response_id, so the count now skips the same turns chat skips. * fix(mistral): count what chat sends and keep the bundled Mistral model list A user turn that mixed file_id files with an image or an inline file lost the image or inline file in the count, since the converted content arrives as a tuple and only lists were read. Both content readers now take either shape A /v1/responses count with a json_schema text.format now sends that format to Mistral, so the count matches what the bridged chat call bills Counts for Mistral models that are not chat models (embeddings, OCR, transcription) now fall back to the local estimate instead of returning Mistral's 400 Count calls now use the deployment's timeout when one is set, and a count response without input_tokens falls back to the local estimate MistralConfig.get_models goes back to the bundled model list, so turning on check_provider_endpoint does not change what /v1/models shows for mistral/* The previous_response_id count test now fakes the spend logs table instead of the history lookup, so the real lookup and its query run * fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral * fix(mistral): answer 408 on a count timeout and use the shared deployment timeout Main's native counters answer a count timeout with 408 and a connection error with 500. The Mistral counter fell back to a local estimate on both, so a hung Mistral endpoint looked like a successful count. It now raises the same count errors, and it reads the deployment timeout with the shared deployment_timeout helper instead of its own copy * fix(mistral): join count text blocks the way Mistral chat does * fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat * fix(mistral): count the thinking chunks Mistral chat replays in assistant history The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is skipped the way chat skips it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| a2a | ||
| aiml | ||
| aiohttp_openai | ||
| anthropic | ||
| apiserpent | ||
| aws_polly | ||
| azure | ||
| azure_ai | ||
| base_llm | ||
| baseten | ||
| bedrock | ||
| bedrock_mantle | ||
| bigmodel | ||
| black_forest_labs | ||
| brave | ||
| bytez | ||
| cerebras | ||
| chat | ||
| chatgpt | ||
| clarifai | ||
| claude_code | ||
| clf_ai_gateway | ||
| cloudflare | ||
| codestral | ||
| codex | ||
| cohere | ||
| cometapi | ||
| compactifai | ||
| crusoe | ||
| custom_httpx | ||
| dashscope | ||
| databricks | ||
| dataforseo | ||
| datarobot | ||
| deepagents | ||
| deepgram | ||
| deepinfra | ||
| deepseek | ||
| docker_model_runner | ||
| duckduckgo | ||
| e2b | ||
| edenai | ||
| elevenlabs | ||
| empiriolabs | ||
| exa_ai | ||
| fal_ai | ||
| fastcrw | ||
| featherless_ai | ||
| firecrawl | ||
| fireworks_ai | ||
| friendliai | ||
| gdc | ||
| gemini | ||
| gigachat | ||
| github_copilot | ||
| google_pse | ||
| gradient_ai | ||
| groq | ||
| heroku | ||
| hosted_vllm | ||
| huggingface | ||
| hyperbolic | ||
| inception | ||
| infinity | ||
| jina_ai | ||
| lambda_ai | ||
| langflow | ||
| langgraph | ||
| laya | ||
| lemonade | ||
| linkup | ||
| litellm_proxy | ||
| llamafile | ||
| lm_studio | ||
| manus | ||
| meta | ||
| meta_llama | ||
| microsoft_365_copilot | ||
| milvus | ||
| minimax | ||
| mistral | ||
| modelscope | ||
| mongodb | ||
| moonshot | ||
| morph | ||
| nadir | ||
| nebius | ||
| neosantara | ||
| nimble | ||
| nlp_cloud | ||
| novita | ||
| nscale | ||
| nvidia_nim | ||
| nvidia_riva | ||
| oci | ||
| ocr | ||
| ollama | ||
| oobabooga | ||
| openai | ||
| openai_like | ||
| opencode | ||
| openrouter | ||
| opensandbox | ||
| opensearch | ||
| ovhcloud | ||
| parallel_ai | ||
| parasail | ||
| pass_through | ||
| perplexity | ||
| pg_vector | ||
| pi | ||
| publicai | ||
| ragflow | ||
| recraft | ||
| reducto | ||
| replicate | ||
| runwayml | ||
| s3_vectors | ||
| sagemaker | ||
| sail | ||
| sambanova | ||
| sap | ||
| scaledown | ||
| scaleway | ||
| searchapi | ||
| searxng | ||
| serper | ||
| snowflake | ||
| soniox | ||
| stability | ||
| strands_decider | ||
| tavily | ||
| tencent | ||
| tinyfish | ||
| together_ai | ||
| tool_loop | ||
| topaz | ||
| triton | ||
| v0 | ||
| valkey | ||
| vercel_ai_gateway | ||
| vertex_ai | ||
| vllm | ||
| volcengine | ||
| voyage | ||
| wandb | ||
| watsonx | ||
| webiq | ||
| xai | ||
| xinference | ||
| you_com | ||
| zai | ||
| __init__.py | ||
| test_api_base_trailing_slash_routes.py | ||
| test_cache_control_and_reasoning.py | ||
| test_custom_llm.py | ||
| test_file_content_block.py | ||
| test_file_search_responses.py | ||
| test_get_complete_url_trailing_slash.py | ||
| test_lifecycle_fix.py | ||
| test_oss_decision.py | ||
| test_polling_url_origin_match.py | ||
| test_predibase_transformation.py | ||
| test_subscription_accounts.py | ||