mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
27c796c412
* fix(proxy): count the request's tool_choice on the token count routes /utils/token_counter, /v1/messages/count_tokens and /v1/responses/input_tokens now pass tool_choice into the local estimate, and every DeepSeek model counts it as 0, as its providers bill * fix(token_counter): count a request with no model name again * fix(token_counter): count grok tool_choice the way xAI bills it xAI bills grok-4.3 and grok-4-1-fast-reasoning 21 tokens for "none", 11 for "required" and 12 plus the name for a named tool, where the OpenAI costs counted 1, 0 and 7 plus the name. The grok-4-1 and grok-4.3 chat template tool sections now carry those costs, so a grok count matches its bill for every tool_choice * fix(token_counter): count xAI's grok tool_choice costs only where xAI serves grok Azure AI Foundry bills grok-4-1's tool_choice differently, so grok on other hosts keeps counting it as 0. A tool_choice sent without tools counts nothing, since OpenAI and xAI reject it * fix(token_counter): key xAI's grok tool_choice costs on the provider that serves the deployment The count routes strip the provider prefix before counting, so Azure, Vertex AI and Perplexity grok matched the xAI name pattern. The routes now resolve the serving provider with get_llm_provider and drop xAI's costs when it is not xAI or OpenRouter's x-ai route. The costs move onto the grok tool sections so one pattern decides them * fix(token_counter): count no tool_choice for Gemini and none without tools in the Rust counter Gemini bills the same prompt tokens for every tool_choice, so it skips the OpenAI choice costs the way DeepSeek does. The Rust reservation counter now adds tool_choice tokens only when tools are present, matching the Python counter * fix(proxy): count no tool_choice a deployment drops, and treat an api.x.ai api_base as xAI A deployment that lists tool_choice in additional_drop_params never sends it, so the count routes no longer add its cost. An openai/ model whose api_base is on api.x.ai is served by xAI, so it gets xAI's tool_choice costs like xai/ grok * fix(proxy): count no tool_choice a request drops in its body /v1/messages/count_tokens takes additional_drop_params from the request body, and inference drops tool_choice when it is listed there, so the count no longer adds its cost either * fix(proxy): take additional_drop_params on /utils/token_counter and treat a malformed api_base as not xAI /utils/token_counter now passes a request body's additional_drop_params into the count, so a tool_choice the request drops counts as 0 there like on the other two count routes. An api_base that does not parse falls back to the deployment's provider instead of failing the count with a 500 * fix(proxy): take chatgpt and github_copilot providers from the model name in the count host check Resolving them through get_llm_provider read the saved login, logged a warning and could refresh a token on every count with a tool_choice * fix(proxy): skip a tool_choice or additional_drop_params too deeply nested to validate on the count routes A value nested 300 levels deep failed TokenCountRequest validation, so the count routes answered 422 or 400 where they used to count the request without it * fix(bedrock): count Kimi K3's forced tool_choice message on the exact count path Kimi K3's chat template adds a tool-choice system message when Bedrock gets toolChoice any or a named tool, and Bedrock bills it (170 input tokens unforced, 206 forced). The exact count path now passes tool_choice through the same Converse mapping inference uses and renders that message * test(proxy): type the Kimi K3 count test's tools parameter * perf(count_tokens): validate each kept count option once instead of twice * fix(dot_notation): parse drop paths in linear time when brackets never close * test(count_tokens): benchmark the unclosed-bracket drop path instead of timing it in a unit test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_a2a_benchmarks.py | ||
| test_benchmarks.py | ||
| test_count_tokens_benchmarks.py | ||
| test_extraction_guardrail_benchmarks.py | ||
| test_inference_benchmarks.py | ||
| test_json_fragment_accumulator_benchmarks.py | ||
| test_mcp_benchmarks.py | ||
| test_prompt_template_benchmarks.py | ||
| test_realtime_session_merge_benchmarks.py | ||
| test_responses_compaction_benchmarks.py | ||
| test_sse_lines_benchmarks.py | ||
| test_token_counter_pdf_benchmarks.py | ||