mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
5849906ddb
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route /v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends. A body inference would reject now answers the same 400 before any provider call. DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning mapping inference uses, and falls back to the estimate only when that route is missing. * perf(proxy): delete count drop paths from one copy of the options additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear * fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400 additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500 The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models * fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any * fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status /utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none * fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it * fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count * fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle * fix(bedrock): send the thinking beta headers on Mantle counts A Mantle count with thinking display updates or block_binding now sends the same anthropic-beta header Mantle inference sends * fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test * fix(deepseek): count only the tools DeepSeek chat inference sends --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_a2a_benchmarks.py | ||
| test_benchmarks.py | ||
| test_count_tokens_benchmarks.py | ||
| test_extraction_guardrail_benchmarks.py | ||
| test_inference_benchmarks.py | ||
| test_json_fragment_accumulator_benchmarks.py | ||
| test_mcp_benchmarks.py | ||
| test_prompt_template_benchmarks.py | ||
| test_realtime_session_merge_benchmarks.py | ||
| test_responses_compaction_benchmarks.py | ||
| test_sse_lines_benchmarks.py | ||
| test_token_counter_pdf_benchmarks.py | ||