rj-cairn 5eb1042bbf fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728) ... * fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): move the o200k provenance note into the assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust_bridge): azure/gpt-4o now resolves to o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know * fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer * fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model. * fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model * fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model * fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted. * test: type the new token-counting tests' parameters * fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it * fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test * perf(token-counter): classify each distinct served model once when walking a model group Budgeted requests walk every deployment in their model group, since the router's cached copy is outside the router surface the proxy may use. Classifying each distinct served model once instead of once per deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms with the same result. --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>		2026-10-29 00:08:12 -07:00
..
a2a	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
aiml	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
aiohttp_openai	fix(aiohttp_openai): forward service_tier in transform_response (#47263)	2026-10-28 13:14:44 -07:00
anthropic	fix(context_management)!: count the system prompt toward the clear_tool_uses input_tokens trigger (#47857)	2026-10-28 21:35:32 -07:00
apiserpent	test(unit): add package markers to migrated unit test directories	2026-09-20 11:53:14 +00:00
aws_polly	fix(aws_polly): read the engine from the bare model name (#47963)	2026-10-23 18:10:51 -07:00
azure	fix(responses): drop params compact rejects and send Azure compaction to the v1 route (#48876)	2026-10-28 19:46:36 -07:00
azure_ai	feat(azure_ai): reach Foundry agents through the project Responses API (#47717)	2026-10-28 20:49:58 -07:00
base_llm	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
baseten	fix(baseten): list current Model APIs instead of retired 2023 deployment IDs (#46945)	2026-10-19 16:40:13 +00:00
bedrock	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
bedrock_mantle	fix(bedrock): resolve AWS region and secret lookups off the event loop (#47331)	2026-10-20 17:04:41 +00:00
black_forest_labs	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
brave	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
bytez	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
cerebras	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
chat	fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493)	2026-10-14 14:19:25 -07:00
chatgpt	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
clarifai	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
claude_code	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
cloudflare	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
codestral	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
codex	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
cohere	fix(cohere): map string stop parameter to list of strings for stop_sequences (#47091)	2026-10-27 19:01:54 -07:00
cometapi	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
compactifai	refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)	2026-10-24 11:31:09 -07:00
crusoe	fix(crusoe): update base URL to current endpoint (#33121)	2026-10-15 19:10:58 -07:00
custom_httpx	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
dashscope	feat(dashscope): add separate explicit and implicit cache pricing (#45567)	2026-10-20 04:54:35 +00:00
databricks	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
dataforseo	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
datarobot	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
deepagents	feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885)	2026-10-01 22:27:49 +00:00
deepgram	refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)	2026-10-24 11:31:09 -07:00
deepinfra	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
deepseek	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)	2026-10-27 10:08:25 -07:00
docker_model_runner	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
duckduckgo	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
e2b	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
edenai	feat: generate provider_endpoints_support.json from the cost map (#48619)	2026-10-27 12:06:55 -07:00
elevenlabs	fix(elevenlabs): map wav and pcm TTS formats to 24kHz output formats (#48455)	2026-10-26 13:42:01 -07:00
exa_ai	fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423)	2026-10-19 15:45:56 -07:00
fal_ai	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
fastcrw	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
featherless_ai	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
firecrawl	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
fireworks_ai	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
friendliai	fix(friendliai): drop litellm-internal message fields before sending (#45092)	2026-10-20 01:22:07 +00:00
gdc	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
gemini	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
gigachat	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
github_copilot	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)	2026-10-27 10:08:25 -07:00
google_pse	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
gradient_ai	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
groq	fix(groq): allow streamed reasoning replies in history (#47931) (#48268)	2026-10-25 00:52:01 +00:00
heroku	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
hosted_vllm	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)	2026-10-11 07:58:53 +00:00
huggingface	fix(huggingface): stop forcing truncate=false on TEI rerank and pass truncate params through (#48706)	2026-10-28 09:27:26 -07:00
hyperbolic	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
inception	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
infinity	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
jina_ai	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
lambda_ai	test: move the unit half of 126 mixed legacy files into tests/unit (#45090)	2026-10-07 14:07:43 -07:00
langflow	fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294)	2026-09-26 09:25:13 -07:00
langgraph	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
laya	feat: add Bespoke Nimble gateway and OSS classifier support (#44246)	2026-10-02 19:37:08 -07:00
lemonade	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
linkup	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
litellm_proxy	fix(responses): drop params compact rejects and send Azure compaction to the v1 route (#48876)	2026-10-28 19:46:36 -07:00
llamafile	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
lm_studio	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
manus	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)	2026-10-11 00:17:21 -07:00
meta	fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939)	2026-10-16 15:24:40 +00:00
meta_llama	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
microsoft_365_copilot	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)	2026-10-27 10:08:25 -07:00
milvus	test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)	2026-10-11 06:24:57 -07:00
minimax	feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos (#43705)	2026-10-15 05:48:36 -07:00
mistral	fix(mistral): replay assistant reasoning as a Mistral thinking chunk (#48778)	2026-10-28 11:41:33 -07:00
modelscope	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
mongodb	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
moonshot	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
morph	feat(morph): support open models and cached input pricing (#48676)	2026-10-27 18:06:14 -07:00
nadir	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
nebius	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
neosantara	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
nimble	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)	2026-10-07 05:02:28 +00:00
nlp_cloud	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
novita	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
nscale	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
nvidia_nim	fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop (#46967)	2026-10-16 18:06:01 -07:00
nvidia_riva	feat(nvidia_riva)!: add riva_offline mode for offline-only ASR models (#41021)	2026-10-20 23:03:56 +00:00
oci	fix(oci): drop non-function tools under drop_params and gate reasoning_effort on declared levels (#40732)	2026-10-15 22:20:02 +00:00
ocr	test(unit): make every tests/unit directory a package so pytest collection is unique	2026-09-20 11:50:59 +00:00
ollama	fix(ollama): forward embedding credentials, headers and timeout (#48707)	2026-10-28 09:32:47 -07:00
oobabooga	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
openai	fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728)	2026-10-29 00:08:12 -07:00
openai_like	feat!: add StreamLake provider metadata (#41006)	2026-10-28 20:19:06 -07:00
opencode	fix(opencode): bill OpenCode Go requests from the reported list_cost (#48254)	2026-10-24 16:44:29 -07:00
openrouter	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
opensandbox	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
ovhcloud	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
parallel_ai	test: add __init__.py to phase 12 unit test directories	2026-09-20 12:36:15 +00:00
parasail	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
pass_through	fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276)	2026-10-26 12:07:58 -07:00
perplexity	fix(responses): drop the null created_by OpenAI rejects when output items are replayed (#48783)	2026-10-28 21:11:49 -07:00
pg_vector	fix(security): remove the publicly known master key from the repo (#44718)	2026-10-06 10:55:24 -07:00
pi	feat(harness): add pi harness for litellm.agent() (#45467)	2026-10-20 09:01:40 +00:00
publicai	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)	2026-10-11 07:58:53 +00:00
ragflow	test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)	2026-10-11 06:24:57 -07:00
recraft	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
reducto	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
replicate	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
runwayml	refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)	2026-10-24 11:31:09 -07:00
s3_vectors	test: add __init__.py to intermediate phase 12 unit test directories	2026-09-20 12:37:21 +00:00
sagemaker	fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)	2026-10-21 02:26:05 +00:00
sail	test(e2e): align completion, SAIL, and spend-log fixtures with supported contracts (#43902)	2026-09-30 14:21:22 -07:00
sambanova	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
sap	fix(sap): correctly handle cache_control (#39122)	2026-10-06 08:54:06 -07:00
scaledown	feat(scaledown): add native text provider support (#44168)	2026-10-10 14:01:17 -07:00
scaleway	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)	2026-10-11 07:58:53 +00:00
searchapi	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
searxng	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
serper	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
snowflake	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
soniox	feat(soniox): decode proxy form fields for context, translation and flags (#40605)	2026-10-16 06:37:34 +00:00
stability	refactor(types): replace Any with proven types in 137 files (#44478)	2026-10-05 11:01:45 -07:00
strands_decider	feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN (#45678)	2026-10-09 22:51:19 -07:00
tavily	fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search (#48684)	2026-10-27 18:52:06 -07:00
tencent	feat(tencent): add Seedream image generation and editing through TokenHub (#48735)	2026-10-28 15:53:00 -07:00
tinyfish	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)	2026-10-07 05:02:28 +00:00
together_ai	fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)	2026-10-14 21:40:14 -07:00
tool_loop	feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391)	2026-10-03 17:33:01 +00:00
topaz	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
triton	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00
v0	test: move the unit half of 126 mixed legacy files into tests/unit (#45090)	2026-10-07 14:07:43 -07:00
valkey	test: migrate legacy provider tests to tests/unit (wave 2, phase 13)	2026-09-20 14:18:49 +00:00
vercel_ai_gateway	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
vertex_ai	fix(vertex_ai)!: send the caller's anthropic-version in the Vertex Claude /v1/messages body, not the header (#48814)	2026-10-28 18:06:58 -07:00
vllm	fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733)	2026-10-14 12:40:48 -07:00
volcengine	fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop (#46967)	2026-10-16 18:06:01 -07:00
voyage	fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)	2026-10-15 02:59:46 -07:00
wandb	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
watsonx	chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)	2026-10-13 15:35:23 -07:00
xai	feat(search): add native xAI X Search support (#38505) (#48677)	2026-10-28 00:28:51 +00:00
xinference	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)	2026-10-07 05:02:28 +00:00
you_com	test: migrate phase 14 wave 2 provider tests to tests/unit	2026-09-20 11:01:40 +00:00
zai	test: add __init__.py to every tests/unit directory this migration touches	2026-09-20 11:52:09 +00:00
__init__.py	test(unit): add package markers to migrated unit test directories	2026-09-20 11:53:14 +00:00
test_api_base_trailing_slash_routes.py	fix(llms): join api_base and route with one shared helper (#47535)	2026-10-21 06:59:32 +00:00
test_cache_control_and_reasoning.py	test: move tests/test_litellm/llms into tests/unit/llms (#43191)	2026-09-25 12:43:23 -07:00
test_custom_llm.py	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)	2026-10-07 05:02:28 +00:00
test_file_content_block.py	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
test_file_search_responses.py	fix(responses)!: count input tokens the way inference sends the request (#47363)	2026-10-20 15:13:59 +00:00
test_get_complete_url_trailing_slash.py	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)	2026-10-27 10:08:25 -07:00
test_lifecycle_fix.py	refactor(llms): expose public names for private provider helpers (#45037)	2026-10-07 08:49:25 -07:00
test_oss_decision.py	feat(decisions): add Laya and Bespoke providers to the Add Model form (#45260)	2026-10-14 08:11:11 -07:00
test_polling_url_origin_match.py	test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199)	2026-09-25 17:10:13 -07:00
test_predibase_transformation.py	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)	2026-10-28 17:00:29 -07:00