* fix(cost): honor per-second custom pricing on chat completions for every provider
* test(cost): assert a per-second priced deployment bills instead of staying at $0
The zero-cost diagnostic tests from #42345 used a per-second-only entry as their
misconfigured fixture, which this branch now bills. Switch that fixture to a
per-query-only entry, which is still selected as the deployment's own pricing and
still prices chat usage at $0, and add a per-second test asserting the call
duration is billed with no diagnostic
Also let a caller's explicit total_time outrank the logging window in
completion_cost, so the SDK precedence stays stamped response, caller, logging
* test(response_metadata): move the per-second pricing regression into the mapped tests/unit file
* fix(cost): bill a deployment's per-second rates on /v1/responses and /v1/messages
The proxy builds the request's logging state before the router picks a
deployment, and only the chat path re-read the deployment's
input_cost_per_second / output_cost_per_second afterwards. The shared
update step every other route runs (Logging.update_from_kwargs) now
seeds those custom pricing keys from the router-merged kwargs too, so
custom_pricing flips on and the cost calculator prices the call against
the router-registered deployment id on every surface.
* fix(cost): price a /v1/messages call off the response time the proxy reports
A /v1/messages result is a plain dict, so the proxy keeps its response time on
the logging object. The cost step now reads it there for a non-streamed call,
so the cost header and the spend log bill the same duration.
* fix(anthropic): bill a deployment's per-second rate only on the native /v1/messages route
Move the custom pricing copy from the shared logging refresh into the native
Messages handler, reusing the helper the Responses handler already had, so
generateContent and other routes keep their current billing
* fix(cost): pass only per-second rates to native /v1/messages cost tracking
Other deployment overrides keep billing as on main, matching chat's per-second gate
* test(anthropic): pin per-second-only pricing on native /v1/messages with a long-context-rate deployment
* fix(cost): bill native /v1/messages from logging setup, as main did, to the end the proxy reports
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds Kyma API (kyma/<model>) as a JSON-configured OpenAI-compatible provider. Spend reads the usage.cost Kyma reports, with 83 catalog rows as the fallback price, and image and audio routes refuse before calling Kyma. Carries over #39799 with its author's commits kept
* fix(azure_ai): price Foundry relays of the native Cohere v2 embed route
A proxy relay of POST /azure_ai/<group>/providers/cohere/v2/embed logged spend 0 because the
Foundry relay shapes had no entry for it, so the body fell through to the passthrough object
with no usage. The Cohere v2 embed body-to-EmbeddingResponse transform is now one shared
function that the direct Cohere path and the relay both call, priced from meta.billed_units
and falling back to the echoed texts' token count when billed_units is absent. A body that is
not a Cohere embed shape raises ValidationError, so the relay fails open as before.
* fix(cohere): price native Cohere embeds on the v1 Foundry relay and the v2 passthrough, billing image tokens per caller
The Foundry relay now also prices /providers/cohere/v1/embed, and the shared Cohere embed parser
accepts the v1 plain list of float vectors next to the by-type object. Direct Cohere bills
billed_units.image_tokens (or v1 images) apart from input_tokens, while Foundry already folds the
image into input_tokens, so the parser takes an explicit CohereImageTokenBilling and each caller
picks its own. The /cohere passthrough now routes and prices /v2/embed through the same parser
as the cohere/ SDK path. The relay prices any body that carries billed units whatever texts holds,
and the no-billed-units fallback counts special-token text as plain text instead of raising
* fix(cohere): keep embed type labels, per-image v3 pricing, and the plain row for unpriceable passthrough embeds
Passthrough and Foundry relay log rows keep each vector's embedding type,
text-only usage keeps image_tokens null, a Cohere v3 billed image count is
priced per image, and an embed call the cost map cannot price falls back to
the plain passthrough row instead of raising inside the logger.
* fix(cohere): warn when a passthrough embed cannot be priced and pin test timestamps
* test(cohere): cover embed-jobs passthrough calls writing the plain passthrough row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch): keep the provider prefix on the agentic follow-up model
The agentic follow-up rebuilt its model string with "a slash means it is already
qualified". Handlers are handed the provider-stripped model, and for a provider
that routes through a sub-path the remainder still holds a slash, so
bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5
and stayed that way. The follow-up acompletion could not resolve a provider from it
and raised "LLM Provider NOT provided".
Prefix unless the model is already qualified with that same provider, in one shared
helper used by both the handler and the websearch patch builder, which carried
copies of the same heuristic. This also fixes openrouter, whose stripped model
keeps a slash for the same reason, and the old startswith test that read openai as
a prefix of openai_like.
Fixes#38829
* test(websearch): cover the agentic follow-up call site
The helper had unit tests but the call site did not, so the two changed lines in
_execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with
mock_response instead of patching litellm, which keeps the test on the behaviour:
against the old code the sub-path case raises "LLM Provider NOT provided", and the
ordinary-model case passes either way as a guard.
* refactor(websearch): trim the comments on the provider-qualify helper
* refactor(websearch): cut the helper and test docstrings to one line
* fix(websearch): leave a hook-qualified follow-up model on its own provider
A hook that patches in its own model owns that choice, so re-qualifying it sent a
cross-provider follow-up to the original request's provider instead. Only a bare
patched model and the request's provider-stripped model take the prefix now, and
the sibling agentic-loop path resolves the model the same way.
* fix(websearch): fall back to the request model when a patch carries no model
An empty patched model dispatched a provider-only name instead of falling back, and a
provider-stripped sub-path copied into the patch lost its prefix. A patched model now
keeps its own provider only when its leading segment names one.
* fix(websearch): keep the provider prefix on the responses follow-up model too
The responses patch builder still used the old "a slash means already qualified" test, so
a provider-stripped sub-path model reached the follow-up unqualified on that surface. It
now resolves the model the same way the chat completion builder does.
* fix(websearch): read custom_llm_provider without a member call on untyped kwargs
* fix(websearch): type the chat patch builder kwargs instead of working around them
* test(websearch): run the real search path and flatten the qualify helper tests
* fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model
* fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider
* fix(websearch): send the agentic follow-up to the model the caller requested
completion() and responses() record the caller's model before routing strips
it, and the chat and Responses follow-up builders prefer that model. This keeps
bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/*
deployments on their own route for the follow-up.
* test(websearch): cover the recorded caller model in Headroom's retrieval follow-up
* test(websearch): cover the recorded caller model in Compresr's retrieval follow-up
Also merges the repeated provider-prefix check in core_helpers into one helper
and drops a test docstring
* test(websearch): drive the real search path through a mocked Perplexity HTTP route
* refactor(websearch): read the provider from kwargs inside _caller_model
* test(http-handler): type the agentic follow-up provider tests
* refactor(websearch): record agentic loop params through a Logging method
Moving the recording out of completion() keeps it under basedpyright's
code-flow complexity limit after the merge with main.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params
A GPT-5 request whose reasoning_effort the model map does not enable for
that model lost the effort with no log line under drop_params. An Azure
deployment name missing from the cost map lost reasoning_effort none this
way and ran at the model's default effort. The gate now logs a warning
naming the model, the level, the map entry it checked, and the
model_info.base_model and model_info.reasoning_effort_levels fixes.
* fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort
The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without
supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both
Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with
model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from
its underlying model.
* fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops
* fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level
* fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses
* fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop
* fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors
A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live.
* fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them
The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels
(github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and
Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact
suggested levels it never reads either.
Declared levels are now None when a caller passes none, so the advice appears only when a caller fed
them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model.
The openai row of the chat warning test now expects no base_model advice, the Responses warning test
goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure
direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped.
* fix: read only the built-in catalog when choosing the reasoning drop warning
* fix(transformation): merge consecutive system and developer messages for non-OpenAI providers and Responses API
* fix(base_utils): keep later cache_control and billing metadata boundaries when merging system messages
* perf(base_utils): fold consecutive system messages in a single linear pass
The tuple accumulator passed to reduce copied the whole prefix on every step,
making map_developer_role_to_system_role O(n^2) in the number of messages.
Message count is not bounded by default, so a caller could tie up a proxy
worker with thousands of small messages before any provider call. Use a list
and merge into its last element in place instead.
* fix(base_utils): hoist later developer messages into the leading system message for non-OpenAI providers
A developer message that arrives after the first user turn (Codex CLI's
per-turn permissions item, a re-injected instruction after compaction, a
mid-conversation policy update) was translated to a system message in
place, and system-first chat templates such as Fireworks' qwen3.8 reject
any system message that is not at the beginning with 400. The fold that
merges consecutive system messages never saw it because a user turn sat
in between.
Move every developer message that follows the leading system block to
the end of that block before the fold, so it merges into the single
leading system message; client-authored mid-conversation system messages
stay where the client put them.
* fix(openai): hoist later developer messages into one leading system message on OpenAI-compatible backends
Developer role translation moves onto the provider config. OpenAI's own endpoint,
Azure OpenAI, and the o-series keep their current behavior. Every other config
inheriting OpenAIGPTConfig (Fireworks, DeepSeek, Together, vLLM, Groq, openai/
with a custom api_base) hoists developer messages that arrive after the leading
system block into that block and folds the block into a single system message,
which is what system-first chat templates require. Providers with their own
message format are untouched.
Messages carrying cache_control or Anthropic billing metadata fold as their own
text blocks, so breakpoints and provider-side stripping survive the merge. The
Responses bridge goes back to prepending instructions as a separate system
message and lets the hoist fold it.
* refactor(base_utils): fold each run of system messages in one pass
The pairwise fold recopied the accumulated system text or block list on every
merge, so a request carrying many consecutive system or developer messages cost
quadratic time in preprocessing. Group consecutive system messages once and
build the merged message from the whole run: plain strings join with a blank
line, and any cached, billing, or block content keeps one text block per
message. The name of the last named message in the run survives, as before.
* fix(base_utils): keep a closing developer message in place and hoist for OpenAIConfig backends
A developer message that closes the conversation stays where the client put it, translated
to system in place, so the request never ends on the assistant's turn. OpenAIConfig and its
subclasses (Azure AI Foundry, GitHub Copilot, ChatGPT, RAGFlow) now share the OpenAI-compatible
hoist, litellm_proxy passes developer messages through to the downstream proxy, and the Anthropic
billing header literals use the shared constant
* fix(base_utils): keep a closing developer message in place only after an assistant turn
A developer message that closes the conversation after a user or tool message hoists like
any other, so the request shape the ticket reports keeps working on system-first templates;
only a closing run that follows an assistant turn stays in place
* refactor(openai): reuse is_openai_backed_api_base for the hosted endpoint check
* test(base_utils): cover an empty system message inside a block-content run
Codecov flagged the empty-string branch of _text_blocks as the one line the
PR left uncovered. The existing empty-string test only hits the plain-text
join; this one forces the block path by merging an empty system message with
a developer message that carries list content.
* fix(openai): hoist developer messages for litellm_proxy and harden the system fold
- drop the LiteLLMProxyChatConfig passthrough so litellm_proxy/ hoists later developer messages like every other OpenAI-compatible endpoint, whichever backend or LiteLLM version sits downstream
- a system message with null or missing content inside a folded run is dropped instead of raising KeyError('content')
- a message-level cache_control on a list-content member lands on its last block so the breakpoint survives the fold
* fix(litellm_proxy): keep the AllMessageValues import the transform signatures still name
* fix(fireworks_ai): hoist developer items after the first turn on the native Responses path
The native Fireworks Responses path landed on staging mapping developer input items to
system in place, so a developer item after the first input item still hit the same
"System message must be at the beginning" 400 on system-first chat templates. The
ordering rules now live in one shared helper over item roles, and the native path
reorders through it before mapping, keeping a closing developer item after an assistant
turn in place exactly like the chat path does
* fix(transformation): carry the developer-message hoist onto main 963333d654
Resolves the main merge for this branch: the tests move to tests/unit, every translate_developer_role_to_system_role override takes and returns a Sequence, Azure OpenAI keeps the in-place mapping, and the system content reads tolerate a missing key
* fix(transformation): join a lone plain text block as text when hoisting developer messages
The Responses bridge turns every developer item into a one-block list, so the hoisted run merged the
instructions string and that block as a block list, and Databricks could no longer collapse it into the
single system string it sends today. A lone text block with no extra keys now joins the run as text,
while multi-block, cached, and billing-header content keep the block path
* fix(transformation): hoist later developer messages only for models that reject mid-conversation system messages
Runs of consecutive system messages still fold into one, which keeps the cached prefix stable. A developer message after the first turn moves into the leading system message only when the model declares supports_mid_conversation_system false, or when it is a Fireworks model with no declaration, matching the native Fireworks Responses route. Every other backend keeps it in place as before, so their prompt cache prefix is unchanged. Malformed system content passes through unmerged so the provider returns its own 400.
* fix(responses): fold later developer items into instructions on hosted_vllm for system-first models
Moves the Responses developer fold shared by Fireworks into one module and applies it to hosted_vllm native Responses when the model declares supports_mid_conversation_system false. Renames the run-merging helper so it no longer shares a name with the Databricks one, and returns tuples from the block helpers.
* fix(llms): place developer messages only for models declared system-first
Models without a supports_mid_conversation_system declaration now keep the
in-place developer-to-system translation they had before, so their cached
prompt prefix does not change. Hoisting and folding run only when the
deployment's model_info or the cost map declares the flag false, with the
deployment value read first.
A developer message that closes the conversation after an assistant turn is
sent as a user message on those models instead of being hoisted, merged list
content keeps a blank-line separator before each later part, and the cost map
declares the flag false for fireworks_ai qwen3p8-max
* fix(proxy): keep a closing developer message's cache mark when it is sent as a user turn
* test: type the developer-hoist tests and fake HTTP with respx
* fix: count non-string roles as turns, keep the Fireworks fold for twins, and copy closing developer content
A message whose role is a list or dict now counts as a conversation turn instead of raising a 500.
Fireworks native Responses skips the developer-item fold only when the deployment's own model_info
declares supports_mid_conversation_system true, since the router registers every deployment's flag
under its shared backend key. A hoisted developer message keeps only role, content, name, and
cache_control, a closing developer message's content is deep-copied so the caller's list is left
alone, and a hosted_vllm developer item with no content is forwarded unchanged.
The Fireworks in-place test now declares true on the deployment itself, because a cost-map true can
no longer be told apart from another deployment's registration.
---------
Co-authored-by: sowmy narayanan <sowmysuresh86@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window
A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window
* fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL
The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend
* fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time
The loaded outcome now holds the tokenizer, so an LRU eviction can no longer
send every count back to an unguarded download. A retry blocks further retries
until it ends, even when it runs past the window, and a download that fails
after another count loaded the tokenizer counts with that tokenizer
* fix(token_counter): retry a failed Hugging Face tokenizer download in the background
After the retry window, the count that finds the stored failure starts one
background download and answers with tiktoken, so no live count waits on a
retry. A failure from a download that started before the first failure no
longer resets a running retry, and a failed retry never replaces a tokenizer
another download loaded. The outcome store takes its clock and retry starter
as dependencies, so the tests inject them instead of patching time.monotonic.
* fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start
If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart.
* fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again
A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load
* fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(github_copilot): treat a malformed api-key.json like a missing one
* fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views
* fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file
Read the cached key file as bytes so a UTF-16 file, which Windows
PowerShell 5.1 writes with >, fails JSON validation instead of raising
the decoder's error, and the request gets the refresh or the login hint
* fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field
* fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one
* fix(github_copilot): use the refreshed key when api-key.json cannot be written
* fix(github_copilot): keep a refreshed key that cannot be saved in memory
* fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired
A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it
* test(github_copilot): type the authenticator test fixture and the tests this branch adds
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): use collection endpoint for custom context cache
* fix(vertex_ai): build the cachedContents collection URL under a custom api_base
* fix(vertex_ai): keep a gateway prefix that contains /publishers/
* test(integration): expect the cachedContents collection URL on a custom Gemini api_base
* fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix
* fix(vertex_ai): cache under a publisher model api_base in that model's location
A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so
no cache is created that the chat call cannot use. The cache body takes its project and
location from the api_base model, matching the collection URL.
* fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base
A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field.
* fix(vertex_ai): check the cache token minimum against the api_base's model
The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create.
* fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached
---------
Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path
* refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge
* fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper
* fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints
Also drop a marker set on the message item itself, sanitize before the
cache_control conversion so a client typo no longer erases that breakpoint,
and skip items that carry no marker so unmarked history costs no extra walk
* fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it
A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a
litellm_proxy alias) kept its valid markers before this PR and must keep them under
drop_params. base_model is read from litellm_params first, then model_info, as chat
completions does.
* fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged
* fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers
* fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied
* fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one
Under drop_params the marker step now hands back the caller's own objects when every marker is
already wire-safe, so a request with valid markers is no longer copied and walked a second time.
A deployment that turns cache_control into markers keeps the caller's own markers too, whatever
its base_model says, so one request never gets two answers about breakpoint support.
On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion,
so the block gets the cache hint cache_control alone would give, as on /v1/responses.
* test(responses): keep a null marker and the original key order on a model that reads breakpoints
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions
Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema
* fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens
Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema.
* fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models
Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them.
* fix(bedrock): drop tool custom object on the Mantle count_tokens fallback
Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way.
* fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages
* test(vertex): annotate count_tokens tool test locals with Final
* fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names
* fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add BigModel provider with native Chat, Responses, and Messages APIs
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* test(providers): make BigModel unit tests a package
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* docs(providers): clarify BigModel parameter interface
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* fix(bigmodel): keep the Coding Plan base, surface Responses auth errors, and scope routes
Chat Completions keeps a GLM Coding Plan api_base. Responses raises the status from
BigModel's success=false body instead of returning an empty 200, and a JSON body
answering a stream request is replayed as a one-shot stream. Messages streams keep
ping and [DONE] lines. bigmodel leaves openai_compatible_providers so speech, image,
and transcription routes send it no request
* fix(responses): replay a JSON body on a stream request only when it has a Content-Length
A gateway that labels a chunked SSE stream as application/json kept streaming before
the replay, so only a sized body is read in full
* fix(bigmodel): scope the JSON stream replay to BigModel, map its business error codes, and echo requested tools
* fix(bigmodel): forward Codex's include and parallel_tool_calls, keep the OpenAI org header off BigModel, and pass unchanged Messages stream events through
* fix(bigmodel): map disabled thinking to low effort, keep Messages temperature, raise context window and content policy errors, and default parallel_tool_calls to true
* fix(bigmodel): keep the caller's reasoning_effort when Messages rewrites disabled thinking
* fix(bigmodel): keep context-window and content-policy errors on streamed Responses replays
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(token_counter): move the o200k provenance note into the assert message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust_bridge): azure/gpt-4o now resolves to o200k_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know
* fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer
* fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer
A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model.
* fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model
* fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k
A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count
base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response
Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model
* fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written
Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted.
* test: type the new token-counting tests' parameters
* fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it
* fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test
* perf(token-counter): classify each distinct served model once when walking a model group
Budgeted requests walk every deployment in their model group, since the
router's cached copy is outside the router surface the proxy may use.
Classifying each distinct served model once instead of once per
deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms
with the same result.
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(context_management): count the system prompt toward the clear_tool_uses input_tokens trigger
Anthropic's API compares an input_tokens trigger with the whole prompt, system
included; the polyfill counted messages and tools only, so it cleared late by the
system prompt's size. compact_20260112 already counts it: share its system_to_text.
* fix(context_management): take read-only types in the system prompt count
The type-discipline gate (LIT001) rejects new list and dict annotations.
* chore: rerun CI (proxy-build-from-pip timed out)
* fix(context_management): count the system prompt through compact's shared counter
Finishes the main merge's conflict in clear_tool_uses.py. The input_tokens trigger now adds the system prompt through count_system_tokens, the same counter compact uses, while the count reused for cleared_input_tokens stays messages and tools only, so the reported savings are just the cleared tool results
* perf(context_management): skip the token count when nothing can be cleared
The keep check runs before the trigger count, the system prompt reader accepts any mapping block, and the system prompt test gets short parameter ids
* test(context_management): keep the off-loop count test counting after the keep check moved first
* test(context_management): cover a client compaction summary counting toward the trigger
The trigger also skips the system prompt count when messages and tools alone already pass it
* fix(context_management): move a client compaction summary into the system prompt when clearing runs before compact
* fix(context_management): clear only the history after a client compaction block so compact still replaces its summary
* fix(context_management): keep tool calls beside a client compaction block clearable
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed
* fix(responses): drop a null created_by from replayed input items too
A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null.
* fix(responses): drop a null created_by from counted input items too
* test(responses): cover saved reasoning nulls and re-compaction of a saved conversation
* fix(responses): drop a set created_by from input under drop_params
* fix(types): type the Responses output serializer's parameters
* fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(azure_ai): cover Foundry agents on the project Responses API
* feat(azure_ai): reach Foundry agents through the project Responses API
* fix(azure_ai): keep system prompts and project-endpoint auth on Foundry agent calls and follow-ups
* fix(azure_ai): read AZURE_SCOPE before the Foundry audience default on project endpoints
* fix(azure_ai): reject or drop instructions on previous_response_id turns and a caller's other agent for Foundry agents
* fix(azure_ai): apply the other-agent rule to agent_reference sent at the top level of a Foundry agent request
* fix(azure_ai): match the caller's agent by name and version, and hide the agent's own instructions
* fix(azure_ai): keep instructions on plain Foundry models that return a null agent_reference
* fix(azure_ai): keep a Foundry agent version requested at the top level when extra_body also names the agent
* fix(azure_ai): answer 400 for /responses/compact on a Foundry agent instead of calling the project with the agent name
* feat: add StreamLake provider metadata
* fix(streamlake): complete provider discovery and test request flows
* test(streamlake): assert cost and tool-param invariants instead of pinning vendor prices
* fix(streamlake): accept tools and reasoning on endpoint ids and match the current price page
Endpoint ids have no cost-map row, so tools and reasoning_effort were rejected
before reaching StreamLake. The provider now declares both capabilities, and
named non-reasoning rows opt out explicitly.
The rows now follow StreamLake's 2026-09-17 price page: drop the KAT-Coder and
Qwen3 rows it no longer serves, add the served models that had no price, drop
account-level rpm and tpm, and fill missing context windows. StreamLake is also
registered for streamlake/* wildcards and its messages and responses support.
* fix(streamlake): regenerate the endpoint matrix and type the new tests
The backup endpoint matrix advertised chat only while the root map listed
Messages and Responses, so /public/endpoints hid both. Regenerating with
scripts/generate_provider_endpoints_support.py makes the two files match.
Types the StreamLake tests and sets the Add Model test's API key with
fireEvent.change, per the dashboard test guide.
* fix(streamlake): drop the retired max_tokens field from the StreamLake cost-map rows
* fix(streamlake): price the four served models StreamLake publishes rates for and add the Qwen3.8 cache prices
* fix(streamlake): send token caps as max_completion_tokens and mark image models
StreamLake ignores max_tokens on kimi-k2.6, kimi-k2.7-code, qwen3.6-27b and
qwen3.6-35b-a3b, while every served model honors max_completion_tokens, so
the JSON provider now maps max_tokens to max_completion_tokens instead of the
reverse. Rows whose models read an image in a live probe set supports_vision,
and the Qwen3.8 rows drop the explicit cache write price, which this route
never uses since cache_control is dropped
* fix(streamlake): keep cache_control marks, price explicit cache reads, and use API input limits
StreamLake reads cache_control, so the provider keeps the marks through a preserve_cache_control flag. Qwen3.8 explicit cache hits now use their own read price, rows with a cache read price declare prompt caching, three models StreamLake serves without reasoning drop reasoning_effort, and max_input_tokens follows the limits the API reports.
* fix(utils): warn when a custom_provider_map name matches a built-in provider
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): scan Responses tool results in texts-mode guardrails
* fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans
Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list.
* fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans
* fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text
* fix(guardrails): scan each Responses tool output text part on its own
* fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id
* fix(guardrails): pair each Responses tool output with its own structured row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop metadata from compact requests to OpenAI and Azure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop user and drop_params-rejected fields on compact and route Azure compaction to v1
* fix(responses): keep stream on compact and keep non-dated Azure api-versions on the v1 compact route
* fix(responses): drop compact params only for providers verified to reject them
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages
Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes.
* fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body
Vertex reads the API version from the body alone, so the caller's
anthropic-version header now becomes the body's anthropic_version instead
of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party
versions keep vertex-2023-10-16, and the header itself is still never sent.
Same change on the Python proxy and the Rust gateway.
* fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept
Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are
carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no
header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under
drop_params and answers a 400 naming drop_params without it, on the Python proxy
and the Rust gateway.
* refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place
* fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python
The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there
* docs(rust router): name Router::from_config in the crate README
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
OpenAI shut down dall-e-2 and dall-e-3 on 2026-05-12. Drop their image
generation and edit configs, the Azure dall-e-2 create-and-poll flow, and
the silent dall-e-2 fallback when no model is passed.
image_generation without a model or provider now fails locally with
"model is required". image_edit and image_variation leave model out of
the provider request instead of sending dall-e-2, so providers that
ignore model (Topaz) keep working and OpenAI returns its own 400.
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream
The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.
SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF
Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed
Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.
Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.
Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.
* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests
Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.
* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines
* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter
* test: type the iterator class parameters in the line-separator stream tests
* fix(responses): stream tool call arguments opened in a text chunk
* fix(responses): hold a tool call that shares a chunk with text until the text ends
* test: type the a2a SSE handler helper and the sandbox fake responses
---------
Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* fix(gemini): preserve every grounding web source
* test(gemini): cover all grounding sources per passage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Zabir Raihan <zabirraihan@icloud.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): replace Any with proven types in 52 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): use typing_extensions.TypedDict for validated seams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert strict Anthropic response validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus
* test(bedrock): annotate the Pegasus null finish reason test
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(tencent): add Seedream image generation and editing through TokenHub
* feat(tencent): price Seedream image generation and edits from TokenHub usage
* test(tencent): cover Seedream image generation, edits, and pricing through TokenHub
* fix(tencent): keep non-Seedream image models on the OpenAI-compatible route
Only seedream- models use TokenHub's wand route, so Hy-Image and other
tencent image models keep working as on main. The response now carries
TokenHub's image size and format, Pro's flat fallback bills its default
2K size, and the new tests are typed.
* fix(tencent): bill other Tencent image models as before and forward user and TokenHub params on edits
* fix(tencent): keep dropped edit params out and parse form-string watermark and seed
* fix(tencent): send extra_body edit fields to TokenHub as given
* fix(tencent): keep the routed model on Seedream edits and bill a deployment's per-token price
* fix(tencent): accept string drop params on edits, rebuild nested form fields, and honor a per-image deployment price
An image edit with a non-list additional_drop_params answered 500 for every provider after this branch validated it as a list, so a non-list value now drops nothing again, as chat treats it.
Seedream edits sent through the OpenAI SDK arrive as multipart form fields, so TokenHub-only booleans and integers are parsed and bracketed keys like optimize_prompt_options[mode] are rebuilt into objects. A deployment that sets only output_cost_per_image is billed that price, and the TokenHub URL uses the shared api_base helpers.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): let the per-model budget field take a model pattern or access group
* feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas
* fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time
Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and
the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting
on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no
longer offers a typed pattern another row already budgets.
* fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica
Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its
own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local
window another charge on the same replica opened meanwhile. Window tests use fixed timestamps.
* fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry
Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin
must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a
provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry
counts as outranking every pattern.
* fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI
* fix(proxy): keep model family budgets through database blips, team models and key entries
A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers
A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments
The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route
* fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry
A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or
a group) at or under the family cap, and its own counter then multiplied the
team's spend on that family. Any new entry that takes precedence over a capped
entry now needs a proxy admin, and group members include the team's own public
model names so team deployments cannot hide a group overlap.
* test(router): cover team_public_model_names_of
* docs(proxy): name pattern and access group keys in model_max_budget docstrings
* fix(proxy): name the entry holding the cap in the team model budget raise refusal
* fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect
* fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap
* fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration
* fix(aiohttp_transport): keep the owned session open while a response stream is still being read (LIT-10237)
* test(aiohttp_transport): annotate the stream regression test locals with Final and drop its docstring
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pricing): add databricks priority pay per token prices
* fix(pricing): bill gemini 3.6 flash cache writes at its databricks input rate
* fix(pricing): drop image token priority rate the cost map schema does not carry
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them
* fix(aiohttp_openai): forward service_tier in transform_response
* chore(aiohttp_openai): mark service_tier assignment rebind-ok for the type gate
* fix(aiohttp_openai): set service_tier with setattr like other providers
ModelResponse has no declared service_tier field, so attribute assignment
added a reportAttributeAccessIssue error to the type gate.
* test(aiohttp_openai): verify serialized served service tier
* fix(aiohttp_openai): copy response when preserving served tier
---------
Co-authored-by: tin-berri <tin@berri.ai>
* feat(providers): add ZeroGPU as an OpenAI-compatible provider
Registers zerogpu through the JSON provider registry so zerogpu/<model> routes to
https://api.zerogpu.ai/v1/chat/completions with ZEROGPU_API_KEY and an optional ZEROGPU_API_BASE.
Adds LlmProviders.ZEROGPU, the constants.py provider and endpoint lists, cost map entries for the
10 ZeroGPU text generation models, the endpoint matrix, the Add Model form fields, the README row
and the dashboard provider entry
* feat(zerogpu): add capability flags verified against the live API
Live calls to https://api.zerogpu.ai/v1/chat/completions confirmed tools, tool_choice and json_schema
response_format on 8 of the 10 catalog text models, cached token reporting on deepseek-v4.1-flash,
gpt-4.1-mini, gpt-5.4-nano and gpt-5.6-luna, and reasoning_effort on gpt-5.6-luna. LFM2.5-1.2B-Instruct
answers stream requests with plain JSON instead of SSE and LFM2.5-1.2B-Thinking returns 500 on every
request, so both stay out of the cost map until the API serves them correctly
* feat(zerogpu): price LFM2.5-1.2B-Instruct
Adds zerogpu/LFM2.5-1.2B-Instruct to the cost map with the catalog prices and context window. It
gets no capability flags because the live API ignores tools, response_format and
max_completion_tokens for this model, so LiteLLM drops tool params for it instead of sending them.
LFM2.5-1.2B-Thinking stays out of the cost map
* fix(zerogpu): reject transcription requests before calling the upstream API
ZeroGPU only serves chat completions, so drop it from the OpenAI transcription transport set. A zerogpu transcription request now fails locally instead of hitting an /audio/transcriptions endpoint that does not exist
* test(zerogpu): cover the responses and messages bridges
Both endpoints reach ZeroGPU through chat completions, so assert the request each one sends upstream and the reply it returns
* refactor(zerogpu): move zerogpu out of the provider list tails
New upstream providers append to the end of openai_compatible_endpoints, openai_compatible_providers and providers.json, so an entry at the tail conflicts with every one of them. Placing zerogpu next to long-stable entries keeps those merges clean. No behavior change
* fix(zerogpu): price cached input for qwen3-30b-a3b-fp8
The ZeroGPU model catalog (https://docs.zerogpu.ai/docs/model-catalog, checked 2026-10-09) lists cached input for qwen3-30b-a3b-fp8 at $0.05 per 1M tokens. Without cache_read_input_token_cost, cached tokens were billed at the full $0.10 input rate
* revert(zerogpu): leave the audio transcription provider set unchanged
Per review, keep OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS as upstream defines it. This PR only covers chat completions, and ZeroGPU now serves /audio/transcriptions anyway, so the local refusal and its test are removed
* fix(zerogpu): add max_tokens so trim_messages can size ZeroGPU history
trim_messages reads max_tokens as the fallback for max_input_tokens and evaluates it eagerly, so without it the lookup raised, got swallowed, and the full history came back untrimmed. max_tokens matches each model's maxTokens in ZeroGPU's models API (https://api-dashboard.zerogpu.ai/api/models, checked 2026-10-10)
* test(zerogpu): record mocked requests on a recorder object
The mock handler appended to a list the caller passed in, which AGENTS.md
forbids. A recorder now owns the captured requests as a tuple
* fix(zerogpu): cap max_tokens at each model's output limit
Each cap is the max_output_tokens ZeroGPU stores for the model in its production models table (read 2026-10-20), the same column its /v1/models serves as max_length. That gives 65536 for deepseek-v4.1-flash, 32768 for gpt-4.1-mini and gpt-oss-120b, 128000 for gpt-5.4-nano and gpt-5.6-luna, 8192 for LFM2.5-1.2B-Instruct and llama-3.1-8b-instruct-fast, and 16384 for qwen3-30b-a3b-fp8. glm-5.3-flash is stored as 32786, which reads as a typo for 32768, so that row uses 32768
trim_messages sizes history from max_input_tokens, so lowering max_tokens leaves it unaffected
* fix(zerogpu): flag prompt caching on glm-5.3-flash
Repeating the same ~4.3K-token prompt against ZeroGPU on 2026-10-20 returned cached_tokens of 4320 on some glm-5.3-flash calls. gpt-oss-120b and llama-3.1-8b-instruct-fast returned 0 on all nine calls, and qwen3-30b-a3b-fp8 never returned cached_tokens, so they stay unflagged. Their cached prices stay as ZeroGPU publishes them, so any cache hit is still billed at the right rate
* fix(zerogpu): set gpt-5.4-nano context window to 272000
ZeroGPU accepts a 271001-token gpt-5.4-nano prompt and rejects 273000 tokens as over the context window (checked 2026-10-21), so max_input_tokens is 272000 rather than the 400000 its catalog lists
* fix(zerogpu): list zerogpu models for zerogpu/* wildcards
zerogpu had no models_by_provider entry, so get_provider_models("zerogpu") returned None and a zerogpu/* wildcard listed no models. zerogpu_models is now filled from the cost map rows like openinfer_models and registered in models_by_provider
* fix(zerogpu): flag gpt-5.4-nano as a reasoning model
ZeroGPU honors reasoning_effort on gpt-5.4-nano (checked 2026-10-21). Without it the model spends 0 reasoning tokens, and with low or high it spends 32 or more. LiteLLM only treated it as a reasoning model by falling back to OpenAI's bare gpt-5.4-nano row, so get_model_info and /model/info reported supports_reasoning as null. The row now carries the flag itself
* fix(zerogpu): flag deepseek, glm, gpt-oss and qwen3 as reasoning models
ZeroGPU's /v1/models marks these four as reasoning models and accepts reasoning_effort on all of them (checked 2026-10-21), but LiteLLM rejected the param with UnsupportedParamsError because their rows had no supports_reasoning flag. glm reports reasoning_tokens under reasoning_effort, and gpt-oss and qwen3 spend hidden reasoning tokens on every reply. The gpt-oss test case now expects reasoning_effort to reach ZeroGPU
* fix(zerogpu): set gpt-oss-120b context window to 128000
ZeroGPU routes gpt-oss-120b across several upstream backends, and at least one of them returns 413 past a 128000-token window, so max_input_tokens uses that as the safe floor. The backends reached on 2026-10-21 accepted up to about 131K prompt tokens, so this only gives up a little headroom on the larger ones
* feat(zerogpu): call ZeroGPU's Responses API natively
ZeroGPU serves /v1/responses for all nine catalog models, so LiteLLM no longer bridges Responses requests through chat completions. Native calls return ZeroGPU's reasoning output items for glm, gpt-oss and qwen3, which the bridge dropped. Checked live on 2026-10-22 through litellm.responses for every model, non-streaming and streaming, plus a gpt-oss tool call. LFM2.5-1.2B-Instruct streams correctly now that ZeroGPU's text_generation lane emits Responses SSE
* fix(zerogpu): stop inferring the zerogpu provider from api_base alone
Listing https://api.zerogpu.ai/v1 in openai_compatible_endpoints rerouted bare model names on that api_base from openai to zerogpu. A bare gpt-5.6-luna call with verbosity used to return 200 and started raising UnsupportedParamsError. Drop the entry and its api_base-alone test, so zerogpu is only selected through the zerogpu/ prefix
* fix(zerogpu): price gpt-6-luna, llama-3.2-3b, LFM2.5-1.2B-Thinking and qwen3.6
A zerogpu/* wildcard served these four chat models with no cost-map row, so their calls logged $0 spend. Add rows at ZeroGPU's catalog prices from https://api-dashboard.zerogpu.ai/api/models, with context windows and output caps from ZeroGPU's models table. LFM2.5-1.2B-Thinking has no cached price, like LFM2.5-1.2B-Instruct. gpt-6-luna is flagged supports_reasoning because LiteLLM already routes gpt-6 names as reasoning models. Checked live on 2026-10-28, all four now report nonzero response_cost
* fix(zerogpu): drop the retired max_tokens field from the zerogpu rows
Upstream #48471 retired max_tokens from the cost map in favor of max_input_tokens and max_output_tokens, and test_no_row_carries_the_retired_max_tokens_field failed on all 13 zerogpu rows. Every row already carries the same value in max_output_tokens, so nothing is lost. The zerogpu limits test now checks that each row's output limit fits its context window
* chore(zerogpu): regenerate provider endpoint support for zerogpu
Upstream #48619 generates provider_endpoints_support.json from the cost map and checks it in CI, and the hand-written zerogpu entry no longer matched. Regenerated with scripts/generate_provider_endpoints_support.py. Only the zerogpu entry changed, and it now matches what the generator emits for other JSON-registry providers such as openinfer
* fix(zerogpu): flag tools, json_schema and prompt caching on gpt-6-luna
ZeroGPU serves function calling, tool_choice, json_schema response_format and prompt caching on gpt-6-luna, so set supports_function_calling, supports_tool_choice, supports_response_schema and supports_prompt_caching on its row in both cost maps. Checked live on 2026-10-28, a weather tool call came back with the right arguments, json_schema output validated, and repeated prompts reported 3260 of 3263 tokens cached
* fix(zerogpu): set qwen3.6-35b-a3b context to 262144 and flag its tools and reasoning
ZeroGPU accepts up to 262,144 input tokens on qwen3.6-35b-a3b and serves tools, tool_choice, json_schema response_format and reasoning_effort on it. Raise max_input_tokens from 16384 and set supports_function_calling, supports_tool_choice, supports_response_schema and supports_reasoning in both cost maps. Before this, LiteLLM rejected tools and reasoning_effort for the model. Checked live on 2026-10-28, a 209,905-token prompt succeeded, an oversized one was refused with a 262144-token limit, and tool calls, json_schema output and reasoning_effort all worked
* fix(zerogpu): flag vision on gpt-6-luna and qwen3.6-35b-a3b
ZeroGPU reads image inputs on gpt-6-luna and qwen3.6-35b-a3b, so set supports_vision on both rows in both cost maps. Checked live on 2026-10-28, both models named the animal in an image_url photo, and the image showed up in their prompt token counts
* test(azure): audit #47348
Add proxy integration tests for #47348's cached Entra token providers and
off-loop token refresh, and fix the token herd the audit found: once the
fetch moved to a worker thread, concurrent cold requests each fetched their
own token. Every cached bearer token provider now serializes its calls so
the first fetch fills the cache and the rest reuse it
* fix(azure): build each cached Entra token provider once under concurrent first calls
lru_cache does not hold back concurrent misses, so worker threads that miss together each built their own credential and token provider, and each fetched its own token. A lock around each provider cache makes the first caller build it and the rest reuse it
* fix(azure): share Entra token fetches only while no valid token is held
Concurrent callers now share one result (or one error) from the cold
fetch, while callers inside the refresh window keep the still-valid
token. The oidc/azure provider build and the client-assertion exchange
are single-flighted per audience, and the shared helpers no longer call
subscripted generic aliases, which crashed on Python 3.10.
* fix(azure): refresh Entra tokens once while other requests keep the valid token
MSAL treats a token with less than five minutes left as expired, so every
request in that window fetched a new one. Now one request refreshes while
the others keep using the still-valid token.
Requests that waited on a fetch that failed retry once through a fresh
fetch instead of all failing with the first error.
Credentials that only implement get_token keep working, and OIDC token
exchanges are cached per scope.
* fix(azure): bound Entra token waits during an outage
Requests waiting on a failed token fetch now retry only when the fetch
before it succeeded, so during an Entra outage later bursts fail after one
fetch instead of two.
Requests during a refresh only get the held token while it has more than
60 seconds left, and a failed refresh falls back to that token instead of
failing its request.
* fix(azure): share one retry per failed Entra token fetch and raise on a failed refresh
Waiters of a failed fetch now retry the flight they joined, so a fast-failing Entra makes one retry instead of one per waiter. A failed refresh raises again instead of returning the held token, which Postgres token auth read as a fresh one. The DefaultAzureCredential fallback reuses one provider per scope.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): audit #47363
Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses.
* fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call
A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name.
* fix(responses): count a lone Anthropic toolset and name the uncountable tool
The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type.
* fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400
The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint.
* fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through
* fix(responses): count tool_search and namespace tools the way Claude chat inference sends them
The count refused a lone tool_search with LiteLLM's 400 and sent a lone
namespace raw or squashed it into an empty tool, while Claude chat
inference maps tool_search to Anthropic's tool search tool and unpacks
a namespace into its function tools. Both now go through the
deployment's own map_openai_params, so the count sends what inference
sends, and tool_search leaves the list of uncountable types.
* fix(bedrock): count a Claude request with its deployment's own chat translation
Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's
tool search tool, but the Bedrock counter always translated with the Anthropic
config, so the count answered 400 for a request inference serves.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mistral): replay assistant reasoning as a Mistral thinking chunk
When a client sends back an earlier Mistral assistant turn, its reasoning
(reasoning_content, thinking_blocks, or Mistral's own thinking chunk in
content) now reaches Mistral as a ThinkChunk inside content. The top-level
LiteLLM fields stay stripped because Mistral rejects them with 422.
* fix(mistral): replay reasoning only to Mistral models that accept it
Mistral answers 400 to a thinking chunk on models without reasoning support,
so a mid-chat model switch, a fallback, or Vertex AI Mistral broke once the
replay was added. Gate the replay on the same check reasoning_effort uses and
keep dropping the reasoning for the other models, as before.
* fix(mistral): drop native thinking chunks for models without reasoning and mark the devstral, code agent, small and leanstral aliases as reasoning
* fix(mistral): keep the leanstral rows as on main, since their reasoning support cannot be checked live
* fix(mistral): send only the text items of a native thinking chunk, so a reference or image item no longer gets Mistral's 422
* fix(mistral): keep closed false only on the last chunk of a final prefix turn, so an earlier turn no longer gets Mistral's 400
* fix(mistral): drop closed false when merged text follows the thinking chunk, so a prefix turn with text first no longer gets Mistral's 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(chatgpt): derive Responses session id from prompt_cache_key
ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.
Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.
Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
per-request and are skipped, checked in both "metadata" and
"litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
to a stable conversation key
Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.
Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.
* style: ruff format
* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments
- hash prompt_cache_keys containing header-unsafe chars instead of
mapping them through _safe_header_value, whose replacement char could
collapse distinct keys onto one cache shard and serialize unrelated
requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
file per AGENTS.md test-placement rule
* fix(chatgpt): keep session typing and test diff scoped
* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests
* test(chatgpt): record upstream requests with a mock instead of appending to a list
* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys
* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid
* test(chatgpt): type the session id test parameters
* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key
* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence
* fix(chatgpt): keep the trace-id session marker out of provider params
The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(huggingface): cover TEI rerank truncate passthrough and server default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): stop forcing truncate=false on TEI rerank and pass truncate params through
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): keep TEI truncate options provider-specific
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(huggingface): use MockTransport for rerank truncation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* test(streaming): cover reasoning text and error frames that mention [DONE]
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): avoid assistant prefill in Responses fallbacks
* fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations
A Responses fallback to a Claude model that takes assistant prefill now gets
the partial response as prefill again, so it continues mid-word without the
prompt-shape drift. Every other fallback keeps the request to continue.
A stream cut after text no longer continues when the request constrains its
output (a json_schema or json_object text format, or a forced tool_choice),
since a continued JSON document or tool call would not parse. The client gets
the primary's error, as on the chat path.
The fallback's re-sent leading space at the seam is dropped, and the prefill
shaping runs per deployment after the encrypted reasoning strip, so stripped
reasoning is never replayed.
* fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output
Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's
error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools
in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its
prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted.
* fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call
* fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls
A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main
* fix(responses): count a stream of tool arguments as a started tool call
* fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls
A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): let drop_params drop speed on claude-opus-4-7 after Anthropic removed its fast mode
* test(anthropic): cite the Claude Code fast-mode page for the Opus 4.7 removal date
* test(anthropic): cover the drop_params speed drop for claude-opus-4-7
* test(anthropic): type the model parameter of the speed drop test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params
* test(bedrock): cover the strict tools 400 retry under drop_params
* fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict
* test(bedrock): return captured requests from the strict tools test helper
* fix(bedrock): retry without strict only when a tool sent strict to Bedrock
A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list
* fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts
An application inference profile with base_model sends strict on the wire, so the retry check now resolves the
capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a
strict-grammar rejection too, so it takes the same single retry under drop_params.
* test(bedrock): annotate the strict-tools retry test helper's values as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add aiand provider
* feat: expose aiand provider in admin UI
* feat(aiand): add ai& model pricing to the cost map
* feat(aiand): list ai& models for aiand/* wildcards and mark /v1/messages supported
* feat(aiand): send Anthropic /v1/messages to ai& natively
* fix(aiand): drop the retired max_tokens field and regenerate endpoint support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
