* fix(proxy): apply published model rows on every worker before the debounced resync
Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path
* fix(proxy): load published model rows from the primary under the reconcile lock
Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s.
* test(proxy): cover a published model row that names no credential
* fix(proxy): keep the unknown model name read-through on the replica and outside the lock
Only the by-id load that follows a published write reads the primary inside
the reconcile lock. A request for a model name a worker does not know reads
through the replica routing before taking the lock again, as on main, so
unknown names put no load on the primary and do not wait behind a reload.
* fix(proxy): keep access group edits from dropping staying deployments and batch published config ids
* fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica
* fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing
A batch of published model ids that held one auto router skipped every row in it, so ordinary
rows written at the same moment waited for the full reload. The by-id load now leaves only the
auto router rows to that reload.
An access group update now reads every deployment from the primary, answers 404 for an unknown
group and 400 for an unknown model id or name before it writes anything, then removes the group
from the deployments that leave before it adds it to the ones that join, so a failed write leaves
the group narrower than asked, never wider.
* fix(proxy): write an access group update in one transaction after a fresh read
The update now checks model names first, reads every deployment from the
primary, and commits every join and leave in one batch right after that
read, so a failed write changes nothing and a field another admin set
during validation is kept. Other workers hear about the rows only after
the commit.
* fix(proxy): lock the model rows while an access group update reads and rewrites them
* test(proxy): run the model resync tests on the real installer and a real router
The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen.
* fix(proxy): lock only the rows an access group update touches and answer 404 before 400
An access group update locked every model row, so a lock on any unrelated
row stalled it for 30 s and then failed it with a 500, and unrelated model
edits on other proxies waited behind it. It now takes a per-group advisory
lock, locks only the rows that carry the group or that it selects, and
reads them again once locked. The group existence check runs before the
model name check again, so a missing group answers 404 as on main.
* fix(proxy): never wait on a locked row in an access group update
The update now takes its row locks with NOWAIT. When another edit holds one
of the group's rows, it rolls back, holding nothing, and tries again for about
10 s before answering 409. Holding rows while waiting deadlocked against team
deletes and let a concurrent model edit undo the update's change. The
per-group lock also moves to the two-key advisory space, apart from team ids.
* fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select
* fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep a deployment's own model_id over the request body's
A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one.
* fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat
Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win
over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists
model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a
client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component
* fix(router): keep a SageMaker deployment's component when it pins it through extra_headers
* fix(router): read the SageMaker component header as a pin only on SageMaker deployments
* fix(sagemaker): send the inference component only as a header, never as a body model_id
SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat
container body and legacy parameters no longer carry model_id. Nova endpoints accept a
fixed body field list, so a pinned Nova deployment could reject the extra field.
* fix(health): probe header-pinned SageMaker deployments at their component
Health checks bypass the router, so a legacy sagemaker/ deployment pinned only
through its inference component header sent no component on /health while routed
requests now send it. Also update the two SageMaker integration tests that still
asserted the old precedence and the dropped body field.
* fix(health): apply the router's model_id pin rule to health probes
Health probes now share the router's pin function, so a deployment that lets
clients set model_id gets no header pin and its own model_id wins without
reading the header. Header keys that are not strings are skipped when
reading the inference component header instead of failing the call.
* fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks
The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): convert PDF image_url content to documents
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt_templates): strip content-type parameters from fetched media
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(prompt_templates): format MIME normalization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
* fix(mistral): forward prompt_cache_key to the Mistral API
* fix(mistral): send prompt_cache_key only to Mistral's API and accept reasoning_effort on devstral and code-agent models
* fix(mistral): keep the upstream api_base out of the prompt_cache_key rejection message
* fix(mistral): honor allowed_openai_params for prompt_cache_key and gate codestral async and env hosts
* test(mistral): annotate the new adapter prompt_cache_key tests
* fix(mistral): drop or reject a prompt_cache_key the Mistral API cannot take
A non-ASCII prompt_cache_key gets a 500 from api.mistral.ai, and a non-string
one a 422, so both now drop under drop_params and answer 400 without it, like
any value the provider cannot take. The /v1/messages adapter derives no key
from a non-ASCII Claude Code session for Mistral, and the 400 names the
caller's provider, so a codestral deployment's error says codestral.
* fix(mistral): send a reasoning_effort object as its level and keep UnsupportedParamsError on sync calls
* fix(mistral): keep async codestral file uploads working by not rewriting the caller's file blocks
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126)
The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests.
* test(proxy): pin provider metadata and key billing on /cursor/chat/completions
A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land
* fix(proxy): pick the Cursor route's metadata field from the body shape
A Responses-style body on /cursor/chat/completions keeps proxy metadata in
litellm_metadata, the same as /v1/responses, while a Chat Completions body
keeps metadata, the same as /v1/chat/completions. Adding the whole route to
the internal-metadata list let a client's chat-body tags replace the key's
tags in spend logs and skip tag budgets.
* fix(proxy): keep get_metadata_variable_name callable with only the request
* fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies
A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field
* fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies
* fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string
* fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list only routable models for a wildcard alias over a provider org
A wildcard alias over an org-scoped deployment (together_ai/* over
together_ai/nosuchorg/*) listed the provider's whole catalog on /v1/models
and /model/info, and every listed id routed to a model the provider 404s.
A skipped deployment path segment now counts as a routing prefix only when
the routed model resolves to a catalog id through the get_model_info
lookup order, so responses/, Bedrock converse/, and region paths list as
before.
Together's model list now also carries the cost map's together_ai rows,
skipping the together-ai-* pricing tiers, so an alias over a real org lists
the models Together serves today.
* fix(proxy): keep flat catalog ids under routes the cost map can't parse, drop retired Together rows
* fix(proxy): check skipped path segments only on Together so Hugging Face provider routes keep listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): count a compaction block instead of answering 500
A message history that replays Anthropic's signed compaction block made
/v1/messages/count_tokens answer HTTP 500: neither the Anthropic nor the
Vertex AI partner count-tokens request carried the compact-2026-09-04 beta,
the provider rejected the block with a 400, the proxy fell back to the local
tokenizer, and that raised on the unknown block type.
Both provider count paths now add the compaction beta whenever the history
carries a compaction block, and the local tokenizer counts the block's
summary text like a text block, skipping the signature.
* fix(token_counter): skip the tokenizer for empty thinking and compaction text
* fix(count_tokens): forward context_management on Anthropic counts and count chat compaction blocks locally
Vertex AI compaction support for inference and counting moves to its own PR, so the
Vertex count handler is back to main here
* fix(anthropic): map list-form context_management to edits for count_tokens and inference betas
Count requests forwarded the OpenAI list form of context_management unchanged, while inference
maps it to Anthropic edits, and the inference beta helper read the list as no edits. Both now
map it once. A replayed compaction block with an empty summary no longer reaches the tokenizer.
* fix(anthropic): forward count context_management only with edits, map Vertex list-form betas, and return provider context_management
* fix(anthropic): drop an unmappable context_management list under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch-interception): stream native server-tool blocks to clients
Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code)
never received the server_tool_use and web_search_tool_result blocks the
interception layer builds for them, so citation panels stayed empty and
result counts read zero. Two gaps in the same flow:
FakeAnthropicMessagesStreamIterator._create_content_block_chunks only
knew text, thinking, redacted_thinking and tool_use; any other block
type emitted a bare content_block_stop with no start and no payload,
silently dropping the block from the fake stream. server_tool_use now
streams as content_block_start plus input_json_delta and any other
server-generated block rides whole in content_block_start, matching
Anthropic's native streaming of these types.
try_short_circuit_search only recognized a native client by finding a
web_search_* tool in the tools list, but the pre-request and deployment
hooks convert that tool to the LiteLLM standard shape before the
short-circuit runs, recording the conversion in
WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so
the synthetic response carries the native blocks on the converted path
instead of only when hook ordering happens to leave the tool untouched.
Fixes#35333
* fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response
The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500
* fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply
* fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply
* test(websearch-interception): build the cost test's logging object with a datetime start time
* fix(websearch-interception): send search snippets once in the direct search answer
* fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks
Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta.
* fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite
A legacy hook that masked a tool call's arguments on a buffered stream made the whole
rewrite undeliverable, text included, so the original stream went out. With web search
blocks in the stream that now hit any hook masking the search query. The adapter hands
same-count tool-call rewrites to the translation's write-back, as a unified guardrail's
already are, and still discards a rewrite that adds or drops a tool call.
* fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream
The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled
ModelResponse, which keeps web_search_tool_result blocks only in provider-specific
fields, so a hook masking result titles on a stream never saw them. The hook's view now
carries those blocks, its rewrite of them goes back through the ended-stream write-back,
and the adapter no longer needs to pass texts outside the hook's view through untouched.
* test(guardrails): type the legacy hook stream test helpers
* fix(guardrails): scan a streamed web search query as text like the non-streaming reply
* test(guardrails): type the masking guardrail test helpers
* fix(guardrails): keep the block id when scanning a streamed web search query
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path
* fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints
* fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts
A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for.
* refactor(azure_ai): read text parts through a walrus and mark new test values Final
* fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts
* fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model
A router deployment without its own supports_prompt_cache_breakpoint flag now
reads the cost map as loaded, so a sibling's opt-in on the same model string no
longer turns its requests into 400s. The shared hook no longer reads router
registered keys for azure_ai.
* fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name
* test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): probe only models a partial wildcard serves
A partial wildcard health check now probes the cheapest cost-map chat models the wildcard matches, keeps provider route prefixes such as Bedrock converse/ on the probe, and falls back only within the wildcard. A wildcard with no matching cost-map chat model reports an error naming model_info.health_check_model
* fix(health): keep the provider-wide probe for an unmatched wildcard, skip retired models, and ignore Bedrock ARNs
A partial wildcard with no served cost-map chat model falls back to the provider's cheapest chat models
instead of failing, and fails with a message naming model_info.health_check_model only when the provider
has none. The cheapest-model picker skips models past their cost-map deprecation_date, and a Bedrock ARN
segment is never read as a route prefix.
* fix(health): fall back to the provider's cheapest probe when a wildcard's own models fail
A partial wildcard now probes up to three of its own cheapest chat models
first and then the provider-wide cheapest ones, all from one cost-map scan,
so stale or unpriced rows the wildcard matches cannot turn a healthy
deployment red. Retired models are read by their resolved cost-map key,
both deprecation checks use the UTC date, Bedrock only treats converse,
invoke, and region names as route segments, and equal-price ties break by
model id
* fix(health): cap wildcard health probes at three and rank retired cost-map rows last
A partial wildcard now probes two of its own cheapest cost-map chat models and then one
provider-wide model, so a failing deployment makes no more calls than before. Rows past
their cost-map deprecation date go last instead of being dropped, since some still serve,
and the shared cheapest-model picker returns what it returned before.
* fix(health): probe a wildcard's own models first, then the same three models main probes
A partial wildcard's health check tries its two cheapest own cost-map models, then the provider's three cheapest models main already probes, so a row main shows healthy stays healthy. The deprecation check uses the server's local date, like the Baseten listing
* fix(health): keep main's provider-wide probe order so equal-price ties match main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): keep the upstream error response on mapped exceptions so e.response.text has the wire body
* fix: keep the upstream error body on 429 and 5xx errors
RateLimitError, InternalServerError, BadGatewayError, and ServiceUnavailableError
rebuilt their response and dropped the upstream body. They now keep the body and
exactly the upstream headers, and an unread streaming response no longer raises.
* fix: keep a gzip-encoded 429 or 5xx error readable
The kept body is already decoded, so the rebuilt response drops content-encoding
instead of decoding the body a second time and raising DecodingError.
* fix: keep the upstream headers and drop credentials on every mapped provider error
The copied error response now keeps every upstream header and carries a
request with only the method and the URL, so no provider key, query
secret, or prompt reaches e.request. Timeout and APIError get the
response too, and azure/ chat, streaming, embedding, and image errors
attach it
* fix: keep the upstream error body on gateway timeouts and re-raised gateway errors
* test(azure): cover the 404, 500, 502, and 504 error bodies and the missing-key placeholder
* fix: keep no e.response on timeouts and errors litellm raises itself
* fix: keep the upstream elapsed time on error responses and the upstream body on azure_text errors
* fix: keep the upstream body without credentials on Azure content filter errors and message-matched OpenAI errors
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): cap the default OTPM reservation for requests without max_tokens
* fix(router): reconcile ITPM and OTPM reservations after every successful call
* fix(router): read the default OTPM reservation from the served model and make its fraction configurable
The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check.
* fix(router): refund a failed attempt's io token reservation in make_call
The proxy passes one logging object to every retry, and it runs failure
callbacks only for the first attempt, so the last attempt's ITPM and
OTPM reservation stayed held until the minute ended. Each attempt that
raises now gives its reservation back before the retry or fallback
* fix(router): count Anthropic cache writes once in ITPM reconcile
Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens.
* fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction
* fix(router): reserve with the deployment cap only through the param the request's API sends
* fix(router): never let a deployment's output cap raise the reservation above the default
* test(router): move the reservation fraction env var test to the integration suite
The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation.
* test(router): retry the reservation fraction scenario when the minute rolls over
The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute.
* test(router): give the rollover retry its own deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(zai): fall back to BIGMODEL/ZHIPUAI api keys and add GLM-5.3 aliases and glm-5v-turbo
Resolve the zai api key from ZAI_API_KEY, BIGMODEL_API_KEY or ZHIPUAI_API_KEY
so deployments predating the zai rename keep working. Accept response_format
and max_completion_tokens as supported openai params. Register zai/GLM-5.3
and zai/GLM-5.3-Flash aliases plus the glm-5v-turbo vision model in both the
main and backup cost maps
* fix(zai): read only ZAI_API_KEY, dropping the BigModel key fallbacks
* fix(zai): map max_completion_tokens to max_tokens, correct glm-5v-turbo pricing, drop redundant aliases
* fix(zai): reject json_schema response_format, add glm-5v-turbo cache read price
* fix(zai): forward every response_format type so allowed_openai_params keeps working
* fix(zai): follow json_schema natively on glm-5.3, prompt other models, drop over-limit max_tokens under drop_params
* fix(zai): keep BigModel's json_schema as sent and never drop the caller's max_tokens under drop_params
* fix(zai): write the prompted JSON schema as JSON so models answer with valid JSON
* test(zai): type the zai provider tests and mark their locals Final
---------
Co-authored-by: DewellNaa <210961864+DewellNaa@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced
* test(integration): a reported cost is attributed to output in the gateway parity cases
* fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects
* fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge
* fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost
* fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill
* fix(cost): read the stored breakdown total through the logging object's typed field
* refactor(cost): name the providers billed from their own usage fields for what they are
* fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Neither chart bundles Lens anymore. The Lens subchart, its lock entry,
the vendored chart archive, the templates/lens resources and the
bundled-Lens install smoke are gone. A connection-only lens block wires
the gateway and backend to a Lens deployed from its own chart, and an
enabled lensWorker value fails the render instead of being ignored.
Supersedes #47869, which made the same change for helm/litellm only.
* fix(cost): bill the requested model's rates when the response names an unpriced model
Fireworks answers chat and /v1/messages calls with a display name such as
"Qwen 3.8 Max" in the response's model field. The cost path priced that
name, the Fireworks calculator fell back to its free default bucket without
raising, and the call billed $0 even though the requested model has a row.
When the response model resolves to no cost-map row or rule that declares
rates and the requested model does, price from the requested model. A priced
response model still wins, so dated snapshots keep their own rates, and an
unpriced requested model such as a router alias leaves the result unchanged.
* perf(cost): check the exact cost-map row before generalization rules when picking the priced model
* fix(cost): apply the priced fallback to the provider response model streaming records
* perf(cost): skip the served model's rule lookup when the requested model has no rates
* fix(cost): price the requested model without a model arg and keep Fireworks size buckets behind the proxy
When completion_cost runs with no model, the requested model now comes from the
call's hidden litellm_model_name, so a display-name response still bills at the
requested model's rates. A Fireworks model whose name carries a parameter size
counts as priced through its size bucket.
The Fireworks per-token lookup now treats a cost-map row that prices nothing, like
the empty row the Router registers for an unlisted deployment, as missing, so an
unlisted Fireworks model bills at its size bucket instead of $0 even when Fireworks
echoes its slug. The "row prices anything" helpers move to llm_cost_calc/utils so
the Fireworks lookup can reuse them without a circular import.
* fix(fireworks): keep prices from a matching fallback rule for an unlisted model
An unlisted Fireworks model that only a fallback generalization rule prices
has no raw cost map row, so the empty-row check sent it to the size bucket
and dropped the rule's rates. The Fireworks lookup now keeps the model info
when the row or a matching rule declares a rate.
* fix(cost): price a bare response model under the requested provider and drop the unlisted Fireworks bucket change
The logging payload picks the pricing model without a provider, so a bare echo
like mistral-large-latest failed its price check and the requested alias took
over model_map_key. The served name is now priced under the requested model's
provider, and an echo that is the requested name minus its provider prefix
skips the checks.
Treating the Router's empty price row as missing let unlisted Fireworks
deployments bill from a name guess, which misprices dedicated deployment ids
and routes some short names to another provider's rates. That change and its
tests go back to the merge base, so unlisted Fireworks models keep billing $0
behind the proxy as on main.
* perf(cost): read the response model's price row without building full model info
An unpriced display name like Qwen 3.8 Max made the price check build full
model info, and Fireworks' supported params lookup scans the whole cost map
for an unknown name. The row check now resolves the cost map key through the
cached model info helper, so that pick takes about 0.45 ms instead of several.
A regional test covers a Bedrock call whose response names an unpriced display
name with no model passed, which bills the requested model's regional row.
The duplicate get_model_info wrapper goes, since its twin already does the
same thing.
* fix(cost): keep the served model's filled-in and size-bucket rates, and cache misses
A served row priced only by a fill_missing_for_providers rule bills those
rates on main, but the price check read only the raw row and billed the
requested alias instead. It now merges the rule's fill-in keys through the
helper model info lookup uses.
A served Fireworks or Together AI id that names its parameter count keeps its
size-bucket price, as on main, instead of switching to the requested row.
The served model is checked first, cost map misses are cached and cleared with
the model info cache, and the rules are read only for a name that resolved
through one. A priced snapshot behind an Azure deployment name takes about
19 us of CPU time against 7 us on the merge base, down from about 300 us.
* fix(cost): bill the requested model's listed rates over a served size-bucket guess
A served Fireworks or Together AI name that is not in the cost map but names
its parameter count, like a display name with a size in it, now bills at the
requested model's listed rates instead of the provider's size-bucket price.
The bucket price still applies when the requested model has no rates, as on
main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): batch Claude models in Anthropic's native row shape
Claude deployments on Vertex were filed under publishers/google/models/<claude>
when the batch input was uploaded, so create_batch asked Vertex for a Google
model that doesn't exist and got a bare 404 on batchPredictionJobs. The rows
were also written as Gemini GenerateContentRequests, which Vertex rejects for
Claude even with the right model path
Claude uploads now go under publishers/anthropic/models/<model> and each line
becomes {"custom_id", "request": <Anthropic Messages body with
anthropic_version vertex-2023-10-16>} using the existing Vertex Claude
transform. Output rows in the Anthropic message shape are converted back to
OpenAI batch rows (custom_id from the row, errors as failed rows), native rows
are costed from Anthropic usage, and create_batch fails fast with a 400 when a
Claude job targets the global location, which Vertex doesn't serve for
partner model batches
Fixes#45671
* test(vertex_ai): cover malformed Claude output rows and rows without custom_id
* fix(vertex_ai): build Claude batch rows with chat's param mapping and surface Vertex batch create errors
Claude batch rows now go through get_optional_params like Vertex Claude chat, so unsupported params, top_k, response_format and extra_body follow drop_params. Tool names Claude would rename get a 400 at upload. A refused batch create returns Vertex's own status and message.
* fix(vertex_ai): price Claude batch output rows without a request echo, keep @ version ids, refuse a json_tool_call user tool, keep Vertex error headers
* fix(vertex_ai): keep httpx.HTTPStatusError on Vertex batch create errors and apply deployment chat params to Claude batch rows
* fix(vertex_ai): keep Vertex's status code on batch create errors through the proxy
* fix(vertex_ai): keep Vertex's error body on batch create errors and require custom_id on Claude lines
* fix(vertex_ai): keep the old batch create error text and Google object names
A failed Vertex batch create keeps httpx's error line first and adds Vertex's body after it, and logs the status and body again. Only Claude object names keep the @ version, so Google models are stored under the same names as before
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(providers): add CLF AI Gateway (clf_ai_gateway), OpenAI-compatible, 9 models
OpenAI-compatible gateway at https://api.clfaigateway.dev/v1 serving open-weight models (GLM, Kimi, DeepSeek, Qwen) on Cloudflare Workers AI upstream. Follows the DeepInfra pattern: ClfAiGatewayConfig(OpenAIGPTConfig), CLF_AI_GATEWAY_API_KEY / _API_BASE, provider registered in enum, provider lists, endpoint inference, lazy-import registry, ProviderConfigManager, validate_environment. Supported params deliberately omit functions/function_call/logit_bias (gateway returns 400 for them). 9 cost-map entries (root + backup) with measured context/output limits and currently billed prices; endpoint-support entry; README row. Mocked tests only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clf_ai_gateway): declare each model's real reasoning_effort levels
The per-level supports_*_reasoning_effort flags cannot express the sets
this gateway actually takes, so all nine models advertised a wrong one:
every model offered "minimal" and eight offered "none", neither of which
the gateway accepts, five dropped "xhigh", which it does accept, and
qwen3.8-27b offered "high", which it rejects. That set surfaces through
/model_group/info and the model picker, so a user picking a level off it
gets a 400 from the gateway.
Declare reasoning_effort_levels instead, taken from the gateway's own
GET /v1/public/models, which resolve_supported_reasoning_efforts reads
first and uses whole. Declaring it also stops clf_ai_gateway/deepseek-v4-pro
and clf_ai_gateway/deepseek-v4-flash inheriting effort metadata from the
unprefixed deepseek entries of the same name.
* test(clf_ai_gateway): name the gateway-rejected params instead of commenting them
* fix(clf_ai_gateway): list the provider in the proxy Add Model form
main now requires every LlmProviders value to have an entry in
provider_create_fields.json (or be frozen as unlisted), so the Add Model
dropdown cannot silently miss a backend provider. Add the clf_ai_gateway
entry in the same shape as the SCX.ai registration: required API key,
optional API base with https://api.clfaigateway.dev/v1 as placeholder, and
a cost-map model as the default model placeholder. A provider test pins
the entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(clf_ai_gateway): sync prices and reasoning_effort levels with the live catalog
CLF AI Gateway raised its launch discount from 40% to 60% off list on
2026-09-26. It also re-measured every model's reasoning_effort levels
against the current Workers AI behaviour.
- input/output/cached per-token prices for all 9 models now match
GET https://api.clfaigateway.dev/v1/public/models to the nano; qwen3.8-27b
cached input also follows the upstream cached rate published on 2026-09-25
- reasoning_effort_levels: add "max" to deepseek-v4-flash, deepseek-v4-pro,
glm-5.2 and glm-5.3-flash; add "none" to kimi-k2.6 (it really disables
thinking there); drop "none" from glm-5.3 (upstream runs it as "max")
Context windows, output limits and vision flags are unchanged and were
checked against the same endpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(clf_ai_gateway): drop a rebind-ok that no longer suppresses anything
After rebasing onto main, the type-discipline gate's new LIT013 rule flags
the `# rebind-ok` on the endpoint-inference `dynamic_api_key` assignment.
The rebind rule no longer fires on that line, so the suppression is inert.
The sibling endpoint branches carry no comment there either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): expect the re-measured reasoning_effort levels
Follows the catalog sync: max on deepseek-v4-flash, deepseek-v4-pro,
glm-5.2 and glm-5.3-flash, none on kimi-k2.6, and no none on glm-5.3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): move the provider tests to tests/unit/llms
main moved every per-provider test from tests/test_litellm/llms/<provider>/
to tests/unit/llms/<provider>/ (DeepInfra lives at
tests/unit/llms/deepinfra/). This branch still added the old path, so no CI
shard claimed it. That failed assert-ci-coverage and assert-shard-coverage,
and the file never ran (hence the low codecov/patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): assert routing, billing, and dropped params instead of pinned gateway facts
* fix(clf_ai_gateway): keep the type-discipline gate at its merge-base count
* chore(clf_ai_gateway): regenerate provider endpoint support
* ci(clf_ai_gateway): run the provider tests in the llms unit shard
* fix(clf_ai_gateway): drop the retired max_tokens field from the cost map rows
* fix(clf_ai_gateway): forward params the gateway accepts, drop undeclared reasoning levels, add deepseek-v4.1-flash
* fix(clf_ai_gateway): drop the unreachable non-chat supported-params branch
* fix(clf_ai_gateway): send a dict reasoning_effort as its level
* fix(clf_ai_gateway): refuse a reasoning_effort dict without a level unless drop_params drops it
---------
Co-authored-by: bap1106 <bap1106@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(types): replace Any with proven types in 35 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): keep anthropic chat transformation json path untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): validate Python client user responses and type delete_user as the count it returns
* fix(types): keep loose payloads working under the sweep's validators
An off-schema Anthropic stream event is now skipped instead of ending the
guardrail's text and stop_reason scan, and MAI image usage accepts the loose
numbers it did before. New dict and list adapters hide their input from
validation errors, RerankResponse indexing keeps its field types, and the
delete_user test mocks the int /user/delete returns.
* fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items
* refactor(types): tighten round 2 parse sites and keep payloads out of errors
Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel
* fix(types): keep MAI image calls alive on unreadable usage counts
An unreadable MAI usage count now counts as absent instead of failing a
billed image call, and the provider's usage fields pass through as sent.
The Anthropic stream scan warnings no longer log raw model output, and the
off-schema skip has tests that fail on the old scan.
* fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(databricks): retirement dates for DeepSeek V4 Pro (0813) and Inkling
* chore(databricks): retirement dates for DeepSeek V4 Flash (0731), V4 Pro (0813) and Inkling
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): read OpenAI batch usage into the batch token counts
* fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller
* fix(batches): keep the chat usage keys next to OpenAI's batch usage fields
Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts.
* test(batches): check the batch usage counts in the e2e terminal-state test
* fix(types): apply nested include and exclude to both batch usage shapes
* fix(batches): log no cost for a batch retrieve that is not final yet
A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve
that saw the batch completed before its output file arrived priced those counts
for the logging callbacks, and the final retrieve priced the whole batch again.
A batch that is not final now logs no cost, as it did before.
Batch usage dumps also pass exclude_unset and exclude_defaults through to the
usage keys.
* fix(batches): pass through every usage detail field the provider sends
Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys
* fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details
* fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape
* fix(batches): store only chat usage extras on batch spend rows
Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens,
output_tokens and their details) into metadata.additional_usage_values next
to the chat counts they duplicate. Build the row from the batch's chat usage,
shared with the logging payload through batch_chat_usage.
* fix(streaming): rebuild each choice of an n>1 chat stream on its own
stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged.
* fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason
* fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices
MCP auto-execute now answers only the first choice's tool calls, since its
follow-up turn continues that choice alone. The Responses bridge streams a
space before each reasoning summary part after the first, matching the
non-streamed reply. build_base_response now takes the per-choice streams
instead of having its choices replaced afterwards
* fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass
* fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages
* fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools
* fix(noma): scan each choice of a multi-choice reply on its own
* fix(noma): keep masked choices when a later choice's scan fails open on a stream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): measure PR gates from the target branch's current tip
The lint gate base, the ui lint and ui unit test scopes, and the spend logs
guard compared the event's pull_request.base.sha with the head to find the
merge base. GitHub can leave base.sha weeks behind the base branch, and when
that old commit is an ancestor of a head that merged newer main, compare
returns the old commit itself, so the strict gates blamed main's own code on
the PR.
Compare base.ref with the head instead, passed through env, so GitHub
resolves the branch tip when the job runs. This is the same base make check
uses locally. The ui unit test push guard now keys on BASE_REF, which is
empty on push and workflow_dispatch.
The regression test runs every workflow step that reads merge_base_commit
against a fake gh backed by a temp repo with a stale base.sha, for main and a
litellm_/ base with a slash.
* fix(ci): check out the base branch's current guard code in the PR guards
* fix(ci): read the PR's current base branch so re-runs survive a deleted base
Every merge-base step now asks the pulls API for the PR's base branch instead of
reading the event's base.ref, which a re-run keeps after a stacked base is merged
and deleted. The guard checkouts go back to base.sha, which stays reachable.
* test(ci): mark the step script path Final
* fix(ci): keep the spend logs guard on main and run both guard scripts from the base branch tip
* fix(ci): measure from the event's base branch unless it was deleted, and test what each step hands on
A re-run after a retarget off main kept following the PR's current base, so lint, ui-unit and the cost map guard could pass without measuring against main. They now read the current base only when the event's base branch no longer exists. The tests run each merge base step whole and check the value its consumer gets
* fix(ci): follow a renamed base branch to its new name before the compare
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding
* fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing
Detect raster images from the page content stream, including inline
images on pages with empty resources, and walk each form once. Strip
only openai/, azure/ and Azure data-zone prefixes, bound the model
family scan, read Azure-only chat models from the cost map, and keep
cl100k_base for listed non-chat OpenAI models.
* fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources
Prices a PDF by the model name's last path segment, the same name the proxy counts with,
keeps listed non-chat models on Claude pricing, walks each form once per inherited resources
dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing
only when a message carries a PDF
* fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only
A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near
it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many
resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes
count like their base model
* fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models
* fix(token_counter): count reservation and project ITPM with the serving deployment's model
Budget reservation and project ITPM counted with the requested group name, so a
GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count
with each deployment's model minus its provider prefix, the name count_tokens uses,
and take the largest. A custom OpenAI tokenizer with no model counts with cl100k
again instead of raising, and listed Azure gpt- chat entries price as OpenAI.
* perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator
pypdf parsed every operator of a page's content stream to find the images it draws, so a
compressed page of a million operators took about 30 seconds to price for an OpenAI model.
A linear scan of the decoded stream for name Do and inline images finds the same draws.
* fix(token-counter): count each team deployment by its PDF pricing and honor base_model
Budget reservation and the TPM limiter now look up the deployments that serve the
caller's team, by base_model when set. A deployment that reads PDF pages and prices
them unlike the group name counts with its own model, any other with the group name.
A comment between an XObject name and Do now reads as whitespace, and the timing
tests moved to CodSpeed benchmarks.
* fix(benchmarks): install pypdf in the benchmark environment
The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path.
* fix(proxy): count PDF pages for deployments served under multi-segment model names
* fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge
* fix(proxy): count routing-group and alias members with the caller's team, as routing serves them
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* build: allow filelock 4.x
filelock 3.x is the only line the current cap permits, and three CVEs
(CVE-2026-112940, CVE-2026-112913, CVE-2026-112907) are fixed only in
4.2.1+. The cap means every downstream that depends on litellm is held
on a vulnerable filelock with no way out, since pip has no override.
litellm uses FileLock(path, timeout=..., mode=...) and Timeout, in
litellm/proxy/client/cli/commands/configure_profiles.py and
claude_settings.py. Both keyword arguments still exist in 4.2.4 and
Timeout is still exported, so the two call sites are unaffected. The
CVEs are in the SQLite-backed ReadWriteLock / AsyncReadWriteLock, which
litellm does not use.
<5 rather than a floor bump because litellm supports Python 3.10 and
filelock 4.x requires >=3.11: widening the cap lets 3.11+ resolve 4.x
while 3.10 keeps a 3.x filelock automatically, through filelock's own
requires_python.
* build: sync core filelock bound and dependency lock
---------
Co-authored-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
* feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex
lite pi, lite opencode and lite codex now read supported_reasoning_efforts from /model_group/info and write them into each client's model config. pi gets reasoning and a thinkingLevelMap, OpenCode gets reasoning_options plus one variant per effort, and Codex fallback entries get supported_reasoning_levels with a default level. The proxy-served Codex catalog on /v1/models?client_version= offers the same levels
Codex before 0.138.0 fails the whole catalog on an effort it cannot name, so those releases, and an unreadable version, get only the six efforts every release parses. A group with unknown efforts keeps today's output, and a group with none is written as non-reasoning
The strength order and nearest_declared_reasoning_effort move to an SDK-free module so the lite CLI, which loads without initializing the SDK, can import the Codex catalog
* fix(lite): describe team models by their public name and keep Codex's none default
/model_group/info now looks up a team's own deployments when the caller's
team serves the requested name, so a team model reports its reasoning levels.
Codex fallback entries default to none when the group lists it, and the proxy
builds the Codex catalog in a worker thread
* fix(lite): round Codex's default level down and describe a team's shared-name model from its own deployments
Move the reasoning strength helper to litellm_core_utils so lite and the Codex catalog import it without the SDK or the router, and let make lint-format-check-changed diff the working tree so a staged rename passes.
* fix(lite): keep pi's system role on reasoning model groups
pi switches its system prompt to the developer role for any model marked reasoning, so an operator's
cache_control_injection_points entry for role system stopped matching on Claude groups. lite pi now sets
compat.supportsDeveloperRole false on those entries, so pi sends system as it did before.
* fix(codex): treat a client_version too long to parse as an old Codex and keep undescribed stock levels
A client_version past Python's integer digit limit made the Codex catalog answer 500. It now gets the
levels every Codex parses. A stock Codex level with no description validates again, as on base, and
lite codex treats undecodable codex --version output as an unknown version
* fix(lite): keep REASONING_EFFORT_STRENGTH_ORDER importable from router_utils and skip the Codex sync on undecodable debug models output
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(caching)!: remove the disk cache backend
Drop the disk cache type and its diskcache dependency, along with the
caching install extra that existed only to pull it in. diskcache has not
had a release since August 2023
Cache now raises a ValueError listing the supported types when it gets a
type it does not know. Before, an unknown type built a Cache with no
backend, which only failed on first use
BREAKING CHANGE: cache_params type "disk" and the litellm[caching] extra
are gone. A proxy configured with type disk now refuses to start. Use
local or redis instead
* test(proxy): expect the unsupported cache type error from _init_cache
The test pinned the old fallthrough, where an unknown type left the cache
without a backend and the proxy hit an AttributeError reading it
* chore(deps): drop the diskcache scan ignore main extended
diskcache is no longer in the lock, so the ignore matches nothing
* fix(caching): ignore a leftover disk_cache_dir instead of crashing redis and s3
Dropping the disk_cache_dir parameter sent the key into **kwargs, which
the redis and s3 branches hand to their clients, so a config that moved
from the disk cache to redis or s3 and kept the key failed with a
TypeError. Accept the key again, ignore it, and log a warning
The unsupported type test now checks the whole supported list, and two
proxy docstrings no longer name disk as a cache backend
* fix(caching): warn about a leftover disk_cache_dir only once the cache is built
The warning ran before the type check, so a config with type disk logged
that the key was ignored and then failed on the type anyway. It now logs
only after a supported backend is built. The test also checks that a
cache without the key logs nothing
* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(dashscope): add image edits and text-to-speech
* fix(dashscope): bill reported speech characters and input images, follow the chat region for edits, keep the call timeout on the audio download
* fix(dashscope): bill deployment image prices as main does and route edits to the host speech uses
A deployment that sets its own input_cost_per_image, output_cost_per_image or input_cost_per_pixel bills through the default image calculator again, so an override is never read as a per-input-image price and pixel pricing on qwen-image-3.0-pro no longer raises.
Edits now resolve the caller's api_base or DASHSCOPE_API_BASE through the same API-root helper speech uses, on any host, with DASHSCOPE_API_BASE_IMAGE still winning. An image response whose usage is null or not an object keeps its images instead of raising.
* fix(images): bill a deployment's own prices on image edits through the proxy
The proxy builds the logging object before the router picks a deployment, so image_edit never handed the deployment's price fields to its cost calculation and billed the cost-map price instead. Pass the custom pricing params with the pre-call update, as the Responses API does.
* fix(images): bill per-pixel image prices at a DashScope W*H size
DashScope's mapped size reads 1328*1328, which the cost size reader rejected, so a deployment's input_cost_per_pixel billed every DashScope generation or edit at the 1024x1024 default.
* fix(dashscope): bill image prices by their meaning, report the rendered size, prefer the deployment edit base, fetch audio over https
* fix(dashscope): read the rendered image size from output_width and output_height
* test(images): bill a fal edit at its deployment's own image price
* fix(dashscope): keep the qwen-image-3.0 input-image fee out of the shared cost map row
Released proxies read the cost map from main and bill input_cost_per_image as the
flat price of each DashScope image, so the row's $0.003 input fee billed every
qwen-image-3.0 image there at a tenth of its $0.03 price. A deployment can still
set input_cost_per_image to bill DashScope's input images
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(context_management): run clear_tool_uses in the gateway on the Responses API path
/v1/messages to openai/ models goes through the Responses API, which translated
only compact_20260112 (to its native compaction) and dropped every other edit
without a warning, so clear_tool_uses_20250919 never cleared anything. Edits
without a Responses API form now run in the same polyfill as the chat completions
path, before the call, and their applied_edits come back on the response and the
final message_delta. compact stays native.
* fix(context_management): check the sync polyfill result instead of casting it
The strict ruff gate (TID251) bans typing.cast.
* test(context_management): cover the Responses path's polyfill error handling
A polyfill crash is a 500 before the call, a rejected edit under drop_params is
skipped, and the sync stream reports its edits. normalize_spec never raises for
a dict spec, so its guard is gone.
* fix(context_management): count unreadable blocks and check tool_result text before clearing
clear_tool_uses now maps blocks the token counter cannot read (server tool, compaction) to JSON text, so an input_tokens trigger no longer 500s on that history. The Responses API path checks malformed tool_result text on the original messages before clearing, as the chat completions path does.
Also removes comments that repeated the code and makes the test fixtures immutable.
* fix(context_management): count only blocks the model receives and keep list specs unsent
clear_tool_uses now leaves blocks token_counter cannot read, like server tool
results and compaction, out of its count instead of counting them as JSON text.
No adapter sends them to the model, so an input_tokens trigger no longer fires
on tokens the model never sees.
On the Responses API path, an OpenAI-style list context_management stays unsent,
as it was before, instead of reaching OpenAI's native compaction, where a
threshold under 1000 is a 400.
On the chat completions path, a clear edit listed before compact_20260112 now
runs on the turns after the latest compaction block, as Anthropic does, instead
of counting and clearing turns the compaction block drops.
The crash test that replaced apply_context_management with a mock is removed.
* fix(context_management): apply clear edits in OpenAI count_tokens and skip nested off-spec blocks when counting
* fix(context_management): run edits on the history drop_params sends, so count_tokens and /v1/messages clear alike
* fix(context_management): count clear_at_least freed tokens over sent blocks only
* fix(context_management): count tools with JSON Schema shorthands and reject a non-object trigger or keep
A tool whose schema has an array without items, a boolean subschema, or a search_result block without a title made the token counter raise, so a clear edit answered 500. A string trigger or keep also answered 500, even under drop_params. Both now count or get the API's 400, and drop_params drops the malformed field.
* fix(token_counter): count a malformed tool schema keyword as absent and move shorthand framed-estimate cases
* fix(count_tokens): pass the system prompt into clearing so count_tokens matches inference
* fix(context_management): keep a malformed system prompt from failing clear_tool_uses on the Responses API
A number, an object, or a list holding non-object items as system made the
polyfill answer 500. The polyfill now keeps a list's object items and treats
any other shape as no system, on both /v1/messages and count_tokens, so the
two paths still count the same text
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): keep the proxy's error body on litellm.responses errors from litellm_proxy models
litellm.responses() with a litellm_proxy/ model raised BadRequestError with
body None when the proxy answered a 4xx, while litellm.completion() on the
same proxy carries the decoded error object. The litellm_proxy Responses
config now decodes the error JSON into the exception body the way the
OpenAI SDK does, so body, code, type and the message match chat completions
on sync, async and streaming calls.
* fix(proxy): relay a downstream proxy's provider_specific_fields in error responses
A LiteLLM proxy routing to another LiteLLM proxy now copies the downstream's
provider_specific_fields (which guardrail fired) from the exception body into
its own error answer, on chat and Responses alike. Adds tests for the two
_openai_sdk_error_body branches the existing tests left open.
* fix(proxy): redact internal details from relayed provider_specific_fields
* fix(proxy): redact relayed provider_specific_fields per key so credential-named values stay hidden
* fix(proxy): redact numeric credentials in relayed provider_specific_fields
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(ui): pin the Workflows page copy with characterization tests
Locks the header, table, filters drawer, detail drawer, relative time and duration text on the Workflows page so the upcoming move onto i18next can be checked against it without editing any test
* feat(ui): add i18next plumbing and move the Workflows page copy onto it
Adds an English-only i18next instance with typed keys from src/i18n/locales/en.json, wraps the root layout in I18nProvider, and loads the instance in the test setup. Every string the Workflows page authors now renders through t(), while backend values such as run status, roles, event types and metadata keys stay raw. The rendered English is unchanged
* fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params
The native /v1/messages path sent tools with `strict` straight through, so
when Claude's strict-tool limits rejected them with a 400 the caller got that
400 even with drop_params set. Under drop_params, a 400 that matches Claude's
strict-limit phrases on a request with a strict tool is now sent once more
with `strict` removed from every tool. Without drop_params nothing changes.
The Claude strict-limit matcher moves from bedrock/common_utils.py to
anthropic/common_utils.py so Bedrock converse and the base messages config
share it, and the Rust bridge routes strict tools under drop_params to Python.
* fix(anthropic): retry without strict only when a tool sends strict true
A tool with only strict false no longer triggers a resend of an identical body, the warning no longer
blames tool schemas when the rejected schema may be the output format, and requests with only
additional_drop_params stay on the Rust messages route since the Python retry ignores them
* fix(anthropic): keep sync messages calls with strict tools on Rust
Python has no sync /v1/messages handler, so routing a sync call with a
strict tool to Python raised ValueError where Rust served it before.
The strict-tools detector now applies to async calls only. The retry
warning also names the strict tool count without a plural.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openapi): build native extension for base counts
* test(openapi): exercise native import in base worktree
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GenericGuardrailAPI.get_supported_event_hooks only listed pre_call,
post_call and during_call, so event hook validation rejected the MCP
modes and the guardrail was skipped at proxy startup instead of being
attached to MCP tool calls. Add pre_mcp_call, during_mcp_call and
post_mcp_call. during_mcp_call runs alongside the tool call, so it can
fail the call but cannot rewrite its arguments.
#48718 added TestLangfuseDestinationFields while #48723 made the unconfigured
stdout placeholder opt-in (console_default=True, the otel callback's setting).
Both merged within a minute, so main's integrations job fails six tests that
read the placeholder's exporter. Build those providers with console_default=True
so the operator-exporter checks keep their target.
