Routes with their own Responses config keep only chat completion request
fields on the way into the bridge. Forwarding every injection point there
let a tool_config point reach the provider body through the generic
unknown-param copy, where no transformation reads it. Forward only the
location: message points the bridge's second pass places.
An end-relative point carried to the bridge's pass gave up its breakpoint slot on the
Responses layer, so a non-negative point listed after it could take the last of the
four slots. It now holds its slot in config order. Routes with their own Responses
config (openrouter, azure and others with use_chat_completions_api) kept only chat
completion request fields on the way into the bridge and dropped
cache_control_injection_points, so every carried point marked nothing there. The
bridge now forwards LITELLM_ONLY_PARAMS.
A /v1/responses request bridged to chat completions ran the cache-control hook on a
provisional message list that holds only the input items with a role, so an index -1
point marked the user message and every function_call and function_call_output after
it went upstream unmarked. Negative ordinals now go to the bridge's pass with the
role-targeted points, where the final list ends in the caller's tool calls and results.
* fix(anthropic): preserve streamed tool argument fragments verbatim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): send the schema-less stream test the null properties form providers accept
* fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece
* fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams
* fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON
A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches.
* fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook
* fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON
* fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers
Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks.
* fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers
* refactor(anthropic): encode JSON mode answers without a recursive helper
* fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation
* fix(anthropic): type the decoded JSON mode answer as object and test non-object answers
json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores
RedisCache and the native Redis client each kept their own breaker, so once
the cache learned the coordination Redis was wedged, a spend-counter write
through the native store still paid a full socket timeout. Every Redis target
now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected
into both the RedisCache guard and the native client. The Python FSM is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(rust): cargo fmt the shared breaker
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(redis): close the breaker when the recovery probe gets a Redis reply error
A command error such as WRONGTYPE means Redis answered, but the probe was left
unsettled and its drop reopened the breaker for another recovery timeout. Both
the RedisCache guard and the native client now settle such a probe as a success.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test
The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw.
* fix(redis): key the shared breaker by server and close the three audit gaps
The breaker registry now keys on the Redis server (host and port, unix socket,
cluster node set or sentinel service) instead of the full connection target, so
caches that differ only in db, credentials or url spelling share one breaker.
A READONLY reply from a replica mid-failover no longer counts as Redis answering,
so it can't reset a run of failures. A new RedisCache refreshes the breaker state
gauge, which the deleted Python breaker did at construction. A shared native client
is only reused by a store that passes the same breaker.
* refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations
lib.rs now only declares modules and re-exports, with resolution, target, tls,
auth and endpoint each in their own file. Derived FromPyObject already reads a
struct field as the attribute of the same name, so #[pyo3(attribute)] on the
breaker and store settings did nothing.
* refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot
Derived FromPyObject already reads each field as the attribute of the same name.
The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in
the same order.
* refactor(redis): return a HashMap from the three memory-tier snapshots
The snapshots read cache-memory's HashMap, so there is no order to keep.
PyO3 builds the same dict from a std HashMap, which replaces the hand-written
PyDict loops in the spend counter and router usage stores and the IndexMap in
the cooldown store. The rate-limit slot registry keeps IndexMap because it
hands back a Python dict in the order Python wrote it.
* refactor(redis): share Redis breakers and clients through one RedisHosts per host
redis-connect's RedisHosts now owns the per-server breaker and the shared clients
that python-bridge kept in two process-wide statics. A client takes breaker
settings and admits through its server's breaker, so no caller can pair a client
with a breaker of its own, and clients are keyed by Resolution alone. This drops
RedisClient::guarded_by and Breaker::same.
python-bridge keeps one RedisHosts per process and calls it with the GIL released.
The gateway builds one per connect, so its coordination and response cache on the
same server now share a breaker. RedisStoreSettings.breaker is back to breaker
settings, as on main.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(vertex_ai): run auth once per Vertex pass-through request
Drive /vertex_ai, /vertex-ai and /vertex_ai/discovery requests through the real
proxy app with a counting custom_auth, and assert each request authenticates
once and is served as the caller that auth returned. The generate-content
routes fail today because the route authenticates a second time.
* fix(vertex_ai): authenticate Vertex pass-through requests once
vertex_proxy_route already resolves the caller through Depends(user_api_key_auth),
but _base_vertex_proxy_route ignored it and authenticated the request again. A
custom_auth function ran twice per request, and a key with max_budget reserved
its estimated cost twice, so the unreleased first reservation exhausted the
budget after one or two calls.
The base route now uses the caller it is given and authenticates once itself
only when none is passed, which is the discovery route's case.
* fix(vertex): read the caller's key only from x-litellm-api-key or Authorization on Vertex routes
The single auth check on /vertex_ai and /vertex-ai now reads the same two headers main's effective check did, so a key sent only in another key header gets 401 again instead of authenticating.
* fix(vertex): authenticate the discovery route through the same Vertex key check
Both Vertex pass-through routes now resolve the caller once through
_authenticate_vertex_caller and hand it to _base_vertex_proxy_route, which
no longer re-runs auth. x-litellm-api-key accepts the same scheme prefixes
on /vertex_ai/discovery as on /vertex_ai and every other proxy route.
* fix(proxy): size a Gemini request's budget hold from its prompt and maxOutputTokens
A Gemini generateContent body's budget hold now counts its contents and
systemInstruction and reads generationConfig.maxOutputTokens. Before, it held
the model's whole context window, about $0.85 on gemini-3.8-flash, so a key
with a small budget got 422s on back-to-back calls it could afford
Also removes get_litellm_virtual_key, which nothing calls since the Vertex
routes stopped using it
* fix(proxy): count every Gemini system part and candidateCount in the budget hold
A Gemini generateContent body with several systemInstruction text parts lost
every part after the first when LiteLLM translated it to a chat request. That
dropped the later system text for non-Gemini models behind
/v1beta/models/<model>:generateContent, and made the budget hold count only the
first part. The adapter now keeps every text part, as a plain string when there
is one part and as text blocks when there are several.
The output hold now also multiplies by generationConfig.candidateCount, the
same way chat bodies use n, so a request asking for several candidates holds
budget for all of them.
* fix(proxy): keep the context-window hold for Gemini bodies the prompt count cannot see
A Gemini generateContent body with media, cachedContent, other roles, or
non-function tools keeps the full context-window hold, since the chat
translation drops or undercounts those parts. The hold also reads the
snake_case generation_config output fields, and several systemInstruction
text parts reach non-Gemini models joined into one system string
* fix(proxy): read only camelCase Gemini output fields in the budget hold
/v1beta forwards only generationConfig.maxOutputTokens to non-Gemini models, so a snake_case cap the model never receives must not shrink the hold. Snake_case bodies now get the default output hold.
* fix(proxy): count snake_case candidate_count in the Gemini budget hold
* fix(vertex_ai): skip malformed system parts and read Gemini output fields only on Gemini bodies
A systemInstruction part whose text is not a string is skipped, as main did, instead of failing the request. The budget hold reads generationConfig output and candidate fields, and counts contents, only when the body has no messages, prompt, input, query or documents field. An output cap of 0 or below counts as no cap. Test recorders keep tuples instead of mutating lists.
* fix(proxy): take the budget hold's request shape from the route
On generateContent routes the hold counts only contents and reads its output
cap and multiplier from generationConfig, so stray fields of other APIs can no
longer hide the prompt. A max_tokens of 0 holds no output again, while negative
values and a Gemini maxOutputTokens of 0 or below hold the default output
* fix(proxy): hold one candidate for a count too large to price and read system parts like content parts
* fix(proxy): keep the spend already counted when a budget hold is far over the budget
* fix(proxy): count candidateCount in the budget hold only for models that generate the candidates
A non-Gemini model reached through /v1beta generateContent never receives candidateCount, so a huge count held the whole budget for the length of the call while the model produced one answer
* fix(proxy): hold a candidate count only when every deployment generates it, and drop n when every deployment drops it
* fix(proxy): count candidateCount when any deployment generates it and always count n
The stricter rules from the previous commit let a key spend past max_budget through a Gemini name no deployment serves, an allowed_openai_params n, and a router fallback. Restore the earlier rules and add regression tests for those three shapes.
* fix(proxy): cap the candidateCount hold at main's default output hold
A candidateCount like 1e300 sent to a mixed Gemini and Claude group, or through a key alias from a Gemini name to Claude, held a whole key while Claude served the call and answered a strict key 422. On generateContent routes the counted candidates now raise the output hold at most to the default output hold main used for the model, so no count makes a call hold more than on main. Chat n is unchanged.
* fix(proxy): hold every Gemini candidate up to the merge base's whole hold
The previous cap stopped the candidateCount output hold at the default
16384 tokens. Because the hold now counts the Gemini prompt instead of the
whole input window, a strict key admitted a call like 4 candidates of 8000
tokens and then spent past max_budget, where the merge base refused it.
Every candidate is now held at its output cap, like chat n, and the whole
hold is clamped at what the merge base held for the same call (the model's
input window plus the default output), never below one candidate's hold.
A huge count still holds no more than the merge base did. Chat n is
unchanged
* fix(proxy): let a Gemini output cap that /v1beta drops only raise the hold
The Vertex and Gemini pass-throughs forward every spelling of the output
cap, but /v1beta passes only generationConfig.maxOutputTokens to other
models. The hold read only that spelling, so a snake_case cap above the
default output held less than the pass-through could spend. A cap in any
other spelling now raises the hold to it and never lowers it
* fix(proxy): hold Gemini candidates of a model with no context window up to the default output
A Gemini model the cost map has no context window for, priced in its deployment, held one
candidate for any candidateCount, so parallel calls could pass their budget holds. Its
candidates are held again up to its counted prompt plus the default output hold
* fix(proxy): keep main's Gemini hold unless the request caps its output
An uncapped generateContent call, or one capped only by a snake_case
spelling, now holds what main holds (the context window plus the default
output). A call with generationConfig.maxOutputTokens holds its counted
prompt plus cap times candidateCount, never clamped to the deployment's
max_output_tokens and never more than main's hold. A model with no context
window keeps main's no-hold, and candidateCount always counts on
generateContent routes. Also keeps get_litellm_virtual_key in
pass_through_endpoints/common_utils.py, so an out-of-tree importer
does not break
* fix(proxy): keep main's budget hold on generateContent routes
The PR now leaves the generateContent budget estimate as main has it.
A live risk check showed the counted Gemini hold under-reserves (the
prompt counter undercounts Gemini bodies, tiered and image output rates
are not read, and request fields the handler merges later are not
counted), so that estimate moves to a follow-up. The single auth pass
on the Vertex pass-through and the systemInstruction adapter fix stay
* fix(vertex_ai): keep main's Gemini adapter and pass a missing Authorization as no key
The Gemini adapter change for later systemInstruction parts goes back to
main, since it is not the double-auth fix this PR is for. The Vertex auth
dependency now hands a missing Authorization header to the auth check as
no key, like main's first auth pass, so custom auth and the 401 text see
what they see on main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(caching): don't cache an Anthropic Messages stream whose content blocks are all empty
A non-Anthropic model routed through the Messages adapter turns an empty reply into a single empty text block. The no-output check counted any content_block_start as output, so the async stream cache stored that empty reply and replayed it on every repeat. Judge each content_block_start and content_block_delta by its payload instead, like chat streams judge a choice: a block carries output when a field other than type holds something besides "", [] or null. Empty replies are no longer stored and stored ones become misses. A bare {} is not treated as empty, since the only block fields that hold one (tool_use input) come with an id and a name
Fixes#49931
* fix(caching): treat thought-signature-only and non-streamed empty replies as no output
* fix(caching): treat whitespace-only replies as no output
* fix(caching): keep whitespace replies and text completion fills cached
* refactor(caching): drop the unused streamed-choice branch of the output check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Add per-key and team prompt storage restrictions for spend logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply spend-log prompt restrictions to passthrough requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply team prompt storage restrictions to queued requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings
The spend-event collector decided whether to ship request and response
bodies from the global setting alone, so an opted-out key or team still
sent prompts through the collector transport. It now uses the same
key/team check as the spend log writer.
A key or team setting of the string "false" was ignored and prompts kept
being stored. It now counts as false, the same way the global setting
parses strings.
Splits the 42-case tenant storage test into one test for the policy and
one for the pass-through logging contexts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows
Failure rows, executed batch rows, semantic index embeddings, websearch
interception calls and errors rejected before request setup built their
logging metadata without the key and team metadata, so a restricted key
or team still had prompts stored on those rows. They now stamp it through
one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata.
The dashboard toggle now shows a saved string "false" as off, matching
the server, and its label and help text come from the translation file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(xai): use billed image cost and resolution prices
* test(xai): mark image generation tests as a package
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): forward Codex agent messages that carry encrypted content intact
Codex multi-agent v2 sends worker messages as agent_message items whose
encrypted_content block only the model server can decrypt. Bedrock Mantle and
Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into
assistant text, pasting the ciphertext into the prompt and dropping the sender
and recipient. Bedrock now decrypts these items itself, so the rewrite left the
recipient model reading an opaque token instead of its task.
Agent messages with an encrypted_content block now go to Bedrock untouched on
both endpoints. The Bedrock configs declare supports_encrypted_agent_messages,
and the shared normalizer keys off that flag so the two cannot drift. Mantle's
/v1/responses route for open-weight models still rejects agent_message in any
form, so it keeps the old rewrite. Plaintext-only agent messages keep the
rewrite on every route.
* fix(bedrock): only keep worker messages locked for OpenAI GPT models
Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so
Codex puts plain text in the locked slot and Bedrock rejected it with
invalid_encrypted_content once the route alone decided to forward it intact.
Decide from the model instead, and restore the full-value rewrite assertion.
* feat(videos): add first-class OpenRouter video request fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): remove recursive image parameter fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve EdenAI native parameter passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): deduplicate frame roles in linear time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve image selection and dropped parameter semantics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve provider scalar parameters and raw extra body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): retain accepted scalar inputs for Vertex and DashScope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): add Kimi K3.1 rows inherited from Kimi K3
* fix(bedrock): sync published Kimi K3.1 Highspeed prices
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): format dashboard values through i18next formatters
Register named formats on i18next's formatter (usd, compact, duration,
relativetime, number and five date presets) with a cache keyed on the
language and options, add format.* keys to en.json, and move the usage
stat strip onto them.
* fix(ui): keep number formats in USD and the format cache bounded
Force USD on the number and compact formats, clean underscore language
codes like i18next does, fall back when options cannot be serialized,
keep formatted placeholders alone in format keys, and pin the remaining
stat strip values in the German test.
* refactor(ui): format durations with Intl.DurationFormat and load its polyfill on demand
Browsers without Intl.DurationFormat (Chrome and Edge before 129, Firefox before 136) load the pinned FormatJS polyfill through a dynamic import and re-render translations once it is installed. The milliseconds-to-units split and the 0s fallback stay, so the output matches the previous hand-built formatter.
* test(ui): cover duration edge cases on the polyfill path
* refactor(ui): drop the unused duration format and its polyfill
Nothing outside tests called format.duration, and its 1m 15s style doesn't match the decimal-second latencies the dashboard shows today. A duration format will come with the pages that need it, in the style they already use.
#50009 moved Aria out of the dashboard header into the sidebar with its own
page, and #50195 renamed the Skills nav link and AI Hub tab to Skills Hub and
relabeled the skills search box. The UI e2e specs still looked for the old
header button, dialog and labels, so the scheduled litellm-e2e run has been
red since build 815.
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations
A team newrelic (or other preset) destination exported spans without the
operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback
was listed, because only the OTel logger's async_pre_call_hook seeded
requester_metadata into Baggage and that hook runs only for loggers in
litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the
request data once requester_metadata exists.
Resolves LIT-9783
* fix(otel): seed only request metadata from the proxy pre-call hook
The pre-call re-seed also passed the request model, which by then can be an
alias target, mcp-tool-call or an internal embedding model, so it rewrote
gen_ai.request.model on the server span for V2 deployments without otel listed.
Seed only the request metadata and keep the model auth seeded. Adds a unit test
for the pre-call seed and the guardrails_only skip, drops the explanatory
comments, and requires one server span plus the model span in the integration
test.
* test(otel): annotate new test variables Final and build test config without mutation
Addresses review: the new unit and integration tests now follow AGENTS.md
(LIT010 Final on every variable, no in-place mutation, read-only parameter
types, one for-clause per comprehension).
* test(otel): cover key-level callbacks and assert tenant spans carry no credentials
Parametrizes the tenant metadata test over team and key callbacks, and adds a
test that sends credential headers and a planted New Relic key with an
allowlist naming credential headers, then asserts no exported span carries the
master key, the virtual key, the team's New Relic key or any credential header
value, and that every export used only the team's own New Relic key.
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy
checked the same response a second time (LIT-10481). The deployment hook now records guardrail +
response hash on the request and on the response, and the proxy skips a guardrail only on a
matching record. The hash covers every field except ids, usage, model and other metadata set
between layers, so a response rewritten between the layers (OCR included) is still checked.
Guardrails with their own post_call hook keep their pre-#40154 behavior
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(usage): tolerate entity metadata database failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(usage): keep spend when metadata lookups hit database failures
Treat transport errors and the query engine crash on a closed connection as database
failures in the key, user and guardrail lookups of the aggregated, keys, users,
model_top_keys and cache_leakage_keys routes, and keep prisma out of module-level imports
so a proxy install without prisma still imports the usage endpoints.
* fix(usage): fall back only on database outages, on every daily activity route
Data-layer Prisma errors such as a rejected query or a missing table return 500 again, a key whose details failed to load reports key_exists null instead of false, and the non-aggregated daily activity routes keep spend when the key or entity metadata lookup hits an outage
* fix(proxy): skip the key metadata lookup when the daily rows carry no keys
* fix(usage): keep the guardrail breakdown fallback for any Prisma error
A missing guardrail usage table or column made the Usage call return 500 after the outage-only narrowing. The guardrail lookup now falls back on outages and on any Prisma error, as it did before this PR
* fix(usage): keep spend when the team, organization, customer, or agent name lookup drops
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): jais-30b-chat price was per 1K tokens, store it per token
* fix(watsonx): jais-13b-chat price was per 1K tokens, store it per token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(proxy): serve Codex its own release's model catalog, fetched from GitHub
When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that
release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the
fetch fails, times out after 2 seconds, does not validate, the version has no tag, or
LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight
most recent versions are kept. The integration tests ask as the bundled release so their expectations
hold whether or not the proxy can reach GitHub
* fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions
Failed versions now live in their own cache so they never push out a fetched catalog, requests for a
version already being fetched share that fetch, each worker starts at most six fetches a minute and
serves the bundled catalog past that, and the version check only accepts ASCII digits
* fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap
Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine
failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the
codebase
* refactor(typing): clear basedpyright Any errors (wave 1)
* refactor(typing): clear basedpyright Any errors (wave 2)
* refactor(typing): clear basedpyright Any errors (wave 3)
* refactor(typing): clear basedpyright Any errors (wave 4)
* refactor(typing): clear basedpyright Any errors (wave 5)
* refactor(typing): clear basedpyright Any errors (wave 6)
* fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves
* fix(logging): keep callbacks running when litellm_params is a params model
should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request
Adds behavior tests for the changed lines codecov listed as not run
* fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter
The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call.
* fix: bill regional uplift multipliers float() accepts, bytes included
* fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type
* fix(bedrock): run realtime guardrails on Nova Sonic sessions
* fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output
* fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames
* fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes
* fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs
Text parts of one message are joined before screening again, so a phrase split
across parts is still blocked, and a mask collapses them into one part. The
spoken notice is realtime_violation_message or the block reason, quoted only on
Nova Sonic. Client-written output_text and text parts are screened. Key and team
guardrail opt-outs reach regular and translation realtime sessions. A non-string
upstream transcript skips screening as before.
* fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages
* fix(realtime): keep every per-part mask when a joined text masks differently
* fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails
Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a
guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the
session request data with its metadata, so key and team policies apply, and a ModifyResponseException
block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock.
* fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items
* fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened
* fix(realtime): screen text parts in a loop so large messages keep their masks
* fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata
* fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): pin /v1/responses requests for prompt caching affinity
* fix(router): pin previous_response_id turns and keep native tools in the Responses pin key
A Responses turn that continues a stored response now goes to the deployment
that answered it, unless that deployment already failed this request. Native
Responses tools are hashed into the pin key as sent instead of converted, so
two requests with different native tools no longer share a pin and a native
request no longer logs a false dropped-tool warning. The pin keeps its
cacheable deployment ids in a sorted tuple so loggers can serialize request
metadata.
* fix(router): leave a Responses request with a malformed tool to the provider
* fix(router): keep the Responses prompt cache pin out of logged metadata
The pre-call check now keeps the pin in a router-owned in-flight store and
puts only a random token in litellm_metadata, so loggers that ship metadata
never see the routing key or deployment ids. The store is shared across check
instances, so turning prompt_caching off and on at runtime keeps pinning, and
a retry drops its previous attempt's token. The skipped-input debug log now
names only the error type.
* fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries
The prompt caching check no longer narrows the candidates inside the callback
filters. It names the deployment it prefers, and the router core applies that
preference at the pinned deployment step, after tag, region, pre-call and plugin
filters, only when the deployment is still a candidate. A pin can no longer
send a request to a deployment its tags do not allow.
A retry skips the pin when the pinned deployment was already tried in this
request, read from the retry breadcrumbs in both metadata buckets and the
excluded and retry-skipped ids. When the retry count outruns the breadcrumbs
the check stands down, so a pinned deployment that keeps failing is not
retried on every attempt.
* test(router): list the prompt caching check as an in-memory cache caller
The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers
* fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records
* test(router): lift test cooldowns through the cooldown store's evict_local
* fix(router): key Responses tools like their chat twin and drop the pin token after success
A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict.
* test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): stop parsing formatted dates and spend back into values
Regenerating a key stored its new created_at as locale text and parsed it
back for display, which shows "Invalid Date" on browsers whose date format
JavaScript cannot parse. It now stores an ISO timestamp.
The model health table stored check times as locale text and sorted by
parsing that text, which misorders or drops rows outside US English. It now
keeps the raw timestamp and formats it in the table cell, so the shown text
is unchanged.
getSpendString decided whether spend rounds to zero by stripping commas from
its own formatted output. It now compares the number against half of the
smallest shown unit, which gives the same output for every value and no
longer depends on how the digits are rendered.
* fix(ui): keep getSpendString exact at high precision
0.5 / 10 ** decimals drifts by one ulp once 10 ** decimals is no longer exact
(past 22 decimals), so a value right at the threshold could flip from "< $..."
to a row of zeros. Parsing 5e-(decimals + 1) gives the correctly rounded
threshold for every precision.
* feat(router): add Switchyard Stage Router strategy
* feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router
Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes
ModelCall.respond and fail async. The classifier is now a tagged union of an LLM
judge and a decision model, and decision calls go through the router's adecisions
so any decisions provider works. Algorithm misbehavior and library errors route to
the capable model instead of failing the request, and the session id plus client
session headers are always passed so capable holds apply without a classifier
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): keep the Switchyard router inside the type-check gate
The gate's environment did not install the optional switchyard extra, so every
libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other
optional runtime packages, and imports go through switchyard_rust.libsy, which
ships py.typed. Step handling moves into helpers so no Final is rebound in the
loop, and the dashboard API types are regenerated for the new routing fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): route Switchyard on every endpoint and reject unsafe router saves
The router now passes each request's call type to routing strategies, and
Switchyard converts the request to chat messages with the existing converter for
that call type before translating it. /v1/messages turns with tool or thinking
blocks used to fall back to the capable model, and now route on their signals.
The Anthropic converter can keep tool_result.is_error on an analysis copy, so
Switchyard sees tool failures there, while requests sent to models never carry it
Saves of a strategy router that lists itself now fail with a 400 instead of a
500 with a dead row, the capable and efficient models must differ, judge and
decision calls are not retried before failing open, the Switchyard routing
fields are classified for redaction, and the edit dialog returns the saved
router instead of the stale one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): catch self-routing on patch and rename, and trim Switchyard tests
The self-routing check now reads the stored params through the decrypting
helper and runs when a patch carries no litellm_params, so a PATCH that points
a router at itself or renames it onto one of its targets returns 400 instead of
a 500 with a dead row. The Switchyard tests share one transcript builder and
router fixture, and reformatting noise in unrelated test files is reverted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(ui): make Switchyard a strategy in the existing auto router form
Switchyard is now a card in the existing strategy picker. Picking it swaps the
tier and classifier fields for the Switchyard fields, and the shared name, team
and save button create it through one submit branch, the same flow as the other
strategies. The separate add form, the extracted picker and the hidden-form
swap are gone, and the edit dialog keeps its own Switchyard form
The judge's session_affinity and message_hash_fallback settings are removed,
since Switchyard rejects both for the stage router
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): accept partial tool semantics and show server errors on model edit
The Switchyard tool semantics field rejected mappings that list only some
categories, although the server accepts them. Model PATCH errors were
reduced to a generic message, so it now goes through the shared API client
like model create and surfaces the server text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): show the routing strategy picker up front in the auto router form
Every strategy was hidden behind an Advanced configuration link, so the
quick setup read as the only option and never said it creates a heuristic
router. The picker now always shows with Heuristics as the labelled
default, the quick setup sits under it, and picking another strategy opens
that strategy prefilled with the chosen models
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show the server's reason when a model settings or API key update fails
Both screens replaced the rejection with a fixed message, so a 400 such as
a router routing to itself read only as a generic failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(router): route Switchyard on the Rust router core
main replaced the Python router core with the Rust one, which left
Switchyard routers registered nowhere. Register them with the other
pre-routing strategies, hand each strategy the endpoint the request came
through, and evict Switchyard routers on reload and delete like the rest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form
Switchyard session state is keyed by a hash of the caller's key hash and session id, and client
session headers no longer reach Switchyard unscoped. The add form carries the quick setup models
into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the
schema reads API-saved configs with nulls and structured decision instructions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): make the Switchyard test folder a package
* fix(router): read the Switchyard session key hash only from the proxy's own metadata
* fix(proxy): keep nested nulls as null when masking deployment params
* fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies
* fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): count embedding batches per input in pre-call checks
Embedding input lists went through the Responses input transform, which
calls .get() on each item, so every list-input /embeddings call logged
"failed to count tokens" and skipped the context window check. Lists of
texts, token IDs or token ID lists now count as their longest input,
since an embedding model's context window applies to each input on its
own
* fix(router): keep empty and boolean input lists on the Responses count path
An empty list carries no embedding evidence, and counting it as a zero
token batch skipped the instructions and tools of an empty Responses
input. Boolean lists are not token IDs
* fix(router): honor disable_token_counter for token ID embedding input
Text inputs already counted as zero through token_counter when token
counting is disabled, while token ID inputs were still measured by
length and could be rejected. The router reads the flag through its
dynamic config like its other process settings
* refactor(router): type the embedding count helper with a read-only sequence and final locals
* fix(router): count embedding batches per input on the media-free text projection
The per-input count now runs on the text projection from the embedding
media split, so data URIs a deployment embeds as media keep the media
path and only text items are checked against the context window. A
caption next to an image is checked on its own, and a text-only
deployment still counts an image data URI as text
* fix(router): keep embedding images out of the pre-call count only on image backends, and skip tokenizing inputs that fit by bytes
Media leaves the embedding count only when every deployment in the group declares image embedding input, so a text-only backend still counts a data URI as text and an image backend outside the TPM media list no longer gets a false 400. A text no longer than the smallest window in UTF-8 bytes counts as its byte length without the tokenizer, since a token is never shorter than a byte.
* fix(router): keep embedding images out of the pre-call count when any backend in the group embeds images
Mark Cohere embed-english-light-v3.0 as embedding images, as its v3 siblings are.
* fix(router): check embedding text batches only with an exact OpenAI tokenizer
litellm's default tokenizer counts non-English text several times higher than
non-OpenAI embedding providers do, so counting batches with it refused inputs
the provider accepts. Text batches on those groups now stay unchecked as at the
merge base, without the error log. Main's per-provider media split and the
cost map are restored, so strings and embedding media keep main's path.
* fix(router): count long embedding batches in slices so chat counts do not wait behind them
A text batch is now tokenized in slices of 65,536 characters, each its own offloaded call, so the FIFO count limiter can serve a chat count between slices. The no-OpenAI-tokenizer rule for text lists moved into the pre-call check, so RoutingContext.count_pre_call_check_tokens counts a text list by its longest text.
* fix(router): read embedding media off the event loop and wait on the count limiter in the fairness test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model-hub): show each deployment's full prices in the model hub
The model hub only showed one input and one output price per model, the highest across its
deployments. Each model group now lists every deployment's prices, read from the same entry
billing charges, including cached tokens, long-context tiers, flex, priority, batch and
tiered ranges. Both the logged-in and public hub detail views render them in one table per
deployment, labeled by provider only
* refactor(ui): move model hub pricing breakdown text onto i18n
Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged
* fix(model-hub): show Vertex regional prices and explicit cached reads as billed
A Vertex deployment outside the global endpoint now shows its token prices with the regional
uplift billing applies at the deployment's location. The explicit cached read price shows per
1M tokens on its own row instead of a raw per-token value. The breakdown also says that
adjustments made per request, like off-peak discounts and regional or data residency
surcharges, aren't included
* fix(model-hub): show prices written as text the way billing reads them
A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but
the hub dropped it, so a deployment showed no price or the cost map's price instead of its own.
The hub now reads numeric text for regular prices and size-based price ranges, and skips text
that isn't a number
* style(ui): format public model hub test
* test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget
RouterEvent and the types it carries (FailureKind, Blocked, MockFailure,
RetryCause, PickError) now serialize, so a host can log or forward events as
JSON instead of parsing their Debug text. Events are tagged by "event",
durations are seconds, failure kinds use litellm's exception class names
* fix(responses): map streamed flex_unavailable errors to 429 not 500
When OpenAI has no flex capacity, a service_tier flex request fails with type
resource_unavailable / code flex_unavailable. Non-streamed, this arrives as HTTP
429 and litellm raises RateLimitError. Streamed /v1/responses delivers the same
error as an in-stream event, and neither field was in _ERROR_CODE_HTTP_STATUS, so
it defaulted to 500 InternalServerError.
The HTTP status then depended only on stream: one provider outage looked like a
429 or a 500. The 500 reads as a server error in metrics and alerting, and callers
that retry on 429 do not retry it.
Map both resource_unavailable and flex_unavailable to 429 so the streamed path
matches the non-streamed path.
Fixes#49736
* test(responses): cover streamed flex outage end to end for error and response.failed events
Streams the flex_unavailable payload from #49736 through litellm.aresponses with an injected client and asserts a 429 RateLimitError for both in-stream event kinds, matching the non-streamed path
* test(responses): drop explanatory comment from flex status mapping test
* test(integration): cover streamed flex outage status on responses and chat bridge
Drives the flex_unavailable payload from #49736 through a real proxy as an in-stream error event and as response.failed. The native /v1/responses stream fails with rate_limit_exceeded and logs 429, and the chat bridge answers a 429 throttling error, where both reported a 500 before the mapping fix
---------
Co-authored-by: BinYi LIU <bliu1@berkeley.edu>
* feat(bedrock): enable native structured output by default for selected Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool
A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too.
* fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action
A role allowed only bedrock:InvokeModel got a 403 on schema streams once these
models went native, since native streams call converse-stream while the JSON
tool path buffered through converse. The retry now sends that request again
as a buffered JSON tool stream, the shape the base sent.
The wire test moves its neither-key and opt-in cells to Opus 4.7, its
Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the
bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native.
The batch wire test now checks a thinking schema record on both sides of the
contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5
and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8
cells assumed those models were opt-in.
* fix(bedrock): move the Claude 5 native output default to a key released proxies do not read
Released proxies load the cost map from main and have none of the native
output retries, so setting supports_native_structured_output on the Opus
5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide
or min/max schemas into 400s on v1.108.1.
Those rows keep supports_native_structured_output false and gain
bedrock_native_structured_output_by_default true, which get_model_info
reports as supports_native_structured_output, so this version behaves as
before while released versions keep the JSON tool.
* fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default
The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it
* fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key
A registered supports_bedrock_native_structured_output false now beats the
native output default, and batch uploads read the deployment's model_info, so
an opted-out deployment keeps the JSON tool on every record. The Opus 5 and
Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default,
so released proxies on main's cost map keep the JSON tool on them. The 64
property batch cutoff now applies only to rows with that key, so models that
were already native keep their batch output shape.
* fix(proxy): report a deployment's Bedrock native output opt-out on /model/info
/model/info merged the deployment's model_info over the backend row, so a
deployment that set only supports_bedrock_native_structured_output: false
still showed the backend's derived supports_native_structured_output: true.
The generic flag is now derived from the merged info unless the deployment
set it itself, through the same helper get_model_info uses.
* fix(bedrock): keep native output when a denied schema stream is resent buffered
A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output.
* fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend
* fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment
The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model
* fix(bedrock): answer a buffered invoke stream from the non-streaming reply
* fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected
* fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info
A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main.
/model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
