* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model-hub): show each deployment's full prices in the model hub
The model hub only showed one input and one output price per model, the highest across its
deployments. Each model group now lists every deployment's prices, read from the same entry
billing charges, including cached tokens, long-context tiers, flex, priority, batch and
tiered ranges. Both the logged-in and public hub detail views render them in one table per
deployment, labeled by provider only
* refactor(ui): move model hub pricing breakdown text onto i18n
Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged
* fix(model-hub): show Vertex regional prices and explicit cached reads as billed
A Vertex deployment outside the global endpoint now shows its token prices with the regional
uplift billing applies at the deployment's location. The explicit cached read price shows per
1M tokens on its own row instead of a raw per-token value. The breakdown also says that
adjustments made per request, like off-peak discounts and regional or data residency
surcharges, aren't included
* fix(model-hub): show prices written as text the way billing reads them
A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but
the hub dropped it, so a deployment showed no price or the cost map's price instead of its own.
The hub now reads numeric text for regular prices and size-based price ranges, and skips text
that isn't a number
* style(ui): format public model hub test
* test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget
RouterEvent and the types it carries (FailureKind, Blocked, MockFailure,
RetryCause, PickError) now serialize, so a host can log or forward events as
JSON instead of parsing their Debug text. Events are tagged by "event",
durations are seconds, failure kinds use litellm's exception class names
* fix(responses): map streamed flex_unavailable errors to 429 not 500
When OpenAI has no flex capacity, a service_tier flex request fails with type
resource_unavailable / code flex_unavailable. Non-streamed, this arrives as HTTP
429 and litellm raises RateLimitError. Streamed /v1/responses delivers the same
error as an in-stream event, and neither field was in _ERROR_CODE_HTTP_STATUS, so
it defaulted to 500 InternalServerError.
The HTTP status then depended only on stream: one provider outage looked like a
429 or a 500. The 500 reads as a server error in metrics and alerting, and callers
that retry on 429 do not retry it.
Map both resource_unavailable and flex_unavailable to 429 so the streamed path
matches the non-streamed path.
Fixes#49736
* test(responses): cover streamed flex outage end to end for error and response.failed events
Streams the flex_unavailable payload from #49736 through litellm.aresponses with an injected client and asserts a 429 RateLimitError for both in-stream event kinds, matching the non-streamed path
* test(responses): drop explanatory comment from flex status mapping test
* test(integration): cover streamed flex outage status on responses and chat bridge
Drives the flex_unavailable payload from #49736 through a real proxy as an in-stream error event and as response.failed. The native /v1/responses stream fails with rate_limit_exceeded and logs 429, and the chat bridge answers a 429 throttling error, where both reported a 500 before the mapping fix
---------
Co-authored-by: BinYi LIU <bliu1@berkeley.edu>
* feat(bedrock): enable native structured output by default for selected Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool
A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too.
* fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action
A role allowed only bedrock:InvokeModel got a 403 on schema streams once these
models went native, since native streams call converse-stream while the JSON
tool path buffered through converse. The retry now sends that request again
as a buffered JSON tool stream, the shape the base sent.
The wire test moves its neither-key and opt-in cells to Opus 4.7, its
Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the
bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native.
The batch wire test now checks a thinking schema record on both sides of the
contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5
and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8
cells assumed those models were opt-in.
* fix(bedrock): move the Claude 5 native output default to a key released proxies do not read
Released proxies load the cost map from main and have none of the native
output retries, so setting supports_native_structured_output on the Opus
5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide
or min/max schemas into 400s on v1.108.1.
Those rows keep supports_native_structured_output false and gain
bedrock_native_structured_output_by_default true, which get_model_info
reports as supports_native_structured_output, so this version behaves as
before while released versions keep the JSON tool.
* fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default
The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it
* fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key
A registered supports_bedrock_native_structured_output false now beats the
native output default, and batch uploads read the deployment's model_info, so
an opted-out deployment keeps the JSON tool on every record. The Opus 5 and
Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default,
so released proxies on main's cost map keep the JSON tool on them. The 64
property batch cutoff now applies only to rows with that key, so models that
were already native keep their batch output shape.
* fix(proxy): report a deployment's Bedrock native output opt-out on /model/info
/model/info merged the deployment's model_info over the backend row, so a
deployment that set only supports_bedrock_native_structured_output: false
still showed the backend's derived supports_native_structured_output: true.
The generic flag is now derived from the merged info unless the deployment
set it itself, through the same helper get_model_info uses.
* fix(bedrock): keep native output when a denied schema stream is resent buffered
A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output.
* fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend
* fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment
The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model
* fix(bedrock): answer a buffered invoke stream from the non-streaming reply
* fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected
* fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info
A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main.
/model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): keep text and accept string input for the Codex backend
Two request-side defects stop chatgpt/* serving structured output or a string
input.
The allowlist at the end of transform_responses_api_request omits "text". The
chat-to-responses bridge translates response_format into text.format, so the
allowlist discards strict schemas silently and the backend answers with prose
instead of JSON. The backend does honour text.format when it receives it.
Separately the backend rejects a bare string input with
{"detail": "Input must be a list"}, while the Responses API itself accepts
either shape, so the string needs wrapping before it is sent.
Both are request-shaping only, so neither depends on the streaming work in
#31332 or #34095, though those two block the same call and have to be in place
to observe this one end to end.
The "text" case was reported in #24356 and closed by the stale bot without a
fix.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): annotate the input coercion for the type-discipline gate
The LIT002 budget counts mutable-collection construction, and wrapping the
string input builds a list and a dict, which put the total two over its
ceiling. Both constructions are the wire payload this backend requires, so the
list cannot be replaced with an immutable equivalent; annotate the line with a
reason instead, matching the convention used elsewhere in the tree.
LIT005 freezes reasonless suppressions at zero, so the reason is required
rather than decorative.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* refactor(chatgpt): bind the coerced input instead of rebinding the parameter
Greptile flagged the parameter rebinding and the explanatory comments against
the repo conventions in CLAUDE.md, which bans rebinding a function parameter
(LIT011) and keeps comments to suppressions, TODOs and genuinely complex logic
Bind a Final local for the coerced input rather than reassigning `input`, and
drop both prose comments; the rationale lives in the commit and the PR body
rather than duplicated at the call site. The mutable-ok suppression stays,
since it is the allowed kind and the list is still constructed
Behaviour is unchanged: 24 tests pass, and the same 4 fail with only the
provider file reverted to base
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): forward max_output_tokens to the Codex backend and return incomplete non-stream responses
The Codex backend accepts and enforces max_output_tokens, so the allowlist now keeps it
instead of dropping it. A structured-output request could otherwise stream until the
backend gave up, ignoring the caller's max_tokens.
A capped request ends with response.incomplete, so the non-stream SSE aggregator now
treats that event as terminal and returns the partial response instead of an error.
---------
Signed-off-by: Alexander Chernov <alexander@chernov.it>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): do not surface server_tool_use as client tool_calls
Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.
Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.
* fix(responses): return Claude web_fetch calls as web_search_call open_page items
* fix(responses): keep Claude server web calls without a result as hosted items
A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call
* fix(anthropic): keep pending server calls native and hide server calls from client tool paths
Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.
* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls
Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again
* fix(anthropic): let guardrails and loggers read Claude's server tool calls again
Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.
* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing
A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.
* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge
Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.
* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses
Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.
* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions
* test(guardrails): type the fake Pillar client like its Protocol
* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool
Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result
* fix(guardrails): scan replayed Claude server calls and hosted web calls
Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.
* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks
* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows
* test: annotate the new server tool call test locals with Final and type their parameters
* test: type the new guardrail test helpers with litellm's tool call types instead of Any
* test: narrow the remaining new test helper parameters from bare dict and object
* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs
Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls
* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera
* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock/converse): omit reasoning signature for non-Anthropic models (#49969)
* test(bedrock): cover signed reasoning replay in mapped tests
* fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target
A signed thinking block from Claude replayed to a Converse model known not to be Claude, such as GPT-6 Luna or Nova 2, now leaves the request instead of going out as reasoningContent. Bedrock rejects that field on those models, so the whole request answered 400
A target that may still be Claude, such as an application inference profile ARN with no base_model, keeps the signed block, since Claude needs it on a tool-use continuation. Unsigned blocks keep the existing text path
The thinking-block error matcher added earlier on this branch is removed, because Converse and non-Claude Bedrock requests never reach that retry
* fix(bedrock): keep signed thinking for Claude behind a region prefix litellm does not list
ca. and in. inference profiles name Claude after a region prefix get_bedrock_cross_region_inference_regions does not have yet, so they counted as non-Claude and lost their signed thinking. The thinking-only turn test now pins the exact merged and continue-message shapes.
* fix(bedrock): limit the region-prefix Claude match to signed thinking and drop a dropped block's cache point
* fix(bedrock): drop Claude's bridged reasoning items before a Bedrock Responses API replay
* fix(bedrock_mantle): drop litellm-minted reasoning items before a Mantle Responses call
* test(integration): expect Bedrock and Bedrock Mantle Responses to drop minted reasoning
Bedrock and Bedrock Mantle answer 400 on a reasoning item litellm minted, so the
wire test now expects both to drop it, like OpenAI and Azure
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Add first-party floating model aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Align Moonshot alias parameter test with its target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Share floating alias proxy budgets and rate limits with targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Prove target-only team budgets count and block floating aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Share floating alias quotas with project and organization targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Coalesce colliding floating alias rate limit counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix latest alias Rust fallback and failure accounting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: adapt latest alias checks to typed rate-limit store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): discover upstream LiteLLM Gateway wildcard models
* fix(proxy): use the typed shared client for gateway discovery
* fix(ui): show the selected LiteLLM Gateway logo
* feat(proxy): hash UI passwords with PBKDF2 in FIPS mode
A FIPS OpenSSL refuses scrypt, so every password write fails there and
nobody can sign in with a password. When OpenSSL enforces FIPS, passwords
now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's
DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is
refused with one error log and left as it is, and the boot migration does
nothing, because FIPS mode is for new deployments.
PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the
five request paths run it on one thread instead of the event loop. A single
thread keeps a login burst from using up a CPU quota, which is what stalls
the loop with asyncio.to_thread. A login that queued for that thread asks the
throttle again before it answers, so a burst of guesses can't get past a block
that the failures ahead of it just set.
The breached password check is off in FIPS mode, because it runs the
password through SHA-1.
A standard OpenSSL reads False at every new branch, so standard mode keeps
scrypt, inline, on the same rows.
Resolves LIT-8425
* fix(proxy): take FIPS logins one at a time and recheck the throttle first
A login that waited for the KDF thread asked the throttle again only after
deriving, so a burst for one user kept that thread busy long after the user
was blocked. Logins now wait for their turn on a lock and ask the throttle
when it comes, so a blocked login costs no PBKDF2 and only one login at a time
holds a place in the KDF queue. Password changes and invite claims no longer
wait behind a whole burst of logins.
* fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check
A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the
real password stopped working and the old row text signed in. The migration now
leaves a row of that exact shape alone. Any other pbkdf2: text still migrates.
FIPS mode now logs a warning when it skips a breach check that is turned on.
The one-thread test no longer depends on timing, and the new test variables are
annotated Final.
* fix(proxy): keep the breached password check on in FIPS mode
* fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL
* test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt
* build(deps): lock filelock 4.2.4 on Python 3.11+ and 3.20.3 on 3.10 to clear osv-scan
Co-authored-by: Gilberto Plaza <keko950@hotmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump @azure/identity to 4.14.1 and @azure/msal-node to 7.1.2 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump oauthlib to 4.0.1 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Gilberto Plaza <keko950@hotmail.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Chirp and other Cloud TTS models now call us-texttospeech or eu-texttospeech.googleapis.com when the deployment's vertex_location is a US or EU location, keep the global host otherwise, and an explicit api_base still wins. Gemini TTS accepts more US and EU regions.
* fix(bedrock): forward the per-turn-control beta to Bedrock Invoke
Claude Code sends `anthropic-beta: per-turn-control-2026-07-01` together
with a mid-conversation `role: system` message carrying a message-level
`output_config` (for example an effort change). The beta header config
mapped the beta to null for `bedrock`, so LiteLLM dropped it while
keeping the message in place on models flagged
`supports_mid_conversation_system`, and Bedrock Invoke rejected the turn
with "messages.N.output_config: Extra inputs are not permitted".
Direct Bedrock InvokeModel probes with a synthetic request show Claude
Opus 5.5 accepts that message shape only with this beta, and that Claude
Haiku 4.5, Sonnet 4.6, Opus 4.8, Opus 5, and Sonnet 5 accept the beta on
a plain request, so map it to itself for `bedrock`. `bedrock_converse`
is unchanged.
Generated with AI
Co-Authored-By: Claude Code
* test(anthropic): expect the per-turn-control beta to reach Bedrock Invoke
Generated with AI
Co-Authored-By: Claude Code
* test(bedrock): drop the duplicate per-turn-control filter test and cite the live check
* fix(rust): allow the per-turn-control beta on bedrock in the Rust beta table
The Rust table must match litellm/anthropic_beta_headers_config.json cell for cell, and the JSON now maps the beta for bedrock. Also type the new Bedrock Invoke test.
* fix(bedrock): forward per-turn-control on Bedrock Invoke only where the cost map flags the model
Bedrock InvokeModel accepts per-turn-control-2026-07-01 on Claude 4.5 and newer and answers 400 invalid beta flag
on Claude 3 Haiku. Released proxies load the beta map from main, so forwarding it in the bedrock row would turn
Claude 3 Haiku requests on every running release from 200 into 400. Keep that row null and forward the client's
beta from the Invoke transformation when a new claude-per-turn-control cost map rule sets supports_per_turn_control.
* fix(bedrock): keep the per-turn-control beta through the messages handler filter on Bedrock Invoke
The /v1/messages handler filtered client betas against the beta map before the
Bedrock Invoke config could add per-turn-control back for models the cost map
flags, so the beta never reached Bedrock. Provider configs now name betas their
model accepts beyond the beta map, and the handler filter keeps those
* fix(bedrock): keep a deployment's per-turn-control flag on that deployment
supports_per_turn_control is now deployment-scoped, so a deployment's opt-in or opt-out no longer changes the beta forward for other deployments of the same Bedrock model. The forward reads the deployment's own model_info first, then the model map.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(redis): route the router, rate limiters and auth through the native store families
* style(redis): satisfy the python lint gates
Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): give the registry's default usage store a memory-only handle
RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): build stores from native handles and wait for router tasks
Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): let admins set tpm_limit and rpm_limit on tags
The tag create modal and edit form said tag TPM/RPM limits were unsupported and never sent them, while the proxy has enforced tag-level rpm_limit and tpm_limit since #41807. Replace the notice with TPM Limit and RPM Limit inputs wired to /tag/new and /tag/update, validated as nonnegative integers, with a cleared limit sent as null so the proxy drops it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): prefill tag budget and limit fields from the Edit Tag button
The edit form only seeded max_budget and budget_duration when opened from the row's Edit action, so the Edit Tag button inside the tag view showed empty budget and limit inputs and a cleared field could not be sent as null. Seed from the loaded tag on both paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): accept null budget fields when editing a tag
A tag whose budget row has max_budget or a limit set to null failed the edit form's zod schema with 'expected string, received null' once the fields were prefilled. Treat null like an empty input and type Tag's budget fields as nullable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): accept a null description when editing a tag
A tag created without a description comes back from /tag/info with description null, which the edit form's zod schema rejected with 'expected string, received null' on save. Allow null and type Tag.description as nullable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): ignore invalid limits left in a collapsed tag budget section
An invalid value typed in Budget & Rate Limits and then hidden by collapsing the section blocked Create Tag and Save Changes with the error out of sight. When the section is collapsed and the only errors are on its fields, drop them and submit the visible values, which already discards the hidden budget fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): name the expected payload so the inline-object lint budget holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move remaining outlier text onto i18n
Moves 146 English messages that the earlier i18n batches missed into en.json.
They live in helpers, zod schemas, label fallbacks and shortcut descriptions
that the batch scope script did not count, so no batch picked them up. The
English on screen is unchanged.
* refactor(ui): keep exported compliance error text in English
The compliance results' "Error: ..." text is also written to the CSV export,
so it stays a literal. The policy table's unnamed fallback now reads i18n.t
inside its helper instead of taking t as a new parameter.
* test(e2e): add a Codex row that calls a local MCP tool on a first and a resumed turn
* test(e2e): assert the codex mcp row got each value back from the mcp tool call itself
* test(e2e): check the codex mcp row exits cleanly on both turns
* test(e2e): drop the unused content type from the codex mcp result model
* feat(team): let default_team_params attach access_group_ids to every new team
* fix(types): keep default access_group_ids as Sequence for the type discipline gate
* fix(settings): chunk the default access group existence lookup
* test(team): prove default access groups reach the saved team row and type the settings fixtures
* fix(team): check default access groups on the writer so a fresh group is not rejected behind a read replica
* feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests
* fix(spend-logs): keep bridged Responses rows and classifier audit in latest_turn mode, reject unknown values at boot
* fix(spend-logs): drop the recursive call in the latest_turn trim and type the stateless test cases
* test(spend-logs): format the stateless latest_turn cases
* test(spend-logs): wrap the classifier audit fixture
* fix(spend-logs): type the latest_turn trim inputs
* fix(spend-logs): leave the raw request body unvalidated before the latest_turn trim
* fix(spend-logs): trim classifier rows and ignore continuation keys off /v1/responses in latest_turn mode
* fix(spend-logs): take the classifier's originating request from the logging payload and redact a copy of vector store results
* fix(spend-logs): redact the body selected for storage, after the latest_turn trim
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
Reworks the redis-client, connect, resolve and testing crates, adds python-compat json loads and
TTL seconds, replaces the redis-store memory tier with a DualCache plus a shared failure log, and
gives the Python side the Redis timeout log throttle and a request OTEL parent span for service events
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise
Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the
value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way.
s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls
* refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map
* fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request
* fix(gemini): omit raw response body from audio transcription error messages
* fix(gemini): raise incomplete transcription interactions as non-retryable 400 and drop response bodies from Vertex transcription errors
* fix(audio_transcription): keep response bodies out of transcription errors
Keep the upstream status on a Gemini interaction that did not complete, so
the router still retries it as it does on main. Gemini stops the same long
clip on some tries and not on others, so a retry can recover it.
Stop quoting the reply in the Deepgram, ElevenLabs, Watsonx, Mistral,
OVHcloud and Scaleway parse errors, and hide pydantic input values on the
Gemini and Vertex response models.
* fix(audio-transcription): keep reply bodies out of watsonx, whisper, xai and validation errors
Watsonx and whisper no longer print the parsed reply after Got: when it has no text field. TranscriptionResponse and the xAI response model hide input in validation errors, so a malformed field cannot quote transcript fragments. Deepgram and ElevenLabs keep the original error text, which is now safe.
* fix(audio-transcription): keep transcripts out of the shared converter's parse errors
* test(audio-transcription): compare wrapped lookup errors instead of pinning upstream wording
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools
* test(integration): build the web search filter proxy config without mutating the shared config
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add a beta language setting to the account menu
The account menu gets a Language field, marked Beta, that lists each language by its own name with the English name beside it. The choice is saved per browser in localStorage and applied with a page reload, so memoized and imperative translations are read fresh.
English stays the default. An unset value, English, an unknown or unsupported code, a locale that fails to load, and any string a locale has not translated all show English. The field stays hidden until a second locale is registered in LOCALES, so this change is invisible until the first translation lands.
* fix(ui): keep the help translate link text inside the link
<link> is a void HTML tag, so the Trans parser dropped its children and rendered an empty anchor. The note now uses <githubLink>, and the language list is at least as wide as its trigger.
* fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check
* test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts
* feat(bedrock): enable native structured output for Opus 5 models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep JSON-tool streaming coverage on unchanged route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
