anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or
Semaphore, so the next task to acquire waits forever on a lock nobody holds
(agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every
httpx request, including the prisma query-engine client
litellm imports anyio directly but never declared it, so pip and uv kept an
already installed anyio 4.13.0 when installing litellm and accepted an
explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker
and CI installs do not change. Both the litellm and litellm-core manifests get
the floor, and the lock only records the new direct dependency
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(openai): drop unsupported reasoning effort for gpt-6.1-sol
gpt-6.1-sol accepts low, medium, high, xhigh, and max. none and minimal were still forwarded, because chat completions ignored an explicit supports_none_reasoning_effort=false and the Responses mapper never applied that contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): mark the stripped reasoning object as outgoing JSON
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Azure responses reading the OpenAI effort map
gpt-6-astra on Azure accepts reasoning.effort none. The responses gate was checking the OpenAI entry, which rejects it.
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): gate every reasoning_effort level from the cost map on chat and Responses
One gate reads the model's cost-map row for whatever effort level a request
asks for, on Chat Completions and the Responses API, for OpenAI and Azure.
xhigh stays opt-in on chat (Azure also keeps none opt-in); every other level
is refused only when its supports_<level>_reasoning_effort flag is false. A
refused level is dropped under drop_params or answered with a local 400, and
Responses keeps the rest of the reasoning object when it drops effort. The
guards that follow judge temperature by the level the provider will see.
Tests move into the mapped test files and derive the expected outcome for
each level from the map row, so a new GPT-6 model needs only its row.
* test(openai): pin gpt-6.1-sol's refused effort levels and prove the row flags alone drive the gate
* fix(cost-map): azure gpt-6-astra refuses none and honors max on Responses, as live calls show
* test(reasoning_effort): pin the astra max and none flips through the adapter and Foundry paths
* fix(reasoning_effort): keep max out of the route gates and correct the gpt-5.3-codex rows
The same model honors max on Responses and refuses it on Chat Completions, so one row flag
cannot gate it on either route. The gates now skip max, the Azure gpt-6-astra rows keep max
off so the /v1/messages adapter degrades to xhigh and /model_group/info stops short of it as
on main, and chat still forwards max for the Responses bridge.
The gpt-5.3-codex rows said none and xhigh off and minimal on; the live model answers the
other way round (2026-10-01), and an opt-out gate would have dropped the levels Codex CLI sends.
* fix(reasoning_effort): bridge function tools to Responses when the chat gate drops none
* docs(reasoning_effort): correct the max advertisement note
* fix(azure_ai): read the Foundry row on the Responses effort gate and turn none off for Azure gpt-6.1-sol
* fix(azure): gate a prefixed Foundry deployment's Responses effort off its row
litellm.responses(model="azure_ai/<deployment>", api_base=<Foundry host>) is
re-routed to the azure provider with the azure_ai/ prefix intact, so the Azure
Responses config saw a prefixed name, skipped it as another provider's
pass-through, and forwarded none and minimal to Azure even under drop_params.
The Azure gpt-5 resolver also read no row at all for a prefixed name.
The Azure Responses gate now detects the family the way the Azure chat gate
does, and an azure_ai/ name on the Azure OpenAI configs reads its azure_ai/ row
when the map has one, else the azure/ twin, the rule the Foundry chat config
already applies to the same deployment.
* fix(cost-map): turn none off on azure/eu/gpt-6-astra like its sibling rows
* docs(reasoning_effort): say how each gate reads none and shorten the module docstring
* fix(azure_ai): call the public effort-level check from the Foundry Responses gate
* test(reasoning_effort): fold the Foundry Responses tests into the mapped file and type the row helpers
---------
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(proxy): opt-in self-serve budget edits for key owners on /key/update
Adds general_settings.self_serve_budget_policy (disabled by default) so a
non-admin can change max_budget or budget_limits on a key they created and
still own. lower_only allows adding or lowering limits. ceiling also allows
raising or removing them when the owner could create that key with the new
budget via /key/generate (same member permissions and delegation ceiling).
soft_budget and spend stay admin-only
* fix(proxy): sync dashboard API types and cover the UI session cap on update
* test(proxy): cover team role restriction for owner budget raises
Drop two unreachable branches from the non-raising key generation check.
* fix(proxy): close self-serve budget bypasses for linked budgets, null caps and retained limits
Keys with a linked budget_id need an admin to change max_budget, owners cannot
leave a key uncapped when an upperbound max_budget is configured, and the
ceiling check now runs on the budget the key would carry after the update.
* fix(proxy): keep open key budget windows and the admin path on self-serve budget edits
A resubmitted budget_limits window now keeps its stored reset_at while that period is still
running, so adding or editing one window no longer restarts the others. Team and org admins
answer False on a ceiling loosen so they fall through to _check_key_admin_access as before.
The self-serve tests feed the team through the cache and prisma mocks instead of patching
get_team_object, and every helper is typed
* perf(proxy): group key budget windows by duration in one pass
The per-duration minimum used to rescan the whole window list once per distinct
duration, so a request with thousands of windows pinned the event loop before any
permission check ran. Sort once and take each group's minimum instead
* refactor(proxy): take budget windows as a Sequence in the delegation ceiling checks
* test(proxy): assert allowed self-serve budget edits return no denial
Also drop the unreachable assert_never arm from the exhaustive budget change match.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* refactor(count_tokens): hold the tool counting wiring for a follow-up commit
* refactor(count_tokens): hold the chat template tool sections for a follow-up commit
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* fix(count_tokens): skip tool_choice and non-function tools for chat template families, leave out Fireworks V4.1 spelling
* fix(count_tokens): count Anthropic tools in the shape the messages bridge sends them
A missing description is no longer rendered as null and strict is counted when set, matching what DeepSeek V4 and Qwen3.5 deployments receive. Adds tests that pin each template's schema rendering and its system turn.
* test(count_tokens): check the prompt caching threshold agrees with token_counter for DeepSeek V4 and Qwen3.5 tools
* fix(count_tokens): skip Anthropic web search tools and count unmapped Anthropic tool keys for chat template families
* test(token_counter): cover chat-template tool counts across count routes, budgets, ITPM and context edits
Integration cells check that DeepSeek V4 and Qwen3.5 tools are counted the way their chat templates render them on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens, strict budget reservation, project ITPM on chat, messages and responses, and the clear_tool_uses trigger, with grok controls keeping OpenAI framing
Show discovered Kubernetes pods on the model page for admins
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Requests with an explicit session id stick to one discovered pod via rendezvous hashing
Requests without a session id keep round robin
Pod routing records are included in logs, spend logs, and traces
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(baseten): list current Model APIs instead of retired 2023 deployment IDs
`baseten/*` wildcard expansion and `/v1/models` read a hardcoded set of
three 2023 deployment IDs (Falcon 7B / WizardLM / MosaicML) that no longer
exist, while the model cost map already carries current Baseten entries.
- Populate `baseten_models` from the model cost map, like fireworks_ai.
- Add `BasetenConfig.get_models()` against the Model API `/v1/models`
endpoint and register it in `ProviderConfigManager.get_provider_model_info`,
so `check_provider_endpoint: true` returns the live catalog.
- Remove the retired deployment IDs from `constants.py`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(baseten): mark retired Model APIs deprecated and skip them in listings
15 Baseten entries in the model cost map are no longer servable: the Model
API returns 410 "model version ... has been deprecated" (DeepSeek-V3.1
returns 400 deactivated), and none appear in `inference.baseten.co/v1/models`.
- Set `deprecation_date` on those entries (kept for spend tracking).
- Skip entries past their `deprecation_date` when building `baseten_models`,
so `baseten/*` expansion only lists models that still serve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(baseten): assert the custom api_base model listing on the returned models
The custom api_base test only inspected the patched client's call args, which trips the
TQ002 mock-echo gate. Serve the custom URL with respx and assert on the models that come
back instead, so a wrong host or a broken parse both fail.
* fix(baseten): drop newly retired models on cost map reload and fake discovery at the HTTP boundary
A cost map refresh only ever added to the Baseten set, so a model that gained a past
deprecation_date after boot stayed listed under baseten/* until restart. Discard it on reload.
Tests now serve the catalog with respx instead of patching our own HTTP client, the
isinstance registration test is gone since the get_valid_models test already proves
discovery end to end, and new test locals carry Final.
---------
Co-authored-by: AlexKer <AlexKer@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
get_azure_ad_token() built a new DefaultAzureCredential for every request when enable_azure_ad_token_refresh is on, so each Azure Responses call fetched a fresh Entra token synchronously on the event loop. Use the per-scope cached provider the SDK-client path already uses, so the credential's own token cache does its job. Fixes#46419
Azure retired the preview realtime surface, so /openai/realtime/client_secrets
and /openai/realtime/calls with an api-version return 404. Both now target
/openai/v1/realtime and the SDP exchange sends the ephemeral key as a Bearer
token, matching the Azure GA WebRTC contract.
Fixes#24659
Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com>
* fix(vertex_ai): price streamed pass-through calls with the regional uplift
The streamed Vertex pass-through handler builds the response with
stream_chunk_builder, which stamps a cost into _hidden_params without the
vertex_location. The handler then prices the response again with the
location from the URL, but only into kwargs, and the logger prefers the
stamped cost. Streamed calls on a regional endpoint were billed at the
global rate.
Stamp the location-aware cost on the response in the shared
generateContent payload builder so the logger records it.
Fixes#44755
* style(vertex_ai): keep the response_cost assignment within the 120-char limit
* refactor(vertex_ai): set the passthrough response cost with set_hidden_param
Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this.
* test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed
* fix(proxy): price streamed Vertex pass-through spend like the non-streamed call
Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls
* fix(proxy): price custom-priced Vertex pass-through deployments at their own rates
The stamped response_cost from the generateContent logging helper is returned before the
logger checks custom pricing, so a deployment with custom input/output rates was billed at
the cost-map price. The helper now passes custom_pricing and the deployment id into
completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls,
and deployments without custom pricing keep the regional uplift.
* fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price
* fix(proxy): give the new pyright ignore a reason for the LIT004 gate
* test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included
* fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same
* fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini
A top-level base_model in a pass-through body now reaches the logging
params only for a key or team with allow_client_pricing_override, so a
plain key cannot pick a cheaper model's price once the logger prices
these calls. The test that pinned every litellm-owned body name into the
logging params now leaves base_model to its own key-gated test.
The Vertex handler now classifies a non-Google upstream host as Vertex
only for Vertex resource paths, so a /gemini stream through a
GEMINI_API_BASE gateway is priced as Gemini without the regional uplift.
Adds the audit's integration spend test for Claude on Vertex, /v1beta,
the named custom-priced /gemini deployment, and a cut OpenAI stream.
* test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing
The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path.
* fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model
The Vertex pass-through handler picked Gemini pricing from the URL path,
so a configured pass-through to a Vertex gateway with a short path (no
/projects/ segment) was billed as Gemini, and its Claude streams at $0.
The provider now comes from the route: a call through /gemini or to
generativelanguage.googleapis.com prices as Gemini, and everything else
as Vertex, the same rule main used. Matching the GEMINI_API_BASE host
was rejected because one gateway can serve both APIs, and the path rule
was rejected because short gateway paths look like Gemini paths.
A non-string base_model (a number, list or object) made the cost
calculation fail and bill $0 once the stamped stream costs were gone.
get_base_model_from_metadata now ignores it and falls back to the
deployment's base_model, so the call bills the list price. A 400 was the
other option, but the logger runs after the upstream answered.
* test(integration): bill a non-string body base_model at the list price on every pass-through route
* fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info
A key allowed client pricing can send metadata.model_info in a pass-through
body. A non-string model_info.base_model made the cost lookup fail, and
once the stamped stream and Gemini costs were gone those calls billed $0
on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores
a base_model that is not a string, the same rule the root base_model got,
so the call bills the list price.
A model_info that is not an object (a string or a list) broke several
logging steps that read it as a mapping, so a non-streamed /gemini call
fired no success callback and no pass-through call wrote a spend row. The
pass-through route now leaves a non-object model_info out of the logging
metadata, so the call bills the list price. A 400 was the other option,
but it would turn calls that answer 200 today into errors, and guarding
every reader of model_info would spread the same check over many files.
* fix: reject a non-object model_info before the provider call again
The previous commit made the base_model lookup return None for a
model_info that is not an object. That lookup also runs during call
setup on every router route and SDK entrypoint, where such a value used
to fail before any provider request. With the early return the call went
upstream, the provider billed it, and the request then failed in a later
reader that still expects an object. On /v1/messages and /v1/responses
the key was also charged for a response the client never got.
The lookup again reads base_model from any truthy model_info, so a
non-object value fails during setup the way it did before this PR. A
non-string base_model inside an object still bills the list price, and
the pass-through route still drops a non-object client model_info at
intake, so pass-through calls keep billing the list price.
* fix: match the Gemini API host exactly when picking the pass-through cost provider
The provider check accepted any host ending in
generativelanguage.googleapis.com, so a Vertex host such as
vertex-generativelanguage.googleapis.com was priced as Gemini and lost
the regional uplift. It now accepts that host or a subdomain of it.
* test: pin that a subdomain of the Gemini API host keeps the Gemini price
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Native Responses now hands back the same managed ids as Python: the
resp_ id carries the provider and deployment, encrypted-content items get
encitem_ ids and wrapped content under affinity routing, and container ids
become cntr_ ids. Encoded encrypted-content items in the input are restored
before the provider request. The codec lives in inference-responses with
goldens taken from litellm/responses/utils.py
Each case runs one aresponses call through Python and through the native
route against the same recorded OpenAI response and compares the dispatch,
provider request, return value and success log. Known gaps are strict xfails
so the PRs that close them must flip the marker
* fix(streaming): map stream read timeouts to litellm.Timeout
* revert(streaming): move the timeout mapping to a later commit
* fix(streaming): map stream read timeouts to litellm.Timeout on chat, messages, and responses
* fix(streaming): keep stream fallback and cooldown unchanged for transport timeouts
A chat stream served through the Responses bridge now keeps mid-stream
fallback on a stall, because the fallback exemption also covers a
litellm.Timeout whose cause is an httpx transport timeout. The Responses
stream hands the raw transport error to failure callbacks again, so
router cooldown sees a stall as it did before. The 408 deployment info
now lists stream_timeout.
* test(integration): cover a stalled hosted_vllm stream on chat, Responses, and Messages with its spend log row
* fix(router): keep a fallback deployment whose stream stalled out of cooldown
* test(integration): cover bridged chat, fallback and cooldown for stalled hosted_vllm streams
* feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes
* fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting
The drop list now runs on the final upstream headers, after x-pass- expansion, through an
upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a
dropped value to Vertex. A malformed setting now answers 500 only for requests that carry
anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working.
* test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests
* fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces
* fix(vertex_ai): split drop-betas list entries on commas like the header
Three integration tests diffed the whole shared Redis keyspace (scan_iter or dbsize before and after), so
a sibling test's cache write could satisfy or break them. Each now finds its own entry by a value only it
wrote: the Gemini reasoning test by a per-run Gemini responseId, the max_messages test by a per-test
answer text, and the S3 cache-hit test by its request marker
The sweep auth tests created models through raw /model/new on the shared gateway and never deleted them.
They now create them through Scenario.model, which deletes each model on exit, including when setup fails
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans
* Preserve main's per-block streaming fixes and regression coverage
* fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs
Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails
* fix(bedrock): accept blank content on tool-call-only Converse turns
A vendor that returns "" or [] where an assistant row carried only tool
calls no longer fails the request closed. Also type the leaf lookup
without Any and restore the original layout of the existing unit tests.
* fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler
Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back.
The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names.
* fix(bedrock): keep new Converse helpers inside the type discipline rules
* test(bedrock): pin Converse row edits that change shape, role or placement
* fix(bedrock): type the Converse body once so its edits keep their reference
* fix(bedrock): list request-field texts where their Converse row sits
* fix(bedrock): send a reshaped Converse tool result as one text block
A vendor that compresses or summarises a tool result, such as Headroom
collapsing a JSON list into a table string, returned a 500 because the
row could not be split back into the original text and JSON parts. The
tool result now carries the vendor's string as one text block where its
parts were, keeping images and documents in place. It still fails
closed when the flat texts masked that tool result or the row changed
its tool call id.
* fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it
A guardrail that masks a Converse tool description or schema now has that
mask written into toolConfig, a tool it removes is dropped, and a tool it
adds, renames or reorders is refused before Bedrock. Headroom, Compresr and
TypeSafe get only the flat texts on Converse again, as on main, because the
pass-through route has no loop to answer their retrieve tool
* test(bedrock): refuse a guardrail that changes a Converse tool's kind
* fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts
OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS
and HiddenLayer scan only part of the rows or rebuild their returned texts from them,
so on Converse they get the flat texts only, as on main.
When a vendor returns changed rows, its texts are applied only if they keep the flat
layout and change only rows the vendor also changed, so row-shaped texts of the same
length no longer overwrite unrelated Converse fields.
* test(bedrock): pin how flat texts sit next to rewritten Converse rows
* fix(bedrock): keep the Bedrock guardrail on flat Converse texts
With experimental_use_latest_role_message_only, the Bedrock guardrail places
its mask by counting text parts per row, and Converse rows do not line up
with the flat texts, so the mask could land in tool call arguments. Give it
the flat texts only, as on main.
* fix(bedrock): keep Akto on flat Converse texts
Akto maps its masked payload back onto the flat texts by value, and a
Converse row joins or serialises tool call arguments and JSON tool results,
so a masked card there could not be placed and the request was blocked.
Give Akto the flat texts only, as on main, where the mask is applied.
* fix(bedrock): list request-field texts last again, as on main
Flat texts listed additionalModelRequestFields strings right after the
system prompt, so flat-text guardrails got a different order than on main.
Akto's live verdict on a benign request depends on that order and started
blocking it. Rows already keep their own order and texts that do not follow
rows are dropped, so the flat texts go back to main's order.
* fix(bedrock): list request-field texts next to their row for row readers
Row readers that rebuild texts from rows could swap a masked request field
with a masked user turn when the field strings came last. Row readers now get
those strings right after the system prompt, where their row sits. Flat-text
guardrails keep the end-of-list order.
* fix(bedrock): list nested request-field texts in their row's order for row readers
Row readers that rebuild texts from rows could write a masked request field
onto a sibling key when the field held nested objects, because the texts walked
the fields depth first and the row breadth first. Row readers now get those
strings in the row's own order.
* fix(bedrock): walk request fields breadth first instead of resolving each path
Resolving every request-field string by its path revalidated the parent
object per string, so a wide additionalModelRequestFields took seconds. The
string walk now goes breadth first for row readers, the same order the row
lists them, in one pass. A unit test pins that texts and rows map the same
way when the vendor masks them differently.
* fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry
When a vendor returns changed rows whose texts do not line up with them, the
rows win. A text the vendor masked on a field its rows left untouched (a tool
call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead,
unless the edited body holds the changed text elsewhere, as texts rebuilt from
the rows do.
* fix(bedrock): read same-length texts by position when rows drop a mask
Matching a dropped text against any string of the edited body let a row
mask on another field with the same token cancel it, so the raw tool call
argument still went out. Texts of the length that was sent are now checked
position by position, and only texts of another length, which cannot be
placed, are matched by value.
* test(bedrock): count each rebuilt text against the rows' edits
* fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed
PromptGuard rebuilds its returned texts from the user rows, and a vendor that
does so while masking only row content let a card in a tool call argument
reach Bedrock, which main masked. PromptGuard now gets the flat texts as on
main, and a reply whose rows changed next to texts of another length fails
closed, since what those texts mask cannot be checked.
* test(bedrock): let the structured checks decide the unsafe tool call and missing row cases
The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for
texts that cannot be placed answered first and the tool call and row count checks went unexercised.
* fix(bedrock): send request field strings to row readers after the conversation, where main puts them
Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user
message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last,
as main sends those strings in texts
* test(bedrock): pin nested request field order for flat readers and the texts view
Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader
getting the row order or a texts view written in main's order fails
* refactor(bedrock): ask each guardrail whether Converse should hand it rows
Converse picked the guardrails that get only the flat texts from a list of
provider names. CustomGuardrail now has supports_structured_messages,
True by default, and the ten classes on that list override it to return
False, so the handler asks the guardrail instead of the list. Every
guardrail class gets the same answer as before
* fix(bedrock): keep Singulr on flat Converse texts as on main
Singulr wraps each flat text as a user message and its backend judges only
the latest user message, so on main a Converse tool result was the turn it
judged. Handed rows, the tool result arrives with role tool and the latest
user row is the original question, so an injection in a tool result reached
Bedrock. Singulr now returns False from supports_structured_messages
* fix(bedrock): keep Qualifire on flat Converse texts as on main
Qualifire builds its evaluation input from the last user row only when that
row's content is a string, so Converse rows (content-block lists) gave an
evaluation_id policy an empty input. Opt it out of structured messages so it
gets main's flat texts again.
* fix(bedrock): keep Conduct on flat Converse texts as on main
The conduct-litellm-guard plugin scans only user rows, so with Converse rows a
system prompt, assistant text or tool result left its scan. Opt it out of
structured messages so it gets main's flat texts again, all sent as user turns.
* fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects
A guardrail that removed every tool, or the tool toolChoice names, produced an
empty tools list or a dangling toolChoice that Bedrock answers with an opaque
ValidationException. Refuse those rewrites before Bedrock, like other tool
changes this route cannot carry.
* fix(bedrock): list request-field strings in main's order for every guardrail
The request-field row walked additionalModelRequestFields breadth first, and
guardrails that get rows had their texts follow that order. With two or more
strings the last text changed, so Ovalix, which blocks only when the last text
is flagged, rewrote an injection main blocked. The row now walks the fields in
the order main lists them in the texts, and the texts use main's order for
every guardrail.
* test(bedrock): pin a container's own request-field strings before its nested ones
* test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools
* fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat
A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock,
converted the way chat completions converts tools, with cache points kept behind their tool.
When every tool is removed, toolConfig is dropped and earlier tool calls and results become text,
as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed.
The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer
hits the recursion limit.
* test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly
* fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply
* fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail
on_error: allow sent the original request on when the endpoint refused a
guardrail rewrite it could not apply, so a Converse request the guardrail
masked reached Bedrock raw. The refusal now counts as a fail on every endpoint.
* fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say
A rewrite the endpoint cannot apply is neither a content verdict nor a
technical error, so on_fail next or allow and on_error allow must not send
the original request on
* test(bedrock): trim the Converse guardrail tests
Drops the chaos and per-vendor integration cells and the duplicate unit cases,
and leaves the existing handler tests as they are on main
* fix(bedrock): put Converse request-field strings in the latest user message for row readers
A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed
after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let
an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their
own only when no user message has text.
* fix(bedrock): create toolConfig when a guardrail adds the first Converse tool
A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main.
* test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail
Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows
* fix(guardrails): keep Converse tool matching and write-back linear in request width
Matching each returned tool rescanned the whole tool list, and every write-back
step revalidated the list or object it passed through, so a request with
thousands of tools or request fields held the event loop for seconds. Returned
tools are now looked up by name in one index, and path steps read the child
directly.
* fix(guardrails): let later Converse guardrails scan guardContent text
When a guardrail removes every tool on a Converse request that carries
guardrailConfig, tool results are rewritten as guardContent blocks. The
flat text extractor read only top-level text, so a later masker in the
same request never saw them and a card in a tool result reached Bedrock
unmasked. Chat completions and /v1/messages keep tool results visible
to later guardrails after the tools are removed.
The extractor now reads guardContent text the same way the structured
rows already do, so flat-text guardrails also scan guardContent text a
caller sends.
* fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text
Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after
the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer,
Qualifire) check what the caller wrote
When a guardrail removes every tool, earlier tool calls and results are now written as plain text
blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own
guardrail keeps checking every message, since a single guardContent block makes it skip all plain text
* fix(guardrails): split a newline-joined Converse message back into its parts
A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline.
* fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns
When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails
after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped
and the history turned into text only when the request is signed, as chat completions does at request
build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400
Request-field strings join the latest user message that has text when a tool-result-only turn follows
it, so a guardrail judging only that message reads the caller's text with them
A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500
Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes
nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* fix(databricks): use the native Anthropic route for Claude /v1/messages
Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge
* fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas
A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again
* fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent
* fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials
Unity Catalog model-service names now reach the AI Gateway's native Anthropic
route instead of a serving-endpoints path that answers 404 for them, and the
native route resolves the key and base through the same order the chat route
uses, so environment-only deployments keep working.
The native route is limited to Databricks foundation-model Claude names, an
unlisted provider forwards safeguards and the beta as sent, and the ten betas
the workspace rejects are marked unsupported for Databricks in the beta map
and its Rust mirror.
* test(rust): expect Databricks to drop the advanced-tool-use beta
* fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback
Only the OpenAI-compatible JSON providers drop safeguards now, so a
deployment opted into /v1/messages passthrough forwards the field and
the beta as sent. When a route drops safeguards and the proxy has a
classifier model, the proxy answers the verdict itself, as the chat
completions bridge did. The Databricks Messages route also reads the
databricks_user_agent param like the chat route.
* fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL
* fix(databricks): drop container on the native Messages route, as the chat completions bridge did
* fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map
Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name
* test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire
* test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas
* fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth
Databricks chat completions, called directly or through the Messages bridge,
no longer receive the client's anthropic-beta and anthropic-version, as at the
merge base. The native Messages route still forwards them, filtered by the
beta map, without counting them as caller credentials, so a deployment with no
api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to
the native route only for Unity Catalog names, matching the chat route.
* test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route
* fix(databricks): keep client anthropic headers off an opted-in passthrough api_base
A databricks Claude deployment whose api_base is no workspace route and that opts
into /v1/messages with supported_endpoints falls back to the OpenAI-like
passthrough. That config does not filter betas through the beta map, yet it
received the native-only Databricks header entry, so Claude Code's
prompt-caching-scope beta reached the upstream raw and got a 400 where the merge
base answered 200.
Each Messages config now says whether it is the provider's native route, the
OpenAI-like passthrough says no, and the handler passes that answer when it
reads provider-specific headers, so native-only entries reach only native configs.
* test(databricks): cover client anthropic betas on an opted-in passthrough api_base
A databricks Claude deployment whose api_base is no workspace route and whose
model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like
passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m
betas through the proxy and asserts the caller gets the upstream's message and
the gateway receives no anthropic-beta, as on the merge base. The scripted
gateway answers 400 on the prompt-caching-scope beta, as the real one does
* fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies
Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do.
* test(databricks): cover a loaded beta map that marks the safeguards beta unsupported
A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict.
* test(anthropic): annotate the MCP beta map recordings as Final
* fix(anthropic): drop safeguards on routes without the classifier only under drop_params
A route whose upstream does not answer safeguards (a JSON provider, or a map
that marks the dangerous-tool-use beta unsupported) now forwards the field and
the beta as the merge base did unless drop_params is set or the proxy runs its
own classifier, matching how litellm treats any unsupported param. Databricks
keeps dropping both, since its merge base bridge never forwarded the field.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
