* fix(watsonx): read passthrough credentials from use_in_pass_through deployments
/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(watsonx): keep env key precedence when no deployment is flagged
Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(streaming): fail on clean EOF without finish reason
* fix(streaming): cover shared OpenAI-compatible EOF paths
* test(router): complete mocked streaming responses
* fix(streaming): keep the EOF error message and cover Anthropic clean EOF
The request-less RemoteProtocolError tripped exception mapping into
'The .request property has not been set.' for non-OpenAI providers.
Adds Anthropic and generic-chunk unit tests, an integration test across
chat, the /v1/messages bridge and the /v1/responses bridge, and moves
main's token-estimate test onto a finished stream with a sibling that
checks the cut stream still bills the estimate on its failure log.
* fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole
Gemini marks every chunk unfinished and a later usage chunk reset
intermittent_finish_reason, so a finished Gemini stream failed at EOF.
LangGraph's __aiter__ opened a new aiter_lines on every wrapper call,
which hit StreamConsumed after the first chunk and now surfaced as a
false EOF error instead of a silent truncation.
* fix(streaming): keep the synthetic stop for a stream that sent no chunks
Gemini's OpenAI-compatible endpoint answers with only [DONE] when
reasoning uses up max_tokens. Failing every openai/azure stream that
sent nothing turned that 200 into a 500, while every other provider
already kept the stop for an empty stream. Adds integration cells for
Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked
deployment and the [DONE]-only answer.
* test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream
An empty stream writes no spend row on main either, so the [DONE]-only
cell checks only what the client sees.
* fix(streaming): read the chunk's finish reason for the sticky flag
* fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message
* test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names
* fix(agentcore): say why the end-of-stream flags rebind
* test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy
* test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream
* test(agentcore): inject the async transport instead of patching the aiohttp flag
* test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does
* test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server
* test(streaming): keep each stream's ending and land every spend row once through a spend log outage
* test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst
* test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop
* test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres
* test(streaming): type the new stream EOF unit tests
* test(streaming): a merged reasoning stream cut mid reasoning now fails
Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content
---------
Co-authored-by: King Star <mcxin.y@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): expect the kyma reported-cost headers to follow the cost map rate split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): keep the kyma test file at its original formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price the response-cost header split from the served model, not the request alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses
pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.
A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.
* fix(anthropic): let a chat client resend a paused turn so Claude continues it
* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message
Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.
* fix(responses): send the streamed refusal part's type and text in content_part.added
* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text
A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does
* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses
* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going
* fix(responses): mark a paused turn's message item completed
* fix(responses): mark every item of a paused turn completed
* fix(responses): keep a refusal's explanation when a refused turn is sent back
A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed
* fix(responses): scan refusal parts in Responses guardrails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits
A native Responses WebSocket turn logs its raw events instead of a usage-carrying
response object, so the v3 rate limiter read no usage and never charged the key
or tag token window. Read the turn's usage from its events the way the cost
calculator does, and charge the TPM-limited tags the socket was opened under.
* fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage
A later response.create turn on an open Responses WebSocket now gets a read-only
check of every TPM window it counts toward (key, team, user, org, end user, team
member, model-scoped limits and the socket's tags). A turn with no tokens left gets
the 429 rate_limit_exceeded frame and the socket stays open. The check never
reserves or increments a counter and leaves RPM, budget, PTU and project windows
to their own paths. It runs on native and managed sockets.
Managed turns carry the socket's call id in their metadata, so they charge the
socket's TPM-limited tags. A failed native turn charges the partial usage its
failure row already carries. A socket refused at open by a rate limit gets the
same refusal frame and closes 1008, logged once without a traceback.
/v1/realtime sessions now count toward TPM through the realtime usage combiner,
and the v1 limiter reads socket usage through the same helper
* fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn
A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with.
* fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero
* fix(proxy): stop charging a realtime session's whole usage into its close minute
Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case.
* test(proxy): check each per-turn TPM window on its own
* fix(proxy): send a fail-closed per-turn socket refusal with its 503 status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags
A Decisions deployment's model_info flags for audio and video now win over
the cost map, per media kind, the same way supports_box_questions already
does. An unlisted model flagged true gets its clips sent upstream, and a
false flag refuses them (or drops them under drop_params). OpenAI-format
providers still refuse clips since their request shape has no field for
them, and the SystemOne request builder no longer re-reads the cost map.
* fix(decisions): ignore media flags on a deployment whose own mode is not decisions
A chat-mode deployment's supports_audio_input describes chat input, so it no longer
turns on audio or video clips for Decisions. The cost map decides for it, as before.
Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params
deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info.
* fix(decisions): follow the cost map's mode for a flagged deployment without its own mode
A chat deployment flagged for audio or video usually leaves mode unset, so
its flags described chat input yet still let clips through on /v1/decisions.
When the deployment sets no mode, the cost map's mode for the model now
decides whether its media flags apply, and an unlisted model still takes
them. The audio and video lookups also run only when the request carries a
clip.
* fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map
The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations
* fix(decisions): route clips in a load-balanced group to a deployment that takes them
A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media.
* fix(decisions): keep deployments that drop or cannot resolve clips in the media filter
* fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering
Running the media filter before cooldown turned the 400 a group answered
when its only clip-taking deployment was cooling down into a 429. It now
runs next to the image filter, just before order filtering, in both the
async and sync selection paths.
* fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments
* test(router): list the router default params helper in the router coverage check
* test(bedrock): stop reloading the cost map module in the GovCloud fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes
* fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes
The whole-object forward sent keys Anthropic refuses and reached other
providers, so a request that used to pass could start failing with a 400.
Read the client's user_id after hooks, drop it unless it is a string with no
email or phone number, and send only that key when the upstream URL path ends
in /v1/messages. Share the validator with the translated route, whose email
check now matches any embedded address as Anthropic's does.
* fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans
Limit the pass-through user_id to Messages create on api.anthropic.com or the
configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep
the field off any request a pre-call guardrail inspects. The translated route
goes back to its own user_id filter.
* fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases
* refactor(proxy): pass the user_id check a plain mapping instead of an optional body
* fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped
Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill Claude web search per request on deployments priced in model_info
A vertex_ai deployment priced in its model_info is registered under its hashed id, so the
web search cost check on the model key never saw "claude" and billed Claude searches with the
Gemini grounding calculator: one search per call, or Gemini's $0.035 default with no search
price. The check now also reads the model the response names.
* fix(vertex_ai): bill token-priced Claude deployments at the served model's search price
A Vertex Claude deployment priced in model_info without a search price now bills each web search at the served model's cost map price, read at cost time and never written into the deployment entry. An explicit search price, zero included, still wins.
* fix(vertex_ai): price web search on the Vertex Claude Sonnet 4.5 and Haiku 4.5 rows
Google bills Claude web search at $10 per 1,000 searches on these models, so token-priced deployments of them no longer bill searches at 0
* fix(cost): bill served-model search price on token-only Anthropic and Azure AI Claude deployments
Token-only Claude deployments on anthropic and azure_ai billed web search at 0 because the deployment entry has no search price. They now fall back to the served model's cost map search price, as vertex_ai already does. Vertex routing reads the served model first, so a Gemini deployment whose id contains claude keeps per-prompt grounding
* fix(proxy): price streamed usage frames under the deployment's model so usage.cost matches the spend row
* test(proxy): give the Vertex streamRawPredict logging doubles the request's model, as a real pass-through Logging has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover text verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity as chat verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): type the bridged text verbosity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity validation on the codex tool search follow-up turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text.verbosity on previous_response_id client tool_search follow-up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp): hand out bytes buffered before a dropped connection so a cut stream bills its partial usage
test(integration): stop the CyberArk override test failing when a short config value lands inside ciphertext
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(aiohttp): cover bytes already read before a dropped connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* refactor(proxy): inject database repositories through per-repository dependencies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): model Prisma client cleanup
* test(rust): include NativeDatabase in bridge API surface
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): give database shutdown one lifespan owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): log database cleanup errors and bound shutdown waits
Database.aclose now logs a failure in either cleanup step instead of raising
out of the lifespan exit, which matches how main handled stop errors and keeps
setup_prisma_client from raising out of cleanup after a failed connect
Shutdown waits for an in-flight reconnect for a fixed five seconds instead of
the reconnect timeout plus grace, and joins engine wait threads with a fixed
limit instead of indefinitely. The threads are daemon threads, so a thread
still alive after the limit cannot keep the process from exiting
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set
* fix(sso): keep the upserted team_id_jwt_field team out of result.team_ids so restricted_sso_group and the access-token group fallback are unchanged
* fix(sso): only upsert the team_id_jwt_field team on UI SSO login when enable_jwt_auth is on
The API JWT path only reaches team_id_upsert inside the enable_jwt_auth branch, so a deployment
with a leftover litellm_jwtauth block and JWT auth off never created teams. Gate the UI SSO upsert
the same way
* fix(sso): read enable_jwt_auth through the proxy_server module so the basedpyright gate stays at its ceiling
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): apply key and team router_settings to /v1/images/generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format router settings helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve key model_group_alias once on image generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): persist sent model_info fields on POST /model/update
POST /model/update rebuilt litellm_params from the request but dropped the
request's model_info, so changing base_model (or mode, tier, etc.) on an
existing DB model was silently ignored. Clients that only use this endpoint,
like the Terraform provider, could never fix a deployment's base_model
without deleting and recreating it.
Persist the model_info fields the caller actually sent, on top of the stored
ones. ModelInfo defaults never overwrite stored values, server-owned, team,
PTU and pricing fields stay on their dedicated write paths, and values echoed
from a /model/info response are not pinned.
* fix(proxy): keep echoed and blank model_info values out of POST /model/update writes
Blank strings from typed clients no longer overwrite stored values. Values echoed from
/model/info (the stored base model's cost map entry and discovered limits) are not
pinned, and an echoed catalog value resets a differing stored override like PATCH does.
direct_access and access_via_team_ids stay display-only. The tests stop patching the
auth check class and give the team lookup a real row instead.
* fix(proxy): lock the model row and skip unchanged model_info on POST /model/update
* fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved
A typed model_info value that matched the cost map, or the previous base model's entry,
was dropped and could erase a stored override. Echo detection now follows PATCH, only a
body carrying the /model/info key counts, compared against the base model the row has
after the write. The locked write patches the stored JSON instead of dumping the hydrated
Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the
real Router with discovery over a mock upstream
* docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform
* test: expect POST /model/update to save the cache point flag, and wrap long test lines
test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info.
This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling
x01 asserts, the stored flag flips to true and cache points come back on.
* fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings
* fix(proxy): read discovered model info through the router registry entry point
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: bedrock claude prompt caching on /v1/messages reports and charges cache tokens
* ci: re-run after ci-coverage and assert-shard-coverage were cancelled by the GitHub Actions incident
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): evict a deleted key from every worker's auth cache
/key/delete dropped the key object only from the in-memory cache of the worker
that served the call. Auth serves key objects cache-first, so every other
worker, and the Redis copy when enable_redis_auth_cache is on, kept
authenticating the deleted key until user_api_key_cache_ttl expired. /key/update
already evicts and broadcasts through evict_and_broadcast, delete now does the
same for the tokens it deletes
* fix(proxy): evict only the deleted rows and broadcast them in one batch
Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker
* refactor(proxy): drop the comment above the delete eviction call
* fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait
* test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests
* test(proxy): wait for the peer to apply the broadcast instead of polling its cache
* test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer
* fix(key_management): evict every requested key on delete, including keys whose row is already gone
A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone
* test(key_management): split the gone-row eviction assertion into one comprehension per worker
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes
A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row.
* fix(spend): price non-finite provider costs on every route and drain parked NaN batches
Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON
* fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains
An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert
* fix(spend): keep non-finite costs out of headers, responses, and admin JSON
Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and
streamed frames, so admin writes and logging integrations keep their output.
The cost calculator records an overflowing total as 0 with a warning, cost
headers leave out non-finite values, a non-streaming NaN usage.cost is dropped
instead of failing the response, and a parked NaN spend log row reads back as 0.
* fix(spend): keep caller-set call ids from splitting the non-finite cost warning
The overflow warning strips newlines from litellm_call_id, and neither
non-finite cost warning prints the raw value, which said nothing beyond
"non-finite"
* fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request
* fix(types): drop a usage cost breakdown that holds a non-finite number
* fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost
* fix(spend): keep the dropped-cost flag out of reach of provider usage keys
A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): check and count deployment itpm and otpm on every Responses WebSocket turn
The success hook only reconciled an ITPM/OTPM reservation found in top-level
metadata or the standard logging object, but on the proxy the reservation lives
in litellm_params.metadata, so no reservation was ever settled there. The hook
now always calls the reconcile, which already returns early without a reservation
and clears what it settled, so a second success event is a no-op.
A Responses WebSocket turn after the first now has its deployment's current-minute
ITPM and OTPM counters checked and is refused with a 429 frame once a counter is at
its limit. An admitted turn carries a zero reservation on the current minute's
keys, so its actual tokens land on those counters on both the native and the
managed relay.
* fix(responses): settle the socket open's itpm/otpm reservation on the first turn only
After the first turn takes its logging copy, both websocket relays drop the
open's ITPM/OTPM reservation keys from the connection metadata, the same way
the budget reservation is detached. Later turns, unsolicited backend events,
turns after limits are removed mid-socket, and sockets with no turn admission
no longer settle or refund the open's reservation a second time.
* refactor(responses): read the itpm/otpm reservation keys from core metadata helpers instead of router_utils
* fix(responses): keep a socket open's hold for its first real turn and check tokens before request limits
A turn that will settle the socket open's itpm/otpm reservation no longer counts
that reservation against itself or swaps it for a zero one, so Codex's warmup
frame on a managed provider, or a refused first frame on the native socket, no
longer blocks the first real prompt or strands the hold for the minute.
The token check now runs before the rpm/rpd checks, so a turn refused for
tokens never uses up a request slot. Token-check tests pin the clock, and the
managed relay settle test drives a real streamed call through a custom provider.
* test(router): pin the router clock in the socket turn admission tests
Both tests seeded this minute and the next to dodge a minute boundary.
They now use the file's pinned-minute fixture and seed only that minute,
so neither depends on when the test runs.
* fix(router): check a responses socket's open without reserving and read shared token counters first
A Responses WebSocket open now runs the same itpm/otpm check a turn runs and
reserves nothing. It used to reserve an estimate that falls back to the
model's max output, so a socket could not open under any otpm below that.
The open still records a zero reservation for its own minute, so the first
frame, which the relay forwards without a turn check, counts its actual
tokens like every later turn.
This drops the plumbing that only carried the open's hold into the first
turn. The turn admission callable takes no arguments again, and the native
relay detaches only the budget reservation, as before this change.
The itpm/otpm turn check reads Redis first and falls back to this
instance's in-memory count only without Redis or when the circuit breaker
is open. These counters go down when a call settles, so an in-memory copy
at the limit could refuse turns after the shared count dropped. The tpm
check keeps its in-memory shortcut, since its counter only grows within a
minute.
Responses usage that reports input_tokens_details.cached_tokens now leaves
cached input out of itpm, as chat usage already did.
* fix(router): leave a socket turn's cached input out of itpm
The native Responses WebSocket relay logs a turn with its list of events,
so the deployment rate limit hook finds no usage on the response and falls
back to the standard logging payload. That fallback read cached input from
metadata.cache_read_input_tokens, a key the payload's cleaned metadata never
carries, so every cached token of a socket turn was counted against itpm.
The fallback now reads the cached count from the payload's usage_object,
the same usage the logging path already built for cost, so a native socket
turn counts only its uncached input. The managed relay and HTTP
/v1/responses already counted correctly through the response's own usage
* fix(router): mark only responses socket opens as check-only and count a turn on the minute it finishes
The check-only branch for a socket open keyed on a websocket value in the request kwargs, which any HTTP body can carry and which the realtime route also sets. The router now passes an opens_responses_socket flag into set_io_token_rate_limit_request_kwargs only for its _aresponses_websocket call, and the flag lives in the same context value as the request kwargs, so no request body can set it and realtime opens reserve and get refused as before
A zero reservation holds nothing to refund, so a socket turn now counts its real tokens on the minute the call finishes, read from the clock at settle time. A non-zero reservation still settles on the keys it reserved on so its refund hits the same counter
When the shared Redis read returns nothing, the check now falls back to this instance's in-memory count instead of admitting the turn with no count at all
* fix(router): keep io token reservation keys out of the metadata sent to providers
The itpm/otpm pre-call check wrote its reservation keys into both metadata and litellm_metadata. On the routes the router serves through its generic helper (Responses, Anthropic Messages, and the rest), metadata is the client's own field and goes to the provider as is. So _litellm_itpm_cache_key and _litellm_otpm_cache_key, which carry the deployment id and the model name, reached the provider, came back in the echoed response metadata, and counted toward the provider's 16-key cap. The managed Responses socket relay merged each turn's reservation into the frame's metadata the same way, and a native socket opened with metadata sent the keys on every turn through its request defaults
The router now passes the limiter the metadata field it uses for its own fields on that call (metadata for chat, litellm_metadata for the generic routes), and the reservation goes only there. metadata_channel is a required keyword, so a new caller has to name the field it writes. The managed relay puts each turn's reservation only on litellm_metadata
With the reservation only on litellm_metadata, a retried call on the generic routes refunded its first attempt twice and pushed the deployment's counters below zero. The failure hook read the reservation from the logging copy in litellm_params.metadata and cleared only that copy, so the router's litellm_metadata still held it and the retry refunded it again. The hooks now also read and clear litellm_params.litellm_metadata, which is the router's own dict
The proxy does not strip client keys that start with _litellm_ on the way in, so dropping that prefix at the provider boundary would also drop keys clients send today
The io token tests now run Redis in process with fakeredis instead of a TCP fake server
* test(responses): count socket turns through real logging callbacks instead of a Logging subclass
The native relay test now hands the rate limit check to a real Logging object as its success and failure callback, so the turn's tokens go through the same dispatch path production uses.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): bill both rounds of a streamed chat completions MCP auto-execute
A streamed /v1/chat/completions request with an MCP tool set to require_approval "never" runs two model calls on one logging object, and the final-stream dedupe kept only the first round's spend row. The final usage chunk also showed only the second round.
The MCP stream iterator now holds the first round's assembled response, prices it, and logs one success row that carries the usage and cost of both rounds, with the fixed margin charged once. The final usage chunk shows the same summed usage. A single-round stream still logs its one row when the answer finishes.
When the follow-up round fails, the first round is logged from a copy of the logging object taken before the follow-up opened, and the failure row carries only the follow-up's partial usage with no second fixed margin. With proxy post-call guardrails armed, only (assembled, cache_hit) tuples are parked for the proxy, and the deferred log is armed only while the iterator reads, so the proxy's orphaned-callback cleanup never sees it after the Router's first-chunk read
* fix(mcp): send the usage chunk of a streamed chat MCP request that ends in one round
When a streamed chat completions MCP request ends in its first round (the model answers without a tool call, or no tool results come back), the iterator drained the inner stream to fire its spend log and dropped the usage chunk that stream produced, so a client that asked for include_usage got no usage chunk at all.
The drained usage chunk is now held and sent to the client after the final content chunk and before [DONE], the same place a plain stream sends it, so the last usage chunk matches the billed row. A two-round request still drops the first round's usage chunk and sends the merged one from the follow-up, and a stream without include_usage still sends none
* fix(mcp): hold round one's log for the guardrail, bill the chunk cost once, and keep round one's id
When the follow-up call of a streamed chat MCP auto-execute fails, round one's
success log now goes through the proxy's deferred guardrail release instead of
straight to the handlers, so a post-call guardrail runs on it and the row
carries it. Without a proxy release it logs directly as before
The last usage chunk now takes its cost from the combined row's billed total,
so a fixed cost margin is counted once in the chunk, as it is on the row. The
follow-up's usage chunk is held until the follow-up ends to make that possible
The combined row now keeps round one's id, the stream's first chunk id, which
matches how a single-round stream is logged
* fix(mcp): bill every chat MCP stream round once on failure, hang-up, and sync drop
When the follow-up call fails, round 1's success row is now logged on a copy of the
logging object that does not share the request's guardrail results, so a post-call
guardrail that reports on the failed request counts once instead of on both rows
A client hang-up during the follow-up now bills round 1 plus the follow-up's partial
answer on one success row with round 1's id and the fixed margin counted once, the
same way a plain stream's hang-up bills its partial answer
The last usage chunk now carries round 1's usage and cost when only round 1 is billed,
and a synchronous reader that drops the stream mid-answer bills round 1 on the loop it
read with instead of discarding it
The deferred release's logged_on argument is typed and every release closure accepts
it, and the parked dispatch constants and protocols both MCP billing paths use now live
in one module. The unit tests wait on a logging signal instead of a fixed delay and
check the billed row through the proxy stream hook instead of private callback state
* test(mcp): drive the unassembled follow-up through an upstream audio delta that is not base64
The test replaced litellm.stream_chunk_builder with a stub that raises. Round 2 now streams an audio delta whose data is not valid base64, so the real builder raises, the stream parks only best-effort usage, and the client's last usage chunk must carry round 1's tokens and cost
* fix(mcp): count the fixed margin once in the combined stream row's usage cost
* fix(mcp): bill round 2's partial answer when a client hangs up on a routed stream
On the proxy the Router wraps the MCP stream, and when a client hangs up during
round 2 the router's own cleanup closes that stream before the proxy's disconnect
cleanup runs. The close logged round 1 alone, so round 2's partial answer was
never billed. The MCP stream now bills round 1 plus round 2's partial answer as
soon as the hang-up's cancellation reaches its read, before anything closes it
* fix(mcp): count the follow-up's own messages in round 2's partial prompt
The follow-up stream shares the request's logger, so it took the client's
original messages as its prompt. When round 2 ended early, from a hang-up or a
broken upstream, its partial usage counted only those messages and left out the
assistant tool call and the tool results the follow-up actually sent. The
follow-up stream now counts the messages the follow-up call sent
* fix(mcp): bill round 2's partial when a routed stream's client hangs up between reads
On a proxy the stream the disconnect cleanup holds is the router's fallback wrapper, so the
cleanup never found the MCP stream's in-flight billing and released round 1 alone when the client
hung up while the proxy sat at a yield. The round billing now registers its in-flight biller on
the request's logging object, next to the parked release, and the cleanup looks it up there, so
any wrapper chain bills round 1 plus round 2's partial once
* fix(mcp): send the MCP stream usage chunk under the billed row id and count the partial off the loop
A streamed chat completion that runs a LiteLLM-hosted MCP tool bills both model calls on one row under round 1's
chatcmpl id, but the usage chunk it sent the client was round 2's held chunk and kept round 2's id, so a caller could
not join the usage chunk to its row. The usage chunk now carries round 1's id in every path that sends it, with or
without a deferred post-call guardrail, while its usage and cost and every content chunk's id stay as they were
When the client hangs up during round 2, assembling round 2's partial answer counts the follow-up messages, tool
result included, with the local tokenizer. That ran on the event loop and stalled every other request on the worker
for about a second on a large tool result. The assembly now runs on a worker thread through asyncify from both the
reader's cancellation and the proxy's disconnect cleanup, shielded from cancellation, and round 1 is checked again
after the await so a release that happened meanwhile is never billed twice
* test(mcp): wait on the loop with a hang guard instead of a 300 ms deadline
* fix(mcp): give a proxy-rejected follow-up's failure row round 1's own cost breakdown
* refactor(streaming): fold the partial response's empty guards into one check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): name shared-handler errors by the status they answer
handle_exception_on_proxy pinned type=internal_server_error on every
exception it wrapped, so a management route rejecting a caller's own
argument answered {"type":"internal_server_error","code":"400"} and an
SDK branching on the type read its own bad request as a gateway outage.
Derive the type from the status the response actually carries, keeping
a type the exception already names.
* test(proxy): pin a 500 for exceptions whose status is None
* fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors
Provider 5xx and timeouts keep internal_server_error so retry loops still retry them,
the shared handler redacts internal hosts from raw exception messages, and /key/list
routes its HTTPException through the shared handler instead of its own copy of the
old label.
* fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures
* fix(proxy): pass shared-handler error messages through as on main
Redacting timeout and 5xx messages changed what every shared-handler route
returns, which relabeling the error type does not need, so messages go back
to the verbatim text main sends.
* fix(proxy): redact only the status-less failure main never sent
* fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403
A failure with no status now answers "Internal server error" instead of its own text, since that text can
name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The
/user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises
* fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise
A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces
* refactor(proxy): parse a three-digit string status without recursing
* fix(bedrock): route unmapped converse-only family model ids to converse
#42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from
falling to the invoke route. The same gap exists for the other families that
have no invoke transformation at all: minimax, nvidia, writer, zai, google and
xai. An id from one of them that is newer than the cost map hits
get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None ->
BedrockError, when Converse would have served it
Add a second family regex for those six and default them to converse, the same
way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because
the prompt-caching rule is openai-specific
Every shipped id of these families already resolves to a converse entry via its
base model (checked against the cost map), so this only changes behaviour for
ids that are not in the map yet
Tests: six unmapped ids route to converse, and an unmapped meta.* id still
routes to invoke since meta has an invoke handler
* fix(bedrock): route a model id with no Invoke provider to Converse
A bedrock/<model> id missing from the cost map fell to the Invoke route,
where an id that names no Invoke provider (a new vendor's model, a bare
provisioned-model, imported-model, or custom-model-deployment ARN) fails
locally with "Unknown provider=None" before any request is sent. Such ids
now go to Converse, which replaces the per-family regex with the general
rule. Ids that name an Invoke provider and explicit route prefixes are
unchanged.
* test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests
* fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id
* fix(bedrock): answer 404 for an invalid model id instead of AWS's 400
AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown
* fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list
* fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about
A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there
* fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows
The regex check now strips route and region prefixes from the model and its
base_model before asking whether LiteLLM knows the model, so
bedrock/converse/<id> gets the same answer in a direct transform call as on the
completion path. A cost-map row counts only when it names a provider, so a
price-only register_model row leaves an unlisted id unknown, as on the proxy.
AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping
an identical copy.
* fix(bedrock): treat a region path like the bare id when the cost map does not list it
A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools,
reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works
on it and a mistyped one still answers 404.
* fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local
* fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local
* fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider
An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse.
atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do.
* fix(text completion): keep /v1/completions error mapping as it is on main
Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(sso): bind CLI SSO and Claude Code gateway logins to the first team on the user record
* fix(proxy): keep SCIM and JWT team sync in record order, and look up SSO groups once
SCIM PATCH and JWT team sync saved a user's teams from a set, so the first
team on the record, the one a Claude Code gateway sign-in binds, could change
on any sync. Both now keep the record's order and append new teams in the
order the IdP sent them.
SSO group sync now drops claim groups that are not LiteLLM teams with one
lookup before adding the missing teams one at a time, so a claim full of
non-team groups no longer costs one failed add per group on every sign-in.
* test(sso): expect the gateway team dropdown in the user record's order
* fix(sso): read team existence from the writer, keep new-user and SCIM replace team order
* fix(scim): keep the teams a user already has in record order on PATCH and PUT
* fix(scim): keep record team order when a SCIM POST matches an existing email
* fix(scim): join new teams in request order so a failed roster write keeps it
* fix(scim): let a groups replace win over later group ops in a PATCH again
Applying group adds and removes after a replace made a PATCH that deactivates a user
and adds a team that does not exist answer 404 and leave the user active, where it
answered 200 before. Keep the record's team order, but ignore group ops after the last
replace as the base did.
* test(sso): type the CLI team detail row helper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: rj-cairn <349020452+rj-cairn@users.noreply.github.com>
* feat(usage): show subscription-covered providers and account fees separately
* fix(usage): write only the named fee columns and sync the root schema
The fee update now writes just the columns the request names, so two admins changing
different fields at once both land, and the dashboard edit form sends only what changed.
The root schema.prisma carries the new table like the two copies, and a billing period
anchored in the calendar's last month ends on December 31, 9999 instead of failing.
* fix(usage): register the subscription table span and keep provider logic under llms
* fix(subscription_accounts): register the fee routes as management routes and reject an empty account id
* fix(usage): run the new tests in CI and close subscription card review gaps
Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee
* fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card
* fix(usage): count a subscription fee only for accounts behind a current deployment
Usage listed a fee row for an account no deployment uses while another ChatGPT
deployment existed and dropped it when none did. It now lists only accounts behind a
current deployment in both cases. Adds a real proxy and Postgres integration test for
the fee API and for ChatGPT Responses rows on a deployment's own login, and names the
card's multi-component test with the integration suffix.
* fix(usage): keep a stored subscription fee listed after its account's deployments are removed
The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): apply published model rows on every worker before the debounced resync
Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path
* fix(proxy): load published model rows from the primary under the reconcile lock
Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s.
* test(proxy): cover a published model row that names no credential
* fix(proxy): keep the unknown model name read-through on the replica and outside the lock
Only the by-id load that follows a published write reads the primary inside
the reconcile lock. A request for a model name a worker does not know reads
through the replica routing before taking the lock again, as on main, so
unknown names put no load on the primary and do not wait behind a reload.
* fix(proxy): keep access group edits from dropping staying deployments and batch published config ids
* fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica
* fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing
A batch of published model ids that held one auto router skipped every row in it, so ordinary
rows written at the same moment waited for the full reload. The by-id load now leaves only the
auto router rows to that reload.
An access group update now reads every deployment from the primary, answers 404 for an unknown
group and 400 for an unknown model id or name before it writes anything, then removes the group
from the deployments that leave before it adds it to the ones that join, so a failed write leaves
the group narrower than asked, never wider.
* fix(proxy): write an access group update in one transaction after a fresh read
The update now checks model names first, reads every deployment from the
primary, and commits every join and leave in one batch right after that
read, so a failed write changes nothing and a field another admin set
during validation is kept. Other workers hear about the rows only after
the commit.
* fix(proxy): lock the model rows while an access group update reads and rewrites them
* test(proxy): run the model resync tests on the real installer and a real router
The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen.
* fix(proxy): lock only the rows an access group update touches and answer 404 before 400
An access group update locked every model row, so a lock on any unrelated
row stalled it for 30 s and then failed it with a 500, and unrelated model
edits on other proxies waited behind it. It now takes a per-group advisory
lock, locks only the rows that carry the group or that it selects, and
reads them again once locked. The group existence check runs before the
model name check again, so a missing group answers 404 as on main.
* fix(proxy): never wait on a locked row in an access group update
The update now takes its row locks with NOWAIT. When another edit holds one
of the group's rows, it rolls back, holding nothing, and tries again for about
10 s before answering 409. Holding rows while waiting deadlocked against team
deletes and let a concurrent model edit undo the update's change. The
per-group lock also moves to the two-key advisory space, apart from team ids.
* fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select
* fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep a deployment's own model_id over the request body's
A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one.
* fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat
Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win
over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists
model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a
client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component
* fix(router): keep a SageMaker deployment's component when it pins it through extra_headers
* fix(router): read the SageMaker component header as a pin only on SageMaker deployments
* fix(sagemaker): send the inference component only as a header, never as a body model_id
SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat
container body and legacy parameters no longer carry model_id. Nova endpoints accept a
fixed body field list, so a pinned Nova deployment could reject the extra field.
* fix(health): probe header-pinned SageMaker deployments at their component
Health checks bypass the router, so a legacy sagemaker/ deployment pinned only
through its inference component header sent no component on /health while routed
requests now send it. Also update the two SageMaker integration tests that still
asserted the old precedence and the dropped body field.
* fix(health): apply the router's model_id pin rule to health probes
Health probes now share the router's pin function, so a deployment that lets
clients set model_id gets no header pin and its own model_id wins without
reading the header. Header keys that are not strings are skipped when
reading the inference component header instead of failing the call.
* fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks
The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126)
The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests.
* test(proxy): pin provider metadata and key billing on /cursor/chat/completions
A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land
* fix(proxy): pick the Cursor route's metadata field from the body shape
A Responses-style body on /cursor/chat/completions keeps proxy metadata in
litellm_metadata, the same as /v1/responses, while a Chat Completions body
keeps metadata, the same as /v1/chat/completions. Adding the whole route to
the internal-metadata list let a client's chat-body tags replace the key's
tags in spend logs and skip tag budgets.
* fix(proxy): keep get_metadata_variable_name callable with only the request
* fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies
A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field
* fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies
* fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string
* fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch-interception): stream native server-tool blocks to clients
Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code)
never received the server_tool_use and web_search_tool_result blocks the
interception layer builds for them, so citation panels stayed empty and
result counts read zero. Two gaps in the same flow:
FakeAnthropicMessagesStreamIterator._create_content_block_chunks only
knew text, thinking, redacted_thinking and tool_use; any other block
type emitted a bare content_block_stop with no start and no payload,
silently dropping the block from the fake stream. server_tool_use now
streams as content_block_start plus input_json_delta and any other
server-generated block rides whole in content_block_start, matching
Anthropic's native streaming of these types.
try_short_circuit_search only recognized a native client by finding a
web_search_* tool in the tools list, but the pre-request and deployment
hooks convert that tool to the LiteLLM standard shape before the
short-circuit runs, recording the conversion in
WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so
the synthetic response carries the native blocks on the converted path
instead of only when hook ordering happens to leave the tool untouched.
Fixes#35333
* fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response
The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500
* fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply
* fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply
* test(websearch-interception): build the cost test's logging object with a datetime start time
* fix(websearch-interception): send search snippets once in the direct search answer
* fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks
Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta.
* fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite
A legacy hook that masked a tool call's arguments on a buffered stream made the whole
rewrite undeliverable, text included, so the original stream went out. With web search
blocks in the stream that now hit any hook masking the search query. The adapter hands
same-count tool-call rewrites to the translation's write-back, as a unified guardrail's
already are, and still discards a rewrite that adds or drops a tool call.
* fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream
The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled
ModelResponse, which keeps web_search_tool_result blocks only in provider-specific
fields, so a hook masking result titles on a stream never saw them. The hook's view now
carries those blocks, its rewrite of them goes back through the ended-stream write-back,
and the adapter no longer needs to pass texts outside the hook's view through untouched.
* test(guardrails): type the legacy hook stream test helpers
* fix(guardrails): scan a streamed web search query as text like the non-streaming reply
* test(guardrails): type the masking guardrail test helpers
* fix(guardrails): keep the block id when scanning a streamed web search query
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path
* fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints
* fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts
A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for.
* refactor(azure_ai): read text parts through a walrus and mark new test values Final
* fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts
* fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model
A router deployment without its own supports_prompt_cache_breakpoint flag now
reads the cost map as loaded, so a sibling's opt-in on the same model string no
longer turns its requests into 400s. The shared hook no longer reads router
registered keys for azure_ai.
* fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name
* test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): cap the default OTPM reservation for requests without max_tokens
* fix(router): reconcile ITPM and OTPM reservations after every successful call
* fix(router): read the default OTPM reservation from the served model and make its fraction configurable
The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check.
* fix(router): refund a failed attempt's io token reservation in make_call
The proxy passes one logging object to every retry, and it runs failure
callbacks only for the first attempt, so the last attempt's ITPM and
OTPM reservation stayed held until the minute ended. Each attempt that
raises now gives its reservation back before the retry or fallback
* fix(router): count Anthropic cache writes once in ITPM reconcile
Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens.
* fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction
* fix(router): reserve with the deployment cap only through the param the request's API sends
* fix(router): never let a deployment's output cap raise the reservation above the default
* test(router): move the reservation fraction env var test to the integration suite
The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation.
* test(router): retry the reservation fraction scenario when the minute rolls over
The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute.
* test(router): give the rollover retry its own deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced
* test(integration): a reported cost is attributed to output in the gateway parity cases
* fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects
* fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge
* fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost
* fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill
* fix(cost): read the stored breakdown total through the logging object's typed field
* refactor(cost): name the providers billed from their own usage fields for what they are
* fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
