* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): bill unitemized output as text when no images return
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): price image calls only by the tier the provider served
---------
Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle a fallback's tokens on the model group that served it
The v3 limiter pinned post-call TPM settlement to the model reserved at
admission, so when the router fell back to another model group the
requested model's counter took the usage and the serving model's counter
stayed empty. When the served model group differs from the admitted one,
charge the served group and refund the abandoned group's per-model
reservation. Same-model retries and alias reloads keep the admission target.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): wait for the primary refund and the fallback charge before reading counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): drop unrelated formatting from the v3 limiter unit test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order
A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create.
* fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments
* fix(router): keep a batch upload off chat deployments while its batch deployment cools down
* fix(router): keep a refused batch upload's retry off the chat deployment
* fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused
* test(router): pin the order fallback for a batch create on a group with no batch deployment
* test(integration): expect an Azure batch upload on the batch deployment alone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): forward message threads on native /v1/messages
Keep the thread body field on native Anthropic legs, add the
message-threads-2026-08-12 beta for it, reject it with a 400 on legs that
cannot continue a stored thread, and relay error.details of an upstream
Anthropic error so thread_not_found reaches SDK thread helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(anthropic): keep a stale thread 404 from cooling the deployment and type the thread beta ignores
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort the cooldown handler imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): dedupe the message-threads beta map rows and annotate test finals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): bind message threads to the key that created them and answer unsupported continues with thread_not_found
Every virtual key on a proxy shares one Anthropic credential, so forwarding thread alone let any key
continue another key's thread by its message id. A new default proxy hook binds each thread reply id
to its virtual key, or to the signed-in identity when there is no key, and answers a continue naming
an id it did not bind with the 404 thread_not_found the SDK replays on.
Legs without message threads now serve a create without the thread and answer a continue with that
404 before any upstream call, instead of answering it with no history.
* fix(proxy): keep a cached message thread reply with the key that got it first
A response cache hit replays the original reply id, and binding it again handed the thread to the second key, so the creator's continue got thread_not_found.
* fix(anthropic): keep message threads bound on every route and stable across turns
Check the thread binding on every route that carries thread except pass-through,
keep bindings in their own cache so they never evict rate-limit counters, add no
tool or message derived betas on thread turns, answer a create on a leg without
threads with a 400 thread_unsupported_request unless drop_params is set, and
forward threads on Claude Platform on AWS
* fix(anthropic): reject message threads on /v1/messages passthrough deployments
A deployment that forwards /v1/messages untranslated to an OpenAI-compatible
server inherited the native config's thread support, so `thread` reached a
server that keeps no thread state. It now answers the same 400
thread_unsupported_request on create and 404 thread_not_found on continue as
a translated leg
* fix(anthropic): resend a thread's create betas on continues and check bound ids on pass-through
A thread create now gets the betas LiteLLM derives from its tools and
messages again, so an advisor tool create that answered 400 on the previous
commit answers 200. Anthropic fingerprints the beta set, so the proxy keeps
the anthropic-beta header the create sent upstream in the thread binding and
adds it to every continue on /v1/messages. It reads that header from the
upstream request, since success logging overwrites the logged request
details before the success hook runs
A continue on the /anthropic pass-through route of a thread the proxy bound
to another key now gets the 404 thread_not_found. Ids the proxy never bound
still pass through, as before
* fix(anthropic): continue unbound threads off /v1/messages and keep every anthropic-beta on chat and Responses
* fix(anthropic): skip non-string anthropic-beta list entries instead of raising
* test(anthropic): keep the header merge regression test clear of main's appended tests
* fix(anthropic): add a thread's create betas to the client's own on continues and keep base header merging on chat and Responses
* fix(anthropic): answer an unbound thread id on chat and Responses with the 404 and drop the stale beta-join tests
* fix(proxy): check a thread continue sent inside extra_body against its owner
Azure AI promotes extra_body into the request body, so another key could continue a thread by putting thread there, past the ownership hook that read only the top-level field
* fix(proxy): write a thread's first owner atomically and bind creates sent inside extra_body
Two workers that both miss a reply's binding now agree on the first key stored in Redis, since the claim uses SET NX and each worker caches the stored winner. A create sent inside extra_body now binds its reply like a top-level create. The redundant Claude Platform supports_message_threads override is removed, since the provider already resolves to the native Anthropic beta list
* fix(proxy): include agent_id in the key-less message thread owner
* fix(proxy): tag message thread ownership Redis calls with their key family
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust_bridge): route every Rust-to-Python call through host modules
Rust now reaches Python only through litellm/rust_bridge/host/ (plus the
execution driver), and no Python module outside host/ imports it. Library
calls that reproduce Python behavior (httpx and OSError error text,
huggingface_hub downloads, the random generator, os.environ) go through
host functions too. The rule is written into host/AGENTS.md
The live litellm.model_cost table, its version and every Python writer move
to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only
the functions Rust calls. The cost map fetch passes use_http_settings instead
of handing Rust an http_settings() snapshot
* refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust
The Redis bridge asked Python for all of os.environ to read one key. CPython's
os.environ writes through to the process environment, so Rust reads it with
ProcessEnvironment like the rest of the module. Also drops doc comments that
restated their code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci(rust): raise the native extension size cap to 55 MB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): re-check and reserve per-model rate limits on fallback targets
A provider error fallback handed the caller the fallback model group without
its per-model TPM/RPM limits ever being checked. The router gets a
fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires
it to the v3 limiter, which reserves the target model's per-model descriptors
before each fallback hop and skips targets that are over their limit.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): dedupe a repeated fallback target before the reserving rate limit check
Both walks now consult the attempted set before the rate limit check, so a
chain like ["B", "B"] reserves B once. The deprecated settings parity test
also learns the new setting.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): a bidirectional fallback chain never re-reserves the requested model group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): match the surfaced failure in the fallback reservation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the admission stash untouched while building fallback descriptors
Building the fallback target's descriptors could rewrite the request stash's
PTU ceiling, which post-call settlement reads for the model the caller named.
The fallback check now hides the stash while it builds, like the frame check.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the streamed Responses fallback metadata as a typed Responses event
With expose_fallback_errors_to_caller on and include_fallback_errors set, a streamed /v1/responses
call that fell back sent its fallback metadata as a chat.completion.chunk with no type, so the OpenAI
SDK read it as an audio delta event with type None. On the Responses route the frame is now
{"type": "litellm.fallback_metadata", "litellm_fallback": {...}} and goes through the same Responses
framing as every other event, so it gets an event: line whenever its neighbours do. The chat route
keeps its chat chunk unchanged
* test(proxy): stop pinning the Responses stream terminator in the fallback metadata test
Main ends Responses streams without data: [DONE] since #47980, so the test reads the event frames whether or not a terminator follows them, and passes both before and after that change merges
* fix(proxy): keep the top-level model on the Responses fallback metadata event
* fix(proxy): keep the fallback metadata id on the Responses event and omit an empty model
* fix(proxy): leave the top-level model off the Responses fallback event when raw model names are on
The complexity router's return_raw_model_name leaves every other event unstamped, so the fallback event now matches. A new test streams a raw upstream frame before the event to cover how main names it.
* chore: push again after GitHub dropped the previous push event
* fix(proxy): send the Responses fallback event without an event line so older OpenAI SDKs read it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes
The /bedrock/model/{model}/invoke, invoke-with-response-stream, converse and
converse-stream routes built the pass-through call from the request body, so
x-pass- headers never reached Bedrock. The processor now hands the caller's
x-pass- headers to the pass-through route, keeps them out of the logged body
snapshot, and on Converse drops the invoke-only guardrail headers when the
deployment sets guardrailConfig and the request metadata header when the proxy
owns request metadata
* fix(bedrock): bill the served service tier on invoke pass-through and scope x-pass- forwarding to the Bedrock routes
* test(bedrock): assert forwarded x-pass- headers on the request Bedrock receives
* test(bedrock): gate the router-model x-pass- forwarding on the request Bedrock receives
* fix(bedrock): bill a caller's forwarded invoke guardrail and keep x-pass- headers out of guardrail vendor payloads
* fix(bedrock): bill a caller guardrail on every invoke model and replace a trace value Bedrock rejects
* fix(bedrock): keep guardrail traces out of chat responses, forward the trace header as sent, and drop credential x-pass- headers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(spend): report guardrail spend separately in daily activity
Model and key spend in /user/daily/activity/aggregated include the cost of
guardrails like Bedrock guardrails, and nothing in the response says how much.
A customer that bills tenants per key and per model could not get model cost
alone, so tenants that pay AWS for their own guardrails were charged for them
twice
Daily spend rows now carry a guardrail_spend column next to spend, and every
breakdown in the daily activity responses exposes it, so model cost is
spend - guardrail_spend. spend keeps its meaning, so budgets still count
guardrail cost as before
The value comes from the request's cost_breakdown.guardrail_cost, capped at
the row's spend because an allowance discount or a repeat job poll can charge
less than the guardrail cost the breakdown records. The delivery ledger
freezes it into the accounting payload next to accounting_savings, since the
stored payload does not keep cost_breakdown, and the legacy daily queue writes
the same value
The migration only adds the column with a 0 default, so rows written before it
read 0
* fix(spend): count guardrail spend on rows billed without a cost breakdown
A request blocked before the LLM call is billed for its guardrails straight from
guardrail_information, so it carries no cost_breakdown and its guardrail_spend
read 0. Both writers now price guardrail_spend from guardrail_information with
guardrail_information_cost, the same function success and failure rows use to add
guardrail cost to spend
Also fills guardrail_spend in the client usage view built from request logs,
qualifies the field description for rows recorded before it existed, and adds
the lock timeout the migration checks require
* fix(ui): avoid a nested ternary when reading guardrail entries from request logs
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): reject client-supplied Vertex location and project unless the admin opts in
A request body could carry vertex_location, vertex_project, or their
vertex_ai_* aliases and override the region and project pinned on a
Vertex AI deployment, sending the admin's credentials to a caller-chosen
project or region. Add the four fields to the banned request params so
they need allow_client_side_credentials or configurable_clientside_auth_params.
/vertex_ai/live and /rag/ingest name their Vertex project and location
by design and have no deployment pin to override, so they keep those two
fields while every other banned param is still rejected there
* fix(proxy): close the remaining Vertex target overrides and keep inert fields working
/vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in
A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL
Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target
* test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads
* fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest
* fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests
* test(proxy): give the batch tests' fake router the alias and routed-group lookups
* fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review
A Vertex opt-in now counts only when every group the request can reach opts in, router,
context window, and content policy fallbacks and one alias hop included, while a group
with no Google deployment never needs the fields. A model named in the route path is
checked next to the body model.
RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a
vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin
saved before the upgrade, and reads a corpus named by project number on the proxy's
own project.
/vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves
a setup frame sent as a binary message onto the session's target, and resolves the
region from default_vertex_config before the model path.
* fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one
With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project,
including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those
projects now connects, and any other project still closes with 1008 before a token is minted.
* fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest
A non-admin ingest into a store an admin created named by project number had its corpus moved onto
the proxy's own project before the admin-store exemption ran, so the upload went to a different
corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the
project-number rewrite and the target check.
* fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request
* fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list
* fix(proxy): check team public models before allowing target overrides
* fix(proxy): include team wildcard targets in opt-in checks
* fix(proxy): validate overrides against authenticated team routes
* fix(proxy): retain URL model during deferred credential checks
The registry lock fix (#50933) makes add_deployment and set_model_list wait
for a held delete on the same lock, taken inside HostedModelList as
self._lock. The helper only knew lines spelled with
self._deployment_index_lock, so it never saw those two writes reach the lock
and timed out after 10 seconds.
The second server in this case is a plain MCP SDK server registered as
oauth_delegate. The gateway's first catalog listing waits for that server's
OAuth metadata lookup, which never resolves and runs for the full
MCP_METADATA_TIMEOUT of 10 seconds. The MCP SDK client also gives
server/discover 10 seconds, so the case passed or failed on about a tenth
of a second of head start.
Setting LITELLM_MCP_METADATA_TIMEOUT to 5 for this case keeps the lookup
well inside the client limit. The case still asserts that subscription
changes are delivered and tools are refreshed across both servers.
* fix(proxy): count a prompt-only call_endpoint request with the provider on every counter
POST /utils/token_counter?call_endpoint=true with only a prompt sent the
prompt to the provider count API on Gemini and Vertex only. Anthropic,
OpenAI, Bedrock and Azure AI (Claude) deployments answered a local
estimate instead. Every counter now gets the prompt as one user message,
so a provider error surfaces the same as the messages form, and the
framed stand-in after a failed provider count frames the prompt as one
user turn, matching the messages form
* fix(proxy): count a call_endpoint prompt like a user message on providers without a count API
* fix(proxy): count a call_endpoint prompt as a turn only on chat deployments
A prompt-only /utils/token_counter request with call_endpoint=true now goes
to the count API as one user message only when the deployment serves chat
(mode chat, responses, or unknown). Embedding rows and models with no
deployment keep the raw local count.
The Anthropic, Azure AI, and OpenAI count calls now honor the deployment's
timeout, and a count error with an empty body names its status code.
* fix(proxy): fall back to router_settings.timeout for count calls and answer 408 on a count timeout
Count calls to Anthropic, Azure AI, and OpenAI now pick their timeout the way
a chat call to the same deployment does: the deployment's timeout, else
router_settings.timeout, else request_timeout. A count call that times out
answers 408 instead of a 500 processing error.
* fix(proxy): count a prompt as a turn only where a count API exists and give count calls chat's timeout order
* fix(proxy): read the count call timeout through the router settings snapshot
The token counter now asks llm_router.settings().timeout_for() for the chat timeout order instead of a new Router member, so the router surface does not grow
* fix(router): answer timeout_for on every router settings snapshot
The proxy reads the count call timeout through llm_router.settings().timeout_for(), but only the Rust core's snapshot had that method, so a plain RouterSettings (the unit tests' router stub) raised AttributeError. timeout_for now lives on RouterSettings and reads the core's _core_deployment_call_timeout, like the snapshot's other members
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(customers): reject unknown fields on customer create and update instead of dropping them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(customers): derive the budget hint from the budget request fields and fix the create example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep a same-model deployment's false box-question flag off the shared cost-map row
* fix(router): take a withdrawn supports_box_questions true off the shared row on update and delete
A deployment's true reached the shared backend row, but a later false no longer
overwrites it, so updating or deleting that deployment left box questions on for
its unflagged siblings until restart. The row now gets the catalog's value back
unless another served deployment of the model still sets true.
* fix(router): count every live router and set_model_list when withdrawing a shared box-questions true
* fix(router): match box-questions siblings by their cost-map row and withdraw on retire
A sibling spelled with its provider registers onto the same litellm.model_cost row, so the withdrawal now compares rows instead of backend keys. Retiring a router takes back the true its deployments put on rows no other live router still sets. Withdrawals are deduped per backend so a large set_model_list stays cheap.
* fix(router): take back a box-questions true an aborted set_model_list registered
A set_model_list that fails partway rolls back the deployment list, but a true the
aborted build already wrote to the shared cost-map row stayed there, so unflagged
same-model deployments kept sending box questions. The rollback now withdraws the
attempted entries too
A withdrawal now only rewrites rows that still hold true, so an entry the failed build
never reached gets no row written for it, even on a model the catalog does not list
The delete wire test now uses a decisions-mode catalog backend whose catalog lists no
box questions, since an unflagged deployment only reads the shared row for a
decisions-mode model
* fix(router): take back an env-named box-questions true a failed set_model_list registered
The rollback took flags back from the raw model_list entries, so a deployment whose model came from os.environ/ was looked up under the reference instead of the model it registered onto, and its true stayed on the shared row
It now takes them back from the entries the failed build reached, whose os.environ/ values are already read
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rate-limit): build model scope settlement targets from one owner tuple
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay Claude tool search on follow-up turns
Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them
A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400
The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors
* fix(responses): emit Claude's tool search output in stream order
The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives.
* fix(anthropic): keep the caller's tool names in tool search results
A tool name Anthropic cannot take gets rewritten on the request, so Claude's
tool_search_tool_result named the rewritten tool. The response now maps those
tool references back to the caller's names, streamed and not, and a replayed
search result maps them forward again on the next turn.
* fix(responses): replay Claude's tool search only on routes that run it
A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept
A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call
* fix(responses): keep a replayed search when the caller sent Anthropic's own search tool
A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it
* fix(responses): replay a BM25 tool search under its own name
* fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools
A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its
search arguments, a stale search drops only the tool references the follow-up no longer sends,
and the stored previous_response_id path keeps the base gate for routes the cost map does not flag
* fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool
A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had
* fix(responses): replay a stored Claude search only when the follow-up still sends a search tool
A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level.
handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors.
* fix(responses): leave out a stored Claude search kept in server_tool_calls
Main stores Claude's server-side tool search on the assistant message in
provider_specific_fields.server_tool_calls and tool_results, not in
tool_calls. The previous_response_id path only looked at tool_calls, so a
follow-up that sent no search tool still replayed the stored search.
The session handler now finds the search in either place and drops it from
both server_tool_calls and tool_results when the follow-up cannot run a
search.
* test(responses): give the replayed search guardrail fixture its own call id
The guardrail test reused toolu_01Weather, an id that streaming and bridge
tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran
first in the same worker, the guardrail write-back took its rebuild path and
the test failed for a reason unrelated to the replayed search.
* fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups
A search-only assistant turn now gets its search result renamed to the sanitized
tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on
Claude routes replay the search as base did, trimmed to the tools still sent, and
Bedrock invoke sends the tool search beta whenever the history holds a search
* test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool
* fix(anthropic): keep Anthropic's tool names in chat tool search results
A chat history whose search result used the request's tool names (weather.get) got
Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a
rollback. Chat completions now returns Claude's search result as main does, and the Responses
bridge maps a rewritten name back to the request's tool when it builds tool_search_output.
That also fills tool_search_output on Bedrock invoke streams, which carry no name map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id
The detail lookup matched request_id OR litellm_call_id, which Postgres runs as a full table scan when the litellm_call_id index is missing or invalid. Read by request_id, the primary key, first and by litellm_call_id only on a miss, both under the spend log statement timeout.
* test(proxy): type the call ID index test helpers
* fix(proxy): limit the Logs drawer call ID lookup to start_date and answer 400 for a malformed date
* fix(proxy): let spend log reads wait for a busy connection like a plain query
* test(proxy): let the session pages read double take max_wait
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill an OpenAI pass-through background response once across repeated reads
A finished background response now bills only on the first pass-through call
that sees it finished with usage. The claim is an insert on the new
LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds
across workers and replicas. Every later retrieve, cancel, streamed create, or
resumed stream logs like an unbilled read. With no database or a failed insert
the call still bills.
* fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse
* fix(proxy): claim background responses on failed streams and bound the claim
A resumed stream that fails after response.completed billed its usage through
the partial-usage path without claiming the response id, so a later read billed
it again. That path now claims the id and logs $0 when an earlier read paid.
The claim uses create_many(skip_duplicates=True) so a repeat read no longer
raises a unique violation in Postgres, and it waits at most 2 s before billing,
since the claim now runs before the client gets its response.
* fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids
* test(pass_through): cover a failed stream whose failure logging cannot be scheduled
* fix(proxy): start the background response settlement migration with a lock timeout
* fix(passthrough): leave a failed background stream's charge to its next read
A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The test stores its recording Redis client in litellm's shared client
cache with the default ttl. When earlier tests in the same process have
filled that cache with longer lived clients, the next insert evicts the
recording client, the pin is read through a fresh client, and the test
sees no commands on the wire.
* fix(proxy): let the OS assign Prisma engine ports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear Prisma connection state after failed startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer
Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525,
first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown
Codex versions still get the catalog cut to 1 MiB, since they drop a larger one.
* fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog
Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those
clients counted as too old for the uncut catalog, their custom reasoning levels and stock
entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does.
* test(proxy): type the Codex catalog byte-limit test
* refactor(proxy): gate the uncut Codex catalog on the 0.166 release line
Read only the major and minor numbers of client_version instead of a shared
semver comparison, and leave the reasoning level and stock entry version
checks as they are on main.
* refactor(proxy): read every Codex client_version with one helper
Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version
parser rejects. The catalog byte limit, the per-model minimum version and the
custom reasoning level checks now share one reader for Codex's format, which
also bounds digit counts so an absurd version can't fail the listing.
* fix: make scheduled spend resets idempotent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover reset roundoff with shared Redis workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate reset workers and type raw adapters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep updated_at moving on zero-spend budget resets
* fix: never leave spend below zero after a budget reset
* fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged
Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time
* fix(proxy): keep a resent budget reset time only once it is due
A save that resends the same budget_duration keeps the stored reset time
only when that time has already passed, so the scheduled sweep still
resets the period it is due for. A row that is not due yet realigns its
reset time on save, the same as before.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(transcription): treat libsndfile's unknown frame count as no duration
* refactor(transcription): inline the unknown frame count check
* fix(transcription): read the frame count once to keep the type gate flat
* test(transcription): annotate the new passthrough test's locals as Final
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: update the spend tests that still expected the pre-ledger accounting queues
#49872 made the SpendLogs ledger the default accounting path on purpose. A batch with no billable request now writes no cost row, a call the guardrail fails closed during a vendor outage keeps one spend row of either status while its key is still charged once, and charges left at a graceful shutdown are applied by the next start. The daily spend batch shutdown tests exercise the old queue, so they pin use_spend_logs_ledger to false.
* test(integration): price the guardrail relay model so the outage test can assert the charge
The relay model had no price, so every stream cost $0 and the outage test could not tell a charged call from an uncharged one. Both rigs now price it, and the test expects each served stream to cost 9 input and 4 output tokens and the key to be charged for all sixteen.
Also notes the skipped batch cost row case in the e2e rules file, next to the other explicit skips.
* test: repair four tests that fail for reasons outside the product
The OpenAPI docs gate test rebuilds the Rust extension with uv, and the integration lane blocks outbound traffic, so uv could not refresh its PyPI index for maturin. The test now runs uv offline against the cache the dependency install already filled.
The response cache test expected a chat reply blocked with empty content to be cached. Since #50057 a reply with no output is never cached, so that case now expects the repeat to reach the upstream and the real answer to be cached.
The Gemini cache hit e2e test gave the model 16 tokens, which its thinking can use up, and a reply with no text is not cached. It now allows 512 and checks the first reply has text.
The Together cache read e2e test picks the cheapest model with a cache read price. Since #50491 that is a -latest alias, whose target reports no cached tokens. Aliases are now left out of the pick.
* fix(proxy): give rag ingest managed file ids and team credentials on vector store file routes
RAG ingest through a team's OpenAI deployment now registers the uploaded file as a LiteLLM managed file and returns its managed id, so GET /v1/files lists it for the caller. Re-ingesting a managed id attaches the provider file and returns the caller's managed id
Model-less vector store file calls (create, list, retrieve, content, update, delete) from a team key now use the team's OpenAI deployment that ingest uploads through, via the router, and fall back to the direct path when the key may not use one. Explicit api_key, api_base, litellm_credential_name, and forwarded provider keys still win, and blank values count as absent
* fix(proxy): route managed vector store file ids to their team deployment and keep missing-file 404s off the router
Model-less team vector store file calls now go straight to the provider with the team deployment's
credentials, so 404s on missing files no longer put the team deployment in cooldown.
A managed file id on the vector store file routes now uses the deployment named by its
llm_output_file_model_id when that deployment is proxy-wide or belongs to the caller's team, and falls
back to target_model_names otherwise.
* fix(rag): keep managed ids and team checks on every file and search path
Ingest no longer registers a raw file id when file_url is empty, a managed
file id on a proxy without the managed files hook answers 400, and vector
store search and /v1/rag/query hits show the managed id ingest returned.
Model-less vector store file calls use the team deployment only after the
key, team, and member model checks pass, fall through to the router when the
team has no OpenAI deployment, and log the deployment they used.
* fix(rag): keep provider ids on lookup failures and mint managed ids only for keys that can use them
A database error while mapping search hits to managed ids keeps OpenAI's ids instead of failing a finished search. Ingest registers a managed id only when the caller passes the team deployment's model checks, so a model-limited key keeps the raw id it can still use, and the file routes share that check.
* fix(rag): attach the managed file copy that matches the ingest credentials
* fix(vector_store_files): route a managed file id to the caller team's copy
* fix(vector_store_files): keep the upload copy for a store registered with its own credentials
* fix(vector_store_files): use a managed file's team copy only when the store holds it
* fix(vector_store_files): attach a managed file's team copy and keep named credential options out of the store check
* fix(vector_store_files): attach the team copy only to a store that routes to the team deployment
* fix(vector_store_files): keep a managed store's routing and check the upload copy through its own deployment
* fix(vector_store_files): route a managed file to its upload deployment only when the key may call it
A managed file id names its upload deployment, and with require_managed_files off that id is not checked against an ownership record. The routes now use that deployment only when the caller may call its public model, else they keep target_model_names routing as on main.
* fix(vector_store_files): log a pinned managed file call under its deployment's public model name
* fix(vector_store_files): send a pinned managed file call's deployment key only to that deployment's base
* fix(vector_store_files): keep stored credential headers intact and fail the store check on errors other than 404
* fix(vector_store_files): answer a failed store check with the provider status and copy credentials read by name
* fix(vector_store_files): send a call to the upload copy when the store check fails without a 404
A store check error other than a 404 now leaves the call on the upload copy, the way main always routed it, so the real call keeps the router's retries, the provider's status, and its failure log instead of failing on one transient error
* fix(vector_store_files): keep a credential's no-log key out of the store check
A named credential carrying a no-log key made the store check call aretrieve with no-log twice, so every managed id call through that deployment answered 500 where main answered 200
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): log keys and teams still on the deprecated empty models defaults at boot
* fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup
* fix(proxy): say which empty-models rows keep access in 2.0.0
Teamless keys listing all-team-models get no models in 2.0.0, so the
boot warning no longer says every counted row keeps its access. It
names each upgrade outcome only when its kind is in use, says the counts
include blocked and expired rows, and words each kind by what the
setting means rather than what it grants
* fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning
The warning no longer says the upgrade keeps access for empty models
lists, since no shipped code or public page backs that yet. It says
2.0.0 removes the defaults and how to keep today's access, and keeps
the advice for teamless all-team-models keys. The integration test
drops an empty-database assertion that ran before the check finished
* fix(proxy): leave rows with access groups out of the empty models boot warning
* fix(proxy): count access group rows, skip management and read-only keys in the empty models warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add customer spend breakdown and filtering by current team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: recover legacy customer team spend and tolerate lookup outages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf: resolve customer team keys once per daily activity query
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: paginate customer team activity in SQL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust
* fix(model-cost): keep main's compat helpers pure and move network tests to integration
ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Google AI Studio answers generationConfig.candidate_count "" with a 400 before it generates anything, so the truncated 200 this case scripts is a reply Google never gives. The other malformed count and cap cases stay.
* fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates
* fix(anthropic-messages): apply additional_drop_params before the unsupported param checks
Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request
* fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body
* test(anthropic-messages): annotate the new test locals with Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): merge default metadata under the request's instead of replacing it
* fix(router): join default metadata tags after the request's tags
A router default_litellm_params.metadata.tags list replaced the request's tags, so the deployment and credential tags the router adds never reached the logged request tags. Default tags now follow the request's tags, skipping ones already present, like the proxy's key tag merge
* fix(router): keep masking a router default turns on
Merging default metadata under the request let a caller's mask_input or mask_output false switch off masking an admin set in default_litellm_params.metadata. A masking flag the default turns on now stays on, and a caller can still turn masking on. Also gives this PR's test suppressions a reason and adds a missing blank line
* fix(router): keep the router's cache namespace and drop malformed caller tags and dicts
A caller's metadata.redis_namespace no longer beats a router default, so it
cannot pick another tenant's cache partition. A caller tags value that is not a
list, or a non-dict value under a dict default, is treated as absent the way the
proxy's merge_tags treats malformed tags, so the router defaults still apply.
* test(router): check that mask_output stays on when the router default turns it on
* fix(router): keep a router default guardrail whatever guardrails the request lists
* fix(router): keep the admin's settings for a router default guardrail when the caller lists it
* test(router): check that a router default guardrail entry keeps the admin's settings
* fix(router): keep a router default private Langfuse trace private whatever the caller sends
* fix(router): keep a router default function such as langfuse_masking_function over the caller's value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import.
The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so.
The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again.
The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths.
The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys.
The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change.
* feat(proxy): let a key's logging entry reference a proxy credential
A virtual key's metadata.logging entry can name a proxy credential with
litellm_credential_name. At request time the credential's values, which config
credential_list resolves through the secret manager at boot, are merged under the
entry's callback_vars, with the entry's own values winning. An entry whose
credential is not defined is dropped, so it never falls back to the proxy's own
callback environment.
Only proxy admins may change the logging entries of a key when the stored or
requested list holds a reference (403), and a reference to an undefined
credential is a 400. The check runs on key generate, update, team bulk update,
regenerate and /user/new, and compares decrypted entries so re-saving metadata
read back from /key/info is not a change.
Resolves LIT-10061
* fix(proxy): drop a key logging entry whose referenced credential holds no logging values
* fix(proxy): only require newly added credential references to exist on key writes
* fix(proxy): log a dropped credential reference as an error so /key/health reports it
* fix(proxy): gate credential references on bulk-created user keys
* fix(proxy): keep failing requests on keys with non-object logging metadata
A key whose stored metadata.logging is not a list of objects failed every
request with a 500 before this branch. The credential reference rewrite
turned those values into no entries, so the request quietly succeeded
without the key's logging. Raise a TypeError for those shapes again so the
request path behaves as it did on main, while empty values and objects with
unknown fields still go through the existing skip.
* test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging
The malformed logging cell expects the base behavior that 7ac4ab1356
restored, a 500 for a key whose stored metadata.logging is not a list of
objects and a 200 for an empty list
* fix(proxy): keep a 500 for teams with non-object logging metadata
On main, a team whose stored metadata.logging held non-object items failed
chat requests with a 500, because AddTeamCallback(**item) raised a TypeError.
This branch switched to AddTeamCallback.model_validate, whose before-validator
raises AttributeError on those items, and the proxy maps AttributeError to a
400 invalid_request_error. Raise a TypeError for non-mapping items before
validation so the status matches main again, as the key path already does.
