* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions
An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects
* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries
An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01
A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses
* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop
Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.
/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.
* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it
Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.
Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.
* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun
The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.
* fix(tool-choice): refuse an allowed_tools entry whose name is not a string
An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.
* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls
* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call
* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks
* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path
Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing
An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left
The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice
* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call
A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11
* fix(allowed_tools): stop the Responses web search follow-up forcing another call
The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up
* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups
A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs
* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists
A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.
* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen
* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools
* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer
An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.
* fix(mcp): match a toolset label's tools by exact native name
A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
* feat(spend): report guardrail spend separately in daily activity
Model and key spend in /user/daily/activity/aggregated include the cost of
guardrails like Bedrock guardrails, and nothing in the response says how much.
A customer that bills tenants per key and per model could not get model cost
alone, so tenants that pay AWS for their own guardrails were charged for them
twice
Daily spend rows now carry a guardrail_spend column next to spend, and every
breakdown in the daily activity responses exposes it, so model cost is
spend - guardrail_spend. spend keeps its meaning, so budgets still count
guardrail cost as before
The value comes from the request's cost_breakdown.guardrail_cost, capped at
the row's spend because an allowance discount or a repeat job poll can charge
less than the guardrail cost the breakdown records. The delivery ledger
freezes it into the accounting payload next to accounting_savings, since the
stored payload does not keep cost_breakdown, and the legacy daily queue writes
the same value
The migration only adds the column with a 0 default, so rows written before it
read 0
* fix(spend): count guardrail spend on rows billed without a cost breakdown
A request blocked before the LLM call is billed for its guardrails straight from
guardrail_information, so it carries no cost_breakdown and its guardrail_spend
read 0. Both writers now price guardrail_spend from guardrail_information with
guardrail_information_cost, the same function success and failure rows use to add
guardrail cost to spend
Also fills guardrail_spend in the client usage view built from request logs,
qualifies the field description for rows recorded before it existed, and adds
the lock timeout the migration checks require
* fix(ui): avoid a nested ternary when reading guardrail entries from request logs
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router): sync responses() applies the Router's num_retries and fallbacks
Sync Router.responses() made one attempt and raised, while aresponses()
retried and fell back on the same Router. The sync plan now walks retries
and fallbacks like the async one, stamps the deployment's num_retries on
the error the same way, and the SDK no longer nests its own retries under
the Router's for sync generic calls
Fixes#49938
* fix(router): sync generate_content() retries and falls back through the Router
Reading litellm_metadata in the SDK retry check left sync generate_content
with no retries at all, since its Router plan made one attempt and the inner
completion() call stopped retrying. Sync generate_content and
generate_content_stream now walk the Router's retries and fallbacks like
their async versions, so retries are neither lost nor nested
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): reject client-supplied Vertex location and project unless the admin opts in
A request body could carry vertex_location, vertex_project, or their
vertex_ai_* aliases and override the region and project pinned on a
Vertex AI deployment, sending the admin's credentials to a caller-chosen
project or region. Add the four fields to the banned request params so
they need allow_client_side_credentials or configurable_clientside_auth_params.
/vertex_ai/live and /rag/ingest name their Vertex project and location
by design and have no deployment pin to override, so they keep those two
fields while every other banned param is still rejected there
* fix(proxy): close the remaining Vertex target overrides and keep inert fields working
/vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in
A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL
Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target
* test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads
* fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest
* fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests
* test(proxy): give the batch tests' fake router the alias and routed-group lookups
* fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review
A Vertex opt-in now counts only when every group the request can reach opts in, router,
context window, and content policy fallbacks and one alias hop included, while a group
with no Google deployment never needs the fields. A model named in the route path is
checked next to the body model.
RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a
vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin
saved before the upgrade, and reads a corpus named by project number on the proxy's
own project.
/vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves
a setup frame sent as a binary message onto the session's target, and resolves the
region from default_vertex_config before the model path.
* fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one
With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project,
including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those
projects now connects, and any other project still closes with 1008 before a token is minted.
* fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest
A non-admin ingest into a store an admin created named by project number had its corpus moved onto
the proxy's own project before the admin-store exemption ran, so the upload went to a different
corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the
project-number rewrite and the target check.
* fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request
* fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list
* fix(proxy): check team public models before allowing target overrides
* fix(proxy): include team wildcard targets in opt-in checks
* fix(proxy): validate overrides against authenticated team routes
* fix(proxy): retain URL model during deferred credential checks
Lowers Together Tev1 4B input price to the pricing page value, adds Mistral Leanstral vision and reasoning flags, and absorbs #50991, #50988 and #50958.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Kerry Lu <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com>
* feat(cost): bill batch cached image input tokens for OpenAI image models
* fix(cost): bill batch cached audio and image tokens without a text cache price
* fix(cost): honor modality cache prices in the half-price batch fallback
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(router): let an explicit routing tag outrank the encrypted content pin
A Responses follow-up that replays encrypted reasoning minted on one
deployment, but asks for another with a routing tag, failed with "Not
allowed to access model due to tags configuration". The affinity check
now runs the router's tag filter on the deployment it is about to pin,
and on encryption-boundary peers. When the filter would reject them, it
releases the pin and strips the reasoning the tagged deployment cannot
decrypt, so the request routes there. Requests the tag filter accepts
behave as before.
Fixes#50926
* fix(router): strip reasoning against only the boundary peers the tags keep
When tags rejected some encryption-boundary peers, reasoning was still
judged against every peer, so a blob only a rejected peer could decrypt
reached the tagged one. Pin and strip against the tag-kept peers.
* fix(router): mark the deployment TypeVar Final
* fix(router): keep every boundary peer for the router's tag check
Returning only the tag-kept peers hid tags the later tag check needs to
fail open, so a request that passed the affinity check was refused.
Return all boundary peers and judge the reasoning against the kept ones.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): let teams send allowed S3 buckets for batch files and read them back
A team's metadata can now list allowed_s3_buckets. Keys on that team may send one
of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on
POST /v1/batches, and the upload and batch-create handlers check the bucket against
the team list plus general_settings.allowed_client_side_s3_buckets. The early auth
body check still rejects bucket fields on every other route and method
Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri
through an internal stored_s3_bucket_name param, so content, retrieve, delete and the
batch cost job can read files written to a request-chosen bucket. Callers can never
send that param, and files in the model bucket resolve exactly as before
* test(proxy): assert outcomes in team bucket tests and type the managed content read
Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count
* fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back
A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix
* fix(proxy): keep the request's approved S3 bucket when merging deployment credentials
Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none
* fix(proxy): keep the request bucket only on upload and batch create, not on reads
The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for
* fix(proxy): only proxy admins can set a team's allowed_s3_buckets
Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes
The registry lock fix (#50933) makes add_deployment and set_model_list wait
for a held delete on the same lock, taken inside HostedModelList as
self._lock. The helper only knew lines spelled with
self._deployment_index_lock, so it never saw those two writes reach the lock
and timed out after 10 seconds.
The second server in this case is a plain MCP SDK server registered as
oauth_delegate. The gateway's first catalog listing waits for that server's
OAuth metadata lookup, which never resolves and runs for the full
MCP_METADATA_TIMEOUT of 10 seconds. The MCP SDK client also gives
server/discover 10 seconds, so the case passed or failed on about a tenth
of a second of head start.
Setting LITELLM_MCP_METADATA_TIMEOUT to 5 for this case keeps the lookup
well inside the client limit. The case still asserts that subscription
changes are delivered and tools are refreshed across both servers.
* fix(proxy): count a prompt-only call_endpoint request with the provider on every counter
POST /utils/token_counter?call_endpoint=true with only a prompt sent the
prompt to the provider count API on Gemini and Vertex only. Anthropic,
OpenAI, Bedrock and Azure AI (Claude) deployments answered a local
estimate instead. Every counter now gets the prompt as one user message,
so a provider error surfaces the same as the messages form, and the
framed stand-in after a failed provider count frames the prompt as one
user turn, matching the messages form
* fix(proxy): count a call_endpoint prompt like a user message on providers without a count API
* fix(proxy): count a call_endpoint prompt as a turn only on chat deployments
A prompt-only /utils/token_counter request with call_endpoint=true now goes
to the count API as one user message only when the deployment serves chat
(mode chat, responses, or unknown). Embedding rows and models with no
deployment keep the raw local count.
The Anthropic, Azure AI, and OpenAI count calls now honor the deployment's
timeout, and a count error with an empty body names its status code.
* fix(proxy): fall back to router_settings.timeout for count calls and answer 408 on a count timeout
Count calls to Anthropic, Azure AI, and OpenAI now pick their timeout the way
a chat call to the same deployment does: the deployment's timeout, else
router_settings.timeout, else request_timeout. A count call that times out
answers 408 instead of a 500 processing error.
* fix(proxy): count a prompt as a turn only where a count API exists and give count calls chat's timeout order
* fix(proxy): read the count call timeout through the router settings snapshot
The token counter now asks llm_router.settings().timeout_for() for the chat timeout order instead of a new Router member, so the router surface does not grow
* fix(router): answer timeout_for on every router settings snapshot
The proxy reads the count call timeout through llm_router.settings().timeout_for(), but only the Rust core's snapshot had that method, so a plain RouterSettings (the unit tests' router stub) raised AttributeError. timeout_for now lives on RouterSettings and reads the core's _core_deployment_call_timeout, like the snapshot's other members
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(customers): reject unknown fields on customer create and update instead of dropping them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(customers): derive the budget hint from the budget request fields and fix the create example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): Customer Portal (Beta) surfacing the existing customer endpoints and customer usage
Adds a Customer Portal mode to the dashboard (product switcher entry, Customers and
Chargebacks nav) built only on endpoints and data the proxy already serves.
Customers: list ranked by this month's spend from the customer daily-activity
aggregate, status (active, over budget, blocked), search, Create Customer through
/customer/new (id, alias, monthly max_budget, tpm_limit, rpm_limit), Edit alias and
budget through /customer/update, Block and Unblock. Customer detail shows the keys
that sent traffic as that customer this month from the api_key_breakdown of the
same aggregate, read only.
Chargebacks: per month, grouped by customer, customer and key, or customer and
model, with CSV export.
All strings go through i18n. Pure helpers (customerForms, customerPortalData,
chargebackRows) carry unit tests, and the ViewSwitcher integration test covers
entering and leaving the portal.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh the customers list after editing or blocking from customer detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): name the expected objects in the customer portal tests so the inline-object lint budget holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): mock next/navigation in the navbar test now that ViewSwitcher routes with useRouter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): format customer portal money and limits through the format keys instead of inline formatters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): note that customer blocking needs blocked_user_check and neutralise formula prefixes in the chargeback CSV
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): hold the registry lock across router-core writes and the mirror reads at their positions
A registry lookup returns positions in router-core's list and the host reads its mirror at them,
while an upsert wrote router-core and read the mirror back unlocked. A read on another thread in
between got an IndexError past the mirror, or after a removal the neighbouring model's entries
* test(router): cover the registry lock's read and write paths
Move the concurrent-read regression into the registry writes characterization, drop the
explanatory lock comments and the unused litellm-model lookup
* Add Claude Mythos 5.2 to Vertex AI cost map
Copies the vertex_ai/claude-fable-5-2 and @default rows as
vertex_ai/claude-mythos-5-2 and @default. Anthropic's Mythos 5.2 model
page lists Google Cloud as a platform and says it has the same
specifications and pricing as Fable 5.2, and Google publishes no Mythos
row yet, so the new rows cite Anthropic's page as their source.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Bill Mythos 5.2 on Vertex through the cost path in tests
Adds cost_per_token checks with cache reads, 5m and 1h cache writes and
the regional uplift so the Vertex Mythos 5.2 rows are proven to charge
what Fable 5.2 charges, and annotates the new test locals with Final.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Read the regional_endpoint_uplift_multiplier key in the Vertex Mythos uplift test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep a same-model deployment's false box-question flag off the shared cost-map row
* fix(router): take a withdrawn supports_box_questions true off the shared row on update and delete
A deployment's true reached the shared backend row, but a later false no longer
overwrites it, so updating or deleting that deployment left box questions on for
its unflagged siblings until restart. The row now gets the catalog's value back
unless another served deployment of the model still sets true.
* fix(router): count every live router and set_model_list when withdrawing a shared box-questions true
* fix(router): match box-questions siblings by their cost-map row and withdraw on retire
A sibling spelled with its provider registers onto the same litellm.model_cost row, so the withdrawal now compares rows instead of backend keys. Retiring a router takes back the true its deployments put on rows no other live router still sets. Withdrawals are deduped per backend so a large set_model_list stays cheap.
* fix(router): take back a box-questions true an aborted set_model_list registered
A set_model_list that fails partway rolls back the deployment list, but a true the
aborted build already wrote to the shared cost-map row stayed there, so unflagged
same-model deployments kept sending box questions. The rollback now withdraws the
attempted entries too
A withdrawal now only rewrites rows that still hold true, so an entry the failed build
never reached gets no row written for it, even on a model the catalog does not list
The delete wire test now uses a decisions-mode catalog backend whose catalog lists no
box questions, since an unflagged deployment only reads the shared row for a
decisions-mode model
* fix(router): take back an env-named box-questions true a failed set_model_list registered
The rollback took flags back from the raw model_list entries, so a deployment whose model came from os.environ/ was looked up under the reference instead of the model it registered onto, and its true stayed on the shared row
It now takes them back from the entries the failed build reached, whose os.environ/ values are already read
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rate-limit): build model scope settlement targets from one owner tuple
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay Claude tool search on follow-up turns
Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them
A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400
The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors
* fix(responses): emit Claude's tool search output in stream order
The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives.
* fix(anthropic): keep the caller's tool names in tool search results
A tool name Anthropic cannot take gets rewritten on the request, so Claude's
tool_search_tool_result named the rewritten tool. The response now maps those
tool references back to the caller's names, streamed and not, and a replayed
search result maps them forward again on the next turn.
* fix(responses): replay Claude's tool search only on routes that run it
A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept
A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call
* fix(responses): keep a replayed search when the caller sent Anthropic's own search tool
A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it
* fix(responses): replay a BM25 tool search under its own name
* fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools
A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its
search arguments, a stale search drops only the tool references the follow-up no longer sends,
and the stored previous_response_id path keeps the base gate for routes the cost map does not flag
* fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool
A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had
* fix(responses): replay a stored Claude search only when the follow-up still sends a search tool
A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level.
handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors.
* fix(responses): leave out a stored Claude search kept in server_tool_calls
Main stores Claude's server-side tool search on the assistant message in
provider_specific_fields.server_tool_calls and tool_results, not in
tool_calls. The previous_response_id path only looked at tool_calls, so a
follow-up that sent no search tool still replayed the stored search.
The session handler now finds the search in either place and drops it from
both server_tool_calls and tool_results when the follow-up cannot run a
search.
* test(responses): give the replayed search guardrail fixture its own call id
The guardrail test reused toolu_01Weather, an id that streaming and bridge
tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran
first in the same worker, the guardrail write-back took its rebuild path and
the test failed for a reason unrelated to the replayed search.
* fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups
A search-only assistant turn now gets its search result renamed to the sanitized
tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on
Claude routes replay the search as base did, trimmed to the tools still sent, and
Bedrock invoke sends the tool search beta whenever the history holds a search
* test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool
* fix(anthropic): keep Anthropic's tool names in chat tool search results
A chat history whose search result used the request's tool names (weather.get) got
Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a
rollback. Chat completions now returns Claude's search result as main does, and the Responses
bridge maps a rewritten name back to the request's tool when it builds tool_search_output.
That also fills tool_search_output on Bedrock invoke streams, which carry no name map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Gemini API and Vertex both include 5,000 grounding requests a month shared across
Gemini 3 and newer, so gemini/gemini-4-carbon, vertex_ai/gemini-4-carbon and
vertex_ai/gemini-4-carbon-cyber-permissive now carry the same search_free_requests
allowance as the Gemini 3.x grounding rows in both catalog copies.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id
The detail lookup matched request_id OR litellm_call_id, which Postgres runs as a full table scan when the litellm_call_id index is missing or invalid. Read by request_id, the primary key, first and by litellm_call_id only on a miss, both under the spend log statement timeout.
* test(proxy): type the call ID index test helpers
* fix(proxy): limit the Logs drawer call ID lookup to start_date and answer 400 for a malformed date
* fix(proxy): let spend log reads wait for a busy connection like a plain query
* test(proxy): let the session pages read double take max_wait
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): transform async chat messages once, whatever configs were built before
* fix(openai): keep tool-result PDF downloads off the event loop and stop Mistral rewriting retried messages
* fix(openai): pre-download PDF URLs into the sync transform so async requests match sync
Async chat requests now download every PDF URL in user and tool messages at once, then run the sync transform_request on a worker thread that reads those downloads. The request, filename, logged messages, and download count match a sync request, and a failed download raises the sync error with no second attempt. DeepSeek joins the configs that flatten content and skip the up-front download.
* fix(openai): skip up-front PDF downloads past a file part the transform rejects
* fix(openai): keep system-message PDF downloads off the event loop and stop downloads after a dead PDF URL
* fix(openai): let PDF downloads after a dead URL finish and retry rate-limited ones one at a time
Cancelling in-flight downloads after a dead PDF URL could abort a proxy worker on uvloop, so the downloads now run to the end in the background. A URL whose up-front download got a 429 downloads again inside the transform one at a time, as on main. A content part whose type is not a string is no longer read as a PDF part, so it reaches the provider instead of raising a TypeError.
* fix(openai): download a request's PDF URLs one at a time and stop at the first dead one
* fix(openai): prefetch the PDF URLs the sync transform downloads, after role mapping and up to the first unreadable part
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): respect use_redis_transaction_buffer when selecting the SpendLogs ledger
* revert: keep the SpendLogs ledger independent of use_redis_transaction_buffer
Measured on a live proxy, ledger on and ledger off grow RSS at the same rate per request,
so gating the ledger on the Redis buffer moved deployments off it without fixing the growth.
* fix(redis): free each request's Redis batch scope once the request ends
The native request batch keeps a copy of the context it is built in. It was built inside the
request scope, so that context pointed back at the scope, and the cycle collector cannot see
through the native object. Every request's batches, spend counter batches and native Redis
state stayed alive, growing worker RSS about 40 KiB per request and slowing each request as
the heap grew. Build the native batch in the context the scope was opened from instead.
* fix(proxy): bill an OpenAI pass-through background response once across repeated reads
A finished background response now bills only on the first pass-through call
that sees it finished with usage. The claim is an insert on the new
LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds
across workers and replicas. Every later retrieve, cancel, streamed create, or
resumed stream logs like an unbilled read. With no database or a failed insert
the call still bills.
* fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse
* fix(proxy): claim background responses on failed streams and bound the claim
A resumed stream that fails after response.completed billed its usage through
the partial-usage path without claiming the response id, so a later read billed
it again. That path now claims the id and logs $0 when an earlier read paid.
The claim uses create_many(skip_duplicates=True) so a repeat read no longer
raises a unique violation in Postgres, and it waits at most 2 s before billing,
since the claim now runs before the client gets its response.
* fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids
* test(pass_through): cover a failed stream whose failure logging cannot be scheduled
* fix(proxy): start the background response settlement migration with a lock timeout
* fix(passthrough): leave a failed background stream's charge to its next read
A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The test stores its recording Redis client in litellm's shared client
cache with the default ttl. When earlier tests in the same process have
filled that cache with longer lived clients, the next insert evicts the
recording client, the pin is read through a fresh client, and the test
sees no commands on the wire.
* fix(ui): keep the request log model cell to one model
A session row rendered every model the conversation ever used, comma
joined. A long conversation produced a huge string that the auto-layout
table turned into a very wide Model column, pushing the rest of the
columns off screen.
Render only the model of the row's own call (the newest call in the
session) and show an overflow count. The full model list stays in the
hover popup, including a note when the server capped the list.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(ui): use the shared Badge and an i18n key in the request log model cell
The "+N" overflow marker now uses the shadcn Badge instead of a hand-styled span, and the
"More models not shown" note reads from en.json so the dashboard's i18next/no-literal-string
lint rule passes.
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(proxy): let the OS assign Prisma engine ports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear Prisma connection state after failed startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer
Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525,
first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown
Codex versions still get the catalog cut to 1 MiB, since they drop a larger one.
* fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog
Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those
clients counted as too old for the uncut catalog, their custom reasoning levels and stock
entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does.
* test(proxy): type the Codex catalog byte-limit test
* refactor(proxy): gate the uncut Codex catalog on the 0.166 release line
Read only the major and minor numbers of client_version instead of a shared
semver comparison, and leave the reasoning level and stock entry version
checks as they are on main.
* refactor(proxy): read every Codex client_version with one helper
Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version
parser rejects. The catalog byte limit, the per-model minimum version and the
custom reasoning level checks now share one reader for Codex's format, which
also bounds digit counts so an absurd version can't fail the listing.
