* perf(spend): narrow user and team spend log reads to the owner's api_keys so the (api_key, startTime) index is used
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): satisfy the type discipline gate for the api_key scope helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): run the api_key lookup under the statement timeout, skip it for search and empty key lists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(search): add Microsoft Web IQ provider
* fix(search): require HTTPS for Web IQ credentials
* fix(search): include Web IQ in the packaged endpoint catalog
* fix(search): register Microsoft Web IQ query pricing
* test(search): fix Web IQ exception assertion lint
* fix(search): type Web IQ response passthrough and search error headers
The type-check gate failed on 8 new unknown-type errors in the Web IQ
transformation. Build passthrough fields from a typed helper and validate
the response models directly, and type the base search error headers as
dict[str, str] so subclasses no longer inherit an unknown parameter.
* fix(search): raise Web IQ parse errors without widening the base header type
Typing the base search error headers as dict[str, str] made the shared
HTTP handler report one more argument-type error. Restore the base
signature and raise the same BaseLLMException directly from Web IQ.
---------
Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(responses): keep the namespace == name shape for deferred top-level tools
The OpenAI Agents SDK matches a call to a deferred top-level function
(defer_loading=True) only when the function_call carries namespace == name,
the shape it reserves for deferred top-level tools. The Responses bridge
returned no namespace, so after a native tool search the SDK failed with
"Tool <name> not found". A client that adds namespace == name then hit the
echo path, which joined it into "<name>__<name>" for the provider.
- namespace_tool_name_map maps deferred top-level functions to (name, name),
so non-streaming and streaming replies carry namespace == name
- the function_call echo does not qualify a name whose namespace equals it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(responses): keep the qualified name for a namespace member that shares its namespace's name
A namespace "search" with a member "search" is advertised as the chat tool
"search__search" and comes back as name == namespace == "search". The echo
rule from the previous commit skipped the join for every namespace == name,
so that replay reached the provider as "search", a tool it was never given.
The bare-name echo now applies only to the request's deferred top-level
tools (deferred_top_level_names), and the reply map uses the same set. A
deferred top-level name that a self-named namespace member also uses is left
out of both directions, so the namespace member keeps its round trip.
The lookup reads defer_loading is True instead of validating it, so
callers that only read the input (token counting under drop_params) do not
raise on a malformed flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(responses): keep the bare name for a namespace == name echo replayed without request tools
Stored history (previous_response_id) is rebuilt with no request tools, so the echo of a deferred
call carrying namespace == name used to join to name__name. Only a namespace that holds a member of
its own name still joins. The lookup also tolerates a tools value that is not a list, and the
comprehensions stay within one for and one if clause.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): read request tools through one validated accessor
The deferred-name lookup and the response-side tool maps take the request's tools from one
TypeAdapter-backed accessor, so a tools value that is not a list of objects reads as no tools and
the type gate does not grow.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): type the request mapping handed to the input transformer
transform_responses_api_input_to_messages takes dict[str, object] instead of a bare dict, so the
tools accessor's argument is known to the type gate and the instructions value is narrowed to str.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): build the namespace tool map from the validated member names
namespace_tool_name_map reads each namespace's members through the same validated accessor as the
self-named check, which keeps the type gate's per-rule counts at their merge-base values.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay stored turns with the tools they were sent with
Stored previous_response_id history is rebuilt with each turn's own tools, so a namespace member
that shares its namespace's name keeps the joined name__name form on replay, while a deferred
top-level call keeps the bare name. Adds the session handler regression.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep the object entries of a tools list and read them on the streaming path too
One accessor hands the request's tools to the request path and the streaming iterator, dropping only
the entries that are not objects instead of the whole list, so both paths see the same custom, namespace
and deferred tools. Trims the explanatory comments.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): count input tokens with the request's tools in hand
/v1/responses/input_tokens hands the request params, tools included, to the input transformer, so an
echoed call for a namespace member named after its namespace counts as name__name there too.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): hand the request tools to every input conversion that reads an echoed call
The guardrail provenance pass, the prompt caching pre-call check and the Rust precall token count convert
Responses input with the request's tools, so a namespace member named after its namespace keeps name__name
there as it does on the request path.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep namespace == name for deferred tools loaded by a client-side tool search
A tool the client's own tool search loaded is still a deferred top-level tool to the Agents SDK, so the bridge keeps its deferral for the response-side name map and the route strips it again before the provider sees the request.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Homen Shum <317650355+homenshum-cheiron@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): connect a per-user GitHub Copilot credential from the command line
lite credentials connect runs the GitHub device login with the caller's own key,
so users can connect Copilot without opening the LiteLLM UI.
* fix(cli): stop connect before the GitHub code expires and explain lost connections
The connect command kept polling until the proxy rejected the expired code with a raw HTTP 400, so it now stops one interval before expiry and prints its own expired message. A dropped connection or timeout while waiting now prints a short rerun message instead of a traceback.
* fix(cli): keep checking for approval until the GitHub code actually expires
Connect stopped one interval early, so an approval in the last seconds was missed. The countdown now starts when the code is requested, which is never later than the proxy's own cutoff, and polling continues while another poll fits.
* fix(proxy): send the streamed Responses fallback metadata as a typed Responses event
With expose_fallback_errors_to_caller on and include_fallback_errors set, a streamed /v1/responses
call that fell back sent its fallback metadata as a chat.completion.chunk with no type, so the OpenAI
SDK read it as an audio delta event with type None. On the Responses route the frame is now
{"type": "litellm.fallback_metadata", "litellm_fallback": {...}} and goes through the same Responses
framing as every other event, so it gets an event: line whenever its neighbours do. The chat route
keeps its chat chunk unchanged
* test(proxy): stop pinning the Responses stream terminator in the fallback metadata test
Main ends Responses streams without data: [DONE] since #47980, so the test reads the event frames whether or not a terminator follows them, and passes both before and after that change merges
* fix(proxy): keep the top-level model on the Responses fallback metadata event
* fix(proxy): keep the fallback metadata id on the Responses event and omit an empty model
* fix(proxy): leave the top-level model off the Responses fallback event when raw model names are on
The complexity router's return_raw_model_name leaves every other event unstamped, so the fallback event now matches. A new test streams a raw upstream frame before the event to cover how main names it.
* chore: push again after GitHub dropped the previous push event
* fix(proxy): send the Responses fallback event without an event line so older OpenAI SDKs read it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes
The /bedrock/model/{model}/invoke, invoke-with-response-stream, converse and
converse-stream routes built the pass-through call from the request body, so
x-pass- headers never reached Bedrock. The processor now hands the caller's
x-pass- headers to the pass-through route, keeps them out of the logged body
snapshot, and on Converse drops the invoke-only guardrail headers when the
deployment sets guardrailConfig and the request metadata header when the proxy
owns request metadata
* fix(bedrock): bill the served service tier on invoke pass-through and scope x-pass- forwarding to the Bedrock routes
* test(bedrock): assert forwarded x-pass- headers on the request Bedrock receives
* test(bedrock): gate the router-model x-pass- forwarding on the request Bedrock receives
* fix(bedrock): bill a caller's forwarded invoke guardrail and keep x-pass- headers out of guardrail vendor payloads
* fix(bedrock): bill a caller guardrail on every invoke model and replace a trace value Bedrock rejects
* fix(bedrock): keep guardrail traces out of chat responses, forward the trace header as sent, and drop credential x-pass- headers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(spend): report guardrail spend separately in daily activity
Model and key spend in /user/daily/activity/aggregated include the cost of
guardrails like Bedrock guardrails, and nothing in the response says how much.
A customer that bills tenants per key and per model could not get model cost
alone, so tenants that pay AWS for their own guardrails were charged for them
twice
Daily spend rows now carry a guardrail_spend column next to spend, and every
breakdown in the daily activity responses exposes it, so model cost is
spend - guardrail_spend. spend keeps its meaning, so budgets still count
guardrail cost as before
The value comes from the request's cost_breakdown.guardrail_cost, capped at
the row's spend because an allowance discount or a repeat job poll can charge
less than the guardrail cost the breakdown records. The delivery ledger
freezes it into the accounting payload next to accounting_savings, since the
stored payload does not keep cost_breakdown, and the legacy daily queue writes
the same value
The migration only adds the column with a 0 default, so rows written before it
read 0
* fix(spend): count guardrail spend on rows billed without a cost breakdown
A request blocked before the LLM call is billed for its guardrails straight from
guardrail_information, so it carries no cost_breakdown and its guardrail_spend
read 0. Both writers now price guardrail_spend from guardrail_information with
guardrail_information_cost, the same function success and failure rows use to add
guardrail cost to spend
Also fills guardrail_spend in the client usage view built from request logs,
qualifies the field description for rows recorded before it existed, and adds
the lock timeout the migration checks require
* fix(ui): avoid a nested ternary when reading guardrail entries from request logs
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): reject client-supplied Vertex location and project unless the admin opts in
A request body could carry vertex_location, vertex_project, or their
vertex_ai_* aliases and override the region and project pinned on a
Vertex AI deployment, sending the admin's credentials to a caller-chosen
project or region. Add the four fields to the banned request params so
they need allow_client_side_credentials or configurable_clientside_auth_params.
/vertex_ai/live and /rag/ingest name their Vertex project and location
by design and have no deployment pin to override, so they keep those two
fields while every other banned param is still rejected there
* fix(proxy): close the remaining Vertex target overrides and keep inert fields working
/vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in
A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL
Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target
* test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads
* fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest
* fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests
* test(proxy): give the batch tests' fake router the alias and routed-group lookups
* fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review
A Vertex opt-in now counts only when every group the request can reach opts in, router,
context window, and content policy fallbacks and one alias hop included, while a group
with no Google deployment never needs the fields. A model named in the route path is
checked next to the body model.
RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a
vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin
saved before the upgrade, and reads a corpus named by project number on the proxy's
own project.
/vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves
a setup frame sent as a binary message onto the session's target, and resolves the
region from default_vertex_config before the model path.
* fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one
With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project,
including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those
projects now connects, and any other project still closes with 1008 before a token is minted.
* fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest
A non-admin ingest into a store an admin created named by project number had its corpus moved onto
the proxy's own project before the admin-store exemption ran, so the upload went to a different
corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the
project-number rewrite and the target check.
* fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request
* fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list
* fix(proxy): check team public models before allowing target overrides
* fix(proxy): include team wildcard targets in opt-in checks
* fix(proxy): validate overrides against authenticated team routes
* fix(proxy): retain URL model during deferred credential checks
* feat(proxy): let teams send allowed S3 buckets for batch files and read them back
A team's metadata can now list allowed_s3_buckets. Keys on that team may send one
of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on
POST /v1/batches, and the upload and batch-create handlers check the bucket against
the team list plus general_settings.allowed_client_side_s3_buckets. The early auth
body check still rejects bucket fields on every other route and method
Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri
through an internal stored_s3_bucket_name param, so content, retrieve, delete and the
batch cost job can read files written to a request-chosen bucket. Callers can never
send that param, and files in the model bucket resolve exactly as before
* test(proxy): assert outcomes in team bucket tests and type the managed content read
Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count
* fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back
A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix
* fix(proxy): keep the request's approved S3 bucket when merging deployment credentials
Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none
* fix(proxy): keep the request bucket only on upload and batch create, not on reads
The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for
* fix(proxy): only proxy admins can set a team's allowed_s3_buckets
Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes
* fix(proxy): count a prompt-only call_endpoint request with the provider on every counter
POST /utils/token_counter?call_endpoint=true with only a prompt sent the
prompt to the provider count API on Gemini and Vertex only. Anthropic,
OpenAI, Bedrock and Azure AI (Claude) deployments answered a local
estimate instead. Every counter now gets the prompt as one user message,
so a provider error surfaces the same as the messages form, and the
framed stand-in after a failed provider count frames the prompt as one
user turn, matching the messages form
* fix(proxy): count a call_endpoint prompt like a user message on providers without a count API
* fix(proxy): count a call_endpoint prompt as a turn only on chat deployments
A prompt-only /utils/token_counter request with call_endpoint=true now goes
to the count API as one user message only when the deployment serves chat
(mode chat, responses, or unknown). Embedding rows and models with no
deployment keep the raw local count.
The Anthropic, Azure AI, and OpenAI count calls now honor the deployment's
timeout, and a count error with an empty body names its status code.
* fix(proxy): fall back to router_settings.timeout for count calls and answer 408 on a count timeout
Count calls to Anthropic, Azure AI, and OpenAI now pick their timeout the way
a chat call to the same deployment does: the deployment's timeout, else
router_settings.timeout, else request_timeout. A count call that times out
answers 408 instead of a 500 processing error.
* fix(proxy): count a prompt as a turn only where a count API exists and give count calls chat's timeout order
* fix(proxy): read the count call timeout through the router settings snapshot
The token counter now asks llm_router.settings().timeout_for() for the chat timeout order instead of a new Router member, so the router surface does not grow
* fix(router): answer timeout_for on every router settings snapshot
The proxy reads the count call timeout through llm_router.settings().timeout_for(), but only the Rust core's snapshot had that method, so a plain RouterSettings (the unit tests' router stub) raised AttributeError. timeout_for now lives on RouterSettings and reads the core's _core_deployment_call_timeout, like the snapshot's other members
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(customers): reject unknown fields on customer create and update instead of dropping them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(customers): derive the budget hint from the budget request fields and fix the create example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rate-limit): build model scope settlement targets from one owner tuple
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay Claude tool search on follow-up turns
Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them
A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400
The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors
* fix(responses): emit Claude's tool search output in stream order
The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives.
* fix(anthropic): keep the caller's tool names in tool search results
A tool name Anthropic cannot take gets rewritten on the request, so Claude's
tool_search_tool_result named the rewritten tool. The response now maps those
tool references back to the caller's names, streamed and not, and a replayed
search result maps them forward again on the next turn.
* fix(responses): replay Claude's tool search only on routes that run it
A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept
A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call
* fix(responses): keep a replayed search when the caller sent Anthropic's own search tool
A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it
* fix(responses): replay a BM25 tool search under its own name
* fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools
A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its
search arguments, a stale search drops only the tool references the follow-up no longer sends,
and the stored previous_response_id path keeps the base gate for routes the cost map does not flag
* fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool
A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had
* fix(responses): replay a stored Claude search only when the follow-up still sends a search tool
A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level.
handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors.
* fix(responses): leave out a stored Claude search kept in server_tool_calls
Main stores Claude's server-side tool search on the assistant message in
provider_specific_fields.server_tool_calls and tool_results, not in
tool_calls. The previous_response_id path only looked at tool_calls, so a
follow-up that sent no search tool still replayed the stored search.
The session handler now finds the search in either place and drops it from
both server_tool_calls and tool_results when the follow-up cannot run a
search.
* test(responses): give the replayed search guardrail fixture its own call id
The guardrail test reused toolu_01Weather, an id that streaming and bridge
tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran
first in the same worker, the guardrail write-back took its rebuild path and
the test failed for a reason unrelated to the replayed search.
* fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups
A search-only assistant turn now gets its search result renamed to the sanitized
tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on
Claude routes replay the search as base did, trimmed to the tools still sent, and
Bedrock invoke sends the tool search beta whenever the history holds a search
* test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool
* fix(anthropic): keep Anthropic's tool names in chat tool search results
A chat history whose search result used the request's tool names (weather.get) got
Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a
rollback. Chat completions now returns Claude's search result as main does, and the Responses
bridge maps a rewritten name back to the request's tool when it builds tool_search_output.
That also fills tool_search_output on Bedrock invoke streams, which carry no name map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id
The detail lookup matched request_id OR litellm_call_id, which Postgres runs as a full table scan when the litellm_call_id index is missing or invalid. Read by request_id, the primary key, first and by litellm_call_id only on a miss, both under the spend log statement timeout.
* test(proxy): type the call ID index test helpers
* fix(proxy): limit the Logs drawer call ID lookup to start_date and answer 400 for a malformed date
* fix(proxy): let spend log reads wait for a busy connection like a plain query
* test(proxy): let the session pages read double take max_wait
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill an OpenAI pass-through background response once across repeated reads
A finished background response now bills only on the first pass-through call
that sees it finished with usage. The claim is an insert on the new
LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds
across workers and replicas. Every later retrieve, cancel, streamed create, or
resumed stream logs like an unbilled read. With no database or a failed insert
the call still bills.
* fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse
* fix(proxy): claim background responses on failed streams and bound the claim
A resumed stream that fails after response.completed billed its usage through
the partial-usage path without claiming the response id, so a later read billed
it again. That path now claims the id and logs $0 when an earlier read paid.
The claim uses create_many(skip_duplicates=True) so a repeat read no longer
raises a unique violation in Postgres, and it waits at most 2 s before billing,
since the claim now runs before the client gets its response.
* fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids
* test(pass_through): cover a failed stream whose failure logging cannot be scheduled
* fix(proxy): start the background response settlement migration with a lock timeout
* fix(passthrough): leave a failed background stream's charge to its next read
A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let the OS assign Prisma engine ports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear Prisma connection state after failed startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer
Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525,
first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown
Codex versions still get the catalog cut to 1 MiB, since they drop a larger one.
* fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog
Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those
clients counted as too old for the uncut catalog, their custom reasoning levels and stock
entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does.
* test(proxy): type the Codex catalog byte-limit test
* refactor(proxy): gate the uncut Codex catalog on the 0.166 release line
Read only the major and minor numbers of client_version instead of a shared
semver comparison, and leave the reasoning level and stock entry version
checks as they are on main.
* refactor(proxy): read every Codex client_version with one helper
Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version
parser rejects. The catalog byte limit, the per-model minimum version and the
custom reasoning level checks now share one reader for Codex's format, which
also bounds digit counts so an absurd version can't fail the listing.
* fix: make scheduled spend resets idempotent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover reset roundoff with shared Redis workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate reset workers and type raw adapters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep updated_at moving on zero-spend budget resets
* fix: never leave spend below zero after a budget reset
* fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged
Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time
* fix(proxy): keep a resent budget reset time only once it is due
A save that resends the same budget_duration keeps the stored reset time
only when that time has already passed, so the scheduled sweep still
resets the period it is due for. A row that is not due yet realigns its
reset time on save, the same as before.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(transcription): treat libsndfile's unknown frame count as no duration
* refactor(transcription): inline the unknown frame count check
* fix(transcription): read the frame count once to keep the type gate flat
* test(transcription): annotate the new passthrough test's locals as Final
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): give rag ingest managed file ids and team credentials on vector store file routes
RAG ingest through a team's OpenAI deployment now registers the uploaded file as a LiteLLM managed file and returns its managed id, so GET /v1/files lists it for the caller. Re-ingesting a managed id attaches the provider file and returns the caller's managed id
Model-less vector store file calls (create, list, retrieve, content, update, delete) from a team key now use the team's OpenAI deployment that ingest uploads through, via the router, and fall back to the direct path when the key may not use one. Explicit api_key, api_base, litellm_credential_name, and forwarded provider keys still win, and blank values count as absent
* fix(proxy): route managed vector store file ids to their team deployment and keep missing-file 404s off the router
Model-less team vector store file calls now go straight to the provider with the team deployment's
credentials, so 404s on missing files no longer put the team deployment in cooldown.
A managed file id on the vector store file routes now uses the deployment named by its
llm_output_file_model_id when that deployment is proxy-wide or belongs to the caller's team, and falls
back to target_model_names otherwise.
* fix(rag): keep managed ids and team checks on every file and search path
Ingest no longer registers a raw file id when file_url is empty, a managed
file id on a proxy without the managed files hook answers 400, and vector
store search and /v1/rag/query hits show the managed id ingest returned.
Model-less vector store file calls use the team deployment only after the
key, team, and member model checks pass, fall through to the router when the
team has no OpenAI deployment, and log the deployment they used.
* fix(rag): keep provider ids on lookup failures and mint managed ids only for keys that can use them
A database error while mapping search hits to managed ids keeps OpenAI's ids instead of failing a finished search. Ingest registers a managed id only when the caller passes the team deployment's model checks, so a model-limited key keeps the raw id it can still use, and the file routes share that check.
* fix(rag): attach the managed file copy that matches the ingest credentials
* fix(vector_store_files): route a managed file id to the caller team's copy
* fix(vector_store_files): keep the upload copy for a store registered with its own credentials
* fix(vector_store_files): use a managed file's team copy only when the store holds it
* fix(vector_store_files): attach a managed file's team copy and keep named credential options out of the store check
* fix(vector_store_files): attach the team copy only to a store that routes to the team deployment
* fix(vector_store_files): keep a managed store's routing and check the upload copy through its own deployment
* fix(vector_store_files): route a managed file to its upload deployment only when the key may call it
A managed file id names its upload deployment, and with require_managed_files off that id is not checked against an ownership record. The routes now use that deployment only when the caller may call its public model, else they keep target_model_names routing as on main.
* fix(vector_store_files): log a pinned managed file call under its deployment's public model name
* fix(vector_store_files): send a pinned managed file call's deployment key only to that deployment's base
* fix(vector_store_files): keep stored credential headers intact and fail the store check on errors other than 404
* fix(vector_store_files): answer a failed store check with the provider status and copy credentials read by name
* fix(vector_store_files): send a call to the upload copy when the store check fails without a 404
A store check error other than a 404 now leaves the call on the upload copy, the way main always routed it, so the real call keeps the router's retries, the provider's status, and its failure log instead of failing on one transient error
* fix(vector_store_files): keep a credential's no-log key out of the store check
A named credential carrying a no-log key made the store check call aretrieve with no-log twice, so every managed id call through that deployment answered 500 where main answered 200
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted
* fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled
A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full.
* fix(responses): finish a turn's early token count when the client leaves
A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors
* fix(router): count one router request per Responses WebSocket turn
A socket whose finished turns each counted their own request in router
usage no longer adds one more request when it closes, so N turns count N
requests, as N HTTP calls do. A socket with no counted turn still counts 1
* fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): log keys and teams still on the deprecated empty models defaults at boot
* fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup
* fix(proxy): say which empty-models rows keep access in 2.0.0
Teamless keys listing all-team-models get no models in 2.0.0, so the
boot warning no longer says every counted row keeps its access. It
names each upgrade outcome only when its kind is in use, says the counts
include blocked and expired rows, and words each kind by what the
setting means rather than what it grants
* fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning
The warning no longer says the upgrade keeps access for empty models
lists, since no shipped code or public page backs that yet. It says
2.0.0 removes the defaults and how to keep today's access, and keeps
the advice for teamless all-team-models keys. The integration test
drops an empty-database assertion that ran before the check finished
* fix(proxy): leave rows with access groups out of the empty models boot warning
* fix(proxy): count access group rows, skip management and read-only keys in the empty models warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list images in the Codex catalog only for models known to read them
A translated Codex entry used to offer image input whenever no deployment declared supports_vision,
so Codex let users attach images to models that may not read them. Image input now goes out only
when every deployment behind the name says it reads images.
* test(proxy): type the vision combination test parameters
* refactor(proxy): rename the vision helper and drop the extra vision docstring sentence
* feat: add customer spend breakdown and filtering by current team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: recover legacy customer team spend and tolerate lookup outages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf: resolve customer team keys once per daily activity query
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: paginate customer team activity in SQL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments
Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog.
* fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them
codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.
* fix(sso): read the SSO client secret before the refresh-token DB check so a concurrent env sync cannot split the client id from its secret
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): type the secret-race test and read the provider from the IdP boundary mock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): record litellm_overhead_latency_metric for streams and count the proxy's per-chunk work in it
* fix(proxy): sum the per-chunk stream work through Logging.add_streaming_overhead so the lint gates hold
* fix(proxy): count the per-chunk work of chat completions streams too, and drop a comment that repeated the code
* fix(prometheus): keep the handoff overhead stamp when a /v1/responses stream finishes
The Responses streaming iterator calls update_response_metadata again at stream end with
include_overhead=False, which replaced the timing stamped at handoff with one that had no
overhead, so litellm_overhead_latency_metric still had no sample for streamed /v1/responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the parameters of the per-chunk overhead tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): add only the proxy's first-chunk work to a stream's overhead, not a sum over every chunk
A 1 ms hook on each of 500 chunks delays the client by 1 ms per chunk, not 500 ms, so the
overhead metric keeps the stamp plus the work on the first chunk and stops summing later ones
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drive the first-chunk overhead tests with a fake clock instead of real sleeps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): a keepalive ping is not a stream's first chunk, and test the overhead with a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust
* fix(model-cost): keep main's compat helpers pure and move network tests to integration
ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(messages): log native messages success once, after the proxy releases it
The proxy defers non-stream success logging past post-call guardrails, but the native host parked its log in its own slot and the release only ran for OCR, so a unary /v1/messages call served by Rust logged no success event, no SpendLogs row and no key, team, user or end-user spend. A deferred native success now parks in enqueue_deferred_logging with deferred_success_result, the same slots Python uses, so the proxy releases it like any other call and its guardrail step pruning and cost preparation see it too
Python runs sync success callbacks inside the deferred logging closure, so a post-call hook that blocks the response drops them. A parked native success now carries its sync callbacks and runs them on release
A refused first attempt and its fallback share one logger. Python logs the async success once, for the first attempt, so the native adapter now claims the async success log once across fallback attempts. Like Python, an attempt that loses the claim runs its sync callbacks only when logging is not deferred, and an internal call logs nothing
* test(messages): cover the deferred native success park and release
* feat(proxy): let a key's logging entry reference a proxy credential
A virtual key's metadata.logging entry can name a proxy credential with
litellm_credential_name. At request time the credential's values, which config
credential_list resolves through the secret manager at boot, are merged under the
entry's callback_vars, with the entry's own values winning. An entry whose
credential is not defined is dropped, so it never falls back to the proxy's own
callback environment.
Only proxy admins may change the logging entries of a key when the stored or
requested list holds a reference (403), and a reference to an undefined
credential is a 400. The check runs on key generate, update, team bulk update,
regenerate and /user/new, and compares decrypted entries so re-saving metadata
read back from /key/info is not a change.
Resolves LIT-10061
* fix(proxy): drop a key logging entry whose referenced credential holds no logging values
* fix(proxy): only require newly added credential references to exist on key writes
* fix(proxy): log a dropped credential reference as an error so /key/health reports it
* fix(proxy): gate credential references on bulk-created user keys
* fix(proxy): keep failing requests on keys with non-object logging metadata
A key whose stored metadata.logging is not a list of objects failed every
request with a 500 before this branch. The credential reference rewrite
turned those values into no entries, so the request quietly succeeded
without the key's logging. Raise a TypeError for those shapes again so the
request path behaves as it did on main, while empty values and objects with
unknown fields still go through the existing skip.
* test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging
The malformed logging cell expects the base behavior that 7ac4ab1356
restored, a 500 for a key whose stored metadata.logging is not a list of
objects and a 200 for an empty list
* fix(proxy): keep a 500 for teams with non-object logging metadata
On main, a team whose stored metadata.logging held non-object items failed
chat requests with a 500, because AddTeamCallback(**item) raised a TypeError.
This branch switched to AddTeamCallback.model_validate, whose before-validator
raises AttributeError on those items, and the proxy maps AttributeError to a
400 invalid_request_error. Raise a TypeError for non-mapping items before
validation so the status matches main again, as the key path already does.
* fix(complexity_router): keep Claude Code subagent and side-request traffic off other agents' tier pins
Claude Code sends one x-claude-code-session-id for the main agent, every
subagent and their side requests. In user_turn mode a subagent's new ask
rewrote the parent's pin (#50099), and the agent progress summaries Claude
Code fires during a subagent run rewrote the subagent's pin to the cheapest
tier (#50444).
Subagent requests now get their own tier pin keyed on session id plus
x-claude-code-agent-id. Requests marked x-claude-code-request-class:
auxiliary route as housekeeping, and only main and subagent turns may write
a pin. Without the hint header, the summary and prompt-suggestion prompts
are matched as housekeeping sentinels.
* chore(ui): regenerate schema.d.ts for the housekeeping_patterns description
* fix(complexity_router): let the request-class hint only mark housekeeping, not suppress pin writes
A client-sent request class must not decide whether a decision is saved.
Tagging a real ask as workflow or auxiliary dropped its routing decision,
so later continuations replayed an older pin with the new history. The
auxiliary hint now only routes the request through the housekeeping path,
which is already never pinned and defers to custom classifiers and
keyword rules.
* test(complexity_router): cover the auxiliary hint on a prompt no sentinel matches
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement
A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously
A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests.
* fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add
Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main.
* test(proxy): drive the no-membership 403 tests through the real membership and spend readers
Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log.
* fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check
Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue.
* fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check
Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat.
* test(proxy): drop docstrings that restate the no-membership tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): answer 400, not 500, for malformed native count-tokens requests
The Bedrock count-tokens handler now checks the body before any AWS call and raises a 400 BedrockError for a missing or empty messages list, a message without role or content, or a missing model. The token counter marks that rejection as local, so it skips the Mantle fallback and the framing estimate, and /v1/messages/count_tokens answers invalid_request_error
get_bedrock_base_model returns a bare cross-region prefix such as us or global unchanged instead of raising IndexError, so Bedrock judges the model id and its own 400 reaches the caller
* fix(bedrock): answer 400 for a malformed message on Mantle token counts
* fix(bedrock): limit the region-only model id fix to token counting
* fix(bedrock): count a routing-prefixed region-only id and leave other providers' count errors as they were
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL
* fix(prometheus): carry live spend for the user, org and customer budget gauges too
The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.
* fix(prometheus): show committed spend in the remaining budget gauges
Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.
* fix(prometheus): count a charge the cached customer row already holds once
* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets
Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.
* fix(prometheus): read the committed org spend on failed requests without the auth object
* test(prometheus): cover a failed request with no carried org
* fix(prometheus): move the committed counters for charges with no auth on the row
A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.
* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes
Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.
* fix(prometheus): count a charged failure's own cost in the org gauge
A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request
* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog
* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete
Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins
* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows
CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values
* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster
The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files
* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): build plugin session claims from FIPS provider services in fips_mode
With general_settings.fips_mode on, the per-plugin key comes from the
salt key KBKDF and the Fernet token is built from ssl.RAND_bytes,
AES-128-CBC and hmac.digest. Standard mode is unchanged.
* test(proxy): freeze the clock in the fips_mode plugin claim test
* feat(proxy): add OpenSearch index pass-through with per-key read/write grants
Adds /opensearch/<index>/<api> for LiteLLM vector store indexes on OpenSearch
stores. A key or team needs read or write on the index in
allowed_vector_store_indexes, and the proxy signs each call with the vector
store's AWS identity, so callers never hold AWS credentials
The route maps the LiteLLM index name to the store's provider index, refuses
any path, query parameter, or body field that could reach another index, and
sends the exact signed bytes upstream with only the signed headers
* fix(proxy): let OpenSearch _bulk and _msearch NDJSON bodies through auth
opensearch-py sends _bulk and _msearch NDJSON with content-type application/json, and the
auth-time body read parsed it as one JSON document and answered 400 before the route ran.
Auth now skips parsing OpenSearch NDJSON bodies the same way it skips Azure Speech audio.
* feat(proxy): accept a _bulk _index naming the path index and allow named search pipelines
A _bulk action line may name the index in the URL path, and LiteLLM rewrites it to the
real index, which is what opensearch-py's helpers.bulk sends. The search_pipeline query
param now goes through so hybrid search works, while an inline pipeline stays refused.
Also types the type-checking Router stub with _router_init's parameters again, which
make lint's tests/e2e check needs since the Rust core change.
* fix(opensearch): answer 401 when the AWS credential source cannot read its token file
* refactor(proxy): check OpenSearch request bodies without recursive helpers
* fix(opensearch): answer 401 when AWS credentials fail to refresh at signing time
* fix(opensearch): keep auth off OpenSearch bodies and widen the read list to index checks
Auth no longer parses any OpenSearch request body, so a document field named model, api_base or base_url is written
instead of answering 401 or 403. GET and HEAD on the index and HEAD on _doc and _source are reads, each operation
checks the exact grant the list names, an index whose store name two stores share answers 400, and an _msearch
header that names the path index is sent to the mapped index
* fix(opensearch): need read for writes that return documents and keep document fields out of LiteLLM params
* fix(opensearch): keep documents out of LiteLLM hooks, keep main's credential statuses, make _msearch admin only, and bill only searches
* fix(proxy): bill only an exact /opensearch/<index>/_search path
A document read such as /opensearch/docs/_doc/_search ended in /_search and was counted as a billable search. The check now matches the index segment and _search exactly
* fix(proxy): only proxy admins set vector store index grants, and bill no OpenSearch document call
An index grant now opens document writes through /opensearch, so key generate, update, bulk update,
regenerate, team new and update, /user/new, and bulk user creation answer 403 when a non-admin
changes allowed_vector_store_indexes, top-level or under metadata. Resending the stored grants is not a change
An /opensearch path is matched before the LLM route suffixes, so a document id such as search or
embeddings never makes a document call count as a billable request
* fix(proxy): treat an empty index grant list from a non-admin as a change
An explicit empty top-level allowed_vector_store_indexes passed the admin-only check as unchanged and then
replaced the stored grants. It now counts as a change, so a non-admin gets 403. The new grant tests also
carry parameter types
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): derive the web identity session policy per AWS caller
Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session.
* fix(bedrock): let async invoke write its S3 output under the web identity session policy
Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com).
* fix(opensearch): give OpenSearch vector store search its own web identity session policy
* fix(bedrock): give every remaining AWS caller its own web identity session policy
Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call
Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting
bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false
KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes
* test(batches): expect config export to skip every underscore attribute
The session policy class attribute is private, so batches config export no longer returns single-underscore names
* fix(bedrock): scope knowledge base ingestion role grants to the roles it creates
Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included
* fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies
* fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity
Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an
account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through,
or realtime
The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions,
document content, and ingested document ACLs, and a test derives that list from botocore
Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and
embeddings never open a bidirectional stream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): price expanded wildcard /model/info rows as the model their calls route to
* fix(proxy): read wildcard routing through the router registry entry points
* fix(proxy): price a wildcard row only when the router sends its name there
Use the router's own route candidates for the listed name, so an alias, a routing group, a more specific pattern, or a team's own deployment keeps a wildcard row unpriced
* fix(proxy): show no price on a wildcard row whose calls go to another deployment
* fix(proxy): keep the listed price on catch-all wildcard rows and match the edit path to the read path
* fix(proxy): price a wildcard row aliased to a sibling name as the alias target
* fix(proxy): price wildcard rows through the caller's alias renames and show nothing for rows another deployment takes
* fix(proxy): keep same-model wildcard row prices, skip deleted team aliases, use the caller's team
* fix(proxy): keep live team aliases when skipping deleted team alias targets
* fix(proxy): show no wildcard row price when the same-model deployment serving it sets custom rates
* fix(model_info): recognize a replayed wildcard row on a worker that has not loaded the wildcard
* test(proxy): feed wildcard discovery real provider data in model info tests
* fix(proxy): price a wildcard row another deployment serves as the model its calls bill
A row whose calls reach another deployment now shows that deployment's model
and limits, and goes blank only when those deployments bill more than one model
or declare their own rate. A name no route serves keeps its own price, and a
replayed row stores none of the metadata it showed.
Drop the router's per-call scan of every deployment for a team key, since the
team and name indexes already cover those names.
* fix(proxy): price a wildcard row aliased to an aliased group as the group its calls reach
An alias target that is itself aliased onward listed the onward group's deployments too, so the row showed no price while calls billed the first target's model. The billed model now comes only from the route candidates
* fix(proxy): price a wildcard row billed by another deployment without the wildcard's base_model
A row aliased to another deployment priced as that deployment's model, but the lookup still swapped in the source wildcard's base_model, so the row showed a rate its calls never bill
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): end the local login when logout cannot check the OS keychain
With no usable keyring, lite logout kept a secret-free token.json as a note
that the keychain went unchecked. Every reader took that note for a live
keychain-backed login, so lite whoami, lite auth print-token, and the bare
lite shell all reported a login the user had just ended, and a keychain that
answered again could pair the note with an earlier login's entry and sign
the machine back in.
Logout now removes token.json whatever the keychain answers. When the
keychain could not be checked and there was a login to end, it records that
in a separate owner-only keychain_unchecked file beside it, written after
the removal so a full disk still has room for it. A repeat logout still
warns while that file is there, and a logout that confirms the keychain is
clean removes it.
* fix(cli): keep warning after a logout the keychain refused to clear
A keychain that will not give up its entry gets the same note as one that
could not be checked, so a later logout made while the keychain is out of
reach does not report a clean logout while the credential is still there.
* fix(cli): revoke the login a logout left in an unreachable keychain once it answers
A logout made while the keychain was locked ended the local login and dropped the metadata that pairs with the refresh token still in the keychain, so the next logout had nothing to revoke. The not-cleared note now keeps that record without its secret, and a later logout pairs it with the keychain entry and revokes it before erasing.
* fix(cli): revoke a locked-keychain login on the next login, and only once after a refused erase
A login made after a logout that could not reach the keychain now revokes the
login it replaces, read from the keychain_not_cleared note, as the merge base
did through the kept token.json. A keychain that refused the erase had already
handed that logout the refresh token, so its note no longer pairs with the entry
and the next logout does not revoke the same token again.
* fix(cli): keep the first login revocable after a second locked-keychain logout
A logout whose login kept its secret in token.json never reached the
keychain, so it now leaves the note an earlier logout wrote in place, or
writes an empty one, instead of overwriting it. The first login's refresh
token stays revocable once the keychain answers, and a keychain-less
machine no longer keeps user and team ids in the note.
Also documents the limits that remain in the CLI README and drops the
docstrings from the tests this PR added.
* fix(cli): revoke the keychain's login after a login made while the keychain was locked
* fix(cli): stop lite from reading the stored login before login and logout
The cli group resolved the stored API key before every command, and that read moved a
locked-keychain login's secret over the keychain entry the logout note names, so logout and
login never saw the first login to revoke it.
* fix(cli): keep the login a locked logout left in the keychain revocable
While the keychain still holds the login an earlier logout could not
end, reading the stored login now serves the secret token.json kept and
leaves that keychain entry alone, whatever the login times say. A clock
that stepped back no longer serves the ended login, and commands other
than logout and login no longer move the newer secret over it. A silent
renewal that writes over the entry revokes that login on the proxy.
* fix(cli): keep a login revocable when its note cannot be written, and never block logout on an earlier login's proxy
A locked logout whose note cannot take the login's record now puts token.json back, so the next logout that reaches the keychain still revokes that login, as the base did. A logout that ends both a current login and an earlier one left in the keychain warns when the earlier login's proxy cannot record the revocation, as lite login does, instead of refusing to log out of the current proxy.
* fix(cli): move a locked keychain login's token file into its note instead of putting it back
* fix(cli): never tell a logout to delete the record of a keychain login it could not move aside
A read-only ~/.litellm made the locked-keychain logout say to delete token.json, which would leave the keychain's login unrevoked. It now says the logout did not happen and asks to make the directory writable. The README also says an older lite cannot finish a logout this release started.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): total_timeout bounds a whole non-streaming request
A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming
/v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries,
moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and
a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a
streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out
request is billed like a client disconnect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission
A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and
background requests it is ignored on. The deadline holds while pre-call admission runs, since admission
reserves shared quota before it records it, and an expired deadline then ends the request before
dispatch, where the failure path hands the reservation back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(key/info): return 400 instead of 500 when no key is provided
When GET /key/info is called without a ?key= query parameter and the
authenticated principal has no virtual key (e.g. an admin-UI session
token whose UserAPIKeyAuth.api_key is None), info_key_fn passed
hashed_key=None to Prisma find_unique(where={"token": None}), which
raised MissingRequiredValueError and surfaced as an unhandled 500.
Add an explicit None/empty guard that raises a clean 400 ProxyException
before the DB lookup, and add a regression test asserting that prisma
find_unique is never called with token=None.
Fixes#43571
Signed-off-by: apex-mochen <2756823972@qq.com>
* test: remove redundant assertion comment
* fix(key/info): remove redundant None check to satisfy basedpyright delta gate
After if not key: raise, key is narrowed to str. The subsequent
if key is not None: guard is always true and triggers basedpyright
reportUnnecessaryComparison, which fails the delta gate.
Collapse to a single hash_token_if_needed(token=key) call.
Signed-off-by: apex-mochen <2756823972@qq.com>
---------
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(accounting): stop delivery metrics from scanning retained receipts on every flush
The backlog query now reads only pending and parked deliveries, so its cost follows the backlog instead
of retained receipt history. Batch count and table size are measured once per check interval per pod,
and the storage and delayed-delivery alerts repeat at most once per interval instead of every flush
Fixes#50475
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): prove the storage query reruns only when its interval is due
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read the check interval on each flush so YAML environment_variables apply
The writer is built at proxy import, before the config's environment_variables load, so reading
the interval there ignored values set in config.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): wait for router accounting instead of sleeping in user-config limit tests
Each request ended with a 0.3s sleep and hoped the router's usage writes had landed, so under CI load a
concurrent request could read counters mid-update. The tests now flush the logging worker and wait for
every task the requests scheduled, and a retired router's counter flush lands before the next router reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
