* feat(s3): support SSE-KMS encryption params on both S3 logging paths
* fix(s3): ignore non-string SSE config values instead of crashing logger init
* Update litellm/integrations/s3.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): invalidate only the mistyped SSE field instead of dropping both
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
The ComplexityRouter's LLM classifier saw only the last user message, so on a
multi-turn conversation it classified whatever happened to be last rather than
what the human actually asked, and a near-constant classifier input pinned a whole
session to one tier.
The blindness turned out to be narrower than first diagnosed, and the fix is
correspondingly smaller. Tool output was never the problem: on the Messages surface
it rides a user turn as tool_result content blocks, which are not text parts, so
flattening to `type == "text"` already dropped those turns; on chat completions it
arrives on a `tool` role the extractor never read. Both surfaces were already
handled before this change. What actually leaked through was the harness
`<system-reminder>` block, which arrives as ordinary text, survives flattening, and
became the current ask on any turn that carried one.
So reminders are stripped rather than used to reject the turn, because a harness
injects them alongside the live ask and not as a turn of their own; rejecting the
turn would lose the ask, and keeping the block would feed the classifier the
near-constant boilerplate that flattens tier selection in the first place. An
earlier revision of this change also pattern-matched serialized tool_result
payloads. That check only ever fired on a hand-serialized string neither request
surface produces, it was where every review finding in this PR lived, and it is
deleted here; the tests now pin the real shapes instead of the synthetic one they
were built on.
The classifier call is split into a system role carrying the rubric plus the
caller's own system prompt, which stays byte-stable across a session so a provider
can prompt-cache it, and a user role carrying the variable context: a bounded
window of prior user turns, a conversation-depth signal, and the current ask. The
caller's system prompt rides every turn, so task constraints are never dropped.
The depth signal measures content-parts messages too, since counting only string
content reported ~0 tokens for exactly the deep Messages-surface conversations that
most need an expensive tier, and it is omitted entirely on the prompt-only path
rather than asserting a false zero.
Prior turns are excluded by matching the current ask rather than by dropping the
newest turn positionally, because `aclassify` takes `prompt` and `messages`
separately and a caller may classify something other than the newest turn.
Truncated turns carry a marker so the classifier can tell a turn was clipped.
Only the LLM classifier's input changes. The heuristic scorer, keyword overrides,
escalation matching and semantic embedding still read the extracted current ask,
which is why that extraction has to yield one clean human-authored string: those
are substring and vector matchers, and an escalation keyword sitting inside a
reminder blob would otherwise trip a tier jump on its own.
Defaults keep single-turn classification equivalent to before. The prior-turn
window is on by default so existing LLM-classifier deployments actually get the
fix; the config field documents that those turns reach the classifier model, which
may be a different provider than the routed completion model, and that the call
already carries the current ask and the caller's system prompt in full.
Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
Move the budgets table onto the paged management list route so sorting,
filtering and search happen server-side instead of over whichever rows
happened to be in memory.
Adds useResourceList, a generic hook that owns page, page_size, sort, q
and filters for a server-driven table, folds them into one JSON:API query
and returns exactly the props DataTable's server modes want. Budgets is
its first consumer.
The budget id column now renders in full with a copy button instead of a
fixed-width cell, and the table gains Reset and Created columns.
* fix(anthropic): split mixed reasoning stream chunks
* style: use builtin generic annotation
* fix(anthropic): split mixed stream chunks by payload kind
The mixed-chunk split cleared only the fields it knew about on each
deep-copied piece, so any other payload riding the chunk survived on
both pieces: tool_calls were emitted as two tool_use blocks with the
same id, thinking_blocks on the text piece emitted duplicated thinking
into a text block while dropping the answer text, and chunks whose
reasoning arrived only as thinking_blocks never split at all
Rebuild each piece's delta from scratch with exactly one payload kind
(reasoning, text, tool calls), ordered to match native Anthropic block
order. Fresh Delta construction keeps unset attributes deleted, which
matters because the translators branch on hasattr, and prevents future
Delta fields from riding along on every piece
* fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once
Adversarial verification against the merge-base found three shapes where
the payload-kind split changed behavior beyond its target: a mixed chunk
carrying a tool argument continuation was torn into a truncated block
plus a fabricated one, a multi-choice chunk lost its secondary choices'
payload, and a signature-less thinking_blocks piece inherited the
non-empty block start body so accumulators collected the thinking twice
Continuation and multi-choice chunks now pass through the splitter
untouched, matching the merge-base byte for byte, and signature-less
thinking_blocks pieces are normalized to reasoning_content so the block
start opens empty and the thinking text is emitted exactly once
---------
Co-authored-by: Napuh <naamanynadiemas@gmail.com>
The Default User Settings form on Internal Users, the org settings form and
the org create dialog all rendered their money fields as
`<input type="number" step={0.01}>` inside a form that never opted out of
native constraint validation. Any value with more than two decimals, such as
a 0.001 max budget, failed the browser's step check, so Chrome vetoed the
submit before react-hook-form ran. No request went out, no field error was
shown, and the read view kept displaying the old value; it looked like the
budget silently refused to stick.
Money fields now use `step="any"`, and the three react-hook-form forms carry
`noValidate` so zod stays the only validator and a DOM-level constraint can
never swallow a submit again.
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
The batch rate limiter counts input tokens by awaiting litellm.afile_content
with no timeout, so a slow Files API holds POST /v1/batches open past any
client deadline; stage saw 63.6s against the harness's 60s read timeout. The
test times out before reaching the unattributed-spend-row assertion it exists
to guard, so it reports an infrastructure hang rather than the contract.
Skipping keeps the signal honest until the fetch is bounded.
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
The keyless flow (gateway as authorization server, no virtual key) worked
only at the aggregate /mcp scope: the session-bearer admission arm was
gated on _is_aggregate_mcp_scope, the 401 fallback only challenged at
aggregate scope, and per-server protected-resource metadata for plain
oauth2 servers pointed clients at the per-server relay, whose flow
returns the raw upstream token that ingress can never accept keylessly
(401 "LiteLLM Virtual Key expected. Received=gho_****").
Per-server spellings now join the same gateway flow for gateway-managed
oauth2 servers (auth_type oauth2 without delegate_auth_to_upstream, new
MCPServer.is_gateway_managed_oauth2 owner):
- the session-bearer arm admits at any MCP scope; downstream grant
resolution already intersects the admitted subject's servers with the
path or header targets fail-closed, so a narrower scope never broadens
- the 401 challenge is scope-aware: a single gateway-managed oauth2 path
target gets the per-server resource_metadata in the spelling the
request used, everything else gets the aggregate document; unknown
names, CSV multi-target paths, and every client-forwarded or delegated
mode keep their existing behavior
- per-server PRM for explicitly named gateway-managed oauth2 servers
advertises the gateway AS ({base}/mcp); delegate, passthrough, bridge,
OBO, and the root-resolved unnamed shape are byte-identical
- the preemptive 401 for an admitted keyless subject with no vaulted
token challenges with resource_metadata (re-entering the gateway flow,
whose authorize interlude vaults the upstream token) instead of the
relay authorization_uri, which cannot vault without a litellm key
The per-server challenge URL builder moved from server.py to
oauth_utils.py (shared with the auth module) and now inserts the
SERVER_ROOT_PATH segment exactly as the discovery routes do.
Resolves LIT-4864
The single-element tuple loop that bound the extracted deployment model
inside the comprehension read poorly; an assignment expression in the
filter clause does the same call-once-and-filter in one line.
Bytez and OCI param maps raise on stream_options when drop_params is
unset, so the default injection would have broken every streamed chat
completion routed to them. Injection now only happens when every router
deployment behind the requested model (wildcards and aliases included)
declares stream_options in its supported OpenAI params; providers that
do not declare it either reject the param or already stream usage
natively, so skipping them keeps old behavior instead of erroring.
_litellm_strip_stream_usage arriving in the client request body is now
overwritten at ingress (and popped in the experimental queue endpoint),
so a client can no longer suppress the usage chunk it explicitly
requested by planting the internal marker.
Streamed chat completions that did not opt into stream_options.include_usage
were logged with tiktoken estimates over the visible text, so hidden
reasoning tokens (billed as output by OpenAI-compatible providers) were
never counted and SpendLogs could undercount output tokens by 90%+ on
reasoning models. The proxy now injects include_usage upstream for
/v1/chat/completions streams by default and strips the injection artifacts
(the final usage chunk and the empty prompt-filter chunk) from the
client-facing SSE stream, so accounting uses provider-billed usage while
the client-visible stream stays byte-identical to today.
always_include_stream_usage keeps its existing semantics: true forwards
the usage chunk to clients as before, and an explicit false now acts as a
kill switch that disables the upstream injection for OpenAI-compatible
backends that reject stream_options.
/tag/list returned HTTP 500 for every internal user with
"LiteLLM_VerificationTokenActions.find_many() got an unexpected keyword
argument 'select'". The non-admin branch scopes the tag list to keys owned
by the caller, and that lookup passed select={"token": True}; prisma-client-py
0.11.0 has no select kwarg on find_many, so the call raised TypeError and the
handler's except block turned it into a 500. Since the Admin UI calls
/tag/list on load, Tags was broken for every non-admin user.
/tag/daily/activity shares the same helper and was failing the same way
The kwarg is dropped rather than replaced; the generated client has no
projection API, and a user's key set is small enough that selecting all
columns is not worth working around
The reason this shipped green is that the existing test asserted the call was
made with select={"token": True} against an AsyncMock, which accepts any
keyword. The verification-token table double now binds each call against the
real find_many signature, so an unsupported kwarg raises the same TypeError
production does
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
Adds search_context_cost_per_query pricing for the 82 OpenAI/Azure models that advertise supports_web_search but had none (gpt-5 family, o-series, deep-research at $0.01/call; gpt-4.1 at $0.025/call), so built-in web search is no longer billed as $0. Also counts web_search_call items in Responses output so N searches bill N times instead of once; usage-count providers (gemini, anthropic, xai, vertex) still route through get_cost_for_web_search_request and are unaffected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Logical replication consumers need FULL replica identity to reconstruct the
old row of an UPDATE or DELETE, and prisma leaves every table it creates at
the postgres default. Operators had to re-apply the setting by hand after
each migration run.
Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM
table at the end of a successful migration run, through the prisma CLI so the
dependency-free proxy-extras package stays that way. Tables that are already
FULL are skipped, foreign tables in the same schema are left alone, and a
database that refuses the ALTER is reported rather than failing the run.
Resolves LIT-3022
