* feat(s3): support SSE-KMS encryption params on both S3 logging paths
* fix(s3): ignore non-string SSE config values instead of crashing logger init
* Update litellm/integrations/s3.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): invalidate only the mistyped SSE field instead of dropping both
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
The ComplexityRouter's LLM classifier saw only the last user message, so on a
multi-turn conversation it classified whatever happened to be last rather than
what the human actually asked, and a near-constant classifier input pinned a whole
session to one tier.
The blindness turned out to be narrower than first diagnosed, and the fix is
correspondingly smaller. Tool output was never the problem: on the Messages surface
it rides a user turn as tool_result content blocks, which are not text parts, so
flattening to `type == "text"` already dropped those turns; on chat completions it
arrives on a `tool` role the extractor never read. Both surfaces were already
handled before this change. What actually leaked through was the harness
`<system-reminder>` block, which arrives as ordinary text, survives flattening, and
became the current ask on any turn that carried one.
So reminders are stripped rather than used to reject the turn, because a harness
injects them alongside the live ask and not as a turn of their own; rejecting the
turn would lose the ask, and keeping the block would feed the classifier the
near-constant boilerplate that flattens tier selection in the first place. An
earlier revision of this change also pattern-matched serialized tool_result
payloads. That check only ever fired on a hand-serialized string neither request
surface produces, it was where every review finding in this PR lived, and it is
deleted here; the tests now pin the real shapes instead of the synthetic one they
were built on.
The classifier call is split into a system role carrying the rubric plus the
caller's own system prompt, which stays byte-stable across a session so a provider
can prompt-cache it, and a user role carrying the variable context: a bounded
window of prior user turns, a conversation-depth signal, and the current ask. The
caller's system prompt rides every turn, so task constraints are never dropped.
The depth signal measures content-parts messages too, since counting only string
content reported ~0 tokens for exactly the deep Messages-surface conversations that
most need an expensive tier, and it is omitted entirely on the prompt-only path
rather than asserting a false zero.
Prior turns are excluded by matching the current ask rather than by dropping the
newest turn positionally, because `aclassify` takes `prompt` and `messages`
separately and a caller may classify something other than the newest turn.
Truncated turns carry a marker so the classifier can tell a turn was clipped.
Only the LLM classifier's input changes. The heuristic scorer, keyword overrides,
escalation matching and semantic embedding still read the extracted current ask,
which is why that extraction has to yield one clean human-authored string: those
are substring and vector matchers, and an escalation keyword sitting inside a
reminder blob would otherwise trip a tier jump on its own.
Defaults keep single-turn classification equivalent to before. The prior-turn
window is on by default so existing LLM-classifier deployments actually get the
fix; the config field documents that those turns reach the classifier model, which
may be a different provider than the routed completion model, and that the call
already carries the current ask and the caller's system prompt in full.
Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
* fix(anthropic): split mixed reasoning stream chunks
* style: use builtin generic annotation
* fix(anthropic): split mixed stream chunks by payload kind
The mixed-chunk split cleared only the fields it knew about on each
deep-copied piece, so any other payload riding the chunk survived on
both pieces: tool_calls were emitted as two tool_use blocks with the
same id, thinking_blocks on the text piece emitted duplicated thinking
into a text block while dropping the answer text, and chunks whose
reasoning arrived only as thinking_blocks never split at all
Rebuild each piece's delta from scratch with exactly one payload kind
(reasoning, text, tool calls), ordered to match native Anthropic block
order. Fresh Delta construction keeps unset attributes deleted, which
matters because the translators branch on hasattr, and prevents future
Delta fields from riding along on every piece
* fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once
Adversarial verification against the merge-base found three shapes where
the payload-kind split changed behavior beyond its target: a mixed chunk
carrying a tool argument continuation was torn into a truncated block
plus a fabricated one, a multi-choice chunk lost its secondary choices'
payload, and a signature-less thinking_blocks piece inherited the
non-empty block start body so accumulators collected the thinking twice
Continuation and multi-choice chunks now pass through the splitter
untouched, matching the merge-base byte for byte, and signature-less
thinking_blocks pieces are normalized to reasoning_content so the block
start opens empty and the thinking text is emitted exactly once
---------
Co-authored-by: Napuh <naamanynadiemas@gmail.com>
The Default User Settings form on Internal Users, the org settings form and
the org create dialog all rendered their money fields as
`<input type="number" step={0.01}>` inside a form that never opted out of
native constraint validation. Any value with more than two decimals, such as
a 0.001 max budget, failed the browser's step check, so Chrome vetoed the
submit before react-hook-form ran. No request went out, no field error was
shown, and the read view kept displaying the old value; it looked like the
budget silently refused to stick.
Money fields now use `step="any"`, and the three react-hook-form forms carry
`noValidate` so zod stays the only validator and a DOM-level constraint can
never swallow a submit again.
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
The batch rate limiter counts input tokens by awaiting litellm.afile_content
with no timeout, so a slow Files API holds POST /v1/batches open past any
client deadline; stage saw 63.6s against the harness's 60s read timeout. The
test times out before reaching the unattributed-spend-row assertion it exists
to guard, so it reports an infrastructure hang rather than the contract.
Skipping keeps the signal honest until the fetch is bounded.
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
The keyless flow (gateway as authorization server, no virtual key) worked
only at the aggregate /mcp scope: the session-bearer admission arm was
gated on _is_aggregate_mcp_scope, the 401 fallback only challenged at
aggregate scope, and per-server protected-resource metadata for plain
oauth2 servers pointed clients at the per-server relay, whose flow
returns the raw upstream token that ingress can never accept keylessly
(401 "LiteLLM Virtual Key expected. Received=gho_****").
Per-server spellings now join the same gateway flow for gateway-managed
oauth2 servers (auth_type oauth2 without delegate_auth_to_upstream, new
MCPServer.is_gateway_managed_oauth2 owner):
- the session-bearer arm admits at any MCP scope; downstream grant
resolution already intersects the admitted subject's servers with the
path or header targets fail-closed, so a narrower scope never broadens
- the 401 challenge is scope-aware: a single gateway-managed oauth2 path
target gets the per-server resource_metadata in the spelling the
request used, everything else gets the aggregate document; unknown
names, CSV multi-target paths, and every client-forwarded or delegated
mode keep their existing behavior
- per-server PRM for explicitly named gateway-managed oauth2 servers
advertises the gateway AS ({base}/mcp); delegate, passthrough, bridge,
OBO, and the root-resolved unnamed shape are byte-identical
- the preemptive 401 for an admitted keyless subject with no vaulted
token challenges with resource_metadata (re-entering the gateway flow,
whose authorize interlude vaults the upstream token) instead of the
relay authorization_uri, which cannot vault without a litellm key
The per-server challenge URL builder moved from server.py to
oauth_utils.py (shared with the auth module) and now inserts the
SERVER_ROOT_PATH segment exactly as the discovery routes do.
Resolves LIT-4864
The single-element tuple loop that bound the extracted deployment model
inside the comprehension read poorly; an assignment expression in the
filter clause does the same call-once-and-filter in one line.
Bytez and OCI param maps raise on stream_options when drop_params is
unset, so the default injection would have broken every streamed chat
completion routed to them. Injection now only happens when every router
deployment behind the requested model (wildcards and aliases included)
declares stream_options in its supported OpenAI params; providers that
do not declare it either reject the param or already stream usage
natively, so skipping them keeps old behavior instead of erroring.
_litellm_strip_stream_usage arriving in the client request body is now
overwritten at ingress (and popped in the experimental queue endpoint),
so a client can no longer suppress the usage chunk it explicitly
requested by planting the internal marker.
Streamed chat completions that did not opt into stream_options.include_usage
were logged with tiktoken estimates over the visible text, so hidden
reasoning tokens (billed as output by OpenAI-compatible providers) were
never counted and SpendLogs could undercount output tokens by 90%+ on
reasoning models. The proxy now injects include_usage upstream for
/v1/chat/completions streams by default and strips the injection artifacts
(the final usage chunk and the empty prompt-filter chunk) from the
client-facing SSE stream, so accounting uses provider-billed usage while
the client-visible stream stays byte-identical to today.
always_include_stream_usage keeps its existing semantics: true forwards
the usage chunk to clients as before, and an explicit false now acts as a
kill switch that disables the upstream injection for OpenAI-compatible
backends that reject stream_options.
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
Logical replication consumers need FULL replica identity to reconstruct the
old row of an UPDATE or DELETE, and prisma leaves every table it creates at
the postgres default. Operators had to re-apply the setting by hand after
each migration run.
Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM
table at the end of a successful migration run, through the prisma CLI so the
dependency-free proxy-extras package stays that way. Tables that are already
FULL are skipped, foreign tables in the same schema are left alone, and a
database that refuses the ALTER is reported rather than failing the run.
Resolves LIT-3022
Two defects combined to make a Redis outage take the proxy down rather than
degrade it.
First, connection kwargs were dropped whenever Redis was configured by url.
_get_redis_url_kwargs built its allowlist from
inspect.getfullargspec(redis.Redis.from_url); from_url is declared
(cls, url, **kwargs), so the argspec carried no connection kwargs and the
function returned ['cls', 'url', 'url']. socket_timeout went with the rest,
and socket_connect_timeout falls back to it, so both ended up None and a
Redis host that drops packets rather than refusing them blocked callers
indefinitely. get_redis_connection_pool's url branch lost the same kwargs by
a different route, rebuilding its pool kwargs from scratch.
The allowlist now comes from the connection class redis-py actually forwards
those kwargs to, walking the MRO because redis-py splits them between
AbstractConnection and its subclasses. Deriving it from the client instead
would admit client-only settings such as single_connection_client and the
SSLConnection-only ssl_* family, which reach AbstractConnection and raise
TypeError on first connect.
Second, the circuit breaker could not trip even once calls failed fast.
_redis_circuit_breaker_guard inferred success from the method returning, but
async_get_cache, async_batch_get_cache, async_set_cache, async_set_cache_pipeline,
async_set_cache_sadd and async_get_ttl catch their own connection errors and
return a default so callers degrade. Each failed call therefore reset the
failure streak and the breaker never opened, so an unreachable Redis stayed in
the pool and every request kept paying a full socket timeout on it. Those
methods now mark the failure and the guard records success only when nothing
failed while the method ran. Lua script execution went through none of this,
which mattered most because the rate limiter issues all of its Redis traffic
that way, so the guard is now a small helper shared by both.
The per-call marker is a ContextVar rather than a counter on the breaker.
Breakers are shared by every concurrent caller, so a shared counter cannot
tell "my call failed" from "some other in-flight call failed", and a success
overlapping someone else's failure would be discarded until a Redis that was
still answering got evicted from the pool anyway.
Only connectivity failures feed the breaker. Command and data errors say
nothing about whether Redis is reachable, and counting them would let a caller
provoke evictions on demand (an INCR against a non-numeric value, say),
dropping rate limiting to per-process counters that spreading traffic across
replicas can outrun.
OpenAI publishes a long-context column on the Flex tier, at half the standard
long-context rate. We had no field for it, so a >272k flex request fell through
to the standard long-context price and billed 2x: Terra $4/$18 instead of
$2/$9, Luna $0.40/$1.80 instead of $0.20/$0.90, Sol $10/$45 instead of $5/$22.50.
Adding the values to the cost map alone does nothing, because get_model_info
builds ModelInfoBase from an explicit kwargs list and silently drops any key
not named there. Declare the four *_above_272k_tokens_flex fields and wire them
through, then add the values for sol, terra, luna, and the gpt-5.6 alias.
That same gap was already swallowing cache_creation_input_token_cost_flex,
_priority, and _above_272k_tokens, which were present in the cost map but never
reached the calculator; they are wired through here too.
Fast mode (ex-Priority) publishes no long-context column, so nothing is added
there rather than deriving a rate by analogy.
Guardrails could only see the MCP tool call request (pre_mcp_call /
during_mcp_call); the tool result went back to the client unscanned, so a tool
that returns sensitive data bypassed every configured guardrail.
Adds a `post_mcp_call` event hook that runs after the tool executes and routes
the result through the unified apply_guardrail seam, so a text guardrail (e.g.
presidio) can mask sensitive values in the tool output or reject the result
without any MCP-specific code of its own.
- MCPGuardrailTranslationHandler.process_output_response now extracts the tool
result's text content into GenericGuardrailAPIInputs["texts"], calls
apply_guardrail with input_type="response", and writes the returned text back
into the content list in place (the logging payload already references that
object, so a copy would leave the unmasked text in the spend log)
- ProxyLogging.post_mcp_call_hook dispatches guardrails that implement
apply_guardrail, gated on should_run_guardrail(post_mcp_call); guardrails
implementing async_post_mcp_tool_call_hook keep their existing dispatch and
are not run twice
- both MCP tool-call paths (mcp_server and the Responses API handler) now honor
the rewritten result, and the REST path no longer swallows a guardrail
rejection as a logging failure
- shared, duck-typed MCP content helpers live in mcp_server/utils.py next to
extract_mcp_tool_result_error_message
- documents that async_post_mcp_tool_call_hook's return value is discarded by
every call site, so that hook only takes effect by mutating in place
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
Reverts the shared IdCell change from the previous commit and scopes the
fix to the budgets table instead
IdCell truncates with `block max-w-[15ch]`, a character-count clamp with
no relationship to the column's width. On budgets the Budget ID column
renders 509px wide at a 1400px container while the ID stays pinned at
108px, so UUIDs ellipsize with ~400px of empty space beside them
Changing that clamp in IdCell itself is wrong today because nothing else
bounds the column. DataTable emits `width: <size>px` on each cell but
leaves the table in `table-auto`, where `width` is only a hint and
`max-width` on a cell is ignored outright (measured: a 120px request
yields a 938px column). Only `table-fixed` binds `size`, and DataTable
enables it solely under `enableColumnResizing`, which 4 of 40 tables use.
So an unbounded IdCell lets content drive the column: Request Logs would
render a 64-char key hash in full, taking its key_hash column from 124px
to 494px and pushing the table from 1918px to 2326px, introducing
horizontal scroll at 1920 where there was none
Scope it to the call site instead. `cn` is tailwind-merge backed, so a
`max-w-*` passed via className dissolves the base clamp while leaving
`truncate` in place; budget IDs render in full and still ellipsize at the
cell edge if one ever outgrows the column. No other table moves
This is a workaround. The real fix is to make column `size` authoritative
by separating a fixed-layout option from `enableColumnResizing`, then
dropping the per-cell clamps; 307 of 321 column defs already declare a
size, so the mechanical gap is small, but ~20 tables would gain
horizontal scroll at 1440 and that needs its own review
The v3 parallel-request limiter stashed its per-request bookkeeping (TPM
reservation, descriptors, parallel slot, rate-limit response snapshot,
released flag) in the request body's metadata channels. On routes where
metadata is a provider request parameter (Responses API and the other
LITELLM_METADATA_ROUTES) that leaked internal keys upstream and produced
HTTP 400s, and it required denylist stripping plus dual-channel writes to
contain.
The stash now lives on an asyncio ContextVar holding a single typed
RequestRateLimiterStash per request. The pre-call hook writes it, and the
success/failure callbacks, disconnect release, and post-call hooks read
and clear the same shared instance, which keeps the refund and slot
release idempotent across sibling callbacks. The request body is never
touched, so the stash-key stripping, the metadata mirror writes, and the
all_litellm_params denylist entries are removed
* fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name
The GenAI metric attribute builder mapped only chat, text completion, embedding,
responses and MCP tool calls to an operation name, so vector-store searches and
A2A agent sends fell through to the "chat" default. Their duration and cost then
landed in the same series a Grafana GenAI dashboard reads chat latency off, with
no way to tell them apart. Both now map to the operation names the convention
defines for them, retrieval and invoke_agent, and an unmapped call type says so
at debug instead of silently becoming chat.
The provider label used gen_ai.system, which the convention deprecated in favor
of gen_ai.provider.name; the dashboards built on that vocabulary find nothing
under the old key. Metrics now carry gen_ai.provider.name with the semconv
provider value (bedrock -> aws.bedrock) via the resolve_provider helper the span
path already uses, and keep dual-emitting gen_ai.system with its raw value so a
dashboard already querying it keeps matching. A request litellm cannot attribute
to a provider gets no provider label at all rather than a placeholder "Unknown"
that minted a permanent series nobody can act on.
Resolves LIT-4954
Resolves LIT-4959
* fix(otel): map the rest of the vector-store call types off the chat default
Mapping only the search left the store lifecycle (create, retrieve, list,
update, delete) and the file operations (create, list, retrieve, content,
update, delete) falling through to chat, so vector-store admin traffic kept
polluting the same series a dashboard reads chat latency off. A live run
confirmed it: all 20 metric datapoints from a create, retrieve, list, file-list
and delete came out labelled chat.
The convention names no operation for vector-store management, so these take
vendor values under the litellm. prefix, litellm.vector_store_management and
litellm.vector_store_file_management, one per REST resource. Its note on
gen_ai.operation.name directs instrumentation to use a system-specific name
when no predefined value applies, which is the same allowance resolve_provider
already relies on for unmapped providers. Excluding them from the GenAI metrics
altogether was the alternative; it deletes series an operator may be watching
today and is far harder to reverse than a rename, so it stays available as a
follow-up rather than being decided here. Mapping them onto the semconv memory
store family was rejected: litellm vector stores hold documents, not agent
memory records, and borrowing those names would put document admin calls into
whatever charts agent-memory operations, which is the bug this fixes.
/rag/query reaches the same recorder and is the same operation as a vector-store
search, so query and aquery map to retrieval too; leaving them would have left
the defect alive on a second retrieval surface. /rag/ingest is a write with no
semconv equivalent and no retrieval or agent confusion, so it is left for the
RAG owners to name.
Resolves LIT-4954
* fix(otel): give the streaming A2A path a call type so it labels as invoke_agent
The streaming logging object is built by hand and never runs through
update_environment_variables, the only place call_type reaches
model_call_details, so every streamed agent turn arrived at the recorder
with no call type and fell back to chat. Stamp it, and map the streaming
spelling alongside the non-streaming ones.
