Vector store create, list, retrieve, update, delete and search never read the
`?model=` query parameter or the `x-litellm-model` header, so with no managed
registry entry and no body model the request fell through to the SDK default
provider (openai) instead of the configured deployment.
Resolve the hint into data["model"] when nothing else established routing, and
add /vector_stores to the auth model-routing markers so the hinted model is
checked against key/team model access.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An MCP permission level answers which servers and tools it permits, and a
level that answers nothing places no restriction. Key auth was reading a
lookup FAULT as that same answer, so the end user, agent and org ceilings
quietly disappeared for as long as one lasted, while the keyless
gateway-admitted path failed closed on the very same fault.
Those levels now separate the two fault classes the user level already
did. A principal row that NAMES an object_permission_id whose contents
cannot be read is a known entitlement with unknown contents, so it denies.
A lookup that fails before we can tell whether the principal is entitled
at all still places no ceiling, that being the state which existed before
the level did; denying there would refuse MCP to the majority of callers,
who have no such entitlement configured. The keyless path is unchanged.
Resolves LIT-4960
PR #35185 added classifier_context_window_size and classifier_context_per_turn_chars
to ComplexityRouterConfig; they worked via config.yaml and the API but had no UI
control on the Add Model or Edit Auto-Router screens. Wires the two fields into
both, shown only when the LLM classifier is selected.
* fix(mcp): annotate connected-app reachability on the gateway connect page
The MCP connect page resolved its server grid through the dashboard identity
(admin shortcut or view_all returns the whole registry) while the gateway DCR
session it sets up resolves servers as an admitted subject through grant
sources only, so the page showed servers and tool counts the session is never
served. GET /v1/mcp/server now accepts connected_app_view=true and stamps each
returned server with connected_app_reachable, computed by the same
_reload_admitted_user + get_allowed_mcp_servers pair the live session uses.
The connect page requests the flag in connect mode and renders unreachable
servers dimmed with a label, excluded from the Connected count and tool-count
fetches. Failure to build the admitted set marks everything unreachable, which
matches what such a session would actually be served. Default behavior without
the param is unchanged for every existing consumer.
* fix(mcp): block connecting unavailable servers from the connect-mode detail view
A server the connect page marks unavailable could still be added through its
detail view Connect action, so the selection could contain servers the
connected-app session is never served. The unavailability decision now lives in
one predicate, connectUnavailabilityLabel, consumed by the card indicator, the
detail view action area, the toggle-on path, the oauth auto-select effect, and
the Connected count, so no interaction path can disagree with the label. This
also closes the same pre-existing hole for servers marked not supported on this
connection, whose detail view likewise offered Connect, and removes a
grandfathered nested ternary, ratcheting the eslint suppressions baseline down
* fix(mcp): hide unreachable servers on the connect page instead of dimming them
Product decision: the connect page should only show what a connected-app
session will actually be served, so annotated-unreachable servers are now
filtered out of the connect-mode list at fetch time rather than rendered
dimmed. Unsupported auth types keep their existing dimmed label since they are
a property of the server, not the caller. A user with zero reachable servers
gets an explanatory empty state pointing at grants. The list filter is the
single source: counts, tabs, auto-select, detail view, and tool-count fetches
all derive from the already-filtered state
* fix(mcp): guarantee the connect view lists every session-reachable server
The connect view's membership came from the dashboard resolver with the
admitted-subject answer only annotated on top, so a server reachable by the
session but missing from the dashboard list would be invisible on the page; an
under-report, the mirror of the bug this PR fixes. The connect view now unions
in any session-reachable server the dashboard resolver did not list, built
from the registry and redacted through the same ladder, so page membership
equals the admitted set by construction in both directions
* fix(mcp): honor connected_app_view only for the dashboard UI session credential
The reachability view resolves through the owning user's admitted identity, so
a caller-passed virtual key could use the param to enumerate servers beyond
its own scope (ids, names, descriptions of the owner's wider grants). The view
is now gated on is_ui_session_credential, a predicate factored out of
resolve_ui_session_team_ids so the two user-identity widening sites share one
trust boundary: the SSO-minted dashboard session token acting as its user. Any
other credential gets the param as a no-op and the admitted resolver is never
consulted for it
* fix(mcp): resolve UI sessions with the admitted-user context everywhere, not per endpoint
The list endpoint unioned in session-reachable servers itself while tool
counts, Connect actions, and credential endpoints still authorized through
build_effective_auth_contexts, whose contexts carry team grants but never the
user row's own object permission; a user-granted server could render on the
connect page while every interaction on it failed. The admitted-user context
(the same auth a gateway session resolves with) is now appended inside
build_effective_auth_contexts for UI session credentials, so the page list and
every per-server action endpoint answer identically, and the list endpoint's
one-off union is deleted. Caller-passed keys are still never widened
(is_ui_session_credential gate inside the context builder) and a reload
failure falls back to team contexts only
* fix(mcp): resolve non-admin dashboard sessions as the admitted subject on tool routes
Server reachability on the REST tool routes came from the widened context
union while tool permission checks ran on the bare session key, which carries
no object permission, so a dashboard user could invoke tools their user-level
grant excludes. Rather than bookkeeping which context granted which server,
the routes now choose one principal at the boundary: acting_user_auth swaps a
non-admin UI session for the admitted-subject auth, the same identity a
gateway session resolves with, so reachability, per-source fail-closed tool
ceilings, rate limits, and billing attribution all bind through the admitted
arms that already exist downstream. Admin sessions keep their operator view
and caller-passed credentials are never widened. One swap point per route,
no per-server principal picking, no parallel permission logic
* fix(mcp): derive the connect page's detail view from the reachable server list
The detail view held its own copy of the server object, so it outlived the list it came
from. When a refetch dropped that server as unreachable, the open detail view kept
rendering it and its Connect action still ran: the guard looked the server back up by id
or name in the current list, found nothing, and fell through, because a missing target
read as "nothing to block" rather than "no longer connectable"
Store the selected server's id and derive the row from the list instead. A server the
list no longer carries cannot be the detail view's subject, so the stale render, the
stale tools query and the guard bypass stop being reachable states rather than being
blocked one at a time. handleToggle now takes the server it is toggling, which deletes
the lookup that could miss at all
* refactor(mcp): one owner for the identity a dashboard session acts as
Three call sites reloaded the admitted subject independently, and the management
endpoint carried its own copy of the reload, the HTTPException swallow and the logging.
admitted_user_context is now the only place that answers "what user identity does this
dashboard session act as", and the connected-app reachability helper reads it, which
also drops its dead empty-user_id branch
That owner now carries the request's tracing span onto the admitted principal.
_reload_admitted_user builds a fresh auth from the user row and has no span of its own,
so swapping it in on the REST tool routes silently detached every downstream lookup and
the tool-call logging from the request's trace
Toolset scoping and the acting-as-user swap are mutually exclusive, so they now share
one owner on the tools list route. The admitted subject resolves per grant source and a
team source deliberately carries none of the caller's object_permission, so a toolset
narrowing layered on top would evaporate on every team-granted server: the request would
be admitted through the toolset grant and then served tools from servers the toolset
never named. A request carrying a toolset name stays on the caller's own credential,
exactly as it did before the swap
* fix(mcp): commit every async connect-page write against the list as it stands
Three continuations in the panel decided against state captured before their await and
committed after it, so a reachability refetch landing in between could not be seen
handleToggle validated the server at click time and then, once listMCPTools resolved,
wrote its name into the selection whatever the list had since become; a server the
refresh had dropped was selected anyway. It now re-asks connectableNow at the commit,
and that predicate resolves the id against the current list, so absence fails closed
instead of reading as nothing to block
The load pipeline was worse, because its cancel flag was shared across runs: the
successor's effect body reset it to false before the predecessor's fetch resolved, so a
superseded load could still run setServers and put the dropped server back on the page
outright. The flag is now a per-effect local that only that run's cleanup can clear,
which is also what makes unmount stop the chunked tool-count loop again. The load
passes its own liveness check down to the tool-count and oauth-status writes rather
than having them consult a flag they share with every other run
* fix(mcp): write the connect-page server list to its ref as it is committed
connectableNow resolves a server id against serversRef, but that ref was a mirror kept
in step by a passive effect, so it lagged the state it mirrored by however long React
took to render and flush. A continuation resolving inside that window read the previous
list: the commit-time reachability check would find a server the refetch had already
dropped, call it connectable, and select it, which is the mismatch the check exists to
prevent
The lag was the whole defect, so the mirror is gone. commitServers writes the ref and
the state together, at the one point the list is ever replaced, and the ref is now
never older than the last committed list. Readers that want the newest answer
(connectableNow, the oauth auto-select effect) get it; rendering still derives from
state, so what is on screen is unchanged
Pinned by a test that resolves the refetch and the in-flight Connect in the same tick,
with no render flushed between them, which is the interleaving the earlier regression
could not reach. The two prop mirrors are deliberately untouched: their staleness is
inherent to appending to a parent-owned list from an async callback rather than caused
by the mirror, and no reachability decision reads them
Greptile found that the OpenAPI fallback added a commit ago collapsed operation
IDs that registration keeps apart: foo/bar and foo.bar register as two tools but
sanitize_openapi_tool_name rewrites both to foo_bar, so a policy naming either
also decided the other.
The cause was two owners for one map, and picking the wrong one. Registration
names an operationId inline at _register_openapi_tools with
operation_id.replace(" ", "_").lower(), which keeps / and . ; the separate
sanitize_openapi_tool_name replaces every character outside [a-zA-Z0-9_-] and
belongs to register_tools_from_openapi, which has no production caller. Nothing
made the matcher use the one that actually registers, so it used the lookalike.
That inline expression is now openapi_tool_name in utils, and both registration
and the matcher call it. Replaying the registering function is the whole safety
argument, and it is structural rather than a claim: two operationIds that
register as two tools normalize to two names here by construction, because this
is the map that registered them. A coarser lookalike cannot be substituted
without a test failing.
The matcher also loses its exact-then-fallback split. The transform is identity
on native servers and idempotent on already-registered names, so normalizing
both sides is exact matching where no OpenAPI spec is involved. Executable lines
drop by three this round; the branch is +5 over the merge-base for four shared
owners that removed duplication at six call sites.
The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
Greptile flagged that match_known_tool_name case-folded both the configured
entry and the derived spellings. Routing keeps two tools whose names differ
only in case as two tools, so folding merged identities the dispatcher
separates: on a server exposing getPet and getpet, an allowlist naming getPet
also granted getpet, and a blocklist naming getPet also denied getpet. That is
unauthorized execution on one arm and the wrong tool denied on the other.
Matching is now exact, which is what identity means here. The case leniency it
replaces was never typo tolerance; _register_openapi_tools rewrites every
operationId through sanitize_openapi_tool_name, so an allowed_tools entry
holding the spec's own spelling never equals the registered name. That link is
recovered by replaying the same rewrite, and only on servers that carry a
spec_path, which is how the rest of the manager already recognizes an OpenAPI
server. Every name that rewrite produces is lowercased, so no two tools on such
a server can differ only in case and the fold cannot merge anything.
Native servers get no folding at all. test_case_folding_applies_to_openapi_
servers_and_not_to_native_ones pins both halves, and two tests pin that a
policy naming one tool leaves its case-variant sibling alone. Dropping the
spec_path guard, dropping the fold, and forcing the fold path are all killed.
The two pre-existing case-insensitivity tests describe OpenAPI servers in their
own docstrings but built fixtures without a spec_path, a shape production never
produces for one; they now set it.
Bugbot flagged REST listing advertising key/team grants that tools/call then
refuses. The listing side was fixed by routing through
filter_tools_by_key_team_permissions, but the two paths still answered the
question with separate implementations that only happened to agree: listing
stripped the known prefix and compared bare, dispatch compared whatever name it
was handed, and each carried its own reading of None and of an empty list.
Changing either side silently diverges from the other, which is how this defect
appeared in the first place.
MCPRequestHandler.tool_is_granted owns the whole decision, and both
is_tool_allowed_for_server and filter_tools_by_key_team_permissions read it.
None still means no tool-level restriction and an empty list still grants
nothing, now stated once. Grants are stored bare by every writer, so matching
stays exact against the bare name, deliberately unlike the server-level lists,
which honor every spelling routing accepts.
test_key_team_listing_and_dispatch_agree drives both production paths over one
matrix. It asserts the expected verdict as well as the agreement, because two
paths reading one predicate makes equality alone tautological: a wrong
predicate keeps them consistent and the agreement assertion alone survived two
mutants that the verdict assertion kills.
The allow list, the deny list, allowed_params and the discovery filter all ask
the same question, "which configured entry names this tool on this server", and
each answered it in its own idiom: any() over a spelling tuple, all() over the
same tuple negated, a next() that pulled a value out of a dict, and a
lowercased set membership. Two review findings on this PR were symptoms of that
duplication. Deriving the operands differently at one site produced the
over-strip; needing a value rather than a boolean at another produced a
truthiness test that read an explicitly empty allowed_params list as "nothing
configured" and allowed every parameter.
match_known_tool_name returns the matching entry or None, and all four sites
read it, so no site can test a container's values to decide membership and the
empty-list fail-open is no longer representable. Matching is case-insensitive
everywhere, which closes the last divergence between discovery and dispatch: a
case-variant disallowed_tools entry used to hide a tool from tools/list while
tools/call still executed it.
Executable lines over the merge-base drop from +9 to +4, all of it the new
owner; mcp_server_manager.py loses 12 lines and the discovery filter loses 17.
The REST listing filter matched key/team grants through _tool_name_matches, which after the prefix-boundary change answers for every spelling routing accepts. Key-level entries in mcp_tool_permissions and toolset rows name a tool on one server and dispatch compares them bare, so a wire-form entry advertised a tool that tools/call then refused. REST listing now goes through filter_tools_by_key_team_permissions, the same function the MCP list path uses.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The gateway publishes a tool as `<server prefix><separator><tool name>` and has to
recover that boundary on the way back in, to compare a called name against a toolset
or allow/deny list and to rebuild the native name sent upstream. Several sites
recovered it by cutting at the FIRST separator and others reconstructed it by hand
from `MCPServer.name` with a literal `-`, so both disagreed with the prefix the
server actually publishes
`get_server_prefix` publishes short_prefix, then alias, then server_name, then
server_id; it never reads `name`. A server with no alias therefore publishes its
hyphen-filled UUID `server_id` as the prefix, and cutting at the first separator
leaves most of the UUID glued to the tool name. Every comparison against the stored
`(server_id, tool_name)` toolset row then misses: an allowlist denies a tool the list
endpoint just advertised, and a disallowed entry stops blocking, which fails open
Recover the boundary in one place instead. `match_known_server_prefix` matches a name
against the server's registered prefixes, longest first so a prefix that itself
contains the separator beats a shorter prefix that is merely its leading segment, and
returns None when the name carries none of them. `strip_known_server_prefix` and
`is_tool_name_prefixed` both delegate to it, and the sites that receive a wire name
call the owner rather than re-deriving the boundary. `split_server_prefix_from_name`
stays for the routing pair it was written for, with a docstring saying so
The server-level permission checks are the other half. They run after the boundary is
already resolved, so their input is bare and the correction there is to derive the
wire form rather than strip it back out; stripping a stored entry a second time cuts a
boundary the caller already consumed, which breaks a native name that itself opens
with the server prefix. Deriving from `get_server_prefix` alone is not enough either,
because routing resolves an inbound name against every prefix from
`iter_known_server_prefixes`, so enforcement keyed to the published spelling answers
for fewer names than are reachable. Turning `LITELLM_USE_SHORT_MCP_TOOL_PREFIX` on
republishes every tool under the short ID while an entry stored under the alias stays
routable and silently stops being enforced, which is a fail-open on a config nobody
edited. `iter_known_tool_name_spellings` yields the bare name plus the wire form under
each accepted prefix, and the allow list, the deny list, `allowed_params` and the
routing map that `_create_prefixed_tools` builds now all key off that one function, so
the set of names enforcement honors and the set routing accepts cannot drift apart
`_tool_name_matches` takes the server as a required argument, so a future caller
cannot silently fall back to guessing, and it matches against that same spelling set,
so `tools/list` hides exactly what dispatch refuses. Answering for fewer spellings in
the filter than enforcement honors leaves a blocked tool advertised, which is how the
alias-form entry above stayed listed even once the call was refused. The OpenAPI
registry lookup builds its key the same way registration does, via `add_server_prefix_to_name` and `get_server_prefix`,
because registration used exactly one key; a server whose `name` differs from its
published prefix stops missing its own tools
* feat(s3): support SSE-KMS encryption params on both S3 logging paths
* fix(s3): ignore non-string SSE config values instead of crashing logger init
* Update litellm/integrations/s3.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): invalidate only the mistyped SSE field instead of dropping both
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
The ComplexityRouter's LLM classifier saw only the last user message, so on a
multi-turn conversation it classified whatever happened to be last rather than
what the human actually asked, and a near-constant classifier input pinned a whole
session to one tier.
The blindness turned out to be narrower than first diagnosed, and the fix is
correspondingly smaller. Tool output was never the problem: on the Messages surface
it rides a user turn as tool_result content blocks, which are not text parts, so
flattening to `type == "text"` already dropped those turns; on chat completions it
arrives on a `tool` role the extractor never read. Both surfaces were already
handled before this change. What actually leaked through was the harness
`<system-reminder>` block, which arrives as ordinary text, survives flattening, and
became the current ask on any turn that carried one.
So reminders are stripped rather than used to reject the turn, because a harness
injects them alongside the live ask and not as a turn of their own; rejecting the
turn would lose the ask, and keeping the block would feed the classifier the
near-constant boilerplate that flattens tier selection in the first place. An
earlier revision of this change also pattern-matched serialized tool_result
payloads. That check only ever fired on a hand-serialized string neither request
surface produces, it was where every review finding in this PR lived, and it is
deleted here; the tests now pin the real shapes instead of the synthetic one they
were built on.
The classifier call is split into a system role carrying the rubric plus the
caller's own system prompt, which stays byte-stable across a session so a provider
can prompt-cache it, and a user role carrying the variable context: a bounded
window of prior user turns, a conversation-depth signal, and the current ask. The
caller's system prompt rides every turn, so task constraints are never dropped.
The depth signal measures content-parts messages too, since counting only string
content reported ~0 tokens for exactly the deep Messages-surface conversations that
most need an expensive tier, and it is omitted entirely on the prompt-only path
rather than asserting a false zero.
Prior turns are excluded by matching the current ask rather than by dropping the
newest turn positionally, because `aclassify` takes `prompt` and `messages`
separately and a caller may classify something other than the newest turn.
Truncated turns carry a marker so the classifier can tell a turn was clipped.
Only the LLM classifier's input changes. The heuristic scorer, keyword overrides,
escalation matching and semantic embedding still read the extracted current ask,
which is why that extraction has to yield one clean human-authored string: those
are substring and vector matchers, and an escalation keyword sitting inside a
reminder blob would otherwise trip a tier jump on its own.
Defaults keep single-turn classification equivalent to before. The prior-turn
window is on by default so existing LLM-classifier deployments actually get the
fix; the config field documents that those turns reach the classifier model, which
may be a different provider than the routed completion model, and that the call
already carries the current ask and the caller's system prompt in full.
Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
* fix(anthropic): split mixed reasoning stream chunks
* style: use builtin generic annotation
* fix(anthropic): split mixed stream chunks by payload kind
The mixed-chunk split cleared only the fields it knew about on each
deep-copied piece, so any other payload riding the chunk survived on
both pieces: tool_calls were emitted as two tool_use blocks with the
same id, thinking_blocks on the text piece emitted duplicated thinking
into a text block while dropping the answer text, and chunks whose
reasoning arrived only as thinking_blocks never split at all
Rebuild each piece's delta from scratch with exactly one payload kind
(reasoning, text, tool calls), ordered to match native Anthropic block
order. Fresh Delta construction keeps unset attributes deleted, which
matters because the translators branch on hasattr, and prevents future
Delta fields from riding along on every piece
* fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once
Adversarial verification against the merge-base found three shapes where
the payload-kind split changed behavior beyond its target: a mixed chunk
carrying a tool argument continuation was torn into a truncated block
plus a fabricated one, a multi-choice chunk lost its secondary choices'
payload, and a signature-less thinking_blocks piece inherited the
non-empty block start body so accumulators collected the thinking twice
Continuation and multi-choice chunks now pass through the splitter
untouched, matching the merge-base byte for byte, and signature-less
thinking_blocks pieces are normalized to reasoning_content so the block
start opens empty and the thinking text is emitted exactly once
---------
Co-authored-by: Napuh <naamanynadiemas@gmail.com>
The Default User Settings form on Internal Users, the org settings form and
the org create dialog all rendered their money fields as
`<input type="number" step={0.01}>` inside a form that never opted out of
native constraint validation. Any value with more than two decimals, such as
a 0.001 max budget, failed the browser's step check, so Chrome vetoed the
submit before react-hook-form ran. No request went out, no field error was
shown, and the read view kept displaying the old value; it looked like the
budget silently refused to stick.
Money fields now use `step="any"`, and the three react-hook-form forms carry
`noValidate` so zod stays the only validator and a DOM-level constraint can
never swallow a submit again.
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
The batch rate limiter counts input tokens by awaiting litellm.afile_content
with no timeout, so a slow Files API holds POST /v1/batches open past any
client deadline; stage saw 63.6s against the harness's 60s read timeout. The
test times out before reaching the unattributed-spend-row assertion it exists
to guard, so it reports an infrastructure hang rather than the contract.
Skipping keeps the signal honest until the fetch is bounded.
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
The keyless flow (gateway as authorization server, no virtual key) worked
only at the aggregate /mcp scope: the session-bearer admission arm was
gated on _is_aggregate_mcp_scope, the 401 fallback only challenged at
aggregate scope, and per-server protected-resource metadata for plain
oauth2 servers pointed clients at the per-server relay, whose flow
returns the raw upstream token that ingress can never accept keylessly
(401 "LiteLLM Virtual Key expected. Received=gho_****").
Per-server spellings now join the same gateway flow for gateway-managed
oauth2 servers (auth_type oauth2 without delegate_auth_to_upstream, new
MCPServer.is_gateway_managed_oauth2 owner):
- the session-bearer arm admits at any MCP scope; downstream grant
resolution already intersects the admitted subject's servers with the
path or header targets fail-closed, so a narrower scope never broadens
- the 401 challenge is scope-aware: a single gateway-managed oauth2 path
target gets the per-server resource_metadata in the spelling the
request used, everything else gets the aggregate document; unknown
names, CSV multi-target paths, and every client-forwarded or delegated
mode keep their existing behavior
- per-server PRM for explicitly named gateway-managed oauth2 servers
advertises the gateway AS ({base}/mcp); delegate, passthrough, bridge,
OBO, and the root-resolved unnamed shape are byte-identical
- the preemptive 401 for an admitted keyless subject with no vaulted
token challenges with resource_metadata (re-entering the gateway flow,
whose authorize interlude vaults the upstream token) instead of the
relay authorization_uri, which cannot vault without a litellm key
The per-server challenge URL builder moved from server.py to
oauth_utils.py (shared with the auth module) and now inserts the
SERVER_ROOT_PATH segment exactly as the discovery routes do.
Resolves LIT-4864
The single-element tuple loop that bound the extracted deployment model
inside the comprehension read poorly; an assignment expression in the
filter clause does the same call-once-and-filter in one line.
Bytez and OCI param maps raise on stream_options when drop_params is
unset, so the default injection would have broken every streamed chat
completion routed to them. Injection now only happens when every router
deployment behind the requested model (wildcards and aliases included)
declares stream_options in its supported OpenAI params; providers that
do not declare it either reject the param or already stream usage
natively, so skipping them keeps old behavior instead of erroring.
_litellm_strip_stream_usage arriving in the client request body is now
overwritten at ingress (and popped in the experimental queue endpoint),
so a client can no longer suppress the usage chunk it explicitly
requested by planting the internal marker.
Streamed chat completions that did not opt into stream_options.include_usage
were logged with tiktoken estimates over the visible text, so hidden
reasoning tokens (billed as output by OpenAI-compatible providers) were
never counted and SpendLogs could undercount output tokens by 90%+ on
reasoning models. The proxy now injects include_usage upstream for
/v1/chat/completions streams by default and strips the injection artifacts
(the final usage chunk and the empty prompt-filter chunk) from the
client-facing SSE stream, so accounting uses provider-billed usage while
the client-visible stream stays byte-identical to today.
always_include_stream_usage keeps its existing semantics: true forwards
the usage chunk to clients as before, and an explicit false now acts as a
kill switch that disables the upstream injection for OpenAI-compatible
backends that reject stream_options.
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
