An MCP permission level answers which servers and tools it permits, and a
level that answers nothing places no restriction. Key auth was reading a
lookup FAULT as that same answer, so the end user, agent and org ceilings
quietly disappeared for as long as one lasted, while the keyless
gateway-admitted path failed closed on the very same fault.
Those levels now separate the two fault classes the user level already
did. A principal row that NAMES an object_permission_id whose contents
cannot be read is a known entitlement with unknown contents, so it denies.
A lookup that fails before we can tell whether the principal is entitled
at all still places no ceiling, that being the state which existed before
the level did; denying there would refuse MCP to the majority of callers,
who have no such entitlement configured. The keyless path is unchanged.
Resolves LIT-4960
Managed files and batches are provider-owned. Cross-model fallbacks can dispatch creation with credentials that cannot access the input file and replace the owning provider's validation error.\n\nCloses #35359
* fix(mcp): annotate connected-app reachability on the gateway connect page
The MCP connect page resolved its server grid through the dashboard identity
(admin shortcut or view_all returns the whole registry) while the gateway DCR
session it sets up resolves servers as an admitted subject through grant
sources only, so the page showed servers and tool counts the session is never
served. GET /v1/mcp/server now accepts connected_app_view=true and stamps each
returned server with connected_app_reachable, computed by the same
_reload_admitted_user + get_allowed_mcp_servers pair the live session uses.
The connect page requests the flag in connect mode and renders unreachable
servers dimmed with a label, excluded from the Connected count and tool-count
fetches. Failure to build the admitted set marks everything unreachable, which
matches what such a session would actually be served. Default behavior without
the param is unchanged for every existing consumer.
* fix(mcp): block connecting unavailable servers from the connect-mode detail view
A server the connect page marks unavailable could still be added through its
detail view Connect action, so the selection could contain servers the
connected-app session is never served. The unavailability decision now lives in
one predicate, connectUnavailabilityLabel, consumed by the card indicator, the
detail view action area, the toggle-on path, the oauth auto-select effect, and
the Connected count, so no interaction path can disagree with the label. This
also closes the same pre-existing hole for servers marked not supported on this
connection, whose detail view likewise offered Connect, and removes a
grandfathered nested ternary, ratcheting the eslint suppressions baseline down
* fix(mcp): hide unreachable servers on the connect page instead of dimming them
Product decision: the connect page should only show what a connected-app
session will actually be served, so annotated-unreachable servers are now
filtered out of the connect-mode list at fetch time rather than rendered
dimmed. Unsupported auth types keep their existing dimmed label since they are
a property of the server, not the caller. A user with zero reachable servers
gets an explanatory empty state pointing at grants. The list filter is the
single source: counts, tabs, auto-select, detail view, and tool-count fetches
all derive from the already-filtered state
* fix(mcp): guarantee the connect view lists every session-reachable server
The connect view's membership came from the dashboard resolver with the
admitted-subject answer only annotated on top, so a server reachable by the
session but missing from the dashboard list would be invisible on the page; an
under-report, the mirror of the bug this PR fixes. The connect view now unions
in any session-reachable server the dashboard resolver did not list, built
from the registry and redacted through the same ladder, so page membership
equals the admitted set by construction in both directions
* fix(mcp): honor connected_app_view only for the dashboard UI session credential
The reachability view resolves through the owning user's admitted identity, so
a caller-passed virtual key could use the param to enumerate servers beyond
its own scope (ids, names, descriptions of the owner's wider grants). The view
is now gated on is_ui_session_credential, a predicate factored out of
resolve_ui_session_team_ids so the two user-identity widening sites share one
trust boundary: the SSO-minted dashboard session token acting as its user. Any
other credential gets the param as a no-op and the admitted resolver is never
consulted for it
* fix(mcp): resolve UI sessions with the admitted-user context everywhere, not per endpoint
The list endpoint unioned in session-reachable servers itself while tool
counts, Connect actions, and credential endpoints still authorized through
build_effective_auth_contexts, whose contexts carry team grants but never the
user row's own object permission; a user-granted server could render on the
connect page while every interaction on it failed. The admitted-user context
(the same auth a gateway session resolves with) is now appended inside
build_effective_auth_contexts for UI session credentials, so the page list and
every per-server action endpoint answer identically, and the list endpoint's
one-off union is deleted. Caller-passed keys are still never widened
(is_ui_session_credential gate inside the context builder) and a reload
failure falls back to team contexts only
* fix(mcp): resolve non-admin dashboard sessions as the admitted subject on tool routes
Server reachability on the REST tool routes came from the widened context
union while tool permission checks ran on the bare session key, which carries
no object permission, so a dashboard user could invoke tools their user-level
grant excludes. Rather than bookkeeping which context granted which server,
the routes now choose one principal at the boundary: acting_user_auth swaps a
non-admin UI session for the admitted-subject auth, the same identity a
gateway session resolves with, so reachability, per-source fail-closed tool
ceilings, rate limits, and billing attribution all bind through the admitted
arms that already exist downstream. Admin sessions keep their operator view
and caller-passed credentials are never widened. One swap point per route,
no per-server principal picking, no parallel permission logic
* fix(mcp): derive the connect page's detail view from the reachable server list
The detail view held its own copy of the server object, so it outlived the list it came
from. When a refetch dropped that server as unreachable, the open detail view kept
rendering it and its Connect action still ran: the guard looked the server back up by id
or name in the current list, found nothing, and fell through, because a missing target
read as "nothing to block" rather than "no longer connectable"
Store the selected server's id and derive the row from the list instead. A server the
list no longer carries cannot be the detail view's subject, so the stale render, the
stale tools query and the guard bypass stop being reachable states rather than being
blocked one at a time. handleToggle now takes the server it is toggling, which deletes
the lookup that could miss at all
* refactor(mcp): one owner for the identity a dashboard session acts as
Three call sites reloaded the admitted subject independently, and the management
endpoint carried its own copy of the reload, the HTTPException swallow and the logging.
admitted_user_context is now the only place that answers "what user identity does this
dashboard session act as", and the connected-app reachability helper reads it, which
also drops its dead empty-user_id branch
That owner now carries the request's tracing span onto the admitted principal.
_reload_admitted_user builds a fresh auth from the user row and has no span of its own,
so swapping it in on the REST tool routes silently detached every downstream lookup and
the tool-call logging from the request's trace
Toolset scoping and the acting-as-user swap are mutually exclusive, so they now share
one owner on the tools list route. The admitted subject resolves per grant source and a
team source deliberately carries none of the caller's object_permission, so a toolset
narrowing layered on top would evaporate on every team-granted server: the request would
be admitted through the toolset grant and then served tools from servers the toolset
never named. A request carrying a toolset name stays on the caller's own credential,
exactly as it did before the swap
* fix(mcp): commit every async connect-page write against the list as it stands
Three continuations in the panel decided against state captured before their await and
committed after it, so a reachability refetch landing in between could not be seen
handleToggle validated the server at click time and then, once listMCPTools resolved,
wrote its name into the selection whatever the list had since become; a server the
refresh had dropped was selected anyway. It now re-asks connectableNow at the commit,
and that predicate resolves the id against the current list, so absence fails closed
instead of reading as nothing to block
The load pipeline was worse, because its cancel flag was shared across runs: the
successor's effect body reset it to false before the predecessor's fetch resolved, so a
superseded load could still run setServers and put the dropped server back on the page
outright. The flag is now a per-effect local that only that run's cleanup can clear,
which is also what makes unmount stop the chunked tool-count loop again. The load
passes its own liveness check down to the tool-count and oauth-status writes rather
than having them consult a flag they share with every other run
* fix(mcp): write the connect-page server list to its ref as it is committed
connectableNow resolves a server id against serversRef, but that ref was a mirror kept
in step by a passive effect, so it lagged the state it mirrored by however long React
took to render and flush. A continuation resolving inside that window read the previous
list: the commit-time reachability check would find a server the refetch had already
dropped, call it connectable, and select it, which is the mismatch the check exists to
prevent
The lag was the whole defect, so the mirror is gone. commitServers writes the ref and
the state together, at the one point the list is ever replaced, and the ref is now
never older than the last committed list. Readers that want the newest answer
(connectableNow, the oauth auto-select effect) get it; rendering still derives from
state, so what is on screen is unchanged
Pinned by a test that resolves the refetch and the in-flight Connect in the same tick,
with no render flushed between them, which is the interleaving the earlier regression
could not reach. The two prop mirrors are deliberately untouched: their staleness is
inherent to appending to a parent-owned list from an async callback rather than caused
by the mirror, and no reachability decision reads them
Greptile found that the OpenAPI fallback added a commit ago collapsed operation
IDs that registration keeps apart: foo/bar and foo.bar register as two tools but
sanitize_openapi_tool_name rewrites both to foo_bar, so a policy naming either
also decided the other.
The cause was two owners for one map, and picking the wrong one. Registration
names an operationId inline at _register_openapi_tools with
operation_id.replace(" ", "_").lower(), which keeps / and . ; the separate
sanitize_openapi_tool_name replaces every character outside [a-zA-Z0-9_-] and
belongs to register_tools_from_openapi, which has no production caller. Nothing
made the matcher use the one that actually registers, so it used the lookalike.
That inline expression is now openapi_tool_name in utils, and both registration
and the matcher call it. Replaying the registering function is the whole safety
argument, and it is structural rather than a claim: two operationIds that
register as two tools normalize to two names here by construction, because this
is the map that registered them. A coarser lookalike cannot be substituted
without a test failing.
The matcher also loses its exact-then-fallback split. The transform is identity
on native servers and idempotent on already-registered names, so normalizing
both sides is exact matching where no OpenAPI spec is involved. Executable lines
drop by three this round; the branch is +5 over the merge-base for four shared
owners that removed duplication at six call sites.
Greptile flagged that match_known_tool_name case-folded both the configured
entry and the derived spellings. Routing keeps two tools whose names differ
only in case as two tools, so folding merged identities the dispatcher
separates: on a server exposing getPet and getpet, an allowlist naming getPet
also granted getpet, and a blocklist naming getPet also denied getpet. That is
unauthorized execution on one arm and the wrong tool denied on the other.
Matching is now exact, which is what identity means here. The case leniency it
replaces was never typo tolerance; _register_openapi_tools rewrites every
operationId through sanitize_openapi_tool_name, so an allowed_tools entry
holding the spec's own spelling never equals the registered name. That link is
recovered by replaying the same rewrite, and only on servers that carry a
spec_path, which is how the rest of the manager already recognizes an OpenAPI
server. Every name that rewrite produces is lowercased, so no two tools on such
a server can differ only in case and the fold cannot merge anything.
Native servers get no folding at all. test_case_folding_applies_to_openapi_
servers_and_not_to_native_ones pins both halves, and two tests pin that a
policy naming one tool leaves its case-variant sibling alone. Dropping the
spec_path guard, dropping the fold, and forcing the fold path are all killed.
The two pre-existing case-insensitivity tests describe OpenAPI servers in their
own docstrings but built fixtures without a spec_path, a shape production never
produces for one; they now set it.
Bugbot flagged REST listing advertising key/team grants that tools/call then
refuses. The listing side was fixed by routing through
filter_tools_by_key_team_permissions, but the two paths still answered the
question with separate implementations that only happened to agree: listing
stripped the known prefix and compared bare, dispatch compared whatever name it
was handed, and each carried its own reading of None and of an empty list.
Changing either side silently diverges from the other, which is how this defect
appeared in the first place.
MCPRequestHandler.tool_is_granted owns the whole decision, and both
is_tool_allowed_for_server and filter_tools_by_key_team_permissions read it.
None still means no tool-level restriction and an empty list still grants
nothing, now stated once. Grants are stored bare by every writer, so matching
stays exact against the bare name, deliberately unlike the server-level lists,
which honor every spelling routing accepts.
test_key_team_listing_and_dispatch_agree drives both production paths over one
matrix. It asserts the expected verdict as well as the agreement, because two
paths reading one predicate makes equality alone tautological: a wrong
predicate keeps them consistent and the agreement assertion alone survived two
mutants that the verdict assertion kills.
The allow list, the deny list, allowed_params and the discovery filter all ask
the same question, "which configured entry names this tool on this server", and
each answered it in its own idiom: any() over a spelling tuple, all() over the
same tuple negated, a next() that pulled a value out of a dict, and a
lowercased set membership. Two review findings on this PR were symptoms of that
duplication. Deriving the operands differently at one site produced the
over-strip; needing a value rather than a boolean at another produced a
truthiness test that read an explicitly empty allowed_params list as "nothing
configured" and allowed every parameter.
match_known_tool_name returns the matching entry or None, and all four sites
read it, so no site can test a container's values to decide membership and the
empty-list fail-open is no longer representable. Matching is case-insensitive
everywhere, which closes the last divergence between discovery and dispatch: a
case-variant disallowed_tools entry used to hide a tool from tools/list while
tools/call still executed it.
Executable lines over the merge-base drop from +9 to +4, all of it the new
owner; mcp_server_manager.py loses 12 lines and the discovery filter loses 17.
The gateway publishes a tool as `<server prefix><separator><tool name>` and has to
recover that boundary on the way back in, to compare a called name against a toolset
or allow/deny list and to rebuild the native name sent upstream. Several sites
recovered it by cutting at the FIRST separator and others reconstructed it by hand
from `MCPServer.name` with a literal `-`, so both disagreed with the prefix the
server actually publishes
`get_server_prefix` publishes short_prefix, then alias, then server_name, then
server_id; it never reads `name`. A server with no alias therefore publishes its
hyphen-filled UUID `server_id` as the prefix, and cutting at the first separator
leaves most of the UUID glued to the tool name. Every comparison against the stored
`(server_id, tool_name)` toolset row then misses: an allowlist denies a tool the list
endpoint just advertised, and a disallowed entry stops blocking, which fails open
Recover the boundary in one place instead. `match_known_server_prefix` matches a name
against the server's registered prefixes, longest first so a prefix that itself
contains the separator beats a shorter prefix that is merely its leading segment, and
returns None when the name carries none of them. `strip_known_server_prefix` and
`is_tool_name_prefixed` both delegate to it, and the sites that receive a wire name
call the owner rather than re-deriving the boundary. `split_server_prefix_from_name`
stays for the routing pair it was written for, with a docstring saying so
The server-level permission checks are the other half. They run after the boundary is
already resolved, so their input is bare and the correction there is to derive the
wire form rather than strip it back out; stripping a stored entry a second time cuts a
boundary the caller already consumed, which breaks a native name that itself opens
with the server prefix. Deriving from `get_server_prefix` alone is not enough either,
because routing resolves an inbound name against every prefix from
`iter_known_server_prefixes`, so enforcement keyed to the published spelling answers
for fewer names than are reachable. Turning `LITELLM_USE_SHORT_MCP_TOOL_PREFIX` on
republishes every tool under the short ID while an entry stored under the alias stays
routable and silently stops being enforced, which is a fail-open on a config nobody
edited. `iter_known_tool_name_spellings` yields the bare name plus the wire form under
each accepted prefix, and the allow list, the deny list, `allowed_params` and the
routing map that `_create_prefixed_tools` builds now all key off that one function, so
the set of names enforcement honors and the set routing accepts cannot drift apart
`_tool_name_matches` takes the server as a required argument, so a future caller
cannot silently fall back to guessing, and it matches against that same spelling set,
so `tools/list` hides exactly what dispatch refuses. Answering for fewer spellings in
the filter than enforcement honors leaves a blocked tool advertised, which is how the
alias-form entry above stayed listed even once the call was refused. The OpenAPI
registry lookup builds its key the same way registration does, via `add_server_prefix_to_name` and `get_server_prefix`,
because registration used exactly one key; a server whose `name` differs from its
published prefix stops missing its own tools
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
The keyless flow (gateway as authorization server, no virtual key) worked
only at the aggregate /mcp scope: the session-bearer admission arm was
gated on _is_aggregate_mcp_scope, the 401 fallback only challenged at
aggregate scope, and per-server protected-resource metadata for plain
oauth2 servers pointed clients at the per-server relay, whose flow
returns the raw upstream token that ingress can never accept keylessly
(401 "LiteLLM Virtual Key expected. Received=gho_****").
Per-server spellings now join the same gateway flow for gateway-managed
oauth2 servers (auth_type oauth2 without delegate_auth_to_upstream, new
MCPServer.is_gateway_managed_oauth2 owner):
- the session-bearer arm admits at any MCP scope; downstream grant
resolution already intersects the admitted subject's servers with the
path or header targets fail-closed, so a narrower scope never broadens
- the 401 challenge is scope-aware: a single gateway-managed oauth2 path
target gets the per-server resource_metadata in the spelling the
request used, everything else gets the aggregate document; unknown
names, CSV multi-target paths, and every client-forwarded or delegated
mode keep their existing behavior
- per-server PRM for explicitly named gateway-managed oauth2 servers
advertises the gateway AS ({base}/mcp); delegate, passthrough, bridge,
OBO, and the root-resolved unnamed shape are byte-identical
- the preemptive 401 for an admitted keyless subject with no vaulted
token challenges with resource_metadata (re-entering the gateway flow,
whose authorize interlude vaults the upstream token) instead of the
relay authorization_uri, which cannot vault without a litellm key
The per-server challenge URL builder moved from server.py to
oauth_utils.py (shared with the auth module) and now inserts the
SERVER_ROOT_PATH segment exactly as the discovery routes do.
Resolves LIT-4864
Bytez and OCI param maps raise on stream_options when drop_params is
unset, so the default injection would have broken every streamed chat
completion routed to them. Injection now only happens when every router
deployment behind the requested model (wildcards and aliases included)
declares stream_options in its supported OpenAI params; providers that
do not declare it either reject the param or already stream usage
natively, so skipping them keeps old behavior instead of erroring.
_litellm_strip_stream_usage arriving in the client request body is now
overwritten at ingress (and popped in the experimental queue endpoint),
so a client can no longer suppress the usage chunk it explicitly
requested by planting the internal marker.
Streamed chat completions that did not opt into stream_options.include_usage
were logged with tiktoken estimates over the visible text, so hidden
reasoning tokens (billed as output by OpenAI-compatible providers) were
never counted and SpendLogs could undercount output tokens by 90%+ on
reasoning models. The proxy now injects include_usage upstream for
/v1/chat/completions streams by default and strips the injection artifacts
(the final usage chunk and the empty prompt-filter chunk) from the
client-facing SSE stream, so accounting uses provider-billed usage while
the client-visible stream stays byte-identical to today.
always_include_stream_usage keeps its existing semantics: true forwards
the usage chunk to clients as before, and an explicit false now acts as a
kill switch that disables the upstream injection for OpenAI-compatible
backends that reject stream_options.
/tag/list returned HTTP 500 for every internal user with
"LiteLLM_VerificationTokenActions.find_many() got an unexpected keyword
argument 'select'". The non-admin branch scopes the tag list to keys owned
by the caller, and that lookup passed select={"token": True}; prisma-client-py
0.11.0 has no select kwarg on find_many, so the call raised TypeError and the
handler's except block turned it into a 500. Since the Admin UI calls
/tag/list on load, Tags was broken for every non-admin user.
/tag/daily/activity shares the same helper and was failing the same way
The kwarg is dropped rather than replaced; the generated client has no
projection API, and a user's key set is small enough that selecting all
columns is not worth working around
The reason this shipped green is that the existing test asserted the call was
made with select={"token": True} against an AsyncMock, which accepts any
keyword. The verification-token table double now binds each call against the
real find_many signature, so an unsupported kwarg raises the same TypeError
production does
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
Logical replication consumers need FULL replica identity to reconstruct the
old row of an UPDATE or DELETE, and prisma leaves every table it creates at
the postgres default. Operators had to re-apply the setting by hand after
each migration run.
Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM
table at the end of a successful migration run, through the prisma CLI so the
dependency-free proxy-extras package stays that way. Tables that are already
FULL are skipped, foreign tables in the same schema are left alone, and a
database that refuses the ALTER is reported rather than failing the run.
Resolves LIT-3022
Guardrails could only see the MCP tool call request (pre_mcp_call /
during_mcp_call); the tool result went back to the client unscanned, so a tool
that returns sensitive data bypassed every configured guardrail.
Adds a `post_mcp_call` event hook that runs after the tool executes and routes
the result through the unified apply_guardrail seam, so a text guardrail (e.g.
presidio) can mask sensitive values in the tool output or reject the result
without any MCP-specific code of its own.
- MCPGuardrailTranslationHandler.process_output_response now extracts the tool
result's text content into GenericGuardrailAPIInputs["texts"], calls
apply_guardrail with input_type="response", and writes the returned text back
into the content list in place (the logging payload already references that
object, so a copy would leave the unmasked text in the spend log)
- ProxyLogging.post_mcp_call_hook dispatches guardrails that implement
apply_guardrail, gated on should_run_guardrail(post_mcp_call); guardrails
implementing async_post_mcp_tool_call_hook keep their existing dispatch and
are not run twice
- both MCP tool-call paths (mcp_server and the Responses API handler) now honor
the rewritten result, and the REST path no longer swallows a guardrail
rejection as a logging failure
- shared, duck-typed MCP content helpers live in mcp_server/utils.py next to
extract_mcp_tool_result_error_message
- documents that async_post_mcp_tool_call_hook's return value is discarded by
every call site, so that hook only takes effect by mutating in place
The v3 parallel-request limiter stashed its per-request bookkeeping (TPM
reservation, descriptors, parallel slot, rate-limit response snapshot,
released flag) in the request body's metadata channels. On routes where
metadata is a provider request parameter (Responses API and the other
LITELLM_METADATA_ROUTES) that leaked internal keys upstream and produced
HTTP 400s, and it required denylist stripping plus dual-channel writes to
contain.
The stash now lives on an asyncio ContextVar holding a single typed
RequestRateLimiterStash per request. The pre-call hook writes it, and the
success/failure callbacks, disconnect release, and post-call hooks read
and clear the same shared instance, which keeps the refund and slot
release idempotent across sibling callbacks. The request body is never
touched, so the stash-key stripping, the metadata mirror writes, and the
all_litellm_params denylist entries are removed
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
The MCP gateway resolved a caller's allowed servers and per-server tool
allowlists from the key, the team, the end user and the agent, but never from
the internal user row, so an admin had no way to bound what a person may call
across every key they hold. Anything the key allowed went through
The internal user now carries the same object_permission an admin already
attaches to a key or a team, and the resolver applies it as a ceiling: the
caller ends up with the intersection of what the key allows and what the user
allows, so adding a user entitlement can only narrow, never widen. A level
that names no server and no tool places no ceiling, which keeps every existing
deployment on its current behavior
/user/new and /user/update accept object_permission and reuse the same
create-or-update helper the team endpoints use, so the row is written once and
the three cached views of it (the user row, the object-permission link and the
permission itself) are invalidated on write. Clearing it with an empty object
now really unlinks the permission instead of being swallowed as an empty value
A row that cannot be read at all places no ceiling, but a row that names a
permission the database cannot return denies the call rather than falling
through to the wider set, so a partial outage cannot hand out access the admin
withheld
The users page grows the MCP servers, access groups, toolsets and per-server
tool pickers the key and team pages already have. A save keeps a tool
allowlist whenever an access group or toolset the admin retained could still
supply that server, since an allowlist is what narrows a grant and an absent
one reads as no restriction; it drops the allowlist once nothing indirect
survives to supply the server, so removing a grant really removes it
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.
The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.
Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.
Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.
routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
Completed-batch cost tracking parsed the whole output file into a list of
dicts, pretty-printed it into debug strings even with debug logging off, and
walked the list three times (cost, usage, models), so a large batch output
could pin a worker's memory. The output is now folded line by line into small
per-line stats records via _aggregate_batch_cost_usage_models, the eager
json.dumps debug calls are gone, and the raw-vertex path computes cost and
usage in one call instead of two. _get_batch_output_file_content_as_dictionary
becomes _fetch_batch_output_file_content (returns bytes); the superseded
three-pass helpers are deleted and their tests migrated
Auto-routers had no home and no list. The create form was mounted in two unrelated places,
inside Models + Endpoints > Add Model and again under Cost Optimization, and neither showed
which auto routers already existed; seeing or editing one meant finding its row in the models
table and drilling in. They now get a dedicated Auto-Routers tab beside All Models, listing
every auto_router/* deployment with create, edit and delete in one place, and both former
entry points are removed.
Creating opens in a shadcn dialog rather than swapping the whole panel out, so the list stays
on screen behind it; the dialog caps its height and scrolls, since the complexity form is long.
The form's own heading goes with it, the dialog header owning that now.
An auto router is a routing construct rather than a deployment, so it also comes off the All
Models table. That table pages server-side off total_count, so a client-side filter would page
over a total including rows it never renders; /v2/model/info therefore gains
exclude_auto_routers (default false, so every existing caller is unaffected) and the filter
runs before the count. /v1/models is untouched, so clients still see auto-routers as models.
Clicking a router opens the same `?model=` drill-in the All Models table uses, so it lands in
ModelInfoView with the full Model Settings, Edit Settings, Edit Auto Router and Delete. An
earlier revision had a bespoke detail page here, which was a partial reimplementation of that
view and showed the router's type twice, once as a Type pill and again as a "Routing strategy"
field saying the same thing. Both are gone.
The auto-router list is keyed under the same `models/list` namespace as the models table
rather than a private one. It reads the same /v2/model/info data, and six call sites across
the app already invalidate ["models","list"] after a write; a separate key meant an edit made
through ModelInfoView left the tab stale until a full reload, and every future writer would
have had to remember a second key.
An auto router has no upstream credential, so its detail header drops Update API Key and
Re-use Credentials, and the destructive action names what it removes rather than saying model.
Test Connection was gated on the editor-aware predicate, which let adaptive and quality routers
through to a check that builds its targets from complexity config they do not have; it now
gates on the deployment predicate.
The edit modal also applies the semantic-matching guard the create form has. It renders those
controls now, and the backend raises on semantic_keyword_matching without an embedding model or
keyword rules, so skipping the shared validator turned an inline message into a raw 400.
Whether a row is writable has two independent axes and the dashboard needs both. STRATEGY:
there are four auto_router/* kinds and only complexity and semantic have a form here, so
adaptive and quality must not be handed an editor that would write auto_router_config onto a
deployment storing its settings elsewhere. ORIGIN: a config.yaml row reports db_model false and
the API refuses it whatever its strategy (PATCH /model/{id}/update 404s, POST /model/delete
400s). Capability is derived per capability rather than as one editable flag, because the
constraints differ: editing needs an editor, deleting removes a row by id and never reads its
config, so a DB-created adaptive router stays deletable. Both axes live in
add_model/auto_router_strategies.ts as a declarative table, one record per strategy, so a fifth
strategy is a table row rather than another branch. That also retired four copies of "is this a
complexity router", one of which was written twice in a row in model_info_view.
Creation narrows to the complexity router, which the UI calls Auto-Router v2; the semantic
option was already badged "to be deprecated" in the picker, so the picker goes away along with
the semantic submit path and its validation helper. Existing semantic routers stay editable.
The edit modal mounted ComplexityRouterConfig without the keyword, escalation and
semantic-matching handlers, so those sections never rendered and could only be set at create
time. It now hydrates them from the stored config, and the five keys become managed only when a
caller supplies that state, so a caller rendering no such control still carries them through. A
component-level round-trip test covers it: a payload-builder test cannot see a hydration bug.
A complexity tier is str | list[str] on the backend, and the UI carried three readers of that
rule, one of which dropped a pinned string. They collapse into one owner,
add_model/complexity_router_tiers.ts.
