The responses API surface still routed tool calls by server name. It built
the caller's reachable set as MCPServer objects, narrowed by both the key's
grants and the requested server filter, then discarded that identity by
flattening to display names. `tool_server_map` carried a name, and dispatch
re-resolved it with `get_mcp_server_by_name`, which walks the whole registry
and returns the first match. Server names are not unique, so a tool listed
from a reachable server could dispatch to a same-named server the caller
cannot reach, sending that server's upstream credential.
That is the bug this PR already fixed for MCP JSON-RPC, on the one surface
that had not been converted. The fix is the same: stop discarding identity.
`tool_server_map` now carries the server_id resolved within the caller's
reachable set through `resolve_tool_route`, the same scoped resolver the
JSON-RPC path uses, so a name two reachable servers share stays ambiguous
here too rather than silently picking one. Dispatch looks the server up by
id. A tool with no reachable owner fails closed and reports a result for its
tool call, matching how every other failure in that loop is surfaced, rather
than being dropped.
`resolved_server` is a parameter this PR introduced, and it let a caller
hand `call_tool` any server at all. That is the same class of defect one
layer down, so the check belongs at the chokepoint rather than at each
caller: `call_tool` now takes the reachable set the server was resolved
against and refuses to dispatch outside it, covering the caller's server and
one it resolves by name itself, which also walks the whole registry. Supplying
`resolved_server` without that set is rejected, so caller-supplied identity
always arrives with its provenance. Both callers already computed the set, so
nothing recomputes it.
Registering a server's tool routes was union-only, so a routing row could
only ever gain owners. Nothing withdrew a tool while its server stayed in
the registry: `_cleanup_server_tool_routing_artifacts` withdraws a server's
id from every row, but it only runs when the server leaves the registry.
So an upstream that stopped exposing a tool left its owner pinned. A name
then served by exactly one reachable server kept resolving as ambiguous and
kept returning the 409, with no way back short of restarting the proxy.
A server's `tools/list` result is its complete listing, so it is the truth
rather than an increment. `_replace_server_tool_routes` replaces one
server's rows instead of accumulating into them, withdrawing its id from a
name it no longer serves and dropping the row once no owner is left. Rows
still accumulate across servers, so a genuinely shared name stays ambiguous.
Treating a listing as the truth is safe because `_fetch_tools_with_timeout`
raises on every failure instead of returning an empty list, and
caller-scoped narrowing (`check_allowed_or_banned_tools`, semantic
filtering) runs downstream, so neither a failed listing nor one caller's
filtered view can evict routes another caller needs.
Eviction is the same operation with an empty set, so it delegates rather
than keeping its own copy of the withdrawal arithmetic. An earlier cut of
this carried a per-server reverse index to avoid scanning the map, which
bought under a millisecond on a path that has just made a network round
trip, in exchange for a second source of truth that can disagree with the
first. It already had: pointing eviction at the index broke
`test_update_server_eviction_clears_openapi_routing_artifacts`, which seeds
the mapping directly and asserts eviction clears rows however they were
written. The scan is exhaustive by construction, so that class is gone.
The OpenAPI path replaces once the whole spec has parsed, so a mid-loop
failure leaves the previous routes intact rather than committing a partial
set that would withdraw operations still served. The startup warm-up no
longer re-registers what listing already recorded, which would have re-added
names outside the replace and reintroduced rows it cannot withdraw.
The OpenAPI registration call site turned out to have no test coverage at
all; deleting it left the suite green. It now has three, including the
mid-parse failure case.
_get_mcp_server_from_tool_name had two competing sources of truth: the ownership map,
and a first-wins prefix-to-server scan of the registry. The prefix path ignored the map's
ambiguity, so a prefixed name owned by two servers sharing a prefix resolved to whichever
was scanned first instead of None, breaking the function's own contract. It also never
checked that the prefix-named server actually owned the tool.
Resolution now goes through the ownership map first: a registered tool name with one owner
resolves by id and with several owners returns None. Only a name that is not itself
registered falls to prefix extraction, and that path resolves to the one server that both
matches the prefix and owns the underlying tool, so a shared prefix or a prefix naming a
non-owner both stay unresolved. This removes the class of arbitrary-owner resolution that
callers were each guarding against individually.
The duplicate-prefix regression test set only name, not server_name, so the prefix was
never recognized and the prefix path was never exercised; it now sets server_name and a
sibling test covers a prefix that names a server which does not own the tool.
resolve_tool_route fails closed when a tool's only owners are outside the caller's
scope, but execute_mcp_tool then re-resolved a still-unset server with a scope-blind
lookup so BYOK and credential injection could run on every path. On the JSON-RPC and
tool-search paths that fallback undid the scope decision: because no server_name was set
for an out-of-scope tool, the permission gate was skipped, and the fallback dispatched to
the out-of-scope server with its credentials. The fallback now only accepts a server the
caller is already allowed to reach, matching how the requested_server lookups are guarded
by the server-id mismatch check.
Cleanup of a departed server's routes withdrew its id per row but did so by clearing the
whole mapping and repopulating it, which momentarily emptied the shared dict the
un-awaited initialize task writes into. It now withdraws the id key by key, so the dict is
never empty and never rebound.
* feat: add Straiker guardrail integration
Implements LLM security guardrails via Straiker with prompt and response inspection, multi-mode execution (pre_call, post_call), and configurable blocking or redaction of flagged content across providers, streaming, images, and tool calls.
* fix(guardrails): harden straiker source attribution and error-path consistency
Use the operator-configured source for Straiker application attribution instead of a caller-supplied agent_id metadata value, so a caller cannot spoof which application a detection is attributed to. Make _fail reuse _block so a post_call error raises ModifyResponseException like a deliberate post_call block rather than GuardrailRaisedException, and type the blocking helper as NoReturn so the type checker enforces that execution never falls through the BLOCKED branch. Serialize the webhook payload once and send it as raw content to avoid re-serializing on the size check and on every retry.
* fix(guardrails): read straiker config and metadata from all supported shapes
Handle a dict optional_params in _get_config_value so nested guardrail
settings loaded from YAML or the DB (timeout, unreachable_fallback, and
the rest) are applied instead of silently falling back to defaults;
previously only attribute-style access was supported. Build the webhook
metadata bag from the merged metadata so client tags stored under
litellm_metadata on routes like /v1/messages reach Straiker the same way
identity and application fields already do, and widen the internal-key
skip prefix to user_api so proxy-injected budget values are not
forwarded.
* fix(guardrails): fail safe on straiker interventions without redactions
Block instead of passing content through when Straiker returns
GUARDRAIL_INTERVENED without replacement texts, so a positive
intervention verdict can never silently forward the original flagged
content. Fix the streamed-request detection to read the request body
from proxy_server_request.body, where the proxy stores it, instead of a
top-level body key that is never populated; the previous fallback was
dead, so a streamed response whose stream flag was not lifted to the top
level would have been redacted rather than blocked while buffering
replayed the original chunks.
* revert(guardrails): restore straiker caller agent_id application attribution
Restore the original behavior where a request-scoped agent_id in metadata
sets the Straiker application source, falling back to the configured
source. This is the integration's intended per-application attribution;
litellm already resolves a key-owned agent_id ahead of any caller-supplied
value, so a configured key cannot be spoofed.
* revert(guardrails): restore straiker webhook metadata scoping
Restore the original behavior where the Straiker webhook metadata bag is
built from request-scoped metadata only. Forwarding litellm_metadata was
a scope change to what the integration sends to Straiker; keep the
author's intended scoping.
* fix(guardrails): keep proxy key material out of straiker webhook metadata
Widen the internal-key skip prefix from user_api_key_ to user_api so the
proxy-injected user_api_key hash and user_api_end_user_max_budget are not
copied into the Straiker webhook metadata bag. The narrower prefix missed
the bare user_api_key name, leaking the hashed key to the vendor. Keeps
the request-scoped metadata source unchanged.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
The proxy already returns x-litellm-model-id (the deployment id) and x-litellm-model-group (the requested model-group alias), but never surfaces the concrete underlying model that served the request; the router rewrites the response model field to the group alias, so callers had no way to read the actual deployment model like anthropic/claude-haiku-4-5. Expose it as x-litellm-model-name, sourced from the deployment recorded in litellm_params metadata.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info
Resolves the per-model get_model_group_info fan-out on GET /v1/models
(and /models) that pegged the event loop on wildcard listings (#33636).
create_model_info_response now reads max_input_tokens/max_output_tokens
from litellm.get_model_info (the static cost map) rather than the router,
which aggregated and deepcopied every deployment in a group per listed
model.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage
Inject the cost-map lookup (defaulting to litellm.get_model_info) so the
except and max_output_tokens branches are exercised deterministically and
the token-limit tests no longer hardcode mutable cost-map values.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup
Add Router.get_configured_token_limits, an O(1) model-name index lookup that
reads a concrete deployment's configured max_input_tokens/max_output_tokens
without triggering pattern matching or deep copies. create_model_info_response
layers this over the cost map so custom deployments absent from the cost map
still surface their limits, and admin-configured limits override cost-map
defaults, while wildcard-expanded names stay on the fast path.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Config.yaml-declared OAuth2 MCP servers using Dynamic Client Registration have no LiteLLM_MCPServerTable row, so the DCR persist path called update_mcp_server, which returns None for a missing row, then update_server(None), which dereferenced .approval_status and raised AttributeError. The exception was swallowed to a warning while /register still returned 200, so the minted client was never stored and every access-token expiry forced a full re-authorization
Persist the acquired DCR client (client_id, client_secret, token_endpoint_auth_method, redirect_uris, encrypted at rest) in a dedicated LiteLLM_MCPServerOAuthClient store keyed by server_id when the server has no row, overlay it onto the in-memory config server so the refresh_token grant can authenticate within the process, and rehydrate it when the registry syncs from the database (which runs after the DB connects, unlike config load) so restarts and other pods pick it up. The store is encrypted at rest and is re-encrypted by the master-key rotation path alongside the server rows, through a shared helper so the two sites cannot diverge. The DB-backed server path is unchanged, and guarding the None return removes the swallowed-crash footgun
Resolves the config.yaml DCR persistence regression introduced in v1.92.0 by #31912
enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are set as live
litellm attributes on the worker that handles the UI save, exactly like
budget_exceeded_throttle_percentage, but they were missing from
LITELLM_SETTINGS_SAFE_DB_OVERRIDES, so a peer worker's config reload merged the DB
value without applying it to the live attribute and stayed stale.
Add both to the allowlist so they behave like the sibling field, and add
test_general_settings_ui_fields_are_db_overridable so the UI registry and the
override allowlist cannot drift again (the exact omission that caused this), plus
a regression test that the flag flips on a simulated peer-worker reload.
PR #33736 made the shielded streaming cleanup await
proxy_logging_obj._arelease_max_parallel_requests_on_disconnect on the
client-disconnect path. The four streaming cancel and disconnect tests in
test_budget_reservation.py drive the generator with a bare MagicMock as
proxy_logging_obj, so the cleanup crashed with TypeError: object MagicMock
can't be used in 'await' expression, breaking proxy-infra CI on every PR
Give the mocks an AsyncMock for the release method and assert it is awaited
exactly once on each disconnect path, pinning the single-owner slot release
contract that PR #33736 introduced without test coverage
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes#33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in c698b88686 to match upstream litellm_internal_staging.
Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update.
* fix: scan system messages as indirect prompt injection in Singulr guardrail
* chore: restore lint budget files to upstream baseline
* fix: resolve ruff UP006 and I001 violations in singulr guardrail
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* resolve review comments on Singulr guardrail
* fix: scan tool call results as indirect prompt injection in Singulr guardrail
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* minor
* formating fix
* refactor: shift extraction logic to singulr side
* refactor:keep precall hook only
* fix:formatting
* fix:linting
* improve config description
* Trigger CI
* fix
* fix:field description
* fix:errors due to change in field names
* style: apply ruff line-wrap formatting to singulr guardrail
* fix:exception
* fix:formatting
* fix playground
* improved
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix
* fix ci issues
* remove uv.lock from pr
* fix
* fix:resolved comments
* chore: trigger CI
* remove uv.lock
* fix
* fix linting
* fix linting
* fix linting
* remove doc strings
* remove test fixes
* chore: retrigger CI
* change in singulr api contract
* remove some ut
* send litellm call_id to singulr
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Fix non-conformant UUIDv7 generation in native Opik integration (#31294)
create_uuid7() encoded the timestamp in units of 16 seconds instead of
milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's
backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067);
the bad encoding decoded to ~year 2201 and every trace/span batch was
rejected with HTTP 400.
Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms),
using the standard library only so no new dependency is added. Add unit
tests covering UUIDv7 validity and millisecond timestamp encoding.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(proxy): expose uvicorn concurrency limit (#33077)
Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and
LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and
accepted connections and returns HTTP 503 once the configured limit is reached.
Reject non-positive limits at CLI parse time and only add the setting to the
uvicorn startup arguments. Because idle connections also consume capacity,
deployments should use upstream connection/header timeouts and per-client
connection limits.
* test: reorder test_utils tail to keep the daily merge conflict-free (#33788)
The daily OSS branch and litellm_internal_staging each appended an
independent test block at the very end of tests/test_litellm/test_utils.py,
so merging the two collides on that shared end-of-file position even though
the additions are unrelated (this branch adds the vertex embedding
encoding-format tests; staging adds the per-model prompt-cache-minimum
tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class
above test_gemini_image_models_do_not_support_reasoning, which both branches
share, gives the two additions different anchors, so git applies both
without a conflict and without pulling staging into this branch. Pure
reorder; no test bodies change
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
Outcome keys in the tools/list _meta, the spend-log outcome and count maps, and the REST error
messages now all use get_server_prefix (alias, or the short prefix when that mode is enabled), the
same naming the caller already sees on tool names. Keying them by canonical server_name let an
authenticated caller enumerate internal server names and their health or auth state that the alias
and short-prefix schemes deliberately hide (Veria finding). One helper decides the key for every
surface; exception messages reaching the multi-server REST error list are mapped to their fault tag
with the display prefix instead of relaying exception text carrying canonical names. Server-side
logs keep the real names
* fix(proxy): bill partial streamed spend when the client disconnects mid-stream
* fix(router): guard FallbackStreamWrapper chunks alias for non-CSW streams
* fix(proxy): await disconnect billing dispatch instead of unrooted create_task
* fix(proxy): make disconnect slot release single-owner to avoid double release
* fix(proxy): use union syntax for disconnect cleanup params (UP045 budget)
Conflict in _list_mcp_tools: staging (#33612) moved toolset-grant expansion into the shared
permission primitives and removed the _merge_toolset_permissions call; resolution applies that
removal to this branch's AggregateToolListing structure
Rather than mixing the flag and its ttl into the generic General settings table
(which also surfaced the confusing Not Set / In Config / In DB provenance badges),
give prompt caching a dedicated tab with a purpose-built toggle and ttl dropdown.
Each registry field gains an optional tab, surfaced as ConfigList.field_tab, so
the General tab renders the ungrouped fields and the caching fields render on
their own tab. The update, persist and reset endpoints are unchanged.
Register enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl on the
General Settings table so caching can be turned on without hand-writing config.
The registry could not express either field: validation was hardcoded to a float in
(0, 1], reset set every field to None (not a bool for a boolean flag), and the listing
reported any non-None value as 'In Config', which a False default would always trip.
Validation now dispatches on the declared type and reset restores each field's own
default. ConfigList carries field_options so the table can render a Select for enums
instead of no editor at all.
Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome
tests and the mode-aware preemptive-401 tests from staging; both kept
resolve_tool_route fell through to a scope-blind lookup when the tool had owners but
none intersected the caller's allowed set, so a caller scoped to one server could
resolve a tool served only by another. The downstream name-based permission check would
then misroute the call to a reachable server sharing that name. The scoped branch is now
authoritative: known owners with none in scope returns not_found rather than resolving a
server the caller cannot reach.
Also seed the logging tests from the specific server under test instead of every
manager server id, so adding a second fixture server cannot make a tool look ambiguous.
* fix(proxy): stop treating upstream model body field as a LiteLLM model on auth-enforced pass-through routes
An auth: true user-defined pass-through endpoint runs full virtual-key auth, and get_model_from_request unconditionally extracted the request body model field, so key/team/user/project model allowlist checks rejected requests whose model only exists upstream (key_model_access_denied), even when the key was explicitly granted the route via allowed_passthrough_routes.
The pass-through route registry moves to a leaf module (route_registry.py) that the auth layer can import without re-entering the pass_through_endpoints -> user_api_key_auth -> auth_utils import cycle. get_model_from_request now returns None for routes registered as user-defined pass-through endpoints (exact and subpath), which skips model allowlist and per-model budget enforcement on those routes while key auth, allowed_passthrough_routes, and spend/budget checks stay intact. Built-in provider passthrough routes (/vertex_ai, /gemini, ...) keep model enforcement.
Resolves LIT-4299
* fix(proxy): key pass-through model-access skip on the dispatched endpoint, not the request path
Addresses a model-authorization bypass: the first version decided whether to skip
model-allowlist extraction by matching the request path against the pass-through
route registry. That ignored the HTTP method and, more importantly, whether the
request was actually dispatched to a pass-through handler. A custom pass-through
whose path collides with a built-in route (e.g. /v1/chat/completions, or an
include_subpath prefix of one) still writes a registry entry even though FastAPI
serves the built-in handler, so a normal request to that route had its model checks
skipped and could reach a model outside the key/team/user/project allowlist.
The skip is now keyed off the FastAPI-resolved endpoint. create_pass_through_route
tags its handler with LITELLM_PASS_THROUGH_ENDPOINT_MARKER, and get_model_from_request
returns None only when request.scope["endpoint"] carries that marker. Because routing
runs before auth dependencies, this reflects the handler that actually serves the
request: on a collision the built-in handler is dispatched and carries no marker, so
model enforcement stays on. This also removes the need for the separate route_registry
module, so that extraction is reverted.
Regression tests cover a pass-through-dispatched request (model suppressed), a
built-in-dispatched request on the same path (model still enforced), and the no-request
budget path.
Resolves LIT-4299
* fix(proxy): enforce max_parallel_requests as a per-slot concurrency gauge
The v3 rate limiter tracked max_parallel_requests with the same
sliding-window machinery as RPM/TPM. A concurrency gauge cannot live on a
windowed counter: every window roll reset the counter to 1 while requests
were still in flight, the completion decrements for those forgotten
requests then drove the counter negative, and rejected requests left
stranded increments that nothing released. Under sustained load a key with
max_parallel_requests=5 let backend concurrency climb to the full client
concurrency (observed 60 on a live proxy) while the proxy kept returning
429s for everyone else
Replace the windowed counter with a per-slot registry (Redis sorted set of
slot ids scored by acquire time, with an asyncio-locked in-memory fallback):
admission atomically prunes expired slots and registers a new slot id only
when in_flight + 1 <= limit, so rejected requests never occupy a slot;
success, failure, and client-disconnect paths release exactly the slot id
this request acquired (stashed in the request metadata channels), so a
release without a matching acquire or a double-fired callback can never
free another request's slot; and a slot leaked by a crashed worker is
pruned individually after its TTL even under continuous traffic
Resolves LIT-4259
Fixes#16011
* fix(proxy): release every acquired gauge and respect mirrored counts in the in-memory fallback
Address review findings on the slot-registry gauge: the acquisition stash
now carries the gauge counter keys alongside the slot id, so the release
paths free the slot from every gauge it was registered under instead of
hardcoding the api_key scope, and the disconnect release keys off the
stashed acquisition instead of the key object's current
max_parallel_requests configuration (which can change mid-request). The
in-memory fallback now treats a cached integer (the count mirrored from
the last successful Redis script call) as real occupancy, carrying it
forward as a floored counter during a Redis outage instead of restarting
from an empty registry
* fix(proxy): release the parallel slot on proxy-level rejections
async_post_call_failure_hook is the only callback that fires when a
downstream hook (guardrail, budget check) rejects a request after the rate
limiter's pre-call hook acquired a slot; async_log_failure_event is a
completion-level callback and never runs for proxy-side rejections.
Release the stashed acquisition at the top of the hook, before the TPM
reservation guard, so those slots do not linger for the full slot TTL and
wedge the key at its limit under moderate rejection rates. Clearing the
acquisition marker keeps the release idempotent when a later failure
callback runs in the same flow
* test(proxy): cover success release, read-only count, Redis release mirror, and TPM rejection release
Four behaviors of the slot-registry gauge had no direct test: a successful
completion releasing exactly its acquired slot, read_only callers counting
in-flight slots through the count script (and degrading to the local
mirror when the script fails) without acquiring, the Redis release script
mirroring returned counts into the local cache, and the TPM reservation
rejection releasing the already-acquired slot before raising
* style(proxy): use builtin generics and union syntax in new rate limiter annotations
The slot-gauge code added Tuple/List/Dict and Optional[...] annotations, pushing
the UP006 and UP045 strict-rule totals past their ceilings in ruff-strict-budget.json.
Convert only the annotations this branch introduces to builtin generics and PEP 604
unions, leaving the rest of the module untouched.
* fix(router): tag-aware pre-routing strategy selection for shared model_name
Complexity/auto/adaptive/quality router registries were keyed by model_name
alone, so a second deployment sharing a model_name but carrying different tags
was rejected and every request used the first config. This made tag-based
routing to distinct provider configs behind one alias impossible, surfacing as
401 'Not allowed to access model due to tags configuration' for the second tag.
Each registry now holds a list of tag-scoped strategies and async_pre_routing_hook
selects the entry whose tags match the request before classification, falling
back to a default-tagged then first-registered entry. A repeat of the same
(model_name, tags) pair is still rejected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover tag-scoped pre-routing strategy registry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: re-trigger CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
server.py selected the route variant with isinstance. A module reload rebinds the
manager and its classes, so resolve_tool_route returned a reloaded variant while
server.py still held the pre-reload class; isinstance was False, the ambiguity branch
never ran, and the call fell through to the local registry as a 404 instead of the
intended 409. Each variant now carries a Literal kind tag and server.py dispatches on
it, which is what makes this a tagged union rather than one that leans on class
identity.
Withdrawing a server's routes also rebuilt the mapping into a fresh dict, orphaning any
holder of the previous one. The initialize task is fired without being awaited and can
be holding it, so the rebuild now updates the mapping in place.
tool_name_to_mcp_server_name_mapping mapped a tool name to a server name and was
written with an unqualified, global, last-writer-wins key. Two server entries sharing
an upstream URL expose the same tool names, so that key always collided and the winner
was whichever server registered last, which meant config declaration order decided
which upstream credential left the gateway.
server_name is not a sound identity: schema.prisma has no unique constraint on
server_name or alias, and the create endpoint only checks server_id. The registry is
already keyed by server_id and config servers derive a deterministic one, so server_id
was always the real identity.
The map now holds frozensets of server ids and accumulates owners instead of
overwriting them, so a name served by several reachable servers stays visibly
ambiguous and is rejected rather than dispatched to an arbitrary server. Ambiguity is
judged against the servers the caller can reach, so a session scoped with
x-mcp-servers still resolves its unprefixed names. All three writers now go through one
registration chokepoint; they previously disagreed on what the value was. Cleanup
withdraws only the departing server's id from each row instead of matching by name,
which stopped removing one server from deleting a same-named server's routes.
execute_mcp_tool now dispatches the server it resolved rather than passing a name that
call_tool re-resolved, closing the same drop on the responses API path.
The consolidation regressed the pre-existing walker semantics: _extract_upstream_auth_failure used
to keep scanning until it found a 401/403, while the consolidated helper took the first response of
any status and then tested it, so a causal 401 sitting behind an unrelated 5xx (retry attempts,
multi-stream task groups) was misclassified as upstream_error and its challenge lost on the listing,
tool-call, and probe paths. The traversal is now an iterator in deliberate order and each consumer
applies its predicate over the stream: the auth scan takes the first 401/403 even behind non-auth
responses, generic classification takes the first response, and classify_list_exception derives its
auth arm from the same scan so the carrier choice and the classification can never disagree
A `lite login` token 429'd with "Budget has been exceeded! Max budget:
0.25" even when no budget was configured anywhere. cli_poll_key stamped
the minted CLI session token with litellm.max_ui_session_budget ($0.25)
as a fallback whenever the user and team had no budget of their own. That
cap was designed for the Admin UI "Test Key" chat pane; the CLI reused
the same session-token machinery, so it inherited a playground-sized
budget baked into the encrypted token at login (unchangeable without
re-login), which trips fast under real CLI/agent use.
The cap is also redundant: the token already carries user_id and team_id,
so the real user/team budgets are enforced independently at request time.
Pass max_budget=None so the CLI token is governed only by those real
budgets, and drop the now-dead user/team budget lookups. The UI login
token's guard (get_experimental_ui_login_jwt_auth_token) is untouched.
Grafted from PR #33582 (closing as superseded by this PR): drives
handle_streamable_http_mcp with real MCPServer objects, parametrized over a
stamped client_credentials row and a legacy unstamped M2M-shape row; both must
reach the session manager without the per-user token store being consulted
The preemptive-401 gate for auth_type=oauth2 MCP servers keyed the challenge
on whether an Authorization header was present (not oauth2_headers). Because
the header parser classifies any Authorization bearer as an OAuth token before
the target server is resolved, a LiteLLM virtual key presented as
Authorization: Bearer sk-... suppressed the challenge on a gateway-managed
authorization_code server; the session then opened with no upstream token and
tools/list masked the failure as 200 with an empty tool list. The same gate
also wrongly challenged client_credentials (M2M) servers, which the gateway
authenticates by minting its own token at egress.
The decision is per oauth2 sub-mode, not per header. Gateway-managed modes
never receive a client-supplied upstream token: client_credentials mints at
egress so it is never challenged, and gateway-managed interactive
(authorization_code, non-delegate) is challenged whenever no stored per-user
token exists, regardless of any bearer. Only the delegate/upstream-PKCE mode,
where a present bearer genuinely is the upstream token, keeps keying on the
Authorization header. oauth2_headers itself is left untouched so the
delegate/passthrough egress paths that forward the client bearer are
unchanged.
* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through
* fix(model_armor): wire skip_unscannable_attachments through guardrail config
* fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping
* fix(model_armor): remove the per-request attachment count cap and scan all attachments
---------
Co-authored-by: yucheng <yucheng@berri.ai>
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
Team/key router_settings.enable_tag_filtering was stored and echoed by
/team/info but never applied at request time: the per-request override
whitelist in route_llm_request.py dropped it, tag filtering only read the
router-level flag, and UpdateRouterConfig silently discarded the field on
/key/generate and /config/update. Requests from teams with the toggle on
were load balanced across all deployments instead of tag-matched ones.
- add enable_tag_filtering to the router_settings_override whitelist and
strip any client-supplied copy from the request body first, so only the
key/team value reaches the router
- run tag filtering when the request carries enable_tag_filtering=True; a
request-level False cannot disable a router-level True, so per-request
settings can only scope down, never escape the global policy
- add the field to UpdateRouterConfig so key and config update paths stop
dropping it, and to all_litellm_params so it never leaks into provider
request bodies
- allow it through Router.update_settings/get_settings so the global UI
toggle persists across DB config reloads
Resolves LIT-4390
