* fix(proxy): bill partial streamed spend when the client disconnects mid-stream
* fix(router): guard FallbackStreamWrapper chunks alias for non-CSW streams
* fix(proxy): await disconnect billing dispatch instead of unrooted create_task
* fix(proxy): make disconnect slot release single-owner to avoid double release
* fix(proxy): use union syntax for disconnect cleanup params (UP045 budget)
The gpt-realtime family (OpenAI and Azure) only serves /v1/realtime and is rejected by /v1/chat/completions with "This is not a chat model", but the cost map tagged them mode=chat. Retag them mode=realtime (a value already used by gemini-live and handled by the health-check realtime handler) and add realtime to the ModelInfoBase mode literal.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity-router): user-triggered escalation keywords
Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.
Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.
Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity-router): validate escalation keywords and pin at tier ceiling
Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop treating upstream model body field as a LiteLLM model on auth-enforced pass-through routes
An auth: true user-defined pass-through endpoint runs full virtual-key auth, and get_model_from_request unconditionally extracted the request body model field, so key/team/user/project model allowlist checks rejected requests whose model only exists upstream (key_model_access_denied), even when the key was explicitly granted the route via allowed_passthrough_routes.
The pass-through route registry moves to a leaf module (route_registry.py) that the auth layer can import without re-entering the pass_through_endpoints -> user_api_key_auth -> auth_utils import cycle. get_model_from_request now returns None for routes registered as user-defined pass-through endpoints (exact and subpath), which skips model allowlist and per-model budget enforcement on those routes while key auth, allowed_passthrough_routes, and spend/budget checks stay intact. Built-in provider passthrough routes (/vertex_ai, /gemini, ...) keep model enforcement.
Resolves LIT-4299
* fix(proxy): key pass-through model-access skip on the dispatched endpoint, not the request path
Addresses a model-authorization bypass: the first version decided whether to skip
model-allowlist extraction by matching the request path against the pass-through
route registry. That ignored the HTTP method and, more importantly, whether the
request was actually dispatched to a pass-through handler. A custom pass-through
whose path collides with a built-in route (e.g. /v1/chat/completions, or an
include_subpath prefix of one) still writes a registry entry even though FastAPI
serves the built-in handler, so a normal request to that route had its model checks
skipped and could reach a model outside the key/team/user/project allowlist.
The skip is now keyed off the FastAPI-resolved endpoint. create_pass_through_route
tags its handler with LITELLM_PASS_THROUGH_ENDPOINT_MARKER, and get_model_from_request
returns None only when request.scope["endpoint"] carries that marker. Because routing
runs before auth dependencies, this reflects the handler that actually serves the
request: on a collision the built-in handler is dispatched and carries no marker, so
model enforcement stays on. This also removes the need for the separate route_registry
module, so that extraction is reverted.
Regression tests cover a pass-through-dispatched request (model suppressed), a
built-in-dispatched request on the same path (model still enforced), and the no-request
budget path.
Resolves LIT-4299
* fix(proxy): enforce max_parallel_requests as a per-slot concurrency gauge
The v3 rate limiter tracked max_parallel_requests with the same
sliding-window machinery as RPM/TPM. A concurrency gauge cannot live on a
windowed counter: every window roll reset the counter to 1 while requests
were still in flight, the completion decrements for those forgotten
requests then drove the counter negative, and rejected requests left
stranded increments that nothing released. Under sustained load a key with
max_parallel_requests=5 let backend concurrency climb to the full client
concurrency (observed 60 on a live proxy) while the proxy kept returning
429s for everyone else
Replace the windowed counter with a per-slot registry (Redis sorted set of
slot ids scored by acquire time, with an asyncio-locked in-memory fallback):
admission atomically prunes expired slots and registers a new slot id only
when in_flight + 1 <= limit, so rejected requests never occupy a slot;
success, failure, and client-disconnect paths release exactly the slot id
this request acquired (stashed in the request metadata channels), so a
release without a matching acquire or a double-fired callback can never
free another request's slot; and a slot leaked by a crashed worker is
pruned individually after its TTL even under continuous traffic
Resolves LIT-4259
Fixes#16011
* fix(proxy): release every acquired gauge and respect mirrored counts in the in-memory fallback
Address review findings on the slot-registry gauge: the acquisition stash
now carries the gauge counter keys alongside the slot id, so the release
paths free the slot from every gauge it was registered under instead of
hardcoding the api_key scope, and the disconnect release keys off the
stashed acquisition instead of the key object's current
max_parallel_requests configuration (which can change mid-request). The
in-memory fallback now treats a cached integer (the count mirrored from
the last successful Redis script call) as real occupancy, carrying it
forward as a floored counter during a Redis outage instead of restarting
from an empty registry
* fix(proxy): release the parallel slot on proxy-level rejections
async_post_call_failure_hook is the only callback that fires when a
downstream hook (guardrail, budget check) rejects a request after the rate
limiter's pre-call hook acquired a slot; async_log_failure_event is a
completion-level callback and never runs for proxy-side rejections.
Release the stashed acquisition at the top of the hook, before the TPM
reservation guard, so those slots do not linger for the full slot TTL and
wedge the key at its limit under moderate rejection rates. Clearing the
acquisition marker keeps the release idempotent when a later failure
callback runs in the same flow
* test(proxy): cover success release, read-only count, Redis release mirror, and TPM rejection release
Four behaviors of the slot-registry gauge had no direct test: a successful
completion releasing exactly its acquired slot, read_only callers counting
in-flight slots through the count script (and degrading to the local
mirror when the script fails) without acquiring, the Redis release script
mirroring returned counts into the local cache, and the TPM reservation
rejection releasing the already-acquired slot before raising
* style(proxy): use builtin generics and union syntax in new rate limiter annotations
The slot-gauge code added Tuple/List/Dict and Optional[...] annotations, pushing
the UP006 and UP045 strict-rule totals past their ceilings in ruff-strict-budget.json.
Convert only the annotations this branch introduces to builtin generics and PEP 604
unions, leaving the rest of the module untouched.
* fix(router): tag-aware pre-routing strategy selection for shared model_name
Complexity/auto/adaptive/quality router registries were keyed by model_name
alone, so a second deployment sharing a model_name but carrying different tags
was rejected and every request used the first config. This made tag-based
routing to distinct provider configs behind one alias impossible, surfacing as
401 'Not allowed to access model due to tags configuration' for the second tag.
Each registry now holds a list of tag-scoped strategies and async_pre_routing_hook
selects the entry whose tags match the request before classification, falling
back to a default-tagged then first-registered entry. A repeat of the same
(model_name, tags) pair is still rejected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover tag-scoped pre-routing strategy registry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: re-trigger CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The read gate cannot cause a wrong pin. A deployment is only pinned when the cache
already holds an entry for the prefix, and async_log_success_event writes entries
against the deployment's real model rather than the group alias, so a model that
will not cache a prefix never records one and there is nothing to pin it to
That makes this gate purely a cheap short-circuit deciding whether the cache lookup
is worth doing, so the threshold must be the lowest minimum in the group. Taking the
highest skipped the lookup for a prefix a lower-minimum member had genuinely cached,
losing a hit it earned, and protected against nothing. It also broke the Fable 5
direction this ticket is meant to fix: its real minimum is 512, so a group gate stuck
at a higher value would skip the lookup for a prefix Fable 5 had actually cached
get_model_info is lru_cached, so swapping litellm.model_cost is not enough on its
own. An earlier test that resolved these models against the remote map, which does
not carry prompt_cache_min_tokens yet, leaves cached entries without it, and the
stale hit resolves to the default. The assertions would then pass for the wrong
reason or fail depending on execution order
Clear on teardown as well, so entries these tests warm against the local map do not
leak into later tests, matching the fixture already used in test_utils.py
Also pin that a wildcard route resolves the underlying model's minimum. That works
only because pattern_match_deployments substitutes the real model name into
litellm_params before the deployment reaches the check; without the assertion that
claim is unpinned and the threshold would silently fall back to the default
MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of
tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is
per-model and ranges from 512 to 4096, and it can differ per platform for the same
model, so one constant is wrong in both directions
is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what
optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is
cacheable, async_filter_deployments pins routing to whichever deployment previously
served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5
or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider
never cached it, so the pin cost load balancing for nothing. In the other direction
Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it
had earned
The minimum now resolves from prompt_cache_min_tokens in the model cost map, which
keeps it current with new models and lets the Bedrock override for Fable 5 fall out
of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT
stays as a global escape hatch when explicitly set, and as the fallback for models the
cost map has no entry for
async_filter_deployments only ever receives the model group alias, never a model name,
so it resolves the threshold from healthy_deployments instead. A group may mix models
with different minimums, so it takes the max: a prompt is only treated as cacheable when
it clears every member's minimum, because an unnecessary pin is the defect being fixed
while a missed pin only forfeits an optimization
Gemini context caching shares this gate and has the same defect; its entries are left
unset so they keep today's behavior, tracked separately in LIT-4525
_request_has_cache_control only looked at messages and system, so a client that
marks cache_control on tools alone did not suppress auto-injection. Tool
breakpoints count toward the provider's four-block limit, so three of them plus
the two injected here is five, which Anthropic rejects. Thread tools through
both entry points and treat a client-marked tool as the stand-down signal it
already is for messages and system.
A `lite login` token 429'd with "Budget has been exceeded! Max budget:
0.25" even when no budget was configured anywhere. cli_poll_key stamped
the minted CLI session token with litellm.max_ui_session_budget ($0.25)
as a fallback whenever the user and team had no budget of their own. That
cap was designed for the Admin UI "Test Key" chat pane; the CLI reused
the same session-token machinery, so it inherited a playground-sized
budget baked into the encrypted token at login (unchangeable without
re-login), which trips fast under real CLI/agent use.
The cap is also redundant: the token already carries user_id and team_id,
so the real user/team budgets are enforced independently at request time.
Pass max_budget=None so the CLI token is governed only by those real
budgets, and drop the now-dead user/team budget lookups. The UI login
token's guard (get_experimental_ui_login_jwt_auth_token) is untouched.
Grafted from PR #33582 (closing as superseded by this PR): drives
handle_streamable_http_mcp with real MCPServer objects, parametrized over a
stamped client_credentials row and a legacy unstamped M2M-shape row; both must
reach the session manager without the per-user token store being consulted
The preemptive-401 gate for auth_type=oauth2 MCP servers keyed the challenge
on whether an Authorization header was present (not oauth2_headers). Because
the header parser classifies any Authorization bearer as an OAuth token before
the target server is resolved, a LiteLLM virtual key presented as
Authorization: Bearer sk-... suppressed the challenge on a gateway-managed
authorization_code server; the session then opened with no upstream token and
tools/list masked the failure as 200 with an empty tool list. The same gate
also wrongly challenged client_credentials (M2M) servers, which the gateway
authenticates by minting its own token at egress.
The decision is per oauth2 sub-mode, not per header. Gateway-managed modes
never receive a client-supplied upstream token: client_credentials mints at
egress so it is never challenged, and gateway-managed interactive
(authorization_code, non-delegate) is challenged whenever no stored per-user
token exists, regardless of any bearer. Only the delegate/upstream-PKCE mode,
where a present bearer genuinely is the upstream token, keeps keying on the
Authorization header. oauth2_headers itself is left untouched so the
delegate/passthrough egress paths that forward the client bearer are
unchanged.
* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through
* fix(model_armor): wire skip_unscannable_attachments through guardrail config
* fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping
* fix(model_armor): remove the per-request attachment count cap and scan all attachments
---------
Co-authored-by: yucheng <yucheng@berri.ai>
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
Team/key router_settings.enable_tag_filtering was stored and echoed by
/team/info but never applied at request time: the per-request override
whitelist in route_llm_request.py dropped it, tag filtering only read the
router-level flag, and UpdateRouterConfig silently discarded the field on
/key/generate and /config/update. Requests from teams with the toggle on
were load balanced across all deployments instead of tag-matched ones.
- add enable_tag_filtering to the router_settings_override whitelist and
strip any client-supplied copy from the request body first, so only the
key/team value reaches the router
- run tag filtering when the request carries enable_tag_filtering=True; a
request-level False cannot disable a router-level True, so per-request
settings can only scope down, never escape the global policy
- add the field to UpdateRouterConfig so key and config update paths stop
dropping it, and to all_litellm_params so it never leaks into provider
request bodies
- allow it through Router.update_settings/get_settings so the global UI
toggle persists across DB config reloads
Resolves LIT-4390
The hook returned early when the semantic filter selected no tools, which
restated a policy that SemanticMCPToolFilter.filter_tools already owns: it
returns the full tool set when nothing matches, so the selection is never
empty. The branch was unreachable, and reachable or not it changed nothing,
since the gateway reads the union of every reference's allowed_tools and
treats an empty union as unset. Its only effect was to suggest the reference
path and the plain tool path resolve a zero-match query differently.
Drop it so a single policy governs both paths, and pin that with a test
covering an unmatched query on each path. Flipping filter_tools to fail
closed now fails the test on both instead of quietly hard-limiting one
surface and not the other.
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials
Key-scoped langfuse_otel callbacks only injected Authorization headers into the
init-time exporter, so a proxy without global LANGFUSE_* env vars kept its
fallback exporter and never exported traces to Langfuse. Dynamic params now
build a full per-request OTLP config (endpoint from the key's langfuse_host,
otlp_http, basic auth from the key's credentials).
Resolves LIT-3976
* fix(otel): log dynamic config endpoint in span processor debug output
* fix(otel): redact authorization headers in exporter debug logs
* fix(router): honor per-request routing_strategy from key/team router_settings
Key and team router_settings.routing_strategy was stored and shown in the
UI but never forwarded to the shared Router, so the global strategy always
won. Forward it through router_settings_override and resolve it in
_get_routing_context: a validated per-request strategy takes precedence
over routing groups and the top-level strategy, with lazily built cached
selectors for strategies that need one. Unknown or unsupported strategy
values are ignored with a warning instead of failing the request, and
routing_strategy is registered in all_litellm_params so it is stripped
before the provider call.
* fix(router): sweep override selectors on strategy re-init and cover coverage-gate helpers
routing_strategy_init now unregisters cached per-request override
selectors so a later update_settings strategy change cannot leave a
zombie selector receiving callback events. Adds direct tests for the
two new helpers so the router code coverage gate passes.
* docs(team): document mcp_rpm_limit in update_team docstring
The documentation CI job walks management_endpoints and requires every
UpdateTeamRequest field to appear in the update_team docstring;
mcp_rpm_limit was added to the model without a docstring line, failing
the job on unrelated PRs depending on walk order. Regenerates
schema.d.ts since the docstring feeds the OpenAPI spec.
* fix(proxy): coerce default_internal_user_params.max_budget to float on config load
* fix(proxy): log coerced default_internal_user_params and cover absent max_budget in tests
The semantic tool filter replaced each litellm_proxy MCP reference in
data["tools"] with the tools it expanded from that reference. The expansion
defaults to the Responses API tool shape, so a /chat/completions request came
out carrying flat {"type": "function", "name": ...} entries where the provider
transformations expect {"type": "function", "function": {...}}. Anthropic then
raised KeyError: 'function' and Bedrock dropped every MCP tool silently, so the
model answered as if no MCP server were connected.
Replacing the reference also removed the marker the MCP gateway matches on, so
acompletion_with_mcp never ran and tool calls were no longer auto-executed for
require_approval="never", on /responses as well as /chat/completions.
Narrow the reference through allowed_tools instead and leave it in place, so the
gateway still owns expansion and keeps both the per-endpoint tool shape and tool
auto-execution. Expansion already applies any caller-supplied allowed_tools, so
the selection can only narrow a reference further, never widen it.
* fix(proxy_cli): reap orphaned prisma query-engine processes when a worker dies
When the proxy runs multi-worker (uvicorn multiprocess supervisor or the
gunicorn arbiter), a worker that crashes or is force-killed never runs its
in-process atexit cleanup, so its prisma query-engine subprocess reparents
to PID 1 and keeps its database connection pool established forever while
the replacement worker opens a fresh pool. Active DB connections then grow
past database_connection_pool_limit with every worker death.
Run a reaper thread in the supervisor process that marks itself a child
subreaper on Linux, scans for adopted query-engine children whose worker
is gone, and terminates them with SIGTERM escalating to SIGKILL after a
bounded grace period. Engines owned by live workers are children of those
workers, never of the supervisor, so they are structurally out of reach.
Resolves LIT-4449
Fixes https://github.com/BerriAI/litellm/issues/33023
* fix(proxy_cli): address review findings on the query-engine reaper
Make start_query_engine_reaper idempotent, reap simultaneous orphans
under one shared grace period instead of serially, and log when a PID
survives SIGKILL. Also regenerate schema.d.ts for the update_team
docstring line that documents the existing mcp_rpm_limit param (fixes
the walk-order-dependent documentation CI failure) and avoid a cast in
the prctl wrapper
* test(proxy): fix reaper idempotency-test isolation and widen coverage
The daemon-thread startup test now stubs threading.enumerate so a
reaper thread left running by an earlier test in the same xdist worker
cannot satisfy the idempotency guard and skip the code under test. Add
coverage for stat-file truncation, non-numeric ppid, non-child reap,
signal-to-dead-pid, subreaper capability, and reaper-loop resilience
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are
now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and
LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on
without a config file. An unsupported ttl falls back to the provider default
rather than reaching the provider verbatim
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
Cost values read from deployment model_info can be strings when the
config YAML contains scientific notation with an integer mantissa
(e.g. 1e-05), which YAML 1.2 parsers such as PyYAML 6.x treat as a
string. Comparing that string against the running float aggregate in
_set_model_group_info raised TypeError and broke /model_group/info,
the prometheus remaining-usage callback, and the
x-litellm-response-cost header. Coerce input/output cost values to
float before comparing and storing them.
