Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.
The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.
The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.
The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.
The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.
The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.
Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.
raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.
The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.
_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.
When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.
to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.
Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).
V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.
Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).
Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a
cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve
today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and
InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared
DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be).
The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a
reread callback for the cross-replica case (losers re-read the persisted token) that the in-process
default ignores.
The refresh seam took only the OAuthToken, but a refresher needs the server's
config (token endpoint, client credentials, scopes) to run the grant and the
(user_id, server_id) key to persist the minted token, neither of which is
derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id,
token) and pass them through from RefreshingTokenStore so each stacked mode PR
plugs into the final seam rather than forcing a later signature change across
the stack.
The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is
an outlier among OAuth clients. Spring Security uses 60s as both its JWT
clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's
"a few minutes" leeway while preserving nearly all of a typical token's life;
30s was untested, so pin the default with two boundary-probe regression tests.
CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the
inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow
is visible on the next request without an invalidation hook, and uniformly across replicas since
the in-process cache holds no stale None to clear. invalidate() now only covers rotation or
revocation of a cached token. Negative caching (with distributed invalidation) can return later
if a slow DB-backed v2-native source makes per-miss reads expensive.
Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it)
and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while
concurrent callers await the same in-flight task and share its result, so the IdP is not
stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by
in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a
cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces
as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore.
OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica
single-flight (Redis) and reactive-401 refresh are the later distributed hardening.
Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token,
expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for
outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while
unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage
without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and
distributed single-flight are deferred to the hardening step.
The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate
header + optional structured body) instead of a bare string, and raise_public emits the header
and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's
provisioning prompt) through the generic resolver edge. of_unauthorized's new params are
keyword-only and default to None, so existing callers and the summary string are unchanged.
* feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution
MCP per-server IP access control reads the client IP from X-Forwarded-For
and trusts the leftmost entry. Behind an append-style proxy or load
balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy,
Cloudflare), a client can prepend an arbitrary value to the header, so the
leftmost entry is attacker-controllable even when the direct peer is a
trusted proxy. An attacker can therefore spoof an internal IP and reach
servers marked available_on_public_internet=false.
This adds an optional mcp_xff_num_trusted_hops general setting modelled on
Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries
from the right of the chain (where N is the number of trusted appending
proxies in front of the gateway) instead of the leftmost value, so any
entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges,
which still validates the direct peer, and only takes effect once that check
passes; without a validated direct peer the gateway keeps failing closed, so
hop counting cannot be abused by a direct-to-pod attacker. The chain must
contain at least N valid entries or resolution fails closed.
Default is unset, preserving existing behaviour.
* chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops
* fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum
A 0 or negative value is silently treated as disabled, which could leave
an operator believing they enabled append-style X-Forwarded-For hardening
while client IP resolution stays on the spoofable leftmost value. Emit a
warning, consistent with how the module already surfaces invalid CIDR
config, so the misconfiguration is visible in logs.
* fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time
Add a ge=1 bound to the ConfigGeneralSettings field so the
update_config_general_settings path rejects 0 and negative values with a
clear validation error instead of accepting them, and self-documents the
valid range. The runtime warning stays as defense-in-depth for raw-dict
config that bypasses model validation.
* style(mcp): black-format ip_address_utils.py
* fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid
A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1)
previously made _resolve_num_trusted_hops return None, which the caller
treated identically to "unset" and silently fell back to the legacy
leftmost X-Forwarded-For value. An operator who set the value to harden
client IP resolution but typo'd it would get weaker security than before,
with no fail-closed signal.
Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid,
_HopCount) so the three states are distinct: unset keeps the legacy path,
a valid count drives hop-counting, and an invalid value fails closed
(returns "") instead of reverting to the spoofable leftmost address. The
caller matches on the union exhaustively.
Add a parametrized regression test asserting get_mcp_client_ip returns ""
for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry,
and update the resolver unit tests for the new return type.
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the
user to sign in. On an unauthenticated initialize the gateway answered locally
(200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop
either connected empty or hit "OAuth probe timeout after 10000ms".
#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers
to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth
servers, expecting the upstream to emit its own challenge. On initialize the
gateway never probes upstream, so no challenge ever reached the client.
Replace the `continue` with a preemptive 401 carrying the proxied
resource_metadata (RFC 9728) challenge, the same form passthrough servers and
MCPUpstreamAuthError already use. This keeps #29770 fixed (still no
authorization_uri) while restoring the upstream PKCE sign-in prompt.
* fix(mcp): resolve toolset tools by the server's known prefix
Toolsets store {server_id, bare tool_name} and reconcile that against the
live prefixed tool name at list time. The reconciliation chopped the live
name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a
server whose prefix contains the separator (a hyphenated alias, or the
UUID server_id used as the prefix when a server has no alias) had its
tools silently dropped from /toolset/<name>/mcp while listing fine
everywhere else. Strip the exact known prefix for the tool's server_id
instead of guessing the boundary, on both the resolve and filter sides
Also render toolset tools as {server-prefix}-{tool} in the dashboard
picker result and chips; this is display only, the persisted record
stays {server_id, bare tool_name}
Resolves LIT-3419
* test(mcp): add focused unit tests for strip_known_server_prefix
Cover the LIT-3419 cases directly on the helper with real MCPServer
objects: clean prefix round-trip, hyphenated alias, UUID server_id
fallback, unprefixed passthrough, and the server=None legacy fallback
* fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off
When a request carries an X-Forwarded-For header but use_x_forwarded_for is
unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load
balancer / reverse proxy). That peer almost always sits inside
mcp_internal_ip_ranges, so the 'Internal network only'
(available_on_public_internet: false) restriction trusts every external caller
as internal and effectively exposes those servers.
Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead
of hard-failing: on a deployment with no load balancer, a crafted
X-Forwarded-For header must not be able to take the service down, and a one-shot
log keeps a flood of crafted headers from spamming the logs.
* fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion
Address PR review: tie the one-shot warning flag to the observed
use_x_forwarded_for value so it re-arms whenever the setting is seen enabled,
restoring the diagnostic on a later rollback to disabled. Also assert against
str(call_args) so the test survives a positional-to-keyword logger refactor.
* fix(mcp): correct misleading no-trusted-proxy warning for XFF access control
* test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration
* fix(proxy): stop double-decrypting email/slack alerting env vars in get_config
proxy_config.get_config() already returns environment_variables decrypted
(the DB overlay decrypts them in _update_config_fields, and YAML values are
plaintext), so the /get/config/callbacks slack and email blocks were running
decrypt_value_helper() a second time on plaintext. That second decrypt always
failed and the helper swallowed the error and returned None, so every SMTP_*
field came back blank when the Admin UI reloaded the email settings, and the
proxy logged a misleading "Did your master_key/salt key change recently?"
error even when nothing changed.
Consume the already-decrypted values directly, matching process_callback's
handling of the same dict for langfuse/datadog/etc. Sensitive-value masking
is preserved.
Fixes#19221
* fix(proxy): preserve a cleared slack webhook instead of falling back to OS env
Use an explicit is-not-None guard rather than truthiness when deciding whether
to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin
cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from
the OS environment; only a truly absent key should trigger the OS lookup. No
decryption is reintroduced.
* feat(proxy): add logging_endpoints package init
* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out
* feat(proxy): register callback_logs_router
* test(proxy): add logging_endpoints test package init
* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling
* refactor(proxy): move callback-logs request/response models to litellm/types/proxy
* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging
* test(proxy): update callback-logs tests for class-based replayer and separated types
* fix(proxy): cover /v1/callbacks/ in backend component allowlist
The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.
* refactor(proxy): use builtin dict/list generics in callback-logs endpoint
Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).
* refactor(proxy): use builtin dict/list generics in callback-logs types
UP006: builtin generics over typing.Dict/List.
* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs
Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.
* fix(proxy): force stream=False when replaying callback logs
A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.
* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs
integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.
* feat(litellm-rust): RealTimeStreaming per-session log collector
1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.
* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)
The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.
* docs(litellm-rust): ai-gateway realtime logging architecture
* docs(litellm-rust): document request-log egress to the LiteLLM control plane
Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.
* feat(litellm-rust): make log-egress tunables env-overridable
Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.
* docs(litellm-rust): document log-egress tuning env vars
* docs(litellm-rust): require constants in a crate-level constants.rs
Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.
* refactor(litellm-rust): move ai-gateway constants into constants.rs
Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.
* ci: run logging_endpoints tests in the proxy-infra coverage shard
tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.
* fix(litellm-rust): hash the master key before logging — never send the raw credential
Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.
* fix(litellm-rust): observe realtime logging on upstream events only
Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.
* feat(proxy): bound callback-logs batch + return per-record failures
Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.
* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field
* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant
It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.
* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes
* docs(litellm-rust): tighten the README request-logging section
* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)
* docs(litellm-rust): drop em-dashes from the request-logging section
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* feat(mcp): add v1 bridge + none/api_key resolver arms (unwired)
PR4a of the MCP v2 outbound-credential migration, stacked on the resolver skeleton.
Builds the bridge for the first live modes without wiring it onto the request path:
- resolver.py: the none arm (NoOpAuth) and the api_key shared-key arm (StaticHeaderAuth
from the config); the BYOK source and the other five arms stay not_implemented.
- adapter.py: the v1 <-> v2 edge (to_subject, to_server_spec, raise_public, should_defer).
to_server_spec maps only none + the static-header family and returns None to defer every
other mode to v1. Imports v1, kept out of the package __init__ so the resolver core stays
v1-free.
- MCPClient gains an optional resolved_auth that feeds the factory's auth= slot, taking
precedence over the SigV4 aws_auth; default None keeps current behavior.
Nothing calls these from _create_mcp_client yet, so production behavior is unchanged; the
graft lands in PR4b. Unit tests cover the two arms, the full mapping table, and the auth
plumbing.
* feat(mcp): graft v2 resolver onto _create_mcp_client for migrated modes
Wire the none + api_key static-family resolver arms from PR4a onto v1's
live request path. In _create_mcp_client's HTTP/SSE branch, to_server_spec
decides per mode: a migrated mode resolves through the injected
UpstreamCredentialProvider and feeds the resulting httpx.Auth into the new
resolved_auth slot; every other mode returns None and falls through to the
unchanged v1 construction. resolve_mcp_auth now runs only when the mode
defers, so a migrated server skips the v1 token-exchange / M2M I/O.
stdio is untouched: auth_type/auth_value never reach the upstream on the
stdio path (_get_auth_headers is HTTP/SSE only), so there is nothing to
graft there. No v1 code is deleted yet; resolve_mcp_auth's static return
still backs stdio and the not-yet-migrated modes until later PRs retire it.
* test(mcp): cover the v2-resolver graft in _create_mcp_client
Regression tests for the PR4 graft. Migrated HTTP modes resolve through the
provider into resolved_auth: none -> NoOpAuth, and the static api_key family
emits the right header per scheme (X-API-Key, Bearer, token, raw authorization,
base64 basic). Deferred modes (oauth2) and a missing static token fall back to
v1's auth_value. A stdio server with a migrated auth_type still defers to v1,
since httpx.Auth never reaches the subprocess. A resolver Error is mapped to the
public HTTP contract (401) via an injected provider, exercising the DI seam.
* fix(mcp): defer to v1 when an inbound credential would be overridden
The graft attaches the resolved static credential as an httpx.Auth, whose auth
flow writes its header after extra_headers. That silently overrode an inbound
Authorization: a per-request mcp_auth_header override, or a header supplied via a
guardrail hook / static_headers / forwarded caller header. v1 lets those win, so
the graft had inverted the credential precedence for the migrated static modes.
Mirror the v2 egress credential-isolation invariant: defer the request to v1 when
mcp_auth_header is set, or when the header the resolved credential would write is
already present in extra_headers. none writes no header, so it never defers.
* test(mcp): cover the credential-isolation defer guard
Regression tests for the precedence fix. A per-request mcp_auth_header override and an
Authorization already present in extra_headers (guardrail hook like the JWT signer,
static_headers, or a forwarded caller header) both defer a migrated static server to v1
so the inbound credential wins; none stays on v2 and does not clobber an inbound
Authorization since NoOpAuth writes nothing. The deferred cases assert resolved_auth is
None, which fails if the guard is removed.
* refactor(mcp): resolve inbound-header conflict on v2 instead of deferring
For an Authorization already supplied via extra_headers (a guardrail hook such as the
JWT signer, static_headers, or a forwarded caller header), keep the request on the v2
path and skip resolved_auth rather than deferring to v1. The inbound header still wins
since nothing overwrites it, but hooks no longer pin a v1 fallback, which is what lets
resolve_mcp_auth be retired once the remaining modes migrate.
The mcp_auth_header per-request override still defers to v1, since that value becomes
the upstream credential rather than sitting in extra_headers; that defer falls away
once the per-user modes stop writing mcp_auth_header.
* fix(mcp): clear UP037 lint gate and fix allowed-servers test under the graft
adapter.py uses `from __future__ import annotations`, so the quoted "UserAPIKeyAuth" /
"MCPServer" annotations in to_subject/to_server_spec/_shared_key_spec were unnecessary
and pushed UP037 over the strict-rule budget; drop the quotes.
test_list_tools_only_returns_allowed_servers passed a MagicMock as user_api_key_auth.
The graft now builds a Subject from the principal, and the MagicMock's non-string
org_id/user_id fail Subject validation, so the listing came back empty. Use a real
UserAPIKeyAuth instead (MagicMock for an injected dependency was the anti-pattern here).
* test(mcp): assert config token via resolved_auth, not the headers dict
test_mcp_server_config_auth_value_header_used inspected _get_auth_headers(), but the
graft now carries the static credential on the client's httpx.Auth (resolved_auth) and
writes the header at send time, so that dict is empty. Assert the header the
StaticHeaderAuth emits onto the request instead. Both config keys (authentication_token,
auth_value) stay covered.
* chore(typecheck): set reportMatchNotExhaustive slack to 0
The previous slack of 3 put the ceiling at baseline + slack = 4, so a newly
non-exhaustive match (for instance dropping an Error arm off a Result match)
could land without tripping the gate. Setting slack to 0 pins the ceiling at
the current baseline of 1, so any added non-exhaustive match now fails CI while
the one pre-existing violation in router.py stays within budget
Creating or updating a key with a specific (non-allow_all_keys) MCP
server or access group failed with a 403 when the key had no team:
Key is not in a team. Only globally available (allow_all_keys) MCP
servers can be assigned
validate_key_mcp_servers_against_team computed the allowed set as
team servers + allow_all_keys servers. For a teamless key the team
set is empty, so the allowed set collapsed to just allow_all_keys
servers and any explicitly-picked server or access group was rejected.
This was asymmetric with runtime: get_allowed_mcp_servers honors a
teamless key's own object_permission.mcp_servers verbatim, with no
team gate and no allow_all_keys filter. So the create/update path
refused to persist a grant the run path would have served.
Thread is_proxy_admin into the validator from both call sites
(/key/generate and /key/update). When a key has no team and the
caller is a proxy admin, the requested servers and access groups are
folded into the allowed set so the existing subset checks pass. A
proxy admin can already reach every MCP server, so there is nothing
to escalate. Non-admins and every team-scoped key are unchanged.
Resolves LIT-3815
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
A user provisioned with "All Proxy Models" stores the literal
"all-proxy-models" sentinel in user.models. get_direct_access_models looked
that string up as a real model_name via get_model_list, which matched no
deployment, so /v2/model/info marked every model direct_access=false and the
Models + Endpoints page rendered empty for such users when they have no teams.
The model dropdown / Playground worked because get_key_models already expands
the sentinel to the full proxy model list, hence the inconsistency in the
report.
Expand the sentinel to all non-team deployment ids via
get_model_ids(exclude_team_models=True), the same call the PROXY_ADMIN branch
in the caller already uses. This fixes both /v1/model/info and /v2/model/info
since they share _populate_team_access_on_models. Empty user.models stays "no
direct access" to match get_key_models semantics.
Fixes#22791
The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the
target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user).
A server defined in config.yaml lives only in the in-memory registry and never
gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server
<id> not found" (or 403 for non-admins) before any credential could be stored,
leaving config-server users unable to connect and forced to re-authorize forever.
Route all three through a single registry-aware resolver: DB first, then the
in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table,
the same fallback fetch_mcp_server already uses), then the canonical
get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls.
Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden
server so server ids stay non-enumerable. This also closes a gap where the two
store endpoints performed no per-server authorization at all.
* fix: redact config and MCP secrets in read-only admin views
GET /config/field/info and the MCP server list/detail endpoints returned
secret-bearing fields to any caller with an admin view, including
read-only admins. They now return those fields in full only to a full
PROXY_ADMIN; every other caller gets the reduced, non-admin view, while
non-sensitive fields remain readable. Regression tests cover the
role-based visibility on both endpoints, including that a full admin
still sees everything needed to populate the edit form.
* fix: redact nested secrets in config field info for non-admins
/config/field/info returned structured general_settings fields verbatim to
any admin-view caller, so a view-only admin reading database_args received
the nested aws_web_identity_token (a DynamoDB role-assumption credential) in
plaintext. Recurse into dict/list field values and redact secret leaves for
non-PROXY_ADMIN callers, leaving non-secret siblings and full-admin reads
unchanged
* fix: redact secret config values in /config/list for non-admins
/config/list shared the same _user_has_admin_view gate as /config/field/info
but returned each field value unredacted, so a view-only admin reading the
list received pass_through_endpoints upstream Authorization headers verbatim.
Route every general_settings value through a shared role-aware redactor
(extracted from /config/field/info) covering the top-level and nested field
paths, so non-PROXY_ADMIN callers get secret-bearing fields redacted while
full-admin reads stay unchanged
* chore(ci): allowlist _redact_secret_values_in_obj in recursive_detector
The config secret redactor recurses over JsonValue, which is acyclic, and
its depth is bounded by the operator-authored general_settings schema. Add
it to the recursive_detector ignore list alongside the other bounded
nested-redaction helpers (mask_dict, _redact_sensitive_litellm_params)
* proxy: cap recursive secret redaction depth at 10
Match the cap on _redact_sensitive_litellm_params (the closest analog
in the proxy, also recursive, key-name driven, returns a sentinel).
The previous justification — bounded by operator-authored schema depth,
JsonValue acyclic — is true today but is a property of the threat model,
not an enforced invariant of the function. If a code path is ever added
that pipes external input into general_settings (config import,
migration tooling, JWT-driven settings, …) the assumption silently
breaks. A local cap makes the invariant local.
The cap branch fails closed: at _REDACT_SECRET_MAX_DEPTH the whole
subtree is replaced with 'REDACTED' rather than returned verbatim. A
future refactor that flips this to fail-open would let a deeply nested
credential leak; the new regression test test_redact_secret_values_in_obj_fails_closed_at_max_depth
guards against that.
Updates the recursive_detector ignore-list rationale to point at the
numeric cap rather than the structural argument.
* test: actually exercise the depth cap in fails-closed test
The previous fixture stored the leaf under the secret-named key
'aws_web_identity_token', which the recursor's key-name short-circuit
redacts regardless of the cap — so the test passed both with and
without the cap in place. Empirically confirmed: under an uncapped
mutant the old fixture still hides the secret (key-name catches it),
the new fixture leaks it (only the cap can stop it). Swap the leaf
key to a non-secret name so the cap is the only redaction path
exercised, making the test fail on mutation as advertised.
PR3 of the MCP v2 outbound-credential migration, stacked on the typed vocabulary.
Adds resolver.py: UpstreamCredentialProvider.resolve_credentials dispatches on the
declared AuthConfig variant with one arm per mode, a wildcard-free match plus an
assert_never tail so a missing arm fails basedpyright's exhaustiveness gate. Every
arm is a not_implemented stub returning a typed CredError; each mode's real body and
seam land in follow-up PRs. Pure v2, no v1 imports, nothing wired onto a request path.
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes#30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.
* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos
test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.
* fix(interactions): drop role from Interaction response to match Google spec
Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.
* fix: align all-team-models sentinel access
* fix(router): forward include_fallback_errors through multi-hop fallbacks
run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.
---------
Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu>
Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com>
Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com>
Co-authored-by: dav nguyxn <hoangson091104@gmail.com>
Co-authored-by: Tal Marian <tal.marian@island.io>
Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
user_api_key_auth raises ProxyException, not HTTPException, on an auth failure.
The streamable-HTTP and SSE MCP handlers only re-raised HTTPException to preserve
status and headers, so a ProxyException fell through to the catch-all and was
flattened to a generic 500, dropping the real status (for example 401) and any
WWW-Authenticate challenge. MCP clients render a 500 on the JSON-RPC POST as a
cancelled or terminated session, and an OAuth client never receives the 401 it
needs to re-authenticate. Because auth runs before server routing, one rejected
credential fails every targeted server at once.
Map ProxyException back to its real status and headers in both handlers
(handle_streamable_http_mcp, handle_sse_mcp) via a small
_proxy_exception_to_http_exception helper inserted before the generic
except Exception. A genuine auth failure now returns its real status; a key sent
without the documented Bearer prefix gets a clear 401 telling the caller to fix
the header rather than a cancelled session.
Regression tests assert that a ProxyException(401) raised during auth propagates
as a 401 with WWW-Authenticate from both the streamable-HTTP and SSE handlers,
and unit-test the converter for the 401/403/non-numeric-code cases.
* feat(mcp): scaffold outbound_credentials package with typed Result
PR1 of the MCP v2 outbound-credential migration. Adds the
litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a
hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new
dependency) and its package surface. Nothing imports this on a live request path
yet, so production behavior is unchanged; later PRs add the typed config
vocabulary, the resolve_credentials dispatch, and the v1 graft.
* feat(mcp): add outbound_credentials typed vocabulary (#31049)
PR2 of the MCP v2 outbound-credential migration, stacked on the result.py
scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen
per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec,
and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth,
StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the
tagged union. Construction-time tests prove illegal mode/field combinations are
rejected. Nothing is wired onto a request path yet; the resolver lands next.
POST /team/new with any budget_limits returned 500 because
jsonify_team_object serialized members_with_roles but left budget_limits
as a raw Python list, which Prisma's Json column rejects. /team/update
and /key/generate worked only because each json.dumps the windows itself.
Serialize budget_limits in the shared helper, guarded by isinstance(list)
so the pre-serialized /team/update path is unaffected.
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools
GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.
Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.
* test(proxy): parametrize MCP inference route check over the full endpoint set
The Add Model provider dropdown is driven by provider_create_fields.json
(served at /public/providers/fields), and Bedrock Mantle had no entry, so it
could not be selected even though the backend provider, its models, and the UI
enum/logo mappings already existed.
Add a bedrock_mantle entry exposing the credential fields the provider actually
honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and
an api_base override. Selecting it now populates the bedrock_mantle models from
the cost map via the existing getProviderModels filter.
Also resolve the provider logo when getProviderLogoAndName is given the enum key
(e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously
only slugs that lowercase-matched their key (like bedrock) resolved a logo.
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row
This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team
Completes the work in #30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced
* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body
* test: cover bedrock key-tag litellm_metadata pre-seed in common_checks
Add a regression test asserting key-level tags on a bedrock passthrough
request land in litellm_metadata and never leak into the provider-facing
metadata field, which closes the codecov/patch gap on the auth_checks
pre-seed line. Also drop the now-stale comment that hardcoded
metadata["headers"]; the headers are written under whichever metadata
field _get_metadata_variable_name selects.
* refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES
The auth-time pre-seed in common_checks hardcoded "bedrock", so any other
route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key
tags leaking into the provider-facing metadata field) without a matching
update here. Key off the shared constant instead, and extend the regression
test to cover a non-bedrock metadata route so the route-agnostic behavior is
locked in.
* fix(auth): pre-seed litellm_metadata before header-tag merge
apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before
common_checks, so it resolved get_metadata_variable_name_from_kwargs to
'metadata' (litellm_metadata was not yet present). common_checks then
pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which
apply_key_tags_pre_auth and _tag_max_budget_check both targeted
litellm_metadata, leaving header tags stranded in metadata and invisible
to per-tag budget enforcement on Bedrock and other matching routes.
Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route
and invoke it before apply_client_tag_policy_pre_auth so all tag merges
and the budget-check read agree on the same metadata key.
* test(auth): guard early litellm_metadata pre-seed wiring for header tags
Bugbot's autofix added a pre-seed of litellm_metadata in
_run_centralized_common_checks before apply_client_tag_policy_pre_auth, so
x-litellm-tags header tags land in litellm_metadata and stay visible to
_tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call
order in the test body, so removing the production call site still passed.
Add a wiring-level regression that drives the real _run_centralized_common_checks
and asserts header tags land in litellm_metadata (not metadata) for bedrock and
/v1/messages. Dropping the pre-seed call site now fails the test.
---------
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and
default_user_id) instead of emails. /user/daily/activity passed
entity_metadata_field=None, so every user entity in the breakdown
carried empty metadata; the chart could only fall back to the user_id.
The frontend resolved labels from a separately paginated user list, so
any spender not on a loaded page showed as a UUID.
Resolve the email/alias for the user_ids actually on the page (mirroring
how api key metadata is already resolved) and attach it to the entity
metadata, so the chart labels each spender with their email and falls
back to the UUID only when no email is on file. get_daily_activity gains
an optional resolve_entity_metadata hook so the user endpoint can do this
page-scoped lookup without loading the whole user table.
Resolves LIT-3889
Map the SCIM enterprise extension block
(urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser
so create and PUT persist employeeNumber, costCenter, organization,
division, department, and manager into LiteLLM_UserTable.metadata under
scim_enterprise, and round-trip them back out on read. This lets financial
reporting group spend by fields like cost center and department.
The enterprise block holds directory-only HR attributes, so it is kept out
of the generic user management responses (/user/info, /v2/user/info, and
/user/list), which non-proxy-admin callers such as team and org admins can
use to read other users. The data still lands in metadata for reporting and
still round-trips through the SCIM read endpoints, which build their response
from the user row directly.
Resolves LIT-3617
