The MCP OAuth DCR relay's client-facing /register response handed back LiteLLM's own
<base>/callback as the client's redirect_uris on every arm except the true bridge relay.
A spec-compliant OAuth 2.1 DCR client (e.g. Open WebUI over Streamable HTTP) adopts that
value for its subsequent /authorize calls, so /callback redirects to itself; the second
hit carries the client's opaque state, fails to decrypt, and surfaces as the LIT-4197
"oauth_state ... Incorrect padding" error, making pass-through MCP OAuth unusable against
real DCR-capable upstreams.
The short-circuit arm keeps registering the gateway callback upstream (unchanged) and now
echoes the client's own redirect_uris back to the client across all register arms. Since
the client then authorizes with its own separate-origin redirect, the /authorize rejection
hint now points operators to MCP_TRUSTED_REDIRECT_ORIGINS, the mechanism a legitimate
cross-origin OAuth client needs (auto-trusting a DCR-registered redirect would reintroduce
the VERIA-57 open-redirect vector, since dynamic registration is unauthenticated).
Staging now contains #33153, whose final rounds made _extract_upstream_auth_failure a thin delegate
to upstream_auth_challenge and introduced the response-level iterator this branch predates. The
resolution completes the consolidation both branches were converging on: iter_exception_tree
(faults/traversal.py) is the one tree walk, _iter_upstream_responses is rebuilt on top of it instead
of carrying a second copy of the traversal, the manager keeps the delegate, and the semantic filter
port from this branch stands. Test conflicts were append-append and both sides are kept
_resolve_v2_auth dropped the resolved client_credentials auth when extra_headers
already carried Authorization (MCPJWTSigner, static_headers), so the upstream got
the injected header instead of the minted token and the one-shot 401 refetch was
lost. M2M now joins token_exchange and authorization_code in the authoritative
set; the conflicting header is dropped
The auth object is the httpx client's auth for the whole MCP session; after a
401 recovery it kept sending the rejected token first, burning a 401 round trip
and the single retry on every subsequent call
An expires_in of zero or below computes a ttl of 0; the entry could never be
served but still occupied a slot in the bounded backend, where it could evict
a live token. The mint still serves the current request and the next get
re-fetches under the per-server lock
Greptile P2s: the per-server lock dict now evicts its oldest entry past
max_locks so ephemeral server ids (REST tools preview) cannot grow it
unbounded, and the min-cache floor is capped at the token's actual lifetime so
an expires_in below the skew is never served past expiry
The graft test pinned the pre-migration contract (M2M defers to v1). Replaced
with two tests pinning the new one: a complete-config M2M server resolves via
the v2 arm into ClientCredentialsBearerAuth, and an incomplete-config server
fails closed with a 500 misconfigured naming the missing grant fields
Replaces the not_implemented stub with a live arm: ClientCredentialsTokenSource
mints and caches the M2M token (rotation-aware identity key, expires_in-driven
TTL, audience and token_endpoint_auth_method support) and
ClientCredentialsBearerAuth retries an upstream 401 exactly once with a freshly
minted token. to_server_spec owns oauth2_flow=client_credentials servers and
fails closed on incomplete grant config instead of connecting unauthenticated
The SERVER_ROOT_PATH fix for the per-server pass-through challenge belongs with its sibling
in exceptions.py (both fabricate a per-server resource_metadata URL and both omit the root
segment), and both are pre-existing paths unrelated to the aggregate discovery this PR adds.
Reverting the server.py change keeps this PR to the aggregate front door and avoids leaving
the two per-server challenge builders inconsistent; the per-server root-path fix lands as its
own change covering both sites.
tests/test_litellm/proxy/test_custom_proxy.py sets SERVER_ROOT_PATH at import time (its app
mounts under a custom path) and never restores it, so in a shared shard the value leaks into the
process. The discovery routes and the 401 challenges now read SERVER_ROOT_PATH to path-insert it
where they previously ignored it, so a leaked value rewrites every resource_metadata URL and the
exact-URL assertions in the delegate, pass-through, and aggregate challenge tests fail depending
on shard order
An autouse fixture clears SERVER_ROOT_PATH for the MCP discovery tests so they deterministically
exercise the default root-mounted deployment; the tests that assert a sub-path deployment set the
value explicitly within their own body. No assertion changed; the leak was invisible before only
because the code ignored the variable
Two RFC 9728 / 8414 discovery fixes on the aggregate front door, both raised by Bugbot on this PR
The aggregate authorization-server document at /.well-known/oauth-authorization-server/mcp used to
defer to a per-server row literally named "mcp", serving issuer {base} while the aggregate
protected-resource document advertises {base}/mcp as its authorization server. A spec client
following that chain fails the RFC 8414 issuer check and cannot sign in. The single segment /mcp is
now reserved for the aggregate so the issuer stays {base}/mcp and matches the protected-resource
document; a server named "mcp" keeps its standard two-segment discovery at
/.well-known/oauth-authorization-server/mcp/mcp
The 401 challenges built the resource_metadata URL as {base}/.well-known/oauth-protected-resource/mcp
with no SERVER_ROOT_PATH segment, but the routes are registered with the path-inserted root segment,
so a proxy mounted under a sub-path pointed DCR clients at a URL that 404s. Both the aggregate
challenge and the pre-existing per-server pass-through challenge now derive the path from one
well_known_root_suffix helper that the route registrations also use, so the advertised URL cannot
drift from the served route
The flag guarded no breaking change: the aggregate discovery lives at new /mcp-suffixed
routes, the challenge only fires at aggregate scope, and the authorize/token/register/admission
arms self-gate on the llm_dcrc_/llm_session_ prefixes. Bare-origin and per-server discovery are
left exactly as they were, and a server literally named mcp keeps its own discovery via
disambiguation, so turning it on for everyone changes nothing about existing flows.
* feat(mcp): add ID-JAG egress auth as a v2 outbound-credentials arm
Adds the oauth2_id_jag MCP egress auth mode (draft-ietf-oauth-identity-assertion-authz-grant,
shipped by Okta as "AI agent token exchange") as a first-class arm of the v2
outbound_credentials resolver rather than a standalone v1 handler.
ID-JAG is a two-leg flow: an RFC 8693 token exchange swaps the caller's id_token for an
ID-JAG assertion at the IdP org authorization server, then an RFC 7523 jwt-bearer grant
presents that assertion to the MCP's resource authorization server for the access token
used to call the upstream. The gateway authenticates to both endpoints with a private-key
JWT client_assertion, falling back to client_secret when no key is configured.
The mode is modeled as IdJagConfig in the AuthConfig discriminated union, with client auth
as a ClientAuth tagged union (private_key_jwt or client_secret) so required fields are
enforced at construction and illegal states are unrepresentable. A new token_endpoint
collaborator performs the authenticated OAuth token-endpoint call and caches the result
with per-key single-flight; the resolver's _id_jag arm runs the two legs and returns an
httpx.Auth or a typed CredError. A missing caller identity token fails closed
(precondition_required), so an ID-JAG server never falls back to a static credential. The
v1->v2 adapter maps oauth2_id_jag servers onto IdJagConfig and the existing live v2 path
resolves them, so no standalone handler, has_id_jag_config flag, or resolve_mcp_auth
precedence branch is needed.
The ID-JAG client_private_key is encrypted at rest alongside client_secret.
* fix(mcp): sort token_endpoint imports to satisfy the I001 budget gate
* fix(mcp): give token_endpoint pyright suppressions reasons for the LIT004 budget
The freshly-merged base ratcheted the LIT004 ceiling down, so the six
unexplained pyright suppressions in token_endpoint.py went over budget.
Annotate each with why the boundary is untyped (litellm http handler and
InMemoryCache are untyped; response.json() is validated by
_TokenEndpointResponse in fetch) so the gate counts them as explained.
* fix(mcp): enforce ID-JAG exchange over caller auth overrides and redact token endpoint from client errors
For oauth2_id_jag servers the v2 resolver mints the upstream assertion from the caller's identity token; a caller-supplied x-mcp-auth / x-mcp-<alias>-authorization override or a conflicting injected Authorization must not disable that exchange and forward an arbitrary bearer, so IdJagConfig now joins authorization_code and token_exchange as a resolver-owned mode that keeps the v2 spec and ignores the override.
The token endpoint error branches previously returned the configured endpoint URL in the client-visible 503 detail. The endpoint now stays in server-side logs and clients get a generic token-exchange failure.
* fix(mcp): bind the ID-JAG token cache to the exchange config and map token endpoint network errors to typed CredErrors
* fix(mcp): fail closed when an oauth2_id_jag server is half-configured instead of deferring to v1 static credentials
* fix(mcp): evict the cached ID-JAG bearer on an upstream 401 so the retry re-exchanges
* fix(mcp): map an unsignable client assertion to a typed misconfigured error instead of an unhandled 500
* fix(mcp): redact credential fields from the server-registry debug dump
Config.yaml-declared OAuth2 MCP servers using Dynamic Client Registration have no LiteLLM_MCPServerTable row, so the DCR persist path called update_mcp_server, which returns None for a missing row, then update_server(None), which dereferenced .approval_status and raised AttributeError. The exception was swallowed to a warning while /register still returned 200, so the minted client was never stored and every access-token expiry forced a full re-authorization
Persist the acquired DCR client (client_id, client_secret, token_endpoint_auth_method, redirect_uris, encrypted at rest) in a dedicated LiteLLM_MCPServerOAuthClient store keyed by server_id when the server has no row, overlay it onto the in-memory config server so the refresh_token grant can authenticate within the process, and rehydrate it when the registry syncs from the database (which runs after the DB connects, unlike config load) so restarts and other pods pick it up. The store is encrypted at rest and is re-encrypted by the master-key rotation path alongside the server rows, through a shared helper so the two sites cannot diverge. The DB-backed server path is unchanged, and guarding the None return removes the swallowed-crash footgun
Resolves the config.yaml DCR persistence regression introduced in v1.92.0 by #31912
Outcome keys in the tools/list _meta, the spend-log outcome and count maps, and the REST error
messages now all use get_server_prefix (alias, or the short prefix when that mode is enabled), the
same naming the caller already sees on tool names. Keying them by canonical server_name let an
authenticated caller enumerate internal server names and their health or auth state that the alias
and short-prefix schemes deliberately hide (Veria finding). One helper decides the key for every
surface; exception messages reaching the multi-server REST error list are mapped to their fault tag
with the display prefix instead of relaying exception text carrying canonical names. Server-side
logs keep the real names
Conflict in _list_mcp_tools: staging (#33612) moved toolset-grant expansion into the shared
permission primitives and removed the _merge_toolset_permissions call; resolution applies that
removal to this branch's AggregateToolListing structure
Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome
tests and the mode-aware preemptive-401 tests from staging; both kept
The consolidation regressed the pre-existing walker semantics: _extract_upstream_auth_failure used
to keep scanning until it found a 401/403, while the consolidated helper took the first response of
any status and then tested it, so a causal 401 sitting behind an unrelated 5xx (retry attempts,
multi-stream task groups) was misclassified as upstream_error and its challenge lost on the listing,
tool-call, and probe paths. The traversal is now an iterator in deliberate order and each consumer
applies its predicate over the stream: the auth scan takes the first 401/403 even behind non-auth
responses, generic classification takes the first response, and classify_list_exception derives its
auth arm from the same scan so the carrier choice and the classification can never disagree
Grafted from PR #33582 (closing as superseded by this PR): drives
handle_streamable_http_mcp with real MCPServer objects, parametrized over a
stamped client_credentials row and a legacy unstamped M2M-shape row; both must
reach the session manager without the per-user token store being consulted
The preemptive-401 gate for auth_type=oauth2 MCP servers keyed the challenge
on whether an Authorization header was present (not oauth2_headers). Because
the header parser classifies any Authorization bearer as an OAuth token before
the target server is resolved, a LiteLLM virtual key presented as
Authorization: Bearer sk-... suppressed the challenge on a gateway-managed
authorization_code server; the session then opened with no upstream token and
tools/list masked the failure as 200 with an empty tool list. The same gate
also wrongly challenged client_credentials (M2M) servers, which the gateway
authenticates by minting its own token at egress.
The decision is per oauth2 sub-mode, not per header. Gateway-managed modes
never receive a client-supplied upstream token: client_credentials mints at
egress so it is never challenged, and gateway-managed interactive
(authorization_code, non-delegate) is challenged whenever no stored per-user
token exists, regardless of any bearer. Only the delegate/upstream-PKCE mode,
where a present bearer genuinely is the upstream token, keeps keying on the
Authorization header. oauth2_headers itself is left untouched so the
delegate/passthrough egress paths that forward the client bearer are
unchanged.
The hook returned early when the semantic filter selected no tools, which
restated a policy that SemanticMCPToolFilter.filter_tools already owns: it
returns the full tool set when nothing matches, so the selection is never
empty. The branch was unreachable, and reachable or not it changed nothing,
since the gateway reads the union of every reference's allowed_tools and
treats an empty union as unset. Its only effect was to suggest the reference
path and the plain tool path resolve a zero-match query differently.
Drop it so a single policy governs both paths, and pin that with a test
covering an unmatched query on each path. Flipping filter_tools to fail
closed now fails the test on both instead of quietly hard-limiting one
surface and not the other.
The semantic tool filter replaced each litellm_proxy MCP reference in
data["tools"] with the tools it expanded from that reference. The expansion
defaults to the Responses API tool shape, so a /chat/completions request came
out carrying flat {"type": "function", "name": ...} entries where the provider
transformations expect {"type": "function", "function": {...}}. Anthropic then
raised KeyError: 'function' and Bedrock dropped every MCP tool silently, so the
model answered as if no MCP server were connected.
Replacing the reference also removed the marker the MCP gateway matches on, so
acompletion_with_mcp never ran and tool calls were no longer auto-executed for
require_approval="never", on /responses as well as /chat/completions.
Narrow the reference through allowed_tools instead and leave it in place, so the
gateway still owns expansion and keeps both the per-endpoint tool shape and tool
auto-execution. Expansion already applies any caller-supplied allowed_tools, so
the selection can only narrow a reference further, never widen it.
Making the in-memory issuer reflect a trust-on-first-use discovered value fixed
the registry/row token-identity drift, but it overloaded a single field: the
carry-forward gate keyed on issuer truthiness as a proxy for "endpoints are
anchored to a pinned issuer, fail-closed". A discovered issuer is truthy yet not
anchored, so a resource-rooted server that had learned its issuer would drop its
last-known-good endpoints on a transient discovery blip instead of carrying them
forward.
Anchoring is now a first-class property rather than a proxy. MCPServer carries
issuer_is_anchored, set at both build paths from the single _uses_issuer_anchor
definition (a pinned issuer on a discovery auth type). issuer stays the identity
value used by the token-identity tuple and the serializers; issuer_is_anchored is
the provenance value the carry-forward gate reads to decide fail-closed. The two
properties can no longer be conflated, so a discovered issuer keeps its
resource-rooted endpoints carrying forward while a pinned issuer still fails
closed.
Regression tests pin both directions: a discovered-but-not-anchored server
restores its endpoints on a discovery blip, an anchored server does not, and the
build sets issuer_is_anchored true only when the issuer is pinned
Two lifecycle gaps let the issuer trust anchor drift out of sync with the
endpoints it governs. Changing or clearing a previously pinned issuer left the
authorization_url and token_url that were resolved under the old issuer in the
row, so clearing the anchor could revive stale, possibly untrusted endpoints
instead of re-discovering. And a build that discovered an issuer
trust-on-first-use persisted it to the row while the returned in-memory server
kept the issuer unset, so the registry and the row disagreed and the per-user
OAuth token identity, which includes the issuer, differed between that build and
the next rebuild and forced a spurious re-auth.
update_mcp_server now treats a change to a previously pinned issuer the same as a
url or auth_type change and clears the auth-flow-scoped endpoint fields that were
resolved under it. The trigger fires only when an issuer was already pinned and
is now changed or cleared, so establishing one for the first time, including the
trust-on-first-use discovery write-back, does not wipe the fields it just
resolved.
Both build paths, build_mcp_server_from_table and load_servers_from_config, now
construct the server with effective_issuer = manual_issuer or the discovered
issuer, skipping an origin-fallback guess exactly as the persistence does, so the
in-memory object always reflects what the row will hold.
Regression tests pin each case: clearing and re-pointing a pinned issuer clear
the stale endpoints, a first-time establish preserves the discovered fields, and
a build reflects the discovered issuer while an origin-fallback guess is not
reflected
When an admin pins an issuer, RFC 8414 section 3.3 makes that issuer the sole
authoritative source of the authorization and token endpoints, so a compromised
or misconfigured upstream cannot smuggle a token endpoint by echoing the pinned
authorize URL. The first cut enforced that only on the database build path; the
carry-forward, persistence, config-load, serialization and sanitization paths
could still restore or emit upstream-derived endpoints for an issuer-anchored
server, which is the class of gap the review flagged.
Every site now routes through one predicate. _endpoints_yield_to_issuer returns
all-None whenever the issuer is the anchor, so both build paths,
has_all_upstream_oauth_fields, needs_discovery and the endpoint merge defer to
the issuer. _carry_forward_resolved_oauth_endpoints carries only scopes for an
issuer-anchored server and fails closed on endpoints.
_persist_discovered_oauth_endpoints skips endpoint writes under the anchor. The
two table serializers round-trip the issuer and both non-admin sanitizers redact
it. Scope selection stays resource-driven per the MCP authorization spec:
_fetch_issuer_anchored_oauth_metadata takes endpoints from the issuer document
and scopes from the resource document.
The OAuth metadata resolution and corroboration gating for the database build
path move into _resolve_table_oauth_metadata so build_mcp_server_from_table
stays within the cyclomatic-complexity budget without changing behavior.
Regression tests pin the invariant at each site: the issuer overrides stored
endpoints even when they are populated, carry-forward does not restore endpoints
under the anchor, persistence does not write endpoints under the anchor, a url or
auth_type change clears stale issuer-scoped fields even when resubmitted
unchanged, the Azure heuristic stays reachable under a required issuer, and
anchored metadata takes endpoints from the issuer while scopes come from the
resource
The issuer anchor is for the token/registration endpoints only (the RFC 9700
mix-up). Scope selection stays resource-driven per the MCP authorization spec
Scope Selection Strategy: _fetch_issuer_anchored_oauth_metadata now validates
the issuer document (RFC 8414 §3.3) for the endpoints and separately fetches the
resource's advertised scopes (WWW-Authenticate challenge, else RFC 9728
scopes_supported) for the scope value, instead of using the issuer document's
own scopes_supported. The resource can influence only the requested scope, which
the authorization server and user consent bound (RFC 6749 §3.3), never the token
endpoint.
Discover the issuer from the upstream and persist it trust-on-first-use (fill-empty-only,
frozen thereafter), so admins do not have to type it; an admin-configured issuer always wins
and is never overwritten. Re-pointing the server url now clears the discovered issuer and
endpoints (matching the existing auth_type-change clearing) so a new upstream re-discovers
instead of anchoring on the previous upstream's issuer. Adds the issuer to the per-user OAuth
token identity so re-pointing it purges stale tokens. Surfaces the issuer as an optional,
auto-discovered, overridable field in the create and edit MCP server forms.
C901 gate shows +1 vs staging; that is inherited from the #33317 stack base (delta 0 against
Adds an admin-configured issuer to MCP servers. When set, OAuth metadata is
fetched from the issuer's own origin and adopted only when the document
self-attests that same issuer (RFC 8414 §3.3), making token_endpoint,
registration_endpoint, and scopes authoritative for the pinned issuer instead
of a document the MCP resource server chose. This closes the mix-up where a
compromised resource echoes a pinned authorization_url to smuggle its own
token endpoint and inflated scopes past the corroboration gate. Discovery is
same-authority against the issuer origin, fails closed on a §3.3 mismatch, and
does not fall back to resource-rooted discovery. Rows without an issuer keep
the existing corroboration-gate behavior unchanged.
Backend + schema only; UI field and live-proxy proof follow.
Reverts the over-correction that restricted a pinned-authorization_url server's
discovered scopes to the authorization server's own scopes_supported. Per the MCP
authorization spec Scope Selection Strategy and RFC 9700 §2.3, the scopes a client
requests are resource-driven: the WWW-Authenticate 401 challenge scope, else the
RFC 9728 protected-resource scopes_supported. The authorization server's RFC 8414
scopes_supported is a non-exhaustive capability list (the server MAY omit supported
scopes) and is never the selection source; scope inflation by a compromised resource
is bounded by the authorization server and user consent (RFC 6749 §3.3), not by the
client restricting the request. The corroboration gate now rejects only the
uncorroborated token_url/registration_url (the RFC 9700 endpoint mix-up) and leaves
scopes untouched. Removes the now-unused authorization_server_scopes field.
A whitespace-only authorization_url was truthy to the row/config merges and
has_all check but blank to the corroboration gate, so discovery and
carry-forward adopted token_url/registration_url/scopes as if unpinned while
the broken whitespace value was still used for redirects. Rather than add
another strip() at each site, the pinned authorization_url/token_url/
registration_url are normalized once per build path (DB and config) via
_blank_to_none, so the merge, has_all gate, discovery gate, persist hook, and
carry-forward all see a single notion of blank. Empty and whitespace pins now
behave identically to an omitted field.
