mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
579 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tin
|
367aa904de |
fix(ui): wire the tool playground and gateway authorize flow for the client-forwarded token modes
The server detail page's Tool Testing Playground gated its browser-held
token handling on the legacy PKCE-passthrough shape, so a
true_passthrough or oauth_delegate server listed tools unauthenticated
and surfaced 'Failed to fetch MCP tools' with no way to authorize. The
playground now treats both modes as browser-held-token servers: it
reads the sessionStorage token established by the create/edit
browser-only Authorize, forwards it via the x-mcp-{alias}-authorization
header, evicts it on a 401, and shows its own Authorize gate when the
token is absent.
That gate's flow uses the gateway's relayed authorize/register/token
endpoints with the real server id, which previously 400ed for anything
but oauth2. Those endpoints now also accept the client-forwarded token
modes (the minted token is upstream-audienced and browser-held; DCR
persistence stays off on this path), and registry builds run the same
RFC 9728/8414 endpoint discovery for these modes that oauth2 rows get,
since their rows never store an authorization_url.
|
||
|
Tin
|
22ab518071 |
feat(ui): browser-only Authorize & Fetch for the client-forwarded token modes
true_passthrough and oauth_delegate persist no upstream credentials, so
the create/edit forms had no way to preview tools or configure the tool
allowlist: tools/list went upstream unauthenticated and came back 401.
This reuses the existing OAuth authorize machinery in browser-only mode
for those two auth types: the admin authorizes against the upstream
(DCR/PKCE, with optional client credentials for IdPs without dynamic
registration), the token lands in sessionStorage exactly like the
legacy PKCE-passthrough path, and the tools preview forwards it via the
per-server x-mcp-{alias}-authorization header, which the passthrough
resolver arm already accepts. Nothing is written to the server row or
the per-user credential store; the create payload keeps excluding
credentials for these auth types via AUTH_TYPES_REQUIRING_CREDENTIALS.
The tools preview endpoint now also extracts the Authorization header
for the two new auth types so the browser-held token reaches the
passthrough arm during create-time previews.
|
||
|
tin-berri
|
4e6ec995e7
|
Merge pull request #31989 from BerriAI/litellm_mcp_passthrough_delegate_modes
feat(mcp): add true_passthrough and oauth_delegate auth modes |
||
|
Tin
|
b2ea36f4f1 |
fix(mcp): match sanitized per-server alias at the connect-time preemptive 401
The connect gate resolved x-mcp-{alias}-authorization by matching the raw
lowercased alias/server_name/name only, but dashboard clients send
x-mcp-{sanitize_mcp_alias_for_header(alias)}-authorization, and egress resolves
those through lookup_mcp_server_auth_in_headers, which also tries the sanitized
alias. So a per-server token bound with a sanitized alias (e.g. alias 'pt-server'
arriving as header key 'pt_server') was forwarded at egress but still triggered a
preemptive 401 at connect. _client_has_per_server_auth_header now resolves through
the same lookup_mcp_server_auth_in_headers egress uses, so connect and egress
agree on which header names match.
|
||
|
Tin
|
ddec3b2b8b |
fix(mcp): plug fan-out Authorization bypass in the extra_headers loop
The listing fan-out withholds the request-wide Authorization from a true_passthrough / oauth_delegate server when another server in scope also consumes it, so one bearer is not replayed across upstreams. The later server.extra_headers copy loop did not honor that decision: a server listing Authorization in extra_headers would re-copy the withheld bearer from raw_headers. The withhold decision is now computed once and applied to both the forwarding branch and the extra_headers loop. |
||
|
Tin
|
edf00bbe23 |
fix(mcp): recognize per-server auth header at the connect-time preemptive 401
The preemptive 401 for true_passthrough and oauth_delegate only inspected the
request-wide Authorization, so a caller who bound the upstream token via the
per-server x-mcp-{alias}-authorization header (the required shape in a
multi-server aggregate, where the request-wide Authorization is withheld) was
spuriously 401'd at initialize even though egress already honors that header.
The gate now recognizes the per-server header for both modes via a shared
helper, mode-correctly: true_passthrough treats any Authorization or the
per-server header as the upstream token, oauth_delegate keeps requiring a
distinct x-litellm-api-key so a lone Authorization consumed for admission is
never mistaken for an upstream token. The preemptive raise is also gated to
single-server scopes so a multi-server aggregate degrades gracefully instead
of one missing token 401-ing the whole connect.
|
||
|
Tin
|
4a25cce114 |
fix(mcp): reject duplicate Authorization headers at MCP ingress
For the client-forwarded token modes the gateway relays the caller's Authorization to the upstream, so a request carrying more than one Authorization header would make which token is forwarded ambiguous (the ASGI header list collapses to last-wins) and could diverge from what admission inspected. Multiple Authorization headers is malformed for bearer auth anyway (RFC 9110: not a comma-combinable field), so the ingress header converter now fails closed with a 400 instead of silently keeping one. Applies to every MCP request, not just passthrough. |
||
|
David Katz
|
c0327cded4 |
fix(mcp): pair token-endpoint client_secret with the same source as client_id
On re-auth against a server with a persisted DCR client, register_client_with_server
short-circuits and returns a placeholder client_secret ("dummy") that the browser
echoes back to /token. exchange_token_with_server overrode the caller's client_id
with the persisted one but still fell back to the caller's secret when the server
had none stored, so a persisted public PKCE client (which has no secret) was paired
with the literal string "dummy" and the IdP rejected the exchange with 401 on every
re-authorization; the proxy surfaced that as a 500. First connects and brand-new
servers worked because a real DCR registration ran and no placeholder existed.
Resolve the secret from the server whenever the server's client_id wins, so a
secretless public client sends no client_secret at all
|
||
|
Yassin Kortam
|
bfff5e8d86
|
fix(mcp): log MCP tool calls returning isError=true as failures (#32238)
An MCP tool call that completes with CallToolResult.isError=true correctly returns HTTP 200 per the MCP spec, but the shared post-call logging helper always fired async_success_handler, so the standard logging payload carried status=success and OTel (whose _parse_error only marks ERROR on status=failure) showed green spans for failed tools. The helper now checks the result after async_post_mcp_tool_call_hook runs (guardrails may flip isError there) and routes error results to the failure path: success gates are consumed so the @client wrapper cannot enqueue a success log, failure_handler and async_failure_handler fire with a new MCPToolResultError carrying the tool's first text content, and post_call_failure_hook records the failure the same way raised exceptions already do. Raised exceptions never reach the helper, so no double failure logging. HTTP wire behavior is unchanged Resolves LIT-4081 |
||
|
tin-berri
|
1fb2b4aef4
|
fix(mcp): drop the cached per-user OAuth token when the credential row changes (#32302)
* fix(mcp): drop the cached per-user OAuth token when the credential row changes The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers, so a re-authorization or revocation wrote the DB while egress kept serving the replaced token from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate, MCPServerManager threads it to the write side, and the three credential write sites (the OAuth callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already feeds the rotated token back into the cache in the same fetch * test(mcp): pin cache invalidation on the revoke already-gone branch Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor moving the call inside the try block would silently skip the cache drop when the row was already deleted by a concurrent request while the cache still held the revoked token. The new test fails on exactly that mutation * test(mcp): cover invalidate on the redis-backed lazy store path Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised; the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain via a fetch and asserts a subsequent invalidate reaches the same store instance without a rebuild |
||
|
tin-berri
|
d6cbf6e7e3
|
feat(ui): expose MCP max_concurrent_requests in server create and edit forms (#32397)
* feat(ui): expose MCP max_concurrent_requests in server create and edit forms The proxy has enforced a per-server outbound tool-call concurrency cap (max_concurrent_requests) across every MCP egress path since #31641, and the management API has accepted the field on create and update all along, but the dashboard offered no way to set it. Add an optional Max Concurrent Requests input to the MCP server create and edit forms; it applies to every auth type and transport, so it renders unconditionally rather than gated on auth mode. Clearing the field on edit sends null so the stored limit is unset. Also rebuild the per-server semaphore when the configured limit changes. Previously the semaphore was created once per server_id and never resized, so an edited limit only took effect after a proxy restart even though the new value was persisted and reloaded into the registry. * feat(ui): mark MCP max concurrent requests field label as optional * test(ui): stop OBO create-form tests from timing out on CI The token-exchange payload test and the Entra scope-required test filled five text fields with user.type, which dispatches a full keystroke sequence per character; every input event runs the antd form onValuesChange handler and re-renders the whole CreateMCPServer tree, roughly 120 renders per test. As the form grew the two tests reached 8s and 18s locally, which crosses the 30s vitest timeout on slower CI containers; ui_unit_tests failed twice this way. Switch the plain text fields to fireEvent.change (one input event per field), matching the existing stdio test pattern. Both tests assert form output, not keystroke behavior, and now run in about 3s each. |
||
|
tin-berri
|
f922be32f0
|
fix(mcp): accept integer progressToken in host progress capture (#32402) | ||
|
Tin
|
732832d342 |
fix(mcp): bind client-forwarded Authorization to a single upstream
In a listing fan-out over a scope containing more than one server that
consumes the caller's Authorization (true_passthrough, oauth_delegate,
or the legacy delegate/passthrough shapes), the request-wide bearer is
now withheld from the new modes instead of being replayed against every
upstream (RFC 9700 cross-resource replay). Explicitly-addressed
operations (tool call, get_prompt, read_resource, single-server routes)
keep forwarding it.
Multi-server aggregates use the per-server x-mcp-{alias}-authorization
header instead: its value now feeds the passthrough resolver arm as the
inbound token and wins over the request-wide header, binding one token
to one server.
|
||
|
Tin Chi Lo
|
9bf3907de5 |
fix(mcp): bind oauth_delegate discovery resource to the upstream
oauth_delegate forwards the caller's token to the upstream, which validates its audience, so the protected-resource metadata must keep resource pointing at the upstream (returned verbatim, like true_passthrough) rather than rewriting it to the gateway. Rewriting to the gateway asks the client to mint a token bound to the gateway audience, which a strict IdP (Entra) refuses to issue for an unregistered resource and a spec-compliant upstream rejects on receipt. The legacy is_oauth_passthrough opt-in keeps the gateway rewrite unchanged. |
||
|
Tin Chi Lo
|
446a6e8cdd |
fix(mcp): route true_passthrough and oauth_delegate through upstream OAuth discovery
Both modes advertised LiteLLM as the authorization server and answered initialize locally, so a client with no token connected empty and was never driven into the upstream OAuth flow. The protected-resource discovery now proxies the upstream metadata for both modes (verbatim for true_passthrough, resource rewritten to the gateway for oauth_delegate), and the preemptive 401 emits the matching challenge: oauth_delegate uses the gateway-proxied resource_metadata once admission passes, true_passthrough probes the upstream anonymously and surfaces its WWW-Authenticate verbatim so the client authorizes directly against the upstream |
||
|
Tin Chi Lo
|
50c90281f4 |
feat(mcp): add true_passthrough and oauth_delegate auth modes
Introduce two first-class MCP server auth_type values that make LiteLLM's role in upstream authentication explicit, added alongside the existing delegate_auth_to_upstream / oauth_passthrough flags without changing their behavior. true_passthrough is a transparent proxy: LiteLLM performs no admission auth, requires no x-litellm-api-key, mints/stores/refreshes nothing, and forwards the client's Authorization to the upstream exactly as received. oauth_delegate keeps normal LiteLLM admission (x-litellm-api-key / SSO / JWT) and then forwards the client's separate upstream Authorization unchanged; the admission credential is never forwarded upstream. Both modes forward the caller's token via the existing extra_headers path and defer egress credential resolution to v1 (the v2 to_server_spec returns None for them). Upstream 401/403 responses are surfaced rather than swallowed so upstream OAuth challenges are preserved. Servers in either mode require per-user auth, so userless health checks are skipped. |
||
|
tin-berri
|
db2402754a
|
feat(mcp): let users select the entra_obo token_exchange profile in the UI and API (#32144)
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
tin-berri
|
9652509e46
|
fix(mcp): apply outbound concurrency limit to OBO tool calls (#32071)
The token_exchange (OBO) branch of _call_regular_mcp_tool built its coroutine by calling _obo_call_tool_with_retry directly, outside the _limit_outbound_concurrency context manager that the regular branch uses. OBO tool calls (and the internal re-mint retry, which issues a second upstream call_tool) therefore bypassed the per-server max_concurrent_requests semaphore, so an authenticated caller could run unlimited concurrent tool calls against an OBO MCP server despite an admin-configured limit. Wrap the OBO coroutine in _limit_outbound_concurrency the same way the regular path does, holding one permit across the initial call, the on-401 re-mint, and the retry, so OBO calls honor the configured cap. |
||
|
tin-berri
|
ff6dc33291
|
feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard (#31772)
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be configured through config.yaml; the create/update REST API and the dashboard had no way to express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end. These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first and falls back to the credentials blob so servers persisted before the columns existed still load. client_id and client_secret continue to ride the existing encrypted credentials path. On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from non-admin and virtual-key responses, matching how token_url is treated. * fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code: when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could not mint a token), the user must re-authorize via the browser flow. token_exchange has no gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was undefined (a compile error) and, per this file's convention and its tests, meant authorization_code; renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode instead and drop the now-unused isTokenExchange * fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes Switching an existing oauth2 server to oauth2_token_exchange left the old flow's token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url as the configured exchange endpoint, so the stale value both suppressed the RFC 9728/8414 discovery this PR adds and sent the exchange grant (client credentials plus the user's subject token) to the previous flow's token endpoint update_mcp_server now mirrors its existing stale-credentials rule for the flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow, token_exchange_endpoint, audience, subject_token_type): when auth_type changes, each one is cleared unless the same request explicitly provides it, so a deliberate override in the switch request still wins. Updates that keep the auth_type never touch these columns, which keeps legacy OBO rows that use token_url as their exchange endpoint working The edit form sends explicit nulls for the previous flow's fields on an auth type switch; antd preserves unmounted field values by default, so without this the old token_url would be re-sent verbatim and read as an explicit override. Transitions are detected against the persisted auth_type, so saves that keep the auth type send nothing extra Reported by Cursor Bugbot on the PR * fix(mcp): lift legacy blob token-exchange settings into their columns on every write The three token-exchange settings live in dedicated columns but also exist on MCPCredentials as the pre-column REST shape. Writes now lift incoming blob values into the columns (an explicit top-level value wins, including an explicit null) and strip them from the stored blob; the same-auth credentials merge migrates legacy rows the same way. The read-time column-or-blob fallback then only ever serves rows current code has never written, so clearing a column to re-enable RFC 9728/8414 discovery can no longer be silently undone by a stale blob copy. Also asserts the auth-switch clearing fires on the external fields_set path (PUT /v1/mcp/server). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(mcp): single source for the RFC 8693 default subject_token_type The default was applied at four egress build sites plus two model defaults, each with its own copy of the literal. All sites now share DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is deliberately not used: Prisma writes explicit values on insert, so a column default would rarely apply, and NULL-means-RFC-default keeps existing rows correct. Also documents two review decisions in place: the audience column keeps the RFC 8693 parameter name (RFC 8707 resource indicators are already a separate concept named resource in the v2 egress types), and the migration's out-of-order timestamp is safe under prisma migrate deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fix import sort order in outbound_credentials/types.py (I001 strict budget) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials The migrate-on-write in the credentials merge lifts blob values into null columns, which is correct for legacy rows but could repopulate a column an admin had cleared in an earlier no-credentials update (that path never touched the blob, so the stale copy survived to be lifted later). An explicit token-exchange column write (set or clear) now migrates the row even when the update carries no credentials: untouched null columns are lifted, every blob copy is stripped, and unrelated blob keys stay as-is. A cleared column can then never be resurrected, because no write path leaves a blob copy behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(mcp): state the blob-to-column lift contract on the legacy credential keys The three token-exchange keys on MCPCredentials are the pre-column REST shape (the only REST shape from 2026-05 until this PR). Document on both the blob type and the request models that the dedicated columns are authoritative and that writes lift blob values into them and strip the stored copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers The other two token-exchange fields were cleared while subject_token_type was left visible. It is a public RFC 8693 URN with no disclosure value, but the sanitizers' rule is that these views receive no token-exchange config at all — cleared for uniformity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
tin-berri
|
12801260ce
|
feat(MCP/UI): add OAuth flow selector on the MCP edit page (#32298)
* feat(ui): OAuth flow selector on the MCP edit page
The edit form had no flow selector: oauth_flow_type was watched but never registered,
so isM2MFlow was always false in edit mode and the flow could only be changed over
REST. That left the backfill's remediation for ambiguous legacy rows (client creds +
token_url, no interactive signal, left unstamped) without a dashboard path
The oauth2 section now opens with an OAuth Flow Type select. Explicit rows prefill
their stored value and re-persist it on save; legacy null rows show a placeholder
instead of a fake preselection, and an untouched save still writes nothing, so the
form never guesses on the admin's behalf. Choosing Machine-to-Machine (M2M) persists
oauth2_flow=client_credentials, choosing Interactive (PKCE) persists
authorization_code, which is exactly the assertion the backfill warning asks for.
Registering the field also brings the existing isM2MFlow gating in the edit form to
life, so M2M rows stop showing the interactive-only token-validation fields
Tests cover the prefill round-trip for both explicit values, the untouched null row
writing nothing, and both selections persisting on a legacy null-flow row
* fix(mcp): registry-to-table conversions must carry oauth2_flow
_build_mcp_server_table and the health-check table builder dropped oauth2_flow when
converting registry servers for GET /v1/mcp/server (list and by-id), so the dashboard
never received the persisted flow: the edit page could not prefill the selector, M2M
gating never activated, and the tools page classifier saw every oauth2 server as
interactive regardless of the column. Found live while proving the edit-selector
persistence path end to end; the write side was fine (PUT persists and the column
reads back correctly), the read side was dropping the field at the conversion
Both builders now carry oauth2_flow; regression test pins the conversion
* docs(mcp): flag _resolve_oauth2_flow as security-sensitive in its docstring
The prior wording ('not called directly by security sites') could read as if the
function has no security relevance, when it is the shape-inference engine both
request-time security helpers delegate to. Reword to state that plainly: it decides
M2M-vs-interactive for an unstamped row, must always be reached through
effective_oauth2_flow or resolve_oauth2_flow_for_request, and its M2M-shape branch
must not be weakened without accounting for those callers. Docstring-only; no logic
change
Raised by review on the stacked PR
* refactor(ui): extract oauth2FlowToFormValue helper for the MCP OAuth flow prefill
The edit form derived the OAuth Flow Type select value from the stored oauth2_flow
with a nested ternary duplicated at two call sites. Extract the mapping into a named
helper in types.tsx (next to getMcpOAuthMode and the flow constants): client_credentials
-> M2M, authorization_code -> Interactive, null/unset -> undefined so the select shows
its placeholder instead of a guessed default. The tool-config call site keeps its
null -> Interactive display fallback via a trailing ?? OAUTH_FLOW.INTERACTIVE, so
behavior is unchanged. Adds unit tests for the helper; the existing prefill/save tests
already cover the call sites
* feat(ui): surface and warn on an unset MCP oauth2_flow (server card + edit page)
An oauth2 MCP server whose oauth2_flow was never classified (legacy null row the
backfill left ambiguous) now advertises that it needs attention instead of silently
falling back. The server card shows an 'OAuth flow not set' warning tag for any
auth_type=oauth2 server with no oauth2_flow, so admins can spot them in the list
without opening each one. The edit page shows a warning alert directly under the new
OAuth Flow Type selector while the flow is unset, and it clears the moment a flow is
picked.
Delegate (delegate_auth_to_upstream) servers are excluded from both: they authenticate
via upstream PKCE passthrough and route to passthrough regardless of oauth2_flow, so
the M2M-vs-interactive classification does not apply and prompting for it would be a
false alarm. The edit page reads the delegate state from the watched switch when it is
mounted and falls back to the stored value otherwise (useWatch returns undefined for an
unmounted field).
Also adds end-to-end coverage of the null-flow chain the selector depends on:
build_mcp_server_from_table carries oauth2_flow=None verbatim into the GET response,
so the dashboard maps it to undefined and shows the placeholder rather than a guessed
default. Tests: backend null carry, the select prefill display for all three states,
the edit-page warning show/hide/clear-on-select and delegate exclusion, and the card
badge across oauth2/non-oauth2, stamped/unstamped, and delegate
|
||
|
tin-berri
|
7ce573e6e8
|
fix(mcp): stop 'Team doesn't exist' warnings for UI dashboard sessions (#32348)
UI session tokens carry the virtual team_id litellm-dashboard (UI_TEAM_ID), which is never persisted. The MCP team-permission helpers passed it to get_team_object anyway, so every dashboard MCP listing raised a 404 per lookup that was swallowed into per-server 'Failed to get allowed tools for server' warnings (plus the sibling 'allowed MCP servers for team' and 'MCP access groups for team' warnings) and wasted DB queries. The 404 also escaped past the key-level permission handling in get_allowed_tools_for_server, dropping key tool restrictions for such sessions. Short-circuit the virtual team before the DB lookup in the three helpers, mirroring the existing UI_TEAM_ID handling in agent_permission_handler. Also reject /team/new with the reserved team_id, since a real row would bind its budget and permissions to every UI session |
||
|
tin-berri
|
733c01902f
|
feat(mcp)!: oauth2_flow read verbatim from DB rows and required in config; inference reduced to the request-time backstop (#32292)
* refactor(mcp): read oauth2_flow verbatim from DB rows; inference stays config-only plus a logged backstop With every DB write site stamping oauth2_flow (#32283, #32288) and the startup backfill healing legacy null rows, the DB build no longer needs to re-derive the flow from field shape. build_mcp_server_from_table now reads the column verbatim via _explicit_oauth2_flow: unknown or null values resolve to None, which needs_user_oauth_token already treats as interactive, so an unstamped row degrades to the safe default instead of guessing M2M from a shape that a DCR-registered interactive server shares whenever discovery is down Field-shape inference survives in exactly two places. config.yaml-loaded servers keep it at load time: they are rebuilt from the config on every boot, so there is no row to backfill and load-time resolution is their write-time stamp. And the request-time backstop in _get_allowed_mcp_servers keeps a not-yet-backfilled M2M row blocking caller Authorization forwarding (the P1 property); it now logs a warning whenever it actually fires, which is the fire-rate signal for deleting it once deployments have booted past the backfill Regression tests pin that the DB build does not infer M2M from the credential shape and reads an explicit column value verbatim Fourth step of the oauth2_flow persistence sequence, stacked on the backfill * feat(mcp): deprecation warning when config-level M2M is inferred rather than declared A config.yaml oauth2 server whose credential shape decides client_credentials without an explicit oauth2_flow now logs a warning at load pointing the admin at the explicit declaration. First rung of the deprecation ladder: the docs make oauth2_flow the recommended path, the warning surfaces configs still relying on inference, and a future breaking release can turn it into a config validation error, at which point config-level shape inference dies entirely. Interactive omissions stay silent since the default matches inference there and nothing load-bearing is being guessed * feat(mcp)!: require explicit oauth2_flow for config-defined oauth2 servers A config.yaml server with auth_type oauth2 must now declare its flow; the load raises a config validation error naming both values and what each means: oauth2_flow: client_credentials for machine-to-machine (the proxy mints a shared token at token_url using client_id/client_secret) or oauth2_flow: authorization_code for interactive (per-user tokens via browser sign-in, including delegate_auth_to_upstream) This replaces the load-time shape inference for config servers entirely. The credential shape is genuinely ambiguous (a DCR-registered interactive server carries client creds + token_url with no authorization_url, identical to M2M), so the config asserts the answer instead of the proxy guessing it. With this, field-shape inference survives in exactly one place: the request-time security backstop, which is telemetry-gated for deletion BREAKING CHANGE: config-defined oauth2 MCP servers without oauth2_flow fail proxy startup with the error above. Add the one line to the server block; the error text says exactly which value to pick * test(mcp): pin the verbatim read for authorization_code alongside client_credentials Raised by review on the PR * fix(mcp): fail closed on the anonymous delegate gate for unstamped M2M-shaped servers Reading oauth2_flow verbatim (this PR) changed has_client_credentials from True to False for a legacy null-flow row that still carries the M2M credential shape. That value is what the anonymous upstream-delegate gate checks before skipping LiteLLM auth entirely, so an M2M-shaped delegate server that was never stamped would newly pass the gate: an unauthenticated caller could get it selected and then list/read upstream data using the client credentials the request-time backstop re-infers, running as LiteLLM's service account. This reopens the hole the gate's existing 'never delegate for M2M' guard was written to close The gate now resolves the flow (column first, shape fallback) instead of reading the bare column, mirroring the request-time backstop in _get_allowed_mcp_servers: both fail closed on the ambiguous M2M shape and are removed together once no null rows remain. A pure-PKCE delegate server (no stored credentials) resolves to a non-M2M flow and keeps its bypass, so the common delegate case is unaffected Tests: an unstamped M2M-shaped delegate server is denied the bypass (mutation-checked against the bare-column regression), and a pure-PKCE delegate server still bypasses Raised by review on the PR * fix(mcp): centralize the request-time oauth2_flow backstop across every security site Reading oauth2_flow verbatim made has_client_credentials unreliable for legacy null rows, and the backstop that compensates was applied at only one reader. Review found three more consequences of that per-site approach: - the anonymous-delegate allowlist in get_allowed_mcp_servers read the bare column, so an unstamped M2M-shape delegate server was surfaced to anonymous callers (High) - call_mcp_tool resolved allowed ids into MCPServer objects without the backstop, so a null-flow M2M-shape row kept has_client_credentials false on tool execution during a backfill gap, though the listing path was covered (High) - the request-time warning claimed the startup backfill would stamp the row next boot, but the backfill deliberately leaves the ambiguous M2M shape unstamped (Low) Rather than patch each site, introduce two helpers on MCPServerManager that are the single choke point for request-time resolution: effective_oauth2_flow(server) for the enum/boolean decisions (allowlist filter, anonymous-delegate gate) and resolve_oauth2_flow_for_request(server) for the egress object copy (listing and tool call). Both fail closed on the M2M shape and leave stamped rows and pure-PKCE rows untouched. The gate now shares effective_oauth2_flow instead of its inline resolution, and the corrected warning lives once inside resolve_oauth2_flow_for_request, so deleting the whole transitional layer later is a single-site change. Tests: helper unit coverage (stamped verbatim, null M2M-shape resolves, pure-PKCE stays None, stamped/pure-PKCE return the same object, corrected warning text), the anonymous allowlist excludes an unstamped M2M-shape delegate server, and the call path resolves the flow like the listing path. The two security-integration tests are mutation-checked against the bare-column regression. Raised by review on the PR |
||
|
tin-berri
|
6041d37414
|
fix(mcp): apply semantic filter to expanded litellm_proxy tools and show filtered-out count (#32285)
* fix(mcp): apply semantic filter to expanded litellm_proxy tools and show filtered-out count * fix(mcp): guard expansion-path filtering on enabled flag and isolate metadata emission |
||
|
Sameer Kankute
|
a78dc69a09
|
fix(mcp): alias/display-name tool routing, REST filters, BYOK auth (#32320)
* fix(mcp): resolve tool name prefix via known server prefixes, not string match When an MCP server's alias differs from its server_name, tool names are listed with the alias prefix but _execute_tool_calls compared that prefix against the server_name stored in tool_server_map. The mismatch silently skipped prefix stripping, forwarding the fully-prefixed tool name upstream and causing "Unknown tool" failures. Resolve the actual MCPServer object and strip using its known prefix forms (alias, server_name, server_id) instead. * fix(mcp): preserve tool overrides and scope REST tool listing Return saved tool display/description overrides from the server table API so the edit UI reloads them, resolve display names before prefix stripping on tool calls, and honor mcp_server_name and toolset_name filters on the REST tools list endpoint. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): inject BYOK credentials on Playground OpenAPI tool calls Playground and Responses API route MCP execution through call_tool, which skipped BYOK lookup and never set the OpenAPI auth ContextVar, so upstream calls went out unauthenticated despite a stored user credential. Co-authored-by: Cursor <cursoragent@cursor.com> * test(mcp): cover alias-mismatch prefix stripping and display-name reverse mapping Regression tests for _execute_tool_calls: an MCP server whose alias differs from its server_name must still have its tool-name prefix stripped correctly, and a tool called by its configured display name must resolve back to the original tool name before dispatch. * fix(mcp): validate tool display names against Bedrock's tool-name pattern A display name replaces the tool name sent to the LLM provider, so a value with spaces or other special characters saves successfully but fails every subsequent Bedrock tool call. Validate tool_name_to_display_name server-side (create/update payload) against Bedrock's [a-zA-Z0-9_-]+ constraint, and add matching inline validation plus a save-blocking guard in the Admin UI's create and edit MCP server forms. * style(mcp): fix ruff/prettier formatting on CI No logic changes; satisfies the format checks flagged on PR #32320. * fix(mcp): fix CI failures on PR - complexity budget and stale test mock Extract toolset-scope resolution and query-param normalization out of list_tool_rest_api into helpers to bring it back under the C901 complexity budget (was 18, now within the 15 threshold). Add the missing get_mcp_server_by_name stub to the streaming iterator test's mock manager; the alias-fallback resolution added for tool-name-prefix stripping calls it unconditionally when _get_mcp_server_from_tool_name misses. * test(mcp): cover BYOK OpenAPI auth-header helpers to close codecov patch gap _format_byok_openapi_auth_header, _openapi_forwarded_extra_headers, and _resolve_byok_mcp_auth_header were only exercised indirectly via a mocked call_tool test, leaving their branches (auth-type formatting, header forwarding/stripping, missing-credential 401) uncovered. * fix(mcp): resolve BYOK auth before queuing the during-hook task _resolve_byok_mcp_auth_header can raise a 401 when no credential is stored. Resolving it after during_hook_task was already queued meant a hook's side effects (audit logging, rate-limit bookkeeping) could run and record success for a tool call that then fails on the missing credential. * fix: correct mcp alias routing regressions --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
tin-berri
|
5e73994441
|
fix(mcp_semantic_filter): keep tool names whole in filter response header (#32282)
The x-litellm-semantic-filter-tools response header was sliced mid-name at MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH with a trailing "...", so the admin UI test panel rendered the last selected tool name chopped. Truncate the CSV at a tool name boundary instead so the header only ever carries complete names, and note in the test panel how many selected tools did not fit in the header |
||
|
tin-berri
|
4e3ebbb164
|
feat(mcp): startup backfill stamping oauth2_flow on legacy null rows (#32290)
* feat(mcp): startup backfill stamping oauth2_flow on legacy null rows Rows created before the write-side stamps carry a null oauth2_flow and rely on read-time field-shape inference, which cannot tell a DCR-registered interactive server (client creds + token_url, no persisted authorization_url) from an M2M server unless endpoint discovery succeeds first; on a transient discovery failure those servers flip to client_credentials for that registry load The backfill classifies each null oauth2 row once, at rest, ordered by signal strength: per-user token rows (only the interactive flow mints them, so this is definitive and catches the DCR-trap cohort), then a persisted authorization_url, then a persisted registration_url (DCR implies interactive; this covers registered-but-never-signed-in rows), then the M2M credential shape mirroring the legacy inference, else the interactive default that matches how needs_user_oauth_token treats a null flow. Every stamp is logged with the rule that fired and written with updated_by=oauth2_flow_backfill for auditability Runs in _init_mcp_servers_in_db before the registry load so the first build of the boot classifies from the column, is isolated so a failure cannot block server loading, and is idempotent: a healed fleet exits after one indexed query. This unblocks deleting the read-time inference for DB rows in the follow-up Third step of the oauth2_flow persistence sequence, after #32283 and #32288 * fix(mcp): backfill leaves the ambiguous M2M shape unstamped instead of guessing client_credentials The credential shape (client_id + client_secret + token_url, no interactive signal) is shared by real M2M servers and DCR-registered interactive servers nobody has signed into: the DCR persist writes creds and token_url but not authorization_url or registration_url. Stamping client_credentials from that shape permanently mislabeled the interactive cohort, and once explicit the value is authoritative, so per-user traffic would run on the proxy's stored client credential with no discovery rescue and no backstop (it only guards null rows) The backfill now stamps only what it can prove. Interactive signals keep stamping authorization_code; the ambiguous shape is left null with an actionable warning naming the server and the fix (set oauth2_flow via the dashboard or PUT /v1/mcp/server). A true M2M row keeps working per-request through the security backstop while the warning nags; an interactive row keeps its Authorize button (null renders interactive), and one completed sign-in creates the per-user token that stamps it authorization_code at the next boot. Mirrors the config-level rule: M2M is asserted by a human, never guessed Raised by review on the PR * perf(mcp): batch the backfill stamps into one update_many per flow value The per-row update loop issued one DB round-trip per legacy row at startup; rows sharing a stamped value now go out as a single update_many, so the DB cost is constant in fleet size. Per-row logging keeps the rule that fired for each server Raised by review on the PR * fix(mcp): backfill stamps only rows still null at write time and counts only real OAuth token rows as sign-in proof Two review findings. The batched update_many matched on server_id alone, so an explicit oauth2_flow set between the backfill's read and its write (an admin PUT or a sign-in's DCR stamp landing in the boot window) would be overwritten with the inferred value; the where clause now also requires oauth2_flow to still be null, so an explicit value can never be clobbered under any interleaving And the per_user_tokens rule counted any LiteLLM_MCPUserCredentials row as proof of an interactive sign-in, but that table doubles as BYOK storage for user-supplied API keys; a BYOK-flavored row would have stamped an M2M-shaped server authorization_code. The rule now counts only rows whose payload decodes as a type oauth2 token via the existing _decode_oauth_payload discriminator, so bare keys, undecodable rows, and stale leftovers from a BYOK-to-oauth2 auth switch prove nothing Raised by review on the PR |
||
|
tin-berri
|
fc3c21e837
|
fix(mcp): forward short OAuth state upstream, keep session in a cookie (#32146)
* fix(mcp): forward short OAuth state upstream, keep session in a cookie Some upstream authorization servers reject the OAuth authorize request with "state parameter too long" because LiteLLM replaced the client's short state with its own long encrypted session blob (base_url, original state, PKCE, client redirect_uri) and sent that upstream as state. Forward a short random handle as the upstream state instead, and carry the encrypted session in a per-flow HttpOnly, SameSite=lax cookie bound to that handle. The browser replays the cookie on /callback, so the session is recovered without any server-side store and the client still gets its own original state back. /callback falls back to decoding state directly when no cookie is present, so flows in flight across a deploy keep working. Resolves LIT-4197 * test(mcp): cover /callback error path cookie read and clear The happy-path regression test already asserts the short-handle -> cookie round trip. Add a focused test for the IdP-error branch of /callback: it must recover the client's original state from the per-flow cookie (not the short handle), propagate the error to the client's redirect_uri, and expire the one-time cookie. Fails if the error path stops reading or clearing the cookie. |
||
|
tin-berri
|
f5ea72b1b8
|
fix(mcp): stamp oauth2_flow=authorization_code when persisting a DCR client registration (#32283)
Only the gateway-managed interactive flow reaches this persist (the public /register routes never pass persist_credentials), so the row it writes is authorization_code by definition. It was not recorded, which left the row as client creds + token_url with no persisted authorization_url and a null oauth2_flow: exactly the shape the legacy M2M inference in _resolve_oauth2_flow matches. The row normally survives because endpoint discovery backfills authorization_url in memory before the inference runs, but on any transient discovery failure at registry build the server flips to client_credentials for that load, routing per-user traffic to the M2M path Stamping the flow at the write site makes the classification explicit and permanent, so a DCR-registered interactive server no longer depends on discovery succeeding to classify correctly. First step of persisting oauth2_flow at every write site so the legacy inference can eventually be deleted |
||
|
tin-berri
|
2e38da6b3e
|
feat(mcp): add entra_obo profile to the token_exchange (OBO) arm (#31983)
* feat(mcp): add entra_obo profile to the token_exchange (OBO) arm
Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects
The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration
Resolves LIT-4163
* feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401
An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with
error=interaction_required and a claims blob the client must satisfy to step up. The arm
dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was
unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims
string off the rejection body (error_description is still never carried; it can leak IdP
internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the
challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain
OAuth token (guards against header injection from a hostile body) and the claims base64-encoded
in a claims parameter, the convention MSAL-family clients decode. With neither field the header
is byte-identical to the static challenge. The multi-server aggregate still absorbs a
step-up 401 to an empty listing; only single-server routes surface it
* fix(mcp): use error=insufficient_claims for the Entra step-up challenge
Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set
error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and
replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims
whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate
error_description in the claims case. The presence of claims now drives the error value, so the
raw oauth_error no longer needs threading from the provider through CredError to the edge; that
plumbing is removed (the provider still reads the error code for its gateway-fault classification).
Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches
the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO
POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and
returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside
resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP)
* fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed
The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the
config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the
exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its
fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of
failing before any network call. Config-load now filters blanks the same way, so an all-blank list
normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo
server and asserts the exchange returns misconfigured without POSTing
|
||
|
tin-berri
|
c737789b26
|
feat(mcp): discover the OBO token endpoint via RFC 9728 -> RFC 8414 (no IdP guessing) (#31762)
* feat(mcp): discover the OBO token endpoint via RFC 9728 to RFC 8414 (no IdP guessing) An oauth2_token_exchange server can now have its token endpoint discovered the same way the oauth2 (authorization_code) flow already does, instead of always requiring token_exchange_endpoint/token_url to be configured by hand. The existing _descovery_metadata chain (RFC 9728 protected-resource metadata -> RFC 8414 authorization-server metadata -> token_endpoint, SSRF-guarded via async_safe_get) is reused; both the config-load and DB-build paths gate on a new _obo_needs_endpoint_discovery so discovery runs only when no endpoint is configured, and an explicitly configured endpoint still wins and skips the round-trip. The discovered token endpoint lands on token_url, which _token_exchange_spec already reads, so no resolver change is needed. _resolve_oauth2_flow returns None for any non-oauth2 auth_type, so a discovered token_url on an OBO server is never mis-inferred as the M2M client_credentials flow. Discovery for OBO is authoritative only: the resolution order is explicitly configured endpoint, then RFC 9728 -> RFC 8414 advertisement, then fail closed (412, on the parent commit). The gateway never guesses the IdP. _descovery_metadata grows an allow_origin_fallback flag, kept True for the browser oauth2 flow (a human sees the redirect) but set False for token_exchange so the last-resort guess that treats the resource server's own origin as its authorization server is skipped; a subject token is never exchanged against an inferred endpoint. * fix(mcp): surface a failed OBO exchange at connect instead of an empty tool list A token_exchange server whose exchange fails with a subject present used to open the MCP session anyway and mask the failure as an empty tools/list. Single-server routes now run the exchange preemptively at the transport edge, where a rejected subject raises the RFC 9728 challenge and a gateway fault its public status; the multi-server aggregate keeps absorbing per-server auth failures. The exchanger caches the preflight result, so the session's list/call reuses it with no extra IdP round-trip. Discovery now also debug-logs the authorization server's advertised issuer, grant types, and client auth methods * fix(mcp): persist the discovered OBO token endpoint to the DB row A DB-backed oauth2_token_exchange server with no configured endpoint had its token_url resolved via RFC 9728 -> RFC 8414 only on the in-memory object returned from build_mcp_server_from_table; the row kept token_url=None, so every rebuild re-ran discovery and a transient upstream outage during a rebuild left the server with no endpoint until the next successful discovery. Write the discovered token_url back onto the row so the guard sees it on the next build. Best-effort and scoped to DB servers: config servers already persist in-memory, and the write-back never fires from a user connect (only from add/update/reload, all admin or system driven). Adds DB-path coverage for discovery firing when unset, skipping when the credentials endpoint is configured, the write-back, and its negative guards |
||
|
tin-berri
|
0e56fc39e2
|
feat(mcp): make token_exchange (OBO) production-ready - discovery threading + audit hardening + RFC 9728 challenge (#31622)
* feat(mcp): thread the caller token into tools/list discovery for token_exchange
A token_exchange (OBO) server's tools could not be discovered through the aggregator: the list path
never threaded the caller's token, so every tools/list hit the no-subject branch. v1 masked this with
its client_credentials fallback (discovery used a service token); v2 dropped that fallback, so listing
had no credential and the OBO server's tools never appeared - and an MCP client lists before it calls.
Thread the inbound subject_token into the list path the same way the call path does, gated on
auth_type oauth2_token_exchange so the caller's bearer never leaks into other modes:
_get_tools_from_server takes an oauth2_headers param, extracts the token via _extract_bearer_token, and
passes it to _create_mcp_client; server.py forwards oauth2_headers at the list call site.
authorization_code (resolves off identity plus stored token), the static/config modes, and the
background registry refresh are unaffected, and the list path's existing graceful degradation
(catch -> empty list) is preserved.
* fix(mcp): harden token_exchange OBO from the audit (strip, TTL/expires_in, subject_token_type)
- _should_strip_caller_authorization returns True for oauth2_token_exchange, so the inbound subject
token is never forwarded upstream raw - only the IdP-exchanged token is (matches authorization_code).
- _parse_expires_in accepts a JSON float / numeric-string expires_in, and _ttl_seconds caps the cache
TTL at the token's real remaining lifetime so a short-lived exchanged token is never served stale.
- to_server_spec normalizes a falsy subject_token_type to the default URN, parity with v1.
The subject/key disambiguation (never exchange the LiteLLM key; Authorization: Bearer <litellm-key>
support for /mcp) is intentionally a separate cross-cutting PR off staging, not part of this OBO work.
* fix(mcp): stop caller header bypassing OBO exchange; thread subject into prompts/resources
The per-server x-mcp-* override guard in _create_mcp_client only kept the v2 spec
for authorization_code, so a caller-supplied header silently disabled the RFC 8693
exchange on a token_exchange server and forwarded the raw bearer upstream. Extend
the guard to token_exchange so the exchange always runs and the caller cannot
substitute an arbitrary upstream credential.
prompts/list+get, resources/list+read, and resource-templates/list never threaded
the OBO subject token, so those operations failed closed (401 / empty) on a
token_exchange server. Thread the caller's bearer as the subject for those paths
too, gated on the token_exchange mode via a shared _obo_subject_token helper.
* fix(mcp): keep the OBO/authz_code resolver credential authoritative; centralize OpenAPI strip
A guardrail (e.g. MCPJWTSigner), static_headers, or any other injected Authorization could
shadow the resolver-owned credential for token_exchange / authorization_code servers, so the
upstream would receive e.g. the signer's JWT instead of the exchanged token and reject it. In
_create_mcp_client the resolver-owned credential now wins: a conflicting header is dropped and
the minted/stored token reaches upstream. No behavior change for none/passthrough/static modes,
where an injected Authorization still wins as before.
The OpenAPI/local _request_extra_headers forwarder gated its Authorization strip on
has_client_credentials only, so an OpenAPI-backed token_exchange server with
extra_headers:[Authorization] forwarded the raw subject token upstream and never exchanged. It
now uses the centralized _should_strip_caller_authorization so it matches the managed paths.
* feat(mcp): RFC 9728 challenge for token_exchange (OBO) unauthorized
OBO previously returned an opaque 401 (Bearer error="invalid_request") with no discovery
info, and any IdP exchange failure collapsed to a retryable 503. Now an OBO server behaves like
a standards-compliant OAuth resource server:
- A missing/rejected subject token returns the RFC 9728 / RFC 6750 challenge: 401 +
WWW-Authenticate: Bearer resource_metadata="...", error="invalid_token", so a spec-compliant
MCP client can discover the IdP, SSO, and retry with a fresh subject token.
- The protected-resource metadata for a token_exchange server advertises the JWT-auth issuer(s)
(JWT_ISSUER / litellm_jwtauth.issuers) as authorization_servers -- the IdP that issues and
validates the subject -- instead of the gateway.
- An IdP 4xx (subject rejected) is now a non-retryable 401 (the challenge) instead of a 503, so a
caller with a dead token re-authenticates rather than looping; 5xx/transport stays retryable 503.
* fix(mcp): emit the OBO RFC 9728 challenge preemptively so a no-subject client can discover the IdP
A token_exchange server's tools are not discoverable without a subject token (list is lenient ->
empty), and a tool-call-time 401 is wrapped into a JSON-RPC error so the WWW-Authenticate header is
lost. So a cold-start client never saw the challenge and could not start discovery. Add a
token_exchange branch to the preemptive-401: a no-subject connect to an OBO server now returns
401 + WWW-Authenticate: Bearer resource_metadata=..., error="invalid_token" at the transport level,
so a spec-compliant client discovers the IdP (the PRM advertises the JWT-auth issuer), SSOs, and
retries with a subject token. Verified live on the per-server endpoint; the with-subject connect
still proceeds (no challenge).
(Also formats two lines from earlier commits in this stack.)
* refactor(mcp): inject root_path into the OBO/OAuth challenge edge
The adapter's raise_user_oauth_challenge and raise_token_exchange_challenge
reached into os.getenv("SERVER_ROOT_PATH") via get_server_root_path(), a
hidden ambient read in a module that is meant to be a pure edge. That coupling
made the preemptive-challenge test order-dependent under xdist: a sibling test
sets SERVER_ROOT_PATH at import without cleanup, leaking the prefix into the
challenge URL and failing the exact-match assertion.
Resolve the root path at the imperative-shell call sites and pass it in
keyword-only, so both challenge builders become pure functions of their inputs.
Extract the shared resource_metadata path construction into a single
oauth_protected_resource_path helper, collapsing the duplicated prefix/name
logic the two functions carried.
Also reduce _create_mcp_client below the strict complexity ceiling by extracting
the v2 credential resolution into _resolve_v2_auth, and extract the OBO
protected-resource-metadata branch into _obo_protected_resource_response (which
shipped without coverage) so discovery can be unit-tested directly.
Tests are now hermetic: the adapter tests pass root_path as a real input rather
than monkeypatching the environment, the stale-session preemptive test asserts
structural invariants instead of the exact prefixed URL, and five new tests
cover the OBO PRM issuer branch end to end.
* feat(mcp): OBO cache-key tenant isolation, reactive 401 retry, v1-parity logs
From a pass over the OBO behavior contract. Three changes to the
token_exchange arm, none of which alters any other auth mode.
The exchanged-token cache key now folds in the caller's tenant alongside
the subject token and exchange config, so two tenants presenting the same
opaque token can never share a cache entry; cross-tenant isolation is
structural rather than incidental to subject-token uniqueness. tenant_id is
threaded from the resolver's Subject; it is keyword-only with an empty
default so the no-tenant case and the existing call sites are unchanged.
The tool-call path gains one reactive retry. When an upstream rejects the
injected token with a 401/403, the gateway invalidates the cached exchange,
re-mints once through the IdP by rebuilding the client, and retries the call
exactly once before surfacing the upstream error, so a token revoked or
rotated upstream mid-TTL self-heals without an infinite loop. It is gated
strictly to oauth2_token_exchange; passthrough, authorization_code,
client_credentials, api_key, and none keep their single-call behavior.
MCPClient.call_tool gains a raise_on_error flag (mirroring list_tools) so
the path can tell an upstream 401 apart from an ordinary tool error and
avoid re-running a non-idempotent tool on a non-auth failure.
The exchanger also emits the v1-parity log lines it had dropped (attempt
with server, endpoint and audience; success; cache hit), while never
logging the form, subject token, secret, or minted token.
* fix(mcp): fail closed with 412 when a token_exchange server has no endpoint
A true token_exchange (OBO) server must use only an explicitly configured
token endpoint; it must never guess an IdP or silently fall back to a weaker
source. Previously an OBO server with client credentials but no
token_exchange_endpoint/token_url deferred to v1, which no-op'd and let the
request connect to the upstream with no credential (an upstream 401 rather
than a clear gateway error).
Now such a server is owned by the v2 arm: _token_exchange_spec builds the spec
even when the endpoint is absent, and the exchanger fails closed with a
precondition_required error that maps to HTTP 412 before any upstream or IdP
call, with the caller's subject token never sent anywhere. A missing
client_id/secret still maps to misconfigured (500); a present-but-rejected
subject still maps to 401; an unreachable IdP still maps to 503. The no-subject
case keeps its existing 401 RFC 9728 challenge.
* feat(mcp): log a refused non-Bearer token_type in the OBO exchange
* fix(mcp): surface OBO/authorization_code list-time 401 as a challenge instead of masking it
* feat(mcp): classify RFC 6749 gateway-fault token-exchange errors as 500, not a caller 401
* test(mcp): absorb fixture uses 500 now that 401/403 are challenge-class at list time
* style(mcp): PEP 604 union in the OBO retry signature to keep the UP007 budget flat
|
||
|
tin-berri
|
f19bf2c984
|
feat(mcp): migrate the token_exchange (OBO) arm to the v2 resolver (#31526)
* feat(mcp): v2-native RFC 8693 token exchanger for the token_exchange mode Adds the pure Rfc8693TokenExchanger plus its composition root: the OBO exchange POSTs the RFC 8693 grant through an injected HTTP edge and returns the upstream-bound token as a typed Result, caching and single-flighting per (subject_token, server) so a repeated caller token skips the IdP round-trip. The audience is carried on TokenExchangeConfig and sent only when the operator set one, matching the spec default behavior. Errors are values: a missing endpoint or client credential is misconfigured, an IdP that returns no usable token is upstream_unavailable. * feat(mcp): migrate the token_exchange arm to the v2 resolve_credentials Routes RFC 8693 OBO servers through the v2 resolver: the resolver arm reads the caller's inbound token and swaps it via the injected TokenExchanger, to_server_spec maps a complete oauth2_token_exchange server (endpoint plus client credentials) to TokenExchangeConfig, and the egress wires the LazyTokenExchanger in. A token_exchange server with no caller token fails closed with a plain 401 rather than v1's fall-through to client_credentials, so the call site now scopes the per-server browser-OAuth challenge to authorization_code and lets other modes raise their own. * fix(mcp): bind the token-exchange cache key to the exchange config The exchanged-token cache was keyed only by (subject_token, server_id), so rotating a server's audience, scope, endpoint, client_id, or secret kept serving a token minted for the old config until TTL. The key now hashes the caller token together with the config that minted it, so a config change forces a fresh exchange. Everything is hashed, so no secret is held in the key. * refactor(mcp): build the token exchanger eagerly, dropping the lazy wrapper The token exchanger reads no runtime global at build time (its httpx client is acquired per call), unlike the per-user store, so it does not need lazy first-use construction. Building it once at egress construction removes the first-use init path entirely and keeps the process-lifetime cache. * fix(mcp): map non-object token-exchange JSON to a miss instead of a 500 The post adapter annotated the parsed body as a dict without checking it, so a valid-but-non-object JSON response (list/string/number) was returned as-is and crashed the field parsing with an AttributeError. It now validates the shape at the boundary and returns None for a non-object body, so a malformed IdP response surfaces as a typed upstream_unavailable rather than a server error. * fix(mcp): fail closed on a non-Bearer token_type in the OBO exchange * feat(mcp): honor token_endpoint_auth_method (client_secret_basic) in the v2 OBO exchange * feat(mcp): reject a non-access issued_token_type in the OBO exchange * fix: include token endpoint auth method in exchange cache key --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
57ca48a863
|
feat(mcp): add all-proxy-mcpservers sentinel to grant teams every MCP server (#32012)
* feat(mcp): add all-proxy-mcpservers sentinel to grant every MCP server Teams can now be scoped to the all-proxy-mcpservers sentinel so they gain access to every MCP server on the proxy without listing each id. The sentinel expands to the live registry at request time, so a server added later is picked up with no change to the team's stored permission. The team ceiling that validates a key's MCP scope expands the sentinel too, so a key can be scoped to any server (including one registered after the team) and still pass subset validation Expose the option in the team create and edit forms via a new exclusive "All Proxy MCP Servers" choice in MCPServerSelector, mirroring the existing "No MCP Servers" sentinel * Update litellm/proxy/management_helpers/object_permission_utils.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mcp): honor all-proxy-mcpservers only on the team path, never per-key The sentinel was expanded inside the shared expand_permission_list, which also feeds the key, org, end_user and agent resolvers. A key whose stored object_permission ever held all-proxy-mcpservers (a stale write, a configured default, or a bug) would silently resolve to every MCP server at runtime, and a teamless key had nothing to cap it, so all servers got injected. Only write-time validation stripping the value stood between that value and a full grant Move the expansion out of expand_permission_list and into _get_allowed_mcp_servers_for_team so the sentinel is honored only where it is settable (a team). Anywhere else it now passes through as an inert literal that matches no registered server and is denied downstream. Reserved-id protection already blocks a real server from taking that id * fix(mcp): require proxy admin to grant a team the all-proxy MCP sentinel Granting a team every MCP server on the proxy is a proxy-wide authorization decision, but team create/update let any caller who can manage a team set object_permission.mcp_servers, with no ceiling check. Org admins reach /team/update by default (org_admin_allowed_routes) and _verify_team_access also admits team admins, so a non-proxy-admin could set all-proxy-mcpservers and self-grant their team access to every MCP server on the proxy, including servers never assigned to that team Gate the grant in new_team and update_team: a non-proxy-admin cannot add the all-proxy-mcpservers sentinel. The check is scoped to newly adding it, so a team a proxy admin already scoped to all-proxy can still be edited by a team admin without being forced to strip the sentinel. The UI only offers the "All Proxy MCP Servers" option to proxy admins in the team create and edit forms * fix(ui): render friendly all-proxy MCP label for non-admins editing an all-proxy team A team scoped to the all-proxy-mcpservers sentinel could be opened in the team edit form by a team admin or org admin (canEditTeam admits them), but the "All Proxy MCP Servers" option in MCPServerSelector was rendered only behind the proxy-admin-gated allowAllProxyMcpServers flag. For a non-proxy-admin the stored sentinel was hydrated into the selected value with no matching Select.Option, so antd showed the raw all-proxy-mcpservers literal as a chip, and adding another server could persist a mixed [all-proxy-mcpservers, <id>] value. Render the option whenever the sentinel is present in the value, not only when the caller may grant it, and drive the real-option disabling off presence too so the selection stays exclusive. A non-proxy-admin now sees the friendly label read-only and cannot build a mixed state; only a proxy admin can newly add it, which the backend already enforces. Adds regression tests: the selector shows the friendly option (not the raw literal) when the sentinel is stored but the grant flag is off, plus exclusive emit and disabled-real-options coverage, and MCPServerPermissions renders the green "All" state instead of the raw sentinel string. * fix(ui): drop redundant "All servers" hint from the all-proxy MCP chip antd renders a Select option's children inside the selected tag, so the all-proxy option showed both "All Proxy MCP Servers" and the green "All servers" type-hint in the chip, which say the same thing. Collapse the option to a single green "All Proxy MCP Servers" label so the dropdown row and the chip read cleanly without the duplication. * fix(ui): color the all-proxy MCP label blue to match server chips Use the same blue (#1890ff) as regular MCP server entries for the "All Proxy MCP Servers" option/chip instead of green. * fix(ui): make the all-proxy MCP permissions display blue, not green Match the blue used by the selector chip and regular server entries so the "All Proxy MCP Servers" badge and row in MCPServerPermissions are consistent across the team/key/org detail views. The red "Blocked" state for no-mcp-servers is unchanged. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
tin-berri
|
15ff389eb4
|
fix(mcp): persist DCR client_id so interactive OAuth token refresh works (#31912)
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works Interactive authorization_code MCP servers register an OAuth client via Dynamic Client Registration (RFC 7591) during the authorize flow, but the minted client_id and the discovered token_url were returned to the caller and never written to the server row. The autonomous refresh_token grant reads client_id, client_secret and token_url off the server, so an expired access token could not be refreshed; the user was bounced back to re-authorize and tools/list returned zero tools Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when the registration returns them) and the discovered token_url onto the server row, reusing the encrypt_credentials write that client_credentials and token exchange already use, then refresh the in-memory registry so the value is live at refresh time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same fields, so egress needs no change * fix: reuse persisted MCP DCR clients * fix: reuse persisted MCP DCR clients --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
tin-berri
|
b9df7fa705
|
fix(mcp): surface tools/list 401 auth failures as a challenge on single-server routes (#31921)
A 401 while listing tools (a missing or expired per-user OAuth token, or an upstream 401 for any auth_type) was swallowed to an empty tool list, so a single-server client got a 200 with no tools and no WWW-Authenticate challenge instead of a 401 it could re-authenticate against. Only oauth pass-through and delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the missing-token case for all of them, masked it. The surface-vs-absorb decision now keys on the route, not the auth_type. An upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError regardless of auth_type, and the per-user OAuth challenge raised during client creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is converted to the same type in _get_tools_from_server. The challenge is scoped to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a re-auth signal and degrades to an empty list like any other non-auth error, and the stdio-allowlist 403 (no challenge header) stays absorbed. The existing routing then does the right thing: single-server routes turn the error into a 401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty list so one unauthenticated server does not fail the whole listing. On the UI tools page, an OBO (per-user authorization_code) server now shows the Authorize gate when the list call returns 401, not only when no credential row exists. The backend already refreshes a still-refreshable token on the list call, so a 401 means there is no valid token and none could be minted (expired with no usable refresh token), which is exactly when the user must reauthorize. |
||
|
tin-berri
|
58de920921
|
feat(mcp): bound outbound tool-call concurrency per MCP server (#31641)
Add an optional per-server max_concurrent_requests that caps how many tool calls LiteLLM sends to one MCP server at once, so batch-processing backends are not overwhelmed by unbounded parallel dispatch. Excess calls queue on a per-server asyncio.Semaphore instead of being rejected. Unset or non-positive means unlimited, preserving existing behavior. Resolves LIT-2749 |
||
|
tin-berri
|
c370503091
|
fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2 (#31736)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2 A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no client_id and no authorization URL, yet the gateway OAuth endpoints did not check auth_type. authorize() raised "client_id is required" before the auth_type was ever examined, and the .well-known discovery builders always advertised authorization_servers / authorization_endpoint / token_endpoint / registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth flow that can never succeed. Add an auth_type != oauth2 guard to the authorize, token, register, protected-resource and authorization-server paths (covering the internal UI OAuth endpoints too). The discovery guard sits after the OAuth pass-through branch so genuine pass-through servers keep proxying their upstream metadata. oauth2 servers are unaffected. * fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle Address review feedback on the auth_type gate. The 400 message no longer claims access is governed by access groups, which is only true for auth_type=none; it now states that the gateway runs the OAuth client_id/authorize/token/register flow only for oauth2 servers and that the server is reached using its configured auth_type, which is accurate for every non-oauth2 type (api_key, oauth2_token_exchange, etc.). The discovery gate previously 404'd a named non-oauth2 server but still returned 200 metadata for an unknown name, which both serves a broken document for a typo and lets an unauthenticated caller enumerate non-OAuth server names by comparing 404 vs 200. A named discovery request now returns 200 only when it resolves to an oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root discovery and pass-through servers are unaffected. * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Sameer Kankute
|
fabe5c283a
|
fix(mcp): roll up MCP tool spend to user counters and usage UI (#31576)
* fix(mcp): roll up MCP tool spend to user counters and usage UI Direct REST MCP tool calls now fire success logging so spend_logs and user/team rollups include configured mcp_server_cost_info charges. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): gate key-info enrichment to requests missing user_id; fix import order - Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is absent, avoiding a cache/DB lookup on every normal LLM request. - Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure Add api_key = ANY($2) to the MCP session aggregate query so it is bounded by the same ownership already applied to the main page query. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix spend logs for call and list mcp tools * Add tags in mcp logging * Fix ruff * fix(lint): replace List/Dict with list/dict in new annotations (UP006) Replace the 8 new UP006 violations introduced by the mcp-tags changes: - Optional[List[str]] → Optional[list[str]] for request_tags params - List[str] return type → list[str] in _get_parent_request_tags - Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError * fix(mcp): keep final streaming chunk when draining inner stream fails * fix: handle MCP logging edge cases * fix: propagate MCP logging cancellation --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
b96f1aa686
|
fix(mcp): byom visibility, preview UX, and admin settings gating (#31809)
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in
|
||
|
Krrish Dholakia
|
cca71a07c2
|
feat(mcp): add mcp_tool_search virtual tools for large tool catalogs (#31777)
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
|
||
|
tin-berri
|
87de0e80a8
|
fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list (#31684)
* fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list On the aggregate MCP route (/mcp), the gateway fans out to every server the caller can access and flattens their tools. _fetch_and_filter_server_tools re-raises MCPUpstreamAuthError unconditionally (added with the OAuth passthrough feature in #28356) so it surfaces a 401 on single-server routes, but on the aggregate route that exception propagates through the asyncio.gather fan-out and the outer handler turns it into an empty list. The result: a single delegate/passthrough OAuth server the user has not authenticated (e.g. a delegate-auth server) zeroes the tools of every other server, including the ones that resolve fine, so the client connects and sees no tools. Surface the upstream auth error only when a single server was explicitly targeted (so that route still drives the upstream OAuth flow); across the aggregate, absorb it to [] for that one server so the rest still list their tools. This restores the graceful per-server degradation that predated #28356. Adds regression tests: the aggregate keeps a healthy server's tools when a sibling raises MCPUpstreamAuthError, and a single-server listing still surfaces it. * fix(mcp): decide aggregate vs single-server listing by route scope, not server count Addresses review: keying the surface-vs-absorb decision off the server count (len(allowed_mcp_servers), and even len(mcp_servers)) misclassifies an aggregate /mcp request from a key that can access exactly one server as a targeted single-server listing, so that one server's MCPUpstreamAuthError re-raises and empties the aggregate again for one-server permission sets. Use the path-derived single-server scope instead: _mcp_gateway_server_name, set by _gateway_initialize_instructions_request_scope only when the request path names exactly one upstream server (/<server>/mcp) and never from client headers, is None on the aggregate route (/mcp) regardless of how many servers the key can access. Single-server routes still surface the upstream-auth challenge; the aggregate absorbs it per server. Adds a regression test that an aggregate request with a single accessible server still absorbs, plus renames the single-server test to drive the route scope explicitly. The new test fails on the count-based logic. * fixing aggregation error * style(mcp): collapse single-line debug log to satisfy ruff format |
||
|
ryan-crabbe-berri
|
468d11f71d
|
feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 (#31525)
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
|
||
|
tin-berri
|
7baf25526f
|
fix(mcp): support client_secret_basic for upstream OAuth token endpoints (#31635)
The MCP gateway authenticated to upstream OAuth token endpoints only with
client_secret_post (client_secret placed in the POST body). Providers that
require HTTP Basic client authentication (client_secret_basic, the OIDC
default) reject that with invalid_client, which surfaced as a 500 on the
/<server>/token exchange and broke both the initial authorization_code
exchange and refresh.
Add a per-server token_endpoint_auth_method ("client_secret_basic" |
"client_secret_post") and a single helper that builds the right headers and
body for the configured method, then route every upstream token-endpoint POST
through it: the inbound exchange and refresh in discoverable_endpoints, the v1
per-user refresh in db, the v2 authorization_code refresher, the M2M
client_credentials fetch, and the RFC 8693 token exchange. The default stays
client_secret_post so existing servers are unaffected; basic sends
Authorization: Basic base64(form-urlencode(client_id):form-urlencode(secret))
per RFC 6749 section 2.3.1 and omits the secret from the body.
client_secret_basic is a confidential-client method, so a server configured for
it with a missing client_id/secret raises rather than silently downgrading to a
body request (no-silent-fallback); the inbound endpoint maps that to a 400 and
the refresh paths to a failed-refresh / needs-reauth. A secretless client_id
under the default method stays valid for public clients authenticating with PKCE.
Resolves LIT-4091
|
||
|
tin-berri
|
ec808edece
|
fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint (#31657)
* fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint The OAuth token endpoint stored a user's per-server token under the identity returned by _extract_user_id_from_request, which read only the Authorization header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup with no model_type rehydration and no DB fallback. That silently returned None in two common cases on a multi-replica gateway: the LiteLLM key arrives on x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code send) rather than Authorization, and a cross-replica cache hit deserializes to a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When it returned None the token was not persisted. This was survivable until the authorization_code v2 migration began stripping the caller's Authorization for migrated per-user OAuth servers and routing the preemptive 401 existence check through the stored token, so a persist miss now hard-fails: the egress challenges with 401 on every reconnect (the client sees "rejected them on reconnect" or a successful connect with zero tools). Resolve identity through get_key_object, the canonical resolver that reads the cache with model_type and falls back to the DB, and accept the key from x-litellm-api-key as well as Authorization. The silent persist skip is now a warning. The caller-Authorization stripping stays as is, since reinstating it would reopen the cross-user credential override it was added to prevent. * fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys (blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed that accidental shield. Validate the resolved key before trusting its identity: return None when blocked or expired, so the upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row. Regression tests cover the blocked and expired cases and fail without the guard. |
||
|
tin-berri
|
5963b9320f
|
feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31493)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5) The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss. * feat(mcp): DualCache-backed token cache backend (step 1b §1.5) The cross-replica TokenCacheBackend implementation that plugs into the foundation's CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected; a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss. * feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5) The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis. * feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5) The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh), release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired / not-held so a cache blip causes an extra refresh, never a crash on the resolve path. * feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5) Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh coordinator when Redis is wired, falling back to the foundation's in-process defaults on a single replica. Layers the cross-replica path on top of the single-replica dispatch store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): refresh on lock-backend error instead of serving a stale token The cross-replica refresh coordinator elected refreshers with a boolean acquire: a Redis transport error was caught and returned as False, which is indistinguishable from "another worker holds the lock". On a total Redis outage every worker therefore took the wait-then-reread branch and served the still-expired token upstream (the upstream then 401s), even though the lock and coordinator docstrings claimed a Redis blip "degrades to an extra refresh". Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the coordinator can tell a busy holder from a dead backend, and refresh anyway on ERROR. This single-flight lock is a load optimization, not a correctness mutex, so failing open is correct: it degrades a lock-backend outage to the no-coordinator behavior (an extra refresh), never a stale bearer. Add a regression test asserting an acquire error refreshes rather than re-reading the expired token, and update the docstrings to match. * style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format * fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed The cross-replica coordinator's losers re-read the token the winner persisted. If the winner's refresh failed, the store still holds the expired token, so the loser re-read it and RefreshingTokenStore handed that expired bearer to the caller (the upstream then 401s) instead of the re-auth challenge the winner returned via None. Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a re-read that is still expired surfaces None so the arm challenges. This only affects the loser path; the winner's freshly refreshed token is returned directly by the coordinator and is unaffected. * fix(mcp): log per-user token decrypt failures at debug, matching v1 When a cached blob cannot be decrypted (e.g. after a salt or master-key rotation) the codec logged a full traceback at error level, since decrypt_value_helper defaults to exception_type=error. v1's MCPPerUserTokenCache passed exception_type=debug on the same path. The blob is ciphertext so this is log noise only, but matching v1 avoids error-level traceback spam on stale entries after a key rotation * fix(mcp): namespace the refresh lock key and fence its release with a token The Redis lock wrote its key through the raw client from init_async_client(), bypassing RedisCache's namespace, so two deployments sharing one Redis collided on mcp:refresh_lock:<user>:<server> for any overlapping (user, server) and a colliding deployment skipped the refresh and challenged its own users. The lock now runs every key through an injected namespace_key wired to RedisCache.check_and_fix_namespace, matching the namespace its token cache already uses release() also deleted the key unconditionally, so a holder whose lock PX-expired and was re-acquired by another worker could delete the new holder's lock and let a third worker run a duplicate refresh, recreating the rotating refresh_token race. acquire now writes a unique per-acquisition token generated by the coordinator and release deletes only when the key still holds that token, via a compare-and-delete Lua script Adds regression tests: release with a stale token is a no-op while the owner's release deletes; keys are namespaced before reaching Redis; the coordinator acquires and releases with the same token * fix(mcp): fail open when the per-user token cache delete errors DualCache swallows get/set errors internally but not delete, and the Redis layer underneath re-raises through its circuit breaker. So a Redis outage on the delete() path escaped CachedOAuthTokenStore.fetch()'s unauthorized branch (which deletes before returning None) and invalidate(), turning a cache blip into a 500 instead of the v1-style fallback. Catch in the backend so delete degrades to the TTL-bounded stale entry like get/set already do. * style(mcp): reformat outbound-credentials files to line-length 120 The merge from staging brought in ruff's line-length 120, but these two PR-authored files were still wrapped at the old width, so the diff-scoped ruff format --check in CI flagged them. Pure reformatting; no behavior change. * fix: harden mcp oauth redis refresh coordination * fix(mcp): make the per-user token cache backend airtight on boundary failures get/set now degrade a cache or codec failure to the safe value (miss / no-op) in the backend itself rather than relying on DualCache and decrypt_value_helper happening to swallow internally, matching delete() and v1's MCPPerUserTokenCache. This upholds the layer's boundary-failure-is-a-miss contract regardless of the injected collaborators, so a Redis outage or an undecryptable entry reads as a cache miss that re-reads the DB instead of a 500. Adds contract tests for the cache raising on get/set/delete and the codec raising on encode. * test(mcp): pin per-user cache get() to a miss when decrypt raises Greptile's out-of-diff repro had the decrypt reject a blob with ValueError (bad ciphertext after key rotation); cover that exact raise path, not just the decrypt-returns-None case, so get() is regression-locked to read it as a miss. * refactor(mcp): use frozen dataclasses for the trivial DI constructors Replace the hand-written self._<arg> = arg constructors on OAuthTokenCacheCodec, RedisRefreshCoordinator, RedisDistributedLock, and DualCacheTokenCacheBackend with frozen slotted dataclasses, matching the rest of this layer. Fields take the former parameter names so the constructor API (and the tests' keyword args) are unchanged; KW_ONLY preserves the keyword-only collaborators. * fix: serialize lazy per-user oauth store rebuild * fix(mcp): stop losers challenging mid-refresh by decoupling wait from lease TTL wait_timeout_seconds defaulted to the same 10s as lock_ttl_seconds, but the holder renews its lease while a slow token endpoint runs, so a loser waiting past 10s bailed and re-read the still-expired DB token, challenging the user even though a valid refresh was in flight. Bound the holder's renewal with a refresh budget so its lock-hold is finite, and set the loser's wait to outlast that budget (refresh_budget_seconds + one lease tail) so a loser only re-reads once the holder has finished or its bounded lease has lapsed, never mid-refresh. * fix: allow concurrent lazy OAuth fetches without Redis --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
tin-berri
|
4b398ef6d4
|
feat(mcp): migrate authorization_code MCP to the v2 resolver (single-replica) [1/2] (#31473)
* feat(mcp): implement the authorization_code resolver arm
Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).
* feat(mcp): v1-backed OAuth token source for authorization_code
V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.
* style(mcp): modern type annotations in the authorization_code arm and source
* refactor(mcp): share v1's OAuth egress core; make V1PerUserTokenStore refresh-capable
Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).
* feat(mcp): route oauth2 per-user (authorization_code) servers through the v2 resolver
to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.
* feat(mcp): per-server fail-closed OAuth challenge at the v2 egress
When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.
* feat(mcp): cut the call_tool egress over to v2 for authorization_code servers
_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.
* feat(mcp): cut the tools/list connection over to v2 for authorization_code servers
The listing connection's per-user OAuth header is no longer built by v1 for migrated servers; the
v2 resolver drives it at connect time, ending the double-resolution where v1 built the token into
extra_headers and the v2 graft then deferred to it. Safe because the preemptive 401 (in the
streamable-http and SSE handlers) already challenges a missing token before the listing connection
runs, so the connection is only reached with a token present. Non-migrated oauth2 (delegate) and
the rest still build their header on v1. With this, resolve_credentials' result is honored on every
authorization_code upstream path: tool calls and listing.
* feat(mcp): route the preemptive 401 existence check through the v2 resolver
The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.
* refactor(mcp): extract the authorization_code arm into a helper
Mirror the api_key arm's structure: the inline AuthorizationCodeConfig body moves into
_authorization_code(subject, server), keeping resolve_credentials a flat one-line-per-arm dispatch.
The helper is annotated with the concrete StaticHeaderAuth it returns rather than the abstract
httpx.Auth (which api_key uses) because a new method carrying the unresolved httpx.Auth return
would add reportUnknownMemberType; the concrete type is both precise and budget-neutral.
* fix(mcp): emit the canonical WWW-Authenticate header name in the OAuth challenge
raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.
* feat(mcp): v2-native per-user token read store (step 1b inner store)
Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.
* feat(mcp): v2-native authorization_code token refresher (step 1b)
The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.
* feat(mcp): wire the v2-native per-user OAuth store into the resolver (step 1b piece 4)
Assemble Cached(Refreshing(V2PerUserTokenStore)) at the composition root and replace
V1PerUserTokenStore in mcp_server_manager. The chain is built lazily on first fetch (its cache/DB/
Redis collaborators are LiteLLM globals not ready at import); when Redis is wired it uses the
cross-replica path (DualCache cache + SET NX PX coordinator), else the in-process defaults. The DB
read, refresh-grant POST, and persist acquire their globals per call like v1. authorization_code
resolution now reads/refreshes through the v2-native lifecycle, not v1's core.
* refactor(mcp): delete the unwired V1PerUserTokenStore adapter (step 1b piece 5)
Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.
* fix(mcp): green CI for authz_code dispatch (format + UTC expiry + v2-seam tests)
- ruff format per_user_oauth_store.py (clears the lint check)
- v2_token_store._iso_to_epoch: anchor a tz-naive expiry to UTC before
.timestamp(), matching v1's db.py _remaining_token_seconds (Greptile P1) so a
non-UTC host doesn't read the expiry as local time and skew refresh timing
- test_mcp_stale_session: repoint the 3 discovery tests off the removed v1
_get_user_oauth_extra_headers_from_db onto the v2 has_user_oauth_token seam;
the delegate test now asserts the existence check is never consulted (delegate
short-circuits to the resource_metadata 401 before any token lookup)
- test_mcp_server_manager: repoint test_deferred_mode_uses_v1_auth_value at M2M
(oauth2 client_credentials), which is still a deferred mode, since per-user
oauth2 (authorization_code) now routes to the v2 resolver
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): caller Authorization must not override the stored per-user OAuth token
A caller with a valid x-litellm-api-key could include their own
"Authorization: Bearer <chosen>" header and have the proxy execute tools against
that bearer instead of the user's stored OAuth credential. For a v2-migrated
authorization_code server the caller's Authorization was seeded into
extra_headers, and the graft's apply-if-absent then dropped the resolved
per-user token in its favor. v1 prevented this by overwriting a stale client
Authorization with the stored token; this restores that precedence on both
egress paths (connect + call_tool).
- _should_strip_caller_authorization: also strip for migrated per-user OAuth
(authorization_code) servers - the v2 resolver injects the stored token, so a
caller-forwarded Authorization must not be forwarded upstream. Delegate /
pass-through (to_server_spec is None) keep forwarding the caller's bearer.
- both seed sites (_prepare_mcp_server_headers, _call_regular_mcp_tool) drop only
the Authorization from the caller's oauth2_headers (via _without_authorization),
keeping any other forwarded header and any hook/static Authorization (which
still wins, as in v1).
- regression test for the call_tool path; updated the two tests that asserted the
old (vulnerable) forwarding to assert the secure behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): preserve recorded OAuth scopes across authorization_code refresh
When a refresh response omits `scope` (RFC 6749 §5.1, where omission means unchanged), the v2 refresher persisted scopes=None and overwrote the user's recorded grant. v1 carried the prior scopes forward via `or cred.get("scopes")`; the v2 path lost that because OAuthToken did not model scopes
OAuthToken now carries scopes, V2PerUserTokenStore populates them on read, and AuthorizationCodeRefresher carries them forward for both the persisted write and the returned/cached token, so repeated refreshes do not erode them. A present `scope` in the response still replaces the prior grant
Adds regression tests: a refresh omitting `scope` preserves the prior scopes, and a present `scope` overrides them
* fix(mcp): keep user token in authorization_code tools preview
After to_server_spec maps oauth2 onto the v2 resolver, the interactive tools preview for an unsaved authorization_code server read the per-user token store, found nothing, and fail-closed with a 401, so the create/test tab could no longer list tools
The preview now routes the just-authorized token (forwarded in oauth2_headers) through mcp_auth_header, so _create_mcp_client takes the per-request-override v1 path and uses it directly, matching v1's preview. Gated to the v2-mapped oauth2 case; M2M, delegate/passthrough, and token-exchange keep their existing preview path
Adds tests: interactive oauth routes the forwarded token to mcp_auth_header, M2M and token-exchange do not
* fix(mcp): stop caller-supplied auth from overriding stored authorization_code tokens
A caller-supplied per-request override (mcp_auth_header / x-mcp-auth / x-mcp-<alias>-authorization) disabled the v2 resolver in _create_mcp_client for any spec, so an authenticated user with a stored authorization_code token could force an arbitrary upstream bearer and bypass the stored credential and its save-time validation. _create_mcp_client now keeps the v2 spec for authorization_code and ignores the override; other modes keep the client-side-credentials override
The create/test tools preview no longer relies on that override path. It resolves the just-authorized, not-yet-persisted token through the v2 resolver via a one-shot PresentedOAuthTokenStore passed as cred_provider - the same path runtime uses for the stored token - so preview and runtime resolve identically. This replaces the mcp_auth_header routing added earlier
Adds tests: a caller override cannot bypass the v2 resolver for authorization_code; the interactive preview resolves via the presented store rather than a caller header; M2M and token-exchange build no presented provider
* feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31474)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5)
The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.
* feat(mcp): DualCache-backed token cache backend (step 1b §1.5)
The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.
* feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5)
The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.
* feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5)
The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.
* feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5)
Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh
coordinator when Redis is wired, falling back to the foundation's in-process defaults on a
single replica. Layers the cross-replica path on top of the single-replica dispatch store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): refresh on lock-backend error instead of serving a stale token
The cross-replica refresh coordinator elected refreshers with a boolean acquire:
a Redis transport error was caught and returned as False, which is
indistinguishable from "another worker holds the lock". On a total Redis
outage every worker therefore took the wait-then-reread branch and served the
still-expired token upstream (the upstream then 401s), even though the lock and
coordinator docstrings claimed a Redis blip "degrades to an extra refresh".
Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the
coordinator can tell a busy holder from a dead backend, and refresh anyway on
ERROR. This single-flight lock is a load optimization, not a correctness mutex,
so failing open is correct: it degrades a lock-backend outage to the
no-coordinator behavior (an extra refresh), never a stale bearer.
Add a regression test asserting an acquire error refreshes rather than
re-reading the expired token, and update the docstrings to match.
* style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format
* fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed
The cross-replica coordinator's losers re-read the token the winner persisted.
If the winner's refresh failed, the store still holds the expired token, so the
loser re-read it and RefreshingTokenStore handed that expired bearer to the
caller (the upstream then 401s) instead of the re-auth challenge the winner
returned via None.
Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a
re-read that is still expired surfaces None so the arm challenges. This only
affects the loser path; the winner's freshly refreshed token is returned
directly by the coordinator and is unaffected.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "feat(mcp): cross-replica single-flight refresh for the v2 per-user OA…" (#31492)
This reverts commit
|
||
|
tin-berri
|
2e69708ef8
|
feat(mcp): shared OAuth token foundation - challenge, store seam, expiry-aware cache, single-flight refresh (#31275)
* feat(mcp): let CredError.of_unauthorized carry a 401 challenge The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate header + optional structured body) instead of a bare string, and raise_public emits the header and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's provisioning prompt) through the generic resolver edge. of_unauthorized's new params are keyword-only and default to None, so existing callers and the summary string are unchanged. * fix(mcp): make Unauthorized a frozen dataclass to keep the type budget flat CredError's unauthorized payload was a pydantic BaseModel, whose base resolves as unknown in this repo's basedpyright (every model in the file trips reportUntypedBaseClass plus an unknown model_config), so the tagged-union case read as unknown and the public edge's challenge access added reportUnknownMemberType errors over the per-rule ceiling. A frozen dataclass is fully typed here, so error.unauthorized resolves directly with no cast or accessor and the per-rule basedpyright counts match base. * feat(mcp): OAuth token store seam + expiry-aware cache for authorization_code Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token, expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and distributed single-flight are deferred to the hardening step. * feat(mcp): proactive token refresh with self-cleaning single-flight Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it) and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while concurrent callers await the same in-flight task and share its result, so the IdP is not stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore. OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica single-flight (Redis) and reactive-401 refresh are the later distributed hardening. * style(mcp): modern type annotations (dict/tuple/X | None) + sorted imports in the token modules * refactor(mcp): FIFO cache eviction, fix stale single-flight comment + refresh_token docstring * refactor(mcp): cache positive tokens only, matching v1 (no negative caching) CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow is visible on the next request without an invalidation hook, and uniformly across replicas since the in-process cache holds no stale None to clear. invalidate() now only covers rotation or revocation of a cached token. Negative caching (with distributed invalidation) can return later if a slow DB-backed v2-native source makes per-miss reads expensive. * fix(mcp): default OAuth expiry skew to 60s, the industry standard The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is an outlier among OAuth clients. Spring Security uses 60s as both its JWT clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's "a few minutes" leeway while preserving nearly all of a typical token's life; 30s was untested, so pin the default with two boundary-probe regression tests. * refactor(mcp): thread user_id/server_id through the TokenRefresher seam The refresh seam took only the OAuthToken, but a refresher needs the server's config (token endpoint, client credentials, scopes) to run the grant and the (user_id, server_id) key to persist the minted token, neither of which is derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id, token) and pass them through from RefreshingTokenStore so each stacked mode PR plugs into the final seam rather than forcing a later signature change across the stack. * feat(mcp): inject cache-backend and refresh-coordinator seams (cross-replica token caching) Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be). The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a reread callback for the cross-replica case (losers re-read the persisted token) that the in-process default ignores. * fix: reread oauth token before refresh --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
7eacdd5258
|
chore: litellm oss staging 250626 (#31305)
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
f16af8853b
|
feat(mcp): opt-in least-privilege default for team key MCP access (#31380)
* feat(mcp): add require_key_mcp_access_defined to stop keys inheriting team MCP servers By default a virtual key that grants no MCP servers of its own inherits its team's full MCP server list. The new general_settings flag require_key_mcp_access_defined (default false) flips this so the team list acts purely as a ceiling: a key reaches only the servers it grants explicitly (or via an access group), and inherits none. This mirrors the existing require_end_user_mcp_access_defined setting. The default is unchanged, so existing deployments keep today's behavior until they opt in. The no-mcp-servers sentinel and key access-group grants are unaffected. * docs(mcp): note require_key_mcp_access_defined effect in resolver docstring |
||
|
tin-berri
|
0f5603895c
|
fix(mcp): challenge delegate-auth OAuth servers with upstream resource_metadata (#31255)
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the user to sign in. On an unauthenticated initialize the gateway answered locally (200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop either connected empty or hit "OAuth probe timeout after 10000ms". #30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth servers, expecting the upstream to emit its own challenge. On initialize the gateway never probes upstream, so no challenge ever reached the client. Replace the `continue` with a preemptive 401 carrying the proxied resource_metadata (RFC 9728) challenge, the same form passthrough servers and MCPUpstreamAuthError already use. This keeps #29770 fixed (still no authorization_uri) while restoring the upstream PKCE sign-in prompt. |