* fix(passthrough): stream non-sse passthrough responses instead of buffering in memory
Non-SSE passthrough responses were fully read into proxy memory (content = await response.aread()) before the first byte reached the client. For large non-JSON bodies such as Anthropic batch results jsonl files this ballooned proxy RSS to a multiple of the file size and produced near-total TTFB dead air, letting intermediaries kill the silent connection and truncate the download.
The upstream request is now sent with httpx stream semantics and the buffering decision is made from the response headers: application/json (and +json) bodies plus upstream errors keep the buffered behavior since spend logging, guardrails and managed-id rewriting inspect them, while every other 2xx body is relayed as a StreamingResponse that iterates upstream bytes without accumulating them, preserving status code and headers (including x-litellm-*) and firing the success-handler logging with response_body=None once the stream completes.
* fix(passthrough): log client disconnects mid-stream and derive test client cache key from production code
* test(passthrough): intercept AsyncClient.send in legacy passthrough tests and assert final wire params
* test(passthrough): fail with a clear assert when the passthrough client cache scan misses
* fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056)
* test(register_model): use a triple provider prefix as the unresolvable-key fixture
get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the
double-prefix fixture stopped exercising the register_model fallback path.
Lock the new double-prefix resolution in as a model-info regression test
In a listing fan-out over a scope containing more than one server that
consumes the caller's Authorization (true_passthrough, oauth_delegate,
or the legacy delegate/passthrough shapes), the request-wide bearer is
now withheld from the new modes instead of being replayed against every
upstream (RFC 9700 cross-resource replay). Explicitly-addressed
operations (tool call, get_prompt, read_resource, single-server routes)
keep forwarding it.
Multi-server aggregates use the per-server x-mcp-{alias}-authorization
header instead: its value now feeds the passthrough resolver arm as the
inbound token and wins over the request-wide header, binding one token
to one server.
* fix(passthrough): stop request params from clobbering merged target query params
* fix(passthrough): rewrite managed ids in query params before folding them into the URL
Root cause: PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials. That is only safe if config-load pre-resolves
os.environ/ refs so the value reaching get_credentials is already the real
secret. The YAML config path has always done this. The DB-load path
(ProxyConfig._resolve_db_litellm_param) only re-expanded keys in a hardcoded
whitelist (_DB_LITELLM_PARAM_ENV_REF_KEYS) plus short-circuited env-ref
resolution entirely for team-scoped rows. PR #32256 extended that whitelist
to 18 keys to unblock a customer whose Bedrock model with aws_role_name:
os.environ/BEDROCK_ASSUME_ROLE_ARN broke on v1.90+, but the whitelist is
structurally fragile: every future auth field breaks the same way until
someone remembers to add it
Fix: remove the whitelist and the team-scope short-circuit. The DB-load
resolver now expands os.environ/ on every string field, matching the YAML
path. Trust boundary stays on the write side: only PROXY_ADMIN can create
team_id=None rows, only team admins of a team can create rows scoped to
that team, and the request-body vector is still blocked by
_BANNED_REQUEST_BODY_PARAMS. Team-scoped rows now resolve env refs — this
is a deliberate LIT-3831 threat-model expansion trusting team admins for
env-var reads
Regression tests in tests/test_litellm/proxy/proxy_server/test_proxy_config.py:
- test_ProxyConfig__add_deployment_resolves_env_refs_after_db_decrypt pins
admin-scoped rows resolve every field (previously api_base stayed literal)
- test_ProxyConfig__add_deployment_resolves_team_env_refs pins team rows
resolve env refs (previously stayed literal)
- test_ProxyConfig__add_deployment_resolves_env_refs_on_arbitrary_field pins
the no-whitelist invariant against a made-up field name
- test_ProxyConfig__add_deployment_resolves_env_refs_for_aws_bedrock_auth_params
(from #32256) still passes
- Path B counterparts (decrypt_model_list_from_db) mirror the above
Left as followups (not fixed here):
- /model/info and /v2/model/info still echo resolved values for fields not
in the current pop-list (aws_role_name, aws_sts_endpoint, api_base, etc.).
Fix is to extend remove_sensitive_info_from_deployment; separate PR
- Master-key rotation reads DB rows via decrypt_model_list_from_db which
now resolves universally, so rotation collapses env-refs into hardcoded
values. Pre-existing bug for the 6 previously-whitelisted fields; wider
surface after this PR. Separate PR
oauth_delegate forwards the caller's token to the upstream, which validates its
audience, so the protected-resource metadata must keep resource pointing at the
upstream (returned verbatim, like true_passthrough) rather than rewriting it to
the gateway. Rewriting to the gateway asks the client to mint a token bound to
the gateway audience, which a strict IdP (Entra) refuses to issue for an
unregistered resource and a spec-compliant upstream rejects on receipt. The
legacy is_oauth_passthrough opt-in keeps the gateway rewrite unchanged.
Both modes advertised LiteLLM as the authorization server and answered initialize locally, so a client with no token connected empty and was never driven into the upstream OAuth flow. The protected-resource discovery now proxies the upstream metadata for both modes (verbatim for true_passthrough, resource rewritten to the gateway for oauth_delegate), and the preemptive 401 emits the matching challenge: oauth_delegate uses the gateway-proxied resource_metadata once admission passes, true_passthrough probes the upstream anonymously and surfaces its WWW-Authenticate verbatim so the client authorizes directly against the upstream
Introduce two first-class MCP server auth_type values that make LiteLLM's
role in upstream authentication explicit, added alongside the existing
delegate_auth_to_upstream / oauth_passthrough flags without changing their
behavior.
true_passthrough is a transparent proxy: LiteLLM performs no admission auth,
requires no x-litellm-api-key, mints/stores/refreshes nothing, and forwards the
client's Authorization to the upstream exactly as received. oauth_delegate keeps
normal LiteLLM admission (x-litellm-api-key / SSO / JWT) and then forwards the
client's separate upstream Authorization unchanged; the admission credential is
never forwarded upstream.
Both modes forward the caller's token via the existing extra_headers path and
defer egress credential resolution to v1 (the v2 to_server_spec returns None for
them). Upstream 401/403 responses are surfaced rather than swallowed so upstream
OAuth challenges are preserved. Servers in either mode require per-user auth, so
userless health checks are skipped.
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The token_exchange (OBO) branch of _call_regular_mcp_tool built its coroutine
by calling _obo_call_tool_with_retry directly, outside the
_limit_outbound_concurrency context manager that the regular branch uses. OBO
tool calls (and the internal re-mint retry, which issues a second upstream
call_tool) therefore bypassed the per-server max_concurrent_requests semaphore,
so an authenticated caller could run unlimited concurrent tool calls against an
OBO MCP server despite an admin-configured limit.
Wrap the OBO coroutine in _limit_outbound_concurrency the same way the regular
path does, holding one permit across the initial call, the on-401 re-mint, and
the retry, so OBO calls honor the configured cap.
* fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign
When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a
Bedrock request, _sign_request received attempt 1's already-signed headers
and copied the old Authorization and X-Amz-Date back over the freshly
computed SigV4 signature, so the retry POSTed the stripped body with a
signature for the original body and AWS returned 403 SignatureDoesNotMatch.
Skip SigV4-computed headers (authorization, x-amz-date,
x-amz-security-token, date) when restoring caller headers after signing,
and only preserve a caller-supplied Authorization that is not itself a
SigV4 header so bearer-token setups keep working.
* fix(bedrock): apply the same stale-header guard to get_request_headers
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard
OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be
configured through config.yaml; the create/update REST API and the dashboard had no way to
express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end.
These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url
and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the
credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first
and falls back to the credentials blob so servers persisted before the columns existed still load.
client_id and client_secret continue to ride the existing encrypted credentials path.
On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field
section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous
catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of
mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from
non-admin and virtual-key responses, matching how token_url is treated.
* fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange
The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code:
when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could
not mint a token), the user must re-authorize via the browser flow. token_exchange has no
gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was
undefined (a compile error) and, per this file's convention and its tests, meant authorization_code;
renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an
authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode
instead and drop the now-unused isTokenExchange
* fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes
Switching an existing oauth2 server to oauth2_token_exchange left the old flow's
token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url
as the configured exchange endpoint, so the stale value both suppressed the RFC
9728/8414 discovery this PR adds and sent the exchange grant (client credentials
plus the user's subject token) to the previous flow's token endpoint
update_mcp_server now mirrors its existing stale-credentials rule for the
flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow,
token_exchange_endpoint, audience, subject_token_type): when auth_type changes,
each one is cleared unless the same request explicitly provides it, so a
deliberate override in the switch request still wins. Updates that keep the
auth_type never touch these columns, which keeps legacy OBO rows that use
token_url as their exchange endpoint working
The edit form sends explicit nulls for the previous flow's fields on an auth type
switch; antd preserves unmounted field values by default, so without this the old
token_url would be re-sent verbatim and read as an explicit override. Transitions
are detected against the persisted auth_type, so saves that keep the auth type
send nothing extra
Reported by Cursor Bugbot on the PR
* fix(mcp): lift legacy blob token-exchange settings into their columns on every write
The three token-exchange settings live in dedicated columns but also exist on
MCPCredentials as the pre-column REST shape. Writes now lift incoming blob
values into the columns (an explicit top-level value wins, including an
explicit null) and strip them from the stored blob; the same-auth credentials
merge migrates legacy rows the same way. The read-time column-or-blob fallback
then only ever serves rows current code has never written, so clearing a column
to re-enable RFC 9728/8414 discovery can no longer be silently undone by a
stale blob copy.
Also asserts the auth-switch clearing fires on the external fields_set path
(PUT /v1/mcp/server).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(mcp): single source for the RFC 8693 default subject_token_type
The default was applied at four egress build sites plus two model defaults,
each with its own copy of the literal. All sites now share
DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is
deliberately not used: Prisma writes explicit values on insert, so a column
default would rarely apply, and NULL-means-RFC-default keeps existing rows
correct.
Also documents two review decisions in place: the audience column keeps the
RFC 8693 parameter name (RFC 8707 resource indicators are already a separate
concept named resource in the v2 egress types), and the migration's
out-of-order timestamp is safe under prisma migrate deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: fix import sort order in outbound_credentials/types.py (I001 strict budget)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials
The migrate-on-write in the credentials merge lifts blob values into null
columns, which is correct for legacy rows but could repopulate a column an
admin had cleared in an earlier no-credentials update (that path never touched
the blob, so the stale copy survived to be lifted later). An explicit
token-exchange column write (set or clear) now migrates the row even when the
update carries no credentials: untouched null columns are lifted, every blob
copy is stripped, and unrelated blob keys stay as-is. A cleared column can then
never be resurrected, because no write path leaves a blob copy behind.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(mcp): state the blob-to-column lift contract on the legacy credential keys
The three token-exchange keys on MCPCredentials are the pre-column REST shape
(the only REST shape from 2026-05 until this PR). Document on both the blob
type and the request models that the dedicated columns are authoritative and
that writes lift blob values into them and strip the stored copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers
The other two token-exchange fields were cleared while subject_token_type was
left visible. It is a public RFC 8693 URN with no disclosure value, but the
sanitizers' rule is that these views receive no token-exchange config at all —
cleared for uniformity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(ui): OAuth flow selector on the MCP edit page
The edit form had no flow selector: oauth_flow_type was watched but never registered,
so isM2MFlow was always false in edit mode and the flow could only be changed over
REST. That left the backfill's remediation for ambiguous legacy rows (client creds +
token_url, no interactive signal, left unstamped) without a dashboard path
The oauth2 section now opens with an OAuth Flow Type select. Explicit rows prefill
their stored value and re-persist it on save; legacy null rows show a placeholder
instead of a fake preselection, and an untouched save still writes nothing, so the
form never guesses on the admin's behalf. Choosing Machine-to-Machine (M2M) persists
oauth2_flow=client_credentials, choosing Interactive (PKCE) persists
authorization_code, which is exactly the assertion the backfill warning asks for.
Registering the field also brings the existing isM2MFlow gating in the edit form to
life, so M2M rows stop showing the interactive-only token-validation fields
Tests cover the prefill round-trip for both explicit values, the untouched null row
writing nothing, and both selections persisting on a legacy null-flow row
* fix(mcp): registry-to-table conversions must carry oauth2_flow
_build_mcp_server_table and the health-check table builder dropped oauth2_flow when
converting registry servers for GET /v1/mcp/server (list and by-id), so the dashboard
never received the persisted flow: the edit page could not prefill the selector, M2M
gating never activated, and the tools page classifier saw every oauth2 server as
interactive regardless of the column. Found live while proving the edit-selector
persistence path end to end; the write side was fine (PUT persists and the column
reads back correctly), the read side was dropping the field at the conversion
Both builders now carry oauth2_flow; regression test pins the conversion
* docs(mcp): flag _resolve_oauth2_flow as security-sensitive in its docstring
The prior wording ('not called directly by security sites') could read as if the
function has no security relevance, when it is the shape-inference engine both
request-time security helpers delegate to. Reword to state that plainly: it decides
M2M-vs-interactive for an unstamped row, must always be reached through
effective_oauth2_flow or resolve_oauth2_flow_for_request, and its M2M-shape branch
must not be weakened without accounting for those callers. Docstring-only; no logic
change
Raised by review on the stacked PR
* refactor(ui): extract oauth2FlowToFormValue helper for the MCP OAuth flow prefill
The edit form derived the OAuth Flow Type select value from the stored oauth2_flow
with a nested ternary duplicated at two call sites. Extract the mapping into a named
helper in types.tsx (next to getMcpOAuthMode and the flow constants): client_credentials
-> M2M, authorization_code -> Interactive, null/unset -> undefined so the select shows
its placeholder instead of a guessed default. The tool-config call site keeps its
null -> Interactive display fallback via a trailing ?? OAUTH_FLOW.INTERACTIVE, so
behavior is unchanged. Adds unit tests for the helper; the existing prefill/save tests
already cover the call sites
* feat(ui): surface and warn on an unset MCP oauth2_flow (server card + edit page)
An oauth2 MCP server whose oauth2_flow was never classified (legacy null row the
backfill left ambiguous) now advertises that it needs attention instead of silently
falling back. The server card shows an 'OAuth flow not set' warning tag for any
auth_type=oauth2 server with no oauth2_flow, so admins can spot them in the list
without opening each one. The edit page shows a warning alert directly under the new
OAuth Flow Type selector while the flow is unset, and it clears the moment a flow is
picked.
Delegate (delegate_auth_to_upstream) servers are excluded from both: they authenticate
via upstream PKCE passthrough and route to passthrough regardless of oauth2_flow, so
the M2M-vs-interactive classification does not apply and prompting for it would be a
false alarm. The edit page reads the delegate state from the watched switch when it is
mounted and falls back to the stored value otherwise (useWatch returns undefined for an
unmounted field).
Also adds end-to-end coverage of the null-flow chain the selector depends on:
build_mcp_server_from_table carries oauth2_flow=None verbatim into the GET response,
so the dashboard maps it to undefined and shows the placeholder rather than a guessed
default. Tests: backend null carry, the select prefill display for all three states,
the edit-page warning show/hide/clear-on-select and delegate exclusion, and the card
badge across oauth2/non-oauth2, stamped/unstamped, and delegate
UI session tokens carry the virtual team_id litellm-dashboard (UI_TEAM_ID),
which is never persisted. The MCP team-permission helpers passed it to
get_team_object anyway, so every dashboard MCP listing raised a 404 per
lookup that was swallowed into per-server 'Failed to get allowed tools for
server' warnings (plus the sibling 'allowed MCP servers for team' and 'MCP
access groups for team' warnings) and wasted DB queries. The 404 also
escaped past the key-level permission handling in
get_allowed_tools_for_server, dropping key tool restrictions for such
sessions.
Short-circuit the virtual team before the DB lookup in the three helpers,
mirroring the existing UI_TEAM_ID handling in agent_permission_handler.
Also reject /team/new with the reserved team_id, since a real row would
bind its budget and permissions to every UI session
* fix(main): stop per-request custom pricing from clobbering shared model_cost pricing
A request routed through a wildcard deployment with explicit zero pricing
(e.g. openai/* with input_cost_per_token: 0) registered that pricing on the
shared {provider}/{model} key in litellm.model_cost, so sibling deployments
relying on built-in pricing logged $0 until process restart (LIT-3991).
Request-time registration in completion()/embedding() now mirrors the
router-startup isolation: router-originated requests register full pricing
under the deployment's unique model id only, while the shared backend key
receives the entry with custom pricing fields stripped. Direct SDK calls
without a router deployment id keep the legacy shared-key registration.
The stripping logic is shared via
CustomPricingLiteLLMParams.strip_custom_pricing_fields and reused by
Router._create_deployment and Router.add_deployment.
* test: update legacy tests that asserted per-request pricing leaking into shared model_cost
test_router_fallbacks_with_custom_model_costs asserted the shared
claude-sonnet-4-5-20250929 entry ends up with the deployment's 30/60
pricing, which is exactly the cross-deployment leak this PR removes; it
now asserts the shared key keeps the built-in pricing, matching the
test's stated goal.
test_cost_calc.py::test_run computed streaming cost via
completion_cost(response), which only matched the non-stream cost while
the shared gpt-3.5-turbo entry was poisoned with the per-request
2/token pricing; it now passes the request's custom pricing explicitly
via custom_cost_per_token.
* refactor(mcp): read oauth2_flow verbatim from DB rows; inference stays config-only plus a logged backstop
With every DB write site stamping oauth2_flow (#32283, #32288) and the startup
backfill healing legacy null rows, the DB build no longer needs to re-derive the
flow from field shape. build_mcp_server_from_table now reads the column verbatim
via _explicit_oauth2_flow: unknown or null values resolve to None, which
needs_user_oauth_token already treats as interactive, so an unstamped row degrades
to the safe default instead of guessing M2M from a shape that a DCR-registered
interactive server shares whenever discovery is down
Field-shape inference survives in exactly two places. config.yaml-loaded servers
keep it at load time: they are rebuilt from the config on every boot, so there is
no row to backfill and load-time resolution is their write-time stamp. And the
request-time backstop in _get_allowed_mcp_servers keeps a not-yet-backfilled M2M
row blocking caller Authorization forwarding (the P1 property); it now logs a
warning whenever it actually fires, which is the fire-rate signal for deleting it
once deployments have booted past the backfill
Regression tests pin that the DB build does not infer M2M from the credential
shape and reads an explicit column value verbatim
Fourth step of the oauth2_flow persistence sequence, stacked on the backfill
* feat(mcp): deprecation warning when config-level M2M is inferred rather than declared
A config.yaml oauth2 server whose credential shape decides client_credentials without
an explicit oauth2_flow now logs a warning at load pointing the admin at the explicit
declaration. First rung of the deprecation ladder: the docs make oauth2_flow the
recommended path, the warning surfaces configs still relying on inference, and a
future breaking release can turn it into a config validation error, at which point
config-level shape inference dies entirely. Interactive omissions stay silent since
the default matches inference there and nothing load-bearing is being guessed
* feat(mcp)!: require explicit oauth2_flow for config-defined oauth2 servers
A config.yaml server with auth_type oauth2 must now declare its flow; the load
raises a config validation error naming both values and what each means:
oauth2_flow: client_credentials for machine-to-machine (the proxy mints a shared
token at token_url using client_id/client_secret) or
oauth2_flow: authorization_code for interactive (per-user tokens via browser
sign-in, including delegate_auth_to_upstream)
This replaces the load-time shape inference for config servers entirely. The
credential shape is genuinely ambiguous (a DCR-registered interactive server
carries client creds + token_url with no authorization_url, identical to M2M),
so the config asserts the answer instead of the proxy guessing it. With this,
field-shape inference survives in exactly one place: the request-time security
backstop, which is telemetry-gated for deletion
BREAKING CHANGE: config-defined oauth2 MCP servers without oauth2_flow fail
proxy startup with the error above. Add the one line to the server block; the
error text says exactly which value to pick
* test(mcp): pin the verbatim read for authorization_code alongside client_credentials
Raised by review on the PR
* fix(mcp): fail closed on the anonymous delegate gate for unstamped M2M-shaped servers
Reading oauth2_flow verbatim (this PR) changed has_client_credentials from True to
False for a legacy null-flow row that still carries the M2M credential shape. That
value is what the anonymous upstream-delegate gate checks before skipping LiteLLM
auth entirely, so an M2M-shaped delegate server that was never stamped would newly
pass the gate: an unauthenticated caller could get it selected and then list/read
upstream data using the client credentials the request-time backstop re-infers,
running as LiteLLM's service account. This reopens the hole the gate's existing
'never delegate for M2M' guard was written to close
The gate now resolves the flow (column first, shape fallback) instead of reading the
bare column, mirroring the request-time backstop in _get_allowed_mcp_servers: both
fail closed on the ambiguous M2M shape and are removed together once no null rows
remain. A pure-PKCE delegate server (no stored credentials) resolves to a non-M2M
flow and keeps its bypass, so the common delegate case is unaffected
Tests: an unstamped M2M-shaped delegate server is denied the bypass (mutation-checked
against the bare-column regression), and a pure-PKCE delegate server still bypasses
Raised by review on the PR
* fix(mcp): centralize the request-time oauth2_flow backstop across every security site
Reading oauth2_flow verbatim made has_client_credentials unreliable for legacy null
rows, and the backstop that compensates was applied at only one reader. Review found
three more consequences of that per-site approach:
- the anonymous-delegate allowlist in get_allowed_mcp_servers read the bare column, so
an unstamped M2M-shape delegate server was surfaced to anonymous callers (High)
- call_mcp_tool resolved allowed ids into MCPServer objects without the backstop, so a
null-flow M2M-shape row kept has_client_credentials false on tool execution during a
backfill gap, though the listing path was covered (High)
- the request-time warning claimed the startup backfill would stamp the row next boot,
but the backfill deliberately leaves the ambiguous M2M shape unstamped (Low)
Rather than patch each site, introduce two helpers on MCPServerManager that are the
single choke point for request-time resolution: effective_oauth2_flow(server) for the
enum/boolean decisions (allowlist filter, anonymous-delegate gate) and
resolve_oauth2_flow_for_request(server) for the egress object copy (listing and tool
call). Both fail closed on the M2M shape and leave stamped rows and pure-PKCE rows
untouched. The gate now shares effective_oauth2_flow instead of its inline resolution,
and the corrected warning lives once inside resolve_oauth2_flow_for_request, so deleting
the whole transitional layer later is a single-site change.
Tests: helper unit coverage (stamped verbatim, null M2M-shape resolves, pure-PKCE stays
None, stamped/pure-PKCE return the same object, corrected warning text), the anonymous
allowlist excludes an unstamped M2M-shape delegate server, and the call path resolves
the flow like the listing path. The two security-integration tests are mutation-checked
against the bare-column regression.
Raised by review on the PR
* fix(spend): bound the logs-tab pagination count to stop full-window scans
The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) computed an
exact pagination total over the whole selected time window on every load. That
was a standalone SELECT COUNT(*) FROM (SELECT request_id FROM LiteLLM_SpendLogs
WHERE ...) which, on a multi-TB SpendLogs table, scans a huge number of rows and
spikes Aurora ACU. The later LIT-4027 change folded it into COUNT(*) OVER (), but
a window count still drains every matching row before the LIMIT applies, so the
full-window scan remained.
Compute the total with a bounded SELECT COUNT(*) FROM (SELECT 1 ... LIMIT $cap+1)
that probes at most cap+1 rows, and drop the window count from the page query so
the page query is a plain indexed ORDER BY ... LIMIT. When more than the cap
match, report the cap and set total_is_capped so the UI renders "<cap>+". The
bounded subquery terminates early rather than aggregating across all tablets, so
it stays safe on sharded engines like YugabyteDB too.
Resolves LIT-4119
* test(spend): make zero-total mock reflect real COUNT(*), add capped-total tooltip
Address Greptile review on #31825:
- the empty-result test now returns [{"total_count": 0}] for the bounded
count query (real COUNT(*) always returns one row) instead of [], so the
zero-total path exercises the normal branch rather than the defensive guard
- the logs toolbar shows a tooltip explaining the cap when total_is_capped is
set, so a disabled Next button at the cap boundary reads as intentional
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a
globally configured percentage of its TPM/RPM instead of being blocked entirely.
A new litellm_settings global, budget_exceeded_throttle_percentage, sets the
fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in
key metadata via the existing management-endpoint metadata routing) opts the key
in. When both are set and the key is over budget, the budget check records the
percentage on a request-scoped budget_throttle_pct instead of raising, and the
rate limiter scales the key's configured TPM/RPM by it. Keys without the flag
keep hard-blocking; team/user/org budgets are unaffected.
The throttle is recomputed from the key's original limits on every request and
the decision is cleared before the auth object is cached, so it never compounds
across requests. Both the budget read-time check and the budget reservation path
honor the opt-in, and both the v3 and legacy rate limiters apply the scaling.
Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an
admin-imposed hard budget block into a soft throttle that keeps spending past
max_budget, so a non-admin must not be able to self-opt-in and bypass their own
spend cap. Both /key/generate and /key/update reject a non-admin setting it to
true (update only gates the transition to enabled, so a non-admin can still edit
other fields and turn the flag off). This matches the feature being wholly
proxy-admin operated: the global percentage is admin-only too.
A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays
hard-blocked rather than serving unlimited requests past its budget (fail-safe).
The global budget_exceeded_throttle_percentage is configurable from the admin UI
(Settings -> General Settings), persisted through litellm_settings so it survives
a restart, not only from config.yaml.
Resolves LIT-3894. Scope for LIT-3893.
* fix(mcp): apply semantic filter to expanded litellm_proxy tools and show filtered-out count
* fix(mcp): guard expansion-path filtering on enabled flag and isolate metadata emission
* fix(mcp): resolve tool name prefix via known server prefixes, not string match
When an MCP server's alias differs from its server_name, tool names are
listed with the alias prefix but _execute_tool_calls compared that prefix
against the server_name stored in tool_server_map. The mismatch silently
skipped prefix stripping, forwarding the fully-prefixed tool name upstream
and causing "Unknown tool" failures. Resolve the actual MCPServer object
and strip using its known prefix forms (alias, server_name, server_id)
instead.
* fix(mcp): preserve tool overrides and scope REST tool listing
Return saved tool display/description overrides from the server table API
so the edit UI reloads them, resolve display names before prefix stripping
on tool calls, and honor mcp_server_name and toolset_name filters on the
REST tools list endpoint.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): inject BYOK credentials on Playground OpenAPI tool calls
Playground and Responses API route MCP execution through call_tool, which
skipped BYOK lookup and never set the OpenAPI auth ContextVar, so upstream
calls went out unauthenticated despite a stored user credential.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(mcp): cover alias-mismatch prefix stripping and display-name reverse mapping
Regression tests for _execute_tool_calls: an MCP server whose alias differs
from its server_name must still have its tool-name prefix stripped correctly,
and a tool called by its configured display name must resolve back to the
original tool name before dispatch.
* fix(mcp): validate tool display names against Bedrock's tool-name pattern
A display name replaces the tool name sent to the LLM provider, so a value
with spaces or other special characters saves successfully but fails every
subsequent Bedrock tool call. Validate tool_name_to_display_name server-side
(create/update payload) against Bedrock's [a-zA-Z0-9_-]+ constraint, and add
matching inline validation plus a save-blocking guard in the Admin UI's
create and edit MCP server forms.
* style(mcp): fix ruff/prettier formatting on CI
No logic changes; satisfies the format checks flagged on PR #32320.
* fix(mcp): fix CI failures on PR - complexity budget and stale test mock
Extract toolset-scope resolution and query-param normalization out of
list_tool_rest_api into helpers to bring it back under the C901 complexity
budget (was 18, now within the 15 threshold).
Add the missing get_mcp_server_by_name stub to the streaming iterator test's
mock manager; the alias-fallback resolution added for tool-name-prefix
stripping calls it unconditionally when _get_mcp_server_from_tool_name misses.
* test(mcp): cover BYOK OpenAPI auth-header helpers to close codecov patch gap
_format_byok_openapi_auth_header, _openapi_forwarded_extra_headers, and
_resolve_byok_mcp_auth_header were only exercised indirectly via a mocked
call_tool test, leaving their branches (auth-type formatting, header
forwarding/stripping, missing-credential 401) uncovered.
* fix(mcp): resolve BYOK auth before queuing the during-hook task
_resolve_byok_mcp_auth_header can raise a 401 when no credential is stored.
Resolving it after during_hook_task was already queued meant a hook's
side effects (audit logging, rate-limit bookkeeping) could run and record
success for a tool call that then fails on the missing credential.
* fix: correct mcp alias routing regressions
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): wire general_settings SSRF allowlist to litellm globals
general_settings.user_url_allowed_hosts was documented in SSRF errors but
never applied at startup, so internal MCP/OpenAPI URLs stayed blocked.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): regenerate dashboard types and satisfy ruff UP006 budget
Use list[str] in ConfigGeneralSettings and run gen:api so schema.d.ts
matches the new SSRF general_settings fields.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: normalize ssrf general settings
* fix: clear ssrf allowlists from null settings
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Once the active Prisma client is in the disconnected state, every DB call
raises ClientNotConnectedError. The reconnect machinery was supposed to
recover from this, but _get_engine_pid() inspected the broken client via
prisma's _engine property, which re-raises that same error, so
recreate_prisma_client failed before it could build a replacement client
and the proxy looped on failed reconnects forever (issue #28322 showed
1486+ consecutive failures over 30 days with zero recoveries)
Guard both _get_engine_pid implementations with is_connected() so a
disconnected client reads as "no engine" (pid 0) and the recreate path
proceeds to construct and connect a fresh client
* fix(caching): pass only metadata to valkey semantic async embedding
ValkeySemanticCache async get/set passed **kwargs into _get_async_embedding,
which raised TypeError on cache_key and other fields and silently skipped
all cache writes. Match redis-semantic by forwarding metadata only.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(caching): add async_get_cache embedding call regression test
Mirror the async_set_cache spy test so async_get_cache passing **kwargs
into _get_async_embedding is caught by a real signature, not AsyncMock.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
The x-litellm-semantic-filter-tools response header was sliced mid-name at
MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH with a trailing "...", so the
admin UI test panel rendered the last selected tool name chopped. Truncate
the CSV at a tool name boundary instead so the header only ever carries
complete names, and note in the test panel how many selected tools did not
fit in the header
Streaming pass-through for native Anthropic /v1/messages and the /v1/responses
streaming iterator never set logging_obj.completion_start_time, so
_success_handler_helper_fn fell back to completion_start_time = end_time.
Downstream TTFT consumers (Prometheus, OTEL, Langfuse, Admin UI, spend logs
completionStartTime) then reported time-to-first-token equal to total request
duration.
Stamp completion_start_time on the first chunk in PassThroughStreamingHandler.
chunk_processor and BaseResponsesAPIStreamingIterator._process_chunk, mirroring
CustomStreamWrapper for /chat/completions.
Resolves LIT-4185
Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
* feat(mcp): startup backfill stamping oauth2_flow on legacy null rows
Rows created before the write-side stamps carry a null oauth2_flow and rely on
read-time field-shape inference, which cannot tell a DCR-registered interactive
server (client creds + token_url, no persisted authorization_url) from an M2M
server unless endpoint discovery succeeds first; on a transient discovery failure
those servers flip to client_credentials for that registry load
The backfill classifies each null oauth2 row once, at rest, ordered by signal
strength: per-user token rows (only the interactive flow mints them, so this is
definitive and catches the DCR-trap cohort), then a persisted authorization_url,
then a persisted registration_url (DCR implies interactive; this covers
registered-but-never-signed-in rows), then the M2M credential shape mirroring the
legacy inference, else the interactive default that matches how
needs_user_oauth_token treats a null flow. Every stamp is logged with the rule
that fired and written with updated_by=oauth2_flow_backfill for auditability
Runs in _init_mcp_servers_in_db before the registry load so the first build of
the boot classifies from the column, is isolated so a failure cannot block server
loading, and is idempotent: a healed fleet exits after one indexed query. This
unblocks deleting the read-time inference for DB rows in the follow-up
Third step of the oauth2_flow persistence sequence, after #32283 and #32288
* fix(mcp): backfill leaves the ambiguous M2M shape unstamped instead of guessing client_credentials
The credential shape (client_id + client_secret + token_url, no interactive signal) is
shared by real M2M servers and DCR-registered interactive servers nobody has signed
into: the DCR persist writes creds and token_url but not authorization_url or
registration_url. Stamping client_credentials from that shape permanently mislabeled
the interactive cohort, and once explicit the value is authoritative, so per-user
traffic would run on the proxy's stored client credential with no discovery rescue
and no backstop (it only guards null rows)
The backfill now stamps only what it can prove. Interactive signals keep stamping
authorization_code; the ambiguous shape is left null with an actionable warning naming
the server and the fix (set oauth2_flow via the dashboard or PUT /v1/mcp/server). A
true M2M row keeps working per-request through the security backstop while the warning
nags; an interactive row keeps its Authorize button (null renders interactive), and one
completed sign-in creates the per-user token that stamps it authorization_code at the
next boot. Mirrors the config-level rule: M2M is asserted by a human, never guessed
Raised by review on the PR
* perf(mcp): batch the backfill stamps into one update_many per flow value
The per-row update loop issued one DB round-trip per legacy row at startup; rows
sharing a stamped value now go out as a single update_many, so the DB cost is
constant in fleet size. Per-row logging keeps the rule that fired for each server
Raised by review on the PR
* fix(mcp): backfill stamps only rows still null at write time and counts only real OAuth token rows as sign-in proof
Two review findings. The batched update_many matched on server_id alone, so an
explicit oauth2_flow set between the backfill's read and its write (an admin PUT or
a sign-in's DCR stamp landing in the boot window) would be overwritten with the
inferred value; the where clause now also requires oauth2_flow to still be null, so
an explicit value can never be clobbered under any interleaving
And the per_user_tokens rule counted any LiteLLM_MCPUserCredentials row as proof of
an interactive sign-in, but that table doubles as BYOK storage for user-supplied API
keys; a BYOK-flavored row would have stamped an M2M-shaped server authorization_code.
The rule now counts only rows whose payload decodes as a type oauth2 token via the
existing _decode_oauth_payload discriminator, so bare keys, undecodable rows, and
stale leftovers from a BYOK-to-oauth2 auth switch prove nothing
Raised by review on the PR
Drive the real parallel_request_limiter through _pre_call_with_fallbacks for
the LIT-3890 customer scenario: a key-level model_tpm_limit raises
ProxyRateLimitError from the pre-call hook and the configured gateway fallback
serves the request instead of returning a 429. Unlike the existing tests, this
exercises the actual limiter rather than a hand-built error.
Also switch the new _pre_call_with_fallbacks return annotation to builtin
tuple to stay within the ruff UP006 strict-rule budget.
* fix(llm_http_handler): send dict transcription request data as a JSON body
httpx form-encodes dicts passed via data= and silently ignores json=, so the
generic audio transcription path never actually sent a JSON body. No provider
hit this before; JSON-body speech APIs need it.
* feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support
Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so
vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the
Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential
resolution (vertex_project/vertex_location/vertex_credentials or ADC); the
location defaults to the us multi-region since chirp_3 is only served from the
us and eu multi-regions, and non-global locations use the regional
<location>-speech.googleapis.com host. Maps language to languageCodes (auto
language detection by default), joins all result alternatives into the
transcript, and tracks cost from totalBilledDuration with a
vertex_ai/chirp_3 price entry at Google's published $0.016/min.
* fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text
OpenAI clients send language codes like "en", which Google rejects with 400
("not supported by the model chirp_3 in the location us"); Speech-to-Text
wants region-qualified BCP-47 like "en-US". Adds a shared
normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's
transcription config already hand-rolled the same table privately) that maps
common bare codes and passes region-qualified ones through, and applies it in
the Vertex transcription request. Also narrows the response JSON parse guard
to ValueError.
* fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works
cost_per_second prefers output_cost_per_second whenever it is not None, so the
0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using
input_cost_per_second. Remove it from both cost maps and pin the behavior with
a regression test computing 18s of chirp_3 audio to ~$0.0048.
* fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text
get_complete_url interpolated vertex_location straight into the request host,
and vertex_location is client-controllable on the proxy (it flows from the
request body and is not on the request-body blocklist). An authenticated caller
could send vertex_location="attacker.example/" to point the host at their own
server, so the proxy would POST the audio plus its admin-minted Google bearer
token and x-goog-user-project header to the attacker, exfiltrating a
cloud-platform-scoped OAuth token minted from the admin's credentials.
Factor the location validation the rest of vertex_ai already applied in
get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared
validate_vertex_location helper in common_utils and call it from both the chat
host builder and the new speech host builder. Invalid locations now raise a 400
VertexAIError instead of building a host. Also reject vertex_project values that
carry URL-structural characters, since it lands in the URL path.
Regression tests assert on the parsed netloc so the security property is pinned:
valid locations always resolve to a *speech.googleapis.com host and injection
inputs are rejected.
* fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
* feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time
The UI create payload never carried oauth2_flow, so every UI-created oauth2 server
persisted a null flow and relied on _resolve_oauth2_flow's field-shape inference at
registry build. That inference cannot tell a DCR-registered interactive server
(client creds + token_url, no persisted authorization_url) from an M2M server unless
endpoint discovery succeeds first, and the dashboard cannot reproduce it at all
because credentials are redacted in responses
The create form now persists the selected flow for oauth2 servers: authorization_code
for Interactive (PKCE), client_credentials for M2M. The REST create endpoints stamp an
omitted oauth2_flow server-side with the same discriminator the legacy inference uses,
run at write time where the payload carries plaintext credentials, so the decision is
made once with full information and stored. Applied to the admin create, the BYOM
submission, and the temporary session-server endpoints
The edit form derives its flow display from oauth2_flow instead of token_url presence
(token_url is present on authorization_code servers too, so it cannot distinguish M2M)
and deliberately never writes oauth2_flow: it has no flow selector, so a write from
edit could only erase an explicit value, including the authorization_code stamp the
DCR flow persists. Regression tests pin all of this down
Second step of persisting oauth2_flow at every write site so the legacy inference can
eventually be deleted; the backfill for existing null rows lands next
* refactor(mcp): name the create-time flow stamp for its fallback-only contract
stamp_omitted_oauth2_flow with a dedicated explicit-value early return and the shape
check renamed to has_m2m_shape, so the precedence (caller's oauth2_flow always wins,
inference only fills an omitted field) reads directly off the code
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
* feat(jwt): fall back to DB team memberships when JWT has no team claims
* style(jwt): use PEP 585/604 annotations in DB team fallback to clear strict gate
* fix(jwt): preserve DB teams on no-claim sync, model-gate DB fallback, stop team-id leak
When fallback_to_db_teams is enabled and a JWT carries no team claims,
sync_user_role_and_teams previously computed teams_to_remove as every existing
DB membership and wiped the user out of all their teams on each request, which
also left the DB fallback nothing to resolve. Skip team removal in that case so
memberships survive and the fallback can attribute usage.
Apply the same per-team model-access check the claim-based path enforces when
selecting a DB fallback team, so a team's models restriction is no longer
bypassed; a team that cannot serve the requested model is skipped in favor of
one that can.
Drop the user's team-id list from the x-litellm-team-id membership 403 detail so
a valid-JWT caller can no longer enumerate team IDs.
* fix(jwt): load team membership on DB fallback; scope header check to provisional teams
The DB-team fallback resolved a team but never loaded its team membership
row, so per-team membership budget limits were silently skipped on that
path. _resolve_db_team_fallback now fetches the resolved team's membership
when a user_id is known and returns it, matching the claim-based path so
downstream LiteLLM_TeamMembership budget enforcement works there too.
The provisional x-litellm-team-id validation also fired on any non-None
team_id, including an RBAC role-derived one, which 403'd RBAC team flows
when the asserted team was not also a DB membership. It now runs only when
team_id actually came from the header (team_id == header_team_id).
* fix(jwt): surface DB-fallback membership lookup failures at warning level
A transient get_team_membership failure on the DB team fallback path is
recoverable: the team is still resolved and the request proceeds, just
without per-team membership budget enforcement for that request. Logging
that at debug hid a silent budget-enforcement gap from operators, so it now
logs at warning and states that enforcement was skipped. Behavior is
otherwise unchanged: the resolved team is returned with a None membership
rather than failing the request, covered by
test_resolve_db_team_fallback_survives_membership_lookup_error.
* fix(jwt-auth): tighten db-team fallback gating and passthrough enforcement
Resolves four issues in the fallback_to_db_teams path:
- _resolve_db_team_fallback now surfaces a model-access denial when memberships
exist but none can access the requested model, instead of always returning
the no-membership message
- auth_builder gates the fallback on real JWT team claims via
get_all_jwt_team_ids so a configured team_id_default does not silently route
claimless tokens to the default team
- A team selected only via _resolve_db_team_fallback is re-validated against
the team's allowed_passthrough_routes; the earlier gate ran while team_id
was still None
- sync_user_role_and_teams considers both plural and singular team claim
shapes when reconciling DB memberships so singular-only tokens
(Okta/Auth0 defaults) no longer leave stale teams behind
* fix(jwt): don't upsert a provisional x-litellm-team-id before membership check
When fallback_to_db_teams is on and the JWT carries no team claims, an
x-litellm-team-id header is accepted provisionally and only validated against
the user's DB memberships later in auth_builder. With team_id_upsert also
enabled, get_team_object ran the upsert on that unvalidated header team first,
so an attacker-supplied header could create an orphaned team row before the
403 membership check. Suppress the upsert whenever the team is provisional
(db_team_fallback), since a genuine membership team already exists and an
invalid one must not be created. Regression:
test_auth_builder_provisional_header_team_is_not_upserted.
* fix(jwt): pin RBAC-asserted team against db-team-fallback header override
When a JWT carries an RBAC team role but no group claims, auth_builder already
sets team_id from the RBAC object_id. db_team_fallback still evaluated true
there, so the provisional x-litellm-team-id path accepted a header team and
silently overrode the RBAC-asserted team with any team the caller belonged to.
Gate db_team_fallback on team_id being unset, and drive the header's provisional
acceptance off db_team_fallback rather than the raw flag, so an RBAC token plus
a non-claim header team is rejected with 403 instead of substituting the team.
Regression: test_auth_builder_header_cannot_override_rbac_team_under_db_fallback.
* fix(jwt): scope dual-claim membership sync to fallback_to_db_teams
The membership sync read both plural and singular JWT team claims via
get_all_jwt_team_ids unconditionally, which silently changed reconciliation
for every deployment using sync_user_role_and_teams, not just those opting
into fallback_to_db_teams: a singular-only IdP token that previously stripped
all DB teams would now be recognized. Gate the dual-claim read on
fallback_to_db_teams so flag-off deployments keep the upstream plural-only
behavior, honoring the PR's contract that existing deployments are unchanged.
Regression: test_sync_user_role_and_teams_singular_claim_only_recognized_under_flag.
* fix(jwt): drop user team IDs from db-fallback model-access 403 detail
The model-access-denied 403 in _resolve_db_team_fallback echoed the user's
full DB team-id list in its detail. It is only the caller's own memberships,
but it is inconsistent with the membership-validation 403 in the same feature
that was deliberately scrubbed of team IDs. Replace the enumerated list with a
generic "no team you are a member of has access" message. Regression extends
test_resolve_db_team_fallback_distinguishes_no_membership_vs_model_denied to
assert the team id is absent from the detail.
* fix(jwt): keep db-team fallback off for alias-only tokens
* test(jwt): cover alias-only token skipping db-team fallback
The autofix in ed21199 added a get_team_alias clause to the db_team_fallback
gate so an alias-only JWT (team_alias_jwt_field set, no team-id claims)
resolves its alias via find_and_validate_specific_team_id instead of being
mis-attributed to the user's first DB team, but it shipped without a
regression test. This drives auth_builder with an alias-only token whose
alias resolves to a different team than the user's DB membership and asserts
the result is the alias-resolved team; reverting the get_team_alias clause
flips the result to the DB-membership team, so the test fails without the fix
* fix(jwt): prefer alias resolution over team_id_default
When the JWT only carries an alias claim and the operator configures
team_id_default, JWTHandler.get_team_id silently substitutes the
default into find_and_validate_specific_team_id. That made the helper
return the default team without ever attempting alias resolution, so
spend and access attached to the default team even though the token
identified a different team via its alias. Use get_all_jwt_team_ids
(which ignores team_id_default) to detect when the resolved team_id is
only the default and clear it so alias resolution runs first; the
default remains the fallback when no alias claim is present.
* fix(jwt): enforce team_allowed_routes in db-team fallback resolution
The claim-based path runs allowed_routes_check when selecting a team, but
_resolve_db_team_fallback selected a team purely on model access, so a
DB-resolved team could reach routes excluded by team_allowed_routes with no
downstream backstop. This mirrors the claim path's route gate in the fallback,
exempting auth-enforced passthrough routes that are gated separately by
allowed_passthrough_routes at the call site
* fix(jwt): enforce team_allowed_routes on header-team db fallback path
The auto-pick DB-team fallback already gates against team_allowed_routes, but a claimless JWT presenting x-litellm-team-id under fallback_to_db_teams set team_id directly from the header and only re-validated DB membership afterwards, skipping the route gate. A caller could reach management/info routes that the JWT config narrowed for team-role callers by supplying the header even though the auto-pick path on the same route returns no team.
* refactor(jwt): narrow db-team fallback except clauses to actual failure types
* fix(jwt): collapse provisional header team lookup failure into membership denial
A caller holding a valid claimless JWT under fallback_to_db_teams could
distinguish nonexistent teams (404 from get_team_object) from existing
teams they do not belong to (membership 403) by varying x-litellm-team-id,
giving an authenticated team-id existence oracle. The provisional header
path now rewrites the lookup failure into the exact 403 the membership
check raises, while claim-backed header teams keep the upstream 404.
Also drop the unreachable falsy-team guard in _resolve_db_team_fallback
(get_team_object returns a team or raises, never None) and stop codecov
carryforward for three dead flags whose stale sessions were measured
against old file revisions and sank patch coverage with phantom
executable lines
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
ModelArmorGuardrail.async_pre_call_hook and async_moderation_hook hardcoded
their inner should_run_guardrail event type to pre_call / during_call. The
central dispatcher already remaps call_mcp_tool -> pre_mcp_call/during_mcp_call
and passes the outer gate, but Model Armor's redundant inner gate then rejected
MCP calls for a guardrail configured with mode pre_mcp_call/during_mcp_call, so
tool-call content was silently skipped.
Remap call_mcp_tool -> pre_mcp_call/during_mcp_call in both hooks, matching the
existing behavior of the noma and cisco guardrails. Adds regression tests
covering both hooks (scan runs on MCP calls, still skipped for chat traffic).
Generated with AI
Co-Authored-By: Claude Code
Co-authored-by: eugene-yao-zocdoc <eugene.yao@zocdoc.com>
* fix(mcp): forward short OAuth state upstream, keep session in a cookie
Some upstream authorization servers reject the OAuth authorize request with
"state parameter too long" because LiteLLM replaced the client's short state
with its own long encrypted session blob (base_url, original state, PKCE, client
redirect_uri) and sent that upstream as state.
Forward a short random handle as the upstream state instead, and carry the
encrypted session in a per-flow HttpOnly, SameSite=lax cookie bound to that
handle. The browser replays the cookie on /callback, so the session is recovered
without any server-side store and the client still gets its own original state
back. /callback falls back to decoding state directly when no cookie is present,
so flows in flight across a deploy keep working.
Resolves LIT-4197
* test(mcp): cover /callback error path cookie read and clear
The happy-path regression test already asserts the short-handle -> cookie round
trip. Add a focused test for the IdP-error branch of /callback: it must recover
the client's original state from the per-flow cookie (not the short handle),
propagate the error to the client's redirect_uri, and expire the one-time
cookie. Fails if the error path stops reading or clearing the cookie.
