Two bugs from the upstream-error fixes: the success handler has no
status-code awareness, so removing raise_for_status() left it firing for
every upstream 4xx/5xx too, meaning the new failure hook and the existing
success handler both logged the same request (corrupting SpendLogs/cost
tracking). Separately, the failure hook was passed the raw
httpx.HTTPStatusError, which ProxyLogging's alerting only excludes
HTTPException/ProxyException from, so a normal upstream 403 would trigger a
"High" severity llm_exceptions alert. Gates the success handler (both
non-streaming and end-of-stream) to status_code < 400, and reports upstream
failures to post_call_failure_hook as an HTTPException instead of the raw
httpx error, matching how auth/rate-limit errors are already excluded from
alerting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(policies): reject non-existent team/key/model scope entries on attachment create
Creating a policy attachment accepted arbitrary team, key, and model values with
no validation, so a typo'd or non-existent team was silently persisted (LIT-4199).
The create endpoint now rejects a concrete (non-wildcard) team, key, or model that
does not resolve to a real entity, wiring the previously-dead PolicyValidator
existence checks and reusing RouteChecks._is_wildcard_pattern so validation agrees
with request-time matching, where only a trailing "*" is a wildcard. Wildcard
patterns are still allowed through since they may match zero entities today and
more later, and tags stay free-form. The Admin UI's Teams field validates the same
rule for immediate feedback when its team list has loaded, deferring to the backend
otherwise.
* style(policies): use builtin list generics and | None in scope validator
Keeps the new find_invalid_scope_entries signature off the UP006/UP045 strict
ruff budgets instead of copying the surrounding legacy typing.List/Optional idiom.
* fix(policies): separate multiple attachment scope errors with ' | '
Addresses Greptile review: joining per-entry validation messages with a bare
space read as one run-on sentence; ' | ' makes the multi-error 400 detail easier
to parse for users and programmatically.
Follow-up to 8c9878025e: returning upstream 4xx/5xx bodies unchanged also
skipped post_call_failure_hook entirely, so spend-tracking and alerting
callbacks never fired for upstream errors, and response_body was hardcoded
to None in the log payload so the actual upstream error body never reached
logging integrations. Adds a small helper that calls post_call_failure_hook
for upstream errors without altering the client-facing response, and parses
response_body unconditionally for logging while still scoping guardrails
and managed-id rewriting to status_code < 400.
Co-authored-by: Cursor <cursoragent@cursor.com>
Generic pass-through endpoints called raise_for_status() on upstream 4xx/5xx
responses and re-raised as HTTPException, which the outer handler reshaped
into a ProxyException with the upstream body stringified into error.message.
Success responses were already forwarded as-is, so failures were the only
case where passthrough wasn't actually transparent. Removes the
raise_for_status() calls for both streaming and non-streaming passthrough so
upstream status, body, and headers reach the client unchanged, while keeping
guardrails/managed-id rewriting scoped to successful responses and leaving
internal proxy failures (auth, config, network errors before any upstream
response) on the existing ProxyException path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(mcp): discover the OBO token endpoint via RFC 9728 to RFC 8414 (no IdP guessing)
An oauth2_token_exchange server can now have its token endpoint discovered the
same way the oauth2 (authorization_code) flow already does, instead of always
requiring token_exchange_endpoint/token_url to be configured by hand. The
existing _descovery_metadata chain (RFC 9728 protected-resource metadata ->
RFC 8414 authorization-server metadata -> token_endpoint, SSRF-guarded via
async_safe_get) is reused; both the config-load and DB-build paths gate on a new
_obo_needs_endpoint_discovery so discovery runs only when no endpoint is
configured, and an explicitly configured endpoint still wins and skips the
round-trip. The discovered token endpoint lands on token_url, which
_token_exchange_spec already reads, so no resolver change is needed.
_resolve_oauth2_flow returns None for any non-oauth2 auth_type, so a discovered
token_url on an OBO server is never mis-inferred as the M2M client_credentials
flow.
Discovery for OBO is authoritative only: the resolution order is explicitly
configured endpoint, then RFC 9728 -> RFC 8414 advertisement, then fail closed
(412, on the parent commit). The gateway never guesses the IdP. _descovery_metadata
grows an allow_origin_fallback flag, kept True for the browser oauth2 flow (a
human sees the redirect) but set False for token_exchange so the last-resort
guess that treats the resource server's own origin as its authorization server
is skipped; a subject token is never exchanged against an inferred endpoint.
* fix(mcp): surface a failed OBO exchange at connect instead of an empty tool list
A token_exchange server whose exchange fails with a subject present used to open the MCP
session anyway and mask the failure as an empty tools/list. Single-server routes now run
the exchange preemptively at the transport edge, where a rejected subject raises the RFC
9728 challenge and a gateway fault its public status; the multi-server aggregate keeps
absorbing per-server auth failures. The exchanger caches the preflight result, so the
session's list/call reuses it with no extra IdP round-trip. Discovery now also debug-logs
the authorization server's advertised issuer, grant types, and client auth methods
* fix(mcp): persist the discovered OBO token endpoint to the DB row
A DB-backed oauth2_token_exchange server with no configured endpoint had its token_url
resolved via RFC 9728 -> RFC 8414 only on the in-memory object returned from
build_mcp_server_from_table; the row kept token_url=None, so every rebuild re-ran discovery
and a transient upstream outage during a rebuild left the server with no endpoint until the
next successful discovery. Write the discovered token_url back onto the row so the guard sees
it on the next build. Best-effort and scoped to DB servers: config servers already persist
in-memory, and the write-back never fires from a user connect (only from add/update/reload,
all admin or system driven). Adds DB-path coverage for discovery firing when unset, skipping
when the credentials endpoint is configured, the write-back, and its negative guards
* Revert "fix(auth): deny model access for teamless keys with all-team-models (#32022)"
This reverts commit dfbbda4f19.
* revert: undo teamless all-team-models denial from PR #29746
Reverts the team_id guard in _resolve_key_models_for_auth_check and
get_key_models so teamless keys with all-team-models resolve to []
(unrestricted = all proxy models) rather than being denied.
Adds hardened regression tests across listing (get_key_models), inference
(_enforce_key_and_fallback_model_access, can_key_call_model,
can_key_call_resolved_model), and batch (_enforce_batch_file_model_access)
paths that enforce teamless all-team-models == all-proxy-models and will
fail if anyone re-introduces a team_id guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(mcp): thread the caller token into tools/list discovery for token_exchange
A token_exchange (OBO) server's tools could not be discovered through the aggregator: the list path
never threaded the caller's token, so every tools/list hit the no-subject branch. v1 masked this with
its client_credentials fallback (discovery used a service token); v2 dropped that fallback, so listing
had no credential and the OBO server's tools never appeared - and an MCP client lists before it calls.
Thread the inbound subject_token into the list path the same way the call path does, gated on
auth_type oauth2_token_exchange so the caller's bearer never leaks into other modes:
_get_tools_from_server takes an oauth2_headers param, extracts the token via _extract_bearer_token, and
passes it to _create_mcp_client; server.py forwards oauth2_headers at the list call site.
authorization_code (resolves off identity plus stored token), the static/config modes, and the
background registry refresh are unaffected, and the list path's existing graceful degradation
(catch -> empty list) is preserved.
* fix(mcp): harden token_exchange OBO from the audit (strip, TTL/expires_in, subject_token_type)
- _should_strip_caller_authorization returns True for oauth2_token_exchange, so the inbound subject
token is never forwarded upstream raw - only the IdP-exchanged token is (matches authorization_code).
- _parse_expires_in accepts a JSON float / numeric-string expires_in, and _ttl_seconds caps the cache
TTL at the token's real remaining lifetime so a short-lived exchanged token is never served stale.
- to_server_spec normalizes a falsy subject_token_type to the default URN, parity with v1.
The subject/key disambiguation (never exchange the LiteLLM key; Authorization: Bearer <litellm-key>
support for /mcp) is intentionally a separate cross-cutting PR off staging, not part of this OBO work.
* fix(mcp): stop caller header bypassing OBO exchange; thread subject into prompts/resources
The per-server x-mcp-* override guard in _create_mcp_client only kept the v2 spec
for authorization_code, so a caller-supplied header silently disabled the RFC 8693
exchange on a token_exchange server and forwarded the raw bearer upstream. Extend
the guard to token_exchange so the exchange always runs and the caller cannot
substitute an arbitrary upstream credential.
prompts/list+get, resources/list+read, and resource-templates/list never threaded
the OBO subject token, so those operations failed closed (401 / empty) on a
token_exchange server. Thread the caller's bearer as the subject for those paths
too, gated on the token_exchange mode via a shared _obo_subject_token helper.
* fix(mcp): keep the OBO/authz_code resolver credential authoritative; centralize OpenAPI strip
A guardrail (e.g. MCPJWTSigner), static_headers, or any other injected Authorization could
shadow the resolver-owned credential for token_exchange / authorization_code servers, so the
upstream would receive e.g. the signer's JWT instead of the exchanged token and reject it. In
_create_mcp_client the resolver-owned credential now wins: a conflicting header is dropped and
the minted/stored token reaches upstream. No behavior change for none/passthrough/static modes,
where an injected Authorization still wins as before.
The OpenAPI/local _request_extra_headers forwarder gated its Authorization strip on
has_client_credentials only, so an OpenAPI-backed token_exchange server with
extra_headers:[Authorization] forwarded the raw subject token upstream and never exchanged. It
now uses the centralized _should_strip_caller_authorization so it matches the managed paths.
* feat(mcp): RFC 9728 challenge for token_exchange (OBO) unauthorized
OBO previously returned an opaque 401 (Bearer error="invalid_request") with no discovery
info, and any IdP exchange failure collapsed to a retryable 503. Now an OBO server behaves like
a standards-compliant OAuth resource server:
- A missing/rejected subject token returns the RFC 9728 / RFC 6750 challenge: 401 +
WWW-Authenticate: Bearer resource_metadata="...", error="invalid_token", so a spec-compliant
MCP client can discover the IdP, SSO, and retry with a fresh subject token.
- The protected-resource metadata for a token_exchange server advertises the JWT-auth issuer(s)
(JWT_ISSUER / litellm_jwtauth.issuers) as authorization_servers -- the IdP that issues and
validates the subject -- instead of the gateway.
- An IdP 4xx (subject rejected) is now a non-retryable 401 (the challenge) instead of a 503, so a
caller with a dead token re-authenticates rather than looping; 5xx/transport stays retryable 503.
* fix(mcp): emit the OBO RFC 9728 challenge preemptively so a no-subject client can discover the IdP
A token_exchange server's tools are not discoverable without a subject token (list is lenient ->
empty), and a tool-call-time 401 is wrapped into a JSON-RPC error so the WWW-Authenticate header is
lost. So a cold-start client never saw the challenge and could not start discovery. Add a
token_exchange branch to the preemptive-401: a no-subject connect to an OBO server now returns
401 + WWW-Authenticate: Bearer resource_metadata=..., error="invalid_token" at the transport level,
so a spec-compliant client discovers the IdP (the PRM advertises the JWT-auth issuer), SSOs, and
retries with a subject token. Verified live on the per-server endpoint; the with-subject connect
still proceeds (no challenge).
(Also formats two lines from earlier commits in this stack.)
* refactor(mcp): inject root_path into the OBO/OAuth challenge edge
The adapter's raise_user_oauth_challenge and raise_token_exchange_challenge
reached into os.getenv("SERVER_ROOT_PATH") via get_server_root_path(), a
hidden ambient read in a module that is meant to be a pure edge. That coupling
made the preemptive-challenge test order-dependent under xdist: a sibling test
sets SERVER_ROOT_PATH at import without cleanup, leaking the prefix into the
challenge URL and failing the exact-match assertion.
Resolve the root path at the imperative-shell call sites and pass it in
keyword-only, so both challenge builders become pure functions of their inputs.
Extract the shared resource_metadata path construction into a single
oauth_protected_resource_path helper, collapsing the duplicated prefix/name
logic the two functions carried.
Also reduce _create_mcp_client below the strict complexity ceiling by extracting
the v2 credential resolution into _resolve_v2_auth, and extract the OBO
protected-resource-metadata branch into _obo_protected_resource_response (which
shipped without coverage) so discovery can be unit-tested directly.
Tests are now hermetic: the adapter tests pass root_path as a real input rather
than monkeypatching the environment, the stale-session preemptive test asserts
structural invariants instead of the exact prefixed URL, and five new tests
cover the OBO PRM issuer branch end to end.
* feat(mcp): OBO cache-key tenant isolation, reactive 401 retry, v1-parity logs
From a pass over the OBO behavior contract. Three changes to the
token_exchange arm, none of which alters any other auth mode.
The exchanged-token cache key now folds in the caller's tenant alongside
the subject token and exchange config, so two tenants presenting the same
opaque token can never share a cache entry; cross-tenant isolation is
structural rather than incidental to subject-token uniqueness. tenant_id is
threaded from the resolver's Subject; it is keyword-only with an empty
default so the no-tenant case and the existing call sites are unchanged.
The tool-call path gains one reactive retry. When an upstream rejects the
injected token with a 401/403, the gateway invalidates the cached exchange,
re-mints once through the IdP by rebuilding the client, and retries the call
exactly once before surfacing the upstream error, so a token revoked or
rotated upstream mid-TTL self-heals without an infinite loop. It is gated
strictly to oauth2_token_exchange; passthrough, authorization_code,
client_credentials, api_key, and none keep their single-call behavior.
MCPClient.call_tool gains a raise_on_error flag (mirroring list_tools) so
the path can tell an upstream 401 apart from an ordinary tool error and
avoid re-running a non-idempotent tool on a non-auth failure.
The exchanger also emits the v1-parity log lines it had dropped (attempt
with server, endpoint and audience; success; cache hit), while never
logging the form, subject token, secret, or minted token.
* fix(mcp): fail closed with 412 when a token_exchange server has no endpoint
A true token_exchange (OBO) server must use only an explicitly configured
token endpoint; it must never guess an IdP or silently fall back to a weaker
source. Previously an OBO server with client credentials but no
token_exchange_endpoint/token_url deferred to v1, which no-op'd and let the
request connect to the upstream with no credential (an upstream 401 rather
than a clear gateway error).
Now such a server is owned by the v2 arm: _token_exchange_spec builds the spec
even when the endpoint is absent, and the exchanger fails closed with a
precondition_required error that maps to HTTP 412 before any upstream or IdP
call, with the caller's subject token never sent anywhere. A missing
client_id/secret still maps to misconfigured (500); a present-but-rejected
subject still maps to 401; an unreachable IdP still maps to 503. The no-subject
case keeps its existing 401 RFC 9728 challenge.
* feat(mcp): log a refused non-Bearer token_type in the OBO exchange
* fix(mcp): surface OBO/authorization_code list-time 401 as a challenge instead of masking it
* feat(mcp): classify RFC 6749 gateway-fault token-exchange errors as 500, not a caller 401
* test(mcp): absorb fixture uses 500 now that 401/403 are challenge-class at list time
* style(mcp): PEP 604 union in the OBO retry signature to keep the UP007 budget flat
* feat(mcp): v2-native RFC 8693 token exchanger for the token_exchange mode
Adds the pure Rfc8693TokenExchanger plus its composition root: the OBO exchange POSTs the
RFC 8693 grant through an injected HTTP edge and returns the upstream-bound token as a typed
Result, caching and single-flighting per (subject_token, server) so a repeated caller token
skips the IdP round-trip. The audience is carried on TokenExchangeConfig and sent only when the
operator set one, matching the spec default behavior. Errors are values: a missing endpoint or
client credential is misconfigured, an IdP that returns no usable token is upstream_unavailable.
* feat(mcp): migrate the token_exchange arm to the v2 resolve_credentials
Routes RFC 8693 OBO servers through the v2 resolver: the resolver arm reads the caller's
inbound token and swaps it via the injected TokenExchanger, to_server_spec maps a complete
oauth2_token_exchange server (endpoint plus client credentials) to TokenExchangeConfig, and the
egress wires the LazyTokenExchanger in. A token_exchange server with no caller token fails closed
with a plain 401 rather than v1's fall-through to client_credentials, so the call site now scopes
the per-server browser-OAuth challenge to authorization_code and lets other modes raise their own.
* fix(mcp): bind the token-exchange cache key to the exchange config
The exchanged-token cache was keyed only by (subject_token, server_id), so rotating a server's
audience, scope, endpoint, client_id, or secret kept serving a token minted for the old config
until TTL. The key now hashes the caller token together with the config that minted it, so a config
change forces a fresh exchange. Everything is hashed, so no secret is held in the key.
* refactor(mcp): build the token exchanger eagerly, dropping the lazy wrapper
The token exchanger reads no runtime global at build time (its httpx client is acquired per call),
unlike the per-user store, so it does not need lazy first-use construction. Building it once at
egress construction removes the first-use init path entirely and keeps the process-lifetime cache.
* fix(mcp): map non-object token-exchange JSON to a miss instead of a 500
The post adapter annotated the parsed body as a dict without checking it, so a valid-but-non-object
JSON response (list/string/number) was returned as-is and crashed the field parsing with an
AttributeError. It now validates the shape at the boundary and returns None for a non-object body,
so a malformed IdP response surfaces as a typed upstream_unavailable rather than a server error.
* fix(mcp): fail closed on a non-Bearer token_type in the OBO exchange
* feat(mcp): honor token_endpoint_auth_method (client_secret_basic) in the v2 OBO exchange
* feat(mcp): reject a non-access issued_token_type in the OBO exchange
* fix: include token endpoint auth method in exchange cache key
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
_with_resolved_session_model was overwriting the nested
input_audio_transcription.model and audio.input.transcription.model with the
realtime conversation model, silently replacing a caller's transcription model
(e.g. whisper-1) since those are a different model than the realtime deployment.
It now only resolves the top-level session model.
Also restores session.model taking precedence over the top-level model in
acreate_realtime_client_secret, matching the proxy's own
_prepare_client_secret_session ordering and avoiding a backwards-incompatible flip.
Adds routing coverage for arealtime_calls (api_base resolution) and
acreate_realtime_transcription_session (api_key resolution) so all three realtime
HTTP endpoints have router credential-resolution tests, plus regression tests for
the two fixes above.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(mcp): add all-proxy-mcpservers sentinel to grant every MCP server
Teams can now be scoped to the all-proxy-mcpservers sentinel so they gain
access to every MCP server on the proxy without listing each id. The
sentinel expands to the live registry at request time, so a server added
later is picked up with no change to the team's stored permission. The team
ceiling that validates a key's MCP scope expands the sentinel too, so a key
can be scoped to any server (including one registered after the team) and
still pass subset validation
Expose the option in the team create and edit forms via a new exclusive
"All Proxy MCP Servers" choice in MCPServerSelector, mirroring the existing
"No MCP Servers" sentinel
* Update litellm/proxy/management_helpers/object_permission_utils.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): honor all-proxy-mcpservers only on the team path, never per-key
The sentinel was expanded inside the shared expand_permission_list, which
also feeds the key, org, end_user and agent resolvers. A key whose stored
object_permission ever held all-proxy-mcpservers (a stale write, a
configured default, or a bug) would silently resolve to every MCP server at
runtime, and a teamless key had nothing to cap it, so all servers got
injected. Only write-time validation stripping the value stood between that
value and a full grant
Move the expansion out of expand_permission_list and into
_get_allowed_mcp_servers_for_team so the sentinel is honored only where it is
settable (a team). Anywhere else it now passes through as an inert literal
that matches no registered server and is denied downstream. Reserved-id
protection already blocks a real server from taking that id
* fix(mcp): require proxy admin to grant a team the all-proxy MCP sentinel
Granting a team every MCP server on the proxy is a proxy-wide authorization
decision, but team create/update let any caller who can manage a team set
object_permission.mcp_servers, with no ceiling check. Org admins reach
/team/update by default (org_admin_allowed_routes) and _verify_team_access
also admits team admins, so a non-proxy-admin could set all-proxy-mcpservers
and self-grant their team access to every MCP server on the proxy, including
servers never assigned to that team
Gate the grant in new_team and update_team: a non-proxy-admin cannot add the
all-proxy-mcpservers sentinel. The check is scoped to newly adding it, so a
team a proxy admin already scoped to all-proxy can still be edited by a team
admin without being forced to strip the sentinel. The UI only offers the
"All Proxy MCP Servers" option to proxy admins in the team create and edit
forms
* fix(ui): render friendly all-proxy MCP label for non-admins editing an all-proxy team
A team scoped to the all-proxy-mcpservers sentinel could be opened in the team
edit form by a team admin or org admin (canEditTeam admits them), but the
"All Proxy MCP Servers" option in MCPServerSelector was rendered only behind the
proxy-admin-gated allowAllProxyMcpServers flag. For a non-proxy-admin the stored
sentinel was hydrated into the selected value with no matching Select.Option, so
antd showed the raw all-proxy-mcpservers literal as a chip, and adding another
server could persist a mixed [all-proxy-mcpservers, <id>] value.
Render the option whenever the sentinel is present in the value, not only when
the caller may grant it, and drive the real-option disabling off presence too so
the selection stays exclusive. A non-proxy-admin now sees the friendly label
read-only and cannot build a mixed state; only a proxy admin can newly add it,
which the backend already enforces.
Adds regression tests: the selector shows the friendly option (not the raw
literal) when the sentinel is stored but the grant flag is off, plus exclusive
emit and disabled-real-options coverage, and MCPServerPermissions renders the
green "All" state instead of the raw sentinel string.
* fix(ui): drop redundant "All servers" hint from the all-proxy MCP chip
antd renders a Select option's children inside the selected tag, so the
all-proxy option showed both "All Proxy MCP Servers" and the green "All servers"
type-hint in the chip, which say the same thing. Collapse the option to a single
green "All Proxy MCP Servers" label so the dropdown row and the chip read cleanly
without the duplication.
* fix(ui): color the all-proxy MCP label blue to match server chips
Use the same blue (#1890ff) as regular MCP server entries for the
"All Proxy MCP Servers" option/chip instead of green.
* fix(ui): make the all-proxy MCP permissions display blue, not green
Match the blue used by the selector chip and regular server entries so the
"All Proxy MCP Servers" badge and row in MCPServerPermissions are consistent
across the team/key/org detail views. The red "Blocked" state for
no-mcp-servers is unchanged.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Two follow-ups from review of the url/db work.
A Redis/Valkey url can embed a password (redis://:secret@host:6379/1), but
_CACHE_SENSITIVE_FIELDS only masked the discrete password and sentinel_password,
so a stored password-bearing url came back in plaintext from every
GET /cache/settings. Add url to the masked set so it gets the same masked-on-read
treatment as password.
The url-precedence resolver dropped host/port/db/password but not username, even
though a url can encode a username too (redis://user:pass@host). Left in, the
discrete username rode along and could contradict the url. Add username to the
overridden set and update the Redis URL help text to list it among the fields url
takes precedence over.
Tests: GET masks a password-bearing url (secret never returned verbatim) while a
non-credential field is untouched, and the resolver drops a discrete username when
a url is present.
Realtime client_secrets, calls, and transcription_sessions were bypassing
the router and falling back to an empty OPENAI_API_KEY for wildcard, team-scoped,
and credential-name deployments.
Co-authored-by: Cursor <cursoragent@cursor.com>
The typed cache-settings form already renders a Redis URL and a Database
Index field, but the backend never defined them, so GET /cache/settings
could not round-trip a saved value into the form and the "URL takes
precedence over Host/Port/Password/Database Index" help text the UI shows
was not actually enforced anywhere.
Add the url and db entries to CACHE_SETTINGS_FIELDS so the endpoint knows
about them, and add _resolve_cache_url_precedence: when a non-empty url is
present it wins and the discrete host/port/db/password fields are dropped
before the settings are tested or persisted, matching how litellm._redis
resolves the connection at runtime (redis.Redis.from_url ignores them).
Cluster mode is exempt because it authenticates via the discrete fields
rather than a url. Both test and save paths go through the resolver so the
stored config is unambiguous.
This finishes LIT-3996: operators can now isolate the cache into a logical
database (e.g. redis://host:6379/1) entirely from the Admin UI instead of
hardcoding REDIS_URL in the environment.
decrypt_value_helper logged `Unable to decrypt value={value}` at DEBUG, which
printed the raw secret whenever decryption failed (for example after a salt or
master key change). This is the same environment_variables config path the
db-config redaction covers, so a DATABASE_URL connection string could still
leak here when the module regex scrubber is bypassed. Drop the value; the key
already identifies the failing pair.
Regression forces a decrypt failure with the redaction filter disabled and
asserts the raw value never reaches a log record while the key stays visible.
Reuse the existing recursive `_redact_secret_values_in_obj` for the worker
config log instead of a hand-rolled top-level pass, so a credential nested
under general_settings is masked at any depth and depth overrun fails closed.
Route the `_update_config_from_db` param_value log (the store_model_in_db
path) and the litellm_settings apply-loop log through the same redactors, so
master_key, database_url, and secret-named settings such as api_key stop
leaking at DEBUG when the module regex scrubber is bypassed. A plain setting
like num_retries still logs its real value.
Regression tests disable _ENABLE_SECRET_REDACTION and cover the nested worker
config shape, the db-config path, and the litellm_settings loop in both
directions.
Three startup log statements in litellm/proxy/proxy_server.py dumped
secret-bearing values in cleartext when the last-line-of-defense regex
scrubber was bypassed (LITELLM_DISABLE_REDACT_SECRETS=true, older versions
that predated the SecretRedactionFilter, or any downstream handler that
snapshots log records before the module filter runs)
ProxyConfig._load_alerting_settings logged the whole general_settings
dict under a label that only referred to the alerting callbacks; a
copy-paste bug that happened to leak master_key, database_url, and every
other secret sitting in general_settings. Now logs only the alerting
callback list
ProxyConfig.load_config logged the resolved DB URL after secret-manager
resolution. The line's stated purpose was to confirm the retrieval ran,
which does not need the value. Now logs a value-less breadcrumb
proxy_startup_event logged the raw WORKER_CONFIG blob, which docker/K8s
deployments hand the proxy as a JSON string containing master_key,
database_url, and provider API keys. Now routes through
_redact_worker_config_for_logging, which combines the segment-matching
SensitiveDataMasker (catches master_key, api_key, *_token) with an
explicit pass over _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS (catches
database_url and other credential-URL fields the segment masker misses)
Regression tests disable the module-level SecretRedactionFilter so
assertions see the raw record; without the fix they would trip on the
secret substring, so a future refactor cannot silently reconstruct the
leaky string
- Prevent fail-open from registering user-supplied hashes as valid for
CCR retrieval; _call_compress now returns (messages, compressed_ok)
so apply_guardrail skips hash extraction and tool injection when
compression did not succeed
- Remove Optional wrapper from HeadroomGuardrailConfigModel.unreachable_fallback
to match BaseLitellmParams typing
- Add fail_open tests for non-JSON response, missing messages key, and
empty message list paths
- Add regression test verifying fail_open does not authorize attacker-planted
hashes
- Regenerate dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
async_handler.post catches httpx.TimeoutException and re-raises it as
litellm.Timeout (a subclass of openai.APITimeoutError). The except
blocks in _call_compress and _call_retrieve only listed httpx exception
types, so litellm.Timeout propagated uncaught and bypassed the
unreachable_fallback=fail_open path.
Add litellm.Timeout to both except clauses and add regression tests for
the fail_closed and fail_open timeout paths.
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix: reuse persisted MCP DCR clients
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
litellm's async httpx client already calls raise_for_status() internally, so a
non-2xx /v1/compress response surfaced as an uncaught httpx.HTTPStatusError
instead of going through the guardrail's status_code check. Caught live by
running the guardrail against a mock headroom endpoint that returns 500:
unreachable_fallback=fail_open silently failed to forward the request until
this fix.
The bulk-update entrypoints `/key/bulk_update` and `/team/key/bulk_update`
route through `_process_single_key_update`, not through
`_validate_update_key_data`, so the `permissions` gate LIT-4092 wired
into the single-key path never fires on bulk. Currently safe by
construction: `BulkUpdateKeyRequestItem` doesn't declare `permissions`
(Pydantic silently drops it), and `KeyUpdateFields` uses
`model_config = ConfigDict(extra="forbid")` (Pydantic 422s at parse
time). Neither structural barrier is enforced by tests on the field
itself; a future widening of either allowlist to include `permissions`
would reopen the class silently.
This wires `_check_permissions_caller_permission` into
`_process_single_key_update` right after `_validate_max_budget`, before
`prepare_key_update_data`. Zero behavior change today for any caller
routing through the current bulk request models; a defense-in-depth
gate for the class.
Tests:
- test_process_single_key_update_non_admin_permissions_rejected
- test_process_single_key_update_non_admin_permissions_explicit_empty_rejected
Both mutation-killed against removing the gate. Full mapped test file
(341 tests) green.
`_check_allowed_routes_caller_permission` previously keyed its
admin-only rule on truthiness. The refactor adds an `allowed_routes_was_provided`
keyword param that raw-body call sites populate from
`"allowed_routes" in data.model_fields_set`, so a caller that omits
the field (default flows through) is distinct from one that sends
any explicit value.
Four raw-body call sites now pass `allowed_routes_was_provided=...`:
`_common_key_generation_helper`, `generate_service_account_key_fn`,
`_validate_update_key_data`, and `regenerate_key_fn`.
Two derived-value call sites keep the pre-fix shape: the
post-`handle_key_type` recheck at `_common_key_generation_helper`
and the mirror in `regenerate_key_fn`. Both pass values produced by
`handle_key_type` (not by the request body), so `allowed_routes_was_provided` stays
False and the `allow_safe_presets=True` carve-out continues to accept
the `llm_api_routes` / `info_routes` presets.
In `regenerate_key_fn` the gate runs before the `premium_user`
license check, matching the LIT-4092 ordering.
`test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate`
pins the ordering; it fails on a swap of the two gates.
Tests in `tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py`
under `TestAllowedRoutesCallerPermission`:
- test_non_admin_generate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_null_allowed_routes_rejected
- test_non_admin_regenerate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate
- test_helper_accepts_derived_safe_preset_for_non_admin
- test_helper_rejects_derived_unsafe_preset_for_non_admin
- test_helper_rejects_when_provided_and_none_without_typeerror
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Three helper-level tests pin the derived-value branch and the
load-bearing None guard; each is mutation-killed against a targeted
change to the frozenset or the guard. Full mapped test file (347 tests)
green.
`NewUserRequest` and `UpdateUserRequest` inherit `permissions` from
`GenerateRequestBase`. `/user/new` passes the field into
`generate_key_helper_fn` which persists it on the auto-created key,
so an org admin who lands on `/user/new` (the route allowlist accepts
org_admin callers when the request body names an org where they hold
that membership) can mint a key with proxy-wide capabilities such as
`get_spend_routes`.
This wires the existing `_check_permissions_caller_permission` helper
into `new_user` and `_update_single_user_helper`. The helper's presence
check keys on `data.model_fields_set`, so an omitted field flows
through untouched and an explicit `{}` / `null` from a non-admin is
rejected 403 the same as any other value.
`_update_single_user_helper` is shared by `/user/update` and
`/user/bulk_update`, so both paths inherit the gate.
Tests in
`tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py`:
- test_new_user_non_admin_permissions_non_empty_rejected
- test_new_user_non_admin_permissions_explicit_empty_rejected
- test_new_user_non_admin_omits_permissions_succeeds (control)
- test_new_user_admin_can_set_permissions (control)
- test_update_single_user_non_admin_permissions_rejected
- test_update_single_user_non_admin_permissions_explicit_empty_rejected
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Full mapped test file (79 tests) green.
Reuses the existing unreachable_fallback flag (already implemented by
generic_guardrail_api, akto, vigil_guard, repelloai) so headroom compression
failures can forward the request uncompressed instead of blocking it with a
502.
_enforce_key_and_fallback_model_access and can_key_call_resolved_model
both unconditionally skipped key-level model checks whenever
all-team-models was present, without verifying the key actually belongs
to a team. PR #29746 fixed the listing path (get_key_models) but these
two call-path checks were left untouched, letting teamless keys call
any model via /chat/completions while seeing an empty model list.
Add team_id is not None guard to both bypass conditions so teamless
keys fall through to can_key_call_model, which already correctly
rejects unresolvable all-team-models sentinels
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
A 401 while listing tools (a missing or expired per-user OAuth token, or an
upstream 401 for any auth_type) was swallowed to an empty tool list, so a
single-server client got a 200 with no tools and no WWW-Authenticate challenge
instead of a 401 it could re-authenticate against. Only oauth pass-through and
delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the
missing-token case for all of them, masked it.
The surface-vs-absorb decision now keys on the route, not the auth_type. An
upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError
regardless of auth_type, and the per-user OAuth challenge raised during client
creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is
converted to the same type in _get_tools_from_server. The challenge is scoped
to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a
re-auth signal and degrades to an empty list like any other non-auth error, and
the stdio-allowlist 403 (no challenge header) stays absorbed. The existing
routing then does the right thing: single-server routes turn the error into a
401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty
list so one unauthenticated server does not fail the whole listing.
On the UI tools page, an OBO (per-user authorization_code) server now shows the
Authorize gate when the list call returns 401, not only when no credential row
exists. The backend already refreshes a still-refreshable token on the list
call, so a 401 means there is no valid token and none could be minted (expired
with no usable refresh token), which is exactly when the user must reauthorize.
Add an optional per-server max_concurrent_requests that caps how many
tool calls LiteLLM sends to one MCP server at once, so batch-processing
backends are not overwhelmed by unbounded parallel dispatch. Excess calls
queue on a per-server asyncio.Semaphore instead of being rejected. Unset
or non-positive means unlimited, preserving existing behavior.
Resolves LIT-2749
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
For LITELLM_METADATA_ROUTES (responses, /v1/messages, batches, etc.),
the proxy stores admin metadata under data["litellm_metadata"] while
user-supplied metadata lives in data["metadata"]. Tags placed in
metadata.tags by the caller were never merged into litellm_metadata.tags,
causing SpendLogs.request_tags to be empty on these routes
Closes#31584
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2
A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no
client_id and no authorization URL, yet the gateway OAuth endpoints did not
check auth_type. authorize() raised "client_id is required" before the
auth_type was ever examined, and the .well-known discovery builders always
advertised authorization_servers / authorization_endpoint / token_endpoint /
registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth
flow that can never succeed.
Add an auth_type != oauth2 guard to the authorize, token, register,
protected-resource and authorization-server paths (covering the internal UI
OAuth endpoints too). The discovery guard sits after the OAuth pass-through
branch so genuine pass-through servers keep proxying their upstream metadata.
oauth2 servers are unaffected.
* fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle
Address review feedback on the auth_type gate.
The 400 message no longer claims access is governed by access groups, which is
only true for auth_type=none; it now states that the gateway runs the OAuth
client_id/authorize/token/register flow only for oauth2 servers and that the
server is reached using its configured auth_type, which is accurate for every
non-oauth2 type (api_key, oauth2_token_exchange, etc.).
The discovery gate previously 404'd a named non-oauth2 server but still returned
200 metadata for an unknown name, which both serves a broken document for a typo
and lets an unauthenticated caller enumerate non-OAuth server names by comparing
404 vs 200. A named discovery request now returns 200 only when it resolves to an
oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root
discovery and pass-through servers are unaffected.
* Apply suggestions from code review
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Route non-full-admin callers through _sanitize_mcp_server_list_for_non_admin,
matching the pattern the fetch and list handlers adopted. Replace the two
regression tests that pinned the old partial-blank behavior with a
sanitize/full-admin pair mirroring the fetch/list coverage.
Resolves LIT-3929
* fix(mcp): roll up MCP tool spend to user counters and usage UI
Direct REST MCP tool calls now fire success logging so spend_logs and
user/team rollups include configured mcp_server_cost_info charges.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate key-info enrichment to requests missing user_id; fix import order
- Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is
absent, avoiding a cache/DB lookup on every normal LLM request.
- Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure
Add api_key = ANY($2) to the MCP session aggregate query so it is
bounded by the same ownership already applied to the main page query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix spend logs for call and list mcp tools
* Add tags in mcp logging
* Fix ruff
* fix(lint): replace List/Dict with list/dict in new annotations (UP006)
Replace the 8 new UP006 violations introduced by the mcp-tags changes:
- Optional[List[str]] → Optional[list[str]] for request_tags params
- List[str] return type → list[str] in _get_parent_request_tags
- Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError
* fix(mcp): keep final streaming chunk when draining inner stream fails
* fix: handle MCP logging edge cases
* fix: propagate MCP logging cancellation
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in 94fd2bf made the no-mcp-servers sentinel return the caller's
submitted BYOM servers, which weakened an explicit key-level opt-out into a
soft preference. Restore the absolute opt-out and additionally skip the BYOM
union for keys with an explicit object_permission.mcp_servers list and for
toolset-scoped requests, mirroring how allow_all_keys servers are handled.
Add unit tests for the sentinel, explicit scoping, toolset scope, the cache
invalidation helper, the cache-miss DB path, and the db.py query helper.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
The handler returned decrypted callback environment values and alerting
routing values verbatim to callers who were not full PROXY_ADMIN. Gate
those on full-admin role, matching the posture used on the sibling
config-inspection endpoints. Non-sensitive routing fields (host / base
URL / port style values) stay visible so the UI can still label which
integration is wired up. Full PROXY_ADMIN sees everything unchanged so
the edit form round-trips on save.
Resolves LIT-4115.
The `_check_permissions_caller_permission` helper introduced in
#31469 was only wired into `_common_key_generation_helper`. This
change wires it into `_validate_update_key_data` and `regenerate_key_fn`
so the three write paths share the admin gate, and refactors the
helper to accept the full request model so it can key on
`"permissions" in data.model_fields_set` rather than truthiness. The
presence check keeps the model-level omit default flowing through
unchanged while treating any explicit value (including `{}` / `null`)
as an admin-only write.
In `regenerate_key_fn` the gate is placed before the `premium_user`
license check so the rejection is consistent across premium and
non-premium deployments. That ordering is pinned by
`test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate`
Tests in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py:
- test_update_key_non_admin_permissions_non_empty_rejected
- test_update_key_non_admin_permissions_explicit_empty_rejected
- test_update_key_non_admin_permissions_explicit_null_rejected
- test_update_key_non_admin_omits_permissions_succeeds (control)
- test_update_key_admin_can_set_permissions (control)
- test_regenerate_key_non_admin_permissions_rejected
- test_regenerate_key_non_admin_permissions_explicit_empty_rejected
- test_permissions_explicit_empty_rejected_for_non_admin_on_generate
- test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate
Mutation-killed against gate removal on either wire, against reverting
the helper to a truthiness check, and against reordering the gate past
the enterprise-license check
* fix(proxy): authorize /health/test_connection against loaded deployment's team_id (VERIA-441)
POST /health/test_connection looked up a deployment by request-supplied model_info.id, dumped its
litellm_params (including api_key) into the outbound probe, merged request params over it, and then
authorized the call against the caller-supplied model_info.team_id. A team admin could pass another
team's deployment id together with their own team_id and an attacker-controlled api_base, sending
the victim team's provider key to that URL.
Capture the loaded deployment's model_info alongside its litellm_params in both the id-lookup and
the model_name fallback paths, and pass that captured value to can_user_make_model_call. When no
deployment is loaded (caller is probing fresh, request-supplied credentials), keep using the
request body's model_info; no foreign deployment is in scope and the existing role check still
requires admin or team-admin.
Add two regression tests that wrap (not mock) ModelManagementAuthChecks.can_user_make_model_call,
one per resolution path, asserting HTTP 403 and that the auth check was reached with the loaded
deployment's team_id. Both fail on the pre-fix code.
* test(health): add positive-path regression through real auth (VERIA-441)
The two deny tests already exercise the real (wrapped) ModelManagementAuthChecks.
Add a matching positive-path test so a mutation that swaps the auth team_id for
a deny-all value on the legit path also fails: loaded deployment owned by team-X,
caller admin of team-X -> asserts HTTP 200 and that the auth check ran with the
LOADED deployment's team_id.
* refactor(test): rename health endpoint tests for clarity (VERIA-441)
Rename test functions and variables from attacker/victim/owner framing to
neutral team-a/team-b terminology. Update docstrings to remove exploit-specific
language. Tests remain functionally identical, covering deny paths (cross-team
deployments) and the positive path (same-team deployments).
Addresses Greptile review feedback: wrap the fallback loop in try/except
BaseException to always restore self.data['model'] to the original value
when a non-ProxyRateLimitError exception escapes a fallback attempt.
Add regression test for this edge case
MCP servers created through the UI are persisted to the database independent of
store_model_in_db, but the in-memory registry that GET /v1/mcp/server reads was
hydrated from the database only through add_deployment, which runs solely when
store_model_in_db is True. On a DB-backed single-instance proxy with
store_model_in_db unset the registry started empty after a restart, so the MCP
Servers page showed nothing until an add or edit triggered a reload.
Hydrate the registry from the database on startup regardless of store_model_in_db
via a new ProxyConfig.init_mcp_servers_from_db, honoring supported_db_objects.
When pre-call hooks (parallel_request_limiter, dynamic_rate_limiter_v3)
reject a request with ProxyRateLimitError, the router's fallback logic
was never reached because the exception was raised before route_request
was called.
Add _pre_call_with_fallbacks that catches ProxyRateLimitError, resolves
configured fallbacks (key-level router_settings -> router-level), and
retries with each fallback model in order. If all fallbacks are also
rate-limited, the original error is re-raised.
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
* feat(proxy): audit remaining system-wide settings updates
Extends the audit logging framework introduced in the parent PR to the
rest of the LiteLLM_Config writers and the two adjacent settings tables:
/config/update (general, environment_variables, litellm_settings,
router_settings sections), /config/field/update, /config/field/delete,
/config/callback/delete, /update/default_team_settings,
/update/mcp_semantic_filter_settings, /add/allowed_ip,
/delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings,
/update/ui_settings.
Each writer records the actor, action, the affected config section, and
a redacted before/after snapshot. SSO and UI settings rows use their own
table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback
and /update/sso_settings audits fire BEFORE the proxy reload and the env
cleanup step respectively, so a failure in either leaves the audit row
intact.
The audit-actor parameter on _update_litellm_setting is now required
rather than optional; the chokepoint covers default_team and
mcp_semantic_filter for free, and a future caller that forgets the
actor fails loudly instead of silently skipping the audit. The two
direct-calling tests pass a dummy actor.
The environment_variables section redacts every value rather than
relying on key-name matching, because it carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
* fix(proxy): capture redacted SSO before-snapshot in audit log
Greptile review of #31754 flagged update_sso_settings as the one endpoint
where before_value is permanently None, so the LiteLLM_SSOConfig audit
trail has no pre-change state. An auditor reviewing a secret-rotation
event could see what the SSO settings were changed to but not what they
were before.
Read the existing SSO row before the upsert, decrypt it via
proxy_config._decrypt_db_variables, and pass it as before_value.
create_config_audit_log's secret-name redaction then masks the
*_client_secret fields, so neither the old nor the new plaintext secret
lands in the audit row.
Add a regression test asserting the before-snapshot reflects the
pre-change values for non-secret fields (google_client_id) and is
redacted for secret fields (google_client_secret). Mutation-checked
against reverting to before_value=None.
The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique
since the endpoint reads it; the read returns None for tests that do not
care about the before-state.
* fix: remove committed zero init migration
* refactor(proxy): audit config writes via asyncio.create_task everywhere
PR A's chokepoint audit call was refactored from a blocking await to
asyncio.create_task so that a post-save audit-log failure could not
surface as a 500 to the caller. The 12 other audit call sites added in
this PR were still using await, reintroducing the exact 500-after-commit
exposure at every sibling endpoint. Wrap them all in asyncio.create_task
to match the model_management_endpoints / key_management_endpoints /
hooks / config_override_endpoints / team_callback_endpoints /
cache_settings_endpoints house pattern, so the codebase tells one story.
The two direct-invocation tests (test_update_config_general_settings and
test_delete_config_general_settings, which call the handler in-process
rather than via TestClient) yield with `await asyncio.sleep(0)` after the
handler returns so the scheduled audit task runs before the assertion.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
