* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix: reuse persisted MCP DCR clients
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The bulk-update entrypoints `/key/bulk_update` and `/team/key/bulk_update`
route through `_process_single_key_update`, not through
`_validate_update_key_data`, so the `permissions` gate LIT-4092 wired
into the single-key path never fires on bulk. Currently safe by
construction: `BulkUpdateKeyRequestItem` doesn't declare `permissions`
(Pydantic silently drops it), and `KeyUpdateFields` uses
`model_config = ConfigDict(extra="forbid")` (Pydantic 422s at parse
time). Neither structural barrier is enforced by tests on the field
itself; a future widening of either allowlist to include `permissions`
would reopen the class silently.
This wires `_check_permissions_caller_permission` into
`_process_single_key_update` right after `_validate_max_budget`, before
`prepare_key_update_data`. Zero behavior change today for any caller
routing through the current bulk request models; a defense-in-depth
gate for the class.
Tests:
- test_process_single_key_update_non_admin_permissions_rejected
- test_process_single_key_update_non_admin_permissions_explicit_empty_rejected
Both mutation-killed against removing the gate. Full mapped test file
(341 tests) green.
`_check_allowed_routes_caller_permission` previously keyed its
admin-only rule on truthiness. The refactor adds an `allowed_routes_was_provided`
keyword param that raw-body call sites populate from
`"allowed_routes" in data.model_fields_set`, so a caller that omits
the field (default flows through) is distinct from one that sends
any explicit value.
Four raw-body call sites now pass `allowed_routes_was_provided=...`:
`_common_key_generation_helper`, `generate_service_account_key_fn`,
`_validate_update_key_data`, and `regenerate_key_fn`.
Two derived-value call sites keep the pre-fix shape: the
post-`handle_key_type` recheck at `_common_key_generation_helper`
and the mirror in `regenerate_key_fn`. Both pass values produced by
`handle_key_type` (not by the request body), so `allowed_routes_was_provided` stays
False and the `allow_safe_presets=True` carve-out continues to accept
the `llm_api_routes` / `info_routes` presets.
In `regenerate_key_fn` the gate runs before the `premium_user`
license check, matching the LIT-4092 ordering.
`test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate`
pins the ordering; it fails on a swap of the two gates.
Tests in `tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py`
under `TestAllowedRoutesCallerPermission`:
- test_non_admin_generate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_null_allowed_routes_rejected
- test_non_admin_regenerate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate
- test_helper_accepts_derived_safe_preset_for_non_admin
- test_helper_rejects_derived_unsafe_preset_for_non_admin
- test_helper_rejects_when_provided_and_none_without_typeerror
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Three helper-level tests pin the derived-value branch and the
load-bearing None guard; each is mutation-killed against a targeted
change to the frozenset or the guard. Full mapped test file (347 tests)
green.
`NewUserRequest` and `UpdateUserRequest` inherit `permissions` from
`GenerateRequestBase`. `/user/new` passes the field into
`generate_key_helper_fn` which persists it on the auto-created key,
so an org admin who lands on `/user/new` (the route allowlist accepts
org_admin callers when the request body names an org where they hold
that membership) can mint a key with proxy-wide capabilities such as
`get_spend_routes`.
This wires the existing `_check_permissions_caller_permission` helper
into `new_user` and `_update_single_user_helper`. The helper's presence
check keys on `data.model_fields_set`, so an omitted field flows
through untouched and an explicit `{}` / `null` from a non-admin is
rejected 403 the same as any other value.
`_update_single_user_helper` is shared by `/user/update` and
`/user/bulk_update`, so both paths inherit the gate.
Tests in
`tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py`:
- test_new_user_non_admin_permissions_non_empty_rejected
- test_new_user_non_admin_permissions_explicit_empty_rejected
- test_new_user_non_admin_omits_permissions_succeeds (control)
- test_new_user_admin_can_set_permissions (control)
- test_update_single_user_non_admin_permissions_rejected
- test_update_single_user_non_admin_permissions_explicit_empty_rejected
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Full mapped test file (79 tests) green.
_enforce_key_and_fallback_model_access and can_key_call_resolved_model
both unconditionally skipped key-level model checks whenever
all-team-models was present, without verifying the key actually belongs
to a team. PR #29746 fixed the listing path (get_key_models) but these
two call-path checks were left untouched, letting teamless keys call
any model via /chat/completions while seeing an empty model list.
Add team_id is not None guard to both bypass conditions so teamless
keys fall through to can_key_call_model, which already correctly
rejects unresolvable all-team-models sentinels
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
A 401 while listing tools (a missing or expired per-user OAuth token, or an
upstream 401 for any auth_type) was swallowed to an empty tool list, so a
single-server client got a 200 with no tools and no WWW-Authenticate challenge
instead of a 401 it could re-authenticate against. Only oauth pass-through and
delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the
missing-token case for all of them, masked it.
The surface-vs-absorb decision now keys on the route, not the auth_type. An
upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError
regardless of auth_type, and the per-user OAuth challenge raised during client
creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is
converted to the same type in _get_tools_from_server. The challenge is scoped
to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a
re-auth signal and degrades to an empty list like any other non-auth error, and
the stdio-allowlist 403 (no challenge header) stays absorbed. The existing
routing then does the right thing: single-server routes turn the error into a
401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty
list so one unauthenticated server does not fail the whole listing.
On the UI tools page, an OBO (per-user authorization_code) server now shows the
Authorize gate when the list call returns 401, not only when no credential row
exists. The backend already refreshes a still-refreshable token on the list
call, so a 401 means there is no valid token and none could be minted (expired
with no usable refresh token), which is exactly when the user must reauthorize.
Add an optional per-server max_concurrent_requests that caps how many
tool calls LiteLLM sends to one MCP server at once, so batch-processing
backends are not overwhelmed by unbounded parallel dispatch. Excess calls
queue on a per-server asyncio.Semaphore instead of being rejected. Unset
or non-positive means unlimited, preserving existing behavior.
Resolves LIT-2749
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
For LITELLM_METADATA_ROUTES (responses, /v1/messages, batches, etc.),
the proxy stores admin metadata under data["litellm_metadata"] while
user-supplied metadata lives in data["metadata"]. Tags placed in
metadata.tags by the caller were never merged into litellm_metadata.tags,
causing SpendLogs.request_tags to be empty on these routes
Closes#31584
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2
A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no
client_id and no authorization URL, yet the gateway OAuth endpoints did not
check auth_type. authorize() raised "client_id is required" before the
auth_type was ever examined, and the .well-known discovery builders always
advertised authorization_servers / authorization_endpoint / token_endpoint /
registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth
flow that can never succeed.
Add an auth_type != oauth2 guard to the authorize, token, register,
protected-resource and authorization-server paths (covering the internal UI
OAuth endpoints too). The discovery guard sits after the OAuth pass-through
branch so genuine pass-through servers keep proxying their upstream metadata.
oauth2 servers are unaffected.
* fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle
Address review feedback on the auth_type gate.
The 400 message no longer claims access is governed by access groups, which is
only true for auth_type=none; it now states that the gateway runs the OAuth
client_id/authorize/token/register flow only for oauth2 servers and that the
server is reached using its configured auth_type, which is accurate for every
non-oauth2 type (api_key, oauth2_token_exchange, etc.).
The discovery gate previously 404'd a named non-oauth2 server but still returned
200 metadata for an unknown name, which both serves a broken document for a typo
and lets an unauthenticated caller enumerate non-OAuth server names by comparing
404 vs 200. A named discovery request now returns 200 only when it resolves to an
oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root
discovery and pass-through servers are unaffected.
* Apply suggestions from code review
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Route non-full-admin callers through _sanitize_mcp_server_list_for_non_admin,
matching the pattern the fetch and list handlers adopted. Replace the two
regression tests that pinned the old partial-blank behavior with a
sanitize/full-admin pair mirroring the fetch/list coverage.
Resolves LIT-3929
* fix(mcp): roll up MCP tool spend to user counters and usage UI
Direct REST MCP tool calls now fire success logging so spend_logs and
user/team rollups include configured mcp_server_cost_info charges.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate key-info enrichment to requests missing user_id; fix import order
- Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is
absent, avoiding a cache/DB lookup on every normal LLM request.
- Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure
Add api_key = ANY($2) to the MCP session aggregate query so it is
bounded by the same ownership already applied to the main page query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix spend logs for call and list mcp tools
* Add tags in mcp logging
* Fix ruff
* fix(lint): replace List/Dict with list/dict in new annotations (UP006)
Replace the 8 new UP006 violations introduced by the mcp-tags changes:
- Optional[List[str]] → Optional[list[str]] for request_tags params
- List[str] return type → list[str] in _get_parent_request_tags
- Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError
* fix(mcp): keep final streaming chunk when draining inner stream fails
* fix: handle MCP logging edge cases
* fix: propagate MCP logging cancellation
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in 94fd2bf made the no-mcp-servers sentinel return the caller's
submitted BYOM servers, which weakened an explicit key-level opt-out into a
soft preference. Restore the absolute opt-out and additionally skip the BYOM
union for keys with an explicit object_permission.mcp_servers list and for
toolset-scoped requests, mirroring how allow_all_keys servers are handled.
Add unit tests for the sentinel, explicit scoping, toolset scope, the cache
invalidation helper, the cache-miss DB path, and the db.py query helper.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
The handler returned decrypted callback environment values and alerting
routing values verbatim to callers who were not full PROXY_ADMIN. Gate
those on full-admin role, matching the posture used on the sibling
config-inspection endpoints. Non-sensitive routing fields (host / base
URL / port style values) stay visible so the UI can still label which
integration is wired up. Full PROXY_ADMIN sees everything unchanged so
the edit form round-trips on save.
Resolves LIT-4115.
The `_check_permissions_caller_permission` helper introduced in
#31469 was only wired into `_common_key_generation_helper`. This
change wires it into `_validate_update_key_data` and `regenerate_key_fn`
so the three write paths share the admin gate, and refactors the
helper to accept the full request model so it can key on
`"permissions" in data.model_fields_set` rather than truthiness. The
presence check keeps the model-level omit default flowing through
unchanged while treating any explicit value (including `{}` / `null`)
as an admin-only write.
In `regenerate_key_fn` the gate is placed before the `premium_user`
license check so the rejection is consistent across premium and
non-premium deployments. That ordering is pinned by
`test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate`
Tests in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py:
- test_update_key_non_admin_permissions_non_empty_rejected
- test_update_key_non_admin_permissions_explicit_empty_rejected
- test_update_key_non_admin_permissions_explicit_null_rejected
- test_update_key_non_admin_omits_permissions_succeeds (control)
- test_update_key_admin_can_set_permissions (control)
- test_regenerate_key_non_admin_permissions_rejected
- test_regenerate_key_non_admin_permissions_explicit_empty_rejected
- test_permissions_explicit_empty_rejected_for_non_admin_on_generate
- test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate
Mutation-killed against gate removal on either wire, against reverting
the helper to a truthiness check, and against reordering the gate past
the enterprise-license check
* fix(proxy): authorize /health/test_connection against loaded deployment's team_id (VERIA-441)
POST /health/test_connection looked up a deployment by request-supplied model_info.id, dumped its
litellm_params (including api_key) into the outbound probe, merged request params over it, and then
authorized the call against the caller-supplied model_info.team_id. A team admin could pass another
team's deployment id together with their own team_id and an attacker-controlled api_base, sending
the victim team's provider key to that URL.
Capture the loaded deployment's model_info alongside its litellm_params in both the id-lookup and
the model_name fallback paths, and pass that captured value to can_user_make_model_call. When no
deployment is loaded (caller is probing fresh, request-supplied credentials), keep using the
request body's model_info; no foreign deployment is in scope and the existing role check still
requires admin or team-admin.
Add two regression tests that wrap (not mock) ModelManagementAuthChecks.can_user_make_model_call,
one per resolution path, asserting HTTP 403 and that the auth check was reached with the loaded
deployment's team_id. Both fail on the pre-fix code.
* test(health): add positive-path regression through real auth (VERIA-441)
The two deny tests already exercise the real (wrapped) ModelManagementAuthChecks.
Add a matching positive-path test so a mutation that swaps the auth team_id for
a deny-all value on the legit path also fails: loaded deployment owned by team-X,
caller admin of team-X -> asserts HTTP 200 and that the auth check ran with the
LOADED deployment's team_id.
* refactor(test): rename health endpoint tests for clarity (VERIA-441)
Rename test functions and variables from attacker/victim/owner framing to
neutral team-a/team-b terminology. Update docstrings to remove exploit-specific
language. Tests remain functionally identical, covering deny paths (cross-team
deployments) and the positive path (same-team deployments).
MCP servers created through the UI are persisted to the database independent of
store_model_in_db, but the in-memory registry that GET /v1/mcp/server reads was
hydrated from the database only through add_deployment, which runs solely when
store_model_in_db is True. On a DB-backed single-instance proxy with
store_model_in_db unset the registry started empty after a restart, so the MCP
Servers page showed nothing until an add or edit triggered a reload.
Hydrate the registry from the database on startup regardless of store_model_in_db
via a new ProxyConfig.init_mcp_servers_from_db, honoring supported_db_objects.
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
* feat(proxy): audit remaining system-wide settings updates
Extends the audit logging framework introduced in the parent PR to the
rest of the LiteLLM_Config writers and the two adjacent settings tables:
/config/update (general, environment_variables, litellm_settings,
router_settings sections), /config/field/update, /config/field/delete,
/config/callback/delete, /update/default_team_settings,
/update/mcp_semantic_filter_settings, /add/allowed_ip,
/delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings,
/update/ui_settings.
Each writer records the actor, action, the affected config section, and
a redacted before/after snapshot. SSO and UI settings rows use their own
table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback
and /update/sso_settings audits fire BEFORE the proxy reload and the env
cleanup step respectively, so a failure in either leaves the audit row
intact.
The audit-actor parameter on _update_litellm_setting is now required
rather than optional; the chokepoint covers default_team and
mcp_semantic_filter for free, and a future caller that forgets the
actor fails loudly instead of silently skipping the audit. The two
direct-calling tests pass a dummy actor.
The environment_variables section redacts every value rather than
relying on key-name matching, because it carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
* fix(proxy): capture redacted SSO before-snapshot in audit log
Greptile review of #31754 flagged update_sso_settings as the one endpoint
where before_value is permanently None, so the LiteLLM_SSOConfig audit
trail has no pre-change state. An auditor reviewing a secret-rotation
event could see what the SSO settings were changed to but not what they
were before.
Read the existing SSO row before the upsert, decrypt it via
proxy_config._decrypt_db_variables, and pass it as before_value.
create_config_audit_log's secret-name redaction then masks the
*_client_secret fields, so neither the old nor the new plaintext secret
lands in the audit row.
Add a regression test asserting the before-snapshot reflects the
pre-change values for non-secret fields (google_client_id) and is
redacted for secret fields (google_client_secret). Mutation-checked
against reverting to before_value=None.
The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique
since the endpoint reads it; the read returns None for tests that do not
care about the before-state.
* fix: remove committed zero init migration
* refactor(proxy): audit config writes via asyncio.create_task everywhere
PR A's chokepoint audit call was refactored from a blocking await to
asyncio.create_task so that a post-save audit-log failure could not
surface as a 500 to the caller. The 12 other audit call sites added in
this PR were still using await, reintroducing the exact 500-after-commit
exposure at every sibling endpoint. Wrap them all in asyncio.create_task
to match the model_management_endpoints / key_management_endpoints /
hooks / config_override_endpoints / team_callback_endpoints /
cache_settings_endpoints house pattern, so the codebase tells one story.
The two direct-invocation tests (test_update_config_general_settings and
test_delete_config_general_settings, which call the handler in-process
rather than via TestClient) yield with `await asyncio.sleep(0)` after the
handler returns so the scheduled audit task runs before the assertion.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(guardrails/headroom): add CCR (compress-cache-retrieve) support via agentic loop
When Headroom's /v1/compress returns messages containing hash markers
(hash=[a-f0-9]{24}), inject a headroom_retrieve tool into the request.
When the LLM calls that tool, intercept via async_should_run_agentic_loop
and async_build_agentic_loop_plan, call GET /v1/retrieve/{hash} on the
Headroom sidecar, and replay the LLM with the original content as a tool
result -- all transparent to the caller.
* style: run ruff format on headroom guardrail and tests
* fix(guardrails/headroom): detect headroom_retrieve calls in both OpenAI and Anthropic response formats
* test(guardrails/headroom): add test for Anthropic content block format detection in CCR loop
* ci: trigger CI checks
* fix(guardrails/headroom): replace List/Dict with list/dict to fix UP006 ruff violations
* fix(guardrails/headroom): replace except Exception with except ValueError to fix BLE001
* fix(guardrails/headroom): add Responses API output format detection for CCR tool calls
* refactor(guardrails/headroom): extract format-specific helpers to fix C901 complexity
* fix(guardrails/headroom): scope CCR retrieval to hashes produced by current request
Previously any LLM-supplied hash in a headroom_retrieve tool call was
forwarded to the Headroom retrieve API, letting a crafted tool call
fetch arbitrary cached content. Validate the hash against the set
produced by compressing the current request's messages before calling
retrieve.
* fix(guardrails/headroom): track issued hashes server-side, fix Responses API replay shape
Hash validation now also checks an in-memory cache of hashes actually
returned by /v1/compress, not just whether the hash text appears
somewhere in the request's messages. The message-text check alone is
forgeable: an attacker can plant a hash-shaped string in their own
prompt and have it treated as valid.
Responses API follow-up now emits function_call/function_call_output
items keyed by call_id instead of chat-style assistant/tool messages,
since the Responses API does not accept the latter as input. Also
fixes call_id/id field priority when extracting tool calls from
Responses API output, since call_id (not id) is what must match
between the function_call and its output.
* fix(guardrails/headroom): drop redundant quoted type annotations
UP037 flags quotes on annotations that are already lazily evaluated
via `from __future__ import annotations`.
* test(guardrails/headroom): add missing pytest.mark.asyncio decorators
Functional under asyncio_mode=auto, but every other async test in the
file has the decorator for consistency.
* fix(guardrails/headroom): scope CCR hashes per call_id, fix Anthropic replay shape
Two real gaps found in review:
1. The instance-wide issued-hash cache combined with a message-text
check did not actually scope retrieval to the request that produced
the hash. A hash issued for request A stays in the shared cache
until TTL expiry, and the message-text check is satisfied by any
request whose own messages happen to echo that hash string. Request
B could plant A's hash in its own prompt and retrieve A's content.
Fixed by keying the issued-hash cache by litellm_call_id, matching
the pattern already used in compression_interception: a hash is
only honored when it was issued under the exact call_id resolving
for the current request.
2. The Anthropic Messages replay path fell through to the chat-style
assistant/tool-message builder, which Anthropic does not accept.
Anthropic requires the tool_use block echoed in an assistant message
paired with a tool_result block in a user message, keyed by
tool_use_id. Added a dedicated branch for this shape.
* docs: note proactive API-fragmentation helper convention
Add a bullet to the coding-conventions list: look for or add a shared
helper when logic branches on API surface (chat completions vs
Anthropic Messages vs Responses API), instead of duplicating
format-detection per module.
* fix(guardrails/headroom): fix Anthropic tool-shape detection, extract shared cross-API tool util
Live e2e testing against the real Anthropic API surfaced two bugs the
mocked unit tests couldn't catch because they used MagicMock responses
instead of realistic response shapes:
1. has_headroom_retrieve_tool only recognized OpenAI-shaped function
tools. By the time an Anthropic Messages response reaches the
agentic-loop gate, the tool this guardrail injected has already been
transformed into Anthropic's native shape (type: "custom", top-level
"name"), so the gate never fired for real Anthropic requests.
2. AnthropicMessagesResponse is a TypedDict, so real responses are
plain dicts at runtime, not objects with attribute access. The
extractors and format detectors used bare getattr(), which silently
returns nothing for dict responses instead of reading the actual
key.
Extracted the cross-API-surface tool-call extraction and tool-presence
check into litellm/litellm_core_utils/prompt_templates/factory.py
(get_tool_calls_from_response, has_tool_with_name) so this format
fragmentation is handled in one place instead of being duplicated
per-guardrail, and reused the existing repair-aware
parse_tool_call_arguments from common_utils instead of a naive
json.loads. headroom.py now delegates to these shared helpers.
Confirmed live against the real Anthropic API: the retrieve loop now
fires and successfully retrieves the correct hash's content through
the full compress -> tool-call -> retrieve -> replay round-trip.
* fix(guardrails/headroom): fix ruff-strict UP006/I001 budget violations
Use lowercase list/dict generics in the new factory.py tool-call
helpers instead of typing.List/Dict, drop the now-unused Tuple import
in headroom.py, and reorder the new factory import ahead of the
llms.custom_httpx import to satisfy import sorting.
* fix(guardrails/headroom): match Anthropic tools without a type field
Anthropic's documented client tool format is just name + input_schema;
type: "custom" is only one possible value, not a requirement. Match
any non-OpenAI-shaped tool on its top-level name instead of requiring
type == "custom".
* feat(proxy): support object_permission in default_key_generate_params
default_key_generate_params filled in a fixed whitelist of scalar fields
plus a full-replace for models/metadata, but never touched object_permission,
so admins had no way to set a default (e.g. mcp_tool_search_enabled,
vector_stores) applied to every new key. Merge object_permission field-by-field
instead of replacing it wholesale, so a caller-supplied field (e.g. mcp_servers)
is preserved alongside defaulted fields the caller left unset.
* ci: retrigger proxy_pass_through_endpoint_tests (suspected flake, unrelated to this PR's diff)
* fix(proxy): apply default object_permission after team-scope validation
Injecting the default before validate_key_vector_stores_against_team /
validate_key_search_tools_against_team ran meant a default containing a
team-scoped field (e.g. vector_stores) looked like a caller-requested
permission, turning ordinary non-admin personal key creation into a 403.
Merge the default into data_json after those checks instead, and guard
against a non-dict default value.
* feat(proxy): audit default user settings updates
Adds audit logging for the customer-impacting path: PATCH
/update/internal_user_settings, which is what the admin dashboard hits
when an admin changes Default User Settings and which today leaves no
record of who changed what.
Introduces the small framework that future system-wide settings audits
will share: a CONFIG_TABLE_NAME enum value, a create_config_audit_log
helper that reuses the existing create_object_audit_log path (so the
enterprise gate and store_audit_logs flag still apply), and a
_dump_redacted_config helper that strips secret leaves before the row is
written using the same matcher /config/field/info applies for non-admins.
The helper handles environment_variables as a special case where every
value is redacted, since that section carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
Only update_internal_user_settings is wired up in this change. Coverage
for the other LiteLLM_Config writers (/config/update sections,
/config/field/update, /config/field/delete, /config/callback/delete,
default_team_settings, mcp_semantic_filter, allowed_ip, sso_settings,
ui_theme, ui_settings) is intentionally a follow-up so each can be
verified live against the credential-bearing fields it actually carries.
The audit-actor parameter on _update_litellm_setting is optional today so
non-audited callers keep working unchanged; the follow-up will make it
required once every caller is wired up.
* fix(proxy): make audit-log call non-blocking and serializer defensive
Greptile review of #31753 surfaced three robustness issues with the
audit-log call path. The settings change always commits; these fixes
prevent post-commit audit failures from surfacing as 500 responses.
Switch the audit-log call in _update_litellm_setting from a blocking
await to asyncio.create_task, matching the create_object_audit_log
pattern every other call site uses (model_management_endpoints etc.).
A transient prisma blip or a JSON serialization error in the audit row
no longer turns a successful save_config into a 500 the caller sees.
Add default=str to both json.dumps calls in _dump_redacted_config so a
YAML-loaded value with a non-JSON-native leaf (datetime, custom object)
serializes cleanly. The sibling audit-log serializers in
team_endpoints.py already pass default=str for the same reason.
Tighten the redact_all_values branch to redact wholesale for non-dict
inputs rather than silently falling through to the key-name matcher;
defensive against a future change that stores a section as a list or
scalar.
Each fix has a regression test mutation-checked against reverting the
fix.
* refactor(proxy): drop unreachable non-dict redact_all_values branch
The defensive non-dict fallback in _dump_redacted_config emitted
json.dumps("REDACTED") which, if ever hit, would crash LiteLLM_AuditLogs
construction (mask_api_keys validator calls json.loads on the already-
parsed bare string). Reachability is zero: redact_all_values is True
only for param_name=="environment_variables", which is always a dict.
Delete the dead branch and its test rather than ship provably-wrong
defensive code with a test that green-lights it.
Two NVIDIA-Riva-specific fields consumed by the audio-transcription
handler via the provider's `optional_params` passthrough were not
covered by the proxy's existing banned-request-body list or the
admin-config clearing list applied on `api_base` BYOK override:
* `nvcf_function_id`
* `use_ssl`
Add both to `_BANNED_REQUEST_BODY_PARAMS` in
`litellm/proxy/auth/auth_utils.py` and to the kwargs-only list in
`_admin_config_fields_to_clear_on_base_override()` in
`litellm/router_utils/clientside_credential_handler.py`, next to the
analogous provider-specific entries already there (`aws_bedrock_*`,
OCI provider fields, etc.). Same admin opt-ins as every other entry
on those lists (`general_settings.allow_client_side_credentials`
proxy-wide, or `configurable_clientside_auth_params` per deployment).
Regression tests in `tests/test_litellm/proxy/auth/test_auth_utils.py`
cover root-level rejection, the historical `api_key` bypass, both
admin opt-in paths (proxy-wide and per-deployment), nested-container
smuggling via the existing recursive walk, and clearing on
`api_base` override. Mutation check verified.
Resolves VERIA-493
The Presidio streaming post-call hooks (_stream_apply_output_masking for
apply_to_output and _stream_pii_unmasking for output_parse_pii) collected every
upstream chunk, reassembled the full completion with stream_chunk_builder at
end-of-stream, ran Presidio over it, then emitted one reconstructed SSE chunk.
Time-to-first-token collapsed to the total generation time and token-by-token
streaming was lost whenever Presidio output handling was enabled. With the
default presidio_filter_scope both, an apply_to_output masking instance is always
created, so even the unmask configuration buffered the stream.
Both paths now transform and forward chunks as they arrive. The unmask path
replaces placeholder tokens per chunk, holding back only the trailing run that
could still grow into a token so a placeholder split across SSE chunks
(<PER + SON_1>) is still rewritten atomically. The mask path emits a prefix only
when masking it in isolation matches the corresponding prefix of masking the
whole buffer, with a lookahead margin still buffered past the cut, so an entity
straddling the cut is detected and held until complete; past
_PRESIDIO_STREAM_MAX_BUFFER the run is bounded without splitting an entity.
Tool-call and legacy function-call argument fragments are accumulated per choice
and transformed once the choice closes, content is buffered independently per
choice index for correct n>1 streaming, raw Anthropic SSE bytes and /v1/responses
events pass through with any held content flushed first so events never reorder,
and a masking error redacts only the affected chunk (fail closed, keeping
finish_reason) while the stream continues.
Resolves LIT-3222
update_spend_logs flushes the queue with a single create_many per batch, so one
row carrying bytes Postgres refuses (a residual NUL byte is the canonical case)
fails the entire insert and drops every good spend log alongside it. PR #29515
strips NUL bytes from the JSON columns, but the scalar string columns (end_user,
model, session_id, ...) still flow through unsanitized, so a poisoned row can
still reach the write and take a batch of up to 1000 good rows down with it.
On a genuine data-layer rejection the batch is now bisected so the good rows
still persist and only the offending row is dropped and logged with its
request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error
(matched by exact type so systemic subclasses like a missing table are not
mistaken for a single poison row), which keeps prisma an in-function import and
litellm.proxy.utils importable without the proxy extra. Transport failures,
including the "can't reach database server" outage that prisma mislabels as a
DataError, are re-raised unchanged so the existing connection-retry path still
runs and a transient outage never turns into silent per-row data loss.
The bisection carries a per-batch isolation budget so an authenticated caller
flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed
inserts and N log lines; once the budget is spent the still-failing remainder
is dropped wholesale under a single log line.
Resolves LIT-4103
* fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list
On the aggregate MCP route (/mcp), the gateway fans out to every server the caller can access and
flattens their tools. _fetch_and_filter_server_tools re-raises MCPUpstreamAuthError unconditionally
(added with the OAuth passthrough feature in #28356) so it surfaces a 401 on single-server routes,
but on the aggregate route that exception propagates through the asyncio.gather fan-out and the
outer handler turns it into an empty list. The result: a single delegate/passthrough OAuth server
the user has not authenticated (e.g. a delegate-auth server) zeroes the tools of every other server,
including the ones that resolve fine, so the client connects and sees no tools.
Surface the upstream auth error only when a single server was explicitly targeted (so that route
still drives the upstream OAuth flow); across the aggregate, absorb it to [] for that one server so
the rest still list their tools. This restores the graceful per-server degradation that predated
#28356.
Adds regression tests: the aggregate keeps a healthy server's tools when a sibling raises
MCPUpstreamAuthError, and a single-server listing still surfaces it.
* fix(mcp): decide aggregate vs single-server listing by route scope, not server count
Addresses review: keying the surface-vs-absorb decision off the server count (len(allowed_mcp_servers),
and even len(mcp_servers)) misclassifies an aggregate /mcp request from a key that can access exactly
one server as a targeted single-server listing, so that one server's MCPUpstreamAuthError re-raises and
empties the aggregate again for one-server permission sets.
Use the path-derived single-server scope instead: _mcp_gateway_server_name, set by
_gateway_initialize_instructions_request_scope only when the request path names exactly one upstream
server (/<server>/mcp) and never from client headers, is None on the aggregate route (/mcp) regardless
of how many servers the key can access. Single-server routes still surface the upstream-auth challenge;
the aggregate absorbs it per server.
Adds a regression test that an aggregate request with a single accessible server still absorbs, plus
renames the single-server test to drive the route scope explicitly. The new test fails on the
count-based logic.
* fixing aggregation error
* style(mcp): collapse single-line debug log to satisfy ruff format
* feat(guardrails): expose streaming knobs on generic_guardrail_api
Wire streaming_end_of_stream_only and streaming_sampling_rate through
optional params, initialize_guardrail, and get_config_model so the
generic guardrail API participates in UnifiedLLMGuardrails streaming
checks with configurable cadence and end-of-stream-only mode.
* fix(guardrails): use builtin type[] in get_config_model return
Avoids a new UP006 violation that tripped the ruff strict-rule budget
gate on the PR lint job.
* fix(guardrails): default optional streaming knobs to None
Non-None Pydantic defaults on GenericGuardrailAPIOptionalParams made
_get_config_value treat unset nested fields as explicit values, which
shadowed top-level litellm_params streaming flags whenever any other
optional_params key was present. Real defaults stay in the constructor.
* fix(guardrails): address review nits on generic_guardrail_api streaming
Validate streaming_sampling_rate >= 1 in the constructor and Pydantic
optional_params (ge=1), and add /v1/responses streaming coverage through
the unified post-call hook so Responses API usage is exercised alongside
chat completions.
* fix(guardrails): read nested streaming config from dict optional_params
Guardrail API/UI delivers optional_params as a plain dict, so getattr was
silently ignoring streaming_sampling_rate and streaming_end_of_stream_only.
Handle both dict and model shapes in _get_config_value with regression tests.
* fix(guardrails): clear ruff findings in generic_guardrail_api tests/types
* style(guardrails): ruff format generic_guardrail_api modules
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
* feat(proxy): type Customer Management response_model for OpenAPI coverage
Add response_model to the five remaining untyped /customer operations
(block, unblock, new, update, delete) so the generated OpenAPI schema
documents a concrete response body. new/update reuse the canonical
LiteLLM_EndUserTable (matching info/list); block, unblock, and delete
get small dedicated models in
litellm/types/proxy/management_endpoints/customer_endpoints.py.
Together with the already-typed info/list/daily-activity routes this
brings the Customer Management group to full response_model coverage.
Regression tests assert each public /customer/* route declares the
expected response_model and that /customer/new surfaces a typed schema
in app.openapi(), so dropping a response_model fails CI.
* fix(proxy): keep budget_id in typed customer responses
Address review feedback on the Customer Management response_model typing.
Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new
and /customer/update silently drops fields the raw Prisma model_dump()
echoed. Checking the schema, budget_id is the only such scalar column that
was missing from the Pydantic model (created_at/updated_at/tpm_limit do not
exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable.
This restores budget_id on new/update and also fixes the pre-existing gap
where /customer/info and /customer/list (already typed) dropped it, which
the UI Customer type expects. A regression test pins budget_id surviving the
response_model filter on /customer/update.
Also document UnblockUsersResponse.blocked_users via a Field description: it
holds the users that remain blocked after the call. The key name predates
this PR and is kept to avoid a backwards-incompatible rename on a beta route.
* fix(proxy): keep nested budget fields in customer responses
response_model=LiteLLM_EndUserTable nests the budget as the narrow write
allowlist LiteLLM_BudgetTable, which silently drops the server-managed
fields the customer endpoints used to return (budget_reset_at, created_at).
Introduce CustomerResponse, a thin response model that nests
LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on
/customer/new, /customer/update, /customer/info and /customer/list. list
also builds CustomerResponse so its budget isn't narrowed at construction
time. created_by/updated_at/updated_by remain omitted, matching how budgets
are returned elsewhere.
The shared LiteLLM_EndUserTable is left untouched: it's constructed in many
places that pass narrow budget instances, and pydantic v2 won't coerce a
budget instance into a wider nested model. Typing only at the response
boundary (where the handler hands FastAPI a dict) sidesteps that. A
regression test pins budget_reset_at + created_at through the filter and
asserts the internal audit fields stay out.
* test(proxy): add golden-master characterization tests for customer responses
Lock the exact JSON body each customer-object endpoint (info/list/new/update)
and delete return today, so the upcoming type-safety refactor of the handlers
is only allowed to land if it reproduces these byte for byte. Pins null-field
inclusion, the nested budget shape (server fields kept, audit fields dropped),
and object_permission reverse-relation stripping. Green against current code.
* refactor(proxy): make the customer response flow type-safe
Replace the untyped dict + bolt-on response_model pattern on the customer
object endpoints with explicit typed construction. A single mapper,
_to_customer_response, validates a DB row into CustomerResponse at one
Any -> typed seam; new/update/info/list now return it (or a list of it) and
carry real -> CustomerResponse / -> List[CustomerResponse] return
annotations, and delete returns DeleteCustomersResponse. basedpyright now
verifies the handlers' return shapes instead of a runtime filter doing it
silently.
This also deletes the four copy-pasted object_permission reverse-relation
cleanup loops: pydantic's extra=ignore drops those undeclared fields during
validation, so the loops were dead code (proven by the golden-master tests,
which stay byte-for-byte green). basedpyright errors on the file drop from
140 to 116, all from removed dict plumbing.
CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits
the existing validators/config unchanged (behavior preservation); only the
nested budget type is widened.
* refactor(proxy): annotate customer response mapper param as BaseModel
Address review nit: the mapper's untyped `record` added an ANN001 violation.
The incoming rows are pydantic v2 models, so type the param as BaseModel
rather than object (object has no model_dump, which would just move the
problem to basedpyright). This clears the ANN001 and also drops three
basedpyright unknown-type violations the untyped param was adding.
* style(test): ruff format customer endpoint tests
* test(proxy): give customer budget test update mocks a valid model_dump
The type-safe response refactor validates the update result via
_to_customer_response (CustomerResponse.model_validate(record.model_dump())).
These budget tests mocked the end-user update to return a bare MagicMock,
so model_dump() yielded a MagicMock that fails validation. Give each update
mock a minimal valid dict; the tests assert on the prisma calls, not the body.
* chore(ui): regenerate API types from proxy OpenAPI spec
* fix(ui): make generated API types stable across Python versions
Python 3.13 strips a docstring's common leading indentation at compile
time while 3.12 keeps it, so app.openapi() emits differently-indented
description strings depending on the interpreter. The dashboard type
generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted
and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed
Normalize every description through inspect.cleandoc in the spec dump so
the output is identical regardless of interpreter, then regenerate
* chore: remove _experimental/out
* fix(ci): recreate _experimental/out before copying UI build output
The build scripts cp the Next.js output into litellm/proxy/_experimental/out,
which was removed from git. cp failed because the target directory no longer
existed; mkdir -p recreates it before the copy.
* fix(proxy): make UI serving resilient to a missing _experimental/out
Removing the committed UI export means the source/test tree no longer
ships litellm/proxy/_experimental/out. Three things assumed it was always
present and broke once it was gone:
- get_favicon hard-coded the built favicon path and 404'd without it; it
now falls back to the bundled swagger/favicon.ico
- the /_next and /ui static mounts raised at construction when the export
was absent, so the whole UI-setup block was swallowed and no mounts
registered; they now use check_dir=False
- _restructure_ui_html_files was a nested function only exposed as a
module attribute when that block happened to succeed; it is now a real
module-level function
test_admin_ui_export_serves_nested_extensionless_routes validated the
committed artifact, whose premise this PR removes; it now drives the same
MCP OAuth callback restructure guarantee through a synthetic export.
* chore(greptile): ignore generated _experimental/out so review fits the file limit
* Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit"
ignorePatterns is applied after Greptile counts the files changed, so it
does not bring the diff under the file limit; the config had no effect.
The MCP gateway authenticated to upstream OAuth token endpoints only with
client_secret_post (client_secret placed in the POST body). Providers that
require HTTP Basic client authentication (client_secret_basic, the OIDC
default) reject that with invalid_client, which surfaced as a 500 on the
/<server>/token exchange and broke both the initial authorization_code
exchange and refresh.
Add a per-server token_endpoint_auth_method ("client_secret_basic" |
"client_secret_post") and a single helper that builds the right headers and
body for the configured method, then route every upstream token-endpoint POST
through it: the inbound exchange and refresh in discoverable_endpoints, the v1
per-user refresh in db, the v2 authorization_code refresher, the M2M
client_credentials fetch, and the RFC 8693 token exchange. The default stays
client_secret_post so existing servers are unaffected; basic sends
Authorization: Basic base64(form-urlencode(client_id):form-urlencode(secret))
per RFC 6749 section 2.3.1 and omits the secret from the body.
client_secret_basic is a confidential-client method, so a server configured for
it with a missing client_id/secret raises rather than silently downgrading to a
body request (no-silent-fallback); the inbound endpoint maps that to a 400 and
the refresh paths to a failed-refresh / needs-reauth. A secretless client_id
under the default method stays valid for public clients authenticating with PKCE.
Resolves LIT-4091
* fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076)
The Anthropic /v1/messages and Google native :generateContent routes return
TypedDict results (AnthropicMessagesResponse, GenerateContentResponseBody) that
are plain dicts at runtime and cannot hold a _hidden_params attribute. The cost
is computed by update_response_metadata, but ResponseMetadata.apply() only
persists _hidden_params back when the result object has that attribute, so for
those two routes the computed response_cost was dropped. The non-streaming
header build in base_process_llm_request then saw an empty response_cost and
get_custom_headers filtered the x-litellm-response-cost header out, even though
the other x-litellm-* headers still appeared.
The non-streaming success path now recovers the cost from the logging object
when the response cannot carry _hidden_params, preferring the value already
stored in model_call_details and recomputing from the same calculator only when
it has not been stored yet. Object responses (ModelResponse, ResponsesAPIResponse)
keep their existing behavior, so chat/completions, /responses, and the Anthropic
error path that intentionally emits a zero cost are unaffected. Streaming stays
out of scope because the header is emitted at stream start, before the cost is
known.
* fix(proxy): also recover response cost header for /generateContent responses with _hidden_params (LIT-4076)
* fix(proxy): compute generateContent response cost synchronously so cost header is emitted (LIT-4076)
* fix(lint): suppress BLE001 on generate_content cost normalization guard
The defensive blind except keeps cost normalization from ever breaking the
response path; mark it noqa so it does not breach the strict-rule budget.
* fix: skip health check for semantic auto_router deployments
auto_router/<name> deployments are semantic meta-routers that select among
real LLM deployments at request time. They have no LLM endpoint to probe.
The health check was passing model=auto_router/router_1 to get_llm_provider(),
which raised BadRequestError: "Unmapped LLM provider for this endpoint" because
auto_router is not a real LLM provider, causing these deployments to always
appear unhealthy and curl requests to hang.
Detect semantic auto_router deployments in _run_model_health_check and return
{} (healthy) without calling litellm.ahealth_check. Sub-strategies
(complexity_router, adaptive_router, quality_router) are excluded from this
fast path and continue to be health-checked normally.
* ci: trigger circleci
The Model Armor guardrail only sent text extracted from user messages to
sanitizeUserPrompt, so harmful content inside attached PDFs, Office docs,
and CSVs reached the LLM unscanned. A file-only message had no extractable
text, so the pre-call and moderation hooks returned early and the document
was never submitted to Model Armor at all.
Wire inline document/file scanning into async_pre_call_hook and
async_moderation_hook. extract_file_attachments walks message content blocks
(OpenAI type:file file_data and Anthropic type:document source), decodes the
base64 bytes, maps the MIME type to a Model Armor byteDataType, and skips
remote URLs, bare file_id references, oversize files past the 4 MB limit, and
unsupported types. Each attachment is sent through the byte API and a
MATCH_FOUND blocks the request before it reaches the LLM.
Resolves LIT-4084
* fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint
The OAuth token endpoint stored a user's per-server token under the identity
returned by _extract_user_id_from_request, which read only the Authorization
header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup
with no model_type rehydration and no DB fallback. That silently returned None
in two common cases on a multi-replica gateway: the LiteLLM key arrives on
x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code
send) rather than Authorization, and a cross-replica cache hit deserializes to
a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When
it returned None the token was not persisted.
This was survivable until the authorization_code v2 migration began stripping
the caller's Authorization for migrated per-user OAuth servers and routing the
preemptive 401 existence check through the stored token, so a persist miss now
hard-fails: the egress challenges with 401 on every reconnect (the client sees
"rejected them on reconnect" or a successful connect with zero tools).
Resolve identity through get_key_object, the canonical resolver that reads the
cache with model_type and falls back to the DB, and accept the key from
x-litellm-api-key as well as Authorization. The silent persist skip is now a
warning. The caller-Authorization stripping stays as is, since reinstating it
would reopen the cross-user credential override it was added to prevent.
* fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity
The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the
blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So
a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with
that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token
for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys
(blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed
that accidental shield.
Validate the resolved key before trusting its identity: return None when blocked or expired, so the
upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row.
Regression tests cover the blocked and expired cases and fail without the guard.
* fix(proxy): count only active users toward license seat limit
SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat
Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected
* fix(proxy): floor billable user count at zero
count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call
Addresses Greptile P1 on the PR
* refactor(proxy): count teams via TeamRepository in available_users
* style: ruff format changed files at line-length 120
Mirror the scalar `max_budget` guard in `_common_key_generation_helper`
for the per-window check: a CLI session token caller (carrying
`max_budget=None`) cannot set `budget_limits` on a personal key. Pass
`team_table` into the helper so it can detect the personal-key shape;
reject before the `delegation_ceiling is None` early return.
Four new regression tests cover the personal-key reject, the team-key
happy path, the team-key over-team-budget path, and the proxy-admin
exemption.
Enforce that every `budget_limits[*].max_budget` is a finite number;
applies to every caller including proxy admin and runs before the
role / ceiling checks. Six parametrized regression tests cover NaN /
+inf / -inf for both non-admin and admin callers.
increment_spend_counters was parallelized in #31578, but the dominant
per-request cost under high concurrency is the pre-call budget enforcement
in common_checks, which still ran a Redis-first get_current_spend per scope
(team, team windows, key windows, org, tag, user, team member, end user)
one sequential await after another inside the auth span.
The per-scope reads target distinct counter keys with no cross-scope
ordering dependency, so they now run concurrently under asyncio.gather.
Key metadata.tags injection still runs before the gather so the tag budget
check sees it, and every scope settles before the first error in
scope-priority order propagates, preserving the previous rejection semantics.
Resolves LIT-4090
/key/generate validated the caller's delegation ceiling against
data.max_budget only. The per-window entries in data.budget_limits
bypassed the check, so a non-admin caller could mint a key whose
1-day window vastly exceeded their own max_budget. The data.permissions
dict also went unvalidated for non-admin callers, so they could
self-grant capabilities like allow_pii_controls (and on Enterprise,
get_spend_routes).
Both gates now live in _common_key_generation_helper, covering
/key/generate and /key/service-account/generate. The existing empty
{} default on permissions still passes for non-admin callers.
