* fix(mcp): surface upstream 401 for token-forwarding MCP servers
For MCP servers configured with extra_headers: [Authorization], the gateway
forwards the client token directly to the upstream. When that token is rejected
(expired or invalid) the upstream returns 401, but the MCP SDK starts the SSE
stream with 200 OK before calling handlers, so the 401 can't be returned
mid-stream.
Fix: add a pre-flight httpx probe in handle_streamable_http_mcp — before the
SDK opens the session — so the gateway can still return HTTP 401 with
WWW-Authenticate: Bearer authorization_uri=<gateway-discovery-url> when the
upstream rejects the token. The probe fails-open (returns 200) on network
errors so a transient hiccup does not block valid requests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): parallelize pre-flight auth probes and use HEAD to avoid side effects
- Extract forwarded_auth outside the pass-through server loop (was called N times for the same scope value)
- Gather all upstream auth probes concurrently with asyncio.gather instead of sequentially; eliminates N×5 s worst-case latency
- Switch probe from POST+initialize JSON-RPC body to HEAD request; HEAD carries the Authorization header so the upstream rejects invalid tokens with 401 but never allocates a session or writes an audit entry
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): use get_async_httpx_client in _probe_upstream_auth
Replaces bare httpx.AsyncClient with the project-standard
get_async_httpx_client(httpxSpecialProvider.MCP) to satisfy the
ensure_async_clients_test code coverage check and avoid the +500 ms
per-request overhead of creating a new client on every probe call.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): extract pre-flight probe into _check_passthrough_upstream_auth
Moves the parallel upstream auth probe logic out of
handle_streamable_http_mcp into a dedicated helper to satisfy
Ruff PLR0915 (Too many statements > 50).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate pre-flight probes on authorized server set to prevent bypass
_check_passthrough_upstream_auth was resolving user-supplied server names
directly before authorization ran, letting any permitted LiteLLM key
trigger an upstream HEAD probe to a server it was not allowed to use.
Changes:
- Call _get_allowed_mcp_servers inside the helper so only servers the
caller's key is authorized for are probed.
- Move the call site to after toolset scoping so the auth context is
fully resolved before the probe list is built.
- Thread user_api_key_auth into the helper signature (replaces the raw
mcp_servers name list).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add async HTTP HEAD support
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): use Scope type annotation in _get_forwarded_auth_from_scope
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix MCP upstream auth probe method
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove unused AsyncHTTPHandler head method
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): exclude has_client_credentials servers from pre-flight auth probe
_prepare_mcp_server_headers skips caller Authorization when the server
uses OAuth client-credentials (M2M), but the pre-flight probe was still
selecting those servers and forwarding the caller's raw token in the HEAD
request. Exclude servers with has_client_credentials from the probe list
to match the actual downstream header-preparation logic.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): propagate upstream 403 as 403, not 401 with WWW-Authenticate
Per RFC 9110, 401 means "go get new credentials." Mapping an upstream 403
to a gateway 401 causes OAuth clients to restart the authorization flow,
obtain a fresh token with identical scopes, hit 403 again, and loop
indefinitely.
401 from upstream → gateway 401 + WWW-Authenticate (re-authorize)
403 from upstream → gateway 403 (no WWW-Authenticate hint)
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): skip auth probe when Authorization may be the LiteLLM proxy key
The pre-flight upstream probe must not forward the caller's Authorization
header when it could itself be the LiteLLM proxy API key. Restrict the
probe to requests that supply x-litellm-api-key explicitly — only then is
the Authorization header unambiguously the upstream OAuth token the
caller wants forwarded.
* Fix MCP ASGI HTTPException propagation
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): use public AsyncHTTPHandler.post() in auth probe
Use AsyncHTTPHandler.post() and catch httpx.HTTPStatusError explicitly so
the 401/403 we want to surface is not silently swallowed by the broad
fail-open except Exception block. Avoids reaching into the handler's
private client attribute, which would silently regress to fail-open if
AsyncHTTPHandler is ever refactored.
* Fix MCP auth probe tests
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(mcp): add coverage for httpx.HTTPStatusError path in auth probe
AsyncHTTPHandler.post() calls raise_for_status() internally, so a real
upstream 401/403 lands as httpx.HTTPStatusError. Add a test that exercises
that specific exception path so a regression that swallows the error in
the broad fail-open except Exception would be caught.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: claude-bot <claude-bot@anthropic.com>
* feat(proxy): skip disable_background_health_check models on GET /health when flag set
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix comment
* fix greptile comments
* Fix health check fallback kwargs
* Format health endpoint
* Harden direct health check kwargs compatibility for monkeypatched perform_health_check
Replace substring-based TypeError detection with unexpected-keyword checks
and a short retry chain (full kwargs, instrumentation only, filter only,
minimal) so partial stubs work regardless of which optional kwarg fails first.
Add proxy unit tests for legacy three-arg stubs and single-kwarg variants.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix black
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
LazyFeatureMiddleware compared the raw scope path against registered
prefixes (e.g. /policies), so requests under a server root path like
/api/v1/policies/... never matched, the feature never loaded, and the
endpoint returned 404. Strip the configured root path before matching,
normalizing trailing slashes and enforcing a component boundary so
/api does not falsely match /apiv2.
`apply_client_tag_policy_pre_auth` overwrote string-typed metadata
with `{}` before merging header tags, dropping any tags inside. A
caller could send `metadata='{"tags":["over-budget"]}'` plus
`x-litellm-tags: within-budget` and bypass `_tag_max_budget_check`
on the body tag. Parse the string via `safe_json_loads` first so
existing tags survive the merge.
Also drop the empty `tests/test_litellm/proxy/credential_endpoints/`
directory — the cascade-rename tests it held imported a function
that was never implemented (out of scope for this PR).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``model_dump(exclude_unset=True)`` in ``prepare_key_update_data``
includes any field the caller explicitly set, even when the value is
``None``. The previous guard short-circuited on ``getattr(data,
'user_id', None) is None``, which conflated "field omitted" (safe)
with "field explicitly set to null" (writes NULL to the token row,
detaching the key from its user and bypassing user-row role
checks).
Switch the omitted-vs-set distinction to ``data.model_fields_set``;
treat explicit-null and explicit-empty-string identically as a
removal attempt, both 403-rejected for non-admin callers.
Parametrized regression adds ``explicit_null_blocked`` alongside the
existing ``rebind_blocked`` / ``empty_blocked`` / ``same_user_id_allowed``
cases.
Removes the allow_client_tags metadata check from apply_client_tag_policy_pre_auth so
x-litellm-tags headers are always merged into request metadata, matching the post-auth
behavior in add_litellm_data_to_request. Updates pre-call tests accordingly and adds a
new test suite covering cascading credential renames into model rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous wording read "User=<new_owner> is not allowed to update the
key to belong to user=<current_owner>" — easy to misread as "caller
wants to keep the key on its current owner". Reframe as
"Non-admin caller is not allowed to rebind the key from
user=<existing> to user=<incoming>" so the direction of the failed
operation is unambiguous.
Same shape preserved (HTTPException 403); only the ``detail`` string
changes. Regression test substring updated.
A non-admin caller could rebind their own key's ``user_id`` via
``/key/regenerate``. ``_execute_virtual_key_regeneration`` had org/team
guards but no ``user_id`` guard, and ``prepare_key_update_data`` did not
strip the field — it survived ``model_dump(exclude_unset=True)`` into
the Prisma update. On the next request,
``_return_user_api_key_auth_obj`` resolved the rebound ``user_id``
against ``litellm_usertable`` and returned ``PROXY_ADMIN`` whenever
the target row's ``user_role`` was admin (e.g. the default
``user_id="default_user_id"`` created on first password-UI login).
``/key/update`` had the equivalent guard inline at
``_validate_update_key_data``; extract it to a shared helper
``_validate_caller_can_change_key_ownership`` and call from both
``/key/update`` and ``_execute_virtual_key_regeneration``. Future
regenerate-style endpoints inherit the guard for free.
Also tighten the premium gate that allowed the master-key rotation
branch to skip the enterprise check. The previous predicate was
``data.new_master_key is not None`` — a field-presence test, not an
identity check. Any non-premium caller could send any value in that
field and the premium check would no-op. Verify the caller actually
holds the master key via ``_is_master_key`` before allowing the
non-premium path.
Tests:
- ``test_regenerate_user_id_rebind_guard`` — parametrized table over
cross-user rebind (blocked), empty-string removal (blocked), and
same-user no-op rebind (allowed).
- ``test_regenerate_premium_gate_requires_actual_master_key`` /
``test_regenerate_premium_gate_allows_actual_master_key_holder`` —
ensure the premium check requires the caller actually present the
master key, and that legitimate master-key rotation still works.
* fix(proxy): resolve cache handling issues in _lookup_deprecated_key
- Updated the in-memory cache for deprecated key lookups to store a 3-tuple (active_token_id, cache_expires_at_ts, revoke_at_ts) instead of a 2-tuple, ensuring proper unpacking and backward compatibility.
- Removed duplicate cache reads and added logic to handle legacy cache entries gracefully.
- Enhanced unit tests to cover scenarios for cache hits, DB misses, and respect for revoke_at timestamps, ensuring robust handling of the grace-period key-rotation feature.
* refactor(proxy): streamline cache handling in _lookup_deprecated_key
- Simplified the cache retrieval logic by directly unpacking the 3-tuple cache entries, removing the need for backward compatibility checks for 2-tuple entries.
- Updated unit tests to ensure that pre-warmed 3-tuple cache entries are served correctly without unnecessary database lookups.
* chore(ci): add new unit test for deprecated key grace period
- Included `test_deprecated_key_grace_period.py` in the CI workflow to enhance coverage for deprecated key handling scenarios.
* fix(proxy): remove unnecessary check for revoke_at in _lookup_deprecated_key
- Eliminated the redundant check for None on revoke_at, streamlining the logic for handling deprecated keys in the cache. This change enhances the efficiency of the key lookup process.
* test(proxy): add end-to-end tests for deprecated key lookup behavior
- Introduced a new test class `TestDeprecatedKeyLookupDbE2E` to validate the behavior of deprecated key lookups against a real Prisma-backed database.
- The test ensures that old key hashes resolve correctly and that repeated lookups utilize the in-memory cache without errors.
- Cleaned up the `_lookup_deprecated_key` function by removing an unnecessary check for `revoke_at`, enhancing the efficiency of the key lookup process.
Caller-supplied tags (`x-litellm-tags` header, body `tags`, `metadata.tags`)
were silently dropped unless the key/team had
`metadata.allow_client_tags: true` set. Restore the documented behavior:
tags from the request always flow into `metadata.tags` and union with any
admin-configured static tags from key/team/project metadata.
Removes the `allow_client_tags` opt-in flag from the pre-call pipeline.
The flag was only ever read here; it has no schema or endpoint footprint,
so leftover values in existing key metadata are inert.
Test cleanup mirrors the simplification: drop the three tests that
verified the strip-when-not-opted-in path, drop the `allow_client_tags`
fixture lines from the merge/union tests.
Caller-supplied tags (`x-litellm-tags` header, body `tags`, `metadata.tags`)
were silently dropped unless the key/team had
`metadata.allow_client_tags: true` set. Restore the documented behavior:
tags from the request always flow into `metadata.tags` and union with any
admin-configured static tags from key/team/project metadata.
Removes the `allow_client_tags` opt-in flag from the pre-call pipeline.
The flag was only ever read here; it has no schema or endpoint footprint,
so leftover values in existing key metadata are inert.
Test cleanup mirrors the simplification: drop the three tests that
verified the strip-when-not-opted-in path, drop the `allow_client_tags`
fixture lines from the merge/union tests.
The enterprise package is installed as `litellm_enterprise` (per
enterprise/pyproject.toml), but several tests imported it as
`enterprise.litellm_enterprise.*` — a path that only resolves
because the repo root happens to sit on sys.path, letting Python's
implicit namespace package machinery discover `enterprise/` as a
directory.
This breaks any test runner that relocates source (e.g. the
mutation-testing workflow, which copies tests under `mutants/`) and
also caused two `patch()` strings to target a module path that does
not match what production code imports — meaning those mocks were
never actually patching the production module's attribute.
Replace `from enterprise.litellm_enterprise.` with the canonical
`from litellm_enterprise.` across 6 test files, and fix two
`patch()` target strings (and one `sys.modules` patch key in the
SSO test) to match.
* feat(ui): search teams by team ID alongside name
The Teams page search box only matched team_alias, so pasting a team UUID
returned zero results. Detect when the input is a full UUID and route it
to the team_id filter instead; otherwise keep the existing alias substring
search. Placeholder now reads "Search teams by name or ID...".
Resolves LIT-2648
* refactor: search teams via backend OR clause, drop client-side UUID detection
Adds a `search` query param to /v2/team/list that ORs across team_id
(exact) and team_alias (case-insensitive contains), so the search box
sends one param regardless of input format. Removes the isLikelyTeamId
helper and the client-side branching it fed.
* fix: invalidate cached tag object on tag budget reset (#27481) (#27572)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* chore(mcp): tighten stdio server registration paths (#27570)
Squash-merged by litellm-agent from stuxf's PR.
* fix(proxy): clear MCP OpenAPI mappings on server eviction; widen budget cache invalidation
Evict OpenAPI tools from global_mcp_tool_registry and strip tool_name_to_mcp_server_name_mapping entries when a server leaves the runtime registry (remove_server and approval-status eviction). Invalidate user_api_key_cache for keys, orgs, and team members on budget-tier spend resets alongside tags.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): align update_server eviction with remove_server name fallback
Document budget-reset test assertion flip (cross-pod cache staleness).
Greptile: eviction now pops by server_id then server_name like remove_server;
test docstring explains assert_not_awaited -> assert_any_await change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix org budget cache invalidation
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Reject fnmatch wildcards on non-scope claims when the claim string contains
whitespace so malformed iss values cannot match patterns like trusted.*.
Merge every entry when team_id_jwt_field resolves to a list instead of
keeping only the first element.
Co-authored-by: Cursor <cursoragent@cursor.com>
Stripping body-supplied tags in apply_client_tag_policy_pre_auth silently
disabled per-tag budget enforcement for non-opted-in keys — pre-PR
behavior was that those tags reached _tag_max_budget_check inside
common_checks. The post-auth strip in add_litellm_data_to_request
continues to remove unauthorized tags before they leave the proxy.
Also moves the LiteLLMProxyRequestSetup import to module-level (no
circular dep with auth/user_api_key_auth.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The x-litellm-tags header was merged into request metadata only after the
auth chain completed, so _tag_max_budget_check (which reads tags from the
request body) silently failed open for header-tagged requests — spend
accumulated past max_budget without any 400 budget_exceeded response.
Move the client-tag policy (strip-or-merge gated on allow_client_tags) to
run before common_checks so header tags are visible to budget enforcement.
The post-auth strip+merge in add_litellm_data_to_request stays as
defense-in-depth; the new pre-auth helper is idempotent with it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mcp): forward extra_headers for OpenAPI MCP tools
OpenAPI-generated tools only applied static closure headers and BYOK
Authorization via ContextVar. Copy MCPServer.extra_headers from the
incoming MCP request into _request_extra_headers (set in server.py before
local tool dispatch), merge in openapi_to_mcp_generator via a small helper.
OAuth2 M2M: do not forward caller Authorization from raw_headers (same rule
as _prepare_mcp_server_headers for managed MCP).
Adds TestRequestExtraHeaders and clarifies mcp_server_manager registration
comment.
Fixes#26794
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): access has_client_credentials on MCPServer directly
Greptile: getattr default was redundant; property exists on MCPServer and
mcp_server is non-None inside the extra_headers forwarding block.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The previous detection treated any model with input_cost_per_image
or output_cost_per_image as image generation. Several chat and
embedding models carry those fields to price multimodal vision input,
not generated images:
- gemini-3.1-pro-preview (mode=chat) has output_cost_per_image=0.00012
alongside input/output token pricing.
- azure/gpt-realtime-* (mode=chat) has input_cost_per_image=5e-6.
- amazon.titan-embed-image-v1 (mode=embedding) has
input_cost_per_image=6e-5.
For these models the image-gen branch fired first and reserved a
fraction of a cent per request, short-circuiting the token-priced
path entirely. Long Gemini chats reserved 1 × $0.00012 instead of
the true token cost.
Gate strictly on mode in {"image_generation", "image_edit"}. All 197
real image_generation entries and all 31 image_edit entries
(Flux Kontext, Stability inpaint/outpaint, etc.) carry the right mode,
so the field-presence fallback was unnecessary.
Adds regression tests for the chat-model-with-image-cost-field case
and for image_edit reservation.
Image-generation routes (dall-e-3, flux, etc.) have no per-token output
cost so they fell through to the no-reservation read-time-only path.
Concurrent image requests against a depleted budget could all pass
common_checks (counter exactly at max_budget passes the strict-`>`
gate) and reach the provider before reconciliation caught up.
Add per-image reservation in _estimate_request_max_cost_for_model:
when the model has a per-image cost field, reserve `n × cost_per_image`
upfront. The atomic counter increment serializes concurrent admissions,
so the second request sees the post-first-reservation counter and
raises BudgetExceededError instead of silently leaking through.
Both `output_cost_per_image` and `input_cost_per_image` are honored —
naming is inconsistent across providers (OpenAI dall-e-3 uses
input_cost_per_image, aiml/dall-e-3 uses output_cost_per_image for
the same per-generated-image price).
Per-pixel pricing (DALL-E 2 size variants) and TTS/STT routes still
fall through to read-time enforcement; those are follow-ups.
- Introduce RoutingPrismaWrapper that transparently routes read operations (find_*, count, group_by, query_raw, query_first) to a reader endpoint while writes remain on the writer, enabling Aurora-style reader/writer endpoint splits
- Add IAMEndpoint dataclass and parse_iam_endpoint_from_url() to capture static connection fields from a reader URL so only the IAM token needs to rotate, avoiding the need for separate DATABASE_HOST_READ_REPLICA/etc. env vars
- Enhance PrismaWrapper with per-instance knobs (db_url_env_var, iam_endpoint, recreate_uses_datasource, log_prefix) so writer and reader wrappers are independent: the reader writes its fresh URL to DATABASE_URL_READ_REPLICA and passes datasource override to Prisma since Prisma only auto-reads DATABASE_URL
- Fix deadlock in PrismaWrapper.__getattr__: when called from inside a running event loop, schedule the token refresh as a background task instead of blocking with run_coroutine_threadsafe + future.result(), which would deadlock the loop thread waiting for a coroutine that needs the loop to run
- Fix botocore crash when DATABASE_PORT is unset by defaulting to "5432" in both proxy_cli.py and PrismaWrapper.get_rds_iam_token(); passing None caused botocore to embed the literal string "None" in the presigned URL
- Implement graceful reader degradation: reader connect/recreate failures are non-fatal; wrapper sets _reader_unavailable=True and silently routes reads to the writer to keep the proxy serving traffic during transient reader outages
- Add PrismaClient.writer_db property so the reconnect smoke-test always validates the writer engine specifically; query_raw on the routing wrapper would route to the reader and not verify the newly-recreated writer
- Expose DATABASE_URL_READ_REPLICA in Helm chart (values.yaml + deployment.yaml) via both plain value and secret key reference, and document the field in docker-compose.yml
- Add 887-line test suite covering routing logic, IAM token refresh paths, reader degradation scenarios, datasource override behavior, and the deadlock regression
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
reserve_budget_for_request fell back to reserving the entire remaining
team/key/user headroom whenever a request omitted max_tokens, which
pinned the spend counter at max_budget for the duration of the
in-flight request and false-positive-blocked every concurrent or
back-to-back request until the success callback reconciled. Surfaced
as an integration-test team being budget-blocked at its $2000 cap
while DB spend was $0.144.
Switch the missing-max_tokens path to a fixed default of 16384 output
tokens (mirrors parallel_request_limiter_v3's DEFAULT_MAX_TOKENS_ESTIMATE
precedent), and clamp explicit max_tokens at the model's
max_output_tokens for reservation accounting only. The outbound request
body is unchanged, so providers see whatever the caller actually sent;
only the local integer used to compute reservation cost is bounded.
This also prevents a hostile max_tokens=999999999 from inflating one
request's reservation up to the entire team headroom.
For Opus 4.7 (output $25/M, max_output 128K) on a $2000 budget the
worst-case per-request reservation drops from "everything left" to
$3.20, raising admittable concurrency from 1 to ~625.
Operators upgrading past 35bbca60b0 (which made /metrics auth
default-on) see "Malformed API Key passed in. Ensure Key has 'Bearer '
prefix." with no hint that
litellm_settings.require_auth_for_metrics_endpoint: false restores the
previous unauthenticated behavior. Append that discovery hint to the
existing 401 body so a Prometheus scraper that breaks after upgrade
has a clear migration path. No behavior change.
* feat(sso): show full IdP claims in /sso/debug/callback
The debug callback only displayed the proxy-parsed OpenID summary, so
customers couldn't verify what custom claims (team_id, team_alias, roles,
etc.) the IdP was actually returning. Render two new sections — Raw
Claims (userinfo) and Access Token Claims (decoded JWT) — alongside the
existing parsed view. Strip bearer tokens defense-in-depth in case a
non-conforming IdP places them in its userinfo response.
Resolves LIT-2838
* Update litellm/proxy/management_endpoints/ui_sso.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(sso): hoist json.dumps out of f-string for py3.10 ruff
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Mirrors the system-message skip in PR #25481 for tool-role messages.
Adds a global litellm.skip_tool_message_in_guardrail flag and a
per-guardrail litellm_params.skip_tool_message_in_guardrail override,
applied in the OpenAI and Anthropic chat translation handlers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
