2527 commits

Author	SHA1	Message	Date
Sameer Kankute	e59e34bed3	Gemini managed agents support (#28270) ... * Add support for environment variable in interactions api * Add sdk support for gemini create agent * Add agents endpoint support via proxy * Add outputs of each api * Add routing for model and agents param * Remove redundant condition in get_provider_agents_api_config LlmProviders.GEMINI.value is literally the string "gemini", so the second clause of the or was checking the exact same thing as the first. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: forward query-param credentials to list/get/delete/versions Gemini agent endpoints The list_gemini_agents, get_gemini_agent, delete_gemini_agent, and list_gemini_agent_versions endpoints previously constructed a hardcoded data dict with no mechanism to pass provider credentials. Unlike create_gemini_agent (POST, reads litellm_params_template from body), these GET/DELETE endpoints gave no way for multi-tenant callers to supply a per-request api_key or other LiteLLM params. Fix: - Add _merge_query_params_into_data() helper that reads query parameters from the request and merges them into the data dict without overwriting already-set keys (e.g. path params like 'name'). - Support a JSON-encoded litellm_params_template query parameter (matching the POST body pattern) as well as flat key=value pairs (e.g. api_key=AIza...). - Apply the helper in all four affected endpoints. - Add 13 unit tests covering the helper and each endpoint. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: pass model=None for managed agent proxy endpoints to prevent agent name polluting data["model"] Endpoints acreate_agent, aget_agent, adelete_agent, and alist_agent_versions were passing model=<agent_name> to base_process_llm_request. This caused common_processing_pre_call_logic to write the agent name into self.data["model"], which then triggered spurious model-alias mapping, rate-limiting lookups, and logging tied to a non-existent model deployment. The agent name is already carried in data["name"] and is passed correctly to the SDK functions (litellm.interactions.agents.*). There is no reason to also set model=<agent_name>; the correct value is model=None for all five managed-agent management routes. Adds tests/test_litellm/proxy/google_endpoints/test_managed_agents_model_param.py to verify all five managed-agent endpoints pass model=None. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: address greptile P1/P2 review comments P1 (router.py): Restore fallback/retry support for acreate_interaction and create_interaction. Both were silently moved to _init_interactions_api_endpoints (direct call, no fallbacks). Moved them back to _ageneric_api_call_with_fallbacks so users with configured fallback models keep retry behaviour. P1 security (agents_endpoints.py): Remove flat query-param credential path (e.g. ?api_key=AIza...) from _merge_query_params_into_data. Credentials in URL query strings appear verbatim in server access logs, CDN edge logs, and browser history. Only the JSON-encoded litellm_params_template query param (matching the POST body pattern) is retained. P2 (interactions/http_handler.py): Extract _BaseHTTPHandler with shared _handle_error, _sync_client, and _async_client helpers. InteractionsHTTPHandler now extends _BaseHTTPHandler. The _async_client reads the provider from litellm_params instead of hardcoding GEMINI. P2 (interactions/agents/http_handler.py): AgentsHTTPHandler now extends InteractionsHTTPHandler (which inherits _BaseHTTPHandler) so all shared HTTP infrastructure is reused rather than duplicated. Removes the hardcoded LlmProviders.GEMINI from the async client path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address CI failures from greptile review fixes - black: format interactions/agents/main.py and utils.py - tests: update test_gemini_agents_endpoints.py to match new _merge_query_params_into_data behaviour (flat credential params are rejected; only JSON-encoded litellm_params_template is accepted) - ci: add test_gemini_agents_endpoints.py to endpoints-and-responses shard in test-unit-proxy-db.yml so assert-shard-coverage passes - tests: add _initialize_managed_agents_endpoints and _init_managed_agents_api_endpoints test coverage so router_code_coverage passes; also fix TestRouterCreateInteractionRouting to reflect that acreate_interaction now correctly routes through _ageneric_api_call_with_fallbacks (restoring fallback support) Co-authored-by: Cursor <cursoragent@cursor.com> * fix: remove InteractionsHTTPHandler._handle_error override to fix type errors AgentsHTTPHandler extends InteractionsHTTPHandler and calls self._handle_error(provider_config=agents_api_config) where agents_api_config is BaseAgentsAPIConfig. Python MRO resolved _handle_error to InteractionsHTTPHandler._handle_error which expected BaseInteractionsAPIConfig, causing 10 mypy arg-type errors in interactions/agents/http_handler.py. Removing the redundant override lets both classes inherit _BaseHTTPHandler._handle_error (provider_config: Any) which is structurally correct for both config types. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: agent-only interactions and managed agents provider routing Resolve None custom_llm_provider in agents HTTP client lookup and set custom_llm_provider on GenericLiteLLMParams for all agent CRUD paths. Stop mapping agent names to proxy model routing; route interactions through _init_interactions_api_endpoints with fallbacks only when model is set. Consolidate duplicate router elif branches for interaction APIs. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix greptile review * test(agents): add unit tests for managed agents SDK and HTTP handler Adds coverage for the new `litellm.interactions.agents` surface area: - main.py: sync/async entry points (create/list/get/delete/list_versions), provider config lookup, logging-obj helper, async error wrapping - http_handler.py: every CRUD method (sync + async paths), `_is_async` dispatch branches, and provider error mapping through GeminiAgentsConfig - utils.py: get_provider_agents_api_config for supported / unsupported providers Brings patch coverage on these files from <25% to ~100% so codecov/patch is satisfied. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(gemini-agents): fix misleading credential-passing examples in GET/DELETE docstrings (#28293) The four GET/DELETE endpoint docstrings (list_gemini_agents, get_gemini_agent, delete_gemini_agent, list_gemini_agent_versions) documented passing per-request credentials as flat query parameters (e.g. ?api_key=AIza...). However, _merge_query_params_into_data only reads the JSON-encoded litellm_params_template query parameter and intentionally ignores flat params (URL query strings appear verbatim in access logs, browser history, and Referer headers). Callers following the documented curl examples would have their credentials silently dropped and hit auth failures against Gemini. Update the examples to use the supported JSON-encoded litellm_params_template query parameter, matching _merge_query_params_into_data's own docstring. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(agents): rename provider-agnostic agent response types Move GeminiAgent{ListResponse,DeleteResult,VersionsResponse} to provider-neutral names (AgentListResponse, AgentDeleteResult, AgentVersionsResponse) so the BaseAgentsAPIConfig interface no longer references Gemini-specific type names. * fix(gemini-agents): close veria-flagged credential-escalation gaps Two high-severity findings from the veria-ai PR review are addressed: 1. **api_base override could leak the shared Gemini key** GeminiAgentsConfig.validate_environment falls back to GOOGLE_API_KEY / GEMINI_API_KEY when no api_key is supplied. Combined with caller-controlled api_base on the proxy CRUD endpoints, an authenticated user could redirect the outbound request to an attacker-controlled host and capture the operator's shared Gemini key from the x-goog-api-key header. The config now refuses env-fallback whenever api_base is explicitly overridden. 2. **Managed-agent CRUD exposed to ordinary LLM keys** The new /v1beta/agents routes live in google_routes (i.e. llm_api_routes), so any non-admin LLM key can reach them. Unlike /v1beta/models/...: generateContent these endpoints are NOT model-routed and have no model_list-supplied credentials, so env-fallback would let any LLM key list / create / delete agents inside the operator's Gemini project. Each endpoint now calls _enforce_caller_supplied_provider_key, which requires non-admin callers to supply their own Gemini api_key via litellm_params_template. Proxy admins keep the env-fallback convenience. Tests cover non-admin rejection, admin allow-through, the api_base override guard, and SDK env-fallback when api_base is not overridden. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(router): restore strict assert_called_once_with on interactions default-provider test --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-05-19 16:02:03 -07:00
Sameer Kankute	0290c7bc00	fix(proxy): decode bytes and pass-through SSE for Google-native streamGenerateContent (#27444) (#28213) ... * fix(proxy): decode bytes and pass-through SSE for Google-native streamGenerateContent (#27444) * fix(proxy): address Greptile review on Google-native SSE bytes path Remove unreachable try/except around SSE pass-through yield and add a unit test covering pre-formatted SSE bytes, terminator padding, and non-SSE byte fallback wrapping. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-18 22:46:11 -07:00
Sameer Kankute	36c494fdd2	Litellm oss staging (#28161) ... * fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455) Squash-merged by litellm-agent from Anai-Guo's PR. * feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508) Squash-merged by litellm-agent from yimao's PR. * fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711) Squash-merged by litellm-agent from krisxia0506's PR. * fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503) Squash-merged by litellm-agent from krisxia0506's PR. * Fix Gemini MIME detection for extensionless GCS URIs (#27278) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107) Squash-merged by litellm-agent from voidborne-d's PR. * feat(chart): add support for autoscaling behavior in HPA (#27990) Squash-merged by litellm-agent from FabrizioCafolla's PR. * feat(proxy): add blocked flag to models for pause/resume from the UI (#27927) Squash-merged by litellm-agent from Cyberfilo's PR. * fix: pass socket timeouts to Redis cluster clients (#27920) Squash-merged by litellm-agent from tomdee's PR. * Fix/cache token (#28009) Squash-merged by litellm-agent from escon1004's PR. * fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080) Squash-merged by litellm-agent from Divyansh8321's PR. * fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617) * fix: reset org and tag budgets (#27326) * reset org budgets * reset tag budgets --------- Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> * fix(ui): omit allowed_routes from key edit save when unchanged (#27553) * fix(ui): omit allowed_routes from key edit save when unchanged When a team admin opens Edit Settings on a key with key_type=AI APIs and saves without changing anything, the UI re-sends the existing allowed_routes value, which the backend's _check_allowed_routes_caller_permission gate rejects for non-proxy-admins (LIT-2681). Strip allowed_routes from the patch in handleSubmit when it deep-equals the original keyData.allowed_routes. The backend treats absence as "leave alone," so no-op saves now succeed for non-admins. Admins explicitly editing the field still send the new value. * fix(ui): order-insensitive allowed_routes diff + cover null-original case Address Greptile review: - Switch the "is allowed_routes unchanged" check to a Set-based comparison so a server-side reorder of the array doesn't register as a user edit and re-trigger LIT-2681. - Add two regression tests: (1) keyData.allowed_routes is null and the form is untouched — patch should strip the field; (2) server returned routes in a different order than the user originally entered — patch should still recognize the value as unchanged. * chore(ui): strip ticket refs and tighten comments in key edit fix - Remove internal-tracker references from in-code comments - Tighten the WHY comment in handleSubmit to two lines - Drop redundant test-block comments — test names already describe the case * fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc * fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests GuardrailRaisedException and BlockedPiiEntityError both lacked a status_code attribute. When these exceptions reached the proxy exception handler (getattr(e, 'status_code', 500)), the fallback defaulted to HTTP 500 — making intentional guardrail blocks indistinguishable from server errors and causing unnecessary client retries. Changes: - Add status_code=400 (keyword-only) to GuardrailRaisedException - Add status_code=400 (keyword-only) to BlockedPiiEntityError - Update _is_guardrail_intervention() to recognize both exceptions so downstream loggers record 'guardrail_intervened' instead of 'guardrail_failed_to_respond' - Add 6 unit tests for default/custom status codes and getattr pattern - Strengthen existing blocked-action test with status_code assertion Fixes #24348 --------- Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> * fix(router/proxy): address Greptile P1+P2 review comments on PR #28161 - router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429) when a specifically-addressed deployment is administratively blocked; 429 misleads retry-enabled clients into spinning forever against a paused model - proxy_server: compute get_fully_blocked_model_names() once before both branches in model_list() instead of duplicating the call in each branch - deepseek: upgrade silent debug log to warning when injecting placeholder reasoning_content so callers are clearly notified of degraded multi-turn quality - tests: update two blocked-deployment assertions to expect ServiceUnavailableError Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address bug detection findings (cache token order, mutable defaults) Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bugs in async pass-through, anthropic cache token detection, rerank tests - async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments - cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0 - dashscope rerank tests: pass request to httpx.Response constructions for consistency Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix code qa * fix(vertex_ai/gemini): strip MIME parameters from GCS contentType GCS object metadata's contentType field can include parameters such as 'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases so downstream get_file_extension_from_mime_type sees a bare MIME type. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai/gemini): clarify mime-type error message string concatenation Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Vincent <yimao1231@gmail.com> Co-authored-by: Kris Xia <xiajiayi0506@gmail.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Tom Denham <tom@tomdee.co.uk> Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com> Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com> Co-authored-by: robin-fiddler <robin@fiddler.ai> Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-18 16:27:44 -07:00
ryan-crabbe-berri	fe63650ebd	fix(proxy): gate team allowed_passthrough_routes to proxy admins (#28097) ... * fix(proxy): gate team allowed_passthrough_routes to proxy admins allowed_passthrough_routes short-circuits the role-based route gate, so the keys endpoints already restrict it to proxy admins. The team writers (/team/new, /team/update) had no equivalent check, letting an org admin (a non-proxy-admin who clears the route gate and _verify_team_access) self-grant pass-through routes on their team. Lift the keys check into a shared helper and apply it to both team endpoints. Resolves LIT-3019 * docs(proxy): note view-only admins are intentionally excluded from passthrough gate Clarifies the proxy-admin guard per review feedback; no behavior change. Refs LIT-3019	2026-05-18 08:56:25 -07:00
Mateo Wang	1b0ae3af83	fix(mcp-oauth): PROXY_BASE_URL escape hatch + diagnostic logging for {"detail":"invalid_request"} (#28086) ... * fix(mcp-oauth): add PROXY_BASE_URL escape hatch + diagnostic logging for invalid_request Customers hitting "{"detail":"invalid_request"}" on the MCP /authorize endpoint had no way to recover when their ingress mangles X-Forwarded-* headers (the same-origin check in validate_trusted_redirect_uri compares the browser-supplied redirect_uri against get_request_base_url, which is reconstructed from those headers). Two contained changes: 1. get_request_base_url now honours PROXY_BASE_URL as the canonical public origin when set, bypassing the X-Forwarded-* trust gate entirely. Operators who know their public URL can set it once instead of debugging ingress header rewrites. 2. The rejection path in validate_trusted_redirect_uri emits a WARN log carrying the redirect_uri, computed proxy base, and the X-Forwarded-* / Host headers seen. A bare 400 was undiagnosable; this turns it into a one-line root-cause. * test(mcp-oauth): capture warnings from correct logger ("LiteLLM") Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp-oauth): reject malformed PROXY_BASE_URL with one-shot diagnostic A scheme-less PROXY_BASE_URL (e.g. "litellm.example.com" instead of "https://litellm.example.com") would sail through urlparse with empty scheme + netloc, silently breaking every same-origin compare in validate_trusted_redirect_uri and leaving the operator staring at the same opaque 400 the env var was meant to fix. Validate it once at read time: only honour values that parse as http(s) URLs with a non-empty netloc; otherwise log a one-shot WARN naming the bad value and fall through to the request-derived origin so the proxy still serves traffic. * fix(mcp/oauth): normalize PROXY_BASE_URL to strip query/fragment Match the X-Forwarded-* path's normalization so a configured PROXY_BASE_URL containing a query string or fragment does not break downstream f-string concatenation like f"{base_url}/callback". Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(mcp-oauth): drop non-essential comments from PROXY_BASE_URL changes Strip narrative comments and verbose docstrings added in this PR; the code is intuitive enough on its own and the log messages already carry their own diagnostic context. Pre-existing comments are left untouched. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-16 17:48:03 -07:00
Shivam Rawat	fbe0ee81f1	fix(proxy): sort BYOK models by their displayed name in /v2/model/info (#28079) ... * fix(proxy): sort BYOK models by team_public_model_name in /v2/model/info Team BYOK rows persist an internal `model_name` like `model_name_{team_id}_{uuid}` and expose the user-facing name via `model_info.team_public_model_name`. The UI's `getDisplayModelName` and the search filter already fall back to that field, but `_sort_models` was keying off the raw `model_name` — so BYOK rows ranked by their opaque IDs and clumped at the end of the alphabetized list instead of interleaving with non-BYOK rows. Match the UI/search behavior: prefer `team_public_model_name` when present, fall back to `model_name` otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): case-insensitive DB-side search for BYOK models `_apply_search_filter_to_models` used Prisma's JSON path `string_contains` to match the BYOK `team_public_model_name` field, but that operator is case-sensitive in Postgres (no `mode: insensitive` flag like column-level string filters have). So a search for "claude" missed a stored "Claude Sonnet" via the DB branch even though the router-side path matched it case-insensitively. Widen the JSON branch to "row has a team_public_model_name set" and filter case-insensitively in Python so DB-only BYOK rows match the same terms users see in the UI. This also drops the now-unused DB-level page-size optimization and `sort_by` knob — the in-Python filter is the source of truth for `db_models_total_count` now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): scope BYOK search results to caller's accessible teams `_apply_search_filter_to_models` was widened to fetch every row with a `team_public_model_name` set so case-insensitive search could match mixed-case stored names. `/v2/model/info` is reachable by non-admin keys though, and the helper ran before `include_team_models` / `teamId` filtering — so a non-admin caller could search a common substring like "claude" and see BYOK rows belonging to teams they're not a member of. Resolve the caller's team membership once (admin → no scoping, else their `user_row.teams`) and drop BYOK rows (those with `model_info.team_id` set) outside that scope on both the router-side matches and the over-broad DB query, before display-name matching. Non-team rows are unaffected and remain gated by the existing `include_team_models` / `direct_access` paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): search by team_public_model_name and scope teamId queries - /v2/model/info search now matches both `model_name` and `model_info.team_public_model_name`, so team BYOK rows (which persist an internal `model_name_{team_id}_{uuid}`) are findable by the public name shown in the UI. DB query OR-includes a JSON-path match on `team_public_model_name` for rows that exist only in the DB. - `_filter_models_by_team_id` no longer short-circuits on the viewer's `direct_access` flag — that describes the admin viewer's own permissions and would leak every public model into a team-scoped view. Models are kept only when they belong to the team (own BYOK, in access_via_team_ids, or reachable via team.models / access groups). - Added `_authorize_team_id_query`: the untrusted `teamId` query parameter now requires the caller to be a proxy admin or a member of the requested team, otherwise returns 403. Without this, any authenticated user could enumerate another team's BYOK metadata by guessing the team id. - `_get_caller_byok_team_scope` now treats `PROXY_ADMIN_VIEW_ONLY` the same as `PROXY_ADMIN` (both are admin roles); previously VIEW_ONLY admins fell through to a user-id team lookup and saw only their own teams' BYOK rows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): bound BYOK search DB fetch in /v2/model/info Previously the DB-side search OR'd a JSON-path predicate `{model_info: {path: [team_public_model_name], string_contains: ""}}` to compensate for Prisma's case-sensitive JSON `string_contains` on Postgres. That predicate matches every row that has any `team_public_model_name` set, so any authenticated caller could force a full BYOK-table read with `/v2/model/info?search=x` regardless of page size. Drop the JSON-path branch. The DB query now does a bounded `model_name contains <search>` lookup. BYOK rows that are loaded into the router are still searchable by their `team_public_model_name` via the router-side filter; only the rare edge case of a BYOK row that exists only in the DB (router sync failed) loses display-name search, which is an acceptable trade-off given the DoS surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): bound DB find_many in /v2/model/info search The previous bounding patch dropped the page-aware `take=N` on `find_many`, so a broad `?search=model` would load and decrypt every matching DB row on each request even though the response only returns one page. Restore bounded fetches in `_apply_search_filter_to_models`: * Unsorted searches use `take = max(0, page * size - router_count)`, i.e. exactly one page worth of remaining DB rows. * Sorted searches need ordering across the full match set, so they cap at `_SORTED_SEARCH_DB_FETCH_CAP = 500` instead of fetching everything. * Total count comes from a cheap `count(...)` query so pagination stays accurate without materializing every row. Wired `page`, `size`, and `sortBy` through from the endpoint and added a regression test covering both `take` values. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(proxy): extract DB-fetch helper to satisfy PLR0915 _apply_search_filter_to_models tripped Ruff's "too many statements" (51 > 50) after the bounded-fetch fix. Move the DB-side block into `_fetch_db_models_for_search`, which keeps the same behavior: * Bounded `take` via page math (unsorted) or `_SORTED_SEARCH_DB_FETCH_CAP` (sorted) * Cheap `count(...)` for accurate pagination totals * Caller-team scope applied to fetched rows before decrypt Pure refactor; no behavior change. All 8 BYOK/team tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: apply black formatting to _fetch_db_models_for_search CI's "Check Black formatting" step flagged one line in the helper added in d55eecf6af. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 15:46:21 -07:00
ryan-crabbe-berri	0300333753	feat(otel): OTel-standard attributes on the proxy SERVER span (status code, route/path, preprocessing latency) (#28040) ... * feat(otel): expose http.response.status_code on failure spans Set the OTel-standard http.response.status_code (integer) on failure spans alongside the existing OpenInference error.code (kept for back-compat). error.type is already emitted via ERROR_TYPE. Crucially, also record structured error attributes on the proxy SERVER span ('Received Proxy Server Request') from async_post_call_failure_hook - the only place the SERVER span is in hand. _handle_failure records on the litellm_request child span (the parent span is not propagated into its kwargs), so prior to this change the SERVER span that dashboards query carried only span status, never error.code/error.type. Reuses _record_exception_on_span + StandardLoggingPayloadSetup.get_error_information so values match the child span. Tests: recorder unit coverage + a hook-driven test asserting the SERVER span is stamped (the gap recorder-only tests missed). Full test_opentelemetry.py suite: 197 passed. * feat(otel): set http.route + url.path on the proxy SERVER span Add the OTel-standard http.route (low-cardinality route template, e.g. /v1/threads/{thread_id}/runs) and url.path (literal path) to the SERVER span ('Received Proxy Server Request') so dashboards can group traffic by endpoint instead of seeing every path param as a unique value. Same architectural gap as the status-code commit: the success/failure logging handlers write the litellm_request CHILD span, and _handle_success explicitly refuses to copy to the SERVER span. Verified with a console-exporter run that the SERVER span was bare on success. Unlike error info, route/path are known at request time, so set them directly on the freshly-created SERVER span in user_api_key_auth (one edit point, works for success and failure, no hook-ordering risk): - http.route from the matched FastAPI route (scope['route'].path), empirically confirmed populated at auth-dependency time. - url.path from the existing literal-path variable. New get_request_route_template helper + set_proxy_request_route_attributes (no-op on None span, so the Langfuse override stays safe). Tests: route-attribute setter + route-template helper edges. Full test_opentelemetry.py and test_auth_utils.py green. * feat(otel): set litellm.preprocessing.duration_ms on the proxy SERVER span Expose the total time LiteLLM spends before the upstream provider request begins (auth + parsing + pre-call hooks) as a single number on the SERVER span ('Received Proxy Server Request'). Window: proxy-receive -> FIRST provider handoff. Retry semantics: first attempt only (pure preprocessing, excludes retry loops + backoff). api_call_start_time is overwritten on every attempt, so a set-once first_api_call_start_time pins the first handoff. Same architectural gap as the prior two commits: the success/failure logging handlers write the litellm_request CHILD span, not the SERVER span. Set it instead from the post-call hooks on user_api_key_dict.parent_otel_span. Failure-path subtlety: request_data.pop('litellm_logging_obj') runs before the failure-hook loop, so the failure hook can't read the logging object. litellm_received_at is propagated via the existing request->metadata channel, and first_api_call_start_time is mirrored onto litellm_params.metadata, so both anchors survive into request_data and the OTel helper reads them uniformly for success and failure. Edits: user_api_key_auth (stash receive instant), litellm_pre_call_utils (propagate it), litellm_logging (set-once first handoff + metadata mirror), opentelemetry (constant + set_preprocessing_duration_attribute, called from both post-call hooks). Tests: duration helper (both container shapes, missing/negative/None edges) + set-once invariant (retry doesn't overwrite, metadata mirror). test_opentelemetry.py + test_auth_utils.py + test_litellm_logging.py: 447 passed. Verified live: SERVER span carries the attribute on success and failure, coexisting with the status-code and route attributes. * fix(otel): MyPy type-narrowing for status-code + preprocessing-duration No behavior change. MyPy (CI lint) flagged: - error_information["error_code"] is str|None: narrow via a None-checked local before int(). - _to_timestamp returns Optional[float]: resolve both anchors and return early if either is None instead of subtracting possibly-None floats. * fix(otel): stop polluting user request metadata with first_api_call_start_time The PR3 set-once preprocessing anchor was mirrored into litellm_params["metadata"] from core litellm_logging.py. That dict is the caller's request metadata, mutated in place and shared across every call path including pure SDK (litellm.acreate_batch). It got echoed into LiteLLMBatch(metadata=...), which the OpenAI batch schema types as Dict[str, str] -> pydantic ValidationError on a datetime value. - litellm_logging.py: set first_api_call_start_time only on model_call_details (success path reads it there directly). - proxy/utils.py: post_call_failure_hook lifts it off the logging object into request_data (internal top-level key, same convention as the other proxy-internal request_data keys) right before the existing litellm_logging_obj pop. Never touches user metadata. - opentelemetry.py: read the anchor from the container top level (model_call_details on success, request_data on failure). - Tests updated; add TestPostCallFailureHookLiftsFirstApiCallStartTime. Fixes the batches_testing regression introduced on this branch. * chore(otel): trim verbose comments to concise rationale Collapse multi-line why-blocks to one or two lines and drop process/plan references (PR-numbering, "the plan") from test comments. No behavior change.	2026-05-16 13:45:08 -07:00
Yassin Kortam	014cb8fa9d	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557) ... Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces - Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass - Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured - Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist - Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes - Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock - Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix - Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set - Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>	2026-05-16 09:25:17 -07:00
Yuneng Jiang	ed8a8a634b	Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/determined-yalow-811fee	2026-05-15 16:51:53 -07:00
Yuneng Jiang	953a8b1b85	test(proxy): isolate run_server CLI tests from prisma DB-setup path ... test_keepalive_timeout_flag and test_timeout_worker_healthcheck_flag were the only run_server tests in test_proxy_cli.py that neither stripped DATABASE_URL/DIRECT_URL nor mocked the prisma DB path. When a DATABASE_URL is present (CI/env leak), run_server --local enters the DB block and blocks in the un-timeout'd subprocess.run(["prisma"]) at proxy_cli.py:987 plus the ProxyExtrasDBManager migrate-deploy retry loops, ~370s per test on the CI runner. --dist=loadscope pins both to one xdist worker, so the proxy-infra job appears stuck at 99% and hits the 20-min timeout. Apply the same isolation every other run_server test in this file already uses: mock PrismaManager.setup_database + should_update_prisma_schema and strip DATABASE_URL/DIRECT_URL. Full module drops from 31.7s to 2.9s locally; both tests fall off the slow list.	2026-05-15 16:51:45 -07:00
Yuneng Jiang	a4a1726d99	test(proxy): add endpoint-level regression for /config/update double-encryption ... Adds test_update_config_env_var_round_trip_not_double_encrypted, which drives the real /config/update handler: first write plaintext, then re-POST the stored ciphertext (the Admin UI round-trip) and assert the value is not stacked with a second encryption layer and untouched keys stay byte-identical. Verified to fail against the pre-fix handler and pass after. Also tightens the unit test to exactly three ciphertext re-feeds.	2026-05-15 15:29:08 -07:00
Yuneng Jiang	0d8c9137fb	fix(proxy): make /config/update env-var encryption idempotent ... A single decrypt-then-encrypt chokepoint (_encrypt_env_variables_for_db) now backs both update_config and save_config. Re-submitting a value the Admin UI read back from /get/config/callbacks as ciphertext no longer stacks a second encryption layer, which previously decrypted to garbage and silently broke the callback. The chokepoint decrypts with the pure _decrypt_db_variables (no os.environ mutation on the write path) and encrypts exactly once; update_config merges only the sent keys so untouched env vars keep their stored ciphertext byte-for-byte.	2026-05-15 15:14:18 -07:00
Sameer Kankute	106b2f2da8	Merge pull request #27977 from BerriAI/litellm_mcp_internal_delegate_pkce ... fix(mcp): delegate PKCE bypass for internal MCP servers	2026-05-15 22:57:48 +05:30
Sameer Kankute	c2efe9e422	fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs (#27912) ... * fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs Vertex batch jobs recorded 0 spend and 0 tokens after PR #25627 added automatic transformation of GCS predictions.jsonl to OpenAI format. Two bugs fixed: 1. batch_utils.py: the Vertex-specific cost/usage reader (calculate_vertex_ai_batch_cost_and_usage) was always invoked and reads raw usageMetadata fields that no longer exist in the OpenAI-shaped output. Now the reader is only used when disable_vertex_batch_output_transformation=True; otherwise the generic path handles the already-transformed OpenAI-shaped content. 2. cost_calculator.py: batch_cost_calculator skipped the global litellm.get_model_info() lookup when a model_info dict was passed in, even when that dict had no pricing fields (e.g. deployment metadata with only id/db_model). It now falls back to the global pricing table when the provided model_info has no pricing data. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm/cost_calculator.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(cost-calculator): use not-any guard for pricing fallback in batch_cost_calculator Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cost-calculator): treat explicit zero batch pricing as set in model_info The fallback to litellm.get_model_info() used truthy checks on pricing fields, so 0.0 was treated as missing and replaced by global rates. Use `is not None` like elsewhere in cost calculation. Add regression test. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>	2026-05-15 04:47:02 -07:00
Sameer Kankute	fe755ee02a	feat(proxy): fix vector store retrieve/list/update/delete without model (#27929) ... * feat(proxy): fix vector store retrieve/list/update/delete routing without model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): remove unchecked query-param injection in vector store management endpoints Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): use subset assertion for vector store route test to allow extra kwargs like shared_session Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-15 04:37:59 -07:00
Sameer Kankute	4e2b2d9d1f	fix(mcp): expose delegate_auth_to_upstream in MCP server list rows (#27936) ... _build_mcp_server_table omitted delegate_auth_to_upstream, so GET /v1/mcp/server always returned the default false while the registry kept the DB value. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-15 04:32:14 -07:00
Sameer Kankute	d855e56333	chore(mcp): warn on internal + upstream PKCE delegate ... Log verbose_logger.warning when loading oauth2 interactive servers with available_on_public_internet=false and delegate_auth_to_upstream=true (config + DB). Dashboard Alert for the same combo. CLAUDE note for operators. Tests for log and M2M skip.	2026-05-15 10:05:35 +05:30
Sameer Kankute	5aabfccf57	fix(mcp): allow delegate PKCE bypass for internal MCP servers ... Remove available_on_public_internet gating from delegate-auth-to-upstream paths so oauth2 + delegate_auth_to_upstream interactive servers behave the same when marked internal. Keeps M2M exclusion. Updates tests.	2026-05-15 08:32:09 +05:30
Dennis Henry	9b6ab55c5f	fix: allow for allowlisted redirect URIs (#27761) ... * fix: allow for allowlisted redirect URIs * github comment addressing * Update litellm/proxy/_experimental/mcp_server/oauth_utils.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * harden oauth wildcard further * test: cover wildcard entry with dot-leading suffix rejection --------- Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>	2026-05-14 11:19:30 -07:00
Mateo Wang	7a462a4220	fix(rate-limit): stop v3 limiter from leaking internal stash to provider body (#27913) ... * fix(rate-limit): stop v3 limiter from leaking internal stash to provider body PR #27001 (atomic TPM rate limit) introduced a reservation flow that writes four LiteLLM-internal keys onto the request data dict: _litellm_rate_limit_descriptors _litellm_tpm_reserved_tokens _litellm_tpm_reserved_model _litellm_tpm_reserved_scopes _litellm_tpm_reservation_released These keys are forwarded as request body params to the upstream provider, which rejects them as unknown fields: OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors' (mapped by litellm to RateLimitError / 429, hiding the bug behind a misleading 'throttling_error' code) Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are not permitted' Net effect: every chat completion against any real provider fails the moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check itself still runs (raises 429 on over-limit), but the success path poisons the upstream body. Reproduced on litellm_internal_staging HEAD (410ce761dc) against gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request fails with the provider's unknown-field error. Fix: the stash is metadata only. - Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS registry so we have a single source of truth for stash keys. - New helper _stash_value_in_metadata_channels writes to data['metadata'] / data['litellm_metadata'] without touching the top level. - _stash_reservation_in_data and the descriptor stash now route through that helper. _mark_reservation_released stops writing top-level. - _lookup_stashed_value also checks kwargs['metadata'] / kwargs['litellm_metadata'] (raw request_data shape) in addition to kwargs['litellm_params']['metadata'] (completion kwargs shape). - async_post_call_failure_hook now reads descriptors via the unified metadata lookup instead of request_data.get(top-level). - Defense in depth: async_pre_call_hook strips any stash key that somehow surfaced at the top level (stale cache, future refactor, test fixture) before returning. Tests: - New regression test asserts no _litellm_* stash key is present at the top level of data after async_pre_call_hook, and that the metadata channel still carries the reservation + descriptors so success / failure reconciliation works. - Existing test_tpm_concurrent.py tests that asserted top-level presence are updated to read from data['metadata'] — the location is an implementation detail; the spec is that post-call callbacks can resolve the stash. Verified end-to-end against OpenAI gpt-4o-mini and Anthropic claude-haiku-4-5 via /v1/chat/completions on a low-rpm key: - With limits not exceeded: HTTP 200, valid completion response, no leaked fields in body. - With RPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: requests'). - With TPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: tokens'). Full v3 hook test suite passes (171 tests). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments Address greptile P2: test fixture now uses the imported constant. Drop comments that re-explain what well-named identifiers already convey. * fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at the start of async_pre_call_hook. Without this, an authenticated caller can inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in body metadata, trigger a proxy-side rejection, and cause async_post_call_failure_hook to refund TPM counters against attacker-named scopes (e.g. another tenant's api_key). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-05-14 10:53:04 -07:00
Yassin Kortam	a6494e6fe3	perf: eliminate per-request callback scanning on proxy hot path (#27858) ... - Introduce `_CallbackCapabilities` dataclass and `ProxyLogging._callback_capabilities()` static method that inspects `litellm.callbacks` once and caches capability flags keyed on (list length, member ids); invalidates automatically when the callback list mutates without per-request iteration overhead - Replace O(n) `litellm.callbacks` walks in `async_pre_call_hook`, `during_call_hook`, `async_post_call_streaming_iterator_hook`, `async_post_call_streaming_hook`, and `post_call_response_headers_hook` with fast-path exits when no relevant callbacks are registered - Add `needs_iterator_wrap()` and `needs_per_chunk_streaming_hook()` instance methods to decouple iterator-level wrapping from per-chunk hook execution; avoids `get_response_string` materialization per chunk when no guardrail or chunk-hook callback is active - Introduce `_fast_serialize_simple_model_response_stream()` using `orjson` for common single-choice text streaming chunks, bypassing the full Pydantic serializer; falls back to `model_dump_json` for tool calls, logprobs, usage, and provider-specific fields - Add early-return in `_restamp_streaming_chunk_model` when downstream model already matches the requested model, avoiding unnecessary string comparisons on every chunk - Fix stale zero-cost cache bug in `_is_model_cost_zero`: move the per-router `_zero_cost_cache` dict onto the `Router` instance and clear it in `_invalidate_model_group_info_cache` so in-place pricing updates via `upsert_deployment` immediately resume budget enforcement - Add `scripts/benchmark_chat_completions_perf.py`: standalone async benchmarking tool with a mock OpenAI provider, LiteLLM proxy process management, non-streaming RPS, streaming TTFT, and full-stream latency measurements with repeat/median run support - Add comprehensive unit tests covering capability detection, cache invalidation, fast-path correctness, zero-cost cache regression, and the no-callback streaming fast path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>	2026-05-14 09:28:31 -07:00
vladpolevoi	65d6ad82ef	feat(lasso): add tool-calling support to LassoGuardrail (#27648) ... * feat(lasso): extend LassoGuardrail to support tool calling (RND-5748) * fix(lasso): PR review followups for tool-calling guardrail (RND-5748) * fix(lasso): handle object-style tool_calls in _update_tool_calls_from_masked (RND-5748) * fix(lasso): use model role for tool_use blocks (RND-5748) * test(lasso): add round-trip tests for message transformation (RND-5748) * fix(lasso): remove unused imports, handle Responses-API input masking, flatten multimodal content (RND-5748) * fix(lasso): inspect Responses-API input field (RND-5748) * fix(lasso): guard text-cursor remap against Lasso count mismatch (RND-5748) * fix(lasso): flatten list content in tool_result.content (RND-5748) * fix(lasso): remap multimodal list content during masking (RND-5748) Bug: _map_masked_messages_back counted list-content messages in original_text_count but the remap loop only handled isinstance(str). The positional text_cursor never advanced for list messages, causing all subsequent masked texts to be written onto the wrong messages. Fix: added elif isinstance(content, list) branch that replaces the list with the masked text string and advances the cursor — mirrors the existing string-content branch. Also handles the assistant + tool_calls combo for list-content messages. Test: test_map_masked_messages_back_list_content verifies a user message with [text + image_url] followed by an assistant message gets correct masked content on both (cursor stays aligned). * refactor(lasso): extract _get_field and _extract_tool_call_fields helpers (RND-5748) The dict-vs-object access pattern (x.get('y') if isinstance(x, dict) else getattr(x, 'y', None)) was duplicated 14 times across 5 methods. _get_field(obj, field) — single-point dict/Pydantic field access. _extract_tool_call_fields(call) — returns (call_id, name, parsed_input) with JSON argument parsing, replacing ~30 duplicate lines in both async_post_call_success_hook and _expand_messages_for_classification. Also simplified _update_tool_calls_from_masked, _prepare_payload tool mapping, and _apply_masking_to_model_response call_id extraction. Net ~60 lines removed. No behavior change — all 32 tests pass. * fix(lasso): add count guard to _apply_masking_to_model_response (RND-5748) _apply_masking_to_model_response used a bare text_cursor without verifying 1:1 correspondence between text-bearing choices and masked text entries. If Lasso returned a different number of text messages than choices with content, masked text would be applied to the wrong choice or silently skip choices. Added the same count-mismatch guard pattern already used in _map_masked_messages_back: count original text-bearing choices, compare to masked_text length, skip text remap on mismatch with a warning log. Tool_call masking via id-based lookup is unaffected. Tests: - test_apply_masking_to_model_response_multiple_choices: verifies correct per-choice masked text with 2 choices - test_apply_masking_to_model_response_count_mismatch: verifies content is left unchanged when counts disagree * fix(lasso): close two guardrail-bypass paths flagged in review (RND-5748) * tool-call args: when function.arguments is malformed JSON or parses to a non-object, preserve the raw string as {"arguments": <raw>} so Lasso still inspects it instead of receiving input=None. Covers both pre-call and post-call extraction (shared helper). Also resolves the CodeQL empty-except warning since the except body now assigns parsed=None. * Responses-API input: when a request carries both "messages" and "input", inspect both. Previously a benign messages array let the guardrail skip data["input"] entirely. The masking write-back is split via a count boundary so masked messages flow back to data["messages"] and masked input flows back to data["input"] without cross-contamination. Tests: malformed/non-object args round-trip, dual-field classification, dual-field masking write-back split. * chore(lasso): black formatting + comment on expand skip branch (RND-5748) * black: wrap two long expressions in lasso.py and reformat dict literals in test_lasso.py to satisfy CI lint. * add a short comment in _expand_messages_for_classification explaining why empty string and None content are intentionally skipped (None is the OpenAI shape for a pure tool-call turn). * fix(lasso): satisfy mypy in _handle_masking, _update_tool_calls_from_masked, _apply_masking_to_model_response (RND-5748) * Narrow `response.get("messages")` into a local before slicing so mypy doesn't see `Optional[List[Dict[str, str]]]` as non-indexable. * Rename the two write-side `func` bindings in `_update_tool_calls_from_masked` to `func_dict` / `func_obj` so mypy doesn't unify the dict and Any|None branches. * Rename the inner loop variable in `_apply_masking_to_model_response` from `msg` to `masked_msg` to avoid clashing with the `msg = choice.message` rebinding below. No behavior change; resolves the 7 mypy errors from the CI lint job.	2026-05-14 08:35:24 -07:00
Kenan Yildirim	649eb2d176	fix(guardrails): improve CrowdStrike AIDR input handling (#26658)	2026-05-14 06:25:47 -07:00
Krrish Dholakia	410ce761dc	fix: tighten budget field validation and authorization checks (#27897) ... * fix: block NaN/Inf budget bypass and add missing non-admin guards Addresses three security issues: GHSA-wvg4-6222-3q4r: /user/update exposes max_budget, soft_budget, spend to self-editing non-admin users with no server-side guard. Non-admin callers now receive HTTP 403 if any of those fields appear in the update payload. GHSA-q775-qw9r-2r4g: _enforce_upperbound_key_params returned early (no-op) when upperbound_key_generate_params was absent from config, letting any authenticated user generate a key with unlimited max_budget. Fix adds a delegated-authority ceiling in _common_key_generation_helper: non-admins cannot grant a key more budget than their own key carries. GHSA-2rv4-xv66-fpjg: float('nan') passes every `value < 0` guard because nan < 0 is False in Python, and spend >= nan is always False, permanently disabling budget enforcement for any entity carrying a NaN max_budget. All write-time budget guards now use `not math.isfinite(v) or v < 0`. _enforce_upperbound_key_params validates finiteness unconditionally (before the early-return). All spend-enforcement comparisons in auth_checks.py are now guarded with math.isfinite(max_budget) as defense-in-depth. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: close budget ceiling bypass for callers with no max_budget (GHSA-q775) Non-admin callers whose API key has no explicit max_budget (None) could bypass the delegated-authority ceiling and create keys with arbitrary budgets. Now blocks budget assignment when caller has no budget configured. Also removes redundant inline import of LitellmUserRoles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: only apply budget ceiling to explicitly requested max_budget Capture the caller-supplied max_budget before _enforce_upperbound_key_params can fill it with a default, so auto-filled defaults don't trigger the ceiling guard for non-admin users with no budget on their own key. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: capture requested max_budget before any defaults are applied Move _requested_max_budget capture before both default_key_generate_params and upperbound_key_generate_params mutations, so auto-filled values don't trigger the ceiling check for non-admin users. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: allow unlimited-budget callers to delegate any budget Callers with max_budget=None (unlimited) can legitimately create budget-capped keys. Only block when caller has an explicit budget and the requested budget exceeds it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>	2026-05-14 05:41:13 +00:00
Yassin Kortam	de1747dca8	fix(spend-logs): redact echoed prompts in error_information (LIT-2992) (#27689) ... Provider validation errors (e.g. OpenAI RateLimitError carrying 178 pydantic errors each with their own 'input': [...]) were stored verbatim in LiteLLM_SpendLogs.metadata.error_information.error_message via str(original_exception), producing rows >12 MB. Sanitize before metadata is serialized: - redact 'input'/'messages' values in both error_message and traceback when store_prompts_in_spend_logs is False (back-door leak paths) - always apply the MAX_STRING_LENGTH_PROMPT_IN_DB size cap to error_message and traceback (DB-storage safeguard) Value scanning uses a parser-based balanced-bracket walk that respects string quoting, so multi-modal payloads ('messages': [{'content': [...]}]) and user text containing literal brackets ("secret[123") are handled correctly instead of leaking past a depth-1 regex. Scoped to the spend-log path so OTEL/Datadog/etc. callbacks still receive the untruncated error per LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE. Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-13 22:11:24 -07:00
Krrish Dholakia	8bbc61e03c	fix: harden /key/update authorization checks (#27878) ... * fix: patch Host-header auth bypass in get_request_route Starlette reconstructs request.url from the Host header. A malformed Host like `localhost/?x=1` causes Starlette to build the full URL as `http://localhost/?x=1/health`, which url-parses to path="/". Since "/" is in LiteLLMRoutes.public_routes, all protected routes became reachable without authentication. Fix: read scope["path"] (set by uvicorn from the HTTP request line, not derivable from headers) instead of request.url.path. Sub-path deployments are handled via scope["app_root_path"] / scope["root_path"], mirroring Starlette's own base_url construction logic. Affected variants confirmed fixed: Host: localhost/?x=1 Host: localhost:4000/?x=1 Host: localhost/#test Host: localhost:4000/#test Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * style: reduce comments in route fix Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block credential fields in RAG ingest vector_store options Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.) in ingest_options.vector_store are now rejected at the API boundary with a 400 error. Credentials must be configured server-side. Previously any authenticated user could supply a vertex_credentials dict with type=external_account pointing credential_source.file at an arbitrary path (e.g. /proc/1/environ) and token_url at an attacker-controlled server. google-auth's identity_pool.Credentials refresh() would read the file and POST its contents to the attacker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block /key/update self-escalation by assigned users Non-admin users who were assigned a key (created_by != caller) could update any non-budget field — models, rpm_limit, guardrails, etc. — without admin authorization, allowing privilege self-escalation. Gate: only the key creator (created_by == caller) may edit their own key without admin check; budget changes always require admin regardless of creator status. All other callers must pass _check_key_admin_access. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block user-controlled api_base in RAG ingest vector_store options A user-supplied api_base in ingest_options.vector_store caused the server to forward its configured provider credentials (Gemini, OpenAI) to an attacker-controlled endpoint via SSRF. Add api_base to the blocked credential params set alongside api_key and the existing credential fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check Any authenticated internal_user could POST arbitrary provider config (aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have the server forward its credentials to an attacker-controlled endpoint. - Gate the endpoint on PROXY_ADMIN role (403 for all other roles) - Call is_request_body_safe() to reject banned params even for admins - Convert ValueError from safety check to HTTP 400 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: apply banned-param check to /utils/transform_request Without is_request_body_safe(), any authenticated user could pass aws_sts_endpoint, api_base, or aws_web_identity_token to /utils/transform_request and have the server forward its configured provider credentials to an attacker-controlled endpoint during SDK credential resolution. Applies the same banned-param blocklist already used by LLM endpoints. Endpoint remains accessible to all authenticated users. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter Any frontmatter key not in ["model","input","output"] flowed into optional_params and was merged into the LLM call data dict, bypassing is_request_body_safe. An attacker with any bearer key could set api_base in YAML to redirect the outbound LLM request — including the provider API key — to an attacker-controlled host. Fix: call is_request_body_safe on the constructed data dict after optional_params are merged, before invoking ProxyBaseLLMRequestProcessing. ValueError from the banned-param check is surfaced as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Update litellm/proxy/rag_endpoints/endpoints.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix: coerce nested config strings before banned-param check _NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently skipped litellm_embedding_config when delivered as a JSON string via multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.) nested inside the stringified value were invisible to is_request_body_safe. _NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: replace substring match with prefix match in is_llm_api_route mapped_pass_through_routes used `_llm_passthrough_route in route` (substring) so any admin-only path whose URL contained a provider name (openai, anthropic, azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the admin gate in non_proxy_admin_allowed_routes_check. Confirmed live: non-admin key could GET /credentials/by_name/openai (read masked provider API key) and DELETE /credentials/openai (delete credential). Fix: use exact match or startswith(prefix + "/") — the same pattern used everywhere else in RouteChecks — so only routes that actually start with a passthrough prefix are allowed through. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: stabilize PR #27878 test failures - key_management_endpoints: extend can_skip_admin_check to team keys so team members with /key/update permission can update non-budget fields. can_team_member_execute_key_management_endpoint already validates team membership + permission and raises if unauthorized; reaching the admin check on a team key means the caller was authorized. - test: set created_by on mock key in test_update_key_non_budget_fields_allowed_for_internal_user so caller_is_creator resolves correctly (MagicMock default ≠ user_id). - auth_utils.get_request_route: guard against non-dict request.scope (e.g. MagicMock in unit tests) to prevent a MagicMock leaking into UserAPIKeyAuth.request_route and failing Pydantic validation. - ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard in test-unit-proxy-db.yml to satisfy the shard-coverage check. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(lint): add explicit str() cast in get_request_route for MyPy scope.get() returns Any|None which MyPy cannot coerce to str implicitly. Wrap both scope.get() calls in str() to satisfy the type checker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: guard bare-/ root_path strip + make total_spend migration idempotent auth_utils.get_request_route: when Starlette sets scope["app_root_path"] to "/" (e.g. behind some middleware), the old stripping logic would remove the leading slash from every path ("/team/new" → "team/new"), breaking route matching and causing auth to misclassify protected routes. Skip stripping when root_path is bare "/". migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration is safe to replay when a prior partial run already created the column. Without this guard, prisma migrate deploy fails on CI DBs that were partially migrated, causing all subsequent DB operations (including /team/new) to 500. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: require creator still owns key for personal-key bypass in /key/update caller_is_creator now requires both created_by == caller AND user_id == caller. Previously checking only created_by let a demoted admin who originally created a key for another user continue editing non-budget fields on it after reassignment, bypassing _check_key_admin_access. Adds regression test: creator whose key was reassigned is blocked (403). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: extract auth checks to fix PLR0915 + broaden max_budget assertion internal_user_endpoints._update_single_user_helper exceeded 50 statements (PLR0915). Extract authorization checks into _check_user_update_authz helper to bring statement count under the limit. test_validate_max_budget: assert "negative" (substring of both the local "cannot be negative" and the CI "non-negative finite number" messages) so the test is stable regardless of which exact wording the function uses. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>	2026-05-14 04:16:04 +00:00
yuneng-jiang	0c4982042a	Merge pull request #27892 from BerriAI/worktree-fix-mcp-byok-oauth ... Worktree fix mcp byok oauth	2026-05-13 21:06:25 -07:00
yuneng-jiang	e3e5209f51	Merge pull request #27801 from stuxf/chore/get-instance-fn-runtime-s3-gate ... chore(proxy): refuse remote-URL instance-fn loads outside config-file path	2026-05-13 20:53:54 -07:00
user	626b768d25	chore(tests): drop redundant membership check; trim test comment ... ``test_azure_ad_token_is_in_banned_list`` only asserted tuple membership of a name the parametrized test already exercises end-to-end through ``is_request_body_safe``. Removed. Tightened the admin-opt-in test comment.	2026-05-14 03:39:15 +00:00
user	5e56022553	chore(proxy): coerce stringified nested-config containers before descent ... ``_NESTED_CONFIG_KEYS`` descent used ``isinstance(nested, dict)``, so a caller sending ``extra_body`` as a JSON-encoded string instead of an object (the same shape multipart/form-data clients use for ``litellm_metadata``) skipped the banned-key check entirely. Switched to ``_coerce_metadata_to_dict`` so the JSON-string path is parsed before descent — mirrors the existing handling on ``_NESTED_METADATA_KEYS``.	2026-05-14 03:37:14 +00:00
user	2519ac161e	chore(proxy): cover extra_body + azure_ad_token in banned-params check ... ``extra_body`` is the OpenAI-SDK passthrough container. Provider modules read provider-auth fields out of it directly (Azure's ``extra_body.azure_ad_token``, Bedrock's ``extra_body.aws_web_identity_token``, etc.) without re-validating, so the boundary check has to walk it the same way it walks ``litellm_embedding_config``. Adding it to ``_NESTED_CONFIG_KEYS`` extends single-level banned-key descent into the container — top-level admin opt-ins (``allow_client_side_credentials`` / ``configurable_clientside_auth_params``) still apply. ``azure_ad_token`` was not in ``_BANNED_REQUEST_BODY_PARAMS`` despite being the bearer-token field the Azure transformer resolves through ``get_secret`` (same shape as ``aws_web_identity_token`` on the Bedrock STS path). Added so it can't be supplied per-request without an admin opt-in.	2026-05-14 03:29:16 +00:00
Sameer Kankute	1294165768	feat(mcp): support MCP access group names in URL-based namespacing (#27726) ... * feat(mcp): support MCP access group names in URL-based namespacing Extends dynamic_mcp_route to resolve /{name}/mcp requests where {name} is an MCP access group tag or a comma-separated list of servers/groups, matching what the documentation promised but the handler did not implement. Resolution order: registered server alias → toolset → comma-separated list → single access group tag (404 if none match). Adds unit tests covering all four resolution paths plus 404 cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): address Greptile review comments on dynamic_mcp_route - Move comma-separated check before toolset DB lookup so comma names short-circuit without hitting the database - Cache access-group DB lookups via user_api_key_cache to avoid a raw find_many on every request (matches toolset caching pattern) - Remove unused response_started variable from _forward_as_mcp_path - Update tests to assert comma list skips toolset call and to mock cache Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(mcp): extract helpers to fix PLR0915 too-many-statements in dynamic_mcp_route Extract _mcp_forward_as_path and _is_mcp_access_group_cached as module-level helpers so dynamic_mcp_route stays under the 50-statement limit. Update tests to patch the new module-level symbols directly. Co-authored-by: Cursor <cursoragent@cursor.com> * Avoid caching missing MCP access groups * fix(mcp): stream MCP responses via _stream_mcp_asgi_response instead of buffering _mcp_forward_as_path previously accumulated the full response body in memory before sending it. Replace the buffering custom_send pattern with _stream_mcp_asgi_response, which uses an asyncio.Queue bridge so chunks are yielded to the client as they arrive, preventing unbounded memory growth on large or long-lived MCP responses. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): short-TTL negative cache for access-group existence lookup An unauthenticated caller could repeatedly request /<unknown>/mcp and force a fresh DB lookup for the access-group existence check on every request (only positive results were cached). Cache negative results for a short DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL window (10s by default) so the DB is shielded from flooding while a transient DB error (which surfaces as an empty list) cannot hide a real group for long. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(mcp): use plain int for access-group negative cache TTL Drop the os.getenv wrapper around DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL to avoid the documentation_test_env_keys check failing on the new variable. The negative-cache window is a small internal tuning constant, not a user-facing knob, so a plain integer is clearer than an env override. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(mcp): validate, dedupe, and cap CSV tokens in dynamic MCP route For /{name1,name2,...}/mcp, validate every token resolves to a known server alias or access group, dedupe case-insensitively, and cap at DEFAULT_MCP_NAMESPACE_CSV_MAX_TOKENS=16 before forwarding. - Bounds the per-request DB / cache fan-out an authenticated caller can trigger by stuffing the path with tokens (raised by veria-ai). - Returns 404 instead of forwarding when no token resolves, so the downstream server filter cannot silently fall back to the full allowed_mcp_servers list (raised by Cursor agentic security review). - Forwards only the resolved subset, so unknown tokens cannot ride along into the downstream filter. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(mcp): exact-match CSV token dedupe to preserve case-sensitive distinct tokens Bugbot flagged that case-insensitive dedup on `MyGroup,mygroup` could collapse to whichever case appeared first and silently drop the matching casing if the downstream resolver is case-sensitive. Switch to exact-match dedup so distinct casings survive; whitespace-only differences still collapse via the .strip() before comparison. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: mateo-berri <mateo@berri.ai> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-05-13 20:20:38 -07:00
user	aced4295bb	chore(proxy): also scrub guardrail callbacks / module paths from DB overlay ... A guardrail entry's ``callbacks`` list (v1: ``{name: {callbacks:[...]}}``, v2: ``{guardrail_name, litellm_params: {callbacks: [...], guardrail: "module.path"}}``) is iterated during config load and threaded through ``get_instance_fn``. A PROXY_ADMIN persisting ``litellm_settings.guardrails[*].callbacks: ["s3://..."]`` or ``litellm_settings.guardrails[*].litellm_params.guardrail: "s3://..."`` via ``/config/update`` was not covered by the previous scrub matrix. Walk both v1 and v2 entry shapes and null out remote-URL callbacks / module-path values before the merge. Adds four regression tests.	2026-05-14 01:24:51 +00:00
Krrish Dholakia	f1d07c13e5	fix: block SSRF fields in RAG ingest vector_store config ... aws_sts_endpoint, aws_web_identity_token, and aws_bedrock_runtime_endpoint in ingest_options.vector_store were passed directly to the Bedrock ingestion class, which reads them into boto3 STS client construction. Any authenticated caller could redirect AssumeRole calls to an attacker-controlled server, leaking the proxy's instance profile credentials. Calls is_request_body_safe() on ingest_options["vector_store"] before forwarding to litellm.aingest(). Same banned-params list and admin opt-in escape hatch (allow_client_side_credentials) as the /chat/completions path. ValueError from the safety check is caught and re-raised as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 18:10:04 -07:00
user	3aef0642a8	chore(proxy): also scrub pass_through_endpoints[].target from DB overlay ... A pass-through endpoint's ``target`` field is passed through ``create_pass_through_route`` into ``get_instance_fn`` during config load. A PROXY_ADMIN persisting ``target: "s3://attacker/m.i"`` via the DB-overlay ``pass_through_endpoints`` write path was not covered by the previous scrub matrix, so the remote module load would still reach the loader because the YAML-load chain has ``config_file_path`` set. Walk each entry in ``general_settings.pass_through_endpoints`` and null out any ``target`` that starts with ``s3://`` or ``gcs://``. The entry itself is preserved so the path-registration helper can choose how to handle a missing target (the existing code skips the route when ``target is None``). Adds two regression tests.	2026-05-14 00:32:01 +00:00
user	7575df2549	chore(proxy): scrub remote-URL module loads from DB-overlay config ... When ``ProxyConfig`` merges DB-persisted ``litellm_settings`` / ``general_settings`` on top of the YAML config, the merged dict is later iterated by ``load_config`` which threads ``config_file_path`` (the YAML path) into ``get_instance_fn``. The runtime gate that refuses ``s3://`` / ``gcs://`` modules when ``config_file_path`` is ``None`` therefore can't distinguish a YAML-sourced value from a DB-sourced one: both look the same to ``get_instance_fn``. Strip ``s3://`` / ``gcs://`` entries from the DB-overlay value for every field whose contents reach ``get_instance_fn`` during config load: - litellm_settings: ``callbacks``, ``success_callback``, ``failure_callback``, ``audit_log_callbacks``, ``post_call_rules``, ``custom_provider_map[].custom_handler`` - general_settings: ``custom_auth``, ``custom_key_generate``, ``custom_key_update``, ``custom_sso``, ``custom_ui_sso_sign_in_handler``, ``litellm_jwtauth.custom_validate`` The YAML config-file load path is unchanged — the documented operator flow (``callbacks: ["s3://bucket/module.instance"]`` in ``config.yaml``) still works. Only DB-overlay writes (e.g. via ``/config/update``) are stripped. Adds 16 regression tests covering the scrub matrix.	2026-05-14 00:15:51 +00:00
Krrish Dholakia	b95130eb32	fix: block client-side pricing injection via request body ... Authenticated clients could supply CustomPricingLiteLLMParams fields (input_cost_per_token, output_cost_per_token, etc.) in the request body. These were forwarded to register_model() in main.py, permanently mutating the shared global litellm.model_cost dict for all users on the instance. Adds all CustomPricingLiteLLMParams fields to _BANNED_REQUEST_BODY_PARAMS so is_request_body_safe() rejects them before they reach completion(). New pricing fields added to CustomPricingLiteLLMParams are auto-covered. Admin opt-in via allow_client_side_credentials or configurable_clientside_auth_params still works as before. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 17:05:36 -07:00
Sameer Kankute	bbeb094d00	Litellm agent oss staging 05 11 2026 (#27733) ... * fix(ollama): Include provider in model list for ollama (#26135) * Include provider in model names for ollama * Fix unit tests * fix(ollama): process both thinking and content in same streaming chunk (#26098) * fix(health_check): skip max_tokens for image_generation mode (#26417) * fix(health_check): skip max_tokens for image_generation mode `_update_litellm_params_for_health_check` injected `max_tokens` for every deployment. OpenAI `/v1/images/generations` strictly rejects unknown fields, so health checks for dall-e-* and gpt-image-1 always failed with `400 "Unknown parameter: 'max_tokens'"` even though the actual image endpoint calls succeed. Skip the `max_tokens` injection when `model_info.mode == "image_generation"`. `messages` still gets injected (downstream `_filter_model_params` already strips it for non-chat handlers). * Switch to allow-list with per-deployment override Per @krrishdholakia review: deny-listing image_generation only re-introduces the same bug for every other non-chat mode (embedding, audio_*, rerank, video_generation, ocr, search, moderation, ...). Replace the single image_generation skip with `_MAX_TOKEN_SUPPORT_MODES = {chat, completion, responses}`. Missing `mode` is treated as chat for backward compatibility. New modes are safe by default. Add `model_info.health_check_supports_max_tokens` as an operator escape hatch — True forces injection on a non-listed deployment (operator wants to bound probe tokens), False suppresses it on a chat-style deployment behind a strict-schema provider. Tests: parametrize over 3 chat-style + 10 non-chat modes, plus override on/off and the no-mode legacy path. * fix(http_handler): handle RequestNotRead in MaskedHTTPStatusError for multipart uploads (#26718) Squash-merged by litellm-agent from dawidkulpa's PR. * fix(ollama): guard against double 'ollama/' prefix in live model listing Greptile flagged that Ollama servers can return names that already start with 'ollama/'. Check the prefix before prepending so we don't produce 'ollama/ollama/...'. Adds a regression test. * Fix Ollama empty reasoning stream chunks Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: James Myatt <james@jamesmyatt.co.uk> Co-authored-by: VHash <225398745+vhash0@users.noreply.github.com> Co-authored-by: hayden <sewhan.kim+@a-bly.com> Co-authored-by: dawidkulpa <84176950+dawidkulpa@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-13 14:09:12 -07:00
Yuneng Jiang	d6fa08307b	fix(proxy): expose db status on public /health/readiness ... External readiness probes consumed the legacy detailed payload's `db` field to drive alerting and pod-rotation decisions. Stripping the body to `{"status": "healthy"}` broke those probes silently — the HTTP code still flipped to 503, but probes checking `body.db == "connected"` treated the response as healthy. Add `db` back to the unauthenticated payload. Keep the rest of the diagnostic fields (litellm_version, callbacks, cache, log_level) gated behind /health/readiness/details so the recon-leak gate from #26912 holds. Values match the legacy contract: "connected", "disconnected", "Not connected".	2026-05-13 13:02:38 -07:00
Sameer Kankute	18f77ff7bc	feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough (#27834) ... * feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough Adds an opt-in per-server flag that lets clients (e.g. VS Code) complete PKCE directly with an upstream OAuth2 MCP server, instead of LiteLLM double-gating with its own API-key/SSO check. Only honored when auth_type=oauth2 and the operator explicitly sets the flag; mixed-target or non-oauth2 requests fail closed. - Adds the field to Pydantic models, Prisma schema, and a migration - New MCPRequestHandler._target_servers_delegate_auth_to_upstream gate that runs only when no x-litellm-api-key is present, so authenticated users still get user_id resolution + stored-credential lookup - Anonymous callers now see delegate servers in get_allowed_mcp_servers (scoped to delegate servers only; the upstream still enforces auth) - mcp_management_endpoints: allow anonymous /authorize and /token for delegate servers so VS Code can complete PKCE without a LiteLLM session - UI toggle (shown only for oauth2) + payload/view wiring - Tests covering: oauth2 on/off, non-oauth2 with flag, mixed targets, no resolvable target, explicit key precedence, and 401 emission Co-authored-by: Cursor <cursoragent@cursor.com> * Enforce oauth2 for delegated MCP auth bypass Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): close secondary Authorization bypass for delegate servers The delegate-auth bypass gated only on the primary `x-litellm-api-key` header, so a LiteLLM key sent via `Authorization: Bearer sk-...` (the secondary header) was silently dropped — skipping spend tracking and rate limiting. Gate on the resolved litellm_api_key (which considers both headers) so the bypass fires only when neither is present. Also update the existing "Authorization header present" test to reflect that an upstream OAuth token now flows through the existing oauth2 fallback (LiteLLM auth attempt → fail → anonymous), not via the delegate branch. Co-authored-by: Cursor <cursoragent@cursor.com> * Avoid duplicate MCP OAuth credential lookup Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): block delegate bypass for M2M and internal-only servers Two security issues flagged in code review: 1. High – client_credentials (M2M) servers must not be delegatable: LiteLLM auto-fetches the upstream token using stored credentials, so allowing anonymous bypass would let any external caller invoke tools authenticated as LiteLLM's service account. Fix: check `server.has_client_credentials` in `_target_servers_delegate_auth_to_upstream`, the anonymous allow-list in `get_allowed_mcp_servers`, and `_mcp_oauth_user_api_key_auth`. 2. Medium – internal-only servers exposed to public internet: The anonymous delegate allow-list was not filtering by `available_on_public_internet`, so external callers with an upstream OAuth token could invoke tools on servers marked internal-only. Fix: add `available_on_public_internet` guard to the anonymous delegate server list in `get_allowed_mcp_servers`. Tests added for both cases. Co-authored-by: Cursor <cursoragent@cursor.com> * Require public MCP delegate auth servers Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): align delegate auth path parsing with downstream routing `_extract_target_server_names_from_path` used a naive segments-based split while `server.py::_get_mcp_servers_in_path` uses a regex that allows server names with one embedded slash and comma-separated lists. With the old parser, a request to `/mcp/<delegated>/<garbage>` was parsed as targeting `<delegated>` by the auth gate (bypassing LiteLLM auth) while the routing layer parsed it as `<delegated>/<garbage>` — when that name did not resolve, the request fell back to the anonymous allow-list, which can include `allow_all_keys` servers that normally require a LiteLLM key. Replace the parser with the same regex logic as `_get_mcp_servers_in_path` so auth gating sees the exact target name(s) downstream routing sees. Add regression tests covering parser parity and the specific extra-path-segment bypass attempt. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(mcp): close header/path TOCTOU in MCP delegate auth gate `_target_servers_delegate_auth_to_upstream` and `_target_servers_use_oauth2` trusted the `x-mcp-servers` header when present, but `server.py::extract_mcp_auth_context` overrides that header with the path-derived list for `/mcp/...` routes. An attacker could set `x-mcp-servers: <delegated>` while pointing the URL path at a non-delegate server, flipping the auth gate without changing the target downstream routing actually uses. Extract a shared `_resolve_target_server_names` helper that mirrors the downstream override (path-derived names for `/mcp/...` routes, header value otherwise). Add regression tests covering the TOCTOU attempt and the helper's path-vs-header precedence. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix delegated MCP OAuth test mock Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): drop unreachable /{server}/mcp branch in auth path parser `_extract_target_server_names_from_path` also matched the ``/{server_name}/mcp`` form, but the downstream parser ``_get_mcp_servers_in_path`` only handles ``/mcp/...`` — and ``dynamic_mcp_route`` in ``proxy_server`` rewrites ``/{name}/mcp`` to ``/mcp/{name}`` on the scope before the MCP handler runs. Parsing the un-rewritten form on the auth side was therefore unreachable in production, and contradicted the docstring's claim of mirroring the downstream parser — exactly the kind of mismatch that risks a future header/path TOCTOU if any new entry point skips the rewrite. Drop the branch; the canonical ``/mcp/...`` path matches both parsers. Update the regression test to assert the new behavior. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP path auth target resolution Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): require auth for refresh_token grants on delegate-auth servers `_mcp_oauth_user_api_key_auth` gates the unauthenticated PKCE flow for ``delegate_auth_to_upstream`` servers, but the bypass applied to BOTH ``/authorize`` and ``/token`` regardless of grant type. ``mcp_token`` accepts ``grant_type=refresh_token`` as well as ``authorization_code``, and ``exchange_token_with_server`` attaches the server's stored ``client_secret`` to whatever is forwarded upstream. An unauthenticated caller holding a refresh token issued to that OAuth client could mint fresh upstream access tokens through LiteLLM. Limit the anonymous bypass on ``/token`` to ``grant_type=authorization_code`` (the only grant PKCE actually protects via ``code_verifier``); fall through to normal LiteLLM auth for ``refresh_token`` and any other grant. ``/authorize`` continues to allow anonymous PKCE redirects. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(ui): clear delegate_auth_to_upstream when switching off oauth2 The ``delegate_auth_to_upstream`` form field is rendered inside an ``isOAuth2 && (...)`` conditional, so the Form.Item unmounts when the user changes ``auth_type`` away from ``oauth2``. The follow-up ``form.setFieldValue("delegate_auth_to_upstream", false)`` runs after the field has already deregistered, so ``onFinish`` receives ``undefined`` and the fallback ``?? mcpServer.delegate_auth_to_upstream`` preserved the old ``true``. The flag then persisted in the database for a non-oauth2 server and silently re-activated if ``auth_type`` was later switched back to ``oauth2``. In the edit payload, force the flag to ``false`` whenever ``auth_type !== oauth2``; only trust the form value (and the existing DB fallback) when the server is actually oauth2. Backend defense-in-depth already ignores the flag for non-oauth2 servers, but the DB state should stay clean too. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP delegate auth reset on edit Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude <claude@anthropic.com>	2026-05-13 12:06:13 -07:00
Sameer Kankute	466f06df6d	fix(mcp): surface upstream 401 for token-forwarding MCP servers (#27847) ... * fix(mcp): surface upstream 401 for token-forwarding MCP servers For MCP servers configured with extra_headers: [Authorization], the gateway forwards the client token directly to the upstream. When that token is rejected (expired or invalid) the upstream returns 401, but the MCP SDK starts the SSE stream with 200 OK before calling handlers, so the 401 can't be returned mid-stream. Fix: add a pre-flight httpx probe in handle_streamable_http_mcp — before the SDK opens the session — so the gateway can still return HTTP 401 with WWW-Authenticate: Bearer authorization_uri=<gateway-discovery-url> when the upstream rejects the token. The probe fails-open (returns 200) on network errors so a transient hiccup does not block valid requests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): parallelize pre-flight auth probes and use HEAD to avoid side effects - Extract forwarded_auth outside the pass-through server loop (was called N times for the same scope value) - Gather all upstream auth probes concurrently with asyncio.gather instead of sequentially; eliminates N×5 s worst-case latency - Switch probe from POST+initialize JSON-RPC body to HEAD request; HEAD carries the Authorization header so the upstream rejects invalid tokens with 401 but never allocates a session or writes an audit entry Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): use get_async_httpx_client in _probe_upstream_auth Replaces bare httpx.AsyncClient with the project-standard get_async_httpx_client(httpxSpecialProvider.MCP) to satisfy the ensure_async_clients_test code coverage check and avoid the +500 ms per-request overhead of creating a new client on every probe call. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(mcp): extract pre-flight probe into _check_passthrough_upstream_auth Moves the parallel upstream auth probe logic out of handle_streamable_http_mcp into a dedicated helper to satisfy Ruff PLR0915 (Too many statements > 50). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): gate pre-flight probes on authorized server set to prevent bypass _check_passthrough_upstream_auth was resolving user-supplied server names directly before authorization ran, letting any permitted LiteLLM key trigger an upstream HEAD probe to a server it was not allowed to use. Changes: - Call _get_allowed_mcp_servers inside the helper so only servers the caller's key is authorized for are probed. - Move the call site to after toolset scoping so the auth context is fully resolved before the probe list is built. - Thread user_api_key_auth into the helper signature (replaces the raw mcp_servers name list). Co-authored-by: Cursor <cursoragent@cursor.com> * Add async HTTP HEAD support Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): use Scope type annotation in _get_forwarded_auth_from_scope Co-authored-by: Cursor <cursoragent@cursor.com> * Fix MCP upstream auth probe method Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove unused AsyncHTTPHandler head method Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): exclude has_client_credentials servers from pre-flight auth probe _prepare_mcp_server_headers skips caller Authorization when the server uses OAuth client-credentials (M2M), but the pre-flight probe was still selecting those servers and forwarding the caller's raw token in the HEAD request. Exclude servers with has_client_credentials from the probe list to match the actual downstream header-preparation logic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): propagate upstream 403 as 403, not 401 with WWW-Authenticate Per RFC 9110, 401 means "go get new credentials." Mapping an upstream 403 to a gateway 401 causes OAuth clients to restart the authorization flow, obtain a fresh token with identical scopes, hit 403 again, and loop indefinitely. 401 from upstream → gateway 401 + WWW-Authenticate (re-authorize) 403 from upstream → gateway 403 (no WWW-Authenticate hint) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): skip auth probe when Authorization may be the LiteLLM proxy key The pre-flight upstream probe must not forward the caller's Authorization header when it could itself be the LiteLLM proxy API key. Restrict the probe to requests that supply x-litellm-api-key explicitly — only then is the Authorization header unambiguously the upstream OAuth token the caller wants forwarded. * Fix MCP ASGI HTTPException propagation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): use public AsyncHTTPHandler.post() in auth probe Use AsyncHTTPHandler.post() and catch httpx.HTTPStatusError explicitly so the 401/403 we want to surface is not silently swallowed by the broad fail-open except Exception block. Avoids reaching into the handler's private client attribute, which would silently regress to fail-open if AsyncHTTPHandler is ever refactored. * Fix MCP auth probe tests Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(mcp): add coverage for httpx.HTTPStatusError path in auth probe AsyncHTTPHandler.post() calls raise_for_status() internally, so a real upstream 401/403 lands as httpx.HTTPStatusError. Add a test that exercises that specific exception path so a regression that swallows the error in the broad fail-open except Exception would be caught. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: claude-bot <claude-bot@anthropic.com>	2026-05-13 12:03:36 -07:00
Sameer Kankute	38709ba9bb	feat(proxy): skip disable_background_health_check models on GET /health when flag set (#27716) ... * feat(proxy): skip disable_background_health_check models on GET /health when flag set Co-authored-by: Cursor <cursoragent@cursor.com> * fix comment * fix greptile comments * Fix health check fallback kwargs * Format health endpoint * Harden direct health check kwargs compatibility for monkeypatched perform_health_check Replace substring-based TypeError detection with unexpected-keyword checks and a short retry chain (full kwargs, instrumentation only, filter only, minimal) so partial stubs work regardless of which optional kwarg fails first. Add proxy unit tests for legacy three-arg stubs and single-kwarg variants. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix black --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>	2026-05-13 09:49:05 -07:00
Yuneng Jiang	83f26d17c1	Strip SERVER_ROOT_PATH before lazy-feature prefix match ... LazyFeatureMiddleware compared the raw scope path against registered prefixes (e.g. /policies), so requests under a server root path like /api/v1/policies/... never matched, the feature never loaded, and the endpoint returned 404. Strip the configured root path before matching, normalizing trailing slashes and enforcing a component boundary so /api does not falsely match /apiv2.	2026-05-12 20:43:08 -07:00
user	14a3083d45	fix(proxy): thread config_file_path into pass-through and MCP-tool YAML loaders ... The previous commit's gate broke two legitimate startup paths for operators using s3://gcs:// remote module loading from their config.yaml: - general_settings.pass_through_endpoints[].custom_handler - mcp_tools[].handler Both call sites called get_instance_fn without a config_file_path, so the new gate rejected them at startup. Thread config_file_path through: - create_pass_through_route accepts config_file_path and forwards it to get_instance_fn. add_exact_path_route, add_subpath_route, _register_pass_through_endpoint, and initialize_pass_through_endpoints accept and propagate it. - The YAML-load call site in proxy_server.load_config now passes config_file_path; the DB-overlay call site in _update_general_settings leaves it as the default None so the gate still fires on admin-written s3:// values. - MCPToolRegistry.load_tools_from_config accepts config_file_path and threads it into get_instance_fn; _init_non_llm_configs forwards it from load_config. Adds two regression tests verifying that the YAML-source callers thread the path through to get_instance_fn.	2026-05-13 03:15:59 +00:00
user	08f3c1aae8	fix(types_utils): drop opt-in env from remote-module runtime gate ... The runtime gate on s3://gcs:// loading in get_instance_fn previously allowed an opt-in via LITELLM_ALLOW_REMOTE_INSTANCE_FN_FROM_API. That env var is admin-flippable at runtime (DB-overlay environment_variables flow into os.environ), which defeats the gate's purpose, and it isn't needed for the documented operator flow: config.yaml callbacks always pass config_file_path through to the loader. Remove the helper, raise unconditionally when config_file_path is None, and drop the corresponding test for the opt-in branch.	2026-05-13 02:52:14 +00:00
yuneng-jiang	a232cd2d19	Merge pull request #27798 from BerriAI/litellm_yj_may12 ... [Infra] Merge dev branch	2026-05-12 18:37:55 -07:00
yuneng-jiang	924e8b243f	Merge pull request #27573 from BerriAI/litellm_tag_budget_header_enforcement ... fix: enforce tag budgets on x-litellm-tags header requests	2026-05-12 18:34:19 -07:00
shivam	a5944140e9	fix(proxy): parse string metadata before pre-auth tag merge ... `apply_client_tag_policy_pre_auth` overwrote string-typed metadata with `{}` before merging header tags, dropping any tags inside. A caller could send `metadata='{"tags":["over-budget"]}'` plus `x-litellm-tags: within-budget` and bypass `_tag_max_budget_check` on the body tag. Parse the string via `safe_json_loads` first so existing tags survive the merge. Also drop the empty `tests/test_litellm/proxy/credential_endpoints/` directory — the cascade-rename tests it held imported a function that was never implemented (out of scope for this PR). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 18:06:12 -07:00
user	dd6eb499e6	chore(proxy): refuse remote-URL instance-fn loads outside config-file path ... ``get_instance_fn`` previously routed any ``s3://`` / ``gcs://`` value into ``_load_instance_from_remote_storage`` regardless of how the value got there. The function ultimately calls ``spec.loader.exec_module(module)`` — Python in the proxy process. On admin-callable endpoints that accept a ``target`` / ``custom_handler`` field from the request body (e.g. ``/config/pass_through_endpoint``, custom-callback registration), that is a one-step admin-to-RCE primitive: any future privilege-escalation bug becomes immediate code execution. The documented operator flow for remote-module loading is ``litellm_settings.callbacks: ["s3://bucket/module.instance"]`` in ``config.yaml``. That path always carries the YAML's ``config_file_path`` through to ``get_instance_fn``. Use the presence of ``config_file_path`` as the discriminator: refuse remote URLs when it is absent (the request-body path) unless the operator explicitly opts back in via ``LITELLM_ALLOW_REMOTE_INSTANCE_FN_FROM_API=true``. The three success/failure/audit-log callback-loop call sites in ``proxy_server.py:load_config`` were already running inside the startup config-file load but had stopped threading ``config_file_path`` through. Pass it through so the documented ``s3://`` callback flow continues to work unchanged. Tests cover: remote URL without ``config_file_path`` raises; remote URL with the opt-in env reaches the loader; remote URL with ``config_file_path`` passes (documented startup flow); local dotted-name imports unaffected.	2026-05-13 01:05:03 +00:00
yuneng-jiang	4c3ff78742	Merge pull request #27793 from stuxf/chore/key-regenerate-ownership-rebind-guard ... chore(proxy): close /key/regenerate ownership-rebind + premium-gate bypass	2026-05-12 17:58:08 -07:00