From b7168ba6886f17c593035913ee4925ff4d47ca88 Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Thu, 21 May 2026 01:02:34 +0530 Subject: [PATCH 01/12] fix(router): use forwarded model_id for native Azure container IDs (#27921) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints Azure code-interpreter containers return provider-native IDs (cntr_ + hex) that carry no LiteLLM routing payload, so _decode_container_id returns model_id=None. The router was falling through to call the handler directly, bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for Azure deployments. Fall back to the model_id forwarded from the proxy ownership check so deployment credentials are always applied. Co-authored-by: Cursor * fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url When a deployment's api_base is the responses endpoint URL (e.g. .../openai/responses?api-version=...), AzureContainerConfig was appending /openai/containers on top of it, producing the broken path .../openai/responses/openai/containers. Azure returns 404 for that URL while the correct path is .../openai/containers. Strip any /openai/responses suffix from api_base before constructing the containers URL so the resource root is always used as the starting point. Co-authored-by: Cursor * fix(azure-containers): prefer api-version from api_base URL over deployment's api_version The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses API and is too old for the containers API, which requires 2025-04-01-preview. The responses endpoint api_base already carries the correct api-version in its query string. Extract it and use it for the containers URL, overriding the stale deployment-level version. Fixes DELETE and file-upload operations returning 404 due to wrong api-version. Co-authored-by: Cursor * fix(containers): pass params=None instead of params={} to httpx to preserve api-version httpx erases a URL's query-string when params={} (empty dict) is passed, silently stripping ?api-version=2025-04-01-preview from every container POST/DELETE request. Azure's GET endpoints tolerate a missing api-version; POST (upload) and DELETE are strict, so those returned 404. Fix: use `params or None` in container_handler._async_handle and llm_http_handler.async_container_delete_handler (and all sibling container handlers) so that an empty params dict falls back to None, leaving httpx to preserve the URL's existing query string intact. Adds a regression test that directly documents the httpx behaviour. Co-authored-by: Cursor * fix(router): remove elif model_id branch from _init_containers_api_endpoints Two reviewer findings addressed: 1. Truncated comment on the model_id fallback line — now complete. 2. Security: the elif branch that fired when container_id was absent allowed any authenticated caller to supply model_id in a POST /v1/containers body and route the request through an arbitrary deployment UUID, bypassing the model-level access checks that only validate `model`. Removed the elif branch; operations without container_id (create, list) route by the caller-supplied `model` field as before. model_id forwarding is kept only inside the container_id block, where the proxy ownership check has already validated the container before forwarding the deployment ID. Adds a regression test pinning the security boundary: no-container-id path calls original_function directly even when model_id is in kwargs. Co-authored-by: Cursor * test(containers): validate proxy-to-router model_id forwarding for managed IDs Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id to verify that get_container_forwarding_params (the proxy-side half of the Azure routing fix) correctly extracts and forwards model_id from a LiteLLM-managed encoded container ID. This closes the gap identified by Greptile P1: the previous regression test only injected model_id as a direct kwarg, validating the router in isolation. The new test exercises the actual proxy-to-router data flow through ownership.get_container_forwarding_params, confirming that kwargs["model_id"] is populated before _init_containers_api_endpoints is reached. Co-authored-by: Cursor * fix(azure-containers): tighten endpoint-path strip to endswith match Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so the suffix strip only fires when api_base actually ends with one of the endpoint-specific path suffixes. This is the more precise check greptile flagged on the original find()-based implementation. * Fix sync container handler to preserve URL query string Mirror the async path fix: pass None instead of an empty params dict so httpx does not strip the URL's existing query string (e.g. ?api-version=...), which is required for Azure container routing. Co-authored-by: Yassin Kortam * fix(azure-containers): strip trailing slash before endpoint suffix match Co-authored-by: Yassin Kortam * fix(containers): recover model_id from stored encoded id for native Azure container IDs get_container_forwarding_params previously only set model_id when the user-supplied container_id was a LiteLLM-managed encoded id. For native upstream IDs (e.g. Azure 'cntr_') the decode fails and model_id was never forwarded — making the router-side fallback in _init_containers_api_endpoints unreachable in production. Fall back to the stored 'unified_object_id' on the ownership row, which is the encoded form captured at create time when the router selected a specific deployment. Decoding that yields the deployment model_id and restores router-based credential application (api_base, api_key) for retrieve/delete and container-file operations on native IDs. Co-authored-by: Cursor --------- Co-authored-by: Cursor Co-authored-by: Claude Co-authored-by: Yassin Kortam (cherry picked from commit 7f563b25937419382e52f3139d766f8e0015be0a) --- .../llms/azure/containers/transformation.py | 41 ++- .../llms/custom_httpx/container_handler.py | 26 +- litellm/llms/custom_httpx/llm_http_handler.py | 20 +- .../proxy/container_endpoints/endpoints.py | 4 +- .../container_endpoints/handler_factory.py | 14 +- .../proxy/container_endpoints/ownership.py | 75 +++++- litellm/router.py | 10 +- .../test_azure_container_transformation.py | 251 ++++++++++++++++++ 8 files changed, 411 insertions(+), 30 deletions(-) diff --git a/litellm/llms/azure/containers/transformation.py b/litellm/llms/azure/containers/transformation.py index 586b2e379a0..cd897511585 100644 --- a/litellm/llms/azure/containers/transformation.py +++ b/litellm/llms/azure/containers/transformation.py @@ -1,9 +1,16 @@ from typing import Optional +from urllib.parse import parse_qs, urlparse, urlunparse from litellm.llms.azure.common_utils import BaseAzureLLM from litellm.llms.openai.containers.transformation import OpenAIContainerConfig from litellm.types.router import GenericLiteLLMParams +# Endpoint-specific path suffixes that may appear in a deployment's api_base +# (e.g. the responses endpoint URL is stored as api_base for Azure models). +# Strip these before building the containers URL so we always start from the +# resource root (https://resource.cognitiveservices.azure.com). +_AZURE_ENDPOINT_PATHS = ("/openai/responses",) + class AzureContainerConfig(OpenAIContainerConfig): """ @@ -27,6 +34,27 @@ class AzureContainerConfig(OpenAIContainerConfig): litellm_params=GenericLiteLLMParams(api_key=api_key), ) + @staticmethod + def _normalize_api_base(api_base: Optional[str]) -> Optional[str]: + """Strip endpoint-specific path suffixes from api_base to get the resource root.""" + if not api_base: + return api_base + parsed = urlparse(api_base) + path = parsed.path.rstrip("/") + for ep in _AZURE_ENDPOINT_PATHS: + if path.endswith(ep): + return urlunparse( + (parsed.scheme, parsed.netloc, path[: -len(ep)], "", "", "") + ) + return api_base + + @staticmethod + def _extract_api_version(api_base: Optional[str]) -> Optional[str]: + """Return the api-version query param from api_base if present.""" + if not api_base: + return None + return parse_qs(urlparse(api_base).query).get("api-version", [None])[0] + def get_complete_url( self, api_base: Optional[str], @@ -39,10 +67,19 @@ class AzureContainerConfig(OpenAIContainerConfig): {endpoint}/openai/v1/containers when api_version is 'v1', 'latest', or 'preview'; otherwise: {endpoint}/openai/containers + + The deployment's api_base may be the responses endpoint URL + (e.g. .../openai/responses?api-version=2025-04-01-preview). We + prefer the api-version embedded there over the deployment's + api_version field, which may point to an older chat API version. """ + effective_params = dict(litellm_params) + api_version_from_base = self._extract_api_version(api_base) + if api_version_from_base: + effective_params["api_version"] = api_version_from_base return BaseAzureLLM._get_base_azure_url( - api_base=api_base, - litellm_params=litellm_params, + api_base=self._normalize_api_base(api_base), + litellm_params=effective_params, route="/openai/containers", default_api_version="v1", ) diff --git a/litellm/llms/custom_httpx/container_handler.py b/litellm/llms/custom_httpx/container_handler.py index 599cd705ebf..501390d840b 100644 --- a/litellm/llms/custom_httpx/container_handler.py +++ b/litellm/llms/custom_httpx/container_handler.py @@ -257,14 +257,19 @@ class GenericContainerHandler: returns_binary = endpoint_config.get("returns_binary", False) is_multipart = endpoint_config.get("is_multipart", False) + # An empty dict passed as `params` to httpx strips any existing query + # string from the URL (e.g. ?api-version=...). Use None instead so + # httpx leaves the URL's own query string intact. + effective_params = query_params or None + try: if method == "GET": response = http_client.get( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) elif method == "DELETE": response = http_client.delete( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) elif method == "POST": if is_multipart and "file" in kwargs: @@ -272,11 +277,11 @@ class GenericContainerHandler: kwargs["file"], headers ) response = http_client.post( - url=url, headers=headers, params=query_params, files=files + url=url, headers=headers, params=effective_params, files=files ) else: response = http_client.post( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) else: raise ValueError(f"Unsupported HTTP method: {method}") @@ -376,14 +381,19 @@ class GenericContainerHandler: returns_binary = endpoint_config.get("returns_binary", False) is_multipart = endpoint_config.get("is_multipart", False) + # An empty dict passed as `params` to httpx strips any existing query + # string from the URL (e.g. ?api-version=...). Use None instead so + # httpx leaves the URL's own query string intact. + effective_params = query_params or None + try: if method == "GET": response = await http_client.get( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) elif method == "DELETE": response = await http_client.delete( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) elif method == "POST": if is_multipart and "file" in kwargs: @@ -391,11 +401,11 @@ class GenericContainerHandler: kwargs["file"], headers ) response = await http_client.post( - url=url, headers=headers, params=query_params, files=files + url=url, headers=headers, params=effective_params, files=files ) else: response = await http_client.post( - url=url, headers=headers, params=query_params + url=url, headers=headers, params=effective_params ) else: raise ValueError(f"Unsupported HTTP method: {method}") diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py index 2ff63cc2d7f..d2af0a3dd52 100644 --- a/litellm/llms/custom_httpx/llm_http_handler.py +++ b/litellm/llms/custom_httpx/llm_http_handler.py @@ -7834,7 +7834,7 @@ class BaseLLMHTTPHandler: response = sync_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_list_response( @@ -7911,7 +7911,7 @@ class BaseLLMHTTPHandler: response = await async_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_list_response( @@ -8001,7 +8001,7 @@ class BaseLLMHTTPHandler: response = sync_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_retrieve_response( @@ -8078,7 +8078,7 @@ class BaseLLMHTTPHandler: response = await async_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_retrieve_response( @@ -8168,7 +8168,7 @@ class BaseLLMHTTPHandler: response = sync_httpx_client.delete( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_delete_response( @@ -8245,7 +8245,7 @@ class BaseLLMHTTPHandler: response = await async_httpx_client.delete( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_delete_response( @@ -8341,7 +8341,7 @@ class BaseLLMHTTPHandler: response = sync_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_file_list_response( @@ -8420,7 +8420,7 @@ class BaseLLMHTTPHandler: response = await async_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_file_list_response( @@ -8508,7 +8508,7 @@ class BaseLLMHTTPHandler: response = sync_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_file_content_response( @@ -8584,7 +8584,7 @@ class BaseLLMHTTPHandler: response = await async_httpx_client.get( url=url, headers=headers, - params=params, + params=params or None, ) return container_provider_config.transform_container_file_content_response( diff --git a/litellm/proxy/container_endpoints/endpoints.py b/litellm/proxy/container_endpoints/endpoints.py index 9650604bf81..fc1f77bb684 100644 --- a/litellm/proxy/container_endpoints/endpoints.py +++ b/litellm/proxy/container_endpoints/endpoints.py @@ -328,7 +328,7 @@ async def retrieve_container( custom_llm_provider=custom_llm_provider, ) data.update( - get_container_forwarding_params( + await get_container_forwarding_params( container_id, original_container_id, custom_llm_provider, @@ -433,7 +433,7 @@ async def delete_container( custom_llm_provider=custom_llm_provider, ) data.update( - get_container_forwarding_params( + await get_container_forwarding_params( container_id, original_container_id, custom_llm_provider, diff --git a/litellm/proxy/container_endpoints/handler_factory.py b/litellm/proxy/container_endpoints/handler_factory.py index 4284cdd5d4a..7eeb11fc372 100644 --- a/litellm/proxy/container_endpoints/handler_factory.py +++ b/litellm/proxy/container_endpoints/handler_factory.py @@ -196,10 +196,12 @@ async def _process_binary_request( ) data: Dict[str, Any] = { "file_id": file_id, - **get_container_forwarding_params( - container_id=container_id, - original_container_id=original_container_id, - custom_llm_provider=resolved_provider, + **( + await get_container_forwarding_params( + container_id=container_id, + original_container_id=original_container_id, + custom_llm_provider=resolved_provider, + ) ), } processor = ProxyBaseLLMRequestProcessing(data=data) @@ -316,7 +318,7 @@ async def _process_multipart_upload_request( ) data.update( - get_container_forwarding_params( + await get_container_forwarding_params( container_id=container_id, original_container_id=original_container_id, custom_llm_provider=resolved_provider, @@ -396,7 +398,7 @@ async def _process_request( ) ) data.update( - get_container_forwarding_params( + await get_container_forwarding_params( container_id=path_params["container_id"], original_container_id=original_container_id, custom_llm_provider=resolved_provider, diff --git a/litellm/proxy/container_endpoints/ownership.py b/litellm/proxy/container_endpoints/ownership.py index 568eca523ae..57de6c4a63d 100644 --- a/litellm/proxy/container_endpoints/ownership.py +++ b/litellm/proxy/container_endpoints/ownership.py @@ -23,6 +23,13 @@ CONTAINER_OBJECT_PURPOSE = "container" _NEGATIVE_OWNER_SENTINEL = "__litellm_container_no_owner__" _CONTAINER_OWNER_CACHE = InMemoryCache(max_size_in_memory=10000, default_ttl=60) +# Caches the stored ``unified_object_id`` (the encoded container ID +# captured at create time) so ``get_container_forwarding_params`` can +# recover the deployment ``model_id`` for native upstream IDs without +# re-hitting Prisma on every retrieve/delete. +_NEGATIVE_STORED_ID_SENTINEL = "__litellm_container_no_stored_id__" +_CONTAINER_STORED_ID_CACHE = InMemoryCache(max_size_in_memory=10000, default_ttl=60) + # Per-caller-scope cache for ``GET /v1/containers`` list filtering. Without # this, every list call issues a fresh ``find_many`` against # ``litellm_managedobjecttable``. The cache key is the sorted owner-scope @@ -56,7 +63,7 @@ def decode_container_id_for_ownership( return original_container_id, custom_llm_provider -def get_container_forwarding_params( +async def get_container_forwarding_params( container_id: str, original_container_id: str, custom_llm_provider: str ) -> Dict[str, str]: params = { @@ -65,6 +72,20 @@ def get_container_forwarding_params( } decoded = ResponsesAPIRequestUtils._decode_container_id(container_id) model_id = decoded.get("model_id") + if not (isinstance(model_id, str) and model_id): + # Native upstream IDs (e.g. Azure ``cntr_``) carry no LiteLLM + # routing payload, so decoding the user-supplied id yields no + # ``model_id``. Recover it from the encoded ``unified_object_id`` + # captured on the ownership row at create time — when the router + # selected a specific deployment that ID embeds the model_id. + stored_id = await _get_stored_container_id( + original_container_id, custom_llm_provider + ) + if stored_id and stored_id != container_id: + stored_decoded = ResponsesAPIRequestUtils._decode_container_id(stored_id) + stored_model_id = stored_decoded.get("model_id") + if isinstance(stored_model_id, str) and stored_model_id: + model_id = stored_model_id if isinstance(model_id, str) and model_id: params["model_id"] = model_id return params @@ -168,6 +189,7 @@ async def record_container_owner( ) _CONTAINER_OWNER_CACHE.set_cache(model_object_id, owner) + _CONTAINER_STORED_ID_CACHE.set_cache(model_object_id, container_id) # Drop the caller's own list-cache entry so the just-created container # shows up on their next ``GET /v1/containers``. Other callers with # disjoint scope tuples have their own entries; intersecting-scope @@ -207,9 +229,60 @@ async def _get_container_owner( _CONTAINER_OWNER_CACHE.set_cache( model_object_id, owner if owner is not None else _NEGATIVE_OWNER_SENTINEL ) + stored_id = getattr(row, "unified_object_id", None) if row is not None else None + _CONTAINER_STORED_ID_CACHE.set_cache( + model_object_id, + ( + stored_id + if isinstance(stored_id, str) and stored_id + else _NEGATIVE_STORED_ID_SENTINEL + ), + ) return owner +async def _get_stored_container_id( + original_container_id: str, custom_llm_provider: str +) -> Optional[str]: + """Return the ``unified_object_id`` stored at create time, if any. + + Used by :func:`get_container_forwarding_params` to recover the + deployment ``model_id`` for native upstream container IDs: the stored + value is the encoded form produced by ``encode_container_id_in_response`` + when the router selected a specific deployment. + """ + model_object_id = _container_model_object_id( + original_container_id, custom_llm_provider + ) + + cached = _CONTAINER_STORED_ID_CACHE.get_cache(model_object_id) + if cached == _NEGATIVE_STORED_ID_SENTINEL: + return None + if isinstance(cached, str) and cached: + return cached + + prisma_client = await _get_prisma_client() + if prisma_client is None: + return None + + row = await prisma_client.db.litellm_managedobjecttable.find_first( + where={ + "model_object_id": model_object_id, + "file_purpose": CONTAINER_OBJECT_PURPOSE, + } + ) + stored_id = getattr(row, "unified_object_id", None) if row is not None else None + _CONTAINER_STORED_ID_CACHE.set_cache( + model_object_id, + ( + stored_id + if isinstance(stored_id, str) and stored_id + else _NEGATIVE_STORED_ID_SENTINEL + ), + ) + return stored_id if isinstance(stored_id, str) and stored_id else None + + async def assert_user_can_access_container( container_id: str, user_api_key_dict: UserAPIKeyAuth, diff --git a/litellm/router.py b/litellm/router.py index fac48b45fb3..6edc10ccc28 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -5587,6 +5587,7 @@ class Router: from litellm.responses.utils import ResponsesAPIRequestUtils container_id = kwargs.get("container_id") + _forwarded_model_id = kwargs.get("model_id") if isinstance(container_id, str): decoded = ResponsesAPIRequestUtils._decode_container_id(container_id) original_id = decoded.get("response_id", container_id) @@ -5595,7 +5596,14 @@ class Router: decoded_provider = decoded.get("custom_llm_provider") if decoded_provider and kwargs.get("custom_llm_provider") == "openai": kwargs["custom_llm_provider"] = decoded_provider - model_id = decoded.get("model_id") + # Fall back to the model_id forwarded by the proxy when the container_id + # is a native upstream ID (e.g. Azure hex cntr_) that carries no LiteLLM + # routing payload, so deployment credentials (api_base, api_key) are applied. + model_id = decoded.get("model_id") or ( + _forwarded_model_id.strip() + if isinstance(_forwarded_model_id, str) and _forwarded_model_id.strip() + else None + ) if model_id: kwargs["model"] = model_id return await self._ageneric_api_call_with_fallbacks( diff --git a/tests/test_litellm/containers/test_azure_container_transformation.py b/tests/test_litellm/containers/test_azure_container_transformation.py index 70181f6f03d..cdcccf7c04e 100644 --- a/tests/test_litellm/containers/test_azure_container_transformation.py +++ b/tests/test_litellm/containers/test_azure_container_transformation.py @@ -109,6 +109,31 @@ class TestAzureContainerConfig: assert "/openai/v1/containers" in url + def test_get_complete_url_strips_responses_path_and_preserves_api_version(self): + """When api_base is the responses endpoint URL, get_complete_url must: + - strip /openai/responses (no double-path) + - use the api-version from api_base query string, NOT the deployment's + older api_version (e.g. 2024-08-01-preview → containers need 2025-04-01-preview) + """ + api_base = "https://my-resource.cognitiveservices.azure.com/openai/responses?api-version=2025-04-01-preview" + + url = self.config.get_complete_url( + api_base=api_base, + litellm_params={"api_version": "2024-08-01-preview"}, + ) + + assert ( + "/openai/responses/openai/containers" not in url + ), "path must not double /openai/responses" + assert "my-resource.cognitiveservices.azure.com" in url + assert "/openai/containers" in url or "/openai/v1/containers" in url + assert ( + "2025-04-01-preview" in url + ), "must use version from api_base, not litellm_params" + assert ( + "2024-08-01-preview" not in url + ), "must not fall back to older chat api_version" + def test_get_complete_url_raises_without_api_base(self, monkeypatch): monkeypatch.delenv("AZURE_API_BASE", raising=False) monkeypatch.setattr(litellm, "api_base", None) @@ -531,6 +556,92 @@ class TestAzureContainerKnownFailureRegressions: assert qs.get("api-version") == ["v1"] assert qs.get("foo") == ["bar"] + @pytest.mark.asyncio + async def test_regression_no_container_id_does_not_use_user_supplied_model_id( + self, monkeypatch + ): + """Operations without container_id (create, list) must NOT route via + _ageneric_api_call_with_fallbacks using a caller-supplied model_id. + + Security boundary: only the path that holds a validated container_id + is trusted to fall back to the forwarded model_id. A caller setting + model_id without container_id on POST /v1/containers must not gain + access to an arbitrary deployment UUID. + """ + from litellm.router import Router + + router = Router( + model_list=[ + { + "model_name": "azure-model", + "litellm_params": { + "model": "azure/gpt-4", + "api_base": "https://my-resource.cognitiveservices.azure.com", + "api_key": "test-key", + "api_version": "2025-04-01-preview", + }, + "model_info": {"id": "deployment-uuid-123"}, + } + ] + ) + + fallback_called = {"called": False} + + async def _mock_fallback(original_function, **kwargs): + fallback_called["called"] = True + return {} + + monkeypatch.setattr(router, "_ageneric_api_call_with_fallbacks", _mock_fallback) + + original_called = {"called": False} + + async def _noop(**kwargs): + original_called["called"] = True + return {} + + # No container_id — simulates create/list; caller injects a model_id + await router._init_containers_api_endpoints( + original_function=_noop, + model_id="deployment-uuid-123", + custom_llm_provider="azure", + ) + + assert not fallback_called["called"], ( + "_ageneric_api_call_with_fallbacks must NOT be called when " + "container_id is absent, even if model_id is supplied" + ) + assert original_called["called"], "original_function must be called directly" + + def test_regression_httpx_empty_params_strips_query_string(self): + """httpx erases the URL query-string when params={} (empty dict) is passed. + + Root cause of the Azure container 404s on POST/DELETE: + _build_query_params returns {} when the endpoint has no extra params; + passing that {} as params= to httpx wiped ?api-version=2025-04-01-preview. + + Fix: every container httpx call now uses `params or None` so an empty + dict falls back to None, which tells httpx to leave the URL untouched. + """ + url = ( + "https://resource.cognitiveservices.azure.com" + "/openai/containers/cntr_123?api-version=2025-04-01-preview" + ) + client = httpx.AsyncClient() + + req_none = client.build_request("DELETE", url, params=None) + assert "api-version=2025-04-01-preview" in str(req_none.url) + + req_empty = client.build_request("DELETE", url, params={}) + assert "api-version" not in str( + req_empty.url + ), "Documents root cause: params={} strips the query string" + + effective: dict = {} + req_guarded = client.build_request("DELETE", url, params=effective or None) + assert "api-version=2025-04-01-preview" in str( + req_guarded.url + ), "`params or None` must preserve ?api-version" + def test_regression_proxy_resolves_azure_text_same_as_azure(self): """Router/proxy treat azure_text like azure for container config.""" from litellm.proxy.container_endpoints.handler_factory import ( @@ -770,3 +881,143 @@ class TestAzureContainerKnownFailureRegressions: assert captured["data"]["container_id"] == "cntr_123" assert captured["data"]["custom_llm_provider"] == "azure" assert captured["data"]["model_id"] == "model_abc123" + + @pytest.mark.asyncio + async def test_regression_get_container_forwarding_params_sets_model_id_for_managed_id( + self, + ): + """get_container_forwarding_params must extract model_id from a + LiteLLM-managed encoded container ID and include it in the forwarding + dict. This is the proxy-side half of the native-Azure-ID routing fix: + the router's _init_containers_api_endpoints reads kwargs["model_id"] + which is set here. + """ + from litellm.proxy.container_endpoints.ownership import ( + get_container_forwarding_params, + ) + + encoded_id = ResponsesAPIRequestUtils._build_container_id( + custom_llm_provider="azure", + model_id="deployment-uuid-123", + container_id="cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df", + ) + + params = await get_container_forwarding_params( + container_id=encoded_id, + original_container_id="cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df", + custom_llm_provider="azure", + ) + + assert ( + params.get("model_id") == "deployment-uuid-123" + ), "model_id must be forwarded to the router for managed container IDs" + assert params.get("container_id") == ( + "cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df" + ) + assert params.get("custom_llm_provider") == "azure" + + @pytest.mark.asyncio + async def test_regression_get_container_forwarding_params_recovers_model_id_for_native_id( + self, monkeypatch + ): + """Native Azure IDs (``cntr_``) cannot be decoded, so model_id + must be recovered from the ownership row's ``unified_object_id`` — + the encoded form captured at create time when the router selected a + specific deployment. Without this, the router-side fallback for + native IDs in ``_init_containers_api_endpoints`` is dead code. + """ + from types import SimpleNamespace + from unittest.mock import AsyncMock + + from litellm.proxy.container_endpoints import ownership + from litellm.proxy.container_endpoints.ownership import ( + get_container_forwarding_params, + ) + + native_id = "cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df" + encoded_stored_id = ResponsesAPIRequestUtils._build_container_id( + custom_llm_provider="azure", + model_id="deployment-uuid-123", + container_id=native_id, + ) + + ownership._CONTAINER_STORED_ID_CACHE.flush_cache() + ownership._CONTAINER_OWNER_CACHE.flush_cache() + + table = AsyncMock() + table.find_first.return_value = SimpleNamespace( + created_by="user-1", + file_purpose=ownership.CONTAINER_OBJECT_PURPOSE, + unified_object_id=encoded_stored_id, + ) + prisma_client = SimpleNamespace( + db=SimpleNamespace(litellm_managedobjecttable=table) + ) + monkeypatch.setattr( + ownership, + "_get_prisma_client", + AsyncMock(return_value=prisma_client), + ) + + params = await get_container_forwarding_params( + container_id=native_id, + original_container_id=native_id, + custom_llm_provider="azure", + ) + + assert params.get("model_id") == "deployment-uuid-123", ( + "model_id must be recovered from the stored unified_object_id " + "for native upstream container IDs" + ) + assert params.get("container_id") == native_id + assert params.get("custom_llm_provider") == "azure" + + @pytest.mark.asyncio + async def test_regression_native_azure_container_id_uses_forwarded_model_id( + self, monkeypatch + ): + """Native Azure container IDs (cntr_ + hex, no LiteLLM payload) must + still route through _ageneric_api_call_with_fallbacks using the + model_id forwarded from the proxy ownership check so that deployment + credentials (api_base) are applied.""" + from litellm.router import Router + + router = Router( + model_list=[ + { + "model_name": "azure-model", + "litellm_params": { + "model": "azure/gpt-4", + "api_base": "https://my-resource.cognitiveservices.azure.com", + "api_key": "test-key", + "api_version": "2025-04-01-preview", + }, + "model_info": {"id": "deployment-uuid-123"}, + } + ] + ) + + called_with: dict = {} + + async def _mock_fallback(original_function, **kwargs): + called_with.update(kwargs) + return {} + + monkeypatch.setattr(router, "_ageneric_api_call_with_fallbacks", _mock_fallback) + + native_azure_id = "cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df" + + async def _noop(**kwargs): + return {} + + await router._init_containers_api_endpoints( + original_function=_noop, + container_id=native_azure_id, + model_id="deployment-uuid-123", + custom_llm_provider="azure", + ) + + assert called_with.get("model") == "deployment-uuid-123", ( + "_ageneric_api_call_with_fallbacks must be called with the forwarded " + "model_id when the container_id carries no LiteLLM routing payload" + ) From 0423d9111d4e2931513315e91a75b91e28a42e28 Mon Sep 17 00:00:00 2001 From: Yassin Kortam Date: Wed, 20 May 2026 17:19:24 -0700 Subject: [PATCH 02/12] fix(proxy): expose Prisma idle/connect timeout + extra DB URL params (#28395) * fix(proxy): expose Prisma idle/connect timeout + extra DB URL params Operators have reported large numbers of idle Prisma connections that never get closed. The proxy already forwards `connection_limit` and `pool_timeout` to the DATABASE_URL, but had no knob for capping idle or slow connections. Add three new `general_settings` keys that thread through to the DATABASE_URL / DIRECT_URL query string: - `database_connect_timeout` -> Prisma `connect_timeout` - `database_socket_timeout` -> Prisma `socket_timeout` (the main knob for closing idle connections from the LiteLLM side) - `database_extra_connection_params` -> untyped passthrough dict for any other Prisma URL param (`pgbouncer`, `statement_cache_size`, `sslmode`, ...); keys here override LiteLLM defaults. Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a single `_build_db_connection_url_params` helper. Co-Authored-By: Claude Opus 4.7 (1M context) * Update litellm/proxy/proxy_cli.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Yassin Kortam Co-authored-by: Claude Opus 4.7 (1M context) Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> (cherry picked from commit 2f9ac77b24cba7d0d282f3afce34cbb28cc7ac28) --- litellm/proxy/_types.py | 24 ++++ litellm/proxy/proxy_cli.py | 62 +++++++--- tests/test_litellm/proxy/test_proxy_cli.py | 130 +++++++++++++++++++++ 3 files changed, 203 insertions(+), 13 deletions(-) diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index e1ccc3d181a..defa40ab08b 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -2357,6 +2357,30 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase): database_connection_timeout: Optional[float] = Field( 60, description="default timeout for a connection to the database" ) + database_connect_timeout: Optional[float] = Field( + None, + description=( + "Prisma `connect_timeout` URL param (seconds). Bounds how long the " + "engine waits to establish a new connection before failing. Defaults " + "to Prisma's built-in value when unset." + ), + ) + database_socket_timeout: Optional[float] = Field( + None, + description=( + "Prisma `socket_timeout` URL param (seconds). When set, an idle/slow " + "connection that has not produced data within this window is closed. " + "This is the main knob for capping idle DB connections from LiteLLM." + ), + ) + database_extra_connection_params: Optional[Dict[str, Any]] = Field( + None, + description=( + "Escape hatch: extra key/value pairs appended verbatim to the Prisma " + "DATABASE_URL / DIRECT_URL query string (e.g. `sslmode`, `pgbouncer`, " + "`statement_cache_size`). Keys here override any default LiteLLM sets." + ), + ) database_type: Optional[Literal["dynamo_db"]] = Field( None, description="to use dynamodb instead of postgres db" ) diff --git a/litellm/proxy/proxy_cli.py b/litellm/proxy/proxy_cli.py index 5fc8c44b2d8..37bab3a45d0 100644 --- a/litellm/proxy/proxy_cli.py +++ b/litellm/proxy/proxy_cli.py @@ -38,6 +38,35 @@ class LiteLLMDatabaseConnectionPool(Enum): database_connection_pool_timeout = 60 +def _build_db_connection_url_params( + connection_limit: int, + pool_timeout: Optional[Union[int, float]], + connect_timeout: Optional[Union[int, float]] = None, + socket_timeout: Optional[Union[int, float]] = None, + extra_params: Optional[dict] = None, +) -> dict: + """Build the Prisma DATABASE_URL query params controlling connection pool behavior. + + `connect_timeout` / `socket_timeout` map to the Prisma URL params of the same + name (https://www.prisma.io/docs/orm/overview/databases/postgresql) and are + omitted when None so Prisma's defaults apply. `extra_params` is an + untyped passthrough — keys it provides win over the named arguments above, + so it can be used to override any default we set here. + """ + params: dict = { + "connection_limit": connection_limit, + } + if pool_timeout is not None: + params["pool_timeout"] = pool_timeout + if connect_timeout is not None: + params["connect_timeout"] = connect_timeout + if socket_timeout is not None: + params["socket_timeout"] = socket_timeout + if extra_params: + params.update(extra_params) + return params + + def append_query_params(url: Optional[str], params: dict) -> str: from litellm._logging import verbose_proxy_logger @@ -807,6 +836,9 @@ def run_server( # noqa: PLR0915 db_connection_pool_limit = 100 # Starts optional due to config fallback checks; guaranteed non-None before use. db_connection_timeout: Optional[Union[int, float]] = 60 + db_connect_timeout: Optional[Union[int, float]] = None + db_socket_timeout: Optional[Union[int, float]] = None + db_extra_connection_params: Optional[dict] = None general_settings = {} ### GET DB TOKEN FOR IAM AUTH ### @@ -924,6 +956,11 @@ def run_server( # noqa: PLR0915 db_connection_timeout = ( LiteLLMDatabaseConnectionPool.database_connection_pool_timeout.value ) + db_connect_timeout = general_settings.get("database_connect_timeout") + db_socket_timeout = general_settings.get("database_socket_timeout") + db_extra_connection_params = general_settings.get( + "database_extra_connection_params" + ) if database_url and database_url.startswith("os.environ/"): original_dir = os.getcwd() # set the working directory to where this script is @@ -963,27 +1000,26 @@ def run_server( # noqa: PLR0915 try: from litellm.secret_managers.main import get_secret + connection_url_params = _build_db_connection_url_params( + connection_limit=db_connection_pool_limit, + pool_timeout=db_connection_timeout, + connect_timeout=db_connect_timeout, + socket_timeout=db_socket_timeout, + extra_params=db_extra_connection_params, + ) if os.getenv("DATABASE_URL", None) is not None: - ### add connection pool + pool timeout args - params = { - "connection_limit": db_connection_pool_limit, - "pool_timeout": db_connection_timeout, - } database_url = get_secret("DATABASE_URL", default_value=None) modified_url = append_query_params( - str(database_url) if database_url else None, params + str(database_url) if database_url else None, + connection_url_params, ) os.environ["DATABASE_URL"] = modified_url if os.getenv("DIRECT_URL", None) is not None: - ### add connection pool + pool timeout args - params = { - "connection_limit": db_connection_pool_limit, - "pool_timeout": db_connection_timeout, - } database_url = os.getenv("DIRECT_URL") - modified_url = append_query_params(database_url, params) + modified_url = append_query_params( + database_url, connection_url_params + ) os.environ["DIRECT_URL"] = modified_url - ### subprocess.run(["prisma"], capture_output=True) is_prisma_runnable = True except FileNotFoundError: diff --git a/tests/test_litellm/proxy/test_proxy_cli.py b/tests/test_litellm/proxy/test_proxy_cli.py index 46a55fc7468..2aacb0299e7 100644 --- a/tests/test_litellm/proxy/test_proxy_cli.py +++ b/tests/test_litellm/proxy/test_proxy_cli.py @@ -483,6 +483,136 @@ class TestProxyInitializationHelpers: assert appended_params["connection_limit"] == 5 assert appended_params["pool_timeout"] == expected_timeout + def test_build_db_connection_url_params_defaults(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params(connection_limit=10, pool_timeout=60) + assert params == {"connection_limit": 10, "pool_timeout": 60} + + def test_build_db_connection_url_params_omits_none_timeouts(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + connect_timeout=None, + socket_timeout=None, + ) + assert "connect_timeout" not in params + assert "socket_timeout" not in params + + def test_build_db_connection_url_params_includes_optional_timeouts(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + connect_timeout=15, + socket_timeout=120, + ) + assert params["connect_timeout"] == 15 + assert params["socket_timeout"] == 120 + + def test_build_db_connection_url_params_extras_override_defaults(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + extra_params={ + "pgbouncer": "true", + "statement_cache_size": 0, + "pool_timeout": 5, + }, + ) + assert params["pgbouncer"] == "true" + assert params["statement_cache_size"] == 0 + assert params["pool_timeout"] == 5 + + @patch("subprocess.run") + @patch("atexit.register") + @patch("litellm.proxy.db.prisma_client.PrismaManager.setup_database") + @patch( + "litellm.proxy.db.prisma_client.should_update_prisma_schema", return_value=False + ) + def test_db_connection_extra_params_forwarded_to_url( + self, + mock_should_update, + mock_setup_db, + mock_atexit_register, + mock_subprocess_run, + ): + from click.testing import CliRunner + + from litellm.proxy.proxy_cli import run_server + + runner = CliRunner() + mock_subprocess_run.return_value = MagicMock(returncode=0) + + mock_proxy_module = MagicMock( + app=MagicMock(), + ProxyConfig=MagicMock(), + KeyManagementSettings=MagicMock(), + save_worker_config=MagicMock(), + ) + mock_proxy_module.ProxyConfig.return_value.get_config = AsyncMock( + return_value={ + "general_settings": { + "database_url": "postgresql://test:test@localhost:5432/test", + "database_connect_timeout": 15, + "database_socket_timeout": 120, + "database_extra_connection_params": { + "pgbouncer": "true", + "statement_cache_size": 0, + }, + } + } + ) + + clean_env = { + k: v + for k, v in os.environ.items() + if k not in ("DATABASE_URL", "DIRECT_URL") + } + + with ( + patch.dict(os.environ, clean_env, clear=True), + patch.dict( + "sys.modules", + { + "proxy_server": mock_proxy_module, + "litellm.proxy.proxy_server": mock_proxy_module, + }, + ), + patch( + "litellm.proxy.proxy_cli.ProxyInitializationHelpers._get_default_unvicorn_init_args" + ) as mock_get_args, + patch( + "litellm.proxy.proxy_cli.append_query_params", + side_effect=lambda url, params: str(url), + ) as mock_append_query_params, + ): + mock_get_args.return_value = { + "app": "litellm.proxy.proxy_server:app", + "host": "localhost", + "port": 8000, + } + + result = runner.invoke( + run_server, + ["--local", "--config", "test-config.yaml", "--skip_server_startup"], + ) + + assert ( + result.exit_code == 0 + ), f"exit_code={result.exit_code}, output={result.output}" + mock_append_query_params.assert_called() + appended_params = mock_append_query_params.call_args.args[1] + assert appended_params["connect_timeout"] == 15 + assert appended_params["socket_timeout"] == 120 + assert appended_params["pgbouncer"] == "true" + assert appended_params["statement_cache_size"] == 0 + @patch("uvicorn.run") @patch("atexit.register") @patch("litellm.proxy.db.prisma_client.PrismaManager.setup_database") From 6344735e3e1627596932992a0ac928df3f094934 Mon Sep 17 00:00:00 2001 From: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Date: Mon, 8 Jun 2026 18:06:51 +0530 Subject: [PATCH 03/12] feat(proxy): add disable_budget_reservation general setting (#27639) (#29493) * feat(proxy): add disable_budget_reservation general setting (#27639) * feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639) * docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639) * ci: re-trigger flaky docker build (prisma generate ECONNRESET) * fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639) Provenance: #29493 landed on litellm_internal_staging inside aggregator 32c88ca74f (Litellm oss staging 080626, #29932). The granular squash 1032dd75 is no longer on any ref; its diff is content-identical to the disable_budget_reservation hunks in the aggregator, verified before this pick. (cherry picked from commit 1032dd751fe612333412fd6673083148b4bd7e09) (cherry picked from commit 32c88ca74f within litellm_internal_staging) --- litellm/proxy/_types.py | 18 ++++++ litellm/proxy/auth/user_api_key_auth.py | 12 ++++ .../proxy/auth/test_user_api_key_auth.py | 60 +++++++++++++++++++ 3 files changed, 90 insertions(+) diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index defa40ab08b..6f04afd1b1d 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -2517,6 +2517,24 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase): None, description="List of MCP server fields that must be filled in for a submission to pass standards checks (e.g. ['description', 'source_url', 'alias']).", ) + disable_budget_reservation: Optional[bool] = Field( + None, + description=( + "If True, disables the optimistic per-request budget reservation " + "introduced in v1.84.0. " + "WARNING: This weakens hard budget enforcement. Without the reservation, " + "a burst of concurrent requests from a single key can each pass the " + "read-time spend check before any of them is charged, allowing a " + "configured budget to be exceeded under high concurrency. " + "Budgets are still evaluated on every request at read time, so " + "an already-exhausted budget is still rejected. " + "Enable only if your deployment is experiencing phantom " + "BudgetExceededError responses caused by leaked reservations " + "(see GitHub issue #27639). " + "A proxy-level WARNING is logged on every request while this flag " + "is active as a reminder that hard enforcement is relaxed." + ), + ) class ConfigYAML(LiteLLMPydanticObjectBase): diff --git a/litellm/proxy/auth/user_api_key_auth.py b/litellm/proxy/auth/user_api_key_auth.py index c4ed662e3f0..8cbce1f8bdb 100644 --- a/litellm/proxy/auth/user_api_key_auth.py +++ b/litellm/proxy/auth/user_api_key_auth.py @@ -2035,6 +2035,7 @@ async def _run_centralized_common_checks( user_api_key_cache=user_api_key_cache, proxy_logging_obj=proxy_logging_obj, skip_budget_checks=skip_budget_checks, + general_settings=general_settings, ) @@ -2055,12 +2056,23 @@ async def _reserve_budget_after_common_checks( user_api_key_cache: UserApiKeyCache, proxy_logging_obj: ProxyLogging, skip_budget_checks: bool, + general_settings: dict, end_user_id: Optional[str] = None, end_user_object: Optional[LiteLLM_EndUserTable] = None, ) -> None: user_api_key_auth_obj.budget_reservation = None if skip_budget_checks: return + if general_settings.get("disable_budget_reservation") is True: + verbose_proxy_logger.warning( + "disable_budget_reservation is enabled: skipping optimistic budget " + "reservation. Budget enforcement is read-time only — concurrent " + "requests can each pass the spend check before their cost is recorded, " + "so a configured budget may be briefly exceeded under high concurrency. " + "Set disable_budget_reservation to False or remove it to restore " + "hard per-request budget enforcement." + ) + return from litellm.proxy.spend_tracking.budget_reservation import ( reserve_budget_for_request, diff --git a/tests/test_litellm/proxy/auth/test_user_api_key_auth.py b/tests/test_litellm/proxy/auth/test_user_api_key_auth.py index 442625c75a7..df29067c5bc 100644 --- a/tests/test_litellm/proxy/auth/test_user_api_key_auth.py +++ b/tests/test_litellm/proxy/auth/test_user_api_key_auth.py @@ -110,11 +110,71 @@ async def test_should_clear_stale_budget_reservation_when_budget_checks_skip(): user_api_key_cache=MagicMock(), proxy_logging_obj=MagicMock(), skip_budget_checks=True, + general_settings={}, ) assert user_api_key_auth_obj.budget_reservation is None +@pytest.mark.asyncio +async def test_disable_budget_reservation_skips_reservation(): + """#27639: general_settings.disable_budget_reservation turns off the optimistic Redis + reservation so operators hit by phantom BudgetExceededError can opt out of it.""" + user_api_key_auth_obj = UserAPIKeyAuth(token="test_token") + + with patch( + "litellm.proxy.spend_tracking.budget_reservation.reserve_budget_for_request", + new=AsyncMock(return_value={"reserved_cost": 0.5, "entries": []}), + ) as mock_reserve: + await _reserve_budget_after_common_checks( + user_api_key_auth_obj=user_api_key_auth_obj, + request_data={"model": "gpt-4o"}, + route="/v1/chat/completions", + llm_router=None, + team_object=None, + user_object=None, + prisma_client=None, + user_api_key_cache=MagicMock(), + proxy_logging_obj=MagicMock(), + skip_budget_checks=False, + general_settings={"disable_budget_reservation": True}, + ) + + mock_reserve.assert_not_called() + assert user_api_key_auth_obj.budget_reservation is None + + +@pytest.mark.asyncio +async def test_budget_reservation_runs_when_not_disabled(): + """Control for #27639: with the flag absent, the reservation still runs and is stored.""" + user_api_key_auth_obj = UserAPIKeyAuth(token="test_token") + reservation = { + "reserved_cost": 0.5, + "entries": [{"counter_key": "spend:key:test_token"}], + } + + with patch( + "litellm.proxy.spend_tracking.budget_reservation.reserve_budget_for_request", + new=AsyncMock(return_value=reservation), + ) as mock_reserve: + await _reserve_budget_after_common_checks( + user_api_key_auth_obj=user_api_key_auth_obj, + request_data={"model": "gpt-4o"}, + route="/v1/chat/completions", + llm_router=None, + team_object=None, + user_object=None, + prisma_client=None, + user_api_key_cache=MagicMock(), + proxy_logging_obj=MagicMock(), + skip_budget_checks=False, + general_settings={}, + ) + + mock_reserve.assert_awaited_once() + assert user_api_key_auth_obj.budget_reservation == reservation + + @pytest.mark.asyncio async def test_should_not_reuse_cached_key_object_for_request_state(): key_cache = DualCache() From 14948b647d0c0e1246bfcb7847f39a9e5fc310cc Mon Sep 17 00:00:00 2001 From: Yassin Kortam Date: Wed, 10 Jun 2026 16:06:01 -0700 Subject: [PATCH 04/12] fix(proxy): recover from cached-plan errors by reconnecting the Prisma client (#29983) (cherry picked from commit 3bd3951e37a0b3201eac9eb1f858d2253aff56e5) --- litellm/proxy/utils.py | 59 ++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/litellm/proxy/utils.py b/litellm/proxy/utils.py index 36fd605cf72..ebff52994c5 100644 --- a/litellm/proxy/utils.py +++ b/litellm/proxy/utils.py @@ -3115,40 +3115,49 @@ class PrismaClient: self, sql_query: str, *args ) -> Optional[dict]: """ - Execute a query with automatic fallback for PostgreSQL cached plan errors. + Execute a query, recovering once from PostgreSQL's "cached plan must not + change result type" error. - This handles the "cached plan must not change result type" error that occurs - during rolling deployments when schema changes are applied while old pods - still have cached query plans expecting the old schema. + That error surfaces during rolling deployments when a schema change + invalidates the prepared-statement plans that pooled connections still + hold. Clearing only the server-side plans with DEALLOCATE ALL makes + things worse: Prisma's query engine keeps a per-connection client-side + cache of prepared-statement names, so once the server drops a plan the + engine re-sends a name PostgreSQL no longer recognizes and the + connection breaks with `prepared statement "sN" does not exist`. With a + small pool that connection stays poisoned and every auth lookup fails. - Args: - sql_query: SQL query string to execute + Recreating the Prisma client kills the engine subprocess and drops the + server-side plans and the engine's client-side name cache together, so + the retried query is prepared fresh. We reconnect through + `attempt_db_reconnect`, which is singleflight: when a schema change + poisons every pooled connection at once, the first cached-plan error + recreates the client and the concurrent waiters reuse that single + recreate instead of racing to kill each other's fresh engine. We then + retry the identical query exactly once. - Returns: - Query result or None + The retry reuses the original query byte-for-byte. Mutating the SQL + (e.g. injecting a unique comment) would defeat PostgreSQL's plan cache, + forcing a fresh plan on every request and pegging the database CPU. - Raises: - Original exception if not a cached plan error + If the reconnect is skipped because a recent reconnect is still within + its cooldown, the retry runs against the same connection and may fail + again; the get_data backoff decorator re-runs the lookup and a later + attempt reconnects once the cooldown elapses. """ try: return await self.db.query_first(sql_query, *args) except Exception as e: - error_str = str(e) - if "cached plan must not change result type" in error_str: - # Force PostgreSQL to re-plan by invalidating the cache - # Add a unique comment to make the query different - sql_query_retry = sql_query.replace( - "SELECT", - f"SELECT /* cache_invalidated_{int(time.time() * 1000)} */", - ) - verbose_proxy_logger.warning( - "PostgreSQL cached plan error detected for token lookup, " - "retrying with fresh plan. This may occur during rolling deployments " - "when schema changes are applied." - ) - return await self.db.query_first(sql_query_retry, *args) - else: + if "cached plan must not change result type" not in str(e): raise + verbose_proxy_logger.warning( + "PostgreSQL cached plan error detected for token lookup; " + "recreating the database connection and retrying with the same " + "query. This may occur during rolling deployments when schema " + "changes are applied." + ) + await self.attempt_db_reconnect(reason="postgres_cached_plan_error") + return await self.db.query_first(sql_query, *args) @backoff.on_exception( backoff.expo, From 0fdccebc3e0071118b9653abbfc17589d9dc8b62 Mon Sep 17 00:00:00 2001 From: Yassin Kortam Date: Wed, 10 Jun 2026 16:06:32 -0700 Subject: [PATCH 05/12] feat(proxy): add option to disable server-side prepared statements for DB lookups (#29984) (cherry picked from commit dff25fef449bc3e2051ee2638e323d09d05a850a) --- litellm/proxy/_types.py | 11 ++ litellm/proxy/proxy_cli.py | 25 ++++- tests/test_litellm/proxy/test_proxy_cli.py | 121 +++++++++++++++++++++ 3 files changed, 154 insertions(+), 3 deletions(-) diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index 6f04afd1b1d..555a82603b4 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -2381,6 +2381,17 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase): "`statement_cache_size`). Keys here override any default LiteLLM sets." ), ) + database_disable_prepared_statements: Optional[bool] = Field( + None, + description=( + "Disable server-side prepared statements by setting Prisma's " + "`pgbouncer=true` URL param. Use this for pgbouncer transaction-pooling " + "deployments, or to prevent the 'cached plan must not change result " + "type' error that pooled connections hit during rolling schema " + "migrations. An explicit `pgbouncer` in `database_extra_connection_params` " + "takes precedence." + ), + ) database_type: Optional[Literal["dynamo_db"]] = Field( None, description="to use dynamodb instead of postgres db" ) diff --git a/litellm/proxy/proxy_cli.py b/litellm/proxy/proxy_cli.py index 37bab3a45d0..23fcc90d880 100644 --- a/litellm/proxy/proxy_cli.py +++ b/litellm/proxy/proxy_cli.py @@ -43,15 +43,19 @@ def _build_db_connection_url_params( pool_timeout: Optional[Union[int, float]], connect_timeout: Optional[Union[int, float]] = None, socket_timeout: Optional[Union[int, float]] = None, + disable_prepared_statements: bool = False, extra_params: Optional[dict] = None, ) -> dict: """Build the Prisma DATABASE_URL query params controlling connection pool behavior. `connect_timeout` / `socket_timeout` map to the Prisma URL params of the same name (https://www.prisma.io/docs/orm/overview/databases/postgresql) and are - omitted when None so Prisma's defaults apply. `extra_params` is an - untyped passthrough — keys it provides win over the named arguments above, - so it can be used to override any default we set here. + omitted when None so Prisma's defaults apply. `disable_prepared_statements` + sets `pgbouncer=true`, which makes Prisma stop using server-side prepared + statements (pgbouncer transaction-pool compatible; also sidesteps the + "cached plan must not change result type" error during rolling migrations). + `extra_params` is an untyped passthrough — keys it provides win over the + named arguments above, so it can be used to override any default we set here. """ params: dict = { "connection_limit": connection_limit, @@ -62,6 +66,8 @@ def _build_db_connection_url_params( params["connect_timeout"] = connect_timeout if socket_timeout is not None: params["socket_timeout"] = socket_timeout + if disable_prepared_statements: + params["pgbouncer"] = "true" if extra_params: params.update(extra_params) return params @@ -838,6 +844,7 @@ def run_server( # noqa: PLR0915 db_connection_timeout: Optional[Union[int, float]] = 60 db_connect_timeout: Optional[Union[int, float]] = None db_socket_timeout: Optional[Union[int, float]] = None + db_disable_prepared_statements: bool = False db_extra_connection_params: Optional[dict] = None general_settings = {} ### GET DB TOKEN FOR IAM AUTH ### @@ -958,6 +965,17 @@ def run_server( # noqa: PLR0915 ) db_connect_timeout = general_settings.get("database_connect_timeout") db_socket_timeout = general_settings.get("database_socket_timeout") + _disable_prepared_statements = general_settings.get( + "database_disable_prepared_statements", False + ) + if isinstance(_disable_prepared_statements, str): + from litellm.secret_managers.main import str_to_bool + + db_disable_prepared_statements = ( + str_to_bool(_disable_prepared_statements) is True + ) + else: + db_disable_prepared_statements = bool(_disable_prepared_statements) db_extra_connection_params = general_settings.get( "database_extra_connection_params" ) @@ -1005,6 +1023,7 @@ def run_server( # noqa: PLR0915 pool_timeout=db_connection_timeout, connect_timeout=db_connect_timeout, socket_timeout=db_socket_timeout, + disable_prepared_statements=db_disable_prepared_statements, extra_params=db_extra_connection_params, ) if os.getenv("DATABASE_URL", None) is not None: diff --git a/tests/test_litellm/proxy/test_proxy_cli.py b/tests/test_litellm/proxy/test_proxy_cli.py index 2aacb0299e7..651bf9caafe 100644 --- a/tests/test_litellm/proxy/test_proxy_cli.py +++ b/tests/test_litellm/proxy/test_proxy_cli.py @@ -613,6 +613,127 @@ class TestProxyInitializationHelpers: assert appended_params["pgbouncer"] == "true" assert appended_params["statement_cache_size"] == 0 + def test_build_db_connection_url_params_disable_prepared_statements(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + disable_prepared_statements=True, + ) + assert params["pgbouncer"] == "true" + + def test_build_db_connection_url_params_no_pgbouncer_by_default(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + ) + assert "pgbouncer" not in params + + def test_build_db_connection_url_params_extra_pgbouncer_overrides_flag(self): + from litellm.proxy.proxy_cli import _build_db_connection_url_params + + params = _build_db_connection_url_params( + connection_limit=10, + pool_timeout=60, + disable_prepared_statements=True, + extra_params={"pgbouncer": "false"}, + ) + assert params["pgbouncer"] == "false" + + @pytest.mark.parametrize( + "config_value, expect_pgbouncer", + [ + (True, True), + (False, False), + ("true", True), + ("false", False), + ("not-a-bool", False), + ], + ) + @patch("subprocess.run") + @patch("atexit.register") + @patch("litellm.proxy.db.prisma_client.PrismaManager.setup_database") + @patch( + "litellm.proxy.db.prisma_client.should_update_prisma_schema", return_value=False + ) + def test_disable_prepared_statements_forwarded_to_url( + self, + mock_should_update, + mock_setup_db, + mock_atexit_register, + mock_subprocess_run, + config_value, + expect_pgbouncer, + ): + from click.testing import CliRunner + + from litellm.proxy.proxy_cli import run_server + + runner = CliRunner() + mock_subprocess_run.return_value = MagicMock(returncode=0) + + mock_proxy_module = MagicMock( + app=MagicMock(), + ProxyConfig=MagicMock(), + KeyManagementSettings=MagicMock(), + save_worker_config=MagicMock(), + ) + mock_proxy_module.ProxyConfig.return_value.get_config = AsyncMock( + return_value={ + "general_settings": { + "database_url": "postgresql://test:test@localhost:5432/test", + "database_disable_prepared_statements": config_value, + } + } + ) + + clean_env = { + k: v + for k, v in os.environ.items() + if k not in ("DATABASE_URL", "DIRECT_URL") + } + + with ( + patch.dict(os.environ, clean_env, clear=True), + patch.dict( + "sys.modules", + { + "proxy_server": mock_proxy_module, + "litellm.proxy.proxy_server": mock_proxy_module, + }, + ), + patch( + "litellm.proxy.proxy_cli.ProxyInitializationHelpers._get_default_unvicorn_init_args" + ) as mock_get_args, + patch( + "litellm.proxy.proxy_cli.append_query_params", + side_effect=lambda url, params: str(url), + ) as mock_append_query_params, + ): + mock_get_args.return_value = { + "app": "litellm.proxy.proxy_server:app", + "host": "localhost", + "port": 8000, + } + + result = runner.invoke( + run_server, + ["--local", "--config", "test-config.yaml", "--skip_server_startup"], + ) + + assert ( + result.exit_code == 0 + ), f"exit_code={result.exit_code}, output={result.output}" + mock_append_query_params.assert_called() + appended_params = mock_append_query_params.call_args.args[1] + if expect_pgbouncer: + assert appended_params["pgbouncer"] == "true" + else: + assert "pgbouncer" not in appended_params + @patch("uvicorn.run") @patch("atexit.register") @patch("litellm.proxy.db.prisma_client.PrismaManager.setup_database") From 7193461559b137150896d0c64d1bbaeead90cca8 Mon Sep 17 00:00:00 2001 From: Yassin Kortam Date: Wed, 10 Jun 2026 16:48:11 -0700 Subject: [PATCH 06/12] fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986) (cherry picked from commit da9d64b4de4b6927d3496f89fa402490a98bfb10) --- litellm/proxy/auth/auth_exception_handler.py | 10 + litellm/proxy/db/exception_handler.py | 86 ++++++++ .../proxy/db/test_exception_handler.py | 195 ++++++++++++++++++ 3 files changed, 291 insertions(+) diff --git a/litellm/proxy/auth/auth_exception_handler.py b/litellm/proxy/auth/auth_exception_handler.py index 431db4254eb..f276a1350f1 100644 --- a/litellm/proxy/auth/auth_exception_handler.py +++ b/litellm/proxy/auth/auth_exception_handler.py @@ -134,6 +134,16 @@ class UserAPIKeyAuthExceptionHandler: ) elif isinstance(e, ProxyException): raise e + if PrismaDBExceptionHandler.is_database_service_unavailable_error(e): + raise ProxyException( + message=( + "Service Unavailable, the authentication database is " + "temporarily unreachable. Please retry shortly." + ), + type=ProxyErrorTypes.no_db_connection, + param="None", + code=status.HTTP_503_SERVICE_UNAVAILABLE, + ) raise ProxyException( message="Authentication Error, " + str(e), type=ProxyErrorTypes.auth_error, diff --git a/litellm/proxy/db/exception_handler.py b/litellm/proxy/db/exception_handler.py index ab9d341aa51..c500e727595 100644 --- a/litellm/proxy/db/exception_handler.py +++ b/litellm/proxy/db/exception_handler.py @@ -109,6 +109,92 @@ class PrismaDBExceptionHandler: return True return False + @staticmethod + def is_prisma_engine_internal_error(e: Exception) -> bool: + """True iff ``e`` is a non-``PrismaError`` exception raised from inside + prisma-client-py's query-engine layer. + + During the instant a DB connection is torn down, the query engine can + return a malformed error payload (``user_facing_error.meta`` is + ``null``). prisma-client-py's ``handle_response_errors`` then crashes + with ``AttributeError: 'NoneType' object has no attribute 'get'`` + before it can raise the proper P1001 "can't reach database server" + error. That AttributeError carries no connection keyword, so it can't + be matched by message; identify it by its ``prisma.engine`` origin + instead. + + Recognized ``PrismaError`` subclasses are excluded: connectivity ones + are already classified by type/keyword above, and data-layer ones + (the DB IS reachable) must stay 401. + """ + import prisma + + if isinstance(e, prisma.errors.PrismaError): + return False + tb = getattr(e, "__traceback__", None) + while tb is not None: + if tb.tb_frame.f_globals.get("__name__", "").startswith("prisma.engine"): + return True + tb = tb.tb_next + return False + + @staticmethod + def is_database_service_unavailable_error(e: Exception) -> bool: + """True iff the exception means the database could not answer at the + infrastructure level (connection refused, socket/interface failure, + timeout) rather than a genuine auth failure (key not found) or a + data-layer error (the DB IS reachable and rejected the data). + + Auth must answer 401 only for a key the DB confirms is invalid. When + the DB itself is unreachable, the request has to surface as 503 so + callers retry instead of treating valid keys as invalid during an + outage. + + Note: prisma-client-py mislabels the P1001 "can't reach database + server" connectivity failure as a ``DataError`` (a data-layer type), + so a type-only check misses real outages. ``is_database_transport_error`` + keyword-matches the connection message and catches that masquerade, + while genuine data errors (no connection keyword) correctly stay 401. + + The Postgres "cached plan must not change result type" error is matched + here, not in ``is_database_transport_error``: it is a transient stale-DB- + state condition (not an invalid key), but the connection is healthy so it + must not trigger a reconnect. + + A non-``PrismaError`` raised from inside the prisma query engine (e.g. + the ``AttributeError`` from ``handle_response_errors`` when the engine + returns a malformed error payload mid-tear-down) is also treated as + unavailable; see ``is_prisma_engine_internal_error``. + """ + import asyncio + + if PrismaDBExceptionHandler.is_database_connection_error(e): + return True + if PrismaDBExceptionHandler.is_database_transport_error(e): + return True + if PrismaDBExceptionHandler.is_prisma_engine_internal_error(e): + return True + if "cached plan must not change result type" in str(e).lower(): + return True + + # OSError already covers ConnectionError and (Py3.3+) TimeoutError. + # asyncio.TimeoutError is a distinct class before Py3.11. + if isinstance(e, (OSError, asyncio.TimeoutError)): + return True + + try: + import asyncpg + except ImportError: + return False + + return isinstance( + e, + ( + asyncpg.exceptions.PostgresConnectionError, + asyncpg.exceptions.InterfaceError, + ), + ) + @staticmethod def handle_db_exception(e: Exception): """ diff --git a/tests/test_litellm/proxy/db/test_exception_handler.py b/tests/test_litellm/proxy/db/test_exception_handler.py index 9dcf5df4aeb..6021c221426 100644 --- a/tests/test_litellm/proxy/db/test_exception_handler.py +++ b/tests/test_litellm/proxy/db/test_exception_handler.py @@ -107,6 +107,201 @@ def test_is_database_connection_generic_errors(): ) +@pytest.mark.parametrize( + "error", + [ + ConnectionError("connection refused"), + TimeoutError("timed out"), + OSError("network is unreachable"), + asyncio.TimeoutError(), + HTTPClientClosedError(), + ClientNotConnectedError(), + PrismaError("can't reach database server"), + PrismaError(), + ], +) +def test_is_database_service_unavailable_error_infra_failures(error): + """Infrastructure-level failures (socket/connection/timeout, prisma + transport, unknown PrismaError) mean the DB could not answer, so auth + must surface 503 instead of treating a valid key as invalid.""" + assert PrismaDBExceptionHandler.is_database_service_unavailable_error(error) is True + + +def test_is_database_service_unavailable_error_prisma_p1001_masquerades_as_dataerror(): + """Real-world regression: prisma-client-py raises the P1001 "can't reach + database server" connectivity failure as a DataError (a data-layer type). + A type-only check would miss it and return 401 during a genuine outage; + the message keyword must still classify it as service-unavailable -> 503.""" + p1001_as_dataerror = DataError( + data={ + "user_facing_error": { + "message": "Can't reach database server at `127.0.0.1`:`5499`", + "meta": {"table": "t"}, + } + } + ) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error( + p1001_as_dataerror + ) + is True + ) + + +def test_is_database_service_unavailable_error_cached_plan_escapes_as_503(): + """Composes with the cached-plan retry: when that recovery fails and the + Postgres "cached plan must not change result type" error escapes (raised by + prisma as a data-layer RawQueryError), it is a transient stale-DB-state + condition, not an invalid key, so it must classify as service-unavailable + -> 503 rather than fall through to 401.""" + cached_plan_error = RawQueryError( + data={ + "user_facing_error": { + "message": "cached plan must not change result type", + "meta": {"table": "t"}, + } + } + ) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error( + cached_plan_error + ) + is True + ) + + +def test_is_database_service_unavailable_error_prisma_engine_malformed_payload(): + """Real-world regression: at the instant the DB socket drops, the prisma + query engine returns a malformed error payload (``user_facing_error.meta`` + is ``null``). prisma-client-py's ``handle_response_errors`` then crashes + with ``AttributeError: 'NoneType' object has no attribute 'get'`` before it + can raise the proper P1001 error. That bare AttributeError has no + connection keyword, so without the prisma-engine-origin check it falls + through to 401 on the first request of an outage. Reproduce the exact + prisma crash and assert it classifies as service-unavailable -> 503.""" + from prisma.engine import utils as prisma_engine_utils + + malformed_payload = [ + { + "error": "Can't reach database server", + "user_facing_error": { + "error_code": "P1001", + "message": "Can't reach database server at `localhost`:`5503`", + "meta": None, + }, + } + ] + with pytest.raises(AttributeError) as exc_info: + prisma_engine_utils.handle_response_errors(None, malformed_payload) + + assert "no attribute 'get'" in str(exc_info.value) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error(exc_info.value) + is True + ) + + +def test_is_prisma_engine_internal_error_excludes_application_attributeerror(): + """The prisma-engine-origin check must stay narrow: a genuine AttributeError + raised by application code (a real bug) must NOT be classified as + service-unavailable, otherwise real bugs would silently become 503s.""" + + def application_bug(): + none_value = None + return none_value.get("oops") + + with pytest.raises(AttributeError) as exc_info: + application_bug() + + assert ( + PrismaDBExceptionHandler.is_prisma_engine_internal_error(exc_info.value) + is False + ) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error(exc_info.value) + is False + ) + + +def test_is_prisma_engine_internal_error_excludes_data_layer_prisma_error(): + """A data-layer ``PrismaError`` (the DB IS reachable and rejected the data) + must stay 401. These are always raised from prisma internals, so the check + excludes any ``PrismaError`` by type before inspecting the traceback.""" + data_layer_error = UniqueViolationError( + data={"user_facing_error": {"meta": {"table": "t"}}} + ) + try: + raise data_layer_error + except UniqueViolationError as e: + assert PrismaDBExceptionHandler.is_prisma_engine_internal_error(e) is False + + +@pytest.mark.parametrize( + "error", + [ + DataError(data={"user_facing_error": {"meta": {"table": "t"}}}), + UniqueViolationError(data={"user_facing_error": {"meta": {"table": "t"}}}), + RecordNotFoundError(data={"user_facing_error": {"meta": {"table": "t"}}}), + Exception("some unrelated error"), + ValueError("bad value"), + ], +) +def test_is_database_service_unavailable_error_excludes_non_infra(error): + """Data-layer errors (the DB IS reachable and answered) and generic + non-DB errors must NOT be classified as service-unavailable, otherwise a + genuine 401 would be masked as a transient 503.""" + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error(error) is False + ) + + +def test_is_database_service_unavailable_error_asyncpg(monkeypatch): + """asyncpg connection/interface errors map to service-unavailable. asyncpg + is not a hard dependency, so inject a stand-in module to exercise the + branch deterministically regardless of the install environment.""" + import sys + import types + + fake_asyncpg = types.ModuleType("asyncpg") + fake_exceptions = types.ModuleType("asyncpg.exceptions") + + class PostgresConnectionError(Exception): + pass + + class InterfaceError(Exception): + pass + + class UniqueViolationError(Exception): # data-layer, must stay False + pass + + fake_exceptions.PostgresConnectionError = PostgresConnectionError + fake_exceptions.InterfaceError = InterfaceError + fake_exceptions.UniqueViolationError = UniqueViolationError + fake_asyncpg.exceptions = fake_exceptions + + monkeypatch.setitem(sys.modules, "asyncpg", fake_asyncpg) + monkeypatch.setitem(sys.modules, "asyncpg.exceptions", fake_exceptions) + + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error( + PostgresConnectionError("connection reset") + ) + is True + ) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error( + InterfaceError("connection was closed") + ) + is True + ) + assert ( + PrismaDBExceptionHandler.is_database_service_unavailable_error( + UniqueViolationError("duplicate key") + ) + is False + ) + + # Test should_allow_request_on_db_unavailable method @patch( "litellm.proxy.proxy_server.general_settings", From 45d467cb0c3e4f788b81782f6d862fee5560f76e Mon Sep 17 00:00:00 2001 From: Yassin Kortam Date: Thu, 11 Jun 2026 14:26:55 -0700 Subject: [PATCH 07/12] fix(passthrough): resolve costing model when body model is unknown (#30160) (cherry picked from commit 1828a7c6f03528b4c174d983bf29d1ca6299439b) --- .../anthropic_passthrough_logging_handler.py | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py index 82260eb2ae5..ffef4a2a865 100644 --- a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py +++ b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py @@ -100,6 +100,42 @@ class AnthropicPassthroughLoggingHandler: return get_end_user_id_from_request_body(request_body) return None + @staticmethod + def _resolve_costing_model(model: str, logging_obj: LiteLLMLoggingObj) -> str: + if model and model != "unknown": + return model + litellm_params = (getattr(logging_obj, "model_call_details", {}) or {}).get( + "litellm_params", {} + ) or {} + deployment_model = litellm_params.get("model") + if deployment_model and deployment_model != "unknown": + return deployment_model + model_group = (litellm_params.get("metadata", {}) or {}).get("model_group") + if model_group: + return model_group.removeprefix("passthrough/") + return model + + @staticmethod + def _extract_model_from_anthropic_chunks( + all_chunks: Sequence[Union[str, bytes]], + ) -> Optional[str]: + for raw in all_chunks: + text = raw.decode("utf-8") if isinstance(raw, bytes) else raw + for line in text.splitlines(): + if not line.startswith("data:"): + continue + try: + data = json.loads(line[len("data:") :].strip()) + except (json.JSONDecodeError, ValueError): + continue + if not isinstance(data, dict): + continue + if data.get("type") == "message_start": + model = (data.get("message") or {}).get("model") + if model: + return model + return None + @staticmethod def _create_anthropic_response_logging_payload( litellm_model_response: Union[ModelResponse, TextCompletionResponse], @@ -127,6 +163,10 @@ class AnthropicPassthroughLoggingHandler: "custom_llm_provider" ) + model = AnthropicPassthroughLoggingHandler._resolve_costing_model( + model, logging_obj + ) + # Prepend custom_llm_provider to model if not already present model_for_cost = model if custom_llm_provider and not model.startswith(f"{custom_llm_provider}/"): @@ -213,6 +253,15 @@ class AnthropicPassthroughLoggingHandler: ): model = cast(str, litellm_logging_obj.model_call_details.get("model")) + if not model or model == "unknown": + chunk_model = ( + AnthropicPassthroughLoggingHandler._extract_model_from_anthropic_chunks( + all_chunks + ) + ) + if chunk_model: + model = chunk_model + complete_streaming_response = ( AnthropicPassthroughLoggingHandler._build_complete_streaming_response( all_chunks=all_chunks, From 5a84642f5410f6a44de5bb3502d9098a671d076e Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Fri, 12 Jun 2026 16:45:31 -0700 Subject: [PATCH 08/12] fix(passthrough): skip [DONE] sentinels and non-JSON SSE frames in Anthropic streaming logging Targeted subset of staging commit cfcdf8714a6968f8248e56f00f37f6f232aa41cd (#30202): only the anthropic_passthrough_logging_handler.py hardening hunks and their four tests are taken; the rest of that staging batch is intentionally excluded. (cherry picked from commit cfcdf8714a6968f8248e56f00f37f6f232aa41cd) (cherry picked from commit 973c7eb8d62b76f1f2724e2ac0ce8b64637089cc) --- .../anthropic_passthrough_logging_handler.py | 15 +++++ ...t_anthropic_passthrough_logging_handler.py | 66 +++++++++++++++++++ 2 files changed, 81 insertions(+) diff --git a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py index ffef4a2a865..6fd62e1a6ff 100644 --- a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py +++ b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py @@ -350,6 +350,13 @@ class AnthropicPassthroughLoggingHandler: # Process each individual event for event_str in individual_events: try: + # Skip OpenAI-style [DONE] sentinels some Anthropic-compatible + # providers emit. Match the whole SSE line so a valid chunk whose + # text payload happens to contain "[DONE]" is not dropped. + if any( + line.strip() == "data: [DONE]" for line in event_str.split("\n") + ): + continue transformed_openai_chunk = anthropic_model_response_iterator.convert_str_chunk_to_generic_chunk( chunk=event_str ) @@ -358,6 +365,14 @@ class AnthropicPassthroughLoggingHandler: except (StopIteration, StopAsyncIteration): break + except json.JSONDecodeError: + # Some upstreams emit non-JSON SSE lines; skip them so the + # logging pipeline is not broken by a single bad frame. + verbose_proxy_logger.debug( + "Skipping non-JSON SSE event: %s", + event_str[:200], + ) + continue complete_streaming_response = litellm.stream_chunk_builder( chunks=all_openai_chunks, diff --git a/tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py b/tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py index 7593ea2f6db..8c3ce348a0e 100644 --- a/tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py +++ b/tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py @@ -686,6 +686,72 @@ class TestAnthropicBatchPassthroughCostTracking: ) +class TestBuildCompleteStreamingResponseRobustness: + """_build_complete_streaming_response must tolerate non-standard SSE frames.""" + + def _build(self, chunks: List[str]): + return AnthropicPassthroughLoggingHandler._build_complete_streaming_response( + all_chunks=chunks, + litellm_logging_obj=MagicMock(), + model="claude-3-sonnet-20240229", + ) + + def test_done_frame_is_skipped(self): + """A bare 'data: [DONE]' control frame must not break reconstruction.""" + chunks = [ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1","type":"message","role":"assistant","content":[],"model":"claude-3-sonnet-20240229","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":1}}}', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hi"}}', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":2}}', + 'event: message_stop\ndata: {"type":"message_stop"}', + "data: [DONE]", + ] + result = self._build(chunks) + assert result is not None + assert result.choices[0].message.content == "Hi" + + def test_non_json_sse_line_is_skipped(self): + """Non-JSON SSE lines (comments, keep-alive pings) must be skipped.""" + chunks = [ + ": ping", + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1","type":"message","role":"assistant","content":[],"model":"claude-3-sonnet-20240229","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":1}}}', + "this is not json at all", + ] + # Must not raise; a malformed stream simply yields no usable response. + result = self._build(chunks) + assert result is None or hasattr(result, "choices") + + def test_mixed_valid_and_invalid_frames(self): + """Valid events are still collected when interleaved with invalid ones.""" + chunks = [ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1","type":"message","role":"assistant","content":[],"model":"claude-3-sonnet-20240229","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":1}}}', + "data: [DONE]", + ": keep-alive", + "not-json", + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":2}}', + 'event: message_stop\ndata: {"type":"message_stop"}', + ] + result = self._build(chunks) + assert result is not None + assert result.choices[0].message.content == "Hello" + + def test_done_in_text_payload_is_not_dropped(self): + """A valid event whose text content contains '[DONE]' must NOT be skipped.""" + chunks = [ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1","type":"message","role":"assistant","content":[],"model":"claude-3-sonnet-20240229","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":1}}}', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The stream ends with [DONE]"}}', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":8}}', + 'event: message_stop\ndata: {"type":"message_stop"}', + ] + result = self._build(chunks) + assert result is not None + assert result.choices[0].message.content == "The stream ends with [DONE]" class TestStreamFalseDeduplication: """ Regression tests for the duplicate-callback bug where a streaming pass-through From 5927f695f768d1a15ea51b973ef233324921db92 Mon Sep 17 00:00:00 2001 From: yuneng-jiang Date: Fri, 12 Jun 2026 17:44:04 -0700 Subject: [PATCH 09/12] fix(proxy): return deprecated-key lookup result directly in get_data combined view (#30327) The grace-period branch assigned the recursive get_data result (a finished LiteLLM_VerificationTokenView) back into the variable that the combined-view dict normalization then subscripts, raising TypeError on every request made with a rotated key inside its grace window; auth surfaced that as a 401. Return the recursive result directly instead. Regression test drives the full get_data flow: old hash misses the view, deprecated table resolves to the active token, and the call must return the view object (cherry picked from commit 5047eaf7f0e7151891d7edbf19f92eb0004ff274) --- litellm/proxy/utils.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/litellm/proxy/utils.py b/litellm/proxy/utils.py index ebff52994c5..0a8cf0f0cd6 100644 --- a/litellm/proxy/utils.py +++ b/litellm/proxy/utils.py @@ -3513,7 +3513,10 @@ class PrismaClient: db=self.db, hashed_token=hashed_token ) if active_token_id: - response = await self.get_data( + # The recursive call returns a finished + # LiteLLM_VerificationTokenView; the dict + # normalization below would crash subscripting it. + deprecated_response = await self.get_data( token=active_token_id, table_name="combined_view", query_type="find_unique", @@ -3521,10 +3524,11 @@ class PrismaClient: proxy_logging_obj=proxy_logging_obj, check_deprecated=False, ) - if response is not None: + if deprecated_response is not None: verbose_proxy_logger.debug( "Deprecated key used during grace period" ) + return deprecated_response if response is not None: if response["team_models"] is None: From 825cb6fd0b8c6403bda480b10ee3c7930031d173 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Sat, 13 Jun 2026 16:25:35 -0700 Subject: [PATCH 10/12] chore(deps): backport #30220 dependency bumps to stable/1.86.x Backports the dependency hardening from #30220 (staging commit d96ab467). Adapted to this line: the proxy-runtime extras are exact-pinned here, so pypdf is moved from ==6.10.2 to ==6.13.1 (staging used >=6.12.0,<7.0); the [tool.uv] constraint-dependencies block (tornado>=6.5.6, aiohttp>=3.13.5,<3.14) matches staging. The dashboard devDependency vitest, @vitest/coverage-v8 and @vitest/ui move 3.2.4 -> 3.2.6 and the transitive brace-expansion 5.0.5 -> 5.0.6. uv.lock and the dashboard package-lock.json were regenerated on this line rather than cherry-picked, so the resolved set reflects 1.86.x: pypdf 6.13.1, tornado 6.5.7, aiohttp held at 3.13.5; vitest 3.2.6, brace-expansion 5.0.6. Adapted from #30220 (cherry picked from commit d96ab467f1dba5e4dbe02de3d5af62ec710c44fd) --- pyproject.toml | 6 +- ui/litellm-dashboard/package-lock.json | 146 ++++++++++--------------- ui/litellm-dashboard/package.json | 6 +- uv.lock | 36 +++--- 4 files changed, 88 insertions(+), 106 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 651129f6dbf..5c73eb6980f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -118,7 +118,7 @@ proxy-runtime = [ "mangum==0.17.0", "azure-ai-contentsafety==1.0.0", "azure-storage-file-datalake==12.20.0", - "pypdf==6.10.2; python_version < '3.14'", + "pypdf==6.13.1; python_version < '3.14'", "llm-sandbox==0.3.39", "detect-secrets==1.5.0", ] @@ -222,6 +222,10 @@ requires = ["uv_build==0.11.8"] build-backend = "uv_build" [tool.uv] +constraint-dependencies = [ + "tornado>=6.5.6", + "aiohttp>=3.13.5,<3.14", +] default-groups = ["dev"] required-version = ">=0.10.9" exclude-newer = "3 days" diff --git a/ui/litellm-dashboard/package-lock.json b/ui/litellm-dashboard/package-lock.json index b33b2a69bee..570cb0a6105 100644 --- a/ui/litellm-dashboard/package-lock.json +++ b/ui/litellm-dashboard/package-lock.json @@ -51,8 +51,8 @@ "@types/react-dom": "18.3.7", "@types/react-syntax-highlighter": "15.5.13", "@types/uuid": "10.0.0", - "@vitest/coverage-v8": "3.2.4", - "@vitest/ui": "3.2.4", + "@vitest/coverage-v8": "3.2.6", + "@vitest/ui": "3.2.6", "autoprefixer": "10.4.24", "dotenv": "17.2.3", "eslint": "9.39.2", @@ -66,7 +66,7 @@ "tailwindcss": "3.4.19", "typescript": "5.9.3", "vite": "7.3.2", - "vitest": "3.2.4" + "vitest": "3.2.6" }, "engines": { "node": ">=20.9.0", @@ -1937,9 +1937,6 @@ "cpu": [ "arm64" ], - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -1956,9 +1953,6 @@ "cpu": [ "arm64" ], - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -1975,9 +1969,6 @@ "cpu": [ "x64" ], - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -1994,9 +1985,6 @@ "cpu": [ "x64" ], - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -4010,9 +3998,9 @@ ] }, "node_modules/@vitest/coverage-v8": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-3.2.4.tgz", - "integrity": "sha512-EyF9SXU6kS5Ku/U82E259WSnvg6c8KTjppUncuNdm5QHpe17mwREHnjDzozC8x9MZ0xfBUFSaLkRv4TMA75ALQ==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-3.2.6.tgz", + "integrity": "sha512-LsAdmUapA0qSN306d8+zOyawM0hFm2m2Hg9IwVNIKBm+qJV8cijiq2c+gxKZcB1HCfIWAy+0qEZDCUQA58A1cw==", "dev": true, "license": "MIT", "dependencies": { @@ -4034,8 +4022,8 @@ "url": "https://opencollective.com/vitest" }, "peerDependencies": { - "@vitest/browser": "3.2.4", - "vitest": "3.2.4" + "@vitest/browser": "3.2.6", + "vitest": "3.2.6" }, "peerDependenciesMeta": { "@vitest/browser": { @@ -4044,15 +4032,15 @@ } }, "node_modules/@vitest/expect": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.4.tgz", - "integrity": "sha512-Io0yyORnB6sikFlt8QW5K7slY4OjqNX9jmJQ02QDda8lyM6B5oNgVWoSoKPac8/kgnCUzuHQKrSLtu/uOqqrig==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.6.tgz", + "integrity": "sha512-1+7q9BtaKzEmO+fmNT3kYvoNn5Y71XWAx2Q5HRim4tTVRQVRv4uJFAQ5FbK0OPUeNP/WmVCpxYxoJdvuHVjzBQ==", "dev": true, "license": "MIT", "dependencies": { "@types/chai": "^5.2.2", - "@vitest/spy": "3.2.4", - "@vitest/utils": "3.2.4", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", "chai": "^5.2.0", "tinyrainbow": "^2.0.0" }, @@ -4061,13 +4049,13 @@ } }, "node_modules/@vitest/mocker": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.4.tgz", - "integrity": "sha512-46ryTE9RZO/rfDd7pEqFl7etuyzekzEhUbTW3BvmeO/BcCMEgq59BKhek3dXDWgAj4oMK6OZi+vRr1wPW6qjEQ==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.6.tgz", + "integrity": "sha512-EZOrpDbkKotFAP7wPAQV1UIyoGOk4oX7ynWhBhLB7v+meMHbQhU16oPpIYGTTe4oFlhpryGpgpcZP/sin3hYuw==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/spy": "3.2.4", + "@vitest/spy": "3.2.6", "estree-walker": "^3.0.3", "magic-string": "^0.30.17" }, @@ -4088,9 +4076,9 @@ } }, "node_modules/@vitest/pretty-format": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.4.tgz", - "integrity": "sha512-IVNZik8IVRJRTr9fxlitMKeJeXFFFN0JaB9PHPGQ8NKQbGpfjlTx9zO4RefN8gp7eqjNy8nyK3NZmBzOPeIxtA==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.6.tgz", + "integrity": "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA==", "dev": true, "license": "MIT", "dependencies": { @@ -4101,13 +4089,13 @@ } }, "node_modules/@vitest/runner": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.4.tgz", - "integrity": "sha512-oukfKT9Mk41LreEW09vt45f8wx7DordoWUZMYdY/cyAk7w5TWkTRCNZYF7sX7n2wB7jyGAl74OxgwhPgKaqDMQ==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.6.tgz", + "integrity": "sha512-HYcoSj1w5tcgUnzoF0HcyaAQjpA1gj9ftUJ7iSJSuipc02jW9gKkigwZbjFldAfYHA1fa8UZVRftdMY5msWM9Q==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/utils": "3.2.4", + "@vitest/utils": "3.2.6", "pathe": "^2.0.3", "strip-literal": "^3.0.0" }, @@ -4116,13 +4104,13 @@ } }, "node_modules/@vitest/snapshot": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.4.tgz", - "integrity": "sha512-dEYtS7qQP2CjU27QBC5oUOxLE/v5eLkGqPE0ZKEIDGMs4vKWe7IjgLOeauHsR0D5YuuycGRO5oSRXnwnmA78fQ==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.6.tgz", + "integrity": "sha512-H+ZjNTWGpObenh0YnlBctAPnJSI20P81PL8BPzWpx54YXLLTm8hEsWawtcYLMrwvpK48hGxLLbCS+1KRXhsKhw==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/pretty-format": "3.2.4", + "@vitest/pretty-format": "3.2.6", "magic-string": "^0.30.17", "pathe": "^2.0.3" }, @@ -4131,9 +4119,9 @@ } }, "node_modules/@vitest/spy": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.4.tgz", - "integrity": "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.6.tgz", + "integrity": "sha512-oq6BbH68WzcWmwtBrU9nqLeaXTR4XwJF7FSLkKEZo4i6eoXcrxjcwSuTvWBIRUTC6VC72nXYunzqgZA+IKdtxg==", "dev": true, "license": "MIT", "dependencies": { @@ -4144,13 +4132,13 @@ } }, "node_modules/@vitest/ui": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/ui/-/ui-3.2.4.tgz", - "integrity": "sha512-hGISOaP18plkzbWEcP/QvtRW1xDXF2+96HbEX6byqQhAUbiS5oH6/9JwW+QsQCIYON2bI6QZBF+2PvOmrRZ9wA==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/ui/-/ui-3.2.6.tgz", + "integrity": "sha512-mATfG3zVdhobE9U1rIpvtYD3DGuSSxqZ3Aj/8ityGqKXy8YDJ9BoAjZmAz6dZ1IZ1xI5V+MerkCczvVa+3QK9Q==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/utils": "3.2.4", + "@vitest/utils": "3.2.6", "fflate": "^0.8.2", "flatted": "^3.3.3", "pathe": "^2.0.3", @@ -4162,17 +4150,17 @@ "url": "https://opencollective.com/vitest" }, "peerDependencies": { - "vitest": "3.2.4" + "vitest": "3.2.6" } }, "node_modules/@vitest/utils": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.4.tgz", - "integrity": "sha512-fB2V0JFrQSMsCo9HiSq3Ezpdv4iYaXRG1Sx8edX3MwxfyNn83mKiGzOcH+Fkxt4MHxr3y42fQi1oeAInqgX2QA==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.6.tgz", + "integrity": "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/pretty-format": "3.2.4", + "@vitest/pretty-format": "3.2.6", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" }, @@ -4751,9 +4739,9 @@ } }, "node_modules/brace-expansion": { - "version": "5.0.5", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", - "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", + "version": "5.0.6", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz", + "integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==", "dev": true, "license": "MIT", "dependencies": { @@ -6524,9 +6512,9 @@ } }, "node_modules/fflate": { - "version": "0.8.2", - "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz", - "integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==", + "version": "0.8.3", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.3.tgz", + "integrity": "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==", "dev": true, "license": "MIT" }, @@ -13014,20 +13002,20 @@ } }, "node_modules/vitest": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.4.tgz", - "integrity": "sha512-LUCP5ev3GURDysTWiP47wRRUpLKMOfPh+yKTx3kVIEiu5KOMeqzpnYNsKyOoVrULivR8tLcks4+lga33Whn90A==", + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.6.tgz", + "integrity": "sha512-xejya+bT/j/+R/AGa1XOfRxLmNUlLtlwjRsFUILF+xHfzElmGcmFydy2gqqIrd62ptIEfwVMofd19uNWD9L7Nw==", "dev": true, "license": "MIT", "dependencies": { "@types/chai": "^5.2.2", - "@vitest/expect": "3.2.4", - "@vitest/mocker": "3.2.4", - "@vitest/pretty-format": "^3.2.4", - "@vitest/runner": "3.2.4", - "@vitest/snapshot": "3.2.4", - "@vitest/spy": "3.2.4", - "@vitest/utils": "3.2.4", + "@vitest/expect": "3.2.6", + "@vitest/mocker": "3.2.6", + "@vitest/pretty-format": "^3.2.6", + "@vitest/runner": "3.2.6", + "@vitest/snapshot": "3.2.6", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", "chai": "^5.2.0", "debug": "^4.4.1", "expect-type": "^1.2.1", @@ -13057,8 +13045,8 @@ "@edge-runtime/vm": "*", "@types/debug": "^4.1.12", "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", - "@vitest/browser": "3.2.4", - "@vitest/ui": "3.2.4", + "@vitest/browser": "3.2.6", + "@vitest/ui": "3.2.6", "happy-dom": "*", "jsdom": "*" }, @@ -13349,8 +13337,9 @@ "version": "3.25.76", "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", - "extraneous": true, "license": "MIT", + "optional": true, + "peer": true, "funding": { "url": "https://github.com/sponsors/colinhacks" } @@ -13364,21 +13353,6 @@ "type": "github", "url": "https://github.com/sponsors/wooorm" } - }, - "node_modules/@next/swc-win32-ia32-msvc": { - "version": "14.2.33", - "resolved": "https://registry.npmjs.org/@next/swc-win32-ia32-msvc/-/swc-win32-ia32-msvc-14.2.33.tgz", - "integrity": "sha512-pc9LpGNKhJ0dXQhZ5QMmYxtARwwmWLpeocFmVG5Z0DzWq5Uf0izcI8tLc+qOpqxO1PWqZ5A7J1blrUIKrIFc7Q==", - "cpu": [ - "ia32" - ], - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">= 10" - } } } } diff --git a/ui/litellm-dashboard/package.json b/ui/litellm-dashboard/package.json index 32c00ac62a8..4540847a695 100644 --- a/ui/litellm-dashboard/package.json +++ b/ui/litellm-dashboard/package.json @@ -63,8 +63,8 @@ "@types/react-dom": "18.3.7", "@types/react-syntax-highlighter": "15.5.13", "@types/uuid": "10.0.0", - "@vitest/coverage-v8": "3.2.4", - "@vitest/ui": "3.2.4", + "@vitest/coverage-v8": "3.2.6", + "@vitest/ui": "3.2.6", "autoprefixer": "10.4.24", "dotenv": "17.2.3", "eslint": "9.39.2", @@ -78,7 +78,7 @@ "tailwindcss": "3.4.19", "typescript": "5.9.3", "vite": "7.3.2", - "vitest": "3.2.4" + "vitest": "3.2.6" }, "overrides": { "prismjs": "1.30.0", diff --git a/uv.lock b/uv.lock index 13e4adceec1..6534fb886e3 100644 --- a/uv.lock +++ b/uv.lock @@ -9,7 +9,7 @@ resolution-markers = [ ] [options] -exclude-newer = "2026-06-08T02:16:23.24531Z" +exclude-newer = "2026-06-10T23:24:01.433096Z" exclude-newer-span = "P3D" [manifest] @@ -18,6 +18,10 @@ members = [ "litellm-enterprise", "litellm-proxy-extras", ] +constraints = [ + { name = "aiohttp", specifier = ">=3.13.5,<3.14" }, + { name = "tornado", specifier = ">=6.5.6" }, +] [[package]] name = "a2a-sdk" @@ -3430,7 +3434,7 @@ requires-dist = [ { name = "pydantic-settings", marker = "extra == 'proxy'", specifier = ">=2.14.1" }, { name = "pyjwt", marker = "extra == 'proxy'", specifier = "==2.12.0" }, { name = "pynacl", marker = "extra == 'proxy'", specifier = "==1.6.2" }, - { name = "pypdf", marker = "python_full_version < '3.14' and extra == 'proxy-runtime'", specifier = "==6.10.2" }, + { name = "pypdf", marker = "python_full_version < '3.14' and extra == 'proxy-runtime'", specifier = "==6.13.1" }, { name = "pyroscope-io", marker = "sys_platform != 'win32' and extra == 'proxy'", specifier = "==0.8.16" }, { name = "python-dotenv", specifier = ">=1.0.0,<2.0" }, { name = "python-multipart", marker = "extra == 'proxy'", specifier = "==0.0.27" }, @@ -5919,14 +5923,14 @@ wheels = [ [[package]] name = "pypdf" -version = "6.10.2" +version = "6.13.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/7b/3f/9f2167401c2e94833ca3b69535bad89e533b5de75fefe4197a2c224baec2/pypdf-6.10.2.tar.gz", hash = "sha256:7d09ce108eff6bf67465d461b6ef352dcb8d84f7a91befc02f904455c6eea11d", size = 5315679, upload-time = "2026-04-15T16:37:36.978Z" } +sdist = { url = "https://files.pythonhosted.org/packages/15/d9/9d12fa0d9660d03320725ff686c961b645a4218940a82296e1272d9e1ff0/pypdf-6.13.1.tar.gz", hash = "sha256:4841d8a4c1589e5833915dc0c7ddfacff80a2e0bcbeb5d1e681fecaa1674b03a", size = 6477811, upload-time = "2026-06-08T11:01:49.344Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/0c/d6/1d5c60cc17bbdf37c1552d9c03862fc6d32c5836732a0415b2d637edc2d0/pypdf-6.10.2-py3-none-any.whl", hash = "sha256:aa53be9826655b51c96741e5d7983ca224d898ac0a77896e64636810517624aa", size = 336308, upload-time = "2026-04-15T16:37:34.851Z" }, + { url = "https://files.pythonhosted.org/packages/fe/dd/8f03e0a5788a5d1feb4550617c3e6db5e9099eaee248a3e482ddaeacbbb0/pypdf-6.13.1-py3-none-any.whl", hash = "sha256:e555e4ce3f561ef069307622f1374136ba964ca6ca24f24158701decaf83ed9b", size = 346259, upload-time = "2026-06-08T11:01:47.741Z" }, ] [[package]] @@ -7413,19 +7417,19 @@ wheels = [ [[package]] name = "tornado" -version = "6.5.5" +version = "6.5.7" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f8/f1/3173dfa4a18db4a9b03e5d55325559dab51ee653763bb8745a75af491286/tornado-6.5.5.tar.gz", hash = "sha256:192b8f3ea91bd7f1f50c06955416ed76c6b72f96779b962f07f911b91e8d30e9", size = 516006, upload-time = "2026-03-10T21:31:02.067Z" } +sdist = { url = "https://files.pythonhosted.org/packages/64/24/95ec527ad67b76d59299e5465b3935d05e4294b7e0290a3924b7487df30b/tornado-6.5.7.tar.gz", hash = "sha256:66c513a76cda70d53907bc27cf1447557699c2e95aa48ba27a442ff61c3ddfc2", size = 519252, upload-time = "2026-06-08T17:34:51.232Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/59/8c/77f5097695f4dd8255ecbd08b2a1ed8ba8b953d337804dd7080f199e12bf/tornado-6.5.5-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:487dc9cc380e29f58c7ab88f9e27cdeef04b2140862e5076a66fb6bb68bb1bfa", size = 445983, upload-time = "2026-03-10T21:30:44.28Z" }, - { url = "https://files.pythonhosted.org/packages/ab/5e/7625b76cd10f98f1516c36ce0346de62061156352353ef2da44e5c21523c/tornado-6.5.5-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:65a7f1d46d4bb41df1ac99f5fcb685fb25c7e61613742d5108b010975a9a6521", size = 444246, upload-time = "2026-03-10T21:30:46.571Z" }, - { url = "https://files.pythonhosted.org/packages/b2/04/7b5705d5b3c0fab088f434f9c83edac1573830ca49ccf29fb83bf7178eec/tornado-6.5.5-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:e74c92e8e65086b338fd56333fb9a68b9f6f2fe7ad532645a290a464bcf46be5", size = 447229, upload-time = "2026-03-10T21:30:48.273Z" }, - { url = "https://files.pythonhosted.org/packages/34/01/74e034a30ef59afb4097ef8659515e96a39d910b712a89af76f5e4e1f93c/tornado-6.5.5-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:435319e9e340276428bbdb4e7fa732c2d399386d1de5686cb331ec8eee754f07", size = 448192, upload-time = "2026-03-10T21:30:51.22Z" }, - { url = "https://files.pythonhosted.org/packages/be/00/fe9e02c5a96429fce1a1d15a517f5d8444f9c412e0bb9eadfbe3b0fc55bf/tornado-6.5.5-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:3f54aa540bdbfee7b9eb268ead60e7d199de5021facd276819c193c0fb28ea4e", size = 448039, upload-time = "2026-03-10T21:30:53.52Z" }, - { url = "https://files.pythonhosted.org/packages/82/9e/656ee4cec0398b1d18d0f1eb6372c41c6b889722641d84948351ae19556d/tornado-6.5.5-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:36abed1754faeb80fbd6e64db2758091e1320f6bba74a4cf8c09cd18ccce8aca", size = 447445, upload-time = "2026-03-10T21:30:55.541Z" }, - { url = "https://files.pythonhosted.org/packages/5a/76/4921c00511f88af86a33de770d64141170f1cfd9c00311aea689949e274e/tornado-6.5.5-cp39-abi3-win32.whl", hash = "sha256:dd3eafaaeec1c7f2f8fdcd5f964e8907ad788fe8a5a32c4426fbbdda621223b7", size = 448582, upload-time = "2026-03-10T21:30:57.142Z" }, - { url = "https://files.pythonhosted.org/packages/2c/23/f6c6112a04d28eed765e374435fb1a9198f73e1ec4b4024184f21faeb1ad/tornado-6.5.5-cp39-abi3-win_amd64.whl", hash = "sha256:6443a794ba961a9f619b1ae926a2e900ac20c34483eea67be4ed8f1e58d3ef7b", size = 448990, upload-time = "2026-03-10T21:30:58.857Z" }, - { url = "https://files.pythonhosted.org/packages/b7/c8/876602cbc96469911f0939f703453c1157b0c826ecb05bdd32e023397d4e/tornado-6.5.5-cp39-abi3-win_arm64.whl", hash = "sha256:2c9a876e094109333f888539ddb2de4361743e5d21eece20688e3e351e4990a6", size = 448016, upload-time = "2026-03-10T21:31:00.43Z" }, + { url = "https://files.pythonhosted.org/packages/02/dc/c7043cab6fed8ae159fc1923ce829ada35c4dbd797d408a43858ffaf9639/tornado-6.5.7-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:148b2eb15c2c765a50796172c1e499649b35f30d2e3c3d3e15913cfa56bfb163", size = 448543, upload-time = "2026-06-08T17:34:38.052Z" }, + { url = "https://files.pythonhosted.org/packages/92/4f/090b1431e5a43df696feceffc268c5383cc079ecb5f08ce58f917109aafe/tornado-6.5.7-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:9da38de27f1da3b78a966f0dae12b5a1ea9afe72ca805d84ff06508272ddf100", size = 446707, upload-time = "2026-06-08T17:34:39.594Z" }, + { url = "https://files.pythonhosted.org/packages/37/d8/ef374952fd5da67d4463122c2b8e5a96536ec10b4b339254c6dcde81d01c/tornado-6.5.7-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8d759e71906ee783f8867b93bf26a265743da4c1e2f4a018464c1ba019862972", size = 449774, upload-time = "2026-06-08T17:34:41.204Z" }, + { url = "https://files.pythonhosted.org/packages/35/37/d434c73f4c6e014b745b9b37085f34f40c022f007efff3d7fe65991899f3/tornado-6.5.7-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a46347a18f23fb92b396beebe0fb78f61dda0cc302445202c16203d8a18848b", size = 450745, upload-time = "2026-06-08T17:34:42.531Z" }, + { url = "https://files.pythonhosted.org/packages/b6/2b/56b9aff361d7f1ab728a805ec7d7ea835f8807afa9f5cc690ea0e630efb9/tornado-6.5.7-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:7778b30bef919231265e91c69963ce0f49a1e9c07ac900bbe75b19ce2575ba92", size = 450578, upload-time = "2026-06-08T17:34:43.787Z" }, + { url = "https://files.pythonhosted.org/packages/02/30/a7444fb23aa76860a14198fab96ac79f1866b0a6e19e26c4381b0938e50f/tornado-6.5.7-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e726f0c75da7726eec023aa62751ff8878bd2737e34fbdd33b1ae5897d2200f5", size = 449985, upload-time = "2026-06-08T17:34:45.326Z" }, + { url = "https://files.pythonhosted.org/packages/5c/42/5f0e56c01e8d9d36f4e23f367b85ae6cae0c1ecddd5e6977d8388ad27488/tornado-6.5.7-cp39-abi3-win32.whl", hash = "sha256:f8de3bf12d3efdd0cbe7c8887868198f8a91415e3f29fcf258d9b8eb7b1d9ae4", size = 451047, upload-time = "2026-06-08T17:34:46.784Z" }, + { url = "https://files.pythonhosted.org/packages/c9/a4/b393076ffb21b469eec5b328a0534cf03a3b90bfc6b1f09507cdd075d938/tornado-6.5.7-cp39-abi3-win_amd64.whl", hash = "sha256:de942f843533a039ef9fa3d9c88c7cd8a7c94553fb5ad0154270989b3d99a2c4", size = 451485, upload-time = "2026-06-08T17:34:48.248Z" }, + { url = "https://files.pythonhosted.org/packages/71/2e/7b1c769803121b809112cf9a00681c472eae1d80e32d7ec0e0bd61d0d0e1/tornado-6.5.7-cp39-abi3-win_arm64.whl", hash = "sha256:ff934fce95643af5f11efdae618eaa73d469dc588641e5c8d19295a0c65c4796", size = 450506, upload-time = "2026-06-08T17:34:49.702Z" }, ] [[package]] From 8e4dedf3a432e9b005c82bc5d48e618e20333a06 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Sat, 13 Jun 2026 16:25:53 -0700 Subject: [PATCH 11/12] =?UTF-8?q?bump:=20version=201.86.5=20=E2=86=92=201.?= =?UTF-8?q?86.6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- pyproject.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 5c73eb6980f..e2445aa9c80 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "litellm" -version = "1.86.5" +version = "1.86.6" description = "Library to easily interface with LLM API providers" readme = "README.md" requires-python = ">=3.10, <3.14" @@ -255,7 +255,7 @@ source-exclude = [ profile = "black" [tool.commitizen] -version = "1.86.5" +version = "1.86.6" version_files = [ "pyproject.toml:^version", ] From ca26d570e30dab755463a225b839fb58bea8f859 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Sat, 13 Jun 2026 16:26:16 -0700 Subject: [PATCH 12/12] chore: refresh uv.lock for 1.86.6 --- uv.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/uv.lock b/uv.lock index 6534fb886e3..971bc9fdd86 100644 --- a/uv.lock +++ b/uv.lock @@ -9,7 +9,7 @@ resolution-markers = [ ] [options] -exclude-newer = "2026-06-10T23:24:01.433096Z" +exclude-newer = "2026-06-10T23:26:07.892848Z" exclude-newer-span = "P3D" [manifest] @@ -3193,7 +3193,7 @@ wheels = [ [[package]] name = "litellm" -version = "1.86.5" +version = "1.86.6" source = { editable = "." } dependencies = [ { name = "aiohttp" },