mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
fix(router): use forwarded model_id for native Azure container IDs (#27921)
* fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints
Azure code-interpreter containers return provider-native IDs (cntr_ + hex)
that carry no LiteLLM routing payload, so _decode_container_id returns
model_id=None. The router was falling through to call the handler directly,
bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for
Azure deployments. Fall back to the model_id forwarded from the proxy
ownership check so deployment credentials are always applied.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url
When a deployment's api_base is the responses endpoint URL
(e.g. .../openai/responses?api-version=...), AzureContainerConfig was
appending /openai/containers on top of it, producing the broken path
.../openai/responses/openai/containers. Azure returns 404 for that URL
while the correct path is .../openai/containers.
Strip any /openai/responses suffix from api_base before constructing
the containers URL so the resource root is always used as the starting point.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): prefer api-version from api_base URL over deployment's api_version
The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses
API and is too old for the containers API, which requires 2025-04-01-preview.
The responses endpoint api_base already carries the correct api-version in its
query string. Extract it and use it for the containers URL, overriding the
stale deployment-level version.
Fixes DELETE and file-upload operations returning 404 due to wrong api-version.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(containers): pass params=None instead of params={} to httpx to preserve api-version
httpx erases a URL's query-string when params={} (empty dict) is passed,
silently stripping ?api-version=2025-04-01-preview from every container
POST/DELETE request. Azure's GET endpoints tolerate a missing api-version;
POST (upload) and DELETE are strict, so those returned 404.
Fix: use `params or None` in container_handler._async_handle and
llm_http_handler.async_container_delete_handler (and all sibling container
handlers) so that an empty params dict falls back to None, leaving httpx to
preserve the URL's existing query string intact.
Adds a regression test that directly documents the httpx behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): remove elif model_id branch from _init_containers_api_endpoints
Two reviewer findings addressed:
1. Truncated comment on the model_id fallback line — now complete.
2. Security: the elif branch that fired when container_id was absent allowed
any authenticated caller to supply model_id in a POST /v1/containers body
and route the request through an arbitrary deployment UUID, bypassing the
model-level access checks that only validate `model`. Removed the elif
branch; operations without container_id (create, list) route by the
caller-supplied `model` field as before. model_id forwarding is kept only
inside the container_id block, where the proxy ownership check has already
validated the container before forwarding the deployment ID.
Adds a regression test pinning the security boundary: no-container-id path
calls original_function directly even when model_id is in kwargs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(containers): validate proxy-to-router model_id forwarding for managed IDs
Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id
to verify that get_container_forwarding_params (the proxy-side half of the Azure
routing fix) correctly extracts and forwards model_id from a LiteLLM-managed
encoded container ID.
This closes the gap identified by Greptile P1: the previous regression test
only injected model_id as a direct kwarg, validating the router in isolation.
The new test exercises the actual proxy-to-router data flow through
ownership.get_container_forwarding_params, confirming that kwargs["model_id"]
is populated before _init_containers_api_endpoints is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): tighten endpoint-path strip to endswith match
Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so
the suffix strip only fires when api_base actually ends with one of the
endpoint-specific path suffixes. This is the more precise check greptile
flagged on the original find()-based implementation.
* Fix sync container handler to preserve URL query string
Mirror the async path fix: pass None instead of an empty params dict so
httpx does not strip the URL's existing query string (e.g.
?api-version=...), which is required for Azure container routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(azure-containers): strip trailing slash before endpoint suffix match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(containers): recover model_id from stored encoded id for native Azure container IDs
get_container_forwarding_params previously only set model_id when the
user-supplied container_id was a LiteLLM-managed encoded id. For native
upstream IDs (e.g. Azure 'cntr_<hex>') the decode fails and model_id was
never forwarded — making the router-side fallback in
_init_containers_api_endpoints unreachable in production.
Fall back to the stored 'unified_object_id' on the ownership row, which
is the encoded form captured at create time when the router selected a
specific deployment. Decoding that yields the deployment model_id and
restores router-based credential application (api_base, api_key) for
retrieve/delete and container-file operations on native IDs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
(cherry picked from commit 7f563b2593)
This commit is contained in:
parent
bbb40d3b92
commit
791b200f43
8 changed files with 411 additions and 30 deletions
|
|
@ -1,9 +1,16 @@
|
|||
from typing import Optional
|
||||
from urllib.parse import parse_qs, urlparse, urlunparse
|
||||
|
||||
from litellm.llms.azure.common_utils import BaseAzureLLM
|
||||
from litellm.llms.openai.containers.transformation import OpenAIContainerConfig
|
||||
from litellm.types.router import GenericLiteLLMParams
|
||||
|
||||
# Endpoint-specific path suffixes that may appear in a deployment's api_base
|
||||
# (e.g. the responses endpoint URL is stored as api_base for Azure models).
|
||||
# Strip these before building the containers URL so we always start from the
|
||||
# resource root (https://resource.cognitiveservices.azure.com).
|
||||
_AZURE_ENDPOINT_PATHS = ("/openai/responses",)
|
||||
|
||||
|
||||
class AzureContainerConfig(OpenAIContainerConfig):
|
||||
"""
|
||||
|
|
@ -27,6 +34,27 @@ class AzureContainerConfig(OpenAIContainerConfig):
|
|||
litellm_params=GenericLiteLLMParams(api_key=api_key),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _normalize_api_base(api_base: Optional[str]) -> Optional[str]:
|
||||
"""Strip endpoint-specific path suffixes from api_base to get the resource root."""
|
||||
if not api_base:
|
||||
return api_base
|
||||
parsed = urlparse(api_base)
|
||||
path = parsed.path.rstrip("/")
|
||||
for ep in _AZURE_ENDPOINT_PATHS:
|
||||
if path.endswith(ep):
|
||||
return urlunparse(
|
||||
(parsed.scheme, parsed.netloc, path[: -len(ep)], "", "", "")
|
||||
)
|
||||
return api_base
|
||||
|
||||
@staticmethod
|
||||
def _extract_api_version(api_base: Optional[str]) -> Optional[str]:
|
||||
"""Return the api-version query param from api_base if present."""
|
||||
if not api_base:
|
||||
return None
|
||||
return parse_qs(urlparse(api_base).query).get("api-version", [None])[0]
|
||||
|
||||
def get_complete_url(
|
||||
self,
|
||||
api_base: Optional[str],
|
||||
|
|
@ -39,10 +67,19 @@ class AzureContainerConfig(OpenAIContainerConfig):
|
|||
{endpoint}/openai/v1/containers
|
||||
when api_version is 'v1', 'latest', or 'preview'; otherwise:
|
||||
{endpoint}/openai/containers
|
||||
|
||||
The deployment's api_base may be the responses endpoint URL
|
||||
(e.g. .../openai/responses?api-version=2025-04-01-preview). We
|
||||
prefer the api-version embedded there over the deployment's
|
||||
api_version field, which may point to an older chat API version.
|
||||
"""
|
||||
effective_params = dict(litellm_params)
|
||||
api_version_from_base = self._extract_api_version(api_base)
|
||||
if api_version_from_base:
|
||||
effective_params["api_version"] = api_version_from_base
|
||||
return BaseAzureLLM._get_base_azure_url(
|
||||
api_base=api_base,
|
||||
litellm_params=litellm_params,
|
||||
api_base=self._normalize_api_base(api_base),
|
||||
litellm_params=effective_params,
|
||||
route="/openai/containers",
|
||||
default_api_version="v1",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -257,14 +257,19 @@ class GenericContainerHandler:
|
|||
returns_binary = endpoint_config.get("returns_binary", False)
|
||||
is_multipart = endpoint_config.get("is_multipart", False)
|
||||
|
||||
# An empty dict passed as `params` to httpx strips any existing query
|
||||
# string from the URL (e.g. ?api-version=...). Use None instead so
|
||||
# httpx leaves the URL's own query string intact.
|
||||
effective_params = query_params or None
|
||||
|
||||
try:
|
||||
if method == "GET":
|
||||
response = http_client.get(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
elif method == "DELETE":
|
||||
response = http_client.delete(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
elif method == "POST":
|
||||
if is_multipart and "file" in kwargs:
|
||||
|
|
@ -272,11 +277,11 @@ class GenericContainerHandler:
|
|||
kwargs["file"], headers
|
||||
)
|
||||
response = http_client.post(
|
||||
url=url, headers=headers, params=query_params, files=files
|
||||
url=url, headers=headers, params=effective_params, files=files
|
||||
)
|
||||
else:
|
||||
response = http_client.post(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported HTTP method: {method}")
|
||||
|
|
@ -376,14 +381,19 @@ class GenericContainerHandler:
|
|||
returns_binary = endpoint_config.get("returns_binary", False)
|
||||
is_multipart = endpoint_config.get("is_multipart", False)
|
||||
|
||||
# An empty dict passed as `params` to httpx strips any existing query
|
||||
# string from the URL (e.g. ?api-version=...). Use None instead so
|
||||
# httpx leaves the URL's own query string intact.
|
||||
effective_params = query_params or None
|
||||
|
||||
try:
|
||||
if method == "GET":
|
||||
response = await http_client.get(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
elif method == "DELETE":
|
||||
response = await http_client.delete(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
elif method == "POST":
|
||||
if is_multipart and "file" in kwargs:
|
||||
|
|
@ -391,11 +401,11 @@ class GenericContainerHandler:
|
|||
kwargs["file"], headers
|
||||
)
|
||||
response = await http_client.post(
|
||||
url=url, headers=headers, params=query_params, files=files
|
||||
url=url, headers=headers, params=effective_params, files=files
|
||||
)
|
||||
else:
|
||||
response = await http_client.post(
|
||||
url=url, headers=headers, params=query_params
|
||||
url=url, headers=headers, params=effective_params
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported HTTP method: {method}")
|
||||
|
|
|
|||
|
|
@ -7816,7 +7816,7 @@ class BaseLLMHTTPHandler:
|
|||
response = sync_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_list_response(
|
||||
|
|
@ -7893,7 +7893,7 @@ class BaseLLMHTTPHandler:
|
|||
response = await async_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_list_response(
|
||||
|
|
@ -7983,7 +7983,7 @@ class BaseLLMHTTPHandler:
|
|||
response = sync_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_retrieve_response(
|
||||
|
|
@ -8060,7 +8060,7 @@ class BaseLLMHTTPHandler:
|
|||
response = await async_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_retrieve_response(
|
||||
|
|
@ -8150,7 +8150,7 @@ class BaseLLMHTTPHandler:
|
|||
response = sync_httpx_client.delete(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_delete_response(
|
||||
|
|
@ -8227,7 +8227,7 @@ class BaseLLMHTTPHandler:
|
|||
response = await async_httpx_client.delete(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_delete_response(
|
||||
|
|
@ -8323,7 +8323,7 @@ class BaseLLMHTTPHandler:
|
|||
response = sync_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_file_list_response(
|
||||
|
|
@ -8402,7 +8402,7 @@ class BaseLLMHTTPHandler:
|
|||
response = await async_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_file_list_response(
|
||||
|
|
@ -8490,7 +8490,7 @@ class BaseLLMHTTPHandler:
|
|||
response = sync_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_file_content_response(
|
||||
|
|
@ -8566,7 +8566,7 @@ class BaseLLMHTTPHandler:
|
|||
response = await async_httpx_client.get(
|
||||
url=url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
params=params or None,
|
||||
)
|
||||
|
||||
return container_provider_config.transform_container_file_content_response(
|
||||
|
|
|
|||
|
|
@ -328,7 +328,7 @@ async def retrieve_container(
|
|||
custom_llm_provider=custom_llm_provider,
|
||||
)
|
||||
data.update(
|
||||
get_container_forwarding_params(
|
||||
await get_container_forwarding_params(
|
||||
container_id,
|
||||
original_container_id,
|
||||
custom_llm_provider,
|
||||
|
|
@ -433,7 +433,7 @@ async def delete_container(
|
|||
custom_llm_provider=custom_llm_provider,
|
||||
)
|
||||
data.update(
|
||||
get_container_forwarding_params(
|
||||
await get_container_forwarding_params(
|
||||
container_id,
|
||||
original_container_id,
|
||||
custom_llm_provider,
|
||||
|
|
|
|||
|
|
@ -196,10 +196,12 @@ async def _process_binary_request(
|
|||
)
|
||||
data: Dict[str, Any] = {
|
||||
"file_id": file_id,
|
||||
**get_container_forwarding_params(
|
||||
container_id=container_id,
|
||||
original_container_id=original_container_id,
|
||||
custom_llm_provider=resolved_provider,
|
||||
**(
|
||||
await get_container_forwarding_params(
|
||||
container_id=container_id,
|
||||
original_container_id=original_container_id,
|
||||
custom_llm_provider=resolved_provider,
|
||||
)
|
||||
),
|
||||
}
|
||||
processor = ProxyBaseLLMRequestProcessing(data=data)
|
||||
|
|
@ -316,7 +318,7 @@ async def _process_multipart_upload_request(
|
|||
)
|
||||
|
||||
data.update(
|
||||
get_container_forwarding_params(
|
||||
await get_container_forwarding_params(
|
||||
container_id=container_id,
|
||||
original_container_id=original_container_id,
|
||||
custom_llm_provider=resolved_provider,
|
||||
|
|
@ -396,7 +398,7 @@ async def _process_request(
|
|||
)
|
||||
)
|
||||
data.update(
|
||||
get_container_forwarding_params(
|
||||
await get_container_forwarding_params(
|
||||
container_id=path_params["container_id"],
|
||||
original_container_id=original_container_id,
|
||||
custom_llm_provider=resolved_provider,
|
||||
|
|
|
|||
|
|
@ -23,6 +23,13 @@ CONTAINER_OBJECT_PURPOSE = "container"
|
|||
_NEGATIVE_OWNER_SENTINEL = "__litellm_container_no_owner__"
|
||||
_CONTAINER_OWNER_CACHE = InMemoryCache(max_size_in_memory=10000, default_ttl=60)
|
||||
|
||||
# Caches the stored ``unified_object_id`` (the encoded container ID
|
||||
# captured at create time) so ``get_container_forwarding_params`` can
|
||||
# recover the deployment ``model_id`` for native upstream IDs without
|
||||
# re-hitting Prisma on every retrieve/delete.
|
||||
_NEGATIVE_STORED_ID_SENTINEL = "__litellm_container_no_stored_id__"
|
||||
_CONTAINER_STORED_ID_CACHE = InMemoryCache(max_size_in_memory=10000, default_ttl=60)
|
||||
|
||||
# Per-caller-scope cache for ``GET /v1/containers`` list filtering. Without
|
||||
# this, every list call issues a fresh ``find_many`` against
|
||||
# ``litellm_managedobjecttable``. The cache key is the sorted owner-scope
|
||||
|
|
@ -56,7 +63,7 @@ def decode_container_id_for_ownership(
|
|||
return original_container_id, custom_llm_provider
|
||||
|
||||
|
||||
def get_container_forwarding_params(
|
||||
async def get_container_forwarding_params(
|
||||
container_id: str, original_container_id: str, custom_llm_provider: str
|
||||
) -> Dict[str, str]:
|
||||
params = {
|
||||
|
|
@ -65,6 +72,20 @@ def get_container_forwarding_params(
|
|||
}
|
||||
decoded = ResponsesAPIRequestUtils._decode_container_id(container_id)
|
||||
model_id = decoded.get("model_id")
|
||||
if not (isinstance(model_id, str) and model_id):
|
||||
# Native upstream IDs (e.g. Azure ``cntr_<hex>``) carry no LiteLLM
|
||||
# routing payload, so decoding the user-supplied id yields no
|
||||
# ``model_id``. Recover it from the encoded ``unified_object_id``
|
||||
# captured on the ownership row at create time — when the router
|
||||
# selected a specific deployment that ID embeds the model_id.
|
||||
stored_id = await _get_stored_container_id(
|
||||
original_container_id, custom_llm_provider
|
||||
)
|
||||
if stored_id and stored_id != container_id:
|
||||
stored_decoded = ResponsesAPIRequestUtils._decode_container_id(stored_id)
|
||||
stored_model_id = stored_decoded.get("model_id")
|
||||
if isinstance(stored_model_id, str) and stored_model_id:
|
||||
model_id = stored_model_id
|
||||
if isinstance(model_id, str) and model_id:
|
||||
params["model_id"] = model_id
|
||||
return params
|
||||
|
|
@ -168,6 +189,7 @@ async def record_container_owner(
|
|||
)
|
||||
|
||||
_CONTAINER_OWNER_CACHE.set_cache(model_object_id, owner)
|
||||
_CONTAINER_STORED_ID_CACHE.set_cache(model_object_id, container_id)
|
||||
# Drop the caller's own list-cache entry so the just-created container
|
||||
# shows up on their next ``GET /v1/containers``. Other callers with
|
||||
# disjoint scope tuples have their own entries; intersecting-scope
|
||||
|
|
@ -207,9 +229,60 @@ async def _get_container_owner(
|
|||
_CONTAINER_OWNER_CACHE.set_cache(
|
||||
model_object_id, owner if owner is not None else _NEGATIVE_OWNER_SENTINEL
|
||||
)
|
||||
stored_id = getattr(row, "unified_object_id", None) if row is not None else None
|
||||
_CONTAINER_STORED_ID_CACHE.set_cache(
|
||||
model_object_id,
|
||||
(
|
||||
stored_id
|
||||
if isinstance(stored_id, str) and stored_id
|
||||
else _NEGATIVE_STORED_ID_SENTINEL
|
||||
),
|
||||
)
|
||||
return owner
|
||||
|
||||
|
||||
async def _get_stored_container_id(
|
||||
original_container_id: str, custom_llm_provider: str
|
||||
) -> Optional[str]:
|
||||
"""Return the ``unified_object_id`` stored at create time, if any.
|
||||
|
||||
Used by :func:`get_container_forwarding_params` to recover the
|
||||
deployment ``model_id`` for native upstream container IDs: the stored
|
||||
value is the encoded form produced by ``encode_container_id_in_response``
|
||||
when the router selected a specific deployment.
|
||||
"""
|
||||
model_object_id = _container_model_object_id(
|
||||
original_container_id, custom_llm_provider
|
||||
)
|
||||
|
||||
cached = _CONTAINER_STORED_ID_CACHE.get_cache(model_object_id)
|
||||
if cached == _NEGATIVE_STORED_ID_SENTINEL:
|
||||
return None
|
||||
if isinstance(cached, str) and cached:
|
||||
return cached
|
||||
|
||||
prisma_client = await _get_prisma_client()
|
||||
if prisma_client is None:
|
||||
return None
|
||||
|
||||
row = await prisma_client.db.litellm_managedobjecttable.find_first(
|
||||
where={
|
||||
"model_object_id": model_object_id,
|
||||
"file_purpose": CONTAINER_OBJECT_PURPOSE,
|
||||
}
|
||||
)
|
||||
stored_id = getattr(row, "unified_object_id", None) if row is not None else None
|
||||
_CONTAINER_STORED_ID_CACHE.set_cache(
|
||||
model_object_id,
|
||||
(
|
||||
stored_id
|
||||
if isinstance(stored_id, str) and stored_id
|
||||
else _NEGATIVE_STORED_ID_SENTINEL
|
||||
),
|
||||
)
|
||||
return stored_id if isinstance(stored_id, str) and stored_id else None
|
||||
|
||||
|
||||
async def assert_user_can_access_container(
|
||||
container_id: str,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
|
|
|
|||
|
|
@ -5551,6 +5551,7 @@ class Router:
|
|||
from litellm.responses.utils import ResponsesAPIRequestUtils
|
||||
|
||||
container_id = kwargs.get("container_id")
|
||||
_forwarded_model_id = kwargs.get("model_id")
|
||||
if isinstance(container_id, str):
|
||||
decoded = ResponsesAPIRequestUtils._decode_container_id(container_id)
|
||||
original_id = decoded.get("response_id", container_id)
|
||||
|
|
@ -5559,7 +5560,14 @@ class Router:
|
|||
decoded_provider = decoded.get("custom_llm_provider")
|
||||
if decoded_provider and kwargs.get("custom_llm_provider") == "openai":
|
||||
kwargs["custom_llm_provider"] = decoded_provider
|
||||
model_id = decoded.get("model_id")
|
||||
# Fall back to the model_id forwarded by the proxy when the container_id
|
||||
# is a native upstream ID (e.g. Azure hex cntr_) that carries no LiteLLM
|
||||
# routing payload, so deployment credentials (api_base, api_key) are applied.
|
||||
model_id = decoded.get("model_id") or (
|
||||
_forwarded_model_id.strip()
|
||||
if isinstance(_forwarded_model_id, str) and _forwarded_model_id.strip()
|
||||
else None
|
||||
)
|
||||
if model_id:
|
||||
kwargs["model"] = model_id
|
||||
return await self._ageneric_api_call_with_fallbacks(
|
||||
|
|
|
|||
|
|
@ -109,6 +109,31 @@ class TestAzureContainerConfig:
|
|||
|
||||
assert "/openai/v1/containers" in url
|
||||
|
||||
def test_get_complete_url_strips_responses_path_and_preserves_api_version(self):
|
||||
"""When api_base is the responses endpoint URL, get_complete_url must:
|
||||
- strip /openai/responses (no double-path)
|
||||
- use the api-version from api_base query string, NOT the deployment's
|
||||
older api_version (e.g. 2024-08-01-preview → containers need 2025-04-01-preview)
|
||||
"""
|
||||
api_base = "https://my-resource.cognitiveservices.azure.com/openai/responses?api-version=2025-04-01-preview"
|
||||
|
||||
url = self.config.get_complete_url(
|
||||
api_base=api_base,
|
||||
litellm_params={"api_version": "2024-08-01-preview"},
|
||||
)
|
||||
|
||||
assert (
|
||||
"/openai/responses/openai/containers" not in url
|
||||
), "path must not double /openai/responses"
|
||||
assert "my-resource.cognitiveservices.azure.com" in url
|
||||
assert "/openai/containers" in url or "/openai/v1/containers" in url
|
||||
assert (
|
||||
"2025-04-01-preview" in url
|
||||
), "must use version from api_base, not litellm_params"
|
||||
assert (
|
||||
"2024-08-01-preview" not in url
|
||||
), "must not fall back to older chat api_version"
|
||||
|
||||
def test_get_complete_url_raises_without_api_base(self, monkeypatch):
|
||||
monkeypatch.delenv("AZURE_API_BASE", raising=False)
|
||||
monkeypatch.setattr(litellm, "api_base", None)
|
||||
|
|
@ -531,6 +556,92 @@ class TestAzureContainerKnownFailureRegressions:
|
|||
assert qs.get("api-version") == ["v1"]
|
||||
assert qs.get("foo") == ["bar"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_regression_no_container_id_does_not_use_user_supplied_model_id(
|
||||
self, monkeypatch
|
||||
):
|
||||
"""Operations without container_id (create, list) must NOT route via
|
||||
_ageneric_api_call_with_fallbacks using a caller-supplied model_id.
|
||||
|
||||
Security boundary: only the path that holds a validated container_id
|
||||
is trusted to fall back to the forwarded model_id. A caller setting
|
||||
model_id without container_id on POST /v1/containers must not gain
|
||||
access to an arbitrary deployment UUID.
|
||||
"""
|
||||
from litellm.router import Router
|
||||
|
||||
router = Router(
|
||||
model_list=[
|
||||
{
|
||||
"model_name": "azure-model",
|
||||
"litellm_params": {
|
||||
"model": "azure/gpt-4",
|
||||
"api_base": "https://my-resource.cognitiveservices.azure.com",
|
||||
"api_key": "test-key",
|
||||
"api_version": "2025-04-01-preview",
|
||||
},
|
||||
"model_info": {"id": "deployment-uuid-123"},
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
fallback_called = {"called": False}
|
||||
|
||||
async def _mock_fallback(original_function, **kwargs):
|
||||
fallback_called["called"] = True
|
||||
return {}
|
||||
|
||||
monkeypatch.setattr(router, "_ageneric_api_call_with_fallbacks", _mock_fallback)
|
||||
|
||||
original_called = {"called": False}
|
||||
|
||||
async def _noop(**kwargs):
|
||||
original_called["called"] = True
|
||||
return {}
|
||||
|
||||
# No container_id — simulates create/list; caller injects a model_id
|
||||
await router._init_containers_api_endpoints(
|
||||
original_function=_noop,
|
||||
model_id="deployment-uuid-123",
|
||||
custom_llm_provider="azure",
|
||||
)
|
||||
|
||||
assert not fallback_called["called"], (
|
||||
"_ageneric_api_call_with_fallbacks must NOT be called when "
|
||||
"container_id is absent, even if model_id is supplied"
|
||||
)
|
||||
assert original_called["called"], "original_function must be called directly"
|
||||
|
||||
def test_regression_httpx_empty_params_strips_query_string(self):
|
||||
"""httpx erases the URL query-string when params={} (empty dict) is passed.
|
||||
|
||||
Root cause of the Azure container 404s on POST/DELETE:
|
||||
_build_query_params returns {} when the endpoint has no extra params;
|
||||
passing that {} as params= to httpx wiped ?api-version=2025-04-01-preview.
|
||||
|
||||
Fix: every container httpx call now uses `params or None` so an empty
|
||||
dict falls back to None, which tells httpx to leave the URL untouched.
|
||||
"""
|
||||
url = (
|
||||
"https://resource.cognitiveservices.azure.com"
|
||||
"/openai/containers/cntr_123?api-version=2025-04-01-preview"
|
||||
)
|
||||
client = httpx.AsyncClient()
|
||||
|
||||
req_none = client.build_request("DELETE", url, params=None)
|
||||
assert "api-version=2025-04-01-preview" in str(req_none.url)
|
||||
|
||||
req_empty = client.build_request("DELETE", url, params={})
|
||||
assert "api-version" not in str(
|
||||
req_empty.url
|
||||
), "Documents root cause: params={} strips the query string"
|
||||
|
||||
effective: dict = {}
|
||||
req_guarded = client.build_request("DELETE", url, params=effective or None)
|
||||
assert "api-version=2025-04-01-preview" in str(
|
||||
req_guarded.url
|
||||
), "`params or None` must preserve ?api-version"
|
||||
|
||||
def test_regression_proxy_resolves_azure_text_same_as_azure(self):
|
||||
"""Router/proxy treat azure_text like azure for container config."""
|
||||
from litellm.proxy.container_endpoints.handler_factory import (
|
||||
|
|
@ -770,3 +881,143 @@ class TestAzureContainerKnownFailureRegressions:
|
|||
assert captured["data"]["container_id"] == "cntr_123"
|
||||
assert captured["data"]["custom_llm_provider"] == "azure"
|
||||
assert captured["data"]["model_id"] == "model_abc123"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_regression_get_container_forwarding_params_sets_model_id_for_managed_id(
|
||||
self,
|
||||
):
|
||||
"""get_container_forwarding_params must extract model_id from a
|
||||
LiteLLM-managed encoded container ID and include it in the forwarding
|
||||
dict. This is the proxy-side half of the native-Azure-ID routing fix:
|
||||
the router's _init_containers_api_endpoints reads kwargs["model_id"]
|
||||
which is set here.
|
||||
"""
|
||||
from litellm.proxy.container_endpoints.ownership import (
|
||||
get_container_forwarding_params,
|
||||
)
|
||||
|
||||
encoded_id = ResponsesAPIRequestUtils._build_container_id(
|
||||
custom_llm_provider="azure",
|
||||
model_id="deployment-uuid-123",
|
||||
container_id="cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df",
|
||||
)
|
||||
|
||||
params = await get_container_forwarding_params(
|
||||
container_id=encoded_id,
|
||||
original_container_id="cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df",
|
||||
custom_llm_provider="azure",
|
||||
)
|
||||
|
||||
assert (
|
||||
params.get("model_id") == "deployment-uuid-123"
|
||||
), "model_id must be forwarded to the router for managed container IDs"
|
||||
assert params.get("container_id") == (
|
||||
"cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df"
|
||||
)
|
||||
assert params.get("custom_llm_provider") == "azure"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_regression_get_container_forwarding_params_recovers_model_id_for_native_id(
|
||||
self, monkeypatch
|
||||
):
|
||||
"""Native Azure IDs (``cntr_<hex>``) cannot be decoded, so model_id
|
||||
must be recovered from the ownership row's ``unified_object_id`` —
|
||||
the encoded form captured at create time when the router selected a
|
||||
specific deployment. Without this, the router-side fallback for
|
||||
native IDs in ``_init_containers_api_endpoints`` is dead code.
|
||||
"""
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
from litellm.proxy.container_endpoints import ownership
|
||||
from litellm.proxy.container_endpoints.ownership import (
|
||||
get_container_forwarding_params,
|
||||
)
|
||||
|
||||
native_id = "cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df"
|
||||
encoded_stored_id = ResponsesAPIRequestUtils._build_container_id(
|
||||
custom_llm_provider="azure",
|
||||
model_id="deployment-uuid-123",
|
||||
container_id=native_id,
|
||||
)
|
||||
|
||||
ownership._CONTAINER_STORED_ID_CACHE.flush_cache()
|
||||
ownership._CONTAINER_OWNER_CACHE.flush_cache()
|
||||
|
||||
table = AsyncMock()
|
||||
table.find_first.return_value = SimpleNamespace(
|
||||
created_by="user-1",
|
||||
file_purpose=ownership.CONTAINER_OBJECT_PURPOSE,
|
||||
unified_object_id=encoded_stored_id,
|
||||
)
|
||||
prisma_client = SimpleNamespace(
|
||||
db=SimpleNamespace(litellm_managedobjecttable=table)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ownership,
|
||||
"_get_prisma_client",
|
||||
AsyncMock(return_value=prisma_client),
|
||||
)
|
||||
|
||||
params = await get_container_forwarding_params(
|
||||
container_id=native_id,
|
||||
original_container_id=native_id,
|
||||
custom_llm_provider="azure",
|
||||
)
|
||||
|
||||
assert params.get("model_id") == "deployment-uuid-123", (
|
||||
"model_id must be recovered from the stored unified_object_id "
|
||||
"for native upstream container IDs"
|
||||
)
|
||||
assert params.get("container_id") == native_id
|
||||
assert params.get("custom_llm_provider") == "azure"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_regression_native_azure_container_id_uses_forwarded_model_id(
|
||||
self, monkeypatch
|
||||
):
|
||||
"""Native Azure container IDs (cntr_ + hex, no LiteLLM payload) must
|
||||
still route through _ageneric_api_call_with_fallbacks using the
|
||||
model_id forwarded from the proxy ownership check so that deployment
|
||||
credentials (api_base) are applied."""
|
||||
from litellm.router import Router
|
||||
|
||||
router = Router(
|
||||
model_list=[
|
||||
{
|
||||
"model_name": "azure-model",
|
||||
"litellm_params": {
|
||||
"model": "azure/gpt-4",
|
||||
"api_base": "https://my-resource.cognitiveservices.azure.com",
|
||||
"api_key": "test-key",
|
||||
"api_version": "2025-04-01-preview",
|
||||
},
|
||||
"model_info": {"id": "deployment-uuid-123"},
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
called_with: dict = {}
|
||||
|
||||
async def _mock_fallback(original_function, **kwargs):
|
||||
called_with.update(kwargs)
|
||||
return {}
|
||||
|
||||
monkeypatch.setattr(router, "_ageneric_api_call_with_fallbacks", _mock_fallback)
|
||||
|
||||
native_azure_id = "cntr_6a058b43d24c8190a226cfb1d35405b20115fb7875ff11df"
|
||||
|
||||
async def _noop(**kwargs):
|
||||
return {}
|
||||
|
||||
await router._init_containers_api_endpoints(
|
||||
original_function=_noop,
|
||||
container_id=native_azure_id,
|
||||
model_id="deployment-uuid-123",
|
||||
custom_llm_provider="azure",
|
||||
)
|
||||
|
||||
assert called_with.get("model") == "deployment-uuid-123", (
|
||||
"_ageneric_api_call_with_fallbacks must be called with the forwarded "
|
||||
"model_id when the container_id carries no LiteLLM routing payload"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue