from the URL, so
countTokens 400s name their model and provider like generateContent ones do.
Non-Google hosts keep the tracked-route rule and still log nothing extra.
* fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases
The failure path only named the Google provider from the URL when the
method was one the success handler tracks or the host was a Google API
host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE
(or a Vertex one behind a custom base) still logged an empty model. A
route that declares gemini or vertex_ai now wins outright, and a URL
whose path carries /projects//locations// counts as Vertex on any
host, while a tracked method on a non-Google host no longer passes as
Vertex.
* fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model
* test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback
* fix(passthrough): scope declared provider to pass-through failures and match Google hosts only
Prometheus trusts the resolved provider as before and reads the request-declared
provider only on pass-through failures. Vertex attribution matches Google hosts,
strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the
URL path only, covers Vertex Live WebSocket failures, and labels a router alias
failure with the alias the client requested
* fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels
A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler.
* fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures
A pre-call guardrail block on a pass-through route now builds its failure
payload the same way upstream failures do, so the model and provider read
from the URL reach spend logs and metrics. Vertex Live websocket failures
(connect errors, upstream rejections, and the credential failure) now carry
the model from the ?model= query parameter under the vertex_ai provider
* fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls
Prometheus reads a failure's declared provider only on pass-through calls, so
without call_type it fell back to guessing from the model name. That labeled a
guardrail block on the Gemini route as vertex_ai and the Live credential
failure as None
* fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row
* fix(pass-through): keep URL model attribution off routes that name another provider
A built-in route that names a non-Google provider, like /openai pointed at
Gemini's OpenAI-compatible host, no longer takes its failure model from the
Google URL. A guardrail block response now returns the x-litellm-call-id of
its spend row.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
---
litellm/integrations/prometheus.py | 53 +-
litellm/llms/vertex_ai/common_utils.py | 2 +-
.../llm_passthrough_endpoints.py | 12 +-
.../gemini_passthrough_logging_handler.py | 8 +-
.../vertex_passthrough_logging_handler.py | 63 +-
.../pass_through_endpoints.py | 136 ++-
.../integrations/test_prometheus_labels.py | 97 +++
.../vertex_ai/test_vertex_ai_common_utils.py | 3 +
.../test_llm_pass_through_endpoints.py | 53 +-
.../test_pass_through_endpoints.py | 772 +++++++++++++++---
10 files changed, 1025 insertions(+), 174 deletions(-)
diff --git a/litellm/integrations/prometheus.py b/litellm/integrations/prometheus.py
index e6318525b88..5c2f0f553a8 100644
--- a/litellm/integrations/prometheus.py
+++ b/litellm/integrations/prometheus.py
@@ -82,6 +82,7 @@ from litellm.types.proxy.carried_budget_state import (
UserBudgetSnapshot,
)
from litellm.types.utils import (
+ CallTypes,
StandardLoggingGuardrailInformation,
StandardLoggingPayload,
StandardLoggingZeroCostDiagnostic,
@@ -279,6 +280,11 @@ def _logged_fields(value: object) -> Mapping[str, object]:
return _NO_LOGGED_FIELDS
+_GOOGLE_PASS_THROUGH_PROVIDERS: Final = frozenset(
+ (litellm.LlmProviders.GEMINI.value, litellm.LlmProviders.VERTEX_AI.value)
+)
+
+
def _is_llm_provider(name: str) -> bool:
routable: Final = (
name in litellm.provider_list or JSONProviderRegistry.exists(name) or is_registered_custom_provider(name)
@@ -2942,6 +2948,22 @@ class PrometheusLogger(CustomLogger):
return False
+ @staticmethod
+ def _google_pass_through_provider(request_data: Mapping[str, object]) -> str | None:
+ declared_provider: Final = request_data.get("custom_llm_provider")
+ if request_data.get("call_type") != CallTypes.pass_through.value or not isinstance(declared_provider, str):
+ return None
+ return declared_provider if declared_provider in _GOOGLE_PASS_THROUGH_PROVIDERS else None
+
+ @staticmethod
+ def _failed_request_requested_model(request_data: Mapping[str, object]) -> object:
+ requested_model: Final = request_data.get("model", "")
+ if PrometheusLogger._google_pass_through_provider(request_data) is None:
+ return requested_model
+ request_metadata: Final = _logged_fields(_logged_fields(request_data.get("litellm_params")).get("metadata"))
+ model_group: Final = request_metadata.get("model_group")
+ return model_group if isinstance(model_group, str) and model_group else requested_model
+
@staticmethod
def _extract_api_provider_from_request_data(request_data: dict) -> str | None:
"""
@@ -2950,22 +2972,27 @@ class PrometheusLogger(CustomLogger):
A request can fail before a deployment is resolved, so the provider is
not always known. Prefer the resolved ``custom_llm_provider`` on
``litellm_params``, then any provider recovered onto a partial
- ``standard_logging_object`` (e.g. a stream that broke mid-flight), and
- finally infer it from the requested model name (e.g. ``gpt-4o-mini`` ->
- ``openai``) since the proxy's failure ``request_data`` usually carries
- only the client-supplied model. Return ``None`` when it cannot be
- determined so the label emits empty rather than a guess.
+ ``standard_logging_object`` (e.g. a stream that broke mid-flight). On a
+ Gemini or Vertex pass-through failure only, next take the provider the
+ failure payload names (``gemini`` or ``vertex_ai``), so every other
+ pass-through route keeps its earlier label. Finally infer it from the
+ requested model name (e.g. ``gpt-4o-mini`` -> ``openai``) since the
+ proxy's failure ``request_data`` usually carries only the client-supplied
+ model. Return ``None`` when it cannot be determined so the label emits
+ empty rather than a guess.
"""
litellm_params: Final = request_data.get("litellm_params") or {}
- provider = litellm_params.get("custom_llm_provider")
- if provider:
- return provider
standard_logging_object: Final = request_data.get("standard_logging_object") or {}
- provider = standard_logging_object.get("custom_llm_provider")
- if provider:
- return provider
+ resolved_provider: Final = litellm_params.get("custom_llm_provider") or standard_logging_object.get(
+ "custom_llm_provider"
+ )
+ if resolved_provider:
+ return resolved_provider
+ google_provider: Final = PrometheusLogger._google_pass_through_provider(request_data)
+ if google_provider is not None:
+ return google_provider
model: Final = litellm_params.get("model") or request_data.get("model")
- if not model:
+ if not isinstance(model, str) or not model:
return None
try:
return litellm.get_llm_provider(model=model)[1] or None
@@ -3052,7 +3079,7 @@ class PrometheusLogger(CustomLogger):
team_alias=user_api_key_dict.team_alias,
org_id=user_api_key_dict.org_id,
org_alias=user_api_key_dict.organization_alias,
- requested_model=_bounded_requested_model_label(request_data.get("model", "")),
+ requested_model=_bounded_requested_model_label(self._failed_request_requested_model(request_data)),
status_code=str(status_code),
exception_status=str(status_code),
exception_class=self._get_exception_class_name(original_exception),
diff --git a/litellm/llms/vertex_ai/common_utils.py b/litellm/llms/vertex_ai/common_utils.py
index 0fa80cb2cad..ad64d852ef9 100644
--- a/litellm/llms/vertex_ai/common_utils.py
+++ b/litellm/llms/vertex_ai/common_utils.py
@@ -1257,7 +1257,7 @@ def get_vertex_model_id_from_url(url: str) -> str | None:
`https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:streamGenerateContent`
"""
- match: Final = re.search(r"/models/([^:]+)", url)
+ match: Final = re.search(r"/models/([^:?#]+)", url)
return match.group(1) if match else None
diff --git a/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py b/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
index 8de7441121f..33b18c3d939 100644
--- a/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
+++ b/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
@@ -32,6 +32,7 @@ from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm import get_llm_provider
from litellm._logging import verbose_proxy_logger
+from litellm._uuid import uuid
from litellm.constants import (
ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS,
AWS_ERROR_TYPE_HEADER,
@@ -149,7 +150,7 @@ from litellm.types.passthrough_endpoints.tinyfish import (
)
from litellm.types.passthrough_endpoints.vertex_ai import VertexPassThroughCredentials
from litellm.types.router import LiteLLMParamsTypedDict
-from litellm.types.utils import LlmProviders
+from litellm.types.utils import CallTypes, LlmProviders
from litellm.types.vector_stores import LiteLLM_ManagedVectorStore
from litellm.utils import ProviderConfigManager
@@ -4105,7 +4106,12 @@ async def vertex_ai_live_websocket_passthrough(
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
- request_data={},
+ request_data={
+ "model": model or "",
+ "custom_llm_provider": litellm.LlmProviders.VERTEX_AI.value,
+ "call_type": CallTypes.pass_through.value,
+ "litellm_call_id": str(uuid.uuid4()),
+ },
)
if websocket.client_state != WebSocketState.DISCONNECTED:
await websocket.close(code=1011, reason=VERTEX_LIVE_UNCONFIGURED_CLOSE_REASON)
@@ -4140,6 +4146,8 @@ async def vertex_ai_live_websocket_passthrough(
forward_headers=False,
endpoint="/vertex_ai/live",
accept_websocket=False,
+ builtin_provider_route=True,
+ model=model,
setup_model_rewriter=_build_vertex_live_setup_model_rewriter(
vertex_project=resolved_project,
vertex_location=resolved_location,
diff --git a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/gemini_passthrough_logging_handler.py b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/gemini_passthrough_logging_handler.py
index 7a5d5a1e259..d449f62af2c 100644
--- a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/gemini_passthrough_logging_handler.py
+++ b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/gemini_passthrough_logging_handler.py
@@ -1,4 +1,3 @@
-import re
from collections.abc import Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final
@@ -9,6 +8,7 @@ import litellm
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.gemini.videos.transformation import GeminiVideoConfig
+from litellm.llms.vertex_ai.common_utils import get_vertex_model_id_from_url
from litellm.proxy._types import PassThroughEndpointLoggingTypedDict
from litellm.proxy.pass_through_endpoints.llm_provider_handlers.vertex_passthrough_logging_handler import (
GeminiSseChunkParser,
@@ -207,11 +207,7 @@ class GeminiPassthroughLoggingHandler:
@staticmethod
def extract_model_from_url(url: str) -> str:
- pattern: Final = r"/models/([^:]+)"
- match: Final = re.search(pattern, url)
- if match:
- return match.group(1)
- return "unknown"
+ return get_vertex_model_id_from_url(url) or "unknown"
@staticmethod
def _create_gemini_response_logging_payload_for_generate_content(
diff --git a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
index b3cbad91624..d5c162296d3 100644
--- a/litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
+++ b/litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
@@ -3,7 +3,7 @@ import re
from collections.abc import Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, cast
-from urllib.parse import urlparse
+from urllib.parse import parse_qs, urlparse
import httpx
from pydantic import ConfigDict, TypeAdapter
@@ -19,6 +19,7 @@ from litellm.litellm_core_utils.llm_cost_calc.usage_object_transformation import
from litellm.llms.vertex_ai.common_utils import (
get_vertex_ai_lyria_generation_cost,
get_vertex_location_from_url,
+ get_vertex_model_id_from_url,
)
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
ModelResponseIterator as VertexModelResponseIterator,
@@ -47,6 +48,19 @@ from litellm.types.utils import (
)
vertex_search_api_config: Final = VertexSearchAPIVectorStoreConfig()
+GEMINI_API_HOST_SUFFIX: Final = "generativelanguage.googleapis.com"
+GOOGLE_AI_HOST_SUFFIXES: Final = (
+ GEMINI_API_HOST_SUFFIX,
+ "aiplatform.googleapis.com",
+ "discoveryengine.googleapis.com",
+)
+_VERTEX_MULTI_REGION_HOST: Final = re.compile(r"aiplatform\.[a-z0-9-]+\.rep\.googleapis\.com")
+
+
+def _is_host_or_subdomain(hostname: str, suffix: str) -> bool:
+ return hostname == suffix or hostname.endswith((f".{suffix}", f"-{suffix}"))
+
+
if TYPE_CHECKING:
from litellm.types.utils import LiteLLMBatch
@@ -684,11 +698,18 @@ class VertexPassthroughLoggingHandler:
@staticmethod
def extract_model_from_url(url: str) -> str:
- pattern: Final = r"/models/([^:]+)"
- match: Final = re.search(pattern, url)
- if match:
- return match.group(1)
- return "unknown"
+ return get_vertex_model_id_from_url(url) or "unknown"
+
+ @staticmethod
+ def model_from_url_route(url_route: str) -> str | None:
+ parsed_url: Final = urlparse(url_route)
+ model: Final = get_vertex_model_id_from_url(parsed_url.path)
+ if model is None:
+ return "vertex_ai/search_api" if ":search" in parsed_url.path else None
+ streamed: Final = "sse" in parse_qs(parsed_url.query).get("alt", ())
+ if parsed_url.path.endswith((":rawPredict", ":streamRawPredict")) and not streamed:
+ return f"vertex_ai/{model}"
+ return model
@staticmethod
def extract_model_name_from_vertex_path(vertex_model_path: str) -> str:
@@ -734,18 +755,32 @@ class VertexPassthroughLoggingHandler:
return None
@staticmethod
- def _get_custom_llm_provider(url: str, logging_obj: LiteLLMLoggingObj) -> str:
+ def is_google_ai_url(url: str) -> bool:
hostname: Final = urlparse(url).hostname
- if logging_obj.model_call_details.get("custom_llm_provider") == litellm.LlmProviders.GEMINI.value or (
- hostname is not None
- and (
- hostname == "generativelanguage.googleapis.com"
- or hostname.endswith(".generativelanguage.googleapis.com")
- )
- ):
+ return hostname is not None and (
+ any(_is_host_or_subdomain(hostname, suffix) for suffix in GOOGLE_AI_HOST_SUFFIXES)
+ or _VERTEX_MULTI_REGION_HOST.fullmatch(hostname) is not None
+ )
+
+ @staticmethod
+ def is_gemini_api_host(url: str) -> bool:
+ hostname: Final = urlparse(url).hostname
+ return hostname is not None and (
+ hostname == GEMINI_API_HOST_SUFFIX or hostname.endswith(f".{GEMINI_API_HOST_SUFFIX}")
+ )
+
+ @staticmethod
+ def custom_llm_provider_from_url(url: str) -> str:
+ if VertexPassthroughLoggingHandler.is_gemini_api_host(url):
return litellm.LlmProviders.GEMINI.value
return litellm.LlmProviders.VERTEX_AI.value
+ @staticmethod
+ def _get_custom_llm_provider(url: str, logging_obj: LiteLLMLoggingObj) -> str:
+ if logging_obj.model_call_details.get("custom_llm_provider") == litellm.LlmProviders.GEMINI.value:
+ return litellm.LlmProviders.GEMINI.value
+ return VertexPassthroughLoggingHandler.custom_llm_provider_from_url(url)
+
@staticmethod
def _is_multimodal_embedding_response(json_response: dict) -> bool:
"""
diff --git a/litellm/proxy/pass_through_endpoints/pass_through_endpoints.py b/litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
index bda2bfb9e04..cfac28db1a9 100644
--- a/litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
+++ b/litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
@@ -3,6 +3,7 @@ import asyncio
import copy
import json
import posixpath
+import re
import traceback
from base64 import b64encode
from collections.abc import AsyncGenerator, AsyncIterator, Callable, Iterable, Mapping, Sequence
@@ -77,6 +78,7 @@ from litellm.llms.base_llm.managed_resources.utils import (
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.llms.eu_jev.common_utils import eu_jev_scoped_model, eu_jev_upstream_body
from litellm.llms.oss_decision import validate_oss_request
+from litellm.llms.vertex_ai.common_utils import get_vertex_model_id_from_url
from litellm.passthrough import BasePassthroughUtils
from litellm.proxy._lazy_features import lazy_owned_routes
from litellm.proxy._types import (
@@ -147,11 +149,12 @@ from litellm.types.passthrough_endpoints.pass_through_endpoints import (
EndpointType,
PassthroughStandardLoggingPayload,
)
-from litellm.types.utils import TRUSTED_CALLBACK_VARS_FIELD, Usage
+from litellm.types.utils import TRUSTED_CALLBACK_VARS_FIELD, CallTypes, Usage
from .llm_provider_handlers.tinyfish_passthrough_logging_handler import (
is_tinyfish_agent_url,
)
+from .llm_provider_handlers.vertex_passthrough_logging_handler import VertexPassthroughLoggingHandler
from .streaming_handler import PassThroughStreamingHandler
from .success_handler import PassThroughEndpointLogging
from .upstream_usage_headers import (
@@ -874,13 +877,45 @@ def _carry_guardrail_logging_info(request_data: dict, guardrail_data: dict | Non
metadata.setdefault("standard_logging_guardrail_information", list(entries))
+_GOOGLE_PASSTHROUGH_PROVIDERS: Final = frozenset(
+ (litellm.LlmProviders.GEMINI.value, litellm.LlmProviders.VERTEX_AI.value)
+)
+
+
+def _passthrough_google_provider(url_route: str, custom_llm_provider: str | None) -> str | None:
+ if custom_llm_provider is not None:
+ return custom_llm_provider if custom_llm_provider in _GOOGLE_PASSTHROUGH_PROVIDERS else None
+ if VertexPassthroughLoggingHandler.is_google_ai_url(url_route):
+ return VertexPassthroughLoggingHandler.custom_llm_provider_from_url(url_route)
+ return None
+
+
+_OPERATION_POLL_SUFFIX: Final = re.compile(r"/operations/[^/]+$")
+
+
+def _without_operation_poll_suffix(model: str | None) -> str | None:
+ return _OPERATION_POLL_SUFFIX.sub("", model) if model else model
+
+
+def _passthrough_google_model(url_route: str, google_provider: str | None) -> str | None:
+ match google_provider:
+ case "gemini":
+ return _without_operation_poll_suffix(get_vertex_model_id_from_url(urlparse(url_route).path))
+ case "vertex_ai":
+ return _without_operation_poll_suffix(VertexPassthroughLoggingHandler.model_from_url_route(url_route))
+ case _:
+ return None
+
+
def _build_passthrough_failure_request_payload(
parsed_body: dict | None,
kwargs: dict | None,
logging_obj: LiteLLMLoggingObj | None,
custom_llm_provider: str | None,
upstream_usage: UpstreamReportedUsage | None = None,
+ url: httpx.URL | None = None,
upstream_model: str | None = None,
+ builtin_provider_route: bool = False,
) -> dict:
"""Build the ``request_data`` dict passed to ``post_call_failure_hook``.
@@ -893,16 +928,25 @@ def _build_passthrough_failure_request_payload(
comes paired with a usage object, so both keys are written together.
``upstream_model`` is the model name the upstream reported, which replaces
the body's ``model`` on the failure row just as it does on a success row.
+ ``builtin_provider_route`` lets a Gemini or Vertex URL name the failure's
+ model and provider. A user-defined route never gets that, since its URL
+ names an upstream model rather than a LiteLLM-managed one.
"""
request_payload: Final[dict] = dict(cast(Mapping[str, object], parsed_body or {})) # cast-ok: json body
if kwargs:
request_payload.update(kwargs)
if logging_obj is not None:
request_payload["litellm_logging_obj"] = logging_obj
- if "model" not in request_payload and parsed_body and isinstance(parsed_body, dict):
- request_payload["model"] = parsed_body.get("model", "")
- if "custom_llm_provider" not in request_payload and custom_llm_provider:
- request_payload["custom_llm_provider"] = custom_llm_provider
+ url_route: Final = str(url) if url is not None else ""
+ google_provider: Final = (
+ _passthrough_google_provider(url_route, custom_llm_provider) if builtin_provider_route else None
+ )
+ if "model" not in request_payload:
+ request_payload["model"] = _passthrough_google_model(url_route, google_provider) or ""
+ resolved_provider: Final = custom_llm_provider or google_provider
+ request_payload.pop("custom_llm_provider", None)
+ if resolved_provider:
+ request_payload["custom_llm_provider"] = resolved_provider
if upstream_usage is not None:
request_payload["response_cost"] = upstream_usage.response_cost or 0.0
request_payload["combined_usage_object"] = Usage(total_tokens=upstream_usage.total_tokens or 0)
@@ -1203,6 +1247,7 @@ async def pass_through_request(
# Initialize variables
#########################################################
litellm_call_id: Final = str(uuid.uuid4())
+ builtin_provider_route: Final = not request_dispatched_to_pass_through_endpoint(request)
url: httpx.URL | None = None
# parsed request body
@@ -1612,6 +1657,8 @@ async def pass_through_request(
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
upstream_usage=upstream_usage,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
upstream_model=upstream_model,
),
logging_obj=logging_obj,
@@ -1651,6 +1698,8 @@ async def pass_through_request(
kwargs=kwargs,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
),
),
managed_id_provider=_managed_id_provider,
@@ -1710,6 +1759,8 @@ async def pass_through_request(
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
upstream_usage=upstream_usage,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
upstream_model=upstream_model,
),
logging_obj=logging_obj,
@@ -1749,6 +1800,8 @@ async def pass_through_request(
kwargs=kwargs,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
),
),
managed_id_provider=_managed_id_provider,
@@ -1811,6 +1864,8 @@ async def pass_through_request(
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
upstream_usage=upstream_usage,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
upstream_model=upstream_model,
)
failure_request_payload["response_body"] = response_body
@@ -1991,7 +2046,14 @@ async def pass_through_request(
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
- request_data=e.request_data,
+ request_data=_build_passthrough_failure_request_payload(
+ parsed_body=e.request_data,
+ kwargs={"call_type": CallTypes.pass_through.value, "litellm_call_id": litellm_call_id},
+ logging_obj=None,
+ custom_llm_provider=custom_llm_provider,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
+ ),
)
except Exception:
verbose_proxy_logger.warning(
@@ -2010,6 +2072,7 @@ async def pass_through_request(
content=json.dumps(error_body),
status_code=200,
media_type="application/json",
+ headers={"x-litellm-call-id": litellm_call_id},
)
except Exception as e:
custom_headers = ProxyBaseLLMRequestProcessing.get_custom_headers(
@@ -2033,19 +2096,14 @@ async def pass_through_request(
# Monitoring: Trigger post_call_failure_hook
# for pass through endpoint failure
#########################################################
- request_payload: Final[dict] = _parsed_body or {}
- # add user_api_key_dict, litellm_call_id, passthrough_logging_payloa for logging
- if kwargs:
- for key, value in kwargs.items():
- request_payload[key] = value
- if logging_obj is not None:
- request_payload["litellm_logging_obj"] = logging_obj
-
- if "model" not in request_payload and _parsed_body and isinstance(_parsed_body, dict):
- request_payload["model"] = _parsed_body.get("model", "")
- if "custom_llm_provider" not in request_payload and custom_llm_provider:
- request_payload["custom_llm_provider"] = custom_llm_provider
-
+ request_payload: Final = _build_passthrough_failure_request_payload(
+ parsed_body=_parsed_body,
+ kwargs=kwargs,
+ logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ url=url,
+ builtin_provider_route=builtin_provider_route,
+ )
_carry_guardrail_logging_info(request_payload, post_call_guardrail_data)
await proxy_logging_obj.post_call_failure_hook(
@@ -2547,6 +2605,8 @@ async def websocket_passthrough_request(
cost_per_request: float | None = None,
accept_websocket: bool = True,
setup_model_rewriter: Callable[[str], str] | None = None,
+ builtin_provider_route: bool = False,
+ model: str | None = None,
):
"""
WebSocket passthrough request handler.
@@ -2560,6 +2620,8 @@ async def websocket_passthrough_request(
endpoint: The endpoint path (for logging purposes)
cost_per_request: Optional field - cost per request to the target endpoint
setup_model_rewriter: Optional rewrite of the setup frame's model before it reaches the upstream
+ builtin_provider_route: Whether a built-in provider route serves the connection, so its URL names the provider
+ model: The model the caller named before connecting, logged on a failure before the setup frame arrives
"""
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.proxy.proxy_server import proxy_config, proxy_logging_obj
@@ -2912,19 +2974,17 @@ async def websocket_passthrough_request(
except InvalidStatus as exc:
verbose_proxy_logger.exception("WebSocket passthrough (%s): upstream rejected WebSocket connection", endpoint)
- # Prepare request payload for logging
- request_payload = {}
- if kwargs:
- for key, value in kwargs.items():
- request_payload[key] = value
- if logging_obj is not None:
- request_payload["litellm_logging_obj"] = logging_obj
-
- # Log the connection failure using the same pattern as HTTP
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=exc,
- request_data=request_payload,
+ request_data=_build_passthrough_failure_request_payload(
+ parsed_body=None if model is None else {"model": model},
+ kwargs=kwargs,
+ logging_obj=logging_obj,
+ custom_llm_provider=None,
+ url=httpx.URL(target),
+ builtin_provider_route=builtin_provider_route,
+ ),
traceback_str=traceback.format_exc(
limit=MAXIMUM_TRACEBACK_LINES_TO_LOG,
),
@@ -2940,19 +3000,17 @@ async def websocket_passthrough_request(
"WebSocket passthrough (%s): unexpected error while proxying WebSocket", endpoint
)
- # Prepare request payload for logging
- request_payload = {}
- if kwargs:
- for key, value in kwargs.items():
- request_payload[key] = value
- if logging_obj is not None:
- request_payload["litellm_logging_obj"] = logging_obj
-
- # Log the unexpected error using the same pattern as HTTP
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
- request_data=request_payload,
+ request_data=_build_passthrough_failure_request_payload(
+ parsed_body=None if model is None else {"model": model},
+ kwargs=kwargs,
+ logging_obj=logging_obj,
+ custom_llm_provider=None,
+ url=httpx.URL(target),
+ builtin_provider_route=builtin_provider_route,
+ ),
traceback_str=traceback.format_exc(
limit=MAXIMUM_TRACEBACK_LINES_TO_LOG,
),
diff --git a/tests/unit/integrations/test_prometheus_labels.py b/tests/unit/integrations/test_prometheus_labels.py
index bff89b6cbf1..878f19c66f8 100644
--- a/tests/unit/integrations/test_prometheus_labels.py
+++ b/tests/unit/integrations/test_prometheus_labels.py
@@ -238,6 +238,53 @@ def test_extract_api_provider_from_request_data_failure_path():
assert extract({"model": "some-unknown-model-xyz"}) is None
+_PASS_THROUGH_CALL_TYPE: Final = "pass_through_endpoint"
+
+
+@pytest.mark.parametrize(
+ "request_data, expected_provider",
+ [
+ ({"model": "gpt-4o-mini", "call_type": "acompletion", "custom_llm_provider": "anthropic"}, "openai"),
+ ({"model": "gpt-4o-mini", "litellm_params": {"custom_llm_provider": "humanloop"}}, "humanloop"),
+ ({"model": "gemini-3.8-flash", "call_type": _PASS_THROUGH_CALL_TYPE}, "vertex_ai"),
+ (
+ {"model": "gemini-3.8-flash", "call_type": _PASS_THROUGH_CALL_TYPE, "custom_llm_provider": "gemini"},
+ "gemini",
+ ),
+ (
+ {
+ "call_type": _PASS_THROUGH_CALL_TYPE,
+ "custom_llm_provider": "gemini",
+ "standard_logging_object": {"custom_llm_provider": "vertex_ai"},
+ },
+ "vertex_ai",
+ ),
+ ],
+)
+def test_extract_api_provider_reads_the_request_declared_provider_only_on_pass_through_failures(
+ request_data: dict[str, object], expected_provider: str
+):
+ from litellm.integrations.prometheus import PrometheusLogger
+
+ assert PrometheusLogger._extract_api_provider_from_request_data(request_data) == expected_provider
+
+
+@pytest.mark.parametrize(
+ "declared_provider",
+ ["azure", "bedrock", "openrouter", "client-supplied-value", "", {"nested": "object"}, 42],
+)
+def test_extract_api_provider_ignores_pass_through_declared_values_outside_gemini_and_vertex(
+ declared_provider: object,
+):
+ from litellm.integrations.prometheus import PrometheusLogger
+
+ extract = PrometheusLogger._extract_api_provider_from_request_data
+ declared = {"call_type": _PASS_THROUGH_CALL_TYPE, "custom_llm_provider": declared_provider}
+
+ assert extract({"model": "gpt-4o-mini", **declared}) == "openai"
+ assert extract({"model": "no-such-model", **declared}) is None
+
+
def test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors():
"""
get_llm_provider raises BadRequestError for a model that maps to no
@@ -849,6 +896,56 @@ async def test_failure_hook_prefers_request_data_provider_over_exception_provide
) == ["azure"]
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "call_type, custom_llm_provider, expected_requested_model",
+ [
+ (_PASS_THROUGH_CALL_TYPE, "vertex_ai", "gcp/google/gemini-2.5-flash"),
+ ("acompletion", "vertex_ai", "other"),
+ (_PASS_THROUGH_CALL_TYPE, "bedrock", "other"),
+ ],
+)
+async def test_failure_hook_labels_a_pass_through_failure_with_the_router_model_the_client_requested(
+ call_type: str, custom_llm_provider: str, expected_requested_model: str
+):
+ from unittest.mock import patch
+
+ from litellm.integrations.prometheus import PrometheusLogger
+ from litellm.proxy._types import UserAPIKeyAuth
+
+ router = litellm.Router(
+ model_list=[
+ {
+ "model_name": "gcp/google/gemini-2.5-flash",
+ "litellm_params": {
+ "model": "vertex_ai/gemini-2.5-flash",
+ "vertex_project": "p",
+ "vertex_location": "us-central1",
+ },
+ }
+ ]
+ )
+ _clear_prometheus_registry()
+ try:
+ with patch("litellm.proxy.proxy_server.llm_router", router, create=True): # test-quality-ok: production reads proxy_server.llm_router lazily, no injection seam
+ await PrometheusLogger().async_post_call_failure_hook(
+ request_data={
+ "model": "gemini-2.5-flash",
+ "call_type": call_type,
+ "custom_llm_provider": custom_llm_provider,
+ "metadata": {},
+ "litellm_params": {"metadata": {"model_group": "gcp/google/gemini-2.5-flash"}},
+ },
+ original_exception=Exception("upstream 400"),
+ user_api_key_dict=UserAPIKeyAuth(token="tok"),
+ )
+ assert {
+ s.labels.get("requested_model") for s in _collected_samples("litellm_proxy_failed_requests_metric_total")
+ } == {expected_requested_model}
+ finally:
+ _clear_prometheus_registry()
+
+
def test_model_group_in_deployment_metrics():
"""
Test that model_group label is present on the deployment-scoped metrics
diff --git a/tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.py b/tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.py
index 152ab2ffc13..3b222be6e5e 100644
--- a/tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.py
+++ b/tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.py
@@ -1293,6 +1293,9 @@ def test_get_vertex_model_id_from_url():
model_id = get_vertex_model_id_from_url(url)
assert model_id == "gemini-pro"
+ url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro?key=provider-secret#frag"
+ assert get_vertex_model_id_from_url(url) == "gemini-pro"
+
# Test with invalid URL
url = "https://invalid-url.com"
model_id = get_vertex_model_id_from_url(url)
diff --git a/tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py b/tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
index aab227e9494..ff27ac54a5b 100644
--- a/tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
+++ b/tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
@@ -25,12 +25,14 @@ from starlette.datastructures import FormData
import litellm
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_logger import CustomLogger
+from litellm.integrations.prometheus import PrometheusLogger
from tests._master_key import MASTER_KEY as SHARED_MASTER_KEY
from litellm.caching.caching import DualCache
-from litellm.types.utils import CallTypesLiteral
+from litellm.types.utils import CallTypes, CallTypesLiteral
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException, ModifyResponseException
from litellm.proxy._types import AwsErrorTypeProxyException, ProxyException
+from litellm.proxy.spend_tracking.spend_tracking_utils import get_spend_logs_id
from litellm.proxy.utils import ProxyLogging
from litellm.passthrough.main import raise_for_provider_status
from litellm.llms.bedrock.common_utils import BedrockError
@@ -6490,6 +6492,55 @@ class TestVertexAILiveWebsocketPassthrough:
assert "default_vertex_config" in close_kwargs["reason"]
assert len(close_kwargs["reason"].encode("utf-8")) <= 123
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("model, expected_model", [("gemini-3.8-live", "gemini-3.8-live"), (None, "")])
+ async def test_credential_failure_logs_the_named_model_under_vertex_ai(
+ self, monkeypatch: pytest.MonkeyPatch, model: str | None, expected_model: str
+ ):
+ from litellm.proxy.pass_through_endpoints import (
+ llm_passthrough_endpoints as passthrough_module,
+ )
+
+ class _FailureRecorder(CustomLogger):
+ def __init__(self) -> None:
+ super().__init__()
+ self.request_data: tuple[dict[str, object], ...] = ()
+
+ async def async_post_call_failure_hook(
+ self,
+ request_data: dict[str, object],
+ original_exception: Exception,
+ user_api_key_dict: UserAPIKeyAuth,
+ traceback_str: str | None = None,
+ ) -> None:
+ self.request_data = (*self.request_data, request_data)
+
+ recorder: Final = _FailureRecorder()
+ monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", None)
+ monkeypatch.setattr(
+ "litellm.proxy.proxy_server.proxy_logging_obj", ProxyLogging(user_api_key_cache=DualCache())
+ )
+ monkeypatch.setattr(litellm, "callbacks", [recorder])
+ monkeypatch.setattr(passthrough_module.passthrough_endpoint_router, "default_vertex_config", None)
+ self._clear_vertex_env(monkeypatch)
+ ensure_token: Final = AsyncMock(side_effect=Exception("Unable to find your credentials"))
+
+ with patch.object(passthrough_module.vertex_llm_base, "ensure_access_token_async", ensure_token):
+ await passthrough_module.vertex_ai_live_websocket_passthrough(
+ websocket=self._websocket(),
+ model=model,
+ user_api_key_dict=UserAPIKeyAuth(),
+ )
+
+ assert [
+ (
+ d["model"],
+ PrometheusLogger._extract_api_provider_from_request_data(d),
+ get_spend_logs_id(CallTypes.pass_through.value, {}, d) is not None,
+ )
+ for d in recorder.request_data
+ ] == [(expected_model, "vertex_ai", True)]
+
class TestAnthropicProxyRoute:
"""The /anthropic passthrough route: custom auth headers must not clobber the
diff --git a/tests/unit/proxy/pass_through_endpoints/test_pass_through_endpoints.py b/tests/unit/proxy/pass_through_endpoints/test_pass_through_endpoints.py
index 7b62853b0e9..a9098ac3d59 100644
--- a/tests/unit/proxy/pass_through_endpoints/test_pass_through_endpoints.py
+++ b/tests/unit/proxy/pass_through_endpoints/test_pass_through_endpoints.py
@@ -30,20 +30,26 @@ from starlette.datastructures import UploadFile as StarletteUploadFile
import litellm
from litellm._logging import verbose_proxy_logger
+from litellm.caching.dual_cache import DualCache
from litellm.constants import DEFAULT_REQUEST_TIMEOUT_SECONDS
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_logger import CustomLogger
+from litellm.integrations.prometheus import PrometheusLogger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._lazy_features import LazyFeature, attach_lazy_features
from litellm.proxy._types import ProxyException, UserAPIKeyAuth
from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import router as llm_passthrough_router
+from litellm.proxy.pass_through_endpoints.llm_provider_handlers.vertex_passthrough_logging_handler import (
+ VertexPassthroughLoggingHandler,
+)
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS,
LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY,
HttpPassThroughEndpointHelpers,
InitPassThroughEndpointHelpers,
SafeRouteAdder,
+ _build_passthrough_failure_request_payload,
_registered_pass_through_routes,
_truncate_upstream_error_body,
_update_metadata_with_tags_in_header,
@@ -61,8 +67,11 @@ from litellm.proxy.pass_through_endpoints.success_handler import (
PassThroughEndpointLogging,
)
from litellm.proxy.route_llm_request import ProxyModelNotFoundError
+from litellm.proxy.spend_tracking.spend_tracking_utils import get_spend_logs_id
+from litellm.proxy.utils import ProxyLogging
from litellm.types import utils as types_utils
from litellm.types.guardrails import GuardrailEventHooks
+from litellm.types.utils import CallTypes
from litellm.types.passthrough_endpoints.pass_through_endpoints import (
LITELLM_PASS_THROUGH_DEPLOYMENT_MODEL_INFO_STATE_KEY,
LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY,
@@ -140,9 +149,7 @@ def test_is_multipart():
),
],
)
-def test_get_endpoint_type_classifies_anthropic_messages_routes(
- url: str, expected_endpoint_type: EndpointType
-) -> None:
+def test_get_endpoint_type_classifies_anthropic_messages_routes(url: str, expected_endpoint_type: EndpointType) -> None:
assert HttpPassThroughEndpointHelpers.get_endpoint_type(url) == expected_endpoint_type
@@ -430,53 +437,18 @@ async def test_pass_through_request_failure_handler():
Critical Test: When a users pass through endpoint request fails, we must log the failure code, exception in litellm spend logs.
"""
- with patch("litellm.proxy.proxy_server.proxy_logging_obj") as mock_proxy_logging:
- with patch("litellm.llms.custom_httpx.http_handler.get_async_httpx_client") as mock_get_client:
- with patch(
- "litellm.proxy.pass_through_endpoints.pass_through_endpoints.ProxyBaseLLMRequestProcessing"
- ) as mock_processing:
- # Setup mock for post_call_failure_hook and pre_call_hook
- mock_proxy_logging.post_call_failure_hook = AsyncMock()
- mock_proxy_logging.pre_call_hook = AsyncMock()
+ request_failure = httpx.HTTPError("Request failed")
+ user_api_key_dict = UserAPIKeyAuth(api_key="sk-test")
- # Setup mock for httpx client
- mock_client = MagicMock()
- mock_client.client = MagicMock()
- mock_client.client.request = AsyncMock(side_effect=httpx.HTTPError("Request failed"))
- mock_get_client.return_value = mock_client
+ failure: Final = await _run_passthrough_failure(
+ "http://test.com", None, request_failure, user_api_key_dict=user_api_key_dict
+ )
- # Mock headers for custom headers
- mock_processing.get_custom_headers.return_value = {}
-
- # Create mock request
- mock_request = MagicMock(spec=Request)
- mock_request.method = "POST"
- mock_request.body = AsyncMock(return_value=b'{"test": "data"}')
- mock_request.headers = Headers({})
-
- # Create a simple empty QueryParams
- mock_request.query_params = QueryParams({})
-
- # Create mock user API key dict
- mock_user_api_key_dict = MagicMock()
-
- # Call the function with a target that will trigger an HTTPError
- with pytest.raises(ProxyException):
- await pass_through_request(
- request=mock_request,
- target="http://test.com",
- custom_headers={},
- user_api_key_dict=mock_user_api_key_dict,
- )
-
- # Assert post_call_failure_hook was called
- mock_proxy_logging.post_call_failure_hook.assert_called_once()
-
- # Verify the arguments to post_call_failure_hook
- call_args = mock_proxy_logging.post_call_failure_hook.call_args[1]
- assert call_args["user_api_key_dict"] == mock_user_api_key_dict
- assert isinstance(call_args["original_exception"], TypeError) # Now expecting TypeError
- assert "traceback_str" in call_args
+ assert failure.user_api_key_dict is user_api_key_dict
+ assert failure.original_exception is request_failure
+ assert "Request failed" in (failure.traceback_str or "")
+ assert failure.request_data["contents"] == "not a list"
+ assert failure.request_data["model"] == ""
@pytest.mark.asyncio
@@ -3656,9 +3628,7 @@ async def test_pass_through_request_preserves_target_query_without_client_query(
@pytest.mark.asyncio
-async def test_config_pass_through_without_configured_headers_preserves_no_authorization(
- tmp_path, monkeypatch
-):
+async def test_config_pass_through_without_configured_headers_preserves_no_authorization(tmp_path, monkeypatch):
proxy = await _boot_db_backed_proxy(
tmp_path,
monkeypatch,
@@ -3725,9 +3695,7 @@ async def test_pass_through_rpm_limit_is_sequential_and_scoped_per_key(tmp_path,
proxy = await _boot_db_backed_proxy(
tmp_path,
monkeypatch,
- config_pass_through_endpoints=[
- {"path": "/rpm-rerank", "target": "http://upstream.test/rerank", "auth": True}
- ],
+ config_pass_through_endpoints=[{"path": "/rpm-rerank", "target": "http://upstream.test/rerank", "auth": True}],
db_pass_through_endpoints=[],
master_key=MASTER_KEY,
)
@@ -4902,8 +4870,10 @@ from fastapi import HTTPException as _FastAPIHTTPException
from litellm.exceptions import (
BlockedPiiEntityError,
GuardrailRaisedException,
+ ModifyResponseException,
)
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
+
_PT_MODULE = "litellm.proxy.pass_through_endpoints.pass_through_endpoints"
@@ -8182,8 +8152,10 @@ async def test_user_defined_passthrough_is_neither_tracked_nor_enforced(metadata
}
)
body: Final = {
- "model": "upstream-only-model", metadata_slot: {
- "model_group": "managed-model", "customer_label": "retained",
+ "model": "upstream-only-model",
+ metadata_slot: {
+ "model_group": "managed-model",
+ "customer_label": "retained",
"user_api_key_team_model_max_budget": budget,
},
}
@@ -8191,48 +8163,84 @@ async def test_user_defined_passthrough_is_neither_tracked_nor_enforced(metadata
assert await limiter.is_team_within_model_budget("shared-team", budget, None, "managed-model")
start: Final = datetime.now()
logging_obj: Final = LiteLLMLoggingObj(
- model="upstream-only-model", messages=[], stream=False, call_type="pass_through_endpoint",
- start_time=start, litellm_call_id="custom-budget", function_id="custom-budget", kwargs={},
+ model="upstream-only-model",
+ messages=[],
+ stream=False,
+ call_type="pass_through_endpoint",
+ start_time=start,
+ litellm_call_id="custom-budget",
+ function_id="custom-budget",
+ kwargs={},
dynamic_async_success_callbacks=[limiter],
)
payload: Final = {
- "url": "https://upstream.test/echo", "request_body": body, "request_method": "POST", "cost_per_request": 0.25,
+ "url": "https://upstream.test/echo",
+ "request_body": body,
+ "request_method": "POST",
+ "cost_per_request": 0.25,
}
kwargs: Final = HttpPassThroughEndpointHelpers._init_kwargs_for_pass_through_endpoint(
- request=request, user_api_key_dict=auth, passthrough_logging_payload=payload, logging_obj=logging_obj,
- _parsed_body=body, litellm_call_id="custom-budget",
+ request=request,
+ user_api_key_dict=auth,
+ passthrough_logging_payload=payload,
+ logging_obj=logging_obj,
+ _parsed_body=body,
+ litellm_call_id="custom-budget",
)
logging_obj.update_environment_variables(
- model="upstream-only-model", user="unknown", optional_params={},
- litellm_params=kwargs["litellm_params"], call_type="pass_through_endpoint",
+ model="upstream-only-model",
+ user="unknown",
+ optional_params={},
+ litellm_params=kwargs["litellm_params"],
+ call_type="pass_through_endpoint",
)
response: Final = httpx.Response(
- 200, request=httpx.Request("POST", "https://upstream.test/echo"), json={"ok": True},
+ 200,
+ request=httpx.Request("POST", "https://upstream.test/echo"),
+ json={"ok": True},
)
await PassThroughEndpointLogging().pass_through_async_success_handler(
- httpx_response=response, response_body={"ok": True}, request_body=body, logging_obj=logging_obj,
- url_route="https://upstream.test/echo", result=response.text, start_time=start, end_time=datetime.now(),
- cache_hit=False, **kwargs,
+ httpx_response=response,
+ response_body={"ok": True},
+ request_body=body,
+ logging_obj=logging_obj,
+ url_route="https://upstream.test/echo",
+ result=response.text,
+ start_time=start,
+ end_time=datetime.now(),
+ cache_hit=False,
+ **kwargs,
)
assert logging_obj.model_call_details["response_cost"] == 0.25
assert await limiter.is_team_within_model_budget("shared-team", budget, None, "managed-model")
metadata: Final = kwargs["litellm_params"]["metadata"]
assert (metadata["model_group"], metadata["customer_label"]) == ("managed-model", "retained")
- assert metadata.keys().isdisjoint({
- "user_api_key_model_max_budget", "user_api_key_team_model_max_budget",
- "user_api_key_user_model_max_budget", "user_api_key_end_user_model_max_budget",
- })
+ assert metadata.keys().isdisjoint(
+ {
+ "user_api_key_model_max_budget",
+ "user_api_key_team_model_max_budget",
+ "user_api_key_user_model_max_budget",
+ "user_api_key_end_user_model_max_budget",
+ }
+ )
@pytest.mark.parametrize("metadata_slot", ["metadata", "litellm_metadata"])
def test_builtin_passthrough_pins_model_group_to_the_resolved_model(metadata_slot: str) -> None:
- request: Final = Request({
- "type": "http", "method": "POST", "path": "/gemini/v1beta/models/gemini-2.5-flash:generateContent",
- "headers": [], "query_string": b"",
- })
+ request: Final = Request(
+ {
+ "type": "http",
+ "method": "POST",
+ "path": "/gemini/v1beta/models/gemini-2.5-flash:generateContent",
+ "headers": [],
+ "query_string": b"",
+ }
+ )
kwargs: Final = HttpPassThroughEndpointHelpers._init_kwargs_for_pass_through_endpoint(
- request=request, user_api_key_dict=UserAPIKeyAuth(token="hash", user_id="u-1"),
- passthrough_logging_payload=MagicMock(), logging_obj=MagicMock(),
+ request=request,
+ user_api_key_dict=UserAPIKeyAuth(token="hash", user_id="u-1"),
+ passthrough_logging_payload=MagicMock(),
+ logging_obj=MagicMock(),
_parsed_body={"contents": [], metadata_slot: {"model_group": "unbounded-client-choice"}},
)
assert kwargs["litellm_params"]["metadata"]["model_group"] == "gemini-2.5-flash"
@@ -8591,6 +8599,581 @@ def test_passthrough_client_cannot_forge_session_id_omission(client_metadata_key
)
+_GEMINI_FLASH_TARGET: Final = "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.8-flash:generateContent"
+_VERTEX_FLASH_TARGET: Final = (
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5"
+ "/publishers/google/models/gemini-3.8-flash:generateContent"
+)
+_VERTEX_CLAUDE_RAW_PREDICT_TARGET: Final = (
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5"
+ "/publishers/anthropic/models/claude-opus-5:rawPredict"
+)
+_VERTEX_SEARCH_TARGET: Final = (
+ "https://discoveryengine.googleapis.com/v1/projects/p/locations/global"
+ "/dataStores/default/servingConfigs/default:search"
+)
+_RELAYED_GEMINI_COUNT_TOKENS_TARGET: Final = "http://127.0.0.1:30732/v1beta/models/gemini-3.8-flash:countTokens"
+_TRANSLATE_TARGET: Final = "https://translate.googleapis.com/v3/projects/p/locations/global:translateText"
+_NON_GOOGLE_VERTEX_SHAPED_TARGET: Final = "https://llm.example.com/v1/projects/a/locations/b/models/my-model:predict"
+_VERTEX_LIVE_TARGET: Final = (
+ "wss://us-central1-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent"
+)
+_VEO_OPERATION_SUFFIX: Final = "/models/veo-3.0-generate-001/operations/op-8f2c1"
+_RELAYED_VERTEX_COUNT_TOKENS_TARGET: Final = (
+ "http://127.0.0.1:30732/v1/projects/p/locations/us-east5/publishers/google/models/gemini-3.8-flash:countTokens"
+)
+
+
+@dataclass(frozen=True, slots=True)
+class _RecordedFailure:
+ request_data: dict[str, object]
+ original_exception: Exception
+ user_api_key_dict: UserAPIKeyAuth
+ traceback_str: str | None
+
+
+class _FailureHookRecorder(CustomLogger):
+ def __init__(self) -> None:
+ super().__init__()
+ self.failures: tuple[_RecordedFailure, ...] = ()
+
+ async def async_post_call_failure_hook(
+ self,
+ request_data: dict[str, object],
+ original_exception: Exception,
+ user_api_key_dict: UserAPIKeyAuth,
+ traceback_str: str | None = None,
+ ) -> None:
+ self.failures = (
+ *self.failures,
+ _RecordedFailure(request_data, original_exception, user_api_key_dict, traceback_str),
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class _PassthroughRun:
+ failure: _RecordedFailure
+ response: Response | None
+
+
+async def _drained_pass_through_response(
+ request: Request,
+ target: str,
+ custom_llm_provider: str | None,
+ user_api_key_dict: UserAPIKeyAuth,
+ stream: bool,
+) -> Response | None:
+ try:
+ received: Final = await pass_through_request(
+ request=request,
+ target=target,
+ custom_headers={},
+ user_api_key_dict=user_api_key_dict,
+ custom_llm_provider=custom_llm_provider,
+ stream=stream,
+ )
+ if isinstance(received, StreamingResponse):
+ async for _ in received.body_iterator:
+ pass
+ except (ProxyException, httpx.HTTPError):
+ return None
+ return received if isinstance(received, Response) else None
+
+
+async def _run_passthrough_failure(
+ target: str,
+ custom_llm_provider: str | None,
+ upstream: httpx.Response | Exception,
+ stream: bool = False,
+ user_api_key_dict: UserAPIKeyAuth | None = None,
+ dispatched_endpoint: Callable[..., object] | None = None,
+ guardrail: CustomGuardrail | None = None,
+ body: bytes = b'{"contents": "not a list"}',
+) -> _RecordedFailure:
+ run: Final = await _run_passthrough(
+ target, custom_llm_provider, upstream, stream, user_api_key_dict, dispatched_endpoint, guardrail, body
+ )
+ return run.failure
+
+
+async def _run_passthrough(
+ target: str,
+ custom_llm_provider: str | None,
+ upstream: httpx.Response | Exception,
+ stream: bool = False,
+ user_api_key_dict: UserAPIKeyAuth | None = None,
+ dispatched_endpoint: Callable[..., object] | None = None,
+ guardrail: CustomGuardrail | None = None,
+ body: bytes = b'{"contents": "not a list"}',
+) -> _PassthroughRun:
+ recorder: Final = _FailureHookRecorder()
+ with (
+ pytest.MonkeyPatch.context() as litellm_patch,
+ patch( # test-quality-ok: proxy_logging_obj is a proxy_server module global read inside pass_through_request; there is no injection seam
+ "litellm.proxy.proxy_server.proxy_logging_obj", ProxyLogging(user_api_key_cache=DualCache())
+ ),
+ respx.mock(assert_all_called=False) as upstream_mock,
+ ):
+ litellm_patch.setattr(litellm, "disable_aiohttp_transport", True)
+ litellm_patch.setattr(litellm, "callbacks", [recorder] if guardrail is None else [guardrail, recorder])
+ litellm.in_memory_llm_clients_cache.flush_cache()
+ upstream_route: Final = upstream_mock.route(method="POST")
+ if isinstance(upstream, Exception):
+ upstream_route.mock(side_effect=upstream)
+ else:
+ upstream_route.mock(return_value=upstream)
+ mock_request: Final = MagicMock(spec=Request)
+ mock_request.method = "POST"
+ proxy_url: Final = httpx.URL(target.replace("https://", "http://test-proxy.com/"))
+ mock_request.url = proxy_url
+ mock_request.scope = {
+ "type": "http",
+ "method": "POST",
+ "path": proxy_url.path,
+ **({"endpoint": dispatched_endpoint} if dispatched_endpoint is not None else {}),
+ }
+ mock_request.body = AsyncMock(return_value=body)
+ mock_request.headers = Headers({"content-type": "application/json"})
+ mock_request.query_params = QueryParams({})
+ response: Final = await _drained_pass_through_response(
+ mock_request,
+ target,
+ custom_llm_provider,
+ user_api_key_dict or UserAPIKeyAuth(api_key="sk-test"),
+ stream,
+ )
+ await asyncio.sleep(0)
+ litellm.in_memory_llm_clients_cache.flush_cache()
+ assert len(recorder.failures) == 1
+ return _PassthroughRun(recorder.failures[0], response)
+
+
+def _upstream_400(target: str) -> httpx.Response:
+ return httpx.Response(
+ status_code=400,
+ headers={"content-type": "application/json"},
+ content=json.dumps({"error": {"code": 400, "status": "INVALID_ARGUMENT"}}).encode(),
+ request=httpx.Request("POST", target),
+ )
+
+
+def _upstream_400_as_event_stream(target: str) -> httpx.Response:
+ return httpx.Response(
+ status_code=400,
+ headers={"content-type": "text/event-stream"},
+ content=b'data: {"error": {"code": 400, "status": "INVALID_ARGUMENT"}}\n\n',
+ request=httpx.Request("POST", target),
+ )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "target, custom_llm_provider, stream, expected_provider",
+ [
+ (_GEMINI_FLASH_TARGET, "gemini", False, "gemini"),
+ (_GEMINI_FLASH_TARGET, "gemini", True, "gemini"),
+ (_VERTEX_FLASH_TARGET, None, False, "vertex_ai"),
+ (_VERTEX_FLASH_TARGET, None, True, "vertex_ai"),
+ (_GEMINI_FLASH_TARGET.replace(":generateContent", ":countTokens"), "gemini", False, "gemini"),
+ (_VERTEX_FLASH_TARGET.replace(":generateContent", ":countTokens"), None, False, "vertex_ai"),
+ (_RELAYED_GEMINI_COUNT_TOKENS_TARGET, "gemini", False, "gemini"),
+ ],
+)
+async def test_passthrough_upstream_error_logs_model_from_url(
+ target: str, custom_llm_provider: str | None, stream: bool, expected_provider: str
+):
+ failure: Final = await _run_passthrough_failure(target, custom_llm_provider, _upstream_400(target), stream=stream)
+ assert failure.request_data["model"] == "gemini-3.8-flash"
+ assert failure.request_data["custom_llm_provider"] == expected_provider
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "target",
+ [
+ _RELAYED_VERTEX_COUNT_TOKENS_TARGET,
+ _TRANSLATE_TARGET,
+ _NON_GOOGLE_VERTEX_SHAPED_TARGET,
+ ],
+)
+async def test_passthrough_upstream_error_to_a_host_outside_gemini_and_vertex_gets_no_url_attribution(target: str):
+ failure: Final = await _run_passthrough_failure(target, None, _upstream_400(target))
+ assert failure.request_data["model"] == ""
+ assert failure.request_data.get("custom_llm_provider") is None
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("target", [_GEMINI_FLASH_TARGET, _VERTEX_FLASH_TARGET])
+async def test_passthrough_upstream_error_on_a_user_defined_route_to_google_gets_no_url_attribution(target: str):
+ failure: Final = await _run_passthrough_failure(
+ target, None, _upstream_400(target), dispatched_endpoint=_marked_pass_through_endpoint()
+ )
+ assert failure.request_data["model"] == ""
+ assert failure.request_data.get("custom_llm_provider") is None
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "target, custom_llm_provider, expected_provider",
+ [
+ (_GEMINI_FLASH_TARGET, "gemini", "gemini"),
+ (_VERTEX_FLASH_TARGET, None, "vertex_ai"),
+ ],
+)
+async def test_passthrough_detected_stream_upstream_error_logs_model_from_url(
+ target: str, custom_llm_provider: str | None, expected_provider: str
+):
+ failure: Final = await _run_passthrough_failure(
+ target, custom_llm_provider, _upstream_400_as_event_stream(target), stream=False
+ )
+ assert failure.request_data["model"] == "gemini-3.8-flash"
+ assert failure.request_data["custom_llm_provider"] == expected_provider
+
+
+@pytest.mark.asyncio
+async def test_passthrough_internal_failure_logs_model_from_url():
+ failure: Final = await _run_passthrough_failure(
+ _GEMINI_FLASH_TARGET, "gemini", httpx.ReadTimeout("upstream timed out")
+ )
+ assert failure.request_data["model"] == "gemini-3.8-flash"
+ assert failure.request_data["custom_llm_provider"] == "gemini"
+
+
+def _upstream_dropping_stream(target: str) -> httpx.Response:
+ return httpx.Response(
+ status_code=200,
+ headers={"content-type": "text/event-stream"},
+ stream=_UpstreamDroppingMidStream(),
+ request=httpx.Request("POST", target),
+ )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("stream", [True, False])
+@pytest.mark.parametrize(
+ "target, custom_llm_provider, expected_provider",
+ [
+ (_GEMINI_FLASH_TARGET.replace(":generateContent", ":streamGenerateContent?alt=sse"), "gemini", "gemini"),
+ (_VERTEX_FLASH_TARGET.replace(":generateContent", ":streamGenerateContent?alt=sse"), None, "vertex_ai"),
+ ],
+)
+async def test_passthrough_mid_stream_drop_logs_model_from_url(
+ target: str, custom_llm_provider: str | None, expected_provider: str, stream: bool
+):
+ failure: Final = await _run_passthrough_failure(
+ target, custom_llm_provider, _upstream_dropping_stream(target), stream=stream
+ )
+ assert isinstance(failure.original_exception, httpx.ReadError)
+ assert failure.request_data["model"] == "gemini-3.8-flash"
+ assert failure.request_data["custom_llm_provider"] == expected_provider
+
+
+class _ModifyResponseOnPreCallGuardrail(CustomGuardrail):
+ async def async_pre_call_hook(
+ self,
+ user_api_key_dict: UserAPIKeyAuth,
+ cache: DualCache,
+ data: dict[str, object],
+ call_type: str,
+ ) -> None:
+ self.raise_passthrough_exception(violation_message="blocked by policy", request_data=data, detection_info=None)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "target, custom_llm_provider, body, expected_model, expected_provider",
+ [
+ (_GEMINI_FLASH_TARGET, "gemini", b'{"contents": "not a list"}', "gemini-3.8-flash", "gemini"),
+ (_VERTEX_FLASH_TARGET, None, b'{"contents": "not a list"}', "gemini-3.8-flash", "vertex_ai"),
+ (_VERTEX_FLASH_TARGET, None, b'{"custom_llm_provider": "gemini"}', "gemini-3.8-flash", "vertex_ai"),
+ ],
+)
+async def test_passthrough_guardrail_block_logs_model_from_url(
+ target: str, custom_llm_provider: str | None, body: bytes, expected_model: str, expected_provider: str
+):
+ guardrail: Final = _ModifyResponseOnPreCallGuardrail(
+ guardrail_name="block-policy", event_hook=GuardrailEventHooks.pre_call, default_on=True
+ )
+ failure: Final = await _run_passthrough_failure(
+ target, custom_llm_provider, _upstream_400(target), guardrail=guardrail, body=body
+ )
+ assert isinstance(failure.original_exception, ModifyResponseException)
+ assert failure.request_data["model"] == expected_model
+ assert failure.request_data["custom_llm_provider"] == expected_provider
+ assert PrometheusLogger._extract_api_provider_from_request_data(failure.request_data) == expected_provider
+ assert get_spend_logs_id(CallTypes.pass_through.value, {}, failure.request_data) is not None
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("body", [b'{"contents": "not a list"}', b'{"litellm_call_id": "client-chosen-id"}'])
+async def test_passthrough_guardrail_block_response_names_the_call_id_of_its_spend_row(body: bytes):
+ guardrail: Final = _ModifyResponseOnPreCallGuardrail(
+ guardrail_name="block-policy", event_hook=GuardrailEventHooks.pre_call, default_on=True
+ )
+ run: Final = await _run_passthrough(
+ _GEMINI_FLASH_TARGET, "gemini", _upstream_400(_GEMINI_FLASH_TARGET), guardrail=guardrail, body=body
+ )
+ assert isinstance(run.failure.original_exception, ModifyResponseException)
+ assert run.response is not None
+ assert run.response.headers["x-litellm-call-id"] == get_spend_logs_id(
+ CallTypes.pass_through.value, {}, run.failure.request_data
+ )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("body", [b'{"contents": "not a list"}', b'{"custom_llm_provider": "vertex_ai"}'])
+async def test_passthrough_guardrail_block_on_a_user_defined_route_gets_no_url_attribution(body: bytes):
+ guardrail: Final = _ModifyResponseOnPreCallGuardrail(
+ guardrail_name="block-policy", event_hook=GuardrailEventHooks.pre_call, default_on=True
+ )
+ failure: Final = await _run_passthrough_failure(
+ _GEMINI_FLASH_TARGET,
+ None,
+ _upstream_400(_GEMINI_FLASH_TARGET),
+ dispatched_endpoint=_marked_pass_through_endpoint(),
+ guardrail=guardrail,
+ body=body,
+ )
+ assert isinstance(failure.original_exception, ModifyResponseException)
+ assert failure.request_data["model"] == ""
+ assert "custom_llm_provider" not in failure.request_data
+
+
+def _websocket_connect_failure(kind: str) -> Exception:
+ from websockets.datastructures import Headers as WebSocketHeaders
+ from websockets.exceptions import InvalidStatus
+ from websockets.http11 import Response as WebSocketHandshakeResponse
+
+ if kind == "rejected_handshake":
+ return InvalidStatus(WebSocketHandshakeResponse(403, "Forbidden", WebSocketHeaders()))
+ return OSError("connection refused")
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("failure_kind", ["rejected_handshake", "unreachable_host"])
+@pytest.mark.parametrize(
+ "target, builtin_provider_route, expected_provider",
+ [
+ (_VERTEX_LIVE_TARGET, True, "vertex_ai"),
+ (_VERTEX_LIVE_TARGET, False, None),
+ ("wss://api.openai.com/v1/realtime?model=gpt-realtime", True, None),
+ ],
+)
+async def test_websocket_passthrough_connect_failure_logs_the_provider_of_the_upstream_url(
+ failure_kind: str, target: str, builtin_provider_route: bool, expected_provider: str | None
+):
+ failure: Final = await _run_websocket_connect_failure(failure_kind, target, builtin_provider_route)
+ assert failure.request_data.get("custom_llm_provider") == expected_provider
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("failure_kind", ["rejected_handshake", "unreachable_host"])
+@pytest.mark.parametrize("model, expected_model", [("gemini-3.8-live", "gemini-3.8-live"), (None, "")])
+async def test_websocket_passthrough_connect_failure_logs_the_model_the_caller_named(
+ failure_kind: str, model: str | None, expected_model: str
+):
+ failure: Final = await _run_websocket_connect_failure(failure_kind, _VERTEX_LIVE_TARGET, True, model=model)
+ assert failure.request_data["model"] == expected_model
+
+
+async def _run_websocket_connect_failure(
+ failure_kind: str, target: str, builtin_provider_route: bool, model: str | None = None
+) -> _RecordedFailure:
+ recorder: Final = _FailureHookRecorder()
+ with (
+ pytest.MonkeyPatch.context() as litellm_patch,
+ patch( # test-quality-ok: proxy_logging_obj is a proxy_server module global read inside websocket_passthrough_request; there is no injection seam
+ "litellm.proxy.proxy_server.proxy_logging_obj", ProxyLogging(user_api_key_cache=DualCache())
+ ),
+ patch(
+ "litellm.proxy.pass_through_endpoints.pass_through_endpoints.connect",
+ side_effect=_websocket_connect_failure(failure_kind),
+ ),
+ ):
+ litellm_patch.setattr(litellm, "callbacks", [recorder])
+ await websocket_passthrough_request(
+ websocket=_client_websocket(_pending_receive),
+ target=target,
+ custom_headers={},
+ user_api_key_dict=UserAPIKeyAuth(api_key="sk-test"),
+ endpoint="/vertex_ai/live",
+ accept_websocket=False,
+ builtin_provider_route=builtin_provider_route,
+ model=model,
+ )
+ assert len(recorder.failures) == 1
+ return recorder.failures[0]
+
+
+@pytest.mark.parametrize(
+ "parsed_body, url, custom_llm_provider, expected_model, expected_provider",
+ [
+ ({"model": "from-body"}, _GEMINI_FLASH_TARGET, "gemini", "from-body", "gemini"),
+ ({"contents": []}, _GEMINI_FLASH_TARGET, "gemini", "gemini-3.8-flash", "gemini"),
+ ({"contents": []}, _VERTEX_FLASH_TARGET, None, "gemini-3.8-flash", "vertex_ai"),
+ ({"contents": []}, _VERTEX_FLASH_TARGET, "anthropic", "", "anthropic"),
+ ({}, "https://generativelanguage.googleapis.com/v1beta/openai/models/gemini-3.8-flash", "openai", "", "openai"),
+ ({"custom_llm_provider": "from-body"}, _VERTEX_FLASH_TARGET, None, "gemini-3.8-flash", "vertex_ai"),
+ ({"custom_llm_provider": "gemini"}, _VERTEX_FLASH_TARGET, "anthropic", "", "anthropic"),
+ ({"contents": []}, _GEMINI_FLASH_TARGET, None, "gemini-3.8-flash", "gemini"),
+ (
+ {},
+ "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.8-flash?key=provider-secret",
+ "gemini",
+ "gemini-3.8-flash",
+ "gemini",
+ ),
+ (
+ {},
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5"
+ "/publishers/google/models/gemini-3.8-flash?key=provider-secret",
+ None,
+ "gemini-3.8-flash",
+ "vertex_ai",
+ ),
+ ({}, _VERTEX_CLAUDE_RAW_PREDICT_TARGET, None, "vertex_ai/claude-opus-5", "vertex_ai"),
+ (
+ {},
+ _VERTEX_CLAUDE_RAW_PREDICT_TARGET.replace(":rawPredict", ":streamRawPredict?alt=sse"),
+ None,
+ "claude-opus-5",
+ "vertex_ai",
+ ),
+ ({}, "https://generativelanguage.googleapis.com/v1beta/models?pageToken=/models/x", "gemini", "", "gemini"),
+ (
+ {"contents": []},
+ "https://evilaiplatform.googleapis.com/v1/projects/p/locations/l"
+ "/publishers/google/models/gemini-3.8-flash:generateContent",
+ None,
+ "",
+ None,
+ ),
+ ({}, _VERTEX_SEARCH_TARGET, None, "vertex_ai/search_api", "vertex_ai"),
+ ({}, f"{_VERTEX_FLASH_TARGET}?tag=rawPredict", None, "gemini-3.8-flash", "vertex_ai"),
+ (
+ {},
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5/batchPredictionJobs"
+ "?filter=:search",
+ None,
+ "",
+ "vertex_ai",
+ ),
+ (
+ {},
+ f"https://generativelanguage.googleapis.com/v1beta{_VEO_OPERATION_SUFFIX}",
+ "gemini",
+ "veo-3.0-generate-001",
+ "gemini",
+ ),
+ (
+ {},
+ "https://us-central1-aiplatform.googleapis.com/v1/projects/p/locations/us-central1"
+ f"/publishers/google{_VEO_OPERATION_SUFFIX}",
+ None,
+ "veo-3.0-generate-001",
+ "vertex_ai",
+ ),
+ (
+ {"contents": []},
+ "https://aiplatform.us.rep.googleapis.com/v1/projects/p/locations/us"
+ "/publishers/google/models/gemini-3.8-flash:generateContent",
+ None,
+ "gemini-3.8-flash",
+ "vertex_ai",
+ ),
+ (
+ {},
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5/batchPredictionJobs",
+ None,
+ "",
+ "vertex_ai",
+ ),
+ (
+ {"contents": []},
+ _GEMINI_FLASH_TARGET.replace(":generateContent", ":countTokens"),
+ "gemini",
+ "gemini-3.8-flash",
+ "gemini",
+ ),
+ (
+ {"contents": []},
+ _VERTEX_FLASH_TARGET.replace(":generateContent", ":countTokens"),
+ None,
+ "gemini-3.8-flash",
+ "vertex_ai",
+ ),
+ (
+ {"contents": []},
+ "https://aiplatform.googleapis.com/v1/projects/p/locations/global"
+ "/publishers/google/models/gemini-3.8-flash:countTokens",
+ None,
+ "gemini-3.8-flash",
+ "vertex_ai",
+ ),
+ (
+ {},
+ "https://us-central1-aiplatform.googleapis.com/v1/projects/p/locations/us-central1/cachedContents",
+ None,
+ "",
+ "vertex_ai",
+ ),
+ ({"contents": []}, _RELAYED_GEMINI_COUNT_TOKENS_TARGET, "gemini", "gemini-3.8-flash", "gemini"),
+ ({"contents": []}, _RELAYED_VERTEX_COUNT_TOKENS_TARGET, None, "", None),
+ ({"contents": []}, _TRANSLATE_TARGET, None, "", None),
+ ({"contents": []}, _NON_GOOGLE_VERTEX_SHAPED_TARGET, None, "", None),
+ ({"contents": []}, "https://api.example.com/v1/models/gpt-x:countTokens", None, "", None),
+ ({"contents": []}, "https://api.example.com/v1/models/gpt-x:predict", None, "", None),
+ ({"contents": []}, "https://api.example.com/v1/models/gpt-x:thing", None, "", None),
+ (None, None, None, "", None),
+ ],
+)
+def test_build_passthrough_failure_request_payload_model_and_provider_precedence(
+ parsed_body: dict | None,
+ url: str | None,
+ custom_llm_provider: str | None,
+ expected_model: str,
+ expected_provider: str | None,
+):
+ request_payload: Final = _build_passthrough_failure_request_payload(
+ parsed_body=parsed_body,
+ kwargs=None,
+ logging_obj=None,
+ custom_llm_provider=custom_llm_provider,
+ url=httpx.URL(url) if url else None,
+ builtin_provider_route=True,
+ )
+ assert request_payload["model"] == expected_model
+ assert request_payload.get("custom_llm_provider") == expected_provider
+
+
+@pytest.mark.parametrize(
+ "url_route, expected_model",
+ [
+ (_VERTEX_FLASH_TARGET, "gemini-3.8-flash"),
+ (_VERTEX_FLASH_TARGET.replace(":generateContent", ":streamGenerateContent?alt=sse"), "gemini-3.8-flash"),
+ (_VERTEX_FLASH_TARGET.replace(":generateContent", ":predict"), "gemini-3.8-flash"),
+ (_VERTEX_FLASH_TARGET.replace(":generateContent", ":embedContent"), "gemini-3.8-flash"),
+ (_VERTEX_CLAUDE_RAW_PREDICT_TARGET, "vertex_ai/claude-opus-5"),
+ (_VERTEX_CLAUDE_RAW_PREDICT_TARGET.replace(":rawPredict", ":streamRawPredict"), "vertex_ai/claude-opus-5"),
+ (
+ _VERTEX_CLAUDE_RAW_PREDICT_TARGET.replace(":rawPredict", ":streamRawPredict?alt=sse"),
+ "claude-opus-5",
+ ),
+ (_VERTEX_CLAUDE_RAW_PREDICT_TARGET.replace(":rawPredict", ":rawPredict?alt=sse"), "claude-opus-5"),
+ (f"{_VERTEX_FLASH_TARGET}?tag=rawPredict", "gemini-3.8-flash"),
+ (_VERTEX_SEARCH_TARGET, "vertex_ai/search_api"),
+ ("https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5/batchPredictionJobs", None),
+ (
+ "https://us-east5-aiplatform.googleapis.com/v1/projects/p/locations/us-east5/batchPredictionJobs"
+ "?filter=:search",
+ None,
+ ),
+ ],
+)
+def test_vertex_failure_model_name_matches_success_handler_naming(url_route: str, expected_model: str | None):
+ assert VertexPassthroughLoggingHandler.model_from_url_route(url_route) == expected_model
+
+
@pytest.mark.parametrize("client_metadata_key", ["litellm_metadata", "metadata"])
def test_passthrough_logs_the_resolved_deployment_model_info_over_the_request_body(client_metadata_key: str):
"""A provider route that resolved a router deployment stashes its model_info on request.state. That
@@ -8598,7 +9181,9 @@ def test_passthrough_logs_the_resolved_deployment_model_info_over_the_request_bo
the call to (LIT-1761: passthrough successes carried model_id="")."""
mock_request = MagicMock(spec=Request)
mock_request.method = "POST"
- mock_request.url = httpx.URL("http://0.0.0.0:4000/vertex_ai/v1/projects/p/locations/global/publishers/google/models/gemini-3.8-flash:generateContent")
+ mock_request.url = httpx.URL(
+ "http://0.0.0.0:4000/vertex_ai/v1/projects/p/locations/global/publishers/google/models/gemini-3.8-flash:generateContent"
+ )
mock_request.headers = Headers({})
mock_request.scope = {}
mock_request.state = SimpleNamespace(
@@ -9668,12 +10253,14 @@ async def _drain_logging_worker():
await GLOBAL_LOGGING_WORKER.stop()
yield
+
@pytest.fixture()
def _vcr_outcome_gate(request, vcr):
install_live_call_probe(request, vcr)
yield
record_vcr_outcome(request, vcr)
+
@pytest.mark.usefixtures("_drain_logging_worker", "_vcr_outcome_gate")
def test_update_pass_through_route_updates_registry():
"""
@@ -9763,9 +10350,7 @@ def mock_request():
return self._dict.values()
class MockRequest:
- def __init__(
- self, headers=None, method="POST", request_body: Optional[dict] = None
- ):
+ def __init__(self, headers=None, method="POST", request_body: Optional[dict] = None):
self.headers = headers or {}
self.query_params = QueryParams()
self.method = method
@@ -9873,9 +10458,9 @@ def test_pass_through_routes_support_all_methods():
path = route.path
methods = set(route.methods)
allowed = PROTOCOL_CONSTRAINED_PASS_THROUGH_ROUTES.get(path, expected_methods)
- assert (
- methods == allowed
- ), f"Route {path} does not support all methods. Supported: {methods}, Expected: {allowed}"
+ assert methods == allowed, (
+ f"Route {path} does not support all methods. Supported: {methods}, Expected: {allowed}"
+ )
check_router_methods(llm_router)
@@ -9905,19 +10490,11 @@ def test_is_bedrock_agent_runtime_route():
# Test agent runtime endpoints (should return True)
assert _is_bedrock_agent_runtime_route("/knowledgebases/kb-123/retrieve") is True
- assert (
- _is_bedrock_agent_runtime_route("/agents/knowledgebases/kb-123/retrieve")
- is True
- )
+ assert _is_bedrock_agent_runtime_route("/agents/knowledgebases/kb-123/retrieve") is True
# Test regular bedrock runtime endpoints (should return False)
- assert (
- _is_bedrock_agent_runtime_route("/guardrail/test-id/version/1/apply") is False
- )
- assert (
- _is_bedrock_agent_runtime_route("/model/cohere.command-r-v1:0/converse")
- is False
- )
+ assert _is_bedrock_agent_runtime_route("/guardrail/test-id/version/1/apply") is False
+ assert _is_bedrock_agent_runtime_route("/model/cohere.command-r-v1:0/converse") is False
assert _is_bedrock_agent_runtime_route("/some/random/endpoint") is False
@@ -9984,9 +10561,7 @@ def test_custom_pricing_used_in_cost_calculation():
# Verify custom cost is different from standard cost (unless prices happen to match)
# This confirms custom pricing is actually being applied
- assert (
- custom_cost != standard_cost
- ), "Custom pricing should produce different cost than standard pricing"
+ assert custom_cost != standard_cost, "Custom pricing should produce different cost than standard pricing"
# Test 3: Custom pricing with cache_read_input_token_cost and input_cost_per_token_batches
# This specifically tests the parameters that were causing the original issue
@@ -10562,6 +11137,7 @@ def test_init_kwargs_client_metadata_cannot_mark_a_pass_through_call_as_a_silent
assert "is_silent_experiment" not in json.loads(row["metadata"])
assert result["litellm_params"]["metadata"]["client_key"] == "kept"
+
@pytest.mark.asyncio
async def test_direct_passthrough_keeps_mapped_baggage_tags_outside_upstream_body():
from litellm.proxy.common_utils.http_parsing_utils import read_request_body
From 8ac9b757370a27f51de80a8dcc84b26f743fe583 Mon Sep 17 00:00:00 2001
From: "devin-ai-integration[bot]"
<158243242+devin-ai-integration[bot]@users.noreply.github.com>
Date: Fri, 30 Oct 2026 09:43:01 -0700
Subject: [PATCH 13/87] fix(model_prices): rolling registry audit, add Gemini
batch and Azure Astra flex tiers (#49114)
* fix(model_prices): add EmpirioLabs model rows from the provider catalog
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): align EmpirioLabs rows with the registry endpoint vocabulary and schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): absorb Gemini long-prompt batch rates and Azure GPT-6 Astra long-context flex rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): drop EmpirioLabs rows from the registry audit
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
...odel_prices_and_context_window_backup.json | 36 ++++++++++++++-----
model_prices_and_context_window.json | 36 ++++++++++++++-----
2 files changed, 54 insertions(+), 18 deletions(-)
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index 0799a298942..28a8fe83fa2 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -8715,7 +8715,7 @@
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014
},
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
+ "source": "https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@@ -8742,7 +8742,11 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
- "supports_xhigh_reasoning_effort": true
+ "supports_xhigh_reasoning_effort": true,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-05,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1e-06,
+ "input_cost_per_token_above_272k_tokens_flex": 1e-05,
+ "output_cost_per_token_above_272k_tokens_flex": 3.75e-05
},
"azure/gpt-6-astra-2026-09-03": {
"cache_creation_input_token_cost": 1.25e-05,
@@ -8763,7 +8767,7 @@
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014
},
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
+ "source": "https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@@ -8790,7 +8794,11 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
- "supports_xhigh_reasoning_effort": true
+ "supports_xhigh_reasoning_effort": true,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-05,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1e-06,
+ "input_cost_per_token_above_272k_tokens_flex": 1e-05,
+ "output_cost_per_token_above_272k_tokens_flex": 3.75e-05
},
"azure/gpt-6-luna": {
"cache_creation_input_token_cost": 1.25e-07,
@@ -30856,8 +30864,9 @@
"output_cost_per_token": 0,
"output_vector_size": 3072,
"rpm": 10000,
- "source": "https://ai.google.dev/gemini-api/docs/embeddings#model-versions",
- "tpm": 10000000
+ "source": "https://ai.google.dev/gemini-api/docs/pricing",
+ "tpm": 10000000,
+ "input_cost_per_token_batches": 7.5e-08
},
"gemini/gemini-embedding-2": {
"input_cost_per_audio_token": 6.5e-06,
@@ -31713,7 +31722,10 @@
"input_cost_per_token_flex": 6.25e-07,
"output_cost_per_token_batches": 5e-06,
"output_cost_per_token_flex": 5e-06,
- "supports_url_context": true
+ "supports_url_context": true,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 2.5e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 1.25e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 7.5e-06
},
"gemini/gemini-2.5-computer-use-preview-10-2025": {
"deprecation_date": "2026-07-28",
@@ -32280,7 +32292,10 @@
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
- "output_cost_per_token_flex": 6e-06
+ "output_cost_per_token_flex": 6e-06,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 4e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 2e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 9e-06
},
"gemini/gemini-3.1-pro-preview-customtools": {
"prompt_cache_min_tokens": 4096,
@@ -32343,7 +32358,10 @@
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
- "output_cost_per_token_flex": 6e-06
+ "output_cost_per_token_flex": 6e-06,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 4e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 2e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 9e-06
},
"gemini-3-flash-preview": {
"cache_read_input_audio_token_cost": 1e-07,
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
index 0799a298942..28a8fe83fa2 100644
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -8715,7 +8715,7 @@
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014
},
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
+ "source": "https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@@ -8742,7 +8742,11 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
- "supports_xhigh_reasoning_effort": true
+ "supports_xhigh_reasoning_effort": true,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-05,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1e-06,
+ "input_cost_per_token_above_272k_tokens_flex": 1e-05,
+ "output_cost_per_token_above_272k_tokens_flex": 3.75e-05
},
"azure/gpt-6-astra-2026-09-03": {
"cache_creation_input_token_cost": 1.25e-05,
@@ -8763,7 +8767,7 @@
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014
},
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
+ "source": "https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@@ -8790,7 +8794,11 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
- "supports_xhigh_reasoning_effort": true
+ "supports_xhigh_reasoning_effort": true,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-05,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1e-06,
+ "input_cost_per_token_above_272k_tokens_flex": 1e-05,
+ "output_cost_per_token_above_272k_tokens_flex": 3.75e-05
},
"azure/gpt-6-luna": {
"cache_creation_input_token_cost": 1.25e-07,
@@ -30856,8 +30864,9 @@
"output_cost_per_token": 0,
"output_vector_size": 3072,
"rpm": 10000,
- "source": "https://ai.google.dev/gemini-api/docs/embeddings#model-versions",
- "tpm": 10000000
+ "source": "https://ai.google.dev/gemini-api/docs/pricing",
+ "tpm": 10000000,
+ "input_cost_per_token_batches": 7.5e-08
},
"gemini/gemini-embedding-2": {
"input_cost_per_audio_token": 6.5e-06,
@@ -31713,7 +31722,10 @@
"input_cost_per_token_flex": 6.25e-07,
"output_cost_per_token_batches": 5e-06,
"output_cost_per_token_flex": 5e-06,
- "supports_url_context": true
+ "supports_url_context": true,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 2.5e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 1.25e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 7.5e-06
},
"gemini/gemini-2.5-computer-use-preview-10-2025": {
"deprecation_date": "2026-07-28",
@@ -32280,7 +32292,10 @@
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
- "output_cost_per_token_flex": 6e-06
+ "output_cost_per_token_flex": 6e-06,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 4e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 2e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 9e-06
},
"gemini/gemini-3.1-pro-preview-customtools": {
"prompt_cache_min_tokens": 4096,
@@ -32343,7 +32358,10 @@
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
- "output_cost_per_token_flex": 6e-06
+ "output_cost_per_token_flex": 6e-06,
+ "cache_read_input_token_cost_above_200k_tokens_batches": 4e-07,
+ "input_cost_per_token_above_200k_tokens_batches": 2e-06,
+ "output_cost_per_token_above_200k_tokens_batches": 9e-06
},
"gemini-3-flash-preview": {
"cache_read_input_audio_token_cost": 1e-07,
From 9432bf6b8ee686d0fbd06863e5f75889d383c8a2 Mon Sep 17 00:00:00 2001
From: tavren27 <348969968+tavren27@users.noreply.github.com>
Date: Fri, 30 Oct 2026 10:01:44 -0700
Subject: [PATCH 14/87] perf(streaming): stop raising caught AttributeErrors
inside pydantic on every streamed chunk (#49166)
* perf(streaming): stop raising AttributeErrors inside pydantic on every streamed chunk
Every later plain chunk on openai and azure streams raised and caught 12 AttributeErrors inside pydantic. Assigning an undeclared extra goes through a class lookup that raises in the model metaclass and is never memoized, and the hidden_params getter went through the private attribute fallback
set_extra_attribute writes the extra and fields_set directly when a cached per-class check says pydantic would do the same, and falls back to setattr otherwise. The hidden_params getter reads the private storage directly. Emitted chunk JSON is unchanged
* fix(streaming): match pydantic's own extra-field bookkeeping on every supported version
Take the direct extra write only when the installed pydantic itself adds an assigned extra to
model_fields_set (2.13 and later) and the class defines no attribute of that name, so pydantic
2.11, 2.12 and 2.14 see the same model state as a plain setattr. A deleted _hidden_params raises
AttributeError again, and the mutable test models carry frozen-ok reasons.
* fix(streaming): keep the normal write for models that do not allow extra fields
openai's construct gives every model an extras dict, so a forbid model built that way still needs
setattr's ValueError instead of a silent extra write.
* test(streaming): cover a ClassVar declared without a value in the extra-field parity test
* fix(streaming): type the streamed choice's logprobs as the parsed value main now stores
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
---
litellm/litellm_core_utils/core_helpers.py | 7 +-
litellm/litellm_core_utils/hidden_params.py | 35 ++-
.../litellm_core_utils/streaming_handler.py | 17 +-
litellm/types/utils.py | 32 ++-
.../litellm_core_utils/test_hidden_params.py | 139 +++++++++++-
.../test_streaming_handler.py | 211 +++++++++++++++---
6 files changed, 388 insertions(+), 53 deletions(-)
diff --git a/litellm/litellm_core_utils/core_helpers.py b/litellm/litellm_core_utils/core_helpers.py
index 2f81154ab83..20e10f63f83 100644
--- a/litellm/litellm_core_utils/core_helpers.py
+++ b/litellm/litellm_core_utils/core_helpers.py
@@ -9,12 +9,13 @@ from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, TypeVar, cast, runtime_checkable
import httpx
-from pydantic import TypeAdapter, ValidationError
+from pydantic import BaseModel, TypeAdapter, ValidationError
from typing_extensions import TypeIs # noqa: TID251 # narrows a duck-typed value to the Protocol its caller reads
import litellm
from litellm._logging import verbose_logger
from litellm.constants import KUBERNETES_POD_ROUTING_KEY
+from litellm.litellm_core_utils.hidden_params import set_extra_attribute
from litellm.litellm_core_utils.llm_response_utils.get_headers import get_provider_request_id
from litellm.types.llms.openai import AllMessageValues, OpenAIChatCompletionFinishReason
@@ -710,9 +711,9 @@ def preserve_upstream_non_openai_attributes(
_assign_attributes(model_response, original_chunk.model_dump(include=frozenset(candidate_keys)).items())
-def _assign_attributes(target: object, values: Iterable[tuple[str, object]]) -> None:
+def _assign_attributes(target: BaseModel, values: Iterable[tuple[str, object]]) -> None:
for key, value in values:
- setattr(target, key, value)
+ set_extra_attribute(target, key, value)
def safe_deep_copy(data):
diff --git a/litellm/litellm_core_utils/hidden_params.py b/litellm/litellm_core_utils/hidden_params.py
index a74337d2549..c04d7614b90 100644
--- a/litellm/litellm_core_utils/hidden_params.py
+++ b/litellm/litellm_core_utils/hidden_params.py
@@ -1,8 +1,9 @@
from collections.abc import Iterator, Mapping, MutableMapping
+from functools import cache, lru_cache
from types import MappingProxyType
from typing import Final, Protocol, cast
-from pydantic import BaseModel
+from pydantic import BaseModel, ConfigDict, create_model
from litellm.types.llms.base import HiddenParams
@@ -60,6 +61,38 @@ def stored_attribute(obj: object, name: str) -> object:
return extras.get(name)
+@cache
+def _assigned_extras_join_fields_set() -> bool:
+ probe: Final = create_model("_ExtraAssignmentProbe", __config__=ConfigDict(extra="allow"))()
+ setattr(probe, "assigned", None)
+ return "assigned" in probe.model_fields_set
+
+
+@lru_cache(maxsize=1024)
+def _writes_plain_extra(model_class: type[BaseModel], name: str) -> bool:
+ config: Final = model_class.model_config
+ return (
+ model_class.__setattr__ is BaseModel.__setattr__
+ and config.get("extra") == "allow"
+ and not config.get("frozen")
+ and not config.get("validate_assignment")
+ and not name.startswith("_")
+ and name not in model_class.__pydantic_fields__
+ and name not in model_class.__class_vars__
+ and not hasattr(model_class, name)
+ and _assigned_extras_join_fields_set()
+ )
+
+
+def set_extra_attribute(model: BaseModel, name: str, value: object) -> None:
+ extras: Final = model.__pydantic_extra__
+ if extras is None or not _writes_plain_extra(type(model), name):
+ setattr(model, name, value)
+ return
+ extras[name] = value
+ model.__pydantic_fields_set__.add(name)
+
+
def _get_hidden_params_storage(obj: object) -> object | None:
if isinstance(obj, dict):
return obj.get(_HIDDEN_PARAMS_ATTR)
diff --git a/litellm/litellm_core_utils/streaming_handler.py b/litellm/litellm_core_utils/streaming_handler.py
index 098a404ea14..b1b3e20f218 100644
--- a/litellm/litellm_core_utils/streaming_handler.py
+++ b/litellm/litellm_core_utils/streaming_handler.py
@@ -28,6 +28,7 @@ from litellm.litellm_core_utils.core_helpers import chunk_usage, provider_report
from litellm.litellm_core_utils.hidden_params import (
HIDDEN_PARAMS_ATTR,
get_hidden_params,
+ set_extra_attribute,
set_hidden_param,
stored_attribute,
)
@@ -1035,7 +1036,7 @@ class CustomStreamWrapper:
_initial_delta: Final = model_response.choices[0].delta.model_dump()
_initial_delta.pop("role", None)
- model_response.choices[0].delta = Delta(**_initial_delta)
+ set_extra_attribute(model_response.choices[0], "delta", Delta(**_initial_delta))
return model_response
def _prebuilt_plain_delta(self, chunk: object) -> Delta | None:
@@ -1061,13 +1062,13 @@ class CustomStreamWrapper:
def _set_role_on_fresh_delta(self, model_response: ModelResponseStream) -> None:
delta: Final = model_response.choices[0].delta
if self.sent_first_chunk is False:
- delta.role = "assistant"
+ set_extra_attribute(delta, "role", "assistant")
self.sent_first_chunk = True
return
if not _holds_only_scalars(delta):
self.strip_role_from_delta(model_response)
return
- delta.role = None
+ set_extra_attribute(delta, "role", None)
if "provider_specific_fields" not in delta.model_fields_set:
delta.provider_specific_fields = None
@@ -1186,8 +1187,8 @@ class CustomStreamWrapper:
and _holds_only_scalars(upstream_delta)
)
if can_reuse_choice and upstream_choice is not None:
- model_response.choices[0].finish_reason = None # rebind-ok: keep reused choice
- model_response.choices[0].index = upstream_choice.index # rebind-ok: keep reused choice
+ set_extra_attribute(model_response.choices[0], "finish_reason", None)
+ set_extra_attribute(model_response.choices[0], "index", upstream_choice.index)
_prepend_upstream_choice_extras(
model_response.choices[0],
upstream_choice,
@@ -1212,11 +1213,7 @@ class CustomStreamWrapper:
else:
return
model_response.system_fingerprint = upstream_chunk.system_fingerprint
- setattr(
- model_response,
- "citations",
- stored_attribute(upstream_chunk, "citations"),
- )
+ set_extra_attribute(model_response, "citations", stored_attribute(upstream_chunk, "citations"))
preserve_upstream_non_openai_attributes(
model_response=model_response,
original_chunk=original_chunk,
diff --git a/litellm/types/utils.py b/litellm/types/utils.py
index 55aaf606175..f8115571dfd 100644
--- a/litellm/types/utils.py
+++ b/litellm/types/utils.py
@@ -75,6 +75,7 @@ from litellm.types.workload_identity import (
)
from ..litellm_core_utils.core_helpers import map_finish_reason, process_response_headers
+from ..litellm_core_utils.hidden_params import set_extra_attribute
from . import litellm_params as _litellm_params
from .agents import LiteLLMSendMessageResponse
from .guardrails import GuardrailEventHooks
@@ -2149,6 +2150,13 @@ class UnreportedUsage(Usage):
class StreamingChoices(OpenAIObject):
+ if TYPE_CHECKING:
+ finish_reason: str | None
+ index: int
+ delta: Delta
+ enhancements: object
+ logprobs: ChoiceLogprobs | JsonValue
+
def __init__(
self,
finish_reason=None,
@@ -2162,25 +2170,22 @@ class StreamingChoices(OpenAIObject):
# https://github.com/BerriAI/litellm/issues/8455
params.pop("message", None)
super().__init__(**params)
- if finish_reason:
- self.finish_reason = map_finish_reason(finish_reason)
- else:
- self.finish_reason = None
- self.index = index
+ set_extra_attribute(self, "finish_reason", map_finish_reason(finish_reason) if finish_reason else None)
+ set_extra_attribute(self, "index", index)
if delta is not None:
if isinstance(delta, Delta):
- self.delta = delta
+ set_extra_attribute(self, "delta", delta)
elif isinstance(delta, dict):
- self.delta = Delta(**delta)
+ set_extra_attribute(self, "delta", Delta(**delta))
else:
- self.delta = Delta()
+ set_extra_attribute(self, "delta", Delta())
if enhancements is not None:
- self.enhancements = enhancements
+ set_extra_attribute(self, "enhancements", enhancements)
if isinstance(logprobs, (dict, BaseModel)):
- self.logprobs = CHOICE_LOGPROBS_OR_JSON.validate_python(logprobs)
+ set_extra_attribute(self, "logprobs", CHOICE_LOGPROBS_OR_JSON.validate_python(logprobs))
else:
- self.logprobs = logprobs
+ set_extra_attribute(self, "logprobs", logprobs)
def __contains__(self, key) -> bool:
# Define custom behavior for the 'in' operator
@@ -2234,7 +2239,10 @@ class ModelResponseBase(OpenAIObject):
@property
def hidden_params(self) -> dict[str, object]: # mutable-ok: API requires mutation
- return self._hidden_params
+ private: Final = self.__pydantic_private__
+ if private is None or "_hidden_params" not in private:
+ return self._hidden_params
+ return private["_hidden_params"]
@hidden_params.setter
def hidden_params(self, hidden_params: dict[str, object]) -> None: # mutable-ok: API requires mutation
diff --git a/tests/unit/litellm_core_utils/test_hidden_params.py b/tests/unit/litellm_core_utils/test_hidden_params.py
index c0485a07696..b32692c67ba 100644
--- a/tests/unit/litellm_core_utils/test_hidden_params.py
+++ b/tests/unit/litellm_core_utils/test_hidden_params.py
@@ -1,7 +1,11 @@
-from typing import Final
+import pickle
+from collections.abc import Callable
+from typing import ClassVar, Final
import pytest
+from openai import BaseModel as OpenAIBaseModel
from openai.types.chat import ChatCompletionChunk
+from pydantic import BaseModel, ConfigDict, PrivateAttr
from litellm.litellm_core_utils.hidden_params import (
HIDDEN_PARAMS_ATTR,
@@ -9,13 +13,14 @@ from litellm.litellm_core_utils.hidden_params import (
get_hidden_params_storage,
get_or_create_hidden_params,
served_from_cache,
+ set_extra_attribute,
set_hidden_param,
set_hidden_params,
)
from litellm.llms.openai.completion.transformation import OpenAITextCompletionConfig
from litellm.types.decisions import DecisionsResponse
from litellm.types.llms.base import HiddenParams
-from litellm.types.utils import ModelResponse, TextChoices, TextCompletionResponse, Usage
+from litellm.types.utils import ModelResponse, ModelResponseStream, TextChoices, TextCompletionResponse, Usage
def test_get_and_set_hidden_params_on_plain_object() -> None:
@@ -309,3 +314,133 @@ def test_served_from_cache_is_true_only_for_a_response_the_cache_stamped(
assert served_from_cache(model_response) is expected
assert served_from_cache({HIDDEN_PARAMS_ATTR: dict(hidden_params)}) is expected
assert served_from_cache(object()) is False
+
+
+class _ExtraModel(BaseModel): # frozen-ok: the assignment tests need a model that accepts attribute writes
+ model_config = ConfigDict(extra="allow")
+
+ declared: str | None = None
+ marker: ClassVar[str | None] = None
+ unset_marker: ClassVar[int]
+ _secret: str = PrivateAttr(default="hidden")
+
+ @property
+ def doubled(self) -> str | None:
+ return self.declared
+
+ @doubled.setter
+ def doubled(self, value: str) -> None:
+ self.declared = value * 2
+
+
+class _FrozenExtraModel(BaseModel):
+ model_config = ConfigDict(extra="allow", frozen=True)
+
+
+class _ValidatedExtraModel(BaseModel): # frozen-ok: the assignment tests need a model that accepts attribute writes
+ model_config = ConfigDict(extra="allow", validate_assignment=True)
+
+ __pydantic_extra__: dict[str, int]
+
+
+class _IgnoredExtraModel(BaseModel): # frozen-ok: the assignment tests need a model that accepts attribute writes
+ model_config = ConfigDict(extra="ignore")
+
+
+class _MirroringModel(BaseModel): # frozen-ok: the assignment tests need a model that accepts attribute writes
+ model_config = ConfigDict(extra="allow")
+
+ def __setattr__(self, name: str, value: object) -> None:
+ super().__setattr__(name, value)
+ super().__setattr__(f"{name}_copy", value)
+
+
+class _NoneClassAttribute:
+ inherited = None
+
+
+class _InheritingExtraModel(_NoneClassAttribute, BaseModel): # frozen-ok: assignment tests need attribute writes
+ model_config = ConfigDict(extra="allow")
+
+
+class _ForbiddingOpenAIModel(OpenAIBaseModel): # frozen-ok: assignment tests need attribute writes
+ model_config = ConfigDict(extra="forbid")
+
+
+def _assignment_outcome(model: BaseModel, assign: Callable[[], None], name: str) -> tuple[object, ...]:
+ try:
+ assign()
+ except (AttributeError, ValueError) as error:
+ return type(error), str(error)
+ return model.model_dump(), getattr(model, name), model.model_fields_set, model.model_dump_json()
+
+
+@pytest.mark.parametrize(
+ "build, name, value",
+ [
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "fresh", 1, id="new-extra"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "existing", 2, id="existing-extra-keeps-order"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "declared", "x", id="declared-field"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "doubled", "ab", id="property"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "_free", 1, id="undeclared-underscore-name"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "_secret", "x", id="private-attribute"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "marker", 1, id="class-var"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "unset_marker", 1, id="class-var-without-value"),
+ pytest.param(lambda: _ExtraModel(existing=1, later=3), "model_config", 1, id="class-attribute"),
+ pytest.param(_FrozenExtraModel, "fresh", 1, id="frozen"),
+ pytest.param(_ValidatedExtraModel, "fresh", "5", id="validate-assignment"),
+ pytest.param(_IgnoredExtraModel, "fresh", 1, id="extra-ignore"),
+ pytest.param(_MirroringModel, "fresh", 1, id="setattr-override"),
+ pytest.param(_InheritingExtraModel, "inherited", 1, id="class-attribute-set-to-none"),
+ pytest.param(
+ lambda: _ForbiddingOpenAIModel.construct(existing=1), "fresh", 1, id="extra-forbid-built-by-construct"
+ ),
+ ],
+)
+def test_set_extra_attribute_matches_setattr(build: Callable[[], BaseModel], name: str, value: object) -> None:
+ through_setattr: Final = build()
+ through_helper: Final = build()
+
+ expected: Final = _assignment_outcome(through_setattr, lambda: setattr(through_setattr, name, value), name)
+ actual: Final = _assignment_outcome(through_helper, lambda: set_extra_attribute(through_helper, name, value), name)
+
+ assert actual == expected
+
+
+@pytest.mark.parametrize("response_type", [ModelResponse, ModelResponseStream])
+def test_hidden_params_raises_attribute_error_once_deleted(
+ response_type: type[ModelResponse | ModelResponseStream],
+) -> None:
+ response: Final = response_type()
+ del response._hidden_params
+
+ assert getattr(response, "hidden_params", "fallback") == "fallback"
+
+
+def _round_trip(response: ModelResponse | ModelResponseStream) -> ModelResponse | ModelResponseStream:
+ return pickle.loads(pickle.dumps(response))
+
+
+@pytest.mark.parametrize("response_type", [ModelResponse, ModelResponseStream])
+@pytest.mark.parametrize(
+ "rebuild",
+ [
+ pytest.param(lambda response: response, id="init"),
+ pytest.param(lambda response: response.model_copy(), id="model-copy"),
+ pytest.param(lambda response: response.model_copy(deep=True), id="deep-copy"),
+ pytest.param(_round_trip, id="pickle"),
+ ],
+)
+def test_hidden_params_is_the_stored_dict_on_every_construction_path(
+ response_type: type[ModelResponse | ModelResponseStream],
+ rebuild: Callable[[ModelResponse | ModelResponseStream], ModelResponse | ModelResponseStream],
+) -> None:
+ original: Final = response_type()
+ original.hidden_params["model_id"] = "deployment-1"
+ response: Final = rebuild(original)
+
+ response.hidden_params["api_base"] = "https://example.com"
+
+ assert response.hidden_params is response._hidden_params
+ assert get_hidden_params(response) == response._hidden_params
+ assert response._hidden_params["api_base"] == "https://example.com"
diff --git a/tests/unit/litellm_core_utils/test_streaming_handler.py b/tests/unit/litellm_core_utils/test_streaming_handler.py
index b9c8feb223d..148b7193ef6 100644
--- a/tests/unit/litellm_core_utils/test_streaming_handler.py
+++ b/tests/unit/litellm_core_utils/test_streaming_handler.py
@@ -11,7 +11,7 @@ import time
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
-from types import CodeType, FrameType, MappingProxyType
+from types import CodeType, FrameType, MappingProxyType, TracebackType
from typing import AsyncIterator, Callable, Final, Iterator, Literal, NoReturn, Optional, cast
from unittest.mock import AsyncMock, MagicMock, Mock, patch
@@ -6292,53 +6292,76 @@ def test_openai_chunk_creator_skips_full_chunk_dump_and_delta_rebuild(
assert finish_output.choices[0].delta.content is None
-def _openai_chunk(chunk_id: str, delta: Mapping[str, object]) -> ChatCompletionChunk:
+def _openai_chunk(
+ chunk_id: str,
+ delta: Mapping[str, object],
+ choice_extras: Mapping[str, object] = MappingProxyType({}),
+ finish_reason: str | None = None,
+ chunk_extras: Mapping[str, object] = MappingProxyType({}),
+) -> ChatCompletionChunk:
return ChatCompletionChunk(
id=chunk_id,
created=1,
model="gpt-4o",
object="chat.completion.chunk",
- choices=[{"index": 0, "delta": delta, "finish_reason": None}],
+ choices=[{"index": 0, "delta": delta, "finish_reason": finish_reason, **choice_extras}],
+ **chunk_extras,
)
-def _missed_attributes(wrapper: CustomStreamWrapper, chunks: Sequence[ChatCompletionChunk]) -> tuple[str, ...]:
- attribute_lookup: Final = BaseModel.__getattr__.__code__
- misses: Final[list[str]] = []
+def _raised_attribute_errors(wrapper: CustomStreamWrapper, chunks: Sequence[ChatCompletionChunk]) -> tuple[str, ...]:
+ raised: Final[list[str]] = []
- def record_miss(frame: FrameType, event: str, arg: object) -> Callable[[FrameType, str, object], object]:
- if event == "exception" and isinstance(arg, tuple) and isinstance(arg[1], AttributeError):
- misses.append(str(frame.f_locals["item"]))
- return record_miss
-
- def trace_lookups(frame: FrameType, event: str, _arg: object) -> Callable[[FrameType, str, object], object] | None:
- return record_miss if event == "call" and frame.f_code is attribute_lookup else None
+ def record_raise(frame: FrameType, event: str, arg: object) -> Callable[[FrameType, str, object], object]:
+ if event == "exception" and isinstance(arg, tuple):
+ _, error, traceback = arg
+ if isinstance(error, AttributeError) and isinstance(traceback, TracebackType) and traceback.tb_next is None:
+ raised.append(f"{frame.f_code.co_qualname}: {error}")
+ return record_raise
previous_tracer: Final = sys.gettrace()
- sys.settrace(trace_lookups)
+ sys.settrace(record_raise)
try:
tuple(wrapper.chunk_creator(chunk=chunk) for chunk in chunks)
finally:
sys.settrace(previous_tracer)
- return tuple(misses)
+ return tuple(raised)
-def test_openai_chunk_creator_builds_one_delta_and_misses_no_attribute_per_later_chunk(
- logging_obj: Logging,
+_AZURE_CHOICE_EXTRAS: Final = MappingProxyType(
+ {"content_filter_results": {"hate": {"filtered": False, "severity": "safe"}}}
+)
+_OPENAI_AND_AZURE_CHOICE_EXTRAS: Final = (
+ pytest.param("openai", MappingProxyType({}), id="openai"),
+ pytest.param("azure", _AZURE_CHOICE_EXTRAS, id="azure"),
+)
+
+
+@pytest.mark.parametrize(("custom_llm_provider", "choice_extras"), _OPENAI_AND_AZURE_CHOICE_EXTRAS)
+def test_openai_chunk_creator_builds_one_delta_and_raises_no_attribute_error_per_later_chunk(
+ logging_obj: Logging, custom_llm_provider: str, choice_extras: Mapping[str, object]
) -> None:
wrapper: Final = CustomStreamWrapper(
completion_stream=iter(()),
model="gpt-4o",
- custom_llm_provider="openai",
+ custom_llm_provider=custom_llm_provider,
logging_obj=logging_obj,
)
- wrapper.chunk_creator(chunk=_openai_chunk("chunk-1", {"role": "assistant", "content": ""}))
- plain_chunks: Final = (
- _openai_chunk("chunk-2", {"content": "hello"}),
- _openai_chunk("chunk-3", {"content": " world"}),
+
+ def later_chunk(chunk_id: str, delta: Mapping[str, object]) -> ChatCompletionChunk:
+ return _openai_chunk(
+ chunk_id, delta, choice_extras, chunk_extras={"service_tier": "default", "obfuscation": chunk_id}
+ )
+
+ nested_delta: Final = {"content": "!", "reasoning_details": [{"type": "reasoning.text", "text": "thought"}]}
+ first_chunk_of_each_shape: Final = (
+ later_chunk("chunk-1", {"role": "assistant", "content": ""}),
+ later_chunk("chunk-2", nested_delta),
)
- nested_chunk: Final = _openai_chunk(
- "chunk-4", {"content": "!", "reasoning_details": [{"type": "reasoning.text", "text": "thought"}]}
+ tuple(wrapper.chunk_creator(chunk=chunk) for chunk in first_chunk_of_each_shape)
+ plain_chunks: Final = (
+ later_chunk("chunk-3", {"content": "hello"}),
+ later_chunk("chunk-4", {"content": " world"}),
)
delta_init: Final = Delta.__init__.__code__
streaming_choices_init: Final = StreamingChoices.__init__.__code__
@@ -6348,7 +6371,145 @@ def test_openai_chunk_creator_builds_one_delta_and_misses_no_attribute_per_later
assert sum(call.code is delta_init for call in calls) == len(plain_chunks)
assert sum(call.code is streaming_choices_init for call in calls) == len(plain_chunks)
assert [output.choices[0].delta.content if output else None for output in outputs] == ["hello", " world"]
- assert _missed_attributes(wrapper, (*plain_chunks, nested_chunk)) == ()
+ assert _raised_attribute_errors(wrapper, (*plain_chunks, later_chunk("chunk-5", nested_delta))) == ()
+
+
+_OPENAI_STREAM_JSON: Final = (
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"finish_reason":null,"index":0,'
+ '"delta":{"provider_specific_fields":null,"refusal":null,"content":"","role":"assistant",'
+ '"function_call":null,"tool_calls":null,"audio":null},"logprobs":null}],'
+ '"provider_specific_fields":null,"citations":null,"service_tier":"default","obfuscation":"o0"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"finish_reason":null,"index":0,'
+ '"delta":{"provider_specific_fields":null,"refusal":null,"content":"one","role":null,'
+ '"function_call":null,"tool_calls":null,"audio":null},"logprobs":null}],'
+ '"provider_specific_fields":null,"citations":null,"service_tier":"default","obfuscation":"o1"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"finish_reason":null,"index":0,'
+ '"delta":{"provider_specific_fields":null,"refusal":null,"content":" two","role":null,'
+ '"function_call":null,"tool_calls":null,"audio":null},"logprobs":null}],'
+ '"provider_specific_fields":null,"citations":null,"service_tier":"default","obfuscation":"o2"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"finish_reason":"stop","index":0,'
+ '"delta":{"provider_specific_fields":null,"content":null,"role":null,"function_call":null,'
+ '"tool_calls":null,"audio":null},"logprobs":null}],"provider_specific_fields":null,'
+ '"service_tier":"default","obfuscation":"o3"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[],"provider_specific_fields":null,'
+ '"usage":{"completion_tokens":3,"prompt_tokens":9,"total_tokens":12,"completion_tokens_details":null,'
+ '"prompt_tokens_details":null}}'
+ ),
+)
+_AZURE_STREAM_JSON: Final = (
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"content_filter_results":{"hate":{"filtered":false,'
+ '"severity":"safe"}},"finish_reason":null,"index":0,"delta":{"provider_specific_fields":null,'
+ '"refusal":null,"content":"","role":"assistant","function_call":null,"tool_calls":null,"audio":null},'
+ '"logprobs":null}],"provider_specific_fields":null,"citations":null,"service_tier":"default",'
+ '"obfuscation":"o0"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"content_filter_results":{"hate":{"filtered":false,'
+ '"severity":"safe"}},"finish_reason":null,"index":0,"delta":{"provider_specific_fields":null,'
+ '"refusal":null,"content":"one","role":null,"function_call":null,"tool_calls":null,"audio":null},'
+ '"logprobs":null}],"provider_specific_fields":null,"citations":null,"service_tier":"default",'
+ '"obfuscation":"o1"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"content_filter_results":{"hate":{"filtered":false,'
+ '"severity":"safe"}},"finish_reason":null,"index":0,"delta":{"provider_specific_fields":null,'
+ '"refusal":null,"content":" two","role":null,"function_call":null,"tool_calls":null,"audio":null},'
+ '"logprobs":null}],"provider_specific_fields":null,"citations":null,"service_tier":"default",'
+ '"obfuscation":"o2"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[{"finish_reason":"stop","index":0,'
+ '"delta":{"provider_specific_fields":null,"content":null,"role":null,"function_call":null,'
+ '"tool_calls":null,"audio":null},"logprobs":null}],"provider_specific_fields":null,'
+ '"service_tier":"default","obfuscation":"o3"}'
+ ),
+ (
+ '{"id":"chatcmpl-1","created":1,"model":"gpt-4o","object":"chat.completion.chunk",'
+ '"system_fingerprint":null,"choices":[],"provider_specific_fields":null,'
+ '"usage":{"completion_tokens":3,"prompt_tokens":9,"total_tokens":12,"completion_tokens_details":null,'
+ '"prompt_tokens_details":null}}'
+ ),
+)
+
+
+@pytest.mark.parametrize(
+ ("custom_llm_provider", "choice_extras", "expected_json"),
+ [
+ pytest.param("openai", MappingProxyType({}), _OPENAI_STREAM_JSON, id="openai"),
+ pytest.param("azure", _AZURE_CHOICE_EXTRAS, _AZURE_STREAM_JSON, id="azure"),
+ ],
+)
+def test_openai_chunk_creator_emits_each_chunk_as_the_same_json(
+ logging_obj: Logging,
+ custom_llm_provider: str,
+ choice_extras: Mapping[str, object],
+ expected_json: tuple[str | None, ...],
+) -> None:
+ wrapper: Final = CustomStreamWrapper(
+ completion_stream=iter(()),
+ model="gpt-4o",
+ custom_llm_provider=custom_llm_provider,
+ logging_obj=logging_obj,
+ stream_options={"include_usage": True},
+ )
+ chunks: Final = (
+ _openai_chunk(
+ "chatcmpl-1",
+ {"role": "assistant", "content": ""},
+ choice_extras,
+ chunk_extras={"service_tier": "default", "obfuscation": "o0"},
+ ),
+ _openai_chunk(
+ "chatcmpl-1",
+ {"content": "one"},
+ choice_extras,
+ chunk_extras={"service_tier": "default", "obfuscation": "o1"},
+ ),
+ _openai_chunk(
+ "chatcmpl-1",
+ {"content": " two"},
+ choice_extras,
+ chunk_extras={"service_tier": "default", "obfuscation": "o2"},
+ ),
+ _openai_chunk(
+ "chatcmpl-1", {}, choice_extras, "stop", chunk_extras={"service_tier": "default", "obfuscation": "o3"}
+ ),
+ ChatCompletionChunk(
+ id="chatcmpl-1",
+ created=1,
+ model="gpt-4o",
+ object="chat.completion.chunk",
+ choices=[],
+ service_tier="default",
+ obfuscation="o4",
+ usage={"prompt_tokens": 9, "completion_tokens": 3, "total_tokens": 12},
+ ),
+ )
+
+ outputs: Final = tuple(wrapper.chunk_creator(chunk=chunk) for chunk in chunks)
+
+ assert tuple(
+ output.model_copy(update={"created": 1}).model_dump_json() if output else None for output in outputs
+ ) == expected_json
def test_chunk_creator_skips_full_dump_of_model_response_stream_chunks(
From ab3f6caddcd096b820359a7ae2829e9440b77c4f Mon Sep 17 00:00:00 2001
From: "devin-ai-integration[bot]"
<158243242+devin-ai-integration[bot]@users.noreply.github.com>
Date: Fri, 30 Oct 2026 10:18:29 -0700
Subject: [PATCH 15/87] fix(gemini): flag computer use support on the four
models the Gemini computer use docs list (#49252)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
litellm/model_prices_and_context_window_backup.json | 10 +++++++---
model_prices_and_context_window.json | 10 +++++++---
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index 28a8fe83fa2..e1131117fcf 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -31883,7 +31883,8 @@
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
- "google_maps_grounding_cost_per_query": 0.014
+ "google_maps_grounding_cost_per_query": 0.014,
+ "supports_computer_use": true
},
"gemini/gemini-3-flash-preview": {
"cache_read_input_audio_token_cost": 1e-07,
@@ -31949,7 +31950,8 @@
"input_cost_per_token_flex": 2.5e-07,
"output_cost_per_token_batches": 1.5e-06,
"output_cost_per_token_flex": 1.5e-06,
- "supports_audio_input": true
+ "supports_audio_input": true,
+ "supports_computer_use": true
},
"gemini/gemini-3.5-flash": {
"prompt_cache_min_tokens": 4096,
@@ -32196,7 +32198,8 @@
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
- "google_maps_grounding_cost_per_query": 0.014
+ "google_maps_grounding_cost_per_query": 0.014,
+ "supports_computer_use": true
},
"gemini/gemini-omni-flash-preview": {
"input_cost_per_audio_token": 1.5e-06,
@@ -76558,6 +76561,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
+ "supports_computer_use": true,
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
index 28a8fe83fa2..e1131117fcf 100644
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -31883,7 +31883,8 @@
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
- "google_maps_grounding_cost_per_query": 0.014
+ "google_maps_grounding_cost_per_query": 0.014,
+ "supports_computer_use": true
},
"gemini/gemini-3-flash-preview": {
"cache_read_input_audio_token_cost": 1e-07,
@@ -31949,7 +31950,8 @@
"input_cost_per_token_flex": 2.5e-07,
"output_cost_per_token_batches": 1.5e-06,
"output_cost_per_token_flex": 1.5e-06,
- "supports_audio_input": true
+ "supports_audio_input": true,
+ "supports_computer_use": true
},
"gemini/gemini-3.5-flash": {
"prompt_cache_min_tokens": 4096,
@@ -32196,7 +32198,8 @@
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
- "google_maps_grounding_cost_per_query": 0.014
+ "google_maps_grounding_cost_per_query": 0.014,
+ "supports_computer_use": true
},
"gemini/gemini-omni-flash-preview": {
"input_cost_per_audio_token": 1.5e-06,
@@ -76558,6 +76561,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
+ "supports_computer_use": true,
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
From 372c096388f3102419e0177f501f42126c7170c7 Mon Sep 17 00:00:00 2001
From: Etienne Chabert
Date: Fri, 30 Oct 2026 18:20:00 +0100
Subject: [PATCH 16/87] fix(proxy): evict a deleted key from every worker's
auth cache (#47609)
* fix(proxy): evict a deleted key from every worker's auth cache
/key/delete dropped the key object only from the in-memory cache of the worker
that served the call. Auth serves key objects cache-first, so every other
worker, and the Redis copy when enable_redis_auth_cache is on, kept
authenticating the deleted key until user_api_key_cache_ttl expired. /key/update
already evicts and broadcasts through evict_and_broadcast, delete now does the
same for the tokens it deletes
* fix(proxy): evict only the deleted rows and broadcast them in one batch
Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker
* refactor(proxy): drop the comment above the delete eviction call
* fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait
* test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests
* test(proxy): wait for the peer to apply the broadcast instead of polling its cache
* test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer
* fix(key_management): evict every requested key on delete, including keys whose row is already gone
A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone
* test(key_management): split the gone-row eviction assertion into one comprehension per worker
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
---
litellm/proxy/auth/auth_checks.py | 37 +--
.../key_management_endpoints.py | 15 +-
.../management/test_key_delete_revocation.py | 15 +-
tests/unit/proxy/auth/test_auth_checks.py | 50 +++-
.../test_key_management_endpoints.py | 231 +++++++++++++++++-
5 files changed, 317 insertions(+), 31 deletions(-)
diff --git a/litellm/proxy/auth/auth_checks.py b/litellm/proxy/auth/auth_checks.py
index f80296f4bd7..79a46ef355e 100644
--- a/litellm/proxy/auth/auth_checks.py
+++ b/litellm/proxy/auth/auth_checks.py
@@ -3457,28 +3457,35 @@ async def delete_cache_key_objects(
proxy_logging_obj: ProxyLogging | None,
) -> None:
"""
- Evict a batch of key objects, for callers that delete keys in bulk rather than through
- `/key/delete`. Auth resolves a cached key object without re-reading its team, so a key left
- cached after its row is gone keeps buying access until its TTL expires.
-
- Evicting locally only reaches this worker; `_delete_cache_key_object` itself broadcasts each
- token, so a deleted key left in a peer worker's in-memory cache still authenticates there until
- its TTL expires.
+ Evict a batch of deleted key objects here and on every other worker. Auth resolves a cached key
+ object without re-reading its team, so a key left cached after its row is gone keeps buying
+ access until its TTL expires.
Best-effort per key: the rows are already deleted by the time this runs, so an unreachable
cache backend must not abort the caller partway through its own cascade.
- Every token is dropped from this worker's memory first, then from the auth cache and the
- internal usage cache in one batch each, so a deleted key stops authenticating here the instant
- its rows are gone instead of refilling from Redis on the next miss. Every other worker then
- gets the eviction in one pipelined broadcast: a publish per key would pass the publisher's
- pending cap on a bulk delete, and the dropped keys would keep authenticating there until their
- TTL.
+ Every token is dropped from this worker's memory first, then from the auth cache's Redis layer,
+ so a deleted key stops authenticating here the instant its rows are gone instead of refilling
+ from Redis on the next miss. Every other worker then gets the eviction in one pipelined
+ broadcast: a publish per key would pass the publisher's pending cap on a bulk delete, and the
+ dropped keys would keep authenticating there until their TTL. The internal usage cache is not
+ an auth refill source, so its batch delete runs beside that chain and a hung Redis costs one
+ bounded wait there instead of another in line.
"""
evict_local(cache_keys=hashed_tokens, user_api_key_cache=user_api_key_cache)
+ usage_eviction: Final = (
+ ()
+ if proxy_logging_obj is None
+ else (evict_shared(cache_keys=hashed_tokens, cache=proxy_logging_obj.internal_usage_cache.dual_cache),)
+ )
+ _ = await asyncio.gather(
+ _evict_auth_cache_then_broadcast(hashed_tokens=hashed_tokens, user_api_key_cache=user_api_key_cache),
+ *usage_eviction,
+ )
+
+
+async def _evict_auth_cache_then_broadcast(hashed_tokens: Sequence[str], user_api_key_cache: UserApiKeyCache) -> None:
await evict_shared(cache_keys=hashed_tokens, cache=user_api_key_cache)
- if proxy_logging_obj is not None:
- await evict_shared(cache_keys=hashed_tokens, cache=proxy_logging_obj.internal_usage_cache.dual_cache)
await broadcast_evictions(cache_keys=hashed_tokens)
diff --git a/litellm/proxy/management_endpoints/key_management_endpoints.py b/litellm/proxy/management_endpoints/key_management_endpoints.py
index a6814b593ca..3021b21de51 100644
--- a/litellm/proxy/management_endpoints/key_management_endpoints.py
+++ b/litellm/proxy/management_endpoints/key_management_endpoints.py
@@ -50,6 +50,7 @@ from litellm.proxy.auth.auth_checks import ( # noqa: F401 # legacy module expo
_delete_cache_key_object, # pyright: ignore[reportPrivateUsage,reportUnusedImport] # backwards-compatible package export
can_team_access_model,
delete_cache_key_object,
+ delete_cache_key_objects,
get_jwt_key_mapping_cache_keys_for_token,
get_key_end_user_budget_id,
get_org_object,
@@ -78,7 +79,7 @@ from litellm.proxy.common_utils.config_sync_pubsub import (
from litellm.proxy.common_utils.openai_error_payload import openai_error_param, proxy_exception_for
from litellm.proxy.common_utils.rbac_utils import check_org_admin_can_generate_keys
from litellm.proxy.common_utils.timezone_utils import get_budget_reset_time
-from litellm.proxy.common_utils.user_api_key_cache import AUTH_OBJECTS_TARGET, UserApiKeyCache
+from litellm.proxy.common_utils.user_api_key_cache import AUTH_OBJECTS_TARGET, UserApiKeyCache, is_user_key_cache_key
from litellm.proxy.hooks.key_management_event_hooks import KeyManagementEventHooks
from litellm.proxy.hooks.model_max_budget_limiter import build_model_max_budget_usage
from litellm.proxy.management.teams.authz import TEAM_ADMIN_ONLY, TEAM_OR_ORG_ADMIN, is_team_admin
@@ -5483,7 +5484,7 @@ async def delete_verification_tokens(
- List of keys being deleted, this contains information about the key_alias, token, and user_id being deleted,
this is passed down to the KeyManagementEventHooks to delete the keys from the secret manager and handle audit logs
"""
- from litellm.proxy.proxy_server import prisma_client
+ from litellm.proxy.proxy_server import prisma_client, proxy_logging_obj
failed_tokens: list = []
try:
@@ -5567,11 +5568,11 @@ async def delete_verification_tokens(
verbose_proxy_logger.debug(traceback.format_exc())
raise e
- for key in tokens:
- user_api_key_cache.delete_cache(key)
- # remove hash token from cache
- hashed_token = hash_token(cast(str, key))
- user_api_key_cache.delete_cache(hashed_token)
+ await delete_cache_key_objects(
+ hashed_tokens=tuple(token for token in hashed_tokens if is_user_key_cache_key(token)),
+ user_api_key_cache=user_api_key_cache,
+ proxy_logging_obj=proxy_logging_obj,
+ )
# After credential invalidation, so a failure here can never keep a deleted key alive.
for deleted_key in authorized_keys:
diff --git a/tests/integration/management/test_key_delete_revocation.py b/tests/integration/management/test_key_delete_revocation.py
index 5b3527a52ae..bdc91dfe5f6 100644
--- a/tests/integration/management/test_key_delete_revocation.py
+++ b/tests/integration/management/test_key_delete_revocation.py
@@ -171,11 +171,8 @@ def test_deleting_a_key_revokes_it(gateway: Gateway, spelling: str) -> None:
assert deleted_rows == (_DeletedKeyRow(token=key_hash),)
-def test_deleting_a_key_revokes_it_on_a_warmed_peer(gateway: Gateway, peer: Gateway) -> None:
- pytest.skip(
- "BUG: POST /key/delete does not broadcast the key eviction, so a warmed peer keeps answering 200 for about "
- "60 seconds until its in-memory cache TTL expires"
- )
+@pytest.mark.parametrize("spelling", ("keys", "hashed_keys", "key_aliases"))
+def test_deleting_a_key_revokes_it_on_a_warmed_peer(gateway: Gateway, peer: Gateway, spelling: str) -> None:
backend: Final = "integration-keys-auth-delete"
prompts: Final = (
"delete warm primary " + uuid.uuid4().hex,
@@ -208,9 +205,13 @@ def test_deleting_a_key_revokes_it_on_a_warmed_peer(gateway: Gateway, peer: Gate
)
assert len(warm_responses) == 2
assert tuple((request.method, request.target) for request in wire.drain()) == (_EXPECTED_CHAT,) * 2
- deleted: Final = gateway.request("POST", "/key/delete", {"keys": [key]})
+ identifier: Final = {"keys": key, "hashed_keys": sha256(key.encode()).hexdigest(), "key_aliases": alias}[
+ spelling
+ ]
+ delete_body: Final = {"key_aliases" if spelling == "key_aliases" else "keys": [identifier]}
+ deleted: Final = gateway.request("POST", "/key/delete", delete_body)
assert deleted.status_code == 200, deleted.text
- assert _DeleteResponse.model_validate_json(deleted.content).deleted_keys == (key,), deleted.text
+ assert _DeleteResponse.model_validate_json(deleted.content).deleted_keys == (identifier,), deleted.text
peer_refusal: Final = eventually(
lambda: _chat(peer, model, key, prompts[2]),
lambda response: response.status_code == 401,
diff --git a/tests/unit/proxy/auth/test_auth_checks.py b/tests/unit/proxy/auth/test_auth_checks.py
index e783f01769e..9d4fe67614e 100644
--- a/tests/unit/proxy/auth/test_auth_checks.py
+++ b/tests/unit/proxy/auth/test_auth_checks.py
@@ -46,7 +46,7 @@ from litellm.proxy.auth.auth_checks import (
_team_soft_budget_check,
common_checks,
)
-from litellm.proxy.utils import ProxyLogging
+from litellm.proxy.utils import InternalUsageCache, ProxyLogging
from litellm.proxy.utils import CallInfo
from litellm.types.passthrough_endpoints.pass_through_endpoints import LITELLM_PASS_THROUGH_ENDPOINT_MARKER
@@ -2425,3 +2425,51 @@ async def test_bulk_eviction_clears_the_redis_copies_before_waiting_on_a_wedged_
assert refilled == []
assert redis.entries == {}
+
+
+class _SignallingPublishRedisClient(_SlowPublishRedisClient):
+ def __init__(self) -> None:
+ super().__init__()
+ self.sent = asyncio.Event()
+
+ async def _send(self, messages: Sequence[str]) -> None:
+ await super()._send(messages)
+ self.sent.set()
+
+
+class _DeleteAfterBroadcastRedisCache(_AuthRedisCache):
+ def __init__(self, broadcast_sent: asyncio.Event) -> None:
+ super().__init__({})
+ self._broadcast_sent = broadcast_sent
+ self.finished = False
+
+ async def delete_cache_keys(self, keys: Sequence[str]) -> None:
+ await self._broadcast_sent.wait()
+ self.finished = True
+
+
+@pytest.mark.asyncio
+async def test_bulk_eviction_broadcasts_while_the_usage_cache_redis_is_wedged(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(pubsub_module, "_in_flight_publishes", asyncio.Semaphore(16))
+ monkeypatch.setattr(pubsub_module, "_pending_publishes", set())
+ client: Final = _SignallingPublishRedisClient()
+ usage_redis: Final = _DeleteAfterBroadcastRedisCache(broadcast_sent=client.sent)
+ proxy_logging_obj: Final = ProxyLogging(user_api_key_cache=None)
+ proxy_logging_obj.internal_usage_cache = InternalUsageCache(
+ dual_cache=DualCache(redis_cache=usage_redis) # pyright: ignore[reportArgumentType] # a dict-backed fake of the Redis layer
+ )
+ hashed_tokens: Final = ("token-a", "token-b")
+
+ with patch(
+ "litellm.proxy.common_utils.auth_cache_invalidation_pubsub.coordination_redis_cache",
+ return_value=_PubSubRedisCache(client),
+ ):
+ await delete_cache_key_objects(
+ hashed_tokens=hashed_tokens, user_api_key_cache=UserApiKeyCache(), proxy_logging_obj=proxy_logging_obj
+ )
+ await _drain_publishes()
+
+ assert usage_redis.finished
+ assert {json.loads(message)["cache_key"] for message in client.published} == set(hashed_tokens)
diff --git a/tests/unit/proxy/management_endpoints/test_key_management_endpoints.py b/tests/unit/proxy/management_endpoints/test_key_management_endpoints.py
index 62dd367b8e5..c26a4bf19b8 100644
--- a/tests/unit/proxy/management_endpoints/test_key_management_endpoints.py
+++ b/tests/unit/proxy/management_endpoints/test_key_management_endpoints.py
@@ -7,7 +7,7 @@ from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
-from uuid import UUID
+from uuid import UUID, uuid4
import pytest
import yaml
@@ -16,6 +16,7 @@ from fastapi.testclient import TestClient
from pydantic import JsonValue, ValidationError
import litellm
+from litellm.caching.in_memory_cache import InMemoryCache
from litellm.constants import PROXY_DB_LOOKUP_STALL_WINDOW_SECONDS
from litellm.litellm_core_utils.duration_parser import duration_in_seconds
from litellm.models.object_permission import LiteLLM_ObjectPermissionTable
@@ -41,12 +42,15 @@ from litellm.proxy._types import (
RegenerateKeyRequest,
ResetSpendRequest,
UpdateKeyRequest,
+ hash_token,
)
from litellm.proxy.auth.auth_checks import (
delete_cache_key_object,
+ get_key_object,
jwt_key_mapping_cache_key,
)
from litellm.proxy.auth.user_api_key_auth import UserAPIKeyAuth
+from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import AuthCacheInvalidationSubscriber
from litellm.proxy.common_utils.user_api_key_cache import UserApiKeyCache, project_cache_key
from litellm.proxy.db.db_lookup_gate import db_lookup_stall_tracker
from litellm.proxy.management_endpoints.key_management_endpoints import (
@@ -5755,6 +5759,231 @@ async def test_delete_verification_tokens_evicts_jwt_key_mapping_cache(monkeypat
)
+class _ChannelSubscription:
+ """Marks a message done when the subscriber comes back for the next one, so ``messages.join()``
+ returns once every delivered message has been applied"""
+
+ def __init__(self) -> None:
+ self.messages: asyncio.Queue[dict[str, object]] = asyncio.Queue()
+ self._holding_message = False
+
+ async def subscribe(self, *channels: str) -> None:
+ return None
+
+ async def get_message(self, *, ignore_subscribe_messages: bool, timeout: float) -> dict[str, object] | None:
+ if self._holding_message:
+ self._holding_message = False
+ self.messages.task_done()
+ try:
+ message: Final = await asyncio.wait_for(self.messages.get(), timeout)
+ except asyncio.TimeoutError:
+ return None
+ self._holding_message = True
+ return message
+
+ async def aclose(self) -> None:
+ return None
+
+
+class _ChannelPipeline:
+ def __init__(self, redis: "_SharedCoordinationRedis") -> None:
+ self._redis = redis
+ self._messages: tuple[str, ...] = ()
+
+ def publish(self, channel: str, message: str) -> "_ChannelPipeline":
+ self._messages = (*self._messages, message)
+ return self
+
+ async def execute(self) -> None:
+ for message in self._messages:
+ self._redis.deliver(message)
+
+
+class _SharedCoordinationRedis:
+ """The coordination Redis two workers share: whatever one worker publishes reaches the other's subscriber"""
+
+ namespace = None
+
+ def __init__(self) -> None:
+ self.published: tuple[str, ...] = ()
+ self._subscriptions: tuple[_ChannelSubscription, ...] = ()
+
+ def init_pubsub_client(self) -> "_SharedCoordinationRedis":
+ return self
+
+ def pubsub(self) -> _ChannelSubscription:
+ subscription: Final = _ChannelSubscription()
+ self._subscriptions = (*self._subscriptions, subscription)
+ return subscription
+
+ def pipeline(self, transaction: bool = True) -> _ChannelPipeline:
+ return _ChannelPipeline(self)
+
+ async def publish(self, channel: str, message: str) -> int:
+ self.deliver(message)
+ return 1
+
+ def deliver(self, message: str) -> None:
+ self.published = (*self.published, message)
+ for subscription in self._subscriptions:
+ subscription.messages.put_nowait({"type": "message", "data": message.encode()})
+
+ async def until_every_subscriber_applied_its_messages(self) -> None:
+ _ = await asyncio.gather(*(subscription.messages.join() for subscription in self._subscriptions))
+
+
+def _owned_key_row(hashed_token: str) -> LiteLLM_VerificationToken:
+ return LiteLLM_VerificationToken(
+ token=hashed_token,
+ user_id="user-123",
+ team_id=None,
+ key_alias="doomed-key",
+ spend=0.0,
+ max_budget=None,
+ models=[],
+ aliases={},
+ config={},
+ permissions={},
+ metadata={},
+ model_max_budget={},
+ model_spend={},
+ soft_budget_cooldown=False,
+ allowed_routes=[],
+ )
+
+
+def _prisma_with_key_row(key_row: LiteLLM_VerificationToken) -> AsyncMock:
+ prisma_client: Final = AsyncMock()
+ prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(return_value=[key_row])
+ prisma_client.db.litellm_jwtkeymapping = _CascadingJWTMappingTable([])
+ prisma_client.db.litellm_deletedverificationtoken.create_many = AsyncMock()
+ prisma_client.delete_data = AsyncMock(return_value=[key_row.token])
+ prisma_client.get_data = AsyncMock(return_value=None)
+ return prisma_client
+
+
+@pytest.mark.asyncio
+async def test_delete_verification_tokens_stops_the_key_on_a_peer_worker(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Auth serves key objects cache-first, so a peer worker that is not told about the delete keeps
+ authenticating the deleted key from its own copy until user_api_key_cache_ttl expires"""
+ raw_key: Final = "sk-" + uuid4().hex
+ hashed_token: Final = hash_token(raw_key)
+ deleting_worker_cache: Final = UserApiKeyCache()
+ peer_worker_cache: Final = UserApiKeyCache()
+ for cache in (deleting_worker_cache, peer_worker_cache):
+ cache.set_cache(key=hashed_token, value=UserAPIKeyAuth(token=hashed_token, user_id="user-123"))
+ coordination_redis: Final = _SharedCoordinationRedis()
+ peer_subscriber: Final = AuthCacheInvalidationSubscriber(
+ redis_cache=coordination_redis, user_api_key_cache=peer_worker_cache
+ )
+ peer_subscriber.start()
+ prisma_client: Final = _prisma_with_key_row(_owned_key_row(hashed_token))
+ monkeypatch.setattr("litellm.proxy.proxy_server.redis_usage_cache", coordination_redis)
+ monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", prisma_client)
+
+ try:
+ await delete_verification_tokens(
+ tokens=[raw_key],
+ user_api_key_cache=deleting_worker_cache,
+ user_api_key_dict=UserAPIKeyAuth(
+ user_id="admin-user", api_key="sk-admin", user_role=LitellmUserRoles.PROXY_ADMIN.value
+ ),
+ )
+ await coordination_redis.until_every_subscriber_applied_its_messages()
+ finally:
+ await peer_subscriber.stop()
+
+ assert deleting_worker_cache.get_cache(key=hashed_token) is None
+ with pytest.raises(ProxyException) as denied:
+ await get_key_object(
+ hashed_token=hashed_token, prisma_client=prisma_client, user_api_key_cache=peer_worker_cache
+ )
+ assert denied.value.code == "401"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "deleter",
+ [
+ UserAPIKeyAuth(user_id="admin-user", api_key="sk-admin", user_role=LitellmUserRoles.PROXY_ADMIN.value),
+ UserAPIKeyAuth(user_id="user-123", api_key="sk-owner", user_role=LitellmUserRoles.INTERNAL_USER.value),
+ ],
+ ids=["proxy_admin", "key_owner"],
+)
+async def test_delete_verification_tokens_evicts_a_requested_key_whose_row_is_already_gone(
+ monkeypatch: pytest.MonkeyPatch, deleter: UserAPIKeyAuth
+) -> None:
+ """A key whose row was removed outside the API stays cached on every worker, so deleting it in a
+ batch beside a live key must still stop it everywhere instead of only evicting the rows found"""
+ live_key: Final = "sk-" + uuid4().hex
+ gone_key: Final = "sk-" + uuid4().hex
+ live_token: Final = hash_token(live_key)
+ gone_token: Final = hash_token(gone_key)
+ deleting_worker_cache: Final = UserApiKeyCache()
+ peer_worker_cache: Final = UserApiKeyCache()
+ for cache in (deleting_worker_cache, peer_worker_cache):
+ for token in (live_token, gone_token):
+ cache.set_cache(key=token, value=UserAPIKeyAuth(token=token, user_id="user-123"))
+ coordination_redis: Final = _SharedCoordinationRedis()
+ peer_subscriber: Final = AuthCacheInvalidationSubscriber(
+ redis_cache=coordination_redis, user_api_key_cache=peer_worker_cache
+ )
+ peer_subscriber.start()
+ monkeypatch.setattr("litellm.proxy.proxy_server.redis_usage_cache", coordination_redis)
+ monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", _prisma_with_key_row(_owned_key_row(live_token)))
+
+ try:
+ await delete_verification_tokens(
+ tokens=[live_key, gone_key], user_api_key_cache=deleting_worker_cache, user_api_key_dict=deleter
+ )
+ await coordination_redis.until_every_subscriber_applied_its_messages()
+ finally:
+ await peer_subscriber.stop()
+
+ assert [deleting_worker_cache.get_cache(key=token) for token in (live_token, gone_token)] == [None, None]
+ assert [peer_worker_cache.get_cache(key=token) for token in (live_token, gone_token)] == [None, None]
+
+
+@pytest.mark.asyncio
+async def test_delete_verification_tokens_leaves_unowned_cache_entries_alone_on_every_worker(monkeypatch: pytest.MonkeyPatch) -> None:
+ """/key/delete accepts any string, and a name that is not a key row passes through unhashed. A key
+ owner padding the request with another team's spend counter name must not wipe that counter from
+ every worker's memory: only the rows the caller was allowed to delete are evicted and broadcast"""
+ raw_key: Final = "sk-" + uuid4().hex
+ hashed_token: Final = hash_token(raw_key)
+ other_team_counter: Final = "spend:team:other-team"
+ deleting_worker_cache: Final = UserApiKeyCache()
+ peer_worker_cache: Final = UserApiKeyCache()
+ peer_spend_counters: Final = InMemoryCache()
+ peer_spend_counters.set_cache(other_team_counter, 42.0)
+ peer_worker_cache.set_cache(key=hashed_token, value=UserAPIKeyAuth(token=hashed_token, user_id="user-123"))
+ coordination_redis: Final = _SharedCoordinationRedis()
+ peer_subscriber: Final = AuthCacheInvalidationSubscriber(
+ redis_cache=coordination_redis,
+ user_api_key_cache=peer_worker_cache,
+ additional_in_memory_caches=(peer_spend_counters,),
+ )
+ peer_subscriber.start()
+ monkeypatch.setattr("litellm.proxy.proxy_server.redis_usage_cache", coordination_redis)
+ monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", _prisma_with_key_row(_owned_key_row(hashed_token)))
+
+ try:
+ await delete_verification_tokens(
+ tokens=[other_team_counter, raw_key],
+ user_api_key_cache=deleting_worker_cache,
+ user_api_key_dict=UserAPIKeyAuth(
+ user_id="user-123", api_key="sk-owner", user_role=LitellmUserRoles.INTERNAL_USER.value
+ ),
+ )
+ await coordination_redis.until_every_subscriber_applied_its_messages()
+ finally:
+ await peer_subscriber.stop()
+
+ assert peer_worker_cache.get_cache(key=hashed_token) is None
+ assert peer_spend_counters.get_cache(other_team_counter) == 42.0
+ assert coordination_redis.published == (json.dumps({"cache_key": hashed_token}),)
+
+
@pytest.mark.asyncio
async def test_delete_key_fn_persists_deleted_keys(monkeypatch):
from litellm.proxy._types import KeyRequest
From 17d1f5833689739003abf379950137d7e8eeb2f5 Mon Sep 17 00:00:00 2001
From: tin-berri
Date: Fri, 30 Oct 2026 10:31:57 -0700
Subject: [PATCH 17/87] fix(ui): preserve team model grants on unrelated
settings saves (#49258)
---
.../ModelSelect/ModelSelect.test.tsx | 33 +++++++++++
.../components/ModelSelect/ModelSelect.tsx | 5 +-
.../team/TeamInfo.integration.test.tsx | 59 ++++++++++++++++++-
.../src/components/team/TeamInfo.test.tsx | 1 -
.../src/components/team/TeamInfo.tsx | 5 +-
5 files changed, 97 insertions(+), 6 deletions(-)
diff --git a/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.test.tsx b/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.test.tsx
index 7ab757cfed9..d7394b1f949 100644
--- a/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.test.tsx
+++ b/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.test.tsx
@@ -775,6 +775,39 @@ describe("ModelSelect", () => {
expect(mockOnChange).toHaveBeenCalledWith(["gpt-4"]);
});
+ it.each([
+ { value: "all-proxy-models", label: "All Proxy Models", includeSpecialOptions: true },
+ { value: "no-default-models", label: "No Default Models", includeSpecialOptions: false },
+ ])(
+ "should label retained $value without offering it again after removal",
+ async ({ value, label, includeSpecialOptions }) => {
+ const user = userEvent.setup();
+ mockUseOrganization.mockReturnValue({
+ data: createMockOrganization(["gpt-4"]),
+ isLoading: false,
+ } as unknown as ReturnType);
+ const props = {
+ onChange: mockOnChange,
+ context: "team" as const,
+ organizationID: "org-1",
+ options: { includeSpecialOptions },
+ };
+ const { rerender } = renderWithProviders();
+
+ expect(screen.getByLabelText(label)).toBeInTheDocument();
+ await openModelList(user);
+ const retainedGroup = within(screen.getByRole("listbox")).getAllByRole("group")[0];
+ expect(within(retainedGroup).getByText("Unavailable")).toBeInTheDocument();
+ await user.click(within(retainedGroup).getByRole("option", { name: label }));
+ expect(mockOnChange).toHaveBeenCalledWith([]);
+
+ rerender();
+ expectNotOffered(label);
+ expectNotOffered("claude-3");
+ expectOffered("gpt-4");
+ },
+ );
+
it("should keep the other selections when removing one unavailable model alongside a special option", async () => {
const user = userEvent.setup();
renderWithProviders(
diff --git a/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.tsx b/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.tsx
index f169382b4ed..c7c428f2ccf 100644
--- a/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.tsx
+++ b/ui/litellm-dashboard/src/components/ModelSelect/ModelSelect.tsx
@@ -240,7 +240,10 @@ export const ModelSelect = (props: ModelSelectProps) => {
];
const offeredByValue = new Map(offeredGroups.flatMap((group) => group.items).map((option) => [option.value, option]));
- const selectedOptions = value.map((v) => offeredByValue.get(v) ?? { label: v, value: v });
+ const selectedOptions = value.map(
+ (v) =>
+ offeredByValue.get(v) ?? MODEL_SENTINEL_OPTIONS.find((option) => option.value === v) ?? { label: v, value: v },
+ );
const groups: ModelOptionGroup[] = [
...unavailableGroups(
selectedOptions,
diff --git a/ui/litellm-dashboard/src/components/team/TeamInfo.integration.test.tsx b/ui/litellm-dashboard/src/components/team/TeamInfo.integration.test.tsx
index fc054e51e5b..545c504281b 100644
--- a/ui/litellm-dashboard/src/components/team/TeamInfo.integration.test.tsx
+++ b/ui/litellm-dashboard/src/components/team/TeamInfo.integration.test.tsx
@@ -1,6 +1,6 @@
import { useTeamMetadataSchema } from "@/app/(dashboard)/hooks/teams/useTeamMetadataSchema";
import * as networking from "@/components/networking";
-import { fireEvent, screen, waitFor } from "@testing-library/react";
+import { fireEvent, screen, waitFor, within } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { renderWithProviders } from "../../../tests/test-utils";
@@ -817,7 +817,7 @@ describe("TeamInfoView - member budget reset prompt", () => {
});
});
-describe("TeamInfoView - model rate limit picker", () => {
+describe("TeamInfoView - model settings", () => {
const props = {
teamId: "123",
onUpdate: vi.fn(),
@@ -836,6 +836,61 @@ describe("TeamInfoView - model rate limit picker", () => {
vi.clearAllMocks();
});
+ it.each([
+ { name: "unrestricted", models: [], organizationId: null },
+ { name: "explicit all models", models: ["all-proxy-models"], organizationId: null },
+ { name: "access groups only", models: ["no-default-models"], organizationId: null },
+ { name: "retained explicit models", models: ["retired-model", "openai/*"], organizationId: null },
+ { name: "unrestricted within an organization", models: [], organizationId: "org-1" },
+ ])("preserves $name model grants when saving only a budget", async ({ models, organizationId }) => {
+ const user = userEvent.setup({ delay: null });
+ vi.mocked(networking.teamInfoCall).mockResolvedValue(
+ createMockTeamData({ models, organization_id: organizationId }),
+ );
+ mockUseTeam.mockReturnValue({
+ data: { team_id: "123", organization_models: organizationId ? ["gpt-4"] : undefined },
+ isLoading: false,
+ } as unknown as ReturnType);
+
+ renderWithProviders();
+ await user.click(await screen.findByRole("tab", { name: "Settings" }));
+ await user.click(await screen.findByRole("button", { name: /edit settings/i }));
+ if (models.length === 0) {
+ expect(await screen.findByLabelText("All Proxy Models")).toBeInTheDocument();
+ }
+ fireEvent.change(await screen.findByLabelText("Max Budget (USD)"), { target: { value: "100" } });
+ await user.click(screen.getByRole("button", { name: /save changes/i }));
+
+ await waitFor(() => expect(networking.teamUpdateCall).toHaveBeenCalledTimes(1));
+ const payload = vi.mocked(networking.teamUpdateCall).mock.calls[0][1];
+ expect(payload).toMatchObject({ team_id: "123", max_budget: 100 });
+ expect(payload).not.toHaveProperty("models");
+ });
+
+ it("discards a cancelled model clear but saves an intentional clear of unrestricted access", async () => {
+ const user = userEvent.setup({ delay: null });
+ vi.mocked(networking.teamInfoCall).mockResolvedValue(createMockTeamData({ models: [] }));
+
+ renderWithProviders();
+ await user.click(await screen.findByRole("tab", { name: "Settings" }));
+ await user.click(await screen.findByRole("button", { name: /edit settings/i }));
+ await user.click(within(await screen.findByLabelText("All Proxy Models")).getByRole("button"));
+ await user.click(screen.getByRole("button", { name: "Cancel" }));
+ await user.click(await screen.findByRole("button", { name: /edit settings/i }));
+ expect(await screen.findByLabelText("All Proxy Models")).toBeInTheDocument();
+ fireEvent.change(screen.getByLabelText("Max Budget (USD)"), { target: { value: "100" } });
+ await user.click(screen.getByRole("button", { name: /save changes/i }));
+
+ await waitFor(() => expect(networking.teamUpdateCall).toHaveBeenCalledTimes(1));
+ expect(vi.mocked(networking.teamUpdateCall).mock.calls[0][1]).not.toHaveProperty("models");
+ await user.click(await screen.findByRole("button", { name: /edit settings/i }));
+ await user.click(within(await screen.findByLabelText("All Proxy Models")).getByRole("button"));
+ await user.click(screen.getByRole("button", { name: /save changes/i }));
+
+ await waitFor(() => expect(networking.teamUpdateCall).toHaveBeenCalledTimes(2));
+ expect(vi.mocked(networking.teamUpdateCall).mock.calls[1][1].models).toEqual(["no-default-models"]);
+ });
+
it("offers access-group models, not no-default-models, in the model rate limit picker", async () => {
const user = userEvent.setup({ delay: null });
vi.mocked(networking.teamInfoCall).mockResolvedValue(
diff --git a/ui/litellm-dashboard/src/components/team/TeamInfo.test.tsx b/ui/litellm-dashboard/src/components/team/TeamInfo.test.tsx
index e4bf67ab72f..46081ae96f5 100644
--- a/ui/litellm-dashboard/src/components/team/TeamInfo.test.tsx
+++ b/ui/litellm-dashboard/src/components/team/TeamInfo.test.tsx
@@ -2421,7 +2421,6 @@ describe("TeamInfoView - the exact bytes the update call sends", () => {
const alwaysSent = {
team_id: "123",
team_alias: "Test Team",
- models: ["gpt-4"],
tpm_limit: 1000,
rpm_limit: 1000,
tpd_limit: null,
diff --git a/ui/litellm-dashboard/src/components/team/TeamInfo.tsx b/ui/litellm-dashboard/src/components/team/TeamInfo.tsx
index 0d20d160283..329552d735a 100644
--- a/ui/litellm-dashboard/src/components/team/TeamInfo.tsx
+++ b/ui/litellm-dashboard/src/components/team/TeamInfo.tsx
@@ -72,6 +72,7 @@ import {
} from "../key_team_helpers/ModelMaxBudgetEditor";
import { modelMaxBudgetUpdate, StoredModelMaxBudget } from "../key_team_helpers/modelMaxBudgetPayload";
import {
+ ALL_PROXY_MODELS,
computeTeamModelBadges,
normalizeTeamModelSelection,
TeamAccessGroupModelGrant,
@@ -515,7 +516,7 @@ const computeEffectiveGuardrails = (info: TeamInfoRecord, globalGuardrailNames:
const toTeamFormValues = (info: TeamInfoRecord, effectiveGuardrails: string[]): TeamUpdateFormValues => ({
team_alias: info.team_alias,
- models: info.models,
+ models: info.models.length === 0 ? [ALL_PROXY_MODELS] : info.models,
max_budget: info.max_budget,
soft_budget: info.soft_budget,
soft_budget_alerting_emails: Array.isArray(info.metadata?.soft_budget_alerting_emails)
@@ -1064,7 +1065,7 @@ const TeamInfoView: React.FC = ({
const updateData: any = {
team_id: teamId,
team_alias: values.team_alias,
- models: normalizeTeamModelSelection(values.models),
+ ...(form.getFieldState("models").isDirty ? { models: normalizeTeamModelSelection(values.models) } : {}),
tpm_limit: numberOrNull(values.tpm_limit),
rpm_limit: numberOrNull(values.rpm_limit),
tpd_limit: numberOrNull(values.tpd_limit),
From 707ac1ffee620e2906ef0fb5f9ad699cfd65a16d Mon Sep 17 00:00:00 2001
From: islomjon-aiand
Date: Sat, 31 Oct 2026 02:55:26 +0900
Subject: [PATCH 18/87] fix(aiand)!: show ai&'s name and logo, and sync ai&'s
model list (#49158)
* feat(ui): show ai&'s name and logo in the dashboard
The Add Model form already lists ai& from its provider metadata, but the dashboard had no entry for it, so the provider showed no logo and model tables printed the raw aiand slug. Adds ai& to the dashboard provider list with its official mark, plated in dark mode so the red and black mark keep their colours, and the aiand/openai/gpt-oss-120b model placeholder
* fix(aiand): retire deepseek-v4-flash, glm-5.2 and kimi-k2.7-code
ai& retired these three models on 2026-10-27 and https://api.aiand.com/v1/models now lists ten. The rows keep their prices with a deprecation_date, so earlier spend still prices, and aiand/* stops listing them, the same way baseten handles retired rows. The other ten rows already match the live prices, context windows and output limits
* feat(aiand): add deepseek-v4.1-flash-1017
ai& launches deepseek-ai/deepseek-v4.1-flash-1017 on 2026-11-02. It is served on https://staging-api.aiand.com/v1/models with the same price, context window, output limit and capabilities as deepseek-v4.1-flash, so its row copies that one
* fix(aiand): retire deepseek-v4.1-flash
ai& retires deepseek-ai/deepseek-v4.1-flash on 2026-10-30, now that deepseek-v4.1-flash-1017 is live in production on https://api.aiand.com/v1/models with the same price, limits and capabilities. Its row keeps its prices with a deprecation_date, like the other three retired ai& rows
* fix(aiand): retire deepseek-v4.1-flash on 2026-11-06
https://api.aiand.com/v1/models now gives deepseek-v4.1-flash a deprecation_date of 2026-11-05T15:00Z, which is 2026-11-06 00:00 in Japan, so its row uses 2026-11-06 instead of 2026-10-30 and aiand/* keeps listing it until then. The wildcard test now leaves a model out only once its retirement date has passed
* test(aiand): derive retired models from the rows instead of pinning dates
The wildcard test now reads each row's deprecation_date, so it holds whichever dates ai& sets, and the test that only restated the JSON dates is gone
* fix(aiand): retire models on their UTC date while the proxy runs, and add muse-spark-open
The wildcard listing now checks each row's deprecation_date when it expands, against the UTC date the field
documents, so a model drops on its day without a restart. deepseek-v4.1-flash now carries 2026-11-05, the
UTC day ai& retires it, and meta/muse-spark-open gets its price row from ai&'s live /v1/models
* fix(aiand): hide retired models only from bundled listings, including the wildcard health check
* fix(aiand): share the UTC retirement check with Together AI and the health check sort
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
---
litellm/__init__.py | 21 ++--
.../litellm_core_utils/llm_request_utils.py | 21 ++--
.../litellm_core_utils/model_retirement.py | 19 ++++
...odel_prices_and_context_window_backup.json | 34 ++++++
litellm/utils.py | 10 +-
model_prices_and_context_window.json | 34 ++++++
.../test_llm_request_utils.py | 38 +++++++
.../baseten/chat/test_baseten_completions.py | 33 +++++-
.../llms/openai_like/test_aiand_provider.py | 100 +++++++++++++++++-
.../public/assets/logos/aiand.svg | 4 +
.../components/provider_info_helpers.test.tsx | 12 +++
.../src/components/provider_info_helpers.tsx | 5 +
.../src/lib/logoTreatments.ts | 1 +
13 files changed, 296 insertions(+), 36 deletions(-)
create mode 100644 litellm/litellm_core_utils/model_retirement.py
create mode 100644 ui/litellm-dashboard/public/assets/logos/aiand.svg
diff --git a/litellm/__init__.py b/litellm/__init__.py
index 5f14c605cd0..f64531f86fb 100644
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -7,7 +7,6 @@ _apply_process_defaults()
### INIT VARIABLES #########################
import threading
import os
-import datetime
from collections.abc import Mapping, Sequence
from typing import (
@@ -550,6 +549,7 @@ _key_management_system: Optional["KeyManagementSystem"] = None
output_parse_pii: bool = False
#############################################
from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map, mark_litellm_import_complete
+from litellm.litellm_core_utils.model_retirement import is_hidden_as_retired
model_cost = get_model_cost_map(url=model_cost_map_url)
cost_discount_config: Dict[str, float] = {} # Provider-specific cost discounts {"vertex_ai": 0.05} = 5% discount
@@ -756,16 +756,6 @@ def is_openai_finetune_model(key: str) -> bool:
return key.startswith("ft:") and not key.count(":") > 1
-def _is_past_deprecation_date(value: Mapping[str, object]) -> bool:
- deprecation_date: Final = value.get("deprecation_date")
- if not isinstance(deprecation_date, str):
- return False
- try:
- return datetime.date.fromisoformat(deprecation_date) <= datetime.date.today()
- except ValueError:
- return False
-
-
def _populate_provider_model_sets(model_cost_map: Mapping[str, Mapping[str, object]]) -> None:
for key, value in tuple(model_cost_map.items()):
if value.get("litellm_provider") == "openai" and not is_openai_finetune_model(key):
@@ -873,12 +863,12 @@ def _populate_provider_model_sets(model_cost_map: Mapping[str, Mapping[str, obje
if "-to-" not in key and "fireworks-ai-default" not in key:
fireworks_ai_models.add(key)
elif value.get("litellm_provider") == "together_ai":
- if _is_past_deprecation_date(value):
+ if is_hidden_as_retired(value):
together_ai_models.discard(key)
elif not key.startswith("together-ai-"):
together_ai_models.add(key)
elif value.get("litellm_provider") == "baseten":
- if _is_past_deprecation_date(value):
+ if is_hidden_as_retired(value):
baseten_models.discard(key)
else:
baseten_models.add(key)
@@ -1051,7 +1041,10 @@ def _populate_provider_model_sets(model_cost_map: Mapping[str, Mapping[str, obje
elif value.get("litellm_provider") == "coralbricks":
coralbricks_models.add(key)
elif value.get("litellm_provider") == "aiand":
- aiand_models.add(key)
+ if is_hidden_as_retired(value):
+ aiand_models.discard(key)
+ else:
+ aiand_models.add(key)
elif value.get("litellm_provider") == "reducto":
reducto_models.add(key)
elif value.get("litellm_provider") == "bedrock_mantle":
diff --git a/litellm/litellm_core_utils/llm_request_utils.py b/litellm/litellm_core_utils/llm_request_utils.py
index 9c2ec0b8984..84f161e0c62 100644
--- a/litellm/litellm_core_utils/llm_request_utils.py
+++ b/litellm/litellm_core_utils/llm_request_utils.py
@@ -1,4 +1,3 @@
-import datetime
from collections.abc import Iterable, Iterator, Mapping
from dataclasses import dataclass
from typing import Final, cast
@@ -9,6 +8,7 @@ from typing_extensions import NotRequired, ReadOnly, TypedDict
import litellm
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
from litellm.litellm_core_utils.get_llm_provider_logic import inferred_provider
+from litellm.litellm_core_utils.model_retirement import is_past_deprecation_date
def _form_field_value(value: object) -> str:
@@ -168,7 +168,9 @@ class RankedChatModel:
def rank_chat_models_from_llm_provider(custom_llm_provider: str) -> tuple[RankedChatModel, ...]:
- known_models: Final[Iterable[str]] = litellm.models_by_provider.get(custom_llm_provider, ())
+ from litellm.utils import bundled_provider_models
+
+ known_models: Final = bundled_provider_models(custom_llm_provider)
ranked_models: Final = (
RankedChatModel(
model=model, route_rank=route_rank, cost=cost, is_past_deprecation_date=is_past_deprecation_date
@@ -187,18 +189,11 @@ _COST_MAP_DEPRECATION: Final = TypeAdapter(_CostMapDeprecation)
def _is_past_deprecation_date(cost_map_key: str) -> bool:
- deprecation_date: Final = _COST_MAP_DEPRECATION.validate_python(
- litellm.model_cost.get(cost_map_key) or {} # pyright: ignore[reportUnknownMemberType] # untyped cost map JSON
- ).get("deprecation_date")
- if not isinstance(deprecation_date, str):
- return False
- try:
- return (
- datetime.date.fromisoformat(deprecation_date)
- <= datetime.datetime.now(datetime.timezone.utc).astimezone().date()
+ return is_past_deprecation_date(
+ _COST_MAP_DEPRECATION.validate_python(
+ litellm.model_cost.get(cost_map_key) or {} # pyright: ignore[reportUnknownMemberType] # untyped cost map JSON
)
- except ValueError:
- return False
+ )
def _chat_row_costs(models: Iterable[str], custom_llm_provider: str) -> Iterator[tuple[str, float, bool]]:
diff --git a/litellm/litellm_core_utils/model_retirement.py b/litellm/litellm_core_utils/model_retirement.py
new file mode 100644
index 00000000000..3bfb834733d
--- /dev/null
+++ b/litellm/litellm_core_utils/model_retirement.py
@@ -0,0 +1,19 @@
+import datetime
+from collections.abc import Mapping
+from typing import Final
+
+PROVIDERS_HIDING_RETIRED_MODELS: Final = frozenset({"aiand", "baseten", "together_ai"})
+
+
+def is_past_deprecation_date(value: Mapping[str, object]) -> bool:
+ deprecation_date: Final = value.get("deprecation_date")
+ if not isinstance(deprecation_date, str):
+ return False
+ try:
+ return datetime.date.fromisoformat(deprecation_date) <= datetime.datetime.now(datetime.timezone.utc).date()
+ except ValueError:
+ return False
+
+
+def is_hidden_as_retired(value: Mapping[str, object]) -> bool:
+ return value.get("litellm_provider") in PROVIDERS_HIDING_RETIRED_MODELS and is_past_deprecation_date(value)
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index e1131117fcf..ea9cacbdabe 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -72249,6 +72249,7 @@
},
"aiand/deepseek-ai/deepseek-v4-flash": {
"cache_read_input_token_cost": 8e-08,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 1.5e-07,
"litellm_provider": "aiand",
"max_input_tokens": 1048576,
@@ -72278,6 +72279,22 @@
"supports_vision": false
},
"aiand/deepseek-ai/deepseek-v4.1-flash": {
+ "cache_read_input_token_cost": 2e-08,
+ "deprecation_date": "2026-11-05",
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "aiand",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 384000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://api.aiand.com/v1/models",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "aiand/deepseek-ai/deepseek-v4.1-flash-1017": {
"cache_read_input_token_cost": 2e-08,
"input_cost_per_token": 3e-07,
"litellm_provider": "aiand",
@@ -72307,8 +72324,24 @@
"supports_tool_choice": true,
"supports_vision": true
},
+ "aiand/meta/muse-spark-open": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "aiand",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 943718,
+ "mode": "chat",
+ "output_cost_per_token": 4.25e-06,
+ "source": "https://api.aiand.com/v1/models",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"aiand/moonshotai/kimi-k2.7-code": {
"cache_read_input_token_cost": 2e-07,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 7.5e-07,
"litellm_provider": "aiand",
"max_input_tokens": 262144,
@@ -72399,6 +72432,7 @@
},
"aiand/zai-org/glm-5.2": {
"cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 1e-06,
"litellm_provider": "aiand",
"max_input_tokens": 1048576,
diff --git a/litellm/utils.py b/litellm/utils.py
index 6f56cc65f45..2ca8a49f968 100644
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -94,6 +94,7 @@ from litellm.litellm_core_utils.fallback_generalizations import (
)
from litellm.litellm_core_utils.get_model_cost_map import AliasedCostEntry, GetModelCostMap
from litellm.litellm_core_utils.model_param_helper import peek_reasoning_summary_aliases
+from litellm.litellm_core_utils.model_retirement import is_hidden_as_retired
from litellm.litellm_core_utils.offline_mode import is_offline_mode
from litellm.litellm_core_utils.sensitive_data_masker import redact_credentials_in_payload
from litellm.litellm_core_utils.tokenizer import strip_special_tokens
@@ -8806,6 +8807,12 @@ def _get_valid_models_from_provider_api(
return []
+def bundled_provider_models(provider: str) -> tuple[str, ...]:
+ bundled: Final[tuple[str, ...]] = tuple(litellm.models_by_provider.get(provider, ()))
+ cost_map: Final = _builtin_cost_map()
+ return tuple(model for model in bundled if not is_hidden_as_retired(cost_map.get(model, {})))
+
+
def get_valid_models(
check_provider_endpoint: bool | None = None,
custom_llm_provider: str | None = None,
@@ -8871,8 +8878,7 @@ def get_valid_models(
)
)
else:
- models_for_provider = copy.deepcopy(litellm.models_by_provider.get(provider, []))
- valid_models.extend(models_for_provider)
+ valid_models.extend(bundled_provider_models(provider))
return valid_models
except Exception as e:
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
index e1131117fcf..ea9cacbdabe 100644
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -72249,6 +72249,7 @@
},
"aiand/deepseek-ai/deepseek-v4-flash": {
"cache_read_input_token_cost": 8e-08,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 1.5e-07,
"litellm_provider": "aiand",
"max_input_tokens": 1048576,
@@ -72278,6 +72279,22 @@
"supports_vision": false
},
"aiand/deepseek-ai/deepseek-v4.1-flash": {
+ "cache_read_input_token_cost": 2e-08,
+ "deprecation_date": "2026-11-05",
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "aiand",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 384000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://api.aiand.com/v1/models",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "aiand/deepseek-ai/deepseek-v4.1-flash-1017": {
"cache_read_input_token_cost": 2e-08,
"input_cost_per_token": 3e-07,
"litellm_provider": "aiand",
@@ -72307,8 +72324,24 @@
"supports_tool_choice": true,
"supports_vision": true
},
+ "aiand/meta/muse-spark-open": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "aiand",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 943718,
+ "mode": "chat",
+ "output_cost_per_token": 4.25e-06,
+ "source": "https://api.aiand.com/v1/models",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"aiand/moonshotai/kimi-k2.7-code": {
"cache_read_input_token_cost": 2e-07,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 7.5e-07,
"litellm_provider": "aiand",
"max_input_tokens": 262144,
@@ -72399,6 +72432,7 @@
},
"aiand/zai-org/glm-5.2": {
"cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2026-10-27",
"input_cost_per_token": 1e-06,
"litellm_provider": "aiand",
"max_input_tokens": 1048576,
diff --git a/tests/unit/litellm_core_utils/test_llm_request_utils.py b/tests/unit/litellm_core_utils/test_llm_request_utils.py
index e7350fae395..5243d308543 100644
--- a/tests/unit/litellm_core_utils/test_llm_request_utils.py
+++ b/tests/unit/litellm_core_utils/test_llm_request_utils.py
@@ -1,3 +1,5 @@
+import datetime
+import time
from typing import Final
import httpx
@@ -245,3 +247,39 @@ def test_rank_chat_models_flags_a_retired_model_the_provider_lists_without_its_c
(ranked_model.model, ranked_model.is_past_deprecation_date)
for ranked_model in rank_chat_models_from_llm_provider("vertex_ai")
] == [("fake-retired-vertex-chat", True), ("fake-live-vertex-chat", False)]
+
+
+@pytest.mark.parametrize("host_zone", ["Etc/GMT+12", "Etc/GMT-14"])
+def test_rank_chat_models_flags_retirement_by_the_utc_date_whatever_the_host_zone(
+ monkeypatch: pytest.MonkeyPatch, host_zone: str
+) -> None:
+ utc_today: Final = datetime.datetime.now(datetime.timezone.utc).date()
+ retirement_dates: Final = {
+ "deepseek/fake-retires-today-chat": utc_today,
+ "deepseek/fake-retires-tomorrow-chat": utc_today + datetime.timedelta(days=1),
+ }
+ for name, retirement_date in retirement_dates.items():
+ monkeypatch.setitem(
+ litellm.model_cost,
+ name,
+ {
+ "input_cost_per_token": 1e-12,
+ "output_cost_per_token": 1e-12,
+ "mode": "chat",
+ "litellm_provider": PICKER_PROVIDER,
+ "deprecation_date": retirement_date.isoformat(),
+ },
+ )
+ monkeypatch.setitem(litellm.models_by_provider, PICKER_PROVIDER, tuple(retirement_dates))
+ monkeypatch.setenv("TZ", host_zone)
+ time.tzset()
+ try:
+ flags: Final = {
+ ranked_model.model: ranked_model.is_past_deprecation_date
+ for ranked_model in rank_chat_models_from_llm_provider(PICKER_PROVIDER)
+ }
+ finally:
+ monkeypatch.undo()
+ time.tzset()
+
+ assert flags == {"deepseek/fake-retires-today-chat": True, "deepseek/fake-retires-tomorrow-chat": False}
diff --git a/tests/unit/llms/baseten/chat/test_baseten_completions.py b/tests/unit/llms/baseten/chat/test_baseten_completions.py
index 4f135a1a106..407b39319e9 100644
--- a/tests/unit/llms/baseten/chat/test_baseten_completions.py
+++ b/tests/unit/llms/baseten/chat/test_baseten_completions.py
@@ -7,8 +7,9 @@ import pytest
import respx
import litellm
+from litellm.litellm_core_utils.model_retirement import is_past_deprecation_date
from litellm.llms.baseten.chat import BasetenConfig
-from litellm.utils import get_valid_models
+from litellm.utils import _model_cache, get_valid_models
class TestBasetenRouting:
@@ -65,7 +66,7 @@ class TestBasetenModelDiscovery:
servable_registry_models: Final = {
key
for key, value in litellm.model_cost.items()
- if value.get("litellm_provider") == "baseten" and not litellm._is_past_deprecation_date(value)
+ if value.get("litellm_provider") == "baseten" and not is_past_deprecation_date(value)
}
assert servable_registry_models
@@ -92,6 +93,20 @@ class TestBasetenModelDiscovery:
litellm.baseten_models.discard(model)
litellm.add_known_models({})
+ def test_wildcard_drops_a_model_whose_retirement_date_passes_while_running(self, monkeypatch: pytest.MonkeyPatch):
+ model: Final = "baseten/acme/retires-while-running"
+ monkeypatch.setitem(litellm.model_cost, model, {"litellm_provider": "baseten", "deprecation_date": "2999-01-01"})
+ try:
+ litellm.add_known_models({model: litellm.model_cost[model]})
+ assert model in get_valid_models(custom_llm_provider="baseten")
+
+ monkeypatch.setitem(litellm.model_cost, model, {"litellm_provider": "baseten", "deprecation_date": "2000-01-01"})
+
+ assert model not in get_valid_models(custom_llm_provider="baseten")
+ finally:
+ litellm.baseten_models.discard(model)
+ litellm.add_known_models({})
+
@pytest.mark.parametrize(
"value, expected",
[
@@ -102,7 +117,7 @@ class TestBasetenModelDiscovery:
],
)
def test_is_past_deprecation_date(self, value, expected):
- assert litellm._is_past_deprecation_date(value) is expected
+ assert is_past_deprecation_date(value) is expected
def test_get_models_calls_model_api(self, respx_mock: respx.MockRouter) -> None:
route: Final = respx_mock.get("https://inference.baseten.co/v1/models").mock(
@@ -156,6 +171,18 @@ class TestBasetenModelDiscovery:
assert models == ["baseten/brand-new/model"]
+ @patch.dict(os.environ, {"BASETEN_API_KEY": "test-key"})
+ def test_live_catalog_keeps_a_model_the_cost_map_dates_as_retired(self, respx_mock: respx.MockRouter) -> None:
+ assert is_past_deprecation_date(litellm.model_cost["baseten/zai-org/GLM-5"])
+ _model_cache.flush_cache()
+ respx_mock.get("https://inference.baseten.co/v1/models").mock(
+ return_value=httpx.Response(200, json={"data": [{"id": "zai-org/GLM-5"}]})
+ )
+
+ models: Final = get_valid_models(check_provider_endpoint=True, custom_llm_provider="baseten")
+
+ assert models == ["baseten/zai-org/GLM-5"]
+
class TestBasetenReasoningEffort:
def test_reasoning_effort_passthrough_with_drop_params(self):
diff --git a/tests/unit/llms/openai_like/test_aiand_provider.py b/tests/unit/llms/openai_like/test_aiand_provider.py
index 804f3a3b6eb..0274b58f19a 100644
--- a/tests/unit/llms/openai_like/test_aiand_provider.py
+++ b/tests/unit/llms/openai_like/test_aiand_provider.py
@@ -1,4 +1,6 @@
+import datetime
import json
+import time
from typing import Final
import httpx
@@ -8,6 +10,8 @@ import respx
import litellm
from litellm.caching.llm_caching_handler import LLMClientCache
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
+from litellm.litellm_core_utils.health_check_helpers import wildcard_health_check_models
+from litellm.litellm_core_utils.model_retirement import is_past_deprecation_date
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
from litellm.proxy.auth.model_checks import get_known_models_from_wildcard
from litellm.utils import ProviderConfigManager
@@ -89,11 +93,99 @@ def _aiand_catalog() -> frozenset[str]:
return frozenset(name for name, info in litellm.model_cost.items() if info.get("litellm_provider") == "aiand")
-def test_aiand_wildcard_lists_its_priced_models():
- catalog: Final = _aiand_catalog()
+def _utc_today() -> datetime.date:
+ return datetime.datetime.now(datetime.timezone.utc).date()
- assert catalog
- assert frozenset(get_known_models_from_wildcard("aiand/*")) == catalog
+
+def test_aiand_wildcard_lists_only_models_not_yet_retired():
+ catalog: Final = _aiand_catalog()
+ retired: Final = frozenset(
+ name
+ for name in catalog
+ if (retired_on := litellm.model_cost[name].get("deprecation_date")) is not None
+ and datetime.date.fromisoformat(retired_on) <= _utc_today()
+ )
+
+ assert catalog - retired
+ assert frozenset(get_known_models_from_wildcard("aiand/*")) == catalog - retired
+
+
+def test_aiand_wildcard_drops_a_model_whose_retirement_date_passes_while_running(monkeypatch: pytest.MonkeyPatch):
+ model: Final = "aiand/acme/retires-while-running"
+ monkeypatch.setitem(litellm.model_cost, model, {"litellm_provider": "aiand", "deprecation_date": "2999-01-01"})
+ try:
+ litellm.add_known_models({model: litellm.model_cost[model]})
+ assert model in get_known_models_from_wildcard("aiand/*")
+
+ monkeypatch.setitem(litellm.model_cost, model, {"litellm_provider": "aiand", "deprecation_date": "2000-01-01"})
+
+ assert model not in get_known_models_from_wildcard("aiand/*")
+ finally:
+ litellm.aiand_models.discard(model)
+ litellm.add_known_models({})
+
+
+def test_aiand_models_by_provider_leaves_out_models_retired_at_load():
+ retired: Final = frozenset(
+ name
+ for name, row in litellm.model_cost.items()
+ if row.get("litellm_provider") == "aiand" and is_past_deprecation_date(row)
+ )
+
+ assert retired
+ assert retired.isdisjoint(litellm.models_by_provider["aiand"])
+
+
+def test_aiand_wildcard_health_check_skips_a_model_whose_retirement_date_passes_while_running(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ model: Final = "aiand/acme/free-and-retiring"
+ live_row: Final = {
+ "litellm_provider": "aiand",
+ "mode": "chat",
+ "input_cost_per_token": 0.0,
+ "output_cost_per_token": 0.0,
+ "deprecation_date": "2999-01-01",
+ }
+ monkeypatch.setitem(litellm.model_cost, model, live_row)
+ try:
+ litellm.add_known_models({model: live_row})
+ assert model in wildcard_health_check_models(wildcard="aiand/*", custom_llm_provider="aiand")
+
+ monkeypatch.setitem(litellm.model_cost, model, {**live_row, "deprecation_date": "2000-01-01"})
+
+ assert model not in wildcard_health_check_models(wildcard="aiand/*", custom_llm_provider="aiand")
+ finally:
+ litellm.aiand_models.discard(model)
+ litellm.add_known_models({})
+
+
+@pytest.mark.parametrize("host_zone", ["Etc/GMT+12", "Etc/GMT-14"])
+def test_aiand_models_retire_on_the_utc_date_whatever_the_host_zone(monkeypatch: pytest.MonkeyPatch, host_zone: str):
+ today: Final = _utc_today()
+ rows: Final = {
+ "aiand/acme/retires-today": {"litellm_provider": "aiand", "deprecation_date": today.isoformat()},
+ "aiand/acme/retires-tomorrow": {
+ "litellm_provider": "aiand",
+ "deprecation_date": (today + datetime.timedelta(days=1)).isoformat(),
+ },
+ }
+ for name, row in rows.items():
+ monkeypatch.setitem(litellm.model_cost, name, row)
+ monkeypatch.setenv("TZ", host_zone)
+ time.tzset()
+ try:
+ litellm.add_known_models(rows)
+ listed: Final = frozenset(get_known_models_from_wildcard("aiand/*"))
+
+ assert "aiand/acme/retires-today" not in listed
+ assert "aiand/acme/retires-tomorrow" in listed
+ finally:
+ monkeypatch.undo()
+ time.tzset()
+ for name in rows:
+ litellm.aiand_models.discard(name)
+ litellm.add_known_models({})
def test_aiand_calls_are_billed_from_their_cost_rows():
diff --git a/ui/litellm-dashboard/public/assets/logos/aiand.svg b/ui/litellm-dashboard/public/assets/logos/aiand.svg
new file mode 100644
index 00000000000..486c73b8dbc
--- /dev/null
+++ b/ui/litellm-dashboard/public/assets/logos/aiand.svg
@@ -0,0 +1,4 @@
+
diff --git a/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx b/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
index 810b010f25e..02c31ecc082 100644
--- a/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
+++ b/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
@@ -32,6 +32,18 @@ describe("provider_info_helpers", () => {
expect(getPlaceholder(Providers.BIGMODEL)).toMatch(/^bigmodel\//);
});
+ it("resolves ai& metadata, name, logo and model placeholder", () => {
+ const metadata = PROVIDER_CREATE_FIELDS.find((provider) => provider.provider === "AIAND");
+ expect(provider_map.AIAND).toBe(metadata?.litellm_provider);
+ expect(getProviderLogoAndName("aiand")).toEqual({
+ displayName: "ai&",
+ logo: providerLogoMap[Providers.AIAND],
+ });
+ expect(getProviderLogoAndName("AIAND")).toEqual(getProviderLogoAndName("aiand"));
+ expect(providerLogoMap[Providers.AIAND]).toContain("aiand");
+ expect(getPlaceholder(Providers.AIAND)).toBe("aiand/openai/gpt-oss-120b");
+ });
+
it("maps Microsoft 365 Copilot to its chat model provider and placeholder", () => {
expect(provider_map.MICROSOFT_365_COPILOT).toBe("microsoft_365_copilot");
expect(getPlaceholder(Providers.MICROSOFT_365_COPILOT)).toBe("microsoft_365_copilot/chat");
diff --git a/ui/litellm-dashboard/src/components/provider_info_helpers.tsx b/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
index 1ed32c160e6..6448bd936cc 100644
--- a/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
+++ b/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
@@ -1,6 +1,7 @@
import { resolveLogoSrc } from "@/lib/assetPaths";
import a2aAgentLogo from "../../public/assets/logos/a2a_agent.png";
import ai21Logo from "../../public/assets/logos/ai21.svg";
+import aiandLogo from "../../public/assets/logos/aiand.svg";
import aimlApiLogo from "../../public/assets/logos/aiml_api.svg";
import anthropicLogo from "../../public/assets/logos/anthropic.svg";
import assemblyaiSmallLogo from "../../public/assets/logos/assemblyai_small.png";
@@ -76,6 +77,7 @@ export enum Providers {
A2A_Agent = "A2A Agent",
AI21 = "Ai21",
AI21_CHAT = "Ai21 Chat",
+ AIAND = "ai&",
AIML = "AI/ML API",
AIOHTTP_OPENAI = "Aiohttp Openai",
Anthropic = "Anthropic",
@@ -207,6 +209,7 @@ export const provider_map: Record = {
A2A_Agent: "a2a_agent",
AI21: "ai21",
AI21_CHAT: "ai21_chat",
+ AIAND: "aiand",
AIML: "aiml",
AIOHTTP_OPENAI: "aiohttp_openai",
Anthropic: "anthropic",
@@ -344,6 +347,7 @@ export const providerLogoMap: Partial> = {
[Providers.A2A_Agent]: a2aAgentLogo.src,
[Providers.AI21]: ai21Logo.src,
[Providers.AI21_CHAT]: ai21Logo.src,
+ [Providers.AIAND]: aiandLogo.src,
[Providers.AIML]: aimlApiLogo.src,
[Providers.AIOHTTP_OPENAI]: openaiSmallLogo.src,
[Providers.Anthropic]: anthropicLogo.src,
@@ -472,6 +476,7 @@ export const getProviderLogoAndName = (providerValue: string): { logo: string; d
const providerPlaceholderMap: Partial> = {
[Providers.AIML]: "aiml/flux-pro/v1.1",
+ [Providers.AIAND]: "aiand/openai/gpt-oss-120b",
[Providers.Anthropic]: "claude-3-opus",
[Providers.Azure]: "my-deployment",
[Providers.Azure_AI_Studio]: "azure_ai/command-r-plus",
diff --git a/ui/litellm-dashboard/src/lib/logoTreatments.ts b/ui/litellm-dashboard/src/lib/logoTreatments.ts
index fc19c5c364d..4cd6f65a613 100644
--- a/ui/litellm-dashboard/src/lib/logoTreatments.ts
+++ b/ui/litellm-dashboard/src/lib/logoTreatments.ts
@@ -32,6 +32,7 @@ const TREATMENT_BY_ASSET: Readonly> = {
"v0.svg": "invert",
"vercel.svg": "invert",
"watsonx.svg": "invert",
+ "aiand.svg": "plate",
"aiml_api.svg": "plate",
"akto.svg": "plate",
"aws.svg": "plate",
From ccdc8c7d10f8b042f576b1adfb670a02011cddcd Mon Sep 17 00:00:00 2001
From: mrinal-berri
Date: Fri, 30 Oct 2026 10:59:28 -0700
Subject: [PATCH 19/87] docs(daily-activity): describe guardrail spend fields
and reporting key IDs in OpenAPI (#49236)
* docs(daily-activity): describe guardrail spend and reporting key IDs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(daily-activity): describe reporting key IDs on /user/daily/activity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
litellm/proxy/_lazy_openapi_snapshot.json | 17 +++++++--
.../daily_activity_scopes.py | 6 ++-
.../internal_user_endpoints.py | 3 +-
.../common_daily_activity.py | 27 +++++++++-----
ui/litellm-dashboard/src/lib/http/schema.d.ts | 37 +++++++++++++------
5 files changed, 64 insertions(+), 26 deletions(-)
diff --git a/litellm/proxy/_lazy_openapi_snapshot.json b/litellm/proxy/_lazy_openapi_snapshot.json
index c76b9be7c33..dc8b9415020 100644
--- a/litellm/proxy/_lazy_openapi_snapshot.json
+++ b/litellm/proxy/_lazy_openapi_snapshot.json
@@ -3491,6 +3491,7 @@
"additionalProperties": {
"$ref": "#/components/schemas/GuardrailMetric"
},
+ "description": "Spend by guardrail ID, already included in key and model spend. Populated on user, team and organization aggregated activity only, without a model filter",
"title": "Guardrails",
"type": "object"
},
@@ -3812,10 +3813,12 @@
"GuardrailKeyMetric": {
"properties": {
"metadata": {
- "$ref": "#/components/schemas/KeyMetadata"
+ "$ref": "#/components/schemas/KeyMetadata",
+ "description": "API key alias and existence information"
},
"metrics": {
- "$ref": "#/components/schemas/GuardrailSpendMetrics"
+ "$ref": "#/components/schemas/GuardrailSpendMetrics",
+ "description": "Guardrail spend and unpriced units for this API key"
}
},
"required": [
@@ -3828,6 +3831,7 @@
"GuardrailMetadata": {
"properties": {
"guardrail_name": {
+ "description": "Guardrail display name, falling back to the guardrail ID",
"title": "Guardrail Name",
"type": "string"
}
@@ -3844,14 +3848,17 @@
"additionalProperties": {
"$ref": "#/components/schemas/GuardrailKeyMetric"
},
+ "description": "Per-key guardrail spend and unpriced units, limited to keys in the day's API key breakdown",
"title": "Api Key Breakdown",
"type": "object"
},
"metadata": {
- "$ref": "#/components/schemas/GuardrailMetadata"
+ "$ref": "#/components/schemas/GuardrailMetadata",
+ "description": "Guardrail name information"
},
"metrics": {
- "$ref": "#/components/schemas/GuardrailSpendMetrics"
+ "$ref": "#/components/schemas/GuardrailSpendMetrics",
+ "description": "Guardrail spend and unpriced units across all matching API keys"
}
},
"required": [
@@ -3865,11 +3872,13 @@
"properties": {
"spend": {
"default": 0.0,
+ "description": "Guardrail spend in USD, already included in key and model spend",
"title": "Spend",
"type": "number"
},
"untracked_units": {
"default": 0,
+ "description": "Guardrail units with no known price, excluded from spend",
"title": "Untracked Units",
"type": "integer"
}
diff --git a/litellm/proxy/management_endpoints/daily_activity_scopes.py b/litellm/proxy/management_endpoints/daily_activity_scopes.py
index ca1a96f9d84..7ff73ac6e2e 100644
--- a/litellm/proxy/management_endpoints/daily_activity_scopes.py
+++ b/litellm/proxy/management_endpoints/daily_activity_scopes.py
@@ -216,7 +216,11 @@ def _user_query(
start_date: str | None = Query(default=None, description="Start date in YYYY-MM-DD format"),
end_date: str | None = Query(default=None, description="End date in YYYY-MM-DD format"),
model: str | None = Query(default=None, description="Filter by specific model"),
- api_key: str | None = Query(default=None, description="Filter by specific API key"),
+ api_key: str | None = Query(
+ default=None,
+ description="Filter by reporting API key ID. Missing keys are reported as unattributed. "
+ "For a literal key unattributed or a key starting with raw:, add an extra raw: prefix",
+ ),
user_id: str | None = Query(
default=None,
description="Filter by specific user ID. Admins can filter by any user or omit for global view. "
diff --git a/litellm/proxy/management_endpoints/internal_user_endpoints.py b/litellm/proxy/management_endpoints/internal_user_endpoints.py
index 6950cf08a25..54a2d4062d2 100644
--- a/litellm/proxy/management_endpoints/internal_user_endpoints.py
+++ b/litellm/proxy/management_endpoints/internal_user_endpoints.py
@@ -2922,7 +2922,8 @@ async def get_user_daily_activity(
),
api_key: str | None = fastapi.Query(
default=None,
- description="Filter by specific API key",
+ description="Filter by reporting API key ID. Missing keys are reported as unattributed. "
+ "For a literal key unattributed or a key starting with raw:, add an extra raw: prefix",
),
user_id: str | None = fastapi.Query(
default=None,
diff --git a/litellm/types/proxy/management_endpoints/common_daily_activity.py b/litellm/types/proxy/management_endpoints/common_daily_activity.py
index 3f1a7980b27..37f59711050 100644
--- a/litellm/types/proxy/management_endpoints/common_daily_activity.py
+++ b/litellm/types/proxy/management_endpoints/common_daily_activity.py
@@ -68,29 +68,34 @@ class MetricWithMetadata(MetricBase):
class GuardrailSpendMetrics(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
- spend: float = 0.0
- untracked_units: int = 0
+ spend: float = Field(default=0.0, description="Guardrail spend in USD, already included in key and model spend")
+ untracked_units: int = Field(default=0, description="Guardrail units with no known price, excluded from spend")
class GuardrailKeyMetric(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
- metrics: GuardrailSpendMetrics
- metadata: KeyMetadata
+ metrics: GuardrailSpendMetrics = Field(description="Guardrail spend and unpriced units for this API key")
+ metadata: KeyMetadata = Field(description="API key alias and existence information")
class GuardrailMetadata(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
- guardrail_name: str
+ guardrail_name: str = Field(description="Guardrail display name, falling back to the guardrail ID")
class GuardrailMetric(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
- metrics: GuardrailSpendMetrics
- metadata: GuardrailMetadata
- api_key_breakdown: Mapping[str, GuardrailKeyMetric] = Field(default_factory=dict)
+ metrics: GuardrailSpendMetrics = Field(
+ description="Guardrail spend and unpriced units across all matching API keys"
+ )
+ metadata: GuardrailMetadata = Field(description="Guardrail name information")
+ api_key_breakdown: Mapping[str, GuardrailKeyMetric] = Field(
+ default_factory=dict,
+ description="Per-key guardrail spend and unpriced units, limited to keys in the day's API key breakdown",
+ )
class BreakdownMetrics(LiteLLMBaseModel):
@@ -103,7 +108,11 @@ class BreakdownMetrics(LiteLLMBaseModel):
endpoints: dict[str, MetricWithMetadata] = Field(default_factory=dict) # endpoint -> {metrics, metadata}
api_keys: dict[str, KeyMetricWithMetadata] = Field(default_factory=dict) # api_key -> {metrics, metadata}
entities: dict[str, MetricWithMetadata] = Field(default_factory=dict) # entity -> {metrics, metadata}
- guardrails: Mapping[str, GuardrailMetric] = Field(default_factory=dict) # guardrail_id -> spend
+ guardrails: Mapping[str, GuardrailMetric] = Field(
+ default_factory=dict,
+ description="Spend by guardrail ID, already included in key and model spend. "
+ "Populated on user, team and organization aggregated activity only, without a model filter",
+ )
class DailySpendData(LiteLLMBaseModel):
diff --git a/ui/litellm-dashboard/src/lib/http/schema.d.ts b/ui/litellm-dashboard/src/lib/http/schema.d.ts
index 960616e2631..aafa732276b 100644
--- a/ui/litellm-dashboard/src/lib/http/schema.d.ts
+++ b/ui/litellm-dashboard/src/lib/http/schema.d.ts
@@ -27870,7 +27870,10 @@ export interface components {
entities?: {
[key: string]: components["schemas"]["MetricWithMetadata"];
};
- /** Guardrails */
+ /**
+ * Guardrails
+ * @description Spend by guardrail ID, already included in key and model spend. Populated on user, team and organization aggregated activity only, without a model filter
+ */
guardrails?: {
[key: string]: components["schemas"]["GuardrailMetric"];
};
@@ -33448,32 +33451,44 @@ export interface components {
};
/** GuardrailKeyMetric */
GuardrailKeyMetric: {
+ /** @description API key alias and existence information */
metadata: components["schemas"]["KeyMetadata"];
+ /** @description Guardrail spend and unpriced units for this API key */
metrics: components["schemas"]["GuardrailSpendMetrics"];
};
/** GuardrailMetadata */
GuardrailMetadata: {
- /** Guardrail Name */
+ /**
+ * Guardrail Name
+ * @description Guardrail display name, falling back to the guardrail ID
+ */
guardrail_name: string;
};
/** GuardrailMetric */
GuardrailMetric: {
- /** Api Key Breakdown */
+ /**
+ * Api Key Breakdown
+ * @description Per-key guardrail spend and unpriced units, limited to keys in the day's API key breakdown
+ */
api_key_breakdown?: {
[key: string]: components["schemas"]["GuardrailKeyMetric"];
};
+ /** @description Guardrail name information */
metadata: components["schemas"]["GuardrailMetadata"];
+ /** @description Guardrail spend and unpriced units across all matching API keys */
metrics: components["schemas"]["GuardrailSpendMetrics"];
};
/** GuardrailSpendMetrics */
GuardrailSpendMetrics: {
/**
* Spend
+ * @description Guardrail spend in USD, already included in key and model spend
* @default 0
*/
spend: number;
/**
* Untracked Units
+ * @description Guardrail units with no known price, excluded from spend
* @default 0
*/
untracked_units: number;
@@ -76633,7 +76648,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76682,7 +76697,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76727,7 +76742,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76773,7 +76788,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76820,7 +76835,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76866,7 +76881,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76912,7 +76927,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
@@ -76958,7 +76973,7 @@ export interface operations {
end_date?: string | null;
/** @description Filter by specific model */
model?: string | null;
- /** @description Filter by specific API key */
+ /** @description Filter by reporting API key ID. Missing keys are reported as unattributed. For a literal key unattributed or a key starting with raw:, add an extra raw: prefix */
api_key?: string | null;
/** @description Filter by specific user ID. Admins can filter by any user or omit for global view. Non-admins must provide their own user_id. */
user_id?: string | null;
From f7f87e6193e151db09ce3db40f26dcff5333a59b Mon Sep 17 00:00:00 2001
From: Ahmet Selim Fedakar <122402435+selimfedakar@users.noreply.github.com>
Date: Fri, 30 Oct 2026 21:01:00 +0300
Subject: [PATCH 20/87] fix(responses): normalize empty function_call_output to
an empty tool result (#48845)
* fix(responses): normalize empty function_call_output to an empty tool result
* test(responses): preserve empty tool outputs through guardrails
---------
Co-authored-by: Tin
---
.../transformation.py | 10 ++++---
...test_openai_responses_guardrail_handler.py | 14 +++++++---
.../test_anthropic_responses_bridge.py | 26 +++++++++++++++++
...test_function_call_output_normalization.py | 17 ++++++++++-
.../test_google_ai_studio_responses_wire.py | 28 +++++++++++++++++++
5 files changed, 86 insertions(+), 9 deletions(-)
diff --git a/litellm/responses/litellm_completion_transformation/transformation.py b/litellm/responses/litellm_completion_transformation/transformation.py
index ce8841c9190..8f0a85d67de 100644
--- a/litellm/responses/litellm_completion_transformation/transformation.py
+++ b/litellm/responses/litellm_completion_transformation/transformation.py
@@ -398,23 +398,25 @@ def tool_output_to_chat_content(
if single_media is not None:
return [single_media]
if isinstance(output, list):
+ parts: Final = cast(Sequence[object], output) # cast-ok: checked list shape, elements remain unvalidated
normalized_blocks: Final[list[ToolMessageContentPart]] = []
text_acc: Final[list[str]] = []
- for part in output:
+ for part in parts:
if not isinstance(part, dict):
continue
part_type = part.get("type")
if part_type in ("input_text", "output_text", "text"):
txt = part.get("text")
- if isinstance(txt, str) and txt:
+ if isinstance(txt, str):
text_acc.append(txt)
- normalized_blocks.append({"type": "text", "text": txt})
+ if txt:
+ normalized_blocks.append({"type": "text", "text": txt})
elif (media := decode_responses_media(part)) is not None:
normalized_blocks.append(media)
if any(b.get("type") != "text" for b in normalized_blocks):
return normalized_blocks
- if text_acc:
+ if any(text_acc) or len(text_acc) == len(parts):
return "".join(text_acc)
try:
return json.dumps(output)
diff --git a/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py b/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
index 673e880a1c0..d732d922375 100644
--- a/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
+++ b/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
@@ -724,16 +724,23 @@ class TestToolResultItemsReachTextsGuardrails:
assert blocked.value.status_code == 400
@pytest.mark.asyncio
- @pytest.mark.parametrize("empty_output", ["", None], ids=["empty_string", "null"])
- async def test_empty_tool_output_sends_no_text_and_stays_as_sent(self, empty_output: str | None) -> None:
+ @pytest.mark.parametrize(("call_item", "output_type"), _TOOL_TURNS, ids=["function", "custom"])
+ @pytest.mark.parametrize(
+ "empty_output",
+ ["", None, [], *[[{"type": kind, "text": ""}] for kind in ("input_text", "output_text", "text")]],
+ ids=["empty_string", "null", "empty_list", "input_text", "output_text", "text"],
+ )
+ async def test_empty_tool_output_sends_no_text_and_stays_as_sent(
+ self, call_item: dict[str, str], output_type: str, empty_output: object
+ ) -> None:
guardrail: Final = RecordingMaskingGuardrail(guardrail_name="test")
- call_item, output_type = _TOOL_TURNS[0]
result: Final = await OpenAIResponsesHandler().process_input_messages(
_tool_result_request(call_item, output_type, empty_output), guardrail
)
assert guardrail.seen_texts == [["Weather?"]]
+ assert guardrail.seen_message_contents == [["Weather?", None]]
assert result["input"][2] == {"type": output_type, "call_id": "call_1", "output": empty_output}
@pytest.mark.asyncio
@@ -1103,7 +1110,6 @@ class TestToolResultItemsReachTextsGuardrails:
["visible"],
id="non_dict_part",
),
- pytest.param([], ["[]"], id="empty"),
pytest.param({"result": "SECRET forecast"}, ['{"result": "SECRET forecast"}'], id="dict"),
pytest.param(
[{"type": "input_image", "file_id": "file-map"}],
diff --git a/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py b/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
index ef198b487b9..0341511496c 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
@@ -23,6 +23,7 @@ from litellm.responses.litellm_completion_transformation.transformation import (
)
from litellm.types.llms.openai import ResponsesAPIStreamEvents
from litellm.types.utils import Message, ModelResponse
+from tests.unit.proxy.conftest import httpx_transport
def test_response_api_handler_merges_metadata_and_service_tier_without_error():
@@ -593,6 +594,31 @@ def _anthropic_message(*content: Mapping[str, object]) -> httpx.Response:
)
+@pytest.mark.usefixtures(httpx_transport.__name__)
+@pytest.mark.parametrize("output", ["", [], [{"type": "input_text", "text": ""}]])
+def test_empty_responses_tool_output_reaches_anthropic_without_json_text(
+ respx_mock: respx.MockRouter, output: str | list[dict[str, str]]
+) -> None:
+ route: Final = respx_mock.post(_ANTHROPIC_MESSAGES_URL).mock(
+ return_value=_anthropic_message({"type": "text", "text": "ok"})
+ )
+ response: Final = litellm.responses(
+ model=_MODEL,
+ api_key="mock_api_key",
+ input=[
+ {"role": "user", "content": "Check the weather"},
+ {"type": "function_call", "call_id": "call_empty", "name": "get_weather", "arguments": "{}"},
+ {"type": "function_call_output", "call_id": "call_empty", "output": output},
+ ],
+ )
+ sent: Final = json.loads(route.calls.last.request.content)
+ assert response.output_text == "ok"
+ assert sent["messages"][-1] == {
+ "role": "user",
+ "content": [{"type": "tool_result", "tool_use_id": "call_empty", "content": ""}],
+ }
+
+
def _replay(output: Sequence[BaseModel]) -> list[object]:
return [*_FIRST_INPUT, *(item.model_dump(exclude_none=True) for item in output), *_NEXT_INPUT]
diff --git a/tests/unit/responses/litellm_completion_transformation/test_function_call_output_normalization.py b/tests/unit/responses/litellm_completion_transformation/test_function_call_output_normalization.py
index 65800e42124..fce06598d29 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_function_call_output_normalization.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_function_call_output_normalization.py
@@ -50,9 +50,10 @@ def test_image_detail_mime_and_adjacent_text_survive_normalization(kind: str, de
else "data:application/octet-stream;base64,AAAA",
**({} if nested_url else metadata),
}
- parts: Final = [{"type": "input_text", "text": "Screenshot follows"}, image]
+ parts: Final = [{"type": "input_text", "text": ""}, {"type": "input_text", "text": "Screenshot follows"}, image]
original: Final = deepcopy(parts)
assert _normalize_media_content(kind, parts) == [
+ *([{"type": "text", "text": ""}] if kind == "direct" else []),
{"type": "text", "text": "Screenshot follows"},
{"type": "image_url", "image_url": {"url": "data:application/octet-stream;base64,AAAA", **metadata}},
]
@@ -97,6 +98,12 @@ def test_media_normalization_preserves_formats_references_and_metadata(kind: str
("output", "expected"),
[
(None, ""),
+ ("", ""),
+ ([], ""),
+ ([{"type": "input_text", "text": ""}], ""),
+ ([{"type": kind, "text": ""} for kind in ("input_text", "output_text", "text")], ""),
+ ("[]", "[]"),
+ ([{"type": "input_text", "text": " \n"}], " \n"),
('{"ok":true}', '{"ok":true}'),
([{"type": "input_text", "text": "hello"}, {"type": "input_text", "text": " world"}], "hello world"),
({"type": "file", "status": "ready"}, '{"type": "file", "status": "ready"}'),
@@ -106,6 +113,14 @@ def test_media_normalization_preserves_formats_references_and_metadata(kind: str
),
({"type": "input_image", "format": 17}, '{"type": "input_image", "format": 17}'),
([{"type": "unknown", "value": 17}], '[{"type": "unknown", "value": 17}]'),
+ ([{"type": "input_text", "text": 17}], '[{"type": "input_text", "text": 17}]'),
+ ([{"type": "input_text"}], '[{"type": "input_text"}]'),
+ ([{"type": "input_text", "text": None}], '[{"type": "input_text", "text": null}]'),
+ (
+ [{"type": "input_text", "text": ""}, {"type": "unknown", "value": 17}],
+ '[{"type": "input_text", "text": ""}, {"type": "unknown", "value": 17}]',
+ ),
+ ([{"type": "input_text", "text": ""}, 17], '[{"type": "input_text", "text": ""}, 17]'),
(
[{"type": "input_text", "text": "keep"}, {"type": "input_image", "file_id": "file-only"}],
"keep",
diff --git a/tests/unit/responses/litellm_completion_transformation/test_google_ai_studio_responses_wire.py b/tests/unit/responses/litellm_completion_transformation/test_google_ai_studio_responses_wire.py
index f63088a606f..c7acc70ba95 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_google_ai_studio_responses_wire.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_google_ai_studio_responses_wire.py
@@ -84,6 +84,34 @@ async def test_web_search_preview_is_sent_as_google_search() -> None:
assert request_body["tools"] == [{"googleSearch": {}}]
+@pytest.mark.asyncio
+@pytest.mark.parametrize("output", ["", [], [{"type": "input_text", "text": ""}]])
+async def test_empty_responses_tool_output_reaches_gemini_without_json_text(
+ output: str | list[dict[str, str]],
+) -> None:
+ with respx.mock() as mock_router:
+ route: Final = mock_router.post(url__regex=_GEMINI_URL).mock(
+ return_value=httpx.Response(status_code=200, json=_function_call_response("signature"))
+ )
+ response: Final = await litellm.aresponses(
+ model="gemini/gemini-2.5-flash",
+ api_key="test-key",
+ input=[
+ {"role": "user", "content": "Check the weather"},
+ {"type": "function_call", "call_id": "call_empty", "name": "get_weather", "arguments": "{}"},
+ {"type": "function_call_output", "call_id": "call_empty", "output": output},
+ ],
+ )
+ request_body: Final = _JSON_OBJECT.validate_json(route.calls.last.request.content)
+
+ assert isinstance(response, ResponsesAPIResponse)
+ assert request_body["contents"] == [
+ {"role": "user", "parts": [{"text": "Check the weather"}]},
+ {"role": "model", "parts": [{"function_call": {"name": "get_weather", "args": {}}}]},
+ {"role": "user", "parts": [{"function_response": {"name": "get_weather", "response": {"content": ""}}}]},
+ ]
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize(
("assistant_content", "assistant_parts"),
From 840c83b4034a266cdd9e0858b9a568e6c348cf22 Mon Sep 17 00:00:00 2001
From: Kerry Lu
Date: Fri, 30 Oct 2026 11:19:08 -0700
Subject: [PATCH 21/87] test: bedrock claude prompt caching on /v1/messages
reports and charges cache tokens (#49265)
* test: bedrock claude prompt caching on /v1/messages reports and charges cache tokens
* ci: re-run after ci-coverage and assert-shard-coverage were cancelled by the GitHub Actions incident
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
...st_bedrock_messages_prompt_caching_wire.py | 191 ++++++++++++++++++
1 file changed, 191 insertions(+)
create mode 100644 tests/integration/messages_endpoint/providers/bedrock/test_bedrock_messages_prompt_caching_wire.py
diff --git a/tests/integration/messages_endpoint/providers/bedrock/test_bedrock_messages_prompt_caching_wire.py b/tests/integration/messages_endpoint/providers/bedrock/test_bedrock_messages_prompt_caching_wire.py
new file mode 100644
index 00000000000..f2622da1bf6
--- /dev/null
+++ b/tests/integration/messages_endpoint/providers/bedrock/test_bedrock_messages_prompt_caching_wire.py
@@ -0,0 +1,191 @@
+import base64
+import json
+import uuid
+from typing import Final
+
+import anthropic
+import httpx
+import pytest
+from integration._support import claude_code as cc
+from integration._support.client import Gateway, eventually
+from integration._support.database import read_rows
+from integration._support.upstream import _aws_event_frame
+from integration._support.wire import Reply, Request, wire_server
+from pydantic import JsonValue
+
+_MODEL_ID: Final = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+_TOKEN: Final = "synthetic-bedrock-bearer"
+_PROMPT: Final = "Reply with exactly the word PONG"
+_INPUT_TOKENS: Final = 10
+_OUTPUT_TOKENS: Final = 4
+_CACHE_READ_TOKENS: Final = 900
+_CACHE_WRITE_TOKENS: Final = 300
+_INPUT_RATE: Final = 0.001
+_OUTPUT_RATE: Final = 0.002
+_CACHE_READ_RATE: Final = 0.0001
+_CACHE_WRITE_RATE: Final = 0.0015
+_EXPECTED_SPEND: Final = (
+ _INPUT_TOKENS * _INPUT_RATE
+ + _CACHE_READ_TOKENS * _CACHE_READ_RATE
+ + _CACHE_WRITE_TOKENS * _CACHE_WRITE_RATE
+ + _OUTPUT_TOKENS * _OUTPUT_RATE
+)
+_PROMPT_TOKENS: Final = _INPUT_TOKENS + _CACHE_READ_TOKENS + _CACHE_WRITE_TOKENS
+_USAGE: Final = {
+ "input_tokens": _INPUT_TOKENS,
+ "output_tokens": _OUTPUT_TOKENS,
+ "cache_read_input_tokens": _CACHE_READ_TOKENS,
+ "cache_creation_input_tokens": _CACHE_WRITE_TOKENS,
+}
+_USER_CONTENT: Final = [{"type": "text", "text": _PROMPT, "cache_control": cc.CACHE}]
+_CACHED_SYSTEM_BLOCKS: Final = [block for block in cc.system_blocks() if "cache_control" in block]
+
+
+def _assert_claude_code_breakpoints_reach_bedrock(request: Request, target: str) -> dict[str, JsonValue]:
+ assert request.method == "POST" and request.target == target, request.target
+ assert request.headers["authorization"] == f"Bearer {_TOKEN}"
+ body: Final = cc.JSON_OBJECT.validate_json(request.body)
+ assert body["anthropic_version"] == "bedrock-2023-05-31", body
+ assert body["system"] == _CACHED_SYSTEM_BLOCKS, f"billing-header block dropped, breakpoints kept: {body}"
+ assert body["messages"] == [{"role": "user", "content": _USER_CONTENT}], body
+ return body
+
+
+def _invoke_chunk(event: dict[str, JsonValue]) -> bytes:
+ return _aws_event_frame("chunk", {"bytes": base64.b64encode(json.dumps(event).encode()).decode()}, "sc", "u")
+
+
+def _stream_frames(message_id: str) -> tuple[bytes, ...]:
+ return (
+ _invoke_chunk(
+ {
+ "type": "message_start",
+ "message": {
+ "id": message_id,
+ "type": "message",
+ "role": "assistant",
+ "content": [],
+ "model": "claude-sonnet-4-5-20250929",
+ "stop_reason": None,
+ "stop_sequence": None,
+ "usage": {**_USAGE, "output_tokens": 1},
+ },
+ }
+ ),
+ _invoke_chunk({"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}),
+ _invoke_chunk({"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "PONG"}}),
+ _invoke_chunk({"type": "content_block_stop", "index": 0}),
+ _invoke_chunk(
+ {
+ "type": "message_delta",
+ "delta": {"stop_reason": "end_turn", "stop_sequence": None},
+ "usage": {"output_tokens": _OUTPUT_TOKENS},
+ }
+ ),
+ _invoke_chunk({"type": "message_stop"}),
+ )
+
+
+def _spend_row(message_id: str) -> dict[str, object]:
+ rows: Final = eventually(
+ lambda: read_rows(
+ 'SELECT spend, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE request_id=%s',
+ (message_id,),
+ ),
+ lambda values: len(values) == 1,
+ seconds=70,
+ )
+ return rows[0]
+
+
+def _client(gateway: Gateway, transport: httpx.Client) -> anthropic.Anthropic:
+ return anthropic.Anthropic(
+ api_key=gateway.key, base_url=str(gateway.client.base_url), max_retries=0, http_client=transport
+ )
+
+
+def test_bedrock_claude_cache_control_reaches_invoke_and_cache_tokens_are_reported_and_charged(
+ gateway: Gateway,
+) -> None:
+ message_id: Final = f"msg_{uuid.uuid4().hex}"
+
+ def respond(request: Request) -> Reply:
+ _assert_claude_code_breakpoints_reach_bedrock(request, f"/model/{_MODEL_ID}/invoke")
+ return Reply(
+ body=cc.message_reply(
+ message_id, "claude-sonnet-4-5-20250929", ({"type": "text", "text": "PONG"},), dict(_USAGE)
+ )
+ )
+
+ with (
+ wire_server(respond) as wire,
+ gateway.scenario() as scenario,
+ httpx.Client(timeout=15, trust_env=False) as transport,
+ ):
+ model: Final = scenario.model(
+ model=f"bedrock/invoke/{_MODEL_ID}",
+ api_key=_TOKEN,
+ aws_region_name="us-east-1",
+ api_base=wire.url,
+ input_cost_per_token=_INPUT_RATE,
+ output_cost_per_token=_OUTPUT_RATE,
+ cache_read_input_token_cost=_CACHE_READ_RATE,
+ cache_creation_input_token_cost=_CACHE_WRITE_RATE,
+ )
+ final: Final = _client(gateway, transport).messages.create(
+ model=model,
+ max_tokens=32,
+ system=list(cc.system_blocks()),
+ messages=[{"role": "user", "content": _USER_CONTENT}],
+ )
+ assert len(wire.drain()) == 1
+ assert final.id == message_id, final
+ assert [(block.type, block.text) for block in final.content] == [("text", "PONG")], final
+ assert final.usage.model_dump(exclude_none=True, include=set(_USAGE)) == _USAGE, final.usage
+ row: Final = _spend_row(message_id)
+
+ assert float(str(row["spend"])) == pytest.approx(_EXPECTED_SPEND), row
+ assert (row["prompt_tokens"], row["completion_tokens"]) == (_PROMPT_TOKENS, _OUTPUT_TOKENS), row
+
+
+def test_bedrock_claude_stream_cache_tokens_from_message_start_are_reported_and_charged(gateway: Gateway) -> None:
+ message_id: Final = f"msg_{uuid.uuid4().hex}"
+
+ def respond(request: Request) -> Reply:
+ body: Final = _assert_claude_code_breakpoints_reach_bedrock(
+ request, f"/model/{_MODEL_ID}/invoke-with-response-stream"
+ )
+ assert "stream" not in body, body
+ return Reply(chunks=_stream_frames(message_id), content_type="application/vnd.amazon.eventstream")
+
+ with (
+ wire_server(respond) as wire,
+ gateway.scenario() as scenario,
+ httpx.Client(timeout=15, trust_env=False) as transport,
+ ):
+ model: Final = scenario.model(
+ model=f"bedrock/invoke/{_MODEL_ID}",
+ api_key=_TOKEN,
+ aws_region_name="us-east-1",
+ api_base=wire.url,
+ input_cost_per_token=_INPUT_RATE,
+ output_cost_per_token=_OUTPUT_RATE,
+ cache_read_input_token_cost=_CACHE_READ_RATE,
+ cache_creation_input_token_cost=_CACHE_WRITE_RATE,
+ )
+ with _client(gateway, transport).messages.stream(
+ model=model,
+ max_tokens=32,
+ system=list(cc.system_blocks()),
+ messages=[{"role": "user", "content": _USER_CONTENT}],
+ ) as stream:
+ deltas: Final = tuple(stream.text_stream)
+ final: Final = stream.get_final_message()
+ assert len(wire.drain()) == 1
+ assert deltas == ("PONG",)
+ assert final.id == message_id, final
+ assert final.usage.model_dump(exclude_none=True, include=set(_USAGE)) == _USAGE, final.usage
+ row: Final = _spend_row(message_id)
+
+ assert float(str(row["spend"])) == pytest.approx(_EXPECTED_SPEND), row
+ assert (row["prompt_tokens"], row["completion_tokens"]) == (_PROMPT_TOKENS, _OUTPUT_TOKENS), row
From d23af8bdd08943fe1a0e1d9e528fcb12000b287a Mon Sep 17 00:00:00 2001
From: Kerry Lu
Date: Fri, 30 Oct 2026 11:24:49 -0700
Subject: [PATCH 22/87] fix(cost): bill the margin once on rebuilt streams and
streamed Responses through the chat bridge (#49254)
* fix(cost): bill the margin once on rebuilt streams and streamed Responses through the chat bridge
The last usage chunk and the Responses terminal event carry litellm's bill in usage.cost, and two readers took that bill for the upstream's own charge and margined it again: completion_cost on a stream_chunk_builder result, and stamp_responses_usage_cost on the chat bridge. The stamp now keeps the upstream's usage.cost under the hidden upstream_usage_cost key, the chunk builder carries it onto the rebuilt response, and the Responses stamp prices that saved figure
* fix(cost): hand the Responses stamp the upstream charge through upstream_usage_cost only, never the response-cost header
response_cost_calculator returns the response-cost header as the final bill without the margin, so a priced copy carrying the upstream's raw usage.cost there billed the client the unmargined figure. The hidden upstream_usage_cost key already feeds billed_usage_cost, which margins it once. The Responses test now prices through a real Logging object instead of a mocked calculator
* fix(cost): leave stamp_responses_usage_cost as on main
The Responses chat bridge copies the inner stream's final usage chunk, so the wrapper stamp keeping upstream_usage_cost already prices it once. Changing the Responses stamp was not needed for that and re-priced xai MCP rounds whose usage.cost is already litellm's bill (0.00512 for 0.00312 in test_xai_stream_auto_execute_bills_each_round_at_its_own_price_with_one_fixed_margin)
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
.../streaming_chunk_builder_utils.py | 18 ++++++++++-
.../litellm_core_utils/streaming_handler.py | 18 ++++++++---
.../test_streaming_chunk_builder_utils.py | 24 ++++++++++++++
.../test_streaming_handler.py | 31 +++++++++++++++++++
4 files changed, 86 insertions(+), 5 deletions(-)
diff --git a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
index 574f1103c1a..0ddfa1210c2 100644
--- a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
+++ b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
@@ -14,12 +14,18 @@ import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.core_helpers import (
PROVIDERS_BILLED_FROM_THEIR_OWN_USAGE_FIELDS,
+ UPSTREAM_USAGE_COST,
chunk_usage,
finite_cost,
provider_reported_usage_cost,
stamp_litellm_bill_on_usage,
)
-from litellm.litellm_core_utils.hidden_params import HIDDEN_PARAMS_ATTR, set_hidden_params
+from litellm.litellm_core_utils.hidden_params import (
+ HIDDEN_PARAMS_ATTR,
+ get_hidden_params,
+ set_hidden_param,
+ set_hidden_params,
+)
from litellm.litellm_core_utils.llm_cost_calc.guardrail_cost import BEDROCK_TRACE_FIELDS, object_items, object_mapping
from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import partition_annotations
from litellm.litellm_core_utils.prompt_templates.common_utils import (
@@ -1444,6 +1450,14 @@ def _set_stream_builder_response_cost(
_set_hidden_response_cost(response, response_cost)
+def _carry_upstream_usage_cost(response: ModelResponse, chunks: Sequence[object]) -> None:
+ chunk_hidden_params: Final = (get_hidden_params(chunk) for chunk in reversed(chunks))
+ stamped: Final = (hidden for hidden in chunk_hidden_params if hidden is not None and UPSTREAM_USAGE_COST in hidden)
+ for hidden in stamped:
+ set_hidden_param(response, UPSTREAM_USAGE_COST, hidden[UPSTREAM_USAGE_COST])
+ return
+
+
def _stamp_streaming_usage_cost(usage: Usage, response: ModelResponse, logging_obj: Optional["Logging"]) -> None:
if logging_obj is None:
return
@@ -1794,6 +1808,7 @@ def stream_chunk_builder(
response,
logging_obj,
)
+ _carry_upstream_usage_cost(response, chunks) # pyright: ignore[reportUnknownArgumentType] # chunks is an untyped list param
_stamp_streaming_usage_cost(usage, response, logging_obj)
_set_stream_builder_response_cost(
response,
@@ -1836,6 +1851,7 @@ def stream_chunk_builder(
)
break
+ _carry_upstream_usage_cost(response, chunks) # pyright: ignore[reportUnknownArgumentType] # chunks is an untyped list param
_stamp_streaming_usage_cost(usage, response, logging_obj)
_set_stream_builder_response_cost(
response,
diff --git a/litellm/litellm_core_utils/streaming_handler.py b/litellm/litellm_core_utils/streaming_handler.py
index b1b3e20f218..9710956c97e 100644
--- a/litellm/litellm_core_utils/streaming_handler.py
+++ b/litellm/litellm_core_utils/streaming_handler.py
@@ -24,7 +24,12 @@ import litellm
from litellm import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.asyncify import asyncify
-from litellm.litellm_core_utils.core_helpers import chunk_usage, provider_reported_usage_cost
+from litellm.litellm_core_utils.core_helpers import (
+ UPSTREAM_USAGE_COST,
+ chunk_usage,
+ provider_reported_usage_cost,
+ stamp_litellm_bill_on_usage,
+)
from litellm.litellm_core_utils.hidden_params import (
HIDDEN_PARAMS_ATTR,
get_hidden_params,
@@ -98,12 +103,17 @@ def _stamp_usage_cost(
usage_chunk: ModelResponse | TextCompletionResponse | ModelResponseStream,
complete_response: ModelResponse | TextCompletionResponse | ModelResponseStream,
) -> None:
- bill: Final = (get_hidden_params(complete_response) or {}).get("response_cost")
+ complete_hidden: Final = get_hidden_params(complete_response) or {}
+ bill: Final = complete_hidden.get("response_cost")
usage: Final[object] = getattr(usage_chunk, "usage", None)
if usage is None or isinstance(bill, bool) or not isinstance(bill, (int, float)):
return
- if bill > 0 or isinstance(getattr(usage, "cost", None), (int, float)):
- setattr(usage, "cost", float(bill)) # noqa: B010 # cost is an optional extra on the usage model
+ if bill <= 0 and not isinstance(getattr(usage, "cost", None), (int, float)):
+ return
+ chunk_hidden: Final = get_hidden_params(usage_chunk)
+ if chunk_hidden is not None and UPSTREAM_USAGE_COST in complete_hidden:
+ chunk_hidden[UPSTREAM_USAGE_COST] = complete_hidden[UPSTREAM_USAGE_COST]
+ stamp_litellm_bill_on_usage(usage_chunk, usage, float(bill))
def _stamp_served_service_tier(response: ModelResponseStream, complete_streaming_response: ModelResponse) -> None:
diff --git a/tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.py b/tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.py
index c8fa9c94c17..9df45b92686 100644
--- a/tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.py
+++ b/tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.py
@@ -1077,6 +1077,30 @@ def test_cost_field_in_usage_chunks():
assert usage.completion_tokens == 5
+def test_the_rebuilt_response_keeps_the_upstream_cost_a_stamped_usage_chunk_saved(monkeypatch: pytest.MonkeyPatch):
+ monkeypatch.setattr(litellm, "cost_margin_config", {"global": 0.5})
+ text_chunk = ModelResponseStream(
+ id="chatcmpl-1",
+ model="gpt-4.1",
+ choices=[StreamingChoices(finish_reason="stop", index=0, delta=Delta(content="Hi"))],
+ )
+ usage_chunk = ModelResponseStream(
+ id="chatcmpl-1",
+ model="gpt-4.1",
+ choices=[],
+ usage=Usage(completion_tokens=5, prompt_tokens=10, total_tokens=15, cost=0.0015),
+ )
+ usage_chunk._hidden_params = {"response_cost": 0.0015, "upstream_usage_cost": 0.001}
+
+ rebuilt = stream_chunk_builder([text_chunk, usage_chunk], messages=[{"role": "user", "content": "hi"}])
+
+ assert rebuilt is not None
+ assert rebuilt._hidden_params["upstream_usage_cost"] == 0.001
+ assert litellm.completion_cost(rebuilt, custom_llm_provider="openai") == pytest.approx(0.0015), (
+ "the upstream's 0.001 gets the 50% margin once, litellm's 0.0015 bill is not margined again"
+ )
+
+
def _perplexity_usage_chunk(cost: Mapping[str, float], finish_reason: str | None) -> ModelResponseStream:
return ModelResponseStream(
diff --git a/tests/unit/litellm_core_utils/test_streaming_handler.py b/tests/unit/litellm_core_utils/test_streaming_handler.py
index 148b7193ef6..6df4731e335 100644
--- a/tests/unit/litellm_core_utils/test_streaming_handler.py
+++ b/tests/unit/litellm_core_utils/test_streaming_handler.py
@@ -6847,6 +6847,37 @@ def test_the_usage_chunk_carries_the_bill_the_builder_priced_for_the_spend_row()
assert usage_chunk.usage is not None and usage_chunk.usage.cost == 0.00032, "the client sees what the row bills"
+@pytest.mark.parametrize("upstream_cost", [None, 0.001])
+def test_rebuilding_a_priced_stream_bills_the_margin_once(monkeypatch: pytest.MonkeyPatch, upstream_cost: float | None) -> None:
+ from litellm.litellm_core_utils.core_helpers import stamp_litellm_bill_on_usage
+ from litellm.litellm_core_utils.streaming_handler import (
+ _stamp_usage_cost, # pyright: ignore[reportPrivateUsage] # the helper under test
+ )
+
+ monkeypatch.setattr(litellm, "cost_margin_config", {"global": 0.5})
+ model: Final = "anthropic/claude-sonnet-4-5"
+ shared_usage: Final = Usage(prompt_tokens=9, completion_tokens=16, total_tokens=25)
+ if upstream_cost is not None:
+ setattr(shared_usage, "cost", upstream_cost)
+ reference: Final = ModelResponse(model=model, usage=shared_usage.model_copy())
+ billed_once: Final = litellm.completion_cost(reference, custom_llm_provider="anthropic")
+ complete: Final = ModelResponse(model=model, usage=shared_usage)
+ stamp_litellm_bill_on_usage(complete, shared_usage, billed_once)
+ text_chunk: Final = ModelResponseStream(
+ id="chunk-1", model=model, choices=[StreamingChoices(index=0, delta=Delta(content="hi"), finish_reason="stop")]
+ )
+ usage_chunk: Final = ModelResponseStream(id="chunk-1", model=model, choices=[], usage=shared_usage)
+
+ _stamp_usage_cost(usage_chunk, complete)
+ rebuilt: Final = litellm.stream_chunk_builder([text_chunk, usage_chunk], messages=[{"role": "user", "content": "hi"}])
+
+ assert rebuilt is not None and rebuilt.usage is not None # pyright: ignore[reportAttributeAccessIssue] # ModelResponse carries usage at runtime
+ assert usage_chunk.usage is not None and usage_chunk.usage.cost == pytest.approx(billed_once)
+ assert litellm.completion_cost(rebuilt, custom_llm_provider="anthropic") == pytest.approx(billed_once), (
+ "pricing the rebuilt stream must not margin litellm's own bill again"
+ )
+
+
def test_chunk_creator_keeps_provider_model_private_across_stream():
from litellm.router_utils.add_retry_fallback_headers import (
get_hidden_params_dict,
From a1c238ff4a1e1e8fc9cf2e9520c3132418e400f6 Mon Sep 17 00:00:00 2001
From: yujonglee
Date: Fri, 30 Oct 2026 11:31:04 -0700
Subject: [PATCH 23/87] refactor(rust): extract SecretValue and use it for
Redis credentials (#49186)
* refactor(rust): move SecretValue into litellm-secret-value
* docs(rust): add rust-redaction skill
* refactor(redis-client): carry credentials as SecretValue
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
.../.agents/skills/rust-redaction/SKILL.md | 43 +++++++++
litellm-rust/AGENTS.md | 2 +
litellm-rust/Cargo.lock | 50 ++++++++---
litellm-rust/Cargo.toml | 1 +
litellm-rust/crates/auth-azure/Cargo.toml | 1 +
litellm-rust/crates/auth-azure/src/native.rs | 6 +-
litellm-rust/crates/auth-azure/src/resolve.rs | 9 +-
litellm-rust/crates/auth-azure/src/types.rs | 4 +-
litellm-rust/crates/auth-gcp/Cargo.toml | 1 +
litellm-rust/crates/auth-gcp/src/auth.rs | 2 +-
litellm-rust/crates/auth-gcp/src/config.rs | 4 +-
litellm-rust/crates/auth-types/Cargo.toml | 2 +-
.../crates/auth-types/src/credential.rs | 5 +-
litellm-rust/crates/auth-types/src/lib.rs | 2 -
litellm-rust/crates/auth-types/src/policy.rs | 3 +-
litellm-rust/crates/auth-types/src/token.rs | 3 +-
litellm-rust/crates/auth-types/tests/token.rs | 6 +-
litellm-rust/crates/auth/Cargo.toml | 3 +
litellm-rust/crates/auth/tests/facade.rs | 3 +-
litellm-rust/crates/cache-gcs/Cargo.toml | 1 +
litellm-rust/crates/cache-gcs/src/token.rs | 4 +-
.../crates/callbacks-legacy-python/Cargo.toml | 2 +-
.../callbacks-legacy-python/src/adapter.rs | 2 +-
litellm-rust/crates/config/Cargo.toml | 2 +-
litellm-rust/crates/config/src/mcp.rs | 2 +-
litellm-rust/crates/config/src/redis.rs | 2 +-
litellm-rust/crates/config/src/settings.rs | 2 +-
litellm-rust/crates/gateway-auth/Cargo.toml | 2 +-
.../crates/gateway-auth/src/authentication.rs | 2 +-
litellm-rust/crates/gateway-auth/src/http.rs | 2 +-
litellm-rust/crates/gateway-auth/src/keys.rs | 2 +-
litellm-rust/crates/gateway-auth/src/ui.rs | 2 +-
.../crates/gateway-auth/tests/auth.rs | 2 +-
.../crates/gateway-auth/tests/keys.rs | 2 +-
litellm-rust/crates/gateway-auth/tests/ui.rs | 2 +-
.../crates/gateway-inference/Cargo.toml | 1 +
.../crates/gateway-inference/src/ocr.rs | 2 +-
.../gateway-inference/tests/support/mod.rs | 4 +-
.../crates/gateway-management/Cargo.toml | 2 +-
.../crates/gateway-management/src/keys.rs | 2 +-
.../crates/gateway-management/tests/keys.rs | 2 +-
litellm-rust/crates/gateway-mcp/Cargo.toml | 2 +-
.../crates/gateway-mcp/src/configured.rs | 2 +-
litellm-rust/crates/gateway-ui/Cargo.toml | 2 +-
.../crates/gateway-ui/tests/routes.rs | 2 +-
litellm-rust/crates/gateway/Cargo.toml | 2 +-
litellm-rust/crates/gateway/src/redis.rs | 10 +--
litellm-rust/crates/gateway/src/secrets.rs | 2 +-
litellm-rust/crates/gateway/src/settings.rs | 2 +-
litellm-rust/crates/gateway/tests/mcp.rs | 2 +-
litellm-rust/crates/gateway/tests/server.rs | 2 +-
litellm-rust/crates/host/Cargo.toml | 2 +-
litellm-rust/crates/host/src/interceptors.rs | 2 +-
litellm-rust/crates/inference-chat/Cargo.toml | 1 +
.../crates/inference-chat/src/handler.rs | 2 +-
.../crates/inference-chat/tests/caching.rs | 4 +-
.../crates/inference-messages/Cargo.toml | 1 +
.../crates/inference-messages/src/handler.rs | 4 +-
.../inference-messages/tests/caching.rs | 4 +-
litellm-rust/crates/inference-ocr/Cargo.toml | 1 +
.../crates/inference-ocr/src/handler.rs | 2 +-
.../inference-ocr/src/provider_config.rs | 16 ++--
.../crates/inference-ocr/src/types.rs | 4 +-
litellm-rust/crates/inference-ocr/src/wire.rs | 4 +-
.../inference-ocr/tests/ocr/azure_ai.rs | 6 +-
.../inference-ocr/tests/ocr/lifecycle.rs | 2 +-
.../crates/inference-ocr/tests/ocr/main.rs | 2 +-
.../crates/inference-responses/Cargo.toml | 1 +
.../inference-responses/tests/caching.rs | 4 +-
.../crates/inference-testing/Cargo.toml | 1 +
.../crates/inference-testing/src/lib.rs | 3 +-
litellm-rust/crates/llms/Cargo.toml | 1 +
.../llms/src/anthropic/chat/transformation.rs | 2 +-
.../crates/llms/src/anthropic/common_utils.rs | 4 +-
.../src/azure_ai/messages/transformation.rs | 2 +-
.../document_intelligence/transformation.rs | 2 +-
.../llms/src/azure_ai/ocr/transformation.rs | 4 +-
litellm-rust/crates/llms/src/base_llm/auth.rs | 4 +-
.../llms/src/base_llm/ocr/transformation.rs | 4 +-
.../bedrock/chat/converse_transformation.rs | 4 +-
.../messages/transformation.rs | 4 +-
.../anthropic_claude_transformation.rs | 4 +-
.../bedrock_mantle/messages/transformation.rs | 4 +-
.../llms/src/cohere/ocr/transformation.rs | 2 +-
.../src/deepseek/messages/transformation.rs | 4 +-
.../src/edenai/messages/transformation.rs | 4 +-
.../src/minimax/messages/transformation.rs | 4 +-
.../llms/src/mistral/ocr/transformation.rs | 2 +-
.../src/openai/responses/transformation.rs | 3 +-
.../src/openai_like/chat/transformation.rs | 4 +-
.../src/openrouter/messages/transformation.rs | 4 +-
.../openrouter/responses/transformation.rs | 4 +-
.../llms/src/reducto/ocr/transformation.rs | 4 +-
.../src/tencent/messages/transformation.rs | 4 +-
.../llms/src/vertex_ai/ocr/transformation.rs | 2 +-
litellm-rust/crates/python-bridge/Cargo.toml | 1 +
.../crates/python-bridge/src/credentials.rs | 4 +-
.../python-bridge/src/routes/ocr/project.rs | 2 +-
.../python-bridge/src/secrets/callback.rs | 4 +-
.../python-bridge/src/secrets/config.rs | 4 +-
.../python-bridge/src/secrets/operations.rs | 8 +-
.../python-bridge/src/secrets/provider.rs | 4 +-
.../python-bridge/src/secrets/python.rs | 4 +-
.../python-bridge/src/secrets/resolved.rs | 5 +-
.../python-bridge/src/secrets/runtime.rs | 4 +-
.../src/secrets/vault/operation.rs | 7 +-
litellm-rust/crates/redis-client/Cargo.toml | 1 +
.../crates/redis-client/src/client.rs | 11 ++-
.../crates/redis-client/src/config.rs | 90 ++++++-------------
.../crates/redis-client/src/connection.rs | 4 +-
.../crates/redis-client/src/test_support.rs | 3 +-
litellm-rust/crates/router-types/Cargo.toml | 1 +
.../crates/router-types/src/litellm_params.rs | 4 +-
.../router-types/tests/litellm_params.rs | 4 +-
litellm-rust/crates/secret-value/Cargo.toml | 14 +++
litellm-rust/crates/secret-value/src/lib.rs | 5 ++
.../src/secret_value.rs} | 6 +-
litellm-rust/crates/secrets-aws/Cargo.toml | 1 +
.../crates/secrets-aws/src/secret_manager.rs | 5 +-
.../secrets-aws/tests/secret_manager.rs | 4 +-
litellm-rust/crates/secrets-azure/Cargo.toml | 1 +
.../crates/secrets-azure/src/key_vault.rs | 4 +-
.../crates/secrets-azure/tests/key_vault.rs | 4 +-
.../crates/secrets-cyberark/Cargo.toml | 1 +
.../secrets-cyberark/src/secret_manager.rs | 5 +-
.../secrets-cyberark/tests/secret_manager.rs | 4 +-
litellm-rust/crates/secrets-google/Cargo.toml | 1 +
.../crates/secrets-google/src/auth.rs | 2 +-
litellm-rust/crates/secrets-google/src/kms.rs | 2 +-
.../secrets-google/src/secret_manager.rs | 6 +-
.../crates/secrets-hashicorp/Cargo.toml | 1 +
.../crates/secrets-hashicorp/src/config.rs | 2 +-
.../secrets-hashicorp/src/secret_manager.rs | 5 +-
.../secrets-hashicorp/tests/secret_manager.rs | 6 +-
litellm-rust/crates/secrets-types/Cargo.toml | 2 +-
.../secrets-types/src/base_secret_manager.rs | 3 +-
.../crates/secrets-types/src/config.rs | 2 +-
.../crates/secrets-types/src/context.rs | 3 +-
litellm-rust/crates/secrets-types/src/lib.rs | 1 -
.../crates/secrets-types/src/value.rs | 2 +-
.../crates/secrets-types/tests/config.rs | 6 +-
.../crates/secrets-types/tests/context.rs | 5 +-
.../crates/secrets-types/tests/rotation.rs | 6 +-
litellm-rust/crates/secrets/Cargo.toml | 1 +
.../crates/secrets/src/compatibility.rs | 3 +-
litellm-rust/crates/secrets/src/handler.rs | 2 +-
litellm-rust/crates/secrets/src/lib.rs | 4 +-
litellm-rust/crates/secrets/src/oidc.rs | 3 +-
litellm-rust/crates/secrets/src/resolver.rs | 3 +-
litellm-rust/crates/secrets/src/source.rs | 3 +-
litellm-rust/crates/secrets/tests/aws.rs | 4 +-
litellm-rust/crates/secrets/tests/azure.rs | 5 +-
.../secrets/tests/common_read_contract.rs | 4 +-
litellm-rust/crates/secrets/tests/cyberark.rs | 4 +-
litellm-rust/crates/secrets/tests/google.rs | 4 +-
litellm-rust/crates/secrets/tests/oidc.rs | 6 +-
.../crates/secrets/tests/resolution.rs | 3 +-
157 files changed, 439 insertions(+), 254 deletions(-)
create mode 100644 litellm-rust/.agents/skills/rust-redaction/SKILL.md
create mode 100644 litellm-rust/crates/secret-value/Cargo.toml
create mode 100644 litellm-rust/crates/secret-value/src/lib.rs
rename litellm-rust/crates/{auth-types/src/secret.rs => secret-value/src/secret_value.rs} (95%)
diff --git a/litellm-rust/.agents/skills/rust-redaction/SKILL.md b/litellm-rust/.agents/skills/rust-redaction/SKILL.md
new file mode 100644
index 00000000000..2e906e10607
--- /dev/null
+++ b/litellm-rust/.agents/skills/rust-redaction/SKILL.md
@@ -0,0 +1,43 @@
+---
+name: rust-redaction
+description: Keep secrets out of Debug output in litellm-rust with SecretValue and veil. Use when a struct or enum holds a password, token, key, credential URL or credential provider, when writing or reviewing a Debug impl, or when a type with secrets gets logged or traced
+metadata:
+ internal: true
+---
+
+# Rust redaction
+
+Every secret in a Debug output is hidden the same way, so a reviewer can tell at a glance that a field is covered and a new field cannot slip through a hand-written impl
+
+## Where SecretValue lives
+
+`SecretValue` lives in `crates/secret-value` and is imported as `litellm_secret_value::SecretValue` everywhere. No crate re-exports it, so there is one import path. Depend on `litellm-secret-value` directly. Do not pull in `litellm-auth-types` or `litellm-auth` just to get it
+
+`crates/secret-value` holds only the type. Credential acquisition (`TokenProvider`, `CredentialPlan`, provider param tables) stays in `litellm-auth-types`, and generic clients such as `litellm-redis-client` must not depend on it
+
+## Rules
+
+- Never hand-write `impl Debug` to hide a value, and never invent markers like `""`, `""` or `.map(|_| "...")`. A hand-written impl silently prints any field added later
+- A secret that crosses a crate boundary or lives in config is typed as `SecretValue`. It redacts itself, compares in constant time and only gives up the plaintext through `expose()`. A type whose only secrets are `SecretValue` fields derives plain `Debug`
+ ```rust
+ #[derive(Clone, Debug, PartialEq)]
+ pub struct HostEndpoint {
+ pub host: String,
+ pub port: u16,
+ pub username: Option,
+ pub password: Option,
+ pub tls: bool,
+ pub db: i64,
+ }
+ ```
+- Keep a secret as `SecretValue` until the last conversion into the third-party type (redis-rs, reqwest, an SDK) and call `expose()` there. Calling `expose()` early spreads plaintext `String`s through our own types
+- A secret that has to stay a raw type, because a serde wire shape or a third-party struct owns the field, is covered by deriving `veil::Redact` on the holder and marking the field `#[redact(with = "[REDACTED]")]`. `Redact` replaces `Debug`, so do not derive both
+- A URL that can carry userinfo is a secret as a whole. Do not parse it to show the host. Keep it as a `SecretValue`
+- A bag of unparsed settings that may hold credentials (`additional_fields`, raw kwargs) is redacted as a whole with `#[redact(with = "[REDACTED]")]`
+- An opaque handle over `Arc` whose trait has no `Debug` gets a newtype whose `Debug` prints only the type name with `finish_non_exhaustive()`. That impl shows nothing, so it is not a redaction impl and the first rule does not apply. A type holding such a handle can derive `Debug`
+- Never put a secret in a tracing field or an error message. Log the type with `?value`, or log only non-secret fields. See the rust-tracing and rust-error skills
+
+## Tests
+
+- One test per type that holds a secret, built the way production builds it, asserting the plaintext is absent from `format!("{value:?}")` and `[REDACTED]` is present
+- Use a distinctive plaintext (`"credential-value"`) so the absence check cannot pass by accident
diff --git a/litellm-rust/AGENTS.md b/litellm-rust/AGENTS.md
index 9e7a55eb232..5eaf876e3b2 100644
--- a/litellm-rust/AGENTS.md
+++ b/litellm-rust/AGENTS.md
@@ -1,5 +1,7 @@
# Rust workspace rules
+For keeping secrets out of Debug output, follow [.agents/skills/rust-redaction/SKILL.md](.agents/skills/rust-redaction/SKILL.md)
+
For diagnostic tracing changes, follow [.agents/skills/rust-tracing/SKILL.md](.agents/skills/rust-tracing/SKILL.md)
For string-valued enums and their Serde conversions, follow [.agents/skills/rust-string-enums/SKILL.md](.agents/skills/rust-string-enums/SKILL.md)
diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock
index e959f77917b..f91deaaa401 100644
--- a/litellm-rust/Cargo.lock
+++ b/litellm-rust/Cargo.lock
@@ -3380,6 +3380,7 @@ dependencies = [
"litellm-auth-azure",
"litellm-auth-gcp",
"litellm-auth-types",
+ "litellm-secret-value",
]
[[package]]
@@ -3416,6 +3417,7 @@ dependencies = [
"azure_identity",
"litellm-auth-types",
"litellm-error",
+ "litellm-secret-value",
"moka",
"rstest",
"serde_json",
@@ -3435,6 +3437,7 @@ dependencies = [
"http 1.4.2",
"litellm-auth-types",
"litellm-error",
+ "litellm-secret-value",
"moka",
"rstest",
"serde",
@@ -3450,10 +3453,10 @@ name = "litellm-auth-types"
version = "0.1.0"
dependencies = [
"litellm-error",
+ "litellm-secret-value",
"rstest",
"serde",
"serde_json",
- "subtle",
"thiserror 2.0.19",
"tokio",
"veil",
@@ -3523,6 +3526,7 @@ dependencies = [
"litellm-cache-testing",
"litellm-error",
"litellm-http",
+ "litellm-secret-value",
"percent-encoding",
"reqwest",
"rstest",
@@ -3668,10 +3672,10 @@ dependencies = [
name = "litellm-callbacks-legacy-python"
version = "0.1.0"
dependencies = [
- "litellm-auth",
"litellm-error",
"litellm-host",
"litellm-host-python",
+ "litellm-secret-value",
"proptest",
"pyo3",
"rstest",
@@ -3692,9 +3696,9 @@ dependencies = [
name = "litellm-config"
version = "0.1.0"
dependencies = [
- "litellm-auth-types",
"litellm-error",
"litellm-router-types",
+ "litellm-secret-value",
"rstest",
"serde",
"serde_json",
@@ -3804,7 +3808,6 @@ dependencies = [
"envy",
"futures-util",
"http-body-util",
- "litellm-auth-types",
"litellm-circuit-breaker",
"litellm-config",
"litellm-error",
@@ -3817,6 +3820,7 @@ dependencies = [
"litellm-llms",
"litellm-redis-client",
"litellm-redis-testing",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-tracing",
"rstest",
@@ -3839,9 +3843,9 @@ dependencies = [
"axum",
"axum-login",
"futures-util",
- "litellm-auth-types",
"litellm-config",
"litellm-error",
+ "litellm-secret-value",
"litellm-secrets",
"rstest",
"serde",
@@ -3879,6 +3883,7 @@ dependencies = [
"litellm-llms",
"litellm-llms-types",
"litellm-router",
+ "litellm-secret-value",
"litellm-secrets",
"rstest",
"serde",
@@ -3895,9 +3900,9 @@ name = "litellm-gateway-management"
version = "0.1.0"
dependencies = [
"base64 0.22.1",
- "litellm-auth-types",
"litellm-error",
"litellm-gateway-auth",
+ "litellm-secret-value",
"rand 0.8.7",
"rstest",
"thiserror 2.0.19",
@@ -3912,9 +3917,9 @@ dependencies = [
"base64 0.22.1",
"futures-util",
"http 1.4.2",
- "litellm-auth-types",
"litellm-config",
"litellm-error",
+ "litellm-secret-value",
"litellm-secrets",
"moka",
"rmcp",
@@ -3942,9 +3947,9 @@ dependencies = [
"futures-util",
"governor",
"jsonwebtoken",
- "litellm-auth-types",
"litellm-error",
"litellm-gateway-auth",
+ "litellm-secret-value",
"rand 0.8.7",
"rstest",
"serde",
@@ -3964,8 +3969,8 @@ name = "litellm-host"
version = "0.1.0"
dependencies = [
"futures-util",
- "litellm-auth",
"litellm-coroutine",
+ "litellm-secret-value",
"rstest",
"serde_json",
"thiserror 2.0.19",
@@ -4094,6 +4099,7 @@ dependencies = [
"litellm-inference-testing",
"litellm-llms",
"litellm-llms-types",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-tracing",
"rstest",
@@ -4124,6 +4130,7 @@ dependencies = [
"litellm-llms",
"litellm-llms-types",
"litellm-router-types",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-tracing",
"reqwest",
@@ -4154,6 +4161,7 @@ dependencies = [
"litellm-inference-testing",
"litellm-llms",
"litellm-llms-types",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-tracing",
"mime_guess",
@@ -4188,6 +4196,7 @@ dependencies = [
"litellm-inference-testing",
"litellm-llms",
"litellm-llms-types",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-tracing",
"reqwest",
@@ -4207,6 +4216,7 @@ dependencies = [
"litellm-auth",
"litellm-http",
"litellm-inference",
+ "litellm-secret-value",
"litellm-secrets",
"tokio",
]
@@ -4254,6 +4264,7 @@ dependencies = [
"litellm-llms-types",
"litellm-python-compat",
"litellm-router-types",
+ "litellm-secret-value",
"litellm-secrets",
"reqwest",
"rstest",
@@ -4347,6 +4358,7 @@ dependencies = [
"litellm-llms-types",
"litellm-model-catalog",
"litellm-router-types",
+ "litellm-secret-value",
"litellm-secrets",
"litellm-secrets-aws",
"litellm-secrets-types",
@@ -4396,6 +4408,7 @@ dependencies = [
"futures-util",
"litellm-circuit-breaker",
"litellm-redis-testing",
+ "litellm-secret-value",
"redis",
"rstest",
"thiserror 2.0.19",
@@ -4432,12 +4445,23 @@ name = "litellm-router-types"
version = "0.1.0"
dependencies = [
"litellm-auth-types",
+ "litellm-secret-value",
"rstest",
"serde",
"serde_json",
"serde_with",
]
+[[package]]
+name = "litellm-secret-value"
+version = "0.1.0"
+dependencies = [
+ "rstest",
+ "serde",
+ "subtle",
+ "veil",
+]
+
[[package]]
name = "litellm-secrets"
version = "0.1.0"
@@ -4451,6 +4475,7 @@ dependencies = [
"litellm-error",
"litellm-http",
"litellm-python-compat",
+ "litellm-secret-value",
"litellm-secrets-aws",
"litellm-secrets-azure",
"litellm-secrets-cyberark",
@@ -4479,6 +4504,7 @@ dependencies = [
"base64 0.22.1",
"litellm-auth-aws",
"litellm-core-utils",
+ "litellm-secret-value",
"litellm-secrets-types",
"litellm-tracing",
"rstest",
@@ -4498,6 +4524,7 @@ dependencies = [
"litellm-auth-types",
"litellm-core-utils",
"litellm-http",
+ "litellm-secret-value",
"litellm-secrets-types",
"percent-encoding",
"reqwest",
@@ -4519,6 +4546,7 @@ dependencies = [
"base64 0.22.1",
"litellm-core-utils",
"litellm-http",
+ "litellm-secret-value",
"litellm-secrets-types",
"litellm-tracing",
"moka",
@@ -4549,6 +4577,7 @@ dependencies = [
"litellm-auth-types",
"litellm-core-utils",
"litellm-http",
+ "litellm-secret-value",
"litellm-secrets-types",
"moka",
"percent-encoding",
@@ -4570,6 +4599,7 @@ dependencies = [
"litellm-core-utils",
"litellm-error",
"litellm-http",
+ "litellm-secret-value",
"litellm-secrets-types",
"reqwest",
"rstest",
@@ -4589,7 +4619,7 @@ dependencies = [
name = "litellm-secrets-types"
version = "0.1.0"
dependencies = [
- "litellm-auth-types",
+ "litellm-secret-value",
"moka",
"rstest",
"serde",
diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml
index 49a33e08456..75cac2a5d63 100644
--- a/litellm-rust/Cargo.toml
+++ b/litellm-rust/Cargo.toml
@@ -37,6 +37,7 @@ litellm-callbacks-legacy-python = { path = "crates/callbacks-legacy-python" }
litellm-framer = { path = "crates/framer" }
litellm-auth = { path = "crates/auth" }
litellm-auth-types = { path = "crates/auth-types" }
+litellm-secret-value = { path = "crates/secret-value" }
litellm-auth-aws = { path = "crates/auth-aws" }
litellm-auth-azure = { path = "crates/auth-azure" }
litellm-auth-gcp = { path = "crates/auth-gcp" }
diff --git a/litellm-rust/crates/auth-azure/Cargo.toml b/litellm-rust/crates/auth-azure/Cargo.toml
index f99972f813b..7dcb0e513a7 100644
--- a/litellm-rust/crates/auth-azure/Cargo.toml
+++ b/litellm-rust/crates/auth-azure/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth-types.workspace = true
moka.workspace = true
diff --git a/litellm-rust/crates/auth-azure/src/native.rs b/litellm-rust/crates/auth-azure/src/native.rs
index 9fda67746d1..f40745ab7a2 100644
--- a/litellm-rust/crates/auth-azure/src/native.rs
+++ b/litellm-rust/crates/auth-azure/src/native.rs
@@ -13,7 +13,8 @@ use azure_identity::{
use sha2::{Digest, Sha256};
use litellm_auth_types::{Error, ErrorDetail};
-use litellm_auth_types::{InputSource, ResolvedCredential, SecretValue, Sourced};
+use litellm_auth_types::{InputSource, ResolvedCredential, Sourced};
+use litellm_secret_value::SecretValue;
use super::credential_provider_cache::{
AzureCredentialProviderCache, AzureCredentialProviderCacheKey,
@@ -498,7 +499,8 @@ mod tests {
use azure_core::{Bytes, Result};
use super::{NativeAzureRequest, NativeAzureTokenAcquirer, ValidatedAzureRequest};
- use litellm_auth_types::{InputSource, SecretValue, Sourced};
+ use litellm_auth_types::{InputSource, Sourced};
+ use litellm_secret_value::SecretValue;
fn deployment(value: T) -> Sourced {
Sourced::new(value, InputSource::Deployment)
diff --git a/litellm-rust/crates/auth-azure/src/resolve.rs b/litellm-rust/crates/auth-azure/src/resolve.rs
index 25b71741342..a92630af26a 100644
--- a/litellm-rust/crates/auth-azure/src/resolve.rs
+++ b/litellm-rust/crates/auth-azure/src/resolve.rs
@@ -1,7 +1,9 @@
use litellm_auth_types::{
- CredentialFileRef, CredentialLookup, CredentialRef, InputSource, ResolvedCredential,
- SecretValue, Sourced, TokenProviderHandle,
+ CredentialFileRef, CredentialLookup, CredentialRef, InputSource, ResolvedCredential, Sourced,
+ TokenProviderHandle,
};
+use litellm_secret_value::SecretValue;
+
use litellm_auth_types::{Error, ErrorDetail};
use std::future::Future;
use std::pin::Pin;
@@ -500,8 +502,9 @@ mod tests {
use litellm_auth_types::ResolvedCredential;
use litellm_auth_types::{
CredentialFileRef, CredentialLookup, CredentialLookupFuture, CredentialRef,
- CredentialResolver, CredentialResolverHandle, InputSource, SecretValue, Sourced,
+ CredentialResolver, CredentialResolverHandle, InputSource, Sourced,
};
+ use litellm_secret_value::SecretValue;
#[derive(Debug)]
struct FileResolver;
diff --git a/litellm-rust/crates/auth-azure/src/types.rs b/litellm-rust/crates/auth-azure/src/types.rs
index a042937a047..6f8290645cf 100644
--- a/litellm-rust/crates/auth-azure/src/types.rs
+++ b/litellm-rust/crates/auth-azure/src/types.rs
@@ -1,8 +1,10 @@
use std::collections::BTreeMap;
use litellm_auth_types::{
- CredentialResolverHandle, Error, InputSource, SecretValue, Sourced, TokenProviderHandle,
+ CredentialResolverHandle, Error, InputSource, Sourced, TokenProviderHandle,
};
+use litellm_secret_value::SecretValue;
+
use serde_json::{Map, Value};
use strum::EnumString;
diff --git a/litellm-rust/crates/auth-gcp/Cargo.toml b/litellm-rust/crates/auth-gcp/Cargo.toml
index 7844b35ecc2..ad60f0db817 100644
--- a/litellm-rust/crates/auth-gcp/Cargo.toml
+++ b/litellm-rust/crates/auth-gcp/Cargo.toml
@@ -9,6 +9,7 @@ repository.workspace = true
google-sdk = ["dep:google-cloud-auth", "dep:http"]
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth-types.workspace = true
base64.workspace = true
diff --git a/litellm-rust/crates/auth-gcp/src/auth.rs b/litellm-rust/crates/auth-gcp/src/auth.rs
index 8ac1e0f54ba..650d5831796 100644
--- a/litellm-rust/crates/auth-gcp/src/auth.rs
+++ b/litellm-rust/crates/auth-gcp/src/auth.rs
@@ -258,7 +258,7 @@ fn auth_acquisition_error(error: gcp_auth::Error) -> Error {
#[cfg(test)]
mod tests {
- use litellm_auth_types::SecretValue;
+ use litellm_secret_value::SecretValue;
use super::*;
diff --git a/litellm-rust/crates/auth-gcp/src/config.rs b/litellm-rust/crates/auth-gcp/src/config.rs
index 5444897aaaf..133e1c26903 100644
--- a/litellm-rust/crates/auth-gcp/src/config.rs
+++ b/litellm-rust/crates/auth-gcp/src/config.rs
@@ -4,7 +4,9 @@ use base64::{
Engine, alphabet,
engine::{DecodePaddingMode, GeneralPurpose, GeneralPurposeConfig},
};
-use litellm_auth_types::{Error, ErrorDetail, InputSource, SecretValue, Sourced};
+use litellm_auth_types::{Error, ErrorDetail, InputSource, Sourced};
+use litellm_secret_value::SecretValue;
+
use serde_json::{Map, Value};
use litellm_auth_types::VertexParams;
diff --git a/litellm-rust/crates/auth-types/Cargo.toml b/litellm-rust/crates/auth-types/Cargo.toml
index 400f2640221..f607f0d1da0 100644
--- a/litellm-rust/crates/auth-types/Cargo.toml
+++ b/litellm-rust/crates/auth-types/Cargo.toml
@@ -7,9 +7,9 @@ repository.workspace = true
[dependencies]
litellm-error.workspace = true
+litellm-secret-value.workspace = true
serde.workspace = true
serde_json.workspace = true
-subtle.workspace = true
thiserror.workspace = true
veil.workspace = true
diff --git a/litellm-rust/crates/auth-types/src/credential.rs b/litellm-rust/crates/auth-types/src/credential.rs
index 4f69c6fffc8..36e21f7794a 100644
--- a/litellm-rust/crates/auth-types/src/credential.rs
+++ b/litellm-rust/crates/auth-types/src/credential.rs
@@ -5,7 +5,8 @@ use std::sync::Arc;
use veil::Redact;
-use crate::{Error, ResolvedCredential, SecretValue, TokenProviderHandle};
+use crate::{Error, ResolvedCredential, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum CredentialFileRef {
@@ -104,8 +105,8 @@ mod tests {
CredentialLookup, CredentialLookupFuture, CredentialPlan, CredentialPlanResolution,
CredentialRef, CredentialResolver, CredentialResolverHandle,
};
- use crate::SecretValue;
use crate::{Error, ErrorDetail};
+ use litellm_secret_value::SecretValue;
#[derive(Debug)]
struct HostResolver;
diff --git a/litellm-rust/crates/auth-types/src/lib.rs b/litellm-rust/crates/auth-types/src/lib.rs
index 99651977867..ab3bd3febbf 100644
--- a/litellm-rust/crates/auth-types/src/lib.rs
+++ b/litellm-rust/crates/auth-types/src/lib.rs
@@ -5,7 +5,6 @@ mod error;
pub mod http;
mod params;
mod policy;
-mod secret;
mod token;
use serde::{Deserialize, Serialize};
@@ -55,5 +54,4 @@ pub use error::{Error, ErrorDetail};
pub use http::CredentialPlacement;
pub use params::{AwsParams, ParamKind, ParamSpec, VertexParams};
pub use policy::{CredentialPlanKind, CredentialRule, ExistingHeaderBehavior, ProviderAuthPolicy};
-pub use secret::SecretValue;
pub use token::{ResolvedCredential, TokenFuture, TokenProvider, TokenProviderHandle};
diff --git a/litellm-rust/crates/auth-types/src/policy.rs b/litellm-rust/crates/auth-types/src/policy.rs
index 8fff59e284a..2232350965a 100644
--- a/litellm-rust/crates/auth-types/src/policy.rs
+++ b/litellm-rust/crates/auth-types/src/policy.rs
@@ -71,7 +71,8 @@ impl ProviderAuthPolicy {
#[cfg(test)]
mod tests {
use super::{CredentialPlanKind, CredentialRule, ExistingHeaderBehavior, ProviderAuthPolicy};
- use crate::{CredentialPlacement, ResolvedCredential, SecretValue};
+ use crate::{CredentialPlacement, ResolvedCredential};
+ use litellm_secret_value::SecretValue;
const RULES: &[CredentialRule] = &[CredentialRule {
kind: CredentialPlanKind::Static,
diff --git a/litellm-rust/crates/auth-types/src/token.rs b/litellm-rust/crates/auth-types/src/token.rs
index a3126052065..b948dda44fc 100644
--- a/litellm-rust/crates/auth-types/src/token.rs
+++ b/litellm-rust/crates/auth-types/src/token.rs
@@ -5,7 +5,8 @@ use std::time::SystemTime;
use veil::Redact;
-use crate::{Error, SecretValue};
+use crate::Error;
+use litellm_secret_value::SecretValue;
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ResolvedCredential {
diff --git a/litellm-rust/crates/auth-types/tests/token.rs b/litellm-rust/crates/auth-types/tests/token.rs
index 7f7027942b0..39908ae7b8a 100644
--- a/litellm-rust/crates/auth-types/tests/token.rs
+++ b/litellm-rust/crates/auth-types/tests/token.rs
@@ -8,9 +8,9 @@ use std::{
time::{Duration, SystemTime},
};
-use litellm_auth_types::{
- Error, ErrorDetail, ResolvedCredential, SecretValue, TokenProviderHandle,
-};
+use litellm_auth_types::{Error, ErrorDetail, ResolvedCredential, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
+
use rstest::rstest;
fn credential(index: usize, access_token: bool) -> ResolvedCredential {
diff --git a/litellm-rust/crates/auth/Cargo.toml b/litellm-rust/crates/auth/Cargo.toml
index ee4900ebcc2..674ae65a59f 100644
--- a/litellm-rust/crates/auth/Cargo.toml
+++ b/litellm-rust/crates/auth/Cargo.toml
@@ -16,3 +16,6 @@ litellm-auth-types.workspace = true
litellm-auth-aws = { workspace = true, optional = true }
litellm-auth-azure = { workspace = true, optional = true }
litellm-auth-gcp = { workspace = true, optional = true }
+
+[dev-dependencies]
+litellm-secret-value.workspace = true
diff --git a/litellm-rust/crates/auth/tests/facade.rs b/litellm-rust/crates/auth/tests/facade.rs
index f1092b15def..5d29bac1072 100644
--- a/litellm-rust/crates/auth/tests/facade.rs
+++ b/litellm-rust/crates/auth/tests/facade.rs
@@ -1,7 +1,8 @@
use litellm_auth::{
CredentialPlacement, CredentialPlanKind, CredentialRule, ExistingHeaderBehavior,
- ProviderAuthPolicy, ResolvedCredential, SecretValue,
+ ProviderAuthPolicy, ResolvedCredential,
};
+use litellm_secret_value::SecretValue;
const RULES: &[CredentialRule] = &[CredentialRule {
kind: CredentialPlanKind::Static,
diff --git a/litellm-rust/crates/cache-gcs/Cargo.toml b/litellm-rust/crates/cache-gcs/Cargo.toml
index dbd8c9ac908..878db4b8010 100644
--- a/litellm-rust/crates/cache-gcs/Cargo.toml
+++ b/litellm-rust/crates/cache-gcs/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-http.workspace = true
futures-util.workspace = true
litellm-auth-gcp.workspace = true
diff --git a/litellm-rust/crates/cache-gcs/src/token.rs b/litellm-rust/crates/cache-gcs/src/token.rs
index 079ec44dbce..852d8bbf035 100644
--- a/litellm-rust/crates/cache-gcs/src/token.rs
+++ b/litellm-rust/crates/cache-gcs/src/token.rs
@@ -1,7 +1,9 @@
use std::{future::Future, pin::Pin};
use litellm_auth_gcp::{VertexAuth, VertexConfig};
-use litellm_auth_types::{InputSource, SecretValue, Sourced};
+use litellm_auth_types::{InputSource, Sourced};
+use litellm_secret_value::SecretValue;
+
use litellm_cache::Error;
pub trait TokenSource: Send + Sync + 'static {
diff --git a/litellm-rust/crates/callbacks-legacy-python/Cargo.toml b/litellm-rust/crates/callbacks-legacy-python/Cargo.toml
index 1cd05710366..0133af16b80 100644
--- a/litellm-rust/crates/callbacks-legacy-python/Cargo.toml
+++ b/litellm-rust/crates/callbacks-legacy-python/Cargo.toml
@@ -16,7 +16,7 @@ serde_json.workspace = true
thiserror.workspace = true
[dev-dependencies]
-litellm-auth.workspace = true
+litellm-secret-value.workspace = true
proptest.workspace = true
rstest.workspace = true
serde_json.workspace = true
diff --git a/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs b/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs
index 493e91bb5ea..9431b9d1326 100644
--- a/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs
+++ b/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs
@@ -925,11 +925,11 @@ assert all(value is failure for name, value in logger.calls if name.endswith('_h
mod payload_tests {
use std::ffi::CStr;
- use litellm_auth::SecretValue;
use litellm_host::hooks::CallHooks;
use litellm_host::interceptors::{RawResponse, RequestContext, WireRequest};
use litellm_host::lifecycle::ExecutionEvent;
use litellm_host_python::{HookStep, PythonCallEvent, PythonOwned, to_py};
+ use litellm_secret_value::SecretValue;
use proptest::prelude::*;
use pyo3::gc::{PyTraverseError, PyVisit};
use pyo3::prelude::*;
diff --git a/litellm-rust/crates/config/Cargo.toml b/litellm-rust/crates/config/Cargo.toml
index 5b0cd22332f..278aea6200e 100644
--- a/litellm-rust/crates/config/Cargo.toml
+++ b/litellm-rust/crates/config/Cargo.toml
@@ -6,7 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
-litellm-auth-types.workspace = true
+litellm-secret-value.workspace = true
litellm-router-types.workspace = true
serde.workspace = true
serde_json.workspace = true
diff --git a/litellm-rust/crates/config/src/mcp.rs b/litellm-rust/crates/config/src/mcp.rs
index cbe992f9956..00480fa5be1 100644
--- a/litellm-rust/crates/config/src/mcp.rs
+++ b/litellm-rust/crates/config/src/mcp.rs
@@ -1,6 +1,6 @@
use std::{collections::BTreeMap, fmt};
-use litellm_auth_types::SecretValue;
+use litellm_secret_value::SecretValue;
use serde::Deserialize;
use crate::Object;
diff --git a/litellm-rust/crates/config/src/redis.rs b/litellm-rust/crates/config/src/redis.rs
index 5674f993025..7a91691c54b 100644
--- a/litellm-rust/crates/config/src/redis.rs
+++ b/litellm-rust/crates/config/src/redis.rs
@@ -1,7 +1,7 @@
use std::{collections::BTreeMap, fmt, time::Duration};
-use litellm_auth_types::SecretValue;
use litellm_router_types::Spelled;
+use litellm_secret_value::SecretValue;
use serde::{Deserialize, Deserializer, de::Error as _};
use crate::{AdditionalFields, Error, Object, Value};
diff --git a/litellm-rust/crates/config/src/settings.rs b/litellm-rust/crates/config/src/settings.rs
index 9571c4fa5de..1bfc19983f3 100644
--- a/litellm-rust/crates/config/src/settings.rs
+++ b/litellm-rust/crates/config/src/settings.rs
@@ -1,6 +1,6 @@
use std::fmt;
-use litellm_auth_types::SecretValue;
+use litellm_secret_value::SecretValue;
use serde::Deserialize;
use crate::{
diff --git a/litellm-rust/crates/gateway-auth/Cargo.toml b/litellm-rust/crates/gateway-auth/Cargo.toml
index 27f20203843..ba9097e85d8 100644
--- a/litellm-rust/crates/gateway-auth/Cargo.toml
+++ b/litellm-rust/crates/gateway-auth/Cargo.toml
@@ -6,9 +6,9 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
axum = { workspace = true, features = ["matched-path"] }
axum-login.workspace = true
-litellm-auth-types.workspace = true
litellm-config.workspace = true
litellm-error.workspace = true
litellm-secrets.workspace = true
diff --git a/litellm-rust/crates/gateway-auth/src/authentication.rs b/litellm-rust/crates/gateway-auth/src/authentication.rs
index 4ecfaa375b4..349913619f5 100644
--- a/litellm-rust/crates/gateway-auth/src/authentication.rs
+++ b/litellm-rust/crates/gateway-auth/src/authentication.rs
@@ -1,8 +1,8 @@
use std::{future::Future, pin::Pin, sync::Arc, time::SystemTime};
use axum::http::{Request, request::Parts};
-use litellm_auth_types::SecretValue;
use litellm_config::Config;
+use litellm_secret_value::SecretValue;
use litellm_secrets::source::SecretSource;
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq;
diff --git a/litellm-rust/crates/gateway-auth/src/http.rs b/litellm-rust/crates/gateway-auth/src/http.rs
index ddbf240509a..ad1718161c0 100644
--- a/litellm-rust/crates/gateway-auth/src/http.rs
+++ b/litellm-rust/crates/gateway-auth/src/http.rs
@@ -4,7 +4,7 @@ use axum::{
middleware::Next,
response::Response,
};
-use litellm_auth_types::SecretValue;
+use litellm_secret_value::SecretValue;
use crate::{AccessRequest, Auth, AuthenticatedRequest, Credential, Error};
diff --git a/litellm-rust/crates/gateway-auth/src/keys.rs b/litellm-rust/crates/gateway-auth/src/keys.rs
index 1fe8fa60e0e..0deaf3f7a82 100644
--- a/litellm-rust/crates/gateway-auth/src/keys.rs
+++ b/litellm-rust/crates/gateway-auth/src/keys.rs
@@ -1,6 +1,6 @@
use std::{future::Future, pin::Pin, time::SystemTime};
-use litellm_auth_types::SecretValue;
+use litellm_secret_value::SecretValue;
use crate::{Error, hash_token};
diff --git a/litellm-rust/crates/gateway-auth/src/ui.rs b/litellm-rust/crates/gateway-auth/src/ui.rs
index 3d655bde187..ee4dd670aa7 100644
--- a/litellm-rust/crates/gateway-auth/src/ui.rs
+++ b/litellm-rust/crates/gateway-auth/src/ui.rs
@@ -3,7 +3,7 @@ use std::{convert::Infallible, sync::Arc};
use axum::extract::FromRequestParts;
use axum::http::{header::AUTHORIZATION, request::Parts};
use axum_login::{AuthUser, AuthnBackend, UserId};
-use litellm_auth_types::SecretValue;
+use litellm_secret_value::SecretValue;
use serde::Deserialize;
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq;
diff --git a/litellm-rust/crates/gateway-auth/tests/auth.rs b/litellm-rust/crates/gateway-auth/tests/auth.rs
index 781ba385381..54180e30504 100644
--- a/litellm-rust/crates/gateway-auth/tests/auth.rs
+++ b/litellm-rust/crates/gateway-auth/tests/auth.rs
@@ -8,9 +8,9 @@ use axum::{
routing::get,
};
use futures_util::future::BoxFuture;
-use litellm_auth_types::SecretValue;
use litellm_config::Config;
use litellm_gateway_auth::{Auth, RequireMasterKey, hash_token};
+use litellm_secret_value::SecretValue;
use litellm_secrets::source::SecretSource;
use rstest::{fixture, rstest};
use tower::ServiceExt;
diff --git a/litellm-rust/crates/gateway-auth/tests/keys.rs b/litellm-rust/crates/gateway-auth/tests/keys.rs
index 4d16d3573bc..1f6fc461a5b 100644
--- a/litellm-rust/crates/gateway-auth/tests/keys.rs
+++ b/litellm-rust/crates/gateway-auth/tests/keys.rs
@@ -1,10 +1,10 @@
use std::time::{Duration, SystemTime};
-use litellm_auth_types::SecretValue;
use litellm_gateway_auth::{
Error,
keys::{KeyHash, KeyLookup, KeyStatus, LookupFuture, verify_key},
};
+use litellm_secret_value::SecretValue;
use rstest::{fixture, rstest};
struct Lookup {
diff --git a/litellm-rust/crates/gateway-auth/tests/ui.rs b/litellm-rust/crates/gateway-auth/tests/ui.rs
index 2e5f72d605d..8d465e1198d 100644
--- a/litellm-rust/crates/gateway-auth/tests/ui.rs
+++ b/litellm-rust/crates/gateway-auth/tests/ui.rs
@@ -1,6 +1,6 @@
use axum_login::{AuthUser, AuthnBackend};
-use litellm_auth_types::SecretValue;
use litellm_gateway_auth::{UiBackend, UiCredentials};
+use litellm_secret_value::SecretValue;
use rstest::{fixture, rstest};
#[fixture]
diff --git a/litellm-rust/crates/gateway-inference/Cargo.toml b/litellm-rust/crates/gateway-inference/Cargo.toml
index 034c3a9826e..69ca561e4b5 100644
--- a/litellm-rust/crates/gateway-inference/Cargo.toml
+++ b/litellm-rust/crates/gateway-inference/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-cache-response.workspace = true
axum = { workspace = true, features = ["json", "multipart", "original-uri"] }
base64.workspace = true
diff --git a/litellm-rust/crates/gateway-inference/src/ocr.rs b/litellm-rust/crates/gateway-inference/src/ocr.rs
index f200fd7b141..b8e6ef4ff4f 100644
--- a/litellm-rust/crates/gateway-inference/src/ocr.rs
+++ b/litellm-rust/crates/gateway-inference/src/ocr.rs
@@ -2,10 +2,10 @@ use litellm_gateway_auth::AuthenticatedRequest;
use std::sync::Arc;
use axum::{Json, extract::State, http::HeaderMap, response::IntoResponse};
-use litellm_auth::SecretValue;
use litellm_inference_ocr::types::{LiteLLMOcrRequest, OcrConnectionInputs, OcrDocumentInput};
use litellm_llms::base_llm::ocr::transformation::decode_request_value;
use litellm_llms_types::formats::ocr::OcrDocument;
+use litellm_secret_value::SecretValue;
use serde_json::Value;
use crate::{
diff --git a/litellm-rust/crates/gateway-inference/tests/support/mod.rs b/litellm-rust/crates/gateway-inference/tests/support/mod.rs
index 737dc0c830c..185cbc8594d 100644
--- a/litellm-rust/crates/gateway-inference/tests/support/mod.rs
+++ b/litellm-rust/crates/gateway-inference/tests/support/mod.rs
@@ -13,7 +13,9 @@ use futures_util::future::BoxFuture;
use litellm_gateway_inference::{Deployment, Gateway, router};
use litellm_http::{HttpClientPool, HttpSettings, Resolution, media::PublicDnsResolver};
use litellm_inference::resources::CoreResources;
-use litellm_secrets::{SecretValue, source::SecretSource};
+use litellm_secret_value::SecretValue;
+use litellm_secrets::source::SecretSource;
+
use serde_json::Value;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tower::ServiceExt;
diff --git a/litellm-rust/crates/gateway-management/Cargo.toml b/litellm-rust/crates/gateway-management/Cargo.toml
index 502760f8fbe..1ff5dd5319d 100644
--- a/litellm-rust/crates/gateway-management/Cargo.toml
+++ b/litellm-rust/crates/gateway-management/Cargo.toml
@@ -6,8 +6,8 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
base64.workspace = true
-litellm-auth-types.workspace = true
litellm-error.workspace = true
litellm-gateway-auth.workspace = true
rand.workspace = true
diff --git a/litellm-rust/crates/gateway-management/src/keys.rs b/litellm-rust/crates/gateway-management/src/keys.rs
index 45d915b7b3d..feeb40b2403 100644
--- a/litellm-rust/crates/gateway-management/src/keys.rs
+++ b/litellm-rust/crates/gateway-management/src/keys.rs
@@ -1,8 +1,8 @@
use std::{future::Future, pin::Pin, sync::Arc, time::SystemTime};
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
-use litellm_auth_types::SecretValue;
use litellm_gateway_auth::keys::{KeyHash, KeyStatus};
+use litellm_secret_value::SecretValue;
use rand::{RngCore, rngs::OsRng};
use crate::Error;
diff --git a/litellm-rust/crates/gateway-management/tests/keys.rs b/litellm-rust/crates/gateway-management/tests/keys.rs
index bf0221cf1be..8a970bb407c 100644
--- a/litellm-rust/crates/gateway-management/tests/keys.rs
+++ b/litellm-rust/crates/gateway-management/tests/keys.rs
@@ -7,7 +7,6 @@ use std::{
};
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
-use litellm_auth_types::SecretValue;
use litellm_gateway_auth::{
hash_token,
keys::{KeyHash, KeyLookup, KeyStatus, verify_key},
@@ -16,6 +15,7 @@ use litellm_gateway_management::{
Error,
keys::{Keys, Revocation},
};
+use litellm_secret_value::SecretValue;
use rstest::{fixture, rstest};
use support::{MemoryStore, Unavailable};
diff --git a/litellm-rust/crates/gateway-mcp/Cargo.toml b/litellm-rust/crates/gateway-mcp/Cargo.toml
index bed48bad58d..65077eec0ea 100644
--- a/litellm-rust/crates/gateway-mcp/Cargo.toml
+++ b/litellm-rust/crates/gateway-mcp/Cargo.toml
@@ -7,11 +7,11 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
axum = { workspace = true, features = ["json", "query"] }
futures-util.workspace = true
litellm-config.workspace = true
litellm-secrets.workspace = true
-litellm-auth-types.workspace = true
litellm-error.workspace = true
base64.workspace = true
http.workspace = true
diff --git a/litellm-rust/crates/gateway-mcp/src/configured.rs b/litellm-rust/crates/gateway-mcp/src/configured.rs
index cc7a52fc029..33cbd243530 100644
--- a/litellm-rust/crates/gateway-mcp/src/configured.rs
+++ b/litellm-rust/crates/gateway-mcp/src/configured.rs
@@ -8,8 +8,8 @@ use std::{
use base64::{Engine, engine::general_purpose::STANDARD};
use futures_util::future::try_join_all;
use http::{HeaderName, HeaderValue};
-use litellm_auth_types::SecretValue;
use litellm_config::{McpAuth, McpServer, McpTransport};
+use litellm_secret_value::SecretValue;
use litellm_secrets::source::SecretSource;
use rmcp::{
RoleClient, ServiceExt,
diff --git a/litellm-rust/crates/gateway-ui/Cargo.toml b/litellm-rust/crates/gateway-ui/Cargo.toml
index 1c65665be72..a6104455e96 100644
--- a/litellm-rust/crates/gateway-ui/Cargo.toml
+++ b/litellm-rust/crates/gateway-ui/Cargo.toml
@@ -23,10 +23,10 @@ tower-http = { version = "0.6.11", features = ["fs", "set-header"] }
tower-sessions.workspace = true
[dev-dependencies]
+litellm-secret-value.workspace = true
futures-util.workspace = true
serde_json.workspace = true
tower = { version = "0.5", features = ["util"] }
-litellm-auth-types.workspace = true
rstest.workspace = true
tempfile.workspace = true
tokio.workspace = true
diff --git a/litellm-rust/crates/gateway-ui/tests/routes.rs b/litellm-rust/crates/gateway-ui/tests/routes.rs
index 95246f33a35..86b7db63ed9 100644
--- a/litellm-rust/crates/gateway-ui/tests/routes.rs
+++ b/litellm-rust/crates/gateway-ui/tests/routes.rs
@@ -16,8 +16,8 @@ use tower_sessions::{
session::{Id, Record},
};
-use litellm_auth_types::SecretValue;
use litellm_gateway_auth::UiBackend;
+use litellm_secret_value::SecretValue;
struct App {
router: Router,
diff --git a/litellm-rust/crates/gateway/Cargo.toml b/litellm-rust/crates/gateway/Cargo.toml
index 4d831aa2314..6eeaa43e013 100644
--- a/litellm-rust/crates/gateway/Cargo.toml
+++ b/litellm-rust/crates/gateway/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
axum.workspace = true
envy = "0.4.2"
http-body-util = "0.1"
@@ -17,7 +18,6 @@ thiserror.workspace = true
futures-util.workspace = true
litellm-gateway-auth.workspace = true
litellm-gateway-ui.workspace = true
-litellm-auth-types.workspace = true
litellm-config.workspace = true
litellm-error.workspace = true
litellm-circuit-breaker.workspace = true
diff --git a/litellm-rust/crates/gateway/src/redis.rs b/litellm-rust/crates/gateway/src/redis.rs
index 197f40e7988..c40b289029b 100644
--- a/litellm-rust/crates/gateway/src/redis.rs
+++ b/litellm-rust/crates/gateway/src/redis.rs
@@ -152,12 +152,12 @@ async fn reachable(client: RedisClient) -> Option {
fn client(params: Params, breaker: Option) -> Result {
tracing::info!(endpoint = ?params.endpoint, namespace = ?params.namespace, "redis client configured");
let endpoint = match params.endpoint {
- RedisEndpoint::Url(url) => Endpoint::Url(url.expose().to_owned()),
+ RedisEndpoint::Url(url) => Endpoint::Url(url),
RedisEndpoint::Host(host) => Endpoint::Host(HostEndpoint {
host: host.host,
port: host.port,
- username: host.username.map(|value| value.expose().to_owned()),
- password: host.password.map(|value| value.expose().to_owned()),
+ username: host.username,
+ password: host.password,
tls: host.tls,
db: host.db,
}),
@@ -170,8 +170,8 @@ fn client(params: Params, breaker: Option) -> Result futures_util::future::BoxFuture<
'a,
- Result, litellm_secrets::Error>,
+ Result, litellm_secrets::Error>,
> {
Box::pin(async { Ok(None) })
}
diff --git a/litellm-rust/crates/gateway/tests/server.rs b/litellm-rust/crates/gateway/tests/server.rs
index c876c1cbcbb..7de7a97c870 100644
--- a/litellm-rust/crates/gateway/tests/server.rs
+++ b/litellm-rust/crates/gateway/tests/server.rs
@@ -159,7 +159,7 @@ async fn mounts_ui_without_exposing_credentials_or_authorizing_inference(inferen
std::fs::write(assets.path().join("index.html"), "dashboard").unwrap();
let backend = litellm_gateway_auth::UiBackend::new(
"admin".into(),
- litellm_auth_types::SecretValue::new("ui-password"),
+ litellm_secret_value::SecretValue::new("ui-password"),
)
.unwrap();
let ui = litellm_gateway_ui::router(
diff --git a/litellm-rust/crates/host/Cargo.toml b/litellm-rust/crates/host/Cargo.toml
index d3354fa8622..e0ccab4e0ad 100644
--- a/litellm-rust/crates/host/Cargo.toml
+++ b/litellm-rust/crates/host/Cargo.toml
@@ -6,8 +6,8 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
futures-util.workspace = true
-litellm-auth.workspace = true
litellm-coroutine.workspace = true
serde_json.workspace = true
thiserror.workspace = true
diff --git a/litellm-rust/crates/host/src/interceptors.rs b/litellm-rust/crates/host/src/interceptors.rs
index 3cff781792f..39af7c410e0 100644
--- a/litellm-rust/crates/host/src/interceptors.rs
+++ b/litellm-rust/crates/host/src/interceptors.rs
@@ -22,7 +22,7 @@ pub struct RequestContext {
/// Optional-param names that carry credentials and must be redacted when logged.
pub secret_fields: Vec,
/// The credential the route resolved for the provider call.
- pub api_key: Option,
+ pub api_key: Option,
}
#[derive(Clone, Debug, PartialEq, Eq)]
diff --git a/litellm-rust/crates/inference-chat/Cargo.toml b/litellm-rust/crates/inference-chat/Cargo.toml
index 8a779d44976..bcde3c8d30f 100644
--- a/litellm-rust/crates/inference-chat/Cargo.toml
+++ b/litellm-rust/crates/inference-chat/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth = { workspace = true, features = ["aws", "azure", "gcp"] }
litellm-cache-response.workspace = true
litellm-core-utils.workspace = true
diff --git a/litellm-rust/crates/inference-chat/src/handler.rs b/litellm-rust/crates/inference-chat/src/handler.rs
index 1c724a47956..f948e8fffaf 100644
--- a/litellm-rust/crates/inference-chat/src/handler.rs
+++ b/litellm-rust/crates/inference-chat/src/handler.rs
@@ -58,7 +58,7 @@ pub(super) async fn execute(
custom_llm_provider: provider_name.to_owned(),
optional_params: Value::Object(call.optional_params),
secret_fields: Vec::new(),
- api_key: call.api_key.map(litellm_auth::SecretValue::new),
+ api_key: call.api_key.map(litellm_secret_value::SecretValue::new),
};
let authenticated = resolve_auth(auth, environment, &|key| secrets.get(key)).await?;
let identity = litellm_host::interceptors::ProviderIdentity {
diff --git a/litellm-rust/crates/inference-chat/tests/caching.rs b/litellm-rust/crates/inference-chat/tests/caching.rs
index 4b3e32f04bc..dd229d3d657 100644
--- a/litellm-rust/crates/inference-chat/tests/caching.rs
+++ b/litellm-rust/crates/inference-chat/tests/caching.rs
@@ -140,7 +140,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
name: &'a str,
) -> futures_util::future::BoxFuture<
'a,
- Result, litellm_secrets::Error>,
+ Result, litellm_secrets::Error>,
> {
Box::pin(async move {
let revision = self.revision.load(Ordering::SeqCst);
@@ -154,7 +154,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
} else {
None
};
- Ok(value.map(litellm_secrets::SecretValue::new))
+ Ok(value.map(litellm_secret_value::SecretValue::new))
})
}
}
diff --git a/litellm-rust/crates/inference-messages/Cargo.toml b/litellm-rust/crates/inference-messages/Cargo.toml
index 694d9b4de31..ac52cb65a0d 100644
--- a/litellm-rust/crates/inference-messages/Cargo.toml
+++ b/litellm-rust/crates/inference-messages/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
bytes.workspace = true
futures-util.workspace = true
litellm-auth = { workspace = true, features = ["aws", "azure", "gcp"] }
diff --git a/litellm-rust/crates/inference-messages/src/handler.rs b/litellm-rust/crates/inference-messages/src/handler.rs
index 9d0bf09197e..2e647688a1a 100644
--- a/litellm-rust/crates/inference-messages/src/handler.rs
+++ b/litellm-rust/crates/inference-messages/src/handler.rs
@@ -2,7 +2,9 @@ use std::time::Duration;
use bytes::Bytes;
use futures_util::{FutureExt, StreamExt, TryStreamExt, stream::BoxStream};
-use litellm_auth::{AuthServices, SecretValue};
+use litellm_auth::AuthServices;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::{
dot_notation_indexing::delete_nested_value,
get_provider_specific_headers::get_provider_specific_headers, settings::Lookup,
diff --git a/litellm-rust/crates/inference-messages/tests/caching.rs b/litellm-rust/crates/inference-messages/tests/caching.rs
index 480b3c4f3b7..663c87453a9 100644
--- a/litellm-rust/crates/inference-messages/tests/caching.rs
+++ b/litellm-rust/crates/inference-messages/tests/caching.rs
@@ -107,7 +107,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
name: &'a str,
) -> futures_util::future::BoxFuture<
'a,
- Result, litellm_secrets::Error>,
+ Result, litellm_secrets::Error>,
> {
Box::pin(async move {
let revision = self.revision.load(Ordering::SeqCst);
@@ -121,7 +121,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
} else {
None
};
- Ok(value.map(litellm_secrets::SecretValue::new))
+ Ok(value.map(litellm_secret_value::SecretValue::new))
})
}
}
diff --git a/litellm-rust/crates/inference-ocr/Cargo.toml b/litellm-rust/crates/inference-ocr/Cargo.toml
index bbbdaddb050..ad53126b9fe 100644
--- a/litellm-rust/crates/inference-ocr/Cargo.toml
+++ b/litellm-rust/crates/inference-ocr/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
base64.workspace = true
bytes.workspace = true
futures-util.workspace = true
diff --git a/litellm-rust/crates/inference-ocr/src/handler.rs b/litellm-rust/crates/inference-ocr/src/handler.rs
index 426d4297f28..c126a192153 100644
--- a/litellm-rust/crates/inference-ocr/src/handler.rs
+++ b/litellm-rust/crates/inference-ocr/src/handler.rs
@@ -128,7 +128,7 @@ mod tests {
decode_request(OcrWireRequest {
model: model.into(),
document,
- api_key: Some(litellm_auth::SecretValue::new("test-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("test-key")),
api_base: Some(base.into()),
custom_llm_provider: None,
extra_headers: None,
diff --git a/litellm-rust/crates/inference-ocr/src/provider_config.rs b/litellm-rust/crates/inference-ocr/src/provider_config.rs
index 19e50af0dd1..e76d291254b 100644
--- a/litellm-rust/crates/inference-ocr/src/provider_config.rs
+++ b/litellm-rust/crates/inference-ocr/src/provider_config.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{InputSource, SecretValue, Sourced};
+use litellm_auth::{InputSource, Sourced};
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::get_llm_provider_logic::{
CustomLlmProvider, LlmProviders, get_custom_llm_provider,
};
@@ -407,7 +409,7 @@ mod tests {
fn connection_resolution_preserves_dynamic_precedence_and_input_sources() {
let connection = OcrConfigKind::Mistral.resolve_connection_params(OcrCredentialInputs {
api_key: Some(Sourced::new(
- litellm_auth::SecretValue::new("explicit-key"),
+ litellm_secret_value::SecretValue::new("explicit-key"),
InputSource::Deployment,
)),
api_base: Some(Sourced::new(
@@ -415,7 +417,7 @@ mod tests {
InputSource::Deployment,
)),
dynamic_api_key: Some(Sourced::new(
- litellm_auth::SecretValue::new("dynamic-key"),
+ litellm_secret_value::SecretValue::new("dynamic-key"),
InputSource::Environment,
)),
dynamic_api_base: Some(Sourced::new(
@@ -455,7 +457,7 @@ mod tests {
) {
let dynamic_key = dynamic_value.map(|value| {
Sourced::new(
- litellm_auth::SecretValue::new(value),
+ litellm_secret_value::SecretValue::new(value),
InputSource::Environment,
)
});
@@ -463,7 +465,7 @@ mod tests {
dynamic_value.map(|value| Sourced::new(value.into(), InputSource::Environment));
let connection = OcrConfigKind::Mistral.resolve_connection_params(OcrCredentialInputs {
api_key: Some(Sourced::new(
- litellm_auth::SecretValue::new("explicit-key"),
+ litellm_secret_value::SecretValue::new("explicit-key"),
InputSource::Deployment,
)),
api_base: Some(Sourced::new(
@@ -502,14 +504,14 @@ mod tests {
OcrCredentialInputs {
api_key: explicit_key.map(|value| {
Sourced::new(
- litellm_auth::SecretValue::new(value),
+ litellm_secret_value::SecretValue::new(value),
InputSource::Deployment,
)
}),
api_base: explicit_base
.map(|value| Sourced::new(value.into(), InputSource::Deployment)),
dynamic_api_key: Some(Sourced::new(
- litellm_auth::SecretValue::new("dynamic-key"),
+ litellm_secret_value::SecretValue::new("dynamic-key"),
InputSource::Environment,
)),
dynamic_api_base: Some(Sourced::new(
diff --git a/litellm-rust/crates/inference-ocr/src/types.rs b/litellm-rust/crates/inference-ocr/src/types.rs
index 98c609a158f..1b2dbb52d77 100644
--- a/litellm-rust/crates/inference-ocr/src/types.rs
+++ b/litellm-rust/crates/inference-ocr/src/types.rs
@@ -1,7 +1,9 @@
use std::{collections::BTreeMap, path::PathBuf, time::Duration};
use bytes::Bytes;
-use litellm_auth::{InputSource, SecretValue, TokenProviderHandle};
+use litellm_auth::{InputSource, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::call_arguments::CallArguments;
use litellm_llms::base_llm::ocr::{
error::Error,
diff --git a/litellm-rust/crates/inference-ocr/src/wire.rs b/litellm-rust/crates/inference-ocr/src/wire.rs
index 56d22be6523..45a2865823f 100644
--- a/litellm-rust/crates/inference-ocr/src/wire.rs
+++ b/litellm-rust/crates/inference-ocr/src/wire.rs
@@ -1,6 +1,8 @@
use std::{collections::BTreeMap, time::Duration};
-use litellm_auth::{InputSource, SecretValue};
+use litellm_auth::InputSource;
+use litellm_secret_value::SecretValue;
+
use litellm_llms::base_llm::ocr::{error::Error, transformation::decode_request_value};
use litellm_llms_types::formats::ocr::OcrDocument;
use serde::Deserialize;
diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs b/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs
index 6f591af73e0..65c7f309b82 100644
--- a/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs
+++ b/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs
@@ -3,9 +3,9 @@ use std::sync::{
atomic::{AtomicUsize, Ordering},
};
-use litellm_auth::{
- ResolvedCredential, SecretValue, TokenFuture, TokenProvider, TokenProviderHandle,
-};
+use litellm_auth::{ResolvedCredential, TokenFuture, TokenProvider, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
+
use rstest::rstest;
use super::*;
diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs
index 75a91092daf..9f59bb9a2a2 100644
--- a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs
+++ b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs
@@ -239,7 +239,7 @@ impl litellm_host_native::services::HostCallHandler for CallerTokenHost {
OcrOp::AcquireAzureAdToken(reply) => {
self.trace.lock().unwrap().push("token".into());
reply.send(litellm_auth::ResolvedCredential::Static(
- litellm_auth::SecretValue::new("caller-token"),
+ litellm_secret_value::SecretValue::new("caller-token"),
));
Ok(())
}
diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs
index 5a48bba9e1a..6871d1ca1f6 100644
--- a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs
+++ b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs
@@ -80,7 +80,7 @@ fn wire(model: &str, base: &str, document: Value, options: Value) -> OcrWireRequ
OcrWireRequest {
model: model.into(),
document,
- api_key: Some(litellm_auth::SecretValue::new("test-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("test-key")),
api_base: Some(base.into()),
custom_llm_provider: None,
extra_headers: None,
diff --git a/litellm-rust/crates/inference-responses/Cargo.toml b/litellm-rust/crates/inference-responses/Cargo.toml
index 083179c50c8..f90651634c5 100644
--- a/litellm-rust/crates/inference-responses/Cargo.toml
+++ b/litellm-rust/crates/inference-responses/Cargo.toml
@@ -25,6 +25,7 @@ tokio-tungstenite.workspace = true
tracing.workspace = true
[dev-dependencies]
+litellm-secret-value.workspace = true
litellm-http = { workspace = true, features = ["test-support"] }
litellm-cache.workspace = true
litellm-cache-memory.workspace = true
diff --git a/litellm-rust/crates/inference-responses/tests/caching.rs b/litellm-rust/crates/inference-responses/tests/caching.rs
index 462461695c3..56e25fbcb98 100644
--- a/litellm-rust/crates/inference-responses/tests/caching.rs
+++ b/litellm-rust/crates/inference-responses/tests/caching.rs
@@ -98,7 +98,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
name: &'a str,
) -> futures_util::future::BoxFuture<
'a,
- Result, litellm_secrets::Error>,
+ Result, litellm_secrets::Error>,
> {
Box::pin(async move {
let revision = self.revision.load(Ordering::SeqCst);
@@ -112,7 +112,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets {
} else {
None
};
- Ok(value.map(litellm_secrets::SecretValue::new))
+ Ok(value.map(litellm_secret_value::SecretValue::new))
})
}
}
diff --git a/litellm-rust/crates/inference-testing/Cargo.toml b/litellm-rust/crates/inference-testing/Cargo.toml
index 748b1df0d3b..5f690532d9c 100644
--- a/litellm-rust/crates/inference-testing/Cargo.toml
+++ b/litellm-rust/crates/inference-testing/Cargo.toml
@@ -7,6 +7,7 @@ repository.workspace = true
publish = false
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth.workspace = true
futures-util.workspace = true
litellm-http.workspace = true
diff --git a/litellm-rust/crates/inference-testing/src/lib.rs b/litellm-rust/crates/inference-testing/src/lib.rs
index 2e20d5a9e91..f330e02e99a 100644
--- a/litellm-rust/crates/inference-testing/src/lib.rs
+++ b/litellm-rust/crates/inference-testing/src/lib.rs
@@ -5,7 +5,8 @@ use litellm_http::{
ClientVariant, HttpClientConfig, HttpClientPool, HttpSettings, Resolution,
media::PublicDnsResolver,
};
-use litellm_secrets::{SecretValue, source::SecretSource};
+use litellm_secret_value::SecretValue;
+use litellm_secrets::source::SecretSource;
use litellm_inference::resources::CoreResources;
diff --git a/litellm-rust/crates/llms/Cargo.toml b/litellm-rust/crates/llms/Cargo.toml
index 962f61ef4ec..79ee17e8b97 100644
--- a/litellm-rust/crates/llms/Cargo.toml
+++ b/litellm-rust/crates/llms/Cargo.toml
@@ -9,6 +9,7 @@ repository.workspace = true
test-support = ["litellm-http/test-support"]
[dependencies]
+litellm-secret-value.workspace = true
litellm-llms-types.workspace = true
litellm-core-utils.workspace = true
litellm-error.workspace = true
diff --git a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs
index eb796a900f5..164dbce1264 100644
--- a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs
+++ b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs
@@ -1,4 +1,3 @@
-use litellm_auth::SecretValue;
use litellm_core_utils::decline::Decline;
use litellm_core_utils::{
core_helpers::{finish_reason_for, unix_now, usage_from_parts},
@@ -10,6 +9,7 @@ use litellm_llms_types::{
},
providers::anthropic::DEFAULT_HEADERS,
};
+use litellm_secret_value::SecretValue;
use serde::Deserialize;
use serde_json::{Map, Value, json};
diff --git a/litellm-rust/crates/llms/src/anthropic/common_utils.rs b/litellm-rust/crates/llms/src/anthropic/common_utils.rs
index 7991a57353e..aec5d9ebc27 100644
--- a/litellm-rust/crates/llms/src/anthropic/common_utils.rs
+++ b/litellm-rust/crates/llms/src/anthropic/common_utils.rs
@@ -1,5 +1,7 @@
use crate::base_llm::messages::context::MessagesModelCapabilities;
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_http::request::{
has_header, header_value, header_values, with_header, without_headers,
diff --git a/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs
index d578acea68c..a2ba2cee0d3 100644
--- a/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs
@@ -1,10 +1,10 @@
-use litellm_auth::SecretValue;
use litellm_http::request::{has_bearer_auth, has_header};
use litellm_llms_types::{
formats::messages::{MessagesOptionalParams, MessagesRequest},
providers::anthropic::{BetaProvider, DEFAULT_HEADERS, MESSAGES_PATH},
};
use litellm_router_types::LitellmParams;
+use litellm_secret_value::SecretValue;
use crate::{
Error,
diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs
index 36dfe2c890e..870306b1a83 100644
--- a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs
+++ b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs
@@ -1250,7 +1250,7 @@ mod tests {
#[tokio::test]
async fn request_endpoint_accepts_request_owned_key() {
let connection = OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new("request-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("request-key")),
api_key_source: InputSource::Request,
api_base: Some("https://request.example".into()),
api_base_source: InputSource::Request,
diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs
index 2f7957fe56e..25f93846967 100644
--- a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs
@@ -211,7 +211,7 @@ mod tests {
#[fixture]
fn connection() -> OcrConnection {
OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new("request-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("request-key")),
api_base: Some("https://example.com".into()),
..Default::default()
}
@@ -312,7 +312,7 @@ mod tests {
#[tokio::test]
async fn request_endpoint_accepts_request_owned_key() {
let connection = OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new("request-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("request-key")),
api_key_source: InputSource::Request,
api_base: Some("https://request.example".into()),
api_base_source: InputSource::Request,
diff --git a/litellm-rust/crates/llms/src/base_llm/auth.rs b/litellm-rust/crates/llms/src/base_llm/auth.rs
index de271a269d3..ab2cf00614e 100644
--- a/litellm-rust/crates/llms/src/base_llm/auth.rs
+++ b/litellm-rust/crates/llms/src/base_llm/auth.rs
@@ -5,7 +5,9 @@
//! that split: `validate_environment` shapes the forwarded headers and names the credential
//! as an [`AuthScheme`], and [`resolve_auth`] turns the scheme into headers and a signer.
-use litellm_auth::{AuthServices, CredentialPlacement, SecretValue, TokenProviderHandle};
+use litellm_auth::{AuthServices, CredentialPlacement, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
+
use litellm_auth_aws::{AwsCredentialSource, SigV4Signer};
use litellm_auth_gcp::VertexConfig;
use litellm_http::request::with_header;
diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs
index c90fd99809e..9574e99d357 100644
--- a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs
@@ -5,7 +5,9 @@ use serde::{
use serde_json::{Map, Value};
use std::{collections::BTreeMap, future::Future, sync::Arc, time::Duration};
-use litellm_auth::{InputSource, SecretValue, Sourced, TokenProviderHandle};
+use litellm_auth::{InputSource, Sourced, TokenProviderHandle};
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::{call_arguments::CallArguments, settings::ProcessEnvironment};
use litellm_http::outbound::RequestSigner;
use litellm_llms_types::formats::ocr::{LiteLLMOcrResponse, OcrDocument, OcrResponseFormat};
diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs
index 976c9161dd2..6cf0167a30f 100644
--- a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs
+++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs
@@ -1,5 +1,7 @@
use litellm_auth::AwsParams;
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_auth_aws::{
AwsCredentialSource, bedrock_model_id_and_region,
constants::{AWS_BEARER_TOKEN_BEDROCK, BEDROCK_RUNTIME_ENDPOINT_TEMPLATE, BEDROCK_SERVICE},
diff --git a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs
index 79c30df9b19..5b89345a1c5 100644
--- a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs
@@ -1,6 +1,8 @@
use std::sync::LazyLock;
-use litellm_auth::{AwsParams, CredentialPlacement, ParamKind, ParamSpec, SecretValue};
+use litellm_auth::{AwsParams, CredentialPlacement, ParamKind, ParamSpec};
+use litellm_secret_value::SecretValue;
+
use litellm_auth_aws::{AwsCredentialSource, resolve_aws_region};
use litellm_core_utils::settings::resolve_non_empty;
use litellm_http::request::{header_value, with_header, without_headers};
diff --git a/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs b/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs
index 3ec60f12021..1d380c917f6 100644
--- a/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs
+++ b/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs
@@ -7,7 +7,9 @@ use crate::{
};
use futures_util::StreamExt;
use litellm_auth::AwsParams;
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_auth_aws::{
AwsCredentialSource, bedrock_model_id_and_region,
constants::{
diff --git a/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs b/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs
index ae13704f4a0..f139b93e259 100644
--- a/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs
@@ -1,6 +1,8 @@
use std::sync::LazyLock;
-use litellm_auth::{AwsParams, CredentialPlacement, SecretValue};
+use litellm_auth::{AwsParams, CredentialPlacement};
+use litellm_secret_value::SecretValue;
+
use litellm_auth_aws::AwsCredentialSource;
use litellm_auth_aws::bedrock_model_id_and_region;
use litellm_core_utils::settings::resolve_non_empty;
diff --git a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs
index 76ff27b56fc..5bad06adda2 100644
--- a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs
@@ -735,7 +735,7 @@ mod tests {
assert!(matches!(
CohereParseConfig.resolve_headers(
&OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new(" ")),
+ api_key: Some(litellm_secret_value::SecretValue::new(" ")),
..Default::default()
},
&|_| None,
diff --git a/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs b/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs
index 168c3e63920..0dbcea01439 100644
--- a/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_llms_types::{
formats::messages::{MessagesOptionalParams, MessagesRequest, MessagesTool},
diff --git a/litellm-rust/crates/llms/src/edenai/messages/transformation.rs b/litellm-rust/crates/llms/src/edenai/messages/transformation.rs
index 2c3a7eaef1d..d5944484522 100644
--- a/litellm-rust/crates/llms/src/edenai/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/edenai/messages/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_llms_types::formats::messages::{
MessagesOptionalParams, MessagesRequest, MessagesResponse,
diff --git a/litellm-rust/crates/llms/src/minimax/messages/transformation.rs b/litellm-rust/crates/llms/src/minimax/messages/transformation.rs
index 012a657fceb..dafe5fbd16b 100644
--- a/litellm-rust/crates/llms/src/minimax/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/minimax/messages/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_llms_types::{
formats::messages::{MessagesOptionalParams, MessagesRequest},
diff --git a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs
index fa4e2254e86..679aa65f7fd 100644
--- a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs
@@ -219,7 +219,7 @@ mod tests {
#[default(vec![])] extra_headers: Vec<(String, String)>,
) -> OcrConnection {
OcrConnection {
- api_key: api_key.map(litellm_auth::SecretValue::new),
+ api_key: api_key.map(litellm_secret_value::SecretValue::new),
extra_headers,
..OcrConnection::default()
}
diff --git a/litellm-rust/crates/llms/src/openai/responses/transformation.rs b/litellm-rust/crates/llms/src/openai/responses/transformation.rs
index cf74871894b..deb163daa07 100644
--- a/litellm-rust/crates/llms/src/openai/responses/transformation.rs
+++ b/litellm-rust/crates/llms/src/openai/responses/transformation.rs
@@ -3,7 +3,8 @@ use litellm_llms_types::formats::responses::{
};
use serde_json::{Map, Value};
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
use crate::{
Error, ErrorDetail,
diff --git a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs
index e998f8785cb..83fb8ac3fa2 100644
--- a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs
+++ b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs
@@ -4,7 +4,9 @@
//! the `max_completion_tokens` -> `max_tokens` rename and the usage
//! `*_tokens` null-to-zero sanitize.
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::core_helpers::unix_now;
use litellm_llms_types::formats::chat_completions::{
ChatCompletionsChoice, ChatCompletionsChoiceMessage, ChatCompletionsResponse,
diff --git a/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs b/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs
index 5b4e4f77f2a..eeb3d1cb7f8 100644
--- a/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_http::request::{has_bearer_auth, without_headers};
use litellm_llms_types::{
diff --git a/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs b/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs
index 900fba9e916..76155ba55e5 100644
--- a/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs
+++ b/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_llms_types::formats::responses::ResponsesApiResponse;
use serde_json::{Map, Value};
diff --git a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs
index bf0ddc1c7a8..beda6899d04 100644
--- a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs
@@ -690,7 +690,7 @@ mod tests {
#[test]
fn explicit_key_precedes_environment_key() {
let connection = OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new("passed-key")),
+ api_key: Some(litellm_secret_value::SecretValue::new("passed-key")),
..Default::default()
};
let headers = resolve_headers(&connection, &|_| Some("env-key".into())).unwrap();
@@ -700,7 +700,7 @@ mod tests {
#[test]
fn blank_explicit_key_uses_environment_key() {
let connection = OcrConnection {
- api_key: Some(litellm_auth::SecretValue::new(" ")),
+ api_key: Some(litellm_secret_value::SecretValue::new(" ")),
..Default::default()
};
let headers = resolve_headers(&connection, &|_| Some(" env-key ".into())).unwrap();
diff --git a/litellm-rust/crates/llms/src/tencent/messages/transformation.rs b/litellm-rust/crates/llms/src/tencent/messages/transformation.rs
index a70336748ad..621a09f4fb0 100644
--- a/litellm-rust/crates/llms/src/tencent/messages/transformation.rs
+++ b/litellm-rust/crates/llms/src/tencent/messages/transformation.rs
@@ -1,4 +1,6 @@
-use litellm_auth::{CredentialPlacement, SecretValue};
+use litellm_auth::CredentialPlacement;
+use litellm_secret_value::SecretValue;
+
use litellm_core_utils::settings::resolve_non_empty;
use litellm_llms_types::{
formats::messages::{MessagesOptionalParams, MessagesRequest},
diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs
index 77881cd2e78..b902967d9dc 100644
--- a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs
+++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs
@@ -137,7 +137,7 @@ impl VertexAiOcrConfig {
connection
.api_key
.as_ref()
- .map(litellm_auth::SecretValue::expose),
+ .map(litellm_secret_value::SecretValue::expose),
config,
&|name: &str| connection.secret(name),
)
diff --git a/litellm-rust/crates/python-bridge/Cargo.toml b/litellm-rust/crates/python-bridge/Cargo.toml
index a4e1c57ab1e..e3ef45f7876 100644
--- a/litellm-rust/crates/python-bridge/Cargo.toml
+++ b/litellm-rust/crates/python-bridge/Cargo.toml
@@ -19,6 +19,7 @@ huggingface = ["litellm-token-counter/huggingface"]
tiktoken = ["litellm-token-counter/tiktoken"]
[dependencies]
+litellm-secret-value.workspace = true
fancy-regex.workspace = true
litellm-tracing.workspace = true
litellm-spend-clickhouse.workspace = true
diff --git a/litellm-rust/crates/python-bridge/src/credentials.rs b/litellm-rust/crates/python-bridge/src/credentials.rs
index a4bb064aae0..adc52a7e77a 100644
--- a/litellm-rust/crates/python-bridge/src/credentials.rs
+++ b/litellm-rust/crates/python-bridge/src/credentials.rs
@@ -2,7 +2,9 @@
//! keyword arguments and acquired on the host's own thread when the call asks for one.
use crate::callable::wrap_failure;
-use litellm_auth::{ResolvedCredential, SecretValue};
+use litellm_auth::ResolvedCredential;
+use litellm_secret_value::SecretValue;
+
use pyo3::{
exceptions::PyTypeError,
gc::{PyTraverseError, PyVisit},
diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs
index c2c3fe51600..73a2922804a 100644
--- a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs
+++ b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs
@@ -1,4 +1,3 @@
-use litellm_auth::SecretValue;
use litellm_auth::VertexParams;
use litellm_host_python::{from_py, present};
use litellm_inference_ocr::{
@@ -6,6 +5,7 @@ use litellm_inference_ocr::{
wire::{OcrWireRequest, consumed_optional_params, decode_document, decode_request_input},
};
use litellm_llms::base_llm::ocr::{error::Error, transformation::DROP_PARAMS_PARAM};
+use litellm_secret_value::SecretValue;
use pyo3::{prelude::*, types::PyDict};
use serde_json::{Map, Value};
diff --git a/litellm-rust/crates/python-bridge/src/secrets/callback.rs b/litellm-rust/crates/python-bridge/src/secrets/callback.rs
index 669c10e602a..af6485a56e1 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/callback.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/callback.rs
@@ -2,9 +2,11 @@ use std::{future::Future, pin::Pin, sync::Arc};
use litellm_core_utils::settings::Lookup;
use litellm_host_python::{PythonContext, attach_blocking};
+use litellm_secret_value::SecretValue;
use litellm_secrets::{
- Error, ExternalSecretManager, KeyManagementSettings, KeyManagementSystem, Secret, SecretValue,
+ Error, ExternalSecretManager, KeyManagementSettings, KeyManagementSystem, Secret,
};
+
use pyo3::{
exceptions::PyException,
prelude::*,
diff --git a/litellm-rust/crates/python-bridge/src/secrets/config.rs b/litellm-rust/crates/python-bridge/src/secrets/config.rs
index 7afd06e2373..c1203dcbd89 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/config.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/config.rs
@@ -1,7 +1,9 @@
use std::sync::Arc;
+use litellm_secret_value::SecretValue;
use litellm_secrets::{SecretManager, SecretManagerState};
-use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem, SecretValue};
+use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem};
+
use pyo3::prelude::*;
use serde_json::Value;
diff --git a/litellm-rust/crates/python-bridge/src/secrets/operations.rs b/litellm-rust/crates/python-bridge/src/secrets/operations.rs
index b458e3f34e8..027da1e4718 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/operations.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/operations.rs
@@ -81,7 +81,7 @@ impl From for PythonMutationError {
pub(super) async fn write_python_provider(
manager: &SecretManager,
name: &str,
- value: &litellm_secrets::SecretValue,
+ value: &litellm_secret_value::SecretValue,
) -> Result {
match manager {
SecretManager::Cyberark(client) => {
@@ -128,7 +128,7 @@ pub(super) async fn rotate_python_provider(
manager: &SecretManager,
current_name: &str,
new_name: &str,
- value: &litellm_secrets::SecretValue,
+ value: &litellm_secret_value::SecretValue,
) -> Result {
match manager {
SecretManager::Cyberark(client) => {
@@ -186,7 +186,7 @@ pub(super) enum PythonMutationResponse {
pub(super) async fn write_python_provider_with_context(
manager: &SecretManager,
name: &str,
- value: &litellm_secrets::SecretValue,
+ value: &litellm_secret_value::SecretValue,
context: &litellm_secrets_types::SecretWriteContext,
) -> Result {
if let (SecretManager::HashicorpVault(client), SecretOperationContext::Hashicorp(operation)) =
@@ -235,7 +235,7 @@ pub(super) async fn rotate_python_provider_with_context(
manager: &SecretManager,
current_name: &str,
new_name: &str,
- value: &litellm_secrets::SecretValue,
+ value: &litellm_secret_value::SecretValue,
context: &SecretOperationContext,
) -> Result {
if let (SecretManager::HashicorpVault(client), SecretOperationContext::Hashicorp(context)) =
diff --git a/litellm-rust/crates/python-bridge/src/secrets/provider.rs b/litellm-rust/crates/python-bridge/src/secrets/provider.rs
index ff10f596311..55343fb63bb 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/provider.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/provider.rs
@@ -1,7 +1,9 @@
use std::time::Duration;
use super::operations::PythonReadRequest;
-use litellm_secrets::{KeyManagementSystem, SecretValue};
+use litellm_secret_value::SecretValue;
+use litellm_secrets::KeyManagementSystem;
+
use litellm_secrets_types::{
AwsOperationContext, CyberarkOperationContext, GoogleOperationContext,
HashicorpOperationContext, SecretOperationContext,
diff --git a/litellm-rust/crates/python-bridge/src/secrets/python.rs b/litellm-rust/crates/python-bridge/src/secrets/python.rs
index 2f040bff773..c5d9e3565b3 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/python.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/python.rs
@@ -2,7 +2,9 @@ use std::sync::Arc;
use futures_util::future::BoxFuture;
use litellm_host_python::{PythonContext, attach_blocking};
-use litellm_secrets::{Error, SecretValue, source::SecretSource};
+use litellm_secret_value::SecretValue;
+use litellm_secrets::{Error, source::SecretSource};
+
use pyo3::prelude::*;
use super::error::external_error;
diff --git a/litellm-rust/crates/python-bridge/src/secrets/resolved.rs b/litellm-rust/crates/python-bridge/src/secrets/resolved.rs
index deb07175e7a..752b533886e 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/resolved.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/resolved.rs
@@ -4,10 +4,9 @@ use futures_util::future::BoxFuture;
use litellm_core_utils::settings::ProcessEnvironment;
use litellm_host_python::PythonContext;
use litellm_http::Client;
+use litellm_secret_value::SecretValue;
use litellm_secrets::source::SecretSource;
-use litellm_secrets::{
- Error, FailurePolicy, OidcResolver, SecretManagerState, SecretResolver, SecretValue,
-};
+use litellm_secrets::{Error, FailurePolicy, OidcResolver, SecretManagerState, SecretResolver};
use super::config::SecretManagerSnapshot;
diff --git a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs
index b956ec297b6..0d52b0aa629 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs
@@ -307,7 +307,7 @@ impl NativeSecretManager {
super::operations::write_python_provider_with_context(
&backend,
&secret_name,
- &litellm_secrets::SecretValue::new(secret_value),
+ &litellm_secret_value::SecretValue::new(secret_value),
&context,
)
.await,
@@ -365,7 +365,7 @@ impl NativeSecretManager {
&backend,
¤t_secret_name,
&new_secret_name,
- &litellm_secrets::SecretValue::new(new_secret_value),
+ &litellm_secret_value::SecretValue::new(new_secret_value),
&context,
)
.await,
diff --git a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs
index 41658f0baf8..b99dff7aeac 100644
--- a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs
+++ b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs
@@ -1,9 +1,8 @@
use std::collections::HashMap;
-use litellm_secrets::{
- SecretValue,
- hashicorp::{HashicorpVault, RawOperationError},
-};
+use litellm_secret_value::SecretValue;
+use litellm_secrets::hashicorp::{HashicorpVault, RawOperationError};
+
use litellm_secrets_types::{HashicorpOperationContext, SecretWriteContext};
use serde_json::value::RawValue;
diff --git a/litellm-rust/crates/redis-client/Cargo.toml b/litellm-rust/crates/redis-client/Cargo.toml
index eab57b36423..3e9447a6bfa 100644
--- a/litellm-rust/crates/redis-client/Cargo.toml
+++ b/litellm-rust/crates/redis-client/Cargo.toml
@@ -7,6 +7,7 @@ repository.workspace = true
[dependencies]
litellm-circuit-breaker.workspace = true
+litellm-secret-value.workspace = true
futures-util.workspace = true
redis = { workspace = true, features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] }
thiserror.workspace = true
diff --git a/litellm-rust/crates/redis-client/src/client.rs b/litellm-rust/crates/redis-client/src/client.rs
index a02b0847663..48dbc2b2dfb 100644
--- a/litellm-rust/crates/redis-client/src/client.rs
+++ b/litellm-rust/crates/redis-client/src/client.rs
@@ -29,7 +29,7 @@ impl RedisClient {
pub fn new(config: RedisConfig) -> Result {
let connection = match &config.endpoint {
Endpoint::Url(url) => {
- Connection::standalone(url.as_str().into_connection_info()?, config.timeout)?
+ Connection::standalone(url.expose().into_connection_info()?, config.timeout)?
}
Endpoint::Host(host) => {
Connection::standalone(host.connection_info()?, config.timeout)?
@@ -204,6 +204,7 @@ mod tests {
use std::time::{Duration, Instant};
use litellm_redis_testing::RedisServer;
+ use litellm_secret_value::SecretValue;
use rstest::rstest;
use litellm_circuit_breaker::{Breaker, Config, State};
@@ -630,8 +631,12 @@ mod tests {
db: 3,
}))
.unwrap();
- let in_db_0 = connect(Endpoint::Url(server.url().to_owned())).unwrap();
- let via_url = connect(Endpoint::Url(format!("{}/3", server.url()))).unwrap();
+ let in_db_0 = connect(Endpoint::Url(SecretValue::new(server.url().to_owned()))).unwrap();
+ let via_url = connect(Endpoint::Url(SecretValue::new(format!(
+ "{}/3",
+ server.url()
+ ))))
+ .unwrap();
in_db_3
.set("k", "3", Duration::from_secs(60))
diff --git a/litellm-rust/crates/redis-client/src/config.rs b/litellm-rust/crates/redis-client/src/config.rs
index 64934c68891..c40b9eb6b36 100644
--- a/litellm-rust/crates/redis-client/src/config.rs
+++ b/litellm-rust/crates/redis-client/src/config.rs
@@ -1,8 +1,9 @@
-use std::{fmt, time::Duration};
+use std::time::Duration;
use redis::{ConnectionAddr, ConnectionInfo, IntoConnectionInfo, RedisConnectionInfo, RedisResult};
use litellm_circuit_breaker::Breaker;
+use litellm_secret_value::SecretValue;
use crate::Namespace;
@@ -15,28 +16,28 @@ pub struct RedisConfig {
pub breaker: Option,
}
-#[derive(Clone, PartialEq)]
+#[derive(Clone, Debug, PartialEq)]
pub enum Endpoint {
- Url(String),
+ Url(SecretValue),
Host(HostEndpoint),
Cluster(ClusterEndpoint),
}
-#[derive(Clone, PartialEq)]
+#[derive(Clone, Debug, PartialEq)]
pub struct HostEndpoint {
pub host: String,
pub port: u16,
- pub username: Option,
- pub password: Option,
+ pub username: Option,
+ pub password: Option,
pub tls: bool,
pub db: i64,
}
-#[derive(Clone, PartialEq)]
+#[derive(Clone, Debug, PartialEq)]
pub struct ClusterEndpoint {
pub nodes: Vec,
- pub username: Option,
- pub password: Option,
+ pub username: Option,
+ pub password: Option,
pub tls: bool,
}
@@ -58,11 +59,11 @@ impl HostEndpoint {
pub(crate) fn connection_info(&self) -> RedisResult {
let settings = RedisConnectionInfo::default().set_db(self.db);
let settings = match &self.username {
- Some(username) => settings.set_username(username),
+ Some(username) => settings.set_username(username.expose()),
None => settings,
};
let settings = match &self.password {
- Some(password) => settings.set_password(password),
+ Some(password) => settings.set_password(password.expose()),
None => settings,
};
Ok(addr(&self.host, self.port, self.tls)
@@ -83,70 +84,35 @@ fn addr(host: &str, port: u16, tls: bool) -> ConnectionAddr {
}
}
-impl fmt::Debug for Endpoint {
- fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
- match self {
- Self::Url(url) => formatter
- .debug_tuple("Url")
- .field(&redacted_url(url))
- .finish(),
- Self::Host(host) => formatter
- .debug_struct("Host")
- .field("host", &host.host)
- .field("port", &host.port)
- .field("username", &host.username)
- .field("password", &host.password.as_ref().map(|_| ""))
- .field("tls", &host.tls)
- .field("db", &host.db)
- .finish(),
- Self::Cluster(cluster) => formatter
- .debug_struct("Cluster")
- .field("nodes", &cluster.nodes)
- .field("username", &cluster.username)
- .field("password", &cluster.password.as_ref().map(|_| ""))
- .field("tls", &cluster.tls)
- .finish(),
- }
- }
-}
-
-fn redacted_url(url: &str) -> String {
- match url
- .split_once("://")
- .and_then(|(scheme, rest)| Some((scheme, rest.rsplit_once('@')?.1)))
- {
- Some((scheme, host)) => format!("{scheme}://@{host}"),
- None => url.to_owned(),
- }
-}
-
#[cfg(test)]
mod tests {
+ use litellm_secret_value::SecretValue;
use rstest::rstest;
- use super::{Endpoint, HostEndpoint};
+ use super::{ClusterEndpoint, Endpoint, HostEndpoint};
#[rstest]
- #[case::url_with_password(Endpoint::Url("redis://user:secret@cache:6379/0".to_owned()), "redis://@cache:6379/0")]
- #[case::url_with_at_in_password(Endpoint::Url("rediss://:p@ss@cache:6380".to_owned()), "rediss://@cache:6380")]
- #[case::host_password(
+ #[case::url(Endpoint::Url(SecretValue::new("redis://credential-value@cache:6379/0")))]
+ #[case::host(
Endpoint::Host(HostEndpoint {
host: "cache".to_owned(),
port: 6379,
- username: None,
- password: Some("secret".to_owned()),
+ username: Some(SecretValue::new("credential-value")),
+ password: Some(SecretValue::new("credential-value")),
tls: false,
db: 0,
- }),
- ""
+ })
)]
- fn debug_output_never_shows_the_password(#[case] endpoint: Endpoint, #[case] shown: &str) {
+ #[case::cluster(Endpoint::Cluster(ClusterEndpoint {
+ nodes: Vec::new(),
+ username: Some(SecretValue::new("credential-value")),
+ password: Some(SecretValue::new("credential-value")),
+ tls: false,
+ }))]
+ fn debug_output_never_shows_the_secret(#[case] endpoint: Endpoint) {
let debug = format!("{endpoint:?}");
- assert!(
- !debug.contains("secret") && !debug.contains("p@ss"),
- "{debug}"
- );
- assert!(debug.contains(shown), "{debug}");
+ assert!(!debug.contains("credential-value"), "{debug}");
+ assert!(debug.contains("[REDACTED]"), "{debug}");
}
}
diff --git a/litellm-rust/crates/redis-client/src/connection.rs b/litellm-rust/crates/redis-client/src/connection.rs
index aaf26f105da..41729bfd107 100644
--- a/litellm-rust/crates/redis-client/src/connection.rs
+++ b/litellm-rust/crates/redis-client/src/connection.rs
@@ -33,11 +33,11 @@ impl Connection {
.response_timeout(timeout)
.retries(RETRIES);
let builder = match &endpoint.username {
- Some(username) => builder.username(username),
+ Some(username) => builder.username(username.expose()),
None => builder,
};
let builder = match &endpoint.password {
- Some(password) => builder.password(password),
+ Some(password) => builder.password(password.expose()),
None => builder,
};
let config = ClusterConfig::new()
diff --git a/litellm-rust/crates/redis-client/src/test_support.rs b/litellm-rust/crates/redis-client/src/test_support.rs
index 64a174b1543..e503739a942 100644
--- a/litellm-rust/crates/redis-client/src/test_support.rs
+++ b/litellm-rust/crates/redis-client/src/test_support.rs
@@ -2,6 +2,7 @@ use std::time::Duration;
use litellm_circuit_breaker::Breaker;
use litellm_redis_testing::{RedisCluster, RedisServer};
+use litellm_secret_value::SecretValue;
use redis::cluster_routing::{Route, RoutingInfo, SingleNodeRoutingInfo, SlotAddr};
use crate::{
@@ -30,7 +31,7 @@ impl Server {
pub(crate) fn endpoint(&self) -> Endpoint {
match self {
- Self::Standalone(server) => Endpoint::Url(server.url().to_owned()),
+ Self::Standalone(server) => Endpoint::Url(SecretValue::new(server.url().to_owned())),
Self::Cluster(cluster) => Endpoint::Cluster(ClusterEndpoint {
nodes: cluster
.nodes()
diff --git a/litellm-rust/crates/router-types/Cargo.toml b/litellm-rust/crates/router-types/Cargo.toml
index 2e9fec28685..f0470ce8a8e 100644
--- a/litellm-rust/crates/router-types/Cargo.toml
+++ b/litellm-rust/crates/router-types/Cargo.toml
@@ -7,6 +7,7 @@ repository.workspace = true
description = "The types of litellm/types/router.py: a deployment's litellm_params as a config or a caller spells them"
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth-types.workspace = true
serde.workspace = true
serde_json.workspace = true
diff --git a/litellm-rust/crates/router-types/src/litellm_params.rs b/litellm-rust/crates/router-types/src/litellm_params.rs
index 186a1d98036..aa2c47f6e3c 100644
--- a/litellm-rust/crates/router-types/src/litellm_params.rs
+++ b/litellm-rust/crates/router-types/src/litellm_params.rs
@@ -1,6 +1,8 @@
use std::{collections::BTreeMap, fmt, ops::Deref};
-use litellm_auth_types::{AwsParams, ParamSpec, SecretValue, VertexParams};
+use litellm_auth_types::{AwsParams, ParamSpec, VertexParams};
+use litellm_secret_value::SecretValue;
+
use serde::Deserialize;
use serde_json::Value;
use serde_with::{OneOrMany, formats::PreferMany, serde_as};
diff --git a/litellm-rust/crates/router-types/tests/litellm_params.rs b/litellm-rust/crates/router-types/tests/litellm_params.rs
index 7d021fd857a..c5c0d16ee8f 100644
--- a/litellm-rust/crates/router-types/tests/litellm_params.rs
+++ b/litellm-rust/crates/router-types/tests/litellm_params.rs
@@ -1,4 +1,6 @@
-use litellm_auth_types::{AwsParams, SecretValue, VertexParams};
+use litellm_auth_types::{AwsParams, VertexParams};
+use litellm_secret_value::SecretValue;
+
use litellm_router_types::{LitellmParams, Spelled};
use rstest::rstest;
use serde_json::{Map, Value, json};
diff --git a/litellm-rust/crates/secret-value/Cargo.toml b/litellm-rust/crates/secret-value/Cargo.toml
new file mode 100644
index 00000000000..467f852a278
--- /dev/null
+++ b/litellm-rust/crates/secret-value/Cargo.toml
@@ -0,0 +1,14 @@
+[package]
+name = "litellm-secret-value"
+version = "0.1.0"
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+
+[dependencies]
+serde.workspace = true
+subtle.workspace = true
+veil.workspace = true
+
+[dev-dependencies]
+rstest.workspace = true
diff --git a/litellm-rust/crates/secret-value/src/lib.rs b/litellm-rust/crates/secret-value/src/lib.rs
new file mode 100644
index 00000000000..f11d22d3701
--- /dev/null
+++ b/litellm-rust/crates/secret-value/src/lib.rs
@@ -0,0 +1,5 @@
+#![forbid(unsafe_code)]
+
+mod secret_value;
+
+pub use secret_value::SecretValue;
diff --git a/litellm-rust/crates/auth-types/src/secret.rs b/litellm-rust/crates/secret-value/src/secret_value.rs
similarity index 95%
rename from litellm-rust/crates/auth-types/src/secret.rs
rename to litellm-rust/crates/secret-value/src/secret_value.rs
index 7e6789deef7..27e4227080d 100644
--- a/litellm-rust/crates/auth-types/src/secret.rs
+++ b/litellm-rust/crates/secret-value/src/secret_value.rs
@@ -32,9 +32,11 @@ impl Hash for SecretValue {
#[cfg(test)]
mod tests {
+ use rstest::rstest;
+
use super::SecretValue;
- #[test]
+ #[rstest]
fn debug_redacts_plaintext() {
let debug = format!("{:?}", SecretValue::new("credential-value"));
@@ -42,7 +44,7 @@ mod tests {
assert!(debug.contains("REDACTED"));
}
- #[test]
+ #[rstest]
fn equality_compares_plaintext_values() {
assert_eq!(SecretValue::new("same"), SecretValue::new("same"));
assert_ne!(SecretValue::new("same"), SecretValue::new("different"));
diff --git a/litellm-rust/crates/secrets-aws/Cargo.toml b/litellm-rust/crates/secrets-aws/Cargo.toml
index 5d3bd413484..d17dfcfeec2 100644
--- a/litellm-rust/crates/secrets-aws/Cargo.toml
+++ b/litellm-rust/crates/secrets-aws/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-auth-aws.workspace = true
litellm-secrets-types.workspace = true
litellm-core-utils.workspace = true
diff --git a/litellm-rust/crates/secrets-aws/src/secret_manager.rs b/litellm-rust/crates/secrets-aws/src/secret_manager.rs
index 2220e387838..b25872ab3f6 100644
--- a/litellm-rust/crates/secrets-aws/src/secret_manager.rs
+++ b/litellm-rust/crates/secrets-aws/src/secret_manager.rs
@@ -21,11 +21,12 @@ use litellm_auth_aws::constants::{
AWS_ACCESS_KEY_ID, AWS_REGION, AWS_REGION_NAME, AWS_SECRET_ACCESS_KEY,
};
use litellm_core_utils::settings::Lookup;
+use litellm_secret_value::SecretValue;
use litellm_secrets_types::{
AwsOperationContext, BaseSecretManager, KeyManagementSettings, RotationError, Secret,
- SecretDeleter, SecretRotator, SecretValue, SecretWriteContext, SecretWriter,
- async_rotate_secret,
+ SecretDeleter, SecretRotator, SecretWriteContext, SecretWriter, async_rotate_secret,
};
+
use serde_json::Value;
use crate::{Error, auth};
diff --git a/litellm-rust/crates/secrets-aws/tests/secret_manager.rs b/litellm-rust/crates/secrets-aws/tests/secret_manager.rs
index c482a168090..56999633a78 100644
--- a/litellm-rust/crates/secrets-aws/tests/secret_manager.rs
+++ b/litellm-rust/crates/secrets-aws/tests/secret_manager.rs
@@ -10,11 +10,13 @@ use aws_sdk_secretsmanager::{
Client,
config::{BehaviorVersion, Credentials, Region, retry::RetryConfig},
};
+use litellm_secret_value::SecretValue;
use litellm_secrets_aws::{AwsSecretsManagerV2, Error, RotationResponse};
use litellm_secrets_types::{
AwsOperationContext, BaseSecretManager, KeyManagementSettings, Secret, SecretDeleter,
- SecretValue, SecretWriteContext, SecretWriter,
+ SecretWriteContext, SecretWriter,
};
+
use rstest::{fixture, rstest};
use serde_json::json;
use wiremock::{
diff --git a/litellm-rust/crates/secrets-azure/Cargo.toml b/litellm-rust/crates/secrets-azure/Cargo.toml
index 2aff6129c37..c4bfb24898c 100644
--- a/litellm-rust/crates/secrets-azure/Cargo.toml
+++ b/litellm-rust/crates/secrets-azure/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-http.workspace = true
tokio.workspace = true
litellm-auth-azure.workspace = true
diff --git a/litellm-rust/crates/secrets-azure/src/key_vault.rs b/litellm-rust/crates/secrets-azure/src/key_vault.rs
index d97b7a22cfc..40028828298 100644
--- a/litellm-rust/crates/secrets-azure/src/key_vault.rs
+++ b/litellm-rust/crates/secrets-azure/src/key_vault.rs
@@ -3,7 +3,9 @@ use std::sync::Arc;
use litellm_auth_azure::{AzureAuthInputs, AzureAuthService, ConfigValue};
use litellm_auth_types::{InputSource, Sourced};
use litellm_core_utils::settings::Lookup;
-use litellm_secrets_types::{AzureOperationContext, BaseSecretManager, Secret, SecretValue};
+use litellm_secret_value::SecretValue;
+use litellm_secrets_types::{AzureOperationContext, BaseSecretManager, Secret};
+
use percent_encoding::{AsciiSet, NON_ALPHANUMERIC};
use serde::Deserialize;
diff --git a/litellm-rust/crates/secrets-azure/tests/key_vault.rs b/litellm-rust/crates/secrets-azure/tests/key_vault.rs
index a0ba9e6ddf8..c3ba01672a8 100644
--- a/litellm-rust/crates/secrets-azure/tests/key_vault.rs
+++ b/litellm-rust/crates/secrets-azure/tests/key_vault.rs
@@ -1,7 +1,9 @@
use std::sync::Arc;
+use litellm_secret_value::SecretValue;
use litellm_secrets_azure::{AzureKeyVault, Error};
-use litellm_secrets_types::{Secret, SecretValue};
+use litellm_secrets_types::Secret;
+
use rstest::{fixture, rstest};
use serde::Deserialize;
use wiremock::{
diff --git a/litellm-rust/crates/secrets-cyberark/Cargo.toml b/litellm-rust/crates/secrets-cyberark/Cargo.toml
index 2ae2c1ea5c3..643f322f7fa 100644
--- a/litellm-rust/crates/secrets-cyberark/Cargo.toml
+++ b/litellm-rust/crates/secrets-cyberark/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-secrets-types.workspace = true
litellm-core-utils.workspace = true
litellm-http.workspace = true
diff --git a/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs b/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs
index c5cb0308b13..cf077a2bbc8 100644
--- a/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs
+++ b/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs
@@ -9,11 +9,12 @@ use litellm_core_utils::settings::Lookup;
use litellm_http::{
Client, ClientIdentity, ClientVariant, HttpClientConfig, HttpClientPool, TlsSource, Verify,
};
+use litellm_secret_value::SecretValue;
use litellm_secrets_types::{
BaseSecretManager, CyberarkOperationContext, RotationError, SecretCache, SecretDeleter,
- SecretRotator, SecretValue, SecretWriteContext, SecretWriter, async_rotate_secret,
- validate_secret_name,
+ SecretRotator, SecretWriteContext, SecretWriter, async_rotate_secret, validate_secret_name,
};
+
use moka::future::Cache;
use percent_encoding::{AsciiSet, NON_ALPHANUMERIC, utf8_percent_encode};
diff --git a/litellm-rust/crates/secrets-cyberark/tests/secret_manager.rs b/litellm-rust/crates/secrets-cyberark/tests/secret_manager.rs
index 783ba6ff67a..e9b480e026b 100644
--- a/litellm-rust/crates/secrets-cyberark/tests/secret_manager.rs
+++ b/litellm-rust/crates/secrets-cyberark/tests/secret_manager.rs
@@ -9,8 +9,10 @@ use std::{
use base64::{Engine, engine::general_purpose::STANDARD};
use litellm_core_utils::settings::Lookup;
use litellm_http::{HttpClientPool, HttpSettings, Resolution, media::PublicDnsResolver};
+use litellm_secret_value::SecretValue;
use litellm_secrets_cyberark::{CyberArkSecretManager, DeleteOutcome, Error};
-use litellm_secrets_types::{BaseSecretManager, CyberarkOperationContext, SecretValue};
+use litellm_secrets_types::{BaseSecretManager, CyberarkOperationContext};
+
use rstest::{fixture, rstest};
use serde::Deserialize;
use wiremock::{
diff --git a/litellm-rust/crates/secrets-google/Cargo.toml b/litellm-rust/crates/secrets-google/Cargo.toml
index 06b3747def5..3c5c2130788 100644
--- a/litellm-rust/crates/secrets-google/Cargo.toml
+++ b/litellm-rust/crates/secrets-google/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-http.workspace = true
moka.workspace = true
tokio.workspace = true
diff --git a/litellm-rust/crates/secrets-google/src/auth.rs b/litellm-rust/crates/secrets-google/src/auth.rs
index 45fc99d8d5f..7022ecc5ae4 100644
--- a/litellm-rust/crates/secrets-google/src/auth.rs
+++ b/litellm-rust/crates/secrets-google/src/auth.rs
@@ -3,7 +3,7 @@ use std::sync::Arc;
use litellm_auth_gcp::{GoogleCredentials, VertexConfig};
use litellm_auth_types::{InputSource, Sourced};
use litellm_core_utils::settings::Lookup;
-use litellm_secrets_types::SecretValue;
+use litellm_secret_value::SecretValue;
pub(crate) fn credentials(
project: Option,
diff --git a/litellm-rust/crates/secrets-google/src/kms.rs b/litellm-rust/crates/secrets-google/src/kms.rs
index 78ebae008f7..5622a3a4d56 100644
--- a/litellm-rust/crates/secrets-google/src/kms.rs
+++ b/litellm-rust/crates/secrets-google/src/kms.rs
@@ -2,7 +2,7 @@ use std::sync::Arc;
use google_cloud_kms_v1::client::KeyManagementService;
use litellm_core_utils::settings::Lookup;
-use litellm_secrets_types::SecretValue;
+use litellm_secret_value::SecretValue;
use crate::{Error, auth};
diff --git a/litellm-rust/crates/secrets-google/src/secret_manager.rs b/litellm-rust/crates/secrets-google/src/secret_manager.rs
index 463d1f6fd73..577a2ccbd4d 100644
--- a/litellm-rust/crates/secrets-google/src/secret_manager.rs
+++ b/litellm-rust/crates/secrets-google/src/secret_manager.rs
@@ -2,9 +2,9 @@ use std::{sync::Arc, time::Duration};
use base64::{Engine, engine::general_purpose::STANDARD};
use litellm_core_utils::settings::Lookup;
-use litellm_secrets_types::{
- BaseSecretManager, GoogleOperationContext, Secret, SecretCache, SecretValue,
-};
+use litellm_secret_value::SecretValue;
+use litellm_secrets_types::{BaseSecretManager, GoogleOperationContext, Secret, SecretCache};
+
use serde::Deserialize;
use litellm_auth_gcp::GoogleCredentials;
diff --git a/litellm-rust/crates/secrets-hashicorp/Cargo.toml b/litellm-rust/crates/secrets-hashicorp/Cargo.toml
index 8731a8deb7a..4fcd496931d 100644
--- a/litellm-rust/crates/secrets-hashicorp/Cargo.toml
+++ b/litellm-rust/crates/secrets-hashicorp/Cargo.toml
@@ -6,6 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
+litellm-secret-value.workspace = true
litellm-error.workspace = true
litellm-core-utils.workspace = true
litellm-http.workspace = true
diff --git a/litellm-rust/crates/secrets-hashicorp/src/config.rs b/litellm-rust/crates/secrets-hashicorp/src/config.rs
index 4232667ae16..a63161b17b2 100644
--- a/litellm-rust/crates/secrets-hashicorp/src/config.rs
+++ b/litellm-rust/crates/secrets-hashicorp/src/config.rs
@@ -1,7 +1,7 @@
use std::{path::PathBuf, time::Duration};
use litellm_core_utils::settings::Lookup;
-use litellm_secrets_types::SecretValue;
+use litellm_secret_value::SecretValue;
use crate::Error;
diff --git a/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs b/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs
index 447beacda85..dae14439630 100644
--- a/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs
+++ b/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs
@@ -12,11 +12,12 @@ use std::{
};
use litellm_core_utils::settings::Lookup;
+use litellm_secret_value::SecretValue;
use litellm_secrets_types::{
BaseSecretManager, HashicorpOperationContext, RotationError, SecretCache, SecretDeleter,
- SecretRotator, SecretValue, SecretWriteContext, SecretWriter, async_rotate_secret,
- validate_secret_name,
+ SecretRotator, SecretWriteContext, SecretWriter, async_rotate_secret, validate_secret_name,
};
+
use rustify::errors::ClientError as RustifyClientError;
use serde_json::Value;
use tokio::sync::Mutex;
diff --git a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager.rs b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager.rs
index 668871d3fc7..d431e472400 100644
--- a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager.rs
+++ b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager.rs
@@ -1,11 +1,13 @@
use std::{collections::HashMap, sync::Arc, time::Duration};
use litellm_core_utils::settings::Lookup;
+use litellm_secret_value::SecretValue;
use litellm_secrets_hashicorp::{Error, HashicorpVault, HashicorpVaultConfig};
use litellm_secrets_types::{
- BaseSecretManager, HashicorpOperationContext, RotationError, SecretDeleter, SecretValue,
- SecretWriteContext, SecretWriter,
+ BaseSecretManager, HashicorpOperationContext, RotationError, SecretDeleter, SecretWriteContext,
+ SecretWriter,
};
+
use rstest::{fixture, rstest};
use serde::Deserialize;
use serde_json::json;
diff --git a/litellm-rust/crates/secrets-types/Cargo.toml b/litellm-rust/crates/secrets-types/Cargo.toml
index 6dd847ec989..b45c41bcf1c 100644
--- a/litellm-rust/crates/secrets-types/Cargo.toml
+++ b/litellm-rust/crates/secrets-types/Cargo.toml
@@ -6,7 +6,7 @@ license.workspace = true
repository.workspace = true
[dependencies]
-litellm-auth-types.workspace = true
+litellm-secret-value.workspace = true
moka.workspace = true
tokio = { workspace = true, features = ["sync"] }
serde.workspace = true
diff --git a/litellm-rust/crates/secrets-types/src/base_secret_manager.rs b/litellm-rust/crates/secrets-types/src/base_secret_manager.rs
index 737fad28662..41dadd1baa6 100644
--- a/litellm-rust/crates/secrets-types/src/base_secret_manager.rs
+++ b/litellm-rust/crates/secrets-types/src/base_secret_manager.rs
@@ -1,4 +1,5 @@
-use crate::{Error, SecretValue, SecretWriteContext};
+use crate::{Error, SecretWriteContext};
+use litellm_secret_value::SecretValue;
pub fn validate_secret_name(name: &str) -> Result<(), Error> {
if name.split('/').any(|segment| segment == "..")
diff --git a/litellm-rust/crates/secrets-types/src/config.rs b/litellm-rust/crates/secrets-types/src/config.rs
index 9b6e82c3a6b..a45acb4df11 100644
--- a/litellm-rust/crates/secrets-types/src/config.rs
+++ b/litellm-rust/crates/secrets-types/src/config.rs
@@ -3,7 +3,7 @@ use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use strum::IntoStaticStr;
-use crate::SecretValue;
+use litellm_secret_value::SecretValue;
#[derive(
Clone,
diff --git a/litellm-rust/crates/secrets-types/src/context.rs b/litellm-rust/crates/secrets-types/src/context.rs
index 1ec493edf42..b71195e2d21 100644
--- a/litellm-rust/crates/secrets-types/src/context.rs
+++ b/litellm-rust/crates/secrets-types/src/context.rs
@@ -1,6 +1,7 @@
use std::{collections::BTreeMap, time::Duration};
-use crate::{Error, KeyManagementSystem, SecretValue};
+use crate::{Error, KeyManagementSystem};
+use litellm_secret_value::SecretValue;
#[derive(Clone, Debug, Default, Eq, PartialEq)]
pub enum SecretOperationContext {
diff --git a/litellm-rust/crates/secrets-types/src/lib.rs b/litellm-rust/crates/secrets-types/src/lib.rs
index 4d7eaf8313e..2563ccbe39d 100644
--- a/litellm-rust/crates/secrets-types/src/lib.rs
+++ b/litellm-rust/crates/secrets-types/src/lib.rs
@@ -18,5 +18,4 @@ pub use context::{
HashicorpOperationContext, SecretOperationContext, SecretWriteContext,
};
pub use error::Error;
-pub use litellm_auth_types::SecretValue;
pub use value::{PythonSecretRead, Secret};
diff --git a/litellm-rust/crates/secrets-types/src/value.rs b/litellm-rust/crates/secrets-types/src/value.rs
index 11f57ef4448..8c92902edad 100644
--- a/litellm-rust/crates/secrets-types/src/value.rs
+++ b/litellm-rust/crates/secrets-types/src/value.rs
@@ -1,4 +1,4 @@
-use crate::SecretValue;
+use litellm_secret_value::SecretValue;
#[derive(Clone, PartialEq, Eq, veil::Redact)]
pub enum Secret {
diff --git a/litellm-rust/crates/secrets-types/tests/config.rs b/litellm-rust/crates/secrets-types/tests/config.rs
index 45fd9d0ee03..e7ba63343da 100644
--- a/litellm-rust/crates/secrets-types/tests/config.rs
+++ b/litellm-rust/crates/secrets-types/tests/config.rs
@@ -1,6 +1,6 @@
-use litellm_secrets_types::{
- AccessMode, KeyManagementSettings, KeyManagementSystem, Secret, SecretValue,
-};
+use litellm_secret_value::SecretValue;
+use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem, Secret};
+
use rstest::{fixture, rstest};
use serde_json::json;
diff --git a/litellm-rust/crates/secrets-types/tests/context.rs b/litellm-rust/crates/secrets-types/tests/context.rs
index 8f8eeabdb62..f8d754564dc 100644
--- a/litellm-rust/crates/secrets-types/tests/context.rs
+++ b/litellm-rust/crates/secrets-types/tests/context.rs
@@ -1,10 +1,11 @@
use std::{collections::BTreeMap, time::Duration};
+use litellm_secret_value::SecretValue;
use litellm_secrets_types::{
AwsOperationContext, AzureOperationContext, CyberarkOperationContext, GoogleOperationContext,
- HashicorpOperationContext, KeyManagementSystem, SecretOperationContext, SecretValue,
- SecretWriteContext,
+ HashicorpOperationContext, KeyManagementSystem, SecretOperationContext, SecretWriteContext,
};
+
use rstest::{fixture, rstest};
#[fixture]
diff --git a/litellm-rust/crates/secrets-types/tests/rotation.rs b/litellm-rust/crates/secrets-types/tests/rotation.rs
index 36c1a065ffa..089890c60bd 100644
--- a/litellm-rust/crates/secrets-types/tests/rotation.rs
+++ b/litellm-rust/crates/secrets-types/tests/rotation.rs
@@ -1,10 +1,12 @@
use std::sync::atomic::{AtomicUsize, Ordering};
+use litellm_secret_value::SecretValue;
use litellm_secrets_types::{
BaseSecretManager, Error, HashicorpOperationContext, RotationError, SecretDeleter,
- SecretOperationContext, SecretRotator, SecretValue, SecretWriteContext, SecretWriter,
- async_rotate_secret, validate_secret_name,
+ SecretOperationContext, SecretRotator, SecretWriteContext, SecretWriter, async_rotate_secret,
+ validate_secret_name,
};
+
use rstest::{fixture, rstest};
struct Manager {
diff --git a/litellm-rust/crates/secrets/Cargo.toml b/litellm-rust/crates/secrets/Cargo.toml
index 023d61136e4..21ee1bdc760 100644
--- a/litellm-rust/crates/secrets/Cargo.toml
+++ b/litellm-rust/crates/secrets/Cargo.toml
@@ -14,6 +14,7 @@ azure = ["dep:litellm-secrets-azure"]
cyberark = ["dep:litellm-secrets-cyberark"]
[dependencies]
+litellm-secret-value.workspace = true
futures-util.workspace = true
litellm-python-compat.workspace = true
litellm-secrets-types.workspace = true
diff --git a/litellm-rust/crates/secrets/src/compatibility.rs b/litellm-rust/crates/secrets/src/compatibility.rs
index 368c9931b51..3d6ba854325 100644
--- a/litellm-rust/crates/secrets/src/compatibility.rs
+++ b/litellm-rust/crates/secrets/src/compatibility.rs
@@ -3,9 +3,10 @@ use litellm_python_compat::{Value, literal::literal_eval};
use litellm_secrets_types::PythonSecretRead;
use crate::{
- Error, KeyManagementSettings, KeyManagementSystem, Secret, SecretManager, SecretValue,
+ Error, KeyManagementSettings, KeyManagementSystem, Secret, SecretManager,
get_secret_from_manager,
};
+use litellm_secret_value::SecretValue;
pub async fn get_secret_from_python_manager(
manager: &SecretManager,
diff --git a/litellm-rust/crates/secrets/src/handler.rs b/litellm-rust/crates/secrets/src/handler.rs
index f42e09c6bdb..2d2b81d197f 100644
--- a/litellm-rust/crates/secrets/src/handler.rs
+++ b/litellm-rust/crates/secrets/src/handler.rs
@@ -5,7 +5,7 @@ use litellm_core_utils::settings::Lookup;
use crate::{Error, KeyManagementSettings, KeyManagementSystem, Secret};
#[cfg(any(feature = "aws", feature = "google"))]
-use crate::SecretValue;
+use litellm_secret_value::SecretValue;
#[cfg(any(
feature = "google",
diff --git a/litellm-rust/crates/secrets/src/lib.rs b/litellm-rust/crates/secrets/src/lib.rs
index 8d0f0561608..e4459576e94 100644
--- a/litellm-rust/crates/secrets/src/lib.rs
+++ b/litellm-rust/crates/secrets/src/lib.rs
@@ -12,9 +12,7 @@ mod state;
pub use compatibility::{get_secret_from_python_manager, read_secret_from_python_manager};
pub use error::Error;
pub use handler::{ExternalSecretManager, SecretManager, get_secret_from_manager};
-pub use litellm_secrets_types::{
- AccessMode, KeyManagementSettings, KeyManagementSystem, Secret, SecretValue,
-};
+pub use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem, Secret};
pub use native::load_native_manager;
pub use oidc::{OidcProvider, OidcReference, OidcResolver};
pub use resolver::{FailurePolicy, SecretResolver, normalize_nonempty_secret_str};
diff --git a/litellm-rust/crates/secrets/src/oidc.rs b/litellm-rust/crates/secrets/src/oidc.rs
index 24e6912c797..18fc9479ef1 100644
--- a/litellm-rust/crates/secrets/src/oidc.rs
+++ b/litellm-rust/crates/secrets/src/oidc.rs
@@ -9,7 +9,8 @@ use litellm_http::Client;
use moka::future::Cache;
use serde::Deserialize;
-use crate::{Error, SecretValue};
+use crate::Error;
+use litellm_secret_value::SecretValue;
const GOOGLE_TOKEN_MAX_TTL: Duration = Duration::from_secs(3540);
const GITHUB_TOKEN_TTL: Duration = Duration::from_secs(295);
diff --git a/litellm-rust/crates/secrets/src/resolver.rs b/litellm-rust/crates/secrets/src/resolver.rs
index 830e7c22cd5..631b60cba10 100644
--- a/litellm-rust/crates/secrets/src/resolver.rs
+++ b/litellm-rust/crates/secrets/src/resolver.rs
@@ -4,7 +4,8 @@ use crate::compatibility::python_manager_string;
use litellm_core_utils::{serde_compat::parse_str_bool, settings::Lookup};
use crate::state::{LookupTarget, normalize_secret_name};
-use crate::{Error, OidcResolver, Secret, SecretManagerState, SecretValue};
+use crate::{Error, OidcResolver, Secret, SecretManagerState};
+use litellm_secret_value::SecretValue;
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum FailurePolicy {
diff --git a/litellm-rust/crates/secrets/src/source.rs b/litellm-rust/crates/secrets/src/source.rs
index 3c86fd25e9a..1ee224b763a 100644
--- a/litellm-rust/crates/secrets/src/source.rs
+++ b/litellm-rust/crates/secrets/src/source.rs
@@ -3,7 +3,8 @@ use std::{collections::HashMap, sync::Arc};
use futures_util::future::{BoxFuture, try_join_all};
use litellm_core_utils::settings::Lookup;
-use crate::{Error, SecretResolver, SecretValue};
+use crate::{Error, SecretResolver};
+use litellm_secret_value::SecretValue;
pub type Secrets = Arc;
diff --git a/litellm-rust/crates/secrets/tests/aws.rs b/litellm-rust/crates/secrets/tests/aws.rs
index abad04f31c5..6f30d1b5161 100644
--- a/litellm-rust/crates/secrets/tests/aws.rs
+++ b/litellm-rust/crates/secrets/tests/aws.rs
@@ -2,11 +2,13 @@
use std::sync::Arc;
+use litellm_secret_value::SecretValue;
use litellm_secrets::{
AccessMode, Error, FailurePolicy, KeyManagementSettings, OidcResolver, Secret, SecretManager,
- SecretManagerState, SecretResolver, SecretValue, aws::AwsSecretsManagerV2,
+ SecretManagerState, SecretResolver, aws::AwsSecretsManagerV2,
secret_manager_would_be_consulted,
};
+
use wiremock::{Mock, MockServer, ResponseTemplate, matchers::method};
fn state(server: &MockServer, settings: KeyManagementSettings) -> SecretManagerState {
diff --git a/litellm-rust/crates/secrets/tests/azure.rs b/litellm-rust/crates/secrets/tests/azure.rs
index 6e81017b0f6..201e31f44b8 100644
--- a/litellm-rust/crates/secrets/tests/azure.rs
+++ b/litellm-rust/crates/secrets/tests/azure.rs
@@ -68,10 +68,11 @@ async fn azure_handler_reads_missing_and_failed_secrets() {
async fn successful_azure_responses_do_not_fall_back_when_the_value_is_empty_or_null(
#[case] body: serde_json::Value,
) {
+ use litellm_secret_value::SecretValue;
use litellm_secrets::{
- OidcResolver, SecretManager, SecretManagerState, SecretResolver, SecretValue,
- azure::AzureKeyVault,
+ OidcResolver, SecretManager, SecretManagerState, SecretResolver, azure::AzureKeyVault,
};
+
use std::sync::Arc;
use wiremock::{Mock, MockServer, ResponseTemplate, matchers::any};
let server = MockServer::start().await;
diff --git a/litellm-rust/crates/secrets/tests/common_read_contract.rs b/litellm-rust/crates/secrets/tests/common_read_contract.rs
index e6928038799..4cf3513fadf 100644
--- a/litellm-rust/crates/secrets/tests/common_read_contract.rs
+++ b/litellm-rust/crates/secrets/tests/common_read_contract.rs
@@ -9,8 +9,9 @@ use std::sync::Arc;
use base64::{Engine, engine::general_purpose::STANDARD};
use litellm_core_utils::settings::Lookup;
+use litellm_secret_value::SecretValue;
use litellm_secrets::{
- KeyManagementSettings, SecretManager, SecretValue,
+ KeyManagementSettings, SecretManager,
aws::AwsSecretsManagerV2,
azure::AzureKeyVault,
cyberark::CyberArkSecretManager,
@@ -18,6 +19,7 @@ use litellm_secrets::{
google::GoogleSecretManager,
hashicorp::{HashicorpVault, HashicorpVaultConfig},
};
+
use rstest::rstest;
use serde_json::json;
use wiremock::{
diff --git a/litellm-rust/crates/secrets/tests/cyberark.rs b/litellm-rust/crates/secrets/tests/cyberark.rs
index 0a6c394937e..09342234118 100644
--- a/litellm-rust/crates/secrets/tests/cyberark.rs
+++ b/litellm-rust/crates/secrets/tests/cyberark.rs
@@ -4,10 +4,12 @@
async fn cyberark_handler_reads_values_and_surfaces_errors() {
use std::time::Duration;
+ use litellm_secret_value::SecretValue;
use litellm_secrets::{
- Error, KeyManagementSettings, SecretManager, SecretValue, cyberark::CyberArkSecretManager,
+ Error, KeyManagementSettings, SecretManager, cyberark::CyberArkSecretManager,
get_secret_from_manager,
};
+
use wiremock::{
Mock, MockServer, ResponseTemplate,
matchers::{body_string, path},
diff --git a/litellm-rust/crates/secrets/tests/google.rs b/litellm-rust/crates/secrets/tests/google.rs
index 5542c864dfd..a75e34335de 100644
--- a/litellm-rust/crates/secrets/tests/google.rs
+++ b/litellm-rust/crates/secrets/tests/google.rs
@@ -7,10 +7,12 @@ use std::sync::Arc;
#[case::failure(503)]
#[tokio::test]
async fn google_resolver_distinguishes_absence_from_failure(#[case] status: u16) {
+ use litellm_secret_value::SecretValue;
use litellm_secrets::{
Error, FailurePolicy, KeyManagementSettings, OidcResolver, SecretManager,
- SecretManagerState, SecretResolver, SecretValue, google::GoogleSecretManager,
+ SecretManagerState, SecretResolver, google::GoogleSecretManager,
};
+
use wiremock::{Mock, MockServer, ResponseTemplate, matchers::method};
let server = MockServer::start().await;
Mock::given(method("GET"))
diff --git a/litellm-rust/crates/secrets/tests/oidc.rs b/litellm-rust/crates/secrets/tests/oidc.rs
index 867a662cbbb..2d250e1de9d 100644
--- a/litellm-rust/crates/secrets/tests/oidc.rs
+++ b/litellm-rust/crates/secrets/tests/oidc.rs
@@ -315,7 +315,7 @@ async fn azure_oidc_acquires_the_requested_scope_and_preserves_failures(#[case]
Box<
dyn std::future::Future<
Output = Result<
- litellm_secrets::SecretValue,
+ litellm_secret_value::SecretValue,
litellm_secrets::azure::Error,
>,
> + Send
@@ -331,7 +331,7 @@ async fn azure_oidc_acquires_the_requested_scope_and_preserves_failures(#[case]
if self.0 {
Err(litellm_secrets::azure::Error::MissingCredentials)
} else {
- Ok(litellm_secrets::SecretValue::new("azure-token"))
+ Ok(litellm_secret_value::SecretValue::new("azure-token"))
}
})
}
@@ -346,7 +346,7 @@ async fn azure_oidc_acquires_the_requested_scope_and_preserves_failures(#[case]
let result = resolver
.get_secret_str(
"oidc/azure/api://audience/path",
- Some(litellm_secrets::SecretValue::new("fallback")),
+ Some(litellm_secret_value::SecretValue::new("fallback")),
)
.await;
if failed {
diff --git a/litellm-rust/crates/secrets/tests/resolution.rs b/litellm-rust/crates/secrets/tests/resolution.rs
index 8435e051315..185f2256780 100644
--- a/litellm-rust/crates/secrets/tests/resolution.rs
+++ b/litellm-rust/crates/secrets/tests/resolution.rs
@@ -1,9 +1,10 @@
use std::{future::Future, pin::Pin, sync::Arc};
use litellm_core_utils::settings::Lookup;
+use litellm_secret_value::SecretValue;
use litellm_secrets::{
Error, ExternalSecretManager, FailurePolicy, KeyManagementSettings, KeyManagementSystem,
- OidcResolver, Secret, SecretManager, SecretManagerState, SecretResolver, SecretValue,
+ OidcResolver, Secret, SecretManager, SecretManagerState, SecretResolver,
secret_manager_would_be_consulted,
};
From 24b8889965a6f86bee6f743d61c77da0896eaa15 Mon Sep 17 00:00:00 2001
From: "devin-ai-integration[bot]"
<158243242+devin-ai-integration[bot]@users.noreply.github.com>
Date: Fri, 30 Oct 2026 11:35:18 -0700
Subject: [PATCH 24/87] chore(cost-map): add openai gpt-6-rosalind from the
pricing page and changelog (#49268)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
...odel_prices_and_context_window_backup.json | 56 +++++++++++++++++++
model_prices_and_context_window.json | 56 +++++++++++++++++++
2 files changed, 112 insertions(+)
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index ea9cacbdabe..dac7af6ab7a 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -81156,5 +81156,61 @@
"output_cost_per_token": 0.0,
"source": "https://inference.baseten.co/v1/models",
"supports_vision": true
+ },
+ "gpt-6-rosalind": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_12hr": 3.75e-05,
+ "cache_creation_input_token_cost_above_12hr_above_272k_tokens": 7.5e-05,
+ "cache_creation_input_token_cost_above_272k_tokens": 3.75e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "cache_read_input_token_cost_above_272k_tokens": 3e-06,
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_above_272k_tokens": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 922000,
+ "max_output_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "output_cost_per_token_above_272k_tokens": 0.0001125,
+ "regional_processing_uplift_multiplier_eu": 1.1,
+ "regional_processing_uplift_multiplier_us": 1.1,
+ "source": "https://developers.openai.com/api/docs/pricing",
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_max_reasoning_effort": true,
+ "supports_minimal_reasoning_effort": false,
+ "supports_native_streaming": true,
+ "supports_none_reasoning_effort": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_cache_breakpoint": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_tool_search": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "supports_xhigh_reasoning_effort": true
+ },
+ "chatgpt/gpt-6-rosalind": {
+ "litellm_provider": "chatgpt",
+ "max_input_tokens": 922000,
+ "max_output_tokens": 128000,
+ "mode": "responses",
+ "source": "https://developers.openai.com/api/docs/pricing",
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supports_function_calling": true,
+ "supports_max_reasoning_effort": true,
+ "supports_minimal_reasoning_effort": false,
+ "supports_none_reasoning_effort": false,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_vision": true,
+ "supports_xhigh_reasoning_effort": true
}
}
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
index ea9cacbdabe..dac7af6ab7a 100644
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -81156,5 +81156,61 @@
"output_cost_per_token": 0.0,
"source": "https://inference.baseten.co/v1/models",
"supports_vision": true
+ },
+ "gpt-6-rosalind": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_12hr": 3.75e-05,
+ "cache_creation_input_token_cost_above_12hr_above_272k_tokens": 7.5e-05,
+ "cache_creation_input_token_cost_above_272k_tokens": 3.75e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "cache_read_input_token_cost_above_272k_tokens": 3e-06,
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_above_272k_tokens": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 922000,
+ "max_output_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "output_cost_per_token_above_272k_tokens": 0.0001125,
+ "regional_processing_uplift_multiplier_eu": 1.1,
+ "regional_processing_uplift_multiplier_us": 1.1,
+ "source": "https://developers.openai.com/api/docs/pricing",
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_max_reasoning_effort": true,
+ "supports_minimal_reasoning_effort": false,
+ "supports_native_streaming": true,
+ "supports_none_reasoning_effort": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_cache_breakpoint": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_tool_search": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "supports_xhigh_reasoning_effort": true
+ },
+ "chatgpt/gpt-6-rosalind": {
+ "litellm_provider": "chatgpt",
+ "max_input_tokens": 922000,
+ "max_output_tokens": 128000,
+ "mode": "responses",
+ "source": "https://developers.openai.com/api/docs/pricing",
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supports_function_calling": true,
+ "supports_max_reasoning_effort": true,
+ "supports_minimal_reasoning_effort": false,
+ "supports_none_reasoning_effort": false,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_vision": true,
+ "supports_xhigh_reasoning_effort": true
}
}
From 1ccb970d520aa24a078d5b2a40ccc7f5249a770c Mon Sep 17 00:00:00 2001
From: Kerry Lu
Date: Fri, 30 Oct 2026 11:43:13 -0700
Subject: [PATCH 25/87] refactor(ui): remove the deprecated Old Usage page
(#49284)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
tests/e2e/ui/fixtures/menuMappings.ts | 1 -
tests/e2e/ui/fixtures/migratedPages.ts | 6 -
tests/e2e/ui/fixtures/pages.ts | 1 -
.../e2e/ui/tests/logs/logsPagination.spec.ts | 2 +-
tests/e2e/ui/tests/usage/usagePage.spec.ts | 2 +-
ui/litellm-dashboard/eslint-suppressions.json | 28 -
.../app/(dashboard)/legacyPageRoutes.test.ts | 2 +-
.../src/app/(dashboard)/legacyPageRoutes.ts | 2 +-
.../old-usage/_components/usage.test.tsx | 257 -----
.../old-usage/_components/usage.tsx | 950 ------------------
.../src/app/(dashboard)/old-usage/page.tsx | 22 -
.../components/UsagePageView.test.tsx | 4 -
.../src/components/leftnav.test.tsx | 26 -
.../src/components/leftnav.tsx | 8 -
.../src/components/page_metadata.ts | 1 -
.../src/components/view_user_spend.tsx | 144 ---
.../tests/CreateKeyPage.expiredToken.test.tsx | 1 -
17 files changed, 4 insertions(+), 1453 deletions(-)
delete mode 100644 ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.test.tsx
delete mode 100644 ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.tsx
delete mode 100644 ui/litellm-dashboard/src/app/(dashboard)/old-usage/page.tsx
delete mode 100644 ui/litellm-dashboard/src/components/view_user_spend.tsx
diff --git a/tests/e2e/ui/fixtures/menuMappings.ts b/tests/e2e/ui/fixtures/menuMappings.ts
index d6e7ea86982..4cf90c00cd0 100644
--- a/tests/e2e/ui/fixtures/menuMappings.ts
+++ b/tests/e2e/ui/fixtures/menuMappings.ts
@@ -32,7 +32,6 @@ export const menuLabelToPage: Record = {
Budgets: Page.Budgets,
"API Playground": Page.TransformRequest,
"Tag Management": Page.TagManagement,
- "Old Usage": Page.Usage,
// Tools submenu items
"MCP Servers": Page.McpServers,
"Vector Stores": Page.VectorStores,
diff --git a/tests/e2e/ui/fixtures/migratedPages.ts b/tests/e2e/ui/fixtures/migratedPages.ts
index 5fa1f9d2444..6fa8d0c2587 100644
--- a/tests/e2e/ui/fixtures/migratedPages.ts
+++ b/tests/e2e/ui/fixtures/migratedPages.ts
@@ -118,12 +118,6 @@ export const MIGRATED_E2E_PAGES: Readonly> = {
content: { role: "heading", name: "AI Hub" },
},
new_usage: { segment: "usage", linkName: "Usage", content: { role: "heading", name: "Usage" } },
- usage: {
- segment: "old-usage",
- linkName: "Old Usage",
- group: "Experimental",
- content: { role: "tab", name: "All Up" },
- },
agents: { segment: "agents", linkName: "Agents", group: "Agentic", content: { role: "heading", name: "Agents" } },
"router-settings": {
segment: "router-settings",
diff --git a/tests/e2e/ui/fixtures/pages.ts b/tests/e2e/ui/fixtures/pages.ts
index 8210334c166..a850dcb4755 100644
--- a/tests/e2e/ui/fixtures/pages.ts
+++ b/tests/e2e/ui/fixtures/pages.ts
@@ -30,5 +30,4 @@ export enum Page {
TagManagement = "tag-management",
VectorStores = "vector-stores",
NewUsage = "new_usage",
- Usage = "usage",
}
diff --git a/tests/e2e/ui/tests/logs/logsPagination.spec.ts b/tests/e2e/ui/tests/logs/logsPagination.spec.ts
index 585eb31c37c..a611f760e8b 100644
--- a/tests/e2e/ui/tests/logs/logsPagination.spec.ts
+++ b/tests/e2e/ui/tests/logs/logsPagination.spec.ts
@@ -84,7 +84,7 @@ test.describe("Logs pagination with Aria in the header", () => {
await expect(aria).toBeFocused();
await visibleTestId(page, "pagination-first").click();
await expect(visibleTestId(page, "pagination-range")).toHaveText("Showing 1-25 of 26");
- for (const destination of [Page.Usage, Page.LlmPlayground, Page.ApiKeys]) {
+ for (const destination of [Page.NewUsage, Page.LlmPlayground, Page.ApiKeys]) {
await navigateToPage(page, destination);
await aria.click();
await expect(page.getByRole("dialog", { name: "Aria Agent", exact: true })).toBeVisible();
diff --git a/tests/e2e/ui/tests/usage/usagePage.spec.ts b/tests/e2e/ui/tests/usage/usagePage.spec.ts
index d532042c7af..2a58a1ef59d 100644
--- a/tests/e2e/ui/tests/usage/usagePage.spec.ts
+++ b/tests/e2e/ui/tests/usage/usagePage.spec.ts
@@ -11,7 +11,7 @@ import {
waitForSpendLog,
} from "../../helpers/traffic";
-/** Covers /ui/usage. The legacy /ui/old-usage view is deprecated and deliberately not covered. */
+/** Covers /ui/usage. */
const topKeysCard = (page: PlaywrightPage): Locator =>
page.locator("section").filter({ has: page.getByRole("heading", { name: "Top Virtual Keys", exact: true }) });
diff --git a/ui/litellm-dashboard/eslint-suppressions.json b/ui/litellm-dashboard/eslint-suppressions.json
index 9fce93782b2..c14f3e6f036 100644
--- a/ui/litellm-dashboard/eslint-suppressions.json
+++ b/ui/litellm-dashboard/eslint-suppressions.json
@@ -625,23 +625,6 @@
"count": 1
}
},
- "src/app/(dashboard)/old-usage/_components/usage.tsx": {
- "local/filename-pascal-case": {
- "count": 1
- },
- "max-lines": {
- "count": 1
- },
- "prefer-const": {
- "count": 6
- },
- "react-hooks/immutability": {
- "count": 1
- },
- "react-hooks/purity": {
- "count": 1
- }
- },
"src/app/(dashboard)/playground/components/chat_ui/AdditionalModelSettings.tsx": {
"react-hooks/set-state-in-effect": {
"count": 2
@@ -2090,17 +2073,6 @@
"count": 1
}
},
- "src/components/view_user_spend.tsx": {
- "local/filename-pascal-case": {
- "count": 1
- },
- "prefer-const": {
- "count": 3
- },
- "react-hooks/set-state-in-effect": {
- "count": 2
- }
- },
"src/contexts/AuthContext.tsx": {
"react-hooks/set-state-in-effect": {
"count": 1
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.test.ts b/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.test.ts
index 6b02b79b04c..315d988db54 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.test.ts
+++ b/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.test.ts
@@ -10,7 +10,7 @@ describe("legacyPageRedirectHref", () => {
expect(redirect("page=models")).toBe("/ui/models-and-endpoints");
expect(redirect("page=llm-playground")).toBe("/ui/playground");
expect(redirect("page=new_usage")).toBe("/ui/usage");
- expect(redirect("page=usage")).toBe("/ui/old-usage");
+ expect(redirect("page=usage")).toBe("/ui/usage");
});
it("keeps the older aliases for renamed pages", () => {
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.ts b/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.ts
index e3025266ddb..fe054c7b39a 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.ts
+++ b/ui/litellm-dashboard/src/app/(dashboard)/legacyPageRoutes.ts
@@ -36,7 +36,7 @@ const LEGACY_PAGE_ROUTES: ReadonlyMap = new Map(
"logging-and-alerts": "logging-and-alerts",
"model-hub-table": "model-hub-table",
new_usage: "usage",
- usage: "old-usage",
+ usage: "usage",
"cost-optimization": "cost-optimization",
"model-insights": "model-insights",
agents: "agents",
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.test.tsx b/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.test.tsx
deleted file mode 100644
index 0e4455ee912..00000000000
--- a/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.test.tsx
+++ /dev/null
@@ -1,257 +0,0 @@
-import React from "react";
-import { describe, it, expect, vi, beforeEach } from "vitest";
-import { act, screen, waitFor, within } from "@testing-library/react";
-import userEvent from "@testing-library/user-event";
-import { renderWithProviders } from "../../../../../tests/test-utils";
-import UsagePage from "./usage";
-
-const networking = vi.hoisted(() => ({
- adminSpendLogsCall: vi.fn(),
- adminTopKeysCall: vi.fn(),
- adminTopModelsCall: vi.fn(),
- adminTopEndUsersCall: vi.fn(),
- teamSpendLogsCall: vi.fn(),
- tagsSpendLogsCall: vi.fn(),
- allTagNamesCall: vi.fn(),
- adminspendByProvider: vi.fn(),
- adminGlobalActivity: vi.fn(),
- adminGlobalActivityPerModel: vi.fn(),
- getProxyUISettings: vi.fn(),
- modelAvailableCall: vi.fn(),
- keyInfoV1Call: vi.fn(),
-}));
-
-vi.mock("@/components/networking", () => networking);
-vi.mock("../../../../components/networking", () => networking);
-
-vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
- default: () => ({
- accessToken: "sk-test",
- token: "tok",
- userRole: "Admin",
- userId: "u1",
- premiumUser: true,
- }),
-}));
-
-const UNLIMITED_SETTINGS = { DISABLE_EXPENSIVE_DB_QUERIES: false, NUM_SPEND_LOGS_ROWS: 10 };
-
-const renderUsage = (overrides: Partial> = {}) =>
- renderWithProviders(
- ,
- );
-
-// Width of this window is guarded by "proves the flush window is wide enough".
-const flushPendingRequests = async () => {
- await act(async () => {
- await new Promise((resolve) => setTimeout(resolve, 0));
- await new Promise((resolve) => setTimeout(resolve, 0));
- });
-};
-
-beforeEach(() => {
- vi.clearAllMocks();
- networking.getProxyUISettings.mockResolvedValue(UNLIMITED_SETTINGS);
- networking.adminSpendLogsCall.mockResolvedValue([{ date: "2026-07-01", spend: 12.5 }]);
- networking.adminTopKeysCall.mockResolvedValue([
- { api_key: "sk-abcdefghijk", key_alias: "prod-key", total_spend: 9.5 },
- ]);
- networking.adminTopModelsCall.mockResolvedValue([{ model: "gpt-5.1", total_spend: 7.25 }]);
- networking.adminTopEndUsersCall.mockResolvedValue([
- { end_user: "customer-alpha", total_spend: 3.5, total_count: 42 },
- ]);
- networking.teamSpendLogsCall.mockResolvedValue({
- daily_spend: [{ date: "2026-07-01", "team-a": 5 }],
- teams: ["team-a"],
- total_spend_per_team: [{ team_id: "team-a", total_spend: 5 }],
- });
- networking.tagsSpendLogsCall.mockResolvedValue({ spend_per_tag: [{ name: "prod", spend: 4 }] });
- networking.allTagNamesCall.mockResolvedValue({ tag_names: ["prod", "staging"] });
- networking.adminspendByProvider.mockResolvedValue([{ provider: "openai", spend: 6.75 }]);
- networking.adminGlobalActivity.mockResolvedValue({
- sum_api_requests: 120,
- sum_total_tokens: 4500,
- daily_data: [{ date: "2026-07-01", api_requests: 120, total_tokens: 4500 }],
- });
- networking.adminGlobalActivityPerModel.mockResolvedValue([]);
- networking.modelAvailableCall.mockResolvedValue({ data: [] });
- networking.keyInfoV1Call.mockResolvedValue({ info: {} });
-});
-
-describe("old usage page", () => {
- describe("when the proxy has disabled expensive DB queries", () => {
- beforeEach(() => {
- networking.getProxyUISettings.mockResolvedValue({
- DISABLE_EXPENSIVE_DB_QUERIES: true,
- NUM_SPEND_LOGS_ROWS: 2500000,
- });
- });
-
- it("shows the database query limit warning instead of the usage dashboard", async () => {
- renderUsage();
-
- expect(await screen.findByText("Database Query Limit Reached")).toBeInTheDocument();
- expect(screen.getByText(/SpendLogs in DB has/)).toHaveTextContent("2500000");
- expect(screen.getByText(/Please follow our guide to view usage when SpendLogs has more than 1M rows/i));
- expect(screen.queryByRole("tab", { name: "All Up" })).not.toBeInTheDocument();
- });
-
- it("links to the cost tracking guide in a new tab", async () => {
- renderUsage();
-
- const link = await screen.findByRole("link", { name: "View Usage Guide" });
- expect(link).toHaveAttribute("href", "https://docs.litellm.ai/docs/proxy/cost_tracking");
- expect(link).toHaveAttribute("target", "_blank");
- });
-
- it("skips every expensive usage query", async () => {
- renderUsage();
-
- await screen.findByText("Database Query Limit Reached");
- await waitFor(() => expect(networking.getProxyUISettings).toHaveBeenCalled());
-
- expect(networking.adminSpendLogsCall).not.toHaveBeenCalled();
- expect(networking.adminspendByProvider).not.toHaveBeenCalled();
- expect(networking.adminTopKeysCall).not.toHaveBeenCalled();
- expect(networking.adminTopModelsCall).not.toHaveBeenCalled();
- expect(networking.adminGlobalActivity).not.toHaveBeenCalled();
- expect(networking.adminGlobalActivityPerModel).not.toHaveBeenCalled();
- expect(networking.teamSpendLogsCall).not.toHaveBeenCalled();
- expect(networking.adminTopEndUsersCall).not.toHaveBeenCalled();
- expect(networking.tagsSpendLogsCall).not.toHaveBeenCalled();
- });
- });
-
- describe("as an admin", () => {
- it("renders the admin tabs", async () => {
- renderUsage();
-
- expect(await screen.findByRole("tab", { name: "All Up" })).toBeInTheDocument();
- expect(screen.getByRole("tab", { name: "Team Based Usage" })).toBeInTheDocument();
- expect(screen.getByRole("tab", { name: "Customer Usage" })).toBeInTheDocument();
- expect(screen.getByRole("tab", { name: "Tag Based Usage" })).toBeInTheDocument();
- });
-
- it("renders the cost panel cards", async () => {
- renderUsage();
-
- expect(await screen.findByText("Monthly Spend")).toBeInTheDocument();
- expect(screen.getByText("Top Virtual Keys")).toBeInTheDocument();
- expect(screen.getByText("Top Models")).toBeInTheDocument();
- expect(screen.getByText("Spend by Provider")).toBeInTheDocument();
- });
-
- it("lists spend by provider in a table", async () => {
- renderUsage();
-
- const providerCell = await screen.findByText("openai");
- const row = providerCell.closest("tr");
- expect(row).not.toBeNull();
- expect(within(row as HTMLElement).getByText("$6.75")).toBeInTheDocument();
- expect(screen.getByRole("columnheader", { name: "Provider" })).toBeInTheDocument();
- });
-
- it("shows the customer usage table when its tab is selected", async () => {
- const user = userEvent.setup();
- renderUsage();
-
- await user.click(await screen.findByRole("tab", { name: "Customer Usage" }));
-
- const customerCell = await screen.findByText("customer-alpha");
- const row = customerCell.closest("tr");
- expect(row).not.toBeNull();
- expect(within(row as HTMLElement).getByText("$3.50")).toBeInTheDocument();
- expect(within(row as HTMLElement).getByText("42")).toBeInTheDocument();
- expect(screen.getByRole("columnheader", { name: "Total Events" })).toBeInTheDocument();
- });
-
- it("shows the tag spend panel when its tab is selected", async () => {
- const user = userEvent.setup();
- renderUsage();
-
- await user.click(await screen.findByRole("tab", { name: "Tag Based Usage" }));
-
- expect(await screen.findByText("Spend Per Tag")).toBeInTheDocument();
- });
-
- it("shows the team spend panel when its tab is selected", async () => {
- const user = userEvent.setup();
- renderUsage();
-
- await user.click(await screen.findByRole("tab", { name: "Team Based Usage" }));
-
- expect(await screen.findByText("Total Spend Per Team")).toBeInTheDocument();
- expect(screen.getByText("Daily Spend Per Team")).toBeInTheDocument();
- });
- });
-
- // org_admin is an organization membership role; those users reach the UI as "Internal User".
- describe.each(["Internal User", "Internal Viewer", "internal_user", "internal_user_viewer", "Org Admin"])(
- "as %s",
- (userRole) => {
- it("shows the admin-only notice instead of the usage dashboard", async () => {
- renderUsage({ userRole });
-
- expect(await screen.findByText(/Proxy-wide usage is only available to admin users/i)).toBeInTheDocument();
- expect(screen.queryByRole("tab", { name: "All Up" })).not.toBeInTheDocument();
- });
-
- it("fires no /global/spend or /global/activity request", async () => {
- renderUsage({ userRole });
-
- await screen.findByText(/Proxy-wide usage is only available to admin users/i);
- await flushPendingRequests();
-
- expect(networking.getProxyUISettings).not.toHaveBeenCalled();
- expect(networking.adminSpendLogsCall).not.toHaveBeenCalled();
- expect(networking.adminTopKeysCall).not.toHaveBeenCalled();
- expect(networking.adminTopModelsCall).not.toHaveBeenCalled();
- expect(networking.adminTopEndUsersCall).not.toHaveBeenCalled();
- expect(networking.teamSpendLogsCall).not.toHaveBeenCalled();
- expect(networking.tagsSpendLogsCall).not.toHaveBeenCalled();
- expect(networking.allTagNamesCall).not.toHaveBeenCalled();
- expect(networking.adminspendByProvider).not.toHaveBeenCalled();
- expect(networking.adminGlobalActivity).not.toHaveBeenCalled();
- expect(networking.adminGlobalActivityPerModel).not.toHaveBeenCalled();
- });
- },
- );
-
- describe("the admin-only gate", () => {
- it("proves the flush window is wide enough to catch a leaked request", async () => {
- renderUsage({ userRole: "Admin" });
-
- await flushPendingRequests();
-
- expect(networking.getProxyUISettings).toHaveBeenCalled();
- expect(networking.adminSpendLogsCall).toHaveBeenCalled();
- expect(networking.tagsSpendLogsCall).toHaveBeenCalled();
- expect(networking.adminGlobalActivity).toHaveBeenCalled();
- });
-
- it("still lets an admin through, so the notice is a real gate and not a dead branch", async () => {
- renderUsage({ userRole: "Admin" });
-
- expect(await screen.findByRole("tab", { name: "All Up" })).toBeInTheDocument();
- expect(screen.queryByText(/Proxy-wide usage is only available to admin users/i)).not.toBeInTheDocument();
- await waitFor(() => expect(networking.adminSpendLogsCall).toHaveBeenCalled());
- });
-
- it("does not put the session token in the provider spend query", async () => {
- renderUsage({ userRole: "Admin", token: "session-jwt-value" });
-
- await waitFor(() => expect(networking.adminspendByProvider).toHaveBeenCalled());
- const callArgs = networking.adminspendByProvider.mock.calls[0];
- expect(callArgs).not.toContain("session-jwt-value");
- expect(callArgs[0]).toBe("sk-test");
- });
- });
-});
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.tsx b/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.tsx
deleted file mode 100644
index 15b2a30b50c..00000000000
--- a/ui/litellm-dashboard/src/app/(dashboard)/old-usage/_components/usage.tsx
+++ /dev/null
@@ -1,950 +0,0 @@
-import React, { useState, useEffect } from "react";
-
-import ViewUserSpend from "@/components/view_user_spend";
-import { ProxySettings } from "@/app/(dashboard)/hooks/proxySettings/useProxySettings";
-import AdvancedDatePicker from "@/components/shared/advanced_date_picker";
-import { Button } from "@/components/ui/button";
-import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
-import {
- Combobox,
- ComboboxChip,
- ComboboxChips,
- ComboboxChipsInput,
- ComboboxContent,
- ComboboxEmpty,
- ComboboxItem,
- ComboboxList,
- ComboboxValue,
- useComboboxAnchor,
-} from "@/components/ui/combobox";
-import { Meter, MeterIndicator, MeterTrack } from "@/components/shared/Meter";
-import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
-import {
- NUMERIC_CELL_CLASS,
- Table,
- TableBody,
- TableCell,
- TableHead,
- TableHeader,
- TableRow,
-} from "@/components/ui/table";
-import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
-import { AreaChart, BarChart, DonutChart } from "@/components/shared/charts";
-
-import {
- adminSpendLogsCall,
- adminTopKeysCall,
- adminTopModelsCall,
- adminTopEndUsersCall,
- teamSpendLogsCall,
- tagsSpendLogsCall,
- allTagNamesCall,
- adminspendByProvider,
- adminGlobalActivity,
- adminGlobalActivityPerModel,
- getProxyUISettings,
-} from "@/components/networking";
-import TopKeyView from "@/components/UsagePage/components/EntityUsage/TopKeyView";
-import { MoneyCell } from "@/components/shared/table_cells";
-import { hasCapability } from "@/utils/capabilities";
-import { formatNumberWithCommas } from "@/utils/dataUtils";
-
-interface UsagePageProps {
- accessToken: string | null;
- token: string | null;
- userRole: string | null;
- userID: string | null;
- keys: any[] | null;
- premiumUser: boolean;
-}
-
-interface GlobalActivityData {
- sum_api_requests: number;
- sum_total_tokens: number;
- daily_data: { date: string; api_requests: number; total_tokens: number }[];
-}
-
-const EMPTY_GLOBAL_ACTIVITY: GlobalActivityData = { sum_api_requests: 0, sum_total_tokens: 0, daily_data: [] };
-
-type UsageDateRange = { from?: Date; to?: Date };
-
-type TeamSpendTotal = { name: string; value: number };
-
-type TagOption = { value: string; label: string; disabled: boolean };
-
-const ALL_TAGS = "all-tags";
-
-const isAdminOrAdminViewer = (role: string | null): boolean => {
- if (role === null) return false;
- return role === "Admin" || role === "Admin Viewer";
-};
-
-const TeamSpendBarList: React.FC<{ data: TeamSpendTotal[] }> = ({ data }) => {
- const max = Math.max(0, ...data.map((team) => team.value));
-
- return (
-
- {data.map((team) => (
-
-
{team.name}
-
-
-
-
-
-
- {formatNumberWithCommas(team.value, 2)}
-
-
- ))}
-
-
-
- Usage
-
-
-
- Proxy-wide usage is only available to admin users. Your own usage is on the Usage page.
-
-
-
-
-
-
- Database Query Limit Reached
-
-
-
- SpendLogs in DB has {proxySettings.NUM_SPEND_LOGS_ROWS} rows.
-
- Please follow our guide to view usage when SpendLogs has more than 1M rows.
-
-
-
-
-
-
- All Up
-
- {isAdminOrAdminViewer(userRole) && (
- <>
- Team Based Usage
- Customer Usage
- Tag Based Usage
-
- )}
-
-
-
-
-
- Cost
- Activity
-
-
-
-
-
-
- Project Spend {new Date().toLocaleString("default", { month: "long" })} 1 -{" "}
- {new Date(new Date().getFullYear(), new Date().getMonth() + 1, 0).getDate()}
-
-
-
-
-
-
- Monthly Spend
-
-
-
-
-
-
-
-
-
- Top Virtual Keys
-
-
- {}} />
-
-
-
-
-
-
- Top Models
-
-
- `$${formatNumberWithCommas(value, 2)}`}
- />
-
-
-
-
-
-
-
- Spend by Provider
-
-
-
-
- `$${formatNumberWithCommas(value, 2)}`}
- />
-
-
-
-
-
- Provider
- Spend
-
-
-
- {spendByProvider.map((provider) => (
-
- {provider.provider}
-
-
-
-
- ))}
-
-
-
-
-
-
-
-
- All Up
-
-
-
-
-
- API Requests {valueFormatterNumbers(globalActivity.sum_api_requests)}
-
-
-
-
-
- Tokens {valueFormatterNumbers(globalActivity.sum_total_tokens)}
-
-
-
-
-
- {globalActivityPerModel.map((globalActivity, index) => (
-
-
- {globalActivity.model}
-
-
-
-
-
- API Requests {valueFormatterNumbers(globalActivity.sum_api_requests)}
-
-
-
-
-
- Tokens {valueFormatterNumbers(globalActivity.sum_total_tokens)}
-
-
-
-
- ))}
-
-
-
-
- Total Spend Per Team
-
-
-
-
-
-
-
- Daily Spend Per Team
-
-
-
-
-
-
-
- Customers of your LLM API calls. Tracked when a `user` param is passed in your LLM calls{" "}
-
- docs here
-
-
-
-
-
{
- setDateValue(value);
- updateEndUserData(value.from, value.to, null);
- }}
- />
-
-
-
Select Key
-
-
-
-
-
-
- Customer
- Spend
- Total Events
-
-
-
-
- {topUsers?.map((user: any, index: number) => (
-
- {user.end_user}
-
-
-
- {user.total_count}
-
- ))}
-
-
-
-
-
{
- setDateValue(value);
- updateTagSpendData(value.from, value.to);
- }}
- />
-
-
-
- selectedTags.includes(option.value))}
- onValueChange={(options: TagOption[]) => setSelectedTags(options.map((option) => option.value))}
- isItemEqualToValue={(a: TagOption, b: TagOption) => a.value === b.value}
- itemToStringLabel={(option: TagOption) => option.label}
- >
- }>
-
- {(options: TagOption[]) =>
- options.map((option) => (
-
- {option.label}
-
- ))
- }
-
-
-
-
- No tags found
-
- {(option: TagOption) => (
-
- {option.label}
-
- )}
-
-
-
-
-
-
-
-
- Spend Per Tag
-
-
-
- Get Started by Tracking cost per tag{" "}
-
- here
-
-
-
-
-
-
-
-
View User Spend
,
-}));
-
vi.mock("@/components/UsagePage/components/EntityUsage/TopKeyView", () => ({
default: () => Top Keys
,
}));
diff --git a/ui/litellm-dashboard/src/components/leftnav.test.tsx b/ui/litellm-dashboard/src/components/leftnav.test.tsx
index 6cf05af1ed1..f3bff85e2d5 100644
--- a/ui/litellm-dashboard/src/components/leftnav.test.tsx
+++ b/ui/litellm-dashboard/src/components/leftnav.test.tsx
@@ -444,30 +444,6 @@ describe("Sidebar (leftnav)", () => {
});
expect(screen.queryByText("Prompts")).not.toBeInTheDocument();
});
-
- it("should hide Old Usage from internal users while keeping other Experimental children", async () => {
- mockUseAuthorized.mockReturnValue(internalAuth);
- renderWithProviders();
-
- act(() => {
- fireEvent.click(screen.getByText("Experimental"));
- });
- await waitFor(() => {
- expect(screen.getByText("API Playground")).toBeInTheDocument();
- });
- expect(screen.queryByText("Old Usage")).not.toBeInTheDocument();
- });
-
- it("should show Old Usage to admins", async () => {
- renderWithProviders();
-
- act(() => {
- fireEvent.click(screen.getByText("Experimental"));
- });
- await waitFor(() => {
- expect(screen.getByText("Old Usage")).toBeInTheDocument();
- });
- });
});
// Workflow Runs, Memory and Guardrails Monitor render a shell and then 401
@@ -704,7 +680,6 @@ describe("Sidebar (leftnav)", () => {
expectHref("Models + Endpoints", "/ui/models-and-endpoints");
expectHref("Usage", "/ui/usage");
expectHref("API Reference", "/ui/api-reference");
- expectHref("Old Usage", "/ui/old-usage");
});
it("never links a leaf to the legacy ?page= switch", () => {
@@ -760,7 +735,6 @@ describe("getBreadcrumb", () => {
it("resolves routes whose segment differs from the sidebar page id", () => {
expect(getBreadcrumb("/ui/models-and-endpoints")).toEqual({ section: "AI Gateway", title: "Models + Endpoints" });
expect(getBreadcrumb("/ui/usage")).toEqual({ section: "Observability", title: "Usage" });
- expect(getBreadcrumb("/ui/old-usage")).toEqual({ section: "Developer Tools", title: "Old Usage" });
});
it("titles the dashboard root as Home", () => {
diff --git a/ui/litellm-dashboard/src/components/leftnav.tsx b/ui/litellm-dashboard/src/components/leftnav.tsx
index 6795cf1c7d1..2cf6f8219db 100644
--- a/ui/litellm-dashboard/src/components/leftnav.tsx
+++ b/ui/litellm-dashboard/src/components/leftnav.tsx
@@ -414,14 +414,6 @@ const menuGroups: MenuGroup[] = [
icon: ,
roles: all_admin_roles,
},
- {
- key: "4",
- page: "usage",
- route: "old-usage",
- label: "Old Usage",
- icon: ,
- roles: rolesWithCapability("viewGlobalSpend"),
- },
],
},
],
diff --git a/ui/litellm-dashboard/src/components/page_metadata.ts b/ui/litellm-dashboard/src/components/page_metadata.ts
index 18f136c3b28..97220a0ef76 100644
--- a/ui/litellm-dashboard/src/components/page_metadata.ts
+++ b/ui/litellm-dashboard/src/components/page_metadata.ts
@@ -41,7 +41,6 @@ export const pageDescriptions: Record = {
"tag-management": "Organize resources with tags",
prompts: "Manage and version prompt templates",
skills: "Browse and manage Claude Code skills",
- usage: "View legacy usage dashboard",
"router-settings": "Configure routing and load balancing settings",
telemetry: "Choose which anonymous usage telemetry this proxy collects and see the last report",
"logging-and-alerts": "Set up logging and alert configurations",
diff --git a/ui/litellm-dashboard/src/components/view_user_spend.tsx b/ui/litellm-dashboard/src/components/view_user_spend.tsx
deleted file mode 100644
index 655fc092673..00000000000
--- a/ui/litellm-dashboard/src/components/view_user_spend.tsx
+++ /dev/null
@@ -1,144 +0,0 @@
-"use client";
-import React, { useEffect, useState } from "react";
-import { modelAvailableCall } from "./networking";
-import { formatNumberWithCommas } from "@/utils/dataUtils";
-import useAuthorized from "@/app/(dashboard)/hooks/useAuthorized";
-
-// Define the props type
-interface ViewUserSpendProps {
- userSpend: number | null;
- userMaxBudget: number | null;
- selectedTeam: any | null;
-}
-const ViewUserSpend: React.FC = ({ userSpend, userMaxBudget, selectedTeam }) => {
- const { accessToken, userRole, userId: userID } = useAuthorized();
- let [spend, setSpend] = useState(userSpend !== null ? userSpend : 0.0);
- const [maxBudget, setMaxBudget] = useState(
- selectedTeam ? Number(formatNumberWithCommas(selectedTeam.max_budget, 4)) : null,
- );
- useEffect(() => {
- if (selectedTeam) {
- if (selectedTeam.team_alias === "Default Team") {
- setMaxBudget(userMaxBudget);
- } else {
- let setMaxBudgetFlag = false;
- if (selectedTeam.team_memberships) {
- /**
- * What 'team_memberships' looks like:
- * "team_memberships": [
- * {
- * "user_id": "2c315de3-e7ce-4269-b73e-b039a06187b1",
- * "team_id": "test-team_515e6f42-ded2-4f0d-8919-0a1f43c5a45f",
- * "budget_id": "0880769f-716a-4149-ab19-7f7651ad4db5",
- * "litellm_budget_table": {
- "soft_budget": null,
- "max_budget": 20.0,
- "max_parallel_requests": null,
- "tpm_limit": null,
- "rpm_limit": null,
- "model_max_budget": null,
- "budget_duration": null
- }
- */
- for (const member of selectedTeam.team_memberships) {
- if (
- member.user_id === userID &&
- "max_budget" in member.litellm_budget_table &&
- member.litellm_budget_table.max_budget !== null
- ) {
- setMaxBudget(member.litellm_budget_table.max_budget);
- setMaxBudgetFlag = true;
- }
- }
- }
- if (!setMaxBudgetFlag) {
- setMaxBudget(selectedTeam.max_budget);
- }
- }
- } else {
- setMaxBudget(userMaxBudget);
- }
- }, [selectedTeam, userMaxBudget]);
- const [userModels, setUserModels] = useState([]);
- useEffect(() => {
- const fetchData = async () => {
- if (!accessToken || !userID || !userRole) {
- return;
- }
- };
- const fetchUserModels = async () => {
- try {
- if (userID === null || userRole === null) {
- return;
- }
-
- if (accessToken !== null) {
- const model_available = await modelAvailableCall(accessToken, userID, userRole);
- let available_model_names = model_available["data"].map((element: { id: string }) => element.id);
- setUserModels(available_model_names);
- }
- } catch (error) {
- console.error("Error fetching user models:", error);
- }
- };
-
- fetchUserModels();
- fetchData();
- }, [userRole, accessToken, userID]);
-
- useEffect(() => {
- if (userSpend !== null) {
- setSpend(userSpend);
- }
- }, [userSpend]);
-
- // logic to decide what models to display
- let modelsToDisplay = [];
- if (selectedTeam && selectedTeam.models) {
- modelsToDisplay = selectedTeam.models;
- }
-
- // check if "all-proxy-models" is in modelsToDisplay
- if (modelsToDisplay && modelsToDisplay.includes("all-proxy-models")) {
- modelsToDisplay = userModels;
- } else if (modelsToDisplay && modelsToDisplay.includes("all-team-models")) {
- modelsToDisplay = selectedTeam.models;
- } else if (modelsToDisplay && modelsToDisplay.length === 0) {
- modelsToDisplay = userModels;
- }
-
- const displayMaxBudget = maxBudget !== null ? `$${formatNumberWithCommas(Number(maxBudget), 4)} limit` : "No limit";
-
- const roundedSpend = spend !== undefined ? formatNumberWithCommas(spend, 4) : null;
-
- return (
-
-
-
-
Total Spend
-
${roundedSpend}
-
-
-
Max Budget
-
{displayMaxBudget}
-
-
- {/*
-
- Team Models
-
-
- {modelsToDisplay.map((model: string) => (
-
- {model}
-
- ))}
-
-
-
-
*/}
-