diff --git a/AGENTS.md b/AGENTS.md
index 01127303da3..d8de447be90 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -52,7 +52,6 @@ If you ever write any human-facing text (pull requests, issues, commit messages,
- don't use "—". Don't use ";" or ":" unless it would be nonsensical not to. Instead, reach for ",", ".", conjunction words, etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them also feels AI-y. A word cap does not penalize you for adding more sentences
- don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
- unless explicitly asked, don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
-- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a ".". Exception is that docs (i.e., litellm-docs) should have trailing "."
- don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
- use plain, simple, everyday engineering language: the common phrase engineers actually say over rare compact phrasing, in grammatically complete sentences. When explicitly asked to use bullets or ordered lists and structure legitimately helps the reader, prefer nested bullets (any depth is fine) over dense lines in a flat structure
diff --git a/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md b/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md
index e1f606a2eb7..538a5f8a404 100644
--- a/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md
+++ b/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md
@@ -46,7 +46,43 @@ For a managed workspace rollout, use OpenAI's [gateway configuration guide](http
## Native compatibility and Pi
-For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials but does not need a model deployment or resolve model aliases. Restricted keys must allow the upstream model name.
+For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials. A ChatGPT model keeps its upstream name and needs no deployment of its own.
+
+### Mix ChatGPT and LiteLLM models
+
+On the native route, Codex's model picker shows OpenAI's catalog first and then the other chat and responses models the virtual key can use. Pick one of those and Codex's requests go through LiteLLM's `/v1/responses` with the key's usual routing, budgets, guardrails, and logging. The proxy removes the ChatGPT bearer and account header from that request, so the other provider and your callbacks never receive them. A name is routed this way when another provider serves it as an exact `model_name`, through a router `model_group_alias`, or as a team's public model name. A ChatGPT model, a name that only a wildcard deployment matches, or a name OpenAI's catalog lists for the caller's ChatGPT account keeps the passthrough, so a model OpenAI adds later still uses the subscription even when a deployment of another provider has the same name. The proxy learns the account's list from the catalog Codex fetches, or asks OpenAI with the caller's login. When it cannot get the list, for example because OpenAI is down, the request times out, or the client is not Codex, every request keeps the passthrough for the next minute, as it did before LiteLLM models were added to the picker. When the key can use many models, the picker lists the ones that fit in Codex's catalog size limit, and the proxy logs a warning with how many it left out.
+
+Codex's picker sends a `none` reasoning effort for a LiteLLM model whose catalog entry lists no effort levels. With `litellm_settings: drop_params: true`, the proxy drops it for a model that does not support `none`, and the provider applies its own default. A model whose cost map entry or `model_info` sets `supports_none_reasoning_effort: true` keeps it. Without `drop_params`, the effort reaches the provider unchanged.
+
+Codex also sends its ChatGPT app tools with every turn, which on a LiteLLM model adds a few hundred thousand prompt tokens to each turn. Set `apps = false` under `[features]` in Codex's `config.toml` to turn them off.
+
+Only a key that may call `/v1/responses` gets these models. A key whose `allowed_routes` covers only the native routes sees OpenAI's catalog unchanged, and every request it sends uses the subscription.
+
+Codex sends earlier reasoning items back with each turn. The proxy tags the reasoning that a LiteLLM model returns on this route, OpenAI and Azure deployments included. When a session switches to a ChatGPT model, the proxy drops those items before forwarding the request to OpenAI, which would reject them. Messages, tool calls, and ChatGPT's own reasoning are kept, and a LiteLLM model still gets its own reasoning back.
+
+When the catalog adds LiteLLM models for a key, its `ETag` is a digest of OpenAI's ETag and those models, and that key's responses carry the same value in `X-Models-Etag`. Codex keeps its cached catalog while the two match and fetches it again when OpenAI's catalog or the key's models change. A key whose catalog adds no LiteLLM models keeps OpenAI's ETags.
+
+### Restrict native models with an access group
+
+A key with a model allowlist needs a rule for ChatGPT names that OpenAI adds over time. Put the subscription wildcards in an access group under `chatgpt/` names:
+
+```yaml
+model_list:
+ - model_name: chatgpt/gpt-*
+ litellm_params:
+ model: chatgpt/gpt-*
+ model_info:
+ access_groups: ["chatgpt-native"]
+ - model_name: chatgpt/codex-*
+ litellm_params:
+ model: chatgpt/codex-*
+ model_info:
+ access_groups: ["chatgpt-native"]
+```
+
+Then add `chatgpt-native` to the key's models, next to any other model names it should reach. On the native route, a key that holds only the `chatgpt/` form of a name is checked as that form, so a request for `gpt-5.6-sol` is allowed when the key may call `chatgpt/gpt-5.6-sol`. The key can call every ChatGPT model that matches the wildcards, including ones the proxy configuration does not name.
+
+On `/v1` these wildcards match only a request that asks for a `chatgpt/` name, so a request for a plain `gpt-` or `codex-` name still goes where it went before, such as a `*` catch-all deployment. A `/v1` request for a `chatgpt/` name without a ChatGPT login falls back to the proxy's own saved login if it has one, so keep that login unset as described above.
### Configure Pi
@@ -71,7 +107,7 @@ Keep Pi's subscription login as the provider credential. The header value names
Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
-The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key must allow that name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
+The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key needs the `chatgpt-native` access group above or that exact name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
diff --git a/litellm-rust/crates/router-core/src/error.rs b/litellm-rust/crates/router-core/src/error.rs
index 926772f435c..1b718ed2c12 100644
--- a/litellm-rust/crates/router-core/src/error.rs
+++ b/litellm-rust/crates/router-core/src/error.rs
@@ -284,6 +284,7 @@ impl Error {
strum::EnumString,
strum::IntoStaticStr,
strum::VariantArray,
+ serde_with::SerializeDisplay,
)]
pub enum FailureKind {
#[strum(serialize = "RateLimitError")]
@@ -471,6 +472,17 @@ mod tests {
assert_eq!(<&'static str>::from(kind), wire);
}
+ #[rstest]
+ #[case::rate_limit(FailureKind::RateLimit, "RateLimitError")]
+ #[case::timeout(FailureKind::Timeout, "Timeout")]
+ #[case::api_connection(FailureKind::ApiConnection, "APIConnectionError")]
+ fn failure_kind_serializes_as_its_exception_name(
+ #[case] kind: FailureKind,
+ #[case] wire: &str,
+ ) {
+ assert_eq!(serde_json::to_value(kind).unwrap(), serde_json::json!(wire));
+ }
+
#[rstest]
#[case::empty("")]
#[case::lowercase("ratelimiterror")]
diff --git a/litellm-rust/crates/router-core/src/host.rs b/litellm-rust/crates/router-core/src/host.rs
index cfc97480bf1..0ef37873e6e 100644
--- a/litellm-rust/crates/router-core/src/host.rs
+++ b/litellm-rust/crates/router-core/src/host.rs
@@ -342,7 +342,10 @@ pub struct Output {
}
/// How a provider blocked a response the call otherwise served.
-#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::IntoStaticStr, strum::EnumString)]
+#[derive(
+ Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, strum::IntoStaticStr, strum::EnumString,
+)]
+#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
pub enum Blocked {
/// A chat response whose first choice was content-filtered (or that has no choices).
@@ -402,8 +405,10 @@ pub enum SelectStage {
OverrideCheck(RoutingStrategy),
}
-/// Routing events, delivered through [`Host::observe`].
-#[derive(Clone, Debug, PartialEq)]
+/// Routing events, delivered through [`Host::observe`]. They serialize as JSON objects tagged by
+/// `event` (the variant name), durations in seconds.
+#[derive(Clone, Debug, PartialEq, serde::Serialize)]
+#[serde(tag = "event")]
pub enum RouterEvent {
/// A pick for `model` (the level's group) begins: its route phase opens.
PickStarted {
@@ -413,6 +418,7 @@ pub enum RouterEvent {
model_group: String,
deployment_id: String,
/// How long the pick took (`_acompletion`'s `async_get_available_deployment` service event).
+ #[serde(serialize_with = "seconds")]
pick: Duration,
},
Succeeded {
@@ -425,11 +431,13 @@ pub enum RouterEvent {
},
CooldownSet {
deployment_id: String,
+ #[serde(serialize_with = "seconds")]
cooldown: Duration,
},
Retry {
model_group: String,
retry: u32,
+ #[serde(serialize_with = "seconds")]
delay: Duration,
},
Fallback {
@@ -584,6 +592,13 @@ impl PickError {
}
}
+/// A pick's error serializes as its message.
+impl serde::Serialize for PickError {
+ fn serialize(&self, serializer: S) -> Result {
+ serializer.collect_str(&self.0)
+ }
+}
+
impl PartialEq for PickError {
fn eq(&self, other: &Self) -> bool {
format!("{:?}", self.0) == format!("{:?}", other.0)
@@ -591,7 +606,8 @@ impl PartialEq for PickError {
}
/// Which error a [`RouterEvent::RetryLogged`] records.
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq, serde::Serialize)]
+#[serde(tag = "cause", content = "error", rename_all = "snake_case")]
pub enum RetryCause {
/// The last attempt's failure.
Attempt,
@@ -745,3 +761,83 @@ pub trait Host: Send + Sync {
Ok(Value::Null)
}
}
+
+fn seconds(duration: &Duration, serializer: S) -> Result {
+ serializer.serialize_f64(duration.as_secs_f64())
+}
+
+#[cfg(test)]
+mod tests {
+ use rstest::rstest;
+ use serde_json::{Value, json};
+
+ use super::*;
+
+ #[rstest]
+ #[case::selected(
+ RouterEvent::Selected {
+ model_group: "gpt-4o".to_owned(),
+ deployment_id: "azure-eastus".to_owned(),
+ pick: Duration::from_millis(1500),
+ },
+ json!({"event": "Selected", "model_group": "gpt-4o", "deployment_id": "azure-eastus", "pick": 1.5})
+ )]
+ #[case::attempt_failed(
+ RouterEvent::AttemptFailed {
+ deployment_id: "openai".to_owned(),
+ kind: FailureKind::RateLimit,
+ status: Some(429),
+ },
+ json!({"event": "AttemptFailed", "deployment_id": "openai", "kind": "RateLimitError", "status": 429})
+ )]
+ #[case::cooldown_set(
+ RouterEvent::CooldownSet { deployment_id: "openai".to_owned(), cooldown: Duration::from_secs(30) },
+ json!({"event": "CooldownSet", "deployment_id": "openai", "cooldown": 30.0})
+ )]
+ #[case::retry(
+ RouterEvent::Retry { model_group: "gpt-4o".to_owned(), retry: 1, delay: Duration::ZERO },
+ json!({"event": "Retry", "model_group": "gpt-4o", "retry": 1, "delay": 0.0})
+ )]
+ #[case::blocked(
+ RouterEvent::Blocked { level: 0, model: "claude".to_owned(), blocked: Blocked::ContentFilter },
+ json!({"event": "Blocked", "level": 0, "model": "claude", "blocked": "content_filter"})
+ )]
+ #[case::retry_logged_after_an_attempt(
+ RouterEvent::RetryLogged {
+ scope: 0,
+ level: 0,
+ model: "gpt-4o".to_owned(),
+ attempted_retries: None,
+ cause: RetryCause::Attempt,
+ },
+ json!({
+ "event": "RetryLogged", "scope": 0, "level": 0, "model": "gpt-4o",
+ "attempted_retries": null, "cause": {"cause": "attempt"},
+ })
+ )]
+ #[case::retry_logged_after_a_pick(
+ RouterEvent::RetryLogged {
+ scope: 1,
+ level: 1,
+ model: "backup".to_owned(),
+ attempted_retries: Some(2),
+ cause: RetryCause::Pick(PickError(Arc::new(Error::UnknownModel { model: "backup".to_owned() }))),
+ },
+ json!({
+ "event": "RetryLogged", "scope": 1, "level": 1, "model": "backup", "attempted_retries": 2,
+ "cause": {"cause": "pick", "error": "no deployment serves model `backup`"},
+ })
+ )]
+ #[case::mock_failed(
+ RouterEvent::MockFailed {
+ mock: MockFailure::RateLimit,
+ level: 0,
+ model: "gpt-4o".to_owned(),
+ num_retries: None,
+ },
+ json!({"event": "MockFailed", "mock": "rate_limit", "level": 0, "model": "gpt-4o", "num_retries": null})
+ )]
+ fn an_event_serializes_tagged_by_its_name(#[case] event: RouterEvent, #[case] expected: Value) {
+ assert_eq!(serde_json::to_value(&event).unwrap(), expected);
+ }
+}
diff --git a/litellm-rust/crates/router-core/src/mock.rs b/litellm-rust/crates/router-core/src/mock.rs
index 898782e6203..1db6c0d8019 100644
--- a/litellm-rust/crates/router-core/src/mock.rs
+++ b/litellm-rust/crates/router-core/src/mock.rs
@@ -7,7 +7,15 @@ use crate::error::{AttemptError, FailureKind};
/// Which mock failure a request asks for.
#[derive(
- Clone, Copy, Debug, PartialEq, Eq, Deserialize, strum::IntoStaticStr, strum::EnumString,
+ Clone,
+ Copy,
+ Debug,
+ PartialEq,
+ Eq,
+ Deserialize,
+ serde::Serialize,
+ strum::IntoStaticStr,
+ strum::EnumString,
)]
#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
diff --git a/litellm-rust/crates/router-core/tests/call_loop.rs b/litellm-rust/crates/router-core/tests/call_loop.rs
index a2a741d6135..cd51e53ec6b 100644
--- a/litellm-rust/crates/router-core/tests/call_loop.rs
+++ b/litellm-rust/crates/router-core/tests/call_loop.rs
@@ -17,7 +17,7 @@ use litellm_router_core::{
};
use parking_lot::Mutex;
use rstest::{fixture, rstest};
-use serde_json::json;
+use serde_json::{Value, json};
#[derive(Default)]
struct ScriptedHost {
@@ -1809,3 +1809,55 @@ fn request_for(model: &str) -> RouterRequest {
..RouterRequest::default()
}
}
+
+#[rstest]
+#[tokio::test]
+async fn every_event_of_a_retried_and_fallen_back_call_serializes_tagged_by_its_name(
+ config: RouterConfig,
+) {
+ let host = Arc::new(ScriptedHost::failing(
+ "primary",
+ vec![failure(FailureKind::RateLimit, 429); 3],
+ ));
+ let router = router(
+ config,
+ json!({"num_retries": 1, "fallbacks": [{"primary": ["backup"]}]}),
+ host.clone(),
+ );
+ router
+ .call(Op::from("acompletion"), request())
+ .await
+ .unwrap();
+ let tags: Value = host
+ .events
+ .lock()
+ .iter()
+ .map(|event| serde_json::to_value(event).unwrap()["event"].clone())
+ .collect();
+ assert_eq!(
+ tags,
+ json!([
+ "RetriesOpened",
+ "PickStarted",
+ "Selected",
+ "AttemptFailed",
+ "CooldownSet",
+ "RetryBudget",
+ "RetryLogged",
+ "Retry",
+ "PickStarted",
+ "Selected",
+ "AttemptFailed",
+ "CooldownSet",
+ "RetryLogged",
+ "FallingBack",
+ "RetryLogged",
+ "Fallback",
+ "RetriesOpened",
+ "PickStarted",
+ "Selected",
+ "Succeeded",
+ "FallbackDone",
+ ])
+ );
+}
diff --git a/litellm/completion_extras/litellm_responses_transformation/transformation.py b/litellm/completion_extras/litellm_responses_transformation/transformation.py
index 9902fe1d8fc..b24c75e69bc 100644
--- a/litellm/completion_extras/litellm_responses_transformation/transformation.py
+++ b/litellm/completion_extras/litellm_responses_transformation/transformation.py
@@ -54,6 +54,7 @@ from litellm.llms.base_llm.bridges.completion_transformation import (
CompletionTransformationBridge,
)
from litellm.llms.custom_httpx.sse_lines import split_sse_lines
+from litellm.responses.additional_tools import additional_tools_in_input
from litellm.responses.litellm_completion_transformation.tool_search import is_anthropic_tool_search_chat_tool
from litellm.responses.sse_output_recovery import (
parse_sse_json_chunk,
@@ -426,6 +427,17 @@ def replayed_tool_search_tool_names(request: object) -> tuple[str, ...]:
return tuple(chain.from_iterable(_tool_definition_name(tool) for tool in replayed_tool_search_tools(request)))
+def _top_level_namespaces(request: object) -> Iterator[object]:
+ tools: Final = _json_list(_json_field(request, "tools")) or ()
+ return (tool for tool in tools if _json_field(tool, "type") == "namespace")
+
+
+def nested_tool_member_names(request: object) -> tuple[str, ...]:
+ tools: Final = (*_top_level_namespaces(request), *additional_tools_in_input(_json_field(request, "input")))
+ members: Final = chain.from_iterable(_namespace_members(tool) for tool in tools)
+ return tuple(chain.from_iterable(_tool_definition_name(member) for member in members))
+
+
def _loads_any_tool(message: object, names: frozenset[str]) -> bool:
loaded_names: Final = chain.from_iterable(_tool_definition_name(tool) for tool in _message_loaded_tools(message))
return any(name in names for name in loaded_names)
diff --git a/litellm/integrations/arize/_utils.py b/litellm/integrations/arize/_utils.py
index f719dacd148..fd912839c95 100644
--- a/litellm/integrations/arize/_utils.py
+++ b/litellm/integrations/arize/_utils.py
@@ -14,6 +14,7 @@ from litellm.litellm_core_utils.redact_messages import (
should_redact_message_logging,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.types.utils import CallTypes, StandardLoggingMCPToolCall, StandardLoggingPayload
if TYPE_CHECKING:
@@ -663,12 +664,11 @@ def _to_plain_dict(value):
def _get_tool_calls(message) -> list | None:
- """Return ``message.tool_calls`` only when it's a non-empty list.
-
- Works for dicts and Pydantic message objects via ``_safe_get``.
+ """Return the message's tool calls, Claude's server tool calls first, or
+ ``None`` when there are none. Works for dicts and Pydantic message objects.
"""
- tool_calls: Final = _safe_get(message, "tool_calls")
- return tool_calls if isinstance(tool_calls, list) and tool_calls else None
+ tool_calls: Final = tool_calls_with_server_tool_calls(message)
+ return list(tool_calls) if tool_calls else None
class _NormalizedToolCallFunction(TypedDict):
diff --git a/litellm/integrations/datadog/datadog_llm_obs.py b/litellm/integrations/datadog/datadog_llm_obs.py
index a72023eaedf..79788d35854 100644
--- a/litellm/integrations/datadog/datadog_llm_obs.py
+++ b/litellm/integrations/datadog/datadog_llm_obs.py
@@ -44,6 +44,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -315,7 +316,10 @@ def _to_dd_tool_calls(message: Mapping[str, object]) -> tuple[ToolCall, ...]:
tool_id=_text_field(tool_call, "id"),
type=_text_field(tool_call, "type", "function"),
)
- for tool_call in (raw_tool_calls if isinstance(raw_tool_calls, list) else ())
+ for tool_call in (
+ *anthropic_server_tool_call_params(message),
+ *(raw_tool_calls if isinstance(raw_tool_calls, list) else ()),
+ )
if isinstance(tool_call, dict)
for function in [_mapping_field(tool_call, "function")]
)
diff --git a/litellm/integrations/helicone.py b/litellm/integrations/helicone.py
index 63acc562d98..a0052069271 100644
--- a/litellm/integrations/helicone.py
+++ b/litellm/integrations/helicone.py
@@ -10,6 +10,23 @@ from litellm.integrations.helicone_mock_client import (
create_mock_helicone_client,
should_use_helicone_mock,
)
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
+from litellm.types.utils import ChatCompletionMessageCustomToolCall, ChatCompletionMessageToolCall
+
+
+def _tool_use_block(
+ tool_call: ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall,
+) -> dict[str, object]:
+ match tool_call:
+ case ChatCompletionMessageCustomToolCall(custom=custom):
+ return {"type": "tool_use", "id": tool_call.id, "name": custom.name, "input": custom.input}
+ case _:
+ return {
+ "type": "tool_use",
+ "id": tool_call.id,
+ "name": tool_call.function.name,
+ "input": tool_call.function.arguments,
+ }
class HeliconeLogger:
@@ -57,23 +74,8 @@ class HeliconeLogger:
message = choice["message"]
content = []
- if "tool_calls" in message and message["tool_calls"]:
- for tool_call in message["tool_calls"]:
- function = tool_call.get("function")
- custom = tool_call.get("custom")
- if not function and not custom:
- continue
- name, tool_input = (
- (function["name"], function["arguments"]) if function else (custom["name"], custom["input"])
- )
- content.append(
- {
- "type": "tool_use",
- "id": tool_call["id"],
- "name": name,
- "input": tool_input,
- }
- )
+ if tool_calls := tool_calls_with_server_tool_calls(message):
+ content = [_tool_use_block(tool_call) for tool_call in tool_calls]
elif "content" in message and message["content"]:
content = [{"type": "text", "text": message["content"]}]
diff --git a/litellm/integrations/langfuse/langfuse_otel.py b/litellm/integrations/langfuse/langfuse_otel.py
index ee75e6501e8..0117348da66 100644
--- a/litellm/integrations/langfuse/langfuse_otel.py
+++ b/litellm/integrations/langfuse/langfuse_otel.py
@@ -26,6 +26,7 @@ from litellm.integrations.opentelemetry import (
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.secret_managers.main import str_to_bool
from litellm.types.integrations.langfuse_otel import (
LangfuseSpanAttributes,
@@ -605,8 +606,8 @@ def _extract_choices_output(response_obj: _Gettable) -> str | None:
message: Final = _as_gettable(first_choice.get("message", {}))
if message is None:
return None
- tool_calls: Final = message.get("tool_calls")
- if isinstance(tool_calls, list) and tool_calls:
+ tool_calls: Final = tool_calls_with_server_tool_calls(message)
+ if tool_calls:
transformed_tool_calls: Final = [
entry for tool_call in tool_calls if (entry := _transformed_tool_call(response_obj, tool_call)) is not None
]
diff --git a/litellm/integrations/lunary.py b/litellm/integrations/lunary.py
index 6c020c78c32..08acb2c80dc 100644
--- a/litellm/integrations/lunary.py
+++ b/litellm/integrations/lunary.py
@@ -6,6 +6,7 @@ from datetime import datetime, timezone
from typing import TYPE_CHECKING, Final
import packaging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
if TYPE_CHECKING:
from openai.types.chat.chat_completion_message_custom_tool_call import Custom
@@ -67,8 +68,8 @@ def parse_messages(input):
}
# Only add tool_calls and function_call to res if they are set
- if message.get("tool_calls"):
- serialized["tool_calls"] = parse_tool_calls(message.get("tool_calls"))
+ if tool_calls := tool_calls_with_server_tool_calls(message):
+ serialized["tool_calls"] = parse_tool_calls(tool_calls)
return serialized
diff --git a/litellm/integrations/newrelic/newrelic.py b/litellm/integrations/newrelic/newrelic.py
index 3623335f599..efc796825e4 100644
--- a/litellm/integrations/newrelic/newrelic.py
+++ b/litellm/integrations/newrelic/newrelic.py
@@ -56,6 +56,7 @@ from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.core_helpers import has_attributes
from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.types.integrations.base_health_check import IntegrationHealthCheckStatus
from litellm.types.integrations.newrelic import NewRelicInitParams
from litellm.types.utils import Message, ModelResponse, StandardLoggingPayload
@@ -479,11 +480,11 @@ class NewRelicLogger(CustomLogger):
content: Final = message.get("content")
# Handle tool calls
- if message.get("tool_calls"):
+ if tool_calls := tool_calls_with_server_tool_calls(message):
try:
- return json.dumps(message["tool_calls"])
+ return json.dumps([tool_call.model_dump() for tool_call in tool_calls])
except Exception:
- return str(message["tool_calls"])
+ return str(tool_calls)
# Handle None or missing content
if content is None:
diff --git a/litellm/integrations/opentelemetry.py b/litellm/integrations/opentelemetry.py
index a9bd93e6420..39c485cb31f 100644
--- a/litellm/integrations/opentelemetry.py
+++ b/litellm/integrations/opentelemetry.py
@@ -42,6 +42,7 @@ from litellm.litellm_core_utils.service_tier_utils import (
get_requested_service_tier,
get_served_service_tier,
)
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params, tool_calls_with_server_tool_calls
from litellm.secret_managers.main import get_secret_bool, str_to_bool
from litellm.types.integrations.otel_span_attributes import SpanAttributes
from litellm.types.services import ServiceLoggerPayload
@@ -2736,9 +2737,8 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
for idx, choice in enumerate(response_obj.get("choices")):
if choice.get("finish_reason"):
message = choice.get("message")
- tool_calls = message.get("tool_calls")
- if tool_calls:
- kv_pairs = OpenTelemetry._tool_calls_kv_pair(tool_calls)
+ if tool_calls := tool_calls_with_server_tool_calls(message):
+ kv_pairs = OpenTelemetry._tool_calls_kv_pair(list(tool_calls))
for key, value in kv_pairs.items():
self.safe_set_attribute(
span=span,
@@ -2856,7 +2856,9 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
transformed_msg = {"role": role, "parts": parts}
if "id" in msg:
transformed_msg["id"] = msg["id"]
- if "tool_calls" in msg:
+ if server_tool_calls := anthropic_server_tool_call_params(cast(object, msg)):
+ transformed_msg["tool_calls"] = [*server_tool_calls, *(msg.get("tool_calls") or [])]
+ elif "tool_calls" in msg:
transformed_msg["tool_calls"] = msg["tool_calls"]
if "tool_call_id" in msg:
transformed_msg["tool_call_id"] = msg["tool_call_id"]
diff --git a/litellm/integrations/otel/mappers/utils.py b/litellm/integrations/otel/mappers/utils.py
index 80ef2dd553a..bfd50fea5d5 100644
--- a/litellm/integrations/otel/mappers/utils.py
+++ b/litellm/integrations/otel/mappers/utils.py
@@ -12,6 +12,8 @@ from typing import Final
from litellm.integrations.otel.mappers.base import AttributeMap, AttrValue
from litellm.integrations.otel.model.payloads import LLMCallSpanData, ToolDefinition
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
@dataclass(frozen=True, slots=True)
@@ -141,12 +143,13 @@ def output_messages(data: LLMCallSpanData) -> list:
def message_tool_calls(message: object) -> tuple[MessageToolCall, ...]:
+ tool_call_values: Final = (
+ *anthropic_server_tool_call_params(message),
+ *parts_of(message_field(message, "tool_calls")),
+ )
if not isinstance(message, dict):
return ()
- tool_calls: Final = message.get("tool_calls")
- if not isinstance(tool_calls, (list, tuple)) or not tool_calls:
- return ()
- return tuple(tool_call for value in tool_calls if (tool_call := _message_tool_call(value)) is not None)
+ return tuple(tool_call for value in tool_call_values if (tool_call := _message_tool_call(value)) is not None)
def _message_tool_call(value: object) -> MessageToolCall | None:
diff --git a/litellm/integrations/otel/model/payloads.py b/litellm/integrations/otel/model/payloads.py
index 7a62ed6aed9..4238d72fc5e 100644
--- a/litellm/integrations/otel/model/payloads.py
+++ b/litellm/integrations/otel/model/payloads.py
@@ -10,6 +10,8 @@ from types import MappingProxyType
from typing import TYPE_CHECKING, ClassVar, Final, Literal, Protocol, cast
from urllib.parse import urlsplit
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from pydantic import TypeAdapter, ValidationError
from typing_extensions import ReadOnly, TypedDict
from litellm.integrations.otel.model.metadata import (
@@ -36,6 +38,7 @@ from litellm.integrations.otel.model.utils import (
as_str_tuple,
)
from litellm.integrations.otel.routing import RoutingAttributeValue, routing_decision_attributes
+from litellm.types.llms.anthropic import claude_hosted_web_tool_call
# ``RequestIdentity`` and the request-metadata translation now live in
# :mod:`metadata`; re-exported here so existing ``model.payloads`` imports keep
@@ -809,6 +812,7 @@ class _Choice(TypedDict):
_RESPONSES_TOOL_CALL_TYPES: Final = frozenset({"function_call", "custom_tool_call"})
+_WEB_SEARCH_CALL: Final = TypeAdapter(ResponseFunctionWebSearch)
def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
@@ -816,9 +820,13 @@ def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
items: Final = _dicts(response.get("output"))
messages: Final = tuple(item for item in items if item.get("type") == "message")
parts: Final = tuple(part for item in messages for part in _dicts(item.get("content")))
- tool_calls: Final = tuple(
+ client_tool_calls: Final = tuple(
_responses_tool_call(item) for item in items if item.get("type") in _RESPONSES_TOOL_CALL_TYPES
)
+ tool_calls: Final = (
+ *(call for item in items if (call := _claude_web_search_tool_call(item)) is not None),
+ *client_tool_calls,
+ )
if not messages and not tool_calls:
return ()
message: Final[_AssistantMessage] = {
@@ -827,7 +835,10 @@ def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
"refusal": _responses_parts_text(parts, "refusal", "refusal"),
"tool_calls": tool_calls or None,
}
- choice: Final[_Choice] = {"message": message, "finish_reason": _responses_finish_reason(response, bool(tool_calls))}
+ choice: Final[_Choice] = {
+ "message": message,
+ "finish_reason": _responses_finish_reason(response, bool(client_tool_calls)),
+ }
return (choice,)
@@ -981,6 +992,20 @@ def _responses_tool_call(item: Mapping[str, object]) -> _ToolCall:
return tool_call
+def _claude_web_search_tool_call(item: Mapping[str, object]) -> _ToolCall | None:
+ if item.get("type") != "web_search_call":
+ return None
+ try:
+ search: Final = _WEB_SEARCH_CALL.validate_python(item)
+ except ValidationError:
+ return None
+ claude_call: Final = claude_hosted_web_tool_call(search)
+ if claude_call is None:
+ return None
+ tool_id, name, arguments = claude_call
+ return {"id": tool_id, "type": "function", "function": {"name": name, "arguments": arguments}}
+
+
def _responses_finish_reason(response: Mapping[str, object], has_tool_calls: bool) -> str | None:
status: Final = as_str(response.get("status"))
if status == "completed":
diff --git a/litellm/integrations/shadow_eval_logger.py b/litellm/integrations/shadow_eval_logger.py
index f683108a51a..2c3d8a731e1 100644
--- a/litellm/integrations/shadow_eval_logger.py
+++ b/litellm/integrations/shadow_eval_logger.py
@@ -39,6 +39,7 @@ from litellm.litellm_core_utils.llm_judge import (
parse_json_verdict,
)
from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.llms.base_llm.base_utils import type_to_response_format_param
from litellm.redis.stores.spend_counters import (
shadow_eval_counter,
@@ -205,7 +206,7 @@ def _chat_final_text(response_obj: object) -> str:
if read is None:
return ""
prose: Final = extract_text_from_content(read("content"))
- if not (read("tool_calls") or read("function_call")):
+ if not _tool_calls_list(read):
return prose
serialized: Final = _serialize_tool_calls(read)
return f"{prose} {serialized}".strip() if prose else serialized
@@ -221,8 +222,9 @@ _RESPONSES_TOOL_CALL_TYPES: Final = frozenset(("function_call", "custom_tool_cal
def _tool_calls_list(read: Callable[[str], object]) -> tuple[object, ...]:
- calls: Final = read("tool_calls")
- listed: Final = tuple(calls) if isinstance(calls, Sequence) and not isinstance(calls, str) else ()
+ listed: Final = tool_calls_with_server_tool_calls(
+ {"tool_calls": read("tool_calls"), "provider_specific_fields": read("provider_specific_fields")}
+ )
single: Final = read("function_call")
return listed if listed else ((single,) if single is not None else ())
diff --git a/litellm/litellm_core_utils/get_llm_provider_logic.py b/litellm/litellm_core_utils/get_llm_provider_logic.py
index 1ffb5e2f8e5..b2851345302 100644
--- a/litellm/litellm_core_utils/get_llm_provider_logic.py
+++ b/litellm/litellm_core_utils/get_llm_provider_logic.py
@@ -12,6 +12,7 @@ from litellm.constants import (
from litellm.litellm_core_utils.fallback_generalizations import (
match_routing_generalization,
)
+from litellm.litellm_core_utils.model_latest_aliases import resolve_model_latest_alias
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
from litellm.llms.opencode.common_utils import is_opencode_api_base
from litellm.secret_managers.main import get_secret, get_secret_str
@@ -189,6 +190,23 @@ def get_llm_provider(
api_base: str | None = None,
api_key: str | None = None,
litellm_params: GenericLiteLLMParams | None = None,
+) -> tuple[str, str, str | None, str | None]:
+ resolved_model, provider, dynamic_api_key, resolved_api_base = _get_llm_provider(
+ model, custom_llm_provider, api_base, api_key, litellm_params
+ )
+ explicit_provider: Final = custom_llm_provider or (litellm_params.custom_llm_provider if litellm_params else None)
+ target: Final = (
+ resolve_model_latest_alias(resolved_model, provider) if explicit_provider or "/" in model else resolved_model
+ )
+ return target, provider, dynamic_api_key, resolved_api_base
+
+
+def _get_llm_provider(
+ model: str,
+ custom_llm_provider: str | None = None,
+ api_base: str | None = None,
+ api_key: str | None = None,
+ litellm_params: GenericLiteLLMParams | None = None,
) -> tuple[str, str, str | None, str | None]:
"""
Returns the provider for a given model name - e.g. 'azure/chatgpt-v-2' -> 'azure'
diff --git a/litellm/litellm_core_utils/llm_cost_calc/utils.py b/litellm/litellm_core_utils/llm_cost_calc/utils.py
index 9ac4f9d5ee8..5eb9cdb11ce 100644
--- a/litellm/litellm_core_utils/llm_cost_calc/utils.py
+++ b/litellm/litellm_core_utils/llm_cost_calc/utils.py
@@ -1464,7 +1464,7 @@ def get_fedramp_uplift_multiplier(model_info: ModelInfo, data_residency: str | N
return 1.0
-def get_vertex_regional_endpoint_uplift(model_info: ModelInfo, vertex_location: str | None) -> float:
+def get_vertex_regional_endpoint_uplift(model_info: Mapping[str, object], vertex_location: str | None) -> float:
"""
Resolve the per-model uplift multiplier for Vertex AI non-global (regional and
multi-region) endpoints.
diff --git a/litellm/litellm_core_utils/model_latest_aliases.py b/litellm/litellm_core_utils/model_latest_aliases.py
new file mode 100644
index 00000000000..e3b1a21dd5a
--- /dev/null
+++ b/litellm/litellm_core_utils/model_latest_aliases.py
@@ -0,0 +1,33 @@
+from collections.abc import Mapping
+from functools import cache
+from importlib.resources import files
+from types import MappingProxyType
+from typing import Final
+
+from pydantic import TypeAdapter
+
+_ALIASES_ADAPTER: Final = TypeAdapter(dict[str, dict[str, str]])
+
+
+@cache
+def get_model_latest_aliases() -> Mapping[str, Mapping[str, str]]:
+ aliases: Final = _ALIASES_ADAPTER.validate_json(files("litellm").joinpath("model_latest_aliases.json").read_bytes())
+ return MappingProxyType({provider: MappingProxyType(models) for provider, models in aliases.items()})
+
+
+def resolve_model_latest_alias(model: str, provider: str) -> str:
+ aliases: Final = get_model_latest_aliases().get(provider)
+ return aliases.get(model, model) if aliases is not None else model
+
+
+def expand_model_latest_alias(model: str) -> tuple[str, ...]:
+ provider, separator, name = model.partition("/")
+ target: Final = resolve_model_latest_alias(name, provider) if separator else name
+ return (model, f"{provider}/{target}") if separator and target != name else (model,)
+
+
+def is_model_latest_alias(model: str, custom_llm_provider: str | None = None) -> bool:
+ prefix, separator, name = model.partition("/")
+ provider: Final = custom_llm_provider or prefix
+ candidate: Final = name if separator and prefix == provider else model
+ return bool(custom_llm_provider or separator) and resolve_model_latest_alias(candidate, provider) != candidate
diff --git a/litellm/litellm_core_utils/prompt_templates/common_utils.py b/litellm/litellm_core_utils/prompt_templates/common_utils.py
index 8f874edfddf..ec857218c79 100644
--- a/litellm/litellm_core_utils/prompt_templates/common_utils.py
+++ b/litellm/litellm_core_utils/prompt_templates/common_utils.py
@@ -604,6 +604,8 @@ def get_content_from_model_response(response: ModelResponse | dict) -> str:
else:
new_response = response
+ from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
+
content = ""
for choice in new_response.choices:
@@ -611,9 +613,8 @@ def get_content_from_model_response(response: ModelResponse | dict) -> str:
content += choice.message.content if choice.message.content else ""
if choice.message.function_call:
content += choice.message.function_call.model_dump_json()
- if choice.message.tool_calls:
- for tc in choice.message.tool_calls:
- content += tc.model_dump_json()
+ for tc in tool_calls_with_server_tool_calls(choice.message):
+ content += tc.model_dump_json()
elif isinstance(choice, StreamingChoices):
content += getattr(choice, "delta", {}).get("content", "") or ""
return content
diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py
index c6288227546..17a1fa7d81b 100644
--- a/litellm/litellm_core_utils/prompt_templates/factory.py
+++ b/litellm/litellm_core_utils/prompt_templates/factory.py
@@ -2012,12 +2012,39 @@ def _tool_call_ids_with_a_kept_server_result(message: object) -> frozenset[str]:
return frozenset()
return frozenset(
tool_call_id
- for tool_call in parts_of(message_field(message, "tool_calls"))
+ for tool_call in (
+ *parts_of(message_field(message, "tool_calls")),
+ *parts_of(message_field(provider_specific_fields, "server_tool_calls")),
+ )
if isinstance(tool_call_id := message_field(tool_call, "id"), str)
and tool_call_is_rebuilt_as_server_tool_use(tool_call_id, provider_specific_fields)
)
+def _assistant_tool_calls_with_server_tool_calls(
+ message: ChatCompletionAssistantMessage,
+) -> Sequence[ChatCompletionAssistantToolCall] | None:
+ from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+ tool_calls: Final = message.get("tool_calls")
+ server_tool_calls: Final = anthropic_server_tool_call_params(message)
+ if not server_tool_calls:
+ return tool_calls
+ client_tool_call_ids: Final = frozenset(tool_call["id"] for tool_call in tool_calls or ())
+ return (
+ *(tool_call for tool_call in server_tool_calls if tool_call["id"] not in client_tool_call_ids),
+ *(tool_calls or ()),
+ )
+
+
+def _server_tool_call_ids(message: object) -> frozenset[str]:
+ from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+ return frozenset(
+ tool_call["id"] for tool_call in anthropic_server_tool_call_params(message) if tool_call["id"] is not None
+ )
+
+
def _server_tool_answers_replaced_after(messages: Sequence[object], index: int) -> frozenset[str]:
kept_server_result_ids: Final = _tool_call_ids_with_a_kept_server_result(messages[index])
if not kept_server_result_ids:
@@ -2102,6 +2129,12 @@ def _unrun_server_tool_use_ids(blocks: Sequence[Mapping[str, object]]) -> frozen
)
+def _paused_server_tool_use_ids(blocks: Sequence[Mapping[str, object]]) -> frozenset[object]:
+ if any(block.get("type") == "tool_use" for block in blocks):
+ return frozenset()
+ return _unrun_server_tool_use_ids(blocks)
+
+
def _last_block_id_among(blocks: Sequence[Mapping[str, object]], ids: frozenset[object]) -> object:
return next(block_id for block in reversed(blocks) if (block_id := _block_id(block)) in ids)
@@ -2115,7 +2148,7 @@ def _paused_turn_replay(
tuple[_AnthropicToolInvokeItem, ...],
tuple[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, ...],
]:
- unrun_ids: Final = _unrun_server_tool_use_ids(tool_invokes)
+ unrun_ids: Final = _paused_server_tool_use_ids(tool_invokes)
if not unrun_ids:
return tuple(tool_invokes), tuple(thinking_blocks)
dropped_ids: Final = (
@@ -2125,10 +2158,11 @@ def _paused_turn_replay(
def convert_to_anthropic_tool_invoke(
- tool_calls: list[ChatCompletionAssistantToolCall],
+ tool_calls: Sequence[ChatCompletionAssistantToolCall],
web_search_results: Sequence[object] | None = None,
tool_results: Sequence[object] | None = None,
answered_tool_ids: Container[str] = frozenset(),
+ server_tool_call_ids: Container[str] = frozenset(),
) -> list[_AnthropicToolInvokeItem]:
"""
OpenAI tool invokes:
@@ -2206,14 +2240,11 @@ def convert_to_anthropic_tool_invoke(
anthropic_tool_invoke.append(server_tool_result)
elif is_anthropic_tool_search_use(tool_id, tool_name) and tool_id not in answered_tool_ids:
continue
- elif is_anthropic_server_tool_use(tool_id) and tool_id not in answered_tool_ids:
+ elif (
+ tool_id in server_tool_call_ids or is_anthropic_server_tool_use(tool_id)
+ ) and tool_id not in answered_tool_ids:
anthropic_tool_invoke.append(
- {
- "type": "server_tool_use",
- "id": tool_id,
- "name": tool_name,
- "input": tool_input,
- }
+ {"type": "server_tool_use", "id": tool_id, "name": tool_name, "input": tool_input}
)
else:
sanitized_tool_id = sanitize_anthropic_tool_use_id(tool_id)
@@ -3174,7 +3205,7 @@ def anthropic_messages_pt(
# If so, we need to interleave thinking blocks with tool call groups
# to preserve the original content block ordering.
# Fixes: https://github.com/BerriAI/litellm/issues/23047
- assistant_tool_calls = assistant_content_block.get("tool_calls")
+ assistant_tool_calls = _assistant_tool_calls_with_server_tool_calls(assistant_content_block)
_has_server_tool_calls = False
if assistant_tool_calls is not None:
for _tc in assistant_tool_calls:
@@ -3198,11 +3229,12 @@ def anthropic_messages_pt(
web_search_results=_provider_specific_fields.get("web_search_results"),
tool_results=_provider_specific_fields.get("tool_results"),
answered_tool_ids=client_answered_tool_ids,
+ server_tool_call_ids=_server_tool_call_ids(assistant_content_block),
)
if assistant_tool_calls is not None
else ()
)
- turn_paused = bool(_unrun_server_tool_use_ids(tool_invoke_results))
+ turn_paused = bool(_paused_server_tool_use_ids(tool_invoke_results))
replayed_tool_invokes, replayed_thinking_blocks = _paused_turn_replay(
tool_invoke_results, thinking_blocks or (), is_final_message=msg_i == len(messages) - 1
)
@@ -5034,11 +5066,15 @@ class BedrockConverseMessagesProcessor:
drop_params: bool = False,
model_info: Mapping[str, object] | None = None,
) -> list[BedrockMessageBlock]:
- from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
+ from litellm.llms.bedrock.common_utils import (
+ bedrock_converse_accepts_redacted_thinking,
+ bedrock_converse_rejects_signed_thinking,
+ )
contents: list[BedrockMessageBlock] = []
msg_i = 0
to_claude: Final = bedrock_converse_accepts_redacted_thinking(model, base_model)
+ drop_signed_thinking: Final = bedrock_converse_rejects_signed_thinking(model, base_model)
messages = BedrockConverseMessagesProcessor._initial_message_setup(
messages, model, llm_provider, user_continue_message
@@ -5194,7 +5230,9 @@ class BedrockConverseMessagesProcessor:
thinking_blocks = assistant_message_block.get("thinking_blocks")
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
- _thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
+ _thinking_in_replay_order(_assistant_content, thinking_blocks),
+ to_claude=to_claude,
+ drop_signed=drop_signed_thinking,
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
@@ -5226,8 +5264,8 @@ class BedrockConverseMessagesProcessor:
deployment_model=deployment_model,
model_info=model_info,
)
- if _cache_point_block is not None and not (
- drop_params and _as_thinking_block(element) is not None
+ if _cache_point_block is not None and BedrockConverseMessagesProcessor.keeps_cache_point(
+ element, drop_params=drop_params, drop_signed=drop_signed_thinking
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
@@ -5286,6 +5324,7 @@ class BedrockConverseMessagesProcessor:
thinking_blocks: Sequence[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
*,
to_claude: bool = False,
+ drop_signed: bool = False,
) -> list[BedrockContentBlock]:
return [
BedrockContentBlock(
@@ -5295,8 +5334,27 @@ class BedrockConverseMessagesProcessor:
)
for thinking_block in thinking_blocks
if not (to_claude and is_encrypted_reasoning_block(thinking_block))
+ and not (drop_signed and BedrockConverseMessagesProcessor._is_signed(thinking_block))
]
+ @staticmethod
+ def keeps_cache_point(element: object, *, drop_params: bool, drop_signed: bool) -> bool:
+ match _as_thinking_block(element):
+ case None:
+ return True
+ case thinking_block:
+ return not (
+ drop_params or (drop_signed and BedrockConverseMessagesProcessor._is_signed(thinking_block))
+ )
+
+ @staticmethod
+ def _is_signed(thinking_block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock) -> bool:
+ match thinking_block:
+ case {"signature": signature} if signature:
+ return True
+ case _:
+ return False
+
@staticmethod
def _to_reasoning_content_block(
thinking_block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
@@ -5455,11 +5513,15 @@ def _bedrock_converse_messages_pt(
- Please ensure that function response turn comes immediately after a function call turn
- Conversation blocks and tool result blocks cannot be provided in the same turn. Issue: https://github.com/BerriAI/litellm/issues/6053
"""
- from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
+ from litellm.llms.bedrock.common_utils import (
+ bedrock_converse_accepts_redacted_thinking,
+ bedrock_converse_rejects_signed_thinking,
+ )
contents: list[BedrockMessageBlock] = []
msg_i = 0
to_claude: Final = bedrock_converse_accepts_redacted_thinking(model, base_model)
+ drop_signed_thinking: Final = bedrock_converse_rejects_signed_thinking(model, base_model)
messages = BedrockConverseMessagesProcessor._initial_message_setup(
messages, model, llm_provider, user_continue_message
@@ -5616,7 +5678,9 @@ def _bedrock_converse_messages_pt(
thinking_blocks = assistant_message_block.get("thinking_blocks")
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
- _thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
+ _thinking_in_replay_order(_assistant_content, thinking_blocks),
+ to_claude=to_claude,
+ drop_signed=drop_signed_thinking,
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
@@ -5649,8 +5713,8 @@ def _bedrock_converse_messages_pt(
deployment_model=deployment_model,
model_info=model_info,
)
- if _cache_point_block is not None and not (
- drop_params and _as_thinking_block(element) is not None
+ if _cache_point_block is not None and BedrockConverseMessagesProcessor.keeps_cache_point(
+ element, drop_params=drop_params, drop_signed=drop_signed_thinking
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
diff --git a/litellm/litellm_core_utils/redact_messages.py b/litellm/litellm_core_utils/redact_messages.py
index ac5c0720eff..a5e79d7b2b4 100644
--- a/litellm/litellm_core_utils/redact_messages.py
+++ b/litellm/litellm_core_utils/redact_messages.py
@@ -14,7 +14,7 @@ from collections.abc import Iterable, Mapping, Sequence
from dataclasses import replace
from typing import TYPE_CHECKING, Any, Final
-from pydantic import JsonValue, TypeAdapter, ValidationError
+from pydantic import BaseModel, JsonValue, TypeAdapter, ValidationError
import litellm
from litellm.constants import REDACTED_BY_LITELLM
@@ -301,9 +301,63 @@ def _redacted_provider_specific_fields(fields: Mapping[str, object]) -> Mapping[
)
if key in fields
},
+ **(
+ {"server_tool_calls": _redacted_tool_call_dicts(fields["server_tool_calls"])}
+ if "server_tool_calls" in fields
+ else {}
+ ),
+ **(
+ {"web_search_calls": _redacted_web_search_calls(fields["web_search_calls"])}
+ if "web_search_calls" in fields
+ else {}
+ ),
}
+def _redacted_tool_call_dict(call: object) -> object:
+ if (mapping := _as_mapping(call)) is None or (function := _as_mapping(mapping.get("function"))) is None:
+ return call
+ return {**mapping, "function": {**function, "arguments": REDACTED_BY_LITELLM}}
+
+
+def _redacted_tool_call_dicts(calls: object) -> object:
+ return [_redacted_tool_call_dict(call) for call in calls] if isinstance(calls, list) else calls
+
+
+_WEB_ACTION_TEXT_FIELDS: Final = ("query", "url", "pattern")
+_WEB_ACTION_LIST_FIELDS: Final = ("queries", "sources")
+
+
+def _web_action_redactions(action: Mapping[str, object]) -> Mapping[str, object]:
+ return {
+ **{key: REDACTED_BY_LITELLM for key in _WEB_ACTION_TEXT_FIELDS if action.get(key) is not None},
+ **{key: None for key in _WEB_ACTION_LIST_FIELDS if action.get(key) is not None},
+ }
+
+
+def _redacted_web_search_action(action: object) -> object:
+ if isinstance(action, BaseModel):
+ return action.model_copy(update=dict(_web_action_redactions(action.model_dump())))
+ mapping: Final = _as_mapping(action)
+ return action if mapping is None else {**mapping, **_web_action_redactions(mapping)}
+
+
+def _redacted_web_search_call(item: object) -> object:
+ if isinstance(item, BaseModel):
+ action: Final = getattr(item, "action", None)
+ return item if action is None else item.model_copy(update={"action": _redacted_web_search_action(action)})
+ mapping: Final = _as_mapping(item)
+ if mapping is None or mapping.get("action") is None:
+ return item
+ return {**mapping, "action": _redacted_web_search_action(mapping["action"])}
+
+
+def _redacted_web_search_calls(calls: object) -> object:
+ if (mapping := _as_mapping(calls)) is not None:
+ return {key: _redacted_web_search_call(call) for key, call in mapping.items()}
+ return [_redacted_web_search_call(call) for call in calls] if isinstance(calls, list) else calls
+
+
def _redact_field_when_present(item: object, field: str) -> None:
if getattr(item, field, None) is not None:
setattr(item, field, REDACTED_BY_LITELLM)
@@ -335,6 +389,8 @@ def _redact_responses_api_output(output_items):
output_item.arguments = REDACTED_BY_LITELLM
if output_type == "custom_tool_call" and hasattr(output_item, "input"):
output_item.input = REDACTED_BY_LITELLM
+ if output_type == "web_search_call" and getattr(output_item, "action", None) is not None:
+ output_item.action = _redacted_web_search_action(output_item.action)
_RESPONSE_ITEMS: Final = TypeAdapter(tuple[object, ...])
@@ -389,6 +445,11 @@ def _redacted_responses_api_output_item(item: object, redacted_str: str) -> obje
if item_mapping.get("type") == "custom_tool_call" and "input" in item_mapping
else {}
),
+ **(
+ {"action": _redacted_web_search_action(item_mapping["action"])}
+ if item_mapping.get("type") == "web_search_call" and item_mapping.get("action") is not None
+ else {}
+ ),
}
diff --git a/litellm/llms/anthropic/chat/guardrail_translation/handler.py b/litellm/llms/anthropic/chat/guardrail_translation/handler.py
index 6436cd832c7..8f1a08d2964 100644
--- a/litellm/llms/anthropic/chat/guardrail_translation/handler.py
+++ b/litellm/llms/anthropic/chat/guardrail_translation/handler.py
@@ -16,7 +16,7 @@ import json
from collections.abc import Iterable, Mapping, MutableSequence, Sequence
from copy import deepcopy
from dataclasses import dataclass
-from itertools import chain, repeat
+from itertools import chain, filterfalse, repeat
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Protocol, TypeVar, cast, overload, runtime_checkable
@@ -26,6 +26,7 @@ from typing_extensions import ReadOnly, TypedDict, assert_never
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.core_helpers import as_str_mapping, iter_items
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_calls
from litellm.llms.anthropic.messages.adapters.transformation import (
OPENAI_MAX_TOOL_NAME_LENGTH,
LiteLLMAnthropicMessagesAdapter,
@@ -206,17 +207,15 @@ class _WebSearchString:
value: str
-def _streamed_web_search_result_blocks(response: ModelResponse) -> tuple[Mapping[str, object], ...]:
- choice: Final = response.choices[0] if response.choices else None
- fields: Final = as_str_mapping(choice.message.provider_specific_fields) if isinstance(choice, Choices) else None
- blocks: Final = (
- (as_str_mapping(item) for item in iter_items(fields.get("web_search_results")))
- if fields is not None and isinstance(fields.get("web_search_results"), list)
- else ()
- )
- return tuple(
- deepcopy(block) for block in blocks if block is not None and block.get("type") == "web_search_tool_result"
- )
+def _legacy_hook_block(block: object) -> object:
+ if not isinstance(block, Mapping):
+ return block
+ fields: Final = _as_str_mapping(block)
+ return {**fields, "type": "tool_use"} if fields.get("type") == "server_tool_use" else fields
+
+
+def _is_web_search_result_block(block: object) -> bool:
+ return isinstance(block, Mapping) and _as_str_mapping(block).get("type") == "web_search_tool_result"
def _web_search_results(block: Mapping[str, object]) -> Iterable[object]:
@@ -519,6 +518,13 @@ def _rewritten_event(event: Mapping[str, object], rewrite_event: _SSEEventRewrit
return {**event, rewrite.section: {**section, rewrite.field: rewrite.value}}
+def _in_stream_block_order(
+ tool_calls: Sequence[ChatCompletionMessageToolCall], blocks: Sequence[tuple[int, Mapping[str, object]]]
+) -> tuple[ChatCompletionMessageToolCall, ...]:
+ positions: Final = MappingProxyType({block.get("id"): position for position, (_, block) in enumerate(blocks)})
+ return tuple(sorted(tool_calls, key=lambda tool_call: positions.get(tool_call.id, len(positions))))
+
+
def _tool_call_shapes(tool_calls: Sequence[object]) -> tuple[_ToolCallShape, ...]:
"""The guardrail-visible shape of each tool call, whether the guardrail handed
back the ``ChatCompletionMessageToolCall`` objects it was given or plain dicts."""
@@ -586,11 +592,14 @@ class AnthropicMessagesHandler(BaseTranslation):
if not isinstance(response, ModelResponse):
return response
translated: Final = self.adapter.translate_openai_response_to_anthropic(response)
- result_blocks: Final = _streamed_web_search_result_blocks(response)
- if not result_blocks:
- return translated
- content: Final = translated.get("content") or ()
- return {**translated, "content": [*result_blocks, *content]}
+ blocks: Final = tuple(map(_legacy_hook_block, translated.get("content") or ()))
+ return {
+ **translated,
+ "content": [
+ *filter(_is_web_search_result_block, blocks),
+ *filterfalse(_is_web_search_result_block, blocks),
+ ],
+ }
@staticmethod
def _build_streaming_usage_response(
@@ -1570,15 +1579,18 @@ class AnthropicMessagesHandler(BaseTranslation):
if built_response is not None and hasattr(built_response, "choices") and built_response.choices:
model_response: Final = cast(ModelResponse, built_response)
first_choice: Final = cast(Choices, model_response.choices[0])
- tool_calls_list: Final = cast(
- list[ChatCompletionMessageToolCall] | None,
- first_choice.message.tool_calls,
+ tool_call_blocks: Final = self._streamed_tool_call_blocks(responses_so_far)
+ tool_calls_list: Final = _in_stream_block_order(
+ (
+ *anthropic_server_tool_calls(first_choice.message),
+ *cast(list[ChatCompletionMessageToolCall], first_choice.message.tool_calls or []),
+ ),
+ tool_call_blocks,
)
string_so_far = first_choice.message.content
pre_guardrail_tool_calls: Final = _tool_call_shapes(tool_calls_list or ())
guardrail_inputs: Final = GenericGuardrailAPIInputs()
leading_texts: Final = (string_so_far,) if isinstance(string_so_far, str) and string_so_far else ()
- tool_call_blocks: Final = self._streamed_tool_call_blocks(responses_so_far)
streamed_query_strings: Final = _streamed_query_strings(tool_call_blocks, pre_guardrail_tool_calls)
streamed_result_strings: Final = self._streamed_web_search_result_strings(responses_so_far)
streamed_web_search_strings: Final = (*streamed_result_strings, *streamed_query_strings)
@@ -1586,7 +1598,7 @@ class AnthropicMessagesHandler(BaseTranslation):
if texts_to_check:
guardrail_inputs["texts"] = texts_to_check
if tool_calls_list:
- guardrail_inputs["tool_calls"] = tool_calls_list
+ guardrail_inputs["tool_calls"] = list(tool_calls_list)
try:
prepared_request_data = self._prepare_request_data(
diff --git a/litellm/llms/anthropic/chat/handler.py b/litellm/llms/anthropic/chat/handler.py
index c273c715ba6..4062a957e00 100644
--- a/litellm/llms/anthropic/chat/handler.py
+++ b/litellm/llms/anthropic/chat/handler.py
@@ -54,6 +54,7 @@ from litellm.types.llms.openai import (
ChatCompletionToolCallFunctionChunk,
)
from litellm.types.responses.main import (
+ HOSTED_WEB_TOOL_NAMES,
OutputCodeInterpreterCall,
build_code_interpreter_log_outputs,
build_web_search_call,
@@ -761,6 +762,8 @@ class ModelResponseIterator:
self._server_tool_inputs: dict[str, object] = {}
self.tool_results: list[dict[str, Any]] = []
self._current_server_tool_id: str | None = None
+ self._current_server_tool_name: str = ""
+ self._server_tool_calls: tuple[ChatCompletionToolCallChunk, ...] = ()
self._container_id: str | None = None
@property
@@ -836,10 +839,9 @@ class ModelResponseIterator:
)
self.content_blocks.append(content_block)
if "partial_json" in content_block["delta"]:
- # Only emit tool calls if we're in a tool_use or server_tool_use block
# web_search_tool_result blocks also have input_json_delta but should not be treated as tool calls
# See: https://github.com/BerriAI/litellm/issues/17254
- if self.current_content_block_type in ("tool_use", "server_tool_use"):
+ if self.current_content_block_type == "tool_use":
tool_use = cast(
ChatCompletionToolCallChunk,
{
@@ -938,7 +940,7 @@ class ModelResponseIterator:
def _web_search_call_snapshot(self) -> dict[str, object]:
return dict(self._web_search_calls)
- def _refresh_pending_web_search_call(self, tool_use_id: str) -> bool:
+ def _refresh_pending_web_search_call(self, tool_use_id: str, tool_name: str) -> bool:
if tool_use_id not in self._web_search_calls:
return False
self._web_search_calls[tool_use_id] = build_web_search_call(
@@ -946,10 +948,11 @@ class ModelResponseIterator:
self._server_tool_inputs.get(tool_use_id, {}),
{"content": []},
status="in_progress",
+ tool_name=tool_name,
)
return True
- def _complete_web_search_call(self, result: dict[str, object]) -> None:
+ def _complete_web_search_call(self, result: dict[str, object], tool_name: str) -> None:
tool_use_id: Final = result.get("tool_use_id")
if not isinstance(tool_use_id, str) or tool_use_id not in self._web_search_calls:
return
@@ -957,6 +960,7 @@ class ModelResponseIterator:
tool_id=tool_use_id,
tool_input=self._server_tool_inputs.get(tool_use_id, {}),
result=result,
+ tool_name=tool_name,
)
def _build_code_interpreter_results(self) -> list:
@@ -1036,7 +1040,6 @@ class ModelResponseIterator:
content_block_start["content_block"]["type"] == "tool_use"
or content_block_start["content_block"]["type"] == "server_tool_use"
):
- self.tool_index += 1
# Reverse-map the (sanitized) tool name back to the
# caller's original. No-op when the map is empty.
_stream_tool_name = content_block_start["content_block"]["name"]
@@ -1045,35 +1048,39 @@ class ModelResponseIterator:
# Use empty string for arguments in content_block_start - actual arguments
# come in subsequent content_block_delta chunks and get accumulated.
# Using str(input) here would prepend '{}' causing invalid JSON accumulation.
- tool_use = ChatCompletionToolCallChunk(
- id=content_block_start["content_block"]["id"],
- type="function",
- function=ChatCompletionToolCallFunctionChunk(
- name=_stream_tool_name,
- arguments="",
- ),
- index=self.tool_index,
- )
+ if content_block_start["content_block"]["type"] == "tool_use":
+ self.tool_index += 1
+ tool_use = ChatCompletionToolCallChunk(
+ id=content_block_start["content_block"]["id"],
+ type="function",
+ function=ChatCompletionToolCallFunctionChunk(
+ name=_stream_tool_name,
+ arguments="",
+ ),
+ index=self.tool_index,
+ )
# Track server tool use inputs for code_interpreter_results.
# The initial input in content_block_start is typically {}
# for streaming; the full input arrives via input_json_delta
# and is assembled at content_block_stop.
if content_block_start["content_block"]["type"] == "server_tool_use":
self._current_server_tool_id = content_block_start["content_block"]["id"]
+ self._current_server_tool_name = _stream_tool_name
tool_input: Final = content_block_start["content_block"].get("input", {})
self._server_tool_inputs[self._current_server_tool_id] = tool_input
- if _stream_tool_name == "web_search":
+ if _stream_tool_name in HOSTED_WEB_TOOL_NAMES:
self._web_search_calls[self._current_server_tool_id] = build_web_search_call(
self._current_server_tool_id,
tool_input,
{"content": []},
status="in_progress",
+ tool_name=_stream_tool_name,
)
provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
# Include caller information if present (for programmatic tool calling)
if "caller" in content_block_start["content_block"]:
caller_data: Final = content_block_start["content_block"]["caller"]
- if caller_data:
+ if caller_data and tool_use is not None:
tool_use["caller"] = cast(dict[str, Any], caller_data)
elif content_block_start["content_block"]["type"] == "redacted_thinking":
(
@@ -1104,7 +1111,7 @@ class ModelResponseIterator:
# The full content comes in content_block_start, not in deltas
# See: https://github.com/BerriAI/litellm/issues/17737
self.web_search_results.append(content_block_start["content_block"])
- self._complete_web_search_call(content_block_start["content_block"])
+ self._complete_web_search_call(content_block_start["content_block"], "web_search")
provider_specific_fields["web_search_results"] = self.web_search_results
provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
elif content_type == "web_fetch_tool_result":
@@ -1112,7 +1119,9 @@ class ModelResponseIterator:
# The full content comes in content_block_start, not in deltas
# Fixes: https://github.com/BerriAI/litellm/issues/18137
self.web_search_results.append(content_block_start["content_block"])
+ self._complete_web_search_call(content_block_start["content_block"], "web_fetch")
provider_specific_fields["web_search_results"] = self.web_search_results
+ provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
else:
self.tool_results.append(content_block_start["content_block"])
provider_specific_fields["tool_results"] = self.tool_results
@@ -1127,7 +1136,7 @@ class ModelResponseIterator:
# check if tool call content block - only for tool_use and server_tool_use blocks
if self.current_content_block_type in ("tool_use", "server_tool_use"):
is_empty: Final = self.check_empty_tool_call_args()
- if is_empty:
+ if is_empty and self.current_content_block_type == "tool_use":
tool_use = ChatCompletionToolCallChunk(
id=None,
type="function",
@@ -1151,8 +1160,24 @@ class ModelResponseIterator:
self._server_tool_inputs[self._current_server_tool_id] = json.loads(args)
except (json.JSONDecodeError, TypeError):
pass
- if self._refresh_pending_web_search_call(self._current_server_tool_id):
+ server_tool_input: Final = self._server_tool_inputs.get(self._current_server_tool_id, {})
+ if self._refresh_pending_web_search_call(
+ self._current_server_tool_id, self._current_server_tool_name
+ ):
provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
+ self._server_tool_calls = (
+ *self._server_tool_calls,
+ ChatCompletionToolCallChunk(
+ id=self._current_server_tool_id,
+ type="function",
+ function=ChatCompletionToolCallFunctionChunk(
+ name=self._current_server_tool_name,
+ arguments=args or json.dumps(server_tool_input),
+ ),
+ index=len(self._server_tool_calls),
+ ),
+ )
+ provider_specific_fields["server_tool_calls"] = list(self._server_tool_calls)
self._current_server_tool_id = None
if self.is_response_format_tool:
text = self._flush_response_format_buffer()
diff --git a/litellm/llms/anthropic/chat/transformation.py b/litellm/llms/anthropic/chat/transformation.py
index a6f6ec3d598..775eb6eda39 100644
--- a/litellm/llms/anthropic/chat/transformation.py
+++ b/litellm/llms/anthropic/chat/transformation.py
@@ -6,10 +6,11 @@ from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from functools import partial
from itertools import chain
from types import MappingProxyType
-from typing import TYPE_CHECKING, Any, ClassVar, Final, NoReturn, cast
+from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Final, NoReturn, cast
import httpx
-from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from pydantic import BaseModel, BeforeValidator, ConfigDict, TypeAdapter, ValidationError
from typing_extensions import ReadOnly, TypedDict
import litellm
@@ -110,11 +111,11 @@ from litellm.types.llms.openai import (
ChatCompletionToolCallChunk,
ChatCompletionToolCallFunctionChunk,
ChatCompletionToolParam,
- OpenAIChatCompletionFinishReason,
OpenAIMcpServerTool,
OpenAIWebSearchOptions,
)
from litellm.types.responses.main import (
+ HOSTED_WEB_TOOL_NAMES,
OutputCodeInterpreterCall,
build_code_interpreter_log_outputs,
build_web_search_call,
@@ -150,6 +151,7 @@ from ..common_utils import (
stop_reason_provider_specific_fields,
strip_advisor_blocks_from_messages,
tools_marked_for_a_tool_config_point,
+ validated_object_mapping,
)
if TYPE_CHECKING:
@@ -187,9 +189,9 @@ _ANTHROPIC_TOOL_NAME_MAX_LEN: Final = 128
_DEFER_LOADING_ADAPTER: Final = TypeAdapter(bool)
_TOOL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_TOOLS_ADAPTER: Final = TypeAdapter(list[object])
-_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
+_CONTENT_BLOCKS_ADAPTER: Final = TypeAdapter(tuple[object, ...])
_RESPONSE_BLOCKS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
-_PENDING_WEB_SEARCH_RESULT: Final = MappingProxyType({"content": ()})
+_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_REQUEST_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_NATIVE_OUTPUT_STRING_FORMATS: Final = frozenset(
("date-time", "time", "date", "duration", "email", "hostname", "uri", "ipv4", "ipv6", "uuid")
@@ -346,6 +348,51 @@ def _validated_defer_loading(value: object) -> bool:
raise ValueError("defer_loading must be a boolean") from exc
+def _hosted_web_tool_use(block: object) -> tuple[str, str, object] | None:
+ match block:
+ case {
+ "type": "server_tool_use",
+ "name": str() as name,
+ "id": str() as call_id,
+ "input": object() as tool_input,
+ } if name in HOSTED_WEB_TOOL_NAMES:
+ return call_id, name, tool_input
+ case _:
+ return None
+
+
+def _server_tool_use_id(block: object) -> str | None:
+ match block:
+ case {"type": "server_tool_use", "id": str() as call_id}:
+ return call_id
+ case _:
+ return None
+
+
+def _dict_or_empty(value: object) -> object:
+ return validated_object_mapping(value) or {}
+
+
+_RESPONSE_BODY_ADAPTER: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(
+ Annotated[Mapping[str, object], BeforeValidator(_dict_or_empty)]
+)
+
+
+def _server_tool_use_ids(completion_response: Mapping[str, object]) -> frozenset[str]:
+ blocks: Final = _CONTENT_BLOCKS_ADAPTER.validate_python(completion_response.get("content") or ())
+ return frozenset(filter(None, map(_server_tool_use_id, blocks)))
+
+
+def _hosted_web_call(call_id: str, tool_name: str, tool_input: object, result: object) -> ResponseFunctionWebSearch:
+ return build_web_search_call(
+ tool_id=call_id,
+ tool_input=tool_input,
+ result=result,
+ status=None if result is not None else "in_progress",
+ tool_name=tool_name,
+ )
+
+
class _AnthropicUsageIteration(TypedDict, total=False):
"""One entry of the ``usage.iterations`` array on an Anthropic response."""
@@ -3079,25 +3126,16 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
self,
web_search_results: Sequence[object],
completion_response: Mapping[str, object],
- ) -> list[object]:
- blocks: Final = _RESPONSE_BLOCKS.validate_python(completion_response.get("content") or ())
- results_by_id: Final = {
- tool_use_id: result
+ ) -> list[ResponseFunctionWebSearch]:
+ blocks: Final = _CONTENT_BLOCKS_ADAPTER.validate_python(completion_response.get("content") or ())
+ results_by_call: Final = {
+ (tool_use_id, result.get("type")): result
for result in _RESPONSE_BLOCKS.validate_python(web_search_results)
- if result.get("type") == "web_search_tool_result"
- and isinstance((tool_use_id := result.get("tool_use_id")), str)
+ if isinstance((tool_use_id := result.get("tool_use_id")), str)
}
return [
- build_web_search_call(
- tool_id=call_id,
- tool_input=block.get("input"),
- result=results_by_id.get(call_id, _PENDING_WEB_SEARCH_RESULT),
- status=None if call_id in results_by_id else "in_progress",
- )
- for block in blocks
- if block.get("type") == "server_tool_use"
- and block.get("name") == "web_search"
- and isinstance((call_id := block.get("id")), str)
+ _hosted_web_call(call_id, tool_name, tool_input, results_by_call.get((call_id, f"{tool_name}_tool_result")))
+ for call_id, tool_name, tool_input in filter(None, map(_hosted_web_tool_use, blocks))
]
def _build_provider_specific_fields(
@@ -3109,6 +3147,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
tool_results: Sequence[_AnthropicToolResultBlock] | None,
compaction_blocks: Sequence[object] | None,
tool_calls: list[ChatCompletionToolCallChunk],
+ server_tool_use_ids: frozenset[str],
) -> dict[str, object]:
provider_specific_fields: Final[dict[str, object]] = {
"citations": citations,
@@ -3148,6 +3187,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if compaction_blocks is not None:
provider_specific_fields["compaction_blocks"] = compaction_blocks
+ server_tool_calls: Final = [tc for tc in tool_calls if tc["id"] in server_tool_use_ids]
+ if server_tool_calls:
+ provider_specific_fields["server_tool_calls"] = server_tool_calls
+
stop_reason_fields: Final = stop_reason_provider_specific_fields(
STOP_REASON_FIELDS.validate_python(completion_response)
)
@@ -3207,6 +3250,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
):
text_content = prefix_prompt + text_content
+ server_tool_use_ids: Final = _server_tool_use_ids(_RESPONSE_BODY_ADAPTER.validate_python(completion_response))
+ client_tool_calls: Final = [tc for tc in tool_calls if tc["id"] not in server_tool_use_ids]
provider_specific_fields: Final = self._build_provider_specific_fields(
completion_response,
citations,
@@ -3215,11 +3260,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
tool_results,
compaction_blocks,
tool_calls,
+ server_tool_use_ids,
)
json_mode_message, tool_calls_for_message, json_extra_content = self._resolve_json_mode_non_streaming(
json_mode=json_mode,
- tool_calls=tool_calls,
+ tool_calls=client_tool_calls,
response_format_schema=response_format_schema,
)
merged_text = text_content or ""
@@ -3258,10 +3304,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
model_response.choices[0].message = _message
model_response.hidden_params["original_response"] = completion_response["content"]
- model_response.choices[0].finish_reason = cast(
- OpenAIChatCompletionFinishReason,
- map_finish_reason(completion_response["stop_reason"]),
- )
+ model_response.choices[0].finish_reason = map_finish_reason(completion_response["stop_reason"])
usage: Final = self.calculate_usage(
usage_object=completion_response["usage"],
diff --git a/litellm/llms/anthropic/common_utils.py b/litellm/llms/anthropic/common_utils.py
index 275cc5cbe5c..5d4957fa166 100644
--- a/litellm/llms/anthropic/common_utils.py
+++ b/litellm/llms/anthropic/common_utils.py
@@ -6,7 +6,7 @@ import copy
import json
import os
import re
-from collections.abc import Container, Mapping, MutableMapping, Sequence
+from collections.abc import Callable, Container, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
from datetime import datetime, timezone
from itertools import chain
@@ -70,10 +70,26 @@ from litellm.types.llms.anthropic import (
AnthropicThinkingParam,
)
from litellm.types.llms.base import LiteLLMBaseModel
-from litellm.types.llms.openai import AllMessageValues
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ ChatCompletionAssistantToolCall,
+ ChatCompletionToolCallFunctionChunk,
+)
from litellm.types.proxy.auth.special_headers import SpecialHeaders
from litellm.types.proxy.model_listing import ModelInfoResponse
-from litellm.types.utils import LlmProviders
+from litellm.types.responses.main import (
+ ResponseFunctionWebSearch,
+ web_search_call_tool_id,
+ with_hosted_web_tool_arguments,
+)
+from litellm.types.utils import (
+ ChatCompletionMessageCustomToolCall,
+ ChatCompletionMessageToolCall,
+ Delta,
+ LlmProviders,
+ Message,
+ chat_completion_tool_call_from_dict,
+)
_MessageT = TypeVar("_MessageT")
@@ -126,6 +142,11 @@ _DOTTED_VERSION_RE: Final = re.compile(r"(\d)\.(\d)")
_CLAUDE_CODE_BILLING_HEADER_PREFIX: Final = "x-anthropic-billing-header:"
OBJECT_MAPPING_ADAPTER: Final = TypeAdapter(dict[object, object])
OBJECT_LIST_ADAPTER: Final = TypeAdapter(list[object])
+_SERVER_TOOL_CALLS_ADAPTER: Final = TypeAdapter(tuple[ChatCompletionAssistantToolCall, ...])
+_SERVER_TOOL_CALL_ADAPTER: Final = TypeAdapter(ChatCompletionAssistantToolCall)
+_SERVER_CALL_FIELD_KEYS: Final = ("server_tool_calls", "web_search_calls")
+_WEB_SEARCH_CALL_ADAPTER: Final = TypeAdapter(ResponseFunctionWebSearch)
+_TOOL_CALL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_DECODED_JSON: Final = TypeAdapter(object)
_JSON_VALUE: Final = TypeAdapter(JsonValue)
@@ -2290,6 +2311,261 @@ def _replayed_server_tool_use(block: object) -> _ReplayedServerToolUse | None:
return None
+def anthropic_server_tool_call_params(message: object) -> tuple[ChatCompletionAssistantToolCall, ...]:
+ fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields"))
+ try:
+ return _SERVER_TOOL_CALLS_ADAPTER.validate_python(fields.get("server_tool_calls", ()) if fields else ())
+ except ValidationError:
+ return ()
+
+
+def anthropic_server_tool_calls(message: object) -> tuple[ChatCompletionMessageToolCall, ...]:
+ return tuple(_message_tool_call_from_params(call) for call in anthropic_server_tool_call_params(message))
+
+
+def tool_calls_with_server_tool_calls(
+ message: object,
+) -> tuple[ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall, ...]:
+ client_tool_calls: Final = tuple(
+ call
+ for tool_call in parts_of(message_field(message, "tool_calls"))
+ if (call := _client_message_tool_call(tool_call)) is not None
+ )
+ return (*anthropic_server_tool_calls(message), *client_tool_calls)
+
+
+def anthropic_server_call_fields(holder: object) -> Mapping[str, object]:
+ fields: Final = validated_object_mapping(message_field(holder, "provider_specific_fields")) or {}
+ return MappingProxyType({key: fields[key] for key in _SERVER_CALL_FIELD_KEYS if fields.get(key)})
+
+
+def message_dict_with_server_tool_calls(message: object) -> object:
+ mapping: Final = validated_object_mapping(message)
+ server_tool_calls: Final = anthropic_server_tool_call_params(message)
+ if mapping is None or not server_tool_calls:
+ return message
+ return {**mapping, "tool_calls": [*server_tool_calls, *parts_of(mapping.get("tool_calls"))]}
+
+
+def _server_tool_call_param(tool_call: object) -> ChatCompletionAssistantToolCall | None:
+ try:
+ call: Final = _SERVER_TOOL_CALL_ADAPTER.validate_python(tool_call)
+ except ValidationError:
+ return None
+ return call if _server_tool_call_id(call).startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX) else None
+
+
+def message_dict_with_server_tool_calls_split_out(message: object) -> object:
+ mapping: Final = validated_object_mapping(message)
+ tool_calls: Final = parts_of(mapping.get("tool_calls")) if mapping is not None else ()
+ server_tool_calls: Final = tuple(
+ call for tool_call in tool_calls if (call := _server_tool_call_param(tool_call)) is not None
+ )
+ if mapping is None or not server_tool_calls:
+ return message
+ fields: Final = validated_object_mapping(mapping.get("provider_specific_fields")) or {}
+ changed: Final = changed_server_tool_call_arguments(anthropic_server_tool_call_params(message), server_tool_calls)
+ client_tool_calls: Final = [tool_call for tool_call in tool_calls if _server_tool_call_param(tool_call) is None]
+ return {
+ **mapping,
+ "tool_calls": client_tool_calls or None,
+ "provider_specific_fields": {
+ **fields,
+ "server_tool_calls": list(server_tool_calls),
+ **(
+ {
+ "web_search_calls": _rewritten_web_search_calls(
+ fields["web_search_calls"], changed, _with_hosted_web_call_arguments
+ )
+ }
+ if changed and fields.get("web_search_calls") is not None
+ else {}
+ ),
+ },
+ }
+
+
+def with_server_tool_call_arguments(
+ call: ChatCompletionAssistantToolCall, arguments: str
+) -> ChatCompletionAssistantToolCall:
+ return ChatCompletionAssistantToolCall(
+ id=call["id"],
+ type=call["type"],
+ function=ChatCompletionToolCallFunctionChunk(name=call["function"].get("name"), arguments=arguments),
+ )
+
+
+def write_anthropic_server_tool_calls(
+ holder: Message | Delta, rewritten: tuple[ChatCompletionAssistantToolCall, ...]
+) -> None:
+ original: Final = anthropic_server_tool_call_params(holder)
+ if rewritten == original:
+ return
+ holder.provider_specific_fields = { # rebind-ok: a guardrail rewrite must land on the message it scanned
+ **(holder.provider_specific_fields or {}),
+ "server_tool_calls": list(rewritten),
+ }
+ write_anthropic_hosted_web_call_arguments(holder, changed_server_tool_call_arguments(original, rewritten))
+
+
+def write_request_server_tool_calls(message: object, rewritten: tuple[ChatCompletionAssistantToolCall, ...]) -> None:
+ original: Final = anthropic_server_tool_call_params(message)
+ if rewritten == original:
+ return
+ write_request_hosted_web_call_arguments(message, changed_server_tool_call_arguments(original, rewritten))
+ fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields")) or {}
+ if not isinstance(message, dict):
+ return
+ message["provider_specific_fields"] = { # rebind-ok: a guardrail rewrite must land on the request it scanned
+ **fields,
+ "server_tool_calls": list(rewritten),
+ }
+
+
+def write_request_hosted_web_call_arguments(message: object, arguments_by_tool_id: Mapping[str, str]) -> None:
+ fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields"))
+ if not arguments_by_tool_id or not isinstance(message, dict) or fields is None:
+ return
+ if fields.get("web_search_calls") is None:
+ return
+ message["provider_specific_fields"] = { # rebind-ok: a rewritten call's hosted copy must not keep its old input
+ **fields,
+ "web_search_calls": _rewritten_web_search_calls(
+ fields["web_search_calls"], arguments_by_tool_id, _jsonable_hosted_web_call
+ ),
+ }
+
+
+def _server_tool_call_id(call: ChatCompletionAssistantToolCall) -> str:
+ return call["id"] or ""
+
+
+def _server_tool_call_arguments(call: ChatCompletionAssistantToolCall) -> str:
+ return call["function"].get("arguments") or ""
+
+
+def changed_server_tool_call_arguments(
+ original: Sequence[ChatCompletionAssistantToolCall], rewritten: Sequence[ChatCompletionAssistantToolCall]
+) -> Mapping[str, str]:
+ original_arguments: Final = MappingProxyType(
+ {_server_tool_call_id(call): _server_tool_call_arguments(call) for call in original}
+ )
+ return MappingProxyType(
+ {
+ _server_tool_call_id(call): _server_tool_call_arguments(call)
+ for call in rewritten
+ if _server_tool_call_arguments(call) != original_arguments.get(_server_tool_call_id(call))
+ }
+ )
+
+
+def write_anthropic_hosted_web_call_arguments(holder: Message | Delta, arguments_by_tool_id: Mapping[str, str]) -> None:
+ fields: Final = holder.provider_specific_fields
+ if not arguments_by_tool_id or not fields or fields.get("web_search_calls") is None:
+ return
+ holder.provider_specific_fields = { # rebind-ok: the hosted copy of a rewritten call must not keep its old input
+ **fields,
+ "web_search_calls": _rewritten_web_search_calls(
+ fields["web_search_calls"], arguments_by_tool_id, _with_hosted_web_call_arguments
+ ),
+ }
+
+
+def _rewritten_web_search_calls(
+ calls: object,
+ arguments_by_tool_id: Mapping[str, str],
+ rewrite: Callable[[object, Mapping[str, str]], object],
+) -> object:
+ mapping: Final = validated_object_mapping(calls)
+ if mapping is not None:
+ return {key: rewrite(call, arguments_by_tool_id) for key, call in mapping.items()}
+ if not isinstance(calls, list):
+ return calls
+ return [rewrite(call, arguments_by_tool_id) for call in parts_of(calls)]
+
+
+def _jsonable_hosted_web_call(item: object, arguments_by_tool_id: Mapping[str, str]) -> object:
+ call: Final = _with_hosted_web_call_arguments(item, arguments_by_tool_id)
+ if not isinstance(call, ResponseFunctionWebSearch):
+ return call
+ return _WEB_SEARCH_CALL_ADAPTER.dump_python(call, mode="json", exclude_none=True)
+
+
+def _with_hosted_web_call_arguments(item: object, arguments_by_tool_id: Mapping[str, str]) -> object:
+ try:
+ call: Final = _WEB_SEARCH_CALL_ADAPTER.validate_python(item)
+ except ValidationError:
+ return item
+ arguments: Final = arguments_by_tool_id.get(web_search_call_tool_id(call.id) or "")
+ if arguments is None:
+ return item
+ return call.model_copy(update={"action": with_hosted_web_tool_arguments(call.action, arguments)})
+
+
+def without_anthropic_server_tool_calls(holder: Message, tool_ids: frozenset[str]) -> None:
+ fields: Final = holder.provider_specific_fields or {}
+ holder.provider_specific_fields = { # rebind-ok: a denied call must leave every copy of the message it was on
+ **fields,
+ "server_tool_calls": [
+ call for call in anthropic_server_tool_call_params(holder) if _server_tool_call_id(call) not in tool_ids
+ ],
+ **(
+ {"web_search_calls": _web_search_calls_without(fields["web_search_calls"], tool_ids)}
+ if fields.get("web_search_calls") is not None
+ else {}
+ ),
+ }
+
+
+def _web_search_calls_without(calls: object, tool_ids: frozenset[str]) -> object:
+ mapping: Final = validated_object_mapping(calls)
+ if mapping is not None:
+ return {key: call for key, call in mapping.items() if _hosted_web_call_tool_id(call) not in tool_ids}
+ items: Final = validated_object_list(calls)
+ if items is None:
+ return calls
+ return [call for call in items if _hosted_web_call_tool_id(call) not in tool_ids]
+
+
+def _hosted_web_call_tool_id(item: object) -> str | None:
+ try:
+ return web_search_call_tool_id(_WEB_SEARCH_CALL_ADAPTER.validate_python(item).id)
+ except ValidationError:
+ return None
+
+
+def write_anthropic_server_tool_call_arguments(holder: Message | Delta, index: int, arguments: str) -> None:
+ write_anthropic_server_tool_calls(
+ holder,
+ tuple(
+ with_server_tool_call_arguments(call, arguments) if call_index == index else call
+ for call_index, call in enumerate(anthropic_server_tool_call_params(holder))
+ ),
+ )
+
+
+def _message_tool_call_from_params(call: ChatCompletionAssistantToolCall) -> ChatCompletionMessageToolCall:
+ return ChatCompletionMessageToolCall(id=call["id"], type=call["type"], function=dict(call["function"]))
+
+
+def _client_message_tool_call(
+ tool_call: object,
+) -> ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall | None:
+ match tool_call:
+ case ChatCompletionMessageToolCall() | ChatCompletionMessageCustomToolCall():
+ return tool_call
+ case _:
+ try:
+ fields: Final = _TOOL_CALL_FIELDS_ADAPTER.validate_python(tool_call)
+ return chat_completion_tool_call_from_dict(
+ MappingProxyType({key: value for key, value in fields.items() if key != "custom"})
+ if fields.get("type") == "function"
+ else fields
+ )
+ except (TypeError, ValidationError):
+ return None
+
+
def tool_call_is_rebuilt_as_server_tool_use(
tool_call_id: object, provider_specific_fields: object, answered_tool_ids: Container[str] = frozenset()
) -> bool:
diff --git a/litellm/llms/anthropic/messages/adapters/handler.py b/litellm/llms/anthropic/messages/adapters/handler.py
index da120cad0e1..64c7ac62d03 100644
--- a/litellm/llms/anthropic/messages/adapters/handler.py
+++ b/litellm/llms/anthropic/messages/adapters/handler.py
@@ -13,7 +13,7 @@ from typing_extensions import TypedDict
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.asyncify import run_async_function
-from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
+from litellm.litellm_core_utils.core_helpers import as_str_mapping, drops_unsupported_params
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObject
from litellm.litellm_core_utils.tool_fields import (
ROUTES_WITHOUT_MCP_SERVERS,
@@ -676,6 +676,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
request_data,
custom_llm_provider=target_provider,
api_base=api_base if isinstance(api_base, str) else None,
+ deployment_model_info=as_str_mapping(provider_kwargs.get("model_info")),
drop_params=drop_params,
)
diff --git a/litellm/llms/anthropic/messages/adapters/transformation.py b/litellm/llms/anthropic/messages/adapters/transformation.py
index 5379dcd6d68..43eaa37aba7 100644
--- a/litellm/llms/anthropic/messages/adapters/transformation.py
+++ b/litellm/llms/anthropic/messages/adapters/transformation.py
@@ -46,6 +46,25 @@ def _thought_signature(provider_specific_fields: object) -> str | None:
return signature if isinstance(signature, str) else None
+def _object_sequence(value: object) -> Sequence[object] | None:
+ return value if isinstance(value, (list, tuple)) else None
+
+
+def _server_tool_blocks(message: object) -> tuple[Mapping[str, object], ...]:
+ server_tool_calls: Final = anthropic_server_tool_call_params(message)
+ if not server_tool_calls:
+ return ()
+ fields: Final = _as_string_mapping(_optional_attr(message, "provider_specific_fields")) or {}
+ return tuple(
+ convert_to_anthropic_tool_invoke(
+ server_tool_calls,
+ web_search_results=_object_sequence(fields.get("web_search_results")),
+ tool_results=_object_sequence(fields.get("tool_results")),
+ server_tool_call_ids=frozenset(tool_call["id"] for tool_call in server_tool_calls),
+ )
+ )
+
+
def _compaction_blocks(provider_specific_fields: object) -> tuple[Mapping[str, object], ...]:
fields: Final = _as_string_mapping(provider_specific_fields)
raw_blocks: Final = fields.get("compaction_blocks") if fields is not None else None
@@ -127,6 +146,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
)
from litellm.litellm_core_utils.prompt_templates.factory import (
THOUGHT_SIGNATURE_SEPARATOR,
+ convert_to_anthropic_tool_invoke,
)
from litellm.litellm_core_utils.prompt_templates.prompt_cache_breakpoint_utils import (
with_prompt_cache_breakpoint,
@@ -145,6 +165,7 @@ from litellm.litellm_core_utils.reasoning_effort_utils import (
reasoning_effort_from_thinking_silently,
)
from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
eager_input_streaming_flag,
is_empty_unsigned_thinking_block,
normalize_anthropic_tool_use_id,
@@ -237,6 +258,29 @@ def target_supports_mid_conversation_system(model: str | None, custom_llm_provid
return supports_mid_conversation_system(model=model, custom_llm_provider=custom_llm_provider)
+def _is_azure_responses_target(
+ model: str | None,
+ custom_llm_provider: str | None,
+ deployment_model_info: Mapping[str, object] | None,
+) -> bool:
+ if not model:
+ return False
+ if custom_llm_provider not in (None, LlmProviders.AZURE.value):
+ return False
+ if custom_llm_provider is None and not model.startswith("azure/"):
+ return False
+
+ from litellm.responses.bridge_check import responses_api_bridge_check
+
+ resolved_model, provider, _, _ = litellm.get_llm_provider(model=model, custom_llm_provider=custom_llm_provider)
+ model_info, _ = responses_api_bridge_check(
+ model=resolved_model,
+ custom_llm_provider=provider,
+ deployment_model_info=deployment_model_info,
+ )
+ return model_info.get("mode") == "responses"
+
+
def _chat_tool_param(function_chunk: ChatCompletionToolParamFunctionChunk, tool: object) -> ChatCompletionToolParam:
eager_input_streaming: Final = eager_input_streaming_flag(tool)
if eager_input_streaming is None:
@@ -272,6 +316,7 @@ class AnthropicAdapter:
*,
custom_llm_provider: str | None = None,
api_base: str | None = None,
+ deployment_model_info: Mapping[str, object] | None = None,
drop_params: bool = True,
) -> tuple[ChatCompletionRequest | None, dict[str, str]]:
"""
@@ -307,6 +352,7 @@ class AnthropicAdapter:
anthropic_message_request=request_body,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
+ deployment_model_info=deployment_model_info,
drop_params=drop_params,
report_reasoning_drops=True,
)
@@ -502,10 +548,15 @@ class LiteLLMAnthropicMessagesAdapter:
model: str | None = None,
*,
custom_llm_provider: str | None = None,
+ deployment_model_info: Mapping[str, object] | None = None,
preserve_midturn_system: bool = False,
) -> list[AllMessageValues]:
new_messages: Final[list[AllMessageValues]] = []
- replayable_messages: Final = strip_encrypted_reasoning_blocks_from_anthropic_messages(messages)
+ replayable_messages: Final = (
+ messages
+ if _is_azure_responses_target(model, custom_llm_provider, deployment_model_info)
+ else strip_encrypted_reasoning_blocks_from_anthropic_messages(messages)
+ )
leading_count: Final = next(
(i for i, m in enumerate(replayable_messages) if not is_system_role_message(m)),
len(replayable_messages),
@@ -1289,6 +1340,7 @@ class LiteLLMAnthropicMessagesAdapter:
*,
custom_llm_provider: str | None = None,
api_base: str | None = None,
+ deployment_model_info: Mapping[str, object] | None = None,
preserve_midturn_system: bool = False,
drop_params: bool = True,
report_reasoning_drops: bool = False,
@@ -1314,6 +1366,7 @@ class LiteLLMAnthropicMessagesAdapter:
messages=messages_list,
model=anthropic_message_request.get("model"),
custom_llm_provider=custom_llm_provider,
+ deployment_model_info=deployment_model_info,
preserve_midturn_system=preserve_midturn_system,
)
## ADD SYSTEM MESSAGE TO MESSAGES
@@ -1476,6 +1529,7 @@ class LiteLLMAnthropicMessagesAdapter:
).model_dump()
)
+ new_content.extend(dict(block) for block in _server_tool_blocks(choice.message))
# Handle text content
if choice.message.content is not None and (choice.message.content != "" or not compaction_blocks):
new_content.append(
diff --git a/litellm/llms/anthropic/messages/responses_adapters/transformation.py b/litellm/llms/anthropic/messages/responses_adapters/transformation.py
index 110e9fc40ff..b4455f1a4e2 100644
--- a/litellm/llms/anthropic/messages/responses_adapters/transformation.py
+++ b/litellm/llms/anthropic/messages/responses_adapters/transformation.py
@@ -584,6 +584,12 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
if block.get("type") == "text" and isinstance(text := block.get("text"), str) and text
)
+ @staticmethod
+ def _resolve_output_format_strict(strict: bool | None) -> bool:
+ if strict is None:
+ return True
+ return strict
+
def translate_request(
self,
anthropic_request: AnthropicMessagesRequest,
@@ -693,7 +699,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
"type": "json_schema",
"name": "structured_output",
"schema": schema,
- "strict": output_format.get("strict", False),
+ "strict": self._resolve_output_format_strict(output_format.get("strict")),
}
}
diff --git a/litellm/llms/base_llm/guardrail_translation/utils.py b/litellm/llms/base_llm/guardrail_translation/utils.py
index fb886e42780..900a9c0712e 100644
--- a/litellm/llms/base_llm/guardrail_translation/utils.py
+++ b/litellm/llms/base_llm/guardrail_translation/utils.py
@@ -4,11 +4,14 @@ import json
from collections.abc import Callable, Iterator, Mapping, Sequence
from typing import Final, TypeVar, cast # noqa: TID251 # a rebuilt chat row has no typed constructor across roles
+from openai.types.responses import ResponseFunctionWebSearch
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
from litellm.llms.custom_httpx.sse_lines import split_sse_lines
+from litellm.types.llms.anthropic import claude_hosted_web_tool_call
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicUsage
from litellm.types.llms.openai import AllMessageValues, ResponseAPIUsage
+from litellm.types.responses.main import with_hosted_web_tool_arguments
_DECODED_JSON: Final = TypeAdapter(object)
@@ -135,6 +138,36 @@ def stream_item_field(item: object, field: str) -> object | None:
return getattr(item, field, None)
+def web_search_call(item: object) -> ResponseFunctionWebSearch | None:
+ if isinstance(item, ResponseFunctionWebSearch):
+ return item
+ if stream_item_field(item, "type") != "web_search_call":
+ return None
+ try:
+ return ResponseFunctionWebSearch.model_validate(item.model_dump() if isinstance(item, BaseModel) else item)
+ except ValidationError:
+ return None
+
+
+def claude_hosted_web_tool_arguments(item: object) -> str | None:
+ search: Final = web_search_call(item)
+ hosted_call: Final = None if search is None else claude_hosted_web_tool_call(search)
+ return None if hosted_call is None else hosted_call[2]
+
+
+def with_web_search_call_arguments(item: object, arguments: str) -> object:
+ search: Final = web_search_call(item)
+ if search is None:
+ return item
+ action: Final = with_hosted_web_tool_arguments(search.action, arguments)
+ if isinstance(item, ResponseFunctionWebSearch):
+ return item.model_copy(update={"action": action})
+ if isinstance(item, BaseModel):
+ return item.model_copy(update={"action": action.model_dump()})
+ fields: Final = cast("Mapping[str, object]", item) # cast-ok: a validated web_search_call dict is str-keyed JSON
+ return {**fields, "action": action.model_dump()}
+
+
def stream_item_fingerprint(item: object) -> str:
plain: Final = item.model_dump() if isinstance(item, BaseModel) else item
return json.dumps(plain, sort_keys=True, default=str)
diff --git a/litellm/llms/bedrock/chat/converse_transformation.py b/litellm/llms/bedrock/chat/converse_transformation.py
index b93d40738ea..12c5031628b 100644
--- a/litellm/llms/bedrock/chat/converse_transformation.py
+++ b/litellm/llms/bedrock/chat/converse_transformation.py
@@ -960,7 +960,8 @@ class AmazonConverseConfig(BaseConfig):
) -> bool:
"""Check if the Bedrock model takes native structured outputs (outputConfig.textFormat) for this request.
- ``supports_native_structured_output`` in ``litellm.model_cost`` turns it on for every request, and
+ ``supports_native_structured_output`` or ``supports_bedrock_native_structured_output_by_default`` in
+ ``litellm.model_cost`` turns it on for every request, and
``supports_bedrock_native_structured_output`` (the deployment's ``model_info`` first) also when Claude
will think, which the json tool path would drop. When the request does not ask for thinking, a model
that cannot force the json tool gets it unless that key is false.
diff --git a/litellm/llms/bedrock/chat/native_output_fallback.py b/litellm/llms/bedrock/chat/native_output_fallback.py
index 79742d0810c..7d5662e2024 100644
--- a/litellm/llms/bedrock/chat/native_output_fallback.py
+++ b/litellm/llms/bedrock/chat/native_output_fallback.py
@@ -9,7 +9,9 @@ from litellm.llms.bedrock.common_utils import (
NativeOutputFallback,
bedrock_strict_tools_fallback,
is_bedrock_native_output_rejection,
+ is_bedrock_stream_permission_denial,
warn_bedrock_native_output_rejected,
+ warn_bedrock_native_output_stream_denied,
warn_bedrock_strict_tools_dropped,
)
from litellm.types.utils import ModelResponse
@@ -25,22 +27,55 @@ NativeOutputFallbackFactory: TypeAlias = Callable[[], NativeOutputFallback | Non
class _SelectedFallback:
fallback: NativeOutputFallback
native_output_retry: bool
+ buffered_native_output: bool = False
_RetryFor: TypeAlias = Callable[[BaseLLMException], _SelectedFallback | None]
+_NextRetryFor: TypeAlias = Callable[[BaseLLMException, _SelectedFallback], _SelectedFallback | None]
-def _native_output_retry(
- error: BaseLLMException, fallback: NativeOutputFallbackFactory | None, model: str
+def _native_output_rejection_retry(
+ error: BaseLLMException, fallback: NativeOutputFallbackFactory, model: str
) -> NativeOutputFallback | None:
- if fallback is None or not is_bedrock_native_output_rejection(error.status_code, error.message):
- return None
retry: Final = fallback()
if retry is not None:
warn_bedrock_native_output_rejected(model, error.message)
return retry
+def _buffered(retry: NativeOutputFallback) -> NativeOutputFallback:
+ return NativeOutputFallback(
+ optional_params={**retry.optional_params, "fake_stream": True}, litellm_params=retry.litellm_params
+ )
+
+
+def _buffered_stream_retry(fallback: NativeOutputFallbackFactory, model: str) -> NativeOutputFallback | None:
+ retry: Final = fallback()
+ if retry is None or retry.optional_params.get("fake_stream") is not True:
+ return None
+ warn_bedrock_native_output_stream_denied(model)
+ return retry
+
+
+def _native_output_retry(
+ error: BaseLLMException,
+ fallback: NativeOutputFallbackFactory | None,
+ buffered_stream_fallback: NativeOutputFallbackFactory | None,
+ model: str,
+) -> _SelectedFallback | None:
+ if fallback is not None and is_bedrock_native_output_rejection(error.status_code, error.message):
+ rejection_retry: Final = _native_output_rejection_retry(error, fallback, model)
+ return None if rejection_retry is None else _SelectedFallback(rejection_retry, native_output_retry=True)
+ if buffered_stream_fallback is not None and is_bedrock_stream_permission_denial(error.status_code, error.message):
+ stream_retry: Final = _buffered_stream_retry(buffered_stream_fallback, model)
+ return (
+ None
+ if stream_retry is None
+ else _SelectedFallback(stream_retry, native_output_retry=True, buffered_native_output=True)
+ )
+ return None
+
+
def _strict_tools_retry(
error: BaseLLMException, fallback: NativeOutputFallbackFactory | None, model: str
) -> NativeOutputFallback | None:
@@ -55,12 +90,13 @@ def _strict_tools_retry(
def _fallback_for(
error: BaseLLMException,
fallback: NativeOutputFallbackFactory | None,
+ buffered_stream_fallback: NativeOutputFallbackFactory | None,
strict_tools_fallback: NativeOutputFallbackFactory | None,
model: str,
) -> _SelectedFallback | None:
- native_output_retry: Final = _native_output_retry(error, fallback, model)
+ native_output_retry: Final = _native_output_retry(error, fallback, buffered_stream_fallback, model)
if native_output_retry is not None:
- return _SelectedFallback(native_output_retry, native_output_retry=True)
+ return native_output_retry
strict_tools_retry: Final = _strict_tools_retry(error, strict_tools_fallback, model)
if strict_tools_retry is None:
return None
@@ -87,12 +123,81 @@ def _strict_tools_retry_after_native_output_retry(
return composed_retry
+def _next_retry(
+ error: BaseLLMException,
+ retry: _SelectedFallback,
+ fallback: NativeOutputFallbackFactory | None,
+ strict_tools_fallback: NativeOutputFallbackFactory | None,
+ model: str,
+) -> _SelectedFallback | None:
+ if (
+ retry.buffered_native_output
+ and fallback is not None
+ and is_bedrock_native_output_rejection(error.status_code, error.message)
+ ):
+ rejection_retry: Final = _native_output_rejection_retry(error, fallback, model)
+ return (
+ None if rejection_retry is None else _SelectedFallback(_buffered(rejection_retry), native_output_retry=True)
+ )
+ composed_retry: Final = _strict_tools_retry_after_native_output_retry(error, retry, strict_tools_fallback, model)
+ return None if composed_retry is None else _SelectedFallback(composed_retry, native_output_retry=False)
+
+
+def _attempt_or_next_retry(
+ attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputAttemptResult | _SelectedFallback:
+ try:
+ return attempt(retry.fallback)
+ except BaseLLMException as error:
+ next_retry: Final = next_retry_for(error, retry)
+ if next_retry is None:
+ raise
+ return next_retry
+
+
+def _attempt_retries(
+ attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputAttemptResult:
+ after_first_retry: Final = _attempt_or_next_retry(attempt, retry, next_retry_for)
+ if not isinstance(after_first_retry, _SelectedFallback):
+ return after_first_retry
+ after_second_retry: Final = _attempt_or_next_retry(attempt, after_first_retry, next_retry_for)
+ if not isinstance(after_second_retry, _SelectedFallback):
+ return after_second_retry
+ return attempt(after_second_retry.fallback)
+
+
+async def _await_or_next_retry(
+ attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputResponse | _SelectedFallback:
+ try:
+ result: Final = attempt(retry.fallback)
+ return await result if isinstance(result, Coroutine) else result
+ except BaseLLMException as error:
+ next_retry: Final = next_retry_for(error, retry)
+ if next_retry is None:
+ raise
+ return next_retry
+
+
+async def _await_retries(
+ attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputResponse:
+ after_first_retry: Final = await _await_or_next_retry(attempt, retry, next_retry_for)
+ if not isinstance(after_first_retry, _SelectedFallback):
+ return after_first_retry
+ after_second_retry: Final = await _await_or_next_retry(attempt, after_first_retry, next_retry_for)
+ if not isinstance(after_second_retry, _SelectedFallback):
+ return after_second_retry
+ last_attempt: Final = attempt(after_second_retry.fallback)
+ return await last_attempt if isinstance(last_attempt, Coroutine) else last_attempt
+
+
async def _await_with_native_output_fallback(
first_attempt: Coroutine[object, object, NativeOutputResponse],
attempt: NativeOutputAttempt,
retry_for: _RetryFor,
- strict_tools_fallback: NativeOutputFallbackFactory | None,
- model: str,
+ next_retry_for: _NextRetryFor,
) -> NativeOutputResponse:
try:
return await first_attempt
@@ -100,17 +205,7 @@ async def _await_with_native_output_fallback(
retry: Final = retry_for(error)
if retry is None:
raise
- try:
- second_attempt: Final = attempt(retry.fallback)
- return await second_attempt if isinstance(second_attempt, Coroutine) else second_attempt
- except BaseLLMException as error:
- composed_retry: Final = _strict_tools_retry_after_native_output_retry(
- error, retry, strict_tools_fallback, model
- )
- if composed_retry is None:
- raise
- third_attempt: Final = attempt(composed_retry)
- return await third_attempt if isinstance(third_attempt, Coroutine) else third_attempt
+ return await _await_retries(attempt, retry, next_retry_for)
def call_with_native_output_fallback(
@@ -118,11 +213,19 @@ def call_with_native_output_fallback(
fallback: NativeOutputFallbackFactory | None,
model: str,
strict_tools_fallback: NativeOutputFallbackFactory | None = None,
+ buffered_stream_fallback: NativeOutputFallbackFactory | None = None,
) -> NativeOutputAttemptResult:
- if fallback is None and strict_tools_fallback is None:
+ if fallback is None and strict_tools_fallback is None and buffered_stream_fallback is None:
return attempt(None)
retry_for: Final = partial(
- _fallback_for, fallback=fallback, strict_tools_fallback=strict_tools_fallback, model=model
+ _fallback_for,
+ fallback=fallback,
+ buffered_stream_fallback=buffered_stream_fallback,
+ strict_tools_fallback=strict_tools_fallback,
+ model=model,
+ )
+ next_retry_for: Final = partial(
+ _next_retry, fallback=fallback, strict_tools_fallback=strict_tools_fallback, model=model
)
try:
first_attempt: Final = attempt(None)
@@ -130,15 +233,7 @@ def call_with_native_output_fallback(
retry: Final = retry_for(error)
if retry is None:
raise
- try:
- return attempt(retry.fallback)
- except BaseLLMException as retry_error:
- composed_retry: Final = _strict_tools_retry_after_native_output_retry(
- retry_error, retry, strict_tools_fallback, model
- )
- if composed_retry is None:
- raise
- return attempt(composed_retry)
+ return _attempt_retries(attempt, retry, next_retry_for)
if isinstance(first_attempt, Coroutine):
- return _await_with_native_output_fallback(first_attempt, attempt, retry_for, strict_tools_fallback, model)
+ return _await_with_native_output_fallback(first_attempt, attempt, retry_for, next_retry_for)
return first_attempt
diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py
index cc81266af9e..0f8694a7e46 100644
--- a/litellm/llms/bedrock/common_utils.py
+++ b/litellm/llms/bedrock/common_utils.py
@@ -297,9 +297,16 @@ def _bedrock_model_supports(model: str, key: str, custom_llm_provider: str = "be
def _bedrock_native_output_enabled(
model: str, custom_llm_provider: str, model_info: Mapping[str, object] | None
) -> bool:
- deployment_flag: Final = (model_info or {}).get("supports_native_structured_output")
+ deployment_info: Final = model_info or {}
+ deployment_flag: Final = deployment_info.get("supports_native_structured_output")
if isinstance(deployment_flag, bool):
return deployment_flag
+ return deployment_info.get("supports_bedrock_native_structured_output") is not False and (
+ bedrock_model_always_takes_native_output(model, custom_llm_provider)
+ )
+
+
+def bedrock_model_always_takes_native_output(model: str, custom_llm_provider: str = "bedrock") -> bool:
return _bedrock_model_supports(model, "supports_native_structured_output", custom_llm_provider)
@@ -443,6 +450,18 @@ def warn_bedrock_native_output_rejected(model: str, error_text: str) -> None:
)
+def is_bedrock_stream_permission_denial(status_code: int, error_text: str) -> bool:
+ return status_code == 403 and "bedrock:InvokeModelWithResponseStream" in error_text
+
+
+def warn_bedrock_native_output_stream_denied(model: str) -> None:
+ verbose_logger.warning(
+ "Bedrock denied bedrock:InvokeModelWithResponseStream for model=%s, so litellm sent the structured output "
+ "request again as a buffered stream. Allow that action to stream it.",
+ model,
+ )
+
+
@dataclass(frozen=True, slots=True)
class NativeOutputFallback:
optional_params: Mapping[str, object]
@@ -543,6 +562,22 @@ def json_object_or_none(value: object) -> Mapping[str, object] | None:
BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES: Final = 64
+def _response_format_json_schema(response_format: object) -> Mapping[str, object] | None:
+ json_schema: Final = json_object_or_none((json_object_or_none(response_format) or {}).get("json_schema"))
+ return json_object_or_none((json_schema or {}).get("schema"))
+
+
+def bedrock_batch_record_takes_native_output(
+ model: str, response_format: object, model_info: Mapping[str, object] | None = None
+) -> bool:
+ schema: Final = _response_format_json_schema(response_format)
+ return _bedrock_native_output_enabled(model, "bedrock", model_info) and (
+ schema is None
+ or schema_property_count(schema) <= BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES
+ or not _bedrock_model_supports(model, "supports_bedrock_native_structured_output_by_default")
+ )
+
+
def bedrock_cleaned_output_schema(
model: str, schema: Mapping[str, object], custom_llm_provider: str = "bedrock"
) -> Mapping[str, object] | None:
@@ -1560,15 +1595,29 @@ def bedrock_converse_accepts_redacted_thinking(model: str, base_model: str | Non
its own. An ARN that hides its model, like an application inference profile, falls back to the
deployment's ``base_model`` and counts as non-Claude without one.
"""
+ return _is_claude(_converse_target_model(model, base_model))
+
+
+def bedrock_converse_rejects_signed_thinking(model: str, base_model: str | None) -> bool:
+ if base_model is None and "." not in get_bedrock_base_model(model):
+ return False
+ return not _is_claude_behind_any_region_prefix(_converse_target_model(model, base_model))
+
+
+def _converse_target_model(model: str, base_model: str | None) -> str:
if base_model is None or "." in get_bedrock_base_model(model):
- return _is_claude(model)
- return _is_claude(base_model)
+ return model
+ return base_model
def _is_claude(model: str) -> bool:
return get_bedrock_base_model(model).startswith(("anthropic", "claude"))
+def _is_claude_behind_any_region_prefix(model: str) -> bool:
+ return _is_claude(model) or get_bedrock_base_model(model).partition(".")[2].startswith("anthropic.")
+
+
def bedrock_converse_supports_parallel_tool_use_config(model: str) -> bool:
return any(
(litellm.model_cost.get(candidate) or {}).get("supports_parallel_tool_use_config") is True
diff --git a/litellm/llms/bedrock/files/transformation.py b/litellm/llms/bedrock/files/transformation.py
index 4d137876890..ef54c85c131 100644
--- a/litellm/llms/bedrock/files/transformation.py
+++ b/litellm/llms/bedrock/files/transformation.py
@@ -71,6 +71,7 @@ from litellm.utils import get_llm_provider, get_optional_params
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import (
BedrockError,
+ bedrock_batch_record_takes_native_output,
deployment_model_info,
merge_bedrock_aws_request_params,
resolve_s3_encryption_key_id,
@@ -167,6 +168,20 @@ _LITELLM_PARAMS_THE_MAPPER_TAKES: Final = frozenset({"allowed_openai_params"})
_MAPPED_PARAMS_THE_REQUEST_HANDLER_STRIPS: Final = frozenset({"json_mode"})
+_UPLOAD_PARAMS: Final = TypeAdapter(dict[str, object])
+
+
+def _upload_params(params: object) -> Mapping[str, object]:
+ return _UPLOAD_PARAMS.validate_python(params) if isinstance(params, Mapping) else {}
+
+
+def _upload_deployment_model_info(litellm_params: object) -> Mapping[str, object]:
+ upload_params: Final = _upload_params(litellm_params)
+ return deployment_model_info(upload_params) or deployment_model_info(
+ _upload_params(upload_params.get("litellm_metadata"))
+ )
+
+
def _invoke_route_model(model: str) -> str:
return f"invoke/{_strip_llm_routing_prefix(model).removeprefix('invoke/')}"
@@ -1034,6 +1049,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
openai_request_body: Mapping[str, Any],
model: str,
provider: str | None = None,
+ model_info: Mapping[str, object] | None = None,
) -> dict[str, object]:
"""
Transform OpenAI request body to Bedrock-compatible modelInput
@@ -1055,7 +1071,15 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
)
config: Final = AmazonAnthropicClaudeConfig()
- litellm_params: Final = without_bedrock_native_output_opt_in({})
+ litellm_params: Final = without_bedrock_native_output_opt_in(
+ {
+ "model_info": {
+ "supports_native_structured_output": bedrock_batch_record_takes_native_output(
+ _invoke_route_model(model), optional_params.get("response_format"), model_info
+ )
+ }
+ }
+ )
mapped_params = get_optional_params(
model=_invoke_route_model(model),
custom_llm_provider="bedrock",
@@ -1150,25 +1174,32 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
]
def transform_batch_input_lines(
- self, batch_input_lines: Sequence[_BatchInputLine], target_model: str = ""
+ self,
+ batch_input_lines: Sequence[_BatchInputLine],
+ target_model: str = "",
+ model_info: Mapping[str, object] | None = None,
) -> tuple[_BedrockBatchRecord, ...]:
return tuple(
- self._transform_batch_input_line(idx, batch_input_line, target_model)
+ self._transform_batch_input_line(idx, batch_input_line, target_model, model_info)
for idx, batch_input_line in enumerate(batch_input_lines)
)
def _transform_batch_input_line(
- self, idx: int, batch_input_line: _BatchInputLine, target_model: str
+ self, idx: int, batch_input_line: _BatchInputLine, target_model: str, model_info: Mapping[str, object] | None
) -> _BedrockBatchRecord:
try:
- return self._transform_openai_batch_record(idx, batch_input_line.record, target_model)
+ return self._transform_openai_batch_record(idx, batch_input_line.record, target_model, model_info)
except _BATCH_RECORD_CONVERSION_ERRORS as e:
raise _batch_line_error(
batch_input_line.line_number, f"could not be converted to a Bedrock batch record: {e}"
) from e
def _transform_openai_batch_record(
- self, idx: int, openai_record: _OpenAIBatchRecord, target_model: str
+ self,
+ idx: int,
+ openai_record: _OpenAIBatchRecord,
+ target_model: str,
+ model_info: Mapping[str, object] | None = None,
) -> _BedrockBatchRecord:
import litellm
@@ -1192,6 +1223,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
openai_request_body=self._transform_batch_body_to_chat_body(openai_body, record_kind),
model=model_for_transform,
provider=provider,
+ model_info=model_info,
)
)
return _BedrockBatchRecord(
@@ -1211,7 +1243,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
model: str,
create_file_data: CreateFileRequest,
optional_params: dict,
- litellm_params: dict,
+ litellm_params: dict[str, object],
) -> bytes | str | dict:
"""
Transform file request and return a pre-signed request for S3.
@@ -1237,7 +1269,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
litellm_params_model: Final = litellm_params.get("model")
target_model: Final = model or (litellm_params_model if isinstance(litellm_params_model, str) else "")
bedrock_jsonl_content: Final = self.transform_batch_input_lines(
- batch_input_lines, target_model=target_model
+ batch_input_lines, target_model=target_model, model_info=_upload_deployment_model_info(litellm_params)
)
file_content = "\n".join(json.dumps(item) for item in bedrock_jsonl_content)
elif isinstance(extracted_file_data_content, bytes):
diff --git a/litellm/llms/chatgpt/responses/transformation.py b/litellm/llms/chatgpt/responses/transformation.py
index 3c7d0b8d673..638a55452d2 100644
--- a/litellm/llms/chatgpt/responses/transformation.py
+++ b/litellm/llms/chatgpt/responses/transformation.py
@@ -143,6 +143,8 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
"reasoning",
"previous_response_id",
"truncation",
+ "text",
+ "max_output_tokens",
}
filtered: Final = {k: v for k, v in request.items() if k in allowed_keys}
@@ -218,7 +220,10 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
)
continue
- if event_type == ResponsesAPIStreamEvents.RESPONSE_COMPLETED:
+ if event_type in (
+ ResponsesAPIStreamEvents.RESPONSE_COMPLETED,
+ ResponsesAPIStreamEvents.RESPONSE_INCOMPLETE,
+ ):
# Real OUTPUT_ITEM_DONE events take precedence at any given
# output_index, but text-only items at indices without a
# matching OUTPUT_ITEM_DONE must still be preserved (e.g.
diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
index 169c21cf635..df82891f789 100644
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -897,7 +897,7 @@ class BaseLLMHTTPHandler:
fake_stream = (
fake_stream
- or optional_params.pop("fake_stream", False)
+ or optional_params.pop("fake_stream", False) is True
or provider_config.should_fake_stream(model=model, custom_llm_provider=custom_llm_provider, stream=stream)
)
@@ -920,7 +920,7 @@ class BaseLLMHTTPHandler:
api_key=api_key,
model=model,
optional_params=optional_params,
- stream=stream,
+ stream=False if fake_stream else stream,
litellm_params=litellm_params,
)
@@ -1056,7 +1056,7 @@ class BaseLLMHTTPHandler:
provider_config=provider_config,
fake_stream=fake_stream,
)
- if provider_config.has_custom_stream_wrapper is True:
+ if provider_config.has_custom_stream_wrapper is True and not fake_stream:
return provider_config.get_sync_custom_stream_wrapper(
model=model,
custom_llm_provider=custom_llm_provider,
@@ -1225,7 +1225,7 @@ class BaseLLMHTTPHandler:
json_mode: bool | None = None,
signed_json_body: bytes | None = None,
):
- if provider_config.has_custom_stream_wrapper is True:
+ if provider_config.has_custom_stream_wrapper is True and not fake_stream:
return await provider_config.get_async_custom_stream_wrapper(
model=model,
custom_llm_provider=custom_llm_provider,
diff --git a/litellm/llms/fireworks_ai/responses/transformation.py b/litellm/llms/fireworks_ai/responses/transformation.py
index d599f0b65f7..a45c25b27f3 100644
--- a/litellm/llms/fireworks_ai/responses/transformation.py
+++ b/litellm/llms/fireworks_ai/responses/transformation.py
@@ -1,10 +1,11 @@
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
from types import MappingProxyType
-from typing import TYPE_CHECKING, Final
+from typing import TYPE_CHECKING, Final, cast # noqa: TID251 # the base signature leaves the params bag untyped
from urllib.parse import unquote
import httpx
+from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.responses.transformation import (
STANDARD_MODE_WITHOUT_CONTEXT_REASONING_FIELD_VALUES,
@@ -20,8 +21,9 @@ from litellm.llms.fireworks_ai.common_utils import (
)
from litellm.llms.openai.responses.instruction_items import developer_items_as_system, instruction_items_folded
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
+from litellm.responses.additional_tools import HoistedAdditionalTools, hoist_additional_tools
from litellm.secret_managers.main import get_secret_str
-from litellm.types.llms.openai import ResponseInputParam
+from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
from litellm.types.responses.main import DeleteResponseResult
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
@@ -98,6 +100,23 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
headers: dict[str, str], # mutable-ok: overrides the base class signature
) -> dict[str, object]: # mutable-ok: overrides the base class signature
validated_input: Final = self._validate_input_param(input)
+ tools_param: Final = response_api_optional_request_params.get("tools")
+ hoisted: Final = (
+ hoist_additional_tools(
+ validated_input,
+ cast( # cast-ok: the base signature leaves the params bag untyped
+ "Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None", tools_param
+ ),
+ drop_params=drops_unsupported_params(litellm_params.drop_params),
+ )
+ if tools_param is None or isinstance(tools_param, (list, tuple))
+ else HoistedAdditionalTools(input=validated_input, tools=(), hoisted=())
+ )
+ hoisted_params: Final = (
+ {**response_api_optional_request_params, "tools": list(hoisted.tools)}
+ if hoisted.hoisted
+ else response_api_optional_request_params
+ )
model_info: Final = deployment_model_info(litellm_params)
declared: Final = declared_mid_conversation_system_support(model, self.custom_llm_provider.value, model_info)
keeps_developer_items_in_place: Final = declared is True and (
@@ -107,14 +126,9 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
folded: Final = (
None
if keeps_developer_items_in_place
- else instruction_items_folded(
- validated_input, response_api_optional_request_params, "user" if declared is False else "system"
- )
- )
- placed_input, placed_params = folded or (
- developer_items_as_system(validated_input),
- response_api_optional_request_params,
+ else instruction_items_folded(hoisted.input, hoisted_params, "user" if declared is False else "system")
)
+ placed_input, placed_params = folded or (developer_items_as_system(hoisted.input), hoisted_params)
request: Final = super().transform_responses_api_request(
model=resolve_fireworks_resource_name(model),
input=placed_input,
diff --git a/litellm/llms/litellm_proxy/chat/transformation.py b/litellm/llms/litellm_proxy/chat/transformation.py
index 4183f900e26..6f55e251f47 100644
--- a/litellm/llms/litellm_proxy/chat/transformation.py
+++ b/litellm/llms/litellm_proxy/chat/transformation.py
@@ -4,9 +4,13 @@ Translate from OpenAI's `/v1/chat/completions` to VLLM's `/v1/chat/completions`
from typing import TYPE_CHECKING, Final
-from litellm.constants import OPENAI_CHAT_COMPLETION_PARAMS
+from pydantic import BaseModel, ConfigDict
+
+import litellm
+from litellm.constants import NON_LLM_CONNECTION_TIMEOUT, OPENAI_CHAT_COMPLETION_PARAMS
from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.types.router import LiteLLM_Params
+from litellm.utils import add_path_to_api_base
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
@@ -14,6 +18,18 @@ if TYPE_CHECKING:
from litellm.types.llms.openai import AllMessageValues
+class _ProxyModel(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ id: str
+
+
+class _ProxyModelList(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ data: tuple[_ProxyModel, ...]
+
+
class LiteLLMProxyChatConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> list:
params_list: Final = super().get_supported_openai_params(model)
@@ -53,8 +69,13 @@ class LiteLLMProxyChatConfig(OpenAIGPTConfig):
api_base, api_key = self._get_openai_compatible_provider_info(api_base, api_key)
if api_base is None:
raise ValueError("api_base not set for LiteLLM Proxy route. Set in env via `LITELLM_PROXY_API_BASE`")
- models: Final = super().get_models(api_key=api_key, api_base=api_base)
- return [f"litellm_proxy/{model}" for model in models]
+ response: Final = litellm.module_level_client.client.get(
+ url=add_path_to_api_base(api_base, "/v1/models"),
+ headers={"Authorization": f"Bearer {api_key}"},
+ timeout=NON_LLM_CONNECTION_TIMEOUT,
+ )
+ response.raise_for_status()
+ return [f"litellm_proxy/{model.id}" for model in _ProxyModelList.model_validate_json(response.content).data]
@staticmethod
def get_api_key(api_key: str | None = None) -> str | None:
diff --git a/litellm/llms/openai/chat/guardrail_translation/handler.py b/litellm/llms/openai/chat/guardrail_translation/handler.py
index 5c5204ef2f4..c5488bc8a98 100644
--- a/litellm/llms/openai/chat/guardrail_translation/handler.py
+++ b/litellm/llms/openai/chat/guardrail_translation/handler.py
@@ -21,6 +21,7 @@ from collections.abc import Iterator, Mapping, Sequence
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Union, cast
+from pydantic import TypeAdapter, ValidationError
from typing_extensions import NotRequired, ReadOnly, TypedDict
import litellm
@@ -29,6 +30,13 @@ from litellm.completion_extras.litellm_responses_transformation.transformation i
replayed_tool_search_tool_names,
)
from litellm.litellm_core_utils.streaming_chunk_builder_utils import stream_chunk_builder
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ changed_server_tool_call_arguments,
+ write_anthropic_hosted_web_call_arguments,
+ write_anthropic_server_tool_calls,
+ write_request_server_tool_calls,
+)
from litellm.llms.base_llm.guardrail_translation.base_translation import (
BaseTranslation,
StreamingScanKey,
@@ -50,7 +58,12 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
stream_item_items,
unappliable_request_rewrite,
)
-from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ ChatCompletionAssistantToolCall,
+ ChatCompletionToolCallFunctionChunk,
+ ChatCompletionToolParam,
+)
from litellm.types.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
coerce_stream_holdback_value,
)
@@ -139,14 +152,27 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
scan_only_tool_results=scan_only_tool_results,
)
+ structured_messages: Final = self.get_structured_messages(data)
+ server_tool_call_tasks: Final = tuple(
+ _input_server_tool_call_tasks(
+ structured_messages or [],
+ skip_system_message=skip_system,
+ skip_tool_message=skip_tool,
+ scan_only_tool_results=scan_only_tool_results,
+ )
+ )
+ tool_calls_with_server_calls: Final = [
+ *tool_calls_to_check,
+ *(_server_tool_call_input(call) for _, call in server_tool_call_tasks),
+ ]
+
# Step 2: Apply guardrail to all texts and tool calls in batch
- if texts_to_check or tool_calls_to_check:
+ if texts_to_check or tool_calls_with_server_calls:
inputs: Final = GenericGuardrailAPIInputs(texts=texts_to_check)
if images_to_check:
inputs["images"] = images_to_check
- if tool_calls_to_check:
- inputs["tool_calls"] = tool_calls_to_check
- structured_messages: Final = self.get_structured_messages(data)
+ if tool_calls_with_server_calls:
+ inputs["tool_calls"] = tool_calls_with_server_calls
scoped_message_indices: Final = scoped_structured_message_indices(
structured_messages or [],
scan_only_tool_results=scan_only_tool_results,
@@ -218,6 +244,12 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls=guardrailed_tool_calls,
task_mappings=tool_call_task_mappings,
)
+ if len(guardrailed_tool_calls) == len(tool_calls_with_server_calls):
+ self._apply_guardrail_responses_to_input_server_tool_calls(
+ messages=structured_messages or [],
+ guardrailed_tool_calls=guardrailed_tool_calls[len(tool_calls_to_check) :],
+ task_mappings=tuple(mapping for mapping, _ in server_tool_call_tasks),
+ )
elif (
not images_to_check
@@ -386,6 +418,22 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# Replace the tool call with the guardrailed version
message_tool_calls[tool_call_idx] = guardrailed_tool_call
+ @staticmethod
+ def _apply_guardrail_responses_to_input_server_tool_calls(
+ messages: Sequence[object],
+ guardrailed_tool_calls: Sequence[object],
+ task_mappings: Sequence[tuple[int, int]],
+ ) -> None:
+ guardrailed_by_task: Final = MappingProxyType(dict(zip(task_mappings, guardrailed_tool_calls)))
+ for msg_idx in sorted(frozenset(msg_idx for msg_idx, _ in task_mappings)):
+ write_request_server_tool_calls(
+ messages[msg_idx],
+ tuple(
+ _rewritten_server_tool_call(server_tool_call, guardrailed_by_task.get((msg_idx, server_idx)))
+ for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(messages[msg_idx]))
+ ),
+ )
+
async def process_output_response(
self,
response: "ModelResponse",
@@ -436,9 +484,14 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
text_task_mappings=text_task_mappings,
tool_call_task_mappings=tool_call_task_mappings,
)
+ server_tool_call_tasks: Final = tuple(_server_tool_call_tasks(response.choices))
+ tool_calls_with_server_calls: Final = [
+ *tool_calls_to_check,
+ *(_server_tool_call_input(call) for _, call in server_tool_call_tasks),
+ ]
# Step 2: Apply guardrail to all texts and tool calls in batch
- if texts_to_check or tool_calls_to_check:
+ if texts_to_check or tool_calls_with_server_calls:
# Use the real request_data if provided (proxy path), otherwise
# create a standalone dict (SDK / direct-call path).
if request_data is None:
@@ -452,8 +505,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
inputs: Final = GenericGuardrailAPIInputs(texts=texts_to_check)
if images_to_check:
inputs["images"] = images_to_check
- if tool_calls_to_check:
- inputs["tool_calls"] = tool_calls_to_check
+ if tool_calls_with_server_calls:
+ inputs["tool_calls"] = tool_calls_with_server_calls
# Include model information from the response if available
if hasattr(response, "model") and response.model:
inputs["model"] = response.model
@@ -469,8 +522,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
returned_tool_calls: Final = guardrailed_inputs.get("tool_calls")
guardrailed_tool_calls: Final[list[dict[str, object]]] = (
cast(list[dict[str, object]], returned_tool_calls)
- if isinstance(returned_tool_calls, list) and len(returned_tool_calls) == len(tool_calls_to_check)
- else tool_calls_to_check
+ if isinstance(returned_tool_calls, list)
+ and len(returned_tool_calls) == len(tool_calls_with_server_calls)
+ else tool_calls_with_server_calls
)
# Step 3: Map guardrail responses back to original response structure
@@ -488,6 +542,11 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls=guardrailed_tool_calls,
task_mappings=tool_call_task_mappings,
)
+ self._apply_guardrail_responses_to_output_server_tool_calls(
+ response=response,
+ guardrailed_tool_calls=guardrailed_tool_calls[len(tool_calls_to_check) :],
+ task_mappings=tuple(mapping for mapping, _ in server_tool_call_tasks),
+ )
verbose_proxy_logger.debug("OpenAI Chat Completions: Processed output response: %s", response)
@@ -681,6 +740,10 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
pre_guardrail_tool_calls=pre_guardrail_tool_calls,
guardrail_name=guardrail_to_apply.guardrail_name or "unknown",
)
+ self._write_ended_stream_server_tool_call_rewrites(
+ responses_so_far=responses_so_far,
+ guardrailed_response=model_response,
+ )
@staticmethod
def _rebuild_ended_stream_per_choice(
@@ -934,6 +997,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if choice.message.tool_calls and isinstance(choice.message.tool_calls, list):
if len(choice.message.tool_calls) > 0:
return True
+ if anthropic_server_tool_call_params(choice.message):
+ return True
elif isinstance(response, ModelResponseStream):
for streaming_choice in response.choices:
if isinstance(streaming_choice, litellm.StreamingChoices):
@@ -944,6 +1009,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if streaming_choice.delta.tool_calls and isinstance(streaming_choice.delta.tool_calls, list):
if len(streaming_choice.delta.tool_calls) > 0:
return True
+ if anthropic_server_tool_call_params(streaming_choice.delta):
+ return True
return False
def _extract_output_text_images_and_tool_calls(
@@ -1094,6 +1161,22 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if "name" in func_dict:
existing_tool_call.function.name = func_dict["name"]
+ @staticmethod
+ def _apply_guardrail_responses_to_output_server_tool_calls(
+ response: "ModelResponse",
+ guardrailed_tool_calls: Sequence[object],
+ task_mappings: Sequence[tuple[int, int]],
+ ) -> None:
+ guardrailed_by_task: Final = MappingProxyType(dict(zip(task_mappings, guardrailed_tool_calls)))
+ for choice_idx, choice in enumerate(response.choices):
+ write_anthropic_server_tool_calls(
+ choice.message,
+ tuple(
+ _rewritten_server_tool_call(server_tool_call, guardrailed_by_task.get((choice_idx, server_idx)))
+ for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(choice.message))
+ ),
+ )
+
@staticmethod
def _string_choice_contents(response: "ModelResponse") -> tuple[str | None, ...]:
return tuple(
@@ -1206,6 +1289,42 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for fragment in tail:
fragment.function.arguments = ""
+ @staticmethod
+ def _write_ended_stream_server_tool_call_rewrites(
+ responses_so_far: Sequence["ModelResponseStream"],
+ guardrailed_response: "ModelResponse",
+ ) -> None:
+ guardrailed_by_id: Final = MappingProxyType(
+ {
+ (guardrailed_response.choices[choice_position].index, server_tool_call["id"]): server_tool_call
+ for (choice_position, _), server_tool_call in _server_tool_call_tasks(guardrailed_response.choices)
+ }
+ )
+ if not guardrailed_by_id:
+ return
+ original_by_id: Final = MappingProxyType(dict(_streamed_server_tool_calls(responses_so_far)))
+ changed_by_choice: Final = MappingProxyType(
+ {
+ choice_index: changed_server_tool_call_arguments(
+ _choice_server_tool_calls(original_by_id, choice_index),
+ _choice_server_tool_calls(guardrailed_by_id, choice_index),
+ )
+ for choice_index in frozenset(index for index, _ in guardrailed_by_id)
+ }
+ )
+ for response in responses_so_far:
+ for choice in response.choices:
+ write_anthropic_server_tool_calls(
+ choice.delta,
+ tuple(
+ guardrailed_by_id.get((choice.index, server_tool_call["id"]), server_tool_call)
+ for server_tool_call in anthropic_server_tool_call_params(choice.delta)
+ ),
+ )
+ write_anthropic_hosted_web_call_arguments(
+ choice.delta, changed_by_choice.get(choice.index, MappingProxyType({}))
+ )
+
async def _apply_guardrail_responses_to_output_streaming(
self,
responses: list["ModelResponseStream"],
@@ -1395,7 +1514,76 @@ def _stream_chunk_choices(item: object) -> Sequence[object]:
def _streamed_delta_tool_calls(delta: object) -> tuple[object, ...]:
function_call: Final = stream_item_field(delta, "function_call")
legacy: Final = () if function_call is None else (function_call,)
- return stream_item_items(delta, "tool_calls") + legacy
+ return anthropic_server_tool_call_params(delta) + stream_item_items(delta, "tool_calls") + legacy
+
+
+def _input_server_tool_call_tasks(
+ messages: Sequence[Mapping[str, object]],
+ *,
+ skip_system_message: bool,
+ skip_tool_message: bool,
+ scan_only_tool_results: bool,
+) -> Iterator[tuple[tuple[int, int], ChatCompletionAssistantToolCall]]:
+ for msg_idx, message in enumerate(messages):
+ if role_out_of_guardrail_scope(
+ str(message.get("role") or "").lower(),
+ skip_system_message=skip_system_message,
+ skip_tool_message=skip_tool_message,
+ scan_only_tool_results=scan_only_tool_results,
+ ):
+ continue
+ for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(message)):
+ yield (msg_idx, server_idx), server_tool_call
+
+
+def _server_tool_call_tasks(
+ choices: Sequence[object],
+) -> Iterator[tuple[tuple[int, int], ChatCompletionAssistantToolCall]]:
+ for choice_idx, choice in enumerate(choices):
+ if not isinstance(choice, Choices):
+ continue
+ for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(choice.message)):
+ yield (choice_idx, server_idx), server_tool_call
+
+
+def _streamed_server_tool_calls(
+ responses: Sequence["ModelResponseStream"],
+) -> Iterator[tuple[tuple[int, str | None], ChatCompletionAssistantToolCall]]:
+ for response in responses:
+ for choice in response.choices:
+ for server_tool_call in anthropic_server_tool_call_params(choice.delta):
+ yield (choice.index, server_tool_call["id"]), server_tool_call
+
+
+def _choice_server_tool_calls(
+ calls_by_id: Mapping[tuple[int, str | None], ChatCompletionAssistantToolCall], choice_index: int
+) -> tuple[ChatCompletionAssistantToolCall, ...]:
+ return tuple(call for (index, _), call in calls_by_id.items() if index == choice_index)
+
+
+def _server_tool_call_input(server_tool_call: ChatCompletionAssistantToolCall) -> dict[str, object]:
+ return {
+ "id": server_tool_call["id"],
+ "type": server_tool_call["type"],
+ "function": dict(server_tool_call["function"]),
+ }
+
+
+_TOOL_CALL_FUNCTION_ADAPTER: Final = TypeAdapter(ChatCompletionToolCallFunctionChunk)
+
+
+def _rewritten_server_tool_call(
+ server_tool_call: ChatCompletionAssistantToolCall, guardrailed: object
+) -> ChatCompletionAssistantToolCall:
+ try:
+ function: Final = _TOOL_CALL_FUNCTION_ADAPTER.validate_python(stream_item_field(guardrailed, "function"))
+ except ValidationError:
+ return server_tool_call
+ return ChatCompletionAssistantToolCall(
+ id=server_tool_call["id"],
+ type=server_tool_call["type"],
+ function={**server_tool_call["function"], **function},
+ )
def _released_choices(responses_so_far: Sequence[object]) -> Iterator[object]:
diff --git a/litellm/llms/openai/responses/guardrail_translation/handler.py b/litellm/llms/openai/responses/guardrail_translation/handler.py
index a18cbe13925..aa4dea196ff 100644
--- a/litellm/llms/openai/responses/guardrail_translation/handler.py
+++ b/litellm/llms/openai/responses/guardrail_translation/handler.py
@@ -71,11 +71,14 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
stream_item_fingerprint,
stream_item_items,
unappliable_request_rewrite,
+ web_search_call,
+ with_web_search_call_arguments,
)
from litellm.llms.openai.responses.guardrail_translation.tool_merge import (
append_returned_tools,
merge_guardrailed_tools,
)
+from litellm.responses.additional_tools import additional_tools_in_input
from litellm.responses.litellm_completion_transformation.compaction_items import (
compaction_summary,
decode_compaction_item,
@@ -86,12 +89,14 @@ from litellm.responses.litellm_completion_transformation.tool_search import is_s
from litellm.responses.litellm_completion_transformation.transformation import (
TOOL_CALL_OUTPUT_ITEM_TYPES,
LiteLLMCompletionResponsesConfig,
+ claude_web_search_call_tool_call,
tool_output_to_chat_content,
)
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.llms.openai import (
AllMessageValues,
BaseLiteLLMOpenAIResponseObject,
+ ChatCompletionAssistantToolCall,
ChatCompletionToolCallChunk,
ChatCompletionToolParam,
ContentPartAddedEvent,
@@ -266,6 +271,8 @@ _TERMINAL_ENVELOPE_EVENT_TYPES: Final = frozenset(
_TOOL_CALL_ITEM_TYPES: Final = frozenset({"function_call", "custom_tool_call"})
_TOOL_SEARCH_CALL_TYPE: Final = "tool_search_call"
+_WEB_SEARCH_CALL_TYPE: Final = "web_search_call"
+_ITEM_ID_KEYED_TOOL_CALL_TYPES: Final = frozenset({_TOOL_SEARCH_CALL_TYPE, _WEB_SEARCH_CALL_TYPE})
_OUTPUT_TOOL_CALL_ITEM_TYPES: Final = _TOOL_CALL_ITEM_TYPES | {_TOOL_SEARCH_CALL_TYPE}
_SCANNED_TOOL_SEARCH_NAME: Final = "tool_search"
_TOOL_SEARCH_ARGUMENTS_ADAPTER: Final = TypeAdapter(dict[str, object])
@@ -335,8 +342,27 @@ def _is_tool_call_item(item: object) -> bool:
return isinstance(item, Mapping) and item.get("type") in _TOOL_CALL_ITEM_TYPES
+def _claude_web_search_tool_call(item: object) -> ChatCompletionAssistantToolCall | None:
+ search: Final = web_search_call(item)
+ return None if search is None else claude_web_search_call_tool_call(search)
+
+
+def _claude_web_tool_call(item: object, index: int) -> ChatCompletionToolCallChunk | None:
+ tool_call: Final = _claude_web_search_tool_call(item)
+ if tool_call is None:
+ return None
+ return ChatCompletionToolCallChunk(id=tool_call["id"], type="function", function=tool_call["function"], index=index)
+
+
+def _is_output_tool_call_item(item: object) -> bool:
+ return (
+ stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
+ or _claude_web_search_tool_call(item) is not None
+ )
+
+
def _tool_call_output_item_mapping(item: object) -> Mapping[str, object] | None:
- if stream_item_field(item, "type") not in _OUTPUT_TOOL_CALL_ITEM_TYPES:
+ if not _is_output_tool_call_item(item):
return None
if isinstance(item, Mapping):
return cast("Mapping[str, object]", item) # cast-ok: output items are str-keyed JSON objects
@@ -348,7 +374,7 @@ def _tool_call_item_key(item: object) -> str | None:
if isinstance(call_id, str) and call_id:
return call_id
item_id: Final = stream_item_field(item, "id")
- if stream_item_field(item, "type") == _TOOL_SEARCH_CALL_TYPE and isinstance(item_id, str) and item_id:
+ if stream_item_field(item, "type") in _ITEM_ID_KEYED_TOOL_CALL_TYPES and isinstance(item_id, str) and item_id:
return item_id
return None
@@ -368,6 +394,17 @@ def _scanned_tool_call_item(item: Mapping[str, object]) -> Mapping[str, object]:
return {**item, "name": _SCANNED_TOOL_SEARCH_NAME, "arguments": json.dumps(item.get("arguments") or {})}
+def _scanned_tool_call(item: Mapping[str, object], index: int) -> ChatCompletionToolCallChunk:
+ web_tool_call: Final = _claude_web_tool_call(item, index)
+ if web_tool_call is not None:
+ return web_tool_call
+ return tool_call_dict_from_output_item(_scanned_tool_call_item(item), index)
+
+
+def _rewritten_web_search_call(item: object, payload: str | None) -> object:
+ return item if payload is None else with_web_search_call_arguments(item, payload)
+
+
def _tool_search_rewrite(payload: str) -> Mapping[str, object]:
try:
return {"arguments": _TOOL_SEARCH_ARGUMENTS_ADAPTER.validate_json(payload)}
@@ -953,9 +990,14 @@ class OpenAIResponsesHandler(BaseTranslation):
def extract_request_tool_names(self, data: Mapping[str, object]) -> list[str]:
"""Extract tool names from Responses API request (tools[].name for function
- and custom, tools[].server_label for mcp)."""
+ and custom, tools[].server_label for mcp), including tools carried by
+ ``additional_tools`` input items."""
names: Final[list[str]] = []
- for tool in list_field(data, "tools"):
+ request_tools: Final[tuple[object, ...]] = (
+ *list_field(data, "tools"),
+ *additional_tools_in_input(data.get("input")),
+ )
+ for tool in request_tools:
if not isinstance(tool, dict):
continue
if tool.get("type") in ("function", "custom") and tool.get("name"):
@@ -1322,17 +1364,22 @@ class OpenAIResponsesHandler(BaseTranslation):
# delivered, so a truncated buffer still reports text undeliverable. #
# ------------------------------------------------------------------ #
if final_chunk.get("type") == "response.output_item.done":
- finished_summary: Final = _output_compaction_summary(stream_item_field(final_chunk, "item"))
+ finished_item: Final = stream_item_field(final_chunk, "item")
+ finished_summary: Final = _output_compaction_summary(finished_item)
model_response_stream: Final = (
OpenAiResponsesToChatCompletionStreamIterator.translate_responses_chunk_to_openai_stream(final_chunk)
)
- tool_calls: Final = model_response_stream.choices[0].delta.tool_calls
+ finished_web_tool_call: Final = _claude_web_tool_call(finished_item, 0)
+ tool_calls: Final = [
+ *cast(list[ChatCompletionToolCallChunk], model_response_stream.choices[0].delta.tool_calls or []),
+ *(() if finished_web_tool_call is None else (finished_web_tool_call,)),
+ ]
if tool_calls or finished_summary is not None:
inputs = GenericGuardrailAPIInputs()
if finished_summary is not None:
inputs["texts"] = [finished_summary]
if tool_calls:
- inputs["tool_calls"] = cast(list[ChatCompletionToolCallChunk], tool_calls)
+ inputs["tool_calls"] = tool_calls
if hasattr(model_response_stream, "model") and model_response_stream.model:
inputs["model"] = model_response_stream.model
await guardrail_to_apply.apply_guardrail(
@@ -1629,7 +1676,7 @@ class OpenAIResponsesHandler(BaseTranslation):
if event_type not in _OUTPUT_ITEM_EVENT_TYPES:
return None
item: Final = stream_item_field(event, "item")
- return _tool_call_item_key(item) if stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES else None
+ return _tool_call_item_key(item) if _is_output_tool_call_item(item) else None
@staticmethod
def _write_tool_call_item(item: object, name: str | None, payload: str | None) -> object:
@@ -1643,6 +1690,8 @@ class OpenAIResponsesHandler(BaseTranslation):
return item
if item_type == _TOOL_SEARCH_CALL_TYPE:
return _rewritten_tool_search_call(item, payload)
+ if item_type == _WEB_SEARCH_CALL_TYPE:
+ return _rewritten_web_search_call(item, payload)
if name is not None:
OpenAIResponsesHandler._write_event_field(item, "name", name)
if payload is not None and isinstance(item_type, str) and item_type in _TOOL_CALL_PAYLOAD_FIELDS:
@@ -1698,7 +1747,7 @@ class OpenAIResponsesHandler(BaseTranslation):
stream_item_field(event, "type") in _TOOL_CALL_PAYLOAD_EVENT_TYPES
or (
stream_item_field(event, "type") in _OUTPUT_ITEM_EVENT_TYPES
- and stream_item_field(stream_item_field(event, "item"), "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
+ and _is_output_tool_call_item(stream_item_field(event, "item"))
)
for event in responses_so_far
)
@@ -1706,9 +1755,7 @@ class OpenAIResponsesHandler(BaseTranslation):
@staticmethod
def _completed_response_scan_key(response: object) -> StreamingScanKey:
output_items: Final = stream_item_items(response, "output")
- message_items: Final = tuple(
- item for item in output_items if stream_item_field(item, "type") not in _OUTPUT_TOOL_CALL_ITEM_TYPES
- )
+ message_items: Final = tuple(item for item in output_items if not _is_output_tool_call_item(item))
message_parts: Final = chain.from_iterable(stream_item_items(item, "content") for item in message_items)
part_texts: Final = tuple(
text for part in message_parts if isinstance(text := stream_item_field(part, "text"), str) and text
@@ -1716,11 +1763,7 @@ class OpenAIResponsesHandler(BaseTranslation):
summaries: Final = tuple(summary for item in output_items if (summary := _output_compaction_summary(item)))
return StreamingScanKey(
texts=(*part_texts, *summaries),
- tool_calls=tuple(
- stream_item_fingerprint(item)
- for item in output_items
- if stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
- ),
+ tool_calls=tuple(stream_item_fingerprint(item) for item in output_items if _is_output_tool_call_item(item)),
stream_ended=True,
)
@@ -1833,9 +1876,7 @@ class OpenAIResponsesHandler(BaseTranslation):
tool_call_item: Final = _tool_call_output_item_mapping(output_item)
if tool_call_item is not None:
if tool_calls_to_check is not None:
- tool_calls_to_check.append(
- tool_call_dict_from_output_item(_scanned_tool_call_item(tool_call_item), output_idx)
- )
+ tool_calls_to_check.append(_scanned_tool_call(tool_call_item, output_idx))
return
summary: Final = _output_compaction_summary(output_item)
diff --git a/litellm/llms/openai/responses/transformation.py b/litellm/llms/openai/responses/transformation.py
index 3d0abb1a375..314a1d22508 100644
--- a/litellm/llms/openai/responses/transformation.py
+++ b/litellm/llms/openai/responses/transformation.py
@@ -77,7 +77,9 @@ _NO_TOOL_UPDATE: Final[Mapping[str, object]] = MappingProxyType({})
_MODEL_FAMILIES_REJECTING_TOP_LEVEL_SCHEMA_COMBINATORS: Final = ("gpt-4", "gpt-3.5", "chatgpt-4o", "o1", "o3", "o4")
_PROVIDERS_WITH_OPENAI_SCHEMA_VALIDATOR: Final = frozenset({LlmProviders.AZURE, LlmProviders.OPENAI})
_PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS: Final = frozenset({LlmProviders.AZURE, LlmProviders.OPENAI})
-_PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING: Final = _PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS
+_PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING: Final = frozenset(
+ {LlmProviders.AZURE, LlmProviders.BEDROCK, LlmProviders.BEDROCK_MANTLE, LlmProviders.OPENAI}
+)
_ROLES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"system", "developer", "user"})
_PART_TYPES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"input_text", "input_image", "input_file"})
_INPUT_PART_LIST_KEYS: Final = frozenset({"content", "output"})
diff --git a/litellm/llms/vertex_ai/text_to_speech/transformation.py b/litellm/llms/vertex_ai/text_to_speech/transformation.py
index c0082bc5ca6..a674ca4130e 100644
--- a/litellm/llms/vertex_ai/text_to_speech/transformation.py
+++ b/litellm/llms/vertex_ai/text_to_speech/transformation.py
@@ -55,17 +55,45 @@ _LyriaVoice: TypeAlias = str | dict | None
_JSON_OBJECT: Final = TypeAdapter(Mapping[str, object], config=ConfigDict(hide_input_in_errors=True))
_JSON_ARRAY: Final = TypeAdapter(tuple[object, ...], config=ConfigDict(hide_input_in_errors=True))
_STR: Final = TypeAdapter(str, config=ConfigDict(hide_input_in_errors=True))
-_GEMINI_CLOUD_TTS_LOCATIONS: Final = MappingProxyType(
+_CLOUD_TTS_LOCATIONS: Final = MappingProxyType(
{
**dict.fromkeys(
- ("us", "us-central1", "us-east1", "us-east4", "us-east5", "us-south1", "us-west1", "us-west4"), "us"
+ (
+ "us",
+ "us-central1",
+ "us-east1",
+ "us-east4",
+ "us-east5",
+ "us-south1",
+ "us-west1",
+ "us-west2",
+ "us-west3",
+ "us-west4",
+ "us-west5",
+ ),
+ "us",
),
**dict.fromkeys(
- ("eu", "europe-central2", "europe-north1", "europe-southwest1", "europe-west1", "europe-west4"), "eu"
+ (
+ "eu",
+ "europe-central2",
+ "europe-north1",
+ "europe-north2",
+ "europe-southwest1",
+ "europe-west1",
+ "europe-west3",
+ "europe-west4",
+ "europe-west8",
+ "europe-west9",
+ "europe-west10",
+ "europe-west12",
+ ),
+ "eu",
),
"northamerica-northeast1": "northamerica-northeast1",
}
)
+_CLOUD_TTS_MULTI_REGIONS: Final = frozenset(("us", "eu"))
_DEFAULT_GEMINI_TTS_VOICE: Final = "Kore"
@@ -563,7 +591,11 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
# Resolve Vertex AI credentials using VertexBase helpers
vertex_credentials: Final = self.safe_get_vertex_ai_credentials(litellm_params_dict)
vertex_project: Final = self.safe_get_vertex_ai_project(litellm_params_dict)
- vertex_location: Final = self.safe_get_vertex_ai_location(litellm_params_dict)
+ vertex_location: Final = (
+ self.safe_get_vertex_ai_location(litellm_params_dict)
+ if self._is_gemini_tts_model(model)
+ else self.explicit_vertex_ai_location(litellm_params_dict)
+ )
mapped_voice, mapped_params = (
self.map_openai_params(
@@ -732,7 +764,11 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
if api_base:
return api_base
- return self.TTS_API_URL
+ location: Final = _JSON_OBJECT.validate_python(litellm_params).get("vertex_location")
+ cloud_tts_location: Final = _CLOUD_TTS_LOCATIONS.get(location) if isinstance(location, str) else None
+ if cloud_tts_location not in _CLOUD_TTS_MULTI_REGIONS:
+ return self.TTS_API_URL
+ return f"https://{cloud_tts_location}-texttospeech.googleapis.com/v1/text:synthesize"
def _get_gemini_tts_url(self, model: str, api_base: str | None, litellm_params: Mapping[str, object]) -> str:
if api_base:
@@ -748,7 +784,7 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
vertex_location: Final = location if isinstance(location, str) else self.get_default_vertex_location()
if vertex_location == "global":
return self.TTS_API_URL
- cloud_tts_location: Final = _GEMINI_CLOUD_TTS_LOCATIONS.get(vertex_location)
+ cloud_tts_location: Final = _CLOUD_TTS_LOCATIONS.get(vertex_location)
if cloud_tts_location is None:
raise UnsupportedParamsError(
message=(
diff --git a/litellm/main.py b/litellm/main.py
index 6b9964a450f..bdb35823757 100644
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -4393,6 +4393,14 @@ def _converse_native_output_fallback(
return None if AmazonConverseConfig.sends_native_text_format(fallback.optional_params) else fallback
+def _buffered_stream_fallback(
+ first_attempt_params: Mapping[str, object], first_attempt_litellm_params: Mapping[str, object]
+) -> NativeOutputFallback:
+ return NativeOutputFallback(
+ optional_params={**first_attempt_params, "fake_stream": True}, litellm_params=first_attempt_litellm_params
+ )
+
+
def _native_output_attempt_timeout(
ctx: CompletionDispatchContext, fallback: NativeOutputFallback | None
) -> float | str | httpx.Timeout | None:
@@ -4496,11 +4504,14 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
)
converse_params: Final = _OPTIONAL_PARAMS_ADAPTER.validate_python(optional_params)
+ converse_uses_native_output: Final = AmazonConverseConfig().uses_native_output(
+ converse_model, converse_params, litellm_params
+ )
response = call_with_native_output_fallback(
converse_attempt,
(
partial(_converse_native_output_fallback, ctx, converse_params, dict(litellm_params))
- if AmazonConverseConfig().uses_native_output(converse_model, converse_params, litellm_params)
+ if converse_uses_native_output
else None
),
converse_model,
@@ -4509,6 +4520,11 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
if drops_unsupported_params(litellm_params.get("drop_params"))
else None
),
+ buffered_stream_fallback=(
+ partial(_buffered_stream_fallback, converse_params, dict(litellm_params))
+ if converse_uses_native_output
+ else None
+ ),
)
elif bedrock_route == "converse_like":
model = model.replace("converse_like/", "")
@@ -4551,15 +4567,20 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
)
invoke_params: Final = _OPTIONAL_PARAMS_ADAPTER.validate_python(optional_params)
+ invoke_uses_native_output: Final = isinstance(
+ provider_config, AmazonAnthropicClaudeConfig
+ ) and provider_config.uses_native_output(model, invoke_params, litellm_params)
response = call_with_native_output_fallback(
invoke_attempt,
- (
- partial(_native_output_fallback, ctx, invoke_params, dict(litellm_params))
- if isinstance(provider_config, AmazonAnthropicClaudeConfig)
- and provider_config.uses_native_output(model, invoke_params, litellm_params)
+ partial(_native_output_fallback, ctx, invoke_params, dict(litellm_params))
+ if invoke_uses_native_output
+ else None,
+ model,
+ buffered_stream_fallback=(
+ partial(_buffered_stream_fallback, invoke_params, dict(litellm_params))
+ if invoke_uses_native_output
else None
),
- model,
)
return response
diff --git a/litellm/model_latest_aliases.json b/litellm/model_latest_aliases.json
new file mode 100644
index 00000000000..7b0fa5d3d39
--- /dev/null
+++ b/litellm/model_latest_aliases.json
@@ -0,0 +1,21 @@
+{
+ "openai": {
+ "gpt-luna-latest": "gpt-6-luna",
+ "gpt-sol-latest": "gpt-6.1-sol",
+ "gpt-terra-latest": "gpt-6.1-terra",
+ "gpt-astra-latest": "gpt-6-astra",
+ "gpt-mini-latest": "gpt-5.4-mini"
+ },
+ "anthropic": {
+ "claude-opus-latest": "claude-opus-5-5",
+ "claude-sonnet-latest": "claude-sonnet-5-5",
+ "claude-haiku-latest": "claude-haiku-5-5",
+ "claude-fable-latest": "claude-fable-5-1"
+ },
+ "xai": {"grok-latest": "grok-4.7"},
+ "moonshot": {"kimi-latest": "kimi-k3"},
+ "zai": {"glm-latest": "glm-5.3", "glm-flash-latest": "glm-5.3-flash"},
+ "deepseek": {"deepseek-pro-latest": "deepseek-v4-pro", "deepseek-flash-latest": "deepseek-v4-flash"},
+ "dashscope": {"qwen-max-latest": "qwen3.8-max", "qwen-flash-latest": "qwen3.8-flash"},
+ "minimax": {"minimax-latest": "MiniMax-M3"}
+}
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index 95adfda8c49..4796ed3ac9c 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -1961,6 +1961,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2041,6 +2042,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2116,6 +2118,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2242,7 +2245,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2287,6 +2291,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2331,7 +2336,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2376,6 +2382,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2420,7 +2427,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2465,6 +2473,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2504,7 +2513,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2549,6 +2559,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2593,7 +2604,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2633,6 +2645,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2715,7 +2728,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2759,7 +2773,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2803,7 +2818,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2847,7 +2863,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2891,7 +2908,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3023,6 +3041,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3068,6 +3087,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3113,6 +3133,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3158,6 +3179,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3203,6 +3225,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -12097,16 +12120,19 @@
},
"azure/tts-1": {
"deprecation_date": "2026-12-15",
+ "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
"input_cost_per_character": 1.5e-05,
"litellm_provider": "azure",
- "mode": "audio_speech"
+ "mode": "audio_speech",
+ "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
},
"azure/tts-1-hd": {
"deprecation_date": "2026-12-15",
+ "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
"input_cost_per_character": 3.3e-05,
"litellm_provider": "azure",
"mode": "audio_speech",
- "source": "https://azure.microsoft.com/en-us/pricing/details/azure-openai/"
+ "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
},
"azure/us/gpt-4.1-2025-04-14": {
"deprecation_date": "2027-04-14",
@@ -12859,11 +12885,13 @@
"supports_vision": true
},
"azure/whisper-1": {
- "deprecation_date": "2026-12-15",
+ "deprecation_date": "2027-02-26",
+ "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
"input_cost_per_second": 0.0001,
"litellm_provider": "azure",
"mode": "audio_transcription",
- "output_cost_per_second": 0.0001
+ "output_cost_per_second": 0.0001,
+ "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
},
"azure_ai/Cohere-embed-v3-english": {
"deprecation_date": "2027-02-15",
@@ -31630,7 +31658,7 @@
"input_cost_per_audio_token_batches": 2.5e-07
},
"gemini/gemini-embedding-2-preview": {
- "deprecation_date": "2026-08-10",
+ "deprecation_date": "2026-11-06",
"deprecation_source": "https://ai.google.dev/gemini-api/docs/deprecations",
"input_cost_per_audio_token": 6.5e-06,
"input_cost_per_audio_token_batches": 3.25e-06,
@@ -44089,6 +44117,7 @@
"supports_vision": true
},
"mistral/mistral-large-2512": {
+ "deprecation_date": "2027-04-30",
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 5e-07,
"litellm_provider": "mistral",
@@ -44164,6 +44193,7 @@
"supports_vision": true
},
"mistral/mistral-medium-3-5": {
+ "deprecation_date": "2027-04-30",
"cache_read_input_token_cost": 1.5e-07,
"input_cost_per_token": 1.5e-06,
"litellm_provider": "mistral",
@@ -45679,6 +45709,7 @@
"o1": {
"cache_read_input_token_cost": 7.5e-06,
"deprecation_date": "2026-10-23",
+ "deprecation_source": "https://developers.openai.com/api/docs/deprecations",
"successor_model": "gpt-5.6-sol",
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
@@ -64725,6 +64756,7 @@
"source": "https://docs.mistral.ai/models/model-cards/mistral-moderation-26-03"
},
"mistral/voxtral-mini-2602": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_second": 5e-05,
"litellm_provider": "mistral",
"mode": "audio_transcription",
@@ -64741,6 +64773,7 @@
"supports_audio_input": true
},
"mistral/voxtral-mini-transcribe-realtime-2602": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_second": 0.0001,
"litellm_provider": "mistral",
"mode": "audio_transcription",
@@ -64757,6 +64790,7 @@
"supports_audio_input": true
},
"mistral/voxtral-mini-tts-2603": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_character": 1.6e-05,
"litellm_provider": "mistral",
"mode": "audio_speech",
@@ -75543,7 +75577,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -75583,7 +75618,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -75623,6 +75659,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -75696,6 +75733,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -76994,13 +77032,13 @@
"supports_vision": true
},
"azure/whisper": {
- "deprecation_date": "2026-12-15",
- "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
+ "deprecation_date": "2027-02-26",
+ "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
"input_cost_per_second": 0.0001,
"litellm_provider": "azure",
"mode": "audio_transcription",
"output_cost_per_second": 0.0001,
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
+ "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
},
"gemini/gemini-nano-banana-2.1": {
"input_cost_per_token": 1.5e-06,
@@ -77411,6 +77449,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77468,6 +77507,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77645,6 +77685,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77697,6 +77738,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77744,6 +77786,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77857,6 +77900,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
diff --git a/litellm/proxy/_lazy_openapi_snapshot.json b/litellm/proxy/_lazy_openapi_snapshot.json
index 5cb63c98d56..137d6016402 100644
--- a/litellm/proxy/_lazy_openapi_snapshot.json
+++ b/litellm/proxy/_lazy_openapi_snapshot.json
@@ -9633,7 +9633,7 @@
"paths": {
"/backend-api/codex/{endpoint}": {
"get": {
- "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
+ "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__get",
"parameters": [
{
@@ -9681,7 +9681,7 @@
]
},
"post": {
- "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
+ "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__post",
"parameters": [
{
@@ -9724,7 +9724,7 @@
}
}
},
- "description": "Unmodified Codex response or server-sent events"
+ "description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events"
},
"422": {
"content": {
diff --git a/litellm/proxy/auth/auth_checks.py b/litellm/proxy/auth/auth_checks.py
index 2df31d7f4a4..9746957760e 100644
--- a/litellm/proxy/auth/auth_checks.py
+++ b/litellm/proxy/auth/auth_checks.py
@@ -180,6 +180,7 @@ from litellm.repositories.table_repositories import (
)
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
+from litellm.responses.additional_tools import additional_tools_in_input
from litellm.router import Router
from litellm.router_api import RegistryReader
from litellm.types.proxy.auth.auth_checks import UserNotFoundError
@@ -1037,9 +1038,6 @@ async def check_tools_allowlist(
call_types: Final = get_call_types_for_route(route)
if not call_types or not any(ct.value in TOOL_CAPABLE_CALL_TYPES for ct in call_types):
return
- tool_names: Final = extract_request_tool_names(route, request_body)
- if not tool_names:
- return
key_meta: Final = (valid_token.metadata or {}) if isinstance(valid_token.metadata, dict) else {}
team_meta: Final = (valid_token.team_metadata or {}) if isinstance(valid_token.team_metadata, dict) else {}
key_allowed: Final = key_meta.get("allowed_tools")
@@ -1047,6 +1045,9 @@ async def check_tools_allowlist(
effective: Final = key_allowed if (isinstance(key_allowed, list) and len(key_allowed) > 0) else team_allowed
if not isinstance(effective, list) or len(effective) == 0:
return
+ tool_names: Final = extract_request_tool_names(route, request_body)
+ if not tool_names:
+ return
allowed_set: Final = {str(t) for t in effective}
disallowed: Final = [n for n in tool_names if n not in allowed_set]
if disallowed:
@@ -7988,6 +7989,10 @@ def _tools(tools: object) -> tuple[object, ...]:
return ()
+def _request_tools(request_body: Mapping[str, object]) -> tuple[object, ...]:
+ return (*_tools(request_body.get("tools")), *additional_tools_in_input(request_body.get("input")))
+
+
def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
try:
tool_fields: Final = _TOOL_ADAPTER.validate_python(tool, strict=True)
@@ -7998,14 +8003,14 @@ def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
def _strict_requested_vector_store_ids(request_body: Mapping[str, object]) -> tuple[str, ...]:
"""
- Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, but a vector_store_ids that is
- not a list of strings is a 400 instead of being skipped or iterated, and tools that are not
- objects name no store.
+ Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, plus the tools of `additional_tools`
+ input items, but a vector_store_ids that is not a list of strings is a 400 instead of being skipped or
+ iterated, and tools that are not objects name no store.
"""
return tuple(
chain(
_validated_vector_store_ids(request_body.get("vector_store_ids")),
- chain.from_iterable(_tool_vector_store_ids(tool) for tool in _tools(request_body.get("tools"))),
+ chain.from_iterable(_tool_vector_store_ids(tool) for tool in _request_tools(request_body)),
)
)
@@ -8044,7 +8049,7 @@ def _legacy_requested_vector_store_ids(
return tuple(
chain(
top_level,
- chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _tools(request_body.get("tools"))),
+ chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _request_tools(request_body)),
)
)
@@ -8113,8 +8118,8 @@ async def vector_store_access_check(
"""
Checks whether the caller may use every vector store the request names.
- Requested stores come from `vector_store_ids`, `tools[].vector_store_ids` and the RAG
- `retrieval_config.vector_store_id`. Grants come from each identity's
+ Requested stores come from `vector_store_ids`, `tools[].vector_store_ids`, the tools of
+ `additional_tools` input items and the RAG `retrieval_config.vector_store_id`. Grants come from each identity's
`object_permission.vector_stores`.
Both modes read the same request fields, with or without a vector store registry, and answer 400
diff --git a/litellm/proxy/auth/auth_utils.py b/litellm/proxy/auth/auth_utils.py
index 2b5fc6d877e..24602dec677 100644
--- a/litellm/proxy/auth/auth_utils.py
+++ b/litellm/proxy/auth/auth_utils.py
@@ -863,7 +863,7 @@ async def pre_db_read_auth_checks(
Raises:
- HTTPException if request fails initial auth checks
"""
- from litellm.proxy.proxy_server import general_settings, llm_router, premium_user
+ from litellm.proxy.proxy_server import general_settings, llm_router
# Check 1. request size
await check_if_request_size_is_safe(request=request)
@@ -891,19 +891,12 @@ async def pre_db_read_auth_checks(
)
# Check 4. Check if request route is an allowed route on the proxy
- if "allowed_routes" in general_settings:
- _allowed_routes: Final = general_settings["allowed_routes"]
- if premium_user is not True:
- verbose_proxy_logger.error(
- "Trying to set allowed_routes. This is an Enterprise feature. %s",
- CommonProxyErrors.not_premium_user.value,
- )
- if route not in _allowed_routes:
- verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, _allowed_routes)
- raise HTTPException(
- status_code=status.HTTP_403_FORBIDDEN,
- detail=f"Access forbidden: Route {route} not allowed",
- )
+ if not RouteChecks.is_allowed_on_proxy(route):
+ verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, general_settings["allowed_routes"])
+ raise HTTPException(
+ status_code=status.HTTP_403_FORBIDDEN,
+ detail=f"Access forbidden: Route {route} not allowed",
+ )
def route_in_additonal_public_routes(current_route: str):
diff --git a/litellm/proxy/auth/login_throttle.py b/litellm/proxy/auth/login_throttle.py
index 0536e293f13..27f049a950e 100644
--- a/litellm/proxy/auth/login_throttle.py
+++ b/litellm/proxy/auth/login_throttle.py
@@ -431,6 +431,10 @@ class LoginAttempt:
throttle: LoginThrottle
username: str
+ async def refuse_if_blocked_while_waiting(self) -> None:
+ """Refuses a login that waited in line while failures ahead of it blocked this key."""
+ await self.throttle.attempt(self.username)
+
async def succeeded(self) -> None:
if not self.throttle.enabled:
return
diff --git a/litellm/proxy/auth/login_utils.py b/litellm/proxy/auth/login_utils.py
index 9783f44b41f..7d413121b56 100644
--- a/litellm/proxy/auth/login_utils.py
+++ b/litellm/proxy/auth/login_utils.py
@@ -5,6 +5,7 @@ This module contains the core login logic that can be reused across different
login endpoints (e.g., /login and /v2/login).
"""
+import asyncio
import os
import secrets
from collections.abc import Mapping
@@ -31,6 +32,7 @@ from litellm.proxy._types import (
from litellm.proxy.auth.auth_utils import is_sso_provider_fully_configured
from litellm.proxy.auth.login_throttle import LoginAttempt, LoginThrottle
from litellm.proxy.auth.password_policy import is_breach_check_enabled, is_password_breached
+from litellm.proxy.common_utils.fips import fips_mode_active
from litellm.proxy.management_endpoints.internal_user_endpoints import user_update
from litellm.proxy.management_endpoints.key_management_endpoints import (
generate_key_helper_fn,
@@ -42,6 +44,7 @@ from litellm.proxy.utils import (
PrismaClient,
get_server_root_path,
hash_password,
+ run_off_loop_if_fips,
verify_password,
)
from litellm.repositories.user_repository import UserRepository
@@ -105,9 +108,21 @@ async def screen_login_password_for_breach(
return breached
+_ONE_FIPS_LOGIN_AT_A_TIME: Final = asyncio.Lock()
+
+
+async def _password_matches(password: str, stored: str, attempt: LoginAttempt) -> bool:
+ """Checks inline, or in FIPS mode one login at a time, refusing a login whose user got blocked while it waited."""
+ if not fips_mode_active():
+ return verify_password(password, stored)
+ async with _ONE_FIPS_LOGIN_AT_A_TIME:
+ await attempt.refuse_if_blocked_while_waiting()
+ return await run_off_loop_if_fips(verify_password, password, stored)
+
+
async def _rehash_password_if_needed(user_id: str, password: str, stored: str) -> None:
"""Rehash legacy password (SHA256) to scrypt on successful login."""
- if stored.startswith("scrypt:"):
+ if stored.startswith(("scrypt:", "pbkdf2:")):
return
from litellm.proxy.proxy_server import prisma_client
@@ -411,7 +426,7 @@ async def _sign_in(
code=401,
)
- if verify_password(password, _password):
+ if await _password_matches(password, _password, attempt):
await _rehash_password_if_needed(_user_row.user_id, password, _password)
breached_now: Final = prisma_client is not None and await screen_login_password_for_breach(
user_id=_user_row.user_id,
diff --git a/litellm/proxy/auth/model_checks.py b/litellm/proxy/auth/model_checks.py
index 73edcd66bd7..c1db8b7b4c7 100644
--- a/litellm/proxy/auth/model_checks.py
+++ b/litellm/proxy/auth/model_checks.py
@@ -41,7 +41,7 @@ _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields)
_MODEL_LISTING_POOL: Final = ThreadPoolExecutor(thread_name_prefix="litellm-model-listing")
_PROVIDERS_LISTED_ONLY_THROUGH_A_DEPLOYMENT: Final = frozenset(
- {LlmProviders.OPENROUTER.value, LlmProviders.HOSTED_VLLM.value}
+ {LlmProviders.OPENROUTER.value, LlmProviders.HOSTED_VLLM.value, LlmProviders.LITELLM_PROXY.value}
)
_Listed: Final = TypeVar("_Listed")
@@ -83,7 +83,11 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N
return None
if ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is None:
return None
- return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
+ return get_valid_models(
+ check_provider_endpoint=llm_provider == LlmProviders.LITELLM_PROXY,
+ custom_llm_provider=provider,
+ litellm_params=litellm_params,
+ )
def _get_models_from_access_groups(
@@ -188,34 +192,25 @@ def get_team_models(
- Empty list if no models set
- If model_access_groups is provided, only return models that are in the access groups
"""
- all_models_set: Final[set[str]] = set()
- if len(team_models) > 0:
- all_models_set.update(team_models)
- if SpecialModelNames.all_team_models.value in all_models_set:
- all_models_set.update(team_models)
- # GH#30619: expand all-team-models sentinel
- # to the actual proxy model list
- all_models_set.discard(SpecialModelNames.all_team_models.value)
- all_models_set.update(proxy_model_list)
- if include_model_access_groups:
- all_models_set.update(model_access_groups.keys())
- if SpecialModelNames.all_proxy_models.value in all_models_set:
- all_models_set.update(proxy_model_list)
- if include_model_access_groups:
- all_models_set.update(model_access_groups.keys())
-
- all_models = _get_models_from_access_groups(
+ expands_to_proxy_models: Final = (
+ SpecialModelNames.all_team_models.value in team_models
+ or SpecialModelNames.all_proxy_models.value in team_models
+ )
+ expansion: Final = (
+ (*proxy_model_list, *(model_access_groups if include_model_access_groups else ()))
+ if expands_to_proxy_models
+ else ()
+ )
+ named_models: Final = (model for model in team_models if model != SpecialModelNames.all_team_models.value)
+ all_models: Final = _get_models_from_access_groups(
model_access_groups=model_access_groups,
- all_models=list(all_models_set),
+ all_models=list(dict.fromkeys((*named_models, *expansion))),
include_model_access_groups=include_model_access_groups,
proxy_model_list=proxy_model_list,
)
- # deduplicate while preserving order
- all_models = list(dict.fromkeys(all_models))
-
verbose_proxy_logger.debug("ALL TEAM MODELS - %s", len(all_models))
- return all_models
+ return list(dict.fromkeys(all_models))
def get_complete_model_list(
diff --git a/litellm/proxy/auth/route_checks.py b/litellm/proxy/auth/route_checks.py
index 125bfab4edf..0aa2528d46b 100644
--- a/litellm/proxy/auth/route_checks.py
+++ b/litellm/proxy/auth/route_checks.py
@@ -439,6 +439,19 @@ class RouteChecks:
detail=f"user not allowed to access this route. Route={route} is an admin only route",
)
+ @staticmethod
+ def is_allowed_on_proxy(route: str) -> bool:
+ from litellm.proxy.proxy_server import general_settings, premium_user
+
+ if "allowed_routes" not in general_settings:
+ return True
+ if premium_user is not True:
+ verbose_proxy_logger.error(
+ "Trying to set allowed_routes. This is an Enterprise feature. %s",
+ CommonProxyErrors.not_premium_user.value,
+ )
+ return route in general_settings["allowed_routes"]
+
@staticmethod
def is_unbilled_route(route: str) -> bool:
resource, _, action = route.rsplit("/", 1)[-1].partition(":")
diff --git a/litellm/proxy/auth/user_api_key_auth.py b/litellm/proxy/auth/user_api_key_auth.py
index ba253b44ea3..a52ad14e293 100644
--- a/litellm/proxy/auth/user_api_key_auth.py
+++ b/litellm/proxy/auth/user_api_key_auth.py
@@ -309,6 +309,17 @@ async def _normalize_claude_model(
rewrite_request_model(request_data, request, source)
+_CODEX_NATIVE_RESPONSES_ROUTE: Final = "/backend-api/codex/responses"
+
+
+def _is_codex_native_litellm_turn(route: str, model: object, llm_router: litellm.Router, team_id: str | None) -> bool:
+ if route != _CODEX_NATIVE_RESPONSES_ROUTE:
+ return False
+ from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import routes_to_litellm
+
+ return routes_to_litellm(model, llm_router, team_id)
+
+
async def _resolve_router_settings_model_group_alias(
request_data: dict[str, object], # mutable-ok: the request body is rewritten in place for every downstream reader
valid_token: UserAPIKeyAuth,
@@ -325,9 +336,12 @@ async def _resolve_router_settings_model_group_alias(
if request.scope.get(MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY) is True:
return
request.scope[MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY] = True
- if request_dispatched_to_pass_through_endpoint(request) or request_dispatched_to_provider_pass_through(request):
- return
requested: Final = request_data.get("model")
+ if request_dispatched_to_pass_through_endpoint(request) or (
+ request_dispatched_to_provider_pass_through(request)
+ and not _is_codex_native_litellm_turn(route, requested, llm_router, valid_token.team_id)
+ ):
+ return
if not isinstance(requested, str) or await read_raw_json_body(request=request) is None:
return
settings: Final = await proxy_config.get_hierarchical_router_settings(
@@ -338,6 +352,11 @@ async def _resolve_router_settings_model_group_alias(
target: Final = resolve_model_group_alias(settings.get("model_group_alias"), requested)
if target is None or target == requested:
return
+ if request_dispatched_to_provider_pass_through(request):
+ from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_turn_serves_through_litellm
+
+ if not await native_turn_serves_through_litellm(requested, llm_router, request, valid_token):
+ return
verbose_proxy_logger.debug(
"router_settings.model_group_alias resolved %s -> %s before auth",
requested.replace("\r", "").replace("\n", ""),
@@ -347,6 +366,27 @@ async def _resolve_router_settings_model_group_alias(
rewrite_request_model(request_data, request, target)
+def _resolve_codex_native_chatgpt_model(
+ request_data: dict[str, object], # mutable-ok: the request body is rewritten in place for every downstream reader
+ valid_token: UserAPIKeyAuth,
+ request: Request | None,
+ route: str,
+) -> None:
+ requested: Final = request_data.get("model")
+ if route != _CODEX_NATIVE_RESPONSES_ROUTE or request is None or not isinstance(requested, str):
+ return
+ from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_chatgpt_model
+ from litellm.proxy.proxy_server import llm_router
+
+ if llm_router is None:
+ return
+ target: Final = native_chatgpt_model(requested, llm_router, valid_token)
+ if target is None:
+ return
+ request.scope.setdefault(CLIENT_REQUESTED_MODEL_SCOPE_KEY, requested)
+ rewrite_request_model(request_data, request, target)
+
+
def _get_model_names_for_budget_checks(
model: str | list[str] | None,
) -> list[str]:
@@ -4172,6 +4212,7 @@ async def _authorize_authenticated_request(
RouteChecks.should_call_route(route=route, valid_token=user_api_key_auth_obj, request=request)
await _normalize_claude_model(request_data, user_api_key_auth_obj, request, route)
await _resolve_router_settings_model_group_alias(request_data, user_api_key_auth_obj, request, route)
+ _resolve_codex_native_chatgpt_model(request_data, user_api_key_auth_obj, request, route)
# Single authorization point. Builder paths MUST NOT call common_checks.
# Route through the same exception handler the builder uses so
@@ -4650,6 +4691,7 @@ async def enforce_key_and_fallback_model_access(
"""
await _normalize_claude_model(request_data, valid_token, request, route)
await _resolve_router_settings_model_group_alias(request_data, valid_token, request, route)
+ _resolve_codex_native_chatgpt_model(request_data, valid_token, request, route)
config: Final = valid_token.config
if config != {}:
diff --git a/litellm/proxy/common_utils/codex_model_catalog.py b/litellm/proxy/common_utils/codex_model_catalog.py
index 462e372f3d8..01cce9a6ce6 100644
--- a/litellm/proxy/common_utils/codex_model_catalog.py
+++ b/litellm/proxy/common_utils/codex_model_catalog.py
@@ -24,8 +24,10 @@ a newer tag.
from __future__ import annotations
+import hashlib
+import json
from collections.abc import Callable, Mapping, Sequence
-from dataclasses import dataclass
+from dataclasses import asdict, dataclass
from functools import cache
from itertools import accumulate, pairwise
from pathlib import Path
@@ -33,7 +35,7 @@ from types import MappingProxyType
from typing import TYPE_CHECKING, Final, Literal, TypeAlias
from packaging.version import Version
-from pydantic import ConfigDict, TypeAdapter, ValidationError
+from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
from litellm._logging import verbose_proxy_logger
from litellm.proxy.common_utils.codex_model_translation import (
@@ -61,6 +63,8 @@ CODEX_CHAT_MODES: Final = frozenset({"chat", "responses"})
_BODY_PREFIX: Final = '{"models":['
_BODY_SUFFIX: Final = "]}"
+_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
+_JSON_OBJECTS: Final = TypeAdapter(tuple[dict[str, object], ...])
_NO_UPDATE: Final[Mapping[str, object]] = MappingProxyType({})
@@ -119,6 +123,7 @@ class CodexStockCatalog(LiteLLMBaseModel):
CodexCatalogEntry = CodexFallbackModel | CodexStockModel
CodexAliasDefaults: TypeAlias = Literal["codex", "deployment"]
+_NATIVE_ALIAS_DEFAULTS: Final[CodexAliasDefaults] = "deployment"
@dataclass(frozen=True, slots=True)
@@ -408,6 +413,7 @@ def _entry(
stock_model.upgrade if stock_model.upgrade is not None and stock_model.upgrade.model in served else None
)
is_alias: Final = row.id != stock_model.slug
+ deployment_decides: Final = is_alias and alias_defaults == "deployment"
stock_name: Final = row.id if is_alias else stock_model.display_name
listing: Final = {
"slug": row.id,
@@ -417,12 +423,18 @@ def _entry(
"supported_in_api": True,
"upgrade": upgrade,
**({} if search_tool is None else {"supports_search_tool": search_tool}),
- **(_deployment_defaults(row, stock_model) if is_alias and alias_defaults == "deployment" else {}),
+ **(_deployment_defaults(row, stock_model) if deployment_decides else {}),
**_configured_window(row, stock_model),
}
if configured is None:
return stock_model.model_copy(update={**_priced_tiers(row, stock_model), **listing})
- return stock_model.model_copy(update={**listing, **_tier_fields(stock_model, configured)})
+ return stock_model.model_copy(
+ update={
+ **listing,
+ **_tier_fields(stock_model, configured),
+ **({"default_service_tier": None} if deployment_decides else {}),
+ }
+ )
def _serialized(entry: CodexCatalogEntry) -> str:
@@ -435,13 +447,12 @@ def _bytes_left_after(left: int, size: int) -> int:
return left - size if size <= left else left
-def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]:
- """Whether each entry, taken in order, fits in what is left of a body of at most `byte_limit`
- bytes: an entry larger than the bytes left is passed over and a smaller one after it is still
- taken, each entry counted with its separating comma and the last one's comma given back."""
- if byte_limit is None:
+def _fits(serialized: Sequence[str], budget: int | None) -> tuple[bool, ...]:
+ """Whether each entry, taken in order, fits in what is left of `budget` bytes: an entry larger
+ than the bytes left is passed over and a smaller one after it is still taken, each entry
+ counted with its separating comma."""
+ if budget is None:
return (True,) * len(serialized)
- budget: Final = byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
left: Final = tuple(
accumulate((len(entry.encode()) + 1 for entry in serialized), _bytes_left_after, initial=budget)
)
@@ -449,17 +460,34 @@ def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]
def _kept_positions(
- entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], byte_limit: int | None
+ entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], budget: int | None
) -> frozenset[int]:
"""The listing positions that survive the byte limit: an entry offering a service tier is taken
ahead of one offering none, each group in listing order, since a model without a tier is driven
by name just as well off Codex's bundled catalog; an entry too large for the bytes left never
costs the smaller ones after it their place."""
survival: Final = sorted(range(len(entries)), key=lambda position: (not entries[position].service_tiers, position))
- fits: Final = _fits(tuple(serialized[position] for position in survival), byte_limit)
+ fits: Final = _fits(tuple(serialized[position] for position in survival), budget)
return frozenset(position for position, fit in zip(survival, fits, strict=True) if fit)
+def _entries(
+ drivable: Sequence[CodexCatalogRow],
+ stock: Mapping[str, CodexStockModel] | None,
+ instructions: str | None,
+ first_priority: int,
+ alias_defaults: CodexAliasDefaults,
+ client_version: str | None,
+) -> tuple[CodexCatalogEntry, ...]:
+ known: Final = bundled_codex_models() if stock is None else stock
+ prompt: Final = codex_base_instructions() if instructions is None else instructions
+ served: Final = frozenset(row.id for row in drivable)
+ return tuple(
+ _entry(first_priority + index, row, known, served, prompt, alias_defaults, client_version)
+ for index, row in enumerate(drivable)
+ )
+
+
@dataclass(frozen=True, slots=True)
class CodexCatalogBody:
"""A `ModelsResponse` body, the ids it lists, and the chat-capable ids the byte limit left out."""
@@ -495,15 +523,12 @@ def codex_models_response_json(
requesting Codex's version, which decides the reasoning levels a fallback entry may offer; None counts
as a version too old for custom levels.
"""
- known: Final = bundled_codex_models() if stock is None else stock
- prompt: Final = codex_base_instructions() if instructions is None else instructions
drivable: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode))
- served: Final = frozenset(row.id for row in drivable)
- entries: Final = tuple(
- _entry(index, row, known, served, prompt, alias_defaults, client_version) for index, row in enumerate(drivable)
- )
+ entries: Final = _entries(drivable, stock, instructions, 0, alias_defaults, client_version)
serialized: Final = tuple(_serialized(entry) for entry in entries)
- kept: Final = _kept_positions(entries, serialized, byte_limit)
+ kept: Final = _kept_positions(
+ entries, serialized, None if byte_limit is None else byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
+ )
return CodexCatalogBody(
json=f"{_BODY_PREFIX}{','.join(serialized[position] for position in sorted(kept))}{_BODY_SUFFIX}",
listed=tuple(row.id for position, row in enumerate(drivable) if position in kept),
@@ -511,6 +536,74 @@ def codex_models_response_json(
)
+def _priority(model: Mapping[str, object]) -> int:
+ priority: Final = model.get("priority")
+ return priority if isinstance(priority, int) else -1
+
+
+def _compact_json(value: object) -> str:
+ return json.dumps(value, separators=(",", ":"))
+
+
+def _entry_json(entry: CodexCatalogEntry) -> Mapping[str, object]:
+ return _JSON_OBJECT.validate_python(entry.model_dump(mode="json", exclude_unset=isinstance(entry, CodexStockModel)))
+
+
+def merged_codex_catalog(
+ upstream_body: bytes,
+ rows: Sequence[CodexCatalogRow],
+ *,
+ stock: Mapping[str, CodexStockModel] | None = None,
+ instructions: str | None = None,
+ byte_limit: int = CODEX_CATALOG_BYTE_LIMIT,
+ client_version: str | None = None,
+) -> str | None:
+ try:
+ top: Final = _JSON_OBJECT.validate_json(upstream_body)
+ upstream: Final = _JSON_OBJECTS.validate_python(top.get("models"))
+ except ValidationError:
+ return None
+ slugs: Final = frozenset(model.get("slug") for model in upstream)
+ added: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode) and row.id not in slugs)
+ first_priority: Final = max((_priority(model) for model in upstream), default=-1) + 1
+ entries: Final = _entries(added, stock, instructions, first_priority, _NATIVE_ALIAS_DEFAULTS, client_version)
+ added_models: Final = tuple(_entry_json(entry) for entry in entries)
+ serialized: Final = tuple(_compact_json(model) for model in added_models)
+ budget: Final = byte_limit - len(_compact_json({**top, "models": upstream}))
+ kept: Final = tuple(sorted(_kept_positions(entries, serialized, budget)))
+ if len(kept) < len(entries):
+ verbose_proxy_logger.warning(
+ "Codex native model catalog left out %d of %d LiteLLM models to stay within Codex's %d byte limit",
+ len(entries) - len(kept),
+ len(entries),
+ byte_limit,
+ )
+ if not kept:
+ return None
+ return _compact_json({**top, "models": [*upstream, *(added_models[position] for position in kept)]})
+
+
+def _canonical_value(value: object) -> object:
+ return value.model_dump(mode="json") if isinstance(value, BaseModel) else repr(value)
+
+
+def codex_catalog_etag(upstream_etag: str, rows: Sequence[CodexCatalogRow]) -> str:
+ canonical: Final = json.dumps(
+ [asdict(row) for row in rows], sort_keys=True, separators=(",", ":"), default=_canonical_value
+ )
+ digest: Final = hashlib.sha256(json.dumps([upstream_etag, canonical]).encode()).hexdigest()
+ return f'"{digest}"'
+
+
+def upstream_codex_slugs(upstream_body: bytes) -> frozenset[str] | None:
+ try:
+ upstream: Final = _JSON_OBJECTS.validate_python(_JSON_OBJECT.validate_json(upstream_body).get("models"))
+ except ValidationError:
+ return None
+ slugs: Final = (model.get("slug") for model in upstream)
+ return frozenset(slug for slug in slugs if isinstance(slug, str))
+
+
def _group_reasoning_efforts(
row: ModelInfoResponse, lookup_id: str, registry: RouterRegistry | None, team_id: str | None
) -> tuple[str, ...] | None:
@@ -520,6 +613,11 @@ def _group_reasoning_efforts(
return group.supported_reasoning_efforts if group is not None else None
+def _shared_pinned_effort(pinned: tuple[object, ...]) -> str | None:
+ first: Final = next(iter(pinned), None)
+ return first if isinstance(first, str) and all(effort == first for effort in pinned) else None
+
+
def deployment_capabilities(model_info: Mapping[str, object] | None, model_id: str) -> CodexDeploymentCapabilities:
"""One deployment's model info, merged with the cost map, as the capabilities a Codex entry describes.
Model info that does not validate declares nothing, logged, so one bad value never fails the listing."""
@@ -562,6 +660,11 @@ def _catalog_row(
registry.resolve(lookup_id, kind="upstream_provider", team_id=team_id) if registry is not None else None
),
display_name=registry.group(label_id, field="display_name") if registry is not None else None,
+ reasoning_effort=(
+ _shared_pinned_effort(registry.group(lookup_id, field="pinned_reasoning_efforts", team_id=team_id))
+ if registry is not None
+ else None
+ ),
service_tiers=registry.group(lookup_id, field="service_tiers", team_id=team_id) if registry is not None else (),
supported_reasoning_efforts=_group_reasoning_efforts(row, lookup_id, registry, team_id),
search_tool_support=(
@@ -619,3 +722,21 @@ def codex_model_list_body(
", ".join(body.left_out),
)
return body.json
+
+
+def codex_entry_names_default_reasoning_level(
+ model_id: str, llm_router: Router, team_id: str | None, client_version: str | None
+) -> bool:
+ """Whether the native catalog entry for `model_id` names a `default_reasoning_level`. Codex's model picker
+ sends reasoning effort `none` for a model whose entry names none, while showing that choice as
+ "default"."""
+ listed: Final[ModelInfoResponse] = {"id": model_id, "object": "model", "created": 0, "owned_by": ""}
+ (entry,) = _entries(
+ (_catalog_row(listed, model_id, model_id, llm_router, team_id),),
+ None,
+ "",
+ 0,
+ _NATIVE_ALIAS_DEFAULTS,
+ client_version,
+ )
+ return _entry_json(entry).get("default_reasoning_level") is not None
diff --git a/litellm/proxy/common_utils/codex_model_translation.py b/litellm/proxy/common_utils/codex_model_translation.py
index 2c4ea8896e3..694dc422994 100644
--- a/litellm/proxy/common_utils/codex_model_translation.py
+++ b/litellm/proxy/common_utils/codex_model_translation.py
@@ -16,7 +16,7 @@ from types import NoneType
from typing import Annotated, Final, Literal, TypeAlias
from packaging.version import Version
-from pydantic import ConfigDict, Field, StringConstraints
+from pydantic import ConfigDict, Field, StringConstraints, field_serializer
from litellm.litellm_core_utils.reasoning_effort_strength import nearest_declared_reasoning_effort
from litellm.types.llms.base_model import LiteLLMBaseModel
@@ -117,6 +117,10 @@ class CodexDeploymentCapabilities(LiteLLMBaseModel):
prices_tokens: bool = False
priced_service_tier_keys: frozenset[str] = frozenset()
+ @field_serializer("priced_service_tier_keys")
+ def _sorted_priced_service_tier_keys(self, keys: frozenset[str]) -> tuple[str, ...]:
+ return tuple(sorted(keys))
+
@cache
def tier_price_keys() -> tuple[str, ...]:
diff --git a/litellm/proxy/common_utils/fips.py b/litellm/proxy/common_utils/fips.py
index 1a4d7ddeadb..6cd23e5c0ff 100644
--- a/litellm/proxy/common_utils/fips.py
+++ b/litellm/proxy/common_utils/fips.py
@@ -1,3 +1,4 @@
+import threading
from collections.abc import Mapping
from typing import Final
@@ -5,6 +6,7 @@ from litellm.secret_managers.main import str_to_bool
FIPS_MODE_SETTING: Final = "fips_mode"
_SETTING: Final = f"general_settings.{FIPS_MODE_SETTING}"
+_FIPS_MODE_ACTIVE: Final = threading.Event()
class FipsModeError(Exception):
@@ -87,3 +89,12 @@ def enforce_fips_mode(general_settings: Mapping[str, object]) -> None:
refusal: Final = fips_mode_refusal(general_settings)
if refusal is not None:
raise FipsModeError(refusal)
+ if _fips_mode_setting(general_settings):
+ _FIPS_MODE_ACTIVE.set()
+ else:
+ _FIPS_MODE_ACTIVE.clear()
+
+
+def fips_mode_active() -> bool:
+ """Whether general_settings.fips_mode was on when enforce_fips_mode last accepted it."""
+ return _FIPS_MODE_ACTIVE.is_set()
diff --git a/litellm/proxy/db/spend_log_tool_index.py b/litellm/proxy/db/spend_log_tool_index.py
index 591ad81a4e5..92181d546eb 100644
--- a/litellm/proxy/db/spend_log_tool_index.py
+++ b/litellm/proxy/db/spend_log_tool_index.py
@@ -63,11 +63,26 @@ def response_tool_call_names(completion_response: object) -> tuple[str, ...]:
)
return tuple(
stripped
- for name in chain(called_names, _auto_executed_mcp_tool_names(completion_response))
+ for name in chain(
+ _server_tool_call_names(completion_response),
+ called_names,
+ _auto_executed_mcp_tool_names(completion_response),
+ )
if isinstance(name, str) and (stripped := name.strip())
)
+def _server_tool_call_names(completion_response: object) -> tuple[str | None, ...]:
+ from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
+ from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+ server_tool_calls: Final = chain.from_iterable(
+ anthropic_server_tool_call_params(message_field(choice, "message"))
+ for choice in parts_of(message_field(completion_response, "choices"))
+ )
+ return tuple(server_tool_call["function"].get("name") for server_tool_call in server_tool_calls)
+
+
class _ExecutedTool(BaseModel):
model_config = ConfigDict(frozen=True)
diff --git a/litellm/proxy/guardrails/_content_utils.py b/litellm/proxy/guardrails/_content_utils.py
index 51934c99db6..1a1670715dc 100644
--- a/litellm/proxy/guardrails/_content_utils.py
+++ b/litellm/proxy/guardrails/_content_utils.py
@@ -18,6 +18,11 @@ from typing import Final
from pydantic import JsonValue, TypeAdapter
from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ changed_server_tool_call_arguments,
+ write_request_hosted_web_call_arguments,
+)
from litellm.responses.litellm_completion_transformation.compaction_items import (
compaction_summary,
input_items,
@@ -407,18 +412,27 @@ def walk_user_text(
return new_parts
return content
- messages: Final = data.get("messages")
- for message in _object_list(messages) or ():
+ def _rewrite_call_arguments(calls: object) -> None:
+ for call in _object_list(calls) or ():
+ function = call.get("function") if isinstance(call, dict) else None
+ if isinstance(function, dict) and "arguments" in function:
+ function["arguments"] = _rewrite_arguments(function["arguments"])
+
+ messages: Final = _object_list(data.get("messages")) or ()
+ server_tool_calls_before: Final = tuple(anthropic_server_tool_call_params(message) for message in messages)
+ for message in messages:
if (skip_tool_message and _is_tool_message(message)) or not isinstance(message, dict):
continue
if "content" in message:
message["content"] = _rewrite_content(message["content"])
- calls = message.get("tool_calls")
- if isinstance(calls, list):
- for call in calls:
- function = call.get("function") if isinstance(call, dict) else None
- if isinstance(function, dict) and "arguments" in function:
- function["arguments"] = _rewrite_arguments(function["arguments"])
+ _rewrite_call_arguments(message.get("tool_calls"))
+ provider_fields = message.get("provider_specific_fields")
+ if isinstance(provider_fields, dict):
+ _rewrite_call_arguments(provider_fields.get("server_tool_calls"))
+ for message, original in zip(messages, server_tool_calls_before, strict=True):
+ write_request_hosted_web_call_arguments(
+ message, changed_server_tool_call_arguments(original, anthropic_server_tool_call_params(message))
+ )
if "system" in data:
data["system"] = _rewrite_content(data["system"]) # rebind-ok: preserve the native hook's in-place contract
diff --git a/litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py b/litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py
index 19b9bccd8aa..c773e88c769 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py
@@ -9,7 +9,7 @@ import contextlib
import json
import os
import ssl
-from collections.abc import AsyncGenerator, AsyncIterable, Mapping, Sequence
+from collections.abc import AsyncGenerator, AsyncIterable, Iterator, Mapping, Sequence
from ssl import SSLContext
from typing import TYPE_CHECKING, Any, Final
@@ -23,7 +23,16 @@ from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm._version import version as litellm_version
from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.litellm_core_utils.core_helpers import attribute_items, iter_items
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_calls,
+ write_anthropic_server_tool_call_arguments,
+)
+from litellm.llms.base_llm.guardrail_translation.utils import (
+ claude_hosted_web_tool_arguments,
+ with_web_search_call_arguments,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
get_ssl_configuration,
@@ -515,27 +524,37 @@ class CatoNetworksGuardrail(CustomGuardrail):
@staticmethod
def _output_fragments(message: Message) -> Sequence[tuple[tuple[str, int | None], str]]:
"""Assistant text the proxy returns to the caller: ``content`` plus every
- ``tool_calls[].function.arguments`` string, each tagged with where a
- redaction must be written back. ``content`` is only included when present
- so a tool-call-only choice keeps its ``None`` content (the text-vs-tool-call
- signal downstream consumers rely on) while its arguments are still inspected."""
- fragments: Final[list] = []
- if message.content is not None:
- fragments.append((("content", None), message.content))
- for idx, tool_call in enumerate(message.tool_calls or []):
- function = getattr(tool_call, "function", None)
- arguments = getattr(function, "arguments", None)
- if isinstance(arguments, str) and arguments:
- fragments.append((("tool_call", idx), arguments))
- return fragments
+ tool call's ``function.arguments`` string, Claude's server tool calls first,
+ each tagged with where a redaction must be written back. ``content`` is only
+ included when present so a tool-call-only choice keeps its ``None`` content
+ (the text-vs-tool-call signal downstream consumers rely on) while its
+ arguments are still inspected."""
+ content: Final = () if message.content is None else ((("content", None), message.content),)
+ return (
+ *content,
+ *CatoNetworksGuardrail._tool_call_fragments("server_tool_call", anthropic_server_tool_calls(message)),
+ *CatoNetworksGuardrail._tool_call_fragments("tool_call", message.tool_calls or ()),
+ )
+
+ @staticmethod
+ def _tool_call_fragments(kind: str, tool_calls: Sequence[object]) -> tuple[tuple[tuple[str, int | None], str], ...]:
+ return tuple(
+ ((kind, idx), arguments)
+ for idx, arguments in enumerate(
+ getattr(getattr(tool_call, "function", None), "arguments", None) for tool_call in tool_calls
+ )
+ if isinstance(arguments, str) and arguments
+ )
@staticmethod
def _apply_output_fragment(message: Any, target: tuple[str, int | None], redacted: str) -> None:
- kind, idx = target
- if kind == "content":
- message.content = redacted
- else:
- message.tool_calls[idx].function.arguments = redacted
+ match target:
+ case ("content", _):
+ message.content = redacted
+ case ("server_tool_call", int() as idx):
+ write_anthropic_server_tool_call_arguments(message, idx, redacted)
+ case (_, idx):
+ message.tool_calls[idx].function.arguments = redacted
@staticmethod
def _responses_output_field(item: object, key: str) -> str | Sequence[object] | None:
@@ -565,12 +584,19 @@ class CatoNetworksGuardrail(CustomGuardrail):
return fragments
@staticmethod
- def _apply_responses_output_fragment(container: object, key: str, redacted: str) -> None:
+ def _apply_responses_output_fragment(container: object, key: str, redacted: object) -> None:
if isinstance(container, dict):
container[key] = redacted
else:
setattr(container, key, redacted)
+ @staticmethod
+ def _responses_hosted_web_calls(response: ResponsesAPIResponse) -> Iterator[tuple[object, str]]:
+ for item in attribute_items(response, "output"):
+ hosted_arguments = claude_hosted_web_tool_arguments(item)
+ if hosted_arguments:
+ yield item, hosted_arguments
+
async def _inspect_output_text(
self,
data: dict,
@@ -612,6 +638,12 @@ class CatoNetworksGuardrail(CustomGuardrail):
redacted_output = await self._inspect_output_text(data, text, user_api_key_dict, user_email)
if redacted_output is not None:
self._apply_responses_output_fragment(container, key, redacted_output)
+ for item, hosted_arguments in self._responses_hosted_web_calls(response):
+ redacted_output = await self._inspect_output_text(data, hosted_arguments, user_api_key_dict, user_email)
+ if redacted_output is not None:
+ self._apply_responses_output_fragment(
+ item, "action", message_field(with_web_search_call_arguments(item, redacted_output), "action")
+ )
return response
async def async_post_call_streaming_iterator_hook(
diff --git a/litellm/proxy/guardrails/guardrail_hooks/cisco_ai_defense/cisco_ai_defense.py b/litellm/proxy/guardrails/guardrail_hooks/cisco_ai_defense/cisco_ai_defense.py
index 3d555707159..2920c6bec9a 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/cisco_ai_defense/cisco_ai_defense.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/cisco_ai_defense/cisco_ai_defense.py
@@ -37,6 +37,16 @@ from litellm.integrations.custom_guardrail import (
log_guardrail_information,
)
from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ tool_calls_with_server_tool_calls,
+ with_server_tool_call_arguments,
+ write_anthropic_server_tool_calls,
+)
+from litellm.llms.base_llm.guardrail_translation.utils import (
+ claude_hosted_web_tool_arguments,
+ with_web_search_call_arguments,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -48,7 +58,9 @@ from litellm.proxy.common_utils.callback_utils import (
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.utils import (
Choices,
+ Delta,
LLMResponseTypes,
+ Message,
ModelResponse,
ModelResponseStream,
TextCompletionResponse,
@@ -1654,6 +1666,7 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
@classmethod
def _clear_tool_call_arguments(cls, message: object) -> None:
"""Clear tool-call / function-call arguments after Cisco redaction."""
+ cls._clear_server_tool_call_arguments(message)
tool_calls = message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
for tc in tool_calls or []:
fn = tc.get("function") if isinstance(tc, dict) else getattr(tc, "function", None)
@@ -1663,9 +1676,26 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
)
cls._clear_arguments_field(function_call)
+ @staticmethod
+ def _clear_server_tool_call_arguments(message: object) -> None:
+ cleared: Final = tuple(
+ with_server_tool_call_arguments(server_tool_call, "{}")
+ for server_tool_call in anthropic_server_tool_call_params(message)
+ )
+ match message:
+ case Message() | Delta():
+ write_anthropic_server_tool_calls(message, cleared)
+ case dict() if cleared:
+ message["provider_specific_fields"] = { # rebind-ok: a redaction must land on the message it scanned
+ **(message.get("provider_specific_fields") or {}),
+ "server_tool_calls": list(cleared),
+ }
+ case _:
+ pass
+
def _redact_responses_api_output(
self,
- output_items: list,
+ output_items: list[object],
sanitized_text: str | None,
sanitized_messages: list[dict[str, object]] | None,
) -> bool:
@@ -1677,7 +1707,10 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
if not replacement_text:
return False
applied = False
- for item in output_items:
+ for index, item in enumerate(output_items):
+ if claude_hosted_web_tool_arguments(item):
+ output_items[index] = with_web_search_call_arguments(item, "{}") # rebind-ok: in-place redaction
+ applied = True
content = getattr(item, "content", None) or (item.get("content") if isinstance(item, dict) else None)
if isinstance(content, list):
for part in content:
@@ -2038,8 +2071,9 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
output_items: Final = getattr(response, "output", None)
if not isinstance(output_items, list):
return []
+ typed_output_items: Final[Sequence[object]] = output_items
output_parts: Final[list[str]] = []
- for item in output_items:
+ for item in typed_output_items:
get = item.get if isinstance(item, dict) else (lambda k: getattr(item, k, None))
for part in get("content") or []:
pget = part.get if isinstance(part, dict) else (lambda k: getattr(part, k, None))
@@ -2053,6 +2087,8 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
direct = get("text")
if isinstance(direct, str) and direct:
output_parts.append(direct)
+ if hosted_arguments := claude_hosted_web_tool_arguments(item):
+ output_parts.append(hosted_arguments)
joined = " ".join(output_parts)
return [{"role": "assistant", "content": joined}] if joined else []
@@ -2095,8 +2131,7 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
@classmethod
def _extract_message_tool_argument_parts(cls, message: object) -> list[str]:
parts: Final[list[str]] = []
- tool_calls = message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
- for tool_call in tool_calls or []:
+ for tool_call in tool_calls_with_server_tool_calls(message):
args = cls._extract_tool_call_arguments(tool_call)
if args:
parts.append(args)
diff --git a/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py b/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py
index cb1b223fb47..9c8c9cbe96d 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py
@@ -22,6 +22,7 @@ from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
)
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -163,11 +164,11 @@ class lakeraAI_Moderation(CustomGuardrail):
for message in data["messages"]:
role = message.get("role")
if role in stringified_roles:
- if "tool_calls" in message:
- tool_call_messages = [
- *tool_call_messages,
- *message["tool_calls"],
- ]
+ tool_call_messages = [
+ *tool_call_messages,
+ *anthropic_server_tool_call_params(message),
+ *(message.get("tool_calls") or ()),
+ ]
if role == Role.SYSTEM.value: # we need this for later
system_message = message
continue
@@ -185,10 +186,11 @@ class lakeraAI_Moderation(CustomGuardrail):
if system_message is not None:
if not litellm.add_function_to_prompt:
content = system_message.get("content")
- function_input: Final = []
- for tool_call in tool_call_messages:
- if "function" in tool_call:
- function_input.append(tool_call["function"]["arguments"])
+ function_input: Final[tuple[str, ...]] = tuple(
+ tool_call["function"]["arguments"]
+ for tool_call in tool_call_messages
+ if "function" in tool_call
+ )
if len(function_input) > 0:
content += " Function Input: " + " ".join(function_input)
diff --git a/litellm/proxy/guardrails/guardrail_hooks/lasso/lasso.py b/litellm/proxy/guardrails/guardrail_hooks/lasso/lasso.py
index 97b35769f63..0530cf22f62 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/lasso/lasso.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/lasso/lasso.py
@@ -40,6 +40,13 @@ from litellm.integrations.custom_guardrail import (
)
from litellm.integrations.custom_guardrail import dc as global_cache
from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ anthropic_server_tool_calls,
+ with_server_tool_call_arguments,
+ write_anthropic_server_tool_calls,
+ write_request_server_tool_calls,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -50,6 +57,7 @@ from litellm.proxy.guardrails._content_utils import (
has_non_string_content,
)
from litellm.types.guardrails import GuardrailEventHooks
+from litellm.types.llms.openai import ChatCompletionAssistantToolCall
class LassoResponse(TypedDict):
@@ -259,7 +267,7 @@ class LassoGuardrail(CustomGuardrail):
msg = choice.message
if msg.content:
response_messages.append({"role": "assistant", "content": msg.content})
- for call in getattr(msg, "tool_calls", None) or []:
+ for call in (*anthropic_server_tool_calls(msg), *(getattr(msg, "tool_calls", None) or ())):
call_id, name, input_data = self._extract_tool_call_fields(call)
if not call_id or not name:
continue
@@ -564,6 +572,15 @@ class LassoGuardrail(CustomGuardrail):
elif role == "assistant" and not content and orig_msg.get("tool_calls"):
msg["tool_calls"] = self._update_tool_calls_from_masked(orig_msg["tool_calls"], masked_tool_use)
+ if role == "assistant":
+ write_request_server_tool_calls(
+ msg,
+ tuple(
+ self._masked_server_tool_call(server_tool_call, masked_tool_use)
+ for server_tool_call in anthropic_server_tool_call_params(orig_msg)
+ ),
+ )
+
result.append(msg)
return result
@@ -704,7 +721,7 @@ class LassoGuardrail(CustomGuardrail):
expanded.append({"role": role, "content": content})
if role == "assistant":
- for call in iter_items(msg.get("tool_calls") or []):
+ for call in (*anthropic_server_tool_call_params(msg), *iter_items(msg.get("tool_calls") or [])):
call_id, name, input_data = self._extract_tool_call_fields(call)
if not call_id or not name:
verbose_proxy_logger.warning(
@@ -962,6 +979,14 @@ class LassoGuardrail(CustomGuardrail):
text_cursor += 1
verbose_proxy_logger.debug("Applied masked text content to choice %s", text_cursor)
+ write_anthropic_server_tool_calls(
+ msg,
+ tuple(
+ self._masked_server_tool_call(server_tool_call, masked_tool_use)
+ for server_tool_call in anthropic_server_tool_call_params(msg)
+ ),
+ )
+
for call in getattr(msg, "tool_calls", None) or []:
call_id = self._get_field(call, "id")
if call_id and call_id in masked_tool_use:
@@ -977,6 +1002,16 @@ class LassoGuardrail(CustomGuardrail):
func.arguments = json.dumps(masked_input)
verbose_proxy_logger.debug("Applied masked tool_call arguments for call_id=%s", call_id)
+ @staticmethod
+ def _masked_server_tool_call(
+ server_tool_call: ChatCompletionAssistantToolCall,
+ masked_tool_use: Mapping[object, Mapping[str, object]],
+ ) -> ChatCompletionAssistantToolCall:
+ masked_input: Final = masked_tool_use.get(server_tool_call["id"], {}).get("input")
+ if masked_input is None:
+ return server_tool_call
+ return with_server_tool_call_arguments(server_tool_call, json.dumps(masked_input))
+
@staticmethod
def get_config_model() -> type["GuardrailConfigModel"] | None:
from litellm.types.proxy.guardrails.guardrail_hooks.lasso import (
diff --git a/litellm/proxy/guardrails/guardrail_hooks/llm_shield_proxy/llm_shield_proxy.py b/litellm/proxy/guardrails/guardrail_hooks/llm_shield_proxy/llm_shield_proxy.py
index d2dafb04fe0..5e7a0240545 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/llm_shield_proxy/llm_shield_proxy.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/llm_shield_proxy/llm_shield_proxy.py
@@ -24,13 +24,17 @@ from litellm.integrations.custom_guardrail import (
log_guardrail_information,
)
from litellm.litellm_core_utils.core_helpers import attribute_items, attribute_or_none, iter_items, require_attributes
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ write_anthropic_server_tool_call_arguments,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.guardrails import GuardrailEventHooks
-from litellm.types.utils import GenericGuardrailAPIInputs, TextChoices
+from litellm.types.utils import GenericGuardrailAPIInputs, Message, TextChoices
if TYPE_CHECKING:
from litellm.caching.caching import DualCache
@@ -87,6 +91,19 @@ _DEFAULT_TIMEOUT_SECONDS: Final = 10.0
_SHIELD_BODY: Final = TypeAdapter(dict[str, object])
+def _server_tool_call_slots(message: object) -> tuple[Slot, ...]:
+ if not isinstance(message, Message):
+ return ()
+ return tuple(
+ (arguments, functools.partial(write_anthropic_server_tool_call_arguments, message, index))
+ for index, arguments in enumerate(
+ server_tool_call["function"].get("arguments")
+ for server_tool_call in anthropic_server_tool_call_params(message)
+ )
+ if isinstance(arguments, str) and arguments
+ )
+
+
def _string_tool_arguments(calls: Sequence[object]) -> Iterator[tuple[object, str]]:
for call in calls:
function = read_field(call, "function")
@@ -382,6 +399,7 @@ class LLMShieldProxyGuardrail(CustomGuardrail):
content = getattr(message, "content", None)
if isinstance(content, str) and content:
pending.append((content, functools.partial(setattr, message, "content")))
+ pending.extend(_server_tool_call_slots(message))
for tool_call in attribute_items(message, "tool_calls"):
function: object = getattr(tool_call, "function", None)
arguments = getattr(function, "arguments", None) if function is not None else None
diff --git a/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/base.py b/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/base.py
index f83314af548..1956170ba67 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/base.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/base.py
@@ -13,6 +13,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
convert_content_list_to_str,
)
from litellm.litellm_core_utils.url_utils import encode_url_path_segment
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -434,9 +435,7 @@ class PurviewGuardrailBase:
args: Final[list[str]] = []
# tool_calls: [{"function": {"arguments": "..."}}]
- tool_calls: Final[Sequence[object] | None] = (
- message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
- )
+ tool_calls: Final = tool_calls_with_server_tool_calls(message)
if tool_calls:
for tc in tool_calls:
fn = tc.get("function") if isinstance(tc, dict) else getattr(tc, "function", None)
diff --git a/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/purview_dlp.py b/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/purview_dlp.py
index 72474ba27c3..0b3d56a016b 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/purview_dlp.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/purview_dlp.py
@@ -22,6 +22,7 @@ from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
)
+from litellm.llms.base_llm.guardrail_translation.utils import claude_hosted_web_tool_arguments
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.utils import (
Choices,
@@ -227,6 +228,9 @@ class MicrosoftPurviewDLPGuardrail(PurviewGuardrailBase, CustomGuardrail):
if not output:
return args
for item in output:
+ hosted_arguments = claude_hosted_web_tool_arguments(item)
+ if hosted_arguments:
+ args.append(hosted_arguments)
if isinstance(item, dict):
item_type = item.get("type")
arguments = item.get("arguments")
diff --git a/litellm/proxy/guardrails/guardrail_hooks/pangea/pangea.py b/litellm/proxy/guardrails/guardrail_hooks/pangea/pangea.py
index 2194238cea0..4ec3ed21359 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/pangea/pangea.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/pangea/pangea.py
@@ -1,5 +1,6 @@
# litellm/proxy/guardrails/guardrail_hooks/pangea.py
import os
+from collections.abc import Callable
from typing import TYPE_CHECKING, Final
from fastapi import HTTPException
@@ -10,6 +11,11 @@ from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
)
+from litellm.llms.anthropic.common_utils import (
+ message_dict_with_server_tool_calls,
+ message_dict_with_server_tool_calls_split_out,
+ validated_object_mapping,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -30,6 +36,13 @@ if TYPE_CHECKING:
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel
+def _choice_with_message(choice: object, rewrite: Callable[[object], object]) -> object:
+ mapping: Final = validated_object_mapping(choice)
+ if mapping is None or "message" not in mapping:
+ return choice
+ return {**mapping, "message": rewrite(mapping["message"])}
+
+
class PangeaGuardrailMissingSecrets(Exception):
"""Custom exception for missing Pangea secrets."""
@@ -242,9 +255,13 @@ class PangeaHandler(CustomGuardrail):
for c in choices:
if isinstance(c, Choices):
try:
- serialized_choices.append(c.model_dump())
+ serialized_choices.append(
+ _choice_with_message(c.model_dump(), message_dict_with_server_tool_calls)
+ )
except Exception:
- serialized_choices.append(c.dict())
+ serialized_choices.append(
+ _choice_with_message(c.dict(), message_dict_with_server_tool_calls)
+ )
else:
serialized_choices.append(c)
choices = serialized_choices
@@ -269,7 +286,9 @@ class PangeaHandler(CustomGuardrail):
return
output: Final = ai_guard_response.get("result", {}).get("output", {})
- response.choices = output["choices"]
+ response.choices = [ # pyright: ignore[reportAttributeAccessIssue] # Pangea returns untyped choice dicts
+ _choice_with_message(choice, message_dict_with_server_tool_calls_split_out) for choice in output["choices"]
+ ]
return response
@log_guardrail_information
diff --git a/litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py b/litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py
index 3cd9dc6a7b4..379c2c4b075 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py
@@ -27,6 +27,12 @@ from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
)
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ tool_calls_with_server_tool_calls,
+ with_server_tool_call_arguments,
+ write_anthropic_server_tool_calls,
+)
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_scan_only_tool_results_for_guardrail,
effective_skip_system_message_for_guardrail,
@@ -364,10 +370,9 @@ class PanwPrismaAirsHandler(CustomGuardrail):
text_parts.append(str(choice.message.content))
# Extract tool call arguments
- if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
- for tool_call in choice.message.tool_calls:
- if hasattr(tool_call, "function") and hasattr(tool_call.function, "arguments"):
- text_parts.append(str(tool_call.function.arguments))
+ for tool_call in tool_calls_with_server_tool_calls(choice.message):
+ if hasattr(tool_call, "function") and hasattr(tool_call.function, "arguments"):
+ text_parts.append(str(tool_call.function.arguments))
# Extract function call arguments (legacy)
if hasattr(choice.message, "function_call") and choice.message.function_call:
@@ -756,6 +761,14 @@ class PanwPrismaAirsHandler(CustomGuardrail):
elif isinstance(content, list):
choice.message.content = self._mask_content_list(content, masked_text)
+ write_anthropic_server_tool_calls(
+ choice.message,
+ tuple(
+ with_server_tool_call_arguments(server_tool_call, masked_text)
+ for server_tool_call in anthropic_server_tool_call_params(choice.message)
+ ),
+ )
+
# Mask tool call arguments
if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
for tool_call in choice.message.tool_calls:
diff --git a/litellm/proxy/guardrails/guardrail_hooks/pillar/pillar.py b/litellm/proxy/guardrails/guardrail_hooks/pillar/pillar.py
index 3f025cfc8b3..e136b9baaa8 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/pillar/pillar.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/pillar/pillar.py
@@ -26,6 +26,7 @@ from litellm.integrations.custom_guardrail import (
log_guardrail_information,
)
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
+from litellm.llms.anthropic.common_utils import message_dict_with_server_tool_calls
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -205,6 +206,7 @@ class PillarGuardrail(CustomGuardrail):
include_evidence: bool | None = None,
fallback_on_error: str | None = None,
timeout: float | None = None,
+ async_handler: _PillarProtectHTTPClient | None = None,
**kwargs,
) -> None:
"""
@@ -223,8 +225,10 @@ class PillarGuardrail(CustomGuardrail):
LiteLLM virtual key context (user_id, team_id, key_alias, etc.) is always
automatically passed as X-LiteLLM-* headers to enable application/user tracking.
"""
- self.async_handler: _PillarProtectHTTPClient = get_async_httpx_client(
- llm_provider=httpxSpecialProvider.GuardrailCallback
+ self.async_handler: _PillarProtectHTTPClient = (
+ async_handler
+ if async_handler is not None
+ else get_async_httpx_client(llm_provider=httpxSpecialProvider.GuardrailCallback)
)
self.api_key = api_key or os.environ.get("PILLAR_API_KEY")
@@ -430,7 +434,9 @@ class PillarGuardrail(CustomGuardrail):
# Extract response messages in the format Pillar expects
response_dict = response.model_dump() if hasattr(response, "model_dump") else {}
response_messages: Final = [
- choice.get("message") for choice in response_dict.get("choices", []) if choice.get("message")
+ message_dict_with_server_tool_calls(choice["message"])
+ for choice in response_dict.get("choices", [])
+ if choice.get("message")
]
if not response_messages:
diff --git a/litellm/proxy/guardrails/guardrail_hooks/presidio.py b/litellm/proxy/guardrails/guardrail_hooks/presidio.py
index 458528e7e13..1e4621622b3 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/presidio.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/presidio.py
@@ -12,7 +12,7 @@ import asyncio
import json
import re
import threading
-from collections.abc import AsyncGenerator, AsyncIterable, AsyncIterator, Awaitable, Iterator, Sequence
+from collections.abc import AsyncGenerator, AsyncIterable, AsyncIterator, Awaitable, Callable, Iterator, Sequence
from contextlib import asynccontextmanager
from dataclasses import dataclass
from datetime import datetime
@@ -43,6 +43,11 @@ from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
)
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ with_server_tool_call_arguments,
+ write_anthropic_server_tool_calls,
+)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.common_utils.sse_keepalive import split_complete_sse_frames
from litellm.proxy.guardrails.anthropic_sse import (
@@ -1404,6 +1409,18 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
verbose_proxy_logger.debug("No pii_tokens found in request_data['metadata'] — nothing to unmask")
presidio_config: Final = self.get_presidio_settings_from_request_data(request_data or {})
+ async def pii_processed_text(text: str) -> str:
+ match mode:
+ case "unmask":
+ return self._unmask_pii_text(text, pii_tokens)
+ case "mask":
+ return await self.check_pii(
+ text=text,
+ output_parse_pii=False,
+ presidio_config=presidio_config,
+ request_data=request_data,
+ )
+
for choice in response.choices:
message: Message | None = getattr(choice, "message", None)
if message is None:
@@ -1412,15 +1429,7 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
# 1. Process content
content = getattr(message, "content", None)
if isinstance(content, str):
- if mode == "unmask":
- message.content = self._unmask_pii_text(content, pii_tokens)
- elif mode == "mask":
- message.content = await self.check_pii(
- text=content,
- output_parse_pii=False,
- presidio_config=presidio_config,
- request_data=request_data,
- )
+ message.content = await pii_processed_text(content)
elif isinstance(content, list):
for item in content:
if not isinstance(item, dict):
@@ -1428,54 +1437,43 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
text_value = item.get("text")
if text_value is None:
continue
- if mode == "unmask":
- item["text"] = self._unmask_pii_text(text_value, pii_tokens)
- elif mode == "mask":
- item["text"] = await self.check_pii(
- text=text_value,
- output_parse_pii=False,
- presidio_config=presidio_config,
- request_data=request_data,
- )
+ item["text"] = await pii_processed_text(text_value)
+
+ await self._process_server_tool_calls_for_pii(message, pii_processed_text)
# 2. Process tool calls
for tool_call in attribute_items(message, "tool_calls"):
function = attribute_or_none(tool_call, "function")
args = attribute_or_none(function, "arguments") if function else None
if isinstance(args, str):
- if mode == "unmask":
- _write_attribute(function, "arguments", self._unmask_pii_text(args, pii_tokens))
- elif mode == "mask":
- _write_attribute(
- function,
- "arguments",
- await self.check_pii(
- text=args,
- output_parse_pii=False,
- presidio_config=presidio_config,
- request_data=request_data,
- ),
- )
+ _write_attribute(function, "arguments", await pii_processed_text(args))
# 3. Process legacy function calls
function_call = attribute_or_none(message, "function_call")
legacy_args = attribute_or_none(function_call, "arguments") if function_call else None
if isinstance(legacy_args, str):
- if mode == "unmask":
- _write_attribute(function_call, "arguments", self._unmask_pii_text(legacy_args, pii_tokens))
- elif mode == "mask":
- _write_attribute(
- function_call,
- "arguments",
- await self.check_pii(
- text=legacy_args,
- output_parse_pii=False,
- presidio_config=presidio_config,
- request_data=request_data,
- ),
- )
+ _write_attribute(function_call, "arguments", await pii_processed_text(legacy_args))
return response
+ @staticmethod
+ async def _process_server_tool_calls_for_pii(
+ message: Message,
+ pii_processed_text: Callable[[str], Awaitable[str]],
+ ) -> None:
+ server_tool_calls: Final = anthropic_server_tool_call_params(message)
+ if not server_tool_calls:
+ return
+ processed_arguments: Final = tuple(
+ [
+ await pii_processed_text(server_tool_call["function"].get("arguments", ""))
+ for server_tool_call in server_tool_calls
+ ]
+ )
+ write_anthropic_server_tool_calls(
+ message,
+ tuple(map(with_server_tool_call_arguments, server_tool_calls, processed_arguments)),
+ )
+
async def _mask_output_response(
self,
response: ModelResponse | EmbeddingResponse | ImageResponse,
diff --git a/litellm/proxy/guardrails/guardrail_hooks/qualifire/qualifire.py b/litellm/proxy/guardrails/guardrail_hooks/qualifire/qualifire.py
index 6150fccac0c..c76e8e9c084 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/qualifire/qualifire.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/qualifire/qualifire.py
@@ -18,6 +18,7 @@ from litellm.integrations.custom_guardrail import (
log_guardrail_information,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@@ -180,8 +181,8 @@ class QualifireGuardrail(CustomGuardrail):
api_message["tool_call_id"] = tool_call_id
# Handle tool calls if present
- tool_calls = msg.get("tool_calls")
- if tool_calls and isinstance(tool_calls, list):
+ tool_calls = [*anthropic_server_tool_call_params(msg), *(msg.get("tool_calls") or [])]
+ if tool_calls:
api_tool_calls = []
for tc in tool_calls:
if isinstance(tc, dict):
diff --git a/litellm/proxy/guardrails/guardrail_hooks/repelloai/repelloai.py b/litellm/proxy/guardrails/guardrail_hooks/repelloai/repelloai.py
index b1f0f588ade..2e1ac62d850 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/repelloai/repelloai.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/repelloai/repelloai.py
@@ -12,6 +12,7 @@ from pydantic import BaseModel, TypeAdapter, ValidationError
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_guardrail import CustomGuardrail
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client, # pyright: ignore[reportUnknownVariableType]
httpxSpecialProvider,
@@ -80,13 +81,11 @@ class RepelloAIGuardrail(CustomGuardrail):
def _extract_tool_call_args_from_message(cls, message: object) -> list[str]:
args: Final[list[str]] = []
- tool_calls: Final = cls._get_field(message, "tool_calls")
- if _is_object_list(tool_calls):
- for tool_call in tool_calls:
- function = cls._get_field(tool_call, "function")
- arguments = cls._get_field(function, "arguments")
- if isinstance(arguments, str) and arguments.strip():
- args.append(arguments)
+ for tool_call in tool_calls_with_server_tool_calls(message):
+ function = cls._get_field(tool_call, "function")
+ arguments = cls._get_field(function, "arguments")
+ if isinstance(arguments, str) and arguments.strip():
+ args.append(arguments)
function_call: Final = cls._get_field(message, "function_call")
arguments = cls._get_field(function_call, "arguments")
diff --git a/litellm/proxy/guardrails/guardrail_hooks/tool_permission.py b/litellm/proxy/guardrails/guardrail_hooks/tool_permission.py
index 5f99b1a4611..f2697c0aa11 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/tool_permission.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/tool_permission.py
@@ -24,6 +24,11 @@ from litellm.integrations.custom_guardrail import (
)
from litellm.litellm_core_utils.core_helpers import as_str_mapping, attribute_or_none
from litellm.litellm_core_utils.tool_fields import declared_function_name, qualified_member_name
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ anthropic_server_tool_calls,
+ without_anthropic_server_tool_calls,
+)
from litellm.llms.base_llm.guardrail_translation.utils import anthropic_tool_names
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.common_utils.callback_utils import (
@@ -983,7 +988,7 @@ class ToolPermissionGuardrail(CustomGuardrail):
for choice_index, choice in enumerate(response.choices):
if isinstance(choice, Choices):
- for tool in choice.message.tool_calls or []:
+ for tool in (*(choice.message.tool_calls or ()), *anthropic_server_tool_calls(choice.message)):
tool_calls.append(tool)
legacy_tool_call = self._legacy_function_call_to_tool_call(
getattr(choice.message, "function_call", None), choice_index
@@ -1493,6 +1498,23 @@ class ToolPermissionGuardrail(CustomGuardrail):
choice.message.tool_calls = filtered_tool_calls if filtered_tool_calls else None
+ server_error_results = tuple(
+ error_results.get(self._judged_tool_call(server_tool_call))
+ for server_tool_call in anthropic_server_tool_calls(choice.message)
+ )
+ if any(server_error_results):
+ error_messages.extend(result.content for result in server_error_results if result is not None)
+ without_anthropic_server_tool_calls(
+ choice.message,
+ frozenset(
+ server_tool_call["id"] or ""
+ for server_tool_call, result in zip(
+ anthropic_server_tool_call_params(choice.message), server_error_results
+ )
+ if result is not None
+ ),
+ )
+
legacy_tool_call = self._legacy_function_call_to_tool_call(
getattr(choice.message, "function_call", None), choice_index
)
diff --git a/litellm/proxy/guardrails/guardrail_hooks/tool_policy/tool_policy_guardrail.py b/litellm/proxy/guardrails/guardrail_hooks/tool_policy/tool_policy_guardrail.py
index fbf1f2576ff..1feb6d533f4 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/tool_policy/tool_policy_guardrail.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/tool_policy/tool_policy_guardrail.py
@@ -27,7 +27,10 @@ from pydantic import TypeAdapter
from typing_extensions import TypedDict, Unpack
from litellm._logging import verbose_proxy_logger
-from litellm.completion_extras.litellm_responses_transformation.transformation import replayed_tool_search_tool_names
+from litellm.completion_extras.litellm_responses_transformation.transformation import (
+ nested_tool_member_names,
+ replayed_tool_search_tool_names,
+)
from litellm.integrations.custom_guardrail import (
CustomGuardrail,
log_guardrail_information,
@@ -47,6 +50,14 @@ class _CustomGuardrailOptions(TypedDict, total=False, extra_items=object):
"""Base-class constructor options this guardrail forwards untouched to CustomGuardrail."""
+def _function_tool_name(tool: object) -> str | None:
+ match tool:
+ case {"function": {"name": str(name)}}:
+ return name
+ case _:
+ return None
+
+
def _attribute(source: object, name: str) -> object:
return getattr(source, name, None)
@@ -140,17 +151,18 @@ class ToolPolicyGuardrail(CustomGuardrail):
"""
if input_type == "request":
tools: Final = inputs.get("tools") or []
- function_tool_names: Final = [
- t["function"]["name"]
- for t in tools
- if isinstance(t, dict) and isinstance(t.get("function"), dict) and t["function"].get("name")
- ]
+ function_tool_names: Final = [name for name in map(_function_tool_name, tools) if name]
route: Final = _get_request_route_from_data(request_data)
- replayed_tool_names: Final = replayed_tool_search_tool_names(_REQUEST_DATA.validate_python(request_data))
- tool_names = (
- extract_request_tool_names(route, request_data)
- if route and not function_tool_names
- else [*function_tool_names, *replayed_tool_names]
+ request: Final = _REQUEST_DATA.validate_python(request_data)
+ tool_names = list(
+ dict.fromkeys(
+ (
+ *function_tool_names,
+ *(extract_request_tool_names(route, request_data) if route else ()),
+ *replayed_tool_search_tool_names(request),
+ *nested_tool_member_names(request),
+ )
+ )
)
else:
tool_calls: Final = inputs.get("tool_calls") or []
diff --git a/litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py b/litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
index 5ce2dfb3b87..6f6d056423f 100644
--- a/litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
+++ b/litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
@@ -25,6 +25,7 @@ from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.api_route_to_call_types import get_call_types_for_route
from litellm.litellm_core_utils.core_helpers import get_or_create_metadata_bucket
from litellm.llms import get_guardrail_translation_mapping, load_guardrail_translation_mappings
+from litellm.llms.anthropic.common_utils import anthropic_server_call_fields
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.utils import (
@@ -144,6 +145,15 @@ def _ends_stream(endpoint_translation: object, item: object) -> bool:
return has_ended is not None and has_ended((item,))
+def _passthrough_delta(delta: object) -> Delta:
+ server_call_fields: Final = anthropic_server_call_fields(delta)
+ role: Final = getattr(delta, "role", None)
+ tool_calls: Final = getattr(delta, "tool_calls", None)
+ if not server_call_fields:
+ return Delta(content=None, role=role, tool_calls=tool_calls)
+ return Delta(content=None, role=role, tool_calls=tool_calls, provider_specific_fields=dict(server_call_fields))
+
+
def _held_choices(held_chars_per_choice: Mapping[int, int]) -> frozenset[int]:
return frozenset(idx for idx, held in held_chars_per_choice.items() if held > 0)
@@ -974,7 +984,7 @@ class UnifiedLLMGuardrails(CustomLogger):
def _chunk_has_tool_calls(item: object) -> bool:
for choice in _chunk_choices(item):
delta = getattr(choice, "delta", None)
- if getattr(delta, "tool_calls", None):
+ if getattr(delta, "tool_calls", None) or anthropic_server_call_fields(delta):
return True
return False
@@ -1042,11 +1052,7 @@ class UnifiedLLMGuardrails(CustomLogger):
synthetic_choices.append(
StreamingChoices(
index=idx,
- delta=Delta(
- content=None,
- role=getattr(delta, "role", None),
- tool_calls=getattr(delta, "tool_calls", None),
- ),
+ delta=_passthrough_delta(delta),
finish_reason=passthrough_finish,
)
)
diff --git a/litellm/proxy/hooks/model_max_budget_limiter.py b/litellm/proxy/hooks/model_max_budget_limiter.py
index 889fb9fb373..4daef2592f7 100644
--- a/litellm/proxy/hooks/model_max_budget_limiter.py
+++ b/litellm/proxy/hooks/model_max_budget_limiter.py
@@ -27,6 +27,7 @@ from litellm.litellm_core_utils.batch_utils import (
)
from litellm.litellm_core_utils.duration_parser import duration_in_seconds
from litellm.litellm_core_utils.litellm_logging import StandardLoggingPayloadSetup
+from litellm.litellm_core_utils.model_latest_aliases import expand_model_latest_alias
from litellm.llms.bedrock.common_utils import get_bedrock_base_model, get_bedrock_cross_region_inference_regions
from litellm.proxy._types import Litellm_EntityType, UserAPIKeyAuth
from litellm.proxy.hooks.model_budget_shared_windows import (
@@ -523,6 +524,25 @@ def _usable_entry(entry: str, raw_budget_config: object) -> ResolvedModelBudget
def team_model_budget_entries(
model: str, team_model_max_budget: Mapping[str, object], key_model_max_budget: Mapping[str, object] | None
+) -> Mapping[str, object]:
+ models: Final = expand_model_latest_alias(model)
+ if len(models) == 1:
+ return _single_team_model_budget_entries(model, team_model_max_budget, key_model_max_budget)
+ entries: Final = tuple(
+ _single_team_model_budget_entries(candidate, team_model_max_budget, key_model_max_budget)
+ for candidate in models
+ )
+ return MappingProxyType(
+ {
+ entry: config
+ for entry, config in team_model_max_budget.items()
+ if all(entry in mapping for mapping in entries)
+ }
+ )
+
+
+def _single_team_model_budget_entries(
+ model: str, team_model_max_budget: Mapping[str, object], key_model_max_budget: Mapping[str, object] | None
) -> Mapping[str, object]:
"""The team entries that budget `model` for this key.
@@ -780,6 +800,31 @@ def _as_float(value: object) -> float | None:
return None
+def _entity_model_budget_entries(
+ model: str,
+ entity_budget: tuple[Litellm_EntityType, str | None, object],
+ model_access_groups: ModelAccessGroups,
+ team_id: str | None,
+) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]:
+ entity_type, entity_id, model_max_budget = entity_budget
+ if entity_id is None or not isinstance(model_max_budget, Mapping) or not model_max_budget:
+ return ()
+ return tuple(
+ (entity_type, entity_id, resolved)
+ for candidate in expand_model_latest_alias(model)
+ if (
+ resolved := resolve_model_budget(
+ model=candidate,
+ model_max_budget=model_max_budget,
+ model_access_groups=model_access_groups,
+ team_id=team_id,
+ )
+ )
+ is not None
+ and resolved.budget_config.budget_duration is not None
+ )
+
+
def _resolve_entity_model_budgets(
model: str,
entity_budgets: Iterable[tuple[Litellm_EntityType, str | None, object]],
@@ -787,19 +832,15 @@ def _resolve_entity_model_budgets(
team_id: str | None,
) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]:
"""Drop the scopes that do not budget `model`, keeping only what can be incremented."""
+ entries: Final = itertools.chain.from_iterable(
+ _entity_model_budget_entries(model, entity_budget, model_access_groups, team_id)
+ for entity_budget in entity_budgets
+ )
return tuple(
- (entity_type, entity_id, resolved)
- for entity_type, entity_id, model_max_budget in entity_budgets
- if entity_id is not None and isinstance(model_max_budget, Mapping) and model_max_budget
- for resolved in (
- resolve_model_budget(
- model=model,
- model_max_budget=model_max_budget,
- model_access_groups=model_access_groups,
- team_id=team_id,
- ),
- )
- if resolved is not None and resolved.budget_config.budget_duration is not None
+ {
+ (entity_type, entity_id, resolved.budget_model): (entity_type, entity_id, resolved)
+ for entity_type, entity_id, resolved in entries
+ }.values()
)
@@ -996,6 +1037,26 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting):
model_max_budget: Mapping[str, object] | None,
model: str,
exceeded_message: str,
+ ) -> bool:
+ for candidate in expand_model_latest_alias(model):
+ await self._is_single_entity_within_model_budget(
+ entity_type=entity_type,
+ entity_id=entity_id,
+ team_id=team_id,
+ model_max_budget=model_max_budget,
+ model=candidate,
+ exceeded_message=exceeded_message,
+ )
+ return True
+
+ async def _is_single_entity_within_model_budget(
+ self,
+ entity_type: Litellm_EntityType,
+ entity_id: str | None,
+ team_id: str | None,
+ model_max_budget: Mapping[str, object] | None,
+ model: str,
+ exceeded_message: str,
) -> bool:
if not model_max_budget:
return True
diff --git a/litellm/proxy/hooks/parallel_request_limiter_v3.py b/litellm/proxy/hooks/parallel_request_limiter_v3.py
index 01766d81150..33a14bfbeda 100644
--- a/litellm/proxy/hooks/parallel_request_limiter_v3.py
+++ b/litellm/proxy/hooks/parallel_request_limiter_v3.py
@@ -50,6 +50,7 @@ from litellm.litellm_core_utils.add_retry_fallback_headers import (
response_has_hidden_params,
)
from litellm.litellm_core_utils.core_helpers import as_str_mapping
+from litellm.litellm_core_utils.model_latest_aliases import expand_model_latest_alias
from litellm.litellm_core_utils.prompt_templates.common_utils import (
get_str_from_messages,
)
@@ -74,6 +75,7 @@ from litellm.proxy.auth.auth_utils import (
get_model_rate_limit_from_metadata,
)
from litellm.proxy.auth.budget_throttle import throttled_limit
+from litellm.proxy.common_utils.callback_utils import get_model_group_from_litellm_kwargs
from litellm.proxy.common_utils.http_parsing_utils import get_tags_from_request_body
from litellm.proxy.common_utils.proxy_rate_limit_error import (
ProxyRateLimitError,
@@ -348,6 +350,44 @@ class RateLimitDescriptor(TypedDict):
rate_limit: RateLimitDescriptorRateLimitObject | None
+def _minimum_descriptor_limit(
+ descriptors: Sequence[RateLimitDescriptor],
+ metric: Literal["requests_per_unit", "tokens_per_unit", "max_parallel_requests"],
+) -> int | None:
+ return min(
+ (limit for descriptor in descriptors if (limit := (descriptor["rate_limit"] or {}).get(metric)) is not None),
+ default=None,
+ )
+
+
+def _merge_rate_limit_descriptors(descriptors: Sequence[RateLimitDescriptor]) -> RateLimitDescriptor:
+ first: Final = descriptors[0]
+ if len(descriptors) == 1:
+ return first
+ return RateLimitDescriptor(
+ key=first["key"],
+ value=first["value"],
+ rate_limit=RateLimitDescriptorRateLimitObject(
+ requests_per_unit=_minimum_descriptor_limit(descriptors, "requests_per_unit"),
+ tokens_per_unit=_minimum_descriptor_limit(descriptors, "tokens_per_unit"),
+ max_parallel_requests=_minimum_descriptor_limit(descriptors, "max_parallel_requests"),
+ window_size=(first["rate_limit"] or {}).get("window_size"),
+ ),
+ )
+
+
+def _coalesce_rate_limit_descriptors(descriptors: Sequence[RateLimitDescriptor]) -> Sequence[RateLimitDescriptor]:
+ identities: Final = tuple(dict.fromkeys((descriptor["key"], descriptor["value"]) for descriptor in descriptors))
+ if len(identities) == len(descriptors):
+ return descriptors
+ return tuple(
+ _merge_rate_limit_descriptors(
+ tuple(descriptor for descriptor in descriptors if (descriptor["key"], descriptor["value"]) == identity)
+ )
+ for identity in identities
+ )
+
+
class ParallelRequestGauge(TypedDict):
scope: RateLimitScope
limit: int
@@ -2405,6 +2445,18 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
group=self._model_group_resolver(requested_model) or requested_model,
)
+ def _rate_limited_models(self, model: RateLimitedModel | None) -> tuple[RateLimitedModel, ...]:
+ if model is None:
+ return ()
+ return (
+ model,
+ *(
+ target
+ for candidate in expand_model_latest_alias(model.requested)[1:]
+ if (target := self._rate_limited_model(candidate)) is not None
+ ),
+ )
+
def create_organization_rate_limit_descriptor(
self, user_api_key_dict: UserAPIKeyAuth, requested_model: str | None = None
) -> list[RateLimitDescriptor]:
@@ -2426,9 +2478,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
)
- model: Final = self._rate_limited_model(requested_model)
- if model is None:
- return descriptors
+ model_descriptors: Final = tuple(
+ descriptor
+ for model in self._rate_limited_models(self._rate_limited_model(requested_model))
+ if (descriptor := self._organization_model_rate_limit_descriptor(user_api_key_dict, model)) is not None
+ )
+ return [*descriptors, *_coalesce_rate_limit_descriptors(model_descriptors)]
+
+ def _organization_model_rate_limit_descriptor(
+ self, user_api_key_dict: UserAPIKeyAuth, model: RateLimitedModel
+ ) -> RateLimitDescriptor | None:
model_specific_tpm_limit: Final = model.limit_in(
get_model_rate_limit_from_metadata(user_api_key_dict, "organization_metadata", "model_tpm_limit")
)
@@ -2436,19 +2495,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
get_model_rate_limit_from_metadata(user_api_key_dict, "organization_metadata", "model_rpm_limit")
)
if model_specific_tpm_limit is None and model_specific_rpm_limit is None:
- return descriptors
- descriptors.append(
- RateLimitDescriptor(
- key="model_per_organization",
- value=f"{user_api_key_dict.org_id}:{model.group}",
- rate_limit={
- "requests_per_unit": model_specific_rpm_limit,
- "tokens_per_unit": model_specific_tpm_limit,
- "window_size": self.window_size,
- },
- )
+ return None
+ return RateLimitDescriptor(
+ key="model_per_organization",
+ value=f"{user_api_key_dict.org_id}:{model.group}",
+ rate_limit={
+ "requests_per_unit": model_specific_rpm_limit,
+ "tokens_per_unit": model_specific_tpm_limit,
+ "window_size": self.window_size,
+ },
)
- return descriptors
def _add_model_per_key_rate_limit_descriptor(
self,
@@ -2935,11 +2991,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
# Model rate limits
requested_model: Final = data.get("model", None)
- self._add_model_per_key_rate_limit_descriptor(
- user_api_key_dict=user_api_key_dict,
- requested_model=requested_model,
- descriptors=descriptors,
- )
+ model_candidates: Final = expand_model_latest_alias(requested_model if isinstance(requested_model, str) else "")
+ for candidate in model_candidates:
+ self._add_model_per_key_rate_limit_descriptor(
+ user_api_key_dict=user_api_key_dict,
+ requested_model=candidate,
+ descriptors=descriptors,
+ )
# Per-request-tag rate limits scoped to this key
self._add_tag_per_key_rate_limit_descriptor(
@@ -2948,11 +3006,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
descriptors=descriptors,
)
- self._add_team_model_rate_limit_descriptor_from_metadata(
- user_api_key_dict=user_api_key_dict,
- requested_model=requested_model if isinstance(requested_model, str) else None,
- descriptors=descriptors,
- )
+ for candidate in model_candidates:
+ self._add_team_model_rate_limit_descriptor_from_metadata(
+ user_api_key_dict=user_api_key_dict,
+ requested_model=candidate,
+ descriptors=descriptors,
+ )
self._add_team_ptu_rate_limit_descriptor(
user_api_key_dict=user_api_key_dict,
requested_model=requested_model if isinstance(requested_model, str) else None,
@@ -2975,7 +3034,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
)
- return descriptors
+ return descriptors if len(model_candidates) == 1 else list(_coalesce_rate_limit_descriptors(descriptors))
_create_rate_limit_descriptors = create_rate_limit_descriptors
@@ -3126,12 +3185,26 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
self,
user_api_key_dict: UserAPIKeyAuth,
requested_model: str | None,
- descriptors: list[RateLimitDescriptor],
+ descriptors: _RateLimitDescriptorSink,
) -> None:
+ candidates: Final = expand_model_latest_alias(requested_model) if requested_model else (None,)
+ model_descriptors: Final = tuple(
+ descriptor
+ for candidate in candidates
+ if (descriptor := self._project_model_rate_limit_descriptor(user_api_key_dict, candidate)) is not None
+ )
+ for descriptor in _coalesce_rate_limit_descriptors(model_descriptors):
+ descriptors.append(descriptor)
+
+ def _project_model_rate_limit_descriptor(
+ self,
+ user_api_key_dict: UserAPIKeyAuth,
+ requested_model: str | None,
+ ) -> RateLimitDescriptor | None:
"""Add project model rate limit descriptor from project_metadata if applicable."""
model: Final = self._rate_limited_model(requested_model)
if model is None:
- return
+ return None
model_specific_tpm_limit: Final = model.limit_in(
get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_tpm_limit")
)
@@ -3139,17 +3212,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_rpm_limit")
)
if model_specific_tpm_limit is None and model_specific_rpm_limit is None:
- return
- descriptors.append(
- RateLimitDescriptor(
- key="model_per_project",
- value=f"{user_api_key_dict.project_id}:{model.group}",
- rate_limit={
- "requests_per_unit": model_specific_rpm_limit,
- "tokens_per_unit": model_specific_tpm_limit,
- "window_size": self.window_size,
- },
- )
+ return None
+ return RateLimitDescriptor(
+ key="model_per_project",
+ value=f"{user_api_key_dict.project_id}:{model.group}",
+ rate_limit={
+ "requests_per_unit": model_specific_rpm_limit,
+ "tokens_per_unit": model_specific_tpm_limit,
+ "window_size": self.window_size,
+ },
)
def add_project_io_token_rate_limit_descriptors_from_metadata(
@@ -3158,6 +3229,20 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
requested_model: str | None,
descriptors: _RateLimitDescriptorSink,
) -> None:
+ candidates: Final = expand_model_latest_alias(requested_model) if requested_model else (None,)
+ model_descriptors: Final = tuple(
+ itertools.chain.from_iterable(
+ self._project_io_token_rate_limit_descriptors(user_api_key_dict, candidate) for candidate in candidates
+ )
+ )
+ for descriptor in _coalesce_rate_limit_descriptors(model_descriptors):
+ descriptors.append(descriptor)
+
+ def _project_io_token_rate_limit_descriptors(
+ self,
+ user_api_key_dict: UserAPIKeyAuth,
+ requested_model: str | None,
+ ) -> tuple[RateLimitDescriptor, ...]:
"""Add project-scoped ITPM/OTPM descriptors from project_metadata.
Enforced independently of, and alongside, the combined ``model_per_project``
@@ -3166,7 +3251,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
"""
model: Final = self._rate_limited_model(requested_model)
if model is None or user_api_key_dict.project_id is None:
- return
+ return ()
model_itpm_limit: Final = model.limit_in(
get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_itpm_limit")
@@ -3175,34 +3260,23 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_otpm_limit")
)
- if model_itpm_limit is None and model_otpm_limit is None:
- return
-
descriptor_value: Final = f"{user_api_key_dict.project_id}:{model.group}"
- if model_itpm_limit is not None:
- descriptors.append(
- RateLimitDescriptor(
- key=PROJECT_ITPM_DESCRIPTOR_KEY,
- value=descriptor_value,
- rate_limit={
- "requests_per_unit": None,
- "tokens_per_unit": model_itpm_limit,
- "window_size": self.window_size,
- },
- )
+ return tuple(
+ RateLimitDescriptor(
+ key=key,
+ value=descriptor_value,
+ rate_limit={
+ "requests_per_unit": None,
+ "tokens_per_unit": limit,
+ "window_size": self.window_size,
+ },
)
- if model_otpm_limit is not None:
- descriptors.append(
- RateLimitDescriptor(
- key=PROJECT_OTPM_DESCRIPTOR_KEY,
- value=descriptor_value,
- rate_limit={
- "requests_per_unit": None,
- "tokens_per_unit": model_otpm_limit,
- "window_size": self.window_size,
- },
- )
+ for key, limit in (
+ (PROJECT_ITPM_DESCRIPTOR_KEY, model_itpm_limit),
+ (PROJECT_OTPM_DESCRIPTOR_KEY, model_otpm_limit),
)
+ if limit is not None
+ )
def _handle_rate_limit_error(
self,
@@ -4416,6 +4490,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
kwargs: object,
model_group: str | None,
tpm_limited_tags: Set[str] = frozenset(),
+ model_groups: tuple[str, ...] = (),
) -> list[tuple[str, str]]:
"""
Enumerate every (scope_key, scope_value) pair that *might* carry a
@@ -4449,24 +4524,25 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
if user_api_key_organization_id:
targets.append(("organization", user_api_key_organization_id))
if model_group:
- if user_api_key:
- targets.append(("model_per_key", f"{user_api_key}:{model_group}"))
- if user_api_key_team_id:
- targets.append(("model_per_team", f"{user_api_key_team_id}:{model_group}"))
- if user_api_key_organization_id:
- targets.append(
- (
- "model_per_organization",
- f"{user_api_key_organization_id}:{model_group}",
+ for group in dict.fromkeys(model_groups or (model_group,)):
+ if user_api_key:
+ targets.append(("model_per_key", f"{user_api_key}:{group}"))
+ if user_api_key_team_id:
+ targets.append(("model_per_team", f"{user_api_key_team_id}:{group}"))
+ if user_api_key_organization_id:
+ targets.append(
+ (
+ "model_per_organization",
+ f"{user_api_key_organization_id}:{group}",
+ )
)
- )
- if user_api_key_project_id:
- targets.append(
- (
- "model_per_project",
- f"{user_api_key_project_id}:{model_group}",
+ if user_api_key_project_id:
+ targets.append(
+ (
+ "model_per_project",
+ f"{user_api_key_project_id}:{group}",
+ )
)
- )
if agent_id:
targets.append(("agent", agent_id))
if session_id:
@@ -4565,9 +4641,6 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
) -> list[ReservationOperation]:
"""Build Redis pipeline increment ops for TPM / parallel-request counters."""
from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs
- from litellm.proxy.common_utils.callback_utils import (
- get_model_group_from_litellm_kwargs,
- )
# Get metadata from standard_logging_object - this correctly handles both
# 'metadata' and 'litellm_metadata' fields from litellm_params
@@ -4617,6 +4690,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
# mid-flight cannot move the charge) is authoritative; otherwise fall
# back to the router's ``model_group`` (the no-reservation charge path).
reconcile_model: Final = reserved_model if reserved_model is not None else self._rate_limited_model(model_group)
+ reconcile_models: Final = self._rate_limited_models(reconcile_model)
pipeline_operations: Final[list[ReservationOperation]] = []
@@ -4636,17 +4710,24 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
kwargs=kwargs,
tpm_limited_tags=_tpm_limited_tags(stash, kwargs),
model_group=reconcile_model.group if reconcile_model is not None else None,
+ model_groups=tuple(model.group for model in reconcile_models),
)
targets.extend(
scope
for scope in sorted(reserved_scopes)
if scope[0] in ("agent", "agent_session") and scope not in targets
)
- charged_targets: Final = (
- [target for target in targets if target[0] != "model_per_team"]
- if self._key_owns_model_tpm_limit_from_request_metadata(request_metadata, reconcile_model)
- else targets
+ logging_metadata: Final = _callback_kwargs_mapping(standard_logging_metadata)
+ key_owned_team_scopes: Final = frozenset(
+ ("model_per_team", f"{logging_metadata.get('user_api_key_team_id')}:{model.group}")
+ for model in reconcile_models
+ if all(
+ self._key_owns_model_tpm_limit_from_request_metadata(request_metadata, candidate)
+ for candidate in reconcile_models
+ if candidate.group == model.group
+ )
)
+ charged_targets: Final = [target for target in targets if target not in key_owned_team_scopes]
if reserved_tokens > 0 and total_tokens < reserved_tokens:
verbose_proxy_logger.debug(
"Releasing unused TPM budget on success: reserved=%s, actual=%s, release=%s",
diff --git a/litellm/proxy/management_endpoints/internal_user_endpoints.py b/litellm/proxy/management_endpoints/internal_user_endpoints.py
index b30d6f01a8a..a76dad8d02a 100644
--- a/litellm/proxy/management_endpoints/internal_user_endpoints.py
+++ b/litellm/proxy/management_endpoints/internal_user_endpoints.py
@@ -85,7 +85,7 @@ from litellm.proxy.management_helpers.object_permission_utils import ( # noqa:
)
from litellm.proxy.management_helpers.team_roster_sync import user_tx_db
from litellm.proxy.management_helpers.utils import management_endpoint_wrapper
-from litellm.proxy.utils import handle_exception_on_proxy, hash_password
+from litellm.proxy.utils import handle_exception_on_proxy, hash_password, run_off_loop_if_fips
from litellm.redis.stores.spend_counters import (
user_counter,
)
@@ -210,7 +210,7 @@ async def _hash_password_in_dict(
if not password_prevalidated:
validate_password_policy(data["password"], general_settings)
await validate_password_not_breached(data["password"], general_settings, hibp_client)
- data["password"] = hash_password(data["password"])
+ data["password"] = await run_off_loop_if_fips(hash_password, data["password"])
data["password_reset_required"] = True
data["last_breach_check_at"] = None
diff --git a/litellm/proxy/management_endpoints/key_management_endpoints.py b/litellm/proxy/management_endpoints/key_management_endpoints.py
index d857f8d87a8..9e4d660277b 100644
--- a/litellm/proxy/management_endpoints/key_management_endpoints.py
+++ b/litellm/proxy/management_endpoints/key_management_endpoints.py
@@ -65,7 +65,6 @@ from litellm.proxy.auth.auth_utils import (
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
evict_and_broadcast,
- publish_auth_cache_invalidation,
)
from litellm.proxy.common_utils.callback_config_validation import logging_metadata_config_error
from litellm.proxy.common_utils.callback_utils import (
@@ -171,7 +170,6 @@ from litellm.proxy.utils import (
require_enterprise_license,
)
from litellm.redis.stores.spend_counters import (
- SPEND_COUNTERS_TARGET,
key_counter,
key_window_counter,
)
@@ -6809,38 +6807,6 @@ def _validate_reset_spend_value(reset_to: object, key_in_db: LiteLLM_Verificatio
return reset_to
-@with_service_target(SPEND_COUNTERS_TARGET)
-async def _set_spend_counter_with_floor_and_broadcast(counter_key: str, value: float) -> None:
- """
- Set a Redis-backed spend counter to `value`, mirror it into the short-lived
- spend_db_floor marker `_authoritative_floor_spend` reads, and broadcast both
- to every worker (LIT-3803 pattern: setting, not deleting, means a worker's
- own self-delivered broadcast still carries the reset value forward).
-
- Without the floor marker, `_authoritative_floor_spend` can re-derive a
- stale, pre-reset value from a marker another worker cached moments earlier
- and raise the just-reset counter right back up via `_repair_stale_spend_counter`.
- Without the broadcast, a worker that already cached the pre-reset key object
- or floor marker keeps enforcing against it until its own TTL expires.
- """
- from litellm.proxy.proxy_server import SPEND_DB_FLOOR_CACHE_TTL_SECONDS, spend_counter_cache, spend_counter_store
-
- try:
- await spend_counter_store.set(counter_key, value, memory_ttl=60)
- except Exception as redis_err:
- verbose_proxy_logger.warning(
- "Failed to update spend counter %s in Redis: %s. Budget checks may use stale value until counter expires.",
- counter_key,
- redis_err,
- )
-
- floor_key: Final = f"spend_db_floor:{counter_key}"
- spend_counter_cache.in_memory_cache.set_cache(key=floor_key, value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
-
- await publish_auth_cache_invalidation(cache_key=counter_key, new_value=value, ttl=60)
- await publish_auth_cache_invalidation(cache_key=floor_key, new_value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
-
-
def _budget_limit_windows(budget_limits: Sequence[object] | str | None) -> tuple[Mapping[str, object], ...]:
"""Coerce a key's stored `budget_limits` into a tuple of plain window dicts.
@@ -6925,6 +6891,8 @@ async def _reset_key_budget_windows(
pre-reset boundary, re-sum the unchanged historical spend, and put the
counter right back where it was before the write ever landed.
"""
+ from litellm.proxy.proxy_server import set_spend_counter_with_floor_and_broadcast
+
windows: Final = _budget_limit_windows(budget_limits)
if not windows:
return
@@ -6943,7 +6911,7 @@ async def _reset_key_budget_windows(
duration = window.get("budget_duration")
if isinstance(duration, str) and duration:
counter_key = key_window_counter(hashed_api_key, duration)
- await _set_spend_counter_with_floor_and_broadcast(counter_key=counter_key, value=0.0)
+ await set_spend_counter_with_floor_and_broadcast(counter_key=counter_key, value=0.0)
@router.post(
@@ -6966,6 +6934,7 @@ async def reset_key_spend_fn(
hash_token,
prisma_client,
proxy_logging_obj,
+ set_spend_counter_with_floor_and_broadcast,
user_api_key_cache,
)
@@ -7017,7 +6986,7 @@ async def reset_key_spend_fn(
# amount for every enforcement check immediately instead of the stale
# pre-reset value.
_counter_key: Final = key_counter(hashed_api_key)
- await _set_spend_counter_with_floor_and_broadcast(counter_key=_counter_key, value=reset_to)
+ await set_spend_counter_with_floor_and_broadcast(counter_key=_counter_key, value=reset_to)
await _reset_key_budget_windows(
prisma_client=prisma_client,
hashed_api_key=hashed_api_key,
diff --git a/litellm/proxy/management_endpoints/password_endpoints.py b/litellm/proxy/management_endpoints/password_endpoints.py
index 16f3e7dfcfb..21172e30639 100644
--- a/litellm/proxy/management_endpoints/password_endpoints.py
+++ b/litellm/proxy/management_endpoints/password_endpoints.py
@@ -33,7 +33,7 @@ from litellm.proxy.auth.password_policy import (
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.management_endpoints.session_endpoints import revoke_ui_session_keys
from litellm.proxy.management_helpers.audit_logs import create_object_audit_log
-from litellm.proxy.utils import hash_password, verify_password
+from litellm.proxy.utils import hash_password, run_off_loop_if_fips, verify_password
from litellm.repositories.prisma_protocols import TableActions
from litellm.repositories.user_repository import UserRepository
@@ -129,7 +129,7 @@ async def change_password(
),
)
- if not verify_password(data.current_password, stored_password):
+ if not await run_off_loop_if_fips(verify_password, data.current_password, stored_password):
raise HTTPException(status_code=400, detail=_error_detail("Current password is incorrect."))
if data.new_password == data.current_password:
@@ -142,7 +142,7 @@ async def change_password(
await validate_password_not_breached(data.new_password, general_settings, hibp_client)
password_update: Final[prisma_types.LiteLLM_UserTableUpdateInput] = {
- "password": hash_password(data.new_password),
+ "password": await run_off_loop_if_fips(hash_password, data.new_password),
"password_reset_required": False,
"last_breach_check_at": None,
}
diff --git a/litellm/proxy/management_endpoints/tag_management_endpoints.py b/litellm/proxy/management_endpoints/tag_management_endpoints.py
index 3ec92165a84..0b7a7ca43cd 100644
--- a/litellm/proxy/management_endpoints/tag_management_endpoints.py
+++ b/litellm/proxy/management_endpoints/tag_management_endpoints.py
@@ -30,6 +30,7 @@ from litellm.proxy.management_endpoints.common_daily_activity import (
SpendAnalyticsPaginatedResponse,
get_daily_activity,
)
+from litellm.proxy.management_endpoints.common_utils import validate_finite_spend
from litellm.proxy.management_helpers.utils import handle_budget_for_entity
from litellm.redis.stores.spend_counters import (
tag_counter,
@@ -415,6 +416,7 @@ async def update_tag(
Parameters:
- name: str - The name of the tag to update
+ - spend: Optional[float] - Updated spend for the tag
- description: Optional[str] - Updated description
- models: List[str] - Updated list of allowed LLM models
- budget_id: Optional[str] - The id for a budget to associate with the tag
@@ -428,11 +430,13 @@ async def update_tag(
- model_max_budget: Optional[dict] - Max budget for a specific model
- budget_duration: Optional[str] - Frequency of resetting tag budget
"""
- from litellm.proxy.proxy_server import prisma_client
+ from litellm.proxy.proxy_server import prisma_client, set_spend_counter_with_floor_and_broadcast
if prisma_client is None:
raise HTTPException(status_code=500, detail="Database not connected")
+ validate_finite_spend(tag.spend)
+
try:
# Check if tag exists
existing_tag: Final = await _table(TagRepository(prisma_client)).find_unique(where={"tag_name": tag.name})
@@ -459,12 +463,10 @@ async def update_tag(
"description": tag.description,
"models": tag.models or [],
"model_info": json.dumps(model_info),
+ **({"spend": tag.spend} if tag.spend is not None else {}),
+ **({"budget_id": budget_id} if budget_id != existing_tag.budget_id else {}),
}
- # Add budget_id if it changed
- if budget_id != existing_tag.budget_id:
- update_data["budget_id"] = budget_id
-
# Update tag in database
updated_tag_record: Final = await _table(TagRepository(prisma_client)).update(
where={"tag_name": tag.name},
@@ -473,6 +475,9 @@ async def update_tag(
await _evict_tag_cache_keys((tag_cache_key(tag.name),))
+ if tag.spend is not None:
+ await set_spend_counter_with_floor_and_broadcast(counter_key=f"spend:tag:{tag.name}", value=tag.spend)
+
# Build response
tag_config: Final = TagConfig(
name=updated_tag_record.tag_name,
diff --git a/litellm/proxy/management_helpers/utils.py b/litellm/proxy/management_helpers/utils.py
index ea19befccc6..1c4eada67fb 100644
--- a/litellm/proxy/management_helpers/utils.py
+++ b/litellm/proxy/management_helpers/utils.py
@@ -217,7 +217,7 @@ async def handle_budget_for_entity(
)
# Get all budget field names
- budget_params: Final = LiteLLM_BudgetTable.model_fields.keys()
+ budget_params: Final = LiteLLM_BudgetTable.model_fields.keys() - {"budget_id"}
# Extract budget fields from data
_json_data: Final = _BUDGET_DATA_MAPPING.validate_python(
diff --git a/litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py b/litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py
index 3087faa9f67..d321f5b5861 100644
--- a/litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py
+++ b/litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py
@@ -1,15 +1,94 @@
+import asyncio
+import hashlib
import hmac
+import json
+import re
+from collections.abc import Iterator, Mapping
+from dataclasses import dataclass
from types import MappingProxyType
-from typing import Annotated, Final, Literal
+from typing import TYPE_CHECKING, Annotated, Final, Literal, Protocol
+import httpx
from fastapi import APIRouter, Depends, HTTPException, Request, Response
+from fastapi.responses import StreamingResponse
+from pydantic import TypeAdapter, ValidationError
+from starlette.types import Message
+import litellm
+from litellm._internal_context import with_service_target
+from litellm._logging import verbose_proxy_logger
+from litellm.exceptions import BadRequestError
+from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, normalize_drop_params
+from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.llms.chatgpt.common_utils import CHATGPT_API_BASE
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.llms.custom_httpx.http_handler import (
+ get_async_httpx_client, # pyright: ignore[reportUnknownVariableType] # shared client factory has untyped params
+)
+from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
+from litellm.proxy.auth.auth_checks import check_model_access_helper, check_tools_allowlist
+from litellm.proxy.auth.handle_jwt import JWTAuthManager
+from litellm.proxy.auth.route_checks import RouteChecks
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
+from litellm.proxy.common_utils.codex_model_catalog import (
+ CodexCatalogRow,
+ bundled_codex_models,
+ codex_catalog_etag,
+ codex_entry_names_default_reasoning_level,
+ merged_codex_catalog,
+ upstream_codex_slugs,
+)
+from litellm.proxy.common_utils.http_parsing_utils import (
+ get_client_requested_model,
+ read_request_body, # pyright: ignore[reportUnknownVariableType] # returns a bare dict, validated at the call
+)
+from litellm.proxy.litellm_pre_call_utils import should_auto_drop_params_for_agentic_cli
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import pass_through_request
+from litellm.proxy.response_api_endpoints.endpoints import responses_api
+from litellm.responses.additional_tools import with_additional_tools_hoisted
+from litellm.responses.litellm_completion_transformation.reasoning_items import is_other_provider_reasoning_item
+from litellm.responses.utils import ResponsesAPIRequestUtils
+from litellm.types.llms.custom_http import httpxSpecialProvider
+from litellm.types.router import Deployment
+from litellm.types.utils import LlmProviders
+from litellm.utils import ProviderConfigManager, supports_none_reasoning_effort
+
+if TYPE_CHECKING:
+ from litellm.router import Router
router: Final = APIRouter()
+_CHATGPT_PROVIDER: Final = "chatgpt"
+_CHATGPT_CREDENTIAL_HEADERS: Final = frozenset({b"authorization", b"chatgpt-account-id"})
+_MERGED_CATALOG_DROPPED_HEADERS: Final = frozenset({"etag", "content-length", "content-type"})
+_ETAG_HEADER: Final = "etag"
+_MODELS_ETAG_HEADER: Final = "x-models-etag"
+_ENCRYPTED_CONTENT_AFFINITY_FLAG: Final = "encrypted_content_affinity_enabled"
+_LITELLM_RESPONSES_PATH: Final = "/v1/responses"
+_CODEX_NO_DEFAULT_EFFORT: Final = "none"
+_NONE_EFFORT_SUPPORT_FLAG: Final = "supports_none_reasoning_effort"
+_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
+_INPUT_ITEMS: Final = TypeAdapter(tuple[object, ...])
+_CACHED_SLUGS: Final = TypeAdapter(frozenset[str])
+_CACHED_MODELS_ETAG: Final = TypeAdapter(dict[Literal["etag"], str])
+_MODEL_NAMES: Final = TypeAdapter(list[str])
+_HEADER_DERIVED_STATE: Final = frozenset({"_cached_headers"})
+_ACCOUNT_CATALOG_HEADERS: Final = frozenset({"authorization", "chatgpt-account-id", "originator", "user-agent"})
+_CODEX_USER_AGENT_PRODUCT: Final = re.compile(r"([^/]+)/(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.+-]+)?)")
+_ACCOUNT_CATALOG_TIMEOUT_SECONDS: Final = 5.0
+_SHARED_CACHE_TTL_SECONDS: Final = 3600
+_ACCOUNT_SLUGS_UNAVAILABLE: Final = "unavailable"
+_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS: Final = 60
+_ACCOUNT_SLUGS_CACHE_PREFIX: Final = "codex_account_slugs:"
+_ACCOUNT_SLUGS_TARGET: Final = "codex_account_slugs"
+_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX: Final = "codex_account_slugs_fetches:"
+_ACCOUNT_SLUGS_FETCHES_PER_CALLER: Final = 5
+_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS: Final = 60
+_MODELS_ETAG_CACHE_PREFIX: Final = "codex_models_etag:"
+_MODELS_ETAG_TTL_SECONDS: Final = 300
+_OPENAI_API_PROVIDERS: Final = frozenset({LlmProviders.OPENAI.value, LlmProviders.AZURE.value})
+_CODEX_TOOL_PROVIDERS: Final = frozenset(
+ {*_OPENAI_API_PROVIDERS, LlmProviders.FIREWORKS_AI.value, LlmProviders.BEDROCK_MANTLE.value}
+)
+_CHAT_COMPLETIONS_BRIDGE: Final = "use_chat_completions_api"
_HEADERS: Final = frozenset(
{
"authorization",
@@ -29,10 +108,637 @@ _HEADERS: Final = frozenset(
)
-@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"])
+def _is_chatgpt(deployment: Deployment) -> bool:
+ return (
+ deployment.litellm_params.custom_llm_provider == _CHATGPT_PROVIDER
+ or deployment.litellm_params.model.partition("/")[0] == _CHATGPT_PROVIDER
+ )
+
+
+def _serves_by_name(deployment: Deployment, names: frozenset[str]) -> bool:
+ return not _is_chatgpt(deployment) and (
+ deployment.model_name in names or deployment.model_info.team_public_model_name in names
+ )
+
+
+def _candidate_deployments(model: str, llm_router: "Router", team_id: str | None) -> tuple[Deployment, ...]:
+ registry: Final = llm_router.registry()
+ candidates: Final = tuple(
+ registry.deployment(model_id) for model_id in registry.names(model, kind="route_candidates", team_id=team_id)
+ )
+ return tuple(deployment for deployment in candidates if deployment is not None)
+
+
+def routes_to_litellm(model: object, llm_router: "Router | None", team_id: str | None) -> bool:
+ if (
+ not isinstance(model, str)
+ or llm_router is None
+ or model in bundled_codex_models()
+ or f"{_CHATGPT_PROVIDER}/{model}" in litellm.model_cost # pyright: ignore[reportUnknownMemberType] # registry is dynamically typed
+ ):
+ return False
+ deployments: Final = _candidate_deployments(model, llm_router, team_id)
+ names: Final = frozenset({model, llm_router.registry().resolve(model, kind="alias") or model})
+ return bool(deployments) and all(_serves_by_name(deployment, names) for deployment in deployments)
+
+
+def _key_and_team_allow(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> bool:
+ allowed_lists: Final = (
+ _MODEL_NAMES.validate_python(user_api_key_dict.models),
+ _MODEL_NAMES.validate_python(user_api_key_dict.team_models),
+ )
+ return all(
+ check_model_access_helper(model, llm_router, allowed, team_id=user_api_key_dict.team_id)
+ for allowed in allowed_lists
+ )
+
+
+def native_chatgpt_model(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> str | None:
+ if model.startswith(f"{_CHATGPT_PROVIDER}/") or _key_and_team_allow(model, llm_router, user_api_key_dict):
+ return None
+ chatgpt_model: Final = f"{_CHATGPT_PROVIDER}/{model}"
+ deployments: Final = _candidate_deployments(chatgpt_model, llm_router, user_api_key_dict.team_id)
+ if not deployments or not all(_is_chatgpt(deployment) for deployment in deployments):
+ return None
+ return chatgpt_model if _key_and_team_allow(chatgpt_model, llm_router, user_api_key_dict) else None
+
+
+class _SharedCache(Protocol):
+ async def async_get_cache(self, key: str) -> object: ...
+
+ async def async_set_cache(self, key: str, value: object, *, ttl: int) -> None: ...
+
+ async def async_increment_cache(self, key: str, value: float, *, ttl: int) -> float | None: ...
+
+
+def codex_shared_cache() -> _SharedCache:
+ from litellm.proxy.proxy_server import proxy_logging_obj
+
+ return proxy_logging_obj.internal_usage_cache.dual_cache
+
+
+def _key_may_call_litellm_responses(user_api_key_dict: UserAPIKeyAuth) -> bool:
+ if not RouteChecks.is_allowed_on_proxy(_LITELLM_RESPONSES_PATH):
+ return False
+ try:
+ _ = RouteChecks.should_call_route(route=_LITELLM_RESPONSES_PATH, valid_token=user_api_key_dict)
+ if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
+ RouteChecks.custom_admin_only_route_check(route=_LITELLM_RESPONSES_PATH)
+ except HTTPException:
+ return False
+ return True
+
+
+async def _jwt_policy_admits_litellm_responses(
+ gateway_token: str, model: str, request_headers: Mapping[str, str]
+) -> bool:
+ from litellm.proxy.proxy_server import (
+ general_settings_view,
+ jwt_handler,
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
+ )
+
+ try:
+ _ = await JWTAuthManager.auth_builder(
+ api_key=gateway_token,
+ jwt_handler=jwt_handler,
+ request_data={"model": model},
+ general_settings=dict(general_settings_view()),
+ route=_LITELLM_RESPONSES_PATH,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=None,
+ proxy_logging_obj=proxy_logging_obj,
+ request_headers=request_headers,
+ request_method="POST",
+ )
+ except Exception: # noqa: BLE001 # any JWT policy refusal keeps the caller on ChatGPT's models
+ return False
+ return True
+
+
+@dataclass(frozen=True, slots=True)
+class _LitellmResponsesAccess:
+ key_admits: bool
+ jwt_token: str | None
+ request_headers: Mapping[str, str]
+
+ async def admits(self, model: str) -> bool:
+ if not self.key_admits:
+ return False
+ if self.jwt_token is None:
+ return True
+ return await _jwt_policy_admits_litellm_responses(self.jwt_token, model, self.request_headers)
+
+
+def _litellm_responses_access(user_api_key_dict: UserAPIKeyAuth, request: Request) -> _LitellmResponsesAccess:
+ standard_jwt: Final = bool(user_api_key_dict.jwt_claims) and not user_api_key_dict.token
+ return _LitellmResponsesAccess(
+ key_admits=_key_may_call_litellm_responses(user_api_key_dict),
+ jwt_token=_gateway_token(request) if standard_jwt else None,
+ request_headers=request.headers,
+ )
+
+
+def _account_slugs_cache_key(headers: Mapping[str, str]) -> str | None:
+ account_id: Final = headers.get("chatgpt-account-id")
+ if not account_id:
+ return None
+ caller: Final = json.dumps([account_id, headers.get("authorization", "")])
+ return f"{_ACCOUNT_SLUGS_CACHE_PREFIX}{hashlib.sha256(caller.encode()).hexdigest()}"
+
+
+def _cached_slugs(cached: object) -> frozenset[str] | None:
+ if cached is None:
+ return None
+ try:
+ return _CACHED_SLUGS.validate_python(cached)
+ except ValidationError:
+ return None
+
+
+async def _remember_account_slugs(cache_key: str, slugs: frozenset[str] | None, cache: _SharedCache) -> None:
+ if slugs is None:
+ await cache.async_set_cache(cache_key, _ACCOUNT_SLUGS_UNAVAILABLE, ttl=_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS)
+ return
+ await cache.async_set_cache(cache_key, tuple(sorted(slugs)), ttl=_SHARED_CACHE_TTL_SECONDS)
+
+
+def _codex_client_version(headers: Mapping[str, str]) -> str | None:
+ product: Final = _CODEX_USER_AGENT_PRODUCT.fullmatch(headers.get("user-agent", "").partition(" ")[0])
+ if product is None or product.group(1) != headers.get("originator"):
+ return None
+ return product.group(2)
+
+
+async def _fetched_account_slugs(headers: Mapping[str, str], client_version: str) -> frozenset[str] | None:
+ client: Final = get_async_httpx_client(
+ llm_provider=httpxSpecialProvider.PassThroughEndpoint,
+ params={"timeout": _ACCOUNT_CATALOG_TIMEOUT_SECONDS},
+ )
+ try:
+ response: Final = await client.client.get(
+ f"{CHATGPT_API_BASE}/models",
+ params={"client_version": client_version},
+ headers={name: value for name, value in headers.items() if name in _ACCOUNT_CATALOG_HEADERS},
+ )
+ except httpx.HTTPError:
+ return None
+ return upstream_codex_slugs(response.content) if response.status_code == 200 else None
+
+
+async def _admits_account_slugs_fetch(caller: str, cache: _SharedCache) -> bool:
+ fetches: Final = await cache.async_increment_cache(
+ f"{_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX}{caller}", 1, ttl=_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS
+ )
+ return fetches is not None and fetches <= _ACCOUNT_SLUGS_FETCHES_PER_CALLER
+
+
+@with_service_target(_ACCOUNT_SLUGS_TARGET)
+async def _account_slugs(headers: Mapping[str, str], caller: str, cache: _SharedCache) -> frozenset[str] | None:
+ cache_key: Final = _account_slugs_cache_key(headers)
+ if cache_key is None:
+ return None
+ cached_value: Final = await cache.async_get_cache(cache_key)
+ if cached_value == _ACCOUNT_SLUGS_UNAVAILABLE:
+ return None
+ cached: Final = _cached_slugs(cached_value)
+ if cached is not None:
+ return cached
+ client_version: Final = _codex_client_version(headers)
+ if client_version is None or not await _admits_account_slugs_fetch(caller, cache):
+ return None
+ fetched: Final = await _fetched_account_slugs(headers, client_version)
+ await _remember_account_slugs(cache_key, fetched, cache)
+ return fetched
+
+
+async def _serves_through_litellm(
+ model: object,
+ llm_router: "Router | None",
+ user_api_key_dict: UserAPIKeyAuth,
+ access: _LitellmResponsesAccess,
+ caller: str,
+ headers: Mapping[str, str],
+ cache: _SharedCache,
+) -> bool:
+ if not isinstance(model, str) or not routes_to_litellm(model, llm_router, user_api_key_dict.team_id):
+ return False
+ if not await access.admits(model):
+ return False
+ account_slugs: Final = await _account_slugs(headers, caller, cache)
+ return account_slugs is not None and model not in account_slugs
+
+
+def _gateway_token(request: Request) -> str:
+ gateway_parts: Final = request.headers.get("x-litellm-api-key", "").split()
+ return (
+ gateway_parts[-1]
+ if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
+ else ""
+ )
+
+
+def _chatgpt_bearer(request: Request) -> str:
+ scheme, _, token = request.headers.get("authorization", "").partition(" ")
+ return token.strip() if scheme.lower() == "bearer" else ""
+
+
+def _forwarded_headers(request: Request) -> Mapping[str, str]:
+ return MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
+
+
+async def native_turn_serves_through_litellm(
+ model: str, llm_router: "Router", request: Request, user_api_key_dict: UserAPIKeyAuth
+) -> bool:
+ chatgpt_bearer: Final = _chatgpt_bearer(request)
+ gateway_token: Final = _gateway_token(request)
+ if not chatgpt_bearer or not gateway_token or hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
+ return False
+ return await _serves_through_litellm(
+ model,
+ llm_router,
+ user_api_key_dict,
+ _litellm_responses_access(user_api_key_dict, request),
+ _models_etag_caller(user_api_key_dict, gateway_token),
+ _forwarded_headers(request),
+ codex_shared_cache(),
+ )
+
+
+def _json_object(value: object) -> Mapping[str, object]:
+ try:
+ return _JSON_OBJECT.validate_python(value)
+ except ValidationError:
+ return MappingProxyType({})
+
+
+def _scope_state(request: Request) -> Mapping[str, object]:
+ return _json_object(request.scope.get("state"))
+
+
+def _as_litellm_responses_request(request: Request, body: bytes, parsed_body: Mapping[str, object]) -> Request:
+ replayed: Final[Iterator[Message]] = iter(({"type": "http.request", "body": body, "more_body": False},))
+
+ async def receive() -> Message:
+ return next(replayed, None) or await request.receive()
+
+ return Request(
+ {
+ **request.scope,
+ "path": _LITELLM_RESPONSES_PATH,
+ "raw_path": _LITELLM_RESPONSES_PATH.encode(),
+ "headers": [
+ (name, value) for name, value in request.headers.raw if name not in _CHATGPT_CREDENTIAL_HEADERS
+ ],
+ "parsed_body": (tuple(parsed_body), parsed_body),
+ "state": {
+ name: value for name, value in _scope_state(request).items() if name not in _HEADER_DERIVED_STATE
+ },
+ },
+ receive,
+ )
+
+
+def _drops_params(body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None) -> bool:
+ deployments: Final = _candidate_deployments(model, llm_router, team_id)
+ return drops_unsupported_params(body.get("drop_params")) or (
+ bool(deployments)
+ and all(normalize_drop_params(deployment.litellm_params.drop_params) is True for deployment in deployments)
+ )
+
+
+def _deployment_supports_no_effort(deployment: Deployment) -> bool:
+ configured: Final = _json_object(deployment.model_info.model_extra).get(_NONE_EFFORT_SUPPORT_FLAG)
+ if isinstance(configured, bool):
+ return configured
+ return supports_none_reasoning_effort(
+ model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
+ )
+
+
+def _supports_no_effort(model: str, llm_router: "Router", team_id: str | None) -> bool:
+ deployments: Final = _candidate_deployments(model, llm_router, team_id)
+ return bool(deployments) and all(_deployment_supports_no_effort(deployment) for deployment in deployments)
+
+
+def _with_provider_default_reasoning(
+ body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, client_version: str | None
+) -> Mapping[str, object]:
+ model: Final = body.get("model")
+ try:
+ reasoning: Final = _JSON_OBJECT.validate_python(body.get("reasoning"))
+ except ValidationError:
+ return body
+ if (
+ reasoning.get("effort") != _CODEX_NO_DEFAULT_EFFORT
+ or not isinstance(model, str)
+ or llm_router is None
+ or not _drops_params(body, model, llm_router, team_id)
+ or codex_entry_names_default_reasoning_level(model, llm_router, team_id, client_version)
+ or _supports_no_effort(model, llm_router, team_id)
+ ):
+ return body
+ return {**body, "reasoning": {key: value for key, value in reasoning.items() if key != "effort"}}
+
+
+def _with_wrapped_reasoning(body: Mapping[str, object]) -> Mapping[str, object]:
+ return {
+ **body,
+ "litellm_metadata": {**_json_object(body.get("litellm_metadata")), _ENCRYPTED_CONTENT_AFFINITY_FLAG: True},
+ }
+
+
+def _with_reasoning_the_targets_can_decrypt(
+ body: Mapping[str, object], model: str, llm_router: "Router", deployments: tuple[Deployment, ...]
+) -> Mapping[str, object]:
+ try:
+ items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
+ except ValidationError:
+ return body
+ model_group: Final = llm_router.registry().resolve(model, kind="alias") or model
+ kept: Final = llm_router.registry().decryptable_reasoning(model_group, items, deployments)
+ if body.get("store") is True:
+ return {**body, "input": list(kept)}
+ resolvable: Final = (_without_unpersisted_reasoning_reference(item) for item in kept)
+ return {**body, "input": [item for item in resolvable if item is not None]}
+
+
+def _without_unpersisted_reasoning_reference(item: object) -> object | None:
+ if not _is_unpersisted_reasoning_reference(item):
+ return item
+ return ResponsesAPIRequestUtils.readable_reasoning(_JSON_OBJECT.validate_python(item))
+
+
+def _model_and_provider(deployment: Deployment) -> tuple[str, str] | None:
+ try:
+ model, provider, _, _ = get_llm_provider(
+ model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
+ )
+ except BadRequestError:
+ return None
+ return model, provider
+
+
+def _is_openai_api(deployment: Deployment) -> bool:
+ resolved: Final = _model_and_provider(deployment)
+ return resolved is not None and resolved[1] in _OPENAI_API_PROVIDERS
+
+
+def _takes_codex_tools(deployment: Deployment) -> bool:
+ resolved: Final = _model_and_provider(deployment)
+ if resolved is None:
+ return False
+ model, provider = resolved
+ return (
+ provider in _CODEX_TOOL_PROVIDERS
+ or ProviderConfigManager.get_provider_responses_api_config(
+ provider=provider, model=model, api_base=deployment.litellm_params.api_base
+ )
+ is None
+ )
+
+
+def _with_tools_the_targets_accept(
+ body: Mapping[str, object], deployments: tuple[Deployment, ...], drop_params: bool
+) -> Mapping[str, object]:
+ if all(_is_openai_api(deployment) for deployment in deployments):
+ return body
+ if all(_takes_codex_tools(deployment) for deployment in deployments):
+ return with_additional_tools_hoisted(body, drop_params=drop_params)
+ return {**body, _CHAT_COMPLETIONS_BRIDGE: True}
+
+
+def _drops_params_for_client(
+ body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None, user_agent: str
+) -> bool:
+ from litellm.proxy.proxy_server import proxy_config
+
+ return _drops_params(body, model, llm_router, team_id) or should_auto_drop_params_for_agentic_cli(
+ user_agent, "drop_params" in body, proxy_config
+ )
+
+
+def _litellm_body(
+ body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, headers: Mapping[str, str]
+) -> Mapping[str, object]:
+ wrapped: Final = _with_wrapped_reasoning(
+ _with_provider_default_reasoning(body, llm_router, team_id, _codex_client_version(headers))
+ )
+ model: Final = body.get("model")
+ if not isinstance(model, str) or llm_router is None:
+ return wrapped
+ deployments: Final = _candidate_deployments(model, llm_router, team_id)
+ return _with_tools_the_targets_accept(
+ _with_reasoning_the_targets_can_decrypt(wrapped, model, llm_router, deployments),
+ deployments,
+ _drops_params_for_client(body, model, llm_router, team_id, headers.get("user-agent", "")),
+ )
+
+
+def _is_unpersisted_reasoning_reference(item: object) -> bool:
+ try:
+ fields: Final = _JSON_OBJECT.validate_python(item)
+ except ValidationError:
+ return False
+ item_id: Final = fields.get("id")
+ encrypted_content: Final = fields.get("encrypted_content")
+ return (
+ fields.get("type") == "reasoning"
+ and isinstance(item_id, str)
+ and bool(item_id)
+ and not (isinstance(encrypted_content, str) and bool(encrypted_content))
+ )
+
+
+def _rejected_by_chatgpt(item: object, stored: bool) -> bool:
+ return is_other_provider_reasoning_item(item) or (not stored and _is_unpersisted_reasoning_reference(item))
+
+
+def _input_chatgpt_accepts(body: Mapping[str, object]) -> tuple[object, ...] | None:
+ try:
+ items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
+ except ValidationError:
+ return None
+ stored: Final = body.get("store") is True
+ kept: Final = tuple(item for item in items if not _rejected_by_chatgpt(item, stored))
+ return kept if len(kept) < len(items) else None
+
+
+def _chatgpt_body(body: Mapping[str, object], client_model: str | None) -> Mapping[str, object] | None:
+ kept_input: Final = _input_chatgpt_accepts(body)
+ restored_model: Final = client_model is not None and body.get("model") == f"{_CHATGPT_PROVIDER}/{client_model}"
+ if kept_input is None and not restored_model:
+ return None
+ return {
+ **body,
+ **({"input": list(kept_input)} if kept_input is not None else {}),
+ **({"model": client_model} if restored_model else {}),
+ }
+
+
+async def _litellm_catalog_rows(
+ user_api_key_dict: UserAPIKeyAuth, access: _LitellmResponsesAccess, llm_router: "Router | None"
+) -> tuple[CodexCatalogRow, ...]:
+ from litellm.proxy.proxy_server import codex_catalog_rows_for_key
+
+ if llm_router is None or not access.key_admits:
+ return ()
+ try:
+ rows: Final = await codex_catalog_rows_for_key(user_api_key_dict)
+ except Exception as error: # noqa: BLE001 # a failed listing must never fail the ChatGPT answer it extends
+ verbose_proxy_logger.warning(
+ "Codex catalog keeps only ChatGPT's models: listing the key's LiteLLM models raised %s",
+ type(error).__name__,
+ )
+ return ()
+ routed: Final = tuple(row for row in rows if routes_to_litellm(row.id, llm_router, user_api_key_dict.team_id))
+ admitted: Final = await asyncio.gather(*(access.admits(row.id) for row in routed))
+ return tuple(row for row, row_admitted in zip(routed, admitted, strict=True) if row_admitted)
+
+
+def _models_etag_caller(user_api_key_dict: UserAPIKeyAuth, gateway_token: str) -> str:
+ return user_api_key_dict.token or hashlib.sha256(gateway_token.encode()).hexdigest()
+
+
+def _models_etag_cache_key(caller: str, upstream_etag: str) -> str:
+ return f"{_MODELS_ETAG_CACHE_PREFIX}{hashlib.sha256(json.dumps([caller, upstream_etag]).encode()).hexdigest()}"
+
+
+async def _remembered_models_etag(
+ caller: str, upstream_etag: str, rows: tuple[CodexCatalogRow, ...], cache: _SharedCache
+) -> str:
+ etag: Final = codex_catalog_etag(upstream_etag, rows) if rows else upstream_etag
+ await cache.async_set_cache(
+ _models_etag_cache_key(caller, upstream_etag), {"etag": etag}, ttl=_MODELS_ETAG_TTL_SECONDS
+ )
+ return etag
+
+
+async def _cached_models_etag(caller: str, upstream_etag: str, cache: _SharedCache) -> str | None:
+ try:
+ return _CACHED_MODELS_ETAG.validate_python(
+ await cache.async_get_cache(_models_etag_cache_key(caller, upstream_etag))
+ )["etag"]
+ except ValidationError:
+ return None
+
+
+async def _key_models_etag(
+ upstream_etag: str,
+ user_api_key_dict: UserAPIKeyAuth,
+ access: _LitellmResponsesAccess,
+ caller: str,
+ llm_router: "Router | None",
+ cache: _SharedCache,
+) -> str:
+ cached: Final = await _cached_models_etag(caller, upstream_etag, cache)
+ if cached is not None:
+ return cached
+ rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
+ return await _remembered_models_etag(caller, upstream_etag, rows, cache)
+
+
+def _with_etag(response: Response, content: bytes | str, etag: str | None) -> Response:
+ kept_headers: Final = {
+ name: value for name, value in response.headers.items() if name not in _MERGED_CATALOG_DROPPED_HEADERS
+ }
+ return Response(
+ content=content,
+ status_code=response.status_code,
+ headers={**kept_headers, **({_ETAG_HEADER: etag} if etag is not None else {})},
+ media_type="application/json",
+ )
+
+
+async def _catalog(
+ request: Request,
+ user_api_key_dict: UserAPIKeyAuth,
+ access: _LitellmResponsesAccess,
+ caller: str,
+ headers: Mapping[str, str],
+ cache: _SharedCache,
+) -> Response:
+ from litellm.proxy.proxy_server import llm_router
+
+ upstream: Final = await pass_through_request(
+ request=request,
+ target=f"{CHATGPT_API_BASE}/models",
+ custom_headers=dict(headers),
+ user_api_key_dict=user_api_key_dict,
+ custom_llm_provider=_CHATGPT_PROVIDER,
+ upstream_headers_filter=lambda _: headers,
+ )
+ if upstream.status_code != 200 or isinstance(upstream, StreamingResponse):
+ return upstream
+ upstream_body: Final = bytes(upstream.body)
+ slugs: Final = upstream_codex_slugs(upstream_body)
+ if slugs is None:
+ return upstream
+ account_slugs_key: Final = _account_slugs_cache_key(headers)
+ if account_slugs_key is not None:
+ await _remember_account_slugs(account_slugs_key, slugs, cache)
+ rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
+ upstream_etag: Final = upstream.headers.get(_ETAG_HEADER)
+ etag: Final = None if upstream_etag is None else await _remembered_models_etag(caller, upstream_etag, rows, cache)
+ if not rows:
+ return upstream
+ merged: Final = await asyncio.to_thread(
+ merged_codex_catalog, upstream_body, rows, client_version=request.query_params.get("client_version")
+ )
+ return _with_etag(upstream, merged if merged is not None else upstream_body, etag)
+
+
+async def _responses(
+ request: Request,
+ fastapi_response: Response,
+ user_api_key_dict: UserAPIKeyAuth,
+ access: _LitellmResponsesAccess,
+ caller: str,
+ headers: Mapping[str, str],
+ cache: _SharedCache,
+) -> object:
+ from litellm.proxy.proxy_server import llm_router
+
+ body: Final = _JSON_OBJECT.validate_python(await read_request_body(request))
+ if await _serves_through_litellm(body.get("model"), llm_router, user_api_key_dict, access, caller, headers, cache):
+ litellm_body: Final = _litellm_body(body, llm_router, user_api_key_dict.team_id, headers)
+ await check_tools_allowlist(
+ request_body=dict(litellm_body),
+ valid_token=user_api_key_dict,
+ team_object=None,
+ route=_LITELLM_RESPONSES_PATH,
+ )
+ return await responses_api(
+ _as_litellm_responses_request(request, json.dumps(litellm_body).encode(), litellm_body),
+ fastapi_response,
+ user_api_key_dict.model_copy(update={"request_route": _LITELLM_RESPONSES_PATH}),
+ )
+ chatgpt_body: Final = _chatgpt_body(body, get_client_requested_model(request))
+ upstream: Final = await pass_through_request(
+ request=request,
+ target=f"{CHATGPT_API_BASE}/responses",
+ custom_headers=dict(headers),
+ user_api_key_dict=user_api_key_dict,
+ custom_body=dict(chatgpt_body) if chatgpt_body is not None else None,
+ custom_llm_provider=_CHATGPT_PROVIDER,
+ upstream_headers_filter=lambda _: headers,
+ )
+ upstream_models_etag: Final = upstream.headers.get(_MODELS_ETAG_HEADER)
+ if upstream_models_etag is not None:
+ upstream.headers[_MODELS_ETAG_HEADER] = await _key_models_etag(
+ upstream_models_etag, user_api_key_dict, access, caller, llm_router, cache
+ )
+ return upstream
+
+
+@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"], response_model=None)
@router.post(
"/backend-api/codex/{endpoint}",
tags=["ChatGPT Pass-through", "pass-through"],
+ response_model=None,
openapi_extra={
"requestBody": {
"required": True,
@@ -41,7 +747,7 @@ _HEADERS: Final = frozenset(
},
responses={
200: {
- "description": "Unmodified Codex response or server-sent events",
+ "description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events",
"content": {
"application/json": {"schema": {"type": "object", "additionalProperties": True}},
"text/event-stream": {"schema": {"type": "string"}},
@@ -52,36 +758,29 @@ _HEADERS: Final = frozenset(
async def codex_passthrough_route(
endpoint: Literal["models", "responses"],
request: Request,
+ fastapi_response: Response,
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
-) -> Response:
- """Forward a client's ChatGPT subscription with separate LiteLLM authentication."""
+ cache: Annotated[_SharedCache, Depends(codex_shared_cache)],
+) -> object:
+ """Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
+ key's other chat models through LiteLLM."""
expected_method: Final = "GET" if endpoint == "models" else "POST"
if request.method != expected_method:
raise HTTPException(status_code=405, detail="Method not allowed", headers={"Allow": expected_method})
- authorization: Final = request.headers.get("authorization", "")
- scheme, _, token = authorization.partition(" ")
- gateway_key: Final = request.headers.get("x-litellm-api-key", "")
- gateway_parts: Final = gateway_key.split()
- gateway_token: Final = (
- gateway_parts[-1]
- if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
- else ""
- )
- if scheme.lower() != "bearer" or not token.strip() or not gateway_token:
+ chatgpt_bearer: Final = _chatgpt_bearer(request)
+ gateway_token: Final = _gateway_token(request)
+ if not chatgpt_bearer or not gateway_token:
raise HTTPException(
status_code=401,
detail="Send the ChatGPT bearer in Authorization and the LiteLLM key in x-litellm-api-key",
)
- if hmac.compare_digest(token.strip().encode(), gateway_token.encode()):
+ if hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
raise HTTPException(status_code=401, detail="ChatGPT and LiteLLM credentials must be separate")
- headers: Final = MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
- return await pass_through_request(
- request=request,
- target=f"{CHATGPT_API_BASE}/{endpoint}",
- custom_headers=dict(headers),
- user_api_key_dict=user_api_key_dict,
- custom_llm_provider="chatgpt",
- upstream_headers_filter=lambda _: headers,
- )
+ headers: Final = _forwarded_headers(request)
+ caller: Final = _models_etag_caller(user_api_key_dict, gateway_token)
+ access: Final = _litellm_responses_access(user_api_key_dict, request)
+ if endpoint == "models":
+ return await _catalog(request, user_api_key_dict, access, caller, headers, cache)
+ return await _responses(request, fastapi_response, user_api_key_dict, access, caller, headers, cache)
diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py
index f02dfadff81..59513c58060 100644
--- a/litellm/proxy/proxy_server.py
+++ b/litellm/proxy/proxy_server.py
@@ -223,7 +223,12 @@ from litellm.types.utils import (
is_server_derived_pricing_key,
pricing_override_fields,
)
-from litellm.utils import acredential_updates, cost_map_omits_token_price, with_named_credential
+from litellm.utils import (
+ acredential_updates,
+ cost_map_omits_token_price,
+ native_structured_output_support,
+ with_named_credential,
+)
if TYPE_CHECKING:
from aiohttp import ClientSession
@@ -233,6 +238,7 @@ if TYPE_CHECKING:
from litellm.integrations.opentelemetry import OpenTelemetry
from litellm.proxy.health_check_utils.shared_health_check_manager import SharedHealthCheckManager
+ from litellm.types.proxy.model_listing import ModelInfoResponse
Span = _Span | Any
else:
@@ -515,10 +521,11 @@ from litellm.proxy.common_utils.anthropic_error_payload import (
)
from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
AuthCacheInvalidationSubscriber,
+ publish_auth_cache_invalidation,
publish_auth_cache_invalidations,
)
from litellm.proxy.common_utils.callback_utils import initialize_callbacks_on_proxy
-from litellm.proxy.common_utils.codex_model_catalog import codex_model_list_body
+from litellm.proxy.common_utils.codex_model_catalog import CodexCatalogRow, codex_catalog_rows, codex_model_list_body
from litellm.proxy.common_utils.config_includes import resolve_include_file_path, resolve_includes
from litellm.proxy.common_utils.config_sync_pubsub import ConfigSyncSubscriber
from litellm.proxy.common_utils.credential_hydration import decrypted_or_stored
@@ -557,6 +564,7 @@ from litellm.proxy.common_utils.http_parsing_utils import (
from litellm.proxy.common_utils.load_config_utils import get_config_from_bucket
from litellm.proxy.common_utils.model_deprecation import collect_model_deprecations
from litellm.proxy.common_utils.model_listing_utils import (
+ CallerAliases,
ClaudeCodeRoutingNames,
DeployedModelGroups,
TeamModelNameTranslator,
@@ -726,6 +734,7 @@ from litellm.proxy.litellm_pre_call_utils import (
SCHEDULER_PRIORITY_HEADER,
LiteLLMProxyRequestSetup,
add_litellm_data_to_request,
+ sanitize_for_log,
)
from litellm.proxy.logging_endpoints.callback_logs_endpoints import (
rust_control_plane_router,
@@ -1045,6 +1054,7 @@ from litellm.proxy.utils import ( # noqa: F401, RUF100 # legacy module exports
migrate_passwords_to_scrypt_async,
model_dump_with_preserved_fields,
prefetch_config_params,
+ run_off_loop_if_fips,
update_spend,
)
from litellm.proxy.video_endpoints.endpoints import router as video_router
@@ -4561,6 +4571,36 @@ def raise_if_spend_counters_unwritten(lost: Exception | None, saved: str) -> Non
)
+@with_service_target(SPEND_COUNTERS_TARGET)
+async def set_spend_counter_with_floor_and_broadcast(counter_key: str, value: float) -> None:
+ """
+ Set a Redis-backed spend counter to `value`, mirror it into the short-lived
+ spend_db_floor marker `_authoritative_floor_spend` reads, and broadcast both
+ to every worker (LIT-3803 pattern: setting, not deleting, means a worker's
+ own self-delivered broadcast still carries the reset value forward).
+
+ Without the floor marker, `_authoritative_floor_spend` can re-derive a
+ stale, pre-reset value from a marker another worker cached moments earlier
+ and raise the just-reset counter right back up via `_repair_stale_spend_counter`.
+ Without the broadcast, a worker that already cached the pre-reset key object
+ or floor marker keeps enforcing against it until its own TTL expires.
+ """
+ try:
+ await spend_counter_store.set(counter_key, value, memory_ttl=60)
+ except Exception as redis_err:
+ verbose_proxy_logger.warning(
+ "Failed to update spend counter %s in Redis: %s. Budget checks may use stale value until counter expires.",
+ sanitize_for_log(counter_key),
+ redis_err,
+ )
+
+ floor_key: Final = f"spend_db_floor:{counter_key}"
+ spend_counter_cache.in_memory_cache.set_cache(key=floor_key, value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
+
+ await publish_auth_cache_invalidation(cache_key=counter_key, new_value=value, ttl=60)
+ await publish_auth_cache_invalidation(cache_key=floor_key, new_value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
+
+
async def _apply_spend_counter_increments(pending: Sequence[PendingSpendIncrement]) -> None:
if _defer_spend_counter_increments(pending):
return
@@ -12721,6 +12761,116 @@ async def _deployment_hidden_by_listing_callbacks(deployment: Deployment, user_a
return listed_name in await _names_hidden_by_listing_callbacks(user_api_key_dict, (listed_name,))
+async def _caller_aliases(user_api_key_dict: UserAPIKeyAuth, team_id: str | None) -> CallerAliases:
+ caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
+ user_api_key_dict, prisma_client, proxy_logging_obj
+ )
+ return caller_alias_maps(
+ user_api_key_dict.aliases,
+ user_api_key_dict.team_model_aliases,
+ user_api_key_dict.team_id,
+ team_id,
+ caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
+ )
+
+
+async def _listing_hidden_names(healthy_only: bool | None) -> frozenset[str]:
+ blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
+ unhealthy_names: Final = await get_hidden_unhealthy_model_names(
+ healthy_only=healthy_only,
+ general_settings=general_settings_view(),
+ llm_router=llm_router,
+ )
+ return frozenset(blocked_names | unhealthy_names)
+
+
+@dataclass(frozen=True, slots=True)
+class _KeyModelListing:
+ rows: "tuple[ModelInfoResponse, ...]"
+ entries: tuple[tuple[str, str], ...]
+ labels: tuple[tuple[str, str], ...]
+
+
+def _listed_model_info(
+ response_id: str, lookup_id: str, include_metadata: bool, fallback_type: str | None
+) -> "ModelInfoResponse":
+ from litellm.proxy.utils import create_model_info_response
+
+ model_info: Final = create_model_info_response(
+ model_id=lookup_id,
+ provider="openai",
+ include_metadata=include_metadata,
+ fallback_type=fallback_type,
+ llm_router=llm_router,
+ )
+ model_info["id"] = response_id
+ return model_info
+
+
+async def _key_model_listing(
+ user_api_key_dict: UserAPIKeyAuth,
+ caller_aliases: CallerAliases,
+ hidden_names: frozenset[str],
+ *,
+ team_id: str | None = None,
+ include_model_access_groups: bool = False,
+ only_model_access_groups: bool = False,
+ include_wildcard_routes: bool = False,
+ include_metadata: bool = False,
+ fallback_type: str | None = None,
+ offered_model_names: Sequence[str] = (),
+) -> _KeyModelListing:
+ from litellm.proxy.utils import get_available_models_for_user
+
+ available: Final = await get_available_models_for_user(
+ user_api_key_dict=user_api_key_dict,
+ llm_router=llm_router,
+ general_settings=general_settings,
+ user_model=user_model,
+ prisma_client=prisma_client,
+ proxy_logging_obj=proxy_logging_obj,
+ team_id=team_id,
+ include_model_access_groups=include_model_access_groups,
+ only_model_access_groups=only_model_access_groups,
+ return_wildcard_routes=include_wildcard_routes,
+ user_api_key_cache=user_api_key_cache,
+ )
+ names: Final = (*available, *(name for name in offered_model_names if name not in available))
+ excluded: Final = hidden_names | undiscoverable_model_names(
+ names, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
+ )
+ listed_entries: Final = await _entries_kept_by_listing_callbacks(
+ TeamModelNameTranslator.listing_entries(
+ [name for name in names if name not in excluded], llm_router, general_settings_view()
+ ),
+ user_api_key_dict,
+ )
+ entries: Final = alias_listing_entries(
+ listed_entries,
+ caller_aliases,
+ _describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
+ )
+ return _KeyModelListing(
+ rows=tuple(
+ _listed_model_info(response_id, lookup_id, include_metadata, fallback_type)
+ for response_id, lookup_id in entries
+ ),
+ entries=entries,
+ labels=label_entries(listed_entries, entries),
+ )
+
+
+async def codex_catalog_rows_for_key(user_api_key_dict: UserAPIKeyAuth) -> tuple[CodexCatalogRow, ...]:
+ listing: Final = await _key_model_listing(
+ user_api_key_dict,
+ await _caller_aliases(user_api_key_dict, None),
+ await _listing_hidden_names(None),
+ )
+ return await asyncio.to_thread(
+ codex_catalog_rows, listing.rows, listing.entries, llm_router, user_api_key_dict.team_id, listing.labels
+ )
+
+
@router.get("/v1/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"])
@router.get(
"/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"]
@@ -12788,10 +12938,7 @@ async def model_list(
from litellm.proxy.management_endpoints.common_utils import (
user_has_admin_privileges,
)
- from litellm.proxy.utils import (
- create_model_info_response,
- get_available_models_for_user,
- )
+ from litellm.proxy.utils import create_model_info_response
from litellm.types.proxy.model_listing import ModelInfoResponse
http_request: Final = cast(Request | None, request) # cast-ok: in-process callers pass no request
@@ -12799,16 +12946,7 @@ async def model_list(
http_request is not None and http_request.headers.get("anthropic-version") is not None
)
client_headers: Final[Mapping[str, str]] = http_request.headers if http_request is not None else _EMPTY_HEADERS
- caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
- user_api_key_dict, prisma_client, proxy_logging_obj
- )
- caller_aliases: Final = caller_alias_maps(
- user_api_key_dict.aliases,
- user_api_key_dict.team_model_aliases,
- user_api_key_dict.team_id,
- team_id,
- caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
- )
+ caller_aliases: Final = await _caller_aliases(user_api_key_dict, team_id)
routing_names: Final = ClaudeCodeRoutingNames(
llm_router,
team_id or user_api_key_dict.team_id,
@@ -12834,18 +12972,7 @@ async def model_list(
proxy_logging_obj=proxy_logging_obj,
)
- # Compute once — used in both branches below to hide paused models from the listing.
- blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
-
- # Opt-in: also hide models whose deployments are all unhealthy per background
- # health checks. Empty when health state is unavailable or stale (fail open).
- unhealthy_names: Final = await get_hidden_unhealthy_model_names(
- healthy_only=healthy_only,
- general_settings=settings,
- llm_router=llm_router,
- )
-
- hidden_names: Final = blocked_names | unhealthy_names
+ hidden_names: Final = await _listing_hidden_names(healthy_only)
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import typesafe_allowlist_model_names
@@ -12943,58 +13070,24 @@ async def model_list(
object="list",
)
- # Otherwise, use the normal behavior (current implementation)
- # Get available models for the user
- available_models: Final = await get_available_models_for_user(
- user_api_key_dict=user_api_key_dict,
- llm_router=llm_router,
- general_settings=general_settings,
- user_model=user_model,
- prisma_client=prisma_client,
- proxy_logging_obj=proxy_logging_obj,
- team_id=team_id,
- include_model_access_groups=include_model_access_groups or False,
- only_model_access_groups=only_model_access_groups or False,
- return_wildcard_routes=include_wildcard_routes,
- user_api_key_cache=user_api_key_cache,
- )
offers_typesafe_allowlist_names: Final = (
asks_for_allowlist_names and team_id is None and user_api_key_has_admin_view(user_api_key_dict)
)
- typesafe_names_to_offer: Final = typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else ()
- all_models = [*available_models, *(name for name in typesafe_names_to_offer if name not in available_models)]
-
- undiscoverable_names: Final = undiscoverable_model_names(
- all_models, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
- )
- if hidden_names or undiscoverable_names:
- all_models = [m for m in all_models if m not in hidden_names and m not in undiscoverable_names]
-
- # Surface the public team name by default; legacy internal keys via flag.
- # The internal routing key drives the metadata/fallback lookup, while the
- # public name is what the client sees as the model id.
- model_data = []
- listed_entries: Final = await _entries_kept_by_listing_callbacks(
- TeamModelNameTranslator.listing_entries(all_models, llm_router, settings), user_api_key_dict
- )
- entries: Final = alias_listing_entries(
- listed_entries,
+ key_listing: Final = await _key_model_listing(
+ user_api_key_dict,
caller_aliases,
- _describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
+ hidden_names,
+ team_id=team_id,
+ include_model_access_groups=include_model_access_groups or False,
+ only_model_access_groups=only_model_access_groups or False,
+ include_wildcard_routes=include_wildcard_routes,
+ include_metadata=include_metadata or False,
+ fallback_type=fallback_type,
+ offered_model_names=typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else (),
)
- labels: Final = label_entries(listed_entries, entries)
- for response_id, lookup_id in entries:
- model_info = create_model_info_response(
- model_id=lookup_id,
- provider="openai",
- include_metadata=include_metadata or False,
- fallback_type=fallback_type,
- llm_router=llm_router,
- )
- model_info["id"] = response_id
- model_data.append(model_info)
-
- listing: Final = cast(Sequence[ModelInfoResponse], model_data) # cast-ok: rows built above
+ listing: Final = key_listing.rows
+ entries: Final = key_listing.entries
+ labels: Final = key_listing.labels
if client_version is not None:
return Response(
content=await asyncio.to_thread(
@@ -13021,7 +13114,7 @@ async def model_list(
)
return dict(
- data=model_data,
+ data=list(listing),
object="list",
)
@@ -16875,7 +16968,7 @@ def _enrich_model_info_with_litellm_data(
stamped_model_info: Final = MappingProxyType(
{**model_info, **_pricing_override_stamps(model_info, model.get("litellm_params") or MappingProxyType({}))}
)
- model["model_info"] = {
+ merged_model_info: Final = {
**stamped_model_info,
**{
k: None if unpriced and k in ("input_cost_per_token", "output_cost_per_token") else v
@@ -16883,6 +16976,18 @@ def _enrich_model_info_with_litellm_data(
if k not in stamped_model_info or (stamped_model_info[k] is None and k in discovered_model_info)
},
}
+ native_output_info: Final = {
+ **merged_model_info,
+ "supports_bedrock_native_structured_output_by_default": litellm_model_info.get(
+ "supports_bedrock_native_structured_output_by_default"
+ ),
+ }
+ model["model_info"] = (
+ {**merged_model_info, "supports_native_structured_output": native_structured_output_support(native_output_info)}
+ if "supports_native_structured_output" in merged_model_info
+ and "supports_native_structured_output" not in stamped_model_info
+ else merged_model_info
+ )
# don't return the api key / vertex credentials
# don't return the llm credentials
model = remove_sensitive_info_from_deployment(model, excluded_keys={"litellm_credential_name"})
@@ -19939,7 +20044,7 @@ async def claim_onboarding_link(data: InvitationClaim, request: Request):
validate_password_policy(data.password, general_settings)
await validate_password_not_breached(data.password, general_settings)
- hashed_pw: Final = hash_password(data.password)
+ hashed_pw: Final = await run_off_loop_if_fips(hash_password, data.password)
current_time = litellm.utils.get_utc_datetime()
tx_manager: Final[_InvitationAcceptTxManager] = prisma_client.db.tx()
async with tx_manager as tx:
diff --git a/litellm/proxy/public_endpoints/provider_create_fields.json b/litellm/proxy/public_endpoints/provider_create_fields.json
index 5ded2fa5f5e..5afc3908820 100644
--- a/litellm/proxy/public_endpoints/provider_create_fields.json
+++ b/litellm/proxy/public_endpoints/provider_create_fields.json
@@ -2267,7 +2267,7 @@
},
{
"provider": "LITELLM_PROXY",
- "provider_display_name": "Litellm Proxy",
+ "provider_display_name": "LiteLLM Gateway",
"litellm_provider": "litellm_proxy",
"credential_fields": [
{
diff --git a/litellm/proxy/utils.py b/litellm/proxy/utils.py
index 3102674376a..64bce0d21f2 100644
--- a/litellm/proxy/utils.py
+++ b/litellm/proxy/utils.py
@@ -6,6 +6,7 @@ import inspect
import json
import math
import os
+import re
import smtplib
import ssl
import sys
@@ -24,6 +25,7 @@ from collections.abc import (
Mapping,
Sequence,
)
+from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta, timezone
from email.mime.multipart import MIMEMultipart
@@ -85,7 +87,9 @@ from litellm.proxy._types import (
SpendLogsMetadata,
SpendLogsPayload,
)
+from litellm.proxy.auth.password_policy import MIN_ALLOWED_LENGTH
from litellm.proxy.bug_report_config import build_proxy_bug_report
+from litellm.proxy.common_utils.fips import fips_mode_active
from litellm.proxy.common_utils.openai_error_payload import (
attribute_of,
error_status_code,
@@ -8399,23 +8403,42 @@ def hash_token(token: str):
return hashed_token
+def _is_pbkdf2_row(stored: str) -> bool:
+ return re.fullmatch(r"pbkdf2:[A-Za-z0-9+/]{64}", stored) is not None
+
+
def hash_password(password: str) -> str:
- """Hash a password using scrypt with a random salt."""
+ """Hash a password using scrypt with a random salt, or PBKDF2 in FIPS mode."""
import base64
import hashlib
import os
+ if fips_mode_active():
+ fips_salt: Final = ssl.RAND_bytes(16)
+ fips_dk: Final = hashlib.pbkdf2_hmac("sha256", password.encode(), fips_salt, iterations=600_000, dklen=32)
+ return "pbkdf2:" + base64.b64encode(fips_salt + fips_dk).decode()
salt: Final = os.urandom(16)
dk: Final = hashlib.scrypt(password.encode(), salt=salt, n=16384, r=8, p=1, dklen=32)
return "scrypt:" + base64.b64encode(salt + dk).decode()
def verify_password(password: str, stored: str) -> bool:
- """Verify a password against a stored hash. Supports scrypt and SHA256."""
+ """Verify a password against a stored hash. Supports scrypt and SHA256, or only PBKDF2 in FIPS mode."""
import base64
import hashlib
import secrets
+ if fips_mode_active():
+ if not _is_pbkdf2_row(stored):
+ verbose_proxy_logger.error("FIPS mode refuses a non-PBKDF2 password row. Reset it with /user/update.")
+ return False
+ if len(password) < MIN_ALLOWED_LENGTH:
+ return False
+ fips_row: Final = base64.b64decode(stored[7:])
+ fips_dk: Final = hashlib.pbkdf2_hmac(
+ "sha256", password.encode(errors="surrogatepass"), fips_row[:16], iterations=600_000, dklen=32
+ )
+ return secrets.compare_digest(fips_row[16:], fips_dk)
if stored.startswith("scrypt:"):
try:
raw: Final = base64.b64decode(stored[7:])
@@ -8430,12 +8453,24 @@ def verify_password(password: str, stored: str) -> bool:
return False
+_FIPS_KDF_POOL: Final = ThreadPoolExecutor(max_workers=1, thread_name_prefix="fips-password-kdf")
+
+
+async def run_off_loop_if_fips(kdf: Callable[..., _T], *args: str) -> _T:
+ """Run a password KDF inline, or on the single KDF thread in FIPS mode."""
+ if not fips_mode_active():
+ return kdf(*args)
+ return await asyncio.get_running_loop().run_in_executor(_FIPS_KDF_POOL, kdf, *args)
+
+
async def migrate_passwords_to_scrypt_async(prisma_client) -> str:
"""
Migrate plaintext passwords in the DB to scrypt. SHA256 passwords
are left alone (they migrate on next login via the SHA256 fallback).
Skips quickly if no plaintext passwords exist.
"""
+ if fips_mode_active():
+ return "Skipped, FIPS mode has no password migration"
all_with_pw: Final = await UserRepository(prisma_client).table.find_many(
where={"password": {"not": None}},
)
@@ -8446,7 +8481,10 @@ async def migrate_passwords_to_scrypt_async(prisma_client) -> str:
plaintext_users: Final = [
(u.user_id, u.password)
for u in all_with_pw
- if u.password and not u.password.startswith("scrypt:") and not _is_sha256_hex(u.password)
+ if u.password
+ and not u.password.startswith("scrypt:")
+ and not _is_sha256_hex(u.password)
+ and not _is_pbkdf2_row(u.password)
]
if not plaintext_users:
return "No plaintext passwords found"
diff --git a/litellm/responses/additional_tools.py b/litellm/responses/additional_tools.py
index 9c7a3a5f72e..879bc8ba41c 100644
--- a/litellm/responses/additional_tools.py
+++ b/litellm/responses/additional_tools.py
@@ -1,21 +1,20 @@
import json
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass
-from itertools import groupby
+from itertools import chain, groupby
from types import MappingProxyType
from typing import Final, cast # noqa: TID251 # validating the openai tool union strips vendor keys from raw tools
-from pydantic import ValidationError
+from pydantic import TypeAdapter, ValidationError
+from typing_extensions import TypeIs # noqa: TID251 # narrows a client input item before its type is read
from litellm._logging import verbose_logger
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
ADDITIONAL_TOOLS_INPUT_ITEM_TYPE: Final = "additional_tools"
-
-
-class _InputItemType(LiteLLMBaseModel):
- type: str = ""
+_NAMESPACE_TOOL_TYPE: Final = "namespace"
+_ITEMS: Final = TypeAdapter(tuple[object, ...])
class _ToolList(LiteLLMBaseModel):
@@ -29,11 +28,14 @@ class HoistedAdditionalTools:
hoisted: tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]
+def _is_an_input_item(
+ item: object,
+) -> TypeIs[Mapping[str, object]]: # guard-ok: auth scans every list input, and a pydantic error per item is slow
+ return isinstance(item, Mapping)
+
+
def _is_additional_tools_item(item: object) -> bool:
- try:
- return _InputItemType.model_validate(item).type == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
- except ValidationError:
- return False
+ return _is_an_input_item(item) and item.get("type") == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
def _tool_list_of(value: object) -> tuple[Mapping[str, object], ...]:
@@ -61,7 +63,7 @@ def _members_of_copies(copies: Sequence[Mapping[str, object]]) -> Iterator[Mappi
def _merged(copies: Sequence[Mapping[str, object]]) -> Mapping[str, object]:
newest: Final = copies[-1]
- if newest.get("type") != "namespace" or len(copies) == 1:
+ if newest.get("type") != _NAMESPACE_TOOL_TYPE or len(copies) == 1:
return newest
members: Final = tuple(_members_of_copies(copies))
return {**newest, "tools": list(_unique_tools(members))} if members else newest
@@ -86,6 +88,11 @@ def _unique_tools(tools: Sequence[Mapping[str, object]]) -> tuple[Mapping[str, o
return tuple(_merged(copies_by_identity[identity]) for identity in first_seen_order)
+def additional_tools_in_input(input: object) -> tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]:
+ items: Final = _ITEMS.validate_python(input) if isinstance(input, list) else ()
+ return tuple(chain.from_iterable(_tools_of_item(item) for item in items if _is_additional_tools_item(item)))
+
+
def hoist_additional_tools(
input: str | ResponseInputParam,
tools: Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None,
@@ -112,3 +119,26 @@ def hoist_additional_tools(
for tool in _unique_tools((*existing, *hoisted))
)
return HoistedAdditionalTools(input=remaining_input, tools=unique, hoisted=hoisted)
+
+
+def _top_level_tools(request: Mapping[str, object]) -> tuple[object, ...] | None:
+ try:
+ return _ITEMS.validate_python(request.get("tools") or ())
+ except ValidationError:
+ return None
+
+
+def with_additional_tools_hoisted(request: Mapping[str, object], *, drop_params: bool) -> Mapping[str, object]:
+ request_input: Final = request.get("input")
+ top_level_tools: Final = _top_level_tools(request)
+ if not isinstance(request_input, list) or top_level_tools is None:
+ return request
+ input_items: Final = cast(ResponseInputParam, request_input) # cast-ok: forwarded as sent, as inference does
+ hoisted: Final = hoist_additional_tools(
+ input_items,
+ cast("tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", top_level_tools), # cast-ok: forwarded as sent
+ drop_params=drop_params,
+ )
+ if not hoisted.hoisted:
+ return request
+ return {**request, "input": hoisted.input, "tools": list(hoisted.tools)}
diff --git a/litellm/responses/litellm_completion_transformation/count_request.py b/litellm/responses/litellm_completion_transformation/count_request.py
index 3deceac07aa..2141b55850b 100644
--- a/litellm/responses/litellm_completion_transformation/count_request.py
+++ b/litellm/responses/litellm_completion_transformation/count_request.py
@@ -12,7 +12,7 @@ from litellm.litellm_core_utils.prompt_templates.file_id_utils import (
)
from litellm.litellm_core_utils.tool_fields import DropPolicy
from litellm.llms.base_llm.files.batch_records import responses_batch_body_to_chat_body
-from litellm.responses.additional_tools import hoist_additional_tools
+from litellm.responses.additional_tools import with_additional_tools_hoisted
from litellm.responses.file_search.emulated_handler import replace_file_search_tools
from litellm.responses.litellm_completion_transformation.compaction_items import holds_compaction_summary
from litellm.responses.litellm_completion_transformation.transformation import (
@@ -20,7 +20,7 @@ from litellm.responses.litellm_completion_transformation.transformation import (
LiteLLMCompletionResponsesConfig,
)
from litellm.responses.main import previous_response_client_ids
-from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
+from litellm.types.llms.openai import ResponseInputParam
from litellm.utils import ProviderConfigManager
_COUNTED_CHAT_PARAMS: Final = (
@@ -114,20 +114,6 @@ def with_provider_file_ids(
}
-def _with_hoisted_additional_tools(responses_request: Mapping[str, object], drop_params: bool) -> Mapping[str, object]:
- responses_input: Final = responses_request.get("input")
- if not isinstance(responses_input, list):
- return responses_request
- input_items: Final = cast(ResponseInputParam, responses_input) # cast-ok: forwarded as sent, as inference does
- top_level_tools: Final = cast( # cast-ok: tools are forwarded as sent, as inference does
- "tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", _ITEMS.validate_python(responses_request.get("tools") or ())
- )
- hoisted: Final = hoist_additional_tools(input_items, top_level_tools, drop_params=drop_params)
- if not hoisted.hoisted:
- return responses_request
- return {**responses_request, "input": hoisted.input, "tools": list(hoisted.tools)}
-
-
def _caller_input(responses_request: Mapping[str, object]) -> ResponseInputParam | None:
responses_input: Final = responses_request.get("input")
return (
@@ -205,7 +191,9 @@ async def chat_count_request(
chat_body: Final = await _with_previous_turns(
responses_batch_body_to_chat_body(
_with_compaction_as_summary_text(
- _with_emulated_file_search(_with_hoisted_additional_tools(responses_request, policy.drop_params))
+ _with_emulated_file_search(
+ with_additional_tools_hoisted(responses_request, drop_params=policy.drop_params)
+ )
),
custom_llm_provider,
policy.drop_params,
diff --git a/litellm/responses/litellm_completion_transformation/reasoning_items.py b/litellm/responses/litellm_completion_transformation/reasoning_items.py
index ab982222902..dda313bfc1b 100644
--- a/litellm/responses/litellm_completion_transformation/reasoning_items.py
+++ b/litellm/responses/litellm_completion_transformation/reasoning_items.py
@@ -6,6 +6,7 @@ from typing import Final
from pydantic import BaseModel, TypeAdapter, ValidationError
REASONING_ITEM_ID_PREFIX: Final = "rs_"
+WRAPPED_ENCRYPTED_CONTENT_PREFIX: Final = "litellm_enc:"
_JSON_LIST: Final = TypeAdapter(list[object])
_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
@@ -58,16 +59,35 @@ def is_minted_reasoning_item_id(item_id: object) -> bool:
return parsed.version == 4 and str(parsed) == suffix
-def is_litellm_minted_reasoning_item(item: object) -> bool:
+def _reasoning_fields(item: object) -> Mapping[str, object] | None:
try:
fields: Final = _JSON_OBJECT.validate_python(
item.model_dump(exclude_none=True) if isinstance(item, BaseModel) else item
)
except ValidationError:
- return False
- if fields.get("type") != "reasoning":
- return False
+ return None
+ return fields if fields.get("type") == "reasoning" else None
+
+
+def _is_minted(fields: Mapping[str, object]) -> bool:
return (
is_minted_reasoning_item_id(fields.get("id"))
or decode_thinking_blocks(fields.get("encrypted_content")) is not None
)
+
+
+def is_litellm_minted_reasoning_item(item: object) -> bool:
+ fields: Final = _reasoning_fields(item)
+ return fields is not None and _is_minted(fields)
+
+
+def is_other_provider_reasoning_item(item: object) -> bool:
+ fields: Final = _reasoning_fields(item)
+ if fields is None:
+ return False
+ encrypted_content: Final = fields.get("encrypted_content")
+ return (
+ _is_minted(fields)
+ or bool(fields.get("content"))
+ or (isinstance(encrypted_content, str) and encrypted_content.startswith(WRAPPED_ENCRYPTED_CONTENT_PREFIX))
+ )
diff --git a/litellm/responses/litellm_completion_transformation/streaming_iterator.py b/litellm/responses/litellm_completion_transformation/streaming_iterator.py
index 521a698b276..0f8ba6ef98a 100644
--- a/litellm/responses/litellm_completion_transformation/streaming_iterator.py
+++ b/litellm/responses/litellm_completion_transformation/streaming_iterator.py
@@ -9,6 +9,7 @@ import litellm
from litellm.litellm_core_utils.core_helpers import as_str_mapping, attribute_items
from litellm.litellm_core_utils.hidden_params import get_or_create_hidden_params, served_from_cache
from litellm.litellm_core_utils.litellm_logging import BilledRoundsSpend
+from litellm.llms.anthropic.common_utils import anthropic_server_call_fields, anthropic_server_tool_calls
from litellm.llms.anthropic.messages.native.utils import openai_chat_refusal_text
from litellm.main import stream_chunk_builder
from litellm.responses.litellm_completion_transformation.compaction_items import compaction_output_item
@@ -133,6 +134,7 @@ def _delta_carries_output(delta: ChatCompletionDelta) -> bool:
getattr(delta, "images", None),
getattr(delta, "audio", None),
openai_chat_refusal_text(delta),
+ anthropic_server_call_fields(delta),
)
return any(fields) or _delta_has_signed_thinking_block(delta)
@@ -228,6 +230,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
self._queued_web_search_call_ids: set[str] = set() # mutable-ok: emitted call ids
self._tool_search = tool_search_routing(request_tools)
self._tool_search_executions: Mapping[str, ToolSearchExecution] = MappingProxyType({})
+ self._done_tool_search_call_ids: frozenset[str] = frozenset()
async def aclose(self) -> None:
await self.litellm_custom_stream_wrapper.aclose()
@@ -265,6 +268,9 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
def _queue_tool_search_call_done(
self, call_id: str, output_index: int, final_args: str, execution: ToolSearchExecution
) -> None:
+ if call_id in self._done_tool_search_call_ids:
+ return
+ self._done_tool_search_call_ids = self._done_tool_search_call_ids | {call_id}
self._queue_tool_search_call_added(call_id, output_index, "", execution)
item: Final = build_tool_search_call(
call_id, final_args or self._tool_args_by_call_id.get(call_id, ""), "completed", execution
@@ -277,6 +283,18 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
)
)
+ def _queue_streamed_tool_search_calls(self, chunk: ModelResponseStream) -> None:
+ if not chunk.choices:
+ return
+ for call in anthropic_server_tool_calls(chunk.choices[0].delta):
+ if (execution := self._tool_search_execution(call.id, call.function.name or "")) is None:
+ continue
+ if call.id not in self._tool_output_index_by_call_id:
+ self._queue_held_annotations()
+ self._queue_tool_search_call_done(
+ call.id, self._get_or_assign_tool_output_index(call.id), call.function.arguments, execution
+ )
+
def _get_or_assign_tool_output_index(self, call_id: str) -> int:
existing: Final = self._tool_output_index_by_call_id.get(call_id)
if existing is not None:
@@ -509,11 +527,17 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
try:
message: Final = litellm_complete_object.choices[0].message
- tool_calls = getattr(message, "tool_calls", None)
+ server_tool_calls: Final = anthropic_server_tool_calls(message)
+ tool_calls: Final = (*server_tool_calls, *_tool_call_objects(getattr(message, "tool_calls", None)))
except Exception:
- tool_calls = None
+ return
- for tc in _tool_call_objects(tool_calls):
+ if not tool_calls:
+ return
+
+ server_tool_call_ids: Final = frozenset(tool_call.id for tool_call in server_tool_calls)
+
+ for tc in tool_calls:
call_id_raw = tc.get("id") if isinstance(tc, dict) else getattr(tc, "id", None)
if not call_id_raw:
continue
@@ -538,6 +562,8 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
if (search_execution := self._tool_search_execution(call_id, fn_name)) is not None:
self._queue_tool_search_call_done(call_id, output_index, fn_args, search_execution)
continue
+ if call_id in server_tool_call_ids:
+ continue
# Track if this is a new tool call that wasn't streamed
is_new_tool_call = call_id not in self._tool_item_id_by_call_id
@@ -1205,7 +1231,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
return
# Tool-first
- if hasattr(delta, "tool_calls") and delta.tool_calls:
+ if (hasattr(delta, "tool_calls") and delta.tool_calls) or anthropic_server_call_fields(delta):
# Tool calls already handled via _queue_tool_call_delta_events
# DO NOT create message item
return
@@ -1383,6 +1409,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
def _queue_chunk_events(self, chunk: ModelResponseStream) -> None:
self._hold_chunk_annotations(chunk)
+ self._queue_streamed_tool_search_calls(chunk)
response_api_chunk: Final = self._transform_chat_completion_chunk_to_response_api_chunk(chunk)
if response_api_chunk:
self._pending_response_events.append(response_api_chunk)
diff --git a/litellm/responses/litellm_completion_transformation/transformation.py b/litellm/responses/litellm_completion_transformation/transformation.py
index 3e605e926f7..33ea85d7d53 100644
--- a/litellm/responses/litellm_completion_transformation/transformation.py
+++ b/litellm/responses/litellm_completion_transformation/transformation.py
@@ -73,7 +73,12 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
is_unhashable_value_behind,
)
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
-from litellm.llms.anthropic.common_utils import supports_anthropic_cache_control
+from litellm.llms.anthropic.common_utils import (
+ anthropic_server_tool_call_params,
+ anthropic_server_tool_calls,
+ supports_anthropic_cache_control,
+ validated_object_mapping,
+)
from litellm.llms.anthropic.messages.native.utils import anthropic_stop_reason, openai_chat_refusal_text
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.bedrock.common_utils import bedrock_declared_cache_point_support
@@ -84,11 +89,13 @@ from litellm.responses.litellm_completion_transformation.session_handler import
ResponsesSessionHandler,
StoredTurns,
)
-from litellm.types.llms.anthropic import AnthropicToolSearchToolRegex
+from litellm.responses.utils import ResponsesAPIRequestUtils
+from litellm.types.llms.anthropic import AnthropicToolSearchToolRegex, claude_hosted_web_tool_call
from litellm.types.llms.base import CachedTokensDetails
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantMessage,
+ ChatCompletionAssistantToolCall,
ChatCompletionAudioObject,
ChatCompletionCachedContent,
ChatCompletionFileObject,
@@ -933,6 +940,22 @@ def _chain_session_metadata(chain_session: str, request_kwargs: Mapping[str, obj
return {key: {**_STR_KEY_DICT_ADAPTER.validate_python(metadata), CALLER_SESSION_ID_METADATA_KEY: chain_session}}
+def claude_web_search_call_tool_call(search: ResponseFunctionWebSearch) -> ChatCompletionAssistantToolCall | None:
+ claude_call: Final = claude_hosted_web_tool_call(search)
+ if claude_call is None:
+ return None
+ tool_id, name, arguments = claude_call
+ return ChatCompletionAssistantToolCall(
+ id=tool_id,
+ type="function",
+ function=ChatCompletionToolCallFunctionChunk(name=name, arguments=arguments),
+ )
+
+
+def _pending_claude_server_tool_call(search: ResponseFunctionWebSearch) -> ChatCompletionAssistantToolCall | None:
+ return claude_web_search_call_tool_call(search) if search.status == "in_progress" else None
+
+
class LiteLLMCompletionResponsesConfig:
@staticmethod
def get_supported_openai_params(model: str) -> list:
@@ -1889,10 +1912,22 @@ class LiteLLMCompletionResponsesConfig:
message_cache_control: Final = (
None if isinstance(content, list) else new_cache_control or previous_cache_control
)
+ new_server_tool_calls: Final = anthropic_server_tool_call_params(new_message)
+ server_tool_call_fields: Final = (
+ {
+ "provider_specific_fields": {
+ **(validated_object_mapping(last_message.get("provider_specific_fields")) or {}),
+ "server_tool_calls": [*anthropic_server_tool_call_params(last_message), *new_server_tool_calls],
+ }
+ }
+ if new_server_tool_calls
+ else {}
+ )
merged: Final = {
**{key: value for key, value in last_message.items() if key != "cache_control"},
"content": content,
**({} if message_cache_control is None else {"cache_control": message_cache_control}),
+ **server_tool_call_fields,
}
return cast( # cast-ok: preserves the assistant fields and content blocks
ChatCompletionAssistantMessage, merged
@@ -2334,10 +2369,15 @@ class LiteLLMCompletionResponsesConfig:
content="Hosted web search: " + replay.model_dump_json(exclude_none=True),
)
]
+ hosted_search_text: Final = "Hosted web search: " + search.model_dump_json(exclude_none=True)
+ pending_server_tool_call: Final = _pending_claude_server_tool_call(search)
+ if pending_server_tool_call is None:
+ return [GenericChatCompletionMessage(role="assistant", content=hosted_search_text)]
return [
- GenericChatCompletionMessage(
+ ChatCompletionResponseMessage(
role="assistant",
- content="Hosted web search: " + search.model_dump_json(exclude_none=True),
+ content=hosted_search_text,
+ provider_specific_fields={"server_tool_calls": [pending_server_tool_call]},
)
]
if LiteLLMCompletionResponsesConfig._is_input_item_tool_call_output(input_item):
@@ -2517,7 +2557,12 @@ class LiteLLMCompletionResponsesConfig:
Returns None for anything this deployment did not write, so a genuinely
opaque blob is still skipped rather than forwarded as garbage.
"""
- decoded: Final = decode_thinking_blocks(input_item.get("encrypted_content"))
+ encrypted_content: Final = input_item.get("encrypted_content")
+ decoded: Final = decode_thinking_blocks(
+ ResponsesAPIRequestUtils.unwrap_encrypted_content_with_model_id(encrypted_content)[1]
+ if isinstance(encrypted_content, str)
+ else encrypted_content
+ )
if decoded is None:
return None
return tuple(
@@ -3085,13 +3130,12 @@ class LiteLLMCompletionResponsesConfig:
all_chat_completion_tools: Final[list[ChatCompletionMessageToolCall]] = []
for choice in chat_completion_response.choices:
if isinstance(choice, Choices):
- if choice.message.tool_calls:
- all_chat_completion_tools.extend(choice.message.tool_calls)
- for tool_call in choice.message.tool_calls:
- TOOL_CALLS_CACHE.set_cache(
- key=tool_call.id,
- value=tool_call,
- )
+ for tool_call in (*anthropic_server_tool_calls(choice.message), *(choice.message.tool_calls or ())):
+ all_chat_completion_tools.append(tool_call)
+ TOOL_CALLS_CACHE.set_cache(
+ key=tool_call.id,
+ value=tool_call,
+ )
request_tools: Final = responses_api_request.get("tools") if responses_api_request is not None else None
custom_tool_names: Final = extract_custom_tool_names(request_tools)
@@ -3522,7 +3566,22 @@ class LiteLLMCompletionResponsesConfig:
for item in responses_output
]
- return responses_output
+ server_tool_call_ids: Final = LiteLLMCompletionResponsesConfig._server_tool_call_ids(choices)
+ return [
+ item
+ for item in responses_output
+ if not (
+ getattr(item, "type", None) == "function_call"
+ and getattr(item, "call_id", None) in server_tool_call_ids
+ )
+ ]
+
+ @staticmethod
+ def _server_tool_call_ids(choices: Sequence[Choices]) -> frozenset[str]:
+ server_tool_calls: Final = chain.from_iterable(
+ anthropic_server_tool_calls(choice.message) for choice in choices
+ )
+ return frozenset(tool_call.id for tool_call in server_tool_calls)
@staticmethod
def _extract_tool_result_output_items(
diff --git a/litellm/responses/streaming_iterator.py b/litellm/responses/streaming_iterator.py
index 288e4a7119d..0c002a770e8 100644
--- a/litellm/responses/streaming_iterator.py
+++ b/litellm/responses/streaming_iterator.py
@@ -250,6 +250,8 @@ _ERROR_CODE_HTTP_STATUS: Final[Mapping[str, int]] = MappingProxyType(
"server_error": 500,
"rate_limit_exceeded": 429,
"insufficient_quota": 429,
+ "resource_unavailable": 429,
+ "flex_unavailable": 429,
"vector_store_timeout": 504,
"invalid_prompt": 400,
"invalid_image": 400,
diff --git a/litellm/responses/utils.py b/litellm/responses/utils.py
index 43ed9a3b96f..7fc8d5aed32 100644
--- a/litellm/responses/utils.py
+++ b/litellm/responses/utils.py
@@ -50,6 +50,7 @@ from litellm.types.utils import (
_PROVIDER_SPECIFIC_HEADER_ADAPTER: Final[
TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]
] = TypeAdapter(ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None)
+_REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES: Final = frozenset({"encrypted_content", "id", "status"})
class _UsageExtraFields(ExtraItemsTypedDict, total=False, extra_items=object):
@@ -678,7 +679,7 @@ class ResponsesAPIRequestUtils:
return
items: Final = cast(list[object], request_input) # cast-ok: untyped client json
stripped: Final = tuple(
- ResponsesAPIRequestUtils._without_encrypted_reasoning(reasoning)
+ ResponsesAPIRequestUtils.readable_reasoning(reasoning)
if (reasoning := ResponsesAPIRequestUtils._encrypted_reasoning_item(item)) is not None
and (should_strip is None or should_strip(reasoning))
else item
@@ -696,14 +697,14 @@ class ResponsesAPIRequestUtils:
return reasoning
@staticmethod
- def _without_encrypted_reasoning(reasoning: Mapping[str, object]) -> object | None:
+ def readable_reasoning(reasoning: Mapping[str, object]) -> Mapping[str, object] | None:
readable: Final = any(
ResponsesAPIRequestUtils._has_readable_text(reasoning.get(key)) for key in ("summary", "content")
)
if not readable:
return None
kept: Final[dict[str, object]] = { # mutable-ok: request item rebuilt without the undecryptable keys
- key: value for key, value in reasoning.items() if key not in ("encrypted_content", "id")
+ key: value for key, value in reasoning.items() if key not in _REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES
}
return kept
diff --git a/litellm/router_api.py b/litellm/router_api.py
index fa6ad69fab4..1ade356f494 100644
--- a/litellm/router_api.py
+++ b/litellm/router_api.py
@@ -158,6 +158,7 @@ GroupField: TypeAlias = Literal[
"display_name",
"mode",
"service_tiers",
+ "pinned_reasoning_efforts",
"max_input_tokens",
"deployment_model_info",
"token_limits",
@@ -939,7 +940,7 @@ class RouterRegistry:
self,
name: str,
*,
- field: Literal["service_tiers"],
+ field: Literal["service_tiers", "pinned_reasoning_efforts"],
include_hidden: bool = False,
team_id: str | None = None,
) -> tuple[object, ...]: ...
@@ -1001,6 +1002,7 @@ class RouterRegistry:
(were ``get_configured_display_name``, ``get_configured_mode``,
``get_configured_token_limits``, ``get_model_listing_info``)
- ``service_tiers``: (was ``get_configured_service_tiers``)
+ - ``pinned_reasoning_efforts``: the ``reasoning_effort`` each routable deployment pins
- ``max_input_tokens``: each routable deployment's own ``model_info.max_input_tokens``
- ``deployment_model_info``: each routable deployment's model info, merged with the pricing map
"""
@@ -1015,6 +1017,8 @@ class RouterRegistry:
return r._core_get_configured_mode(name)
if field == "service_tiers":
return r._core_get_configured_service_tiers(name, team_id=team_id)
+ if field == "pinned_reasoning_efforts":
+ return r._core_get_pinned_reasoning_efforts(name, team_id=team_id)
if field == "max_input_tokens":
return r._core_get_configured_max_input_tokens(name, team_id=team_id)
if field == "deployment_model_info":
@@ -1136,6 +1140,18 @@ class RouterRegistry:
return r._core_prompt_caching_pin_cache
raise ValueError(f"unknown plugin kind {kind!r}")
+ def decryptable_reasoning(
+ self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
+ ) -> tuple[object, ...]:
+ """
+ ``request_input`` without the encrypted reasoning that ``targets``, the deployments a
+ ``model_group`` request may go to, cannot decrypt, keeping each item's readable summary.
+ When encrypted content affinity is on for the group, the router sends the request to the
+ reasoning's origin, so reasoning one target can decrypt stays; otherwise every target must
+ decrypt it (was reading ``optional_callbacks`` for ``EncryptedContentAffinityCheck``).
+ """
+ return self._router._core_decryptable_reasoning(model_group, request_input, targets)
+
# Settings a caller may read. Assigning one from outside the router goes through apply(ChangeSet(assign=...)).
SETTING_FIELDS: Final[tuple[str, ...]] = (
diff --git a/litellm/router_rust/steps/callbacks.py b/litellm/router_rust/steps/callbacks.py
index 75afe38548e..c96a9c1bbae 100644
--- a/litellm/router_rust/steps/callbacks.py
+++ b/litellm/router_rust/steps/callbacks.py
@@ -407,6 +407,28 @@ class OptionalChecks:
bound: Final = request.get("litellm_logging_obj") in self._bound_requests
return {"success": bound and not sync, "failure": bound, "make_call": make_call}
+ def decryptable_reasoning(
+ self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
+ ) -> tuple[object, ...]:
+ """``registry().decryptable_reasoning``: one target decrypting is enough only when encrypted
+ content affinity pins ``model_group`` to the reasoning's origin."""
+ target_dicts: Final = tuple(target.model_dump(exclude_none=True) for target in targets)
+ pinned: Final = any(
+ isinstance(callback, EncryptedContentAffinityCheck) and callback.is_enabled_for_model_group(model_group)
+ for callback in self.optional_callbacks or ()
+ )
+ target_groups: Final = (target_dicts,) if pinned else tuple((target,) for target in target_dicts)
+ kept: Final = list(request_input)
+ for group in target_groups:
+ EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
+ self._ctx, # pyright: ignore[reportArgumentType] # the core's RoutingContext serves registry()
+ kept,
+ None,
+ group,
+ unmarked_origin=None,
+ )
+ return tuple(kept)
+
def routing_callbacks(self) -> Iterable[object]:
"""``_routing_callbacks``: the host's callbacks, or after ``observe(request_scoped=True)`` the
host's callbacks with the router's own selectors and policies in place of theirs."""
diff --git a/litellm/router_rust/surface/model_info.py b/litellm/router_rust/surface/model_info.py
index b1649617557..508d2bb1145 100644
--- a/litellm/router_rust/surface/model_info.py
+++ b/litellm/router_rust/surface/model_info.py
@@ -46,6 +46,7 @@ from litellm.litellm_core_utils.add_retry_fallback_headers import (
from litellm.litellm_core_utils.batch_utils import deployment_supported_endpoints
from litellm.litellm_core_utils.core_helpers import as_str_mapping, coerce_token_limit
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
+from litellm.litellm_core_utils.litellm_logging import resolve_vertex_location_for_cost
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_is_catalog_mapped,
intersect_supported_reasoning_efforts,
@@ -56,11 +57,13 @@ from litellm.llms.anthropic.common_utils import AnthropicModelInfo
from litellm.llms.bedrock.common_utils import get_bedrock_base_model
from litellm.redis.stores.router_usage import ROUTER_USAGE_TARGET, RouterUsage, UsageCounter
from litellm.router_python_host import python_host
+from litellm.router_utils.model_group_pricing import deployment_pricing
from litellm.router_utils.pre_call_checks.io_token_rate_limit_check import build_io_token_rate_limit_headers
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
from litellm.types.router import (
Deployment,
DeploymentModelListingInfo,
+ DeploymentPricing,
DeploymentTypedDict,
DiscoveredDeploymentModelInfo,
LiteLLM_Params,
@@ -78,6 +81,7 @@ if TYPE_CHECKING:
)
_MODEL_INFO_ADAPTER: Final = TypeAdapter(Mapping[str, object])
+_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
# The key router-core's merge reads as an opaque value (``router-core/src/model_info.rs``).
_OPAQUE_KEY: Final = "__litellm_python_value__"
@@ -140,6 +144,11 @@ def _configured_model_info(deployment: Mapping[str, object]) -> Mapping[str, obj
return _MODEL_INFO_ADAPTER.validate_python(deployment.get("model_info") or {})
+def _configured_deployment_id(deployment: Mapping[str, object]) -> str | None:
+ deployment_id: Final = _configured_model_info(deployment).get("id")
+ return deployment_id if isinstance(deployment_id, str) else None
+
+
def _operator_max_input_tokens(deployment: Mapping[str, object]) -> int | None:
"""``_operator_max_input_tokens``: the deployment's own limit, skipping one a saved
``/model/info`` response copied from the cost map."""
@@ -187,6 +196,7 @@ class _GroupBuild:
# Filled as the lookups arrive.
params: dict[int, LiteLLM_Params] = dataclasses.field(default_factory=dict) # mutable-ok: see above
info: ModelGroupInfo | None = None
+ pricing: tuple[DeploymentPricing, ...] = ()
def _usage_total(values: Sequence[object]) -> int | None:
@@ -358,6 +368,7 @@ class HostModelInfo:
return None
for field, value in (as_str_mapping(answer["assignments"]) or {}).items():
setattr(group.info, field, _group_field(field, wire.from_wire(value)))
+ group.info.deployment_pricing = group.pricing
return group.info
def _answer_group_lookup(self, step: Mapping[str, object], group: "_GroupBuild", wire: "_Wire") -> object:
@@ -384,6 +395,17 @@ class HostModelInfo:
efforts: Final = resolve_supported_reasoning_efforts(
model_info, deployment_is_mapped=step["mapped"] is True
)
+ pricing: Final = deployment_pricing(
+ llm_provider,
+ _configured_deployment_id(deployment),
+ group.params[int(str(step["index"]))].model,
+ model_info,
+ resolve_vertex_location_for_cost(
+ llm_provider, deployment.get("litellm_params"), None, litellm_model
+ ),
+ )
+ if pricing is not None and pricing not in group.pricing:
+ group.pricing = (*group.pricing, pricing) # rebind-ok: the merge's state
return {
"supported_endpoints": wire.to_wire(
list(
@@ -740,6 +762,20 @@ class HostModelInfo:
)
return tuple(wire.from_wire(tier) for tier in tiers)
+ def _core_get_pinned_reasoning_efforts(self, model_name: str, team_id: str | None = None) -> tuple[object, ...]:
+ """
+ Return the litellm_params.reasoning_effort each deployment a request
+ from team_id to model_name can route to (see _routable_deployments)
+ pins, unvalidated and in model_list order, None for a deployment that
+ pins none; the caller decides how the deployments combine.
+
+ Returns an empty tuple for wildcard-expanded or unknown names.
+ """
+ return tuple(
+ _LITELLM_PARAMS_ADAPTER.validate_python(deployment["litellm_params"]).get("reasoning_effort")
+ for deployment in self._routable_deployments(model_name, team_id)
+ )
+
def _core_get_configured_max_input_tokens(
self, model_name: str, team_id: str | None = None
) -> tuple[int | None, ...]:
diff --git a/litellm/router_rust/surface/registry.py b/litellm/router_rust/surface/registry.py
index eaeac10cb5f..7c3abdcd45e 100644
--- a/litellm/router_rust/surface/registry.py
+++ b/litellm/router_rust/surface/registry.py
@@ -1510,6 +1510,14 @@ class HostRegistry(HostModelInfo):
return self.__getattr__("_core_prompt_caching_pin_cache") # pyright: ignore[reportReturnType] # raises RustCoreUnsupported
return self.checks.prompt_caching_pin_cache
+ def _core_decryptable_reasoning(
+ self, model_group: str, request_input: Sequence[object], targets: Sequence[Deployment]
+ ) -> tuple[object, ...]:
+ """The input ``targets`` can decrypt (``registry().decryptable_reasoning``)."""
+ if self.checks is None:
+ return self.__getattr__("_core_decryptable_reasoning") # pyright: ignore[reportReturnType] # raises RustCoreUnsupported
+ return self.checks.decryptable_reasoning(model_group, request_input, targets)
+
def replay_model_cost_registrations(self) -> None:
"""Re-register every served deployment's cost-map entries (``replay_model_cost_registrations``)."""
provider_unresolved: Final = self._provider_unresolved_deployments
diff --git a/litellm/router_utils/model_group_pricing.py b/litellm/router_utils/model_group_pricing.py
new file mode 100644
index 00000000000..afadbc3eb38
--- /dev/null
+++ b/litellm/router_utils/model_group_pricing.py
@@ -0,0 +1,69 @@
+import math
+from collections.abc import Mapping
+from typing import Final
+
+from pydantic import TypeAdapter, ValidationError
+
+from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info
+from litellm.litellm_core_utils.llm_cost_calc.utils import get_vertex_regional_endpoint_uplift
+from litellm.types.router import DeploymentPricing, PriceTier
+
+_RAW_PRICE_TIERS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
+
+
+def _numeric_text(value: str) -> float | None:
+ try:
+ parsed: Final = float(value)
+ except ValueError:
+ return None
+ return parsed if math.isfinite(parsed) else None
+
+
+def _cost_value(value: object) -> float | None:
+ match value:
+ case bool():
+ return None
+ case int() | float():
+ return float(value)
+ case str():
+ return _numeric_text(value)
+ case _:
+ return None
+
+
+def _cost_fields(fields: Mapping[str, object], token_uplift: float) -> Mapping[str, float]:
+ return {
+ name: cost * token_uplift if "token" in name else cost
+ for name, value in fields.items()
+ if "cost" in name and (cost := _cost_value(value)) is not None
+ }
+
+
+def _price_tiers(tiered_pricing: object, token_uplift: float) -> tuple[PriceTier, ...]:
+ try:
+ return tuple(
+ PriceTier.model_validate({"range": tier.get("range"), "costs": _cost_fields(tier, token_uplift)})
+ for tier in _RAW_PRICE_TIERS.validate_python(tiered_pricing)
+ )
+ except ValidationError:
+ return ()
+
+
+def deployment_pricing(
+ provider: str,
+ deployment_id: str | None,
+ deployment_model: str,
+ resolved_model_info: Mapping[str, object],
+ vertex_location: str | None,
+) -> DeploymentPricing | None:
+ """Prices from the entry billing charges this deployment at, so a cost-map variant its own pricing shadows
+ is never shown. A deployment that declares no pricing is billed at its resolved cost-map entry. Token prices
+ carry the Vertex regional-endpoint uplift billing applies at the deployment's location."""
+ billed_model_info: Final = deployment_pricing_model_info(model_id=deployment_id, deployment_model=deployment_model)
+ model_info: Final = resolved_model_info if billed_model_info is None else billed_model_info
+ token_uplift: Final = get_vertex_regional_endpoint_uplift(model_info, vertex_location)
+ costs: Final = _cost_fields(model_info, token_uplift)
+ tiers: Final = _price_tiers(model_info.get("tiered_pricing"), token_uplift)
+ if not costs and not tiers:
+ return None
+ return DeploymentPricing(provider=provider, costs=costs, tiered_pricing=tiers)
diff --git a/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py b/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py
index 062886b44fd..9badee854fb 100644
--- a/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py
+++ b/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py
@@ -148,7 +148,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
return any("encrypted_content_affinity" in checks for checks in model_group_affinity_config.values())
- def _is_enabled_for_model_group(self, model_group: str) -> bool:
+ def is_enabled_for_model_group(self, model_group: str) -> bool:
group_checks: Final = self.model_group_affinity_config.get(model_group)
return self.enable_global_affinity or (
group_checks is not None and "encrypted_content_affinity" in group_checks
@@ -530,7 +530,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
"""
request_kwargs = request_kwargs or {}
typed_healthy_deployments: Final = cast(list[dict[str, object]], healthy_deployments)
- if not self._is_enabled_for_model_group(model):
+ if not self.is_enabled_for_model_group(model):
return typed_healthy_deployments
typed_request_kwargs: Final = _REQUEST_KWARGS_ADAPTER.validate_python(request_kwargs)
target_order: Final = typed_request_kwargs.get("_target_order")
diff --git a/litellm/rust_bridge/catalog.py b/litellm/rust_bridge/catalog.py
index 9ba8cbc35a7..9adfcb60d74 100644
--- a/litellm/rust_bridge/catalog.py
+++ b/litellm/rust_bridge/catalog.py
@@ -110,6 +110,7 @@ class Feature(str, Enum):
"""A model_list that Python fans out across its deployments."""
MODEL_ALIAS_MAP = "model_alias_map"
"""A model that litellm.model_alias_map rewrites before the call."""
+ LATEST_ALIAS = "latest_alias"
@dataclass(frozen=True, slots=True)
@@ -153,6 +154,7 @@ COMMON_PYTHON_FEATURES: Final = frozenset(
Feature.TIMEOUT_VARIANTS,
Feature.AGENTIC_LOOP_HOOK,
Feature.TLS_POLICY,
+ Feature.LATEST_ALIAS,
}
)
diff --git a/litellm/rust_bridge/chat_completions/detectors.py b/litellm/rust_bridge/chat_completions/detectors.py
index 5aa65a945b1..eb8f8bc49f9 100644
--- a/litellm/rust_bridge/chat_completions/detectors.py
+++ b/litellm/rust_bridge/chat_completions/detectors.py
@@ -38,7 +38,8 @@ def model_list(request: NativeCall) -> bool:
def model_alias_map(request: NativeCall) -> bool:
"""Python rewrites this model through the configured alias map."""
- return bool(litellm.model_alias_map) and str(request.resolved["model"]) in litellm.model_alias_map
+ model: Final = str(request.resolved["model"])
+ return model in litellm.model_alias_map
def chat_retries_fallbacks(request: NativeCall) -> bool:
diff --git a/litellm/rust_bridge/detectors.py b/litellm/rust_bridge/detectors.py
index 0af3a2e6fa6..f2591bdbcd1 100644
--- a/litellm/rust_bridge/detectors.py
+++ b/litellm/rust_bridge/detectors.py
@@ -12,6 +12,7 @@ import httpx
import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.core_helpers import normalize_drop_params
+from litellm.litellm_core_utils.model_latest_aliases import is_model_latest_alias
from litellm.rust_bridge.catalog import Feature
from litellm.rust_bridge.public_call import NativeCall, optional_mapping, optional_sequence, optional_str
@@ -35,6 +36,13 @@ def mock_response(request: NativeCall) -> bool:
return present(request, "mock_response")
+def latest_alias(request: NativeCall) -> bool:
+ return is_model_latest_alias(
+ optional_str(request.resolved.get("model")) or "",
+ optional_str(request.resolved.get("custom_llm_provider")),
+ )
+
+
def caller_client(request: NativeCall) -> bool:
"""Python sends through the caller's HTTP client or shared session."""
return present(request, "client") or present(request, "shared_session")
@@ -179,5 +187,6 @@ COMMON: Final[Detectors] = MappingProxyType(
Feature.TIMEOUT_VARIANTS: timeout_variants,
Feature.AGENTIC_LOOP_HOOK: agentic_loop_hook,
Feature.TLS_POLICY: tls_policy,
+ Feature.LATEST_ALIAS: latest_alias,
}
)
diff --git a/litellm/tool_loop.py b/litellm/tool_loop.py
index d0b2850bb5b..554fdff4fce 100644
--- a/litellm/tool_loop.py
+++ b/litellm/tool_loop.py
@@ -120,9 +120,12 @@ def _replays_provider_specific_fields(
client_tool_calls: tuple[ChatCompletionMessageToolCall, ...],
) -> bool:
from litellm.completion_extras.litellm_responses_transformation.transformation import carries_tool_search_items
+ from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
- return client_tool_calls != tool_calls or carries_tool_search_items(
- getattr(message, "provider_specific_fields", None)
+ return (
+ client_tool_calls != tool_calls
+ or bool(anthropic_server_tool_call_params(message))
+ or carries_tool_search_items(getattr(message, "provider_specific_fields", None))
)
diff --git a/litellm/types/llms/anthropic.py b/litellm/types/llms/anthropic.py
index 802f6794839..c322eb25b3e 100644
--- a/litellm/types/llms/anthropic.py
+++ b/litellm/types/llms/anthropic.py
@@ -2,10 +2,12 @@ from collections.abc import Iterable, Mapping, Sequence
from enum import Enum
from typing import Final, Literal, TypeAlias
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
from pydantic import ConfigDict
from typing_extensions import NotRequired, ReadOnly, Required, TypedDict
from litellm.types.llms.base import LiteLLMBaseModel
+from litellm.types.responses.main import hosted_web_tool_call, web_search_call_tool_id
from .openai import (
ChatCompletionCachedContent,
@@ -862,6 +864,15 @@ ANTHROPIC_TOOL_SEARCH_TOOL_TYPES: Final = frozenset(
ANTHROPIC_TOOL_SEARCH_TOOL_NAMES: Final = frozenset({"tool_search_tool_regex", "tool_search_tool_bm25"})
ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX: Final = "srvtoolu_"
+
+def claude_hosted_web_tool_call(search: ResponseFunctionWebSearch) -> tuple[str, str, str] | None:
+ tool_id: Final = web_search_call_tool_id(search.id)
+ if tool_id is None or not tool_id.startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX):
+ return None
+ hosted_call: Final = hosted_web_tool_call(search.action)
+ return None if hosted_call is None else (tool_id, *hosted_call)
+
+
# Effort beta header constant
ANTHROPIC_EFFORT_BETA_HEADER: Final = "effort-2025-11-24"
diff --git a/litellm/types/responses/main.py b/litellm/types/responses/main.py
index 86dc857cfd0..7a2f58e9c16 100644
--- a/litellm/types/responses/main.py
+++ b/litellm/types/responses/main.py
@@ -1,11 +1,18 @@
import builtins
+import json
from collections.abc import Mapping, Sequence
from typing import Final, Literal, Optional, Union
from openai.types.responses.response_function_tool_call import ResponseFunctionToolCall
-from openai.types.responses.response_function_web_search import ActionSearchSource, ResponseFunctionWebSearch
+from openai.types.responses.response_function_web_search import (
+ Action,
+ ActionOpenPage,
+ ActionSearch,
+ ActionSearchSource,
+ ResponseFunctionWebSearch,
+)
from openai.types.responses.response_output_refusal import ResponseOutputRefusal
-from pydantic import ConfigDict, PrivateAttr
+from pydantic import ConfigDict, PrivateAttr, TypeAdapter, ValidationError
from typing_extensions import TypedDict
from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
@@ -43,14 +50,11 @@ class OutputFunctionToolCall(BaseLiteLLMOpenAIResponseObject):
phase: Phase = None
-def build_web_search_call(
- tool_id: str,
- tool_input: object,
- result: object,
- status: Literal["in_progress", "searching", "completed", "failed"] | None = None,
-) -> ResponseFunctionWebSearch:
+HOSTED_WEB_TOOL_NAMES: Final = frozenset({"web_search", "web_fetch"})
+
+
+def _web_search_action(tool_input: object, content: object) -> ActionSearch:
query: Final = tool_input.get("query", "") if isinstance(tool_input, Mapping) else ""
- content: Final = result.get("content") if isinstance(result, Mapping) else None
result_items: Final = content if isinstance(content, Sequence) and not isinstance(content, (str, bytes)) else ()
sources: Final = [
ActionSearchSource(type="url", url=url)
@@ -59,20 +63,96 @@ def build_web_search_call(
and item.get("type") == "web_search_result"
and isinstance((url := item.get("url")), str)
]
- failed: Final = isinstance(content, Mapping) and content.get("type") == "web_search_tool_result_error"
- return ResponseFunctionWebSearch(
- id=f"ws_{tool_id}",
- type="web_search_call",
- status=status or ("failed" if failed else "completed"),
- action={
- "type": "search",
- "query": query if isinstance(query, str) else "",
- "queries": [query] if isinstance(query, str) and query else [],
- "sources": sources,
- },
+ return ActionSearch(
+ type="search",
+ query=query if isinstance(query, str) else "",
+ queries=[query] if isinstance(query, str) and query else [],
+ sources=sources,
)
+def _web_fetch_action(tool_input: object) -> ActionOpenPage:
+ match tool_input:
+ case {"url": str() as url}:
+ return ActionOpenPage(type="open_page", url=url)
+ case _:
+ return ActionOpenPage(type="open_page", url=None)
+
+
+def _result_content(result: object) -> object:
+ match result:
+ case {"content": object() as content}:
+ return content
+ case _:
+ return None
+
+
+def _is_tool_result_error(content: object, tool_name: str) -> bool:
+ match content:
+ case {"type": str() as content_type}:
+ return content_type == f"{tool_name}_tool_result_error"
+ case _:
+ return False
+
+
+WEB_SEARCH_CALL_ID_PREFIX: Final = "ws_"
+
+
+def build_web_search_call(
+ tool_id: str,
+ tool_input: object,
+ result: object,
+ status: Literal["in_progress", "searching", "completed", "failed"] | None = None,
+ tool_name: str = "web_search",
+) -> ResponseFunctionWebSearch:
+ content: Final = _result_content(result)
+ return ResponseFunctionWebSearch(
+ id=f"{WEB_SEARCH_CALL_ID_PREFIX}{tool_id}",
+ type="web_search_call",
+ status=status or ("failed" if _is_tool_result_error(content, tool_name) else "completed"),
+ action=_web_fetch_action(tool_input) if tool_name == "web_fetch" else _web_search_action(tool_input, content),
+ )
+
+
+def web_search_call_tool_id(call_id: str) -> str | None:
+ tool_id: Final = call_id.removeprefix(WEB_SEARCH_CALL_ID_PREFIX)
+ return None if tool_id == call_id else tool_id
+
+
+_HOSTED_WEB_TOOL_ARGUMENTS: Final = TypeAdapter(dict[str, object])
+
+
+def hosted_web_tool_call(action: object) -> tuple[str, str] | None:
+ match action:
+ case ActionSearch(query=str() as query):
+ return "web_search", json.dumps({"query": query})
+ case ActionOpenPage(url=str() as url):
+ return "web_fetch", json.dumps({"url": url})
+ case _:
+ return None
+
+
+def _decoded_hosted_web_tool_arguments(arguments: str) -> Mapping[str, object]:
+ try:
+ return _HOSTED_WEB_TOOL_ARGUMENTS.validate_json(arguments)
+ except ValidationError:
+ return {}
+
+
+def with_hosted_web_tool_arguments(action: Action, arguments: str) -> Action:
+ match action, _decoded_hosted_web_tool_arguments(arguments):
+ case ActionSearch(), {"query": str() as query}:
+ return action.model_copy(update={"query": query, "queries": [query] if query else []})
+ case ActionSearch(), _:
+ return action.model_copy(update={"query": "", "queries": []})
+ case ActionOpenPage(), {"url": str() as url}:
+ return action.model_copy(update={"url": url})
+ case ActionOpenPage(), _:
+ return action.model_copy(update={"url": None})
+ case _:
+ return action
+
+
class OutputImageGenerationCall(BaseLiteLLMOpenAIResponseObject):
"""An image generation call output"""
diff --git a/litellm/types/router.py b/litellm/types/router.py
index a9f28fbc673..3dcfc694003 100644
--- a/litellm/types/router.py
+++ b/litellm/types/router.py
@@ -1041,6 +1041,21 @@ def _resolved_annotations(model_class: type[object]) -> Mapping[str, object]:
return MappingProxyType(get_type_hints(model_class))
+class PriceTier(LiteLLMBaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ range: tuple[float, float]
+ costs: Mapping[str, float]
+
+
+class DeploymentPricing(LiteLLMBaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ provider: str
+ costs: Mapping[str, float]
+ tiered_pricing: tuple[PriceTier, ...] = ()
+
+
class ModelGroupInfo(LiteLLMBaseModel):
model_group: str
providers: list[str]
@@ -1071,6 +1086,7 @@ class ModelGroupInfo(LiteLLMBaseModel):
configurable_clientside_auth_params: CONFIGURABLE_CLIENTSIDE_AUTH_PARAMS = None
description: str | None = None
supported_endpoints: tuple[str, ...] | None = None
+ deployment_pricing: tuple[DeploymentPricing, ...] = ()
def __init__(self, **data) -> None:
for field_name, field_type in _resolved_annotations(self.__class__).items():
diff --git a/litellm/types/tag_management.py b/litellm/types/tag_management.py
index e59b02af71a..2958c07ce95 100644
--- a/litellm/types/tag_management.py
+++ b/litellm/types/tag_management.py
@@ -27,6 +27,7 @@ class TagNewRequest(TagBase):
class TagUpdateRequest(TagBase):
+ spend: float | None = None
budget_id: str | None = None
# Budget fields - if provided, will update the budget
max_budget: float | None = None
diff --git a/litellm/types/utils.py b/litellm/types/utils.py
index 5a14dc95e7c..b082198d3f2 100644
--- a/litellm/types/utils.py
+++ b/litellm/types/utils.py
@@ -217,6 +217,7 @@ class ProviderSpecificModelInfo(TypedDict, total=False):
supports_native_streaming: bool | None
supports_native_structured_output: bool | None
supports_bedrock_native_structured_output: ReadOnly[bool | None]
+ supports_bedrock_native_structured_output_by_default: ReadOnly[bool | None]
supports_parallel_function_calling: bool | None
supports_web_search: bool | None
supports_reasoning: bool | None
@@ -4303,6 +4304,7 @@ DEPLOYMENT_SCOPED_CAPABILITY_FIELDS: Final[frozenset[str]] = frozenset(
"supports_regex_lookaround",
"supports_assistant_prefill",
"supports_bedrock_native_structured_output",
+ "supports_bedrock_native_structured_output_by_default",
"supports_per_turn_control",
"supports_pdf_input",
"supports_continuation",
diff --git a/litellm/utils.py b/litellm/utils.py
index 8944ecb5648..13f22ddc36e 100644
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -4167,6 +4167,13 @@ def _registration_base(
return _RegistrationWrite(key=key, entry=dict(fallback.entry))
+def native_structured_output_support(model_info: Mapping[str, object]) -> bool | None:
+ if model_info.get("supports_bedrock_native_structured_output_by_default"):
+ return model_info.get("supports_bedrock_native_structured_output") is not False
+ declared: Final = model_info.get("supports_native_structured_output")
+ return declared if isinstance(declared, bool) else None
+
+
def _base_fields_replaced_by(value: Mapping[str, object]) -> frozenset[str]:
return (
frozenset({"explicit_cache_read_input_token_cost"})
@@ -7587,10 +7594,13 @@ def get_model_info_helper(
supports_pdf_input=_model_info.get("supports_pdf_input", None),
supports_embedding_image_input=_model_info.get("supports_embedding_image_input", None),
supports_native_streaming=_model_info.get("supports_native_streaming", None),
- supports_native_structured_output=_model_info.get("supports_native_structured_output", None),
+ supports_native_structured_output=native_structured_output_support(_model_info),
supports_bedrock_native_structured_output=_model_info.get(
"supports_bedrock_native_structured_output", None
),
+ supports_bedrock_native_structured_output_by_default=_model_info.get(
+ "supports_bedrock_native_structured_output_by_default", None
+ ),
supports_web_search=_model_info.get("supports_web_search", None),
supports_url_context=_model_info.get("supports_url_context", None),
supports_continuation=_model_info.get("supports_continuation", None),
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
index 95adfda8c49..4796ed3ac9c 100644
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -1961,6 +1961,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2041,6 +2042,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2116,6 +2118,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
@@ -2242,7 +2245,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2287,6 +2291,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2331,7 +2336,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2376,6 +2382,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2420,7 +2427,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2465,6 +2473,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2504,7 +2513,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2549,6 +2559,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2593,7 +2604,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2633,6 +2645,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -2715,7 +2728,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2759,7 +2773,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2803,7 +2818,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2847,7 +2863,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -2891,7 +2908,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3023,6 +3041,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3068,6 +3087,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3113,6 +3133,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3158,6 +3179,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -3203,6 +3225,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -12097,16 +12120,19 @@
},
"azure/tts-1": {
"deprecation_date": "2026-12-15",
+ "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
"input_cost_per_character": 1.5e-05,
"litellm_provider": "azure",
- "mode": "audio_speech"
+ "mode": "audio_speech",
+ "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
},
"azure/tts-1-hd": {
"deprecation_date": "2026-12-15",
+ "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
"input_cost_per_character": 3.3e-05,
"litellm_provider": "azure",
"mode": "audio_speech",
- "source": "https://azure.microsoft.com/en-us/pricing/details/azure-openai/"
+ "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
},
"azure/us/gpt-4.1-2025-04-14": {
"deprecation_date": "2027-04-14",
@@ -12859,11 +12885,13 @@
"supports_vision": true
},
"azure/whisper-1": {
- "deprecation_date": "2026-12-15",
+ "deprecation_date": "2027-02-26",
+ "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
"input_cost_per_second": 0.0001,
"litellm_provider": "azure",
"mode": "audio_transcription",
- "output_cost_per_second": 0.0001
+ "output_cost_per_second": 0.0001,
+ "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
},
"azure_ai/Cohere-embed-v3-english": {
"deprecation_date": "2027-02-15",
@@ -31630,7 +31658,7 @@
"input_cost_per_audio_token_batches": 2.5e-07
},
"gemini/gemini-embedding-2-preview": {
- "deprecation_date": "2026-08-10",
+ "deprecation_date": "2026-11-06",
"deprecation_source": "https://ai.google.dev/gemini-api/docs/deprecations",
"input_cost_per_audio_token": 6.5e-06,
"input_cost_per_audio_token_batches": 3.25e-06,
@@ -44089,6 +44117,7 @@
"supports_vision": true
},
"mistral/mistral-large-2512": {
+ "deprecation_date": "2027-04-30",
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 5e-07,
"litellm_provider": "mistral",
@@ -44164,6 +44193,7 @@
"supports_vision": true
},
"mistral/mistral-medium-3-5": {
+ "deprecation_date": "2027-04-30",
"cache_read_input_token_cost": 1.5e-07,
"input_cost_per_token": 1.5e-06,
"litellm_provider": "mistral",
@@ -45679,6 +45709,7 @@
"o1": {
"cache_read_input_token_cost": 7.5e-06,
"deprecation_date": "2026-10-23",
+ "deprecation_source": "https://developers.openai.com/api/docs/deprecations",
"successor_model": "gpt-5.6-sol",
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
@@ -64725,6 +64756,7 @@
"source": "https://docs.mistral.ai/models/model-cards/mistral-moderation-26-03"
},
"mistral/voxtral-mini-2602": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_second": 5e-05,
"litellm_provider": "mistral",
"mode": "audio_transcription",
@@ -64741,6 +64773,7 @@
"supports_audio_input": true
},
"mistral/voxtral-mini-transcribe-realtime-2602": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_second": 0.0001,
"litellm_provider": "mistral",
"mode": "audio_transcription",
@@ -64757,6 +64790,7 @@
"supports_audio_input": true
},
"mistral/voxtral-mini-tts-2603": {
+ "deprecation_date": "2027-04-30",
"input_cost_per_character": 1.6e-05,
"litellm_provider": "mistral",
"mode": "audio_speech",
@@ -75543,7 +75577,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -75583,7 +75618,8 @@
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
- "supports_native_structured_output": true,
+ "supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -75623,6 +75659,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
@@ -75696,6 +75733,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -76994,13 +77032,13 @@
"supports_vision": true
},
"azure/whisper": {
- "deprecation_date": "2026-12-15",
- "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
+ "deprecation_date": "2027-02-26",
+ "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
"input_cost_per_second": 0.0001,
"litellm_provider": "azure",
"mode": "audio_transcription",
"output_cost_per_second": 0.0001,
- "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
+ "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
},
"gemini/gemini-nano-banana-2.1": {
"input_cost_per_token": 1.5e-06,
@@ -77411,6 +77449,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77468,6 +77507,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77645,6 +77685,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77697,6 +77738,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77744,6 +77786,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
@@ -77857,6 +77900,7 @@
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
+ "supports_bedrock_native_structured_output_by_default": true,
"supports_bedrock_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
diff --git a/osv-scanner.toml b/osv-scanner.toml
index 11c2d4fbd87..7097188c3c7 100644
--- a/osv-scanner.toml
+++ b/osv-scanner.toml
@@ -1,86 +1,31 @@
-[[IgnoredVulns]]
-id = "GHSA-v4x8-3wr5-3wq8"
-ignoreUntil = 2026-11-20
-reason = "filelock ReadWriteLock advisory, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-567q-5qvw-pf92"
-ignoreUntil = 2026-11-20
-reason = "Windows-only local DoS (CVSS 2.5) in filelock soft locks; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-9vhv-jxww-xfm5"
-ignoreUntil = 2026-11-20
-reason = "Windows-only local DoS (CVSS 3.3) in filelock ReadWriteLock, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-6627-m3xw-93mp"
-ignoreUntil = 2026-11-20
-reason = "filelock StrictSoftFileLock claim directories ignore the lock mode, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
[[IgnoredVulns]]
id = "GHSA-69hj-f68j-2879"
-ignoreUntil = 2026-11-20
-reason = "filelock UnixFileLock truncates and chmods a file another user renames or hard-links in at the lock path; litellm's CLI takes FileLock only inside ~/.litellm, which it creates owner-only, and huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting; the attack needs another local user who can rename or link files in that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
+ignoreUntil = 2027-02-01
+reason = "filelock UnixFileLock truncates and chmods a file another user renames or hard-links in at the lock path, in every release before 4.2.1, which needs Python >= 3.11. Python 3.11+ now locks 4.2.4 and only the Python 3.10 resolution stays on 3.20.3, held below 3.20.4 by a [tool.uv] constraint because every newer 3.10-compatible release carries more filelock advisories. litellm's CLI takes FileLock only inside ~/.litellm, which it creates owner-only, and huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, so the attack needs another local user who can rename or link files in that cache. Remove once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-8mx5-7m5m-r77g"
-ignoreUntil = 2026-11-20
-reason = "filelock WindowsFileLock ignores mode; the proxy image is Linux, so only Windows installs run it; there litellm's CLI takes FileLock only inside the owner-only ~/.litellm, whose lock file holds no data, and huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-9c4x-gm8v-wgc4"
-ignoreUntil = 2026-11-20
-reason = "filelock StrictSoftFileLock and SoftReadWriteLock adopt another user's protocol directory, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-9j6q-wvj5-jgjg"
-ignoreUntil = 2026-11-20
-reason = "filelock SoftFileLock symlink swap leaks whether a path exists; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-cwqr-hh33-gh44"
-ignoreUntil = 2026-11-20
-reason = "filelock ReadWriteLock stalls on a crafted lock database, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-fq56-3cx3-fjxp"
-ignoreUntil = 2026-11-20
-reason = "filelock SoftFileLock stale-lock break removes a live lock on a reused inode; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-j8f7-rjxc-mr56"
-ignoreUntil = 2026-11-20
-reason = "filelock ReadWriteLock follows a symlink into SQLite, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed in 4.0.9, which litellm's own filelock<4.0 cap blocks on every Python until it is lifted (#48635, #48802); remove once that lands"
-
-[[IgnoredVulns]]
-id = "GHSA-m462-7rxw-wvpg"
-ignoreUntil = 2026-11-20
-reason = "filelock StrictSoftFileLock follows a symlink swapped in at .filelock, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
-
-[[IgnoredVulns]]
-id = "GHSA-rfpc-qcw5-3pc4"
-ignoreUntil = 2026-11-20
-reason = "filelock SoftFileLock removes a live holder's marker when another user forges its PID; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
+ignoreUntil = 2027-02-01
+reason = "filelock WindowsFileLock ignores mode in every release from 3.10.0 before 4.2.1, which needs Python >= 3.11. Python 3.11+ now locks 4.2.4 and only the Python 3.10 resolution stays on 3.20.3, held below 3.20.4 by a [tool.uv] constraint because every newer 3.10-compatible release carries more filelock advisories. The proxy image is Linux, so only Windows installs on Python 3.10 run it, and there litellm's CLI takes FileLock only inside the owner-only ~/.litellm, whose lock file holds no data, while huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer. Remove once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-pcg2-g6cj-q92c"
-ignoreUntil = 2026-11-27
+ignoreUntil = 2027-02-01
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's HTTP header parsing is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-pw9p-v7r6-gx4p"
-ignoreUntil = 2026-11-27
+ignoreUntil = 2027-02-01
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's RedirectHandler is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-r4j5-9xpp-3985"
-ignoreUntil = 2026-11-27
+ignoreUntil = 2027-02-01
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's chunked body reading is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-45c3-73f7-pv4m"
-ignoreUntil = 2026-11-27
+ignoreUntil = 2027-02-01
reason = "dnspython 2.9.0 (published 2026-10-09) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 2.8.0 while 3.11+ locks 2.9.0; the advisory is CPU use in dns.name.from_wire_parser and litellm only reaches dnspython through pydantic EmailStr, which never does DNS lookups; remove this entry once 3.10 is dropped or a 3.10-compatible fix ships"
[[PackageOverrides]]
@@ -88,5 +33,5 @@ name = "pillow"
version = "12.3.0"
ecosystem = "PyPI"
vulnerability.ignore = true
-effectiveUntil = 2026-11-27
-reason = "pillow 13.0.0 requires Python >= 3.11, so the Python 3.10 resolution of the ci and dev groups and of the mlflow extra stays on 12.3.0; 3.11+ is on 13.0.0; pillow has never shipped an older-major release after the next major (no 10.x after 11.0.0, no 11.x after 12.0.0), so 13.x is the only fix line; extended to 2026-11-27 with maintainer approval, same date as the dnspython 3.10 entry; remove once 3.10 is dropped"
+effectiveUntil = 2027-02-01
+reason = "pillow 13.0.0 requires Python >= 3.11, so the Python 3.10 resolution of the ci and dev groups and of the mlflow extra stays on 12.3.0; 3.11+ is on 13.0.0; pillow has never shipped an older-major release after the next major (no 10.x after 11.0.0, no 11.x after 12.0.0), so 13.x is the only fix line; extended to 2027-02-01 with maintainer approval, same date as every other Python 3.10 entry; remove once 3.10 is dropped"
diff --git a/pyproject.toml b/pyproject.toml
index 47bed23c093..c53fbb4d888 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -378,6 +378,8 @@ constraint-dependencies = [
"httplib2>=0.32.0",
"setuptools>=84.1.0",
"dnspython>=2.9.0; python_version >= '3.11'",
+ "filelock<3.20.4; python_version < '3.11'",
+ "filelock>=4.2.1; python_version >= '3.11'",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.
diff --git a/tests/e2e/ui/tests/integrationCritical/mcpUserEnvVars.spec.ts b/tests/e2e/ui/tests/integrationCritical/mcpUserEnvVars.spec.ts
index 11cc1c948a3..b37ff903121 100644
--- a/tests/e2e/ui/tests/integrationCritical/mcpUserEnvVars.spec.ts
+++ b/tests/e2e/ui/tests/integrationCritical/mcpUserEnvVars.spec.ts
@@ -380,15 +380,19 @@ test("clearing credentials for a server deleted underneath the modal reports the
await openMcpServers(page);
const card = cardFor(page, server);
const dialog = credentialsDialog(page);
+ const clear = dialog.getByRole("button", { name: "Clear", exact: true });
await card.getByRole("button", { name: "Update", exact: true }).click();
- await expect(dialog).toBeVisible();
+ await expect(
+ clear,
+ "the modal must load the saved credentials before the server is deleted",
+ ).toBeVisible();
await server.remove();
const cleared = page.waitForResponse(
(response) =>
response.request().method() === "DELETE" &&
response.url().includes(server.statusUrl),
);
- await dialog.getByRole("button", { name: "Clear", exact: true }).click();
+ await clear.click();
await page
.getByRole("alertdialog", { name: "Clear saved credentials" })
.getByRole("button", {
diff --git a/tests/integration/_support/codex_vendor.py b/tests/integration/_support/codex_vendor.py
index 2ec9c4e8b96..a588fa8db3d 100644
--- a/tests/integration/_support/codex_vendor.py
+++ b/tests/integration/_support/codex_vendor.py
@@ -46,6 +46,9 @@ FORWARDED_KEYS: Final = frozenset(
"reasoning",
"previous_response_id",
"truncation",
+ "text",
+ "service_tier",
+ "max_output_tokens",
}
)
diff --git a/tests/integration/cost_calculation/cost_map.py b/tests/integration/cost_calculation/cost_map.py
index 8f11b3ce2f1..ffdb6752bf2 100644
--- a/tests/integration/cost_calculation/cost_map.py
+++ b/tests/integration/cost_calculation/cost_map.py
@@ -1603,32 +1603,44 @@ COST_MAP_ENTRIES: Final[dict[str, dict[str, JsonValue]]] = {
},
"azure/gpt-6-luna": {
"cache_read_input_token_cost": 1.3e-08,
+ "cache_read_input_token_cost_flex": 6.5e-09,
"cache_creation_input_token_cost": 1.625e-07,
+ "cache_creation_input_token_cost_flex": 8.125e-08,
"cache_creation_input_token_cost_above_272k_tokens": 3.25e-07,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 1.625e-07,
"input_cost_per_token": 1.3e-07,
+ "input_cost_per_token_flex": 6.5e-08,
"litellm_provider": "azure",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.5e-07,
+ "output_cost_per_token_flex": 3.25e-07,
"supports_function_calling": True,
"supports_none_reasoning_effort": True,
"input_cost_per_token_above_272k_tokens": 2.6e-07,
+ "input_cost_per_token_above_272k_tokens_flex": 1.3e-07,
"output_cost_per_token_above_272k_tokens": 1.3e-06,
+ "output_cost_per_token_above_272k_tokens_flex": 4.875e-07,
"cache_read_input_token_cost_above_272k_tokens": 2.6e-08,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1.3e-08,
},
"azure/gpt-6-astra": {
"cache_read_input_token_cost": 1.25e-06,
+ "cache_read_input_token_cost_flex": 6.25e-07,
"cache_creation_input_token_cost": 1.5625e-05,
+ "cache_creation_input_token_cost_flex": 7.8125e-06,
"cache_creation_input_token_cost_above_272k_tokens": 3.125e-05,
"input_cost_per_token": 1.25e-05,
+ "input_cost_per_token_flex": 6.25e-06,
"litellm_provider": "azure",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.25e-05,
+ "output_cost_per_token_flex": 3.125e-05,
"supports_function_calling": True,
"supports_none_reasoning_effort": True,
"input_cost_per_token_above_272k_tokens": 2.5e-05,
@@ -1637,20 +1649,28 @@ COST_MAP_ENTRIES: Final[dict[str, dict[str, JsonValue]]] = {
},
"azure/gpt-6.1-sol": {
"cache_read_input_token_cost": 2.4e-07,
+ "cache_read_input_token_cost_flex": 6e-08,
"cache_creation_input_token_cost": 3e-06,
+ "cache_creation_input_token_cost_flex": 1.5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 6e-06,
+ "cache_creation_input_token_cost_above_272k_tokens_flex": 3e-06,
"input_cost_per_token": 2.4e-06,
+ "input_cost_per_token_flex": 1.2e-06,
"litellm_provider": "azure",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
+ "output_cost_per_token_flex": 6e-06,
"supports_function_calling": True,
"supports_none_reasoning_effort": True,
"input_cost_per_token_above_272k_tokens": 4.8e-06,
+ "input_cost_per_token_above_272k_tokens_flex": 2.4e-06,
"output_cost_per_token_above_272k_tokens": 2.4e-05,
+ "output_cost_per_token_above_272k_tokens_flex": 9e-06,
"cache_read_input_token_cost_above_272k_tokens": 4.8e-07,
+ "cache_read_input_token_cost_above_272k_tokens_flex": 1.2e-07,
},
}
diff --git a/tests/integration/management/test_tag_budget_update.py b/tests/integration/management/test_tag_budget_update.py
new file mode 100644
index 00000000000..c8ce1bdc8f4
--- /dev/null
+++ b/tests/integration/management/test_tag_budget_update.py
@@ -0,0 +1,20 @@
+from typing import Final
+from uuid import uuid4
+
+from tests.integration._support.client import Gateway, object_value, string_value
+
+
+def test_tag_budget_can_be_updated_with_its_existing_budget_id(gateway: Gateway) -> None:
+ name: Final = f"integration-tag-{uuid4().hex}"
+ gateway.post("/tag/new", {"name": name, "max_budget": 10})
+ with gateway.scenario() as scenario:
+ scenario.cleanups.callback(gateway.post, "/tag/delete", {"name": name})
+ before: Final = object_value(gateway.post("/tag/info", {"names": [name]})[name])
+ budget_id: Final = string_value(object_value(before["litellm_budget_table"])["budget_id"])
+ scenario.cleanups.callback(gateway.post, "/budget/delete", {"id": budget_id})
+ response: Final = gateway.request(
+ "POST", "/tag/update", {"name": name, "budget_id": budget_id, "max_budget": 20}
+ )
+ assert response.status_code == 200, response.text
+ after: Final = object_value(gateway.post("/tag/info", {"names": [name]})[name])
+ assert object_value(after["litellm_budget_table"])["max_budget"] == 20, after
diff --git a/tests/integration/providers/_tool_search_support.py b/tests/integration/providers/_tool_search_support.py
index fa6b53040e4..b2a156d238d 100644
--- a/tests/integration/providers/_tool_search_support.py
+++ b/tests/integration/providers/_tool_search_support.py
@@ -317,6 +317,11 @@ def tool_call_ids(assistant: Event) -> list[JsonValue]:
return [JSON_OBJECT.validate_python(call)["id"] for call in calls]
+def server_tool_call_ids(assistant: Event) -> list[JsonValue]:
+ calls: Final = JSON_LIST.validate_python(provider_fields(assistant).get("server_tool_calls") or [])
+ return [JSON_OBJECT.validate_python(call)["id"] for call in calls]
+
+
def wire_content(message: Event) -> list[JsonValue]:
content: Final = message["content"]
if isinstance(content, str):
diff --git a/tests/integration/providers/test_anthropic_tool_search_result_matrix_wire.py b/tests/integration/providers/test_anthropic_tool_search_result_matrix_wire.py
index d4c0e2651d8..adce80d3c75 100644
--- a/tests/integration/providers/test_anthropic_tool_search_result_matrix_wire.py
+++ b/tests/integration/providers/test_anthropic_tool_search_result_matrix_wire.py
@@ -66,6 +66,7 @@ from integration.providers._tool_search_support import (
response_id,
scripted_peer,
search_result,
+ server_tool_call_ids,
spend_row,
target,
target_path,
@@ -136,7 +137,8 @@ def test_tool_search_result_round_trips_on_every_anthropic_wire_provider(
assert [target_path(request) for request in received] == [target(provider, stream)] * 2, received
assert kept_results(assistant) == [SEARCH_RESULT], assistant
assert "code_interpreter_results" not in provider_fields(assistant), assistant
- assert tool_call_ids(assistant) == [SEARCH_ID, WEATHER_ID], assistant
+ assert server_tool_call_ids(assistant) == [SEARCH_ID], assistant
+ assert tool_call_ids(assistant) == [WEATHER_ID], assistant
assert wire_assistant(received[1]) == list(SEARCH_TURN), received[1].body
assert answered_ids(received[1]) == [WEATHER_ID], received[1].body
for identity in (response_id(first, stream), response_id(second, stream)):
@@ -157,7 +159,8 @@ def test_snowflake_claude_keeps_the_tool_search_result_on_the_response(gateway:
assistant: Final = assistant_from(first, stream)
assert [target_path(request) for request in received] == [target("snowflake", stream)], received
assert kept_results(assistant) == [SEARCH_RESULT], assistant
- assert tool_call_ids(assistant) == [SEARCH_ID, WEATHER_ID], assistant
+ assert server_tool_call_ids(assistant) == [SEARCH_ID], assistant
+ assert tool_call_ids(assistant) == [WEATHER_ID], assistant
assert spend_row(response_id(first, stream))["model_group"] == model
diff --git a/tests/integration/providers/test_anthropic_tool_search_result_wire.py b/tests/integration/providers/test_anthropic_tool_search_result_wire.py
index b46187e3e71..491cd5acbe8 100644
--- a/tests/integration/providers/test_anthropic_tool_search_result_wire.py
+++ b/tests/integration/providers/test_anthropic_tool_search_result_wire.py
@@ -1,6 +1,7 @@
import json
import uuid
from collections.abc import Sequence
+from functools import reduce
from typing import Final
from integration._support.anthropic_thinking import sse_chunks
@@ -142,13 +143,8 @@ def _streamed_assistant(lines: Sequence[str]) -> dict[str, JsonValue]:
entry["id"] = call.get("id") or entry["id"]
entry["name"] += call["function"].get("name") or ""
entry["arguments"] += call["function"].get("arguments") or ""
- fields: Final = next(
- (
- delta["provider_specific_fields"]
- for delta in reversed(deltas)
- if (delta.get("provider_specific_fields") or {}).get("tool_results")
- ),
- {},
+ fields: Final = reduce(
+ lambda merged, delta: {**merged, **(delta.get("provider_specific_fields") or {})}, deltas, {}
)
return {
"role": "assistant",
diff --git a/tests/integration/providers/test_bedrock_batch_files_wire.py b/tests/integration/providers/test_bedrock_batch_files_wire.py
index 823bcda7e06..162fc5aac27 100644
--- a/tests/integration/providers/test_bedrock_batch_files_wire.py
+++ b/tests/integration/providers/test_bedrock_batch_files_wire.py
@@ -88,10 +88,17 @@ _SCHEMA: Final = {
@pytest.mark.parametrize(
- "backend", ["us.anthropic.claude-haiku-5-5", "us.anthropic.claude-opus-4-8"], ids=["haiku_5_5", "opus_4_8"]
+ ("backend", "native_output"),
+ [
+ ("us.anthropic.claude-sonnet-5-5", False),
+ ("us.anthropic.claude-haiku-5-5", True),
+ ("us.anthropic.claude-opus-4-7", False),
+ ("us.anthropic.claude-opus-4-8", True),
+ ],
+ ids=["opt_in_native_output_model", "native_output_model", "opus_4_7", "opus_4_8"],
)
-def test_json_schema_batch_record_with_thinking_uploads_the_json_tool_and_no_native_output(
- gateway: Gateway, backend: str
+def test_json_schema_batch_record_with_thinking_uploads_native_output_only_on_models_that_always_take_it(
+ gateway: Gateway, backend: str, native_output: bool
) -> None:
record: Final = {
"custom_id": "schema-record",
@@ -125,7 +132,16 @@ def test_json_schema_batch_record_with_thinking_uploads_the_json_tool_and_no_nat
(upload,) = wire.drain()
(stored,) = tuple(JSON_OBJECT.validate_json(line) for line in upload.body.decode().splitlines() if line.strip())
model_input: Final = object_value(stored["modelInput"])
- assert "format" not in object_value(model_input.get("output_config", {})), model_input
- (tool,) = (object_value(entry) for entry in list_value(model_input["tools"]))
+ output_format: Final = object_value(model_input.get("output_config", {})).get("format")
+ tools: Final = tuple(object_value(entry) for entry in list_value(model_input.get("tools", [])))
+ if native_output:
+ assert model_input["output_config"] == {"format": {"type": "json_schema", "schema": _SCHEMA}}, model_input
+ assert model_input["anthropic_beta"] == ["structured-outputs-2025-11-13"], model_input
+ assert "tools" not in model_input, model_input
+ assert model_input["thinking"] == {"type": "adaptive"}, model_input
+ assert not tools, model_input
+ return
+ assert output_format is None, model_input
+ (tool,) = tools
assert tool["name"] == _JSON_TOOL, model_input
assert object_value(tool["input_schema"])["properties"] == _SCHEMA["properties"], model_input
diff --git a/tests/integration/providers/test_bedrock_converse_redacted_thinking_wire.py b/tests/integration/providers/test_bedrock_converse_redacted_thinking_wire.py
index efb0609d560..0b5cc3cdb23 100644
--- a/tests/integration/providers/test_bedrock_converse_redacted_thinking_wire.py
+++ b/tests/integration/providers/test_bedrock_converse_redacted_thinking_wire.py
@@ -429,6 +429,65 @@ _CONVERSE_SIGNED_PART: Final[dict[str, JsonValue]] = {
}
+@pytest.mark.parametrize(
+ ("deployment", "replayed_thinking"),
+ [
+ pytest.param({"model": f"bedrock/converse/{NOVA_MODEL_ID}"}, (), id="nova"),
+ pytest.param(
+ {
+ "model": f"bedrock/converse/{APPLICATION_PROFILE_ARN}",
+ "model_info": {"base_model": f"bedrock/{NOVA_MODEL_ID}"},
+ },
+ (),
+ id="application-profile-arn-with-nova-base-model",
+ ),
+ pytest.param(
+ {"model": f"bedrock/converse/{APPLICATION_PROFILE_ARN}"},
+ (_CONVERSE_SIGNED_PART,),
+ id="application-profile-arn-without-base-model",
+ ),
+ pytest.param(
+ {
+ "model": f"bedrock/converse/{APPLICATION_PROFILE_ARN}",
+ "model_info": {"base_model": f"bedrock/{CLAUDE_MODEL_ID}"},
+ },
+ (_CONVERSE_SIGNED_PART,),
+ id="application-profile-arn-with-claude-base-model",
+ ),
+ pytest.param(_CLAUDE, (_CONVERSE_SIGNED_PART,), id="claude"),
+ ],
+)
+def test_signed_history_keeps_its_thinking_unless_the_converse_target_is_known_not_to_be_claude(
+ gateway: Gateway, deployment: dict[str, JsonValue], replayed_thinking: tuple[dict[str, JsonValue], ...]
+) -> None:
+ model_info: Final = deployment.get("model_info")
+ parameters: Final = {key: value for key, value in deployment.items() if key != "model_info"}
+ with wire_server(_bedrock_peer()) as wire, gateway.scenario() as scenario:
+ model: Final = scenario.model(
+ model_info=model_info if isinstance(model_info, dict) else None,
+ api_key=TOKEN,
+ aws_region_name="us-east-1",
+ aws_bedrock_runtime_endpoint=wire.url,
+ **parameters,
+ )
+
+ reply: Final = _chat(
+ gateway,
+ {
+ "model": model,
+ "max_tokens": 256,
+ "messages": [
+ {"role": "user", "content": QUESTION},
+ {"role": "assistant", "content": FIRST_ANSWER, "thinking_blocks": [_SIGNED_THINKING]},
+ {"role": "user", "content": FOLLOW_UP},
+ ],
+ },
+ )
+
+ assert reply.content == FINAL_ANSWER
+ assert _replayed_assistant_turn(wire).content == (*replayed_thinking, {"text": FIRST_ANSWER})
+
+
def _replayed_to_converse(
gateway: Gateway, wire: Wire, model: str, assistants: list[JsonValue], **fields: JsonValue
) -> tuple[dict[str, JsonValue], ...]:
diff --git a/tests/integration/providers/test_bedrock_native_structured_output_wire.py b/tests/integration/providers/test_bedrock_native_structured_output_wire.py
index 2502bd90bd6..ceb40879b9f 100644
--- a/tests/integration/providers/test_bedrock_native_structured_output_wire.py
+++ b/tests/integration/providers/test_bedrock_native_structured_output_wire.py
@@ -18,9 +18,10 @@ from pydantic import JsonValue, TypeAdapter
_HAIKU: Final = "us.anthropic.claude-haiku-5-5"
_FABLE: Final = "us.anthropic.claude-fable-5"
_SONNET: Final = "us.anthropic.claude-sonnet-4-6"
-_OPUS: Final = "us.anthropic.claude-opus-5-5"
-_OPT_IN: Final = _OPUS
-_UNFORCEABLE_TOOL: Final = _OPUS
+_NEITHER_KEY: Final = "us.anthropic.claude-opus-4-7"
+_OPT_IN: Final = _NEITHER_KEY
+_BEDROCK_KEY_ONLY: Final = "us.anthropic.claude-sonnet-5-5"
+_UNFORCEABLE_TOOL: Final = "anthropic.claude-opus-5-5"
_OPUS_4_8: Final = "us.anthropic.claude-opus-4-8"
_DEEPSEEK: Final = "deepseek.v3.2"
_AWS: Final[dict[str, JsonValue]] = {
@@ -622,11 +623,9 @@ def test_messages_thinking_on_a_bedrock_converse_native_model_gets_native_output
@pytest.mark.parametrize("route", ["converse", "invoke"])
-@pytest.mark.parametrize(
- ("backend", "thinks_by_default"), [(_OPUS_4_8, False), (_HAIKU, True)], ids=["opus_4_8", "haiku_5_5"]
-)
-def test_bedrock_native_model_with_no_thinking_param_gets_native_output_only_when_it_thinks_by_default(
- gateway: Gateway, route: str, backend: str, thinks_by_default: bool
+@pytest.mark.parametrize("backend", [_OPUS_4_8, _HAIKU], ids=["opus_4_8", "haiku_5_5"])
+def test_bedrock_native_model_with_no_thinking_param_gets_native_output(
+ gateway: Gateway, route: str, backend: str
) -> None:
marker, marker_hex = _marker()
with wire_server(_bedrock) as wire, gateway.scenario() as scenario:
@@ -639,11 +638,9 @@ def test_bedrock_native_model_with_no_thinking_param_gets_native_output_only_whe
assert not message.get("tool_calls"), response.text
_, body = _one_call(wire, marker_hex)
invoke: Final = route == "invoke"
- if thinks_by_default:
- _assert_native_without_tools(body, invoke, _SCHEMA)
- return
- assert _native_schema(body, invoke) is None, body
- assert _tools(body, invoke)[_JSON_TOOL]["properties"] == _SCHEMA["properties"], body
+ _assert_native_without_tools(body, invoke, _SCHEMA)
+ if invoke:
+ assert body["anthropic_beta"] == ["structured-outputs-2025-11-13"], body
@pytest.mark.parametrize("route", ["converse", "invoke"])
@@ -683,11 +680,16 @@ def test_a_model_that_cannot_force_the_json_tool_retries_a_rejected_native_outpu
@pytest.mark.parametrize(
("backend", "model_info", "native"),
[
- (_OPUS, None, False),
+ (_NEITHER_KEY, None, False),
(_OPT_IN, {"supports_bedrock_native_structured_output": True}, True),
(_OPUS_4_8, {"supports_bedrock_native_structured_output": False}, False),
+ (
+ _OPUS_4_8,
+ {"supports_native_structured_output": True, "supports_bedrock_native_structured_output": False},
+ True,
+ ),
],
- ids=["neither_key", "deployment_opt_in", "deployment_opt_out"],
+ ids=["neither_key", "deployment_opt_in", "deployment_opt_out", "explicit_native_with_bedrock_key_off"],
)
def test_thinking_request_follows_the_bedrock_native_key_the_model_resolves_to(
gateway: Gateway, backend: str, model_info: Mapping[str, JsonValue] | None, native: bool
@@ -897,7 +899,7 @@ def test_messages_on_bedrock_invoke_follows_the_bedrock_native_key_without_think
) -> None:
marker, marker_hex = _marker()
with wire_server(_bedrock) as wire, gateway.scenario() as scenario:
- model: Final = _deployment(scenario, wire, "invoke", _HAIKU)
+ model: Final = _deployment(scenario, wire, "invoke", _BEDROCK_KEY_ONLY)
response: Final = gateway.request(
"POST",
"/v1/messages",
@@ -945,9 +947,9 @@ def test_malformed_schema_on_a_native_model_reaches_bedrock_once_without_a_proxy
("route", "backend", "thinking", "schema", "native"),
[
("converse", _SONNET, False, _NUMERIC_FORMATS_SCHEMA, True),
- ("invoke", _HAIKU, True, _NUMERIC_FORMATS_SCHEMA, True),
+ ("invoke", _BEDROCK_KEY_ONLY, True, _NUMERIC_FORMATS_SCHEMA, True),
("converse", _SONNET, False, _WIDE_SCHEMA, True),
- ("converse", _HAIKU, True, _WIDE_SCHEMA, False),
+ ("converse", _BEDROCK_KEY_ONLY, True, _WIDE_SCHEMA, False),
],
ids=["numeric_formats_old_key", "numeric_formats_new_key_invoke", "wide_old_key", "wide_new_key"],
)
@@ -1046,7 +1048,7 @@ def test_bedrock_rejecting_opt_in_native_output_gets_one_retry_without_it(
ids=["other_400", "native_model"],
)
@pytest.mark.parametrize("route", ["converse", "invoke"])
-def test_bedrock_400_is_not_retried_unless_it_rejects_opt_in_native_output(
+def test_bedrock_400_is_not_retried_unless_it_rejects_native_output(
gateway: Gateway, route: str, backend: str, rejection: str
) -> None:
marker, marker_hex = _marker()
@@ -1057,10 +1059,19 @@ def test_bedrock_400_is_not_retried_unless_it_rejects_opt_in_native_output(
"/v1/chat/completions",
_chat_body(model, marker, thinking=_ADAPTIVE, response_format=_response_format(_SCHEMA)),
)
- assert response.status_code == 400, response.text
- assert rejection.split(": ", 1)[1] in response.text, response.text
- _, body = _one_call(wire, marker_hex)
- assert _native_schema(body, route == "invoke") == _SCHEMA, body
+ invoke: Final = route == "invoke"
+ if rejection == _MAX_TOKENS_400:
+ assert response.status_code == 400, response.text
+ assert rejection.split(": ", 1)[1] in response.text, response.text
+ _, body = _one_call(wire, marker_hex)
+ _assert_native_without_tools(body, invoke, _SCHEMA)
+ return
+ assert _content_json(_chat_message(response)) == _payload(marker_hex), response.text
+ rejected, retried = (_JSON.validate_json(request.body) for request in wire.drain())
+ _assert_native_without_tools(rejected, invoke, _SCHEMA)
+ assert _native_schema(retried, invoke) is None, retried
+ assert _tools(retried, invoke)[_JSON_TOOL]["properties"] == _SCHEMA["properties"], retried
+ assert marker_hex in json.dumps(retried), retried
def _always_rejecting(message: str) -> Callable[[Request], Reply]:
@@ -1075,7 +1086,7 @@ def _always_rejecting(message: str) -> Callable[[Request], Reply]:
[(_OPUS_4_8, _MAX_TOKENS_400), (_SONNET, _FABLE_REJECTION)],
ids=["other_400", "native_model"],
)
-def test_messages_400_on_bedrock_invoke_reaches_the_caller_after_one_call_unless_it_rejects_opt_in_native_output(
+def test_messages_400_on_bedrock_invoke_reaches_the_caller_after_one_call_unless_it_rejects_native_output(
gateway: Gateway, backend: str, rejection: str
) -> None:
marker, marker_hex = _marker()
@@ -1088,7 +1099,17 @@ def test_messages_400_on_bedrock_invoke_reaches_the_caller_after_one_call_unless
)
assert response.status_code == 400, response.text
assert rejection.split(": ", 1)[1] in response.text, response.text
- _one_call(wire, marker_hex)
+ if rejection == _MAX_TOKENS_400:
+ _, body = _one_call(wire, marker_hex)
+ _assert_native_without_tools(body, True, _SCHEMA)
+ return
+ rejected, retried = (_JSON.validate_json(request.body) for request in wire.drain())
+ _assert_native_without_tools(rejected, True, _SCHEMA)
+ assert _native_schema(retried, True) is None, retried
+ assert not _tools(retried, True), retried
+ prompt: Final = json.dumps(retried["messages"])
+ assert marker_hex in prompt, retried
+ assert all(name in prompt for name in object_value(_SCHEMA["properties"])), retried
def test_a_messages_retry_without_native_output_still_omits_the_deployments_dropped_params(gateway: Gateway) -> None:
diff --git a/tests/integration/providers/test_chat_completions_tool_search_wire.py b/tests/integration/providers/test_chat_completions_tool_search_wire.py
index f99db3fe735..51c44996d31 100644
--- a/tests/integration/providers/test_chat_completions_tool_search_wire.py
+++ b/tests/integration/providers/test_chat_completions_tool_search_wire.py
@@ -352,28 +352,41 @@ def _first_choice(payload: Mapping[str, JsonValue]) -> dict[str, JsonValue]:
return rv.ITEMS.validate_python(payload["choices"])[0]
-def _streamed_tool_calls(frames: Sequence[Mapping[str, JsonValue]]) -> list[tuple[str, str]]:
- deltas: Final = [
- rv.JSON_OBJECT.validate_python(_first_choice(frame)["delta"]) for frame in frames if frame.get("choices")
- ]
- opened: Final = rv.ITEMS.validate_python(
- list(chain.from_iterable(rv.ITEMS.validate_python(delta.get("tool_calls") or []) for delta in deltas))
- )
+def _call_names(calls: Sequence[Mapping[str, JsonValue]]) -> list[tuple[str, str]]:
return [
(str(call["id"]), str(rv.JSON_OBJECT.validate_python(call["function"])["name"]))
- for call in opened
+ for call in calls
if call.get("id")
]
+def _deltas(frames: Sequence[Mapping[str, JsonValue]]) -> list[dict[str, JsonValue]]:
+ return [rv.JSON_OBJECT.validate_python(_first_choice(frame)["delta"]) for frame in frames if frame.get("choices")]
+
+
+def _server_tool_calls_of(holder: Mapping[str, JsonValue]) -> list[dict[str, JsonValue]]:
+ fields: Final = rv.JSON_OBJECT.validate_python(holder.get("provider_specific_fields") or {})
+ return rv.ITEMS.validate_python(fields.get("server_tool_calls") or [])
+
+
def answered_tool_calls(answer: Answer) -> list[tuple[str, str]]:
if answer.frames:
- return _streamed_tool_calls(answer.frames)
+ return _call_names(
+ list(
+ chain.from_iterable(rv.ITEMS.validate_python(d.get("tool_calls") or []) for d in _deltas(answer.frames))
+ )
+ )
message: Final = rv.JSON_OBJECT.validate_python(_first_choice(answer.payload)["message"])
- return [
- (str(call["id"]), str(rv.JSON_OBJECT.validate_python(call["function"])["name"]))
- for call in rv.ITEMS.validate_python(message.get("tool_calls") or [])
- ]
+ return _call_names(rv.ITEMS.validate_python(message.get("tool_calls") or []))
+
+
+def answered_server_tool_calls(answer: Answer) -> list[tuple[str, str]]:
+ if answer.frames:
+ streamed: Final = _call_names(
+ list(chain.from_iterable(_server_tool_calls_of(d) for d in _deltas(answer.frames)))
+ )
+ return list(dict.fromkeys(streamed))
+ return _call_names(_server_tool_calls_of(rv.JSON_OBJECT.validate_python(_first_choice(answer.payload)["message"])))
def response_id(answer: Answer) -> str:
@@ -382,10 +395,8 @@ def response_id(answer: Answer) -> str:
def assert_search_call_answered(answer: Answer, marker: str) -> None:
assert answer.status == 200, answer.text
- assert answered_tool_calls(answer) == [
- (f"srvtoolu_{marker}", _REGEX_TOOL_NAME),
- (f"toolu_{marker}", "get_weather"),
- ], answer.text
+ assert answered_server_tool_calls(answer) == [(f"srvtoolu_{marker}", _REGEX_TOOL_NAME)], answer.text
+ assert answered_tool_calls(answer) == [(f"toolu_{marker}", "get_weather")], answer.text
if answer.frames:
return
usage: Final = rv.JSON_OBJECT.validate_python(answer.payload["usage"])
diff --git a/tests/integration/providers/test_responses_bridge_tool_search_wire.py b/tests/integration/providers/test_responses_bridge_tool_search_wire.py
index 57d9655b7d5..95c05610fba 100644
--- a/tests/integration/providers/test_responses_bridge_tool_search_wire.py
+++ b/tests/integration/providers/test_responses_bridge_tool_search_wire.py
@@ -998,7 +998,7 @@ def test_a_callers_function_named_like_the_search_tool_stays_a_function_call(gat
assert [tool["name"] for tool in _ITEMS.validate_python(body["tools"])] == [_REGEX_TOOL_NAME], body
-def test_an_unrequested_server_tool_use_from_the_vendor_stays_a_function_call(gateway: Gateway) -> None:
+def test_an_unrequested_server_tool_use_from_the_vendor_is_not_returned_as_a_function_call(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(_Vendor(searched=True).respond) as wire, gateway.scenario() as scenario:
model: Final = _deployment(scenario, "anthropic", wire.url, gateway.upstream_url)
@@ -1007,7 +1007,11 @@ def test_an_unrequested_server_tool_use_from_the_vendor_stays_a_function_call(ga
)
assert "tool_search_call" not in _output_types(answer.payload), answer
calls: Final = {str(item["call_id"]): str(item["name"]) for item in _tool_items(answer.payload)}
- assert calls == {f"srvtoolu_{marker}": _REGEX_TOOL_NAME, f"toolu_{marker}": "get_weather"}, answer
+ assert calls == {f"toolu_{marker}": "get_weather"}, answer
+ server_calls: Final = _ITEMS.validate_python(
+ _JSON_OBJECT.validate_python(answer.payload["provider_specific_fields"])["server_tool_calls"]
+ )
+ assert [call["id"] for call in server_calls] == [f"srvtoolu_{marker}"], answer
_, body = _only_request(wire)
_assert_dropped_on_wire(body, ("get_weather",))
@@ -1224,15 +1228,17 @@ def test_chat_completions_keep_forwarding_the_regex_tool_and_defer_loading(gatew
if stream:
assert "data: [DONE]" in response.text and "get_weather" in response.text, response.text
else:
- calls: Final = _ITEMS.validate_python(
- _JSON_OBJECT.validate_python(
- _ITEMS.validate_python(_JSON_OBJECT.validate_json(response.content)["choices"])[0]["message"]
- )["tool_calls"]
+ message: Final = _JSON_OBJECT.validate_python(
+ _ITEMS.validate_python(_JSON_OBJECT.validate_json(response.content)["choices"])[0]["message"]
)
- assert [_JSON_OBJECT.validate_python(call["function"])["name"] for call in calls] == [
- _REGEX_TOOL_NAME,
- "get_weather",
- ], calls
+ calls: Final = _ITEMS.validate_python(message["tool_calls"])
+ server_calls: Final = _ITEMS.validate_python(
+ _JSON_OBJECT.validate_python(message["provider_specific_fields"])["server_tool_calls"]
+ )
+ assert [_JSON_OBJECT.validate_python(call["function"])["name"] for call in calls] == ["get_weather"], calls
+ assert [_JSON_OBJECT.validate_python(call["function"])["name"] for call in server_calls] == [
+ _REGEX_TOOL_NAME
+ ], message
request, sent = _only_request(wire)
_assert_anthropic_wire(request, sent, ("get_weather",))
diff --git a/tests/integration/providers/test_responses_minted_reasoning_replay_wire.py b/tests/integration/providers/test_responses_minted_reasoning_replay_wire.py
index 2253443dfc0..4ff5871ff97 100644
--- a/tests/integration/providers/test_responses_minted_reasoning_replay_wire.py
+++ b/tests/integration/providers/test_responses_minted_reasoning_replay_wire.py
@@ -77,7 +77,18 @@ _AZURE_AI_OPENAI_HOST: Final = _Deployment(
"/openai/v1/responses?api-version=preview",
strips_message_status=True,
)
-_DROPPING: Final = (_OPENAI, _AZURE, _AZURE_AI_OPENAI_HOST)
+_BEDROCK_MANTLE: Final = _Deployment(
+ "bedrock_mantle", "bedrock_mantle/openai.gpt-oss-120b", "synthetic-mantle-key", "/v1/responses"
+)
+_BEDROCK: Final = _Deployment(
+ "bedrock",
+ "bedrock/openai.gpt-oss-120b-1:0",
+ "synthetic-bedrock-key",
+ "/openai/v1/responses",
+ MappingProxyType({"aws_region_name": "us-east-1"}),
+ model_info=MappingProxyType({"supported_endpoints": ["/v1/responses"]}),
+)
+_DROPPING: Final = (_OPENAI, _AZURE, _AZURE_AI_OPENAI_HOST, _BEDROCK_MANTLE, _BEDROCK)
_KEEPING: Final = (
_Deployment("litellm_proxy", f"litellm_proxy/{_GPT}", "synthetic-proxy-key", "/responses"),
_Deployment("databricks", "databricks/gpt-5.6", "synthetic-databricks-key", "/responses"),
@@ -95,15 +106,6 @@ _KEEPING: Final = (
"/v1/responses",
types_untyped_items_as_messages=True,
),
- _Deployment("bedrock_mantle", "bedrock_mantle/openai.gpt-oss-120b", "synthetic-mantle-key", "/v1/responses"),
- _Deployment(
- "bedrock",
- "bedrock/openai.gpt-oss-120b-1:0",
- "synthetic-bedrock-key",
- "/openai/v1/responses",
- MappingProxyType({"aws_region_name": "us-east-1"}),
- model_info=MappingProxyType({"supported_endpoints": ["/v1/responses"]}),
- ),
*(
_Deployment(slug, f"{slug}/{model}", f"synthetic-{slug}-key", "/responses")
for slug, model in (
diff --git a/tests/integration/providers/test_vertex_anthropic_structured_output_wire.py b/tests/integration/providers/test_vertex_anthropic_structured_output_wire.py
index 19928ecd6e0..95e07593b2a 100644
--- a/tests/integration/providers/test_vertex_anthropic_structured_output_wire.py
+++ b/tests/integration/providers/test_vertex_anthropic_structured_output_wire.py
@@ -48,7 +48,7 @@ _THINKING_OFF_TOOL_MODEL: Final = "claude-opus-4-8"
_UNMAPPED_MODEL: Final = "claude-scripted-9"
_BEDROCK_INVOKE_MODEL: Final = "anthropic.claude-sonnet-5"
_BEDROCK_TOOL_PATH_MODEL: Final = "anthropic.claude-opus-4-8"
-_BEDROCK_JSON_TOOL_MODEL: Final = "us.anthropic.claude-opus-4-8"
+_BEDROCK_JSON_TOOL_MODEL: Final = "us.anthropic.claude-opus-4-7"
_THINKING_SIGNATURE: Final = "c2NyaXB0ZWQtc2lnbmF0dXJl"
_PROJECT: Final = "scripted-project"
_LOCATION: Final = "us-east5"
diff --git a/tests/integration/spend/test_redis_ttl_preserving_token_increment.py b/tests/integration/spend/test_redis_ttl_preserving_token_increment.py
index 8b7a21f73d5..daf76bf4740 100644
--- a/tests/integration/spend/test_redis_ttl_preserving_token_increment.py
+++ b/tests/integration/spend/test_redis_ttl_preserving_token_increment.py
@@ -11,7 +11,7 @@ from litellm.proxy.hooks.parallel_request_limiter_v3 import (
PROXY_MaxParallelRequestsHandler_v3 as _PROXY_MaxParallelRequestsHandler,
)
from litellm.proxy.utils import InternalUsageCache
-from litellm.types.caching import RedisPipelineIncrementOperation
+from litellm.redis.stores.rate_limits import CounterIncrement, RateLimitScope
@pytest.mark.asyncio
@@ -22,11 +22,12 @@ async def test_async_increment_tokens_with_ttl_preservation() -> None:
handler: Final = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache(redis_cache=redis_cache))
)
- assert handler.token_increment_script is not None
suffix: Final = uuid.uuid4().hex[:8]
- key_with_ttl: Final = f"{{test_ttl}}:with_ttl:{suffix}"
- key_without_ttl: Final = f"{{test_ttl}}:without_ttl:{suffix}"
+ scope_with_ttl: Final = RateLimitScope("test_ttl", f"with_ttl:{suffix}")
+ scope_without_ttl: Final = RateLimitScope("test_ttl", f"without_ttl:{suffix}")
+ key_with_ttl: Final = f"{{test_ttl:with_ttl:{suffix}}}:tokens"
+ key_without_ttl: Final = f"{{test_ttl:without_ttl:{suffix}}}:tokens"
try:
await redis_cache.async_delete_cache(key_with_ttl)
@@ -34,8 +35,8 @@ async def test_async_increment_tokens_with_ttl_preservation() -> None:
await handler.async_increment_tokens_with_ttl_preservation(
pipeline_operations=[
- RedisPipelineIncrementOperation(key=key_with_ttl, increment_value=10.0, ttl=60),
- RedisPipelineIncrementOperation(key=key_without_ttl, increment_value=5.0, ttl=None),
+ CounterIncrement(scope_with_ttl.counter("tokens"), 10.0, 60),
+ CounterIncrement(scope_without_ttl.counter("tokens"), 5.0, None),
]
)
@@ -50,8 +51,8 @@ async def test_async_increment_tokens_with_ttl_preservation() -> None:
await handler.async_increment_tokens_with_ttl_preservation(
pipeline_operations=[
- RedisPipelineIncrementOperation(key=key_with_ttl, increment_value=15.0, ttl=60),
- RedisPipelineIncrementOperation(key=key_without_ttl, increment_value=7.0, ttl=None),
+ CounterIncrement(scope_with_ttl.counter("tokens"), 15.0, 60),
+ CounterIncrement(scope_without_ttl.counter("tokens"), 7.0, None),
]
)
diff --git a/tests/integration/spend/test_tag_spend_reset.py b/tests/integration/spend/test_tag_spend_reset.py
new file mode 100644
index 00000000000..2104539f5d7
--- /dev/null
+++ b/tests/integration/spend/test_tag_spend_reset.py
@@ -0,0 +1,75 @@
+import json
+import uuid
+from typing import Final
+
+import httpx
+
+from tests.integration._support.client import Gateway, eventually
+from tests.integration._support.database import read_rows
+from tests.integration._support.provider import PROVIDER_URL, SharedProvider
+from tests.integration._support.wire import Reply
+
+
+def test_admin_can_reset_tag_spend_and_resume_budget_blocked_requests(
+ gateway: Gateway, provider: SharedProvider
+) -> None:
+ tag: Final = f"tag-reset-{uuid.uuid4().hex}"
+ completion: Final = Reply(
+ body=json.dumps(
+ {
+ "id": f"chatcmpl-{uuid.uuid4().hex}",
+ "object": "chat.completion",
+ "created": 1,
+ "model": "gpt-4o-mini",
+ "choices": [
+ {"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
+ ],
+ "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
+ }
+ ).encode()
+ )
+ with gateway.scenario() as scenario:
+ model: Final = scenario.model(
+ api_base=f"{PROVIDER_URL}/v1", input_cost_per_token=0.001, output_cost_per_token=0.002
+ )
+ gateway.post("/tag/new", {"name": tag, "max_budget": 0.0001, "budget_duration": "30d"})
+
+ def delete_tag() -> None:
+ gateway.post("/tag/delete", {"name": tag})
+
+ scenario.cleanups.callback(delete_tag)
+
+ def tagged_request() -> httpx.Response:
+ return gateway.request(
+ "POST",
+ "/v1/chat/completions",
+ {
+ "model": model,
+ "messages": [{"role": "user", "content": f"tag reset {uuid.uuid4().hex}"}],
+ "metadata": {"tags": [tag]},
+ },
+ )
+
+ provider.expect(completion)
+ first: Final = tagged_request()
+ assert first.status_code == 200, first.text
+ assert len(provider.received()) == 1
+ eventually(
+ lambda: read_rows('SELECT spend FROM "LiteLLM_TagTable" WHERE tag_name=%s', (tag,)),
+ lambda rows: len(rows) == 1 and float(str(rows[0]["spend"])) > 0.0001,
+ seconds=70,
+ )
+ blocked: Final = tagged_request()
+ assert blocked.status_code == 422, blocked.text
+ assert blocked.json()["error"]["type"] == "budget_exceeded", blocked.text
+ assert f"Tag={tag}" in blocked.text, blocked.text
+ assert provider.received() == ()
+ reset: Final = gateway.request("POST", "/tag/update", {"name": tag, "spend": 0})
+ assert reset.status_code == 200, reset.text
+ provider.expect(completion)
+ try:
+ resumed: Final = tagged_request()
+ assert resumed.status_code == 200, resumed.text
+ assert len(provider.received()) == 1
+ finally:
+ provider.replies.clear()
diff --git a/tests/integration/streaming/test_responses_top_level_error_events.py b/tests/integration/streaming/test_responses_top_level_error_events.py
index 63bdadceebc..2cbc4570dec 100644
--- a/tests/integration/streaming/test_responses_top_level_error_events.py
+++ b/tests/integration/streaming/test_responses_top_level_error_events.py
@@ -22,12 +22,14 @@ from integration._support.responses_stream import (
data_frames,
delta,
error_body,
+ error_event,
frame,
function_tools,
healthy_stream,
messages_body,
openai_error_event,
openai_error_stream,
+ response_object,
serve,
spend_row,
sse_events,
@@ -277,6 +279,38 @@ def test_native_responses_stream_fails_with_the_provider_code_and_message(gatewa
_assert_failure_logged(response.headers["x-litellm-call-id"], "429", RATE_LIMIT_MESSAGE)
+_FLEX_MESSAGE: Final = "Flex processing is temporarily unavailable. Please try again later or use standard processing."
+_FLEX_ERROR: Final = {"type": "resource_unavailable", "code": "flex_unavailable", "message": _FLEX_MESSAGE}
+
+
+def _flex_outage(identity: str, kind: str) -> tuple[bytes, ...]:
+ failure: Final = (
+ error_event(_FLEX_ERROR)
+ if kind == "error"
+ else {
+ "type": "response.failed",
+ "sequence_number": 1,
+ "response": response_object(identity, "failed", error=_FLEX_ERROR),
+ }
+ )
+ return (frame(created(identity)), frame(failure))
+
+
+@pytest.mark.parametrize("kind", ["error", "response.failed"])
+def test_native_responses_stream_flex_outage_fails_as_a_rate_limit(gateway: Gateway, kind: str) -> None:
+ response, error = _responses_frames(gateway, _flex_outage(_identity(), kind))
+ assert error["code"] == "rate_limit_exceeded" and _FLEX_MESSAGE in string_value(error["message"]), error
+ _assert_failure_logged(response.headers["x-litellm-call-id"], "429", _FLEX_MESSAGE)
+
+
+@pytest.mark.parametrize("kind", ["error", "response.failed"])
+def test_chat_stream_flex_outage_is_a_throttling_error_like_the_non_streamed_429(gateway: Gateway, kind: str) -> None:
+ response, _ = _chat(gateway, _flex_outage(_identity(), kind))
+ assert response.status_code == 429, response.text
+ assert_provider_typed_rate_limit(error_body(response), _FLEX_MESSAGE)
+ _assert_failure_logged(response.headers["x-litellm-call-id"], "429", _FLEX_MESSAGE)
+
+
def test_native_responses_stream_error_after_text_fails_with_the_provider_code(gateway: Gateway) -> None:
identity: Final = _identity()
stream: Final = (
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
index 146a6a3ff1c..0a9dc0ab3cf 100644
--- a/tests/unit/conftest.py
+++ b/tests/unit/conftest.py
@@ -496,7 +496,7 @@ def declare_mid_conversation_system(monkeypatch: pytest.MonkeyPatch) -> Callable
return declare
-NativeStructuredOutputKey: TypeAlias = Literal["legacy", "bedrock"] | None
+NativeStructuredOutputKey: TypeAlias = Literal["legacy", "bedrock", "default"] | None
@pytest.fixture
@@ -506,7 +506,8 @@ def set_native_structured_output(
def set_flags(model_cost_key: str, native_key: NativeStructuredOutputKey) -> None:
entry: Final = litellm.model_cost[model_cost_key]
monkeypatch.setitem(entry, "supports_native_structured_output", native_key == "legacy")
- monkeypatch.setitem(entry, "supports_bedrock_native_structured_output", native_key == "bedrock")
+ monkeypatch.setitem(entry, "supports_bedrock_native_structured_output", native_key in ("bedrock", "default"))
+ monkeypatch.setitem(entry, "supports_bedrock_native_structured_output_by_default", native_key == "default")
litellm.get_model_info.cache_clear()
return set_flags
diff --git a/tests/unit/enterprise/enterprise_callbacks/test_secret_detection.py b/tests/unit/enterprise/enterprise_callbacks/test_secret_detection.py
index 9816eb91102..3599c938c28 100644
--- a/tests/unit/enterprise/enterprise_callbacks/test_secret_detection.py
+++ b/tests/unit/enterprise/enterprise_callbacks/test_secret_detection.py
@@ -29,6 +29,7 @@ from litellm_enterprise.enterprise_callbacks.secret_detection import (
from litellm.caching.caching import DualCache
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.utils import hash_token
+from litellm.types.responses.main import build_web_search_call
AWS_KEY = "AKIAIOSFODNN7EXAMPLE"
OPENAI_KEY = "sk-test-abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGH"
@@ -114,6 +115,51 @@ async def test_deeply_nested_tool_arguments_are_redacted():
assert json.loads(rewritten) == json.loads(arguments.replace(AWS_KEY, "[REDACTED]"))
+@pytest.mark.asyncio
+async def test_a_replayed_server_tool_call_cannot_forward_unredacted_credentials():
+ arguments = json.dumps({"query": f"AWS_ACCESS_KEY_ID={AWS_KEY}"})
+ server_call = {"id": "srvtoolu_1", "type": "function", "function": {"name": "web_search", "arguments": arguments}}
+ data = {
+ "messages": [
+ {
+ "role": "assistant",
+ "content": "Searching.",
+ "provider_specific_fields": {"server_tool_calls": [server_call]},
+ }
+ ]
+ }
+ await _guardrail().async_pre_call_hook(UserAPIKeyAuth(), DualCache(), data, "acompletion")
+ rewritten = data["messages"][0]["provider_specific_fields"]["server_tool_calls"][0]["function"]["arguments"]
+ assert json.loads(rewritten) == {"query": "AWS_ACCESS_KEY_ID=[REDACTED]"}
+
+
+@pytest.mark.asyncio
+async def test_a_replayed_server_web_search_keeps_no_credentials_in_its_hosted_copy():
+ query = f"AWS_ACCESS_KEY_ID={AWS_KEY}"
+ server_call = {
+ "id": "srvtoolu_1",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": json.dumps({"query": query})},
+ }
+ data = {
+ "messages": [
+ {
+ "role": "assistant",
+ "content": "Searching.",
+ "provider_specific_fields": {
+ "server_tool_calls": [server_call],
+ "web_search_calls": [
+ build_web_search_call("srvtoolu_1", {"query": query}, {"content": []}).model_dump()
+ ],
+ },
+ }
+ ]
+ }
+ await _guardrail().async_pre_call_hook(UserAPIKeyAuth(), DualCache(), data, "acompletion")
+ hosted = json.loads(json.dumps(data["messages"][0]["provider_specific_fields"]["web_search_calls"]))
+ assert [call["action"]["query"] for call in hosted] == ["AWS_ACCESS_KEY_ID=[REDACTED]"]
+
+
@pytest.mark.parametrize("password", ("hunter2\\brahms9x", "hunter2\tbrahms9x"))
def test_tool_payload_redacts_keyword_values_that_need_json_escaping(password):
assert _guardrail().redact_tool_payload({"password": password, "path": ".env"}) == {
diff --git a/tests/unit/integrations/arize/test_arize_utils.py b/tests/unit/integrations/arize/test_arize_utils.py
index 31699359bbe..db7144b125a 100644
--- a/tests/unit/integrations/arize/test_arize_utils.py
+++ b/tests/unit/integrations/arize/test_arize_utils.py
@@ -725,6 +725,73 @@ def test_arize_output_value_falls_back_to_tool_calls_summary():
assert "SF" in out
+def test_arize_server_tool_calls_lead_the_output_summary_and_tool_call_attributes() -> None:
+ span: Final = TracerProvider().get_tracer(__name__).start_span("litellm_request")
+ kwargs: Final = {
+ "model": "claude-opus-5-5",
+ "messages": [{"role": "user", "content": "persimmon facts?"}],
+ "standard_logging_object": {"model_parameters": {}, "metadata": {}, "call_type": "completion"},
+ "optional_params": {},
+ "litellm_params": {"custom_llm_provider": "anthropic"},
+ }
+ response_obj: Final = ModelResponse(
+ usage={"total_tokens": 10, "completion_tokens": 4, "prompt_tokens": 6},
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ],
+ model="claude-opus-5-5",
+ id="r-server-tools",
+ )
+
+ ArizeLogger.set_arize_attributes(span, kwargs, response_obj)
+
+ tool_calls_prefix: Final = f"{SpanAttributes.LLM_OUTPUT_MESSAGES}.0.{MessageAttributes.MESSAGE_TOOL_CALLS}."
+ tool_call_attributes: Final = {
+ key: value for key, value in span.attributes.items() if key.startswith(tool_calls_prefix)
+ }
+ assert json.loads(span.attributes[SpanAttributes.OUTPUT_VALUE]) == {
+ "tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ },
+ {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "SF"}'}},
+ ]
+ }
+ assert tool_call_attributes == {
+ f"{tool_calls_prefix}0.{ToolCallAttributes.TOOL_CALL_ID}": "srvtoolu_01",
+ f"{tool_calls_prefix}0.{ToolCallAttributes.TOOL_CALL_FUNCTION_NAME}": "web_search",
+ f"{tool_calls_prefix}0.{ToolCallAttributes.TOOL_CALL_FUNCTION_ARGUMENTS_JSON}": '{"query": "persimmon"}',
+ f"{tool_calls_prefix}1.{ToolCallAttributes.TOOL_CALL_ID}": "call_1",
+ f"{tool_calls_prefix}1.{ToolCallAttributes.TOOL_CALL_FUNCTION_NAME}": "get_weather",
+ f"{tool_calls_prefix}1.{ToolCallAttributes.TOOL_CALL_FUNCTION_ARGUMENTS_JSON}": '{"city": "SF"}',
+ }
+
+
def test_arize_output_value_unchanged_when_content_present():
"""Regression guard: when content is non-empty, OUTPUT_VALUE must be
exactly that content (no summary written)."""
diff --git a/tests/unit/integrations/datadog/test_datadog_llm_obs.py b/tests/unit/integrations/datadog/test_datadog_llm_obs.py
index a7826e11483..3604ed223e1 100644
--- a/tests/unit/integrations/datadog/test_datadog_llm_obs.py
+++ b/tests/unit/integrations/datadog/test_datadog_llm_obs.py
@@ -110,6 +110,41 @@ def test_output_tool_calls_use_the_datadog_tool_call_schema(logger: DataDogLLMOb
assert "function" not in message["tool_calls"][0]
+
+def test_claude_server_tool_calls_are_reported_before_client_tool_calls(logger: DataDogLLMObsLogger) -> None:
+ payload: Final = build(
+ logger,
+ response_message={
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [ASSISTANT_TOOL_CALL],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01Search",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "paris weather"}'},
+ }
+ ]
+ },
+ },
+ )
+
+ assert payload["meta"]["output"]["messages"][0]["tool_calls"] == [
+ {
+ "name": "web_search",
+ "arguments": {"query": "paris weather"},
+ "tool_id": "srvtoolu_01Search",
+ "type": "function",
+ },
+ {
+ "name": "get_weather",
+ "arguments": {"city": "Paris", "unit": "c"},
+ "tool_id": "call_abc123",
+ "type": "function",
+ },
+ ]
+
def test_tool_call_identifiers_that_are_not_strings_are_blanked_not_stringified(logger: DataDogLLMObsLogger) -> None:
payload = build(
logger,
diff --git a/tests/unit/integrations/newrelic/test_newrelic.py b/tests/unit/integrations/newrelic/test_newrelic.py
index d09732c6eb2..312457d07e7 100644
--- a/tests/unit/integrations/newrelic/test_newrelic.py
+++ b/tests/unit/integrations/newrelic/test_newrelic.py
@@ -1,3 +1,4 @@
+import json
import os
import sys
from datetime import datetime, timezone
@@ -45,6 +46,12 @@ NR_ENV = {
"NEW_RELIC_APP_NAME": "test-app",
}
+_SERVER_CALL: Final = {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+}
+
def make_logger(**kwargs) -> NewRelicLogger:
"""Instantiate NewRelicLogger with NR agent calls mocked out."""
@@ -361,6 +368,21 @@ class TestExtractMessageContent:
assert "get_weather" in result
assert "call_1" in result
+ @pytest.mark.parametrize(
+ "message",
+ [
+ {"role": "assistant", "content": None, "provider_specific_fields": {"server_tool_calls": [_SERVER_CALL]}},
+ litellm.Message(
+ role="assistant", content=None, provider_specific_fields={"server_tool_calls": [_SERVER_CALL]}
+ ),
+ ],
+ ids=["dict", "message"],
+ )
+ def test_server_tool_call_only_message_serializes_the_server_call(
+ self, message: dict[str, object] | litellm.Message
+ ) -> None:
+ assert json.loads(self.logger._extract_message_content(message)) == [_SERVER_CALL]
+
def test_multimodal_list_serialized_as_json(self):
msg = {
"content": [
diff --git a/tests/unit/integrations/otel/test_otel_v2_sources_of_truth.py b/tests/unit/integrations/otel/test_otel_v2_sources_of_truth.py
index 2d0b4701c9f..c29da4607eb 100644
--- a/tests/unit/integrations/otel/test_otel_v2_sources_of_truth.py
+++ b/tests/unit/integrations/otel/test_otel_v2_sources_of_truth.py
@@ -48,6 +48,7 @@ from litellm.integrations.otel.model.spans import (
validate_registry,
)
from litellm.integrations.otel.model.trace_controls import TraceControls, caller_trace_controls
+from litellm.types.responses.main import build_web_search_call
@pytest.fixture(autouse=True)
@@ -798,6 +799,39 @@ def test_responses_tool_calls_fold_into_the_assistant_message_with_tool_calls_fi
assert data.finish_reasons == ("tool_calls",)
+def test_responses_claude_web_calls_lead_the_tool_calls_without_ending_the_turn_on_tool_calls():
+ claude_fetch: Final = (
+ build_web_search_call("srvtoolu_01", {"url": "https://example.com/"}, None, tool_name="web_fetch")
+ )
+ openai_search: Final = {
+ "type": "web_search_call",
+ "id": "ws_1",
+ "status": "completed",
+ "action": {"type": "search", "query": "news"},
+ }
+ client_call: Final = {"type": "function_call", "call_id": "call_1", "name": "get_weather", "arguments": "{}"}
+ claude_tool_call: Final = {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://example.com/"}'},
+ }
+
+ hosted_only: Final = LLMCallSpanData.from_standard_logging_payload(
+ _responses_payload([_RESPONSES_TEXT_ITEM, claude_fetch.model_dump(), openai_search]), capture_content=True
+ )
+ with_client_call: Final = LLMCallSpanData.from_standard_logging_payload(
+ _responses_payload([_RESPONSES_TEXT_ITEM, claude_fetch.model_dump(), client_call]), capture_content=True
+ )
+
+ assert json.loads(json.dumps(hosted_only.choices_out[0]["message"]["tool_calls"])) == [claude_tool_call]
+ assert hosted_only.finish_reasons == ("stop",)
+ assert json.loads(json.dumps(with_client_call.choices_out[0]["message"]["tool_calls"])) == [
+ claude_tool_call,
+ {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}},
+ ]
+ assert with_client_call.finish_reasons == ("tool_calls",)
+
+
def test_responses_tool_call_only_output_has_no_content():
data = LLMCallSpanData.from_standard_logging_payload(
_responses_payload([{"type": "function_call", "id": "fc_1", "name": "get_weather", "arguments": "{}"}]),
diff --git a/tests/unit/integrations/otel/test_otel_v2_vendor_mappers.py b/tests/unit/integrations/otel/test_otel_v2_vendor_mappers.py
index 57764bf902b..611e828cf84 100644
--- a/tests/unit/integrations/otel/test_otel_v2_vendor_mappers.py
+++ b/tests/unit/integrations/otel/test_otel_v2_vendor_mappers.py
@@ -174,6 +174,45 @@ def test_openinference_output_tool_calls_preserve_calls_in_attributes_and_value(
]
+def test_openinference_output_lists_claude_server_tool_calls_before_client_tool_calls():
+ data: Final = _llm_call(
+ choices_out=(
+ {
+ "finish_reason": "tool_calls",
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_paris",
+ "type": "function",
+ "function": {"name": "lookup_weather", "arguments": '{"city": "Paris"}'},
+ }
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01Search",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "paris weather"}'},
+ }
+ ]
+ },
+ },
+ },
+ )
+ )
+ attrs: Final = OpenInferenceMapper().map(data)
+ assert {key: value for key, value in attrs.items() if ".tool_calls." in key} == {
+ "llm.output_messages.0.message.tool_calls.0.tool_call.id": "srvtoolu_01Search",
+ "llm.output_messages.0.message.tool_calls.0.tool_call.function.name": "web_search",
+ "llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments": '{"query": "paris weather"}',
+ "llm.output_messages.0.message.tool_calls.1.tool_call.id": "call_paris",
+ "llm.output_messages.0.message.tool_calls.1.tool_call.function.name": "lookup_weather",
+ "llm.output_messages.0.message.tool_calls.1.tool_call.function.arguments": '{"city": "Paris"}',
+ }
+
+
def test_openinference_responses_tool_calls_are_emitted_as_output_attributes():
data: Final = LLMCallSpanData.from_standard_logging_payload(
_responses_payload(
diff --git a/tests/unit/integrations/test_helicone.py b/tests/unit/integrations/test_helicone.py
index ce3f61a61eb..a4c23a35b40 100644
--- a/tests/unit/integrations/test_helicone.py
+++ b/tests/unit/integrations/test_helicone.py
@@ -17,7 +17,8 @@ def _claude_mapping(messages, response_obj):
return logger.claude_mapping(model="gpt-5.6", messages=messages, response_obj=response_obj)
-def test_claude_mapping_serializes_custom_tool_calls(monkeypatch):
+@pytest.fixture()
+def anthropic_prompts(monkeypatch: pytest.MonkeyPatch) -> None:
"""
Stub the anthropic module unconditionally: the SDK may be absent (it lives in the
proxy-runtime extra), and the tests/unit/llms/anthropic test package can
@@ -27,6 +28,10 @@ def test_claude_mapping_serializes_custom_tool_calls(monkeypatch):
stub.HUMAN_PROMPT = "\n\nHuman:"
stub.AI_PROMPT = "\n\nAssistant:"
monkeypatch.setitem(sys.modules, "anthropic", stub)
+
+
+@pytest.mark.usefixtures("anthropic_prompts")
+def test_claude_mapping_serializes_custom_tool_calls():
response_obj = {
"id": "chatcmpl-1",
"choices": [
@@ -58,6 +63,44 @@ def test_claude_mapping_serializes_custom_tool_calls(monkeypatch):
assert {"type": "tool_use", "id": "call_f", "name": "read_file", "input": '{"path": "a.py"}'} in tool_use_blocks
+@pytest.mark.usefixtures("anthropic_prompts")
+def test_claude_mapping_puts_server_tool_calls_first_as_tool_use_blocks():
+ response_obj: Final = {
+ "id": "chatcmpl-1",
+ "choices": [
+ {
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_f",
+ "type": "function",
+ "function": {"name": "read_file", "arguments": '{"path": "a.py"}'},
+ },
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ },
+ "finish_reason": "tool_calls",
+ }
+ ],
+ "usage": {"prompt_tokens": 1, "completion_tokens": 2},
+ }
+ mapped: Final = _claude_mapping([{"role": "user", "content": "hi"}], response_obj)
+ assert mapped["content"] == [
+ {"type": "tool_use", "id": "srvtoolu_01", "name": "web_search", "input": '{"query": "persimmon"}'},
+ {"type": "tool_use", "id": "call_f", "name": "read_file", "input": '{"path": "a.py"}'},
+ ]
+
+
@pytest.fixture()
def _vcr_outcome_gate(request, vcr):
install_live_call_probe(request, vcr)
diff --git a/tests/unit/integrations/test_langfuse_otel.py b/tests/unit/integrations/test_langfuse_otel.py
index a2803719c75..71087b4de51 100644
--- a/tests/unit/integrations/test_langfuse_otel.py
+++ b/tests/unit/integrations/test_langfuse_otel.py
@@ -1309,6 +1309,55 @@ class TestDerivedTraceFields:
assert json.loads(str(attributes["langfuse.observation.output"])) == {"content": "hi"}
+ def test_server_tool_calls_lead_the_observation_output(self) -> None:
+ response_obj: Final = ModelResponse(
+ id="chatcmpl-server",
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ],
+ )
+ attributes: Final = _emitted({"call_type": "acompletion", "litellm_params": {"metadata": {}}}, response_obj)
+
+ assert json.loads(str(attributes["langfuse.observation.output"])) == [
+ {
+ "id": "chatcmpl-server",
+ "name": "web_search",
+ "call_id": "srvtoolu_01",
+ "type": "function_call",
+ "arguments": {"query": "persimmon"},
+ },
+ {
+ "id": "chatcmpl-server",
+ "name": "get_weather",
+ "call_id": "call_1",
+ "type": "function_call",
+ "arguments": {"city": "SF"},
+ },
+ ]
+
def test_no_messages_leaves_trace_input_unset_but_emits_output(self) -> None:
attributes: Final = _emitted(
{"call_type": "acompletion", "litellm_params": {"metadata": {}}}, _chat_response("hi")
diff --git a/tests/unit/integrations/test_lunary.py b/tests/unit/integrations/test_lunary.py
index 6491f5c8b82..51353f938b3 100644
--- a/tests/unit/integrations/test_lunary.py
+++ b/tests/unit/integrations/test_lunary.py
@@ -1,6 +1,9 @@
-from litellm.integrations.lunary import parse_tool_calls
+from typing import Final
+
+import litellm
+from litellm.integrations.lunary import parse_messages, parse_tool_calls
from litellm.types.utils import (
ChatCompletionMessageCustomToolCall,
ChatCompletionMessageToolCall,
@@ -35,3 +38,46 @@ def test_parse_tool_calls_serializes_custom_tool_calls():
def test_parse_tool_calls_none_passthrough():
assert parse_tool_calls(None) is None
+
+
+def test_parse_messages_lists_server_tool_calls_before_client_calls():
+ response: Final = litellm.ModelResponse(
+ choices=[
+ litellm.Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ],
+ )
+ assert parse_messages(response.choices) == {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "type": "function",
+ "id": "srvtoolu_01",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ },
+ {"type": "function", "id": "call_1", "function": {"name": "get_weather", "arguments": '{"city": "SF"}'}},
+ ],
+ }
diff --git a/tests/unit/integrations/test_opentelemetry.py b/tests/unit/integrations/test_opentelemetry.py
index 5f749e951b1..393905da6a2 100644
--- a/tests/unit/integrations/test_opentelemetry.py
+++ b/tests/unit/integrations/test_opentelemetry.py
@@ -372,6 +372,112 @@ def test_v1_routing_diagnostics_add_scalar_attributes_and_preserve_legacy_metada
assert finished.attributes["metadata.routing_decision"] == str(decision)
+def test_v1_server_tool_calls_lead_the_completion_function_call_attributes() -> None:
+ exporter: Final = InMemorySpanExporter()
+ provider: Final = TracerProvider()
+ provider.add_span_processor(SimpleSpanProcessor(exporter))
+ otel: Final = OpenTelemetry(tracer_provider=provider)
+ response_obj: Final = litellm.ModelResponse(
+ choices=[
+ litellm.Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ],
+ )
+ with provider.get_tracer(__name__).start_as_current_span("inference") as span:
+ otel.set_attributes(
+ span,
+ {
+ "model": "claude-opus-5-5",
+ "standard_logging_object": {"id": "server-tools-call", "call_type": "acompletion", "metadata": {}},
+ },
+ response_obj,
+ )
+ (finished,) = exporter.get_finished_spans()
+ function_call_attributes: Final = {
+ key: value for key, value in finished.attributes.items() if ".function_call." in key
+ }
+ assert function_call_attributes == {
+ f"{SpanAttributes.LLM_COMPLETIONS.value}.0.function_call.name": "web_search",
+ f"{SpanAttributes.LLM_COMPLETIONS.value}.0.function_call.arguments": '{"query": "persimmon"}',
+ f"{SpanAttributes.LLM_COMPLETIONS.value}.1.function_call.name": "get_weather",
+ f"{SpanAttributes.LLM_COMPLETIONS.value}.1.function_call.arguments": '{"city": "SF"}',
+ }
+
+
+def test_v1_server_tool_calls_stay_in_the_gen_ai_input_and_output_messages() -> None:
+ exporter: Final = InMemorySpanExporter()
+ provider: Final = TracerProvider()
+ provider.add_span_processor(SimpleSpanProcessor(exporter))
+ otel: Final = OpenTelemetry(tracer_provider=provider)
+ weather_call: Final = {
+ "id": "toolu_01",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ search_call: Final = {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ replayed_assistant: Final = {
+ "role": "assistant",
+ "content": "Searching.",
+ "tool_calls": [weather_call],
+ "provider_specific_fields": {"server_tool_calls": [search_call]},
+ }
+ response_obj: Final = litellm.ModelResponse(
+ choices=[
+ litellm.Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=[weather_call],
+ provider_specific_fields={"server_tool_calls": [search_call]},
+ ),
+ )
+ ],
+ )
+ with provider.get_tracer(__name__).start_as_current_span("inference") as span:
+ otel.set_attributes(
+ span,
+ {
+ "model": "claude-opus-5-5",
+ "messages": [{"role": "user", "content": "Weather?"}, replayed_assistant],
+ "standard_logging_object": {"id": "server-tools-call", "call_type": "acompletion", "metadata": {}},
+ },
+ response_obj,
+ )
+ (finished,) = exporter.get_finished_spans()
+ input_messages: Final = json.loads(finished.attributes[SpanAttributes.GEN_AI_INPUT_MESSAGES.value])
+ output_messages: Final = json.loads(finished.attributes[SpanAttributes.GEN_AI_OUTPUT_MESSAGES.value])
+ assert [call["id"] for call in input_messages[1]["tool_calls"]] == ["srvtoolu_01", "toolu_01"]
+ assert [call["id"] for call in output_messages[0]["tool_calls"]] == ["srvtoolu_01", "toolu_01"]
+
+
def test_v1_pod_routing_none_matches_other_none_metadata_attributes() -> None:
exporter: Final = InMemorySpanExporter()
provider: Final = TracerProvider()
diff --git a/tests/unit/integrations/test_shadow_eval_logger.py b/tests/unit/integrations/test_shadow_eval_logger.py
index f802dbfdba9..ab8563c7ced 100644
--- a/tests/unit/integrations/test_shadow_eval_logger.py
+++ b/tests/unit/integrations/test_shadow_eval_logger.py
@@ -1905,6 +1905,40 @@ class TestShadowPipeline:
assert "Let me look that up. [tool call] Read({})" in judge_prompt
+ async def test_a_server_tool_call_reaches_the_judge_ahead_of_client_tool_calls(self):
+ router: Final = _shadow_reply_router(
+ {
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+ }
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ },
+ finish_reason="tool_calls",
+ )
+ await self._judged_shadow_row(router)
+
+ judge_prompt: Final = next(
+ call.kwargs["messages"][-1]["content"]
+ for call in router.acompletion.call_args_list
+ if call.kwargs["metadata"].get(INTERNAL_CALL_ORIGIN_METADATA_KEY) != SHADOW_EVAL_ROUTER_CALL_ORIGIN
+ )
+
+ shadow_text: Final = '[tool call] web_search({"query": "persimmon"}), [tool call] get_weather({"city": "SF"})'
+ assert f":\n{shadow_text}\n\n" in judge_prompt, judge_prompt
+
async def test_an_empty_shadow_reply_names_the_finish_reason_and_the_routed_model(self):
"""A reply that really carried no text is diagnosable only if the row says what
the arm was doing when it produced none: a truncated turn and a model that answers
diff --git a/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_common_utils.py b/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_common_utils.py
index 390da500c32..9d3c19b7460 100644
--- a/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_common_utils.py
+++ b/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_common_utils.py
@@ -11,6 +11,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
TOOL_RESULT_MEDIA_BOUNDARY,
TOOL_RESULT_MEDIA_PLACEHOLDER,
add_system_prompt_to_messages,
+ get_content_from_model_response,
get_semantic_cache_prompt_from_messages,
get_str_from_messages,
handle_any_messages_to_chat_completion_str_messages_conversion,
@@ -21,6 +22,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
system_messages_first,
)
from litellm.types.llms.openai import ChatCompletionAudioObject, ChatCompletionFileObject, ChatCompletionImageObject
+from litellm.types.utils import ChatCompletionMessageToolCall, Choices, Function, Message, ModelResponse
_ARTIFACT_FIELD_PATTERN: Final = r'^(?!__.*__$)[^\p{Cc}\p{Cf}\p{Zl}\p{Zp}"\\./[\]]{1,200}$'
_ARTIFACT_DATA_ID_PATTERN: Final = r"^(?!\.\.?(?:\/|$))[A-Za-z0-9_\-.~:@+]{1,200}$"
@@ -983,3 +985,30 @@ def test_content_list_flattening_returns_new_messages_and_leaves_the_callers_unt
assert flattened == [{"role": "user", "content": "Hello world"}, {"role": "assistant", "content": "Hi"}]
assert messages == messages_before
+
+
+def test_get_content_from_model_response_includes_server_tool_calls_before_client_calls() -> None:
+ server_call: Final = ChatCompletionMessageToolCall(
+ id="srvtoolu_01", type="function", function=Function(name="web_search", arguments='{"query": "persimmon"}')
+ )
+ client_call: Final = ChatCompletionMessageToolCall(
+ id="call_1", type="function", function=Function(name="get_weather", arguments='{"city": "SF"}')
+ )
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content="Looking it up.",
+ tool_calls=[client_call.model_dump()],
+ provider_specific_fields={"server_tool_calls": [server_call.model_dump()]},
+ ),
+ )
+ ],
+ )
+
+ assert get_content_from_model_response(response) == (
+ "Looking it up." + server_call.model_dump_json() + client_call.model_dump_json()
+ )
diff --git a/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py b/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py
index bb88f632073..3270cad80c9 100644
--- a/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py
+++ b/tests/unit/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py
@@ -74,7 +74,12 @@ from litellm.llms.vertex_ai.gemini.transformation import gemini_convert_messages
from litellm.llms.vllm.completion import handler as vllm_handler
from litellm.llms.watsonx.common_utils import _convert_watsonx_messages_core
from litellm.types.llms.bedrock import BedrockConverseReasoningContentBlock
-from litellm.types.llms.openai import AllMessageValues, ChatCompletionAssistantMessage, ChatCompletionToolMessage
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
from litellm.types.utils import ChatCompletionMessageToolCall, Function, ModelResponse, TextCompletionResponse
from litellm.utils import (
Rules,
@@ -435,6 +440,46 @@ def test_anthropic_messages_pt_drops_the_stream_index_from_replayed_thinking_blo
]
+@pytest.mark.parametrize(
+ ("model", "expected_content"),
+ (
+ (
+ "global.anthropic.claude-haiku-5-5",
+ [
+ {"reasoningContent": {"reasoningText": {"text": "pondering", "signature": "signed-reasoning"}}},
+ {"text": "hello"},
+ ],
+ ),
+ ("global.zai.glm-5.3", [{"text": "hello"}]),
+ ("amazon.nova-pro-v1:0", [{"text": "hello"}]),
+ ("converse/us.openai.gpt-6-luna", [{"text": "hello"}]),
+ ),
+)
+def test_bedrock_converse_replays_signed_thinking_only_to_claude(
+ model: str, expected_content: Sequence[Mapping[str, object]]
+) -> None:
+ body: Final = litellm.AmazonConverseConfig().transform_request(
+ model=model,
+ messages=[
+ ChatCompletionUserMessage(role="user", content="hi"),
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ content="hello",
+ thinking_blocks=[{"type": "thinking", "thinking": "pondering", "signature": "signed-reasoning"}],
+ ),
+ ChatCompletionUserMessage(role="user", content="continue"),
+ ],
+ optional_params={},
+ litellm_params={},
+ headers={},
+ )
+ assert body["messages"] == [
+ {"role": "user", "content": [{"text": "hi"}]},
+ {"role": "assistant", "content": expected_content},
+ {"role": "user", "content": [{"text": "continue"}]},
+ ]
+
+
async def test_anthropic_bedrock_thinking_blocks_with_none_content():
"""
Test the specific function that processes thinking_blocks when content is None
@@ -1032,6 +1077,192 @@ async def test_bedrock_converse_keeps_dropping_redacted_thinking_for_non_claude_
assert assistant_content == [{"text": "hello"}]
+_HIDDEN_MODEL_ARN: Final = "arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/abc123"
+
+
+async def _converse_messages(
+ messages: list[dict[str, object]],
+ model: str,
+ base_model: str | None,
+ use_async: bool,
+ assistant_continue_message: str | None = None,
+) -> list[dict[str, object]]:
+ return (
+ await BedrockConverseMessagesProcessor._bedrock_converse_messages_pt_async(
+ messages=messages,
+ model=model,
+ llm_provider="bedrock",
+ base_model=base_model,
+ assistant_continue_message=assistant_continue_message,
+ )
+ if use_async
+ else _bedrock_converse_messages_pt(
+ messages=messages,
+ model=model,
+ llm_provider="bedrock",
+ base_model=base_model,
+ assistant_continue_message=assistant_continue_message,
+ )
+ )
+
+
+def _assistant_with_thinking(placement: str, block: dict[str, object]) -> dict[str, object]:
+ return (
+ {"role": "assistant", "content": "hello", "thinking_blocks": [block]}
+ if placement == "thinking_blocks"
+ else {"role": "assistant", "content": [block, {"type": "text", "text": "hello"}]}
+ )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("use_async", [False, True])
+@pytest.mark.parametrize("placement", ["thinking_blocks", "content"])
+@pytest.mark.parametrize(
+ "signature", ["claude-signature", encrypted_reasoning_signature("gAAAA_1")], ids=["claude", "encrypted"]
+)
+@pytest.mark.parametrize(
+ ("model", "base_model"),
+ [
+ ("converse/us.openai.gpt-6-luna", None),
+ ("us.amazon.nova-pro-v1:0", None),
+ ("us.amazon.nova-pro-v1:0", "anthropic.claude-sonnet-4-6"),
+ (_HIDDEN_MODEL_ARN, "openai.gpt-oss-120b-1:0"),
+ ],
+)
+async def test_bedrock_converse_drops_signed_thinking_for_a_known_non_claude_target(
+ use_async: bool, placement: str, signature: str, model: str, base_model: str | None
+):
+ block: Final = {"type": "thinking", "thinking": "private plan", "signature": signature}
+ messages: Final = [
+ {"role": "user", "content": "hi"},
+ _assistant_with_thinking(placement, block),
+ {"role": "user", "content": "again"},
+ ]
+
+ assert await _converse_messages(messages, model, base_model, use_async) == [
+ {"role": "user", "content": [{"text": "hi"}]},
+ {"role": "assistant", "content": [{"text": "hello"}]},
+ {"role": "user", "content": [{"text": "again"}]},
+ ]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("use_async", [False, True])
+@pytest.mark.parametrize("placement", ["thinking_blocks", "content"])
+@pytest.mark.parametrize(
+ ("model", "base_model"),
+ [
+ ("us.anthropic.claude-sonnet-4-6", None),
+ ("ca.anthropic.claude-haiku-5-5", None),
+ ("converse/in.anthropic.claude-sonnet-5", None),
+ ("claude-sonnet-4-6", None),
+ (_HIDDEN_MODEL_ARN, None),
+ (_HIDDEN_MODEL_ARN, "anthropic.claude-sonnet-4-6"),
+ ],
+)
+async def test_bedrock_converse_keeps_signed_thinking_for_a_target_that_may_be_claude(
+ use_async: bool, placement: str, model: str, base_model: str | None
+):
+ block: Final = {"type": "thinking", "thinking": "private plan", "signature": "claude-signature"}
+ messages: Final = [
+ {"role": "user", "content": "hi"},
+ _assistant_with_thinking(placement, block),
+ {"role": "user", "content": "again"},
+ ]
+
+ result: Final = await _converse_messages(messages, model, base_model, use_async)
+
+ assistant_content: Final = next(m["content"] for m in result if m["role"] == "assistant")
+ assert assistant_content == [
+ {"reasoningContent": {"reasoningText": {"text": "private plan", "signature": "claude-signature"}}},
+ {"text": "hello"},
+ ]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("use_async", [False, True])
+async def test_bedrock_converse_keeps_the_tool_call_when_dropping_signed_thinking(use_async: bool):
+ messages: Final = [
+ {"role": "user", "content": "weather?"},
+ {
+ "role": "assistant",
+ "content": None,
+ "thinking_blocks": [{"type": "thinking", "thinking": "private plan", "signature": "claude-signature"}],
+ "tool_calls": [
+ {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}}
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call_1", "content": "sunny"},
+ ]
+
+ result: Final = await _converse_messages(messages, "converse/us.openai.gpt-6-luna", None, use_async)
+
+ assert [m["role"] for m in result] == ["user", "assistant", "user"]
+ assert [list(part) for part in result[1]["content"]] == [["toolUse"]]
+ assert result[1]["content"][0]["toolUse"]["toolUseId"] == "call_1"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("use_async", [False, True])
+@pytest.mark.parametrize(
+ ("assistant_continue_message", "expected"),
+ [
+ (None, [{"role": "user", "content": [{"text": "hi"}, {"text": "again"}]}]),
+ (
+ "Please continue.",
+ [
+ {"role": "user", "content": [{"text": "hi"}]},
+ {"role": "assistant", "content": [{"text": "Please continue."}]},
+ {"role": "user", "content": [{"text": "again"}]},
+ ],
+ ),
+ ],
+ ids=["merged_into_one_user_turn", "split_by_the_continue_message"],
+)
+async def test_bedrock_converse_drops_a_turn_that_held_only_signed_thinking(
+ use_async: bool, assistant_continue_message: str | None, expected: list[dict[str, object]]
+):
+ messages: Final = [
+ {"role": "user", "content": "hi"},
+ {
+ "role": "assistant",
+ "content": None,
+ "thinking_blocks": [{"type": "thinking", "thinking": "private plan", "signature": "claude-signature"}],
+ },
+ {"role": "user", "content": "again"},
+ ]
+
+ result: Final = await _converse_messages(
+ messages, "converse/us.openai.gpt-6-luna", None, use_async, assistant_continue_message
+ )
+
+ assert result == expected
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("use_async", [False, True])
+async def test_bedrock_converse_drops_the_cache_point_of_a_dropped_signed_thinking_block(use_async: bool):
+ messages: Final = [
+ {"role": "user", "content": "hi"},
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "thinking",
+ "thinking": "private plan",
+ "signature": "claude-signature",
+ "cache_control": {"type": "ephemeral"},
+ }
+ ],
+ },
+ {"role": "user", "content": "again"},
+ ]
+
+ result: Final = await _converse_messages(messages, "converse/us.amazon.nova-2-lite-v1:0", None, use_async)
+
+ assert result == [{"role": "user", "content": [{"text": "hi"}, {"text": "again"}]}]
+
+
@pytest.mark.parametrize(
"thinking_block",
[
diff --git a/tests/unit/litellm_core_utils/test_get_llm_provider_logic.py b/tests/unit/litellm_core_utils/test_get_llm_provider_logic.py
index 5779250950d..de252b17f00 100644
--- a/tests/unit/litellm_core_utils/test_get_llm_provider_logic.py
+++ b/tests/unit/litellm_core_utils/test_get_llm_provider_logic.py
@@ -19,6 +19,7 @@ from litellm.litellm_core_utils.get_llm_provider_logic import (
)
from litellm.llms.custom_httpx.http_handler import HTTPHandler
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
+from litellm.litellm_core_utils.model_latest_aliases import get_model_latest_aliases
from litellm.types.router import LiteLLM_Params
from litellm.utils import _invalidate_model_cost_lowercase_map
from tests._vcr_conftest_common import install_live_call_probe, record_vcr_outcome
@@ -27,6 +28,38 @@ from unittest.mock import patch
CUSTOM_PROVIDER: Final = "test-onprem-llm"
+@pytest.mark.parametrize(
+ ("model", "provider"),
+ (("openai/gpt-luna-latest", None), ("gpt-luna-latest", "openai")),
+)
+def test_latest_alias_resolves_to_the_bundled_target(model: str, provider: str | None) -> None:
+ target: Final = get_model_latest_aliases()["openai"]["gpt-luna-latest"]
+ resolved, resolved_provider, _, api_base = get_llm_provider(
+ model, custom_llm_provider=provider, api_base="https://api.openai.com/v1"
+ )
+ assert (resolved, resolved_provider, api_base) == (target, "openai", "https://api.openai.com/v1")
+
+
+@pytest.mark.parametrize(
+ "model",
+ ("gemini/gemini-pro-latest", "gemini/gemini-flash-latest", "dashscope/qwen-plus-latest", "mistral/mistral-large-latest"),
+)
+def test_provider_native_latest_alias_passes_through(model: str) -> None:
+ provider, name = model.split("/", 1)
+ assert get_llm_provider(model)[:2] == (name, provider)
+
+
+def test_latest_alias_completion_reports_and_prices_the_target() -> None:
+ target: Final = get_model_latest_aliases()["anthropic"]["claude-opus-latest"]
+ response: Final = litellm.completion(
+ model="anthropic/claude-opus-latest", messages=[{"role": "user", "content": "hi"}], mock_response="hi"
+ )
+ assert response.model == target
+ cost: Final = litellm.completion_cost(completion_response=response)
+ assert cost is not None and cost > 0
+ assert cost == litellm.completion_cost(completion_response=response, model=target, custom_llm_provider="anthropic")
+
+
@pytest.fixture
def registered_custom_provider(monkeypatch: pytest.MonkeyPatch) -> str:
monkeypatch.setattr(litellm, "custom_provider_map", [{"provider": CUSTOM_PROVIDER, "custom_handler": CustomLLM()}])
diff --git a/tests/unit/litellm_core_utils/test_model_latest_aliases.py b/tests/unit/litellm_core_utils/test_model_latest_aliases.py
new file mode 100644
index 00000000000..c052a24b4dc
--- /dev/null
+++ b/tests/unit/litellm_core_utils/test_model_latest_aliases.py
@@ -0,0 +1,47 @@
+from typing import Final
+
+import pytest
+
+import litellm
+from litellm.litellm_core_utils.model_latest_aliases import (
+ expand_model_latest_alias,
+ get_model_latest_aliases,
+ is_model_latest_alias,
+)
+
+
+@pytest.mark.parametrize("model", ("", "gpt-luna-latest", "openai/unknown", "gemini/gemini-pro-latest"))
+def test_non_alias_expansion_keeps_the_requested_name(model: str) -> None:
+ assert expand_model_latest_alias(model) == (model,)
+
+
+def test_alias_expansion_keeps_requested_and_target_names() -> None:
+ for provider, aliases in get_model_latest_aliases().items():
+ for alias, target in aliases.items():
+ assert expand_model_latest_alias(f"{provider}/{alias}") == (f"{provider}/{alias}", f"{provider}/{target}")
+
+
+def test_latest_alias_targets_have_chat_pricing_without_shadowing_native_models() -> None:
+ for provider, aliases in get_model_latest_aliases().items():
+ for alias, target in aliases.items():
+ target_key: Final = f"{provider}/{target}" if f"{provider}/{target}" in litellm.model_cost else target
+ entry: Final = litellm.model_cost[target_key]
+ assert (entry["litellm_provider"], entry["mode"]) == (provider, "chat")
+ assert f"{provider}/{alias}" not in litellm.model_cost
+ assert litellm.model_cost.get(alias, {}).get("litellm_provider") != provider
+ assert litellm.get_llm_provider(f"{provider}/{alias}")[:2] == (target, provider)
+
+
+@pytest.mark.parametrize(
+ ("model", "provider", "expected"),
+ (
+ ("openai/gpt-luna-latest", None, True),
+ ("gpt-luna-latest", "openai", True),
+ ("gpt-luna-latest", None, False),
+ ("gemini/gemini-pro-latest", None, False),
+ ("", None, False),
+ ("openai/unknown-model", None, False),
+ ),
+)
+def test_latest_alias_detection_requires_an_explicit_provider(model: str, provider: str | None, expected: bool) -> None:
+ assert is_model_latest_alias(model, provider) is expected
diff --git a/tests/unit/litellm_core_utils/test_redact_messages.py b/tests/unit/litellm_core_utils/test_redact_messages.py
index 491ea5315ff..8e4bed02d7e 100644
--- a/tests/unit/litellm_core_utils/test_redact_messages.py
+++ b/tests/unit/litellm_core_utils/test_redact_messages.py
@@ -11,7 +11,7 @@ import importlib
import json
import os
import threading
-from collections.abc import AsyncIterator, Mapping
+from collections.abc import AsyncIterator, Iterable, Mapping
from datetime import datetime
from types import MappingProxyType, SimpleNamespace
from typing import Final, Optional, Union
@@ -20,10 +20,11 @@ from unittest.mock import patch
import httpx
import pytest
import pytest_asyncio
-from pydantic import JsonValue
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from pydantic import JsonValue, TypeAdapter
import litellm
-from litellm.constants import LOGGING_WORKER_MAX_TIME_PER_COROUTINE
+from litellm.constants import LOGGING_WORKER_MAX_TIME_PER_COROUTINE, REDACTED_BY_LITELLM
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
from litellm.litellm_core_utils.redact_messages import (
@@ -37,6 +38,7 @@ from litellm.litellm_core_utils.redact_messages import (
)
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.responses.main import mock_responses_api_response
+from litellm.types.responses.main import build_web_search_call
from litellm.types.router import BaselineRouteStamp
from litellm.types.utils import (
ModelResponse,
@@ -236,6 +238,116 @@ def test_streaming_redaction_redacts_delta_provider_fields_without_touching_the_
assert fields == _provider_fields()
+_SECRET_URL: Final = f"https://example.com/?q={SECRET}"
+_WEB_SEARCH_CALL: Final = TypeAdapter(ResponseFunctionWebSearch)
+
+
+def _claude_fetch_call() -> ResponseFunctionWebSearch:
+ return build_web_search_call(
+ "srvtoolu_01Fetch", {"url": _SECRET_URL}, {"content": []}, status="in_progress", tool_name="web_fetch"
+ )
+
+
+def _web_call_summaries(
+ calls: Mapping[str, object] | Iterable[object],
+) -> list[tuple[str, str | None, dict[str, object]]]:
+ items: Final = calls.values() if isinstance(calls, dict) else calls
+ return [
+ (call.id, call.status, call.action.model_dump(exclude_none=True))
+ for call in (_WEB_SEARCH_CALL.validate_python(item) for item in items)
+ ]
+
+
+@pytest.mark.parametrize("shape", ["object", "dict"])
+def test_redaction_hides_claude_server_call_inputs_but_keeps_the_calls(shape: str) -> None:
+ response: Final = ModelResponse(
+ model="claude-opus-5-5",
+ choices=[
+ {
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01Fetch",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": json.dumps({"url": _SECRET_URL})},
+ }
+ ],
+ "web_search_calls": [
+ _claude_fetch_call(),
+ build_web_search_call(
+ "srvtoolu_01Search", {"query": SECRET}, {"content": []}, status="in_progress"
+ ),
+ ],
+ },
+ }
+ }
+ ],
+ )
+ result: Final = response if shape == "object" else response.model_dump()
+
+ redacted: Final = perform_redaction({}, result)
+
+ message: Final = (
+ redacted.choices[0].message if shape == "object" else SimpleNamespace(**redacted["choices"][0]["message"])
+ )
+ fields: Final = message.provider_specific_fields
+ assert fields["server_tool_calls"][0]["id"] == "srvtoolu_01Fetch"
+ assert fields["server_tool_calls"][0]["function"] == {"name": "web_fetch", "arguments": REDACTED_BY_LITELLM}
+ assert _web_call_summaries(fields["web_search_calls"]) == [
+ ("ws_srvtoolu_01Fetch", "in_progress", {"type": "open_page", "url": REDACTED_BY_LITELLM}),
+ ("ws_srvtoolu_01Search", "in_progress", {"type": "search", "query": REDACTED_BY_LITELLM}),
+ ]
+ assert SECRET not in str(redacted), redacted
+ assert SECRET in str(result)
+
+
+@pytest.mark.parametrize("key", ["complete_streaming_response", "async_complete_streaming_response"])
+def test_streaming_redaction_hides_the_hosted_copy_of_a_claude_web_fetch(key: str) -> None:
+ response: Final = ModelResponseStream(
+ choices=[
+ litellm.utils.StreamingChoices(
+ delta=litellm.utils.Delta(
+ provider_specific_fields={"web_search_calls": {"srvtoolu_01Fetch": _claude_fetch_call()}}
+ )
+ )
+ ]
+ )
+
+ perform_redaction({"stream": True, key: response}, None)
+
+ assert _web_call_summaries(response.choices[0].delta.provider_specific_fields["web_search_calls"]) == [
+ ("ws_srvtoolu_01Fetch", "in_progress", {"type": "open_page", "url": REDACTED_BY_LITELLM})
+ ]
+
+
+@pytest.mark.parametrize("shape", ["object", "dict"])
+def test_redaction_hides_web_search_call_actions_in_responses_api_output(shape: str) -> None:
+ response: Final = ResponsesAPIResponse(
+ id="resp_1",
+ created_at=1,
+ model="claude-opus-5-5",
+ object="response",
+ status="completed",
+ output=[
+ build_web_search_call("srvtoolu_01Fetch", {"url": _SECRET_URL}, {"content": []}, tool_name="web_fetch"),
+ build_web_search_call("srvtoolu_01Search", {"query": SECRET}, {"content": []}),
+ ],
+ )
+ result: Final = response if shape == "object" else response.model_dump()
+
+ redacted: Final = perform_redaction({}, result)
+
+ output: Final = redacted.output if shape == "object" else redacted["output"]
+ assert _web_call_summaries(output) == [
+ ("ws_srvtoolu_01Fetch", "completed", {"type": "open_page", "url": REDACTED_BY_LITELLM}),
+ ("ws_srvtoolu_01Search", "completed", {"type": "search", "query": REDACTED_BY_LITELLM}),
+ ]
+ assert SECRET in str(result)
+
+
@pytest.mark.parametrize("shape", ["object", "dict"])
def test_redaction_hides_a_compaction_summary_in_responses_api_output(shape: str) -> None:
response: Final = ResponsesAPIResponse(
diff --git a/tests/unit/llms/anthropic/chat/guardrail_translation/test_anthropic_guardrail_handler.py b/tests/unit/llms/anthropic/chat/guardrail_translation/test_anthropic_guardrail_handler.py
index c2ee58b67ed..8affe2e136e 100644
--- a/tests/unit/llms/anthropic/chat/guardrail_translation/test_anthropic_guardrail_handler.py
+++ b/tests/unit/llms/anthropic/chat/guardrail_translation/test_anthropic_guardrail_handler.py
@@ -594,6 +594,57 @@ class TestAnthropicMessagesHandlerStreamingOutputProcessing:
assert '"id": "srvtoolu_1"' in raw and '"id": "toolu_1"' in raw
assert "persim" not in raw and '"query"' not in raw
+ @pytest.mark.asyncio
+ async def test_deliver_ended_stream_tool_use_rewrite_lands_on_its_own_block_when_server_tool_use_follows(self):
+ class MaskFruitLookup(CustomGuardrail):
+ async def apply_guardrail(
+ self,
+ inputs: GenericGuardrailAPIInputs,
+ request_data: dict[str, object],
+ input_type: Literal["request", "response"],
+ logging_obj: LiteLLMLoggingObj | None = None,
+ ) -> GenericGuardrailAPIInputs:
+ for tool_call in inputs.get("tool_calls", []):
+ if tool_call.function.name == "lookup_fruit":
+ tool_call.function.arguments = '{"fruit": "[MASKED]"}'
+ return inputs
+
+ handler: Final = AnthropicMessagesHandler()
+ tool_use: Final = self._ended_tool_use_sse_chunks()
+ server_tool_use: Final = [
+ (
+ "content_block_start",
+ {
+ "type": "content_block_start",
+ "index": 1,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_1", "name": "web_search", "input": {}},
+ },
+ ),
+ (
+ "content_block_delta",
+ {
+ "type": "content_block_delta",
+ "index": 1,
+ "delta": {"type": "input_json_delta", "partial_json": '{"query": "fruit"}'},
+ },
+ ),
+ ("content_block_stop", {"type": "content_block_stop", "index": 1}),
+ ]
+ chunks: Final = (
+ tool_use[:-2]
+ + [f"event: {name}\ndata: {json.dumps(payload)}\n\n".encode() for name, payload in server_tool_use]
+ + tool_use[-2:]
+ )
+
+ await handler.process_output_streaming_response(
+ responses_so_far=chunks,
+ guardrail_to_apply=MaskFruitLookup(guardrail_name="test"),
+ litellm_logging_obj=MagicMock(),
+ deliver_ended_stream_rewrites=True,
+ )
+
+ assert self._partial_jsons(chunks) == ['{"fruit": "[MASKED]"}', "", "", '{"query": "fruit"}']
+
@pytest.mark.asyncio
async def test_deliver_ended_stream_tool_use_rewrite_on_an_unindexed_tool_block_fails_closed(self):
from litellm.proxy.policy_engine.pipeline_executor import UndeliverableStreamRewrite
diff --git a/tests/unit/llms/anthropic/chat/test_anthropic_chat_handler.py b/tests/unit/llms/anthropic/chat/test_anthropic_chat_handler.py
index e81ee2a4e02..2fb4eb9931c 100644
--- a/tests/unit/llms/anthropic/chat/test_anthropic_chat_handler.py
+++ b/tests/unit/llms/anthropic/chat/test_anthropic_chat_handler.py
@@ -17,7 +17,9 @@ from litellm._uuid import uuid
from litellm.constants import NON_LLM_CONNECTION_TIMEOUT, RESPONSE_FORMAT_TOOL_NAME
from litellm.litellm_core_utils.prompt_templates import image_handling
from litellm.litellm_core_utils.prompt_templates.factory import anthropic_messages_pt
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
from litellm.llms.anthropic.chat.handler import ModelResponseIterator, _envelope_verdict, make_call, make_sync_call
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
from litellm.llms.base_llm.base_utils import response_format_schema, unwrap_response_format_envelope
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.types.llms.openai import (
@@ -25,7 +27,7 @@ from litellm.types.llms.openai import (
ChatCompletionToolCallFunctionChunk,
)
from litellm.types.responses.main import OutputCodeInterpreterCall
-from litellm.types.utils import ChatCompletionDeltaToolCall, ModelResponseStream
+from litellm.types.utils import ChatCompletionDeltaToolCall, ChatCompletionMessageToolCall, Choices, ModelResponseStream
from tests.unit.llms.anthropic.anthropic_docs_refusal import ANTHROPIC_DOCS_REFUSAL_MESSAGE
@@ -1966,19 +1968,113 @@ def test_web_search_tool_result_no_extra_tool_calls():
for tc in parsed.choices[0].delta.tool_calls:
tool_calls_emitted.append(tc)
- # Should have exactly 2 tool calls:
- # 1. From content_block_start (server_tool_use) with id and name
- # 2. From content_block_delta with the actual query
- assert len(tool_calls_emitted) == 2, f"Expected 2 tool calls, got {len(tool_calls_emitted)}"
+ assert tool_calls_emitted == []
- # First tool call should have the id and name
- assert tool_calls_emitted[0]["id"] == "srvtoolu_01ABC123"
- assert tool_calls_emitted[0]["function"]["name"] == "web_search"
- # Second tool call should have the query arguments
- assert tool_calls_emitted[1]["function"]["arguments"] == '{"query": "test"}'
+def _server_tool_input_delta(fragment: str) -> dict[str, object]:
+ return {"type": "content_block_delta", "index": 0, "delta": {"type": "input_json_delta", "partial_json": fragment}}
- # The {} chunks from web_search_tool_result should NOT have been emitted as tool calls
+
+def _server_search_then_answer(*partial_json: str) -> tuple[dict[str, object], ...]:
+ return (
+ {
+ "type": "message_start",
+ "message": {"id": "msg_01", "model": "claude-opus-5-5", "usage": {"input_tokens": 9}},
+ },
+ {
+ "type": "content_block_start",
+ "index": 0,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_01Web", "name": "web_search", "input": {}},
+ },
+ *(_server_tool_input_delta(fragment) for fragment in partial_json),
+ {"type": "content_block_stop", "index": 0},
+ {
+ "type": "content_block_start",
+ "index": 1,
+ "content_block": {"type": "web_search_tool_result", "tool_use_id": "srvtoolu_01Web", "content": []},
+ },
+ {"type": "content_block_stop", "index": 1},
+ {"type": "content_block_start", "index": 2, "content_block": {"type": "text", "text": ""}},
+ {"type": "content_block_delta", "index": 2, "delta": {"type": "text_delta", "text": "Paris is in France."}},
+ {"type": "content_block_stop", "index": 2},
+ )
+
+
+_SERVER_SEARCH_THEN_ANSWER: Final = _server_search_then_answer('{"query": "Paris"}')
+_CLIENT_TOOL_CALL: Final[tuple[dict[str, object], ...]] = (
+ {
+ "type": "content_block_start",
+ "index": 3,
+ "content_block": {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {}},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 3,
+ "delta": {"type": "input_json_delta", "partial_json": '{"city": "Paris"}'},
+ },
+ {"type": "content_block_stop", "index": 3},
+)
+
+
+def _message_delta(stop_reason: str) -> dict[str, object]:
+ return {"type": "message_delta", "delta": {"stop_reason": stop_reason}, "usage": {"output_tokens": 20}}
+
+
+_PARIS_SEARCH: Final = ("srvtoolu_01Web", "web_search", '{"query": "Paris"}')
+
+
+@pytest.mark.parametrize(
+ ("events", "expected_tool_calls", "expected_finish_reason", "expected_server_tool_call"),
+ [
+ pytest.param(
+ (*_SERVER_SEARCH_THEN_ANSWER, _message_delta("end_turn")), [], "stop", _PARIS_SEARCH, id="server_tool_only"
+ ),
+ pytest.param(
+ (*_SERVER_SEARCH_THEN_ANSWER, *_CLIENT_TOOL_CALL, _message_delta("tool_use")),
+ [("toolu_01Weather", "get_weather", '{"city": "Paris"}')],
+ "tool_calls",
+ _PARIS_SEARCH,
+ id="server_tool_then_client_tool",
+ ),
+ pytest.param(
+ (*_server_search_then_answer('{"que', 'ry": "Pa', 'ris"}'), _message_delta("end_turn")),
+ [],
+ "stop",
+ _PARIS_SEARCH,
+ id="server_tool_input_split_across_deltas",
+ ),
+ pytest.param(
+ (*_server_search_then_answer(), _message_delta("end_turn")),
+ [],
+ "stop",
+ ("srvtoolu_01Web", "web_search", "{}"),
+ id="server_tool_with_empty_input",
+ ),
+ ],
+)
+def test_streamed_server_tool_use_is_not_returned_as_a_client_tool_call(
+ events: tuple[dict[str, object], ...],
+ expected_tool_calls: list[tuple[str, str, str]],
+ expected_finish_reason: str,
+ expected_server_tool_call: tuple[str, str, str],
+):
+ choice: Final = _assemble_stream(events).choices[0]
+ assert isinstance(choice, Choices)
+
+ assert (choice.message.tool_calls is None) == (not expected_tool_calls)
+ assert [
+ (call.id, call.function.name, call.function.arguments)
+ for call in choice.message.tool_calls or ()
+ if isinstance(call, ChatCompletionMessageToolCall)
+ ] == expected_tool_calls
+ assert choice.finish_reason == expected_finish_reason
+ assert choice.message.content == "Paris is in France."
+ assert [
+ (call["id"], call["function"].get("name"), call["function"].get("arguments"))
+ for call in anthropic_server_tool_call_params(choice.message)
+ ] == [expected_server_tool_call]
+ web_search_results: Final = message_field(choice.message.provider_specific_fields, "web_search_results")
+ assert [message_field(result, "tool_use_id") for result in parts_of(web_search_results)] == ["srvtoolu_01Web"]
def test_current_content_block_type_tracking():
diff --git a/tests/unit/llms/anthropic/chat/test_anthropic_chat_transformation.py b/tests/unit/llms/anthropic/chat/test_anthropic_chat_transformation.py
index 95bc8af7270..ae72af200d4 100644
--- a/tests/unit/llms/anthropic/chat/test_anthropic_chat_transformation.py
+++ b/tests/unit/llms/anthropic/chat/test_anthropic_chat_transformation.py
@@ -2,9 +2,9 @@ import copy
import itertools
import json
import logging
-from collections.abc import Iterator
+from collections.abc import Iterator, Mapping
from itertools import chain
-from typing import Final
+from typing import Final, cast
from unittest.mock import MagicMock, patch
import httpx
@@ -25,12 +25,18 @@ from litellm.constants import (
RESPONSE_FORMAT_UNFORCED_TOOL_DESCRIPTION,
)
from litellm.litellm_core_utils.prompt_templates.factory import anthropic_messages_pt
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
from litellm.litellm_core_utils.prompt_templates.reasoning_utils import (
encrypted_reasoning_signature,
)
from litellm.llms.anthropic.chat import ModelResponseIterator
+from litellm.llms.anthropic.common_utils import (
+ AnthropicError,
+ anthropic_server_tool_call_params,
+ process_anthropic_headers,
+ validated_object_mapping,
+)
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
-from litellm.llms.anthropic.common_utils import AnthropicError, process_anthropic_headers
from litellm.llms.anthropic.messages.native.transformation import (
AnthropicMessagesConfig,
)
@@ -43,10 +49,14 @@ from litellm.llms.bedrock.claude_platform.transformation import BedrockClaudePla
from litellm.llms.vertex_ai.vertex_ai_partner_models.anthropic.transformation import (
VertexAIAnthropicConfig,
)
-from litellm.types.llms.anthropic import ANTHROPIC_BETA_HEADER_VALUES, ANTHROPIC_TOOL_SEARCH_TOOL_TYPES
+from litellm.types.llms.anthropic import (
+ ANTHROPIC_BETA_HEADER_VALUES,
+ ANTHROPIC_TOOL_SEARCH_TOOL_TYPES,
+ AllAnthropicPassThroughMessageValues,
+)
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicMessagesResponse
-from litellm.types.llms.openai import ChatCompletionToolCallFunctionChunk
-from litellm.types.utils import ChatCompletionToolCallChunk, ModelResponse, ServerToolUse, Usage
+from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolCallFunctionChunk
+from litellm.types.utils import ChatCompletionToolCallChunk, Choices, Message, ModelResponse, ServerToolUse, Usage
from tests.unit.llms.anthropic.anthropic_docs_refusal import ANTHROPIC_DOCS_REFUSAL_MESSAGE
@@ -3624,21 +3634,17 @@ def test_code_execution_tool_results_extraction():
prefix_prompt=None,
)
- # Verify tool calls are present
- assert transformed_response.choices[0].message.tool_calls is not None
- assert len(transformed_response.choices[0].message.tool_calls) == 2
+ assert transformed_response.choices[0].message.tool_calls is None
- # Verify first tool call
- assert transformed_response.choices[0].message.tool_calls[0].id == "srvtoolu_01ABC"
- assert transformed_response.choices[0].message.tool_calls[0].function.name == "bash_code_execution"
-
- # Verify second tool call
- assert transformed_response.choices[0].message.tool_calls[1].id == "srvtoolu_01DEF"
- assert transformed_response.choices[0].message.tool_calls[1].function.name == "text_editor_code_execution"
-
- # Verify tool results are in provider_specific_fields
provider_fields = transformed_response.choices[0].message.provider_specific_fields
assert provider_fields is not None
+ server_tool_calls: Final = anthropic_server_tool_call_params(transformed_response.choices[0].message)
+ assert [(call["id"], call["function"].get("name")) for call in server_tool_calls] == [
+ ("srvtoolu_01ABC", "bash_code_execution"),
+ ("srvtoolu_01DEF", "text_editor_code_execution"),
+ ]
+
+ # Verify tool results are in provider_specific_fields
assert "tool_results" in provider_fields
assert provider_fields["tool_results"] is not None
assert len(provider_fields["tool_results"]) == 2
@@ -3661,6 +3667,45 @@ def test_code_execution_tool_results_extraction():
assert "Done!" in transformed_response.choices[0].message.content
+def test_server_and_client_tool_calls_split_by_block_type_not_id_prefix():
+ completion_response: Final = {
+ "id": "msg_split",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-sonnet-4-5-20250929",
+ "content": [
+ {"type": "server_tool_use", "id": "toolu_server", "name": "web_search", "input": {"query": "persimmon"}},
+ {"type": "tool_use", "id": "srvtoolu_client", "name": "get_weather", "input": {"city": "Paris"}},
+ ],
+ "stop_reason": "tool_use",
+ "stop_sequence": None,
+ "usage": {"input_tokens": 10, "output_tokens": 5},
+ }
+ raw_response: Final = MagicMock(spec=httpx.Response)
+ raw_response.headers = {}
+
+ message: Final = (
+ AnthropicConfig()
+ .transform_parsed_response(
+ completion_response=completion_response,
+ raw_response=raw_response,
+ model_response=ModelResponse(),
+ json_mode=False,
+ prefix_prompt=None,
+ )
+ .choices[0]
+ .message
+ )
+
+ assert [
+ (call["id"], call["function"].get("name"), call["function"].get("arguments"))
+ for call in anthropic_server_tool_call_params(message)
+ ] == [("toolu_server", "web_search", '{"query": "persimmon"}')]
+ assert [
+ (tool_call.id, tool_call.function.name, tool_call.function.arguments) for tool_call in message.tool_calls or ()
+ ] == [("srvtoolu_client", "get_weather", '{"city": "Paris"}')]
+
+
def test_code_execution_tool_results_in_hidden_params():
"""
Test that tool_results reaches _hidden_params so the Responses API adapter
@@ -3748,7 +3793,7 @@ def _tool_search_turn(tool_references: list[str], last_block: dict[str, object])
]
-def _parsed_assistant_message(content: list[dict[str, object]], stop_reason: str):
+def _parsed_choice(content: list[dict[str, object]], stop_reason: str) -> Choices:
anthropic_response: Final = {
"id": "msg_01XYZ",
"type": "message",
@@ -3763,7 +3808,7 @@ def _parsed_assistant_message(content: list[dict[str, object]], stop_reason: str
raw_response.json.return_value = anthropic_response
raw_response.status_code = 200
raw_response.headers = {}
- return (
+ choice: Final = (
AnthropicConfig()
.transform_parsed_response(
completion_response=anthropic_response,
@@ -3773,14 +3818,19 @@ def _parsed_assistant_message(content: list[dict[str, object]], stop_reason: str
prefix_prompt=None,
)
.choices[0]
- .message
)
+ assert isinstance(choice, Choices)
+ return choice
-_WEATHER_CALL: Final = {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {"city": "Paris"}}
+def _parsed_assistant_message(content: list[dict[str, object]], stop_reason: str) -> Message:
+ return _parsed_choice(content, stop_reason).message
-def _web_search_turn(last_block: dict[str, object]) -> list[dict[str, object]]:
+_WEATHER_TOOL_USE: Final = {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {"city": "Paris"}}
+
+
+def _web_search_turn(last_block: Mapping[str, object]) -> list[dict[str, object]]:
return [
{"type": "server_tool_use", "id": "srvtoolu_01Web", "name": "web_search", "input": {"query": "Paris facts"}},
{
@@ -3796,7 +3846,7 @@ def _web_search_turn(last_block: dict[str, object]) -> list[dict[str, object]]:
}
],
},
- last_block,
+ dict(last_block),
]
@@ -3811,7 +3861,7 @@ def _web_search_turn(last_block: dict[str, object]) -> list[dict[str, object]]:
)
@pytest.mark.parametrize(
"content",
- [_tool_search_turn(["get_weather"], _WEATHER_CALL), _web_search_turn(_WEATHER_CALL)],
+ [_tool_search_turn(["get_weather"], _WEATHER_TOOL_USE), _web_search_turn(_WEATHER_TOOL_USE)],
ids=["tool_search", "web_search"],
)
def test_server_tool_result_round_trips_right_after_its_call(
@@ -3843,8 +3893,148 @@ def test_server_tool_result_round_trips_right_after_its_call(
assert [block["tool_use_id"] for block in replayed[2]["content"]] == ["toolu_01Weather"]
-def _block_ids(message: dict[str, object], block_type: str, id_key: str) -> list[object]:
- return sorted(block[id_key] for block in message["content"] if block["type"] == block_type)
+_FETCH_STILL_WAITING_ON_ITS_RESULT: Final = {
+ "type": "server_tool_use",
+ "id": "srvtoolu_01Fetch",
+ "name": "web_fetch",
+ "input": {"url": "https://example.com/"},
+}
+
+
+@pytest.mark.parametrize("modify_params", [False, True])
+def test_server_call_still_waiting_on_its_result_is_replayed_as_a_server_call(
+ modify_params: bool, monkeypatch: pytest.MonkeyPatch
+):
+ content: Final = [_FETCH_STILL_WAITING_ON_ITS_RESULT, _WEATHER_TOOL_USE]
+ message: Final = _parsed_assistant_message(content, "tool_use")
+ monkeypatch.setattr(litellm, "modify_params", modify_params)
+
+ replayed: Final = _replayed(
+ {"role": "user", "content": "Fetch example.com, then tell me the weather in Paris"},
+ message.model_dump(),
+ {"role": "tool", "tool_call_id": "toolu_01Weather", "content": "18C"},
+ )
+
+ assert replayed[1]["content"] == content
+ assert [turn["role"] for turn in replayed] == ["user", "assistant", "user"]
+ assert [block["tool_use_id"] for block in replayed[2]["content"]] == ["toolu_01Weather"]
+
+
+_FETCH_AND_WEATHER_THINKING: Final = {
+ "type": "thinking",
+ "thinking": "Fetch the page, then check the weather.",
+ "signature": "sig_fetch_weather",
+}
+_CHECKING_BOTH: Final = {"type": "text", "text": "Checking both."}
+
+
+@pytest.mark.parametrize("thinking_inline", [False, True], ids=["thinking_blocks", "inline_thinking"])
+def test_tool_use_turn_waiting_on_a_server_call_keeps_its_thinking_and_both_calls(thinking_inline: bool):
+ parsed: Final = _parsed_assistant_message(
+ [_FETCH_AND_WEATHER_THINKING, _CHECKING_BOTH, _FETCH_STILL_WAITING_ON_ITS_RESULT, _WEATHER_TOOL_USE], "tool_use"
+ ).model_dump()
+ message: Final = (
+ {**parsed, "thinking_blocks": None, "content": [_FETCH_AND_WEATHER_THINKING, _CHECKING_BOTH]}
+ if thinking_inline
+ else parsed
+ )
+
+ replayed: Final = _replayed(
+ {"role": "user", "content": "Fetch example.com, then tell me the weather in Paris"},
+ message,
+ {"role": "tool", "tool_call_id": "toolu_01Weather", "content": "18C"},
+ )
+
+ assistant_content: Final = replayed[1]["content"]
+ assert assistant_content[:2] == [_FETCH_AND_WEATHER_THINKING, _CHECKING_BOTH]
+ assert sorted(assistant_content[2:], key=lambda block: block["id"]) == [
+ _FETCH_STILL_WAITING_ON_ITS_RESULT,
+ _WEATHER_TOOL_USE,
+ ]
+ assert [block["tool_use_id"] for block in replayed[2]["content"]] == ["toolu_01Weather"]
+
+
+_PARIS_TEXT: Final = "Paris is the capital of France."
+_PARIS_ANSWER: Final[Mapping[str, str]] = {"type": "text", "text": _PARIS_TEXT}
+
+
+@pytest.mark.parametrize(
+ ("content", "stop_reason", "expected_tool_call_ids", "expected_finish_reason"),
+ [
+ pytest.param(_web_search_turn(_PARIS_ANSWER), "end_turn", [], "stop", id="web_search_then_answer"),
+ pytest.param(
+ _web_search_turn(_WEATHER_TOOL_USE), "tool_use", ["toolu_01Weather"], "tool_calls", id="web_search_then_tool"
+ ),
+ pytest.param([_WEATHER_TOOL_USE], "tool_use", ["toolu_01Weather"], "tool_calls", id="client_tool_only"),
+ ],
+)
+def test_server_tool_use_is_not_returned_as_a_client_tool_call(
+ content: list[dict[str, object]],
+ stop_reason: str,
+ expected_tool_call_ids: list[str],
+ expected_finish_reason: str,
+):
+ choice: Final = _parsed_choice(content, stop_reason)
+
+ assert (choice.message.tool_calls is None) == (not expected_tool_call_ids)
+ assert [call.id for call in choice.message.tool_calls or ()] == expected_tool_call_ids
+ assert choice.finish_reason == expected_finish_reason
+ assert [
+ (call["id"], call["function"].get("name"), call["function"].get("arguments"))
+ for call in anthropic_server_tool_call_params(choice.message)
+ ] == [
+ (block["id"], block["name"], json.dumps(block["input"]))
+ for block in content
+ if block["type"] == "server_tool_use"
+ ]
+ if not expected_tool_call_ids:
+ assert choice.message.content == _PARIS_TEXT
+
+
+def _replayed(*messages: Mapping[str, object]) -> list[AllAnthropicPassThroughMessageValues]:
+ return anthropic_messages_pt(
+ messages=cast(list[AllMessageValues], list(messages)), # cast-ok: replays history the way clients send it back
+ model="claude-opus-5-5",
+ llm_provider="anthropic",
+ )
+
+
+def test_web_search_answer_round_trips_without_any_client_tool_call():
+ message: Final = _parsed_assistant_message(_web_search_turn(_PARIS_ANSWER), "end_turn")
+
+ replayed: Final = _replayed(
+ {"role": "user", "content": "Tell me about Paris"},
+ message.model_dump(),
+ {"role": "user", "content": "And Lyon?"},
+ )
+
+ assert _block_ids(replayed[1], "server_tool_use", "id") == ["srvtoolu_01Web"]
+ assert _block_ids(replayed[1], "web_search_tool_result", "tool_use_id") == ["srvtoolu_01Web"]
+ assert [
+ message_field(block, "text")
+ for block in parts_of(replayed[1]["content"])
+ if message_field(block, "type") == "text"
+ ] == [_PARIS_TEXT]
+
+
+def _block_ids(message: Mapping[str, object], block_type: str, id_key: str) -> list[str]:
+ return sorted(
+ block_id
+ for block in parts_of(message["content"])
+ if message_field(block, "type") == block_type and isinstance(block_id := message_field(block, id_key), str)
+ )
+
+
+def _as_returned_before_server_tool_calls_moved(message: Message) -> Mapping[str, object]:
+ dumped: Final[Mapping[str, object]] = message.model_dump()
+ provider_fields: Final = validated_object_mapping(dumped.get("provider_specific_fields")) or {}
+ return {
+ **dumped,
+ "tool_calls": [*anthropic_server_tool_call_params(message), *parts_of(dumped.get("tool_calls"))],
+ "provider_specific_fields": {
+ key: value for key, value in provider_fields.items() if key != "server_tool_calls"
+ },
+ }
_SEARCH_THAT_FOUND_NOTHING: Final = _tool_search_turn(
@@ -3879,21 +4069,33 @@ def test_replayed_search_that_found_nothing_answers_every_tool_use_and_ends_on_a
server_tool_use_ids: list[str],
monkeypatch: pytest.MonkeyPatch,
):
- from litellm.litellm_core_utils.prompt_templates.factory import anthropic_messages_pt
-
message: Final = _parsed_assistant_message(_SEARCH_THAT_FOUND_NOTHING, "end_turn")
monkeypatch.setattr(litellm, "modify_params", modify_params)
- replayed: Final = anthropic_messages_pt(
- messages=[{"role": "user", "content": "Find a tool and use it"}, message.model_dump(), *follow_ups],
- model="claude-opus-5-5",
- llm_provider="anthropic",
+ replayed: Final = _replayed(
+ {"role": "user", "content": "Find a tool and use it"},
+ _as_returned_before_server_tool_calls_moved(message),
+ *follow_ups,
)
assert [turn["role"] for turn in replayed] == ["user", "assistant", "user"]
assert _block_ids(replayed[2], "tool_result", "tool_use_id") == _block_ids(replayed[1], "tool_use", "id")
assert _block_ids(replayed[1], "server_tool_use", "id") == server_tool_use_ids
+def test_replayed_search_that_found_nothing_keeps_its_result_when_the_call_never_reached_the_client(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ message: Final = _parsed_assistant_message(_SEARCH_THAT_FOUND_NOTHING, "end_turn")
+ monkeypatch.setattr(litellm, "modify_params", True)
+
+ replayed: Final = _replayed({"role": "user", "content": "Find a tool and use it"}, message.model_dump())
+
+ assert [turn["role"] for turn in replayed] == ["user", "assistant"]
+ assert _block_ids(replayed[1], "server_tool_use", "id") == ["srvtoolu_01Search"]
+ assert _block_ids(replayed[1], "tool_search_tool_result", "tool_use_id") == ["srvtoolu_01Search"]
+ assert _block_ids(replayed[1], "tool_use", "id") == []
+
+
_CODE_EXECUTION_BLOCKS: Final = [
{
"type": "server_tool_use",
diff --git a/tests/unit/llms/anthropic/messages/adapters/test_anthropic_messages_adapters_transformation.py b/tests/unit/llms/anthropic/messages/adapters/test_anthropic_messages_adapters_transformation.py
index fb4f19a862c..420de9cb4f5 100644
--- a/tests/unit/llms/anthropic/messages/adapters/test_anthropic_messages_adapters_transformation.py
+++ b/tests/unit/llms/anthropic/messages/adapters/test_anthropic_messages_adapters_transformation.py
@@ -80,6 +80,59 @@ def test_native_compaction_response_roundtrip(text: str | None, count: int, expe
assert result["stop_reason"] == expected_stop
+_SERVER_AND_CLIENT_TOOL_TURN: Final[tuple[dict[str, object], ...]] = (
+ {
+ "type": "server_tool_use",
+ "id": "srvtoolu_01Fetch",
+ "name": "web_fetch",
+ "input": {"url": "https://example.com/"},
+ },
+ {
+ "type": "web_fetch_tool_result",
+ "tool_use_id": "srvtoolu_01Fetch",
+ "content": {"type": "web_fetch_result", "url": "https://example.com/"},
+ },
+ {"type": "server_tool_use", "id": "srvtoolu_02Search", "name": "web_search", "input": {"query": "Paris"}},
+ {"type": "text", "text": "The page is titled Example."},
+ {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {"city": "Paris"}},
+)
+
+
+def test_claude_server_tool_blocks_survive_the_round_trip_through_chat_completions() -> None:
+ from unittest.mock import MagicMock
+
+ from litellm.llms.anthropic.chat.transformation import AnthropicConfig
+
+ anthropic_response: Final = {
+ "id": "msg_01Fetch",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-sonnet-5-5",
+ "content": list(_SERVER_AND_CLIENT_TOOL_TURN),
+ "stop_reason": "tool_use",
+ "stop_sequence": None,
+ "usage": {"input_tokens": 100, "output_tokens": 50},
+ }
+ raw_response: Final = MagicMock(spec=httpx.Response)
+ raw_response.json.return_value = anthropic_response
+ raw_response.status_code = 200
+ raw_response.headers = {}
+ chat_response: Final = AnthropicConfig().transform_parsed_response(
+ completion_response=anthropic_response,
+ raw_response=raw_response,
+ model_response=ModelResponse(),
+ json_mode=False,
+ prefix_prompt=None,
+ )
+
+ result: Final = LiteLLMAnthropicMessagesAdapter().translate_openai_response_to_anthropic(chat_response)
+
+ assert [(block["type"], block.get("id") or block.get("tool_use_id")) for block in result["content"]] == [
+ (block["type"], block.get("id") or block.get("tool_use_id")) for block in _SERVER_AND_CLIENT_TOOL_TURN
+ ]
+ assert result["stop_reason"] == "tool_use"
+
+
def test_translate_chat_refusal_to_anthropic_response():
response = ModelResponse(
id="chatcmpl-refusal",
diff --git a/tests/unit/llms/anthropic/messages/native/test_anthropic_messages_handler.py b/tests/unit/llms/anthropic/messages/native/test_anthropic_messages_handler.py
index 8a924dd32d6..3296163c64e 100644
--- a/tests/unit/llms/anthropic/messages/native/test_anthropic_messages_handler.py
+++ b/tests/unit/llms/anthropic/messages/native/test_anthropic_messages_handler.py
@@ -2716,3 +2716,120 @@ async def test_anthropic_messages_strict_tools_rejection_is_sent_again_without_s
assert first["tools"] == [tool]
assert second == {**first, "tools": [{"name": "lookup_record", "input_schema": {"type": "object"}}]}
assert _JSON_BODY.validate_python(response)["id"] == "msg_strict_retry"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("is_async", [False, True], ids=["sync", "async"])
+@pytest.mark.parametrize("stream", [False, True], ids=["complete", "stream"])
+@pytest.mark.parametrize(
+ "model,model_info,responses_target",
+ [
+ ("azure/responses/qa-deployment", None, True),
+ ("azure/qa-deployment", {"mode": "responses"}, True),
+ ("azure/qa-deployment", None, False),
+ ("azure/qa-deployment", {"mode": "chat"}, False),
+ ],
+ ids=["responses-prefix", "responses-pin", "chat-deployment", "chat-pin"],
+)
+async def test_azure_messages_replay_encrypted_reasoning_only_to_responses(
+ monkeypatch: pytest.MonkeyPatch,
+ model: str,
+ model_info: dict[str, str] | None,
+ responses_target: bool,
+ stream: bool,
+ is_async: bool,
+) -> None:
+ from litellm.litellm_core_utils.prompt_templates.reasoning_utils import encrypted_reasoning_signature
+
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ monkeypatch.setattr(
+ litellm,
+ "model_cost",
+ {
+ **litellm.model_cost,
+ "azure/qa-deployment": {
+ "litellm_provider": "azure",
+ "mode": "chat",
+ "supported_endpoints": ["/v1/chat/completions", "/v1/responses"],
+ "supports_native_streaming": True,
+ },
+ },
+ )
+ _invalidate_model_cost_lowercase_map()
+ litellm.in_memory_llm_clients_cache.flush_cache()
+ chat_reply: Final = {
+ "id": "chatcmpl-azure",
+ "object": "chat.completion",
+ "created": 1,
+ "model": "qa-deployment",
+ "choices": [{"index": 0, "message": {"role": "assistant", "content": "hello there"}, "finish_reason": "stop"}],
+ "usage": {"prompt_tokens": 3, "completion_tokens": 2, "total_tokens": 5},
+ }
+ chat_stream: Final = (
+ 'data: {"id":"chatcmpl-azure","object":"chat.completion.chunk","created":1,'
+ '"model":"qa-deployment","choices":[{"index":0,"delta":{"role":"assistant","content":"hello there"},'
+ '"finish_reason":null}]}\n\n'
+ 'data: {"id":"chatcmpl-azure","object":"chat.completion.chunk","created":1,'
+ '"model":"qa-deployment","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],'
+ '"usage":{"prompt_tokens":3,"completion_tokens":2,"total_tokens":5}}\n\n'
+ "data: [DONE]\n\n"
+ ).encode()
+ with respx.mock(assert_all_called=True) as upstream:
+ route: Final = upstream.route(host="qa-resource.openai.azure.com").mock(
+ return_value=(
+ httpx.Response(
+ 200,
+ headers={"content-type": "text/event-stream"},
+ content=_sse_body(_RESPONSES_SSE_EVENTS) if responses_target else chat_stream,
+ )
+ if stream
+ else httpx.Response(200, json=_RESPONSES_COMPLETED_BODY if responses_target else chat_reply)
+ )
+ )
+ create: Final = litellm.anthropic.messages.acreate if is_async else litellm.anthropic.messages.create
+ pending: Final = create(
+ model=model,
+ model_info=model_info,
+ max_tokens=100,
+ api_base="https://qa-resource.openai.azure.com/openai/v1",
+ api_version="v1",
+ api_key="test-api-key",
+ stream=stream,
+ messages=[
+ {"role": "user", "content": "Use calc for 17*23."},
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "thinking",
+ "thinking": "Use the calculator.",
+ "signature": encrypted_reasoning_signature("qa-opaque-reasoning"),
+ },
+ {"type": "tool_use", "id": "call_1", "name": "calc", "input": {"x": 391}},
+ ],
+ },
+ {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "call_1", "content": "391"}]},
+ ],
+ )
+ result: Final = await pending if is_async else pending
+ if stream:
+ chunks: Final = [chunk async for chunk in result] if hasattr(result, "__aiter__") else list(result)
+ assert "message_stop" in "".join(chunk.decode() if isinstance(chunk, bytes) else chunk for chunk in chunks)
+ else:
+ assert result["content"] == [{"type": "text", "text": "hello there"}]
+ body: Final = json.loads(route.calls.last.request.content)
+ if responses_target:
+ assert route.calls.last.request.url.path.endswith("/responses")
+ assert [item for item in body["input"] if item.get("type") == "reasoning"] == [
+ {
+ "type": "reasoning",
+ "summary": [{"type": "summary_text", "text": "Use the calculator."}],
+ "encrypted_content": "qa-opaque-reasoning",
+ }
+ ]
+ assert [item for item in body["input"] if item.get("type") == "function_call_output"] == [
+ {"type": "function_call_output", "call_id": "call_1", "output": [{"type": "input_text", "text": "391"}]}
+ ]
+ else:
+ assert route.calls.last.request.url.path.endswith("/chat/completions")
+ assert "qa-opaque-reasoning" not in json.dumps(body)
diff --git a/tests/unit/llms/anthropic/messages/responses_adapters/test_responses_adapters_transformation.py b/tests/unit/llms/anthropic/messages/responses_adapters/test_responses_adapters_transformation.py
index dbb8c925fa0..660f00cd722 100644
--- a/tests/unit/llms/anthropic/messages/responses_adapters/test_responses_adapters_transformation.py
+++ b/tests/unit/llms/anthropic/messages/responses_adapters/test_responses_adapters_transformation.py
@@ -5,7 +5,8 @@ Tests for LiteLLMAnthropicToResponsesAPIAdapter
import json
import os
-from typing import Any, Dict, Final, List
+from collections.abc import Mapping
+from typing import Any, Dict, Final, List, Literal
from unittest.mock import MagicMock
import pytest
@@ -24,6 +25,9 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.litellm_core_utils.prompt_templates.reasoning_utils import (
encrypted_reasoning_signature,
)
+from litellm.llms.anthropic.messages.adapters.transformation import (
+ LiteLLMAnthropicMessagesAdapter,
+)
from litellm.llms.anthropic.messages.responses_adapters.transformation import (
LiteLLMAnthropicToResponsesAPIAdapter,
)
@@ -141,21 +145,23 @@ class TestOutputConfigStructuredOutput:
}
def test_output_config_format_json_schema_converted(self):
- """output_config.format.json_schema is converted to OpenAI text.format, defaulting strict to False."""
+ """output_config.format.json_schema becomes strict OpenAI text.format."""
req = _make_request(output_config={"format": {"type": "json_schema", "schema": self._SCHEMA}})
kwargs = _ADAPTER.translate_request(req)
assert "text" in kwargs
fmt = kwargs["text"]["format"]
assert fmt["type"] == "json_schema"
assert fmt["schema"] == self._SCHEMA
- assert fmt["strict"] is False
+ assert fmt["strict"] is True
assert fmt["name"] == "structured_output"
- def test_output_config_format_explicit_strict_true_is_preserved(self):
- """Nested output_config.format with explicit strict=True is preserved."""
- req = _make_request(output_config={"format": {"type": "json_schema", "schema": self._SCHEMA, "strict": True}})
- kwargs = _ADAPTER.translate_request(req)
- assert kwargs["text"]["format"]["strict"] is True
+ @pytest.mark.parametrize("strict", (True, False))
+ def test_output_config_format_explicit_strict_is_preserved(self, strict: bool) -> None:
+ req: Final = _make_request(
+ output_config={"format": {"type": "json_schema", "schema": self._SCHEMA, "strict": strict}}
+ )
+ kwargs: Final = _ADAPTER.translate_request(req)
+ assert kwargs["text"]["format"]["strict"] is strict
def test_output_config_without_format_does_not_set_text(self):
"""output_config with only non-format keys doesn't produce text.format."""
@@ -164,12 +170,12 @@ class TestOutputConfigStructuredOutput:
assert "text" not in kwargs
def test_output_format_still_works(self):
- """The original output_format field still takes precedence when present, defaulting strict to False."""
+ """The original output_format field still defaults to strict mode."""
req = _make_request(output_format={"type": "json_schema", "schema": self._SCHEMA})
kwargs = _ADAPTER.translate_request(req)
assert "text" in kwargs
assert kwargs["text"]["format"]["type"] == "json_schema"
- assert kwargs["text"]["format"]["strict"] is False
+ assert kwargs["text"]["format"]["strict"] is True
def test_output_format_explicit_strict_false_is_preserved(self):
"""output_format with an explicit strict=False is preserved as False."""
@@ -183,6 +189,23 @@ class TestOutputConfigStructuredOutput:
kwargs = _ADAPTER.translate_request(req)
assert kwargs["text"]["format"]["strict"] is True
+ @pytest.mark.parametrize("format_field", ("output_format", "output_config"))
+ @pytest.mark.parametrize("strict_options", ({}, {"strict": None}))
+ def test_default_strict_matches_messages_to_chat_adapter(
+ self,
+ format_field: Literal["output_format", "output_config"],
+ strict_options: Mapping[str, None],
+ ) -> None:
+ output_format: Final = {"type": "json_schema", "schema": self._SCHEMA, **strict_options}
+ request: Final = _make_request(
+ **{format_field: output_format if format_field == "output_format" else {"format": output_format}}
+ )
+ chat, _ = LiteLLMAnthropicMessagesAdapter().translate_anthropic_to_openai(request)
+ responses: Final = _ADAPTER.translate_request(request)
+
+ assert chat["response_format"]["json_schema"]["strict"] is True
+ assert responses["text"]["format"]["strict"] is chat["response_format"]["json_schema"]["strict"]
+
def test_output_format_takes_precedence_over_output_config(self):
"""output_format takes precedence over output_config.format, for both schema and strict."""
other_schema = {"type": "object", "properties": {"id": {"type": "integer"}}}
diff --git a/tests/unit/llms/anthropic/test_anthropic_common_utils.py b/tests/unit/llms/anthropic/test_anthropic_common_utils.py
index c97585e02c2..14cabb683f1 100644
--- a/tests/unit/llms/anthropic/test_anthropic_common_utils.py
+++ b/tests/unit/llms/anthropic/test_anthropic_common_utils.py
@@ -14,6 +14,7 @@ import itertools
import json
import logging
import os
+import subprocess
import sys
import threading
from collections.abc import Mapping
@@ -4869,3 +4870,40 @@ def test_a_register_model_opt_out_for_one_claude_model_leaves_every_other_routed
litellm.register_model({f"anthropic/{opted_out}": {"supports_prompt_caching": False}})
assert supports_anthropic_cache_control(untouched, "anthropic", {"id": "plain-id"}) is True
+
+
+_MASKED_HOSTED_WEB_CALL_PROBE: Final = """
+import json
+
+from litellm.llms.anthropic.common_utils import write_request_hosted_web_call_arguments
+
+message = {
+ "role": "assistant",
+ "provider_specific_fields": {
+ "web_search_calls": [
+ {
+ "id": "ws_srvtoolu_1",
+ "type": "web_search_call",
+ "status": "completed",
+ "action": {"type": "search", "query": "deploy token 12345", "queries": ["deploy token 12345"]},
+ }
+ ]
+ },
+}
+write_request_hosted_web_call_arguments(message, {"srvtoolu_1": json.dumps({"query": "deploy token [REDACTED]"})})
+print(json.dumps(message["provider_specific_fields"]["web_search_calls"]))
+"""
+
+
+def test_a_masked_replayed_hosted_web_call_serializes_before_its_openai_model_was_ever_built() -> None:
+ result: Final = subprocess.run(
+ (sys.executable, "-I", "-c", _MASKED_HOSTED_WEB_CALL_PROBE),
+ capture_output=True,
+ text=True,
+ check=True,
+ timeout=120,
+ )
+ hosted: Final = json.loads(result.stdout.strip().splitlines()[-1])
+ assert [call["action"] for call in hosted] == [
+ {"type": "search", "query": "deploy token [REDACTED]", "queries": ["deploy token [REDACTED]"]}
+ ]
diff --git a/tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude_transformation.py b/tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude_transformation.py
index 053574d7dd6..4f17f8b10e1 100644
--- a/tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude_transformation.py
+++ b/tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude_transformation.py
@@ -570,6 +570,7 @@ def _invoke_params_for_schema(
("us.anthropic.claude-opus-4-8", "bedrock", {"thinking": {"type": "adaptive"}}, True),
("us.anthropic.claude-opus-4-8", "bedrock", {"reasoning_effort": "low"}, True),
("us.anthropic.claude-haiku-5-5", "bedrock", {"thinking": {"type": "disabled"}}, False),
+ ("us.anthropic.claude-haiku-5-5", "default", {"thinking": {"type": "disabled"}}, True),
],
ids=[
"legacy-key",
@@ -578,6 +579,7 @@ def _invoke_params_for_schema(
"bedrock-key-adaptive",
"bedrock-key-effort",
"bedrock-key-thinking-disabled",
+ "default-key-thinking-disabled",
],
)
def test_bedrock_native_output_key_applies_only_when_the_model_will_think(
@@ -2047,9 +2049,9 @@ def _invoke_chat_params(
)
@pytest.mark.parametrize("model", ["us.anthropic.claude-opus-4-8", "global.anthropic.claude-sonnet-5"])
def test_invoke_uses_opt_in_native_output_when_the_opt_in_drove_native_output(
- local_model_cost_map, model, thinking_params, monkeypatch
+ set_native_structured_output, model, thinking_params
):
- monkeypatch.setitem(litellm.model_cost[model], "supports_native_structured_output", False)
+ set_native_structured_output(model, "bedrock")
optional_params: Final = _invoke_chat_params(
model, None, {"response_format": _CITY_RESPONSE_FORMAT, **thinking_params}
)
@@ -2084,8 +2086,12 @@ def test_invoke_uses_opt_in_native_output_is_false_when_native_output_did_not_co
)
-def test_invoke_uses_default_native_output_for_schema_retry(local_model_cost_map: None) -> None:
- model: Final = "us.anthropic.claude-opus-5"
+@pytest.mark.parametrize(
+ "model",
+ ["us.anthropic.claude-opus-5", "us.anthropic.claude-sonnet-5"],
+ ids=["opus-5", "sonnet-5"],
+)
+def test_invoke_uses_default_native_output_for_schema_retry(local_model_cost_map: None, model: str) -> None:
optional_params: Final = _invoke_chat_params(
model, None, {"response_format": _CITY_RESPONSE_FORMAT, "thinking": {"type": "disabled"}}
)
diff --git a/tests/unit/llms/bedrock/chat/test_bedrock_converse_handler.py b/tests/unit/llms/bedrock/chat/test_bedrock_converse_handler.py
index 7d1b502df2b..d43cc771185 100644
--- a/tests/unit/llms/bedrock/chat/test_bedrock_converse_handler.py
+++ b/tests/unit/llms/bedrock/chat/test_bedrock_converse_handler.py
@@ -599,6 +599,139 @@ async def test_native_output_retry_is_bounded_by_the_request_timeout_not_the_str
assert _sent_paths(respx_mock) == ["converse-stream", "converse"]
+# Bedrock's 403 for a role without the streaming action, seen live on us.anthropic.claude-sonnet-5 on 2026-11-06
+_STREAM_PERMISSION_DENIAL: Final = json.dumps(
+ {
+ "Message": "User: arn:aws:sts::123456789012:assumed-role/invoke-only/qa is not authorized to perform: "
+ "bedrock:InvokeModelWithResponseStream on resource: arn:aws:bedrock:us-east-1:123456789012:inference-profile/"
+ "us.anthropic.claude-opus-5-5 because no identity-based policy allows the "
+ "bedrock:InvokeModelWithResponseStream action"
+ }
+).encode()
+
+
+def _denies_streaming_then_answers(request: httpx.Request) -> httpx.Response:
+ if request.url.path.endswith("/converse-stream"):
+ return httpx.Response(403, content=_STREAM_PERMISSION_DENIAL, request=request)
+ return httpx.Response(200, content=CONVERSE_ANSWER, headers={"content-type": "application/json"}, request=request)
+
+
+@pytest.mark.asyncio
+async def test_a_denied_native_output_stream_is_sent_again_buffered_with_native_output(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ respx_mock.post(url__regex=_BEDROCK_RUNTIME_URL).mock(side_effect=_denies_streaming_then_answers)
+ router: Final = _bedrock_router(
+ "bedrock/converse/us.anthropic.claude-opus-5-5", timeout=60, stream_timeout=STREAM_TIMEOUT_SECONDS
+ )
+
+ response: Final = await router.acompletion(
+ model="bedrock",
+ messages=[{"role": "user", "content": "hi"}],
+ stream=True,
+ thinking={"type": "disabled"},
+ response_format={"type": "json_schema", "json_schema": {"name": "answer", "schema": _ANSWER_SCHEMA}},
+ )
+
+ assert "".join([chunk.choices[0].delta.content or "" async for chunk in response]) == "hi"
+ assert _sent_paths(respx_mock) == ["converse-stream", "converse"]
+ retry_body: Final = json.loads(respx_mock.calls[1].request.content)
+ assert "textFormat" in retry_body.get("outputConfig", {}) and "toolConfig" not in retry_body
+
+
+_INVOKE_ANSWER: Final = json.dumps(
+ {
+ "id": "msg_1",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-opus-5-5",
+ "content": [{"type": "text", "text": '{"answer": "hi"}'}],
+ "stop_reason": "end_turn",
+ "stop_sequence": None,
+ "usage": {"input_tokens": 10, "output_tokens": 5},
+ }
+).encode()
+
+
+def _denies_invoke_streaming_then_answers(request: httpx.Request) -> httpx.Response:
+ if request.url.path.endswith("/invoke-with-response-stream"):
+ return httpx.Response(403, content=_STREAM_PERMISSION_DENIAL, request=request)
+ return httpx.Response(200, content=_INVOKE_ANSWER, headers={"content-type": "application/json"}, request=request)
+
+
+@pytest.mark.asyncio
+async def test_a_denied_native_output_stream_on_the_invoke_route_is_sent_again_buffered_to_invoke(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ respx_mock.post(url__regex=_BEDROCK_RUNTIME_URL).mock(side_effect=_denies_invoke_streaming_then_answers)
+ router: Final = _bedrock_router(
+ "bedrock/invoke/us.anthropic.claude-opus-5-5", timeout=60, stream_timeout=STREAM_TIMEOUT_SECONDS
+ )
+
+ response: Final = await router.acompletion(
+ model="bedrock",
+ messages=[{"role": "user", "content": "hi"}],
+ stream=True,
+ thinking={"type": "disabled"},
+ response_format={"type": "json_schema", "json_schema": {"name": "answer", "schema": _ANSWER_SCHEMA}},
+ )
+
+ assert "".join([chunk.choices[0].delta.content or "" async for chunk in response]) == '{"answer": "hi"}'
+ assert _sent_paths(respx_mock) == ["invoke-with-response-stream", "invoke"]
+ retry_body: Final = json.loads(respx_mock.calls[1].request.content)
+ assert "format" in retry_body.get("output_config", {}) and "tools" not in retry_body
+
+
+_INVOKE_JSON_TOOL_ANSWER: Final = json.dumps(
+ {
+ "id": "msg_2",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-opus-5-5",
+ "content": [{"type": "tool_use", "id": "toolu_1", "name": "json_tool_call", "input": {"answer": "hi"}}],
+ "stop_reason": "tool_use",
+ "stop_sequence": None,
+ "usage": {"input_tokens": 10, "output_tokens": 5},
+ }
+).encode()
+
+
+def _denies_invoke_streaming_and_rejects_native_output(request: httpx.Request) -> httpx.Response:
+ if request.url.path.endswith("/invoke-with-response-stream"):
+ return httpx.Response(403, content=_STREAM_PERMISSION_DENIAL, request=request)
+ if "format" in json.loads(request.content).get("output_config", {}):
+ return httpx.Response(400, content=_NATIVE_OUTPUT_REJECTION, request=request)
+ return httpx.Response(
+ 200, content=_INVOKE_JSON_TOOL_ANSWER, headers={"content-type": "application/json"}, request=request
+ )
+
+
+@pytest.mark.asyncio
+async def test_a_rejected_schema_after_a_denied_invoke_stream_falls_back_to_the_json_tool_still_buffered(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ respx_mock.post(url__regex=_BEDROCK_RUNTIME_URL).mock(side_effect=_denies_invoke_streaming_and_rejects_native_output)
+ router: Final = _bedrock_router(
+ "bedrock/invoke/us.anthropic.claude-opus-5-5", timeout=60, stream_timeout=STREAM_TIMEOUT_SECONDS
+ )
+
+ response: Final = await router.acompletion(
+ model="bedrock",
+ messages=[{"role": "user", "content": "hi"}],
+ stream=True,
+ thinking={"type": "disabled"},
+ response_format={"type": "json_schema", "json_schema": {"name": "answer", "schema": _ANSWER_SCHEMA}},
+ )
+
+ assert "".join([chunk.choices[0].delta.content or "" async for chunk in response]) == '{"answer": "hi"}'
+ assert _sent_paths(respx_mock) == ["invoke-with-response-stream", "invoke", "invoke"]
+ json_tool_body: Final = json.loads(respx_mock.calls[2].request.content)
+ assert "format" not in json_tool_body.get("output_config", {}) and "tools" in json_tool_body
+
+
@pytest.mark.asyncio
async def test_failed_native_output_retry_reports_the_request_timeout_it_was_bounded_by(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
diff --git a/tests/unit/llms/bedrock/chat/test_converse_transformation.py b/tests/unit/llms/bedrock/chat/test_converse_transformation.py
index b37d6977c10..4dcf312737d 100644
--- a/tests/unit/llms/bedrock/chat/test_converse_transformation.py
+++ b/tests/unit/llms/bedrock/chat/test_converse_transformation.py
@@ -5225,9 +5225,9 @@ def test_a_model_that_cannot_force_the_json_tool_gets_native_output_without_thin
("model", "thinking_params", "native"),
[
("us.anthropic.claude-opus-5-5", None, True),
- ("us.anthropic.claude-opus-5-5", {"thinking": {"type": "adaptive"}}, False),
+ ("us.anthropic.claude-opus-5-5", {"thinking": {"type": "adaptive"}}, True),
("global.anthropic.claude-fable-5-1", None, True),
- ("global.anthropic.claude-fable-5-1", {"reasoning_effort": "low"}, False),
+ ("global.anthropic.claude-fable-5-1", {"reasoning_effort": "low"}, True),
("us-gov.anthropic.claude-opus-5-5", None, False),
("us-gov.anthropic.claude-fable-5-1", None, False),
("us-gov-west-1/anthropic.claude-opus-5-5", None, False),
@@ -5235,7 +5235,7 @@ def test_a_model_that_cannot_force_the_json_tool_gets_native_output_without_thin
("us-east-1/anthropic.claude-opus-5-5", None, True),
],
)
-def test_models_whose_thinking_stops_under_native_output_keep_the_json_tool_only_when_thinking(
+def test_native_output_follows_model_cost_map_for_bedrock_profiles(
model: str, thinking_params: Mapping[str, object] | None, native: bool
) -> None:
_, request = _converse_request_for_json_schema(model, False, thinking_params)
@@ -5248,6 +5248,17 @@ def test_models_whose_thinking_stops_under_native_output_keep_the_json_tool_only
@pytest.mark.parametrize(
("model", "thinking_params", "native"),
[
+ ("us.anthropic.claude-opus-5-5", {"thinking": {"type": "disabled"}}, True),
+ ("global.anthropic.claude-fable-5-1", {"thinking": {"type": "adaptive"}}, True),
+ ("global.anthropic.claude-fable-5-1", {"thinking": {"type": "disabled"}}, True),
+ ("us.anthropic.claude-sonnet-5", None, True),
+ ("us.anthropic.claude-sonnet-5", {"thinking": {"type": "adaptive"}}, True),
+ ("us.anthropic.claude-sonnet-5", {"thinking": {"type": "disabled"}}, True),
+ ("us.anthropic.claude-haiku-5-5", None, True),
+ ("us.anthropic.claude-haiku-5-5", {"thinking": {"type": "adaptive"}}, True),
+ ("us.anthropic.claude-haiku-5-5", {"thinking": {"type": "disabled"}}, True),
+ ("us-gov.anthropic.claude-sonnet-5", None, False),
+ ("us-gov.anthropic.claude-haiku-5-5", None, False),
("us.anthropic.claude-opus-5", None, True),
("us.anthropic.claude-opus-5", {"thinking": {"type": "adaptive"}}, True),
("us.anthropic.claude-opus-5", {"thinking": {"type": "disabled"}}, True),
@@ -5258,6 +5269,17 @@ def test_models_whose_thinking_stops_under_native_output_keep_the_json_tool_only
("us-gov.anthropic.claude-opus-4-8", None, False),
],
ids=[
+ "opus-5-5-disabled",
+ "fable-5-1-adaptive",
+ "fable-5-1-disabled",
+ "sonnet-5-unset",
+ "sonnet-5-adaptive",
+ "sonnet-5-disabled",
+ "haiku-5-5-unset",
+ "haiku-5-5-adaptive",
+ "haiku-5-5-disabled",
+ "sonnet-5-us-gov",
+ "haiku-5-5-us-gov",
"opus-5-unset",
"opus-5-adaptive",
"opus-5-disabled",
@@ -5268,7 +5290,7 @@ def test_models_whose_thinking_stops_under_native_output_keep_the_json_tool_only
"opus-4-8-us-gov",
],
)
-def test_opus_5_and_4_8_cost_map_defaults_use_native_output_across_thinking_modes(
+def test_model_cost_native_output_defaults_hold_across_thinking_modes(
model: str, thinking_params: Mapping[str, object] | None, native: bool
) -> None:
_, request = _converse_request_for_json_schema(model, False, thinking_params)
@@ -5282,6 +5304,30 @@ def test_opus_5_and_4_8_cost_map_defaults_use_native_output_across_thinking_mode
assert [tool["toolSpec"]["name"] for tool in request["toolConfig"]["tools"]] == ["json_tool_call"]
+@pytest.mark.usefixtures("local_model_cost_map")
+@pytest.mark.parametrize(
+ ("model", "thinking_params"),
+ [
+ ("us.anthropic.claude-opus-5-5", {"thinking": {"type": "adaptive"}}),
+ ("global.anthropic.claude-fable-5-1", {"thinking": {"type": "adaptive"}}),
+ ("us.anthropic.claude-sonnet-5", {"thinking": {"type": "disabled"}}),
+ ("us.anthropic.claude-haiku-5-5", None),
+ ("us.anthropic.claude-opus-5", None),
+ ],
+)
+@pytest.mark.parametrize("stream", [False, True])
+def test_deployment_opt_out_keeps_the_json_tool_on_models_that_default_to_native_output(
+ model: str, thinking_params: Mapping[str, object] | None, stream: bool
+) -> None:
+ optional_params, request = _converse_request_for_json_schema(
+ model, stream, thinking_params, model_info={"supports_bedrock_native_structured_output": False}
+ )
+
+ assert "outputConfig" not in request
+ assert [tool["toolSpec"]["name"] for tool in request["toolConfig"]["tools"]] == ["json_tool_call"]
+ assert optional_params.get("fake_stream", False) is stream
+
+
def _converse_capital_answer(request: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
@@ -5295,9 +5341,7 @@ def _converse_capital_answer(request: httpx.Request) -> httpx.Response:
@pytest.mark.usefixtures("local_model_cost_map")
-@pytest.mark.parametrize(
- "model", ["us.anthropic.claude-opus-5-5", "global.anthropic.claude-sonnet-5-5", "global.anthropic.claude-fable-5-1"]
-)
+@pytest.mark.parametrize("model", ["global.anthropic.claude-sonnet-5-5"])
def test_streamed_json_schema_on_a_model_that_cannot_force_a_tool_still_calls_converse_without_streaming(
model: str,
) -> None:
@@ -5610,7 +5654,9 @@ def test_schema_past_a_native_output_size_limit_keeps_the_tool_path(
@pytest.mark.parametrize(
- ("native_key", "native_past_limit"), [("legacy", True), ("bedrock", False)], ids=["legacy-key", "bedrock-key"]
+ ("native_key", "native_past_limit"),
+ [("legacy", True), ("default", True), ("bedrock", False)],
+ ids=["legacy-key", "default-key", "bedrock-key"],
)
def test_field_count_limit_sends_the_json_tool_only_on_the_bedrock_native_key(
set_native_structured_output, native_key, native_past_limit
@@ -10335,7 +10381,6 @@ def test_converse_reserves_the_tool_cache_point_within_the_cap(tool_cache_params
assert request["toolConfig"]["tools"][-1] == cache_point
-
@pytest.mark.parametrize(
"unsent_tool_params",
[
@@ -10385,6 +10430,7 @@ def test_converse_leaves_the_cap_to_markers_when_a_marked_tool_is_not_sent(
]
assert cache_point not in request.get("toolConfig", {}).get("tools", [])
+
def _converse_tool_turn(*result_parts: dict[str, object]) -> list[dict[str, object]]:
return [
{
@@ -10506,7 +10552,9 @@ def test_converse_translates_prompt_cache_breakpoint_on_an_unmapped_arn_only_whe
messages=[
{
"role": "user",
- "content": [{"type": "text", "text": "Hello", "prompt_cache_breakpoint": {"mode": "explicit", **ttl}}],
+ "content": [
+ {"type": "text", "text": "Hello", "prompt_cache_breakpoint": {"mode": "explicit", **ttl}}
+ ],
}
],
optional_params={},
@@ -10518,7 +10566,6 @@ def test_converse_translates_prompt_cache_breakpoint_on_an_unmapped_arn_only_whe
assert user_content(unflagged_arn, {"ttl": "30m"}) == [{"text": "Hello"}]
-
def test_converse_prompt_cache_breakpoint_follows_the_arn_flag_over_its_base_model(local_model_cost_map):
from litellm import Router
@@ -10545,7 +10592,9 @@ def test_converse_prompt_cache_breakpoint_follows_the_arn_flag_over_its_base_mod
messages=[
{
"role": "user",
- "content": [{"type": "text", "text": "Hello", "prompt_cache_breakpoint": {"mode": "explicit", **ttl}}],
+ "content": [
+ {"type": "text", "text": "Hello", "prompt_cache_breakpoint": {"mode": "explicit", **ttl}}
+ ],
}
],
optional_params={},
@@ -10633,7 +10682,9 @@ def test_converse_prompt_cache_breakpoint_on_guarded_user_text_keeps_its_cache_p
def test_converse_guarded_user_text_keeps_no_cache_point_on_a_model_not_known_to_take_one(local_model_cost_map):
request: Final = litellm.AmazonConverseConfig()._transform_request(
model="arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/unmapped1",
- messages=[{"role": "user", "content": [{"type": "text", "text": "Hello", "cache_control": {"type": "ephemeral"}}]}],
+ messages=[
+ {"role": "user", "content": [{"type": "text", "text": "Hello", "cache_control": {"type": "ephemeral"}}]}
+ ],
optional_params={"guardrailConfig": {"guardrailIdentifier": "gr-1", "guardrailVersion": "DRAFT"}},
litellm_params={},
)
@@ -11638,9 +11689,9 @@ _OUTPUT_CONFIG_FORMAT: Final = {
)
@pytest.mark.parametrize("model", ["us.anthropic.claude-opus-4-8", "global.anthropic.claude-sonnet-5"])
def test_uses_opt_in_native_output_when_the_opt_in_drove_native_output(
- local_model_cost_map, monkeypatch, model, request_params
+ set_native_structured_output, model, request_params
):
- monkeypatch.setitem(litellm.model_cost[model], "supports_native_structured_output", False)
+ set_native_structured_output(model, "bedrock")
optional_params: Final = _converse_params(model, None, request_params)
assert AmazonConverseConfig().uses_opt_in_native_output(model, optional_params, {"model_info": {"id": "d1"}})
@@ -11672,9 +11723,9 @@ def test_uses_opt_in_native_output_is_false_when_native_output_did_not_come_from
def test_uses_opt_in_native_output_reads_the_base_model_of_an_application_inference_profile(
- local_model_cost_map, monkeypatch
+ set_native_structured_output,
):
- monkeypatch.setitem(litellm.model_cost["us.anthropic.claude-opus-4-8"], "supports_native_structured_output", False)
+ set_native_structured_output("us.anthropic.claude-opus-4-8", "bedrock")
profile: Final = "arn:aws:bedrock:us-east-1:000000000000:application-inference-profile/profile-id"
litellm_params: Final = {"base_model": "bedrock/us.anthropic.claude-opus-4-8"}
optional_params: Final = _converse_params(
@@ -11768,8 +11819,9 @@ def test_a_refused_native_schema_is_sent_again_only_when_the_retry_drops_native_
@pytest.mark.respx(assert_all_called=True)
+@pytest.mark.parametrize("model", ["us.anthropic.claude-sonnet-5", "us.anthropic.claude-opus-5"])
def test_a_default_native_schema_rejection_retries_with_the_json_tool(
- respx_mock: respx.Router, local_model_cost_map: None
+ respx_mock: respx.Router, local_model_cost_map: None, model: str
) -> None:
route: Final = respx_mock.post(url__regex=r"/converse$").mock(
return_value=httpx.Response(400, json={"message": _BEDROCK_SCHEMA_REFUSAL})
@@ -11777,7 +11829,7 @@ def test_a_default_native_schema_rejection_retries_with_the_json_tool(
with pytest.raises(litellm.BadRequestError, match=re.escape(_BEDROCK_SCHEMA_REFUSAL)):
completion(
- model="bedrock/converse/us.anthropic.claude-opus-5",
+ model=f"bedrock/converse/{model}",
messages=[{"role": "user", "content": "What is the capital of France?"}],
response_format=CAPITAL_RESPONSE_FORMAT,
thinking={"type": "disabled"},
@@ -11938,7 +11990,9 @@ def test_converse_sends_no_tools_when_drop_params_drops_every_hosted_mcp_tool(mo
def _warnings_naming(caplog: pytest.LogCaptureFixture, param: str) -> tuple[str, ...]:
return tuple(
- r.getMessage() for r in caplog.records if r.levelno == logging.WARNING and f"Dropping {param}=" in r.getMessage()
+ r.getMessage()
+ for r in caplog.records
+ if r.levelno == logging.WARNING and f"Dropping {param}=" in r.getMessage()
)
@@ -12102,9 +12156,15 @@ def test_converse_sends_only_the_header_betas_when_header_and_body_both_carry_be
def _five_cache_hint_conversation() -> list[dict[str, object]]:
return [
- {"role": "system", "content": [{"type": "text", "text": "system prompt", "cache_control": {"type": "ephemeral"}}]},
+ {
+ "role": "system",
+ "content": [{"type": "text", "text": "system prompt", "cache_control": {"type": "ephemeral"}}],
+ },
{"role": "user", "content": [{"type": "text", "text": "turn 1", "cache_control": {"type": "ephemeral"}}]},
- {"role": "assistant", "content": [{"type": "text", "text": "answer 1", "cache_control": {"type": "ephemeral"}}]},
+ {
+ "role": "assistant",
+ "content": [{"type": "text", "text": "answer 1", "cache_control": {"type": "ephemeral"}}],
+ },
{"role": "user", "content": [{"type": "text", "text": "turn 2", "cache_control": {"type": "ephemeral"}}]},
{"role": "assistant", "content": "answer 2"},
{"role": "user", "content": [{"type": "text", "text": "turn 3", "cache_control": {"type": "ephemeral"}}]},
@@ -12121,7 +12181,10 @@ def _five_cache_hint_conversation() -> list[dict[str, object]]:
],
)
def test_converse_claude_request_over_the_cache_control_cap_drops_the_oldest_hints_only_under_drop_params(
- drop_params: bool | str, expected_system_cache_points: int, monkeypatch: pytest.MonkeyPatch, local_model_cost_map: None
+ drop_params: bool | str,
+ expected_system_cache_points: int,
+ monkeypatch: pytest.MonkeyPatch,
+ local_model_cost_map: None,
) -> None:
monkeypatch.setattr(litellm, "drop_params", False)
messages: Final = _five_cache_hint_conversation()
diff --git a/tests/unit/llms/bedrock/chat/test_converse_transformation_nova_2.py b/tests/unit/llms/bedrock/chat/test_converse_transformation_nova_2.py
index 58058a2e1d4..f29ad3b5844 100644
--- a/tests/unit/llms/bedrock/chat/test_converse_transformation_nova_2.py
+++ b/tests/unit/llms/bedrock/chat/test_converse_transformation_nova_2.py
@@ -637,8 +637,7 @@ class TestNova2MultiTurnMessageTranslation:
assert "Simple addition" in texts
assert "4." in texts
- def test_should_keep_signed_thinking_blocks_as_reasoning_content(self):
- """With a signature, reasoning is preserved as a reasoningContent block."""
+ def test_should_drop_signed_thinking_blocks_from_another_model(self):
bedrock_msgs = self._to_bedrock(
[
{"role": "user", "content": "What is 2+2?"},
@@ -653,13 +652,9 @@ class TestNova2MultiTurnMessageTranslation:
]
)
assistant = next(m for m in bedrock_msgs if m["role"] == "assistant")
- rc_blocks = [b for b in assistant["content"] if "reasoningContent" in b]
- assert len(rc_blocks) >= 1
- assert rc_blocks[0]["reasoningContent"]["reasoningText"]["text"] == "math"
- assert rc_blocks[0]["reasoningContent"]["reasoningText"]["signature"] == "sig-1"
+ assert assistant["content"] == [{"text": "4."}]
- def test_should_translate_inline_content_list_thinking_type(self):
- """content=[{type:'thinking',...},{type:'text',...}] should also round-trip."""
+ def test_should_drop_inline_signed_thinking_and_keep_the_answer(self):
bedrock_msgs = self._to_bedrock(
[
{"role": "user", "content": "Hi"},
@@ -674,11 +669,4 @@ class TestNova2MultiTurnMessageTranslation:
]
)
assistant = next(m for m in bedrock_msgs if m["role"] == "assistant")
- rc_blocks = [b for b in assistant["content"] if "reasoningContent" in b]
- text_blocks = [
- b
- for b in assistant["content"]
- if "text" in b and "reasoningContent" not in b
- ]
- assert len(rc_blocks) >= 1
- assert any("Hello!" in b["text"] for b in text_blocks)
+ assert assistant["content"] == [{"text": "Hello!"}]
diff --git a/tests/unit/llms/bedrock/chat/test_native_output_fallback.py b/tests/unit/llms/bedrock/chat/test_native_output_fallback.py
index 81832d7d8bf..c87451d9577 100644
--- a/tests/unit/llms/bedrock/chat/test_native_output_fallback.py
+++ b/tests/unit/llms/bedrock/chat/test_native_output_fallback.py
@@ -149,6 +149,122 @@ def test_a_rejection_reaches_the_caller_when_the_retry_would_send_native_output_
assert bedrock.sent == (None,)
+# Bedrock's 403 for a role without the streaming action, seen live on us.anthropic.claude-sonnet-5 on 2026-11-06
+_STREAM_PERMISSION_DENIAL: Final = BedrockError(
+ status_code=403,
+ message='{"Message":"User: arn:aws:sts::123456789012:assumed-role/invoke-only/qa is not authorized to perform: '
+ "bedrock:InvokeModelWithResponseStream on resource: arn:aws:bedrock:us-west-2:123456789012:inference-profile/"
+ "us.anthropic.claude-sonnet-5 because no identity-based policy allows the "
+ 'bedrock:InvokeModelWithResponseStream action"}',
+)
+_INVOKE_PERMISSION_DENIAL: Final = BedrockError(
+ status_code=403,
+ message='{"Message":"User: arn:aws:sts::123456789012:assumed-role/no-bedrock/qa is not authorized to perform: '
+ 'bedrock:InvokeModel because no identity-based policy allows the bedrock:InvokeModel action"}',
+)
+_BUFFERED_NATIVE_STREAM: Final = NativeOutputFallback(
+ optional_params={"outputConfig": "native", "fake_stream": True}, litellm_params={}
+)
+
+
+def _buffered(fallback: NativeOutputFallback) -> NativeOutputFallback:
+ return NativeOutputFallback(
+ optional_params={**fallback.optional_params, "fake_stream": True}, litellm_params=fallback.litellm_params
+ )
+
+
+def _call_with_buffered_stream_fallback(
+ bedrock: ScriptedBedrock, fallback: NativeOutputFallback | None, buffered_stream_fallback: NativeOutputFallback
+) -> object:
+ return _resolve(
+ call_with_native_output_fallback(
+ bedrock,
+ None if fallback is None else lambda: fallback,
+ _MODEL,
+ buffered_stream_fallback=lambda: buffered_stream_fallback,
+ )
+ )
+
+
+@_MODES
+def test_a_denied_native_stream_is_sent_again_once_as_a_buffered_native_request(
+ asynchronous: bool, caplog: pytest.LogCaptureFixture
+) -> None:
+ bedrock: Final = ScriptedBedrock(errors=(_STREAM_PERMISSION_DENIAL, None), asynchronous=asynchronous)
+
+ with caplog.at_level("WARNING", logger="LiteLLM"):
+ response: Final = _call_with_buffered_stream_fallback(bedrock, _FALLBACK, _BUFFERED_NATIVE_STREAM)
+
+ assert isinstance(response, ModelResponse) and response.id == "answer-2"
+ assert bedrock.sent == (None, _BUFFERED_NATIVE_STREAM)
+ assert "bedrock:InvokeModelWithResponseStream" in caplog.text and _MODEL in caplog.text
+
+
+@_MODES
+def test_a_buffered_native_request_bedrock_rejects_is_sent_again_buffered_with_the_fallback_params(
+ asynchronous: bool,
+) -> None:
+ bedrock: Final = ScriptedBedrock(
+ errors=(_STREAM_PERMISSION_DENIAL, _GRAMMAR_REJECTION, None), asynchronous=asynchronous
+ )
+
+ response: Final = _call_with_buffered_stream_fallback(bedrock, _FALLBACK, _BUFFERED_NATIVE_STREAM)
+
+ assert isinstance(response, ModelResponse) and response.id == "answer-3"
+ assert bedrock.sent == (None, _BUFFERED_NATIVE_STREAM, _buffered(_FALLBACK))
+
+
+@_MODES
+def test_strict_tools_bedrock_rejects_after_a_buffered_native_retry_are_dropped_on_the_buffered_json_tool_retry(
+ asynchronous: bool,
+) -> None:
+ bedrock: Final = ScriptedBedrock(
+ errors=(_STREAM_PERMISSION_DENIAL, _GRAMMAR_REJECTION, _GRAMMAR_REJECTION, None), asynchronous=asynchronous
+ )
+
+ response: Final = _resolve(
+ call_with_native_output_fallback(
+ bedrock,
+ lambda: _NATIVE_RETRY_WITH_STRICT_TOOLS,
+ "us.anthropic.claude-sonnet-4-6",
+ strict_tools_fallback=lambda: _STRICT_FALLBACK,
+ buffered_stream_fallback=lambda: _BUFFERED_NATIVE_STREAM,
+ )
+ )
+
+ assert isinstance(response, ModelResponse) and response.id == "answer-4"
+ assert bedrock.sent == (
+ None,
+ _BUFFERED_NATIVE_STREAM,
+ _buffered(_NATIVE_RETRY_WITH_STRICT_TOOLS),
+ _buffered(_COMPOSED_FALLBACK),
+ )
+
+
+@_MODES
+@pytest.mark.parametrize(
+ ("error", "buffered_stream_fallback"),
+ [
+ (
+ _STREAM_PERMISSION_DENIAL,
+ NativeOutputFallback(optional_params={"outputConfig": "native"}, litellm_params={}),
+ ),
+ (_INVOKE_PERMISSION_DENIAL, _BUFFERED_NATIVE_STREAM),
+ ],
+ ids=["retry_would_stream_again", "invoke_denied"],
+)
+def test_a_permission_denial_reaches_the_caller_unless_the_retry_buffers_the_stream(
+ asynchronous: bool, error: BedrockError, buffered_stream_fallback: NativeOutputFallback
+) -> None:
+ bedrock: Final = ScriptedBedrock(errors=(error,), asynchronous=asynchronous)
+
+ with pytest.raises(BedrockError) as raised:
+ _call_with_buffered_stream_fallback(bedrock, _FALLBACK, buffered_stream_fallback)
+
+ assert raised.value is error
+ assert bedrock.sent == (None,)
+
+
@pytest.mark.parametrize(
("error", "advises_opting_out"),
[(_REJECTION, True), (_SCHEMA_REFUSAL, False)],
diff --git a/tests/unit/llms/bedrock/files/test_bedrock_files_transformation.py b/tests/unit/llms/bedrock/files/test_bedrock_files_transformation.py
index 93bda5b444d..fdd6d5e61bc 100644
--- a/tests/unit/llms/bedrock/files/test_bedrock_files_transformation.py
+++ b/tests/unit/llms/bedrock/files/test_bedrock_files_transformation.py
@@ -21,7 +21,7 @@ from botocore.credentials import Credentials
import litellm
from litellm.constants import DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET
-from litellm.llms.bedrock.common_utils import BedrockError
+from litellm.llms.bedrock.common_utils import BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES, BedrockError
from litellm.llms.bedrock.files.transformation import BedrockFilesConfig, BedrockJsonlFilesTransformation
from tests.unit.proxy.conftest import httpx_transport
@@ -2069,6 +2069,76 @@ class TestBedrockBatchAnthropicRowParams:
assert ("format" in model_input.get("output_config", {})) is native_output
assert (tool_names == {"json_tool_call"}) is not native_output
+ @pytest.mark.parametrize(
+ ("model", "native_over_the_limit"),
+ [(MODEL, True), ("bedrock/us.anthropic.claude-opus-5-5", False)],
+ ids=["already_native_model", "native_by_default_model"],
+ )
+ @pytest.mark.parametrize("over_the_limit", [False, True], ids=["at_the_limit", "over_the_limit"])
+ def test_only_native_by_default_models_send_json_schemas_wider_than_the_limit_through_the_json_tool(
+ self, local_model_cost_map, model, native_over_the_limit, over_the_limit
+ ):
+ native_output: Final = native_over_the_limit or not over_the_limit
+ field_names: Final = tuple(
+ f"field_{index:03d}" for index in range(BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES + over_the_limit)
+ )
+ wide_schema: Final = {
+ "type": "object",
+ "properties": {name: {"type": "string"} for name in field_names},
+ "required": list(field_names),
+ "additionalProperties": False,
+ }
+ model_input: Final = self._transform(
+ "/v1/chat/completions",
+ {
+ "messages": [{"role": "user", "content": "Describe a ceramic coffee mug"}],
+ "response_format": {"type": "json_schema", "json_schema": {"name": "wide", "schema": wide_schema}},
+ },
+ model=model,
+ )
+ tool_names: Final = frozenset(tool["name"] for tool in model_input.get("tools", []))
+
+ assert ("format" in model_input.get("output_config", {})) is native_output
+ assert (tool_names == {"json_tool_call"}) is not native_output
+
+ def test_a_deployment_opt_out_sends_uploaded_json_schema_rows_through_the_json_tool(self, local_model_cost_map):
+ from litellm.llms.bedrock.files.transformation import BedrockFilesConfig
+
+ record: Final = {
+ "custom_id": "row-1",
+ "method": "POST",
+ "url": "/v1/chat/completions",
+ "body": {
+ "model": self.MODEL,
+ "messages": [{"role": "user", "content": "Weather in Paris?"}],
+ "response_format": {
+ "type": "json_schema",
+ "json_schema": {"name": "weather", "schema": self.PARAMETERS},
+ },
+ },
+ }
+ request: Final = BedrockFilesConfig().transform_create_file_request(
+ model=self.MODEL,
+ create_file_data={
+ "file": ("batch.jsonl", json.dumps(record).encode(), "application/jsonl"),
+ "purpose": "batch",
+ },
+ optional_params={
+ "aws_access_key_id": "test-key-id",
+ "aws_secret_access_key": "test-secret",
+ "aws_region_name": "us-west-2",
+ },
+ litellm_params={
+ "s3_bucket_name": "litellm-batch-bucket",
+ "litellm_metadata": {"model_info": {"supports_bedrock_native_structured_output": False}},
+ },
+ )
+ assert isinstance(request, dict)
+ model_input: Final = json.loads(request["data"].splitlines()[0])["modelInput"]
+
+ assert "format" not in model_input.get("output_config", {})
+ assert [tool["name"] for tool in model_input["tools"]] == ["json_tool_call"]
+
def test_provider_native_params_still_pass_through(self):
model_input = self._transform(
"/v1/chat/completions",
diff --git a/tests/unit/llms/bedrock/test_bedrock_common_utils.py b/tests/unit/llms/bedrock/test_bedrock_common_utils.py
index 0c1c4f5475a..7cc4629e497 100644
--- a/tests/unit/llms/bedrock/test_bedrock_common_utils.py
+++ b/tests/unit/llms/bedrock/test_bedrock_common_utils.py
@@ -1,11 +1,10 @@
-
import asyncio, importlib, litellm, pytest
import respx
from httpx import Response
from typing import Final
-from litellm.llms.bedrock.common_utils import(
+from litellm.llms.bedrock.common_utils import (
AmazonBedrockGlobalConfig,
BedrockModelInfo,
NativeOutputFallback,
@@ -54,9 +53,7 @@ def test_bedrock_response_stream_shape_lazy_loads_once():
import litellm.llms.bedrock.common_utils as mod
sentinel = MagicMock()
- with patch.object(
- mod, "_load_bedrock_response_stream_shape", return_value=sentinel
- ) as mock_load:
+ with patch.object(mod, "_load_bedrock_response_stream_shape", return_value=sentinel) as mock_load:
assert mod.get_bedrock_response_stream_shape() is sentinel
assert mod.get_bedrock_response_stream_shape() is sentinel
mock_load.assert_called_once()
@@ -103,9 +100,7 @@ def test_bedrock_response_stream_shape_is_structure_shape():
from litellm.llms.bedrock.common_utils import get_bedrock_response_stream_shape
loaded_shape = get_bedrock_response_stream_shape()
- assert (
- loaded_shape is not None
- ), "get_bedrock_response_stream_shape() is None — botocore may not be installed"
+ assert loaded_shape is not None, "get_bedrock_response_stream_shape() is None — botocore may not be installed"
shape: StructureShape = loaded_shape
assert isinstance(shape, StructureShape)
assert shape.name == "ResponseStream"
@@ -170,9 +165,7 @@ def test_deepseek_cris():
Test that DeepSeek models with cross-region inference prefix use converse route
"""
bedrock_model_info = BedrockModelInfo
- bedrock_route = bedrock_model_info.get_bedrock_route(
- model="bedrock/us.deepseek.r1-v1:0"
- )
+ bedrock_route = bedrock_model_info.get_bedrock_route(model="bedrock/us.deepseek.r1-v1:0")
assert bedrock_route == "converse"
@@ -228,7 +221,10 @@ def test_bare_opaque_arn_with_no_invoke_provider_routes_to_converse(arn_segment:
model: Final = f"bedrock/arn:aws:bedrock:us-east-1:123412341234:{arn_segment}"
assert BaseAWSLLM.get_bedrock_invoke_provider(model) is None
assert BedrockModelInfo.get_bedrock_route(model) == "converse"
- assert BedrockModelInfo.get_bedrock_route(f"bedrock/llama/arn:aws:bedrock:us-east-1:123412341234:{arn_segment}") == "invoke"
+ assert (
+ BedrockModelInfo.get_bedrock_route(f"bedrock/llama/arn:aws:bedrock:us-east-1:123412341234:{arn_segment}")
+ == "invoke"
+ )
def test_govcloud_cross_region_inference_prefix():
@@ -238,27 +234,19 @@ def test_govcloud_cross_region_inference_prefix():
bedrock_model_info = BedrockModelInfo
# Test us-gov prefix is stripped correctly for Claude models
- base_model = bedrock_model_info.get_base_model(
- model="bedrock/us-gov.anthropic.claude-haiku-4-5-20251001-v1:0"
- )
+ base_model = bedrock_model_info.get_base_model(model="bedrock/us-gov.anthropic.claude-haiku-4-5-20251001-v1:0")
assert base_model == "anthropic.claude-haiku-4-5-20251001-v1:0"
# Test us-gov prefix is stripped correctly for different Claude versions
- base_model = bedrock_model_info.get_base_model(
- model="bedrock/us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0"
- )
+ base_model = bedrock_model_info.get_base_model(model="bedrock/us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0")
assert base_model == "anthropic.claude-sonnet-4-5-20250929-v1:0"
# Test us-gov prefix is stripped correctly for Haiku models
- base_model = bedrock_model_info.get_base_model(
- model="bedrock/us-gov.anthropic.claude-3-haiku-20240307-v1:0"
- )
+ base_model = bedrock_model_info.get_base_model(model="bedrock/us-gov.anthropic.claude-3-haiku-20240307-v1:0")
assert base_model == "anthropic.claude-3-haiku-20240307-v1:0"
# Test us-gov prefix is stripped correctly for Meta models
- base_model = bedrock_model_info.get_base_model(
- model="bedrock/us-gov.meta.llama3-8b-instruct-v1:0"
- )
+ base_model = bedrock_model_info.get_base_model(model="bedrock/us-gov.meta.llama3-8b-instruct-v1:0")
assert base_model == "meta.llama3-8b-instruct-v1:0"
@@ -272,23 +260,14 @@ def test_context_window_suffix_stripped_for_cost_lookup():
"""
from litellm.llms.bedrock.common_utils import get_bedrock_base_model
- assert (
- get_bedrock_base_model("us.anthropic.claude-opus-4-6-v1[1m]")
- == "anthropic.claude-opus-4-6-v1"
- )
- assert (
- get_bedrock_base_model("us.anthropic.claude-sonnet-4-6[1m]")
- == "anthropic.claude-sonnet-4-6"
- )
+ assert get_bedrock_base_model("us.anthropic.claude-opus-4-6-v1[1m]") == "anthropic.claude-opus-4-6-v1"
+ assert get_bedrock_base_model("us.anthropic.claude-sonnet-4-6[1m]") == "anthropic.claude-sonnet-4-6"
assert (
get_bedrock_base_model("global.anthropic.claude-opus-4-5-20251101-v1:0[1m]")
== "anthropic.claude-opus-4-5-20251101-v1:0"
)
# Ensure models without suffix are unaffected
- assert (
- get_bedrock_base_model("us.anthropic.claude-opus-4-6-v1")
- == "anthropic.claude-opus-4-6-v1"
- )
+ assert get_bedrock_base_model("us.anthropic.claude-opus-4-6-v1") == "anthropic.claude-opus-4-6-v1"
# Ensure :51k throughput suffix still works
assert (
get_bedrock_base_model("anthropic.claude-3-5-sonnet-20241022-v2:0:51k")
@@ -358,9 +337,7 @@ def test_output_config_effort_normalization_uses_model_info_ceiling(monkeypatch)
("us.anthropic.claude-opus-4-7", "xhigh"),
],
)
-def test_bundled_bedrock_opus_model_info_declares_output_config_effort_ceiling(
- model, expected_ceiling
-):
+def test_bundled_bedrock_opus_model_info_declares_output_config_effort_ceiling(model, expected_ceiling):
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
model_info = GetModelCostMap.load_local_model_cost_map()[model]
@@ -379,54 +356,24 @@ def test_route_prefix_matched_as_path_segment_not_substring():
or a ``/`` boundary.
"""
# The bedrock_mantle/ provider prefix must NOT be read as the mantle/ route.
- assert (
- BedrockModelInfo.get_bedrock_route("bedrock_mantle/openai.gpt-5.5") != "mantle"
- )
- assert (
- BedrockModelInfo.get_bedrock_route("bedrock_mantle/openai.gpt-5.4") == "converse"
- )
- assert (
- BedrockModelInfo._explicit_mantle_route("bedrock_mantle/openai.gpt-5.5")
- is False
- )
+ assert BedrockModelInfo.get_bedrock_route("bedrock_mantle/openai.gpt-5.5") != "mantle"
+ assert BedrockModelInfo.get_bedrock_route("bedrock_mantle/openai.gpt-5.4") == "converse"
+ assert BedrockModelInfo._explicit_mantle_route("bedrock_mantle/openai.gpt-5.5") is False
# A genuine mantle route still resolves, via the startswith branch...
- assert (
- BedrockModelInfo.get_bedrock_route("mantle/anthropic.claude-mythos-preview")
- == "mantle"
- )
+ assert BedrockModelInfo.get_bedrock_route("mantle/anthropic.claude-mythos-preview") == "mantle"
# ...and via the mid-path "/mantle/" branch (after the bedrock/ provider prefix).
- assert (
- BedrockModelInfo.get_bedrock_route(
- "bedrock/mantle/anthropic.claude-mythos-preview"
- )
- == "mantle"
- )
+ assert BedrockModelInfo.get_bedrock_route("bedrock/mantle/anthropic.claude-mythos-preview") == "mantle"
def test_model_has_route_prefix_exercises_both_branches():
"""``_model_has_route_prefix`` matches on ``startswith`` or a ``/`` boundary only."""
# startswith branch
- assert (
- BedrockModelInfo._model_has_route_prefix(
- "mantle/anthropic.claude-mythos-preview", "mantle/"
- )
- is True
- )
+ assert BedrockModelInfo._model_has_route_prefix("mantle/anthropic.claude-mythos-preview", "mantle/") is True
# f"/{prefix}" boundary branch
- assert (
- BedrockModelInfo._model_has_route_prefix(
- "bedrock/mantle/anthropic.claude-mythos-preview", "mantle/"
- )
- is True
- )
+ assert BedrockModelInfo._model_has_route_prefix("bedrock/mantle/anthropic.claude-mythos-preview", "mantle/") is True
# neither branch: the token only appears glued to another segment
- assert (
- BedrockModelInfo._model_has_route_prefix(
- "bedrock_mantle/openai.gpt-5.5", "mantle/"
- )
- is False
- )
+ assert BedrockModelInfo._model_has_route_prefix("bedrock_mantle/openai.gpt-5.5", "mantle/") is False
@pytest.mark.parametrize(
@@ -476,16 +423,10 @@ def test_explicit_invoke_route_does_not_match_async_invoke():
"""
async_invoke_model = "async_invoke/twelvelabs.marengo-embed-2-7-v1:0"
assert BedrockModelInfo._explicit_invoke_route(async_invoke_model) is False
- assert (
- BedrockModelInfo._explicit_invoke_route(f"bedrock/{async_invoke_model}")
- is False
- )
+ assert BedrockModelInfo._explicit_invoke_route(f"bedrock/{async_invoke_model}") is False
# ...while async_invoke/ is still detected as its own route.
assert BedrockModelInfo._explicit_async_invoke_route(async_invoke_model) is True
- assert (
- BedrockModelInfo._explicit_async_invoke_route(f"bedrock/{async_invoke_model}")
- is True
- )
+ assert BedrockModelInfo._explicit_async_invoke_route(f"bedrock/{async_invoke_model}") is True
def test_capability_lookups_fall_back_to_base_model_when_regional_entry_lacks_field(monkeypatch):
@@ -605,7 +546,6 @@ def test_deployment_model_info_breakpoint_flag_covers_an_unmapped_arn(local_mode
assert bedrock_model_accepts_cache_points(unflagged_arn) is True
-
_CONVERSE_REPLY: Final = {
"output": {"message": {"role": "assistant", "content": [{"text": "ok"}]}},
"stopReason": "end_turn",
@@ -638,7 +578,11 @@ _MARKED_TOOL_CONVERSATION: Final = (
}
],
},
- {"role": "tool", "tool_call_id": "call_1", "content": [{"type": "text", "text": "Sunny", "cache_control": _CACHE_MARK}]},
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": [{"type": "text", "text": "Sunny", "cache_control": _CACHE_MARK}],
+ },
{"role": "tool", "tool_call_id": "call_1_1", "content": "Cloudy"},
{"role": "assistant", "content": "Paris is sunny, Rome is cloudy.", "cache_control": _CACHE_MARK},
{"role": "user", "content": "Thanks"},
@@ -711,6 +655,7 @@ async def test_a_bedrock_deployment_sends_cache_points_by_its_own_flag_not_a_sib
"flagless": not declared,
}
+
async def test_a_routed_bedrock_deployment_sends_cache_points_when_register_model_opted_its_model_in(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch, local_model_cost_map: None
) -> None:
@@ -738,7 +683,9 @@ async def test_a_routed_bedrock_deployment_sends_cache_points_when_register_mode
num_retries=0,
)
- await router.acompletion(model="flagless", messages=[{"role": "system", "content": "stable"}, {"role": "user", "content": "hi"}])
+ await router.acompletion(
+ model="flagless", messages=[{"role": "system", "content": "stable"}, {"role": "user", "content": "hi"}]
+ )
assert b"cachePoint" in converse.calls.last.request.content
@@ -783,10 +730,13 @@ def test_a_deployments_prompt_caching_flag_written_as_a_string_or_number_decides
)
assert (
- bedrock_model_accepts_cache_points(model, model_info={"id": "qa-deployment", "supports_prompt_caching": declared})
+ bedrock_model_accepts_cache_points(
+ model, model_info={"id": "qa-deployment", "supports_prompt_caching": declared}
+ )
is expected
)
+
def test_merge_bedrock_aws_request_params_strips_caller_identity_when_deployment_has_static_credentials():
from litellm.llms.bedrock.common_utils import merge_bedrock_aws_request_params
@@ -1491,6 +1441,7 @@ def _vcr_outcome_gate(request, vcr):
yield
record_vcr_outcome(request, vcr)
+
@pytest.fixture(scope="session")
def event_loop():
try:
@@ -1500,6 +1451,7 @@ def event_loop():
yield loop
loop.close()
+
@pytest.fixture(scope="function")
def setup_and_teardown(event_loop):
import litellm
@@ -1533,6 +1485,7 @@ def setup_and_teardown(event_loop):
if pending:
event_loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True))
+
_SCALAR_DEFAULTS = {
"num_retries": getattr(litellm, "num_retries", None),
"set_verbose": getattr(litellm, "set_verbose", False),
@@ -1547,6 +1500,7 @@ _SCALAR_DEFAULTS = {
"cohere_key": getattr(litellm, "cohere_key", None),
}
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestStripBedrockRoutingPrefix:
"""Tests for strip_bedrock_routing_prefix function."""
@@ -1577,6 +1531,7 @@ class TestStripBedrockRoutingPrefix:
== "anthropic.claude-3-sonnet-20240229-v1:0"
)
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestStripBedrockThroughputSuffix:
"""Tests for strip_bedrock_throughput_suffix function."""
@@ -1604,6 +1559,7 @@ class TestStripBedrockThroughputSuffix:
def test_strip_throughput_suffix(self, input_model, expected):
assert strip_bedrock_throughput_suffix(input_model) == expected
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestExtractModelNameFromBedrockArn:
"""Tests for extract_model_name_from_bedrock_arn function."""
@@ -1624,6 +1580,7 @@ class TestExtractModelNameFromBedrockArn:
arn = "ARN:aws:bedrock:us-east-1:123456789012:model/my-model"
assert extract_model_name_from_bedrock_arn(arn) == "my-model"
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestGetBedrockCrossRegionInferenceRegions:
"""Tests for get_bedrock_cross_region_inference_regions function."""
@@ -1639,6 +1596,7 @@ class TestGetBedrockCrossRegionInferenceRegions:
regions = get_bedrock_cross_region_inference_regions()
assert isinstance(regions, list)
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestGetBedrockBaseModel:
"""Tests for get_bedrock_base_model function."""
@@ -1702,6 +1660,7 @@ class TestGetBedrockBaseModel:
"""Test that throughput tier suffixes like :51k are stripped. Issue #19113."""
assert get_bedrock_base_model(input_model) == expected
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestBedrockModelInfoWrappers:
"""Tests that BedrockModelInfo methods correctly wrap standalone functions."""
@@ -1723,6 +1682,7 @@ class TestBedrockModelInfoWrappers:
model = "bedrock/converse/claude-3"
assert BedrockModelInfo.get_non_litellm_routing_model_name(model) == strip_bedrock_routing_prefix(model)
+
@pytest.mark.usefixtures("_vcr_outcome_gate", "setup_and_teardown")
class TestBedrockTokenCounter:
"""Tests for BedrockTokenCounter class."""
@@ -1759,12 +1719,14 @@ class TestBedrockTokenCounter:
)
assert result is None
+
@pytest.fixture
def _pr4_bedrock_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("AWS_ACCESS_KEY_ID", "pr4-test-aws-access-key")
monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "pr4-test-aws-secret-key")
monkeypatch.setenv("AWS_REGION_NAME", "us-east-1")
+
@pytest.fixture(scope="module")
def _use_local_model_cost_map() -> Iterator[None]:
with pytest.MonkeyPatch.context() as monkeypatch:
@@ -1772,6 +1734,7 @@ def _use_local_model_cost_map() -> Iterator[None]:
importlib.reload(litellm)
yield
+
@pytest.mark.usefixtures(
"_pr4_bedrock_env",
"_use_local_model_cost_map",
@@ -2348,8 +2311,12 @@ def test_bedrock_capability_model_reads_base_model_only_for_application_profile_
assert bedrock_capability_model(model, base_model) == expected
-_FORMAT_REJECTION: Final = "The model returned the following errors: output_config.format: Extra inputs are not permitted"
-_COMPILE_REJECTION: Final = "The model returned the following errors: compilation exceeded limits: Schema is too complex."
+_FORMAT_REJECTION: Final = (
+ "The model returned the following errors: output_config.format: Extra inputs are not permitted"
+)
+_COMPILE_REJECTION: Final = (
+ "The model returned the following errors: compilation exceeded limits: Schema is too complex."
+)
_GRAMMAR_REJECTION: Final = (
"The model returned the following errors: The compiled grammar is too large, which would cause performance issues. "
"Simplify your tool schemas or reduce the number of strict tools."
@@ -2394,10 +2361,10 @@ def test_is_bedrock_native_output_rejection_matches_only_the_400s_that_reject_na
("model", "model_info", "expected"),
[
("us.anthropic.claude-opus-4-8", None, False),
- ("global.anthropic.claude-sonnet-5", None, True),
+ ("global.anthropic.claude-sonnet-5", None, False),
("us-gov.anthropic.claude-opus-4-8", None, False),
("us.anthropic.claude-opus-4-8", {"supports_bedrock_native_structured_output": False}, False),
- ("us.anthropic.claude-opus-5-5", {"supports_bedrock_native_structured_output": True}, True),
+ ("us.anthropic.claude-opus-5-5", {"supports_bedrock_native_structured_output": True}, False),
("us.anthropic.claude-sonnet-4-6", {"supports_bedrock_native_structured_output": True}, False),
],
ids=["opus_4_8", "sonnet_5", "govcloud_opus_4_8", "deployment_opt_out", "deployment_opt_in", "native_model"],
@@ -2490,7 +2457,10 @@ def test_bedrock_strict_tools_fallback_drops_only_strict_and_keeps_everything_el
assert fallback == NativeOutputFallback(
optional_params={
"tools": [
- {**strict_tool, "function": {"name": "lookup", "description": "Look up", "parameters": {"type": "object"}}},
+ {
+ **strict_tool,
+ "function": {"name": "lookup", "description": "Look up", "parameters": {"type": "object"}},
+ },
plain_tool,
anthropic_tool,
{"toolSpec": {"name": "spec", "inputSchema": {"json": {"type": "object"}}}},
diff --git a/tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py b/tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py
index 635363e77ea..c456e782c09 100644
--- a/tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py
+++ b/tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py
@@ -19,6 +19,7 @@ from litellm.llms.chatgpt.authenticator import Authenticator
from litellm.llms.custom_httpx.http_handler import HTTPHandler
from litellm.llms.chatgpt.responses.transformation import ChatGPTResponsesAPIConfig
from litellm.llms.openai.common_utils import OpenAIError
+from litellm.llms.openai.responses.transformation import OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS
from litellm.main import responses_api_bridge_check
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
@@ -59,7 +60,7 @@ class TestChatGPTResponsesAPITransformation:
assert request["reasoning"] == {"effort": effort}
assert request["stream"] is True
assert request["store"] is False
- assert "max_output_tokens" not in request
+ assert request["max_output_tokens"] == 16
assert "prompt_cache_options" not in request
@pytest.mark.parametrize("requested_tier", [None, "auto", "flex", "unknown"])
@@ -236,9 +237,9 @@ class TestChatGPTResponsesAPITransformation:
"top_p": 0.9,
"context_management": [{"type": "compaction", "compact_threshold": 200000}],
"metadata": {"foo": "bar"},
- "max_output_tokens": 123,
"stream_options": {"include_usage": True},
# supported and should be preserved
+ "max_output_tokens": 123,
"truncation": "auto",
"previous_response_id": "resp_123",
"reasoning": {"effort": "medium"},
@@ -254,9 +255,9 @@ class TestChatGPTResponsesAPITransformation:
assert "top_p" not in request
assert "context_management" not in request
assert "metadata" not in request
- assert "max_output_tokens" not in request
assert "stream_options" not in request
+ assert request["max_output_tokens"] == 123
assert request["truncation"] == "auto"
assert request["previous_response_id"] == "resp_123"
assert request["reasoning"] == {"effort": "medium"}
@@ -618,22 +619,25 @@ def test_chatgpt_non_stream_sse_keeps_an_output_item_whose_text_holds_unicode_li
assert parsed.output_text == text
-@pytest.mark.parametrize("call", ["responses", "completion"])
-def test_prompt_cache_key_keeps_one_session_header_across_requests(
- call: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
-) -> None:
+@pytest.fixture
+def chatgpt_token_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
(tmp_path / "auth.json").write_text(
json.dumps({"access_token": "token", "account_id": "acct", "expires_at": 4102444800})
)
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(tmp_path))
monkeypatch.delenv("CHATGPT_AUTH_FILE", raising=False)
+
+
+def _completed_codex_upstream(output_text: str, incomplete_reason: str | None = None) -> MagicMock:
+ status: Final = "completed" if incomplete_reason is None else "incomplete"
completed: Final = {
- "type": "response.completed",
+ "type": f"response.{status}",
"response": {
"id": "resp_1",
"object": "response",
"created_at": 1,
- "status": "completed",
+ "status": status,
+ "incomplete_details": None if incomplete_reason is None else {"reason": incomplete_reason},
"model": "gpt-5.5",
"output": [
{
@@ -641,19 +645,25 @@ def test_prompt_cache_key_keeps_one_session_header_across_requests(
"id": "msg_1",
"status": "completed",
"role": "assistant",
- "content": [{"type": "output_text", "text": "OK", "annotations": []}],
+ "content": [{"type": "output_text", "text": output_text, "annotations": []}],
}
],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
},
}
- upstream: Final = MagicMock(
+ return MagicMock(
side_effect=lambda _request: httpx.Response(
200,
headers={"content-type": "text/event-stream"},
- content=f"event: response.completed\ndata: {json.dumps(completed)}\n\n".encode(),
+ content=f"event: response.{status}\ndata: {json.dumps(completed)}\n\n".encode(),
)
)
+
+
+@pytest.mark.usefixtures("chatgpt_token_dir")
+@pytest.mark.parametrize("call", ["responses", "completion"])
+def test_prompt_cache_key_keeps_one_session_header_across_requests(call: str) -> None:
+ upstream: Final = _completed_codex_upstream("OK")
client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(upstream)))
for _ in range(2):
if call == "responses":
@@ -667,3 +677,89 @@ def test_prompt_cache_key_keeps_one_session_header_across_requests(
)
assert [call.args[0].headers["session_id"] for call in upstream.call_args_list] == ["conv-123", "conv-123"]
+
+
+_CITY_SCHEMA: Final = {
+ "type": "object",
+ "properties": {"city": {"type": "string"}, "country": {"type": "string"}},
+ "required": ["city", "country"],
+ "additionalProperties": False,
+}
+
+
+@pytest.mark.usefixtures("chatgpt_token_dir")
+def test_responses_text_reaches_the_codex_backend_whole() -> None:
+ text: Final = {
+ "format": {"type": "json_schema", "name": "city", "schema": _CITY_SCHEMA, "strict": True},
+ "verbosity": "low",
+ }
+ upstream: Final = _completed_codex_upstream('{"city": "Paris", "country": "France"}')
+ client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(upstream)))
+
+ litellm.responses(model="chatgpt/gpt-5.5", input="Where is the Eiffel Tower?", text=text, client=client)
+
+ assert json.loads(upstream.call_args.args[0].content)["text"] == text
+
+
+@pytest.mark.usefixtures("chatgpt_token_dir")
+@pytest.mark.parametrize(
+ ("response_format", "expected_format"),
+ [
+ (
+ {"type": "json_schema", "json_schema": {"name": "city", "schema": _CITY_SCHEMA, "strict": True}},
+ {"type": "json_schema", "name": "city", "schema": _CITY_SCHEMA, "strict": True},
+ ),
+ ({"type": "json_object"}, {"type": "json_object"}),
+ ],
+)
+def test_chat_response_format_reaches_the_codex_backend_as_text_format(
+ response_format: dict[str, object], expected_format: dict[str, object]
+) -> None:
+ upstream: Final = _completed_codex_upstream('{"city": "Paris", "country": "France"}')
+ client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(upstream)))
+
+ litellm.completion(
+ model="chatgpt/gpt-5.5",
+ messages=[{"role": "user", "content": "Where is the Eiffel Tower? Answer in JSON."}],
+ response_format=response_format,
+ client=client,
+ )
+
+ assert json.loads(upstream.call_args.args[0].content)["text"]["format"] == expected_format
+
+
+@pytest.mark.usefixtures("chatgpt_token_dir")
+@pytest.mark.parametrize(
+ ("max_tokens", "expected_max_output_tokens"),
+ [(300, 300), (OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS - 1, OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS)],
+)
+def test_chat_max_tokens_reaches_the_codex_backend_no_lower_than_its_minimum(
+ max_tokens: int, expected_max_output_tokens: int
+) -> None:
+ upstream: Final = _completed_codex_upstream("OK")
+ client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(upstream)))
+
+ litellm.completion(
+ model="chatgpt/gpt-5.5",
+ messages=[{"role": "user", "content": "hi"}],
+ max_tokens=max_tokens,
+ client=client,
+ )
+
+ assert json.loads(upstream.call_args.args[0].content)["max_output_tokens"] == expected_max_output_tokens
+
+
+@pytest.mark.usefixtures("chatgpt_token_dir")
+def test_non_stream_chat_cut_off_by_max_tokens_returns_the_partial_answer() -> None:
+ upstream: Final = _completed_codex_upstream('{"capitals": ["Paris", "Ber', incomplete_reason="max_output_tokens")
+ client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(upstream)))
+
+ response: Final = litellm.completion(
+ model="chatgpt/gpt-5.5",
+ messages=[{"role": "user", "content": "List capitals as JSON."}],
+ max_tokens=16,
+ client=client,
+ )
+
+ assert response.choices[0].finish_reason == "length"
+ assert response.choices[0].message.content == '{"capitals": ["Paris", "Ber'
diff --git a/tests/unit/llms/custom_httpx/test_llm_http_handler.py b/tests/unit/llms/custom_httpx/test_llm_http_handler.py
index dd0e5762845..a758f4d4250 100644
--- a/tests/unit/llms/custom_httpx/test_llm_http_handler.py
+++ b/tests/unit/llms/custom_httpx/test_llm_http_handler.py
@@ -4890,6 +4890,92 @@ async def test_json_mode_stream_reaches_the_iterator_of_a_config_without_a_reque
]
+class _PlainStreamRouteConfig(_TransformRecordingConfig):
+ def __init__(self):
+ super().__init__(transform_async=False)
+
+ def get_complete_url(self, api_base, api_key, model, optional_params, litellm_params, stream=None):
+ return f"{api_base}/{'stream' if stream else 'answer'}"
+
+
+class _StreamRouteConfig(_PlainStreamRouteConfig):
+ @property
+ def has_custom_stream_wrapper(self) -> bool:
+ return True
+
+
+def _answers_only_the_non_streaming_route(request: httpx.Request) -> httpx.Response:
+ if request.url.path.endswith("/answer"):
+ return httpx.Response(200, json=json.loads(request.content))
+ return httpx.Response(403, json={"message": "not authorized to stream"})
+
+
+def _fake_streamed_completion(
+ acompletion: bool,
+ client: HTTPHandler | AsyncHTTPHandler,
+ fake_stream: object = True,
+ provider_config: BaseConfig | None = None,
+):
+ return BaseLLMHTTPHandler().completion(
+ model="stub-model",
+ messages=[{"role": "user", "content": "hi"}],
+ api_base="https://llm.example/v1/chat",
+ custom_llm_provider="openai",
+ model_response=ModelResponse(),
+ encoding=None,
+ logging_obj=_streaming_logging_obj(),
+ optional_params={"fake_stream": fake_stream},
+ timeout=10.0,
+ litellm_params={},
+ acompletion=acompletion,
+ stream=True,
+ client=client,
+ provider_config=provider_config or _StreamRouteConfig(),
+ )
+
+
+async def test_a_fake_streamed_completion_is_sent_to_the_non_streaming_route():
+ sync_client: Final = HTTPHandler(
+ client=httpx.Client(transport=httpx.MockTransport(_answers_only_the_non_streaming_route))
+ )
+ async_client: Final = AsyncHTTPHandler()
+ async_client.client = httpx.AsyncClient(transport=httpx.MockTransport(_answers_only_the_non_streaming_route))
+
+ sync_chunks: Final = list(_fake_streamed_completion(acompletion=False, client=sync_client))
+ async_chunks: Final = [
+ chunk async for chunk in await _fake_streamed_completion(acompletion=True, client=async_client)
+ ]
+
+ assert (_streamed_text(sync_chunks), _streamed_text(async_chunks)) == ("sync", "sync")
+
+
+def _answers_only_the_streaming_route(request: httpx.Request) -> httpx.Response:
+ if request.url.path.endswith("/stream"):
+ return _sse_echoing_transformed_by(request)
+ return httpx.Response(403, json={"message": "only streaming is allowed"})
+
+
+@pytest.mark.parametrize("fake_stream", ["true", 1], ids=["string", "int"])
+async def test_a_fake_stream_value_other_than_true_still_streams_from_the_streaming_route(fake_stream: object):
+ sync_client: Final = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(_answers_only_the_streaming_route)))
+ async_client: Final = AsyncHTTPHandler()
+ async_client.client = httpx.AsyncClient(transport=httpx.MockTransport(_answers_only_the_streaming_route))
+
+ sync_chunks: Final = list(
+ _fake_streamed_completion(
+ acompletion=False, client=sync_client, fake_stream=fake_stream, provider_config=_PlainStreamRouteConfig()
+ )
+ )
+ async_chunks: Final = [
+ chunk
+ async for chunk in await _fake_streamed_completion(
+ acompletion=True, client=async_client, fake_stream=fake_stream, provider_config=_PlainStreamRouteConfig()
+ )
+ ]
+
+ assert (_streamed_text(sync_chunks), _streamed_text(async_chunks)) == ("sync", "sync")
+
+
class _ImageEditRecordingConfig(BaseImageEditConfig):
def __init__(self):
self.transform_calls = []
diff --git a/tests/unit/llms/fireworks_ai/responses/test_fireworks_ai_responses_transformation.py b/tests/unit/llms/fireworks_ai/responses/test_fireworks_ai_responses_transformation.py
index 6538ccf1cac..fe33a26dfd7 100644
--- a/tests/unit/llms/fireworks_ai/responses/test_fireworks_ai_responses_transformation.py
+++ b/tests/unit/llms/fireworks_ai/responses/test_fireworks_ai_responses_transformation.py
@@ -477,6 +477,84 @@ def test_transform_request_forwards_non_string_instructions_and_input_untouched(
)
+CODEX_CLOCK_NAMESPACE_TOOL: Final = {
+ "type": "namespace",
+ "name": "clock",
+ "description": "Clock tools.",
+ "tools": [
+ {
+ "type": "function",
+ "name": "sleep",
+ "description": "Sleep for some seconds.",
+ "parameters": {"type": "object", "properties": {"seconds": {"type": "number"}}, "required": ["seconds"]},
+ }
+ ],
+}
+CODEX_ADDITIONAL_TOOLS_ITEM: Final = {
+ "type": "additional_tools",
+ "role": "developer",
+ "tools": [CODEX_CLOCK_NAMESPACE_TOOL],
+}
+
+
+@pytest.mark.parametrize("instructions", [None, "Answer with exactly one word."], ids=["no-instructions", "instructions"])
+def test_transform_request_moves_codex_additional_tools_item_into_tools(instructions: str | None) -> None:
+ user_item: Final = {"role": "user", "content": "Sleep for two seconds."}
+ request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
+ model="accounts/fireworks/models/glm-5p3",
+ input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item]),
+ response_api_optional_request_params={} if instructions is None else {"instructions": instructions},
+ litellm_params=GenericLiteLLMParams(),
+ headers={},
+ )
+ assert tuple(request["tools"]) == (CODEX_CLOCK_NAMESPACE_TOOL,)
+ assert tuple(request["input"]) == (user_item,)
+ assert request.get("instructions") == instructions
+
+
+@pytest.mark.parametrize(
+ ("drop_params", "expected_clock_copies"),
+ [
+ pytest.param(True, 1, id="merged once under drop_params"),
+ pytest.param(None, 2, id="forwarded as sent without drop_params"),
+ ],
+)
+@pytest.mark.parametrize("tools_as_tuple", [False, True], ids=["tools-list", "tools-tuple"])
+def test_transform_request_merges_codex_additional_tools_into_existing_tools_only_under_drop_params(
+ drop_params: bool | None, expected_clock_copies: int, tools_as_tuple: bool
+) -> None:
+ weather_tool: Final = {
+ "type": "function",
+ "name": "get_weather",
+ "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
+ }
+ user_item: Final = {"role": "user", "content": "Sleep for two seconds, then check the weather in Paris."}
+ request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
+ model="accounts/fireworks/models/glm-5p3",
+ input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item, CODEX_ADDITIONAL_TOOLS_ITEM]),
+ response_api_optional_request_params={"tools": (weather_tool,) if tools_as_tuple else [weather_tool]},
+ litellm_params=GenericLiteLLMParams(drop_params=drop_params),
+ headers={},
+ )
+ assert tuple(request["tools"]) == (weather_tool, *(CODEX_CLOCK_NAMESPACE_TOOL,) * expected_clock_copies)
+ assert tuple(request["input"]) == (user_item,)
+
+
+@pytest.mark.parametrize("drop_params", [True, None], ids=["drop_params", "no-drop_params"])
+def test_transform_request_leaves_codex_additional_tools_in_input_when_tools_is_malformed(
+ drop_params: bool | None,
+) -> None:
+ user_item: Final = {"role": "user", "content": "Sleep for two seconds."}
+ request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
+ model="accounts/fireworks/models/glm-5p3",
+ input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item]),
+ response_api_optional_request_params={"tools": {"type": "function", "name": "get_weather"}},
+ litellm_params=GenericLiteLLMParams(drop_params=drop_params),
+ headers={},
+ )
+ assert tuple(request["input"]) == (CODEX_ADDITIONAL_TOOLS_ITEM, user_item)
+
+
def test_responses_call_maps_pydantic_developer_items_and_replays_pydantic_output_items() -> None:
client: Final = _mock_http_client(_fireworks_response("accounts/fireworks/models/qwen3p8-2p4t-a95b"))
pydantic_input: Final = cast(
diff --git a/tests/unit/llms/litellm_proxy/chat/test_litellm_proxy_chat_transformation.py b/tests/unit/llms/litellm_proxy/chat/test_litellm_proxy_chat_transformation.py
index 8a690a70a97..f516d03c499 100644
--- a/tests/unit/llms/litellm_proxy/chat/test_litellm_proxy_chat_transformation.py
+++ b/tests/unit/llms/litellm_proxy/chat/test_litellm_proxy_chat_transformation.py
@@ -7,6 +7,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
+import respx
from openai import AsyncOpenAI, OpenAI
from openai._legacy_response import LegacyAPIResponse
from openai.types import CreateEmbeddingResponse, Embedding
@@ -27,6 +28,31 @@ _GATEWAY_EMBEDDING_RESPONSE: Final = CreateEmbeddingResponse(
)
+@pytest.mark.parametrize(
+ ("base_path", "models_path"),
+ [
+ ("", "/v1/models"),
+ ("/", "/v1/models"),
+ ("/v1", "/v1/models"),
+ ("/v1/", "/v1/models"),
+ ("/gateway/v1", "/gateway/v1/models"),
+ ("/gateway/v1/", "/gateway/v1/models"),
+ ],
+)
+@respx.mock
+def test_proxy_model_discovery_preserves_the_api_base_path_and_upstream_namespaces(
+ base_path: str, models_path: str
+) -> None:
+ respx.get(
+ f"http://upstream-listing.test:4000{models_path}",
+ headers={"Authorization": "Bearer test-proxy-key"},
+ ).mock(return_value=httpx.Response(200, json={"data": [{"id": "chat-model"}, {"id": "vendor/chat-model"}]}))
+
+ assert LiteLLMProxyChatConfig().get_models(
+ api_base=f"http://upstream-listing.test:4000{base_path}", api_key="test-proxy-key"
+ ) == ["litellm_proxy/chat-model", "litellm_proxy/vendor/chat-model"]
+
+
async def _gateway_embedding_via_injected_client(
is_async: bool,
) -> tuple[CapturingTransport, litellm.EmbeddingResponse]:
diff --git a/tests/unit/llms/moonshot/test_moonshot_chat_transformation.py b/tests/unit/llms/moonshot/test_moonshot_chat_transformation.py
index ee2dd627563..b054dd84c22 100644
--- a/tests/unit/llms/moonshot/test_moonshot_chat_transformation.py
+++ b/tests/unit/llms/moonshot/test_moonshot_chat_transformation.py
@@ -5,6 +5,7 @@ These tests validate the MoonshotChatConfig class which extends OpenAIGPTConfig.
Moonshot AI is an OpenAI-compatible provider with minor customizations.
"""
+from typing import Final
from unittest.mock import patch
@@ -13,6 +14,7 @@ import pytest
import litellm
import litellm.utils
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
+from litellm.litellm_core_utils.model_latest_aliases import get_model_latest_aliases
from litellm.llms.moonshot.chat.transformation import MoonshotChatConfig
@@ -719,10 +721,16 @@ class TestMoonshotReasoningEffort:
def test_reasoning_model_supports_reasoning_effort(self, model):
assert "reasoning_effort" in MoonshotChatConfig().get_supported_openai_params(model)
- @pytest.mark.parametrize("model", ["moonshot-v1-8k", "kimi-latest", "kimi-k2-turbo-preview"])
+ @pytest.mark.parametrize("model", ["moonshot-v1-8k", "kimi-k2-turbo-preview"])
def test_non_reasoning_model_does_not_support_reasoning_effort(self, model):
assert "reasoning_effort" not in MoonshotChatConfig().get_supported_openai_params(model)
+ def test_latest_alias_uses_target_supported_params(self) -> None:
+ target: Final = get_model_latest_aliases()["moonshot"]["kimi-latest"]
+ assert MoonshotChatConfig().get_supported_openai_params(
+ "kimi-latest"
+ ) == MoonshotChatConfig().get_supported_openai_params(target)
+
@pytest.mark.parametrize("effort", ["low", "high", "max"])
def test_declared_effort_reaches_optional_params(self, effort):
optional_params = litellm.get_optional_params(
diff --git a/tests/unit/llms/openai/chat/guardrail_translation/test_openai_guardrail_handler.py b/tests/unit/llms/openai/chat/guardrail_translation/test_openai_guardrail_handler.py
index 5d71cbe9afc..54832bb35cc 100644
--- a/tests/unit/llms/openai/chat/guardrail_translation/test_openai_guardrail_handler.py
+++ b/tests/unit/llms/openai/chat/guardrail_translation/test_openai_guardrail_handler.py
@@ -6,17 +6,21 @@ with guardrail transformations, including tool calls.
"""
import json
-from collections.abc import Mapping
-from typing import Any, Literal, Optional
+from collections.abc import Iterable, Mapping
+from typing import Any, Final, Literal, Optional
import pytest
+from openai.types.responses import ResponseFunctionWebSearch
from litellm.integrations.custom_guardrail import CustomGuardrail
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.guardrail_translation.base_translation import StreamingScanKey
from litellm.llms.openai.chat.guardrail_translation.handler import (
OpenAIChatCompletionsHandler,
)
+from litellm.types.llms.openai import ChatCompletionToolCallChunk
+from litellm.types.responses.main import build_web_search_call
from litellm.types.utils import (
ChatCompletionMessageToolCall,
Choices,
@@ -144,6 +148,55 @@ class MockMisalignedToolCallGuardrail(CustomGuardrail):
)
+def _server_tool_call(call_id: str, name: str, arguments: Mapping[str, object]) -> dict[str, object]:
+ return {"id": call_id, "type": "function", "function": {"name": name, "arguments": json.dumps(arguments)}}
+
+
+def _tool_call_ids_names_and_arguments(
+ tool_calls: Iterable[ChatCompletionToolCallChunk],
+) -> list[tuple[str | None, str | None, str]]:
+ return [
+ (tool_call["id"], tool_call["function"]["name"], tool_call["function"]["arguments"])
+ for tool_call in tool_calls
+ ]
+
+
+def _hosted_web_call_inputs(web_search_calls: Iterable[ResponseFunctionWebSearch]) -> list[tuple[str, str, Optional[str]]]:
+ return [
+ (call.id, call.status, call.action.query if call.action.type == "search" else call.action.url)
+ for call in web_search_calls
+ ]
+
+
+_SEARCH_INPUT: Final = {"query": "jane@example.com"}
+_FETCH_INPUT: Final = {"url": "https://example.com/?email=jane@example.com"}
+
+
+class QueryRedactingGuardrail(CustomGuardrail):
+ """Returns copied tool calls with any ``query`` or ``url`` argument redacted, leaving other calls as they came."""
+
+ async def apply_guardrail(
+ self,
+ inputs: GenericGuardrailAPIInputs,
+ request_data: dict,
+ input_type: Literal["request", "response"],
+ logging_obj: Optional[LiteLLMLoggingObj] = None,
+ ) -> GenericGuardrailAPIInputs:
+ def redacted(tool_call: ChatCompletionToolCallChunk) -> ChatCompletionToolCallChunk:
+ arguments: Final[dict[str, object]] = json.loads(tool_call["function"]["arguments"])
+ redacted_arguments: Final = {
+ key: "[REDACTED]" if key in ("query", "url") else value for key, value in arguments.items()
+ }
+ if redacted_arguments == arguments:
+ return tool_call
+ return {**tool_call, "function": {**tool_call["function"], "arguments": json.dumps(redacted_arguments)}}
+
+ return GenericGuardrailAPIInputs(
+ texts=inputs.get("texts", []),
+ tool_calls=[redacted(tool_call) for tool_call in inputs.get("tool_calls", [])],
+ )
+
+
class TestOpenAIChatCompletionsHandlerToolsInput:
"""Test input processing with tools (function definitions)"""
@@ -906,6 +959,192 @@ class TestOpenAIChatCompletionsHandlerToolCallsOutput:
assert tool_calls[0].function.arguments == first_args
assert tool_calls[1].function.arguments == second_args
+ @pytest.mark.asyncio
+ async def test_input_guardrail_rewrite_of_a_replayed_server_tool_call_lands_on_the_request(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ data: Final = {
+ "messages": [
+ {"role": "user", "content": "look up jane and the weather"},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [_server_tool_call("toolu_01Weather", "get_weather", {"city": "Paris"})],
+ "provider_specific_fields": {
+ "server_tool_calls": [_server_tool_call("srvtoolu_01Search", "web_search", _SEARCH_INPUT)]
+ },
+ },
+ ]
+ }
+
+ await handler.process_input_messages(data, QueryRedactingGuardrail(guardrail_name="redact"))
+
+ assistant: Final = data["messages"][1]
+ assert _tool_call_ids_names_and_arguments(assistant["provider_specific_fields"]["server_tool_calls"]) == [
+ ("srvtoolu_01Search", "web_search", '{"query": "[REDACTED]"}')
+ ]
+ assert _tool_call_ids_names_and_arguments(assistant["tool_calls"]) == [
+ ("toolu_01Weather", "get_weather", '{"city": "Paris"}')
+ ]
+
+ @pytest.mark.asyncio
+ async def test_input_guardrail_rewrite_of_a_replayed_server_web_call_reaches_its_hosted_copy(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ data: Final = {
+ "messages": [
+ {"role": "user", "content": "look up jane"},
+ {
+ "role": "assistant",
+ "content": "Searching.",
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ _server_tool_call("srvtoolu_01Search", "web_search", _SEARCH_INPUT),
+ _server_tool_call("srvtoolu_01Fetch", "web_fetch", _FETCH_INPUT),
+ ],
+ "web_search_calls": [
+ build_web_search_call("srvtoolu_01Search", _SEARCH_INPUT, {"content": []}).model_dump(),
+ build_web_search_call(
+ "srvtoolu_01Fetch", _FETCH_INPUT, {"content": []}, tool_name="web_fetch"
+ ).model_dump(),
+ ],
+ },
+ },
+ ]
+ }
+
+ await handler.process_input_messages(data, QueryRedactingGuardrail(guardrail_name="redact"))
+
+ hosted: Final = json.loads(json.dumps(data["messages"][1]["provider_specific_fields"]["web_search_calls"]))
+ assert [(call["id"], call["action"].get("query") or call["action"].get("url")) for call in hosted] == [
+ ("ws_srvtoolu_01Search", "[REDACTED]"),
+ ("ws_srvtoolu_01Fetch", "[REDACTED]"),
+ ]
+
+ @pytest.mark.asyncio
+ async def test_output_guardrail_sees_a_response_that_only_carries_server_tool_calls(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ guardrail: Final = TestOpenAIChatCompletionsHandlerStreamingOutput._recording_guardrail()
+ response: Final = ModelResponse(
+ model="claude-sonnet-4-5",
+ choices=[
+ Choices(
+ finish_reason="stop",
+ index=0,
+ message=Message(
+ content=None,
+ role="assistant",
+ provider_specific_fields={
+ "server_tool_calls": [_server_tool_call("srvtoolu_01Search", "web_search", {"query": "x"})]
+ },
+ ),
+ )
+ ],
+ )
+
+ await handler.process_output_response(response, guardrail)
+
+ assert _tool_call_ids_names_and_arguments(guardrail.seen_inputs[-1]["tool_calls"]) == [
+ ("srvtoolu_01Search", "web_search", '{"query": "x"}')
+ ]
+
+ @pytest.mark.asyncio
+ async def test_output_guardrail_rewrite_of_a_server_tool_call_lands_in_server_tool_calls_only(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ weather_args: Final = json.dumps({"city": "Paris", "query": "forecast"})
+ response: Final = ModelResponse(
+ model="claude-sonnet-4-5",
+ choices=[
+ Choices(
+ finish_reason="tool_calls",
+ index=0,
+ message=Message(
+ content="Looking it up",
+ role="assistant",
+ tool_calls=[
+ ChatCompletionMessageToolCall(
+ id="call_1",
+ type="function",
+ function=Function(name="get_weather", arguments=json.dumps({"city": "Paris"})),
+ )
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ _server_tool_call("srvtoolu_01Search", "web_search", {"query": "jane@example.com"})
+ ]
+ },
+ ),
+ ),
+ Choices(
+ finish_reason="tool_calls",
+ index=1,
+ message=Message(
+ content=None,
+ role="assistant",
+ tool_calls=[
+ ChatCompletionMessageToolCall(
+ id="call_2",
+ type="function",
+ function=Function(name="get_weather", arguments=weather_args),
+ )
+ ],
+ ),
+ ),
+ ],
+ )
+
+ await handler.process_output_response(response, QueryRedactingGuardrail(guardrail_name="redact"))
+
+ first, second = response.choices
+ assert _tool_call_ids_names_and_arguments(first.message.provider_specific_fields["server_tool_calls"]) == [
+ ("srvtoolu_01Search", "web_search", '{"query": "[REDACTED]"}')
+ ]
+ assert [(call.id, call.function.arguments) for call in first.message.tool_calls] == [
+ ("call_1", '{"city": "Paris"}')
+ ]
+ assert [(call.id, call.function.arguments) for call in second.message.tool_calls] == [
+ ("call_2", '{"city": "Paris", "query": "[REDACTED]"}')
+ ]
+ assert not (second.message.provider_specific_fields or {}).get("server_tool_calls")
+
+ @pytest.mark.asyncio
+ async def test_output_guardrail_rewrite_of_a_server_web_call_reaches_its_hosted_copy(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ response: Final = ModelResponse(
+ model="claude-sonnet-4-5",
+ choices=[
+ Choices(
+ finish_reason="stop",
+ index=0,
+ message=Message(
+ content=None,
+ role="assistant",
+ provider_specific_fields={
+ "server_tool_calls": [
+ _server_tool_call("srvtoolu_01Search", "web_search", _SEARCH_INPUT),
+ _server_tool_call("srvtoolu_01Fetch", "web_fetch", _FETCH_INPUT),
+ ],
+ "web_search_calls": [
+ build_web_search_call("srvtoolu_01Search", _SEARCH_INPUT, {"content": []}),
+ build_web_search_call(
+ "srvtoolu_01Fetch",
+ _FETCH_INPUT,
+ {"content": []},
+ status="in_progress",
+ tool_name="web_fetch",
+ ),
+ ],
+ },
+ ),
+ )
+ ],
+ )
+
+ await handler.process_output_response(response, QueryRedactingGuardrail(guardrail_name="redact"))
+
+ assert _hosted_web_call_inputs(response.choices[0].message.provider_specific_fields["web_search_calls"]) == [
+ ("ws_srvtoolu_01Search", "completed", "[REDACTED]"),
+ ("ws_srvtoolu_01Fetch", "in_progress", "[REDACTED]"),
+ ]
+
class MockPassThroughGuardrail(CustomGuardrail):
"""Mock guardrail that passes through without blocking - for testing streaming fallback behavior"""
@@ -1462,6 +1701,136 @@ class TestOpenAIChatCompletionsHandlerStreamingOutput:
assert chunks[0].choices[0].delta.content == "hello [MASKED]"
assert chunks[1].choices[0].delta.content in (None, "")
+ @staticmethod
+ def _ended_server_tool_call_stream_chunks() -> list:
+ from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices
+
+ def chunk(delta: Delta, finish_reason: Optional[str] = None) -> ModelResponseStream:
+ return ModelResponseStream(
+ id="chatcmpl-123",
+ created=1234567890,
+ model="claude-sonnet-4-5",
+ object="chat.completion.chunk",
+ choices=[StreamingChoices(index=0, delta=delta, finish_reason=finish_reason)],
+ )
+
+ search: Final = (
+ {**_server_tool_call("srvtoolu_01Search", "web_search", {"query": "jane@example.com"}), "index": 0}
+ )
+ return [
+ chunk(Delta(content="Searching")),
+ chunk(Delta(content=None, provider_specific_fields={"server_tool_calls": [search]})),
+ chunk(Delta(content=None), finish_reason="stop"),
+ ]
+
+ @staticmethod
+ def _ended_server_fetch_stream_chunks() -> list:
+ from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices
+
+ def chunk(delta: Delta, finish_reason: Optional[str] = None) -> ModelResponseStream:
+ return ModelResponseStream(
+ id="chatcmpl-123",
+ created=1234567890,
+ model="claude-sonnet-4-5",
+ object="chat.completion.chunk",
+ choices=[StreamingChoices(index=0, delta=delta, finish_reason=finish_reason)],
+ )
+
+ def hosted_fetch(
+ tool_input: Mapping[str, str],
+ result: Mapping[str, object],
+ status: Literal["in_progress", "completed"] | None,
+ ) -> dict[str, ResponseFunctionWebSearch]:
+ return {
+ "srvtoolu_01Fetch": build_web_search_call(
+ "srvtoolu_01Fetch", tool_input, result, status=status, tool_name="web_fetch"
+ )
+ }
+
+ fetch: Final = {**_server_tool_call("srvtoolu_01Fetch", "web_fetch", _FETCH_INPUT), "index": 0}
+ return [
+ chunk(
+ Delta(
+ content=None,
+ provider_specific_fields={"web_search_calls": hosted_fetch({}, {"content": []}, "in_progress")},
+ )
+ ),
+ chunk(
+ Delta(
+ content=None,
+ provider_specific_fields={
+ "server_tool_calls": [fetch],
+ "web_search_calls": hosted_fetch(_FETCH_INPUT, {"content": []}, "in_progress"),
+ },
+ )
+ ),
+ chunk(
+ Delta(
+ content=None,
+ provider_specific_fields={
+ "web_search_calls": hosted_fetch(_FETCH_INPUT, {"content": {"type": "web_fetch_result"}}, None)
+ },
+ )
+ ),
+ chunk(Delta(content="Fetched"), finish_reason="stop"),
+ ]
+
+ @pytest.mark.asyncio
+ async def test_deliver_ended_stream_rewrites_reach_every_hosted_copy_of_a_server_web_call(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ chunks: Final = self._ended_server_fetch_stream_chunks()
+
+ await handler.process_output_streaming_response(
+ responses_so_far=chunks,
+ guardrail_to_apply=QueryRedactingGuardrail(guardrail_name="redact"),
+ litellm_logging_obj=None,
+ deliver_ended_stream_rewrites=True,
+ )
+
+ assert [
+ _hosted_web_call_inputs(chunk.choices[0].delta.provider_specific_fields["web_search_calls"].values())
+ for chunk in chunks[:3]
+ ] == [
+ [("ws_srvtoolu_01Fetch", "in_progress", "[REDACTED]")],
+ [("ws_srvtoolu_01Fetch", "in_progress", "[REDACTED]")],
+ [("ws_srvtoolu_01Fetch", "completed", "[REDACTED]")],
+ ]
+
+ @pytest.mark.asyncio
+ async def test_ended_stream_guardrail_sees_streamed_server_tool_calls(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ guardrail: Final = self._recording_guardrail()
+
+ await handler.process_output_streaming_response(
+ responses_so_far=self._ended_server_tool_call_stream_chunks(),
+ guardrail_to_apply=guardrail,
+ litellm_logging_obj=None,
+ deliver_ended_stream_rewrites=True,
+ )
+
+ assert _tool_call_ids_names_and_arguments(guardrail.seen_inputs[-1]["tool_calls"]) == [
+ ("srvtoolu_01Search", "web_search", '{"query": "jane@example.com"}')
+ ]
+
+ @pytest.mark.asyncio
+ async def test_deliver_ended_stream_rewrites_writes_server_tool_call_arguments_back_into_chunks(self):
+ handler: Final = OpenAIChatCompletionsHandler()
+ chunks: Final = self._ended_server_tool_call_stream_chunks()
+
+ await handler.process_output_streaming_response(
+ responses_so_far=chunks,
+ guardrail_to_apply=QueryRedactingGuardrail(guardrail_name="redact"),
+ litellm_logging_obj=None,
+ deliver_ended_stream_rewrites=True,
+ )
+
+ delta: Final = chunks[1].choices[0].delta
+ assert _tool_call_ids_names_and_arguments(delta.provider_specific_fields["server_tool_calls"]) == [
+ ("srvtoolu_01Search", "web_search", '{"query": "[REDACTED]"}')
+ ]
+ assert delta.tool_calls is None
+ assert chunks[0].choices[0].delta.content == "Searching"
+
class TestUndecoratedGuardrailIsRecorded:
"""LIT-5983 regression: the handler calls apply_guardrail bare, so a custom guardrail
@@ -2323,6 +2692,29 @@ class TestStreamingScanKey:
assert len(ended_key.tool_calls) == 1 and "run_shell" in ended_key.tool_calls[0]
assert ended_key.tool_calls_in_flight is False
+ def test_server_tool_calls_hold_the_window_and_enter_the_key_once_the_stream_has_ended(self):
+ from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices
+
+ handler: Final = OpenAIChatCompletionsHandler()
+ server_chunk: Final = ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=0,
+ delta=Delta(
+ content=None,
+ provider_specific_fields={
+ "server_tool_calls": [_server_tool_call("srvtoolu_01Search", "web_search", {"query": "x"})]
+ },
+ ),
+ )
+ ]
+ )
+ open_key: Final = handler.get_streaming_scan_key([self._chunk("hi"), server_chunk])
+ ended_key: Final = handler.get_streaming_scan_key([self._chunk("hi"), server_chunk, self._chunk(None, "stop")])
+ assert open_key.tool_calls_in_flight is True
+ assert len(ended_key.tool_calls) == 1 and "web_search" in ended_key.tool_calls[0]
+ assert ended_key != open_key
+
def test_text_after_the_first_choice_finishes_still_changes_the_key(self):
handler = OpenAIChatCompletionsHandler()
first_done = [self._chunk("a", index=0), self._chunk("b", finish_reason="stop", index=0)]
diff --git a/tests/unit/llms/openai/responses/count_tokens/test_token_counter.py b/tests/unit/llms/openai/responses/count_tokens/test_token_counter.py
index 29c25c5fdbf..35b544be31f 100644
--- a/tests/unit/llms/openai/responses/count_tokens/test_token_counter.py
+++ b/tests/unit/llms/openai/responses/count_tokens/test_token_counter.py
@@ -301,7 +301,7 @@ async def test_anthropic_format_counts_the_structured_output_schema(monkeypatch:
assert sent == {
"model": "gpt-4o",
"input": [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Answer in JSON."}]}],
- "text": {"format": {"type": "json_schema", "name": "structured_output", "schema": schema, "strict": False}},
+ "text": {"format": {"type": "json_schema", "name": "structured_output", "schema": schema, "strict": True}},
}
diff --git a/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py b/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
index d8954c4072a..a5bb49f0793 100644
--- a/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
+++ b/tests/unit/llms/openai/responses/test_openai_responses_guardrail_handler.py
@@ -20,7 +20,9 @@ import pytest
from fastapi import HTTPException
from pydantic import BaseModel
+from openai.types.responses.response_function_web_search import ActionSearch
from openai.types.responses import (
+ ResponseFunctionWebSearch,
ResponseCustomToolCall,
ResponseCustomToolCallInputDeltaEvent,
ResponseCustomToolCallInputDoneEvent,
@@ -61,6 +63,7 @@ from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.responses.litellm_completion_transformation.tool_search import build_tool_search_call
from litellm.types.responses.main import (
CustomToolCallOutputItem,
+ build_web_search_call,
GenericResponseOutputItem,
OutputText,
OutputToolSearchCall,
@@ -250,6 +253,30 @@ CUSTOM_TOOL_CALL_ITEM = {
}
+def _claude_web_calls_and_a_native_search() -> list[ResponseFunctionWebSearch]:
+ return [
+ build_web_search_call("srvtoolu_1", {"query": "persimmon"}, None, "completed"),
+ build_web_search_call("srvtoolu_2", {"url": "https://persimmon.example"}, None, "in_progress", "web_fetch"),
+ ResponseFunctionWebSearch(
+ id="ws_native",
+ type="web_search_call",
+ status="completed",
+ action=ActionSearch(type="search", query="persimmon"),
+ ),
+ ]
+
+
+def _web_call_actions(items: list[object]) -> list[dict[str, object]]:
+ return [ResponseFunctionWebSearch.model_validate(item, from_attributes=True).action.model_dump() for item in items]
+
+
+_MASKED_WEB_CALL_ACTIONS: Final = [
+ {"type": "search", "query": "[MASKED]", "queries": ["[MASKED]"], "sources": []},
+ {"type": "open_page", "url": "https://[MASKED].example"},
+ {"type": "search", "query": "persimmon", "queries": None, "sources": None},
+]
+
+
class TestOpenAIResponsesHandlerDiscovery:
"""Test that the handler is properly discovered by the guardrail system"""
@@ -1823,6 +1850,27 @@ class TestOpenAIResponsesHandlerToolCallExtraction:
assert search_call.call_id == "srvtoolu_1"
assert "name" not in search_call.model_dump()
+ @pytest.mark.asyncio
+ async def test_process_output_response_scans_and_rewrites_claude_web_calls_but_not_native_search(self):
+ guardrail: Final = RecordingPersimmonMaskingGuardrail(guardrail_name="mask")
+ response: Final = ResponsesAPIResponse(
+ id="resp_1",
+ created_at=1,
+ model="claude-sonnet-5-5",
+ object="response",
+ status="completed",
+ output=_claude_web_calls_and_a_native_search(),
+ )
+
+ result: Final = await OpenAIResponsesHandler().process_output_response(response, guardrail)
+
+ assert guardrail.seen_tool_calls == [
+ ("web_search", '{"query": "persimmon"}'),
+ ("web_fetch", '{"url": "https://persimmon.example"}'),
+ ]
+ assert [item.type for item in result.output] == ["web_search_call"] * 3
+ assert _web_call_actions(result.output) == _MASKED_WEB_CALL_ACTIONS
+
@pytest.mark.asyncio
async def test_process_output_response_never_returns_original_search_arguments_after_a_non_json_rewrite(self):
response = ResponsesAPIResponse(
@@ -2758,6 +2806,117 @@ class TestOpenAIResponsesHandlerStreamingOutputProcessing:
assert completed.response.output[0].arguments == {"query": "[MASKED]"}
assert "name" not in done.item.model_dump()
+ @pytest.mark.asyncio
+ async def test_deliver_ended_stream_scans_and_rewrites_claude_web_calls_but_not_native_search(self):
+ from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
+ from litellm.types.llms.openai import (
+ OutputItemAddedEvent,
+ OutputItemDoneEvent,
+ ResponseCompletedEvent,
+ ResponsesAPIStreamEvents,
+ )
+
+ items: Final = _claude_web_calls_and_a_native_search()
+ added: Final = [
+ OutputItemAddedEvent(
+ type=ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
+ output_index=index,
+ item=BaseLiteLLMOpenAIResponseObject(**{**item.model_dump(), "action": None}),
+ )
+ for index, item in enumerate(items)
+ ]
+ done: Final = [
+ OutputItemDoneEvent(
+ type=ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
+ output_index=index,
+ sequence_number=index + 3,
+ item=BaseLiteLLMOpenAIResponseObject(**item.model_dump()),
+ )
+ for index, item in enumerate(items)
+ ]
+ completed: Final = ResponseCompletedEvent(
+ type=ResponsesAPIStreamEvents.RESPONSE_COMPLETED,
+ response=ResponsesAPIResponse(
+ id="resp_1",
+ created_at=1,
+ model="claude-sonnet-5-5",
+ object="response",
+ status="completed",
+ output=_claude_web_calls_and_a_native_search(),
+ ),
+ )
+ guardrail: Final = RecordingPersimmonMaskingGuardrail(guardrail_name="mask")
+
+ await OpenAIResponsesHandler().process_output_streaming_response(
+ responses_so_far=[*added, *done, completed],
+ guardrail_to_apply=guardrail,
+ litellm_logging_obj=None,
+ deliver_ended_stream_rewrites=True,
+ )
+
+ assert sorted(guardrail.seen_tool_calls) == [
+ ("web_fetch", '{"url": "https://persimmon.example"}'),
+ ("web_search", '{"query": "persimmon"}'),
+ ]
+ assert [event.item.model_dump()["action"] for event in added] == [None, None, None]
+ assert _web_call_actions([event.item for event in done]) == _MASKED_WEB_CALL_ACTIONS
+ assert _web_call_actions(completed.response.output) == _MASKED_WEB_CALL_ACTIONS
+
+ @pytest.mark.asyncio
+ async def test_deliver_ended_stream_rewrites_dict_shaped_claude_web_calls_but_not_native_search(self):
+ items: Final = [item.model_dump() for item in _claude_web_calls_and_a_native_search()]
+ done: Final = [
+ {"type": "response.output_item.done", "output_index": index, "item": dict(item)}
+ for index, item in enumerate(items)
+ ]
+ completed: Final = {
+ "type": "response.completed",
+ "response": {
+ "id": "resp_1",
+ "model": "claude-sonnet-5-5",
+ "status": "completed",
+ "output": [dict(item) for item in items],
+ },
+ }
+ guardrail: Final = RecordingPersimmonMaskingGuardrail(guardrail_name="mask")
+
+ await OpenAIResponsesHandler().process_output_streaming_response(
+ responses_so_far=[*done, completed],
+ guardrail_to_apply=guardrail,
+ litellm_logging_obj=None,
+ deliver_ended_stream_rewrites=True,
+ )
+
+ assert sorted(guardrail.seen_tool_calls) == [
+ ("web_fetch", '{"url": "https://persimmon.example"}'),
+ ("web_search", '{"query": "persimmon"}'),
+ ]
+ assert _web_call_actions([event["item"] for event in done]) == _MASKED_WEB_CALL_ACTIONS
+ assert _web_call_actions(completed["response"]["output"]) == _MASKED_WEB_CALL_ACTIONS
+
+ @pytest.mark.asyncio
+ async def test_streaming_output_item_done_reports_a_claude_web_search_call(self):
+ from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
+ from litellm.types.llms.openai import OutputItemDoneEvent, ResponsesAPIStreamEvents
+
+ done: Final = OutputItemDoneEvent(
+ type=ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
+ output_index=0,
+ sequence_number=1,
+ item=BaseLiteLLMOpenAIResponseObject(
+ **build_web_search_call("srvtoolu_1", {"query": "persimmon"}, None, "completed").model_dump()
+ ),
+ )
+ guardrail: Final = RecordingPersimmonMaskingGuardrail(guardrail_name="mask")
+
+ await OpenAIResponsesHandler().process_output_streaming_response(
+ responses_so_far=[done],
+ guardrail_to_apply=guardrail,
+ litellm_logging_obj=None,
+ )
+
+ assert guardrail.seen_tool_calls == [("web_search", '{"query": "persimmon"}')]
+
@pytest.mark.asyncio
async def test_deliver_ended_stream_rewrites_a_native_tool_search_call_that_has_no_call_id(self):
def search_call(arguments: dict, status: str) -> dict:
diff --git a/tests/unit/llms/openai/responses/test_openai_responses_transformation.py b/tests/unit/llms/openai/responses/test_openai_responses_transformation.py
index 6b36a683df7..cd300d74b59 100644
--- a/tests/unit/llms/openai/responses/test_openai_responses_transformation.py
+++ b/tests/unit/llms/openai/responses/test_openai_responses_transformation.py
@@ -17,6 +17,8 @@ from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from litellm.llms.azure_ai.responses.transformation import AzureAIResponsesAPIConfig
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
+from litellm.llms.bedrock.responses.transformation import BedrockOpenAIResponsesConfig
+from litellm.llms.bedrock_mantle.responses.transformation import BedrockMantleResponsesAPIConfig
from litellm.llms.hosted_vllm.responses.transformation import HostedVLLMResponsesAPIConfig
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.llms.xai.responses.transformation import XAIResponsesAPIConfig
@@ -1350,7 +1352,15 @@ class TestOpenAIResponsesAPIConfig:
)
return list(bridged.output)
- @pytest.mark.parametrize("config", [OpenAIResponsesAPIConfig(), AzureOpenAIResponsesAPIConfig()])
+ @pytest.mark.parametrize(
+ "config",
+ [
+ OpenAIResponsesAPIConfig(),
+ AzureOpenAIResponsesAPIConfig(),
+ BedrockOpenAIResponsesConfig(),
+ BedrockMantleResponsesAPIConfig(),
+ ],
+ )
def test_claude_reasoning_minted_by_the_bridge_is_dropped_before_the_history_reaches_openai(self, config):
saved_claude_turn = json.loads(
json.dumps([item.model_dump() for item in self._claude_turn_bridged_to_responses_output()])
diff --git a/tests/unit/llms/vertex_ai/text_to_speech/test_transformation.py b/tests/unit/llms/vertex_ai/text_to_speech/test_transformation.py
index b3d6d5b952f..59c781c5a59 100644
--- a/tests/unit/llms/vertex_ai/text_to_speech/test_transformation.py
+++ b/tests/unit/llms/vertex_ai/text_to_speech/test_transformation.py
@@ -102,6 +102,61 @@ class TestVertexAITextToSpeechConfig:
assert url == custom_url
+ @pytest.mark.parametrize(
+ ("vertex_location", "expected_host"),
+ [
+ ("eu", "eu-texttospeech.googleapis.com"),
+ ("europe-west4", "eu-texttospeech.googleapis.com"),
+ ("europe-west1", "eu-texttospeech.googleapis.com"),
+ ("europe-west3", "eu-texttospeech.googleapis.com"),
+ ("europe-west8", "eu-texttospeech.googleapis.com"),
+ ("europe-west9", "eu-texttospeech.googleapis.com"),
+ ("europe-west12", "eu-texttospeech.googleapis.com"),
+ ("us", "us-texttospeech.googleapis.com"),
+ ("us-central1", "us-texttospeech.googleapis.com"),
+ ("us-west2", "us-texttospeech.googleapis.com"),
+ ("us-west5", "us-texttospeech.googleapis.com"),
+ ],
+ )
+ def test_get_complete_url_uses_regional_endpoint_for_vertex_location(
+ self, vertex_location: str, expected_host: str
+ ) -> None:
+ config: Final = VertexAITextToSpeechConfig()
+
+ url: Final = config.get_complete_url(
+ model="vertex_ai/chirp",
+ api_base=None,
+ litellm_params={"vertex_location": vertex_location},
+ )
+
+ assert url == f"https://{expected_host}/v1/text:synthesize"
+
+ @pytest.mark.parametrize(
+ "vertex_location", ["global", "europe-west2", "asia-northeast1", "northamerica-northeast1", None]
+ )
+ def test_get_complete_url_keeps_global_endpoint_without_a_regional_match(self, vertex_location: str | None) -> None:
+ config: Final = VertexAITextToSpeechConfig()
+
+ url: Final = config.get_complete_url(
+ model="vertex_ai/chirp",
+ api_base=None,
+ litellm_params={"vertex_location": vertex_location},
+ )
+
+ assert url == "https://texttospeech.googleapis.com/v1/text:synthesize"
+
+ def test_get_complete_url_api_base_wins_over_vertex_location(self) -> None:
+ config: Final = VertexAITextToSpeechConfig()
+
+ custom_url: Final = "https://custom-tts-endpoint.example.com/v1/synthesize"
+ url: Final = config.get_complete_url(
+ model="vertex_ai/chirp",
+ api_base=custom_url,
+ litellm_params={"vertex_location": "eu"},
+ )
+
+ assert url == custom_url
+
@patch.object( # test-quality-ok: isolates provider credentials while testing request serialization
VertexAITextToSpeechConfig, "_ensure_access_token"
)
@@ -1332,8 +1387,7 @@ def test_litellm_speech_vertex_ai_chirp(mock_get_token, mock_ensure_token, mock_
mock_post.assert_called_once()
call_kwargs = mock_post.call_args.kwargs
- # Verify the URL is the Google Cloud TTS API
- assert call_kwargs["url"] == "https://texttospeech.googleapis.com/v1/text:synthesize"
+ assert call_kwargs["url"] == "https://us-texttospeech.googleapis.com/v1/text:synthesize"
# Verify request body structure
assert "json" in call_kwargs
@@ -1863,9 +1917,13 @@ async def test_aspeech_vertex_ai_gemini_tts_mp3_with_a_numeric_speaking_rate_ret
(
("us-central1", "https://us-texttospeech.googleapis.com/v1/text:synthesize"),
("us-east5", "https://us-texttospeech.googleapis.com/v1/text:synthesize"),
+ ("us-west3", "https://us-texttospeech.googleapis.com/v1/text:synthesize"),
+ ("us-west5", "https://us-texttospeech.googleapis.com/v1/text:synthesize"),
("us", "https://us-texttospeech.googleapis.com/v1/text:synthesize"),
("europe-west4", "https://eu-texttospeech.googleapis.com/v1/text:synthesize"),
("europe-north1", "https://eu-texttospeech.googleapis.com/v1/text:synthesize"),
+ ("europe-west9", "https://eu-texttospeech.googleapis.com/v1/text:synthesize"),
+ ("europe-west10", "https://eu-texttospeech.googleapis.com/v1/text:synthesize"),
("eu", "https://eu-texttospeech.googleapis.com/v1/text:synthesize"),
("northamerica-northeast1", "https://northamerica-northeast1-texttospeech.googleapis.com/v1/text:synthesize"),
("global", _SYNTHESIZE_URL),
@@ -1914,6 +1972,26 @@ async def test_aspeech_vertex_ai_gemini_tts_mp3_rejects_a_region_cloud_tts_does_
assert not respx_mock.calls
+@pytest.mark.asyncio
+async def test_aspeech_vertex_ai_chirp_keeps_the_global_host_when_only_vertexai_location_is_set(
+ respx_mock: respx.MockRouter, google_token_endpoint: responses.RequestsMock, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setenv("VERTEXAI_LOCATION", "europe-west4")
+ route: Final = respx_mock.post(_SYNTHESIZE_URL).mock(
+ return_value=httpx.Response(200, json={"audioContent": base64.b64encode(b"global-audio").decode()})
+ )
+
+ response: Final = await litellm.aspeech(
+ model="vertex_ai/chirp",
+ input="Bonjour",
+ voice="fr-FR-Chirp3-HD-Aoede",
+ vertex_credentials=_AUTHORIZED_USER,
+ )
+
+ assert route.call_count == 1
+ assert response.content == b"global-audio"
+
+
@pytest.mark.asyncio
async def test_router_gemini_tts_mp3_unknown_voice_is_a_bad_request_that_keeps_the_deployment_serving(
respx_mock: respx.MockRouter, google_token_endpoint: responses.RequestsMock
diff --git a/tests/unit/proxy/auth/test_auth_checks_object_access_and_lookup.py b/tests/unit/proxy/auth/test_auth_checks_object_access_and_lookup.py
index 7162667288d..0e761eb5fd1 100644
--- a/tests/unit/proxy/auth/test_auth_checks_object_access_and_lookup.py
+++ b/tests/unit/proxy/auth/test_auth_checks_object_access_and_lookup.py
@@ -2401,6 +2401,37 @@ async def test_deny_by_default_ignores_tools_that_name_no_vector_store(
)
+@pytest.mark.asyncio
+@pytest.mark.parametrize("deny_by_default", [False, True], ids=["legacy", "deny-by-default"])
+@pytest.mark.parametrize(
+ ("key_vector_stores", "denied_by"),
+ [(["KBSTOREA"], _KEY_DENIED), (["KBSTOREB"], None)],
+ ids=["ungranted", "granted"],
+)
+async def test_file_search_inside_additional_tools_input_item_needs_a_vector_store_grant(
+ deny_by_default: bool, key_vector_stores: list[str], denied_by: ProxyErrorTypes | None
+):
+ await _assert_rag_query_outcome(
+ _common_checks_for_rag_query(
+ {"vector_store_deny_by_default": deny_by_default},
+ _virtual_key(object_permission_id="key-permission"),
+ SimpleNamespace(vector_stores=key_vector_stores),
+ request_body={
+ "model": "gpt-4o-mini",
+ "input": [
+ {
+ "type": "additional_tools",
+ "role": "developer",
+ "tools": [{"type": "file_search", "vector_store_ids": ["KBSTOREB"]}],
+ },
+ {"type": "message", "role": "user", "content": "what is in this KB?"},
+ ],
+ },
+ ),
+ denied_by,
+ )
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize(
"request_body",
diff --git a/tests/unit/proxy/auth/test_login_utils.py b/tests/unit/proxy/auth/test_login_utils.py
index 5148e38aa61..bd5bdeafe95 100644
--- a/tests/unit/proxy/auth/test_login_utils.py
+++ b/tests/unit/proxy/auth/test_login_utils.py
@@ -5,6 +5,7 @@ This module tests the refactored login logic that was moved from proxy_server.py
to login_utils.py for better reusability.
"""
+import asyncio
import hashlib
import os
from collections.abc import Mapping
@@ -2365,3 +2366,160 @@ class TestScreenLoginPasswordForBreach:
)
is True
)
+
+
+_PBKDF2_PASSWORD: Final = "Seed-Lamp-4-River"
+_PBKDF2_ROW: Final = "pbkdf2:AAECAwQFBgcICQoLDA0OD+uxhNqRo/3ZFLEaKARqFBT+N3Kqek/SGM/5AvocSN0T"
+_SCRYPT_ROW: Final = "scrypt:AAECAwQFBgcICQoLDA0OD7hIwlM/k8ytnffjFDt6AuO+BvqtmZvDkWXJP+A3bf0N"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "stored, schemes_written",
+ [
+ (_SCRYPT_ROW, []),
+ (_PBKDF2_ROW, []),
+ (hashlib.sha256(_PBKDF2_PASSWORD.encode()).hexdigest(), ["scrypt"]),
+ ],
+ ids=["scrypt", "pbkdf2", "sha256"],
+)
+async def test_login_rehash_rewrites_only_a_legacy_row(stored: str, schemes_written: list[str]) -> None:
+ from litellm.proxy.auth.login_utils import _rehash_password_if_needed
+
+ prisma: Final = MagicMock()
+ prisma.db.litellm_usertable.update = AsyncMock()
+ with patch( # test-quality-ok: the rehash reads proxy_server's prisma_client global; no injection seam
+ "litellm.proxy.proxy_server.prisma_client", prisma
+ ):
+ await _rehash_password_if_needed("u-1", _PBKDF2_PASSWORD, stored)
+
+ updates: Final = prisma.db.litellm_usertable.update.await_args_list
+ assert [update.kwargs["data"]["password"].split(":")[0] for update in updates] == schemes_written
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_login_runs_one_pbkdf2_off_the_event_loop_and_writes_no_row(
+ monkeypatch: pytest.MonkeyPatch, pbkdf2_on_event_loop: list[bool]
+) -> None:
+ from litellm.proxy.auth.login_utils import authenticate_user
+
+ monkeypatch.setenv("DATABASE_URL", "postgresql://stub")
+ repository: Final = _known_user("user@corp.com")
+ repository.return_value.table.find_first.return_value.password = _PBKDF2_ROW
+ repository.return_value.table.update = AsyncMock()
+ with (
+ patch( # test-quality-ok: the user lookup and rehash write are the database boundary
+ "litellm.proxy.auth.login_utils.UserRepository", repository
+ ),
+ patch( # test-quality-ok: the rehash reads proxy_server's prisma_client global; no injection seam
+ "litellm.proxy.proxy_server.prisma_client", MagicMock()
+ ),
+ patch( # test-quality-ok: success mints a UI key; faked so no DB is needed
+ "litellm.proxy.auth.login_utils.generate_key_helper_fn", new=AsyncMock(return_value={"token": "sk-ui"})
+ ),
+ ):
+ result: Final = await authenticate_user(
+ username="user@corp.com",
+ password=_PBKDF2_PASSWORD,
+ master_key="sk-master",
+ prisma_client=MagicMock(),
+ throttle=_unlimited_throttle(),
+ general_settings=_POLICY_NO_BREACH_CHECK,
+ )
+
+ assert result.key == "sk-ui"
+ assert pbkdf2_on_event_loop == [False]
+ repository.return_value.table.update.assert_not_awaited()
+
+
+async def _login_whose_user_gets_blocked_during_the_row_lookup(monkeypatch: pytest.MonkeyPatch, row: str):
+ from litellm.proxy.auth.login_utils import authenticate_user
+
+ throttle: Final = _throttle(user_limit=1)
+ repository: Final = _known_user()
+ user: Final = repository.return_value.table.find_first.return_value
+ user.password = row
+
+ async def block_the_user_then_return_the_row(**_: object) -> MagicMock:
+ await throttle.record_failure("known@example.com")
+ await throttle.record_failure("known@example.com")
+ return user
+
+ repository.return_value.table.find_first = AsyncMock(side_effect=block_the_user_then_return_the_row)
+ monkeypatch.setenv("DATABASE_URL", "postgresql://stub")
+ with (
+ patch( # test-quality-ok: the user lookup is the database boundary
+ "litellm.proxy.auth.login_utils.UserRepository", repository
+ ),
+ patch( # test-quality-ok: the rehash reads proxy_server's prisma_client global; no injection seam
+ "litellm.proxy.proxy_server.prisma_client", MagicMock()
+ ),
+ patch( # test-quality-ok: success mints a UI key; faked so no DB is needed
+ "litellm.proxy.auth.login_utils.generate_key_helper_fn", new=AsyncMock(return_value={"token": "sk-ui"})
+ ),
+ ):
+ return await authenticate_user(
+ username="known@example.com",
+ password=_PBKDF2_PASSWORD,
+ master_key="sk-master",
+ prisma_client=MagicMock(),
+ throttle=throttle,
+ general_settings=_POLICY_NO_BREACH_CHECK,
+ )
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_login_hides_the_right_password_when_its_user_got_blocked_while_it_waited(
+ monkeypatch: pytest.MonkeyPatch, pbkdf2_on_event_loop: list[bool]
+) -> None:
+ from litellm.proxy._types import ProxyException
+
+ with pytest.raises(ProxyException) as blocked:
+ await _login_whose_user_gets_blocked_during_the_row_lookup(monkeypatch, _PBKDF2_ROW)
+
+ assert blocked.value.code == "429"
+ assert pbkdf2_on_event_loop == []
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_burst_of_wrong_passwords_stops_deriving_once_its_user_is_blocked(
+ monkeypatch: pytest.MonkeyPatch, pbkdf2_on_event_loop: list[bool]
+) -> None:
+ from litellm.proxy.auth.login_utils import authenticate_user
+
+ monkeypatch.setenv("DATABASE_URL", "postgresql://stub")
+ repository: Final = _known_user()
+ repository.return_value.table.find_first.return_value.password = _PBKDF2_ROW
+ throttle: Final = _throttle(user_limit=1)
+ with patch( # test-quality-ok: the user lookup is the database boundary
+ "litellm.proxy.auth.login_utils.UserRepository", repository
+ ):
+ refusals: Final = await asyncio.gather(
+ *(
+ authenticate_user(
+ username="known@example.com",
+ password="Wrong-Lamp-4-River",
+ master_key="sk-master",
+ prisma_client=MagicMock(),
+ throttle=throttle,
+ general_settings=_POLICY_NO_BREACH_CHECK,
+ )
+ for _ in range(6)
+ ),
+ return_exceptions=True,
+ )
+
+ assert [refusal.code for refusal in refusals] == ["401", "401", "429", "429", "429", "429"]
+ assert pbkdf2_on_event_loop == [False, False]
+
+
+@pytest.mark.asyncio
+async def test_standard_login_past_the_throttle_keeps_its_answer_when_its_user_gets_blocked(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ result: Final = await _login_whose_user_gets_blocked_during_the_row_lookup(monkeypatch, _SCRYPT_ROW)
+
+ assert result.key == "sk-ui"
diff --git a/tests/unit/proxy/auth/test_model_checks.py b/tests/unit/proxy/auth/test_model_checks.py
index bc9366aa607..cd5410ffd18 100644
--- a/tests/unit/proxy/auth/test_model_checks.py
+++ b/tests/unit/proxy/auth/test_model_checks.py
@@ -35,6 +35,15 @@ def test_get_team_models_for_all_models_and_team_only_models():
assert set(result) == set(combined_models)
+def test_get_team_models_lists_models_in_the_order_the_team_set_them() -> None:
+ from litellm.proxy.auth.model_checks import get_team_models
+
+ team_models: Final = ["zeta", "alpha", "mu", "beta", "omega", "kappa", "delta", "all-proxy-models"]
+ result: Final = get_team_models(team_models, ["proxy-b", "proxy-a"], {"grp": ["g1"]}, True)
+
+ assert result == [*team_models, "proxy-b", "proxy-a", "grp", "g1"]
+
+
def test_get_team_models_all_proxy_models_includes_access_groups():
"""
When a team has 'all-proxy-models' and include_model_access_groups=True,
@@ -1411,24 +1420,21 @@ def test_get_complete_model_list_sentinel_only_grants_nothing():
assert result == []
-def test_get_provider_models_admits_providers_without_a_static_catalog():
- """Providers without a static model list are no longer rejected up front.
+@pytest.fixture
+def fresh_available_models_cache(monkeypatch: pytest.MonkeyPatch) -> None:
+ import litellm.utils as litellm_utils
- With endpoint discovery off, get_valid_models falls back to the (empty)
- static list, so the result is [] rather than None. Before the fix this
- returned None and the wildcard was never expanded.
- """
- import litellm
+ monkeypatch.setattr(litellm_utils, "_model_cache", litellm_utils.AvailableModelsCache())
+
+
+def test_get_provider_models_admits_providers_without_a_static_catalog():
from litellm.proxy.auth.model_checks import get_provider_models
from litellm.types.router import LiteLLM_Params
- assert "litellm_proxy" not in litellm.models_by_provider
- assert "hosted_vllm" not in litellm.models_by_provider
-
result = get_provider_models(
- "litellm_proxy",
+ "hosted_vllm",
litellm_params=LiteLLM_Params(
- model="litellm_proxy/*",
+ model="hosted_vllm/*",
api_base="http://upstream:4000",
api_key="sk-upstream",
),
@@ -1437,47 +1443,103 @@ def test_get_provider_models_admits_providers_without_a_static_catalog():
assert result == []
-def test_get_complete_model_list_discovers_litellm_proxy_wildcard_models(monkeypatch):
- """A litellm_proxy/* deployment lists the upstream proxy's models when endpoint discovery is on."""
+@pytest.mark.parametrize("wildcard", ["litellm_proxy/*", "team-proxy/*"])
+@pytest.mark.usefixtures("fresh_available_models_cache")
+@respx.mock
+def test_get_complete_model_list_discovers_litellm_proxy_wildcard_models_by_default(
+ monkeypatch: pytest.MonkeyPatch, wildcard: str
+) -> None:
import litellm
- from litellm import Router
- from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.proxy.auth.model_checks import get_complete_model_list
- monkeypatch.setattr(litellm, "check_provider_endpoint", True)
- captured = {}
-
- def fake_get_models(self, api_key=None, api_base=None):
- captured["api_key"] = api_key
- captured["api_base"] = api_base
- return ["gpt-4o", "claude-sonnet"]
-
- monkeypatch.setattr(OpenAIGPTConfig, "get_models", fake_get_models)
-
- router = Router(
+ monkeypatch.setattr(litellm, "check_provider_endpoint", False)
+ catalog: Final = respx.get(
+ "http://upstream-discovery.test:4000/gateway/v1/models",
+ headers={"Authorization": "Bearer test-upstream-key"},
+ ).mock(return_value=httpx.Response(200, json={"data": [{"id": "chat-model"}, {"id": "vendor/chat-model"}]}))
+ router: Final = Router(
model_list=[
{
- "model_name": "litellm_proxy/*",
+ "model_name": wildcard,
"litellm_params": {
"model": "litellm_proxy/*",
- "api_base": "http://upstream:4000",
- "api_key": "sk-upstream",
+ "api_base": "http://upstream-discovery.test:4000/gateway/v1",
+ "api_key": "test-upstream-key",
},
}
]
)
- result = get_complete_model_list(
+ result: Final = get_complete_model_list(
key_models=[],
team_models=[],
- proxy_model_list=["litellm_proxy/*"],
+ proxy_model_list=list(router.registry().names(kind="group_set")),
user_model=None,
infer_model_from_keys=False,
llm_router=router,
)
- assert captured == {"api_key": "sk-upstream", "api_base": "http://upstream:4000"}
- assert "litellm_proxy/gpt-4o" in result
- assert "litellm_proxy/claude-sonnet" in result
+ assert result == [wildcard.replace("*", "chat-model"), wildcard.replace("*", "vendor/chat-model")]
+ assert catalog.call_count == 1
+
+
+@pytest.mark.usefixtures("fresh_available_models_cache")
+@respx.mock
+def test_litellm_proxy_discovery_cache_is_scoped_to_the_upstream_key_and_gateway(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import litellm
+ from litellm.proxy.auth.model_checks import get_known_models_from_wildcard
+ from litellm.types.router import LiteLLM_Params
+
+ monkeypatch.setattr(litellm, "check_provider_endpoint", False)
+ first: Final = respx.get(
+ "http://first-proxy.test:4000/v1/models", headers={"Authorization": "Bearer test-first-key"}
+ ).mock(return_value=httpx.Response(200, json={"data": [{"id": "first-key-model"}]}))
+ second: Final = respx.get(
+ "http://first-proxy.test:4000/v1/models", headers={"Authorization": "Bearer test-second-key"}
+ ).mock(return_value=httpx.Response(200, json={"data": [{"id": "second-key-model"}]}))
+ other_gateway: Final = respx.get(
+ "http://other-proxy.test:4000/v1/models", headers={"Authorization": "Bearer test-first-key"}
+ ).mock(return_value=httpx.Response(200, json={"data": [{"id": "other-gateway-model"}]}))
+ deployment: Final = LiteLLM_Params(
+ model="litellm_proxy/*", api_base="http://first-proxy.test:4000/v1", api_key="test-first-key"
+ )
+
+ assert get_known_models_from_wildcard("litellm_proxy/*", deployment) == ["litellm_proxy/first-key-model"]
+ assert get_known_models_from_wildcard(
+ "litellm_proxy/*", deployment.model_copy(update={"api_key": "test-second-key"})
+ ) == ["litellm_proxy/second-key-model"]
+ assert get_known_models_from_wildcard(
+ "litellm_proxy/*", deployment.model_copy(update={"api_base": "http://other-proxy.test:4000/v1"})
+ ) == ["litellm_proxy/other-gateway-model"]
+ assert get_known_models_from_wildcard("litellm_proxy/*", deployment) == ["litellm_proxy/first-key-model"]
+ assert (first.call_count, second.call_count, other_gateway.call_count) == (1, 1, 1)
+
+
+@pytest.mark.usefixtures("fresh_available_models_cache")
+@respx.mock
+def test_undeployed_litellm_proxy_wildcard_does_not_list_the_environment_gateway(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import litellm
+ from litellm.proxy.auth.model_checks import get_complete_model_list
+
+ monkeypatch.setattr(litellm, "check_provider_endpoint", False)
+ monkeypatch.setenv("LITELLM_PROXY_API_BASE", "http://environment-proxy.test:4000")
+ monkeypatch.setenv("LITELLM_PROXY_API_KEY", "test-environment-key")
+ catalog: Final = respx.get("http://environment-proxy.test:4000/v1/models").mock(
+ return_value=httpx.Response(200, json={"data": [{"id": "environment-only-model"}]})
+ )
+ listed: Final = get_complete_model_list(
+ key_models=["litellm_proxy/*"],
+ team_models=[],
+ proxy_model_list=[],
+ user_model=None,
+ infer_model_from_keys=False,
+ llm_router=Router(model_list=[]),
+ )
+
+ assert (listed, catalog.call_count) == ([], 0)
def test_get_provider_models_returns_none_for_an_unknown_provider():
diff --git a/tests/unit/proxy/auth/test_onboarding.py b/tests/unit/proxy/auth/test_onboarding.py
index 46a48c21353..c449ffed806 100644
--- a/tests/unit/proxy/auth/test_onboarding.py
+++ b/tests/unit/proxy/auth/test_onboarding.py
@@ -462,7 +462,7 @@ async def test_claim_token_sets_accepted_at_after_password_written():
prisma.db.litellm_usertable.update.assert_called_once()
call_kwargs = prisma.db.litellm_usertable.update.call_args
assert call_kwargs.kwargs["where"] == {"user_id": "user-123"}
- assert "password" in call_kwargs.kwargs["data"]
+ assert call_kwargs.kwargs["data"]["password"].startswith(("scrypt:", "pbkdf2:"))
# A freshly claimed, policy-screened password lifts any pending forced
# reset and re-arms the login-time breach screen.
assert call_kwargs.kwargs["data"]["password_reset_required"] is False
@@ -477,6 +477,13 @@ async def test_claim_token_sets_accepted_at_after_password_written():
assert outer_claims["key"] == "sk-generated-key"
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_claim_hashes_off_the_event_loop(pbkdf2_on_event_loop: list[bool]) -> None:
+ await test_claim_token_sets_accepted_at_after_password_written()
+ assert pbkdf2_on_event_loop == [False]
+
+
@pytest.mark.asyncio
async def test_claim_token_revokes_existing_ui_sessions():
"""A claimed invite/reset link changes the password; any UI session minted
diff --git a/tests/unit/proxy/auth/test_password_policy.py b/tests/unit/proxy/auth/test_password_policy.py
index 0ad1083ebe0..ea60c08a4ea 100644
--- a/tests/unit/proxy/auth/test_password_policy.py
+++ b/tests/unit/proxy/auth/test_password_policy.py
@@ -9,6 +9,7 @@ httpx.MockTransport, so no network is touched and nothing is monkeypatched.
import asyncio
import hashlib
+from typing import Final
import httpx
import pytest
@@ -20,6 +21,7 @@ from litellm.proxy.auth.password_policy import (
MIN_ALLOWED_LENGTH,
PasswordPolicy,
get_password_policy,
+ is_password_breached,
validate_password_not_breached,
validate_password_policy,
validate_passwords_bulk,
@@ -355,3 +357,10 @@ async def test_validate_passwords_bulk_mixed_verdicts():
async def test_validate_passwords_bulk_empty_batch_makes_no_lookups():
verdicts = await validate_passwords_bulk((), {}, client=_client_never_called())
assert verdicts == {}
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_mode_keeps_the_breach_check() -> None:
+ client_says_breached: Final = _client_returning(f"{_sha1_upper(STRONG_PASSWORD)[5:]}:1387")
+ assert await is_password_breached(STRONG_PASSWORD, {}, client_says_breached) is True
diff --git a/tests/unit/proxy/auth/test_user_api_key_auth_request_flow.py b/tests/unit/proxy/auth/test_user_api_key_auth_request_flow.py
index 741669337cd..8440d3d2ab2 100644
--- a/tests/unit/proxy/auth/test_user_api_key_auth_request_flow.py
+++ b/tests/unit/proxy/auth/test_user_api_key_auth_request_flow.py
@@ -12752,6 +12752,19 @@ def _user_defined_pass_through_endpoint():
return endpoint
+@pytest.mark.asyncio
+async def test_router_settings_model_group_alias_leaves_a_deployed_name_alone_on_provider_pass_through(monkeypatch):
+ router = _alias_router()
+ monkeypatch.setattr(litellm.proxy.proxy_server, "llm_router", router)
+ data = {"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}
+ route = "/anthropic/v1/messages"
+ request = _alias_request(route, data, path_params={"endpoint": "v1/messages"})
+ token = _alias_token(monkeypatch, "key", {"claude-sonnet": "claude-haiku"}, ["claude-haiku", "claude-sonnet"])
+ await _enforce_alias_access(token, data, route, request, router)
+ assert data["model"] == "claude-sonnet"
+ assert (await request.json())["model"] == "claude-sonnet"
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize("user_defined", [False, True])
async def test_router_settings_model_group_alias_leaves_pass_through_bodies_alone(monkeypatch, user_defined):
diff --git a/tests/unit/proxy/common_utils/test_codex_model_catalog.py b/tests/unit/proxy/common_utils/test_codex_model_catalog.py
index 760f79f4ec2..f0b3a8fabc7 100644
--- a/tests/unit/proxy/common_utils/test_codex_model_catalog.py
+++ b/tests/unit/proxy/common_utils/test_codex_model_catalog.py
@@ -1,4 +1,5 @@
import json
+import logging
from collections.abc import Mapping
from dataclasses import replace
from types import MappingProxyType
@@ -7,6 +8,7 @@ from typing import Final, Literal
import pytest
import litellm
+from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
from litellm.litellm_core_utils.llm_cost_calc.utils import get_service_tier_cost_key
from litellm.proxy.common_utils.codex_model_catalog import (
@@ -15,11 +17,14 @@ from litellm.proxy.common_utils.codex_model_catalog import (
CodexStockModel,
CodexStockUpgrade,
bundled_codex_models,
+ codex_catalog_etag,
codex_catalog_rows,
+ codex_entry_names_default_reasoning_level,
codex_model_list_body,
codex_models_response_json,
configured_service_tiers,
deployment_capabilities,
+ merged_codex_catalog,
)
from litellm.proxy.common_utils.codex_model_translation import CodexDeploymentCapabilities, CodexServiceTier
@@ -628,6 +633,15 @@ def test_default_service_tier_survives_only_while_still_offered(configured, expe
assert entry["default_service_tier"] == expected
+
+@pytest.mark.parametrize(("alias_defaults", "expected"), [("codex", "priority"), ("deployment", None)])
+def test_alias_with_offered_tiers_takes_codexs_default_tier_only_under_codex_defaults(alias_defaults, expected):
+ (entry,) = _models(
+ _row("team-sol", upstream_model="gpt-6-sol", service_tiers=["priority"]), alias_defaults=alias_defaults
+ )
+
+ assert (_tiers(entry), entry["default_service_tier"]) == ([("priority", "Fast", "2x speed, increased usage")], expected)
+
def test_non_chat_and_wildcard_rows_are_left_out_of_the_catalog():
body = _body(
_row("gpt-6-astra"),
@@ -705,6 +719,55 @@ def test_an_entry_too_large_for_the_bytes_left_is_passed_over_and_the_smaller_on
assert len(body.json.encode()) <= limit
+def test_merged_catalog_keeps_upstream_whole_and_adds_only_the_entries_that_fit_the_byte_limit():
+ upstream = {"models": [{"slug": "upstream-model", "priority": 4, "base_instructions": "u" * 500}]}
+ upstream_only = len(json.dumps(upstream, separators=(",", ":")))
+ rows = tuple(_row(f"model-{index}") for index in range(4))
+
+ def merged(*merged_rows, byte_limit):
+ return merged_codex_catalog(
+ json.dumps(upstream).encode(), merged_rows, stock=_STOCK, instructions=_PROMPT, byte_limit=byte_limit
+ )
+
+ unlimited = json.loads(merged(*rows, byte_limit=upstream_only * 100))["models"]
+ limit = len(merged(*rows[:2], byte_limit=upstream_only * 100).encode())
+
+ body = merged(*rows, byte_limit=limit)
+
+ assert len(body.encode()) <= limit
+ assert json.loads(body)["models"] == unlimited[:3]
+ assert merged(*rows, byte_limit=upstream_only) is None
+
+
+def test_merged_catalog_gives_an_alias_no_service_tier_or_upgrade_of_codexs_own():
+ upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
+ rows = (_row("team-sol", upstream_model="openai/gpt-6-sol"), _row("team-55", upstream_model="openai/gpt-5.5"))
+
+ added = json.loads(merged_codex_catalog(upstream, rows, stock=_STOCK, instructions=_PROMPT))["models"][1:]
+
+ assert [
+ (entry["slug"], entry["service_tiers"], entry["default_service_tier"], entry["upgrade"]) for entry in added
+ ] == [
+ ("team-sol", [], None, None),
+ ("team-55", [], None, None),
+ ]
+
+
+def test_merged_catalog_warns_once_with_how_many_models_the_byte_limit_left_out(caplog, monkeypatch):
+ upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
+ rows = tuple(_row(f"model-{index}") for index in range(4))
+ limit = len(merged_codex_catalog(upstream, rows[:1], stock=_STOCK, instructions=_PROMPT, byte_limit=10**6).encode())
+ monkeypatch.setattr(verbose_proxy_logger, "propagate", True)
+
+ with caplog.at_level(logging.WARNING, logger=verbose_proxy_logger.name):
+ merged_codex_catalog(upstream, rows, stock=_STOCK, instructions=_PROMPT, byte_limit=limit)
+ merged_codex_catalog(upstream, rows[:1], stock=_STOCK, instructions=_PROMPT, byte_limit=limit)
+
+ assert [record.getMessage() for record in caplog.records] == [
+ f"Codex native model catalog left out 3 of 4 LiteLLM models to stay within Codex's {limit} byte limit"
+ ]
+
+
def test_unlimited_body_keeps_every_entry():
rows = tuple(_row(f"model-{index}") for index in range(3))
@@ -947,6 +1010,38 @@ def test_catalog_rows_without_a_router_carry_only_the_listing():
assert codex_catalog_rows(listing, (("plain", "plain"),), None) == (CodexCatalogRow(id="plain", mode="chat"),)
+@pytest.mark.parametrize("client_version", [None, "0.165.0"])
+def test_entry_names_a_default_reasoning_level_exactly_when_the_native_entry_carries_one(client_version):
+ router = litellm.Router(
+ model_list=[
+ {"model_name": "open-weights", "litellm_params": {"model": "fireworks_ai/open-weights"}},
+ {"model_name": "coding-model", "litellm_params": {"model": "openai/gpt-5.5"}},
+ {"model_name": "pinned-model", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "high"}},
+ {"model_name": "split-pin", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "high"}},
+ {"model_name": "split-pin", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "low"}},
+ {
+ "model_name": "max-only",
+ "litellm_params": {"model": "hosted_vllm/qa-local-coder", "api_base": "https://vllm.example/v1"},
+ "model_info": {"supports_reasoning": True, "reasoning_effort_levels": ["max"]},
+ },
+ ]
+ )
+ names = ("open-weights", "coding-model", "pinned-model", "split-pin", "max-only")
+ listing = tuple(
+ {"id": name, "object": "model", "created": 0, "owned_by": "openai", "mode": "chat"} for name in names
+ )
+ upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
+ rows = codex_catalog_rows(listing, tuple((name, name) for name in names), router)
+ served = json.loads(merged_codex_catalog(upstream, rows, client_version=client_version))["models"][1:]
+ served_names_one = [entry.get("default_reasoning_level") is not None for entry in served]
+
+ assert [
+ codex_entry_names_default_reasoning_level(entry["slug"], router, None, client_version) for entry in served
+ ] == served_names_one
+ assert served_names_one == [False, False, True, False, client_version is not None]
+ assert served[2]["default_reasoning_level"] == "high"
+
+
def test_codex_tool_search_is_offered_off_the_route_and_model_info_and_only_when_every_deployment_answers_it():
claude = "anthropic/claude-sonnet-5-5"
converse = "bedrock/us.anthropic.claude-sonnet-5-5"
@@ -995,6 +1090,34 @@ def test_codex_tool_search_is_offered_off_the_route_and_model_info_and_only_when
assert models[0]["context_window"] == 1000000
+def test_native_merged_catalog_offers_codex_tool_search_as_the_proxy_listing_does():
+ claude = "anthropic/claude-sonnet-5-5"
+ router = litellm.Router(
+ model_list=[
+ {"model_name": "claude", "litellm_params": {"model": claude}},
+ {
+ "model_name": "claude-off",
+ "litellm_params": {"model": claude},
+ "model_info": {"supports_search_tool": False},
+ },
+ {"model_name": "coding-model", "litellm_params": {"model": "openai/gpt-5.5"}},
+ ]
+ )
+ names = ("claude", "claude-off", "coding-model")
+ listing = tuple({"id": name, "object": "model", "created": 0, "owned_by": "x", "mode": "chat"} for name in names)
+ entries = tuple((name, name) for name in names)
+ upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
+
+ listed = json.loads(codex_model_list_body(listing, entries, router))["models"]
+ merged = json.loads(merged_codex_catalog(upstream, codex_catalog_rows(listing, entries, router)))["models"][1:]
+
+ assert [entry["slug"] for entry in merged] == list(names)
+ assert [entry.get("supports_search_tool") for entry in merged] == [
+ entry.get("supports_search_tool") for entry in listed
+ ]
+ assert [entry["supports_search_tool"] for entry in merged[:2]] == [True, False]
+
+
_SOL_SLUG: Final = next(
slug for slug, model in bundled_codex_models().items() if model.default_service_tier is not None
)
@@ -1282,3 +1405,20 @@ def test_model_info_that_does_not_validate_declares_no_capability_but_keeps_its_
)
assert capabilities == CodexDeploymentCapabilities(priced_service_tier_keys=frozenset({priced}))
+
+
+class _CollidingName(str):
+ def __hash__(self) -> int:
+ return 1
+
+
+def test_catalog_etag_ignores_the_order_a_process_iterates_priced_service_tiers_in() -> None:
+ def row_with_tiers(*tiers: str) -> CodexCatalogRow:
+ capabilities: Final = CodexDeploymentCapabilities.model_construct(
+ priced_service_tier_keys=frozenset(_CollidingName(tier) for tier in tiers)
+ )
+ return replace(_row("gpt-6-astra"), deployment_capabilities=(capabilities,))
+
+ assert codex_catalog_etag("upstream", [row_with_tiers("flex", "priority")]) == codex_catalog_etag(
+ "upstream", [row_with_tiers("priority", "flex")]
+ )
diff --git a/tests/unit/proxy/common_utils/test_fips.py b/tests/unit/proxy/common_utils/test_fips.py
index 0884d0bbbed..d22684ea051 100644
--- a/tests/unit/proxy/common_utils/test_fips.py
+++ b/tests/unit/proxy/common_utils/test_fips.py
@@ -1,5 +1,5 @@
import _hashlib
-from collections.abc import Mapping
+from collections.abc import Iterator, Mapping
from contextlib import nullcontext
from unittest.mock import Mock
@@ -12,10 +12,17 @@ from litellm.proxy.common_utils.fips import (
FipsModeError,
cryptography_enforces_fips,
enforce_fips_mode,
+ fips_mode_active,
openssl_enforces_fips,
)
+@pytest.fixture(autouse=True)
+def _fips_mode_off_after_each_test() -> Iterator[None]:
+ yield
+ enforce_fips_mode({})
+
+
def _fake_openssl(monkeypatch: pytest.MonkeyPatch, *, fips_flag: int, scrypt_refused: bool) -> None:
monkeypatch.setattr(_hashlib, "get_fips_mode", lambda: fips_flag)
if scrypt_refused:
@@ -119,3 +126,29 @@ def test_enforce_fips_mode(
with pytest.raises(FipsModeError, match=refusal) if refusal else nullcontext():
enforce_fips_mode(general_settings)
+
+
+@pytest.mark.parametrize(
+ "general_settings, active",
+ [({"fips_mode": True}, True), ({"fips_mode": "true"}, True), ({"fips_mode": False}, False), ({}, False)],
+ ids=["on", "on-from-a-string", "off", "unset"],
+)
+def test_fips_mode_active_follows_the_setting_the_gate_accepted(
+ monkeypatch: pytest.MonkeyPatch, general_settings: Mapping[str, object], active: bool
+) -> None:
+ _fake_openssl(monkeypatch, fips_flag=1, scrypt_refused=True)
+ monkeypatch.setattr(rust_openssl, "is_fips_enabled", lambda: True)
+ enforce_fips_mode(general_settings)
+ assert fips_mode_active() is active
+
+
+def test_fips_mode_stays_inactive_on_a_refused_boot(monkeypatch: pytest.MonkeyPatch) -> None:
+ _fake_openssl(monkeypatch, fips_flag=0, scrypt_refused=False)
+ with pytest.raises(FipsModeError):
+ enforce_fips_mode({"fips_mode": True})
+ assert fips_mode_active() is False
+
+
+def test_a_fips_openssl_alone_does_not_activate_fips_mode(monkeypatch: pytest.MonkeyPatch) -> None:
+ _fake_openssl(monkeypatch, fips_flag=1, scrypt_refused=True)
+ assert fips_mode_active() is False
diff --git a/tests/unit/proxy/conftest.py b/tests/unit/proxy/conftest.py
index 298d32def7d..d08a98242ef 100644
--- a/tests/unit/proxy/conftest.py
+++ b/tests/unit/proxy/conftest.py
@@ -1,7 +1,9 @@
# conftest.py
+import _hashlib
import asyncio
import copy
+import hashlib
import inspect
import logging
import os
@@ -21,6 +23,7 @@ from prisma.errors import ClientNotConnectedError
import litellm
import litellm.proxy.proxy_server
from litellm._logging import verbose_proxy_logger
+from litellm.proxy.common_utils.fips import enforce_fips_mode
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
InitPassThroughEndpointHelpers,
_registered_pass_through_routes,
@@ -514,3 +517,31 @@ def postgres_span_names() -> Iterator[Callable[[], Awaitable[tuple[str, ...]]]]:
with patch("litellm.proxy.proxy_server.proxy_logging_obj", MagicMock(service_logging_obj=service_logging)):
yield rendered
+
+
+@pytest.fixture
+def fips_openssl(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(_hashlib, "get_fips_mode", lambda: 1)
+ monkeypatch.setattr(_hashlib, "scrypt", MagicMock(side_effect=ValueError("[digital envelope routines] unsupported")))
+
+
+@pytest.fixture
+def fips_mode(fips_openssl: None, monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
+ monkeypatch.setattr("litellm.proxy.common_utils.fips.cryptography_enforces_fips", lambda: True)
+ enforce_fips_mode({"fips_mode": True})
+ yield
+ enforce_fips_mode({})
+
+
+@pytest.fixture
+def pbkdf2_on_event_loop(monkeypatch: pytest.MonkeyPatch) -> list[bool]:
+ """One entry per PBKDF2 call: whether it ran on the event loop's thread."""
+ real_pbkdf2: Final = hashlib.pbkdf2_hmac
+ on_event_loop: Final[list[bool]] = []
+
+ def recording_pbkdf2(hash_name: str, password: bytes, salt: bytes, iterations: int, dklen: int) -> bytes:
+ on_event_loop.append(asyncio._get_running_loop() is not None)
+ return real_pbkdf2(hash_name, password, salt, iterations, dklen)
+
+ monkeypatch.setattr(hashlib, "pbkdf2_hmac", recording_pbkdf2)
+ return on_event_loop
diff --git a/tests/unit/proxy/db/test_spend_log_tool_index.py b/tests/unit/proxy/db/test_spend_log_tool_index.py
index af3268a7cc9..901a9ea8b69 100644
--- a/tests/unit/proxy/db/test_spend_log_tool_index.py
+++ b/tests/unit/proxy/db/test_spend_log_tool_index.py
@@ -6,7 +6,7 @@ plus the LiteLLM_DailyToolSpend rollup in one transaction.
from collections.abc import Awaitable, Callable
from types import SimpleNamespace
-from typing import Any
+from typing import Any, Final
from unittest.mock import AsyncMock, MagicMock
import httpx
@@ -23,6 +23,7 @@ from litellm.proxy.db.spend_log_tool_index import (
)
from litellm.responses.mcp.litellm_proxy_mcp_handler import LiteLLM_Proxy_MCP_Handler
from litellm.types.llms.openai import ResponsesAPIResponse
+from litellm.types.utils import ModelResponse
from tests.unit.proxy.db.fake_prisma_engine import engine_call
@@ -204,6 +205,36 @@ class TestResponseToolCallNames:
def test_blank_names_are_dropped(self):
assert response_tool_call_names(_response_with_tool_calls(" ", "real_tool")) == ("real_tool",)
+ @pytest.mark.parametrize("as_object", [False, True], ids=["dict", "model_response"])
+ def test_server_tool_calls_of_every_choice_are_named_before_client_tool_calls(self, as_object: bool):
+ def tool_call(call_id: str, name: str) -> dict[str, object]:
+ return {"id": call_id, "type": "function", "function": {"name": name, "arguments": "{}"}}
+
+ response: dict[str, object] = {
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [tool_call("call_1", "get_weather")],
+ "provider_specific_fields": {"server_tool_calls": [tool_call("srvtoolu_01", "web_search")]},
+ },
+ },
+ {
+ "index": 1,
+ "message": {
+ "role": "assistant",
+ "content": "done",
+ "provider_specific_fields": {"server_tool_calls": [tool_call("srvtoolu_02", "web_fetch")]},
+ },
+ },
+ ]
+ }
+ completion_response: Final = ModelResponse(**response) if as_object else response
+
+ assert response_tool_call_names(completion_response) == ("web_search", "web_fetch", "get_weather")
+
def test_responses_api_output_function_calls(self):
# Regression: /v1/responses carries invocations in output[] items of
# type function_call, not in choices; they must reach the rollup.
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_cato_networks.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_cato_networks.py
index 349030c6c75..a8f68f038c0 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_cato_networks.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_cato_networks.py
@@ -1,6 +1,7 @@
import asyncio
import json
import ssl
+from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@@ -2237,6 +2238,83 @@ async def test_post_call_success_hook_redacts_both_content_and_tool_arguments():
assert message.tool_calls[0].function.arguments == '{"to": "[NAME_1]"}'
+@pytest.mark.asyncio
+async def test_post_call_success_hook_inspects_and_redacts_server_tool_call_arguments():
+ guard: Final = _make_guardrail()
+ request_data: Final = {"messages": [{"role": "user", "content": "look up persimmon"}]}
+
+ def side_effect(url: str, *args: object, **kwargs: dict[str, list[dict[str, str]]]) -> Response:
+ last: Final = kwargs["json"]["messages"][-1]["content"]
+ return _make_response(
+ {
+ "analysis_result": {"policy_drill_down": {"PII": {}}},
+ "required_action": {"action_type": "anonymize_action", "policy_name": "PII"},
+ "redacted_chat": {
+ "all_redacted_messages": [
+ {"role": "user", "content": "look up persimmon"},
+ {"role": "assistant", "content": last.replace("persimmon", "[FRUIT_1]")},
+ ]
+ },
+ }
+ )
+
+ llm_response: Final = ModelResponse(
+ choices=[
+ {
+ "finish_reason": "tool_calls",
+ "index": 0,
+ "message": {
+ "content": None,
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'},
+ }
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ },
+ }
+ ]
+ )
+ with patch(
+ "litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
+ side_effect=side_effect,
+ ) as mock_post:
+ result: Final = await guard.async_post_call_success_hook(
+ data=request_data,
+ response=llm_response,
+ user_api_key_dict=UserAPIKeyAuth(),
+ )
+
+ inspected: Final = [call.kwargs["json"]["messages"][-1] for call in mock_post.call_args_list]
+ assert inspected == [
+ {"role": "assistant", "content": '{"query": "persimmon"}'},
+ {"role": "assistant", "content": '{"city": "Paris"}'},
+ ]
+ message: Final = result.choices[0].message
+ assert message.provider_specific_fields == {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "[FRUIT_1]"}'},
+ }
+ ]
+ }
+ assert message.content is None
+ assert message.tool_calls[0].function.arguments == '{"city": "Paris"}'
+
+
def _make_responses_api_response(output: list) -> ResponsesAPIResponse:
return ResponsesAPIResponse(id="resp-1", created_at=0, output=output)
@@ -2332,6 +2410,53 @@ async def test_post_call_success_hook_redacts_responses_api_function_call_argume
assert result.output[0].arguments == '{"recipient": "[NAME_1]"}'
+@pytest.mark.asyncio
+async def test_post_call_success_hook_redacts_claude_hosted_web_fetch_url():
+ from openai.types.responses import ResponseFunctionWebSearch
+ from openai.types.responses.response_function_web_search import ActionOpenPage
+
+ guard: Final = _make_guardrail()
+ request_data: Final = {"messages": [{"role": "user", "content": "read the page"}]}
+ anonymize_response: Final = _make_response(
+ {
+ "analysis_result": {"policy_drill_down": {"PII": {}}},
+ "required_action": {"action_type": "anonymize_action", "policy_name": "PII"},
+ "redacted_chat": {
+ "all_redacted_messages": [
+ {"role": "user", "content": "read the page"},
+ {"role": "assistant", "content": '{"url": "https://example.com/?ssn=[SSN_1]"}'},
+ ]
+ },
+ }
+ )
+ response: Final = _make_responses_api_response(
+ [
+ ResponseFunctionWebSearch(
+ id="ws_srvtoolu_01",
+ type="web_search_call",
+ status="completed",
+ action=ActionOpenPage(type="open_page", url="https://example.com/?ssn=123-45-6789"),
+ )
+ ]
+ )
+ with patch(
+ "litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
+ new_callable=AsyncMock,
+ ) as mock_post:
+ mock_post.return_value = anonymize_response
+ result: Final = await guard.async_post_call_success_hook(
+ data=request_data,
+ response=response,
+ user_api_key_dict=UserAPIKeyAuth(),
+ )
+ posted: Final = mock_post.call_args.kwargs["json"]["messages"]
+ assert posted[-1] == {"role": "assistant", "content": '{"url": "https://example.com/?ssn=123-45-6789"}'}
+ assert (result.output[0].id, result.output[0].action.url) == (
+ "ws_srvtoolu_01",
+ "https://example.com/?ssn=[SSN_1]",
+ )
+
+
@pytest.mark.asyncio
async def test_post_call_success_hook_blocks_responses_api_output():
"""A ``block_action`` on Responses API output must raise so the blocked text never
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_cisco_ai_defense_chat.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_cisco_ai_defense_chat.py
index 5ce78fd05d1..4cdabe5c360 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_cisco_ai_defense_chat.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_cisco_ai_defense_chat.py
@@ -40,6 +40,14 @@ from tests.unit.proxy.guardrails.guardrail_hooks._cisco_ai_defense_test_utils im
patch,
pytest,
)
+from typing import Final
+
+from openai.types.responses import ResponseFunctionWebSearch
+from openai.types.responses.response_function_web_search import (
+ Action,
+ ActionOpenPage,
+ ActionSearch,
+)
def test_cisco_ai_defense_config_via_init_v2_chat(monkeypatch):
@@ -404,6 +412,31 @@ class TestCiscoAIDefenseChatMode:
)
+_CLAUDE_HOSTED_WEB_ACTIONS: Final = (
+ ActionSearch(type="search", query="ssn 123-45-6789"),
+ ActionOpenPage(type="open_page", url="https://example.com/?ssn=123-45-6789"),
+)
+
+
+def _claude_hosted_web_call_response(action: Action):
+ from litellm.types.llms.openai import ResponsesAPIResponse
+
+ return ResponsesAPIResponse(
+ id="resp_1",
+ created_at=0,
+ output=[
+ ResponseFunctionWebSearch(
+ id="ws_srvtoolu_01", type="web_search_call", status="completed", action=action
+ )
+ ],
+ parallel_tool_calls=False,
+ tool_choice=None,
+ tools=None,
+ top_p=None,
+ usage=None,
+ )
+
+
class TestCiscoAIDefenseResponsesAPIOutput:
@staticmethod
@@ -505,6 +538,23 @@ class TestCiscoAIDefenseResponsesAPIOutput:
joined = " ".join(m.get("content", "") for m in (sent.get("messages") or []))
assert "4111-1111-1111-1111" in joined
+ @pytest.mark.parametrize("action", _CLAUDE_HOSTED_WEB_ACTIONS)
+ @pytest.mark.asyncio
+ async def test_post_call_scans_claude_hosted_web_call(self, action: Action) -> None:
+ g: Final = _make_guardrail(event_hook="post_call")
+ post_mock: Final = AsyncMock(return_value=_safe_response())
+
+ with _patch_inspection_post(g, post_mock):
+ await g.async_post_call_success_hook(
+ data={"input": [{"role": "user", "content": "look it up"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=_claude_hosted_web_call_response(action),
+ )
+
+ sent: Final = post_mock.call_args.kwargs["json"]
+ joined: Final = " ".join(m.get("content", "") for m in (sent.get("messages") or []))
+ assert "123-45-6789" in joined
+
@pytest.mark.asyncio
async def test_post_call_responses_api_violation_is_blocked(self):
g = _make_guardrail(event_hook="post_call")
@@ -573,6 +623,24 @@ class TestCiscoAIDefenseResponsesAPIOutputRedaction:
)
+ @pytest.mark.parametrize("action", _CLAUDE_HOSTED_WEB_ACTIONS)
+ @pytest.mark.asyncio
+ async def test_redact_clears_claude_hosted_web_call(self, action: Action) -> None:
+ g: Final = _make_guardrail(event_hook="post_call", on_flagged_action="monitor")
+ cisco_resp: Final = _redact_response(sanitized_text="[REDACTED]")
+
+ with _patch_inspection_post(g, AsyncMock(return_value=cisco_resp)):
+ result: Final = await g.async_post_call_success_hook(
+ data={"input": [{"role": "user", "content": "look it up"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=_claude_hosted_web_call_response(action),
+ )
+
+ call: Final = result.output[0]
+ assert (call.id, call.action.type) == ("ws_srvtoolu_01", action.type)
+ assert "123-45-6789" not in call.action.model_dump_json()
+
+
class TestCiscoAIDefenseResponsesAPIInputRedaction:
@pytest.mark.parametrize(
@@ -1660,6 +1728,38 @@ class TestCiscoAIDefenseToolCallBypass:
f"included in the scanned text. Sent: {sent!r}"
)
+ @pytest.mark.asyncio
+ async def test_pre_call_scans_function_tool_call_that_also_carries_a_custom_key(self):
+ g: Final = _make_guardrail(event_hook="pre_call")
+ post_mock: Final = AsyncMock(return_value=_safe_response())
+ data: Final = {
+ "messages": [
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "custom": {},
+ "function": {"name": "send_data", "arguments": '{"data":"SSN 123-45-6789"}'},
+ }
+ ],
+ }
+ ]
+ }
+
+ with _patch_inspection_post(g, post_mock):
+ await g.async_pre_call_hook(
+ user_api_key_dict=UserAPIKeyAuth(),
+ cache=DualCache(),
+ data=data,
+ call_type="completion",
+ )
+
+ sent: Final = post_mock.call_args.kwargs["json"]
+ assert "123-45-6789" in " ".join(m.get("content", "") for m in (sent.get("messages") or []))
+
@pytest.mark.parametrize(
"data",
[
@@ -1777,6 +1877,78 @@ class TestCiscoAIDefenseToolCallBypass:
f"included in the scanned text. Sent: {sent!r}"
)
+ @staticmethod
+ def _server_tool_call_response() -> ModelResponse:
+ from litellm.types.utils import ChatCompletionMessageToolCall, Function
+
+ return ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ ChatCompletionMessageToolCall(
+ id="call_1",
+ type="function",
+ function=Function(name="get_weather", arguments='{"city": "Paris"}'),
+ )
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ]
+ )
+
+ @pytest.mark.asyncio
+ async def test_post_call_scan_sends_server_tool_call_arguments_to_cisco(self):
+ g: Final = _make_guardrail(event_hook="post_call")
+ post_mock: Final = AsyncMock(return_value=_safe_response())
+
+ with _patch_inspection_post(g, post_mock):
+ await g.async_post_call_success_hook(
+ data={"messages": [{"role": "user", "content": "look up persimmon"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=self._server_tool_call_response(),
+ )
+
+ assert [call.kwargs["json"]["messages"] for call in post_mock.call_args_list] == [
+ [
+ {"role": "user", "content": "look up persimmon"},
+ {"role": "assistant", "content": '{"query": "persimmon"} {"city": "Paris"}'},
+ ]
+ ]
+
+ @pytest.mark.asyncio
+ async def test_post_call_redact_clears_server_tool_call_arguments(self):
+ g: Final = _make_guardrail(event_hook="post_call", on_flagged_action="monitor")
+ cisco_resp: Final = _redact_response(sanitized_text="[REDACTED]")
+
+ with _patch_inspection_post(g, AsyncMock(return_value=cisco_resp)):
+ result: Final = await g.async_post_call_success_hook(
+ data={"messages": [{"role": "user", "content": "look up persimmon"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=self._server_tool_call_response(),
+ )
+
+ message: Final = result.choices[0].message
+ assert message.provider_specific_fields == {
+ "server_tool_calls": [
+ {"id": "srvtoolu_01", "type": "function", "function": {"name": "web_search", "arguments": "{}"}}
+ ]
+ }
+ assert message.tool_calls[0].function.arguments == "{}"
+
class TestCiscoAIDefenseToolDefinitionBypass:
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_lakera_ai.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_lakera_ai.py
new file mode 100644
index 00000000000..cb7fb7898ab
--- /dev/null
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_lakera_ai.py
@@ -0,0 +1,98 @@
+import json
+from typing import Final
+
+import httpx
+import pytest
+import respx
+
+import litellm
+from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.guardrails.guardrail_hooks.lakera_ai import lakeraAI_Moderation
+
+
+@pytest.mark.asyncio
+async def test_prompt_check_appends_replayed_server_tool_call_arguments_to_system_message(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ monkeypatch.setattr(litellm, "add_function_to_prompt", False)
+ route: Final = respx_mock.post("https://api.lakera.ai/v1/prompt_injection").mock(
+ return_value=httpx.Response(200, json={"results": [{"flagged": False}]})
+ )
+ moderation: Final = lakeraAI_Moderation(api_key="test-key", moderation_check="pre_call")
+
+ await moderation._check(
+ data={
+ "messages": [
+ {"role": "system", "content": "You are a shopping assistant."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_01",
+ "type": "function",
+ "function": {"name": "lookup_order", "arguments": '{"order_id": "A-17"}'},
+ }
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://example.com"}'},
+ }
+ ]
+ },
+ },
+ ]
+ },
+ user_api_key_dict=UserAPIKeyAuth(),
+ call_type="completion",
+ )
+
+ assert json.loads(route.calls.last.request.content)["input"][0] == {
+ "role": "system",
+ "content": 'You are a shopping assistant. Function Input: {"url": "https://example.com"} {"order_id": "A-17"}',
+ }
+
+
+@pytest.mark.asyncio
+async def test_prompt_check_scans_a_replayed_turn_that_only_has_server_tool_calls(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ monkeypatch.setattr(litellm, "add_function_to_prompt", False)
+ route: Final = respx_mock.post("https://api.lakera.ai/v1/prompt_injection").mock(
+ return_value=httpx.Response(200, json={"results": [{"flagged": False}]})
+ )
+ moderation: Final = lakeraAI_Moderation(api_key="test-key", moderation_check="pre_call")
+
+ await moderation._check(
+ data={
+ "messages": [
+ {"role": "system", "content": "You are a shopping assistant."},
+ {
+ "role": "assistant",
+ "content": "Searching.",
+ "tool_calls": None,
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "order A-17"}'},
+ }
+ ]
+ },
+ },
+ ]
+ },
+ user_api_key_dict=UserAPIKeyAuth(),
+ call_type="completion",
+ )
+
+ assert json.loads(route.calls.last.request.content)["input"][0] == {
+ "role": "system",
+ "content": 'You are a shopping assistant. Function Input: {"query": "order A-17"}',
+ }
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_lasso.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_lasso.py
index b8a219ba863..634e511fb91 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_lasso.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_lasso.py
@@ -1,3 +1,4 @@
+import json
import os
import uuid
from typing import Final
@@ -1549,3 +1550,254 @@ async def test_api_error_handling_legacy_messages(
assert "Failed to verify request safety with Lasso API" in str(excinfo.value)
assert error_message in str(excinfo.value)
assert route.called
+
+
+def _reply_with_web_search_call(
+ arguments: str, tool_calls: list[dict[str, object]] | None = None
+) -> litellm.ModelResponse:
+ return litellm.ModelResponse(
+ choices=[
+ litellm.Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=litellm.Message(
+ role="assistant",
+ content=None,
+ tool_calls=tool_calls,
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": arguments},
+ }
+ ]
+ },
+ ),
+ )
+ ]
+ )
+
+
+@pytest.mark.asyncio
+async def test_post_call_scan_sends_server_tool_calls_as_tool_use_entries(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ lasso_guardrail: Final = LassoGuardrail(
+ lasso_api_key="test-key",
+ user_id="test-user",
+ conversation_id="test-conversation",
+ guardrail_name="test-guard",
+ event_hook="post_call",
+ default_on=True,
+ )
+ route: Final = respx_mock.post("https://server.lasso.security/gateway/v3/classify").mock(
+ return_value=httpx.Response(200, json={"violations_detected": False, "deputies": {}, "findings": {}})
+ )
+ response: Final = _reply_with_web_search_call(
+ '{"query": "persimmon"}',
+ tool_calls=[
+ {
+ "id": "call_01",
+ "type": "function",
+ "function": {"name": "lookup_order", "arguments": '{"order_id": "A-17"}'},
+ }
+ ],
+ )
+
+ await lasso_guardrail.async_post_call_success_hook(
+ data={"messages": [{"role": "user", "content": "search persimmons"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=response,
+ )
+
+ assert json.loads(route.calls.last.request.content) == {
+ "messages": [
+ {
+ "role": "model",
+ "content": {
+ "type": "tool_use",
+ "id": "srvtoolu_01",
+ "name": "web_search",
+ "input": {"query": "persimmon"},
+ },
+ },
+ {
+ "role": "model",
+ "content": {
+ "type": "tool_use",
+ "id": "call_01",
+ "name": "lookup_order",
+ "input": {"order_id": "A-17"},
+ },
+ },
+ ],
+ "messageType": "COMPLETION",
+ "source": {"type": "litellm"},
+ "userId": "test-user",
+ "sessionId": "test-conversation",
+ }
+
+
+@pytest.mark.asyncio
+async def test_post_call_masking_rewrites_server_tool_call_arguments(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ lasso_guardrail: Final = LassoGuardrail(
+ lasso_api_key="test-key",
+ user_id="test-user",
+ conversation_id="test-conversation",
+ mask=True,
+ guardrail_name="test-guard",
+ event_hook="post_call",
+ default_on=True,
+ )
+ respx_mock.post("https://server.lasso.security/gateway/v3/classifix").mock(
+ return_value=httpx.Response(
+ 200,
+ json={
+ "violations_detected": True,
+ "deputies": {"pattern-detection": True},
+ "findings": {"pattern-detection": [{"action": "AUTO_MASKING", "severity": "HIGH"}]},
+ "messages": [
+ {
+ "role": "model",
+ "content": {
+ "type": "tool_use",
+ "id": "srvtoolu_01",
+ "name": "web_search",
+ "input": {"query": " persimmon"},
+ },
+ }
+ ],
+ },
+ )
+ )
+
+ result: Final = await lasso_guardrail.async_post_call_success_hook(
+ data={"messages": [{"role": "user", "content": "search persimmons"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=_reply_with_web_search_call('{"query": "jane@example.com persimmon"}'),
+ )
+
+ assert result.choices[0].message.provider_specific_fields == {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": " persimmon"}'},
+ }
+ ]
+ }
+
+
+def _replayed_web_fetch_request(url: str) -> dict[str, object]:
+ return {
+ "messages": [
+ {"role": "user", "content": "open the order page"},
+ {
+ "role": "assistant",
+ "content": None,
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": json.dumps({"url": url})},
+ }
+ ]
+ },
+ },
+ {"role": "user", "content": "summarize it"},
+ ]
+ }
+
+
+@pytest.mark.asyncio
+async def test_pre_call_scan_sends_replayed_server_tool_calls_as_tool_use_entries(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ lasso_guardrail: Final = LassoGuardrail(
+ lasso_api_key="test-key",
+ user_id="test-user",
+ conversation_id="test-conversation",
+ guardrail_name="test-guard",
+ event_hook="pre_call",
+ default_on=True,
+ )
+ route: Final = respx_mock.post("https://server.lasso.security/gateway/v3/classify").mock(
+ return_value=httpx.Response(200, json={"violations_detected": False, "deputies": {}, "findings": {}})
+ )
+
+ await lasso_guardrail.async_pre_call_hook(
+ user_api_key_dict=UserAPIKeyAuth(),
+ cache=DualCache(),
+ data=_replayed_web_fetch_request("https://example.com/?email=jane@example.com"),
+ call_type="completion",
+ )
+
+ assert json.loads(route.calls.last.request.content)["messages"] == [
+ {"role": "user", "content": "open the order page"},
+ {
+ "role": "model",
+ "content": {
+ "type": "tool_use",
+ "id": "srvtoolu_01",
+ "name": "web_fetch",
+ "input": {"url": "https://example.com/?email=jane@example.com"},
+ },
+ },
+ {"role": "user", "content": "summarize it"},
+ ]
+
+
+@pytest.mark.asyncio
+async def test_pre_call_masking_rewrites_replayed_server_tool_call_arguments(
+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ lasso_guardrail: Final = LassoGuardrail(
+ lasso_api_key="test-key",
+ user_id="test-user",
+ conversation_id="test-conversation",
+ mask=True,
+ guardrail_name="test-guard",
+ event_hook="pre_call",
+ default_on=True,
+ )
+ respx_mock.post("https://server.lasso.security/gateway/v3/classifix").mock(
+ return_value=httpx.Response(
+ 200,
+ json={
+ "violations_detected": True,
+ "deputies": {"pattern-detection": True},
+ "findings": {"pattern-detection": [{"action": "AUTO_MASKING", "severity": "HIGH"}]},
+ "messages": [
+ {"role": "user", "content": "open the order page"},
+ {
+ "role": "model",
+ "content": {
+ "type": "tool_use",
+ "id": "srvtoolu_01",
+ "name": "web_fetch",
+ "input": {"url": "https://example.com/?email="},
+ },
+ },
+ {"role": "user", "content": "summarize it"},
+ ],
+ },
+ )
+ )
+
+ result: Final = await lasso_guardrail.async_pre_call_hook(
+ user_api_key_dict=UserAPIKeyAuth(),
+ cache=DualCache(),
+ data=_replayed_web_fetch_request("https://example.com/?email=jane@example.com"),
+ call_type="completion",
+ )
+
+ assert isinstance(result, dict)
+ assert result["messages"] == _replayed_web_fetch_request("https://example.com/?email=")["messages"]
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_llm_shield_proxy.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_llm_shield_proxy.py
index 2603cf92f30..8a1d7f0b4e9 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_llm_shield_proxy.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_llm_shield_proxy.py
@@ -893,6 +893,55 @@ class TestRestoration:
assert result.choices[0].message.content == "a@b.com"
+ @pytest.mark.asyncio
+ async def test_server_tool_call_arguments_are_restored(self):
+ guardrail, _ = _shielded({"[EMAIL_1]": "a@b.com", "[URL_1]": "https://b.com/a"})
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "[EMAIL_1] persimmon"}'},
+ },
+ {
+ "id": "srvtoolu_02",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "[URL_1]"}'},
+ },
+ ]
+ },
+ ),
+ )
+ ]
+ )
+
+ result: Final = await guardrail.async_post_call_success_hook(
+ data={"messages": []}, user_api_key_dict=None, response=response
+ )
+
+ assert result.choices[0].message.provider_specific_fields == {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "a@b.com persimmon"}'},
+ },
+ {
+ "id": "srvtoolu_02",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://b.com/a"}'},
+ },
+ ]
+ }
+
@pytest.mark.asyncio
async def test_responses_api_shape_is_restored(self):
"""The Responses API reply carries output items, not choices.
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_microsoft_purview.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_microsoft_purview.py
index 25d2556bf26..34221720205 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_microsoft_purview.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_microsoft_purview.py
@@ -2072,6 +2072,63 @@ class TestCompletionResponseTextPartsToolCalls:
sent_text = mock_check.call_args.kwargs["text"]
assert '{"credit_card": "4111-1111-1111-1111"}' in sent_text
+ @pytest.mark.asyncio
+ async def test_post_call_hook_sends_server_tool_call_args_to_purview(self):
+ from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
+ from litellm.types.utils import Choices, Message, ModelResponse
+
+ def graph(request: httpx.Request) -> httpx.Response:
+ if request.url.host == "login.microsoftonline.com":
+ return httpx.Response(200, json={"access_token": "graph-token", "expires_in": 3600})
+ if not request.url.path.endswith("/processContent"):
+ return httpx.Response(200, json={"value": []}, headers={"ETag": "scope-etag"})
+ return httpx.Response(200, json={"protectionScopeState": "unchanged", "policyActions": []})
+
+ transport: Final = Mock(side_effect=graph)
+ guardrail: Final = _make_guardrail()
+ guardrail.async_handler = AsyncHTTPHandler(transport=httpx.MockTransport(transport))
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'},
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ],
+ )
+
+ await guardrail.async_post_call_success_hook(
+ data={},
+ user_api_key_dict=UserAPIKeyAuth(api_key="test", user_id="user-123"),
+ response=response,
+ )
+
+ scanned: Final = [
+ json.loads(call.args[0].content)["contentToProcess"]["contentEntries"][0]["content"]["data"]
+ for call in transport.call_args_list
+ if call.args[0].url.path.endswith("/processContent")
+ ]
+ assert scanned == ['{"query": "persimmon"}\n\n---\n\n{"city": "Paris"}']
+
def test_responses_api_function_call_args_included(self):
"""Function-call arguments in ``ResponsesAPIResponse.output`` must be DLP-scanned."""
from litellm.types.llms.openai import ResponsesAPIResponse
@@ -2123,6 +2180,29 @@ class TestCompletionResponseTextPartsToolCalls:
parts = guardrail._completion_response_text_parts(response)
assert any('{"secret": "hunter2"}' in p for p in parts)
+ def test_responses_api_claude_hosted_web_fetch_url_included(self):
+ from openai.types.responses import ResponseFunctionWebSearch
+ from openai.types.responses.response_function_web_search import ActionOpenPage
+
+ from litellm.types.llms.openai import ResponsesAPIResponse
+
+ guardrail: Final = _make_guardrail()
+ response: Final = ResponsesAPIResponse(
+ id="resp-ws-1",
+ created_at=0,
+ output=[
+ ResponseFunctionWebSearch(
+ id="ws_srvtoolu_01",
+ type="web_search_call",
+ status="completed",
+ action=ActionOpenPage(type="open_page", url="https://example.com/?ssn=123-45-6789"),
+ )
+ ],
+ )
+ assert guardrail._completion_response_text_parts(response) == [
+ '{"url": "https://example.com/?ssn=123-45-6789"}'
+ ]
+
# ---------------------------------------------------------------
# Graph user id path encoding
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_pangea.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_pangea.py
index 4e31a3c2055..145e03c718b 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_pangea.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_pangea.py
@@ -1,5 +1,9 @@
+from typing import Final
from unittest.mock import AsyncMock, patch
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from typing_extensions import ReadOnly, TypedDict
+
import httpx
import pytest
from fastapi import HTTPException
@@ -11,6 +15,8 @@ from litellm.proxy.guardrails.guardrail_hooks.pangea.pangea import (
)
from litellm.proxy.guardrails.init_guardrails import init_guardrails_v2
from litellm.types.guardrails import GuardrailEventHooks, LitellmParams
+from litellm.types.llms.openai import ChatCompletionAssistantToolCall
+from litellm.types.responses.main import build_web_search_call
from litellm.types.utils import Choices, Message, ModelResponse
@@ -368,3 +374,86 @@ async def test_pangea_ai_guard_response_transformed(pangea_guardrail):
)
assert response.choices[0]["message"]["content"] == "Yes, here is an SSN: "
+
+
+class _GuardedMessage(TypedDict):
+ tool_calls: ReadOnly[list[ChatCompletionAssistantToolCall]]
+
+
+class _GuardedChoice(TypedDict):
+ message: ReadOnly[_GuardedMessage]
+
+
+class _GuardedOutput(TypedDict):
+ choices: ReadOnly[list[_GuardedChoice]]
+
+
+class _AiGuardRequest(TypedDict):
+ input: ReadOnly[_GuardedOutput]
+
+
+def _claude_fetch_call(url: str) -> ChatCompletionAssistantToolCall:
+ return {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "%s"}' % url},
+ }
+
+
+def _with_masked_tool_calls(choice: _GuardedChoice) -> _GuardedChoice:
+ message: Final = choice["message"]
+ masked_calls: Final = [_claude_fetch_call("") for _ in message["tool_calls"]]
+ return {**choice, "message": {**message, "tool_calls": masked_calls}}
+
+
+@pytest.mark.asyncio
+async def test_pangea_ai_guard_response_masks_claude_server_tool_call_inputs(pangea_guardrail: PangeaHandler) -> None:
+ data: Final = {"messages": [{"role": "user", "content": "Read https://example.com/"}]}
+ fetch: Final = build_web_search_call("srvtoolu_01", {"url": "https://example.com/"}, None, tool_name="web_fetch")
+ sent_choices: Final[list[_GuardedChoice]] = []
+
+ async def _mask_tool_call_urls(
+ _handler: object, url: str, json: _AiGuardRequest, **_kwargs: object
+ ) -> httpx.Response:
+ choices: Final = json["input"]["choices"]
+ sent_choices.extend(choices)
+ return httpx.Response(
+ status_code=200,
+ json={
+ "result": {
+ "blocked": False,
+ "transformed": True,
+ "output": {"choices": [_with_masked_tool_calls(choice) for choice in choices]},
+ }
+ },
+ request=httpx.Request(method="POST", url=url),
+ )
+
+ with patch("litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post", new=_mask_tool_call_urls):
+ response: Final = await pangea_guardrail.async_post_call_success_hook(
+ data=data,
+ user_api_key_dict=None,
+ response=ModelResponse(
+ choices=[
+ Choices(
+ message=Message(
+ role="assistant",
+ content="Done",
+ provider_specific_fields={
+ "server_tool_calls": [_claude_fetch_call("https://example.com/")],
+ "web_search_calls": [fetch],
+ },
+ )
+ )
+ ]
+ ),
+ )
+
+ assert sent_choices[0]["message"]["tool_calls"] == [_claude_fetch_call("https://example.com/")]
+ message: Final = response.choices[0]["message"]
+ assert message["tool_calls"] is None
+ assert message["provider_specific_fields"]["server_tool_calls"] == [_claude_fetch_call("")]
+ hosted_copy: Final = (
+ ResponseFunctionWebSearch.model_validate(message["provider_specific_fields"]["web_search_calls"][0])
+ )
+ assert hosted_copy.action.url == ""
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_panw_prisma_airs.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_panw_prisma_airs.py
index a41331230c2..35cea840df3 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_panw_prisma_airs.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_panw_prisma_airs.py
@@ -6513,5 +6513,90 @@ class TestPanwAirsToolCallBlockMaskedDataRouting:
assert error["scan_id"] == "scan-tool-2"
+class TestPanwAirsServerToolCalls:
+ SERVER_TOOL_CALL: Final = {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "persimmon"}'},
+ }
+
+ @classmethod
+ def _response(cls, *client_tool_calls: ChatCompletionMessageToolCall) -> ModelResponse:
+ return ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ tool_calls=list(client_tool_calls) or None,
+ provider_specific_fields={"server_tool_calls": [copy.deepcopy(cls.SERVER_TOOL_CALL)]},
+ ),
+ )
+ ],
+ model="claude-opus-4-1",
+ )
+
+ @staticmethod
+ def _airs(scan_result: Mapping[str, object]) -> tuple[PanwPrismaAirsHandler, MagicMock]:
+ transport: Final = MagicMock(side_effect=lambda request: httpx.Response(200, json=scan_result))
+ handler: Final = make_handler(
+ mask_response_content=True,
+ http_client=AsyncHTTPHandler(transport=httpx.MockTransport(transport)),
+ )
+ return handler, transport
+
+ @pytest.mark.asyncio
+ async def test_post_call_scan_sends_server_tool_call_query_to_airs(
+ self, user_api_key_dict: UserAPIKeyAuth
+ ) -> None:
+ handler, transport = self._airs({"action": "allow", "category": "benign", "scan_id": "s1"})
+ response: Final = self._response(
+ ChatCompletionMessageToolCall(
+ id="call_1",
+ type="function",
+ function=Function(name="get_weather", arguments='{"city": "Paris"}'),
+ )
+ )
+
+ await handler.async_post_call_success_hook(
+ data={"model": "claude-opus-4-1", "litellm_call_id": "call-srv", "metadata": {}},
+ user_api_key_dict=user_api_key_dict,
+ response=response,
+ )
+
+ sent_contents: Final = [json.loads(call.args[0].content)["contents"] for call in transport.call_args_list]
+ assert sent_contents == [[{"response": '{"query": "persimmon"} {"city": "Paris"}'}]]
+
+ @pytest.mark.asyncio
+ async def test_post_call_masking_rewrites_server_tool_call_arguments(
+ self, user_api_key_dict: UserAPIKeyAuth
+ ) -> None:
+ masked: Final = '{"query": "*********"}'
+ handler, _ = self._airs(
+ {
+ "action": "block",
+ "category": "sensitive_data",
+ "scan_id": "s1",
+ "response_masked_data": {"data": masked},
+ }
+ )
+
+ result: Final = await handler.async_post_call_success_hook(
+ data={"model": "claude-opus-4-1", "litellm_call_id": "call-srv-mask", "metadata": {}},
+ user_api_key_dict=user_api_key_dict,
+ response=self._response(),
+ )
+
+ message: Final = result.choices[0].message
+ assert message.provider_specific_fields == {
+ "server_tool_calls": [
+ {"id": "srvtoolu_01", "type": "function", "function": {"name": "web_search", "arguments": masked}}
+ ]
+ }
+ assert message.tool_calls is None
+
+
if __name__ == "__main__":
pytest.main([__file__, "-v"])
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_presidio.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_presidio.py
index 563b2a0f2b0..cc0d2a8008f 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_presidio.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_presidio.py
@@ -1783,6 +1783,48 @@ async def test_pii_tokens_in_metadata_used_for_unmasking():
assert response.choices[0].message.content == "Hello John, how can I help you?"
+@pytest.mark.asyncio
+async def test_output_unmasking_restores_server_tool_call_arguments(mock_user_api_key: UserAPIKeyAuth) -> None:
+ guardrail: Final = OPTIONAL_PresidioPIIMasking(mock_testing=True, output_parse_pii=True)
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": " persimmon"}'},
+ }
+ ]
+ },
+ ),
+ )
+ ]
+ )
+
+ result: Final = await guardrail.async_post_call_success_hook(
+ data={"metadata": {"pii_tokens": {"": "John Smith"}}},
+ user_api_key_dict=mock_user_api_key,
+ response=response,
+ )
+
+ assert result.choices[0].message.provider_specific_fields == {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "John Smith persimmon"}'},
+ }
+ ]
+ }
+
+
@pytest.mark.parametrize(
"initial_hook",
["pre_call", "during_call", "pre_mcp_call"],
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_qualifire.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_qualifire.py
index dfd54cff730..ef97a335562 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_qualifire.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_qualifire.py
@@ -2,6 +2,7 @@
Unit tests for Qualifire guardrail integration.
"""
+from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@@ -735,3 +736,41 @@ class TestQualifireGuardrailRegistry:
assert "qualifire" in guardrail_class_registry
assert guardrail_class_registry["qualifire"] == QualifireGuardrail
+
+
+def test_convert_messages_sends_replayed_server_tool_calls_with_client_tool_calls() -> None:
+ from litellm.proxy.guardrails.guardrail_hooks.qualifire.qualifire import (
+ QualifireGuardrail,
+ )
+
+ guardrail: Final = QualifireGuardrail(api_key="test_key", guardrail_name="test_guardrail")
+
+ result: Final = guardrail._convert_messages_to_api_format(
+ [
+ {
+ "role": "assistant",
+ "content": "",
+ "tool_calls": [
+ {
+ "id": "call_123",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"location": "NYC"}'},
+ }
+ ],
+ "provider_specific_fields": {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://example.com"}'},
+ }
+ ]
+ },
+ },
+ ]
+ )
+
+ assert result[0]["tool_calls"] == [
+ {"id": "srvtoolu_01", "name": "web_fetch", "arguments": {"url": "https://example.com"}},
+ {"id": "call_123", "name": "get_weather", "arguments": {"location": "NYC"}},
+ ]
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_repelloai.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_repelloai.py
index 77883e9af0e..ff61af9ee62 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_repelloai.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_repelloai.py
@@ -1,4 +1,6 @@
import os
+from collections.abc import Mapping
+from typing import Final
import pytest
from fastapi import HTTPException
@@ -852,6 +854,61 @@ class TestRepelloAIPostCall:
in captured["json"]["scan_data"]["response"]
)
+ @pytest.mark.asyncio
+ async def test_server_tool_call_arguments_scanned(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ guardrail: Final = _guardrail(event_hook="post_call")
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ index=0,
+ finish_reason="tool_calls",
+ message=Message(
+ role="assistant",
+ content=None,
+ tool_calls=[
+ {
+ "id": "call_01",
+ "type": "function",
+ "function": {
+ "name": "lookup_order",
+ "arguments": '{"order_id": "A-17"}',
+ },
+ }
+ ],
+ provider_specific_fields={
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {
+ "name": "web_search",
+ "arguments": '{"query": "persimmon"}',
+ },
+ }
+ ]
+ },
+ ),
+ )
+ ]
+ )
+ captured: Final = {}
+
+ async def capture(
+ url: str, headers: Mapping[str, str], json: Mapping[str, object], **_kwargs: object
+ ) -> Response:
+ captured["json"] = json
+ return _verdict_response("passed", url)
+
+ monkeypatch.setattr(guardrail.async_handler, "post", capture)
+ await guardrail.async_post_call_success_hook(
+ data={"messages": [{"role": "user", "content": "q"}]},
+ user_api_key_dict=UserAPIKeyAuth(),
+ response=response,
+ )
+ assert captured["json"]["scan_data"] == {
+ "response": '{"query": "persimmon"}\n{"order_id": "A-17"}'
+ }
+
# ----------------------------------------------------------------------
# verdict handling: unknown / malformed responses must not fail open
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_permission.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_permission.py
index 46e766f753c..a46f1ddeed9 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_permission.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_permission.py
@@ -23,6 +23,7 @@ from litellm.proxy.guardrails.guardrail_hooks.tool_permission import (
ToolPermissionGuardrail,
)
from litellm.types.guardrails import GuardrailEventHooks, LitellmParams
+from litellm.types.responses.main import build_web_search_call
from litellm.types.proxy.guardrails.guardrail_hooks.tool_permission import (
PermissionError,
)
@@ -41,6 +42,14 @@ from litellm.types.utils import (
)
+_SERVER_WEB_SEARCH_CALL: Final = {
+ "id": "srvtoolu_01Search",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query": "weather in Paris"}'},
+}
+_ALLOWED_BASH_CALL: Final = {"id": "toolu_01Bash", "type": "function", "function": {"name": "Bash", "arguments": "{}"}}
+
+
class TestToolPermissionGuardrail:
"""Test class for Tool Permission Guardrail functionality"""
@@ -1376,6 +1385,93 @@ class TestToolPermissionGuardrail:
assert "Permission denied" in choice.message.content
+ def _server_search_response(self) -> ModelResponse:
+ return ModelResponse(
+ choices=[
+ Choices(
+ finish_reason="tool_calls",
+ message={
+ "content": "Searching.",
+ "tool_calls": [_ALLOWED_BASH_CALL],
+ "provider_specific_fields": {"server_tool_calls": [_SERVER_WEB_SEARCH_CALL]},
+ },
+ )
+ ]
+ )
+
+ @pytest.mark.asyncio
+ async def test_denied_server_tool_call_is_blocked(self):
+ with patch.object(self.guardrail, "should_run_guardrail", return_value=True):
+ with pytest.raises(GuardrailRaisedException):
+ await self.guardrail.async_post_call_success_hook(
+ data={}, user_api_key_dict=UserAPIKeyAuth(), response=self._server_search_response()
+ )
+
+ @pytest.mark.asyncio
+ async def test_denied_server_tool_call_in_a_chat_stream_is_blocked(self):
+ async def _stream():
+ yield ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=0,
+ delta=Delta(
+ content="Searching.",
+ provider_specific_fields={"server_tool_calls": [{**_SERVER_WEB_SEARCH_CALL, "index": 0}]},
+ ),
+ finish_reason="stop",
+ )
+ ]
+ )
+
+ with patch.object(self.guardrail, "should_run_guardrail", return_value=True):
+ with pytest.raises(GuardrailRaisedException):
+ async for _ in self.guardrail.async_post_call_streaming_iterator_hook(
+ user_api_key_dict=UserAPIKeyAuth(), response=_stream(), request_data={}
+ ):
+ pass
+
+ @pytest.mark.asyncio
+ async def test_rewrite_mode_removes_the_denied_server_tool_call(self):
+ rewriting: Final = ToolPermissionGuardrail(
+ guardrail_name="test-tool-permission",
+ rules=self.test_rules,
+ default_action="deny",
+ on_disallowed_action="rewrite",
+ )
+
+ with patch.object(rewriting, "should_run_guardrail", return_value=True):
+ result: Final = await rewriting.async_post_call_success_hook(
+ data={}, user_api_key_dict=UserAPIKeyAuth(), response=self._server_search_response()
+ )
+
+ message: Final = result.choices[0].message
+ assert message.provider_specific_fields["server_tool_calls"] == []
+ assert [tool_call.id for tool_call in message.tool_calls] == [_ALLOWED_BASH_CALL["id"]]
+ assert "Permission denied" in message.content
+
+ @pytest.mark.asyncio
+ async def test_rewrite_mode_removes_the_denied_server_tool_calls_hosted_copy(self):
+ rewriting: Final = ToolPermissionGuardrail(
+ guardrail_name="test-tool-permission",
+ rules=self.test_rules,
+ default_action="deny",
+ on_disallowed_action="rewrite",
+ )
+ response: Final = self._server_search_response()
+ response.choices[0].message.provider_specific_fields = {
+ **response.choices[0].message.provider_specific_fields,
+ "web_search_calls": [
+ build_web_search_call("srvtoolu_01Search", {"query": "weather in Paris"}, {"content": []})
+ ],
+ }
+
+ with patch.object(rewriting, "should_run_guardrail", return_value=True):
+ result: Final = await rewriting.async_post_call_success_hook(
+ data={}, user_api_key_dict=UserAPIKeyAuth(), response=response
+ )
+
+ assert result.choices[0].message.provider_specific_fields["web_search_calls"] == []
+
def test_modify_response_with_permission_errors_keeps_custom_tool_call_in_another_choice(self):
denied_call = ChatCompletionMessageToolCall(
id="call_bash", type="function", function=Function(name="Bash", arguments="{}")
@@ -1405,6 +1501,7 @@ class TestToolPermissionGuardrail:
]
assert kept == [[], ["call_patch"]]
+
class TestToolPermissionGuardrailIntegration:
"""Integration tests for Tool Permission Guardrail"""
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_policy_guardrail.py b/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_policy_guardrail.py
index f40465e8d7b..bac470b1365 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_policy_guardrail.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/test_tool_policy_guardrail.py
@@ -2,13 +2,16 @@
Unit tests for ToolPolicyGuardrail.
"""
-from typing import Any
+from collections.abc import Mapping, Sequence
+from types import SimpleNamespace
+from typing import Any, Final
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from fastapi import HTTPException
+from litellm.proxy.db import tool_registry_writer
from litellm.proxy.guardrails.guardrail_hooks.tool_policy.tool_policy_guardrail import (
ToolPolicyGuardrail,
)
@@ -222,3 +225,104 @@ async def test_blocked_tool_loaded_by_a_replayed_tool_search_raises_next_to_allo
)
assert exc_info.value.status_code == 400
assert exc_info.value.detail["blocked_tools"] == ["run_shell"]
+
+
+async def _install_tool_policies(monkeypatch: pytest.MonkeyPatch, input_policies: Mapping[str, str]) -> None:
+ prisma_client = MagicMock()
+ prisma_client.db.litellm_tooltable.find_many = AsyncMock(
+ return_value=[
+ SimpleNamespace(tool_name=name, input_policy=policy, output_policy="untrusted")
+ for name, policy in input_policies.items()
+ ]
+ )
+ prisma_client.db.litellm_objectpermissiontable.find_many = AsyncMock(return_value=[])
+ registry = tool_registry_writer.ToolPolicyRegistry()
+ await registry.sync_tool_policy_from_db(prisma_client)
+ monkeypatch.setattr(tool_registry_writer, "_tool_policy_registry", registry)
+
+
+def _responses_request(
+ tools: Sequence[Mapping[str, object]], item_tools: Sequence[Mapping[str, object]]
+) -> dict[str, object]:
+ return {
+ "user_api_key_request_route": "/v1/responses",
+ "tools": tools,
+ "input": [
+ {"type": "additional_tools", "role": "developer", "tools": item_tools},
+ {"role": "user", "content": "clean up the disk"},
+ ],
+ }
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "request_data",
+ [
+ _responses_request(
+ [{"type": "function", "name": "get_weather"}],
+ [{"type": "function", "name": "drop_database"}],
+ ),
+ _responses_request(
+ [{"type": "function", "name": "get_weather"}],
+ [
+ {
+ "type": "namespace",
+ "name": "mcp__db",
+ "tools": [{"type": "function", "name": "drop_database"}],
+ }
+ ],
+ ),
+ _responses_request(
+ [
+ {"type": "function", "name": "get_weather"},
+ {"type": "mcp", "server_label": "drop_database"},
+ ],
+ [],
+ ),
+ _responses_request(
+ [
+ {"type": "function", "name": "get_weather"},
+ {"type": "namespace", "name": "mcp__db", "tools": [{"type": "function", "name": "drop_database"}]},
+ ],
+ [],
+ ),
+ ],
+ ids=["additional_tools_item", "namespace_in_item", "top_level_mcp", "top_level_namespace"],
+)
+async def test_blocked_tool_raises_next_to_a_top_level_function_tool(
+ guardrail: ToolPolicyGuardrail, request_data: dict[str, object], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ await _install_tool_policies(monkeypatch, {"get_weather": "untrusted", "drop_database": "blocked"})
+ with pytest.raises(HTTPException) as exc_info:
+ await guardrail.apply_guardrail(
+ inputs=_tool_request_inputs(["get_weather"]),
+ request_data=request_data,
+ input_type="request",
+ )
+ assert exc_info.value.status_code == 400
+ assert exc_info.value.detail["blocked_tools"] == ["drop_database"]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("name", [["drop_database"], {"name": "drop_database"}], ids=["list", "dict"])
+async def test_function_tool_name_that_is_not_a_string_is_left_for_the_provider_to_refuse(
+ guardrail: ToolPolicyGuardrail, name: object, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ await _install_tool_policies(monkeypatch, {"get_weather": "untrusted"})
+ inputs: Final = {"tools": [{"type": "function", "function": {"name": name}}]}
+ assert await guardrail.apply_guardrail(inputs=inputs, request_data={}, input_type="request") == inputs
+
+
+@pytest.mark.asyncio
+async def test_blocked_tool_defined_at_the_top_level_and_in_an_item_is_named_once(
+ guardrail: ToolPolicyGuardrail, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ tool = {"type": "function", "name": "drop_database"}
+ await _install_tool_policies(monkeypatch, {"drop_database": "blocked"})
+ with pytest.raises(HTTPException) as exc_info:
+ await guardrail.apply_guardrail(
+ inputs=_tool_request_inputs(["drop_database"]),
+ request_data=_responses_request([tool], [tool]),
+ input_type="request",
+ )
+ assert exc_info.value.detail["blocked_tools"] == ["drop_database"]
diff --git a/tests/unit/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py b/tests/unit/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py
index de3ced0c583..2229741fce5 100644
--- a/tests/unit/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py
+++ b/tests/unit/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py
@@ -50,6 +50,7 @@ from litellm.proxy.mcp_server.guardrail_translation.handler import (
)
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.llms.openai import ResponsesAPIResponse
+from litellm.types.responses.main import build_web_search_call
from litellm.types.utils import (
CallTypes,
CallTypesLiteral,
@@ -1288,6 +1289,38 @@ class TestStreamingTransform:
# end of stream (it can block), matching block_only.
assert guardrail.received_tool_calls
+ @pytest.mark.asyncio
+ async def test_claude_server_tool_call_chunks_keep_their_provider_fields(self):
+ server_fields: Final = {
+ "server_tool_calls": [
+ {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://example.com/"}'},
+ }
+ ],
+ "web_search_calls": {
+ "srvtoolu_01": build_web_search_call(
+ "srvtoolu_01", {"url": "https://example.com/"}, None, tool_name="web_fetch"
+ ).model_dump()
+ },
+ }
+ server_chunk: Final = ModelResponseStream(
+ choices=[StreamingChoices(index=0, delta=Delta(content=None, provider_specific_fields=server_fields))],
+ )
+
+ out: Final = await _drive_stream(
+ UnifiedLLMGuardrails(),
+ _StreamingTextGuardrail(),
+ [server_chunk, _stream_chunk("done", finish_reason="stop")],
+ )
+
+ delivered: Final = (
+ [getattr(item.choices[0].delta, "provider_specific_fields", None) for item in out if item.choices]
+ )
+ assert server_fields in delivered
+ assert "".join(_delta_text(item) for item in out) == "DONE"
+
@pytest.mark.asyncio
async def test_per_choice_finish_reason_when_choices_finish_in_different_chunks(self):
"""n>1: a choice finishing before the stream's last chunk keeps its own
diff --git a/tests/unit/proxy/guardrails/test_pillar_guardrails.py b/tests/unit/proxy/guardrails/test_pillar_guardrails.py
index c6be433399a..86c1b1de8a1 100644
--- a/tests/unit/proxy/guardrails/test_pillar_guardrails.py
+++ b/tests/unit/proxy/guardrails/test_pillar_guardrails.py
@@ -7,7 +7,7 @@ and following LiteLLM testing patterns and best practices.
# Standard library imports
import importlib
-from typing import Any, Dict
+from typing import Any, Dict, Final
from unittest.mock import Mock, patch
# Add parent directory to path for imports
@@ -18,7 +18,7 @@ from urllib.parse import unquote
import pytest
from fastapi.exceptions import HTTPException
-from httpx import Request, Response
+from httpx import Request, Response, Timeout
# LiteLLM imports
import litellm
@@ -34,6 +34,7 @@ from litellm.proxy.guardrails.guardrail_hooks.pillar.pillar import (
build_pillar_response_headers,
)
from litellm.proxy.guardrails.init_guardrails import init_guardrails_v2
+from litellm.types.utils import Choices, Message, ModelResponse
# ============================================================================
# FIXTURES
@@ -669,6 +670,63 @@ async def test_post_call_hook_with_tool_calls(
assert result == mock_llm_response_with_tools
+@pytest.mark.asyncio
+async def test_post_call_hook_blocks_a_flagged_claude_server_tool_call(
+ env_setup: None,
+ sample_request_data: dict[str, object],
+ user_api_key_dict: UserAPIKeyAuth,
+ pillar_clean_response: Response,
+ pillar_flagged_response: Response,
+) -> None:
+ server_call: Final = {
+ "id": "srvtoolu_01",
+ "type": "function",
+ "function": {"name": "web_fetch", "arguments": '{"url": "https://example.com/"}'},
+ }
+ response: Final = ModelResponse(
+ choices=[
+ Choices(
+ message=Message(
+ role="assistant",
+ content="Fetched it",
+ provider_specific_fields={"server_tool_calls": [server_call]},
+ )
+ )
+ ]
+ )
+
+ class _PillarFlagsTheServerCall:
+ async def post(
+ self,
+ *,
+ url: str,
+ headers: dict[str, str],
+ json: dict[str, object],
+ timeout: float | Timeout | None,
+ ) -> Response:
+ messages: Final = json["messages"]
+ assert isinstance(messages, list)
+ scanned_tool_calls: Final = messages[-1].get("tool_calls") or []
+ return pillar_flagged_response if server_call in scanned_tool_calls else pillar_clean_response
+
+ guardrail: Final = PillarGuardrail(
+ guardrail_name="pillar-test",
+ api_key="test-pillar-key",
+ api_base="https://api.pillar.security",
+ on_flagged_action="block",
+ async_handler=_PillarFlagsTheServerCall(),
+ )
+
+ with pytest.raises(HTTPException) as excinfo:
+ await guardrail.async_post_call_success_hook(
+ data=sample_request_data,
+ user_api_key_dict=user_api_key_dict,
+ response=response,
+ )
+
+ assert excinfo.value.status_code == 400
+
+
# =========================================================================
# HEADER CONFIGURATION TESTS
# =========================================================================
diff --git a/tests/unit/proxy/hooks/test_model_max_budget_limiter.py b/tests/unit/proxy/hooks/test_model_max_budget_limiter.py
index f1fd02981d6..7572321014d 100644
--- a/tests/unit/proxy/hooks/test_model_max_budget_limiter.py
+++ b/tests/unit/proxy/hooks/test_model_max_budget_limiter.py
@@ -12,6 +12,7 @@ import pytest
import litellm
from litellm.caching.caching import DualCache
+from litellm.litellm_core_utils.model_latest_aliases import get_model_latest_aliases
from litellm.proxy._types import Litellm_EntityType, UserAPIKeyAuth
from litellm.proxy.db.db_transaction_queue.window_spend_update_queue import WindowSpendTransaction
from litellm.proxy.hooks.model_budget_shared_windows import DatabaseSharedWindows, SharedWindows
@@ -331,7 +332,6 @@ async def test_a_batch_polled_within_every_budget_window_is_never_charged_again(
assert _local_spend(limiter, KEY_SPEND_KEY) == pytest.approx(BATCH_COST)
-
async def _poll_video(
limiter: PROXY_VirtualKeyModelMaxBudgetLimiter, video_id: str, status: str, response_cost: float
) -> None:
@@ -382,6 +382,7 @@ async def test_only_the_finished_poll_pays_the_job_and_only_a_video_repoll_pays_
assert await _spend(limiter, KEY_SPEND_KEY) == pytest.approx(BATCH_COST + guardrail_charges * guardrail_cost)
+
OPUS_GROUP: Final = "claude-opus"
OPUS_GROUP_MEMBERS: Final = MappingProxyType(
{"claude-opus-4-8": frozenset({OPUS_GROUP}), "claude-opus-5-5": frozenset({OPUS_GROUP})}
@@ -487,6 +488,91 @@ def _key(model_max_budget: Mapping[str, object]) -> UserAPIKeyAuth:
return UserAPIKeyAuth(token=KEY_HASH, key_alias="opus-family", model_max_budget=dict(model_max_budget))
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team"))
+@pytest.mark.parametrize("prefix", ("", "anthropic/"))
+@pytest.mark.parametrize("budget_alias", (False, True))
+async def test_latest_alias_is_blocked_by_either_model_budget(scope: str, prefix: str, budget_alias: bool) -> None:
+ target: Final = get_model_latest_aliases()["anthropic"]["claude-opus-latest"]
+ name: Final = "claude-opus-latest" if budget_alias else target
+ budget: Final = {f"{prefix}{name}": {"max_budget": 0, "budget_duration": "1d"}}
+ limiter: Final = _family_limiter()
+ admission: Final = (
+ limiter.is_key_within_model_budget(_key(budget), "anthropic/claude-opus-latest")
+ if scope == "key"
+ else limiter.is_team_within_model_budget(TEAM_ID, budget, None, "anthropic/claude-opus-latest")
+ )
+ with pytest.raises(litellm.BudgetExceededError, match="exceeded budget"):
+ await admission
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("model", ("anthropic/claude-opus-5-5", "anthropic/unknown-model"))
+async def test_latest_alias_budget_does_not_govern_other_requests(model: str) -> None:
+ limiter: Final = _family_limiter()
+ budget: Final = {"anthropic/claude-opus-latest": {"max_budget": 0, "budget_duration": "1d"}}
+ assert await limiter.is_key_within_model_budget(_key(budget), model)
+ assert await limiter.is_team_within_model_budget(TEAM_ID, budget, None, model)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", (Litellm_EntityType.KEY, Litellm_EntityType.TEAM))
+@pytest.mark.parametrize(
+ "entries",
+ (
+ ("anthropic/claude-opus-latest", "claude-opus-5-5"),
+ ("claude-opus*",),
+ ("anthropic/claude-opus-5-5",),
+ ("claude-opus-5-5",),
+ ),
+)
+@pytest.mark.parametrize("requested", ("anthropic/claude-opus-latest", "anthropic/claude-opus-5-5"))
+async def test_latest_alias_spend_charges_each_governing_counter_once(
+ scope: Litellm_EntityType, entries: tuple[str, ...], requested: str
+) -> None:
+ budget: Final = {entry: FAMILY_CAP for entry in entries}
+ cache: Final = DualCache()
+ limiter: Final = _family_limiter(dual_cache=cache)
+ await _log_success(
+ limiter,
+ {
+ "call_type": "acompletion",
+ "standard_logging_object": {
+ "model": "claude-opus-5-5",
+ "model_group": requested,
+ "response_cost": CHAT_COST,
+ "metadata": {"user_api_key_hash": KEY_HASH, "user_api_key_team_id": TEAM_ID},
+ },
+ "litellm_params": {
+ "metadata": {
+ "user_api_key_model_max_budget": budget if scope == Litellm_EntityType.KEY else {},
+ "user_api_key_team_model_max_budget": budget if scope == Litellm_EntityType.TEAM else {},
+ }
+ },
+ },
+ )
+ for entry in entries:
+ spend_key: Final = model_budget_spend_cache_key(
+ entity_type=scope,
+ entity_id=KEY_HASH if scope == Litellm_EntityType.KEY else TEAM_ID,
+ budget_model=entry,
+ budget_duration="1d",
+ )
+ expected: Final = (
+ CHAT_COST
+ if requested == "anthropic/claude-opus-latest" or entry != "anthropic/claude-opus-latest"
+ else None
+ )
+ assert await cache.async_get_cache(spend_key, local_only=True) == expected
+ admission: Final = (
+ limiter.is_key_within_model_budget(_key(budget), requested)
+ if scope == Litellm_EntityType.KEY
+ else limiter.is_team_within_model_budget(TEAM_ID, budget, None, requested)
+ )
+ with pytest.raises(litellm.BudgetExceededError, match="exceeded budget"):
+ await admission
+
+
async def _spend_on(
limiter: PROXY_VirtualKeyModelMaxBudgetLimiter, model: str, model_max_budget: Mapping[str, object]
) -> None:
diff --git a/tests/unit/proxy/hooks/test_parallel_request_limiter_v3.py b/tests/unit/proxy/hooks/test_parallel_request_limiter_v3.py
index cf0d53dbf59..64e7cde4034 100644
--- a/tests/unit/proxy/hooks/test_parallel_request_limiter_v3.py
+++ b/tests/unit/proxy/hooks/test_parallel_request_limiter_v3.py
@@ -28,6 +28,8 @@ from litellm.exceptions import ServiceUnavailableError
from litellm.integrations.custom_secret_manager import CustomSecretManager
from litellm.litellm_core_utils import ptu_pricing
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.litellm_core_utils.model_latest_aliases import get_model_latest_aliases
+from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.ptu_shares import PTUTeamCeiling
from litellm.llms.azure.ptu_capacity import AZURE_PTU_CAPACITY
from litellm.proxy._types import UserAPIKeyAuth
@@ -133,6 +135,533 @@ def _isolated_request_stash():
_request_stash.reset(token)
+@pytest.fixture
+def reservation() -> bool:
+ return True
+
+
+@pytest.fixture
+def collision_limiter(
+ reservation: bool, monkeypatch: pytest.MonkeyPatch,
+) -> tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str]:
+ monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", str(reservation).lower())
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ router: Final = Router(
+ model_list=[{"model_name": "anthropic/*", "litellm_params": {"model": "anthropic/*"}}],
+ model_group_alias={alias: target},
+ )
+ cache: Final = DualCache()
+ handler: Final = PROXY_MaxParallelRequestsHandler_v3(
+ internal_usage_cache=InternalUsageCache(cache),
+ model_group_resolver=partial(resolve_model_group_alias, router.registry().aliases()),
+ time_provider=lambda: datetime(2026, 1, 1, tzinfo=timezone.utc),
+ )
+ return cache, handler, alias, target
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "scope,metric",
+ tuple(
+ (scope, metric)
+ for scope in ("key", "team", "project", "organization")
+ for metric in ("model_rpm_limit", "model_tpm_limit")
+ ) + (("project", "model_itpm_limit"), ("project", "model_otpm_limit")),
+)
+@pytest.mark.parametrize("limit_alias", (False, True))
+async def test_latest_alias_collision_keeps_the_stricter_limit(
+ collision_limiter: tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str],
+ scope: str, metric: str, limit_alias: bool,
+) -> None:
+ cache, handler, alias, target = collision_limiter
+ limits: Final = {metric: {alias: 0 if limit_alias else 100, target: 100 if limit_alias else 0}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="collision-key", team_id="collision-team", project_id="collision-project", org_id="collision-org",
+ **{("metadata" if scope == "key" else f"{scope}_metadata"): limits},
+ )
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.async_pre_call_hook(
+ auth, cache,
+ {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1}, "acompletion",
+ )
+ assert exc.value.status_code == 429
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+@pytest.mark.parametrize("reservation", (False, True))
+@pytest.mark.parametrize("event", ("success", "failure", "partial"))
+async def test_latest_alias_collision_charges_and_settles_one_counter_per_metric(
+ collision_limiter: tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str],
+ scope: str, reservation: bool, event: str,
+) -> None:
+ cache, handler, alias, target = collision_limiter
+ limits: Final = {"model_rpm_limit": {alias: 2}, "model_tpm_limit": {target: 10000}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="collision-key", team_id="collision-team", project_id="collision-project", org_id="collision-org",
+ **{("metadata" if scope == "key" else f"{scope}_metadata"): limits},
+ )
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}, "acompletion"
+ )
+ entity: Final = {
+ "key": auth.api_key, "team": auth.team_id, "project": auth.project_id, "organization": auth.org_id
+ }[scope]
+ rpm_key: Final = _counter(f"model_per_{scope}", f"{entity}:{target}", "requests")
+ tpm_key: Final = _counter(f"model_per_{scope}", f"{entity}:{target}", "tokens")
+ stash: Final = get_request_stash()
+ assert stash is not None
+ assert await handler.rate_limits.get(rpm_key) == 1
+ assert await handler.rate_limits.get(tpm_key) == (stash.reserved_tokens if reservation else 1)
+ usage: Final = Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5)
+ kwargs: Final = {
+ "standard_logging_object": {
+ "model_group": alias,
+ "metadata": {
+ "user_api_key_hash": auth.api_key, "user_api_key_team_id": auth.team_id,
+ "user_api_key_project_id": auth.project_id, "user_api_key_org_id": auth.org_id,
+ },
+ },
+ "litellm_params": {"metadata": {"model_group": alias, "user_api_key_metadata": auth.metadata}},
+ **({"combined_usage_object": usage} if event == "partial" else {}),
+ }
+ if event == "success":
+ await handler.async_log_success_event(kwargs, ModelResponse(model=target, usage=usage), None, None)
+ else:
+ await handler.async_log_failure_event(kwargs, None, None, None)
+ actual_tokens: Final = 5 if event == "success" or (event == "partial" and reservation) else 0
+ assert await handler.rate_limits.get(tpm_key) == actual_tokens + (0 if reservation else 1)
+ alias_key: Final = _counter(f"model_per_{scope}", f"{entity}:{alias}", "tokens")
+ assert await handler.rate_limits.get(alias_key) is None
+ assert await handler.async_pre_call_hook(auth, cache, {"model": alias}, "acompletion") is None
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.async_pre_call_hook(auth, cache, {"model": alias}, "acompletion")
+ assert exc.value.status_code == 429
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("event", ("success", "failure", "partial"))
+async def test_latest_alias_collision_project_io_reservations_settle_once(
+ collision_limiter: tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str], event: str,
+) -> None:
+ cache, handler, alias, target = collision_limiter
+ auth: Final = UserAPIKeyAuth(
+ api_key="collision-io-key", project_id="collision-project",
+ project_metadata={metric: {alias: 10000, target: 10000} for metric in ("model_itpm_limit", "model_otpm_limit")},
+ )
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}, "acompletion"
+ )
+ stash: Final = get_request_stash()
+ assert stash is not None
+ for scope, reserved in (
+ ("model_per_project_itpm", stash.itpm_reserved_tokens), ("model_per_project_otpm", stash.otpm_reserved_tokens),
+ ):
+ assert reserved > 0
+ counter: Final = _counter(scope, f"{auth.project_id}:{target}", "tokens")
+ assert await handler.rate_limits.get(counter) == reserved
+ usage: Final = Usage(
+ prompt_tokens=3, completion_tokens=2, total_tokens=5, prompt_tokens_details={"cached_tokens": 1}
+ )
+ kwargs: Final = {
+ "standard_logging_object": {
+ "model_group": alias,
+ "metadata": {"user_api_key_hash": auth.api_key, "user_api_key_project_id": auth.project_id},
+ },
+ "litellm_params": {"metadata": {"model_group": alias}},
+ **({"combined_usage_object": usage} if event == "partial" else {}),
+ }
+ if event == "success":
+ await handler.async_log_success_event(kwargs, ModelResponse(model=target, usage=usage), None, None)
+ else:
+ await handler.async_log_failure_event(kwargs, None, None, None)
+ for scope in ("model_per_project_itpm", "model_per_project_otpm"):
+ counter: Final = _counter(scope, f"{auth.project_id}:{target}", "tokens")
+ assert int(await handler.rate_limits.get(counter) or 0) == (0 if event == "failure" else 2)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("reservation", (False, True))
+@pytest.mark.parametrize("event", ("success", "partial"))
+async def test_latest_alias_collision_settles_the_target_team_limit_beside_the_key_alias_limit(
+ collision_limiter: tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str],
+ reservation: bool, event: str,
+) -> None:
+ cache, handler, alias, target = collision_limiter
+ auth: Final = UserAPIKeyAuth(
+ api_key="collision-inherited-key", team_id="collision-team",
+ metadata={"model_tpm_limit": {alias: 10000}}, team_metadata={"model_tpm_limit": {target: 10000}},
+ )
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}, "acompletion"
+ )
+ usage: Final = Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5)
+ kwargs: Final = {
+ "standard_logging_object": {
+ "model_group": alias,
+ "metadata": {"user_api_key_hash": auth.api_key, "user_api_key_team_id": auth.team_id},
+ },
+ "litellm_params": {"metadata": {"model_group": alias, "user_api_key_metadata": auth.metadata}},
+ **({"combined_usage_object": usage} if event == "partial" else {}),
+ }
+ if event == "success":
+ await handler.async_log_success_event(kwargs, ModelResponse(model=target, usage=usage), None, None)
+ else:
+ await handler.async_log_failure_event(kwargs, None, None, None)
+ actual_tokens: Final = 5 if reservation or event == "success" else 0
+ for scope, entity in (("model_per_key", auth.api_key), ("model_per_team", auth.team_id)):
+ counter: Final = _counter(scope, f"{entity}:{target}", "tokens")
+ assert await handler.rate_limits.get(counter) == actual_tokens + (0 if reservation else 1)
+
+
+@pytest.mark.asyncio
+async def test_latest_alias_collision_project_frames_charge_once_and_enforce_the_lower_cap(
+ collision_limiter: tuple[DualCache, PROXY_MaxParallelRequestsHandler_v3, str, str],
+) -> None:
+ cache, handler, alias, target = collision_limiter
+ auth: Final = UserAPIKeyAuth(
+ api_key="collision-frame-key", project_id="collision-project",
+ project_metadata={metric: {alias: 5, target: 100} for metric in ("model_itpm_limit", "model_otpm_limit")},
+ )
+ await handler.enforce_project_io_token_quota_for_frame(auth, alias, 3, 2)
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.enforce_project_io_token_quota_for_frame(auth, alias, 3, 2)
+ assert exc.value.status_code == 429
+ for scope, expected in (("model_per_project_itpm", 3), ("model_per_project_otpm", 2)):
+ counter: Final = _counter(scope, f"{auth.project_id}:{target}", "tokens")
+ assert await handler.rate_limits.get(counter) == expected
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+@pytest.mark.parametrize("metric", ("model_rpm_limit", "model_tpm_limit"))
+@pytest.mark.parametrize("budget_alias", (False, True))
+async def test_latest_alias_is_blocked_by_either_rate_limit(scope: str, metric: str, budget_alias: bool) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ target: Final = get_model_latest_aliases()["anthropic"]["claude-opus-latest"]
+ limited_model: Final = "anthropic/claude-opus-latest" if budget_alias else f"anthropic/{target}"
+ limits: Final = {metric: {limited_model: 0}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-rate-key",
+ team_id="latest-rate-team",
+ project_id="latest-rate-project",
+ org_id="latest-rate-org",
+ metadata=limits if scope == "key" else {},
+ team_metadata=limits if scope == "team" else {},
+ project_metadata=limits if scope == "project" else {},
+ organization_metadata=limits if scope == "organization" else {},
+ )
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.async_pre_call_hook(
+ auth,
+ cache,
+ {"model": "anthropic/claude-opus-latest", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1},
+ "acompletion",
+ )
+ assert exc.value.status_code == 429
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+@pytest.mark.parametrize("metric", ("model_rpm_limit", "model_tpm_limit"))
+@pytest.mark.parametrize("model", ("anthropic/claude-opus-5-5", "anthropic/unknown-model", None))
+async def test_latest_alias_rate_limit_does_not_govern_other_requests(scope: str, metric: str, model: str | None) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ limits: Final = {metric: {"anthropic/claude-opus-latest": 0}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-other-key",
+ team_id="latest-other-team",
+ project_id="latest-other-project",
+ org_id="latest-other-org",
+ metadata=limits if scope == "key" else {},
+ team_metadata=limits if scope == "team" else {},
+ project_metadata=limits if scope == "project" else {},
+ organization_metadata=limits if scope == "organization" else {},
+ )
+ assert await handler.async_pre_call_hook(auth, cache, {"model": model}, "acompletion") is None
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+async def test_latest_alias_rpm_draws_from_both_model_counters(scope: str) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ limits: Final = {"model_rpm_limit": {alias: 2, target: 1}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-rpm-key",
+ team_id="latest-rpm-team",
+ project_id="latest-rpm-project",
+ org_id="latest-rpm-org",
+ organization_rpm_limit=10 if scope == "organization" else None,
+ metadata=limits if scope == "key" else {},
+ team_metadata=limits if scope == "team" else {},
+ project_metadata=limits if scope == "project" else {},
+ organization_metadata=limits if scope == "organization" else {},
+ )
+ assert await handler.async_pre_call_hook(auth, cache, {"model": alias}, "acompletion") is None
+ descriptor: Final = f"model_per_{scope}"
+ entity: Final = {
+ "key": auth.api_key, "team": auth.team_id, "project": auth.project_id, "organization": auth.org_id
+ }[scope]
+ for model in (alias, target):
+ counter: Final = _counter(descriptor, f"{entity}:{model}", "requests")
+ assert await handler.rate_limits.get(counter) == 1
+ if scope == "organization":
+ global_counter: Final = _counter("organization", str(auth.org_id), "requests")
+ assert await handler.rate_limits.get(global_counter) == 1
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.async_pre_call_hook(auth, cache, {"model": alias}, "acompletion")
+ assert exc.value.status_code == 429
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+@pytest.mark.parametrize("event", ("success", "failure", "partial"))
+@pytest.mark.parametrize("reservation", (False, True))
+async def test_latest_alias_tpm_settles_both_counters(
+ scope: str, event: str, reservation: bool, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", str(reservation).lower())
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(
+ internal_usage_cache=InternalUsageCache(cache),
+ time_provider=lambda: datetime(2026, 1, 1, tzinfo=timezone.utc),
+ )
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ limits: Final = {"model_tpm_limit": {alias: 10000, target: 10000}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-tpm-key",
+ team_id="latest-tpm-team",
+ project_id="latest-tpm-project",
+ org_id="latest-tpm-org",
+ tpm_limit=10000,
+ metadata=limits if scope == "key" else {},
+ team_metadata=limits if scope == "team" else {},
+ project_metadata=limits if scope == "project" else {},
+ organization_metadata=limits if scope == "organization" else {},
+ )
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}, "acompletion"
+ )
+ stash: Final = get_request_stash()
+ assert stash is not None
+ descriptor: Final = f"model_per_{scope}"
+ entity: Final = {
+ "key": auth.api_key, "team": auth.team_id, "project": auth.project_id, "organization": auth.org_id
+ }[scope]
+ keys: Final = tuple(
+ _counter(descriptor, f"{entity}:{model}", "tokens") for model in (alias, target)
+ )
+ for key in keys:
+ assert int(await handler.rate_limits.get(key) or 0) == (stash.reserved_tokens if reservation else 1)
+ kwargs: Final = {
+ "standard_logging_object": {
+ "model_group": alias,
+ "metadata": {
+ "user_api_key_hash": auth.api_key,
+ "user_api_key_team_id": auth.team_id,
+ "user_api_key_project_id": auth.project_id,
+ "user_api_key_org_id": auth.org_id,
+ },
+ },
+ "litellm_params": {"metadata": {"model_group": alias, "user_api_key_metadata": auth.metadata}},
+ **(
+ {"combined_usage_object": Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5)}
+ if event == "partial"
+ else {}
+ ),
+ }
+ if event == "success":
+ await handler.async_log_success_event(
+ kwargs,
+ ModelResponse(model=target, usage=Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5)),
+ None,
+ None,
+ )
+ else:
+ await handler.async_log_failure_event(kwargs, None, None, None)
+ actual_tokens: Final = 5 if event == "success" or (event == "partial" and reservation) else 0
+ for key in (*keys, _counter("api_key", auth.api_key, "tokens")):
+ assert int(await handler.rate_limits.get(key) or 0) == actual_tokens + (0 if reservation else 1)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("scope", ("key", "team", "project", "organization"))
+@pytest.mark.parametrize("request_alias", (False, True))
+@pytest.mark.parametrize("reservation_released", (False, True))
+async def test_failed_latest_alias_without_open_reservation_matches_direct_model_accounting(
+ scope: str,
+ request_alias: bool,
+ reservation_released: bool,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setenv("LITELLM_TPM_TOKEN_RESERVATION_ENABLED", str(reservation_released).lower())
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(
+ internal_usage_cache=InternalUsageCache(cache),
+ time_provider=lambda: datetime(2026, 1, 1, tzinfo=timezone.utc),
+ )
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ model: Final = alias if request_alias else target
+ limits: Final = {"model_tpm_limit": {alias: 10000, target: 10000}}
+ auth: Final = UserAPIKeyAuth(
+ api_key="failed-latest-key",
+ team_id="failed-latest-team",
+ project_id="failed-latest-project",
+ org_id="failed-latest-org",
+ tpm_limit=10000,
+ metadata=limits if scope == "key" else {},
+ team_metadata=limits if scope == "team" else {},
+ project_metadata=limits if scope == "project" else {},
+ organization_metadata=limits if scope == "organization" else {},
+ )
+ request_data: Final = {"model": model, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}
+ await handler.async_pre_call_hook(auth, cache, request_data, "acompletion")
+ if reservation_released:
+ await handler.async_post_call_failure_hook(
+ request_data=request_data,
+ original_exception=Exception("rejected"),
+ user_api_key_dict=auth,
+ )
+ entity: Final = {
+ "key": auth.api_key,
+ "team": auth.team_id,
+ "project": auth.project_id,
+ "organization": auth.org_id,
+ }[scope]
+ models: Final = (alias, target) if request_alias else (target,)
+ keys: Final = (
+ _counter("api_key", auth.api_key, "tokens"),
+ *(_counter(f"model_per_{scope}", f"{entity}:{name}", "tokens") for name in models),
+ )
+ await handler.async_log_failure_event(
+ {
+ "standard_logging_object": {
+ "model_group": model,
+ "metadata": {
+ "user_api_key_hash": auth.api_key,
+ "user_api_key_team_id": auth.team_id,
+ "user_api_key_project_id": auth.project_id,
+ "user_api_key_org_id": auth.org_id,
+ },
+ },
+ "litellm_params": {"metadata": {"model_group": model, "user_api_key_metadata": auth.metadata}},
+ "combined_usage_object": Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5),
+ },
+ None,
+ None,
+ None,
+ )
+ for key in keys:
+ assert int(await handler.rate_limits.get(key) or 0) == (0 if reservation_released else 1)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("metric", ("model_itpm_limit", "model_otpm_limit"))
+@pytest.mark.parametrize("limit_alias", (False, True))
+async def test_latest_alias_project_io_limits_block_either_name(metric: str, limit_alias: bool) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-io-key",
+ project_id="latest-io-project",
+ project_metadata={metric: {alias if limit_alias else target: 0}},
+ )
+ with pytest.raises(ProxyRateLimitError) as exc:
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1}, "acompletion"
+ )
+ assert exc.value.status_code == 429
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("metric", ("model_itpm_limit", "model_otpm_limit"))
+@pytest.mark.parametrize("model", ("anthropic/claude-opus-5-5", "anthropic/unknown-model", None))
+async def test_latest_alias_project_io_limits_leave_other_requests_unchanged(metric: str, model: str | None) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-io-other-key",
+ project_id="latest-io-other-project",
+ project_metadata={metric: {"anthropic/claude-opus-latest": 0}},
+ )
+ assert await handler.async_pre_call_hook(auth, cache, {"model": model}, "acompletion") is None
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("event", ("success", "failure", "partial"))
+async def test_latest_alias_project_io_reservations_settle_both_counters(event: str) -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(
+ internal_usage_cache=InternalUsageCache(cache),
+ time_provider=lambda: datetime(2026, 1, 1, tzinfo=timezone.utc),
+ )
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-io-settle-key",
+ project_id="latest-io-settle-project",
+ project_metadata={metric: {alias: 10000, target: 10000} for metric in ("model_itpm_limit", "model_otpm_limit")},
+ )
+ await handler.async_pre_call_hook(
+ auth, cache, {"model": alias, "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}, "acompletion"
+ )
+ stash: Final = get_request_stash()
+ assert stash is not None
+ for scope, reserved in (
+ ("model_per_project_itpm", stash.itpm_reserved_tokens),
+ ("model_per_project_otpm", stash.otpm_reserved_tokens),
+ ):
+ assert reserved > 0
+ for model in (alias, target):
+ counter: Final = _counter(scope, f"{auth.project_id}:{model}", "tokens")
+ assert await handler.rate_limits.get(counter) == reserved
+ usage: Final = Usage(prompt_tokens=3, completion_tokens=2, total_tokens=5, prompt_tokens_details={"cached_tokens": 1})
+ kwargs: Final = {
+ "standard_logging_object": {
+ "model_group": alias,
+ "metadata": {"user_api_key_hash": auth.api_key, "user_api_key_project_id": auth.project_id},
+ },
+ "litellm_params": {"metadata": {"model_group": alias}},
+ **({"combined_usage_object": usage} if event == "partial" else {}),
+ }
+ if event == "success":
+ await handler.async_log_success_event(kwargs, ModelResponse(model=target, usage=usage), None, None)
+ else:
+ await handler.async_log_failure_event(kwargs, None, None, None)
+ for scope in ("model_per_project_itpm", "model_per_project_otpm"):
+ for model in (alias, target):
+ counter: Final = _counter(scope, f"{auth.project_id}:{model}", "tokens")
+ assert int(await handler.rate_limits.get(counter) or 0) == (0 if event == "failure" else 2)
+
+
+@pytest.mark.asyncio
+async def test_latest_alias_project_io_frame_reservations_charge_both_names() -> None:
+ cache: Final = DualCache()
+ handler: Final = _PROXY_MaxParallelRequestsHandler(internal_usage_cache=InternalUsageCache(cache))
+ alias: Final = "anthropic/claude-opus-latest"
+ target: Final = f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}"
+ auth: Final = UserAPIKeyAuth(
+ api_key="latest-frame-key",
+ project_id="latest-frame-project",
+ project_metadata={metric: {alias: 20, target: 20} for metric in ("model_itpm_limit", "model_otpm_limit")},
+ )
+ await handler.enforce_project_io_token_quota_for_frame(auth, alias, 3, 2)
+ for scope, expected in (("model_per_project_itpm", 3), ("model_per_project_otpm", 2)):
+ for model in (alias, target):
+ counter: Final = _counter(scope, f"{auth.project_id}:{model}", "tokens")
+ assert await handler.rate_limits.get(counter) == expected
+
+
@pytest.mark.parametrize(
"throttle_pct, expected_rpm, expected_tpm",
[
diff --git a/tests/unit/proxy/management_endpoints/test_internal_user_endpoints.py b/tests/unit/proxy/management_endpoints/test_internal_user_endpoints.py
index b96a5b606eb..120cf8a2b84 100644
--- a/tests/unit/proxy/management_endpoints/test_internal_user_endpoints.py
+++ b/tests/unit/proxy/management_endpoints/test_internal_user_endpoints.py
@@ -5483,3 +5483,16 @@ async def test_user_update_reports_a_bad_budget_duration_as_a_request_error_not_
assert (exc_info.value.code, exc_info.value.type) == ("400", "invalid_request_error")
assert "Invalid budget_duration" in exc_info.value.message
assert "Authentication Error" not in exc_info.value.message
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_user_update_hashes_off_the_event_loop(pbkdf2_on_event_loop: list[bool]) -> None:
+ from litellm.proxy.management_endpoints.internal_user_endpoints import _hash_password_in_dict
+
+ data: Final = {"password": "Str0ng!Passw0rd"}
+
+ await _hash_password_in_dict(data, {"password_policy_check_breached_passwords": False})
+
+ assert pbkdf2_on_event_loop == [False]
+ assert data["password"].startswith("pbkdf2:")
diff --git a/tests/unit/proxy/management_endpoints/test_password_endpoints.py b/tests/unit/proxy/management_endpoints/test_password_endpoints.py
index adff4eda47c..d1e9f80129b 100644
--- a/tests/unit/proxy/management_endpoints/test_password_endpoints.py
+++ b/tests/unit/proxy/management_endpoints/test_password_endpoints.py
@@ -490,3 +490,30 @@ async def test_change_password_requires_db():
)
assert exc_info.value.status_code == 500
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_password_change_runs_both_pbkdf2_off_the_event_loop(pbkdf2_on_event_loop: list[bool]) -> None:
+ from litellm.proxy._types import ChangePasswordRequest
+
+ prisma: Final = _make_prisma(
+ _make_user_row("pbkdf2:AAECAwQFBgcICQoLDA0OD+uxhNqRo/3ZFLEaKARqFBT+N3Kqek/SGM/5AvocSN0T")
+ )
+
+ with (
+ patch( # test-quality-ok: change_password reads proxy_server module globals; no injection seam
+ "litellm.proxy.proxy_server.prisma_client", prisma
+ ),
+ patch( # test-quality-ok: change_password reads proxy_server module globals; no injection seam
+ "litellm.proxy.proxy_server.general_settings", _POLICY_NO_BREACH_CHECK
+ ),
+ ):
+ await change_password(
+ data=ChangePasswordRequest(current_password="Seed-Lamp-4-River", new_password=NEW_PASSWORD),
+ user_api_key_dict=_caller(),
+ hibp_client=_hibp_client_never_called(),
+ )
+
+ assert pbkdf2_on_event_loop == [False, False]
+ assert prisma.db.litellm_usertable.update.call_args.kwargs["data"]["password"].startswith("pbkdf2:")
diff --git a/tests/unit/proxy/management_endpoints/test_tag_management_endpoints.py b/tests/unit/proxy/management_endpoints/test_tag_management_endpoints.py
index 531bf98b60e..813875bce7c 100644
--- a/tests/unit/proxy/management_endpoints/test_tag_management_endpoints.py
+++ b/tests/unit/proxy/management_endpoints/test_tag_management_endpoints.py
@@ -16,6 +16,7 @@ from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.proxy_server import app
from litellm.proxy.utils import PrismaClient
from litellm.types.tag_management import TagNewRequest
+from tests.unit.redis.stores.spend_counters_double import FakeSpendCounters
client = TestClient(app)
@@ -230,6 +231,138 @@ async def test_update_tag():
app.dependency_overrides.clear()
+@pytest.mark.asyncio
+@pytest.mark.parametrize("spend_fields", [{"spend": 0}, {}, {"spend": None}])
+async def test_update_tag_changes_spend_and_counter_only_when_provided(
+ spend_fields: Mapping[str, float | None],
+) -> None:
+ from litellm.proxy.management_endpoints.tag_management_endpoints import update_tag
+ from litellm.types.tag_management import TagUpdateRequest
+
+ updated_tag: Final = SimpleNamespace(
+ tag_name="reset-tag",
+ description=None,
+ models=[],
+ budget_id=None,
+ created_at=datetime(2024, 1, 1),
+ updated_at=datetime(2024, 1, 1),
+ created_by="admin",
+ )
+ update_row: Final = AsyncMock(return_value=updated_tag)
+ spend: Final = FakeSpendCounters(redis=None)
+ mock_prisma: Final = SimpleNamespace(
+ db=SimpleNamespace(
+ litellm_tagtable=SimpleNamespace(
+ find_unique=AsyncMock(return_value=updated_tag), update=update_row
+ ),
+ litellm_proxymodeltable=SimpleNamespace(find_many=AsyncMock(return_value=[])),
+ )
+ )
+ with (
+ patch("litellm.proxy.proxy_server.prisma_client", mock_prisma),
+ _tag_cache_doubles(),
+ patch("litellm.proxy.proxy_server.invalidate_spend_counter", new_callable=AsyncMock) as invalidate,
+ patch("litellm.proxy.proxy_server.spend_counter_cache") as counter_cache,
+ patch("litellm.proxy.proxy_server.spend_counter_store", spend.store()),
+ ):
+ counter_cache.redis_cache = None
+ response: Final = await update_tag(
+ tag=TagUpdateRequest.model_validate({"name": "reset-tag", **spend_fields}),
+ user_api_key_dict=UserAPIKeyAuth(user_id="admin", user_role=LitellmUserRoles.PROXY_ADMIN),
+ )
+
+ assert response["message"] == "Tag reset-tag updated successfully"
+ update_row.assert_awaited_once_with(
+ where={"tag_name": "reset-tag"},
+ data={
+ "description": None,
+ "models": [],
+ "model_info": "{}",
+ **({"spend": spend_fields["spend"]} if spend_fields.get("spend") is not None else {}),
+ },
+ )
+ if spend_fields.get("spend") is not None:
+ assert spend.memory["spend:tag:reset-tag"] == 0
+ else:
+ assert "spend:tag:reset-tag" not in spend.memory
+ counter_cache.in_memory_cache.set_cache.assert_not_called()
+ invalidate.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("spend", [float("nan"), float("inf"), float("-inf")])
+async def test_update_tag_rejects_non_finite_spend_before_writing(spend: float) -> None:
+ from litellm.proxy.management_endpoints.tag_management_endpoints import update_tag
+ from litellm.types.tag_management import TagUpdateRequest
+
+ update_row: Final = AsyncMock()
+ mock_prisma: Final = SimpleNamespace(
+ db=SimpleNamespace(litellm_tagtable=SimpleNamespace(update=update_row))
+ )
+ with (
+ patch("litellm.proxy.proxy_server.prisma_client", mock_prisma),
+ patch("litellm.proxy.proxy_server.invalidate_spend_counter", new_callable=AsyncMock) as invalidate,
+ pytest.raises(HTTPException) as error,
+ ):
+ await update_tag(
+ tag=TagUpdateRequest(name="reset-tag", spend=spend),
+ user_api_key_dict=UserAPIKeyAuth(user_id="admin", user_role=LitellmUserRoles.PROXY_ADMIN),
+ )
+ assert error.value.status_code == 400
+ assert "spend must be a finite number" in str(error.value.detail)
+ update_row.assert_not_awaited()
+ invalidate.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("redis_state", ["available", "absent", "error"])
+async def test_update_tag_overwrites_stale_counter_repopulated_during_tag_cache_eviction(redis_state: str) -> None:
+ from litellm.proxy.management_endpoints.tag_management_endpoints import update_tag
+ from litellm.types.tag_management import TagUpdateRequest
+
+ counter_key: Final = "spend:tag:overlap-reset-tag"
+ spend: Final = FakeSpendCounters(redis=None if redis_state == "absent" else {}, fail=redis_state == "error")
+ updated_tag: Final = SimpleNamespace(
+ tag_name="overlap-reset-tag",
+ description=None,
+ models=[],
+ budget_id=None,
+ created_at=datetime(2024, 1, 1),
+ updated_at=datetime(2024, 1, 1),
+ created_by="admin",
+ )
+ mock_prisma: Final = SimpleNamespace(
+ db=SimpleNamespace(
+ litellm_tagtable=SimpleNamespace(
+ find_unique=AsyncMock(return_value=updated_tag), update=AsyncMock(return_value=updated_tag)
+ ),
+ litellm_proxymodeltable=SimpleNamespace(find_many=AsyncMock(return_value=[])),
+ )
+ )
+
+ async def overlapping_completion(cache_keys: Sequence[str]) -> None:
+ spend.set_local(counter_key, 1.5, ttl_seconds=60)
+
+ with (
+ patch("litellm.proxy.proxy_server.prisma_client", mock_prisma),
+ patch(
+ "litellm.proxy.proxy_server.spend_counter_store", spend.store(),
+ ),
+ patch(
+ "litellm.proxy.management_endpoints.tag_management_endpoints._evict_tag_cache_keys",
+ overlapping_completion,
+ ),
+ ):
+ await update_tag(
+ tag=TagUpdateRequest(name="overlap-reset-tag", spend=0),
+ user_api_key_dict=UserAPIKeyAuth(user_id="admin", user_role=LitellmUserRoles.PROXY_ADMIN),
+ )
+ assert spend.memory[counter_key] == 0
+ if redis_state == "available":
+ assert spend.redis is not None
+ assert spend.redis[counter_key] == 0
+
+
@pytest.mark.asyncio
async def test_new_tag_persists_a_budget():
from datetime import datetime
diff --git a/tests/unit/proxy/management_helpers/test_management_helpers_utils.py b/tests/unit/proxy/management_helpers/test_management_helpers_utils.py
index 644f0656c7a..640e17aa58e 100644
--- a/tests/unit/proxy/management_helpers/test_management_helpers_utils.py
+++ b/tests/unit/proxy/management_helpers/test_management_helpers_utils.py
@@ -249,6 +249,35 @@ async def test_add_new_member_links_default_team_budget_id():
assert create_data["budget_id"] == test_default_budget_id
+@pytest.mark.asyncio
+async def test_handle_budget_for_entity_updates_with_existing_budget_id_in_request() -> None:
+ from litellm.proxy.management_helpers.utils import handle_budget_for_entity
+ from litellm.types.tag_management import TagUpdateRequest
+
+ budget_update: Final = AsyncMock()
+ prisma_client: Final = cast(
+ PrismaClient,
+ SimpleNamespace(db=SimpleNamespace(litellm_budgettable=SimpleNamespace(update=budget_update))),
+ )
+ with (
+ patch("litellm.proxy.proxy_server.prisma_client", prisma_client),
+ patch("litellm.proxy.proxy_server.litellm_proxy_admin_name", "admin"),
+ ):
+ budget_id: Final = await handle_budget_for_entity(
+ data=TagUpdateRequest(name="budget-tag", budget_id="budget-1", max_budget=20),
+ existing_budget_id="budget-1",
+ user_api_key_dict=UserAPIKeyAuth(user_id="admin"),
+ prisma_client=prisma_client,
+ litellm_proxy_admin_name="admin",
+ )
+
+ assert budget_id == "budget-1"
+ budget_update.assert_awaited_once_with(
+ where={"budget_id": "budget-1"},
+ data={"budget_id": "budget-1", "max_budget": 20.0, "updated_by": "admin"},
+ )
+
+
@pytest.mark.parametrize(
("request_fields", "cleared_budget_fields", "should_update", "expected_max_budget"),
cast(
diff --git a/tests/unit/proxy/pass_through_endpoints/test_codex_passthrough_endpoints.py b/tests/unit/proxy/pass_through_endpoints/test_codex_passthrough_endpoints.py
index 354c65776dc..1a4751b4598 100644
--- a/tests/unit/proxy/pass_through_endpoints/test_codex_passthrough_endpoints.py
+++ b/tests/unit/proxy/pass_through_endpoints/test_codex_passthrough_endpoints.py
@@ -1,17 +1,39 @@
import asyncio
+import base64
import json
-from collections.abc import Iterator
+import time
+from collections.abc import Callable, Iterator, Sequence
from datetime import datetime
from typing import Final
import httpx
+import jwt
import pytest
import respx
+from cryptography.hazmat.primitives.asymmetric import rsa
from fastapi.testclient import TestClient
+from jwt.algorithms import RSAAlgorithm
import litellm
+from litellm import Router
+from litellm.caching.dual_cache import DualCache
from litellm.integrations.custom_logger import CustomLogger
+from litellm.llms.chatgpt.common_utils import OPENAI_AUTH_CLAIM_KEY
from litellm.proxy import proxy_server
+from litellm.proxy._types import LiteLLM_JWTAuth, LiteLLM_TeamTableCachedObj, LitellmUserRoles, UserAPIKeyAuth
+from litellm.proxy.auth.handle_jwt import JWTHandler
+from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
+from litellm.proxy.common_utils.codex_model_catalog import bundled_codex_models
+from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import (
+ _ACCOUNT_SLUGS_FETCHES_PER_CALLER,
+ codex_shared_cache,
+)
+from litellm.proxy.utils import hash_token
+from litellm.responses.litellm_completion_transformation.reasoning_items import (
+ encode_thinking_blocks,
+ mint_reasoning_item_id,
+)
+from litellm.responses.utils import ResponsesAPIRequestUtils
from tests._master_key import MASTER_KEY
_BASE: Final = "https://chatgpt.com/backend-api/codex"
@@ -22,6 +44,7 @@ _HEADERS: Final = {"authorization": f"Bearer {_TOKEN}", "x-litellm-api-key": MAS
@pytest.fixture
def client(monkeypatch: pytest.MonkeyPatch, httpx_transport: None) -> Iterator[TestClient]:
monkeypatch.setattr(proxy_server, "master_key", MASTER_KEY)
+ monkeypatch.setattr(proxy_server.proxy_logging_obj.internal_usage_cache, "dual_cache", DualCache())
yield TestClient(proxy_server.app)
@@ -183,3 +206,1860 @@ async def test_usage_logged_without_credentials(
serialized: Final = json.dumps(logged, default=str)
for secret in (_TOKEN, MASTER_KEY, "test-hidden-token", "test-hidden-cookie"):
assert secret not in serialized
+
+
+_PROVIDER_BASE: Final = "https://open-weights.test/v1"
+_CHATGPT_JWT: Final = (
+ ".".join(
+ base64.urlsafe_b64encode(json.dumps(part).encode()).decode().rstrip("=")
+ for part in ({"alg": "none"}, {OPENAI_AUTH_CLAIM_KEY: {"chatgpt_account_id": "account-a"}})
+ )
+ + "."
+)
+_PROVIDER_KEY: Final = "test-provider-key"
+_CODEX_USER_AGENT: Final = "codex_cli_rs/0.165.0 (Ubuntu 24.4.0; x86_64) xterm-256color"
+_ACCOUNT_HEADERS: Final = {
+ **_HEADERS,
+ "authorization": f"Bearer {_CHATGPT_JWT}",
+ "chatgpt-account-id": "account-a",
+ "originator": "codex_cli_rs",
+ "user-agent": _CODEX_USER_AGENT,
+}
+_BUNDLED_SLUG: Final = next(iter(bundled_codex_models()))
+_PRICED_CHATGPT_SLUG: Final = "priced-subscription-model"
+_UPSTREAM_CATALOG: Final = {
+ "models": [
+ {"slug": "subscription-model", "priority": 0, "supported_reasoning_levels": ["high"]},
+ {"slug": "shared-name", "priority": 7, "visibility": "list"},
+ {"slug": "subscription-hidden", "priority": 3, "visibility": "hide"},
+ ]
+}
+
+
+def _open_weights(model_name: str, mode: str | None = None, upstream: str | None = None) -> dict[str, object]:
+ return {
+ "model_name": model_name,
+ "litellm_params": {
+ "model": f"fireworks_ai/{upstream or model_name}",
+ "api_base": _PROVIDER_BASE,
+ "api_key": _PROVIDER_KEY,
+ },
+ "model_info": {"mode": mode} if mode else {},
+ }
+
+
+@pytest.fixture
+def mixed_router(monkeypatch: pytest.MonkeyPatch) -> Router:
+ router: Final = Router(
+ model_list=[
+ {
+ "model_name": "chatgpt/gpt-*",
+ "litellm_params": {"model": "chatgpt/gpt-*"},
+ "model_info": {"access_groups": ["chatgpt-native"]},
+ },
+ {
+ "model_name": "chatgpt/codex-*",
+ "litellm_params": {"model": "chatgpt/codex-*"},
+ "model_info": {"access_groups": ["chatgpt-native"]},
+ },
+ {"model_name": "chatgpt-subscription", "litellm_params": {"model": "chatgpt/subscription-model"}},
+ _open_weights("open-weights"),
+ _open_weights("shared-name"),
+ _open_weights(_BUNDLED_SLUG),
+ _open_weights(_PRICED_CHATGPT_SLUG),
+ _open_weights("open-embedder", mode="embedding"),
+ _open_weights("*"),
+ ]
+ )
+ monkeypatch.setattr(proxy_server, "llm_router", router)
+ unpriced_bundled: Final = {
+ name: price for name, price in litellm.model_cost.items() if name != f"chatgpt/{_BUNDLED_SLUG}"
+ }
+ monkeypatch.setattr(
+ litellm,
+ "model_cost",
+ {**unpriced_bundled, f"chatgpt/{_PRICED_CHATGPT_SLUG}": {"litellm_provider": "chatgpt", "mode": "responses"}},
+ )
+ return router
+
+
+def test_native_catalog_lists_the_keys_other_chat_models_after_chatgpts(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": '"upstream-v1"'})
+ response: Final = client.get("/backend-api/codex/models?client_version=1.0.0", headers=_HEADERS)
+ assert response.status_code == 200
+ models: Final = response.json()["models"]
+ assert models[: len(_UPSTREAM_CATALOG["models"])] == _UPSTREAM_CATALOG["models"]
+ added: Final = models[len(_UPSTREAM_CATALOG["models"]) :]
+ assert [model["slug"] for model in added] == ["open-weights"]
+ assert added[0]["visibility"] == "list"
+ assert added[0]["priority"] > max(model["priority"] for model in _UPSTREAM_CATALOG["models"])
+
+
+@pytest.mark.parametrize(
+ ("client_version", "expected_efforts"),
+ [
+ pytest.param("0.165.0", ["low", "max"], id="a codex that parses custom efforts gets them all"),
+ pytest.param("0.137.0", ["low"], id="an older codex gets only the efforts it parses"),
+ ],
+)
+def test_native_catalog_offers_the_declared_reasoning_efforts_the_requesting_codex_parses(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ monkeypatch: pytest.MonkeyPatch,
+ client_version: str,
+ expected_efforts: list[str],
+) -> None:
+ declared: Final = {
+ **_open_weights("open-weights"),
+ "model_info": {"supports_reasoning": True, "reasoning_effort_levels": ["low", "max"]},
+ }
+ monkeypatch.setattr(proxy_server, "llm_router", Router(model_list=[declared]))
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": '"upstream-v1"'})
+ response: Final = client.get(f"/backend-api/codex/models?client_version={client_version}", headers=_HEADERS)
+ assert response.status_code == 200
+ (added,) = response.json()["models"][len(_UPSTREAM_CATALOG["models"]) :]
+ assert [level["effort"] for level in added["supported_reasoning_levels"]] == expected_efforts
+
+
+@pytest.mark.parametrize(
+ "upstream",
+ [
+ pytest.param(httpx.Response(503, json={"detail": "busy"}, headers={"etag": '"e"'}), id="non-200"),
+ pytest.param(httpx.Response(200, content=b"login", headers={"etag": '"e"'}), id="not json"),
+ pytest.param(httpx.Response(200, json={"models": "none"}, headers={"etag": '"e"'}), id="no model list"),
+ ],
+)
+def test_native_catalog_relays_an_unmergeable_upstream_answer_unchanged(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, upstream: httpx.Response
+) -> None:
+ respx_mock.get(f"{_BASE}/models").mock(return_value=upstream)
+ response: Final = client.get("/backend-api/codex/models?client_version=1.0.0", headers=_HEADERS)
+ assert response.status_code == upstream.status_code
+ assert response.content == upstream.content
+ assert response.headers["etag"] == '"e"'
+
+
+def _provider_completed(text: str) -> dict[str, object]:
+ return {
+ "id": "resp_open_weights",
+ "object": "response",
+ "created_at": 1700000000,
+ "status": "completed",
+ "model": "open-weights",
+ "output": [
+ {
+ "type": "message",
+ "id": "msg_open_weights",
+ "role": "assistant",
+ "status": "completed",
+ "content": [{"type": "output_text", "text": text, "annotations": []}],
+ }
+ ],
+ "usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
+ }
+
+
+def _provider_stream(text: str) -> bytes:
+ completed: Final = _provider_completed(text)
+ events: Final = (
+ {
+ "type": "response.created",
+ "sequence_number": 0,
+ "response": {**completed, "status": "in_progress", "output": []},
+ },
+ {
+ "type": "response.output_text.delta",
+ "sequence_number": 1,
+ "item_id": "msg_open_weights",
+ "output_index": 0,
+ "content_index": 0,
+ "delta": text,
+ },
+ {"type": "response.completed", "sequence_number": 2, "response": completed},
+ )
+ return "".join(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n" for event in events).encode()
+
+
+def _event_types(stream: bytes) -> list[str]:
+ payloads: Final = [line.removeprefix(b"data: ") for line in stream.splitlines() if line.startswith(b"data: {")]
+ return [json.loads(payload)["type"] for payload in payloads]
+
+
+class PreCallRecorder(CustomLogger):
+ def __init__(self) -> None:
+ super().__init__()
+ self.requests: Final[list[tuple[str | None, str]]] = []
+ self.raw_header_names: Final[list[frozenset[str]]] = []
+
+ async def async_pre_call_hook(
+ self, user_api_key_dict: UserAPIKeyAuth, cache: object, data: dict[str, object], call_type: str
+ ) -> dict[str, object]:
+ self.requests.append((user_api_key_dict.request_route, repr(data)))
+ secret_fields: Final = data.get("secret_fields")
+ raw_headers: Final = secret_fields.get("raw_headers") if isinstance(secret_fields, dict) else None
+ self.raw_header_names.append(frozenset(raw_headers) if isinstance(raw_headers, dict) else frozenset())
+ return data
+
+
+def _provider_reply(request: httpx.Request) -> httpx.Response:
+ if json.loads(request.content).get("stream") is not True:
+ return httpx.Response(200, json=_provider_completed("Hi"))
+ return httpx.Response(200, content=_provider_stream("Hi"), headers={"content-type": "text/event-stream"})
+
+
+def _open_weights_provider(respx_mock: respx.MockRouter) -> respx.Route:
+ return respx_mock.post(f"{_PROVIDER_BASE}/responses").mock(side_effect=_provider_reply)
+
+
+_UPSTREAM_MODELS_ETAG: Final = '"upstream-v2"'
+
+
+def _chatgpt_responses(respx_mock: respx.MockRouter) -> respx.Route:
+ return respx_mock.post(f"{_BASE}/responses").respond(
+ content=b"event: response.completed\ndata: {}\n\n",
+ headers={"content-type": "text/event-stream", "x-models-etag": _UPSTREAM_MODELS_ETAG},
+ )
+
+
+def _account_catalog(respx_mock: respx.MockRouter) -> respx.Route:
+ return respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG)
+
+
+def test_litellm_model_on_native_route_streams_like_v1_without_chatgpt_credentials(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ recorder: Final = PreCallRecorder()
+ monkeypatch.setattr(litellm, "callbacks", [recorder])
+ _account_catalog(respx_mock)
+ chatgpt: Final = respx_mock.post(f"{_BASE}/responses").respond(json={})
+ provider: Final = respx_mock.post(f"{_PROVIDER_BASE}/responses").mock(
+ side_effect=lambda _: httpx.Response(
+ 200, content=_provider_stream("Hi"), headers={"content-type": "text/event-stream"}
+ )
+ )
+ body: Final = {"model": "open-weights", "input": "Say hi", "stream": True, "store": False}
+ native: Final = client.post("/backend-api/codex/responses", json=body, headers=_ACCOUNT_HEADERS)
+ v1: Final = client.post("/v1/responses", json=body, headers={"authorization": f"Bearer {MASTER_KEY}"})
+ assert native.status_code == v1.status_code == 200
+ assert native.headers["content-type"].startswith("text/event-stream")
+ assert _event_types(native.content) == _event_types(v1.content)
+ assert "response.completed" in _event_types(native.content)
+ assert not chatgpt.called
+ provider_headers: Final = provider.calls[0].request.headers
+ assert provider_headers["authorization"] == f"Bearer {_PROVIDER_KEY}"
+ assert "chatgpt-account-id" not in provider_headers
+ assert all(_CHATGPT_JWT not in value for value in provider_headers.values())
+ (native_route, native_call), (v1_route, _) = recorder.requests
+ assert native_route == v1_route == "/v1/responses"
+ assert _CHATGPT_JWT not in native_call
+ assert "account-a" not in native_call
+
+
+def test_litellm_model_on_native_route_keeps_chatgpt_credentials_out_of_hook_raw_headers(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ recorder: Final = PreCallRecorder()
+ monkeypatch.setattr(litellm, "callbacks", [recorder])
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Say hi", "stream": True, "store": False},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200
+ assert provider.called
+ (raw_header_names,) = recorder.raw_header_names
+ assert "x-litellm-api-key" in raw_header_names
+ assert not raw_header_names & {"authorization", "chatgpt-account-id"}, sorted(raw_header_names)
+
+
+@pytest.mark.parametrize(
+ ("model", "sent", "drop_params", "expected"),
+ [
+ pytest.param("open-weights", "none", None, "none", id="none without drop_params"),
+ pytest.param("open-weights", "none", "global", None, id="none under litellm_settings drop_params"),
+ pytest.param("open-weights", "none", "deployment", None, id="none under the deployment's drop_params"),
+ pytest.param("open-weights", "none", "request", None, id="none under the request's drop_params"),
+ pytest.param("open-weights", "high", "global", "high", id="effort set for a model without a default effort"),
+ pytest.param("coding-model", "none", "global", None, id="none on an alias whose native entry names no effort"),
+ pytest.param("api-model", "none", "global", "none", id="none on a model that supports it"),
+ pytest.param("glm-model", "none", "global", None, id="none on a model that does not support it"),
+ pytest.param("declared-model", "none", "global", "none", id="none on a deployment whose model_info allows it"),
+ ],
+)
+def test_litellm_model_on_native_route_gets_the_reasoning_effort_codex_m_would_send(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ monkeypatch: pytest.MonkeyPatch,
+ model: str,
+ sent: str,
+ drop_params: str | None,
+ expected: str | None,
+) -> None:
+ monkeypatch.setattr(litellm, "drop_params", drop_params == "global")
+ open_weights: Final = _open_weights("open-weights")
+ monkeypatch.setattr(
+ proxy_server,
+ "llm_router",
+ Router(
+ model_list=[
+ {
+ **open_weights,
+ "litellm_params": {**open_weights["litellm_params"], "drop_params": drop_params == "deployment"},
+ },
+ {
+ **_open_weights("coding-model", upstream="gpt-5.5"),
+ "model_info": {"supports_none_reasoning_effort": False},
+ },
+ _open_weights("glm-model", upstream="glm-5p3"),
+ {**_open_weights("declared-model"), "model_info": {"supports_none_reasoning_effort": True}},
+ {
+ "model_name": "api-model",
+ "litellm_params": {"model": "openai/gpt-5.4", "api_base": _PROVIDER_BASE, "api_key": _PROVIDER_KEY},
+ },
+ ]
+ ),
+ )
+ _account_catalog(respx_mock)
+ provider: Final = respx_mock.post(f"{_PROVIDER_BASE}/responses").mock(
+ side_effect=lambda _: httpx.Response(
+ 200, content=_provider_stream("Hi"), headers={"content-type": "text/event-stream"}
+ )
+ )
+ body: Final = {
+ "model": model,
+ "input": "Say hi",
+ "stream": True,
+ "store": False,
+ **({"drop_params": True} if drop_params == "request" else {}),
+ }
+ expected_reasoning: Final = {"summary": "auto"} if expected is None else {"effort": expected, "summary": "auto"}
+ native: Final = client.post(
+ "/backend-api/codex/responses",
+ json={**body, "reasoning": {"effort": sent, "summary": "auto"}},
+ headers=_ACCOUNT_HEADERS,
+ )
+ v1: Final = client.post(
+ "/v1/responses",
+ json={**body, "reasoning": expected_reasoning},
+ headers={"authorization": f"Bearer {MASTER_KEY}"},
+ )
+ assert native.status_code == v1.status_code == 200
+ native_request, v1_request = (json.loads(call.request.content) for call in provider.calls)
+ assert native_request == v1_request
+ assert native_request["reasoning"].get("effort") == expected
+
+
+@pytest.mark.parametrize(
+ "model", ["gpt-unlisted", "chatgpt-subscription", _BUNDLED_SLUG, _PRICED_CHATGPT_SLUG, "newly-released"]
+)
+def test_chatgpt_model_keeps_passthrough_without_other_providers_reasoning(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, model: str
+) -> None:
+ provider: Final = respx_mock.post(f"{_PROVIDER_BASE}/responses").respond(json={})
+ chatgpt: Final = respx_mock.post(f"{_BASE}/responses").respond(
+ content=b"event: response.completed\ndata: {}\n\n",
+ headers={"content-type": "text/event-stream", "x-models-etag": '"upstream-v2"'},
+ )
+ user_turn: Final = {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Hello"}]}
+ chatgpt_reasoning: Final = {
+ "type": "reasoning",
+ "id": "rs_upstream",
+ "summary": [],
+ "encrypted_content": "opaque-upstream-blob",
+ }
+ assistant_turn: Final = {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "Hi"}]}
+ other_providers_reasoning: Final = [
+ {"type": "reasoning", "id": mint_reasoning_item_id(), "summary": []},
+ {
+ "type": "reasoning",
+ "summary": [],
+ "encrypted_content": encode_thinking_blocks([{"type": "thinking", "thinking": "t", "signature": "s"}]),
+ },
+ {"type": "reasoning", "summary": [], "content": [{"type": "reasoning_text", "text": "t"}]},
+ {
+ "type": "reasoning",
+ "summary": [],
+ "encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id("blob", "deployment"),
+ },
+ ]
+ body: Final = {
+ "model": model,
+ "input": [user_turn, *other_providers_reasoning, chatgpt_reasoning, assistant_turn],
+ "stream": True,
+ }
+ response: Final = client.post("/backend-api/codex/responses", json=body, headers=_HEADERS)
+ assert response.status_code == 200
+ assert not provider.called
+ assert json.loads(chatgpt.calls.last.request.content) == {
+ **body,
+ "input": [user_turn, chatgpt_reasoning, assistant_turn],
+ }
+
+
+@pytest.fixture
+def restricted_key(monkeypatch: pytest.MonkeyPatch) -> Callable[..., str]:
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+
+ def mint(
+ models: list[str],
+ allowed_routes: tuple[str, ...] = (),
+ user_role: LitellmUserRoles | None = None,
+ allowed_tools: tuple[str, ...] = (),
+ ) -> str:
+ key: Final = f"sk-test-{'-'.join((*models, *allowed_routes, str(user_role), *allowed_tools))}"
+ proxy_server.user_api_key_cache.set_cache(
+ key=hash_token(key),
+ value=UserAPIKeyAuth(
+ token=hash_token(key),
+ models=models,
+ allowed_routes=list(allowed_routes),
+ user_role=user_role,
+ metadata={"allowed_tools": list(allowed_tools)} if allowed_tools else {},
+ ),
+ )
+ return key
+
+ return mint
+
+
+@pytest.mark.parametrize(
+ ("models", "added"),
+ [
+ pytest.param(["chatgpt-native", "open-weights"], ["open-weights"], id="key with a LiteLLM model"),
+ pytest.param(["chatgpt-native"], [], id="key with ChatGPT models only"),
+ ],
+)
+def test_native_catalog_adds_only_the_litellm_models_the_key_can_call(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ models: list[str],
+ added: list[str],
+) -> None:
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": '"upstream-v1"'})
+ response: Final = client.get(
+ "/backend-api/codex/models?client_version=1.0.0",
+ headers={**_HEADERS, "x-litellm-api-key": restricted_key(models)},
+ )
+ assert response.status_code == 200
+ assert [model["slug"] for model in response.json()["models"]] == [
+ *(model["slug"] for model in _UPSTREAM_CATALOG["models"]),
+ *added,
+ ]
+ assert (response.headers["etag"] == '"upstream-v1"') is not added
+
+
+@pytest.mark.parametrize(
+ ("models", "model", "allowed"),
+ [
+ pytest.param(["chatgpt-native", "open-weights"], "gpt-unlisted", True, id="chatgpt access group"),
+ pytest.param(["chatgpt-subscription", "open-weights"], "gpt-unlisted", False, id="single subscription model"),
+ pytest.param(["chatgpt-native"], "open-weights", False, id="litellm model outside the key"),
+ ],
+)
+def test_native_route_serves_only_the_models_the_key_allows(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ models: list[str],
+ model: str,
+ allowed: bool,
+) -> None:
+ provider: Final = respx_mock.post(f"{_PROVIDER_BASE}/responses").respond(json={})
+ chatgpt: Final = respx_mock.post(f"{_BASE}/responses").respond(
+ content=b"event: response.completed\ndata: {}\n\n", headers={"content-type": "text/event-stream"}
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": model, "input": "Hello", "stream": True},
+ headers={**_HEADERS, "x-litellm-api-key": restricted_key(models)},
+ )
+ assert (response.status_code == 200) is allowed
+ assert chatgpt.called is allowed
+ assert not provider.called
+ assert [json.loads(call.request.content)["model"] for call in chatgpt.calls] == ([model] if allowed else [])
+
+
+def _key_with_alias_to_budgeted_target(monkeypatch: pytest.MonkeyPatch, target_budget: float) -> str:
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+ monkeypatch.setattr(
+ proxy_server, "llm_router", Router(model_list=[_open_weights("group-a"), _open_weights("group-b")])
+ )
+ key: Final = f"sk-test-alias-budget-{target_budget}"
+ proxy_server.user_api_key_cache.set_cache(
+ key=hash_token(key),
+ value=UserAPIKeyAuth(
+ token=hash_token(key),
+ models=["group-a", "group-b"],
+ router_settings={"model_group_alias": {"group-a": "group-b"}},
+ model_max_budget={"group-b": {"max_budget": target_budget, "budget_duration": "1d"}},
+ ),
+ )
+ return key
+
+
+def test_native_route_refuses_a_saved_alias_when_its_target_is_over_budget(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ key: Final = _key_with_alias_to_budgeted_target(monkeypatch, target_budget=0.0)
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "group-a", "input": "Hello"},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert response.json()["error"]["type"] == "budget_exceeded", response.text
+ assert "model=group-b" in response.json()["error"]["message"]
+ assert not provider.called
+
+
+def test_native_route_serves_a_saved_alias_target_within_budget_under_the_requested_name(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ key: Final = _key_with_alias_to_budgeted_target(monkeypatch, target_budget=5.0)
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "group-a", "input": "Hello"},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert response.status_code == 200, response.text
+ assert response.json()["model"] == "group-a"
+ assert json.loads(provider.calls.last.request.content)["model"].endswith("/group-b")
+
+
+def test_native_route_keeps_a_chatgpt_turn_on_the_subscription_despite_a_saved_alias(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+ key: Final = "sk-test-alias-on-a-chatgpt-name"
+ proxy_server.user_api_key_cache.set_cache(
+ key=hash_token(key),
+ value=UserAPIKeyAuth(
+ token=hash_token(key),
+ models=[_BUNDLED_SLUG, "open-weights"],
+ router_settings={"model_group_alias": {_BUNDLED_SLUG: "open-weights"}},
+ ),
+ )
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": _BUNDLED_SLUG, "input": "Hello", "stream": True},
+ headers={**_HEADERS, "x-litellm-api-key": key},
+ )
+ assert response.status_code == 200, response.text
+ assert not provider.called
+ assert [json.loads(call.request.content)["model"] for call in chatgpt.calls] == [_BUNDLED_SLUG]
+
+
+def test_native_route_keeps_a_name_the_account_lists_on_the_subscription_despite_a_saved_alias(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+ monkeypatch.setattr(
+ proxy_server, "llm_router", Router(model_list=[_open_weights("gpt-reserve"), _open_weights("group-b")])
+ )
+ key: Final = "sk-test-alias-on-a-name-the-account-lists"
+ proxy_server.user_api_key_cache.set_cache(
+ key=hash_token(key),
+ value=UserAPIKeyAuth(
+ token=hash_token(key),
+ models=["gpt-reserve", "group-b"],
+ router_settings={"model_group_alias": {"gpt-reserve": "group-b"}},
+ model_max_budget={"group-b": {"max_budget": 0.0, "budget_duration": "1d"}},
+ ),
+ )
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(json={"models": [{"slug": "gpt-reserve"}]})
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "gpt-reserve", "input": "Hello", "stream": True},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert response.status_code == 200, response.text
+ assert not provider.called
+ assert [json.loads(call.request.content)["model"] for call in chatgpt.calls] == ["gpt-reserve"]
+ assert catalog.call_count == 1
+
+
+_BEARER_KEY: Final = "sk-test-alias-key-sent-as-the-bearer"
+
+
+@pytest.mark.parametrize(
+ "gateway_header",
+ [{}, {"x-litellm-api-key": _BEARER_KEY}],
+ ids=["key only in the bearer", "same key in both headers"],
+)
+def test_native_route_never_sends_a_litellm_key_sent_as_the_bearer_to_chatgpt(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch, gateway_header: dict[str, str]
+) -> None:
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+ monkeypatch.setattr(
+ proxy_server, "llm_router", Router(model_list=[_open_weights("gpt-reserve"), _open_weights("group-b")])
+ )
+ proxy_server.user_api_key_cache.set_cache(
+ key=hash_token(_BEARER_KEY),
+ value=UserAPIKeyAuth(
+ token=hash_token(_BEARER_KEY),
+ models=["gpt-reserve", "group-b"],
+ router_settings={"model_group_alias": {"gpt-reserve": "group-b"}},
+ ),
+ )
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(json={"models": []})
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "gpt-reserve", "input": "Hello", "stream": True},
+ headers={
+ "authorization": f"Bearer {_BEARER_KEY}",
+ "chatgpt-account-id": "account-a",
+ "originator": "codex_cli_rs",
+ "user-agent": _CODEX_USER_AGENT,
+ **gateway_header,
+ },
+ )
+ assert response.status_code == 401, response.text
+ assert not catalog.called
+ assert not provider.called
+ assert not chatgpt.called
+
+
+def test_v1_request_for_a_gpt_name_still_reaches_the_catch_all_with_the_access_group_recipe(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/v1/responses",
+ json={"model": "gpt-unlisted", "input": "Hello", "stream": True},
+ headers={"authorization": f"Bearer {MASTER_KEY}"},
+ )
+ assert response.status_code == 200, response.text
+ assert provider.called
+ assert not chatgpt.called
+
+
+_NATIVE_ROUTES: Final = ("/backend-api/codex/models", "/backend-api/codex/responses")
+
+
+@pytest.mark.parametrize(
+ ("allowed_routes", "added"),
+ [
+ pytest.param(_NATIVE_ROUTES, [], id="key limited to the native routes"),
+ pytest.param((*_NATIVE_ROUTES, "/v1/responses"), ["open-weights"], id="key that may call /v1/responses"),
+ ],
+)
+def test_native_catalog_adds_litellm_models_only_for_a_key_that_may_call_v1_responses(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ allowed_routes: tuple[str, ...],
+ added: list[str],
+) -> None:
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG)
+ response: Final = client.get(
+ "/backend-api/codex/models?client_version=1.0.0",
+ headers={**_HEADERS, "x-litellm-api-key": restricted_key(["chatgpt-native", "open-weights"], allowed_routes)},
+ )
+ assert response.status_code == 200, response.text
+ assert [model["slug"] for model in response.json()["models"]] == [
+ *(model["slug"] for model in _UPSTREAM_CATALOG["models"]),
+ *added,
+ ]
+
+
+@pytest.mark.parametrize(
+ ("allowed_routes", "through_litellm"),
+ [
+ pytest.param(_NATIVE_ROUTES, False, id="key limited to the native routes"),
+ pytest.param((*_NATIVE_ROUTES, "/v1/responses"), True, id="key that may call /v1/responses"),
+ ],
+)
+def test_native_responses_reach_litellm_models_only_for_a_key_that_may_call_v1_responses(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ allowed_routes: tuple[str, ...],
+ through_litellm: bool,
+) -> None:
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Hello", "stream": True},
+ headers={
+ **_ACCOUNT_HEADERS,
+ "x-litellm-api-key": restricted_key(["chatgpt-native", "open-weights"], allowed_routes),
+ },
+ )
+ assert response.status_code == 200, response.text
+ assert provider.called is through_litellm
+ assert chatgpt.called is not through_litellm
+
+
+
+@pytest.mark.parametrize(
+ ("user_role", "through_litellm"),
+ [
+ pytest.param(LitellmUserRoles.INTERNAL_USER, False, id="non-admin key"),
+ pytest.param(LitellmUserRoles.PROXY_ADMIN, True, id="proxy admin key"),
+ ],
+)
+def test_admin_only_v1_responses_keeps_litellm_models_to_proxy_admins_on_native_routes(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ monkeypatch: pytest.MonkeyPatch,
+ user_role: LitellmUserRoles,
+ through_litellm: bool,
+) -> None:
+ monkeypatch.setattr(proxy_server, "general_settings", {"admin_only_routes": ["/v1/responses"]})
+ monkeypatch.setattr(proxy_server, "premium_user", True)
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ key: Final = restricted_key(["chatgpt-native", "open-weights"], user_role=user_role)
+ catalog: Final = client.get(
+ "/backend-api/codex/models?client_version=1.0.0", headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key}
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Hello", "stream": True},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert (catalog.status_code, response.status_code) == (200, 200), response.text
+ assert ("open-weights" in [model["slug"] for model in catalog.json()["models"]]) is through_litellm
+ assert provider.called is through_litellm
+ assert chatgpt.called is not through_litellm
+
+
+@pytest.mark.parametrize(
+ ("allowed_routes", "through_litellm"),
+ [
+ pytest.param(["/backend-api/codex/models", "/backend-api/codex/responses"], False, id="native routes only"),
+ pytest.param(
+ ["/backend-api/codex/models", "/backend-api/codex/responses", "/v1/responses"],
+ True,
+ id="v1 responses allowed too",
+ ),
+ ],
+)
+def test_proxy_allowed_routes_keep_litellm_models_off_native_routes_unless_v1_responses_is_allowed(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ monkeypatch: pytest.MonkeyPatch,
+ allowed_routes: list[str],
+ through_litellm: bool,
+) -> None:
+ monkeypatch.setattr(proxy_server, "general_settings", {"allowed_routes": allowed_routes})
+ monkeypatch.setattr(proxy_server, "premium_user", True)
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ key: Final = restricted_key(["chatgpt-native", "open-weights"])
+ catalog: Final = client.get(
+ "/backend-api/codex/models?client_version=1.0.0", headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key}
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Hello", "stream": True},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert (catalog.status_code, response.status_code) == (200, 200), response.text
+ assert ("open-weights" in [model["slug"] for model in catalog.json()["models"]]) is through_litellm
+ assert provider.called is through_litellm
+ assert chatgpt.called is not through_litellm
+
+
+_JWT_ISSUER: Final = "https://idp.codex-test.example"
+_JWT_JWKS_URL: Final = f"{_JWT_ISSUER}/keys"
+_JWT_AUDIENCE: Final = "litellm-proxy"
+_JWT_KID: Final = "codex-test-key"
+_JWT_TEAM: Final = "codex-team"
+
+
+def _team_jwt(monkeypatch: pytest.MonkeyPatch, team_allowed_routes: list[str]) -> str:
+ private_key: Final = rsa.generate_private_key(public_exponent=65537, key_size=2048)
+ public_jwk: Final = {
+ **json.loads(RSAAlgorithm.to_jwk(private_key.public_key())),
+ "kid": _JWT_KID,
+ "alg": "RS256",
+ "use": "sig",
+ }
+ keys_cache: Final = DualCache()
+ keys_cache.set_cache(key=f"litellm_jwt_auth_keys_{_JWT_JWKS_URL}", value=[public_jwk])
+ handler: Final = JWTHandler()
+ handler.update_environment(
+ prisma_client=None,
+ user_api_key_cache=keys_cache,
+ litellm_jwtauth=LiteLLM_JWTAuth(
+ issuers=[{"issuer": _JWT_ISSUER, "jwks_url": _JWT_JWKS_URL, "audience": _JWT_AUDIENCE}],
+ team_ids_jwt_field="groups",
+ team_allowed_routes=team_allowed_routes,
+ ),
+ )
+ monkeypatch.setattr(proxy_server, "jwt_handler", handler)
+ monkeypatch.setattr(proxy_server, "general_settings", {"enable_jwt_auth": True})
+ monkeypatch.setattr(proxy_server, "premium_user", True)
+ monkeypatch.setattr(proxy_server, "prisma_client", "test-database")
+ proxy_server.user_api_key_cache.set_cache(
+ key=f"team_id:{_JWT_TEAM}",
+ value=LiteLLM_TeamTableCachedObj(team_id=_JWT_TEAM, models=["chatgpt-native", "open-weights"]),
+ )
+ issued_at: Final = int(time.time())
+ return jwt.encode(
+ {
+ "sub": "codex-user",
+ "iss": _JWT_ISSUER,
+ "aud": _JWT_AUDIENCE,
+ "iat": issued_at,
+ "exp": issued_at + 300,
+ "groups": [_JWT_TEAM],
+ },
+ private_key,
+ algorithm="RS256",
+ headers={"kid": _JWT_KID},
+ )
+
+
+@pytest.mark.parametrize(
+ ("team_allowed_routes", "through_litellm"),
+ [
+ pytest.param(["/backend-api/codex/*"], False, id="team allowed only the native routes"),
+ pytest.param(["/backend-api/codex/*", "openai_routes"], True, id="team allowed openai routes too"),
+ ],
+)
+def test_jwt_team_routes_keep_litellm_models_off_native_routes_unless_v1_responses_is_allowed(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ monkeypatch: pytest.MonkeyPatch,
+ team_allowed_routes: list[str],
+ through_litellm: bool,
+) -> None:
+ token: Final = _team_jwt(monkeypatch, team_allowed_routes)
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ catalog: Final = client.get(
+ "/backend-api/codex/models?client_version=1.0.0", headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": token}
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Hello", "stream": True},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": token},
+ )
+ assert (catalog.status_code, response.status_code) == (200, 200), response.text
+ assert ("open-weights" in [model["slug"] for model in catalog.json()["models"]]) is through_litellm
+ assert provider.called is through_litellm
+ assert chatgpt.called is not through_litellm
+
+
+_USER_TURN: Final = {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Hello"}]}
+_ASSISTANT_TURN: Final = {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "Hi"}]}
+_FIREWORKS_REASONING: Final = {
+ "type": "reasoning",
+ "id": f"rs_{'0f' * 16}",
+ "summary": [{"type": "summary_text", "text": "Thinking it over"}],
+}
+
+
+@pytest.mark.parametrize(
+ ("item", "store", "forwarded"),
+ [
+ pytest.param(_FIREWORKS_REASONING, False, False, id="another provider's summary-only item"),
+ pytest.param({**_FIREWORKS_REASONING, "encrypted_content": None}, None, False, id="null blob, store unset"),
+ pytest.param(
+ {"type": "reasoning", "id": "rs_upstream", "summary": [], "encrypted_content": "gAAAAB-test-blob"},
+ False,
+ True,
+ id="ChatGPT's own item",
+ ),
+ pytest.param({**_FIREWORKS_REASONING, "id": None}, False, True, id="summary-only item without an id"),
+ pytest.param(_FIREWORKS_REASONING, True, True, id="stored conversation"),
+ ],
+)
+def test_chatgpt_passthrough_drops_reasoning_ids_chatgpt_never_persisted(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ item: dict[str, object],
+ store: bool | None,
+ forwarded: bool,
+) -> None:
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ body: Final = {
+ "model": "subscription-model",
+ "input": [_USER_TURN, item, _ASSISTANT_TURN],
+ "stream": True,
+ **({} if store is None else {"store": store}),
+ }
+ response: Final = client.post("/backend-api/codex/responses", json=body, headers=_HEADERS)
+ assert response.status_code == 200
+ assert json.loads(chatgpt.calls.last.request.content)["input"] == (
+ [_USER_TURN, item, _ASSISTANT_TURN] if forwarded else [_USER_TURN, _ASSISTANT_TURN]
+ )
+
+
+def test_exact_deployment_named_like_an_account_slug_stays_on_the_subscription_after_the_catalog_read(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ assert client.get("/backend-api/codex/models?client_version=0.165.0", headers=_ACCOUNT_HEADERS).status_code == 200
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "shared-name", "input": "Hello", "stream": True},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200
+ assert chatgpt.called
+ assert not provider.called
+ assert catalog.call_count == 1
+
+
+_LISTED: Final = httpx.Response(200, json=_UPSTREAM_CATALOG)
+
+
+@pytest.mark.parametrize(
+ ("model", "user_agent", "catalog_answer", "subscription"),
+ [
+ pytest.param("shared-name", _CODEX_USER_AGENT, _LISTED, True, id="listed"),
+ pytest.param("open-weights", _CODEX_USER_AGENT, _LISTED, False, id="unlisted"),
+ pytest.param("open-weights", _CODEX_USER_AGENT, httpx.Response(503), True, id="catalog unavailable"),
+ pytest.param("open-weights", _CODEX_USER_AGENT, httpx.ReadTimeout("slow"), True, id="catalog timeout"),
+ pytest.param("open-weights", "codex_cli_rs", _LISTED, True, id="no version"),
+ pytest.param("open-weights", "curl/8.5.0", _LISTED, True, id="user agent of another client"),
+ ],
+)
+def test_native_responses_ask_chatgpt_which_models_the_account_lists_on_a_cache_miss(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ model: str,
+ user_agent: str,
+ catalog_answer: httpx.Response | httpx.TimeoutException,
+ subscription: bool,
+) -> None:
+ catalog: Final = respx_mock.get(f"{_BASE}/models").mock(side_effect=[catalog_answer])
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": model, "input": "Hello", "stream": True},
+ headers={**_ACCOUNT_HEADERS, "user-agent": user_agent},
+ )
+ assert response.status_code == 200
+ assert chatgpt.called is subscription
+ assert provider.called is not subscription
+ assert catalog.called is (user_agent == _CODEX_USER_AGENT)
+
+
+def test_a_failed_account_catalog_read_keeps_the_next_turns_on_the_subscription_without_asking_again(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(503)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ responses: Final = tuple(
+ client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": "Hello", "stream": True},
+ headers=_ACCOUNT_HEADERS,
+ )
+ for _ in range(2)
+ )
+ assert [response.status_code for response in responses] == [200, 200]
+ assert chatgpt.call_count == 2
+ assert not provider.called
+ assert catalog.call_count == 1
+
+
+def test_a_failed_account_catalog_read_with_another_login_leaves_the_accounts_own_list_reachable(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ own_login: Final = f"Bearer {_CHATGPT_JWT}"
+ catalog: Final = respx_mock.get(f"{_BASE}/models").mock(
+ side_effect=lambda request: (
+ httpx.Response(200, json=_UPSTREAM_CATALOG)
+ if request.headers["authorization"] == own_login
+ else httpx.Response(401)
+ )
+ )
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ turn: Final = {"model": "open-weights", "input": "Hello", "stream": True}
+ other_login: Final = client.post(
+ "/backend-api/codex/responses",
+ json=turn,
+ headers={**_ACCOUNT_HEADERS, "authorization": "Bearer test-someone-elses-login"},
+ )
+ own: Final = client.post("/backend-api/codex/responses", json=turn, headers=_ACCOUNT_HEADERS)
+ assert other_login.status_code == own.status_code == 200
+ assert chatgpt.call_count == 1
+ assert provider.call_count == 1
+ assert [call.request.headers["authorization"] for call in catalog.calls] == [
+ "Bearer test-someone-elses-login",
+ own_login,
+ ]
+
+
+def test_rotating_chatgpt_logins_on_one_key_reads_the_account_catalog_a_bounded_number_of_times_per_key(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, restricted_key: Callable[..., str]
+) -> None:
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(401)
+ provider: Final = _open_weights_provider(respx_mock)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ turn: Final = {"model": "open-weights", "input": "Hello", "stream": True}
+ rotated: Final = tuple(
+ client.post(
+ "/backend-api/codex/responses",
+ json=turn,
+ headers={**_ACCOUNT_HEADERS, "authorization": f"Bearer test-rotated-login-{attempt}"},
+ )
+ for attempt in range(_ACCOUNT_SLUGS_FETCHES_PER_CALLER + 2)
+ )
+ teammate: Final = client.post(
+ "/backend-api/codex/responses",
+ json=turn,
+ headers={
+ **_ACCOUNT_HEADERS,
+ "authorization": "Bearer test-teammate-login",
+ "x-litellm-api-key": restricted_key(["open-weights"]),
+ },
+ )
+ assert {response.status_code for response in (*rotated, teammate)} == {200}
+ assert chatgpt.call_count == len(rotated) + 1
+ assert not provider.called
+ assert [call.request.headers["authorization"] for call in catalog.calls][-1] == "Bearer test-teammate-login"
+ assert catalog.call_count == _ACCOUNT_SLUGS_FETCHES_PER_CALLER + 1
+
+
+def test_account_catalog_lookup_sends_the_callers_chatgpt_login_and_codex_version(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ catalog: Final = respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG)
+ _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "shared-name", "input": "Hello", "stream": True},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200
+ fetched: Final = catalog.calls.last.request
+ assert fetched.url.params["client_version"] == "0.165.0"
+ assert {name: fetched.headers[name] for name in ("authorization", "chatgpt-account-id", "originator")} == {
+ "authorization": f"Bearer {_CHATGPT_JWT}",
+ "chatgpt-account-id": "account-a",
+ "originator": "codex_cli_rs",
+ }
+ assert fetched.headers["user-agent"] == _CODEX_USER_AGENT
+ assert "x-litellm-api-key" not in fetched.headers
+
+
+def _models_etags(client: TestClient, headers: dict[str, str]) -> tuple[str | None, str | None]:
+ catalog: Final = client.get("/backend-api/codex/models?client_version=0.165.0", headers=headers)
+ responses: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "gpt-unlisted", "input": "Hello", "stream": True},
+ headers=headers,
+ )
+ assert catalog.status_code == responses.status_code == 200, (catalog.text, responses.text)
+ return catalog.headers.get("etag"), responses.headers.get("x-models-etag")
+
+
+@pytest.mark.parametrize(
+ ("models", "merged"),
+ [
+ pytest.param(["chatgpt-native", "open-weights"], True, id="catalog with LiteLLM models"),
+ pytest.param(["chatgpt-native"], False, id="catalog of ChatGPT models only"),
+ ],
+)
+def test_native_catalog_and_responses_carry_the_same_models_etag(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ restricted_key: Callable[..., str],
+ models: list[str],
+ merged: bool,
+) -> None:
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": _UPSTREAM_MODELS_ETAG})
+ _chatgpt_responses(respx_mock)
+ catalog_etag, responses_etag = _models_etags(client, {**_HEADERS, "x-litellm-api-key": restricted_key(models)})
+ assert catalog_etag is not None
+ assert catalog_etag == responses_etag
+ assert (catalog_etag != _UPSTREAM_MODELS_ETAG) is merged
+
+
+def test_native_models_etag_is_the_same_on_replicas_with_empty_caches(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setitem(proxy_server.app.dependency_overrides, codex_shared_cache, lambda: DualCache())
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": _UPSTREAM_MODELS_ETAG})
+ _chatgpt_responses(respx_mock)
+ first_replica: Final = _models_etags(client, _HEADERS)
+ second_replica: Final = _models_etags(client, _HEADERS)
+ assert first_replica[0] not in (None, _UPSTREAM_MODELS_ETAG)
+ assert len({*first_replica, *second_replica}) == 1
+
+
+def test_native_models_etag_changes_with_the_keys_litellm_models_and_with_upstreams_etag(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ open_weights: Final = _open_weights("open-weights")
+ wider_window: Final = {**open_weights, "model_info": {"max_input_tokens": 64000}}
+
+ def catalog_etag(deployment: dict[str, object], upstream_etag: str) -> str | None:
+ monkeypatch.setattr(proxy_server, "llm_router", Router(model_list=[deployment]))
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": upstream_etag})
+ return client.get("/backend-api/codex/models?client_version=0.165.0", headers=_HEADERS).headers.get("etag")
+
+ baseline: Final = catalog_etag(open_weights, '"upstream-v1"')
+ assert baseline == catalog_etag(open_weights, '"upstream-v1"')
+ assert (
+ len({baseline, catalog_etag(wider_window, '"upstream-v1"'), catalog_etag(open_weights, '"upstream-v3"')}) == 3
+ )
+ assert baseline not in (None, '"upstream-v1"')
+
+
+class FailingModelListing(CustomLogger):
+ async def async_filter_listed_models(
+ self, user_api_key_dict: UserAPIKeyAuth, model_names: Sequence[str]
+ ) -> Sequence[str]:
+ raise RuntimeError("model listing backend unavailable")
+
+
+def test_native_catalog_relays_chatgpts_answer_when_listing_the_keys_models_fails(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "callbacks", [FailingModelListing()])
+ respx_mock.get(f"{_BASE}/models").respond(json=_UPSTREAM_CATALOG, headers={"etag": '"upstream-v1"'})
+ response: Final = client.get("/backend-api/codex/models?client_version=0.165.0", headers=_HEADERS)
+ assert response.status_code == 200
+ assert response.json() == _UPSTREAM_CATALOG
+ assert response.headers["etag"] == '"upstream-v1"'
+
+
+def test_native_responses_keep_chatgpts_models_etag_when_listing_the_keys_models_fails(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "callbacks", [FailingModelListing()])
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "gpt-unlisted", "input": "Hello", "stream": True},
+ headers=_HEADERS,
+ )
+ assert response.status_code == 200
+ assert chatgpt.called
+ assert response.headers["x-models-etag"] == _UPSTREAM_MODELS_ETAG
+
+
+_OPENAI_BASE: Final = "https://openai-api.test/v1"
+_OPENAI_BLOB: Final = "gAAAAB-openai-reasoning-blob"
+_OPENAI_REASONING: Final = {"type": "reasoning", "id": "rs_openai", "summary": [], "encrypted_content": _OPENAI_BLOB}
+
+
+def _openai_reply(request: httpx.Request) -> httpx.Response:
+ completed: Final = {
+ "id": "resp_openai",
+ "object": "response",
+ "created_at": 1700000000,
+ "status": "completed",
+ "model": "gpt-5.6",
+ "output": [
+ _OPENAI_REASONING,
+ {
+ "type": "message",
+ "id": "msg_openai",
+ "role": "assistant",
+ "status": "completed",
+ "content": [{"type": "output_text", "text": "Hi", "annotations": []}],
+ },
+ ],
+ "usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
+ }
+ if json.loads(request.content).get("stream") is not True:
+ return httpx.Response(200, json=completed)
+ events: Final = (
+ {
+ "type": "response.created",
+ "sequence_number": 0,
+ "response": {**completed, "status": "in_progress", "output": []},
+ },
+ {"type": "response.output_item.done", "sequence_number": 1, "output_index": 0, "item": _OPENAI_REASONING},
+ {"type": "response.completed", "sequence_number": 2, "response": completed},
+ )
+ return httpx.Response(
+ 200,
+ content="".join(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n" for event in events).encode(),
+ headers={"content-type": "text/event-stream"},
+ )
+
+
+def _reasoning_item_of(response: httpx.Response, stream: bool) -> dict[str, object]:
+ if not stream:
+ return response.json()["output"][0]
+ payloads: Final = (
+ json.loads(line.removeprefix(b"data: "))
+ for line in response.content.splitlines()
+ if line.startswith(b"data: {")
+ )
+ return next(event["item"] for event in payloads if event["type"] == "response.output_item.done")
+
+
+@pytest.mark.parametrize("stream", [False, True])
+def test_openai_reasoning_from_the_native_route_is_dropped_for_chatgpt_and_unwrapped_for_litellm(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch, stream: bool
+) -> None:
+ monkeypatch.setattr(
+ proxy_server,
+ "llm_router",
+ Router(
+ model_list=[
+ {
+ "model_name": "api-model",
+ "litellm_params": {
+ "model": "openai/gpt-5.6",
+ "api_base": _OPENAI_BASE,
+ "api_key": "test-openai-key",
+ },
+ }
+ ]
+ ),
+ )
+ _account_catalog(respx_mock)
+ provider: Final = respx_mock.post(f"{_OPENAI_BASE}/responses").mock(side_effect=_openai_reply)
+ chatgpt: Final = _chatgpt_responses(respx_mock)
+ turn: Final = {"model": "api-model", "input": [_USER_TURN], "stream": stream, "store": False}
+ first: Final = client.post("/backend-api/codex/responses", json=turn, headers=_ACCOUNT_HEADERS)
+ assert first.status_code == 200, first.text
+ reasoning: Final = _reasoning_item_of(first, stream)
+ model_id, blob = ResponsesAPIRequestUtils.unwrap_encrypted_content_with_model_id(
+ str(reasoning["encrypted_content"])
+ )
+ assert (model_id is not None, blob) == (True, _OPENAI_BLOB)
+ history: Final = [_USER_TURN, reasoning, _ASSISTANT_TURN, _USER_TURN]
+ to_chatgpt: Final = client.post(
+ "/backend-api/codex/responses",
+ json={**turn, "model": "subscription-model", "input": history, "stream": True},
+ headers=_ACCOUNT_HEADERS,
+ )
+ back_to_litellm: Final = client.post(
+ "/backend-api/codex/responses", json={**turn, "input": history}, headers=_ACCOUNT_HEADERS
+ )
+ v1: Final = client.post(
+ "/v1/responses", json={**turn, "stream": False}, headers={"authorization": f"Bearer {MASTER_KEY}"}
+ )
+ assert to_chatgpt.status_code == back_to_litellm.status_code == v1.status_code == 200
+ assert json.loads(chatgpt.calls.last.request.content)["input"] == [_USER_TURN, _ASSISTANT_TURN, _USER_TURN]
+ replayed: Final = json.loads(provider.calls[1].request.content)["input"][1]
+ assert (replayed["id"], replayed["encrypted_content"]) == ("rs_openai", _OPENAI_BLOB)
+ assert v1.json()["output"][0]["encrypted_content"] == _OPENAI_BLOB
+
+
+_XAI_BASE: Final = "https://xai-api.test/v1"
+_LUNA_ID: Final = "oai-luna-deployment"
+_LUNA_BLOB: Final = "gAAAAB-luna-reasoning-blob"
+_LUNA_REASONING: Final = {
+ "type": "reasoning",
+ "summary": [],
+ "encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(_LUNA_BLOB, _LUNA_ID),
+}
+_CHATGPT_REASONING: Final = {"type": "reasoning", "summary": [], "encrypted_content": "gAAAAB-chatgpt-reasoning-blob"}
+
+
+@pytest.fixture
+def xai_fireworks_and_openai_router(monkeypatch: pytest.MonkeyPatch) -> Router:
+ router: Final = Router(
+ model_list=[
+ {
+ "model_name": "grok-4.7",
+ "litellm_params": {"model": "xai/grok-4.7", "api_base": _XAI_BASE, "api_key": "test-xai-key"},
+ },
+ _open_weights("open-weights"),
+ {
+ "model_name": "oai-luna",
+ "litellm_params": {
+ "model": "openai/gpt-6-luna",
+ "api_base": _OPENAI_BASE,
+ "api_key": "test-openai-key",
+ },
+ "model_info": {"id": _LUNA_ID},
+ },
+ ]
+ )
+ monkeypatch.setattr(proxy_server, "llm_router", router)
+ return router
+
+
+@pytest.mark.parametrize(
+ ("model", "replayed", "expected"),
+ [
+ pytest.param("open-weights", _CHATGPT_REASONING, None, id="chatgpt reasoning to fireworks is dropped"),
+ pytest.param("open-weights", _LUNA_REASONING, None, id="another deployment's reasoning to fireworks is dropped"),
+ pytest.param("oai-luna", _LUNA_REASONING, _LUNA_BLOB, id="the minting deployment's reasoning is kept"),
+ ],
+)
+def test_litellm_model_on_native_route_gets_only_the_reasoning_it_can_decrypt(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ xai_fireworks_and_openai_router: Router,
+ model: str,
+ replayed: dict[str, object],
+ expected: str | None,
+) -> None:
+ _account_catalog(respx_mock)
+ fireworks: Final = _open_weights_provider(respx_mock)
+ luna: Final = respx_mock.post(f"{_OPENAI_BASE}/responses").mock(side_effect=_provider_reply)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": model,
+ "input": [_USER_TURN, replayed, _ASSISTANT_TURN, _USER_TURN],
+ "stream": False,
+ "store": False,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ provider: Final = fireworks if model == "open-weights" else luna
+ sent: Final = json.loads(provider.calls.last.request.content)["input"]
+ assert [item.get("encrypted_content") for item in sent if item.get("type") == "reasoning"] == (
+ [] if expected is None else [expected]
+ )
+
+
+_WHOLE_FIREWORKS_REASONING: Final = {**_FIREWORKS_REASONING, "status": "completed"}
+_SUMMARY_ONLY_REASONING: Final = {key: value for key, value in _FIREWORKS_REASONING.items() if key != "id"}
+_OTHER_DEPLOYMENTS_BLOB: Final = ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
+ "gAAAAB-other-deployment-blob", "other-deployment"
+)
+
+
+@pytest.mark.parametrize(
+ ("model", "replayed", "store", "expected"),
+ [
+ pytest.param(
+ "oai-luna", _WHOLE_FIREWORKS_REASONING, False, [_SUMMARY_ONLY_REASONING], id="whole fireworks item to openai"
+ ),
+ pytest.param(
+ "oai-luna",
+ {**_WHOLE_FIREWORKS_REASONING, "encrypted_content": _OTHER_DEPLOYMENTS_BLOB},
+ False,
+ [_SUMMARY_ONLY_REASONING],
+ id="whole item with another deployment's blob to openai",
+ ),
+ pytest.param(
+ "oai-luna",
+ {**_SUMMARY_ONLY_REASONING, "content": [], "encrypted_content": _OTHER_DEPLOYMENTS_BLOB},
+ False,
+ [{**_SUMMARY_ONLY_REASONING, "content": []}],
+ id="codex's replay of another deployment's item to openai",
+ ),
+ pytest.param(
+ "oai-luna",
+ {**_SUMMARY_ONLY_REASONING, "content": None, "encrypted_content": None},
+ False,
+ [_SUMMARY_ONLY_REASONING],
+ id="codex's replay of a blobless item to openai",
+ ),
+ pytest.param(
+ "oai-luna",
+ {"type": "reasoning", "id": "rs_unreadable", "summary": []},
+ False,
+ [],
+ id="an unresolvable id with nothing readable is dropped",
+ ),
+ pytest.param(
+ "open-weights",
+ {**_SUMMARY_ONLY_REASONING, "id": "rs_openai"},
+ False,
+ [_SUMMARY_ONLY_REASONING],
+ id="openai summary-only item to fireworks",
+ ),
+ pytest.param(
+ "oai-luna",
+ {**_LUNA_REASONING, "id": "rs_luna", "content": []},
+ False,
+ [{**_LUNA_REASONING, "id": "rs_luna", "content": [], "encrypted_content": _LUNA_BLOB}],
+ id="the minting deployment gets its item whole",
+ ),
+ pytest.param(
+ "oai-luna", _FIREWORKS_REASONING, True, [_FIREWORKS_REASONING], id="a stored conversation keeps the id"
+ ),
+ ],
+)
+def test_reasoning_the_target_cannot_resolve_reaches_it_without_its_id(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ xai_fireworks_and_openai_router: Router,
+ model: str,
+ replayed: dict[str, object],
+ store: bool,
+ expected: list[dict[str, object]],
+) -> None:
+ _account_catalog(respx_mock)
+ fireworks: Final = _open_weights_provider(respx_mock)
+ luna: Final = respx_mock.post(f"{_OPENAI_BASE}/responses").mock(side_effect=_provider_reply)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": model,
+ "input": [_USER_TURN, replayed, _ASSISTANT_TURN, _USER_TURN],
+ "stream": False,
+ "store": store,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ sent: Final = json.loads((fireworks if model == "open-weights" else luna).calls.last.request.content)["input"]
+ assert [item for item in sent if item.get("type") == "reasoning"] == expected
+
+
+def test_reasoning_the_chat_bridge_target_cannot_resolve_reaches_it_as_its_summary(
+ client: TestClient, respx_mock: respx.MockRouter, xai_fireworks_and_openai_router: Router
+) -> None:
+ _account_catalog(respx_mock)
+ chat_endpoint: Final = respx_mock.post(f"{_XAI_BASE}/chat/completions").mock(side_effect=_xai_search_call_reply)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": "grok-4.7",
+ "input": [_USER_TURN, _WHOLE_FIREWORKS_REASONING, _ASSISTANT_TURN, _USER_TURN],
+ "stream": False,
+ "store": False,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ messages: Final = json.loads(chat_endpoint.calls.last.request.content)["messages"]
+ assert [message.get("reasoning_content") for message in messages if message["role"] == "assistant"] == [
+ "Thinking it over"
+ ]
+
+
+def _namespace(name: str, *members: str) -> dict[str, object]:
+ return {
+ "type": "namespace",
+ "name": name,
+ "description": "",
+ "tools": [{"type": "function", "name": member, "parameters": {"type": "object"}} for member in members],
+ }
+
+
+_EXEC_TOOL: Final = {"type": "function", "name": "exec_command", "parameters": {"type": "object"}}
+_ADDITIONAL_TOOLS: Final = {
+ "type": "additional_tools",
+ "role": "developer",
+ "tools": [_namespace("collaboration", "followup_task"), _namespace("mcp__docs__", "search")],
+}
+
+
+@pytest.mark.parametrize(
+ ("model", "expected_input", "expected_tool_names"),
+ [
+ pytest.param(
+ "open-weights",
+ [_USER_TURN],
+ ["exec_command", "collaboration", "mcp__docs__"],
+ id="fireworks gets the item's tools as top-level tools",
+ ),
+ pytest.param("oai-luna", [_ADDITIONAL_TOOLS, _USER_TURN], ["exec_command"], id="the openai api keeps the item"),
+ ],
+)
+def test_codex_additional_tools_item_reaches_each_provider_in_a_shape_it_accepts(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ xai_fireworks_and_openai_router: Router,
+ model: str,
+ expected_input: list[dict[str, object]],
+ expected_tool_names: list[str],
+) -> None:
+ _account_catalog(respx_mock)
+ fireworks: Final = _open_weights_provider(respx_mock)
+ luna: Final = respx_mock.post(f"{_OPENAI_BASE}/responses").mock(side_effect=_provider_reply)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": model,
+ "input": [_ADDITIONAL_TOOLS, _USER_TURN],
+ "tools": [_EXEC_TOOL],
+ "stream": False,
+ "store": False,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ sent: Final = json.loads((fireworks if model == "open-weights" else luna).calls.last.request.content)
+ assert sent["input"] == expected_input
+ assert [tool["name"] for tool in sent["tools"]] == expected_tool_names
+
+
+_FORBIDDEN_TOOL: Final = {"type": "function", "name": "delete_user", "parameters": {"type": "object"}}
+
+
+@pytest.mark.parametrize(
+ ("tools", "input_items", "allowed"),
+ [
+ pytest.param([_EXEC_TOOL, _FORBIDDEN_TOOL], [_USER_TURN], False, id="a forbidden top-level tool"),
+ pytest.param(
+ [_EXEC_TOOL],
+ [{"type": "additional_tools", "role": "developer", "tools": [_FORBIDDEN_TOOL]}, _USER_TURN],
+ False,
+ id="a forbidden tool in an additional_tools item",
+ ),
+ pytest.param([_EXEC_TOOL], [_USER_TURN], True, id="only allowed tools"),
+ ],
+)
+def test_native_litellm_turn_enforces_the_key_tool_allowlist(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ xai_fireworks_and_openai_router: Router,
+ restricted_key: Callable[..., str],
+ tools: list[dict[str, object]],
+ input_items: list[dict[str, object]],
+ allowed: bool,
+) -> None:
+ _account_catalog(respx_mock)
+ fireworks: Final = _open_weights_provider(respx_mock)
+ key: Final = restricted_key(
+ ["open-weights"], (*_NATIVE_ROUTES, "/v1/responses"), allowed_tools=(str(_EXEC_TOOL["name"]),)
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": input_items, "tools": tools, "stream": False, "store": False},
+ headers={**_ACCOUNT_HEADERS, "x-litellm-api-key": key},
+ )
+ assert response.status_code == (200 if allowed else 403), response.text
+ assert fireworks.called is allowed
+
+
+_ANTHROPIC_BASE: Final = "https://anthropic-api.test"
+
+
+def test_codex_namespace_tool_called_through_the_chat_bridge_comes_back_with_its_namespace(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(
+ proxy_server,
+ "llm_router",
+ Router(
+ model_list=[
+ {
+ "model_name": "claude-sonnet-5-5",
+ "litellm_params": {
+ "model": "anthropic/claude-sonnet-5-5",
+ "api_base": _ANTHROPIC_BASE,
+ "api_key": "test-anthropic-key",
+ },
+ }
+ ]
+ ),
+ )
+ _account_catalog(respx_mock)
+ anthropic: Final = respx_mock.post(f"{_ANTHROPIC_BASE}/v1/messages").respond(
+ json={
+ "id": "msg_test",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-sonnet-5-5",
+ "content": [
+ {"type": "tool_use", "id": "toolu_test", "name": "mcp__docs____search", "input": {"query": "proxy"}}
+ ],
+ "stop_reason": "tool_use",
+ "usage": {"input_tokens": 3, "output_tokens": 1},
+ }
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": "claude-sonnet-5-5",
+ "input": [_ADDITIONAL_TOOLS, _USER_TURN],
+ "tools": [_EXEC_TOOL],
+ "stream": False,
+ "store": False,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ assert anthropic.called
+ (call,) = (item for item in response.json()["output"] if item["type"] == "function_call")
+ assert (call["namespace"], call["name"]) == ("mcp__docs__", "search")
+
+
+_CLOCK_NAMESPACE: Final = _namespace("clock", "sleep")
+_CLOCK_TOOLS: Final = {"type": "additional_tools", "role": "developer", "tools": [_CLOCK_NAMESPACE]}
+def test_codex_namespace_tool_reaches_fireworks_as_a_namespace(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router
+) -> None:
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "open-weights", "input": [_CLOCK_TOOLS, _USER_TURN], "stream": False, "store": False},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ assert json.loads(provider.calls.last.request.content)["tools"] == [_CLOCK_NAMESPACE]
+
+
+
+@pytest.mark.parametrize(
+ ("drop_params", "expected_tools"),
+ [
+ pytest.param({}, [_CLOCK_NAMESPACE], id="codex's drop_params default merges the copies"),
+ pytest.param({"drop_params": False}, [_CLOCK_NAMESPACE, _CLOCK_NAMESPACE], id="the caller's off sends each"),
+ ],
+)
+def test_codex_tool_list_repeated_on_every_turn_reaches_fireworks_merged_only_under_drop_params(
+ client: TestClient,
+ respx_mock: respx.MockRouter,
+ mixed_router: Router,
+ drop_params: dict[str, bool],
+ expected_tools: list[dict[str, object]],
+) -> None:
+ _account_catalog(respx_mock)
+ provider: Final = _open_weights_provider(respx_mock)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": "open-weights",
+ "input": [_CLOCK_TOOLS, _USER_TURN, _CLOCK_TOOLS],
+ "stream": False,
+ "store": False,
+ **drop_params,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ assert json.loads(provider.calls.last.request.content)["tools"] == expected_tools
+
+_CUSTOM_EXEC_TOOL: Final = {"type": "custom", "name": "exec", "description": "Run a command", "format": {"type": "text"}}
+_XAI_SEARCH_CALL: Final = {
+ "id": "call_search",
+ "type": "function",
+ "function": {"name": "mcp__docs____search", "arguments": '{"query":"proxy"}'},
+}
+_XAI_USAGE: Final = {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4}
+
+
+def _xai_chat_chunk(choice: dict[str, object], **fields: object) -> str:
+ chunk: Final = {
+ "id": "chatcmpl-xai",
+ "object": "chat.completion.chunk",
+ "created": 1700000000,
+ "model": "grok-4.7",
+ "choices": [{"index": 0, **choice}],
+ **fields,
+ }
+ return f"data: {json.dumps(chunk)}\n\n"
+
+
+def _xai_search_call_reply(request: httpx.Request) -> httpx.Response:
+ if json.loads(request.content).get("stream") is not True:
+ return httpx.Response(
+ 200,
+ json={
+ "id": "chatcmpl-xai",
+ "object": "chat.completion",
+ "created": 1700000000,
+ "model": "grok-4.7",
+ "choices": [
+ {
+ "index": 0,
+ "message": {"role": "assistant", "content": None, "tool_calls": [_XAI_SEARCH_CALL]},
+ "finish_reason": "tool_calls",
+ }
+ ],
+ "usage": _XAI_USAGE,
+ },
+ )
+ stream: Final = (
+ _xai_chat_chunk(
+ {"delta": {"role": "assistant", "tool_calls": [{"index": 0, **_XAI_SEARCH_CALL}]}, "finish_reason": None}
+ )
+ + _xai_chat_chunk({"delta": {}, "finish_reason": "tool_calls"}, usage=_XAI_USAGE)
+ + "data: [DONE]\n\n"
+ )
+ return httpx.Response(200, content=stream.encode(), headers={"content-type": "text/event-stream"})
+
+
+def _data_lines(stream: bytes) -> list[bytes]:
+ return [line.removeprefix(b"data: ") for line in stream.splitlines() if line.startswith(b"data: {")]
+
+
+def _function_calls_of(response: httpx.Response, stream: bool) -> list[dict[str, object]]:
+ items: Final = (
+ response.json()["output"]
+ if not stream
+ else [
+ event["item"]
+ for event in map(json.loads, _data_lines(response.content))
+ if event["type"] == "response.output_item.done"
+ ]
+ )
+ return [item for item in items if item["type"] == "function_call"]
+
+
+@pytest.mark.parametrize("stream", [False, True])
+def test_codex_turn_for_xai_goes_through_chat_completions_and_its_namespaced_call_comes_back_whole(
+ client: TestClient, respx_mock: respx.MockRouter, xai_fireworks_and_openai_router: Router, stream: bool
+) -> None:
+ _account_catalog(respx_mock)
+ responses_endpoint: Final = respx_mock.post(f"{_XAI_BASE}/responses").respond(json=_provider_completed("Hi"))
+ chat_endpoint: Final = respx_mock.post(f"{_XAI_BASE}/chat/completions").mock(side_effect=_xai_search_call_reply)
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={
+ "model": "grok-4.7",
+ "input": [_ADDITIONAL_TOOLS, _USER_TURN],
+ "tools": [_EXEC_TOOL, _CUSTOM_EXEC_TOOL],
+ "stream": stream,
+ "store": False,
+ },
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ assert not responses_endpoint.called
+ sent_tools: Final = json.loads(chat_endpoint.calls.last.request.content)["tools"]
+ assert [(tool["type"], tool["function"]["name"]) for tool in sent_tools] == [
+ ("function", "exec_command"),
+ ("function", "exec"),
+ ("function", "collaboration__followup_task"),
+ ("function", "mcp__docs____search"),
+ ]
+ (call,) = _function_calls_of(response, stream)
+ assert (call["name"], call["namespace"], call["call_id"]) == ("search", "mcp__docs__", "call_search")
+
+
+def test_encrypted_reasoning_stays_only_when_every_deployment_the_router_may_pick_can_decrypt_it(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ second_base: Final = "https://second-openai-org.test/v1"
+
+ def replayed_blobs(optional_pre_call_checks: list[str]) -> list[object]:
+ monkeypatch.setattr(
+ proxy_server,
+ "llm_router",
+ Router(
+ model_list=[
+ {
+ "model_name": "oai-pair",
+ "litellm_params": {"model": "openai/gpt-6-luna", "api_base": base, "api_key": "test-key"},
+ "model_info": {"id": model_id},
+ }
+ for base, model_id in ((_OPENAI_BASE, _LUNA_ID), (second_base, "second-org-deployment"))
+ ],
+ optional_pre_call_checks=optional_pre_call_checks,
+ ),
+ )
+ providers: Final = tuple(
+ respx_mock.post(f"{base}/responses").mock(side_effect=_provider_reply)
+ for base in (_OPENAI_BASE, second_base)
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "oai-pair", "input": [_USER_TURN, _LUNA_REASONING, _USER_TURN], "store": False},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ (sent,) = (json.loads(call.request.content)["input"] for provider in providers for call in provider.calls)
+ return [item.get("encrypted_content") for item in sent if item.get("type") == "reasoning"]
+
+ _account_catalog(respx_mock)
+ assert replayed_blobs([]) == []
+ respx_mock.reset()
+ _account_catalog(respx_mock)
+ assert replayed_blobs(["encrypted_content_affinity"]) == [_LUNA_BLOB]
+
+
+def test_encrypted_reasoning_stays_on_a_global_alias_of_a_group_pinned_to_its_origin(
+ client: TestClient, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ second_base: Final = "https://second-openai-org.test/v1"
+ monkeypatch.setattr(
+ proxy_server,
+ "llm_router",
+ Router(
+ model_list=[
+ {
+ "model_name": "oai-pair",
+ "litellm_params": {"model": "openai/gpt-6-luna", "api_base": base, "api_key": "test-key"},
+ "model_info": {"id": model_id},
+ }
+ for base, model_id in ((_OPENAI_BASE, _LUNA_ID), (second_base, "second-org-deployment"))
+ ],
+ model_group_alias={"pair-alias": "oai-pair"},
+ model_group_affinity_config={"oai-pair": ["encrypted_content_affinity"]},
+ ),
+ )
+ _account_catalog(respx_mock)
+ providers: Final = tuple(
+ respx_mock.post(f"{base}/responses").mock(side_effect=_provider_reply) for base in (_OPENAI_BASE, second_base)
+ )
+ response: Final = client.post(
+ "/backend-api/codex/responses",
+ json={"model": "pair-alias", "input": [_USER_TURN, _LUNA_REASONING, _USER_TURN], "store": False},
+ headers=_ACCOUNT_HEADERS,
+ )
+ assert response.status_code == 200, response.text
+ (sent,) = (json.loads(call.request.content)["input"] for provider in providers for call in provider.calls)
+ assert [item.get("encrypted_content") for item in sent if item.get("type") == "reasoning"] == [_LUNA_BLOB]
+
+
+class ModelListingCounter(CustomLogger):
+ def __init__(self) -> None:
+ super().__init__()
+ self.listings: Final[list[tuple[str, ...]]] = []
+
+ async def async_filter_listed_models(
+ self, user_api_key_dict: UserAPIKeyAuth, model_names: Sequence[str]
+ ) -> Sequence[str]:
+ self.listings.append(tuple(model_names))
+ return model_names
+
+
+def test_jwt_caller_builds_the_models_etag_once_across_chatgpt_turns(
+ client: TestClient, respx_mock: respx.MockRouter, mixed_router: Router, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ counter: Final = ModelListingCounter()
+ monkeypatch.setattr(litellm, "callbacks", [counter])
+ monkeypatch.setitem(
+ proxy_server.app.dependency_overrides,
+ user_api_key_auth,
+ lambda: UserAPIKeyAuth(api_key=None, user_id="jwt-user", user_role=LitellmUserRoles.PROXY_ADMIN),
+ )
+ _chatgpt_responses(respx_mock)
+ turns: Final = tuple(
+ client.post(
+ "/backend-api/codex/responses",
+ json={"model": "gpt-unlisted", "input": "Hello", "stream": True},
+ headers={**_HEADERS, "x-litellm-api-key": "test-jwt-credential"},
+ )
+ for _ in range(2)
+ )
+ assert [turn.status_code for turn in turns] == [200, 200]
+ assert len(counter.listings) == 1
+ first_etag, second_etag = (turn.headers["x-models-etag"] for turn in turns)
+ assert first_etag == second_etag != _UPSTREAM_MODELS_ETAG
diff --git a/tests/unit/proxy/proxy_server/test_routes_model_info.py b/tests/unit/proxy/proxy_server/test_routes_model_info.py
index 6ad5ef1a4eb..79f3e697a4f 100644
--- a/tests/unit/proxy/proxy_server/test_routes_model_info.py
+++ b/tests/unit/proxy/proxy_server/test_routes_model_info.py
@@ -490,6 +490,51 @@ def test_model_info_reports_null_cost_for_unpriced_deployment_and_zero_for_decla
assert input_cost > 0 and output_cost > 0
+@pytest.mark.parametrize(
+ ("deployment_model_info", "native"),
+ [
+ ({}, True),
+ ({"supports_bedrock_native_structured_output": False}, False),
+ ({"supports_native_structured_output": True, "supports_bedrock_native_structured_output": False}, True),
+ ],
+ ids=["native-by-default", "bedrock-opt-out", "explicit-generic-flag"],
+)
+def test_model_info_reports_a_deployment_bedrock_native_output_opt_out(
+ local_model_cost_map: None, deployment_model_info: dict[str, object], native: bool
+) -> None:
+ info: Final = proxy_server._enrich_model_info_with_litellm_data(
+ {
+ "model_name": "sonnet",
+ "litellm_params": {"model": "bedrock/us.anthropic.claude-sonnet-5"},
+ "model_info": {"id": "sonnet-deployment", **deployment_model_info},
+ }
+ )["model_info"]
+
+ assert info["supports_native_structured_output"] is native
+
+
+@pytest.mark.parametrize(
+ ("model", "by_default"),
+ [("bedrock/us.anthropic.claude-opus-4-7", True), ("bedrock/us.anthropic.claude-sonnet-5", False)],
+ ids=["set-on-a-model-the-cost-map-leaves-on-the-json-tool", "unset-on-a-model-the-cost-map-makes-native"],
+)
+def test_model_info_ignores_a_deployment_by_default_key_the_router_reads_only_from_the_cost_map(
+ local_model_cost_map: None, model: str, by_default: bool
+) -> None:
+ def reported_native(deployment_model_info: dict[str, object]) -> object:
+ return proxy_server._enrich_model_info_with_litellm_data(
+ {
+ "model_name": "claude",
+ "litellm_params": {"model": model},
+ "model_info": {"id": "claude-deployment", **deployment_model_info},
+ }
+ )["model_info"]["supports_native_structured_output"]
+
+ assert reported_native({"supports_bedrock_native_structured_output_by_default": by_default}) is reported_native(
+ {}
+ )
+
+
def test_model_info_id_lookup_reports_the_same_cost_as_the_list(
client: TestClient,
auth_as: Callable[[], AbstractContextManager[object]],
diff --git a/tests/unit/proxy/public_endpoints/test_public_endpoints.py b/tests/unit/proxy/public_endpoints/test_public_endpoints.py
index d1afcf80ec9..27ba8968200 100644
--- a/tests/unit/proxy/public_endpoints/test_public_endpoints.py
+++ b/tests/unit/proxy/public_endpoints/test_public_endpoints.py
@@ -987,6 +987,57 @@ def test_public_model_hub_reports_the_newest_check_of_a_group_not_the_last_row_r
app.dependency_overrides.clear()
+def test_public_model_hub_lists_each_deployments_prices_without_exposing_the_deployment() -> None:
+ llm_router: Final = litellm.Router(
+ model_list=[
+ {
+ "model_name": "hub-public",
+ "litellm_params": {
+ "model": "openai/hub-secret-backend-a",
+ "api_base": "https://secret-a.internal",
+ "api_key": "sk-secret-a",
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 2e-06,
+ },
+ "model_info": {"id": "hub-secret-id-a"},
+ },
+ {
+ "model_name": "hub-public",
+ "litellm_params": {
+ "model": "openai/hub-secret-backend-b",
+ "api_base": "https://secret-b.internal",
+ "api_key": "sk-secret-b",
+ "input_cost_per_token": 3e-06,
+ "output_cost_per_token": 4e-06,
+ },
+ "model_info": {"id": "hub-secret-id-b"},
+ },
+ ]
+ )
+ app: Final = FastAPI()
+ app.include_router(router)
+
+ with (
+ patch("litellm.public_model_groups", ["hub-public"]),
+ patch("litellm.proxy.proxy_server.llm_router", llm_router),
+ patch("litellm.proxy.proxy_server.prisma_client", None),
+ ):
+ response: Final = TestClient(app).get("/public/model_hub")
+
+ assert response.status_code == 200, response.text
+ groups: Final = TypeAdapter(list[ModelGroupInfoProxy]).validate_json(response.text)
+ assert [dict(p.costs) for p in groups[0].deployment_pricing] == [
+ {"input_cost_per_token": 1e-06, "output_cost_per_token": 2e-06},
+ {"input_cost_per_token": 3e-06, "output_cost_per_token": 4e-06},
+ ]
+ leaked: Final = [
+ secret
+ for secret in ("secret-a.internal", "secret-b.internal", "sk-secret", "hub-secret-backend", "hub-secret-id")
+ if secret in response.text
+ ]
+ assert leaked == [], response.text
+
+
# ---------------------------------------------------------------------------
# /public/agent_hub
# ---------------------------------------------------------------------------
diff --git a/tests/unit/proxy/router_entry_point_stubs.py b/tests/unit/proxy/router_entry_point_stubs.py
index 1d30cb99b44..1445fe517c6 100644
--- a/tests/unit/proxy/router_entry_point_stubs.py
+++ b/tests/unit/proxy/router_entry_point_stubs.py
@@ -125,6 +125,8 @@ def route_entry_points_to_stubs(router: _RouterDouble) -> _RouterDouble:
return router.get_configured_display_name(name)
if field == "service_tiers":
return router.get_configured_service_tiers(name, **_given(team_id=team_id))
+ if field == "pinned_reasoning_efforts":
+ return router.get_pinned_reasoning_efforts(name, **_given(team_id=team_id))
if field == "max_input_tokens":
return router.get_configured_max_input_tokens(name, **_given(team_id=team_id))
if field == "deployment_model_info":
diff --git a/tests/unit/proxy/test_tools_allowlist_enforcement.py b/tests/unit/proxy/test_tools_allowlist_enforcement.py
index e306f7be4f8..efaf25c480c 100644
--- a/tests/unit/proxy/test_tools_allowlist_enforcement.py
+++ b/tests/unit/proxy/test_tools_allowlist_enforcement.py
@@ -242,6 +242,42 @@ class TestCheckToolsAllowlist:
assert exc_info.value.type == ProxyErrorTypes.tool_access_denied
assert "restricted_tool" in str(exc_info.value.message)
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(
+ ("allowed_tools", "denied"),
+ [
+ pytest.param(["get_weather"], True, id="forbidden"),
+ pytest.param(["get_weather", "delete_user"], False, id="allowed"),
+ ],
+ )
+ async def test_tool_defined_in_an_additional_tools_input_item_is_checked_on_responses_route(
+ self, allowed_tools: list[str], denied: bool
+ ) -> None:
+ body = {
+ "model": "fireworks_ai/glm-5p3",
+ "input": [
+ {
+ "type": "additional_tools",
+ "role": "developer",
+ "tools": [{"type": "function", "name": "delete_user", "parameters": {"type": "object"}}],
+ },
+ {"type": "message", "role": "user", "content": "Delete user 42"},
+ ],
+ }
+ check = check_tools_allowlist(
+ request_body=body,
+ valid_token=_token(metadata={"allowed_tools": allowed_tools}),
+ team_object=None,
+ route="/v1/responses",
+ )
+ if not denied:
+ await check
+ return
+ with pytest.raises(ProxyException) as exc_info:
+ await check
+ assert exc_info.value.type == ProxyErrorTypes.tool_access_denied
+ assert "delete_user" in str(exc_info.value.message)
+
@pytest.mark.parametrize(
"loaded_tool",
[
diff --git a/tests/unit/proxy/utils/prisma_and_spend/test_password_helpers.py b/tests/unit/proxy/utils/prisma_and_spend/test_password_helpers.py
index f241cd00cf5..2df009a72a9 100644
--- a/tests/unit/proxy/utils/prisma_and_spend/test_password_helpers.py
+++ b/tests/unit/proxy/utils/prisma_and_spend/test_password_helpers.py
@@ -4,6 +4,7 @@ Symbols pinned here:
- ``hash_token``
- ``hash_password``
- ``verify_password``
+ - ``run_off_loop_if_fips``
- ``migrate_passwords_to_scrypt_async``
- ``_hash_token_if_needed``
- ``PrismaClient._is_sha256_hex`` (a nested helper inside
@@ -13,9 +14,12 @@ Symbols pinned here:
from __future__ import annotations
+import asyncio
import hashlib
+import ssl
+import threading
from types import SimpleNamespace
-from typing import List
+from typing import Final, List
from unittest.mock import AsyncMock, MagicMock
import pytest
@@ -25,9 +29,17 @@ from litellm.proxy.utils import (
hash_password,
hash_token,
migrate_passwords_to_scrypt_async,
+ run_off_loop_if_fips,
verify_password,
)
+PASSWORD: Final = "Seed-Lamp-4-River"
+PBKDF2_ROW: Final = "pbkdf2:AAECAwQFBgcICQoLDA0OD+uxhNqRo/3ZFLEaKARqFBT+N3Kqek/SGM/5AvocSN0T"
+EIGHT_CHARACTER_PBKDF2_ROW: Final = "pbkdf2:AAECAwQFBgcICQoLDA0ODxhpezknSiGV0cY8wgaSXYf/xzxcPw49uGdOt/TBvPPH"
+SEVEN_CHARACTER_PBKDF2_ROW: Final = "pbkdf2:AAECAwQFBgcICQoLDA0ODxFIwl/XKIAeyWEnfqz8H9vNQX4uo7/u1HiJhB7Ro+N7"
+SCRYPT_ROW: Final = "scrypt:AAECAwQFBgcICQoLDA0OD7hIwlM/k8ytnffjFDt6AuO+BvqtmZvDkWXJP+A3bf0N"
+SHA256_ROW: Final = hashlib.sha256(PASSWORD.encode()).hexdigest()
+
def test_hash_token_returns_sha256_hex_of_input() -> None:
token = "sk-abcDEF12345"
@@ -217,3 +229,133 @@ async def test_migrate_passwords_raises_on_db_failure() -> None:
pc.db.litellm_usertable.find_many = AsyncMock(side_effect=RuntimeError("db unavailable"))
with pytest.raises(RuntimeError, match="db unavailable"):
await migrate_passwords_to_scrypt_async(pc)
+
+
+@pytest.mark.usefixtures("fips_openssl")
+def test_a_fips_openssl_without_fips_mode_stays_on_scrypt() -> None:
+ assert hash_password(PASSWORD).startswith("scrypt:")
+
+
+@pytest.mark.usefixtures("fips_mode")
+def test_fips_hash_is_pbkdf2_sha256_at_600k_salted_by_openssl(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(ssl, "RAND_bytes", lambda size: bytes(range(size)))
+ assert hash_password(PASSWORD) == PBKDF2_ROW
+
+
+@pytest.mark.usefixtures("fips_mode")
+def test_fips_hash_salts_every_row_and_verifies() -> None:
+ first: Final = hash_password(PASSWORD)
+ assert first != hash_password(PASSWORD)
+ assert verify_password(PASSWORD, first)
+
+
+@pytest.mark.usefixtures("fips_mode")
+def test_fips_verify_needs_the_right_password_of_at_least_eight_characters() -> None:
+ outcome: Final = {
+ "right": verify_password(PASSWORD, PBKDF2_ROW),
+ "wrong": verify_password("Seed-Lamp-4-Rivet", PBKDF2_ROW),
+ "lone_surrogate": verify_password("\ud800" + PASSWORD, PBKDF2_ROW),
+ "eight_characters": verify_password("Seed-Lam", EIGHT_CHARACTER_PBKDF2_ROW),
+ "seven_characters": verify_password("Seed-La", SEVEN_CHARACTER_PBKDF2_ROW),
+ }
+ assert outcome == {
+ "right": True,
+ "wrong": False,
+ "lone_surrogate": False,
+ "eight_characters": True,
+ "seven_characters": False,
+ }
+
+
+@pytest.mark.usefixtures("fips_mode")
+@pytest.mark.parametrize(
+ "row",
+ [
+ SCRYPT_ROW,
+ SHA256_ROW,
+ PASSWORD,
+ "pbkdf2:",
+ PBKDF2_ROW[:-1],
+ PBKDF2_ROW + "A",
+ PBKDF2_ROW + "\n",
+ "pbkdf2:" + "@" * 64,
+ "pbkdf2:sha256:1000:AAECAwQFBgcICQoLDA0ODw==:67GE2pGj/dkUsRooBGoUFP43cqp6T9IYz/kC+hxI3RM=",
+ ],
+ ids=[
+ "scrypt",
+ "sha256",
+ "plaintext",
+ "prefix-only",
+ "short",
+ "long",
+ "newline",
+ "not-base64",
+ "stored-iterations",
+ ],
+)
+def test_fips_refuses_any_other_row_with_one_error_that_hides_it(caplog: pytest.LogCaptureFixture, row: str) -> None:
+ with caplog.at_level("ERROR", logger="LiteLLM Proxy"):
+ assert verify_password(PASSWORD, row) is False
+ assert [record.levelname for record in caplog.records] == ["ERROR"]
+ assert row not in caplog.text
+
+
+@pytest.mark.parametrize(
+ "row, verifies",
+ [(SCRYPT_ROW, True), (SHA256_ROW, True), (PBKDF2_ROW, False)],
+ ids=["scrypt", "sha256", "pbkdf2"],
+)
+def test_standard_mode_verifies_the_rows_it_always_did(row: str, verifies: bool) -> None:
+ assert verify_password(PASSWORD, row) is verifies
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_fips_boot_migration_writes_no_row() -> None:
+ pc: Final = MagicMock()
+ pc.db.litellm_usertable.find_many = AsyncMock(
+ return_value=[_make_user("plaintext-user", PASSWORD), _make_user("fips-user", PBKDF2_ROW)]
+ )
+ pc.db.litellm_usertable.update = AsyncMock()
+
+ await migrate_passwords_to_scrypt_async(pc)
+
+ pc.db.litellm_usertable.update.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_standard_boot_migration_hashes_a_plaintext_row_but_not_a_pbkdf2_row() -> None:
+ pc: Final = MagicMock()
+ pc.db.litellm_usertable.find_many = AsyncMock(
+ return_value=[_make_user("plaintext-user", "pbkdf2:plaintext"), _make_user("fips-user", PBKDF2_ROW)]
+ )
+ pc.db.litellm_usertable.update = AsyncMock()
+
+ await migrate_passwords_to_scrypt_async(pc)
+
+ updated: Final = pc.db.litellm_usertable.update.await_args_list
+ assert [update.kwargs["where"]["user_id"] for update in updated] == ["plaintext-user"]
+
+
+def _on_event_loop() -> bool:
+ return asyncio._get_running_loop() is not None
+
+
+@pytest.mark.asyncio
+async def test_password_kdf_runs_on_the_event_loop_in_standard_mode() -> None:
+ assert await run_off_loop_if_fips(_on_event_loop) is True
+
+
+@pytest.mark.asyncio
+@pytest.mark.usefixtures("fips_mode")
+async def test_password_kdfs_leave_the_event_loop_on_a_single_thread_in_fips_mode() -> None:
+ release_first: Final = threading.Event()
+ first_kdf: Final = asyncio.ensure_future(run_off_loop_if_fips(lambda: release_first.wait(timeout=5)))
+ second_kdf: Final = asyncio.ensure_future(run_off_loop_if_fips(_on_event_loop))
+ await asyncio.sleep(0)
+ kdf_threads: Final = [thread for thread in threading.enumerate() if thread.name.startswith("fips-password-kdf")]
+ release_first.set()
+
+ assert len(kdf_threads) == 1
+ assert await first_kdf is True
+ assert await second_kdf is False
diff --git a/tests/unit/responses/litellm_completion_transformation/test_litellm_completion_responses.py b/tests/unit/responses/litellm_completion_transformation/test_litellm_completion_responses.py
index 2427e7eacb1..d5860b7519b 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_litellm_completion_responses.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_litellm_completion_responses.py
@@ -34,7 +34,7 @@ from litellm.responses.litellm_completion_transformation.transformation import (
LiteLLMCompletionResponsesConfig,
)
from litellm.types.llms.base import HiddenParams
-from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
+from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams, ResponsesAPIResponse
from litellm.types.responses.main import OutputToolSearchCall, build_web_search_call
from litellm.types.utils import (
ChatCompletionMessageToolCall,
@@ -4957,6 +4957,159 @@ def test_function_call_tool_id_falls_back_to_unique_id_for_degenerate_call_id():
assert convert(openai)["id"] == "call_tokyo"
+_WEATHER_TOOL_USE: Final = {
+ "type": "tool_use",
+ "id": "toolu_01Weather",
+ "name": "get_weather",
+ "input": {"city": "Paris"},
+}
+_FETCHED_URL: Final = "https://example.com/"
+_BLOCKED_URL: Final = "https://blocked.invalid/"
+_ANTHROPIC_WEB_FETCH_TURN: Final[tuple[dict[str, object], ...]] = (
+ {"type": "server_tool_use", "id": "srvtoolu_01Fetch", "name": "web_fetch", "input": {"url": _FETCHED_URL}},
+ {
+ "type": "web_fetch_tool_result",
+ "tool_use_id": "srvtoolu_01Fetch",
+ "content": {
+ "type": "web_fetch_result",
+ "url": _FETCHED_URL,
+ "content": {"type": "document", "source": {"type": "text", "media_type": "text/plain", "data": "Example"}},
+ },
+ },
+ {"type": "server_tool_use", "id": "srvtoolu_02Fetch", "name": "web_fetch", "input": {"url": _BLOCKED_URL}},
+ {
+ "type": "web_fetch_tool_result",
+ "tool_use_id": "srvtoolu_02Fetch",
+ "content": {"type": "web_fetch_tool_result_error", "error_code": "url_not_allowed"},
+ },
+ {"type": "text", "text": "The page is titled Example."},
+ _WEATHER_TOOL_USE,
+)
+
+
+def _anthropic_turn_on_responses(content: Sequence[Mapping[str, object]]) -> ResponsesAPIResponse:
+ from unittest.mock import MagicMock
+
+ import httpx
+
+ from litellm.llms.anthropic.chat.transformation import AnthropicConfig
+
+ anthropic_response: Final = {
+ "id": "msg_01Fetch",
+ "type": "message",
+ "role": "assistant",
+ "model": "claude-sonnet-5-5",
+ "content": list(content),
+ "stop_reason": "tool_use",
+ "stop_sequence": None,
+ "usage": {"input_tokens": 100, "output_tokens": 50},
+ }
+ raw_response: Final = MagicMock(spec=httpx.Response)
+ raw_response.json.return_value = anthropic_response
+ raw_response.status_code = 200
+ raw_response.headers = {}
+ chat_response: Final = AnthropicConfig().transform_parsed_response(
+ completion_response=anthropic_response,
+ raw_response=raw_response,
+ model_response=ModelResponse(),
+ json_mode=False,
+ prefix_prompt=None,
+ )
+ return LiteLLMCompletionResponsesConfig.transform_chat_completion_response_to_responses_api_response(
+ request_input="Use the web and check the weather in Paris.",
+ responses_api_request={
+ "tools": [
+ {"type": "web_search_20250305", "name": "web_search"},
+ {"type": "web_fetch_20250910", "name": "web_fetch"},
+ {"type": "function", "name": "get_weather", "parameters": {"type": "object"}},
+ ]
+ },
+ chat_completion_response=chat_response,
+ )
+
+
+def test_anthropic_web_fetch_is_a_hosted_open_page_call_on_responses() -> None:
+ response: Final = _anthropic_turn_on_responses(_ANTHROPIC_WEB_FETCH_TURN)
+
+ assert [
+ (item.id, item.status, item.action.model_dump()) for item in response.output if item.type == "web_search_call"
+ ] == [
+ ("ws_srvtoolu_01Fetch", "completed", {"type": "open_page", "url": _FETCHED_URL}),
+ ("ws_srvtoolu_02Fetch", "failed", {"type": "open_page", "url": _BLOCKED_URL}),
+ ]
+ assert [item.name for item in response.output if item.type == "function_call"] == ["get_weather"]
+
+
+@pytest.mark.parametrize(
+ ("server_tool_use", "expected_action"),
+ [
+ (
+ {
+ "type": "server_tool_use",
+ "id": "srvtoolu_01Pending",
+ "name": "web_fetch",
+ "input": {"url": _FETCHED_URL},
+ },
+ {"type": "open_page", "url": _FETCHED_URL},
+ ),
+ (
+ {"type": "server_tool_use", "id": "srvtoolu_01Pending", "name": "web_search", "input": {"query": "python"}},
+ {"type": "search", "query": "python"},
+ ),
+ ],
+)
+def test_anthropic_server_web_call_without_a_result_is_an_in_progress_hosted_call_on_responses(
+ server_tool_use: Mapping[str, object], expected_action: Mapping[str, object]
+) -> None:
+ response: Final = _anthropic_turn_on_responses((server_tool_use, _WEATHER_TOOL_USE))
+
+ web_calls: Final = [item for item in response.output if item.type == "web_search_call"]
+ assert [(item.id, item.status) for item in web_calls] == [("ws_srvtoolu_01Pending", "in_progress")]
+ assert expected_action.items() <= web_calls[0].action.model_dump().items()
+ assert [item.name for item in response.output if item.type == "function_call"] == ["get_weather"]
+
+
+def test_anthropic_code_execution_is_never_a_client_function_call_on_responses() -> None:
+ response: Final = _anthropic_turn_on_responses(
+ (
+ {
+ "type": "server_tool_use",
+ "id": "srvtoolu_01Bash",
+ "name": "bash_code_execution",
+ "input": {"command": "ls"},
+ },
+ {
+ "type": "bash_code_execution_tool_result",
+ "tool_use_id": "srvtoolu_01Bash",
+ "content": {
+ "type": "bash_code_execution_result",
+ "stdout": "notes.txt\n",
+ "stderr": "",
+ "return_code": 0,
+ },
+ },
+ {
+ "type": "server_tool_use",
+ "id": "srvtoolu_02Edit",
+ "name": "text_editor_code_execution",
+ "input": {"command": "view", "path": "notes.txt"},
+ },
+ {
+ "type": "server_tool_use",
+ "id": "srvtoolu_03Bash",
+ "name": "bash_code_execution",
+ "input": {"command": "pwd"},
+ },
+ _WEATHER_TOOL_USE,
+ )
+ )
+
+ assert [(item.type, getattr(item, "call_id", item.id)) for item in response.output if item.type != "message"] == [
+ ("code_interpreter_call", "srvtoolu_01Bash"),
+ ("function_call", "toolu_01Weather"),
+ ]
+
+
class TestHostedWebSearchReplay:
def test_emitted_hosted_search_output_round_trips_with_client_tool_result(self) -> None:
search_result: Final = {
@@ -5027,6 +5180,75 @@ class TestHostedWebSearchReplay:
assert messages[2]["content"] == "Paris is sunny."
assert history == original
+ @pytest.mark.parametrize(
+ "leading_text", ((), ({"type": "text", "text": "Let me look that up."},)), ids=("alone", "after_text")
+ )
+ @pytest.mark.parametrize(
+ ("server_tool_use", "expected_input"),
+ (
+ (
+ {"type": "server_tool_use", "id": "srvtoolu_01Pending", "name": "web_fetch", "input": {"url": _FETCHED_URL}},
+ {"url": _FETCHED_URL},
+ ),
+ (
+ {"type": "server_tool_use", "id": "srvtoolu_01Pending", "name": "web_search", "input": {"query": "python"}},
+ {"query": "python"},
+ ),
+ ),
+ ids=("web_fetch", "web_search"),
+ )
+ def test_pending_claude_web_call_replays_to_anthropic_as_its_own_server_tool_use(
+ self,
+ leading_text: tuple[Mapping[str, object], ...],
+ server_tool_use: Mapping[str, object],
+ expected_input: Mapping[str, object],
+ ) -> None:
+ response: Final = _anthropic_turn_on_responses((*leading_text, server_tool_use, _WEATHER_TOOL_USE))
+ history: Final = [
+ {"role": "user", "content": "Use the web and check the weather in Paris."},
+ *(item.model_dump(exclude_none=True) for item in response.output),
+ {"type": "function_call_output", "call_id": "toolu_01Weather", "output": "Paris is sunny."},
+ ]
+
+ anthropic_turns: Final = anthropic_messages_pt(
+ messages=LiteLLMCompletionResponsesConfig.transform_responses_api_input_to_messages(
+ input=history, responses_api_request={}
+ ),
+ model="claude-sonnet-5-5",
+ llm_provider="anthropic",
+ )
+
+ assert [turn["role"] for turn in anthropic_turns] == ["user", "assistant", "user"]
+ assert [block for block in anthropic_turns[1]["content"] if block["type"] != "text"] == [
+ {"type": "server_tool_use", "id": "srvtoolu_01Pending", "name": server_tool_use["name"], "input": expected_input},
+ {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {"city": "Paris"}},
+ ]
+ assert [(block["type"], block["tool_use_id"]) for block in anthropic_turns[2]["content"]] == [
+ ("tool_result", "toolu_01Weather")
+ ]
+
+ @pytest.mark.parametrize(
+ ("call_id", "status"),
+ (("ws_srvtoolu_01Fetch", "completed"), ("ws_68a1b2c3d4", "in_progress"), ("srvtoolu_01Fetch", "in_progress")),
+ ids=("completed-claude-call", "pending-non-claude-call", "pending-unprefixed-id"),
+ )
+ def test_web_call_that_is_not_a_pending_claude_call_replays_as_text_only(
+ self, call_id: str, status: Literal["completed", "in_progress"]
+ ) -> None:
+ search: Final = ResponseFunctionWebSearch(
+ id=call_id, type="web_search_call", status=status, action=ActionOpenPage(type="open_page", url=_FETCHED_URL)
+ )
+
+ messages: Final = (
+ LiteLLMCompletionResponsesConfig._transform_responses_api_input_item_to_chat_completion_message(
+ input_item=search.model_dump(exclude_none=True)
+ )
+ )
+
+ assert messages == [
+ {"role": "assistant", "content": "Hosted web search: " + search.model_dump_json(exclude_none=True)}
+ ]
+
@pytest.mark.parametrize(
"action",
(
diff --git a/tests/unit/responses/litellm_completion_transformation/test_reasoning_items.py b/tests/unit/responses/litellm_completion_transformation/test_reasoning_items.py
index 093d1744418..580ee672500 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_reasoning_items.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_reasoning_items.py
@@ -1,12 +1,18 @@
import json
+import pytest
+
from litellm.responses.litellm_completion_transformation.reasoning_items import (
decode_thinking_blocks,
encode_thinking_blocks,
is_litellm_minted_reasoning_item,
is_minted_reasoning_item_id,
+ is_other_provider_reasoning_item,
mint_reasoning_item_id,
)
+from litellm.responses.litellm_completion_transformation.transformation import LiteLLMCompletionResponsesConfig
+from litellm.responses.utils import ResponsesAPIRequestUtils
+from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
A_PROVIDER_OWNED_REASONING_ITEM_ID = "rs_08d3a89dbb92277a006abf04f4266087d0b4eedacd7848f306"
A_PROVIDER_OWNED_ENCRYPTED_BLOB = "gAAAAABo-opaque-provider-blob"
@@ -58,5 +64,82 @@ def test_a_reasoning_item_is_litellm_minted_by_its_id_or_by_its_encoded_thinking
"encrypted_content": A_PROVIDER_OWNED_ENCRYPTED_BLOB,
}
)
- assert not is_litellm_minted_reasoning_item({"type": "message", "id": mint_reasoning_item_id(), "role": "assistant"})
+ assert not is_litellm_minted_reasoning_item(
+ {"type": "message", "id": mint_reasoning_item_id(), "role": "assistant"}
+ )
assert not is_litellm_minted_reasoning_item("a bare string input")
+
+
+SUMMARY = [{"type": "summary_text", "text": "Paris first."}]
+PROVIDER_OWNED_ITEM = {
+ "type": "reasoning",
+ "id": A_PROVIDER_OWNED_REASONING_ITEM_ID,
+ "summary": SUMMARY,
+ "encrypted_content": A_PROVIDER_OWNED_ENCRYPTED_BLOB,
+}
+
+
+@pytest.mark.parametrize(
+ "item",
+ [
+ pytest.param({"type": "reasoning", "id": mint_reasoning_item_id(), "summary": SUMMARY}, id="minted id"),
+ pytest.param(
+ {"type": "reasoning", "summary": SUMMARY, "encrypted_content": encode_thinking_blocks([SIGNED_BLOCK])},
+ id="encoded thinking blocks",
+ ),
+ pytest.param(
+ {"type": "reasoning", "summary": [], "content": [{"type": "reasoning_text", "text": "Paris first."}]},
+ id="plain-text reasoning content",
+ ),
+ pytest.param(
+ {
+ **PROVIDER_OWNED_ITEM,
+ "encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
+ A_PROVIDER_OWNED_ENCRYPTED_BLOB, "another-deployment"
+ ),
+ },
+ id="blob wrapped with a litellm deployment id",
+ ),
+ ],
+)
+def test_reasoning_items_another_provider_produced_are_recognized(item):
+ assert is_other_provider_reasoning_item(item)
+
+
+@pytest.mark.parametrize(
+ "item",
+ [
+ pytest.param(PROVIDER_OWNED_ITEM, id="provider-owned blob"),
+ pytest.param({"type": "reasoning", "summary": SUMMARY}, id="summary only, no id"),
+ pytest.param(
+ {"type": "reasoning", "summary": SUMMARY, "content": None, "encrypted_content": None}, id="nulled replay"
+ ),
+ pytest.param({"type": "reasoning", "summary": SUMMARY, "content": []}, id="empty content"),
+ pytest.param(
+ {"type": "message", "id": mint_reasoning_item_id(), "role": "user", "content": "hi"}, id="not reasoning"
+ ),
+ pytest.param("a stray string", id="not an object"),
+ ],
+)
+def test_reasoning_items_the_provider_can_replay_are_kept(item):
+ assert not is_other_provider_reasoning_item(item)
+
+
+def test_signed_thinking_survives_the_model_id_wrap_on_its_way_back_to_the_chat_completions_bridge():
+ wrapped = ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
+ encode_thinking_blocks([SIGNED_BLOCK]), "anthropic-deployment"
+ )
+ messages = LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request(
+ model="anthropic/claude-sonnet-5-5",
+ input=[
+ {"role": "user", "content": "Capital of France?"},
+ {"type": "reasoning", "id": mint_reasoning_item_id(), "summary": [], "encrypted_content": wrapped},
+ {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "Paris."}]},
+ {"role": "user", "content": "And of Italy?"},
+ ],
+ responses_api_request=ResponsesAPIOptionalRequestParams(),
+ )["messages"]
+
+ assert [message.get("thinking_blocks") for message in messages if message["role"] == "assistant"] == [
+ [SIGNED_BLOCK]
+ ]
diff --git a/tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.py b/tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.py
index 87858741908..1a6e3cc7832 100644
--- a/tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.py
@@ -356,6 +356,319 @@ async def test_web_search_stream_preserves_hosted_and_client_calls(sync_mode, to
)
+_STREAMED_WEB_FETCH_CALL: Final[tuple[dict[str, object], ...]] = (
+ {
+ "type": "message_start",
+ "message": {
+ "id": "msg_01Fetch",
+ "type": "message",
+ "role": "assistant",
+ "content": [],
+ "usage": {"input_tokens": 10, "output_tokens": 1},
+ },
+ },
+ {
+ "type": "content_block_start",
+ "index": 0,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_01Fetch", "name": "web_fetch"},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 0,
+ "delta": {"type": "input_json_delta", "partial_json": '{"url": "https://example.com/"}'},
+ },
+ {"type": "content_block_stop", "index": 0},
+)
+_STREAMED_WEB_FETCH_RESULT: Final[tuple[dict[str, object], ...]] = (
+ {
+ "type": "content_block_start",
+ "index": 1,
+ "content_block": {
+ "type": "web_fetch_tool_result",
+ "tool_use_id": "srvtoolu_01Fetch",
+ "content": {"type": "web_fetch_result", "url": "https://example.com/"},
+ },
+ },
+ {"type": "content_block_stop", "index": 1},
+)
+
+
+def _streamed_text_then_weather_call(first_index: int) -> tuple[dict[str, object], ...]:
+ return (
+ {"type": "content_block_start", "index": first_index, "content_block": {"type": "text", "text": ""}},
+ {"type": "content_block_delta", "index": first_index, "delta": {"type": "text_delta", "text": "Fetched."}},
+ {"type": "content_block_stop", "index": first_index},
+ {
+ "type": "content_block_start",
+ "index": first_index + 1,
+ "content_block": {"type": "tool_use", "id": "toolu_01Weather", "name": "get_weather", "input": {}},
+ },
+ {
+ "type": "content_block_delta",
+ "index": first_index + 1,
+ "delta": {"type": "input_json_delta", "partial_json": '{"city": "Paris"}'},
+ },
+ {"type": "content_block_stop", "index": first_index + 1},
+ {
+ "type": "message_delta",
+ "delta": {"stop_reason": "tool_use", "stop_sequence": None},
+ "usage": {"output_tokens": 20},
+ },
+ )
+
+
+def _anthropic_web_fetch_stream_events(fetch_finished: bool) -> tuple[dict[str, object], ...]:
+ if fetch_finished:
+ return (*_STREAMED_WEB_FETCH_CALL, *_STREAMED_WEB_FETCH_RESULT, *_streamed_text_then_weather_call(2))
+ return (*_STREAMED_WEB_FETCH_CALL, *_streamed_text_then_weather_call(1))
+
+
+@pytest.mark.parametrize("sync_mode", [True, False])
+@pytest.mark.parametrize(
+ ("fetch_finished", "expected_status", "expected_search_events"),
+ [
+ pytest.param(
+ True,
+ "completed",
+ [
+ ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
+ ResponsesAPIStreamEvents.WEB_SEARCH_CALL_IN_PROGRESS,
+ ResponsesAPIStreamEvents.WEB_SEARCH_CALL_SEARCHING,
+ ResponsesAPIStreamEvents.WEB_SEARCH_CALL_COMPLETED,
+ ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
+ ],
+ id="fetch_finished_in_this_turn",
+ ),
+ pytest.param(
+ False,
+ "in_progress",
+ [
+ ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
+ ResponsesAPIStreamEvents.WEB_SEARCH_CALL_IN_PROGRESS,
+ ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
+ ],
+ id="fetch_still_waiting_on_its_result",
+ ),
+ ],
+)
+@pytest.mark.asyncio
+async def test_streamed_anthropic_web_fetch_is_a_hosted_open_page_call(
+ sync_mode: bool,
+ fetch_finished: bool,
+ expected_status: str,
+ expected_search_events: list[ResponsesAPIStreamEvents],
+) -> None:
+ from litellm.llms.anthropic.chat.handler import ModelResponseIterator
+
+ parser: Final = ModelResponseIterator(streaming_response=MagicMock(), sync_stream=sync_mode, json_mode=False)
+ chunks: Final = [parser.chunk_parser(event) for event in _anthropic_web_fetch_stream_events(fetch_finished)]
+ iterator: Final = LiteLLMCompletionStreamingIterator(
+ model="claude-sonnet-5-5",
+ litellm_custom_stream_wrapper=_FakeStreamWrapper(chunks),
+ request_input="Fetch example.com and check the weather in Paris.",
+ responses_api_request={
+ "tools": [
+ {"type": "web_fetch_20250910", "name": "web_fetch"},
+ {"type": "function", "name": "get_weather", "parameters": {"type": "object"}},
+ ]
+ },
+ custom_llm_provider="anthropic",
+ )
+
+ events: Final = await _collect_events(iterator, sync_mode)
+
+ completed: Final = next(
+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED
+ )
+ assert [
+ (item.id, item.status, item.action.model_dump())
+ for item in completed.response.output
+ if item.type == "web_search_call"
+ ] == [("ws_srvtoolu_01Fetch", expected_status, {"type": "open_page", "url": "https://example.com/"})]
+ assert [
+ event.type
+ for event in events
+ if getattr(event, "item_id", getattr(getattr(event, "item", None), "id", None)) == "ws_srvtoolu_01Fetch"
+ ] == expected_search_events
+ assert [item.name for item in completed.response.output if item.type == "function_call"] == ["get_weather"]
+ assert not any(
+ event.item.name == "web_fetch"
+ for event in events
+ if getattr(event, "type", None) == ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED
+ and event.item.type == "function_call"
+ )
+
+
+_STREAMED_PENDING_CODE_EXECUTION: Final[tuple[dict[str, object], ...]] = (
+ _STREAMED_WEB_FETCH_CALL[0],
+ {
+ "type": "content_block_start",
+ "index": 0,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_01Bash", "name": "bash_code_execution"},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 0,
+ "delta": {"type": "input_json_delta", "partial_json": '{"command": "ls"}'},
+ },
+ {"type": "content_block_stop", "index": 0},
+ *_streamed_text_then_weather_call(1),
+)
+
+
+@pytest.mark.parametrize("sync_mode", [True, False])
+@pytest.mark.asyncio
+async def test_streamed_anthropic_code_execution_is_never_a_client_function_call(sync_mode: bool) -> None:
+ from litellm.llms.anthropic.chat.handler import ModelResponseIterator
+
+ parser: Final = ModelResponseIterator(streaming_response=MagicMock(), sync_stream=sync_mode, json_mode=False)
+ iterator: Final = LiteLLMCompletionStreamingIterator(
+ model="claude-sonnet-5-5",
+ litellm_custom_stream_wrapper=_FakeStreamWrapper(
+ [parser.chunk_parser(event) for event in _STREAMED_PENDING_CODE_EXECUTION]
+ ),
+ request_input="List the files, then check the weather in Paris.",
+ responses_api_request={
+ "tools": [
+ {"type": "code_execution_20250825", "name": "code_execution"},
+ {"type": "function", "name": "get_weather", "parameters": {"type": "object"}},
+ ]
+ },
+ custom_llm_provider="anthropic",
+ )
+
+ events: Final = await _collect_events(iterator, sync_mode)
+
+ item_events: Final = [event for event in events if getattr(event, "item", None) is not None]
+ assert {event.item.call_id for event in item_events if event.item.type == "function_call"} == {"toolu_01Weather"}
+ completed: Final = next(
+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED
+ )
+ assert [item.call_id for item in completed.response.output if item.type == "function_call"] == ["toolu_01Weather"]
+
+
+def _streamed_anthropic_tool_search(search_arguments: str) -> tuple[dict[str, object], ...]:
+ half: Final = len(search_arguments) // 2
+ return (
+ _STREAMED_WEB_FETCH_CALL[0],
+ {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
+ {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Searching."}},
+ {"type": "content_block_stop", "index": 0},
+ {
+ "type": "content_block_start",
+ "index": 1,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_01Search", "name": "tool_search_tool_regex"},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 1,
+ "delta": {"type": "input_json_delta", "partial_json": search_arguments[:half]},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 1,
+ "delta": {"type": "input_json_delta", "partial_json": search_arguments[half:]},
+ },
+ {"type": "content_block_stop", "index": 1},
+ *_streamed_text_then_weather_call(2)[3:],
+ )
+
+
+@pytest.mark.parametrize("sync_mode", [True, False])
+@pytest.mark.parametrize(
+ ("search_arguments", "expected_status", "expected_arguments"),
+ [
+ pytest.param('{"pattern": "weather"}', "completed", {"pattern": "weather"}, id="arguments_parse"),
+ pytest.param('{"pattern": ', "incomplete", {}, id="arguments_cut_off"),
+ ],
+)
+@pytest.mark.asyncio
+async def test_streamed_anthropic_tool_search_keeps_its_place_before_the_function_call(
+ sync_mode: bool, search_arguments: str, expected_status: str, expected_arguments: dict[str, str]
+) -> None:
+ from litellm.llms.anthropic.chat.handler import ModelResponseIterator
+
+ parser: Final = ModelResponseIterator(streaming_response=MagicMock(), sync_stream=sync_mode, json_mode=False)
+ iterator: Final = LiteLLMCompletionStreamingIterator(
+ model="claude-sonnet-5-5",
+ litellm_custom_stream_wrapper=_FakeStreamWrapper(
+ [parser.chunk_parser(event) for event in _streamed_anthropic_tool_search(search_arguments)]
+ ),
+ request_input="What is the weather in Paris right now?",
+ responses_api_request={
+ "tools": [
+ {"type": "tool_search", "execution": "server"},
+ {"type": "function", "name": "get_weather", "parameters": {"type": "object"}, "defer_loading": True},
+ ]
+ },
+ custom_llm_provider="anthropic",
+ )
+
+ events: Final = await _collect_events(iterator, sync_mode)
+
+ completed: Final = next(
+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED
+ )
+ output: Final = completed.response.output
+ assert [item.type for item in output] == ["message", "tool_search_call", "function_call"]
+ assert (output[1].status, output[1].arguments) == (expected_status, expected_arguments)
+ search_item_events: Final = [
+ event for event in events if getattr(getattr(event, "item", None), "type", None) == "tool_search_call"
+ ]
+ assert [(event.type, event.output_index) for event in search_item_events] == [
+ (ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED, 1),
+ (ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE, 1),
+ ]
+
+
+_STREAMED_PAUSED_WEB_SEARCH: Final[tuple[dict[str, object], ...]] = (
+ _STREAMED_WEB_FETCH_CALL[0],
+ {
+ "type": "content_block_start",
+ "index": 0,
+ "content_block": {"type": "server_tool_use", "id": "srvtoolu_01Pending", "name": "web_search"},
+ },
+ {
+ "type": "content_block_delta",
+ "index": 0,
+ "delta": {"type": "input_json_delta", "partial_json": '{"query": "next city"}'},
+ },
+ {"type": "content_block_stop", "index": 0},
+ {
+ "type": "message_delta",
+ "delta": {"stop_reason": "pause_turn", "stop_sequence": None},
+ "usage": {"output_tokens": 5},
+ },
+)
+
+
+@pytest.mark.parametrize("sync_mode", [True, False])
+@pytest.mark.asyncio
+async def test_streamed_paused_anthropic_web_search_lists_only_the_unrun_search(sync_mode: bool) -> None:
+ from litellm.llms.anthropic.chat.handler import ModelResponseIterator
+
+ parser: Final = ModelResponseIterator(streaming_response=MagicMock(), sync_stream=sync_mode, json_mode=False)
+ iterator: Final = LiteLLMCompletionStreamingIterator(
+ model="claude-sonnet-5-5",
+ litellm_custom_stream_wrapper=_FakeStreamWrapper(
+ [parser.chunk_parser(event) for event in _STREAMED_PAUSED_WEB_SEARCH]
+ ),
+ request_input="Find the next city.",
+ responses_api_request={"tools": [{"type": "web_search_20250305", "name": "web_search"}]},
+ custom_llm_provider="anthropic",
+ )
+
+ events: Final = await _collect_events(iterator, sync_mode)
+
+ completed: Final = next(
+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED
+ )
+ assert [(item.type, item.status, item.action.query) for item in completed.response.output] == [
+ ("web_search_call", "in_progress", "next city")
+ ]
+ assert not any(_is_message_item(event) for event in events)
+
+
def test_tool_calls_present_only_in_final_response_are_emitted_before_completed():
iterator = LiteLLMCompletionStreamingIterator(
model="test-model",
diff --git a/tests/unit/responses/test_additional_tools.py b/tests/unit/responses/test_additional_tools.py
index 290670a7c0a..d2c8cbacf78 100644
--- a/tests/unit/responses/test_additional_tools.py
+++ b/tests/unit/responses/test_additional_tools.py
@@ -1,8 +1,9 @@
from typing import Final
+import pytest
from pydantic import JsonValue
-from litellm.responses.additional_tools import hoist_additional_tools
+from litellm.responses.additional_tools import hoist_additional_tools, with_additional_tools_hoisted
_EXEC_TOOL = {
"type": "custom",
@@ -134,3 +135,10 @@ def test_tool_list_repeated_on_every_turn_is_sent_as_is_without_drop_params() ->
assert hoisted.tools == (_TOP_LEVEL_TOOL, _function("exec", "old"), _function("exec", "new"))
assert hoisted.input == [_USER_MESSAGE, _USER_MESSAGE]
+
+
+@pytest.mark.parametrize("tools", [7, "exec", {"type": "function"}], ids=["int", "str", "dict"])
+def test_request_whose_top_level_tools_is_not_a_list_stays_as_sent_for_the_provider_to_refuse(tools: object) -> None:
+ request: Final = {"input": [_tool_list(_function("exec")), _USER_MESSAGE], "tools": tools}
+
+ assert with_additional_tools_hoisted(request, drop_params=True) == request
diff --git a/tests/unit/responses/test_streaming_iterator_error_events.py b/tests/unit/responses/test_streaming_iterator_error_events.py
index 14ecb185f06..0acf5d9ce62 100644
--- a/tests/unit/responses/test_streaming_iterator_error_events.py
+++ b/tests/unit/responses/test_streaming_iterator_error_events.py
@@ -37,6 +37,7 @@ from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.get_litellm_params import get_litellm_params
from litellm.litellm_core_utils.litellm_logging import _get_provider_request_id
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
+from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.responses.streaming_iterator import (
_ERROR_CODE_HTTP_STATUS,
BaseResponsesAPIStreamingIterator,
@@ -75,6 +76,83 @@ def _make_error_chunk(error_type: str, code: str, message: str = "err") -> Error
return ErrorEvent(type=ResponsesAPIStreamEvents.ERROR, sequence_number=0, error=error_obj)
+@pytest.mark.parametrize(
+ "error_type,error_code",
+ (
+ ("resource_unavailable", "flex_unavailable"),
+ ("resource_unavailable", None),
+ (None, "flex_unavailable"),
+ ),
+)
+def test_streamed_flex_unavailable_maps_to_429(error_type: str | None, error_code: str | None) -> None:
+ assert _status_code_for_error_fields(error_type, error_code) == 429
+
+
+def test_streamed_unrelated_server_error_still_maps_to_500() -> None:
+ assert _status_code_for_error_fields("server_error", "server_error") == 500
+
+
+FLEX_UNAVAILABLE_ERROR: Final = {
+ "type": "resource_unavailable",
+ "code": "flex_unavailable",
+ "message": "Flex processing is temporarily unavailable. Please try again later or use standard processing.",
+}
+
+
+def _flex_outage_sse(kind: str) -> str:
+ pending: Final = {
+ "id": "resp_flex",
+ "object": "response",
+ "created_at": 1700000000,
+ "model": "gpt-5",
+ "status": "in_progress",
+ "output": [],
+ }
+ failure: Final = (
+ {"type": "error", "sequence_number": 1, "error": FLEX_UNAVAILABLE_ERROR}
+ if kind == "error"
+ else {
+ "type": "response.failed",
+ "sequence_number": 1,
+ "response": {**pending, "status": "failed", "error": FLEX_UNAVAILABLE_ERROR},
+ }
+ )
+ events: Final = ({"type": "response.created", "sequence_number": 0, "response": pending}, failure)
+ return "".join(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n" for event in events)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("kind", ("error", "response.failed"))
+async def test_streamed_flex_outage_raises_rate_limit_like_the_non_streamed_path(
+ kind: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+ litellm.in_memory_llm_clients_cache.flush_cache()
+ async_client: Final = AsyncHTTPHandler(
+ transport=httpx.MockTransport(
+ lambda request: httpx.Response(
+ 200, headers={"content-type": "text/event-stream"}, text=_flex_outage_sse(kind)
+ )
+ )
+ )
+ try:
+ stream: Final = await litellm.aresponses(
+ model="openai/gpt-5",
+ input="hi",
+ service_tier="flex",
+ stream=True,
+ api_key="sk-test",
+ client=async_client,
+ )
+ with pytest.raises(MidStreamFallbackError) as raised:
+ _ = [event async for event in stream]
+ finally:
+ await async_client.close()
+ assert raised.value.status_code == 429, raised.value
+ assert isinstance(raised.value.original_exception, litellm.RateLimitError), raised.value.original_exception
+ assert "Flex processing is temporarily unavailable" in str(raised.value.original_exception)
+
+
@pytest.mark.parametrize("provider", ("openai", "azure", "bedrock"))
@pytest.mark.parametrize("header", ("x-amzn-RequestId", "x-request-id", None))
@pytest.mark.parametrize(
diff --git a/tests/unit/rust_bridge/test_catalog.py b/tests/unit/rust_bridge/test_catalog.py
index 704a4f73fe8..1994dd00709 100644
--- a/tests/unit/rust_bridge/test_catalog.py
+++ b/tests/unit/rust_bridge/test_catalog.py
@@ -5,7 +5,8 @@ from typing import Final
import pytest
-from litellm.rust_bridge import catalog, configuration
+from litellm.litellm_core_utils.model_latest_aliases import get_model_latest_aliases
+from litellm.rust_bridge import catalog, configuration, detectors
from litellm.rust_bridge.catalog import (
Feature,
Route,
@@ -15,6 +16,32 @@ from litellm.rust_bridge.catalog import (
Target,
)
from litellm.rust_bridge.configuration import Decision, Rollout
+from litellm.rust_bridge.public_call import NativeCall
+
+
+@pytest.mark.parametrize(
+ ("model", "expected"),
+ (
+ ("anthropic/claude-opus-latest", Rollout.PYTHON_ONLY),
+ (f"anthropic/{get_model_latest_aliases()['anthropic']['claude-opus-latest']}", Rollout.RUST_OPT_IN),
+ ),
+)
+def test_latest_alias_messages_falls_back_without_changing_the_pinned_model_rollout(
+ model: str,
+ expected: Rollout,
+) -> None:
+ request: Final = NativeCall(args=(), kwargs={"model": model}, base={})
+ context: Final = RouteContext(
+ Route.MESSAGES,
+ provider="anthropic",
+ model=model,
+ features=detectors.detected(detectors.COMMON, request),
+ )
+ rules: Final = (
+ RouteRule(Route.MESSAGES, Rollout.PYTHON_ONLY, features=catalog.MESSAGES_PYTHON_FEATURES),
+ RouteRule(Route.MESSAGES, Rollout.RUST_OPT_IN, providers=frozenset({"anthropic"})),
+ )
+ assert catalog.rollout(context, rules) is expected
@pytest.fixture(autouse=True)
diff --git a/tests/unit/rust_bridge/test_detectors.py b/tests/unit/rust_bridge/test_detectors.py
index e4c39bb40f9..3001d23e57e 100644
--- a/tests/unit/rust_bridge/test_detectors.py
+++ b/tests/unit/rust_bridge/test_detectors.py
@@ -45,6 +45,22 @@ def call(**kwargs: object) -> NativeCall:
return NativeCall(args=(), kwargs=kwargs, base={"model": "gpt-5"})
+@pytest.mark.parametrize(
+ ("model", "provider", "expected"),
+ (
+ ("openai/gpt-luna-latest", None, True),
+ ("gpt-luna-latest", "openai", True),
+ ("gemini/gemini-pro-latest", None, False),
+ ("gpt-luna-latest", None, False),
+ ("anthropic/claude-opus-latest", None, True),
+ ("claude-opus-latest", "anthropic", True),
+ ("", None, False),
+ ),
+)
+def test_floating_alias_requires_python(model: str, provider: str | None, expected: bool) -> None:
+ assert detectors.latest_alias(call(model=model, custom_llm_provider=provider)) is expected
+
+
@pytest.mark.parametrize(
("model", "provider", "tools", "expected"),
(
diff --git a/tests/unit/test_router/core/test_parity_regress5_errors.py b/tests/unit/test_router/core/test_parity_regress5_errors.py
index a0122f8eda6..3e8537af878 100644
--- a/tests/unit/test_router/core/test_parity_regress5_errors.py
+++ b/tests/unit/test_router/core/test_parity_regress5_errors.py
@@ -69,8 +69,9 @@ def cooled(core: RouterCore, *ids: str) -> Mapping[str, Mapping[str, object]]:
async def settle() -> None:
"""Let the tasks a failed stream scheduled run."""
- for _ in range(20):
- await asyncio.sleep(0)
+ while pending := asyncio.all_tasks() - {asyncio.current_task()}:
+ _, still_running = await asyncio.wait(pending, timeout=5)
+ assert not still_running, still_running
async def failed_with(core: RouterCore, **request: object) -> BaseException: # kwargs-ok: request kwargs
diff --git a/tests/unit/test_router/core/test_registry_characterization_model_info.py b/tests/unit/test_router/core/test_registry_characterization_model_info.py
index d79aebf75b9..578183d7f02 100644
--- a/tests/unit/test_router/core/test_registry_characterization_model_info.py
+++ b/tests/unit/test_router/core/test_registry_characterization_model_info.py
@@ -273,6 +273,27 @@ def test_max_input_tokens_leave_out_blocked_deployments_and_other_teams(make_cor
assert core.registry().group("mt", field="max_input_tokens", team_id="TA") == (2, 3)
+def test_pinned_reasoning_efforts_are_each_routable_deployments_own_value_in_order(make_core: MakeCore) -> None:
+ core = make_core(
+ model_list=[
+ deployment("pe", "p1", reasoning_effort="high"),
+ deployment("pe", "p2"),
+ deployment("pe", "p3", reasoning_effort="low"),
+ deployment("pe", "p4", reasoning_effort="minimal", model_info={"blocked": True}),
+ deployment("pe", "p5", reasoning_effort="medium", model_info={"team_id": "TA"}),
+ ]
+ )
+
+ assert core.registry().group("pe", field="pinned_reasoning_efforts") == ("high", None, "low")
+ assert core.registry().group("pe", field="pinned_reasoning_efforts", team_id="TA") == (
+ "high",
+ None,
+ "low",
+ "medium",
+ )
+ assert core.registry().group("not-configured", field="pinned_reasoning_efforts") == ()
+
+
def test_max_input_tokens_skip_a_value_a_saved_model_info_copied_from_the_cost_map(make_core: MakeCore) -> None:
catalog_limit = GetModelCostMap.loaded_model_cost_map()["gpt-4o"]["max_input_tokens"]
core = make_core(
diff --git a/tests/unit/test_router/test_router.py b/tests/unit/test_router/test_router.py
index 271615b18cf..8f8676103b2 100644
--- a/tests/unit/test_router/test_router.py
+++ b/tests/unit/test_router/test_router.py
@@ -1889,6 +1889,281 @@ def test_model_group_info_cost_none_for_unpriced_deployment_but_zero_when_declar
assert priced.output_cost_per_token is not None and priced.output_cost_per_token > 0
+_HUB_PRICES: Final = MappingProxyType(
+ {
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 4e-06,
+ "cache_read_input_token_cost": 1e-07,
+ "cache_creation_input_token_cost": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2e-06,
+ "output_cost_per_token_above_200k_tokens": 8e-06,
+ "input_cost_per_token_priority": 3e-06,
+ "input_cost_per_token_flex": 5e-07,
+ "input_cost_per_token_batches": 5e-07,
+ }
+)
+
+
+def _cost_map_model(key: str, entry: Mapping[str, object]) -> str:
+ litellm.register_model(
+ {key: {"litellm_provider": "openai", "mode": "chat", **entry}},
+ persist_across_reloads=False,
+ add_to_provider_model_sets=False,
+ )
+ return f"openai/{key}"
+
+
+def test_model_group_info_lists_every_cost_map_price_of_a_deployment(local_model_cost_map: None) -> None:
+ model: Final = _cost_map_model("hub-pricing-model", {**_HUB_PRICES, "max_input_tokens": 400000})
+ router: Final = litellm.Router(model_list=[{"model_name": "hub", "litellm_params": {"model": model}}])
+
+ group: Final = router.registry().group("hub")
+
+ assert group is not None
+ assert [(p.provider, dict(p.costs), p.tiered_pricing) for p in group.deployment_pricing] == [
+ ("openai", dict(_HUB_PRICES), ())
+ ]
+
+
+def test_model_group_info_lists_each_deployments_price_once_next_to_the_unchanged_max(
+ local_model_cost_map: None,
+) -> None:
+ model: Final = _cost_map_model("hub-pricing-model", _HUB_PRICES)
+ override: Final = 3 * _HUB_PRICES["input_cost_per_token"]
+ router: Final = litellm.Router(
+ model_list=[
+ {"model_name": "hub", "litellm_params": {"model": model, "api_base": "https://a.example"}},
+ {"model_name": "hub", "litellm_params": {"model": model, "api_base": "https://b.example"}},
+ {"model_name": "hub", "litellm_params": {"model": model, "input_cost_per_token": override}},
+ ]
+ )
+
+ group: Final = router.registry().group("hub")
+
+ assert group is not None
+ assert [p.costs["input_cost_per_token"] for p in group.deployment_pricing] == [
+ _HUB_PRICES["input_cost_per_token"],
+ override,
+ ]
+ assert group.input_cost_per_token == override
+ assert group.output_cost_per_token == _HUB_PRICES["output_cost_per_token"]
+
+
+def _usage_billed_at(cost_key: str, prompt_tokens: int = 1000) -> tuple[Usage, str | None]:
+ match cost_key:
+ case "input_cost_per_token":
+ return Usage(prompt_tokens=prompt_tokens, completion_tokens=0, total_tokens=prompt_tokens), None
+ case "output_cost_per_token":
+ return Usage(prompt_tokens=0, completion_tokens=1000, total_tokens=1000), None
+ case "cache_read_input_token_cost":
+ return Usage(
+ prompt_tokens=1000,
+ completion_tokens=0,
+ total_tokens=1000,
+ prompt_tokens_details={"cached_tokens": 1000},
+ ), None
+ case "cache_creation_input_token_cost":
+ return Usage(
+ prompt_tokens=1000, completion_tokens=0, total_tokens=1000, cache_creation_input_tokens=1000
+ ), None
+ case "input_cost_per_token_priority":
+ return Usage(prompt_tokens=1000, completion_tokens=0, total_tokens=1000), "priority"
+ case "input_cost_per_token_flex":
+ return Usage(prompt_tokens=1000, completion_tokens=0, total_tokens=1000), "flex"
+ case _:
+ raise AssertionError(f"no billing scenario for {cost_key}")
+
+
+def _billed_rate(model: str, deployment_id: str, usage: Usage, service_tier: str | None) -> float:
+ cost: Final = litellm.completion_cost(
+ completion_response=litellm.ModelResponse(usage=usage),
+ model=model,
+ custom_llm_provider="openai",
+ custom_pricing=True,
+ router_model_id=deployment_id,
+ service_tier=service_tier,
+ )
+ return cost / max(usage.prompt_tokens, usage.completion_tokens)
+
+
+def test_model_group_info_shows_an_overridden_deployment_only_the_prices_billing_charges(
+ local_model_cost_map: None,
+) -> None:
+ model: Final = _cost_map_model("hub-pricing-model", _HUB_PRICES)
+ router: Final = litellm.Router(
+ model_list=[
+ {
+ "model_name": "hub",
+ "litellm_params": {
+ "model": model,
+ "input_cost_per_token": 3 * _HUB_PRICES["input_cost_per_token"],
+ "output_cost_per_token": 3 * _HUB_PRICES["output_cost_per_token"],
+ },
+ "model_info": {"id": "hub-override"},
+ }
+ ]
+ )
+
+ group: Final = router.registry().group("hub")
+
+ assert group is not None
+ costs: Final = group.deployment_pricing[0].costs
+ assert {key: _billed_rate(model, "hub-override", *_usage_billed_at(key)) for key in costs} == pytest.approx(
+ dict(costs)
+ )
+ long_prompt: Final = _usage_billed_at("input_cost_per_token", prompt_tokens=250_000)
+ assert _billed_rate(model, "hub-override", *long_prompt) == pytest.approx(costs["input_cost_per_token"])
+ shadowed: Final = {
+ "input_cost_per_token_above_200k_tokens",
+ "output_cost_per_token_above_200k_tokens",
+ "input_cost_per_token_batches",
+ }
+ assert shadowed.isdisjoint(costs), dict(costs)
+
+
+@pytest.mark.parametrize("vertex_location", ["us-east5", "global"])
+def test_model_group_info_shows_a_vertex_deployment_the_token_prices_billing_charges_at_its_location(
+ local_model_cost_map: None, vertex_location: str
+) -> None:
+ model: Final = "vertex_ai/hub-vertex-model"
+ litellm.register_model(
+ {
+ model: {
+ "litellm_provider": "vertex_ai-language-models",
+ "mode": "chat",
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 4e-06,
+ "cache_read_input_token_cost": 1e-07,
+ "regional_endpoint_uplift_multiplier": 1.1,
+ }
+ },
+ persist_across_reloads=False,
+ add_to_provider_model_sets=False,
+ )
+ router: Final = litellm.Router(
+ model_list=[
+ {
+ "model_name": "hub",
+ "litellm_params": {"model": model, "vertex_project": "p", "vertex_location": vertex_location},
+ "model_info": {"id": "hub-vertex"},
+ }
+ ]
+ )
+
+ group: Final = router.registry().group("hub")
+
+ assert group is not None
+ costs: Final = group.deployment_pricing[0].costs
+
+ def billed(cost_key: str) -> float:
+ usage, _ = _usage_billed_at(cost_key)
+ cost: Final = litellm.completion_cost(
+ completion_response=litellm.ModelResponse(usage=usage),
+ model=model,
+ custom_llm_provider="vertex_ai",
+ router_model_id="hub-vertex",
+ vertex_location=vertex_location,
+ )
+ return cost / max(usage.prompt_tokens, usage.completion_tokens)
+
+ shown: Final = ("input_cost_per_token", "output_cost_per_token", "cache_read_input_token_cost")
+ assert {key: costs[key] for key in shown} == pytest.approx({key: billed(key) for key in shown})
+
+
+def test_model_group_info_lists_tiered_pricing_ranges(local_model_cost_map: None) -> None:
+ tiers: Final = (
+ {"range": [0, 256000], "input_cost_per_token": 5e-08, "output_cost_per_token": 4e-07},
+ {"range": [256000, 1000000], "input_cost_per_token": 2.5e-07, "output_cost_per_token": 2e-06},
+ )
+ model: Final = _cost_map_model("hub-tiered-model", {"tiered_pricing": list(tiers)})
+ router: Final = litellm.Router(
+ model_list=[
+ {"model_name": "tiered", "litellm_params": {"model": model}},
+ {
+ "model_name": "tiered",
+ "litellm_params": {"model": model, "input_cost_per_token": 1e-06, "output_cost_per_token": 4e-06},
+ },
+ ]
+ )
+
+ group: Final = router.registry().group("tiered")
+
+ assert group is not None
+ assert [
+ [(tier.range, dict(tier.costs)) for tier in pricing.tiered_pricing] for pricing in group.deployment_pricing
+ ] == [
+ [
+ ((0.0, 256000.0), {"input_cost_per_token": 5e-08, "output_cost_per_token": 4e-07}),
+ ((256000.0, 1000000.0), {"input_cost_per_token": 2.5e-07, "output_cost_per_token": 2e-06}),
+ ],
+ [],
+ ]
+
+
+def test_model_group_info_reads_prices_written_as_text_the_way_billing_does(local_model_cost_map: None) -> None:
+ tier: Final = {"range": [0, 256000], "input_cost_per_token": "5e-08", "output_cost_per_token": "4e-07"}
+ model: Final = _cost_map_model(
+ "hub-text-priced-model", {"tiered_pricing": [tier], "cache_read_input_token_cost": "N/A"}
+ )
+ router: Final = litellm.Router(
+ model_list=[
+ {"model_name": "tiered", "litellm_params": {"model": model}},
+ {
+ "model_name": "flat",
+ "litellm_params": {"model": model, "input_cost_per_token": "3e-7", "output_cost_per_token": "1.2e-6"},
+ "model_info": {"id": "hub-text-override"},
+ },
+ ]
+ )
+
+ tiered: Final = router.registry().group("tiered")
+ flat: Final = router.registry().group("flat")
+
+ assert tiered is not None
+ assert flat is not None
+ assert [dict(t.costs) for t in tiered.deployment_pricing[0].tiered_pricing] == [
+ {"input_cost_per_token": 5e-08, "output_cost_per_token": 4e-07}
+ ]
+ assert "cache_read_input_token_cost" not in tiered.deployment_pricing[0].costs
+ flat_costs: Final = flat.deployment_pricing[0].costs
+ shown: Final = ("input_cost_per_token", "output_cost_per_token")
+ assert {key: flat_costs[key] for key in shown} == pytest.approx(
+ {key: _billed_rate(model, "hub-text-override", *_usage_billed_at(key)) for key in shown}
+ )
+
+
+def test_model_group_info_lists_custom_prices_of_an_unmapped_deployment_and_skips_an_unpriced_one() -> None:
+ router: Final = litellm.Router(
+ model_list=[
+ {
+ "model_name": "self-hosted",
+ "litellm_params": {"model": "openai/hub-unpriced", "api_base": "http://localhost:8000/v1"},
+ "model_info": {"cost_tracking_exempt": True},
+ },
+ {
+ "model_name": "self-hosted",
+ "litellm_params": {
+ "model": "openai/hub-priced",
+ "api_base": "http://localhost:8001/v1",
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 2e-06,
+ },
+ "model_info": {"cache_read_input_token_cost": 1e-07},
+ },
+ ]
+ )
+
+ group: Final = router.registry().group("self-hosted")
+
+ assert group is not None
+ assert [(p.provider, dict(p.costs)) for p in group.deployment_pricing] == [
+ (
+ "openai",
+ {"input_cost_per_token": 1e-06, "output_cost_per_token": 2e-06, "cache_read_input_token_cost": 1e-07},
+ )
+ ]
+
+
def test_model_group_info_description_from_model_info():
"""
model_info.description set in the config should surface on ModelGroupInfo,
diff --git a/tests/unit/test_router/test_router_entry_points.py b/tests/unit/test_router/test_router_entry_points.py
index c3d5d962123..06287ba7e0b 100644
--- a/tests/unit/test_router/test_router_entry_points.py
+++ b/tests/unit/test_router/test_router_entry_points.py
@@ -514,6 +514,81 @@ def test_attribute_errors_about_other_objects_pass_through_unchanged(router):
assert _public_attribute_error(router, "_core_x", "x", other) is other
+_MINTING_DEPLOYMENT = "dep-minting-org"
+_OTHER_ORG_DEPLOYMENT = "dep-other-org"
+_BOTH_ORGS = (_MINTING_DEPLOYMENT, _OTHER_ORG_DEPLOYMENT)
+_GROUP_AFFINITY = ["encrypted_content_affinity"]
+
+
+@pytest.mark.parametrize(
+ "affinity, target_ids, reasoning_stays",
+ [
+ pytest.param({}, _BOTH_ORGS, False, id="no affinity, one target cannot decrypt"),
+ pytest.param({}, (_MINTING_DEPLOYMENT,), True, id="no affinity, the only target minted it"),
+ pytest.param(
+ {"optional_pre_call_checks": _GROUP_AFFINITY}, _BOTH_ORGS, True, id="global affinity, origin among targets"
+ ),
+ pytest.param(
+ {"optional_pre_call_checks": _GROUP_AFFINITY},
+ (_OTHER_ORG_DEPLOYMENT,),
+ False,
+ id="global affinity, origin not among targets",
+ ),
+ pytest.param(
+ {"optional_pre_call_checks": [], "model_group_affinity_config": {"pair": _GROUP_AFFINITY}},
+ _BOTH_ORGS,
+ True,
+ id="affinity on for this group",
+ ),
+ pytest.param(
+ {"optional_pre_call_checks": [], "model_group_affinity_config": {"other-group": _GROUP_AFFINITY}},
+ _BOTH_ORGS,
+ False,
+ id="affinity on for another group only",
+ ),
+ ],
+)
+def test_decryptable_reasoning_keeps_encrypted_reasoning_only_when_the_router_sends_it_to_a_deployment_that_decrypts_it(
+ affinity, target_ids, reasoning_stays
+):
+ from litellm.responses.utils import ResponsesAPIRequestUtils
+
+ r = Router(
+ model_list=[
+ {
+ "model_name": "pair",
+ "litellm_params": {"model": "openai/gpt-4o-mini", "api_base": api_base, "api_key": "sk-pair"},
+ "model_info": {"id": model_id},
+ }
+ for api_base, model_id in (
+ ("https://minting-org.test/v1", _MINTING_DEPLOYMENT),
+ ("https://other-org.test/v1", _OTHER_ORG_DEPLOYMENT),
+ )
+ ],
+ **affinity,
+ )
+ summary = [{"type": "summary_text", "text": "multiply 17 by 23"}]
+ user_turn = {"type": "message", "role": "user", "content": "What is 17*23?"}
+ reasoning = {
+ "type": "reasoning",
+ "id": "rs_minted",
+ "encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
+ "gAAAAA-blob", _MINTING_DEPLOYMENT
+ ),
+ "summary": summary,
+ }
+ request_input = [user_turn, reasoning]
+ targets = tuple(r.registry().deployment(model_id) for model_id in target_ids)
+ try:
+ kept = r.registry().decryptable_reasoning("pair", request_input, targets)
+ finally:
+ r.retire()
+
+ readable_only = {"type": "reasoning", "summary": summary}
+ assert kept == (user_turn, reasoning if reasoning_stays else readable_only)
+ assert request_input == [user_turn, reasoning], "the caller's input must not change"
+
+
# Router-family helpers whose Router parameter became a RoutingContext keep the old spellings.
diff --git a/tests/unit/test_router_model_cost_isolation.py b/tests/unit/test_router_model_cost_isolation.py
index c0c8a0655fe..cd71c3f9bee 100644
--- a/tests/unit/test_router_model_cost_isolation.py
+++ b/tests/unit/test_router_model_cost_isolation.py
@@ -542,7 +542,13 @@ def test_should_not_pollute_shared_key_with_custom_nonzero_pricing():
@pytest.mark.parametrize(
"flag",
- ["supports_regex_lookaround", "supports_assistant_prefill", "supports_sampling_params", "supports_continuation"],
+ [
+ "supports_regex_lookaround",
+ "supports_assistant_prefill",
+ "supports_sampling_params",
+ "supports_continuation",
+ "supports_bedrock_native_structured_output_by_default",
+ ],
)
def test_capability_flag_stays_on_the_deployment_that_set_it(flag: str) -> None:
"""A deployment's capability override must not land on the shared
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
index c41c4db2c02..b0c55d3f0f2 100644
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -1354,6 +1354,7 @@ _COST_MAP_ROW_SCHEMA: Final = {
"supports_image_size": {"type": "boolean"},
"supports_native_structured_output": {"type": "boolean"},
"supports_bedrock_native_structured_output": {"type": "boolean"},
+ "supports_bedrock_native_structured_output_by_default": {"type": "boolean"},
"use_openai_responses_path": {"type": "boolean"},
"off_peak_pricing": {
"type": "object",
@@ -3011,52 +3012,37 @@ if __name__ == "__main__":
class TestGetValidModelsWithCLI:
"""Test get_valid_models function as used in CLI token usage"""
- def test_get_valid_models_with_cli_pattern(self):
+ @respx.mock
+ def test_get_valid_models_with_cli_pattern(self, reset_mock_cache: None) -> None:
"""Test get_valid_models with litellm_proxy provider and CLI token pattern"""
- # Mock the HTTP request that get_valid_models makes to the proxy
- mock_response = MagicMock()
- mock_response.status_code = 200
- mock_response.json.return_value = {
- "data": [
- {"id": "gpt-3.5-turbo", "object": "model"},
- {"id": "gpt-4", "object": "model"},
- {"id": "litellm_proxy/gemini/gemini-2.5-flash", "object": "model"},
- {"id": "claude-3-sonnet", "object": "model"},
- ]
- }
+ route: Final = respx.get("https://gateway-cli.example.test/v1/models").respond(
+ 200,
+ json={
+ "data": [
+ {"id": "gpt-3.5-turbo", "object": "model"},
+ {"id": "gpt-4", "object": "model"},
+ {"id": "litellm_proxy/gemini/gemini-2.5-flash", "object": "model"},
+ {"id": "claude-3-sonnet", "object": "model"},
+ ]
+ },
+ )
- with patch.object(litellm.module_level_client, "get", return_value=mock_response) as mock_get:
- # Test the exact pattern used in cli_token_usage.py
- result = litellm.get_valid_models(
- check_provider_endpoint=True,
- custom_llm_provider="litellm_proxy",
- api_key="sk-test-cli-key-123",
- api_base="http://localhost:4000/",
- )
+ result: Final = litellm.get_valid_models(
+ check_provider_endpoint=True,
+ custom_llm_provider="litellm_proxy",
+ api_key="sk-test-cli-key-123",
+ api_base="https://gateway-cli.example.test/",
+ )
- # Verify the function returns a list of model names
- assert isinstance(result, list)
- assert len(result) == 4
- # All models get prefixed with "litellm_proxy/" by the get_models method
- assert "litellm_proxy/gpt-3.5-turbo" in result
- assert "litellm_proxy/gpt-4" in result
- # Note: This model already had the prefix, so it gets double-prefixed
- assert "litellm_proxy/litellm_proxy/gemini/gemini-2.5-flash" in result
- assert "litellm_proxy/claude-3-sonnet" in result
-
- # Verify the HTTP request was made with correct parameters
- mock_get.assert_called_once()
- _, call_kwargs = mock_get.call_args
-
- # Check that the request was made to the correct endpoint
- assert call_kwargs["url"].startswith("http://localhost:4000/")
- assert call_kwargs["url"].endswith("/v1/models")
-
- # Check that the API key was included in headers
- assert "headers" in call_kwargs
- headers = call_kwargs["headers"]
- assert headers.get("Authorization") == "Bearer sk-test-cli-key-123"
+ assert result == [
+ "litellm_proxy/gpt-3.5-turbo",
+ "litellm_proxy/gpt-4",
+ "litellm_proxy/litellm_proxy/gemini/gemini-2.5-flash",
+ "litellm_proxy/claude-3-sonnet",
+ ]
+ assert route.call_count == 1
+ assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test-cli-key-123"
@respx.mock
@@ -10307,36 +10293,33 @@ def test_logprobs_type():
assert logprobs.top_logprobs is None
-def test_get_valid_models_openai_proxy(monkeypatch):
- from litellm.utils import get_valid_models
- import litellm
-
- litellm.turn_on_debug()
-
+@respx.mock
+def test_get_valid_models_openai_proxy(monkeypatch: pytest.MonkeyPatch, reset_mock_cache: None) -> None:
monkeypatch.setenv("LITELLM_PROXY_API_KEY", "sk-9876")
- monkeypatch.setenv("LITELLM_PROXY_API_BASE", "https://litellm-api.up.railway.app/")
+ monkeypatch.setenv("LITELLM_PROXY_API_BASE", "https://gateway-env.example.test/")
monkeypatch.delenv("FIREWORKS_AI_ACCOUNT_ID", None)
monkeypatch.delenv("FIREWORKS_AI_API_KEY", None)
- mock_response_data = {
- "object": "list",
- "data": [
- {
- "id": "gpt-5.5",
- "object": "model",
- "created": 1686935002,
- "owned_by": "organization-owner",
- },
- ],
- }
+ route: Final = respx.get("https://gateway-env.example.test/v1/models").respond(
+ 200,
+ json={
+ "object": "list",
+ "data": [
+ {
+ "id": "gpt-5.5",
+ "object": "model",
+ "created": 1686935002,
+ "owned_by": "organization-owner",
+ },
+ ],
+ },
+ )
- mock_response = MagicMock()
- mock_response.status_code = 200
- mock_response.json.return_value = mock_response_data
+ valid_models: Final = litellm.get_valid_models(check_provider_endpoint=True)
- with patch.object(litellm.module_level_client, "get", return_value=mock_response) as mock_post:
- valid_models = get_valid_models(check_provider_endpoint=True)
- assert "litellm_proxy/gpt-5.5" in valid_models
+ assert "litellm_proxy/gpt-5.5" in valid_models
+ assert route.call_count == 1
+ assert route.calls.last.request.headers["Authorization"] == "Bearer sk-9876"
def test_pick_cheapest_chat_model_from_llm_provider():
diff --git a/tests/unit/test_utils_get_model_info.py b/tests/unit/test_utils_get_model_info.py
index d08742be4af..7548001ce4c 100644
--- a/tests/unit/test_utils_get_model_info.py
+++ b/tests/unit/test_utils_get_model_info.py
@@ -13,7 +13,7 @@ from litellm import get_model_info
from litellm.llms.bedrock.common_utils import BedrockModelInfo
from litellm.llms.custom_httpx.http_handler import HTTPHandler
from litellm.types.utils import ModelInfoBase
-from litellm.utils import _invalidate_model_cost_lowercase_map
+from litellm.utils import _invalidate_model_cost_lowercase_map, supports_native_structured_output
import os
from unittest.mock import MagicMock, patch
@@ -388,6 +388,29 @@ def test_get_model_info_bedrock_cross_region_capability_parity():
assert checked > 0, "no cross-region bedrock profiles found - the filter is inert"
+def test_bedrock_native_output_default_reaches_only_versions_that_read_its_key(local_model_cost_map):
+ flagged: Final = tuple(
+ key for key, row in litellm.model_cost.items() if row.get("supports_bedrock_native_structured_output_by_default")
+ )
+
+ assert flagged
+ assert [key for key in flagged if litellm.model_cost[key].get("supports_native_structured_output")] == []
+ assert [key for key in flagged if not get_model_info(key)["supports_native_structured_output"]] == []
+
+
+@pytest.mark.parametrize(
+ ("registered", "native"),
+ [({"supports_bedrock_native_structured_output": False}, False), ({"max_output_tokens": 1024}, True)],
+ ids=["bedrock-opt-out", "unrelated-override"],
+)
+def test_a_registered_bedrock_opt_out_beats_the_native_output_default(local_model_cost_map, registered, native):
+ model: Final = "us.anthropic.claude-sonnet-5"
+ litellm.register_model({model: registered}, persist_across_reloads=False)
+
+ assert get_model_info(f"bedrock/{model}")["supports_native_structured_output"] is native
+ assert supports_native_structured_output(f"bedrock/{model}") is native
+
+
def _is_positive_cost(value: object) -> bool:
return isinstance(value, (int, float)) and value > 0
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/AllModelsTable.test.tsx b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/AllModelsTable.test.tsx
index b00affc50f5..3de187cbf55 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/AllModelsTable.test.tsx
+++ b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/AllModelsTable.test.tsx
@@ -132,6 +132,22 @@ describe("AllModelsTable", () => {
expect(onModelIdClick).toHaveBeenCalledWith("model-1");
});
+ it("should display the model metadata logo for a gateway-routed model", () => {
+ const logoUrl = "https://cdn.example.com/gateway.svg";
+ render(
+ ,
+ );
+
+ expect(screen.getByRole("img", { name: "openai logo" })).toHaveAttribute("src", logoUrl);
+ });
+
it("opens the team detail from the team ID cell", async () => {
const user = userEvent.setup();
const onTeamIdClick = vi.fn();
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelsTableColumns.tsx b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelsTableColumns.tsx
index 0e286418355..65fd8be97c2 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelsTableColumns.tsx
+++ b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelsTableColumns.tsx
@@ -70,7 +70,7 @@ function ModelInformationCell({ model, displayName }: { model: ModelData; displa
}
>
{model.provider ? (
-
+
) : (
-
@@ -88,7 +88,9 @@ function ModelInformationCell({ model, displayName }: { model: ModelData; displa
- {model.provider ?
: null}
+ {model.provider ? (
+
+ ) : null}
{model.provider || t("modelsAndEndpoints.modelsTableColumns.unknownProvider")}
diff --git a/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.test.tsx b/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.test.tsx
index 123c7431732..1b30b060ab0 100644
--- a/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.test.tsx
+++ b/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.test.tsx
@@ -2,6 +2,7 @@ import * as networking from "@/components/networking";
import userEvent from "@testing-library/user-event";
import { act } from "@testing-library/react";
import type { MCPServerData } from "./MCPHubTableColumns";
+import type { DeploymentPricing } from "@/components/molecules/models/deploymentPricing";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { renderWithProviders, screen, testQueryClient, waitFor, within } from "../../../tests/test-utils";
import ModelHubTable from "./ModelHubTable";
@@ -374,7 +375,7 @@ describe("ModelHubTable", () => {
describe("model details usage example", () => {
const openDetails = async (
- model: { model_group: string; providers: string[]; mode: string },
+ model: { model_group: string; providers: string[]; mode: string; deployment_pricing?: DeploymentPricing[] },
canOpenPlayground = true,
) => {
vi.mocked(networking.modelHubCall).mockResolvedValue({ data: [model] });
@@ -425,6 +426,34 @@ describe("ModelHubTable", () => {
expect(screen.queryByRole("link", { name: "Try it in the Playground" })).not.toBeInTheDocument();
});
+ it("shows each deployment's own prices, by service tier, in the model details", async () => {
+ const gpt5OnOpenAIAndAzure = {
+ model_group: "gpt-5",
+ providers: ["openai", "azure"],
+ mode: "chat",
+ deployment_pricing: [
+ {
+ provider: "openai",
+ costs: { input_cost_per_token: 2.5e-6, input_cost_per_token_priority: 5e-6 },
+ tiered_pricing: [],
+ },
+ { provider: "azure", costs: { input_cost_per_token: 2.75e-6 }, tiered_pricing: [] },
+ ],
+ };
+ const dialog = await openDetails(gpt5OnOpenAIAndAzure);
+
+ const openai = within(dialog).getByRole("region", { name: "OpenAI pricing" });
+ expect(within(openai).getByRole("row", { name: /Input \/ 1M tokens/ })).toHaveTextContent("$2.50$5.00");
+ expect(within(dialog).getByRole("region", { name: "Azure pricing" })).toHaveTextContent("$2.75");
+ });
+
+ it("leaves the pricing breakdown out when the proxy sends no deployment prices", async () => {
+ const dialog = await openDetails({ model_group: "gpt-5", providers: ["openai"], mode: "chat" });
+
+ expect(within(dialog).getByText("Input Cost per 1M Tokens:")).toBeInTheDocument();
+ expect(within(dialog).queryByText("Pricing by Deployment:")).not.toBeInTheDocument();
+ });
+
it("keeps the chat completions example for a chat model", async () => {
const dialog = await openDetails({ model_group: "claude-opus-4-8", providers: ["anthropic"], mode: "chat" });
diff --git a/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.tsx b/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.tsx
index 4d44d0bb809..60259b1f7f0 100644
--- a/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.tsx
+++ b/ui/litellm-dashboard/src/components/AIHub/ModelHubTable.tsx
@@ -57,6 +57,7 @@ import {
isDecisionMode,
} from "@/lib/decisionModels";
import DecisionModelGarden from "@/components/AIHub/DecisionModelGarden";
+import { DeploymentPricingBreakdown } from "@/components/molecules/models/DeploymentPricingBreakdown";
import { ModelCustomMetadata } from "./ModelCustomMetadata";
interface ModelHubTableProps {
@@ -819,6 +820,11 @@ const ModelHubTable: React.FC
= ({
+ {t("aiHub.modelHubTable.modelDialog.pricingByDeployment")}
}
+ />
{/* Capabilities */}
diff --git a/ui/litellm-dashboard/src/components/AIHub/ModelHubTableColumns.tsx b/ui/litellm-dashboard/src/components/AIHub/ModelHubTableColumns.tsx
index 2e4af18455f..7f2f3a49d44 100644
--- a/ui/litellm-dashboard/src/components/AIHub/ModelHubTableColumns.tsx
+++ b/ui/litellm-dashboard/src/components/AIHub/ModelHubTableColumns.tsx
@@ -22,6 +22,7 @@ import { copyToClipboard } from "@/utils/dataUtils";
import { ModelCustomTags, type ModelCustomMetadataValue } from "./ModelCustomMetadata";
import { formatModelModeLabel, ProviderLogo } from "./ModelHubProviders";
import { getProviderLogoAndName } from "@/components/provider_info_helpers";
+import type { DeploymentPricing } from "@/components/molecules/models/deploymentPricing";
export interface ModelHubData {
model_group: string;
@@ -41,6 +42,7 @@ export interface ModelHubData {
description?: string;
custom_tags?: string[] | null;
custom_metadata?: Record | null;
+ deployment_pricing?: DeploymentPricing[] | null;
[key: string]: any;
}
diff --git a/ui/litellm-dashboard/src/components/Navbar/ViewSwitcher.tsx b/ui/litellm-dashboard/src/components/Navbar/ViewSwitcher.tsx
index 93b5894d9c6..e9eb5586c42 100644
--- a/ui/litellm-dashboard/src/components/Navbar/ViewSwitcher.tsx
+++ b/ui/litellm-dashboard/src/components/Navbar/ViewSwitcher.tsx
@@ -9,7 +9,7 @@ import {
DropdownMenuItem,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
-import { Check, ChevronDown, ChevronsUpDown, MessageSquare, PanelsTopLeft } from "lucide-react";
+import { Check, ChevronDown, ChevronsUpDown } from "lucide-react";
import { useSidebar } from "@/components/shared/Sidebar";
import { usePluginMode } from "@/contexts/PluginModeContext";
import { useUISettings } from "@/app/(dashboard)/hooks/uiSettings/useUISettings";
@@ -65,7 +65,6 @@ export default function ViewSwitcher({ className, contentClassName }: ViewSwitch
const isChatRoute = isWorkspaceRoute(normalizedPathname, CHAT);
const isMoyaiRoute = isWorkspaceRoute(normalizedPathname, MOYAI);
const isStandaloneRoute = isChatRoute || isMoyaiRoute;
- const WorkspaceIcon = isChatRoute ? MessageSquare : PanelsTopLeft;
const gatewayLabel = term(t("shell.viewSwitcher.aiGateway"));
const activeLabel = activeWorkspaceLabel(
@@ -150,16 +149,13 @@ export default function ViewSwitcher({ className, contentClassName }: ViewSwitch
"max-w-[220px] gap-1.5 px-3 dark:bg-transparent",
className,
collapsed &&
- "mx-auto flex size-9 justify-center gap-0.5 border-sidebar-border p-0 text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground",
+ "mx-auto flex size-9 justify-center gap-0 border-sidebar-border p-0 text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground",
)}
/>
}
>
{collapsed ? (
- <>
-
-
- >
+
) : (
<>
diff --git a/ui/litellm-dashboard/src/components/PublicModelHubTableColumns.tsx b/ui/litellm-dashboard/src/components/PublicModelHubTableColumns.tsx
index 08d6b973b87..eb89af22f17 100644
--- a/ui/litellm-dashboard/src/components/PublicModelHubTableColumns.tsx
+++ b/ui/litellm-dashboard/src/components/PublicModelHubTableColumns.tsx
@@ -11,6 +11,7 @@ import i18n from "@/i18n/i18n";
import { displayMode } from "@/lib/decisionModels";
import type { ModelCustomMetadataValue } from "@/components/AIHub/ModelCustomMetadata";
import { formatModelModeLabel, ProviderLogo } from "@/components/AIHub/ModelHubProviders";
+import type { DeploymentPricing } from "@/components/molecules/models/deploymentPricing";
export interface ModelGroupInfo {
model_group: string;
@@ -32,6 +33,7 @@ export interface ModelGroupInfo {
description?: string;
custom_tags?: string[] | null;
custom_metadata?: Record | null;
+ deployment_pricing?: DeploymentPricing[] | null;
[key: string]: any;
}
diff --git a/ui/litellm-dashboard/src/components/add_model/AddModelForm.integration.test.tsx b/ui/litellm-dashboard/src/components/add_model/AddModelForm.integration.test.tsx
index a95d9ffbfb8..7fab7219f7a 100644
--- a/ui/litellm-dashboard/src/components/add_model/AddModelForm.integration.test.tsx
+++ b/ui/litellm-dashboard/src/components/add_model/AddModelForm.integration.test.tsx
@@ -1,5 +1,6 @@
import {
chooseSelectOption,
+ act,
fireEvent,
renderHook,
screen,
@@ -48,6 +49,15 @@ vi.mock("../networking", async () => {
credentialCreateCall: vi.fn().mockResolvedValue({ success: true }),
modelCreateCall: vi.fn().mockResolvedValue({ success: true }),
getProviderCreateMetadata: vi.fn().mockResolvedValue([
+ {
+ provider: "LITELLM_PROXY",
+ provider_display_name: "LiteLLM Gateway",
+ litellm_provider: "litellm_proxy",
+ credential_fields: [
+ { key: "api_base", label: "API Base", field_type: "text" },
+ { key: "api_key", label: "API Key", field_type: "password" },
+ ],
+ },
{
provider: "OpenAI",
provider_display_name: "OpenAI",
@@ -103,6 +113,15 @@ vi.mock("../networking", async () => {
vi.mock("@/app/(dashboard)/hooks/providers/useProviderFields", () => ({
useProviderFields: vi.fn().mockReturnValue({
data: [
+ {
+ provider: "LITELLM_PROXY",
+ provider_display_name: "LiteLLM Gateway",
+ litellm_provider: "litellm_proxy",
+ credential_fields: [
+ { key: "api_base", label: "API Base", field_type: "text" },
+ { key: "api_key", label: "API Key", field_type: "password" },
+ ],
+ },
{
provider: "OpenAI",
provider_display_name: "OpenAI",
@@ -300,6 +319,44 @@ describe("AddModelForm", () => {
expect(await screen.findByRole("heading", { name: "Add Model" })).toBeInTheDocument();
});
+ it("should add a LiteLLM Gateway wildcard with its upstream URL and key when no catalog models are known", async () => {
+ const mockUseAuthorized = vi.mocked(await import("@/app/(dashboard)/hooks/useAuthorized"));
+ mockUseAuthorized.default.mockReturnValue(mockAuthorizedUser("proxy_admin", "user-1", true));
+ vi.mocked(modelCreateCall).mockClear();
+ const user = userEvent.setup({ pointerEventsCheck: PointerEventsCheckLevel.Never });
+ const props = { ...createTestProps(), selectedProvider: "LITELLM_PROXY", providerModels: [] };
+ props.handleOk.mockImplementation(async () => {
+ await handleAddModelSubmit(props.mountedValues(), "test-access-token", { resetFields: vi.fn() });
+ return true;
+ });
+ renderWithProviders();
+
+ await pickProvider(user, /LiteLLM Gateway/);
+ await pickModel(user, "All LiteLLM Gateway Models (Wildcard)");
+ act(() => {
+ fireEvent.change(screen.getByLabelText("API Base"), { target: { value: "https://gateway.example.test/v1" } });
+ fireEvent.change(screen.getByLabelText("API Key"), { target: { value: "upstream-test-key" } });
+ });
+ await user.click(screen.getByTestId("add-model-btn"));
+
+ const expectedParams = {
+ model: "litellm_proxy/*",
+ custom_llm_provider: "litellm_proxy",
+ api_base: "https://gateway.example.test/v1",
+ api_key: "upstream-test-key",
+ };
+ await waitFor(() =>
+ expect(modelCreateCall).toHaveBeenCalledWith(
+ "test-access-token",
+ expect.objectContaining({
+ model_name: "litellm_proxy/*",
+ litellm_params: expect.objectContaining(expectedParams),
+ }),
+ ),
+ );
+ expect(modelCreateCall).toHaveBeenCalledOnce();
+ });
+
it("should show proxy admin only (not team admin) - should not see Select Team dropdown unless switch is toggled", async () => {
const mockUseAuthorized = vi.mocked(await import("@/app/(dashboard)/hooks/useAuthorized"));
mockUseAuthorized.default.mockReturnValue(mockAuthorizedUser("proxy_admin", "user-1", true));
diff --git a/ui/litellm-dashboard/src/components/add_model/litellm_model_name.tsx b/ui/litellm-dashboard/src/components/add_model/litellm_model_name.tsx
index edb02da5d0d..97c6769db54 100644
--- a/ui/litellm-dashboard/src/components/add_model/litellm_model_name.tsx
+++ b/ui/litellm-dashboard/src/components/add_model/litellm_model_name.tsx
@@ -23,6 +23,7 @@ const LiteLLMModelNameField: React.FC = ({
const form = useFormContext();
const modelValue = useWatch({ control: form.control, name: "model" });
const selectedModels = Array.isArray(modelValue) ? modelValue : [modelValue];
+ const isLiteLLMGateway = selectedProvider === "LITELLM_PROXY" || selectedProvider === Providers.LITELLM_PROXY;
const handleModelChange = (value: string | string[]) => {
// Ensure value is always treated as an array
@@ -142,7 +143,8 @@ const LiteLLMModelNameField: React.FC = ({
}
}}
/>
- ) : providerModels.length > 0 && !(typeof control.value === "string" && control.value !== "") ? (
+ ) : (providerModels.length > 0 || isLiteLLMGateway) &&
+ !(typeof control.value === "string" && control.value !== "") ? (
= ({
},
{
label: t("addModel.litellmModelName.wildcardOption", {
- provider: selectedProvider ?? t("addModel.litellmModelName.providerFallback"),
+ provider: isLiteLLMGateway
+ ? Providers.LITELLM_PROXY
+ : selectedProvider ?? t("addModel.litellmModelName.providerFallback"),
}),
value: "all-wildcard",
},
diff --git a/ui/litellm-dashboard/src/components/logs/detail/PrettyMessagesView.test.tsx b/ui/litellm-dashboard/src/components/logs/detail/PrettyMessagesView.test.tsx
index 807aaeeae00..6dbca340b4e 100644
--- a/ui/litellm-dashboard/src/components/logs/detail/PrettyMessagesView.test.tsx
+++ b/ui/litellm-dashboard/src/components/logs/detail/PrettyMessagesView.test.tsx
@@ -17,6 +17,45 @@ describe("PrettyMessagesView", () => {
expect(screen.getByText("Hi there!")).toBeInTheDocument();
});
+ it("shows Claude's server tool calls next to the client tool calls on chat completions", () => {
+ const serverToolCalls = [
+ { id: "srvtoolu_01", type: "function", function: { name: "web_search", arguments: '{"query": "persimmon"}' } },
+ ];
+ const weatherCall = {
+ id: "toolu_01",
+ type: "function",
+ function: { name: "get_weather", arguments: '{"city": "Paris"}' },
+ };
+ const request = {
+ messages: [
+ { role: "user", content: "Search, then get the weather" },
+ {
+ role: "assistant",
+ content: "Searching.",
+ tool_calls: [weatherCall],
+ provider_specific_fields: { server_tool_calls: serverToolCalls },
+ },
+ { role: "tool", tool_call_id: "toolu_01", content: "Sunny" },
+ ],
+ };
+ const response = {
+ choices: [
+ {
+ message: {
+ role: "assistant",
+ content: null,
+ tool_calls: [weatherCall],
+ provider_specific_fields: { server_tool_calls: serverToolCalls },
+ },
+ },
+ ],
+ };
+
+ render();
+ expect(screen.getAllByText("web_search").length).toBeGreaterThanOrEqual(2);
+ expect(screen.getAllByText("get_weather").length).toBeGreaterThanOrEqual(2);
+ });
+
it("renders input when request is a bare messages array (cold storage payload)", () => {
const request = [{ role: "user", content: "Write me a poem" }];
const response = {
diff --git a/ui/litellm-dashboard/src/components/logs/detail/prettyMessagesUtils.ts b/ui/litellm-dashboard/src/components/logs/detail/prettyMessagesUtils.ts
index 43bbd691490..5482b034419 100644
--- a/ui/litellm-dashboard/src/components/logs/detail/prettyMessagesUtils.ts
+++ b/ui/litellm-dashboard/src/components/logs/detail/prettyMessagesUtils.ts
@@ -81,7 +81,7 @@ const parseResponseMessage = (payload: ResponsePayload): ParsedMessage | null =>
return {
role: toRole(message.role, "assistant"),
content: parseMessageContent(message.content),
- toolCalls: parseChatToolCalls(message.tool_calls),
+ toolCalls: parseChatMessageToolCalls(message),
};
}
case "responses": {
@@ -104,7 +104,7 @@ const parseChatMessage = (message: unknown): ParsedMessage => {
return {
role: toRole(message.role, "user"),
content: parseMessageContent(message.content),
- toolCalls: parseChatToolCalls(message.tool_calls),
+ toolCalls: parseChatMessageToolCalls(message),
toolCallId: typeof message.tool_call_id === "string" ? message.tool_call_id : undefined,
};
};
@@ -169,6 +169,13 @@ const parseContentPart = (part: unknown): string => {
/**
* Parse tool calls from response message
*/
+const parseChatMessageToolCalls = (message: UnknownRecord): ToolCall[] | undefined => {
+ const fields = message.provider_specific_fields;
+ const serverToolCalls = isRecord(fields) ? parseChatToolCalls(fields.server_tool_calls) : undefined;
+ const toolCalls = parseChatToolCalls(message.tool_calls);
+ return serverToolCalls?.length ? [...serverToolCalls, ...(toolCalls ?? [])] : toolCalls;
+};
+
const parseChatToolCalls = (toolCalls: unknown): ToolCall[] | undefined => {
if (!Array.isArray(toolCalls)) return undefined;
return toolCalls.map((toolCall) => {
diff --git a/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.test.ts b/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.test.ts
index 78372fe56d9..a6693794df6 100644
--- a/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.test.ts
+++ b/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.test.ts
@@ -131,6 +131,52 @@ describe("ToolsSection utils", () => {
expect(result[1].callData).toBeUndefined();
});
+ it("should mark Claude server tools as called from provider_specific_fields.server_tool_calls", () => {
+ const log: Partial = {
+ request_id: "test-server-tools",
+ proxy_server_request: {
+ tools: [
+ { type: "web_search_20250305", name: "web_search", max_uses: 3 },
+ { type: "function", function: { name: "get_weather", description: "Get weather" } },
+ { type: "function", function: { name: "send_email", description: "Send email" } },
+ ],
+ },
+ response: {
+ choices: [
+ {
+ message: {
+ tool_calls: [
+ {
+ id: "toolu_01Weather",
+ type: "function",
+ function: { name: "get_weather", arguments: '{"location": "Paris"}' },
+ },
+ ],
+ provider_specific_fields: {
+ server_tool_calls: [
+ {
+ id: "srvtoolu_01Search",
+ type: "function",
+ function: { name: "web_search", arguments: '{"query": "paris weather"}' },
+ },
+ ],
+ },
+ },
+ },
+ ],
+ },
+ };
+
+ const result = parseToolsFromLog(log as LogEntry);
+
+ expect(result.map((tool) => [tool.name, tool.called, tool.callData?.id])).toEqual([
+ ["web_search", true, "srvtoolu_01Search"],
+ ["get_weather", true, "toolu_01Weather"],
+ ["send_email", false, undefined],
+ ]);
+ expect(result[0].callData?.arguments).toEqual({ query: "paris weather" });
+ });
+
it("should handle string format request and response", () => {
const log: Partial = {
request_id: "test-5",
diff --git a/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.ts b/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.ts
index 4fef4a93497..c489f3b2f49 100644
--- a/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.ts
+++ b/ui/litellm-dashboard/src/components/logs/detail/sections/ToolsSection/utils.ts
@@ -43,6 +43,24 @@ function extractToolsFromRequest(log: LogEntry): ToolDefinition[] {
return [];
}
+interface ChatMessageToolCallFields {
+ tool_calls?: unknown;
+ provider_specific_fields?: { server_tool_calls?: unknown } | null;
+}
+
+function toolCallList(value: unknown): ToolCall[] {
+ return Array.isArray(value) ? value : [];
+}
+
+function extractChatMessageToolCalls(message: ChatMessageToolCallFields | undefined): ToolCall[] | null {
+ const serverToolCalls = message?.provider_specific_fields?.server_tool_calls;
+ const clientToolCalls = message?.tool_calls;
+ if (!Array.isArray(serverToolCalls) && !Array.isArray(clientToolCalls)) {
+ return null;
+ }
+ return [...toolCallList(serverToolCalls), ...toolCallList(clientToolCalls)];
+}
+
/**
* Extract tool calls from response data
*/
@@ -54,10 +72,9 @@ function extractToolCallsFromResponse(log: LogEntry): ToolCall[] {
// OpenAI format: response.choices[0].message.tool_calls
const choices = responseData.choices;
if (Array.isArray(choices) && choices.length > 0) {
- const firstChoice = choices[0];
- const message = firstChoice.message;
- if (message && Array.isArray(message.tool_calls)) {
- return message.tool_calls;
+ const chatToolCalls = extractChatMessageToolCalls(choices[0].message);
+ if (chatToolCalls) {
+ return chatToolCalls;
}
}
diff --git a/ui/litellm-dashboard/src/components/model_dashboard/types.ts b/ui/litellm-dashboard/src/components/model_dashboard/types.ts
index 47c7eab8ba6..2a8ec028aff 100644
--- a/ui/litellm-dashboard/src/components/model_dashboard/types.ts
+++ b/ui/litellm-dashboard/src/components/model_dashboard/types.ts
@@ -5,6 +5,7 @@ export interface PerSecondCostTier {
export interface ModelInfo {
id: string;
+ logo_url?: string | null;
created_at: string;
updated_at: string;
created_by: string;
diff --git a/ui/litellm-dashboard/src/components/model_info_view.integration.test.tsx b/ui/litellm-dashboard/src/components/model_info_view.integration.test.tsx
index a42db06d251..404ef6e7f20 100644
--- a/ui/litellm-dashboard/src/components/model_info_view.integration.test.tsx
+++ b/ui/litellm-dashboard/src/components/model_info_view.integration.test.tsx
@@ -205,6 +205,22 @@ describe("ModelInfoView", () => {
});
});
+ it("should display the model metadata logo in the provider details", async () => {
+ const logoUrl = "https://cdn.example.com/gateway.svg";
+ mockUseModelsInfo.mockReturnValue({
+ data: {
+ data: [{ ...defaultModelData, model_info: { ...defaultModelData.model_info, logo_url: logoUrl } }],
+ },
+ isLoading: false,
+ error: null,
+ });
+
+ render(, { wrapper });
+
+ expect(await screen.findByRole("img", { name: "openai logo" })).toHaveAttribute("src", logoUrl);
+ expect(screen.getByRole("heading", { name: "openai" })).toBeInTheDocument();
+ });
+
it("shows Kubernetes pods for discovery-enabled models", async () => {
mockUseModelsInfo.mockReturnValue({
data: {
diff --git a/ui/litellm-dashboard/src/components/model_info_view.tsx b/ui/litellm-dashboard/src/components/model_info_view.tsx
index efa7caf550d..5a52006632c 100644
--- a/ui/litellm-dashboard/src/components/model_info_view.tsx
+++ b/ui/litellm-dashboard/src/components/model_info_view.tsx
@@ -45,7 +45,7 @@ import {
tagListCall,
testConnectionRequest,
} from "./networking";
-import { Logo } from "@/components/molecules/logo/Logo";
+import { ProviderLogo } from "@/components/molecules/models/ProviderLogo";
import { ModelPricingSummary } from "@/components/molecules/models/ModelPricingSummary";
import UpdateModelCredentialsModal from "./update_model_credentials_modal";
import ModelInfoEditForm, { type ModelEditSubmitValues, type TouchedPricingField } from "./ModelInfoEditForm";
@@ -626,7 +626,13 @@ export default function ModelInfoView({
{t("modelInfoView.overview.provider")}
- {modelData.provider &&
}
+ {modelData.provider && (
+
+ )}
{modelData.provider || t("modelInfoView.notSet")}
diff --git a/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.test.tsx b/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.test.tsx
new file mode 100644
index 00000000000..08ee05aeb04
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.test.tsx
@@ -0,0 +1,97 @@
+import { render, screen, within } from "@testing-library/react";
+import { describe, expect, it } from "vitest";
+
+import { DeploymentPricingBreakdown } from "./DeploymentPricingBreakdown";
+import type { DeploymentPricing } from "./deploymentPricing";
+
+const OPENAI: DeploymentPricing = {
+ provider: "openai",
+ costs: {
+ input_cost_per_token: 2.5e-6,
+ output_cost_per_token: 1e-5,
+ input_cost_per_token_priority: 5e-6,
+ cache_read_input_token_cost_above_272k_tokens_priority: 1e-6,
+ output_cost_per_second_720p: 0.05,
+ },
+ tiered_pricing: [{ range: [0, 256_000], costs: { input_cost_per_token: 5e-8, output_cost_per_token: 4e-7 } }],
+};
+
+const AZURE: DeploymentPricing = { provider: "azure", costs: { input_cost_per_token: 2.75e-6 }, tiered_pricing: [] };
+
+const pricesOf = (region: HTMLElement, group: string, charge: string) => {
+ const rows = within(within(region).getByRole("rowgroup", { name: group })).getAllByRole("row");
+ const row = rows.find((candidate) => within(candidate).queryByRole("rowheader", { name: charge }));
+ expect(row).toBeDefined();
+ return within(row as HTMLElement)
+ .getAllByRole("cell")
+ .map((cell) => cell.textContent);
+};
+
+describe("DeploymentPricingBreakdown", () => {
+ it("renders one block per deployment, headed by its provider, with that deployment's own prices", () => {
+ render(Pricing} />);
+
+ const openai = screen.getByRole("region", { name: "OpenAI pricing" });
+ const azure = screen.getByRole("region", { name: "Azure pricing" });
+ expect(within(openai).getByRole("heading", { name: /OpenAI/ })).toBeInTheDocument();
+ expect(pricesOf(openai, "Base prices", "Input / 1M tokens")).toEqual(["$2.50", "$5.00"]);
+ expect(pricesOf(azure, "Base prices", "Input / 1M tokens")).toEqual(["$2.75"]);
+ expect(within(azure).queryByText(/Blank cells/)).not.toBeInTheDocument();
+ expect(screen.getByText(/This model has 2 deployments/)).toBeInTheDocument();
+ expect(screen.getAllByText(/Adjustments made per request/)).toHaveLength(1);
+ });
+
+ it("lays prices out by service tier column and context threshold group", () => {
+ render();
+
+ const openai = screen.getByRole("region", { name: "OpenAI pricing" });
+ const header = within(openai).getAllByRole("rowgroup")[0];
+ expect(
+ within(header)
+ .getAllByRole("columnheader")
+ .map((cell) => cell.textContent),
+ ).toEqual(["Charge", "Standard", "Priority"]);
+ expect(pricesOf(openai, "Base prices", "Output / 1M tokens")).toEqual(["$10.00", ""]);
+ expect(pricesOf(openai, "Above 272K tokens", "Cache read / 1M tokens")).toEqual(["", "$1.00"]);
+ expect(within(openai).getByText("Blank cells aren't priced separately for this deployment.")).toBeInTheDocument();
+ expect(screen.queryByText(/This model has/)).not.toBeInTheDocument();
+ });
+
+ it("shows tiered pricing ranges and keys it could not read", () => {
+ render();
+
+ expect(screen.getByText("0–256K tokens: $0.05 input / $0.40 output")).toBeInTheDocument();
+ expect(screen.getByText("Output / second 720p")).toBeInTheDocument();
+ expect(screen.queryByText("output_cost_per_second_720p")).not.toBeInTheDocument();
+ expect(screen.getByText("$0.05")).toBeInTheDocument();
+ });
+
+ it("shows only the request-size ranges for a deployment priced purely by tiers", () => {
+ render(
+ ,
+ );
+
+ expect(screen.getByText("0–256K tokens: $0.05 input")).toBeInTheDocument();
+ expect(screen.queryByRole("table")).not.toBeInTheDocument();
+ });
+
+ it.each([
+ ["missing", undefined],
+ ["null", null],
+ ["empty", []],
+ ])("renders nothing, heading included, when deployment pricing is %s", (_, deployments) => {
+ const { container } = render(
+ Pricing by deployment} />,
+ );
+
+ expect(container).toBeEmptyDOMElement();
+ });
+});
diff --git a/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.tsx b/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.tsx
new file mode 100644
index 00000000000..3d967e1d43e
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/molecules/models/DeploymentPricingBreakdown.tsx
@@ -0,0 +1,139 @@
+import type * as React from "react";
+import { useTranslation } from "react-i18next";
+
+import { ProviderLogo } from "@/components/molecules/models/ProviderLogo";
+import { getProviderLogoAndName } from "@/components/provider_info_helpers";
+import { cn } from "@/lib/cva.config";
+import {
+ NUMERIC_CELL_CLASS,
+ Table,
+ TableBody,
+ TableCell,
+ TableHead,
+ TableHeader,
+ TableRow,
+} from "@/components/ui/table";
+
+import {
+ buildPricingTable,
+ type DeploymentPricing,
+ describePricingTier,
+ formatTokenCount,
+ type PriceGroup,
+ SERVICE_TIER_LABELS,
+ type ServiceTier,
+} from "./deploymentPricing";
+
+function PriceGroupRows({ group, tiers }: { group: PriceGroup; tiers: readonly ServiceTier[] }) {
+ const { t } = useTranslation();
+ const name =
+ group.threshold === null
+ ? t("deploymentPricing.basePrices")
+ : t("deploymentPricing.aboveThreshold", { threshold: formatTokenCount(group.threshold) });
+ return (
+
+ {group.threshold !== null && (
+
+
+ {name}
+
+
+ )}
+ {group.rows.map((row) => (
+
+
+ {row.label}
+
+ {tiers.map((tier) => (
+
+ {row.prices[tier]}
+
+ ))}
+
+ ))}
+
+ );
+}
+
+function DeploymentPrices({ deployment }: { deployment: DeploymentPricing }) {
+ const { t } = useTranslation();
+ const table = buildPricingTable(deployment.costs);
+ const tiers = deployment.tiered_pricing;
+ const providerName = getProviderLogoAndName(deployment.provider).displayName;
+ return (
+
+
+
+ {providerName}
+
+ {table.groups.length > 0 && (
+
+
+
+ {t("deploymentPricing.chargeColumn")}
+ {table.tiers.map((tier) => (
+
+ {t(SERVICE_TIER_LABELS[tier])}
+
+ ))}
+
+
+ {table.groups.map((group) => (
+
+ ))}
+
+ )}
+ {table.hasBlankCells && {t("deploymentPricing.blankCells")}
}
+ {tiers.length > 0 && (
+
+
{t("deploymentPricing.tieredPricing")}
+
+ {tiers.map((tier) => (
+ - {describePricingTier(tier)}
+ ))}
+
+
+ )}
+ {table.unknown.length > 0 && (
+
+ {table.unknown.map(({ key, label, price }) => (
+
+
- {label}
+ - {price}
+
+ ))}
+
+ )}
+
+ );
+}
+
+export function DeploymentPricingBreakdown({
+ deployments,
+ heading,
+ className,
+}: {
+ deployments: readonly DeploymentPricing[] | null | undefined;
+ heading?: React.ReactNode;
+ className?: string;
+}) {
+ const { t } = useTranslation();
+ if (!deployments || deployments.length === 0) return null;
+ return (
+
+ {heading}
+ {deployments.length > 1 && (
+
+ {t("deploymentPricing.deploymentCount", { count: deployments.length })}
+
+ )}
+ {deployments.map((deployment, index) => (
+
+ ))}
+
{t("deploymentPricing.requestAdjustments")}
+
+ );
+}
diff --git a/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.test.tsx b/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.test.tsx
index 9a9c631890b..b42719510ea 100644
--- a/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.test.tsx
+++ b/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.test.tsx
@@ -5,10 +5,15 @@ import { ProviderLogo } from "./ProviderLogo";
import * as providerInfoHelpers from "../../provider_info_helpers";
vi.mock("../../provider_info_helpers");
+vi.mock("@/lib/serverRootPath", () => ({ serverRootPath: "/litellm" }));
describe("ProviderLogo", () => {
beforeEach(() => {
vi.resetAllMocks();
+ vi.mocked(providerInfoHelpers.getProviderLogoAndName).mockReturnValue({
+ logo: "/assets/provider-logo.svg",
+ displayName: "Test Provider",
+ });
});
it("should render the component", () => {
@@ -20,6 +25,39 @@ describe("ProviderLogo", () => {
expect(screen.getByText("t")).toBeInTheDocument();
});
+ it.each([
+ ["https://cdn.example.com/gateway.svg", "https://cdn.example.com/gateway.svg"],
+ ["/ui/assets/logos/gateway.svg", "/litellm/ui/assets/logos/gateway.svg"],
+ ])("should display a custom logo from %s", (logoUrl, expectedSrc) => {
+ render();
+
+ expect(screen.getByRole("img", { name: "test logo" })).toHaveAttribute("src", expectedSrc);
+ });
+
+ it.each([undefined, null, ""])("should use the provider logo when the override is %s", (logoUrl) => {
+ render();
+
+ expect(screen.getByRole("img", { name: "test logo" })).toHaveAttribute("src", "/assets/provider-logo.svg");
+ });
+
+ it("should recover when a failed custom logo is replaced", () => {
+ const { rerender } = render();
+
+ act(() => {
+ fireEvent.error(screen.getByRole("img", { name: "test logo" }));
+ });
+
+ expect(screen.queryByRole("img")).not.toBeInTheDocument();
+ expect(screen.getByText("t")).toBeInTheDocument();
+
+ rerender();
+
+ expect(screen.getByRole("img", { name: "test logo" })).toHaveAttribute(
+ "src",
+ "https://cdn.example.com/gateway.svg",
+ );
+ });
+
it("should show fallback when image fails to load", () => {
vi.mocked(providerInfoHelpers.getProviderLogoAndName).mockReturnValue({
logo: "/path/to/logo.png",
diff --git a/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.tsx b/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.tsx
index 4bc11126ae4..eb93dd77e30 100644
--- a/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.tsx
+++ b/ui/litellm-dashboard/src/components/molecules/models/ProviderLogo.tsx
@@ -3,9 +3,13 @@ import { Logo } from "@/components/molecules/logo/Logo";
interface ProviderLogoProps {
provider: string;
+ logoUrl?: string | null;
className?: string;
}
-export const ProviderLogo: React.FC = ({ provider, className = "w-4 h-4" }) => (
-
-);
+export const ProviderLogo: React.FC = ({ provider, logoUrl, className = "w-4 h-4" }) =>
+ logoUrl ? (
+
+ ) : (
+
+ );
diff --git a/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.test.ts b/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.test.ts
new file mode 100644
index 00000000000..4e38f55d1b3
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.test.ts
@@ -0,0 +1,186 @@
+import { describe, expect, it } from "vitest";
+
+import { buildPricingTable, describePricingTier, fallbackCostLabel, parseCostKey } from "./deploymentPricing";
+
+describe("parseCostKey", () => {
+ it.each([
+ ["input_cost_per_token", { charge: "input", modality: "", unit: "token", threshold: null, tier: "standard" }],
+ ["output_cost_per_token_flex", { charge: "output", unit: "token", threshold: null, tier: "flex" }],
+ ["input_cost_per_token_batches", { charge: "input", tier: "batches" }],
+ ["cache_read_input_token_cost", { charge: "cache_read", unit: "token", tier: "standard" }],
+ ["explicit_cache_read_input_token_cost", { charge: "explicit_cache_read", unit: "token", tier: "standard" }],
+ [
+ "cache_read_input_token_cost_above_272k_tokens_priority",
+ { charge: "cache_read", threshold: 272_000, tier: "priority" },
+ ],
+ [
+ "cache_creation_input_token_cost_above_12hr_above_272k_tokens_flex",
+ { charge: "cache_write", cacheDuration: "12hr", threshold: 272_000, tier: "flex" },
+ ],
+ ["cache_creation_input_token_cost_above_1hr", { charge: "cache_write", cacheDuration: "1hr", threshold: null }],
+ ["cache_read_input_audio_token_cost_priority", { charge: "cache_read", modality: "audio", tier: "priority" }],
+ ["input_cost_per_audio_token", { charge: "input", modality: "audio", unit: "token" }],
+ ["output_cost_per_reasoning_token", { charge: "output", modality: "reasoning", unit: "token" }],
+ [
+ "input_cost_per_audio_per_second_above_128k_tokens",
+ { charge: "input", modality: "audio", unit: "second", threshold: 128_000 },
+ ],
+ ["output_cost_per_second", { charge: "output", modality: "", unit: "second" }],
+ ["input_cost_per_image", { charge: "input", modality: "", unit: "image" }],
+ ["input_cost_per_image_token", { charge: "input", modality: "image", unit: "token" }],
+ ["output_cost_per_pixel", { unit: "pixel" }],
+ ["input_cost_per_character", { unit: "character" }],
+ ["input_cost_per_query", { unit: "query" }],
+ ["input_cost_per_token_above_200k_tokens_ultrafast", { threshold: 200_000, tier: "ultrafast" }],
+ ["output_cost_per_token_balanced", { tier: "balanced" }],
+ ])("reads %s", (key, expected) => {
+ expect(parseCostKey(key)).toMatchObject({ kind: "price", ...expected });
+ });
+
+ it.each([
+ "output_cost_per_second_720p",
+ "output_cost_per_image_1024",
+ "input_cost_per_token_cache_hit",
+ "input_cost_per_token_above_272k_tokens_decisions",
+ "input_cost_per_video_per_second_above_8s_interval",
+ "ocr_cost_per_page",
+ "input_dbu_cost_per_token",
+ "input_cost_per_text_message",
+ ])("keeps %s as an unknown key rather than guessing", (key) => {
+ expect(parseCostKey(key)).toEqual({ kind: "unknown", key });
+ });
+});
+
+const CONTRACT_COSTS = {
+ input_cost_per_token: 2.5e-6,
+ output_cost_per_token: 1e-5,
+ cache_read_input_token_cost: 1.25e-6,
+ cache_creation_input_token_cost: 3e-6,
+ input_cost_per_token_priority: 5e-6,
+ input_cost_per_token_flex: 1.25e-6,
+ input_cost_per_token_above_200k_tokens: 5e-6,
+ cache_read_input_token_cost_above_272k_tokens_priority: 1e-6,
+ input_cost_per_token_batches: 1.25e-6,
+ output_cost_per_second: 0.001,
+ input_cost_per_image: 0.002,
+};
+
+describe("buildPricingTable", () => {
+ const table = buildPricingTable(CONTRACT_COSTS);
+
+ it("offers a column per service tier the deployment prices, in a fixed order", () => {
+ expect(table.tiers).toEqual(["standard", "flex", "priority", "batches"]);
+ });
+
+ it("groups prices by context threshold, base prices first", () => {
+ expect(table.groups.map((group) => group.threshold)).toEqual([null, 200_000, 272_000]);
+ });
+
+ it("puts every tier of one charge on one row, token prices scaled to 1M tokens", () => {
+ expect(table.groups[0].rows).toEqual([
+ { label: "Input / 1M tokens", prices: { standard: "$2.50", flex: "$1.25", priority: "$5.00", batches: "$1.25" } },
+ { label: "Input / image", prices: { standard: "$0.002" } },
+ { label: "Cache read / 1M tokens", prices: { standard: "$1.25" } },
+ { label: "Cache write / 1M tokens", prices: { standard: "$3.00" } },
+ { label: "Output / 1M tokens", prices: { standard: "$10.00" } },
+ { label: "Output / second", prices: { standard: "$0.001" } },
+ ]);
+ });
+
+ it("shows an explicit cached read per 1M tokens on its own row", () => {
+ const { groups, unknown } = buildPricingTable({
+ cache_read_input_token_cost: 1e-7,
+ explicit_cache_read_input_token_cost: 5e-8,
+ });
+ expect(unknown).toEqual([]);
+ expect(groups[0].rows).toEqual([
+ { label: "Cache read / 1M tokens", prices: { standard: "$0.10" } },
+ { label: "Explicit cache read / 1M tokens", prices: { standard: "$0.05" } },
+ ]);
+ });
+
+ it("answers a cached read on priority above a threshold from its own group", () => {
+ expect(table.groups[2].rows).toEqual([{ label: "Cache read / 1M tokens", prices: { priority: "$1.00" } }]);
+ });
+
+ it("lists keys it cannot read under their raw name with the unscaled price", () => {
+ const { groups, unknown } = buildPricingTable({ input_cost_per_token: 1e-6, output_cost_per_second_720p: 0.05 });
+ expect(unknown).toEqual([{ key: "output_cost_per_second_720p", label: "Output / second 720p", price: "$0.05" }]);
+ expect(groups.flatMap((group) => group.rows.map((row) => row.label))).toEqual(["Input / 1M tokens"]);
+ });
+
+ it("shows a zero price as free and keeps precision for tiny per-unit prices", () => {
+ const { groups } = buildPricingTable({ input_cost_per_token: 0, output_cost_per_pixel: 1e-8 });
+ expect(groups[0].rows).toEqual([
+ { label: "Input / 1M tokens", prices: { standard: "Free" } },
+ { label: "Output / pixel", prices: { standard: "$0.00000001" } },
+ ]);
+ });
+
+ it("labels modality and cache duration in the row", () => {
+ const { groups } = buildPricingTable({
+ cache_creation_input_token_cost_above_1hr: 6e-6,
+ input_cost_per_audio_token: 4e-5,
+ output_cost_per_reasoning_token: 2e-5,
+ });
+ expect(groups[0].rows.map((row) => row.label)).toEqual([
+ "Audio input / 1M tokens",
+ "Cache write (1hr) / 1M tokens",
+ "Reasoning output / 1M tokens",
+ ]);
+ });
+
+ it("returns an empty table for a deployment with no prices", () => {
+ const empty = { tiers: [], groups: [], hasBlankCells: false, unknown: [] };
+ expect(buildPricingTable({})).toEqual(empty);
+ });
+
+ it("flags blank cells only when some charge lacks a price in a tier the deployment offers", () => {
+ const everyTierPriced = {
+ input_cost_per_token: 1e-6,
+ input_cost_per_token_priority: 2e-6,
+ output_cost_per_token: 3e-6,
+ output_cost_per_token_priority: 6e-6,
+ };
+ expect(buildPricingTable(CONTRACT_COSTS).hasBlankCells).toBe(true);
+ expect(buildPricingTable(everyTierPriced).hasBlankCells).toBe(false);
+ });
+});
+
+describe("fallbackCostLabel", () => {
+ it.each([
+ ["google_maps_grounding_cost_per_query", "Google maps grounding / query"],
+ ["file_search_cost_per_gb_per_day", "File search / gb per day"],
+ ["cost_per_second", "Cost / second"],
+ ["output_cost_per_second_720p", "Output / second 720p"],
+ ["explicit_cache_read_input_token_cost", "Explicit cache read input token"],
+ ["some_new_price", "Some new price"],
+ ])("names %s as %s", (key, label) => {
+ expect(fallbackCostLabel(key)).toBe(label);
+ });
+});
+
+describe("describePricingTier", () => {
+ it("states the token range and each price per 1M tokens, input before output", () => {
+ expect(
+ describePricingTier({
+ range: [0, 256_000],
+ costs: { output_cost_per_token: 4e-7, input_cost_per_token: 5e-8 },
+ }),
+ ).toBe("0–256K tokens: $0.05 input / $0.40 output");
+ });
+
+ it("names extra charges and non-token units", () => {
+ expect(
+ describePricingTier({
+ range: [256_000, 1_000_000],
+ costs: {
+ input_cost_per_token: 2.5e-7,
+ output_cost_per_reasoning_token: 1.2e-5,
+ input_cost_per_query: 0.01,
+ web_search_cost_per_call: 0.02,
+ },
+ }),
+ ).toBe("256K–1M tokens: $0.25 input / $0.01 input per query / $12.00 reasoning output / $0.02 web search / call");
+ });
+});
diff --git a/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.ts b/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.ts
new file mode 100644
index 00000000000..337a8900b7d
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/molecules/models/deploymentPricing.ts
@@ -0,0 +1,305 @@
+import type { ParseKeys } from "i18next";
+
+import i18n from "@/i18n/i18n";
+import type { components } from "@/lib/http/schema";
+
+export type PricingTier = components["schemas"]["PriceTier"];
+export type DeploymentPricing = components["schemas"]["DeploymentPricing"];
+
+export const SERVICE_TIERS = ["standard", "flex", "balanced", "priority", "ultrafast", "batches"] as const;
+export type ServiceTier = (typeof SERVICE_TIERS)[number];
+
+export const SERVICE_TIER_LABELS: Readonly> = {
+ standard: "deploymentPricing.tiers.standard",
+ flex: "deploymentPricing.tiers.flex",
+ balanced: "deploymentPricing.tiers.balanced",
+ priority: "deploymentPricing.tiers.priority",
+ ultrafast: "deploymentPricing.tiers.ultrafast",
+ batches: "deploymentPricing.tiers.batches",
+};
+
+const UNITS = ["token", "second", "image", "pixel", "character", "query", "request"] as const;
+type Unit = (typeof UNITS)[number];
+
+const CHARGES = ["input", "cache_read", "explicit_cache_read", "cache_write", "output"] as const;
+type Charge = (typeof CHARGES)[number];
+
+const MODALITIES = ["", "text", "image", "audio", "video", "reasoning"] as const;
+type Modality = (typeof MODALITIES)[number];
+
+export type CostKey =
+ | {
+ readonly kind: "price";
+ readonly charge: Charge;
+ readonly modality: Modality;
+ readonly cacheDuration: string;
+ readonly unit: Unit;
+ readonly threshold: number | null;
+ readonly tier: ServiceTier;
+ }
+ | { readonly kind: "unknown"; readonly key: string };
+
+const TIER_SUFFIX = /^(.+)_(flex|balanced|priority|ultrafast|batches)$/;
+const THRESHOLD_SUFFIX = /^(.+)_above_(\d+)k_tokens$/;
+const CACHE_DURATION_SUFFIX = /^(.+)_above_(\d+hr)$/;
+const DIRECT_PRICE = /^(input|output)_cost_per_(?:(text|image|audio|video|reasoning)_(?:per_)?)?([a-z]+)$/;
+const CACHE_PRICE = /^(explicit_)?cache_(read|creation)_input_(?:(image|audio)_)?token_cost$/;
+
+const splitSuffix = (key: string, pattern: RegExp): readonly [string, string | null] => {
+ const match = pattern.exec(key);
+ return match ? [match[1], match[2]] : [key, null];
+};
+
+const isUnit = (value: string): value is Unit => (UNITS as readonly string[]).includes(value);
+
+const parseBase = (base: string): Pick, "charge" | "modality" | "unit"> | null => {
+ const direct = DIRECT_PRICE.exec(base);
+ if (direct) {
+ const [, charge, modality = "", unit] = direct;
+ return isUnit(unit) ? { charge: charge as Charge, modality: modality as Modality, unit } : null;
+ }
+ const cache = CACHE_PRICE.exec(base);
+ if (cache) {
+ const [, explicit, kind, modality = ""] = cache;
+ if (kind === "creation")
+ return explicit ? null : { charge: "cache_write", modality: modality as Modality, unit: "token" };
+ return { charge: explicit ? "explicit_cache_read" : "cache_read", modality: modality as Modality, unit: "token" };
+ }
+ return null;
+};
+
+export const parseCostKey = (key: string): CostKey => {
+ const [withoutTier, tier] = splitSuffix(key, TIER_SUFFIX);
+ const [withoutThreshold, thousands] = splitSuffix(withoutTier, THRESHOLD_SUFFIX);
+ const [base, cacheDuration] = splitSuffix(withoutThreshold, CACHE_DURATION_SUFFIX);
+ const parsed = parseBase(base);
+ if (!parsed) return { kind: "unknown", key };
+ return {
+ kind: "price",
+ ...parsed,
+ cacheDuration: cacheDuration ?? "",
+ threshold: thousands === null ? null : Number(thousands) * 1000,
+ tier: (tier ?? "standard") as ServiceTier,
+ };
+};
+
+type Price = Extract;
+
+type LabelForm = "start" | "inline";
+
+const CHARGE_LABELS: Readonly>>> = {
+ start: {
+ input: "deploymentPricing.charges.input",
+ cache_read: "deploymentPricing.charges.cacheRead",
+ explicit_cache_read: "deploymentPricing.charges.explicitCacheRead",
+ cache_write: "deploymentPricing.charges.cacheWrite",
+ output: "deploymentPricing.charges.output",
+ },
+ inline: {
+ input: "deploymentPricing.chargesInline.input",
+ cache_read: "deploymentPricing.chargesInline.cacheRead",
+ explicit_cache_read: "deploymentPricing.chargesInline.explicitCacheRead",
+ cache_write: "deploymentPricing.chargesInline.cacheWrite",
+ output: "deploymentPricing.chargesInline.output",
+ },
+};
+
+const MODALITY_LABELS: Readonly, ParseKeys>>>> = {
+ start: {
+ text: "deploymentPricing.modalities.text",
+ image: "deploymentPricing.modalities.image",
+ audio: "deploymentPricing.modalities.audio",
+ video: "deploymentPricing.modalities.video",
+ reasoning: "deploymentPricing.modalities.reasoning",
+ },
+ inline: {
+ text: "deploymentPricing.modalitiesInline.text",
+ image: "deploymentPricing.modalitiesInline.image",
+ audio: "deploymentPricing.modalitiesInline.audio",
+ video: "deploymentPricing.modalitiesInline.video",
+ reasoning: "deploymentPricing.modalitiesInline.reasoning",
+ },
+};
+
+const UNIT_LABELS: Readonly> = {
+ token: "deploymentPricing.units.token",
+ second: "deploymentPricing.units.second",
+ image: "deploymentPricing.units.image",
+ pixel: "deploymentPricing.units.pixel",
+ character: "deploymentPricing.units.character",
+ query: "deploymentPricing.units.query",
+ request: "deploymentPricing.units.request",
+};
+
+const chargeName = ({ charge, modality }: Price, form: LabelForm): string =>
+ modality
+ ? i18n.t("deploymentPricing.withModality", {
+ modality: i18n.t(MODALITY_LABELS[form][modality]),
+ charge: i18n.t(CHARGE_LABELS.inline[charge]),
+ })
+ : i18n.t(CHARGE_LABELS[form][charge]);
+
+const chargeLabel = (price: Price, form: LabelForm): string =>
+ price.cacheDuration
+ ? i18n.t("deploymentPricing.withCacheDuration", { charge: chargeName(price, form), duration: price.cacheDuration })
+ : chargeName(price, form);
+
+const unitLabel = (unit: Unit): string => i18n.t(UNIT_LABELS[unit]);
+
+const rowLabel = (price: Price): string =>
+ i18n.t("deploymentPricing.rowLabel", { charge: chargeLabel(price, "start"), unit: unitLabel(price.unit) });
+
+const rowRank = (price: Price): readonly number[] => [
+ CHARGES.indexOf(price.charge),
+ UNITS.indexOf(price.unit),
+ MODALITIES.indexOf(price.modality),
+ parseInt(price.cacheDuration || "0", 10),
+];
+
+const compareRanks = (a: readonly number[], b: readonly number[]): number =>
+ a.reduce((result, value, index) => result || value - b[index], 0);
+
+const PRICE_FORMAT_OPTIONS: Intl.NumberFormatOptions = {
+ style: "currency",
+ currency: "USD",
+ minimumFractionDigits: 2,
+ maximumFractionDigits: 10,
+ useGrouping: false,
+};
+
+const PRICE_FORMAT = new Intl.NumberFormat("en-US", PRICE_FORMAT_OPTIONS);
+
+export const formatRawPrice = (value: number): string =>
+ value === 0 ? i18n.t("deploymentPricing.free") : PRICE_FORMAT.format(value);
+
+const formatUnitPrice = (value: number, unit: Unit): string =>
+ formatRawPrice(unit === "token" ? value * 1_000_000 : value);
+
+export interface PriceRow {
+ readonly label: string;
+ readonly prices: Readonly>>;
+}
+
+export interface PriceGroup {
+ readonly threshold: number | null;
+ readonly rows: readonly PriceRow[];
+}
+
+export interface UnknownPrice {
+ readonly key: string;
+ readonly label: string;
+ readonly price: string;
+}
+
+export interface PricingTable {
+ readonly tiers: readonly ServiceTier[];
+ readonly groups: readonly PriceGroup[];
+ readonly hasBlankCells: boolean;
+ readonly unknown: readonly UnknownPrice[];
+}
+
+const PER_UNIT_KEY = /^(.*?)_?cost_per_(.+)$/;
+const COST_SUFFIX = /^(.+?)_cost(?:_(.+))?$/;
+
+const sentenceCase = (text: string): string => {
+ const words = text.replaceAll("_", " ").trim();
+ return words.charAt(0).toUpperCase() + words.slice(1);
+};
+
+export const fallbackCostLabel = (key: string): string => {
+ const perUnit = PER_UNIT_KEY.exec(key);
+ if (perUnit) {
+ return i18n.t("deploymentPricing.rowLabel", {
+ charge: perUnit[1] ? sentenceCase(perUnit[1]) : i18n.t("deploymentPricing.fallbackCost"),
+ unit: perUnit[2].replaceAll("_", " "),
+ });
+ }
+ const suffixed = COST_SUFFIX.exec(key);
+ if (suffixed) return sentenceCase([suffixed[1], suffixed[2]].filter(Boolean).join(" "));
+ return sentenceCase(key);
+};
+
+interface ParsedEntry {
+ readonly key: string;
+ readonly value: number;
+ readonly cost: CostKey;
+}
+
+interface PriceEntry extends ParsedEntry {
+ readonly cost: Price;
+}
+
+const isPriceEntry = (entry: ParsedEntry): entry is PriceEntry => entry.cost.kind === "price";
+
+const parseCosts = (costs: Readonly>): readonly ParsedEntry[] =>
+ Object.entries(costs).map(([key, value]) => ({ key, value, cost: parseCostKey(key) }));
+
+const uniqueBy = (items: readonly T[], keyOf: (item: T) => K): readonly T[] =>
+ items.filter((item, index) => items.findIndex((other) => keyOf(other) === keyOf(item)) === index);
+
+const byRowRank = (a: PriceEntry, b: PriceEntry): number => compareRanks(rowRank(a.cost), rowRank(b.cost));
+
+const groupRows = (entries: readonly PriceEntry[]): readonly PriceRow[] =>
+ [...uniqueBy(entries, ({ cost }) => rowLabel(cost))].sort(byRowRank).map(({ cost }) => ({
+ label: rowLabel(cost),
+ prices: Object.fromEntries(
+ entries
+ .filter((other) => rowLabel(other.cost) === rowLabel(cost))
+ .map((other) => [other.cost.tier, formatUnitPrice(other.value, other.cost.unit)]),
+ ),
+ }));
+
+export const buildPricingTable = (costs: Readonly>): PricingTable => {
+ const parsed = parseCosts(costs);
+ const prices = parsed.filter(isPriceEntry);
+ const thresholds = [...uniqueBy(prices, ({ cost }) => cost.threshold)]
+ .map(({ cost }) => cost.threshold)
+ .sort((a, b) => (a ?? 0) - (b ?? 0));
+ const tiers = SERVICE_TIERS.filter((tier) => prices.some(({ cost }) => cost.tier === tier));
+ const groups = thresholds.map((threshold) => ({
+ threshold,
+ rows: groupRows(prices.filter(({ cost }) => cost.threshold === threshold)),
+ }));
+ return {
+ tiers,
+ groups,
+ hasBlankCells: groups.some(({ rows }) => rows.some(({ prices: row }) => tiers.some((tier) => !(tier in row)))),
+ unknown: parsed
+ .filter((entry) => !isPriceEntry(entry))
+ .map(({ key, value }) => ({ key, label: fallbackCostLabel(key), price: formatRawPrice(value) })),
+ };
+};
+
+export const formatTokenCount = (tokens: number): string => {
+ if (tokens >= 1_000_000) return `${+(tokens / 1_000_000).toFixed(1)}M`;
+ if (tokens >= 1_000) return `${+(tokens / 1_000).toFixed(1)}K`;
+ return String(tokens);
+};
+
+const tierPart = ({ cost, value }: PriceEntry): string => {
+ const charge = chargeLabel(cost, "inline");
+ const price = formatUnitPrice(value, cost.unit);
+ return cost.unit === "token"
+ ? i18n.t("deploymentPricing.tierPrice", { price, charge })
+ : i18n.t("deploymentPricing.tierPricePerUnit", { price, charge, unit: unitLabel(cost.unit) });
+};
+
+export const describePricingTier = ({ range: [low, high], costs }: PricingTier): string => {
+ const parsed = parseCosts(costs);
+ const parts = [
+ ...[...parsed.filter(isPriceEntry)].sort(byRowRank).map(tierPart),
+ ...parsed
+ .filter((entry) => !isPriceEntry(entry))
+ .map(({ key, value }) =>
+ i18n.t("deploymentPricing.tierPrice", {
+ price: formatRawPrice(value),
+ charge: fallbackCostLabel(key).toLowerCase(),
+ }),
+ ),
+ ];
+ return i18n.t("deploymentPricing.tierRange", {
+ low: formatTokenCount(low),
+ high: formatTokenCount(high),
+ prices: parts.join(" / "),
+ });
+};
diff --git a/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx b/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
index 604d5923caa..6a78fe9441a 100644
--- a/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
+++ b/ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
@@ -11,15 +11,27 @@ import {
resolveLitellmProviderSlug,
} from "./provider_info_helpers";
-const PROVIDER_CREATE_FIELDS: { provider: string; litellm_provider: string }[] = JSON.parse(
- readFileSync(resolve(__dirname, "../../../../litellm/proxy/public_endpoints/provider_create_fields.json"), "utf8"),
-);
+const PROVIDER_CREATE_FIELDS: { provider: string; provider_display_name: string; litellm_provider: string }[] =
+ JSON.parse(
+ readFileSync(resolve(__dirname, "../../../../litellm/proxy/public_endpoints/provider_create_fields.json"), "utf8"),
+ );
const BUNDLED_MODEL_MAP: Record = JSON.parse(
readFileSync(resolve(__dirname, "../../../../model_prices_and_context_window.json"), "utf8"),
);
describe("provider_info_helpers", () => {
+ it("should identify the upstream LiteLLM Gateway consistently with provider metadata", () => {
+ const metadata = PROVIDER_CREATE_FIELDS.find((provider) => provider.provider === "LITELLM_PROXY");
+ expect(resolveLitellmProviderSlug("LITELLM_PROXY")).toBe(metadata?.litellm_provider);
+ expect(Providers.LITELLM_PROXY).toBe(metadata?.provider_display_name);
+ expect(getProviderLogoAndName("litellm_proxy")).toEqual({
+ displayName: Providers.LITELLM_PROXY,
+ logo: providerLogoMap[Providers.LITELLM_PROXY],
+ });
+ expect(providerLogoMap[Providers.LITELLM_PROXY]).toContain("litellm_monogram");
+ });
+
it("resolves Meta Model API separately from Meta Llama and isolates their bundled models", () => {
expect(resolveLitellmProviderSlug("META")).toBe("meta");
expect(getPlaceholder(Providers.META)).toBe("meta/muse-spark-1.1");
diff --git a/ui/litellm-dashboard/src/components/provider_info_helpers.tsx b/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
index 167343fc05e..8efd42f52e5 100644
--- a/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
+++ b/ui/litellm-dashboard/src/components/provider_info_helpers.tsx
@@ -34,6 +34,7 @@ import hostingerLogo from "../../public/assets/logos/hostinger.png";
import infinityLogo from "../../public/assets/logos/infinity.png";
import jinaLogo from "../../public/assets/logos/jina.png";
import lambdaLogo from "../../public/assets/logos/lambda.svg";
+import litellmLogo from "../../public/assets/logos/litellm_monogram.svg";
import lmstudioLogo from "../../public/assets/logos/lmstudio.svg";
import metaLlamaLogo from "../../public/assets/logos/meta_llama.svg";
import microsoft365CopilotLogo from "../../public/assets/logos/microsoft_365_copilot.svg";
@@ -141,6 +142,7 @@ export enum Providers {
LAMBDA_AI = "Lambda Ai",
Laya = "Laya",
LEMONADE = "Lemonade",
+ LITELLM_PROXY = "LiteLLM Gateway",
LLAMAFILE = "Llamafile",
LM_STUDIO = "Lm Studio",
LLAMA = "Meta Llama",
@@ -276,6 +278,7 @@ export const provider_map: Record = {
LAMBDA_AI: "lambda_ai",
Laya: "laya",
LEMONADE: "lemonade",
+ LITELLM_PROXY: "litellm_proxy",
LLAMAFILE: "llamafile",
LLAMA: "meta_llama",
META: "meta",
@@ -402,6 +405,7 @@ export const providerLogoMap: Partial> = {
[Providers.JinaAI]: jinaLogo.src,
[Providers.OPENCODE_GO]: opencodeLogo.src,
[Providers.LAMBDA_AI]: lambdaLogo.src,
+ [Providers.LITELLM_PROXY]: litellmLogo.src,
[Providers.LM_STUDIO]: lmstudioLogo.src,
[Providers.LLAMA]: metaLlamaLogo.src,
[Providers.META]: metaLlamaLogo.src,
@@ -506,6 +510,7 @@ const providerPlaceholderMap: Partial> = {
[Providers.Google_AI_Studio]: "gemini-pro",
[Providers.HOSTINGER_ROUTER]: "hostinger_router/claude-sonnet-5",
[Providers.JinaAI]: "jina_ai/",
+ [Providers.LITELLM_PROXY]: "litellm_proxy/*",
[Providers.META]: "meta/muse-spark-1.1",
[Providers.MICROSOFT_365_COPILOT]: "microsoft_365_copilot/chat",
[Providers.NVIDIA_RIVA]: "nvidia_riva/nvidia/parakeet-ctc-1_1b-asr",
diff --git a/ui/litellm-dashboard/src/components/publicModelHub/ModelInspector.tsx b/ui/litellm-dashboard/src/components/publicModelHub/ModelInspector.tsx
index 95901f2bf1d..39495baeea3 100644
--- a/ui/litellm-dashboard/src/components/publicModelHub/ModelInspector.tsx
+++ b/ui/litellm-dashboard/src/components/publicModelHub/ModelInspector.tsx
@@ -4,6 +4,7 @@ import { ChevronDown } from "lucide-react";
import * as React from "react";
import { Trans, useTranslation } from "react-i18next";
+import { DeploymentPricingBreakdown } from "@/components/molecules/models/DeploymentPricingBreakdown";
import type { ModelGroupInfo } from "@/components/PublicModelHubTableColumns";
import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert";
import { Button } from "@/components/ui/button";
@@ -17,6 +18,7 @@ import {
modeLabel,
modelCapabilities,
providerName,
+ SECTION_TITLE,
wildcardExample,
} from "./hubFormat";
import { Badges, CodeBlock, type Fact, Facts, PaneSection, ProviderAvatar, ProviderName } from "./HubParts";
@@ -175,6 +177,10 @@ export function ModelInspector({ model }: { model: ModelGroupInfo }) {
)}
+ {t("publicModelHub.modelInspector.pricingByDeployment")}}
+ />
{capabilities.length > 0 && (
diff --git a/ui/litellm-dashboard/src/components/publicModelHub/PublicAiHub.integration.test.tsx b/ui/litellm-dashboard/src/components/publicModelHub/PublicAiHub.integration.test.tsx
index e966836fa3a..cf40a685d76 100644
--- a/ui/litellm-dashboard/src/components/publicModelHub/PublicAiHub.integration.test.tsx
+++ b/ui/litellm-dashboard/src/components/publicModelHub/PublicAiHub.integration.test.tsx
@@ -77,6 +77,13 @@ const MODELS = [
output_cost_per_token: 0,
max_input_tokens: 0,
description: "General purpose",
+ deployment_pricing: [
+ {
+ provider: "azure",
+ costs: { output_cost_per_token: 1e-5, output_cost_per_token_priority: 2e-5 },
+ tiered_pricing: [],
+ },
+ ],
},
];
@@ -362,6 +369,24 @@ describe("PublicAiHub", () => {
expect(within(pricedRow).queryByRole("cell", { name: "Free" })).not.toBeInTheDocument();
});
+ it("shows the open model's prices for each of its deployments", async () => {
+ const user = userEvent.setup();
+ renderWithProviders();
+
+ await user.click(await screen.findByRole("link", { name: /gpt-5/ }));
+
+ expect(await screen.findByRole("heading", { name: "Pricing by deployment" })).toBeInTheDocument();
+ const azure = screen.getByRole("region", { name: "Azure pricing" });
+ expect(within(azure).getByRole("row", { name: /Output \/ 1M tokens/ })).toHaveTextContent("$10.00$20.00");
+ });
+
+ it("leaves the pricing breakdown out for a model without deployment prices", async () => {
+ renderWithProviders(, { searchParams: { open: "claude-sonnet-4-5" } });
+
+ expect(await screen.findByRole("heading", { level: 2, name: "claude-sonnet-4-5" })).toBeInTheDocument();
+ expect(screen.queryByRole("heading", { name: "Pricing by deployment" })).not.toBeInTheDocument();
+ });
+
it("treats an empty open param as nothing opened", async () => {
renderWithProviders(, { searchParams: { open: "" } });
diff --git a/ui/litellm-dashboard/src/components/public_model_hub.test.tsx b/ui/litellm-dashboard/src/components/public_model_hub.test.tsx
index 515b8230d5b..4081751ca73 100644
--- a/ui/litellm-dashboard/src/components/public_model_hub.test.tsx
+++ b/ui/litellm-dashboard/src/components/public_model_hub.test.tsx
@@ -327,6 +327,33 @@ describe("PublicModelHub", () => {
expect(example).toHaveTextContent('model="gpt-4"');
});
+ it("shows each deployment's full pricing in the model details", async () => {
+ respondWith([
+ model({
+ model_group: "gpt-4",
+ input_cost_per_token: 2.5e-6,
+ deployment_pricing: [
+ {
+ provider: "openai",
+ costs: { input_cost_per_token: 2.5e-6, cache_read_input_token_cost_above_272k_tokens_priority: 1e-6 },
+ tiered_pricing: [{ range: [0, 256_000], costs: { input_cost_per_token: 5e-8 } }],
+ },
+ ],
+ }),
+ ]);
+ renderHub();
+
+ fireEvent.click(await screen.findByRole("button", { name: "gpt-4" }));
+
+ const openai = await screen.findByRole("region", { name: "OpenAI pricing" });
+ const aboveThreshold = within(openai).getByRole("rowgroup", { name: "Above 272K tokens" });
+ expect(within(aboveThreshold).getByRole("row", { name: /Cache read/ })).toHaveTextContent(
+ "Cache read / 1M tokens$1.00",
+ );
+ expect(within(openai).getByText("0–256K tokens: $0.05 input")).toBeInTheDocument();
+ expect(within(screen.getByRole("dialog")).getByText("$2.5000")).toBeInTheDocument();
+ });
+
it("filters by several providers at once, and returns to the first page", async () => {
const user = userEvent.setup();
respondWith(DEFAULT_MODELS, 300);
diff --git a/ui/litellm-dashboard/src/components/public_model_hub.tsx b/ui/litellm-dashboard/src/components/public_model_hub.tsx
index a80bf939165..552ed6fc490 100644
--- a/ui/litellm-dashboard/src/components/public_model_hub.tsx
+++ b/ui/litellm-dashboard/src/components/public_model_hub.tsx
@@ -56,6 +56,7 @@ import { displayMode, isDecisionMode } from "@/lib/decisionModels";
import { modelUsageExample } from "@/components/AIHub/modelUsageExample";
import { filterBySearchTerm, rankBySearchRelevance } from "@/utils/searchUtils";
import { ModelCustomMetadata } from "@/components/AIHub/ModelCustomMetadata";
+import { DeploymentPricingBreakdown } from "@/components/molecules/models/DeploymentPricingBreakdown";
import ModelHubProviders, { formatModelModeLabel } from "@/components/AIHub/ModelHubProviders";
interface PublicModelHubProps {
@@ -954,6 +955,13 @@ const PublicModelHub: React.FC = ({ accessToken, isEmbedded
+ {t("aiHub.publicModelHub.modelDialog.pricingByDeployment")}
+ }
+ />
{/* Capabilities */}
diff --git a/ui/litellm-dashboard/src/components/shared/SearchSelect.integration.test.tsx b/ui/litellm-dashboard/src/components/shared/SearchSelect.integration.test.tsx
index ecb8dd161f3..75ba453c501 100644
--- a/ui/litellm-dashboard/src/components/shared/SearchSelect.integration.test.tsx
+++ b/ui/litellm-dashboard/src/components/shared/SearchSelect.integration.test.tsx
@@ -5,6 +5,7 @@ import { describe, expect, it, vi } from "vitest";
import { SearchSelect } from "./SearchSelect";
import { chooseSelectOption } from "../../../tests/test-utils";
+import { ProviderLogo } from "../molecules/models/ProviderLogo";
const OPTIONS = [
{ label: "Acme Prod", value: "team-1" },
@@ -30,6 +31,38 @@ describe("SearchSelect", () => {
expect(screen.getByRole("combobox")).toHaveValue("Growth");
});
+ it("should keep the LiteLLM Gateway logo visible after selection and remove it when cleared", async () => {
+ const user = userEvent.setup();
+ const options = [
+ {
+ label: "LiteLLM Gateway",
+ value: "LITELLM_PROXY",
+ icon: ,
+ },
+ ];
+ function Controlled() {
+ const [value, setValue] = useState(null);
+ return ;
+ }
+ render();
+ const input = screen.getByRole("combobox", { name: "Provider" });
+
+ await chooseSelectOption(user, input, /LiteLLM Gateway/);
+ await user.tab();
+
+ expect(input).toHaveValue("LiteLLM Gateway");
+ expect(screen.getByRole("img", { name: "litellm_proxy logo" })).toHaveAttribute(
+ "src",
+ expect.stringContaining("litellm_monogram"),
+ );
+
+ await user.click(screen.getByRole("button", { name: "Clear" }));
+ await user.tab();
+
+ expect(input).toHaveValue("");
+ expect(screen.queryByRole("img", { name: "litellm_proxy logo" })).not.toBeInTheDocument();
+ });
+
it("shows a value the options do not carry yet instead of blanking the field", () => {
const { rerender } = render();
expect(screen.getByRole("combobox")).toHaveValue("team-2");
diff --git a/ui/litellm-dashboard/src/components/shared/SearchSelect.tsx b/ui/litellm-dashboard/src/components/shared/SearchSelect.tsx
index 897782b6c7f..b2188267001 100644
--- a/ui/litellm-dashboard/src/components/shared/SearchSelect.tsx
+++ b/ui/litellm-dashboard/src/components/shared/SearchSelect.tsx
@@ -9,6 +9,7 @@ import {
ComboboxItem,
ComboboxList,
} from "@/components/ui/combobox";
+import { InputGroupAddon } from "@/components/ui/input-group";
import i18n from "@/i18n/i18n";
export interface SearchSelectOption {
@@ -47,6 +48,11 @@ const matchesQuery = (option: SearchSelectOption, query: string): boolean => {
return option.label.toLowerCase().includes(q) || (option.sublabel?.toLowerCase().includes(q) ?? false);
};
+const SelectedOptionIcon = ({ option }: { option: SearchSelectOption | null }) => {
+ if (!option?.icon) return null;
+ return {option.icon};
+};
+
export function SearchSelect({
options,
value,
@@ -91,7 +97,9 @@ export function SearchSelect({
placeholder={placeholder}
showClear={allowClear && value != null && value !== ""}
className={`h-8 w-full text-sm ${className ?? ""}`}
- />
+ >
+
+
{emptyText}
diff --git a/ui/litellm-dashboard/src/i18n/locales/en.json b/ui/litellm-dashboard/src/i18n/locales/en.json
index 919d6cd25cc..3c9dbc99a6b 100644
--- a/ui/litellm-dashboard/src/i18n/locales/en.json
+++ b/ui/litellm-dashboard/src/i18n/locales/en.json
@@ -7217,7 +7217,8 @@
"rpm": "Requests per Minute:",
"openaiParams": "Supported OpenAI Parameters",
"usageExample": "Usage Example",
- "copyToClipboard": "Copy to clipboard"
+ "copyToClipboard": "Copy to clipboard",
+ "pricingByDeployment": "Pricing by Deployment:"
},
"agentDialog": {
"fallbackTitle": "Agent Details",
@@ -7370,7 +7371,8 @@
"usageExample": "Usage Example",
"decisionNote": "Decision models answer on /v1/decisions and /v1/systemone, not /chat/completions.",
"howToCall": "How to call decision models",
- "tryPlayground": "Try it in the Playground"
+ "tryPlayground": "Try it in the Playground",
+ "pricingByDeployment": "Pricing by Deployment:"
},
"agentDialog": {
"fallbackTitle": "Agent Details",
@@ -7905,7 +7907,8 @@
"title": "Wildcard model",
"description": "Call it as , with any model name this provider offers"
},
- "capabilities": "Capabilities"
+ "capabilities": "Capabilities",
+ "pricingByDeployment": "Pricing by deployment"
},
"agentInspector": {
"copyName": "Copy agent name",
@@ -20300,5 +20303,69 @@
"save": "Save Changes"
}
}
+ },
+ "deploymentPricing": {
+ "deploymentCount_one": "This model has {{count}} deployment. Each request is billed at the prices of the deployment that served it.",
+ "deploymentCount_other": "This model has {{count}} deployments. Each request is billed at the prices of the deployment that served it.",
+ "requestAdjustments": "These are list prices. Adjustments made per request, like off-peak discounts, regional or data residency surcharges, and some providers' own pricing rules, aren't included.",
+ "providerPricing": "{{provider}} pricing",
+ "chargeColumn": "Charge",
+ "basePrices": "Base prices",
+ "aboveThreshold": "Above {{threshold}} tokens",
+ "blankCells": "Blank cells aren't priced separately for this deployment.",
+ "tieredPricing": "Prices by request size, per 1M tokens",
+ "tierRange": "{{low}}–{{high}} tokens: {{prices}}",
+ "tierPrice": "{{price}} {{charge}}",
+ "tierPricePerUnit": "{{price}} {{charge}} per {{unit}}",
+ "free": "Free",
+ "rowLabel": "{{charge}} / {{unit}}",
+ "withModality": "{{modality}} {{charge}}",
+ "withCacheDuration": "{{charge}} ({{duration}})",
+ "fallbackCost": "Cost",
+ "tiers": {
+ "standard": "Standard",
+ "flex": "Flex",
+ "balanced": "Balanced",
+ "priority": "Priority",
+ "ultrafast": "Ultrafast",
+ "batches": "Batch"
+ },
+ "charges": {
+ "input": "Input",
+ "cacheRead": "Cache read",
+ "explicitCacheRead": "Explicit cache read",
+ "cacheWrite": "Cache write",
+ "output": "Output"
+ },
+ "chargesInline": {
+ "input": "input",
+ "cacheRead": "cache read",
+ "explicitCacheRead": "explicit cache read",
+ "cacheWrite": "cache write",
+ "output": "output"
+ },
+ "modalities": {
+ "text": "Text",
+ "image": "Image",
+ "audio": "Audio",
+ "video": "Video",
+ "reasoning": "Reasoning"
+ },
+ "modalitiesInline": {
+ "text": "text",
+ "image": "image",
+ "audio": "audio",
+ "video": "video",
+ "reasoning": "reasoning"
+ },
+ "units": {
+ "token": "1M tokens",
+ "second": "second",
+ "image": "image",
+ "pixel": "pixel",
+ "character": "character",
+ "query": "query",
+ "request": "request"
+ }
}
}
diff --git a/ui/litellm-dashboard/src/lib/http/schema.d.ts b/ui/litellm-dashboard/src/lib/http/schema.d.ts
index 2272b4bea43..5cef303ee67 100644
--- a/ui/litellm-dashboard/src/lib/http/schema.d.ts
+++ b/ui/litellm-dashboard/src/lib/http/schema.d.ts
@@ -1864,13 +1864,15 @@ export interface paths {
};
/**
* Codex Passthrough Route
- * @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
+ * @description Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
+ * key's other chat models through LiteLLM.
*/
get: operations["codex_passthrough_route_backend_api_codex__endpoint__get"];
put?: never;
/**
* Codex Passthrough Route
- * @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
+ * @description Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
+ * key's other chat models through LiteLLM.
*/
post: operations["codex_passthrough_route_backend_api_codex__endpoint__post"];
delete?: never;
@@ -17159,6 +17161,7 @@ export interface paths {
*
* Parameters:
* - name: str - The name of the tag to update
+ * - spend: Optional[float] - Updated spend for the tag
* - description: Optional[str] - Updated description
* - models: List[str] - Updated list of allowed LLM models
* - budget_id: Optional[str] - The id for a budget to associate with the tag
@@ -32836,6 +32839,20 @@ export interface components {
} & {
[key: string]: unknown;
};
+ /** DeploymentPricing */
+ DeploymentPricing: {
+ /** Costs */
+ costs: {
+ [key: string]: number;
+ };
+ /** Provider */
+ provider: string;
+ /**
+ * Tiered Pricing
+ * @default []
+ */
+ tiered_pricing: components["schemas"]["PriceTier"][];
+ };
/** DeploymentWarning */
DeploymentWarning: {
/** Id */
@@ -39686,6 +39703,11 @@ export interface components {
} | null;
/** Custom Tags */
custom_tags?: string[] | null;
+ /**
+ * Deployment Pricing
+ * @default []
+ */
+ deployment_pricing: components["schemas"]["DeploymentPricing"][];
/** Description */
description?: string | null;
/** Health Checked At */
@@ -42890,6 +42912,18 @@ export interface components {
*/
version_status: string;
};
+ /** PriceTier */
+ PriceTier: {
+ /** Costs */
+ costs: {
+ [key: string]: number;
+ };
+ /** Range */
+ range: [
+ number,
+ number
+ ];
+ };
/** Prompt */
Prompt: {
litellm_params: components["schemas"]["PromptLiteLLMParams"];
@@ -47982,6 +48016,8 @@ export interface components {
rpm_limit?: number | null;
/** Soft Budget */
soft_budget?: number | null;
+ /** Spend */
+ spend?: number | null;
/** Tpm Limit */
tpm_limit?: number | null;
};
@@ -55936,7 +55972,7 @@ export interface operations {
};
};
responses: {
- /** @description Unmodified Codex response or server-sent events */
+ /** @description Codex response, the model catalog with the key's LiteLLM models, or server-sent events */
200: {
headers: {
[name: string]: unknown;
diff --git a/uv.lock b/uv.lock
index fb591264fc3..b6a9a599ba2 100644
--- a/uv.lock
+++ b/uv.lock
@@ -28,6 +28,8 @@ members = [
constraints = [
{ name = "aiohttp", specifier = ">=3.14.2,<4.0" },
{ name = "dnspython", marker = "python_full_version >= '3.11'", specifier = ">=2.9.0" },
+ { name = "filelock", marker = "python_full_version < '3.11'", specifier = "<3.20.4" },
+ { name = "filelock", marker = "python_full_version >= '3.11'", specifier = ">=4.2.1" },
{ name = "httplib2", specifier = ">=0.32.0" },
{ name = "packaging", specifier = ">=24.0" },
{ name = "setuptools", specifier = ">=84.1.0" },
@@ -2238,11 +2240,30 @@ wheels = [
[[package]]
name = "filelock"
-version = "3.32.7"
+version = "3.20.3"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/0f/59/e19834834cb01a32febfbb0f8a23a9088088f5d45991824ff2bc3b5e8acb/filelock-3.32.7.tar.gz", hash = "sha256:37b8a3d9811b0f9aef7e5ec5c71bb320de52df51e6ca9bcd6f5ad81187660da7", size = 225154, upload-time = "2026-09-16T00:24:20.907Z" }
+resolution-markers = [
+ "python_full_version < '3.11'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1d/65/ce7f1b70157833bf3cb851b556a37d4547ceafc158aa9b34b36782f23696/filelock-3.20.3.tar.gz", hash = "sha256:18c57ee915c7ec61cff0ecf7f0f869936c7c30191bb0cf406f1341778d0834e1", size = 19485, upload-time = "2026-01-09T17:55:05.421Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/15/df/31098c5aeb4d966b553641472bd55fcf5fdfac953549894b8a765ba44e91/filelock-3.32.7-py3-none-any.whl", hash = "sha256:65ff0d0190ea42038b32bda4b77834fb05be2cad4c5b9b01aa4dfb3614536e52", size = 100157, upload-time = "2026-09-16T00:24:19.543Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/36/7fb70f04bf00bc646cd5bb45aa9eddb15e19437a28b8fb2b4a5249fac770/filelock-3.20.3-py3-none-any.whl", hash = "sha256:4b0dda527ee31078689fc205ec4f1c1bf7d56cf88b6dc9426c4f230e46c2dce1", size = 16701, upload-time = "2026-01-09T17:55:04.334Z" },
+]
+
+[[package]]
+name = "filelock"
+version = "4.2.4"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.15'",
+ "python_full_version == '3.14.*'",
+ "python_full_version == '3.13.*'",
+ "python_full_version == '3.12.*'",
+ "python_full_version == '3.11.*'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d8/28/3893324bcadf8f89b1616637468d766af3f625008c0735bf704fdfa86617/filelock-4.2.4.tar.gz", hash = "sha256:c6d8663535e1650ee430507b4ffbac7b464dd64b7647ea0112a9f406499d163d", size = 652444, upload-time = "2026-10-19T01:57:51.001Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e0/1b/5e74289a1878e666bb4b65fe5805a04cafcac2aa5f519ef1ead67ef91c56/filelock-4.2.4-py3-none-any.whl", hash = "sha256:301241438c57936ef93f4243a94f2559022268bfcf9b375a9a72eb0f8db8358e", size = 162475, upload-time = "2026-10-19T01:57:49.419Z" },
]
[[package]]
@@ -3829,7 +3850,8 @@ version = "1.32.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
- { name = "filelock" },
+ { name = "filelock", version = "3.20.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
+ { name = "filelock", version = "4.2.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
{ name = "fsspec" },
{ name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" },
{ name = "httpx" },
@@ -5008,7 +5030,8 @@ dependencies = [
{ name = "certifi" },
{ name = "click" },
{ name = "fastuuid", marker = "python_full_version < '3.15'" },
- { name = "filelock" },
+ { name = "filelock", version = "3.20.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
+ { name = "filelock", version = "4.2.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
{ name = "httpx", extra = ["http2"] },
{ name = "huggingface-hub" },
{ name = "importlib-metadata" },
@@ -6732,11 +6755,11 @@ wheels = [
[[package]]
name = "oauthlib"
-version = "4.0.0"
+version = "4.0.1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/7a/d8/a1bcc8ba112a627f8ffbdc212a78ce18d3ac07e91a5ca65d27918eee25a1/oauthlib-4.0.0.tar.gz", hash = "sha256:efb274799819440f95b4ab3b818869f1ce9ae26c5beacba0201d1a1b76b54f86", size = 187232, upload-time = "2026-09-28T06:01:18.77Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/33/17/2bb072de0b990303c985f26869fd47bb179024a051f47e7f9d208eb92f8e/oauthlib-4.0.1.tar.gz", hash = "sha256:3d73febdb6faa00f277c81fb04be7b7586fb96fc3b7f0b9c772dada69a25ede7", size = 191219, upload-time = "2026-10-31T09:52:58.008Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d9/f4/78229a1066068ca14fc60fb26cf7381cabe4382261392b90e5f9552722d4/oauthlib-4.0.0-py3-none-any.whl", hash = "sha256:624c28c13a0a59cabf9747dfa52af63be3e512a7f2714df16e91b5b3a145e6cd", size = 159715, upload-time = "2026-09-28T06:01:17.008Z" },
+ { url = "https://files.pythonhosted.org/packages/df/ef/3b8adde1b32a6f540758bff734d3442c621734c83708e5f02b0ba2f43b84/oauthlib-4.0.1-py3-none-any.whl", hash = "sha256:498a298876e996301950a5b053d12c31628bb7b895db03666430c69ce182eb61", size = 161934, upload-time = "2026-10-31T09:52:56.206Z" },
]
[[package]]