chore: merge main into litellm_spend_counter_ttl

AGENTS.md

@@ -52,7 +52,6 @@ If you ever write any human-facing text (pull requests, issues, commit messages,
 - don't use "—". Don't use ";" or ":" unless it would be nonsensical not to. Instead, reach for ",", ".", conjunction words, etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them also feels AI-y. A word cap does not penalize you for adding more sentences
 - don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
 - unless explicitly asked, don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
-- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a ".". Exception is that docs (i.e., litellm-docs) should have trailing "."
 - don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
 - use plain, simple, everyday engineering language: the common phrase engineers actually say over rare compact phrasing, in grammatically complete sentences. When explicitly asked to use bullets or ordered lists and structure legitimately helps the reader, prefer nested bullets (any depth is fine) over dense lines in a flat structure
 

cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md

@@ -46,7 +46,43 @@ For a managed workspace rollout, use OpenAI's [gateway configuration guide](http
 
 ## Native compatibility and Pi
 
-For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials but does not need a model deployment or resolve model aliases. Restricted keys must allow the upstream model name.
+For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials. A ChatGPT model keeps its upstream name and needs no deployment of its own.
+
+### Mix ChatGPT and LiteLLM models
+
+On the native route, Codex's model picker shows OpenAI's catalog first and then the other chat and responses models the virtual key can use. Pick one of those and Codex's requests go through LiteLLM's `/v1/responses` with the key's usual routing, budgets, guardrails, and logging. The proxy removes the ChatGPT bearer and account header from that request, so the other provider and your callbacks never receive them. A name is routed this way when another provider serves it as an exact `model_name`, through a router `model_group_alias`, or as a team's public model name. A ChatGPT model, a name that only a wildcard deployment matches, or a name OpenAI's catalog lists for the caller's ChatGPT account keeps the passthrough, so a model OpenAI adds later still uses the subscription even when a deployment of another provider has the same name. The proxy learns the account's list from the catalog Codex fetches, or asks OpenAI with the caller's login. When it cannot get the list, for example because OpenAI is down, the request times out, or the client is not Codex, every request keeps the passthrough for the next minute, as it did before LiteLLM models were added to the picker. When the key can use many models, the picker lists the ones that fit in Codex's catalog size limit, and the proxy logs a warning with how many it left out.
+
+Codex's picker sends a `none` reasoning effort for a LiteLLM model whose catalog entry lists no effort levels. With `litellm_settings: drop_params: true`, the proxy drops it for a model that does not support `none`, and the provider applies its own default. A model whose cost map entry or `model_info` sets `supports_none_reasoning_effort: true` keeps it. Without `drop_params`, the effort reaches the provider unchanged.
+
+Codex also sends its ChatGPT app tools with every turn, which on a LiteLLM model adds a few hundred thousand prompt tokens to each turn. Set `apps = false` under `[features]` in Codex's `config.toml` to turn them off.
+
+Only a key that may call `/v1/responses` gets these models. A key whose `allowed_routes` covers only the native routes sees OpenAI's catalog unchanged, and every request it sends uses the subscription.
+
+Codex sends earlier reasoning items back with each turn. The proxy tags the reasoning that a LiteLLM model returns on this route, OpenAI and Azure deployments included. When a session switches to a ChatGPT model, the proxy drops those items before forwarding the request to OpenAI, which would reject them. Messages, tool calls, and ChatGPT's own reasoning are kept, and a LiteLLM model still gets its own reasoning back.
+
+When the catalog adds LiteLLM models for a key, its `ETag` is a digest of OpenAI's ETag and those models, and that key's responses carry the same value in `X-Models-Etag`. Codex keeps its cached catalog while the two match and fetches it again when OpenAI's catalog or the key's models change. A key whose catalog adds no LiteLLM models keeps OpenAI's ETags.
+
+### Restrict native models with an access group
+
+A key with a model allowlist needs a rule for ChatGPT names that OpenAI adds over time. Put the subscription wildcards in an access group under `chatgpt/` names:
+
+```yaml
+model_list:
+  - model_name: chatgpt/gpt-*
+    litellm_params:
+      model: chatgpt/gpt-*
+    model_info:
+      access_groups: ["chatgpt-native"]
+  - model_name: chatgpt/codex-*
+    litellm_params:
+      model: chatgpt/codex-*
+    model_info:
+      access_groups: ["chatgpt-native"]
+```
+
+Then add `chatgpt-native` to the key's models, next to any other model names it should reach. On the native route, a key that holds only the `chatgpt/` form of a name is checked as that form, so a request for `gpt-5.6-sol` is allowed when the key may call `chatgpt/gpt-5.6-sol`. The key can call every ChatGPT model that matches the wildcards, including ones the proxy configuration does not name.
+
+On `/v1` these wildcards match only a request that asks for a `chatgpt/` name, so a request for a plain `gpt-` or `codex-` name still goes where it went before, such as a `*` catch-all deployment. A `/v1` request for a `chatgpt/` name without a ChatGPT login falls back to the proxy's own saved login if it has one, so keep that login unset as described above.
 
 ### Configure Pi
 
@@ -71,7 +107,7 @@ Keep Pi's subscription login as the provider credential. The header value names
 
 Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
 
-The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key must allow that name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
+The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key needs the `chatgpt-native` access group above or that exact name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
 
 The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
 

litellm-rust/crates/router-core/src/error.rs

@@ -284,6 +284,7 @@ impl Error {
     strum::EnumString,
     strum::IntoStaticStr,
     strum::VariantArray,
+    serde_with::SerializeDisplay,
 )]
 pub enum FailureKind {
     #[strum(serialize = "RateLimitError")]
@@ -471,6 +472,17 @@ mod tests {
         assert_eq!(<&'static str>::from(kind), wire);
     }
 
+    #[rstest]
+    #[case::rate_limit(FailureKind::RateLimit, "RateLimitError")]
+    #[case::timeout(FailureKind::Timeout, "Timeout")]
+    #[case::api_connection(FailureKind::ApiConnection, "APIConnectionError")]
+    fn failure_kind_serializes_as_its_exception_name(
+        #[case] kind: FailureKind,
+        #[case] wire: &str,
+    ) {
+        assert_eq!(serde_json::to_value(kind).unwrap(), serde_json::json!(wire));
+    }
+
     #[rstest]
     #[case::empty("")]
     #[case::lowercase("ratelimiterror")]

litellm-rust/crates/router-core/src/host.rs

@@ -342,7 +342,10 @@ pub struct Output {
 }
 
 /// How a provider blocked a response the call otherwise served.
-#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::IntoStaticStr, strum::EnumString)]
+#[derive(
+    Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, strum::IntoStaticStr, strum::EnumString,
+)]
+#[serde(rename_all = "snake_case")]
 #[strum(serialize_all = "snake_case")]
 pub enum Blocked {
     /// A chat response whose first choice was content-filtered (or that has no choices).
@@ -402,8 +405,10 @@ pub enum SelectStage {
     OverrideCheck(RoutingStrategy),
 }
 
-/// Routing events, delivered through [`Host::observe`].
-#[derive(Clone, Debug, PartialEq)]
+/// Routing events, delivered through [`Host::observe`]. They serialize as JSON objects tagged by
+/// `event` (the variant name), durations in seconds.
+#[derive(Clone, Debug, PartialEq, serde::Serialize)]
+#[serde(tag = "event")]
 pub enum RouterEvent {
     /// A pick for `model` (the level's group) begins: its route phase opens.
     PickStarted {
@@ -413,6 +418,7 @@ pub enum RouterEvent {
         model_group: String,
         deployment_id: String,
         /// How long the pick took (`_acompletion`'s `async_get_available_deployment` service event).
+        #[serde(serialize_with = "seconds")]
         pick: Duration,
     },
     Succeeded {
@@ -425,11 +431,13 @@ pub enum RouterEvent {
     },
     CooldownSet {
         deployment_id: String,
+        #[serde(serialize_with = "seconds")]
         cooldown: Duration,
     },
     Retry {
         model_group: String,
         retry: u32,
+        #[serde(serialize_with = "seconds")]
         delay: Duration,
     },
     Fallback {
@@ -584,6 +592,13 @@ impl PickError {
     }
 }
 
+/// A pick's error serializes as its message.
+impl serde::Serialize for PickError {
+    fn serialize<S: serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
+        serializer.collect_str(&self.0)
+    }
+}
+
 impl PartialEq for PickError {
     fn eq(&self, other: &Self) -> bool {
         format!("{:?}", self.0) == format!("{:?}", other.0)
@@ -591,7 +606,8 @@ impl PartialEq for PickError {
 }
 
 /// Which error a [`RouterEvent::RetryLogged`] records.
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq, serde::Serialize)]
+#[serde(tag = "cause", content = "error", rename_all = "snake_case")]
 pub enum RetryCause {
     /// The last attempt's failure.
     Attempt,
@@ -745,3 +761,83 @@ pub trait Host: Send + Sync {
         Ok(Value::Null)
     }
 }
+
+fn seconds<S: serde::Serializer>(duration: &Duration, serializer: S) -> Result<S::Ok, S::Error> {
+    serializer.serialize_f64(duration.as_secs_f64())
+}
+
+#[cfg(test)]
+mod tests {
+    use rstest::rstest;
+    use serde_json::{Value, json};
+
+    use super::*;
+
+    #[rstest]
+    #[case::selected(
+        RouterEvent::Selected {
+            model_group: "gpt-4o".to_owned(),
+            deployment_id: "azure-eastus".to_owned(),
+            pick: Duration::from_millis(1500),
+        },
+        json!({"event": "Selected", "model_group": "gpt-4o", "deployment_id": "azure-eastus", "pick": 1.5})
+    )]
+    #[case::attempt_failed(
+        RouterEvent::AttemptFailed {
+            deployment_id: "openai".to_owned(),
+            kind: FailureKind::RateLimit,
+            status: Some(429),
+        },
+        json!({"event": "AttemptFailed", "deployment_id": "openai", "kind": "RateLimitError", "status": 429})
+    )]
+    #[case::cooldown_set(
+        RouterEvent::CooldownSet { deployment_id: "openai".to_owned(), cooldown: Duration::from_secs(30) },
+        json!({"event": "CooldownSet", "deployment_id": "openai", "cooldown": 30.0})
+    )]
+    #[case::retry(
+        RouterEvent::Retry { model_group: "gpt-4o".to_owned(), retry: 1, delay: Duration::ZERO },
+        json!({"event": "Retry", "model_group": "gpt-4o", "retry": 1, "delay": 0.0})
+    )]
+    #[case::blocked(
+        RouterEvent::Blocked { level: 0, model: "claude".to_owned(), blocked: Blocked::ContentFilter },
+        json!({"event": "Blocked", "level": 0, "model": "claude", "blocked": "content_filter"})
+    )]
+    #[case::retry_logged_after_an_attempt(
+        RouterEvent::RetryLogged {
+            scope: 0,
+            level: 0,
+            model: "gpt-4o".to_owned(),
+            attempted_retries: None,
+            cause: RetryCause::Attempt,
+        },
+        json!({
+            "event": "RetryLogged", "scope": 0, "level": 0, "model": "gpt-4o",
+            "attempted_retries": null, "cause": {"cause": "attempt"},
+        })
+    )]
+    #[case::retry_logged_after_a_pick(
+        RouterEvent::RetryLogged {
+            scope: 1,
+            level: 1,
+            model: "backup".to_owned(),
+            attempted_retries: Some(2),
+            cause: RetryCause::Pick(PickError(Arc::new(Error::UnknownModel { model: "backup".to_owned() }))),
+        },
+        json!({
+            "event": "RetryLogged", "scope": 1, "level": 1, "model": "backup", "attempted_retries": 2,
+            "cause": {"cause": "pick", "error": "no deployment serves model `backup`"},
+        })
+    )]
+    #[case::mock_failed(
+        RouterEvent::MockFailed {
+            mock: MockFailure::RateLimit,
+            level: 0,
+            model: "gpt-4o".to_owned(),
+            num_retries: None,
+        },
+        json!({"event": "MockFailed", "mock": "rate_limit", "level": 0, "model": "gpt-4o", "num_retries": null})
+    )]
+    fn an_event_serializes_tagged_by_its_name(#[case] event: RouterEvent, #[case] expected: Value) {
+        assert_eq!(serde_json::to_value(&event).unwrap(), expected);
+    }
+}

litellm-rust/crates/router-core/src/mock.rs

@@ -7,7 +7,15 @@ use crate::error::{AttemptError, FailureKind};
 
 /// Which mock failure a request asks for.
 #[derive(
-    Clone, Copy, Debug, PartialEq, Eq, Deserialize, strum::IntoStaticStr, strum::EnumString,
+    Clone,
+    Copy,
+    Debug,
+    PartialEq,
+    Eq,
+    Deserialize,
+    serde::Serialize,
+    strum::IntoStaticStr,
+    strum::EnumString,
 )]
 #[serde(rename_all = "snake_case")]
 #[strum(serialize_all = "snake_case")]

litellm-rust/crates/router-core/tests/call_loop.rs

@@ -17,7 +17,7 @@ use litellm_router_core::{
 };
 use parking_lot::Mutex;
 use rstest::{fixture, rstest};
-use serde_json::json;
+use serde_json::{Value, json};
 
 #[derive(Default)]
 struct ScriptedHost {
@@ -1809,3 +1809,55 @@ fn request_for(model: &str) -> RouterRequest {
         ..RouterRequest::default()
     }
 }
+
+#[rstest]
+#[tokio::test]
+async fn every_event_of_a_retried_and_fallen_back_call_serializes_tagged_by_its_name(
+    config: RouterConfig,
+) {
+    let host = Arc::new(ScriptedHost::failing(
+        "primary",
+        vec![failure(FailureKind::RateLimit, 429); 3],
+    ));
+    let router = router(
+        config,
+        json!({"num_retries": 1, "fallbacks": [{"primary": ["backup"]}]}),
+        host.clone(),
+    );
+    router
+        .call(Op::from("acompletion"), request())
+        .await
+        .unwrap();
+    let tags: Value = host
+        .events
+        .lock()
+        .iter()
+        .map(|event| serde_json::to_value(event).unwrap()["event"].clone())
+        .collect();
+    assert_eq!(
+        tags,
+        json!([
+            "RetriesOpened",
+            "PickStarted",
+            "Selected",
+            "AttemptFailed",
+            "CooldownSet",
+            "RetryBudget",
+            "RetryLogged",
+            "Retry",
+            "PickStarted",
+            "Selected",
+            "AttemptFailed",
+            "CooldownSet",
+            "RetryLogged",
+            "FallingBack",
+            "RetryLogged",
+            "Fallback",
+            "RetriesOpened",
+            "PickStarted",
+            "Selected",
+            "Succeeded",
+            "FallbackDone",
+        ])
+    );
+}

litellm/completion_extras/litellm_responses_transformation/transformation.py

@@ -54,6 +54,7 @@ from litellm.llms.base_llm.bridges.completion_transformation import (
     CompletionTransformationBridge,
 )
 from litellm.llms.custom_httpx.sse_lines import split_sse_lines
+from litellm.responses.additional_tools import additional_tools_in_input
 from litellm.responses.litellm_completion_transformation.tool_search import is_anthropic_tool_search_chat_tool
 from litellm.responses.sse_output_recovery import (
     parse_sse_json_chunk,
@@ -426,6 +427,17 @@ def replayed_tool_search_tool_names(request: object) -> tuple[str, ...]:
     return tuple(chain.from_iterable(_tool_definition_name(tool) for tool in replayed_tool_search_tools(request)))
 
 
+def _top_level_namespaces(request: object) -> Iterator[object]:
+    tools: Final = _json_list(_json_field(request, "tools")) or ()
+    return (tool for tool in tools if _json_field(tool, "type") == "namespace")
+
+
+def nested_tool_member_names(request: object) -> tuple[str, ...]:
+    tools: Final = (*_top_level_namespaces(request), *additional_tools_in_input(_json_field(request, "input")))
+    members: Final = chain.from_iterable(_namespace_members(tool) for tool in tools)
+    return tuple(chain.from_iterable(_tool_definition_name(member) for member in members))
+
+
 def _loads_any_tool(message: object, names: frozenset[str]) -> bool:
     loaded_names: Final = chain.from_iterable(_tool_definition_name(tool) for tool in _message_loaded_tools(message))
     return any(name in names for name in loaded_names)

litellm/integrations/arize/_utils.py

@@ -14,6 +14,7 @@ from litellm.litellm_core_utils.redact_messages import (
     should_redact_message_logging,
 )
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.types.utils import CallTypes, StandardLoggingMCPToolCall, StandardLoggingPayload
 
 if TYPE_CHECKING:
@@ -663,12 +664,11 @@ def _to_plain_dict(value):
 
 
 def _get_tool_calls(message) -> list | None:
-    """Return ``message.tool_calls`` only when it's a non-empty list.
-
-    Works for dicts and Pydantic message objects via ``_safe_get``.
+    """Return the message's tool calls, Claude's server tool calls first, or
+    ``None`` when there are none. Works for dicts and Pydantic message objects.
     """
-    tool_calls: Final = _safe_get(message, "tool_calls")
-    return tool_calls if isinstance(tool_calls, list) and tool_calls else None
+    tool_calls: Final = tool_calls_with_server_tool_calls(message)
+    return list(tool_calls) if tool_calls else None
 
 
 class _NormalizedToolCallFunction(TypedDict):

litellm/integrations/datadog/datadog_llm_obs.py

@@ -44,6 +44,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
 from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
 from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -315,7 +316,10 @@ def _to_dd_tool_calls(message: Mapping[str, object]) -> tuple[ToolCall, ...]:
             tool_id=_text_field(tool_call, "id"),
             type=_text_field(tool_call, "type", "function"),
         )
-        for tool_call in (raw_tool_calls if isinstance(raw_tool_calls, list) else ())
+        for tool_call in (
+            *anthropic_server_tool_call_params(message),
+            *(raw_tool_calls if isinstance(raw_tool_calls, list) else ()),
+        )
         if isinstance(tool_call, dict)
         for function in [_mapping_field(tool_call, "function")]
     )

litellm/integrations/helicone.py

@@ -10,6 +10,23 @@ from litellm.integrations.helicone_mock_client import (
     create_mock_helicone_client,
     should_use_helicone_mock,
 )
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
+from litellm.types.utils import ChatCompletionMessageCustomToolCall, ChatCompletionMessageToolCall
+
+
+def _tool_use_block(
+    tool_call: ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall,
+) -> dict[str, object]:
+    match tool_call:
+        case ChatCompletionMessageCustomToolCall(custom=custom):
+            return {"type": "tool_use", "id": tool_call.id, "name": custom.name, "input": custom.input}
+        case _:
+            return {
+                "type": "tool_use",
+                "id": tool_call.id,
+                "name": tool_call.function.name,
+                "input": tool_call.function.arguments,
+            }
 
 
 class HeliconeLogger:
@@ -57,23 +74,8 @@ class HeliconeLogger:
         message = choice["message"]
 
         content = []
-        if "tool_calls" in message and message["tool_calls"]:
-            for tool_call in message["tool_calls"]:
-                function = tool_call.get("function")
-                custom = tool_call.get("custom")
-                if not function and not custom:
-                    continue
-                name, tool_input = (
-                    (function["name"], function["arguments"]) if function else (custom["name"], custom["input"])
-                )
-                content.append(
-                    {
-                        "type": "tool_use",
-                        "id": tool_call["id"],
-                        "name": name,
-                        "input": tool_input,
-                    }
-                )
+        if tool_calls := tool_calls_with_server_tool_calls(message):
+            content = [_tool_use_block(tool_call) for tool_call in tool_calls]
         elif "content" in message and message["content"]:
             content = [{"type": "text", "text": message["content"]}]
 

litellm/integrations/langfuse/langfuse_otel.py

@@ -26,6 +26,7 @@ from litellm.integrations.opentelemetry import (
 from litellm.litellm_core_utils.core_helpers import as_str_mapping
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
 from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.secret_managers.main import str_to_bool
 from litellm.types.integrations.langfuse_otel import (
     LangfuseSpanAttributes,
@@ -605,8 +606,8 @@ def _extract_choices_output(response_obj: _Gettable) -> str | None:
     message: Final = _as_gettable(first_choice.get("message", {}))
     if message is None:
         return None
-    tool_calls: Final = message.get("tool_calls")
-    if isinstance(tool_calls, list) and tool_calls:
+    tool_calls: Final = tool_calls_with_server_tool_calls(message)
+    if tool_calls:
         transformed_tool_calls: Final = [
             entry for tool_call in tool_calls if (entry := _transformed_tool_call(response_obj, tool_call)) is not None
         ]

litellm/integrations/lunary.py

@@ -6,6 +6,7 @@ from datetime import datetime, timezone
 from typing import TYPE_CHECKING, Final
 
 import packaging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 
 if TYPE_CHECKING:
     from openai.types.chat.chat_completion_message_custom_tool_call import Custom
@@ -67,8 +68,8 @@ def parse_messages(input):
         }
 
         # Only add tool_calls and function_call to res if they are set
-        if message.get("tool_calls"):
-            serialized["tool_calls"] = parse_tool_calls(message.get("tool_calls"))
+        if tool_calls := tool_calls_with_server_tool_calls(message):
+            serialized["tool_calls"] = parse_tool_calls(tool_calls)
 
         return serialized
 

litellm/integrations/newrelic/newrelic.py

@@ -56,6 +56,7 @@ from litellm._logging import verbose_logger
 from litellm.integrations.custom_logger import CustomLogger
 from litellm.litellm_core_utils.core_helpers import has_attributes
 from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.types.integrations.base_health_check import IntegrationHealthCheckStatus
 from litellm.types.integrations.newrelic import NewRelicInitParams
 from litellm.types.utils import Message, ModelResponse, StandardLoggingPayload
@@ -479,11 +480,11 @@ class NewRelicLogger(CustomLogger):
         content: Final = message.get("content")
 
         # Handle tool calls
-        if message.get("tool_calls"):
+        if tool_calls := tool_calls_with_server_tool_calls(message):
             try:
-                return json.dumps(message["tool_calls"])
+                return json.dumps([tool_call.model_dump() for tool_call in tool_calls])
             except Exception:
-                return str(message["tool_calls"])
+                return str(tool_calls)
 
         # Handle None or missing content
         if content is None:

litellm/integrations/opentelemetry.py

@@ -42,6 +42,7 @@ from litellm.litellm_core_utils.service_tier_utils import (
     get_requested_service_tier,
     get_served_service_tier,
 )
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params, tool_calls_with_server_tool_calls
 from litellm.secret_managers.main import get_secret_bool, str_to_bool
 from litellm.types.integrations.otel_span_attributes import SpanAttributes
 from litellm.types.services import ServiceLoggerPayload
@@ -2736,9 +2737,8 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
                     for idx, choice in enumerate(response_obj.get("choices")):
                         if choice.get("finish_reason"):
                             message = choice.get("message")
-                            tool_calls = message.get("tool_calls")
-                            if tool_calls:
-                                kv_pairs = OpenTelemetry._tool_calls_kv_pair(tool_calls)
+                            if tool_calls := tool_calls_with_server_tool_calls(message):
+                                kv_pairs = OpenTelemetry._tool_calls_kv_pair(list(tool_calls))
                                 for key, value in kv_pairs.items():
                                     self.safe_set_attribute(
                                         span=span,
@@ -2856,7 +2856,9 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
             transformed_msg = {"role": role, "parts": parts}
             if "id" in msg:
                 transformed_msg["id"] = msg["id"]
-            if "tool_calls" in msg:
+            if server_tool_calls := anthropic_server_tool_call_params(cast(object, msg)):
+                transformed_msg["tool_calls"] = [*server_tool_calls, *(msg.get("tool_calls") or [])]
+            elif "tool_calls" in msg:
                 transformed_msg["tool_calls"] = msg["tool_calls"]
             if "tool_call_id" in msg:
                 transformed_msg["tool_call_id"] = msg["tool_call_id"]

litellm/integrations/otel/mappers/utils.py

@@ -12,6 +12,8 @@ from typing import Final
 
 from litellm.integrations.otel.mappers.base import AttributeMap, AttrValue
 from litellm.integrations.otel.model.payloads import LLMCallSpanData, ToolDefinition
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
 
 
 @dataclass(frozen=True, slots=True)
@@ -141,12 +143,13 @@ def output_messages(data: LLMCallSpanData) -> list:
 
 
 def message_tool_calls(message: object) -> tuple[MessageToolCall, ...]:
+    tool_call_values: Final = (
+        *anthropic_server_tool_call_params(message),
+        *parts_of(message_field(message, "tool_calls")),
+    )
     if not isinstance(message, dict):
         return ()
-    tool_calls: Final = message.get("tool_calls")
-    if not isinstance(tool_calls, (list, tuple)) or not tool_calls:
-        return ()
-    return tuple(tool_call for value in tool_calls if (tool_call := _message_tool_call(value)) is not None)
+    return tuple(tool_call for value in tool_call_values if (tool_call := _message_tool_call(value)) is not None)
 
 
 def _message_tool_call(value: object) -> MessageToolCall | None:

litellm/integrations/otel/model/payloads.py

@@ -10,6 +10,8 @@ from types import MappingProxyType
 from typing import TYPE_CHECKING, ClassVar, Final, Literal, Protocol, cast
 from urllib.parse import urlsplit
 
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from pydantic import TypeAdapter, ValidationError
 from typing_extensions import ReadOnly, TypedDict
 
 from litellm.integrations.otel.model.metadata import (
@@ -36,6 +38,7 @@ from litellm.integrations.otel.model.utils import (
     as_str_tuple,
 )
 from litellm.integrations.otel.routing import RoutingAttributeValue, routing_decision_attributes
+from litellm.types.llms.anthropic import claude_hosted_web_tool_call
 
 # ``RequestIdentity`` and the request-metadata translation now live in
 # :mod:`metadata`; re-exported here so existing ``model.payloads`` imports keep
@@ -809,6 +812,7 @@ class _Choice(TypedDict):
 
 
 _RESPONSES_TOOL_CALL_TYPES: Final = frozenset({"function_call", "custom_tool_call"})
+_WEB_SEARCH_CALL: Final = TypeAdapter(ResponseFunctionWebSearch)
 
 
 def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
@@ -816,9 +820,13 @@ def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
     items: Final = _dicts(response.get("output"))
     messages: Final = tuple(item for item in items if item.get("type") == "message")
     parts: Final = tuple(part for item in messages for part in _dicts(item.get("content")))
-    tool_calls: Final = tuple(
+    client_tool_calls: Final = tuple(
         _responses_tool_call(item) for item in items if item.get("type") in _RESPONSES_TOOL_CALL_TYPES
     )
+    tool_calls: Final = (
+        *(call for item in items if (call := _claude_web_search_tool_call(item)) is not None),
+        *client_tool_calls,
+    )
     if not messages and not tool_calls:
         return ()
     message: Final[_AssistantMessage] = {
@@ -827,7 +835,10 @@ def _responses_choices(response: Mapping[str, object]) -> tuple[_Choice, ...]:
         "refusal": _responses_parts_text(parts, "refusal", "refusal"),
         "tool_calls": tool_calls or None,
     }
-    choice: Final[_Choice] = {"message": message, "finish_reason": _responses_finish_reason(response, bool(tool_calls))}
+    choice: Final[_Choice] = {
+        "message": message,
+        "finish_reason": _responses_finish_reason(response, bool(client_tool_calls)),
+    }
     return (choice,)
 
 
@@ -981,6 +992,20 @@ def _responses_tool_call(item: Mapping[str, object]) -> _ToolCall:
     return tool_call
 
 
+def _claude_web_search_tool_call(item: Mapping[str, object]) -> _ToolCall | None:
+    if item.get("type") != "web_search_call":
+        return None
+    try:
+        search: Final = _WEB_SEARCH_CALL.validate_python(item)
+    except ValidationError:
+        return None
+    claude_call: Final = claude_hosted_web_tool_call(search)
+    if claude_call is None:
+        return None
+    tool_id, name, arguments = claude_call
+    return {"id": tool_id, "type": "function", "function": {"name": name, "arguments": arguments}}
+
+
 def _responses_finish_reason(response: Mapping[str, object], has_tool_calls: bool) -> str | None:
     status: Final = as_str(response.get("status"))
     if status == "completed":

litellm/integrations/shadow_eval_logger.py

@@ -39,6 +39,7 @@ from litellm.litellm_core_utils.llm_judge import (
     parse_json_verdict,
 )
 from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.llms.base_llm.base_utils import type_to_response_format_param
 from litellm.redis.stores.spend_counters import (
     shadow_eval_counter,
@@ -205,7 +206,7 @@ def _chat_final_text(response_obj: object) -> str:
     if read is None:
         return ""
     prose: Final = extract_text_from_content(read("content"))
-    if not (read("tool_calls") or read("function_call")):
+    if not _tool_calls_list(read):
         return prose
     serialized: Final = _serialize_tool_calls(read)
     return f"{prose} {serialized}".strip() if prose else serialized
@@ -221,8 +222,9 @@ _RESPONSES_TOOL_CALL_TYPES: Final = frozenset(("function_call", "custom_tool_cal
 
 
 def _tool_calls_list(read: Callable[[str], object]) -> tuple[object, ...]:
-    calls: Final = read("tool_calls")
-    listed: Final = tuple(calls) if isinstance(calls, Sequence) and not isinstance(calls, str) else ()
+    listed: Final = tool_calls_with_server_tool_calls(
+        {"tool_calls": read("tool_calls"), "provider_specific_fields": read("provider_specific_fields")}
+    )
     single: Final = read("function_call")
     return listed if listed else ((single,) if single is not None else ())
 

litellm/litellm_core_utils/get_llm_provider_logic.py

@@ -12,6 +12,7 @@ from litellm.constants import (
 from litellm.litellm_core_utils.fallback_generalizations import (
     match_routing_generalization,
 )
+from litellm.litellm_core_utils.model_latest_aliases import resolve_model_latest_alias
 from litellm.llms.openai_like.json_loader import JSONProviderRegistry
 from litellm.llms.opencode.common_utils import is_opencode_api_base
 from litellm.secret_managers.main import get_secret, get_secret_str
@@ -189,6 +190,23 @@ def get_llm_provider(
     api_base: str | None = None,
     api_key: str | None = None,
     litellm_params: GenericLiteLLMParams | None = None,
+) -> tuple[str, str, str | None, str | None]:
+    resolved_model, provider, dynamic_api_key, resolved_api_base = _get_llm_provider(
+        model, custom_llm_provider, api_base, api_key, litellm_params
+    )
+    explicit_provider: Final = custom_llm_provider or (litellm_params.custom_llm_provider if litellm_params else None)
+    target: Final = (
+        resolve_model_latest_alias(resolved_model, provider) if explicit_provider or "/" in model else resolved_model
+    )
+    return target, provider, dynamic_api_key, resolved_api_base
+
+
+def _get_llm_provider(
+    model: str,
+    custom_llm_provider: str | None = None,
+    api_base: str | None = None,
+    api_key: str | None = None,
+    litellm_params: GenericLiteLLMParams | None = None,
 ) -> tuple[str, str, str | None, str | None]:
     """
     Returns the provider for a given model name - e.g. 'azure/chatgpt-v-2' -> 'azure'

litellm/litellm_core_utils/llm_cost_calc/utils.py

@@ -1464,7 +1464,7 @@ def get_fedramp_uplift_multiplier(model_info: ModelInfo, data_residency: str | N
         return 1.0
 
 
-def get_vertex_regional_endpoint_uplift(model_info: ModelInfo, vertex_location: str | None) -> float:
+def get_vertex_regional_endpoint_uplift(model_info: Mapping[str, object], vertex_location: str | None) -> float:
     """
     Resolve the per-model uplift multiplier for Vertex AI non-global (regional and
     multi-region) endpoints.

litellm/litellm_core_utils/model_latest_aliases.py

@@ -0,0 +1,33 @@
+from collections.abc import Mapping
+from functools import cache
+from importlib.resources import files
+from types import MappingProxyType
+from typing import Final
+
+from pydantic import TypeAdapter
+
+_ALIASES_ADAPTER: Final = TypeAdapter(dict[str, dict[str, str]])
+
+
+@cache
+def get_model_latest_aliases() -> Mapping[str, Mapping[str, str]]:
+    aliases: Final = _ALIASES_ADAPTER.validate_json(files("litellm").joinpath("model_latest_aliases.json").read_bytes())
+    return MappingProxyType({provider: MappingProxyType(models) for provider, models in aliases.items()})
+
+
+def resolve_model_latest_alias(model: str, provider: str) -> str:
+    aliases: Final = get_model_latest_aliases().get(provider)
+    return aliases.get(model, model) if aliases is not None else model
+
+
+def expand_model_latest_alias(model: str) -> tuple[str, ...]:
+    provider, separator, name = model.partition("/")
+    target: Final = resolve_model_latest_alias(name, provider) if separator else name
+    return (model, f"{provider}/{target}") if separator and target != name else (model,)
+
+
+def is_model_latest_alias(model: str, custom_llm_provider: str | None = None) -> bool:
+    prefix, separator, name = model.partition("/")
+    provider: Final = custom_llm_provider or prefix
+    candidate: Final = name if separator and prefix == provider else model
+    return bool(custom_llm_provider or separator) and resolve_model_latest_alias(candidate, provider) != candidate

litellm/litellm_core_utils/prompt_templates/common_utils.py

@@ -604,6 +604,8 @@ def get_content_from_model_response(response: ModelResponse | dict) -> str:
     else:
         new_response = response
 
+    from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
+
     content = ""
 
     for choice in new_response.choices:
@@ -611,9 +613,8 @@ def get_content_from_model_response(response: ModelResponse | dict) -> str:
             content += choice.message.content if choice.message.content else ""
             if choice.message.function_call:
                 content += choice.message.function_call.model_dump_json()
-            if choice.message.tool_calls:
-                for tc in choice.message.tool_calls:
-                    content += tc.model_dump_json()
+            for tc in tool_calls_with_server_tool_calls(choice.message):
+                content += tc.model_dump_json()
         elif isinstance(choice, StreamingChoices):
             content += getattr(choice, "delta", {}).get("content", "") or ""
     return content

litellm/litellm_core_utils/prompt_templates/factory.py

@@ -2012,12 +2012,39 @@ def _tool_call_ids_with_a_kept_server_result(message: object) -> frozenset[str]:
         return frozenset()
     return frozenset(
         tool_call_id
-        for tool_call in parts_of(message_field(message, "tool_calls"))
+        for tool_call in (
+            *parts_of(message_field(message, "tool_calls")),
+            *parts_of(message_field(provider_specific_fields, "server_tool_calls")),
+        )
         if isinstance(tool_call_id := message_field(tool_call, "id"), str)
         and tool_call_is_rebuilt_as_server_tool_use(tool_call_id, provider_specific_fields)
     )
 
 
+def _assistant_tool_calls_with_server_tool_calls(
+    message: ChatCompletionAssistantMessage,
+) -> Sequence[ChatCompletionAssistantToolCall] | None:
+    from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+    tool_calls: Final = message.get("tool_calls")
+    server_tool_calls: Final = anthropic_server_tool_call_params(message)
+    if not server_tool_calls:
+        return tool_calls
+    client_tool_call_ids: Final = frozenset(tool_call["id"] for tool_call in tool_calls or ())
+    return (
+        *(tool_call for tool_call in server_tool_calls if tool_call["id"] not in client_tool_call_ids),
+        *(tool_calls or ()),
+    )
+
+
+def _server_tool_call_ids(message: object) -> frozenset[str]:
+    from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+    return frozenset(
+        tool_call["id"] for tool_call in anthropic_server_tool_call_params(message) if tool_call["id"] is not None
+    )
+
+
 def _server_tool_answers_replaced_after(messages: Sequence[object], index: int) -> frozenset[str]:
     kept_server_result_ids: Final = _tool_call_ids_with_a_kept_server_result(messages[index])
     if not kept_server_result_ids:
@@ -2102,6 +2129,12 @@ def _unrun_server_tool_use_ids(blocks: Sequence[Mapping[str, object]]) -> frozen
     )
 
 
+def _paused_server_tool_use_ids(blocks: Sequence[Mapping[str, object]]) -> frozenset[object]:
+    if any(block.get("type") == "tool_use" for block in blocks):
+        return frozenset()
+    return _unrun_server_tool_use_ids(blocks)
+
+
 def _last_block_id_among(blocks: Sequence[Mapping[str, object]], ids: frozenset[object]) -> object:
     return next(block_id for block in reversed(blocks) if (block_id := _block_id(block)) in ids)
 
@@ -2115,7 +2148,7 @@ def _paused_turn_replay(
     tuple[_AnthropicToolInvokeItem, ...],
     tuple[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, ...],
 ]:
-    unrun_ids: Final = _unrun_server_tool_use_ids(tool_invokes)
+    unrun_ids: Final = _paused_server_tool_use_ids(tool_invokes)
     if not unrun_ids:
         return tuple(tool_invokes), tuple(thinking_blocks)
     dropped_ids: Final = (
@@ -2125,10 +2158,11 @@ def _paused_turn_replay(
 
 
 def convert_to_anthropic_tool_invoke(
-    tool_calls: list[ChatCompletionAssistantToolCall],
+    tool_calls: Sequence[ChatCompletionAssistantToolCall],
     web_search_results: Sequence[object] | None = None,
     tool_results: Sequence[object] | None = None,
     answered_tool_ids: Container[str] = frozenset(),
+    server_tool_call_ids: Container[str] = frozenset(),
 ) -> list[_AnthropicToolInvokeItem]:
     """
     OpenAI tool invokes:
@@ -2206,14 +2240,11 @@ def convert_to_anthropic_tool_invoke(
             anthropic_tool_invoke.append(server_tool_result)
         elif is_anthropic_tool_search_use(tool_id, tool_name) and tool_id not in answered_tool_ids:
             continue
-        elif is_anthropic_server_tool_use(tool_id) and tool_id not in answered_tool_ids:
+        elif (
+            tool_id in server_tool_call_ids or is_anthropic_server_tool_use(tool_id)
+        ) and tool_id not in answered_tool_ids:
             anthropic_tool_invoke.append(
-                {
-                    "type": "server_tool_use",
-                    "id": tool_id,
-                    "name": tool_name,
-                    "input": tool_input,
-                }
+                {"type": "server_tool_use", "id": tool_id, "name": tool_name, "input": tool_input}
             )
         else:
             sanitized_tool_id = sanitize_anthropic_tool_use_id(tool_id)
@@ -3174,7 +3205,7 @@ def anthropic_messages_pt(
             # If so, we need to interleave thinking blocks with tool call groups
             # to preserve the original content block ordering.
             # Fixes: https://github.com/BerriAI/litellm/issues/23047
-            assistant_tool_calls = assistant_content_block.get("tool_calls")
+            assistant_tool_calls = _assistant_tool_calls_with_server_tool_calls(assistant_content_block)
             _has_server_tool_calls = False
             if assistant_tool_calls is not None:
                 for _tc in assistant_tool_calls:
@@ -3198,11 +3229,12 @@ def anthropic_messages_pt(
                     web_search_results=_provider_specific_fields.get("web_search_results"),
                     tool_results=_provider_specific_fields.get("tool_results"),
                     answered_tool_ids=client_answered_tool_ids,
+                    server_tool_call_ids=_server_tool_call_ids(assistant_content_block),
                 )
                 if assistant_tool_calls is not None
                 else ()
             )
-            turn_paused = bool(_unrun_server_tool_use_ids(tool_invoke_results))
+            turn_paused = bool(_paused_server_tool_use_ids(tool_invoke_results))
             replayed_tool_invokes, replayed_thinking_blocks = _paused_turn_replay(
                 tool_invoke_results, thinking_blocks or (), is_final_message=msg_i == len(messages) - 1
             )
@@ -5034,11 +5066,15 @@ class BedrockConverseMessagesProcessor:
         drop_params: bool = False,
         model_info: Mapping[str, object] | None = None,
     ) -> list[BedrockMessageBlock]:
-        from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
+        from litellm.llms.bedrock.common_utils import (
+            bedrock_converse_accepts_redacted_thinking,
+            bedrock_converse_rejects_signed_thinking,
+        )
 
         contents: list[BedrockMessageBlock] = []
         msg_i = 0
         to_claude: Final = bedrock_converse_accepts_redacted_thinking(model, base_model)
+        drop_signed_thinking: Final = bedrock_converse_rejects_signed_thinking(model, base_model)
 
         messages = BedrockConverseMessagesProcessor._initial_message_setup(
             messages, model, llm_provider, user_continue_message
@@ -5194,7 +5230,9 @@ class BedrockConverseMessagesProcessor:
                 thinking_blocks = assistant_message_block.get("thinking_blocks")
                 converted_thinking_blocks = (
                     BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
-                        _thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
+                        _thinking_in_replay_order(_assistant_content, thinking_blocks),
+                        to_claude=to_claude,
+                        drop_signed=drop_signed_thinking,
                     )
                 )
                 assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
@@ -5226,8 +5264,8 @@ class BedrockConverseMessagesProcessor:
                             deployment_model=deployment_model,
                             model_info=model_info,
                         )
-                        if _cache_point_block is not None and not (
-                            drop_params and _as_thinking_block(element) is not None
+                        if _cache_point_block is not None and BedrockConverseMessagesProcessor.keeps_cache_point(
+                            element, drop_params=drop_params, drop_signed=drop_signed_thinking
                         ):
                             assistants_parts.append(_cache_point_block)
                     assistant_content.extend(assistants_parts)
@@ -5286,6 +5324,7 @@ class BedrockConverseMessagesProcessor:
         thinking_blocks: Sequence[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
         *,
         to_claude: bool = False,
+        drop_signed: bool = False,
     ) -> list[BedrockContentBlock]:
         return [
             BedrockContentBlock(
@@ -5295,8 +5334,27 @@ class BedrockConverseMessagesProcessor:
             )
             for thinking_block in thinking_blocks
             if not (to_claude and is_encrypted_reasoning_block(thinking_block))
+            and not (drop_signed and BedrockConverseMessagesProcessor._is_signed(thinking_block))
         ]
 
+    @staticmethod
+    def keeps_cache_point(element: object, *, drop_params: bool, drop_signed: bool) -> bool:
+        match _as_thinking_block(element):
+            case None:
+                return True
+            case thinking_block:
+                return not (
+                    drop_params or (drop_signed and BedrockConverseMessagesProcessor._is_signed(thinking_block))
+                )
+
+    @staticmethod
+    def _is_signed(thinking_block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock) -> bool:
+        match thinking_block:
+            case {"signature": signature} if signature:
+                return True
+            case _:
+                return False
+
     @staticmethod
     def _to_reasoning_content_block(
         thinking_block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
@@ -5455,11 +5513,15 @@ def _bedrock_converse_messages_pt(
     - Please ensure that function response turn comes immediately after a function call turn
     - Conversation blocks and tool result blocks cannot be provided in the same turn. Issue: https://github.com/BerriAI/litellm/issues/6053
     """
-    from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
+    from litellm.llms.bedrock.common_utils import (
+        bedrock_converse_accepts_redacted_thinking,
+        bedrock_converse_rejects_signed_thinking,
+    )
 
     contents: list[BedrockMessageBlock] = []
     msg_i = 0
     to_claude: Final = bedrock_converse_accepts_redacted_thinking(model, base_model)
+    drop_signed_thinking: Final = bedrock_converse_rejects_signed_thinking(model, base_model)
 
     messages = BedrockConverseMessagesProcessor._initial_message_setup(
         messages, model, llm_provider, user_continue_message
@@ -5616,7 +5678,9 @@ def _bedrock_converse_messages_pt(
             thinking_blocks = assistant_message_block.get("thinking_blocks")
             converted_thinking_blocks = (
                 BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
-                    _thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
+                    _thinking_in_replay_order(_assistant_content, thinking_blocks),
+                    to_claude=to_claude,
+                    drop_signed=drop_signed_thinking,
                 )
             )
             assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
@@ -5649,8 +5713,8 @@ def _bedrock_converse_messages_pt(
                             deployment_model=deployment_model,
                             model_info=model_info,
                         )
-                        if _cache_point_block is not None and not (
-                            drop_params and _as_thinking_block(element) is not None
+                        if _cache_point_block is not None and BedrockConverseMessagesProcessor.keeps_cache_point(
+                            element, drop_params=drop_params, drop_signed=drop_signed_thinking
                         ):
                             assistants_parts.append(_cache_point_block)
                 assistant_content.extend(assistants_parts)

litellm/litellm_core_utils/redact_messages.py

@@ -14,7 +14,7 @@ from collections.abc import Iterable, Mapping, Sequence
 from dataclasses import replace
 from typing import TYPE_CHECKING, Any, Final
 
-from pydantic import JsonValue, TypeAdapter, ValidationError
+from pydantic import BaseModel, JsonValue, TypeAdapter, ValidationError
 
 import litellm
 from litellm.constants import REDACTED_BY_LITELLM
@@ -301,9 +301,63 @@ def _redacted_provider_specific_fields(fields: Mapping[str, object]) -> Mapping[
             )
             if key in fields
         },
+        **(
+            {"server_tool_calls": _redacted_tool_call_dicts(fields["server_tool_calls"])}
+            if "server_tool_calls" in fields
+            else {}
+        ),
+        **(
+            {"web_search_calls": _redacted_web_search_calls(fields["web_search_calls"])}
+            if "web_search_calls" in fields
+            else {}
+        ),
     }
 
 
+def _redacted_tool_call_dict(call: object) -> object:
+    if (mapping := _as_mapping(call)) is None or (function := _as_mapping(mapping.get("function"))) is None:
+        return call
+    return {**mapping, "function": {**function, "arguments": REDACTED_BY_LITELLM}}
+
+
+def _redacted_tool_call_dicts(calls: object) -> object:
+    return [_redacted_tool_call_dict(call) for call in calls] if isinstance(calls, list) else calls
+
+
+_WEB_ACTION_TEXT_FIELDS: Final = ("query", "url", "pattern")
+_WEB_ACTION_LIST_FIELDS: Final = ("queries", "sources")
+
+
+def _web_action_redactions(action: Mapping[str, object]) -> Mapping[str, object]:
+    return {
+        **{key: REDACTED_BY_LITELLM for key in _WEB_ACTION_TEXT_FIELDS if action.get(key) is not None},
+        **{key: None for key in _WEB_ACTION_LIST_FIELDS if action.get(key) is not None},
+    }
+
+
+def _redacted_web_search_action(action: object) -> object:
+    if isinstance(action, BaseModel):
+        return action.model_copy(update=dict(_web_action_redactions(action.model_dump())))
+    mapping: Final = _as_mapping(action)
+    return action if mapping is None else {**mapping, **_web_action_redactions(mapping)}
+
+
+def _redacted_web_search_call(item: object) -> object:
+    if isinstance(item, BaseModel):
+        action: Final = getattr(item, "action", None)
+        return item if action is None else item.model_copy(update={"action": _redacted_web_search_action(action)})
+    mapping: Final = _as_mapping(item)
+    if mapping is None or mapping.get("action") is None:
+        return item
+    return {**mapping, "action": _redacted_web_search_action(mapping["action"])}
+
+
+def _redacted_web_search_calls(calls: object) -> object:
+    if (mapping := _as_mapping(calls)) is not None:
+        return {key: _redacted_web_search_call(call) for key, call in mapping.items()}
+    return [_redacted_web_search_call(call) for call in calls] if isinstance(calls, list) else calls
+
+
 def _redact_field_when_present(item: object, field: str) -> None:
     if getattr(item, field, None) is not None:
         setattr(item, field, REDACTED_BY_LITELLM)
@@ -335,6 +389,8 @@ def _redact_responses_api_output(output_items):
             output_item.arguments = REDACTED_BY_LITELLM
         if output_type == "custom_tool_call" and hasattr(output_item, "input"):
             output_item.input = REDACTED_BY_LITELLM
+        if output_type == "web_search_call" and getattr(output_item, "action", None) is not None:
+            output_item.action = _redacted_web_search_action(output_item.action)
 
 
 _RESPONSE_ITEMS: Final = TypeAdapter(tuple[object, ...])
@@ -389,6 +445,11 @@ def _redacted_responses_api_output_item(item: object, redacted_str: str) -> obje
             if item_mapping.get("type") == "custom_tool_call" and "input" in item_mapping
             else {}
         ),
+        **(
+            {"action": _redacted_web_search_action(item_mapping["action"])}
+            if item_mapping.get("type") == "web_search_call" and item_mapping.get("action") is not None
+            else {}
+        ),
     }
 
 

litellm/llms/anthropic/chat/guardrail_translation/handler.py

@@ -16,7 +16,7 @@ import json
 from collections.abc import Iterable, Mapping, MutableSequence, Sequence
 from copy import deepcopy
 from dataclasses import dataclass
-from itertools import chain, repeat
+from itertools import chain, filterfalse, repeat
 from types import MappingProxyType
 from typing import TYPE_CHECKING, Any, Final, Protocol, TypeVar, cast, overload, runtime_checkable
 
@@ -26,6 +26,7 @@ from typing_extensions import ReadOnly, TypedDict, assert_never
 from litellm._logging import verbose_proxy_logger
 from litellm.litellm_core_utils.core_helpers import as_str_mapping, iter_items
 from litellm.llms.anthropic.chat.transformation import AnthropicConfig
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_calls
 from litellm.llms.anthropic.messages.adapters.transformation import (
     OPENAI_MAX_TOOL_NAME_LENGTH,
     LiteLLMAnthropicMessagesAdapter,
@@ -206,17 +207,15 @@ class _WebSearchString:
     value: str
 
 
-def _streamed_web_search_result_blocks(response: ModelResponse) -> tuple[Mapping[str, object], ...]:
-    choice: Final = response.choices[0] if response.choices else None
-    fields: Final = as_str_mapping(choice.message.provider_specific_fields) if isinstance(choice, Choices) else None
-    blocks: Final = (
-        (as_str_mapping(item) for item in iter_items(fields.get("web_search_results")))
-        if fields is not None and isinstance(fields.get("web_search_results"), list)
-        else ()
-    )
-    return tuple(
-        deepcopy(block) for block in blocks if block is not None and block.get("type") == "web_search_tool_result"
-    )
+def _legacy_hook_block(block: object) -> object:
+    if not isinstance(block, Mapping):
+        return block
+    fields: Final = _as_str_mapping(block)
+    return {**fields, "type": "tool_use"} if fields.get("type") == "server_tool_use" else fields
+
+
+def _is_web_search_result_block(block: object) -> bool:
+    return isinstance(block, Mapping) and _as_str_mapping(block).get("type") == "web_search_tool_result"
 
 
 def _web_search_results(block: Mapping[str, object]) -> Iterable[object]:
@@ -519,6 +518,13 @@ def _rewritten_event(event: Mapping[str, object], rewrite_event: _SSEEventRewrit
     return {**event, rewrite.section: {**section, rewrite.field: rewrite.value}}
 
 
+def _in_stream_block_order(
+    tool_calls: Sequence[ChatCompletionMessageToolCall], blocks: Sequence[tuple[int, Mapping[str, object]]]
+) -> tuple[ChatCompletionMessageToolCall, ...]:
+    positions: Final = MappingProxyType({block.get("id"): position for position, (_, block) in enumerate(blocks)})
+    return tuple(sorted(tool_calls, key=lambda tool_call: positions.get(tool_call.id, len(positions))))
+
+
 def _tool_call_shapes(tool_calls: Sequence[object]) -> tuple[_ToolCallShape, ...]:
     """The guardrail-visible shape of each tool call, whether the guardrail handed
     back the ``ChatCompletionMessageToolCall`` objects it was given or plain dicts."""
@@ -586,11 +592,14 @@ class AnthropicMessagesHandler(BaseTranslation):
         if not isinstance(response, ModelResponse):
             return response
         translated: Final = self.adapter.translate_openai_response_to_anthropic(response)
-        result_blocks: Final = _streamed_web_search_result_blocks(response)
-        if not result_blocks:
-            return translated
-        content: Final = translated.get("content") or ()
-        return {**translated, "content": [*result_blocks, *content]}
+        blocks: Final = tuple(map(_legacy_hook_block, translated.get("content") or ()))
+        return {
+            **translated,
+            "content": [
+                *filter(_is_web_search_result_block, blocks),
+                *filterfalse(_is_web_search_result_block, blocks),
+            ],
+        }
 
     @staticmethod
     def _build_streaming_usage_response(
@@ -1570,15 +1579,18 @@ class AnthropicMessagesHandler(BaseTranslation):
             if built_response is not None and hasattr(built_response, "choices") and built_response.choices:
                 model_response: Final = cast(ModelResponse, built_response)
                 first_choice: Final = cast(Choices, model_response.choices[0])
-                tool_calls_list: Final = cast(
-                    list[ChatCompletionMessageToolCall] | None,
-                    first_choice.message.tool_calls,
+                tool_call_blocks: Final = self._streamed_tool_call_blocks(responses_so_far)
+                tool_calls_list: Final = _in_stream_block_order(
+                    (
+                        *anthropic_server_tool_calls(first_choice.message),
+                        *cast(list[ChatCompletionMessageToolCall], first_choice.message.tool_calls or []),
+                    ),
+                    tool_call_blocks,
                 )
                 string_so_far = first_choice.message.content
                 pre_guardrail_tool_calls: Final = _tool_call_shapes(tool_calls_list or ())
                 guardrail_inputs: Final = GenericGuardrailAPIInputs()
                 leading_texts: Final = (string_so_far,) if isinstance(string_so_far, str) and string_so_far else ()
-                tool_call_blocks: Final = self._streamed_tool_call_blocks(responses_so_far)
                 streamed_query_strings: Final = _streamed_query_strings(tool_call_blocks, pre_guardrail_tool_calls)
                 streamed_result_strings: Final = self._streamed_web_search_result_strings(responses_so_far)
                 streamed_web_search_strings: Final = (*streamed_result_strings, *streamed_query_strings)
@@ -1586,7 +1598,7 @@ class AnthropicMessagesHandler(BaseTranslation):
                 if texts_to_check:
                     guardrail_inputs["texts"] = texts_to_check
                 if tool_calls_list:
-                    guardrail_inputs["tool_calls"] = tool_calls_list
+                    guardrail_inputs["tool_calls"] = list(tool_calls_list)
 
                 try:
                     prepared_request_data = self._prepare_request_data(

litellm/llms/anthropic/chat/handler.py

@@ -54,6 +54,7 @@ from litellm.types.llms.openai import (
     ChatCompletionToolCallFunctionChunk,
 )
 from litellm.types.responses.main import (
+    HOSTED_WEB_TOOL_NAMES,
     OutputCodeInterpreterCall,
     build_code_interpreter_log_outputs,
     build_web_search_call,
@@ -761,6 +762,8 @@ class ModelResponseIterator:
         self._server_tool_inputs: dict[str, object] = {}
         self.tool_results: list[dict[str, Any]] = []
         self._current_server_tool_id: str | None = None
+        self._current_server_tool_name: str = ""
+        self._server_tool_calls: tuple[ChatCompletionToolCallChunk, ...] = ()
         self._container_id: str | None = None
 
     @property
@@ -836,10 +839,9 @@ class ModelResponseIterator:
             )
         self.content_blocks.append(content_block)
         if "partial_json" in content_block["delta"]:
-            # Only emit tool calls if we're in a tool_use or server_tool_use block
             # web_search_tool_result blocks also have input_json_delta but should not be treated as tool calls
             # See: https://github.com/BerriAI/litellm/issues/17254
-            if self.current_content_block_type in ("tool_use", "server_tool_use"):
+            if self.current_content_block_type == "tool_use":
                 tool_use = cast(
                     ChatCompletionToolCallChunk,
                     {
@@ -938,7 +940,7 @@ class ModelResponseIterator:
     def _web_search_call_snapshot(self) -> dict[str, object]:
         return dict(self._web_search_calls)
 
-    def _refresh_pending_web_search_call(self, tool_use_id: str) -> bool:
+    def _refresh_pending_web_search_call(self, tool_use_id: str, tool_name: str) -> bool:
         if tool_use_id not in self._web_search_calls:
             return False
         self._web_search_calls[tool_use_id] = build_web_search_call(
@@ -946,10 +948,11 @@ class ModelResponseIterator:
             self._server_tool_inputs.get(tool_use_id, {}),
             {"content": []},
             status="in_progress",
+            tool_name=tool_name,
         )
         return True
 
-    def _complete_web_search_call(self, result: dict[str, object]) -> None:
+    def _complete_web_search_call(self, result: dict[str, object], tool_name: str) -> None:
         tool_use_id: Final = result.get("tool_use_id")
         if not isinstance(tool_use_id, str) or tool_use_id not in self._web_search_calls:
             return
@@ -957,6 +960,7 @@ class ModelResponseIterator:
             tool_id=tool_use_id,
             tool_input=self._server_tool_inputs.get(tool_use_id, {}),
             result=result,
+            tool_name=tool_name,
         )
 
     def _build_code_interpreter_results(self) -> list:
@@ -1036,7 +1040,6 @@ class ModelResponseIterator:
                     content_block_start["content_block"]["type"] == "tool_use"
                     or content_block_start["content_block"]["type"] == "server_tool_use"
                 ):
-                    self.tool_index += 1
                     # Reverse-map the (sanitized) tool name back to the
                     # caller's original. No-op when the map is empty.
                     _stream_tool_name = content_block_start["content_block"]["name"]
@@ -1045,35 +1048,39 @@ class ModelResponseIterator:
                     # Use empty string for arguments in content_block_start - actual arguments
                     # come in subsequent content_block_delta chunks and get accumulated.
                     # Using str(input) here would prepend '{}' causing invalid JSON accumulation.
-                    tool_use = ChatCompletionToolCallChunk(
-                        id=content_block_start["content_block"]["id"],
-                        type="function",
-                        function=ChatCompletionToolCallFunctionChunk(
-                            name=_stream_tool_name,
-                            arguments="",
-                        ),
-                        index=self.tool_index,
-                    )
+                    if content_block_start["content_block"]["type"] == "tool_use":
+                        self.tool_index += 1
+                        tool_use = ChatCompletionToolCallChunk(
+                            id=content_block_start["content_block"]["id"],
+                            type="function",
+                            function=ChatCompletionToolCallFunctionChunk(
+                                name=_stream_tool_name,
+                                arguments="",
+                            ),
+                            index=self.tool_index,
+                        )
                     # Track server tool use inputs for code_interpreter_results.
                     # The initial input in content_block_start is typically {}
                     # for streaming; the full input arrives via input_json_delta
                     # and is assembled at content_block_stop.
                     if content_block_start["content_block"]["type"] == "server_tool_use":
                         self._current_server_tool_id = content_block_start["content_block"]["id"]
+                        self._current_server_tool_name = _stream_tool_name
                         tool_input: Final = content_block_start["content_block"].get("input", {})
                         self._server_tool_inputs[self._current_server_tool_id] = tool_input
-                        if _stream_tool_name == "web_search":
+                        if _stream_tool_name in HOSTED_WEB_TOOL_NAMES:
                             self._web_search_calls[self._current_server_tool_id] = build_web_search_call(
                                 self._current_server_tool_id,
                                 tool_input,
                                 {"content": []},
                                 status="in_progress",
+                                tool_name=_stream_tool_name,
                             )
                             provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
                     # Include caller information if present (for programmatic tool calling)
                     if "caller" in content_block_start["content_block"]:
                         caller_data: Final = content_block_start["content_block"]["caller"]
-                        if caller_data:
+                        if caller_data and tool_use is not None:
                             tool_use["caller"] = cast(dict[str, Any], caller_data)
                 elif content_block_start["content_block"]["type"] == "redacted_thinking":
                     (
@@ -1104,7 +1111,7 @@ class ModelResponseIterator:
                         # The full content comes in content_block_start, not in deltas
                         # See: https://github.com/BerriAI/litellm/issues/17737
                         self.web_search_results.append(content_block_start["content_block"])
-                        self._complete_web_search_call(content_block_start["content_block"])
+                        self._complete_web_search_call(content_block_start["content_block"], "web_search")
                         provider_specific_fields["web_search_results"] = self.web_search_results
                         provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
                     elif content_type == "web_fetch_tool_result":
@@ -1112,7 +1119,9 @@ class ModelResponseIterator:
                         # The full content comes in content_block_start, not in deltas
                         # Fixes: https://github.com/BerriAI/litellm/issues/18137
                         self.web_search_results.append(content_block_start["content_block"])
+                        self._complete_web_search_call(content_block_start["content_block"], "web_fetch")
                         provider_specific_fields["web_search_results"] = self.web_search_results
+                        provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
                     else:
                         self.tool_results.append(content_block_start["content_block"])
                         provider_specific_fields["tool_results"] = self.tool_results
@@ -1127,7 +1136,7 @@ class ModelResponseIterator:
                 # check if tool call content block - only for tool_use and server_tool_use blocks
                 if self.current_content_block_type in ("tool_use", "server_tool_use"):
                     is_empty: Final = self.check_empty_tool_call_args()
-                    if is_empty:
+                    if is_empty and self.current_content_block_type == "tool_use":
                         tool_use = ChatCompletionToolCallChunk(
                             id=None,
                             type="function",
@@ -1151,8 +1160,24 @@ class ModelResponseIterator:
                                 self._server_tool_inputs[self._current_server_tool_id] = json.loads(args)
                             except (json.JSONDecodeError, TypeError):
                                 pass
-                        if self._refresh_pending_web_search_call(self._current_server_tool_id):
+                        server_tool_input: Final = self._server_tool_inputs.get(self._current_server_tool_id, {})
+                        if self._refresh_pending_web_search_call(
+                            self._current_server_tool_id, self._current_server_tool_name
+                        ):
                             provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
+                        self._server_tool_calls = (
+                            *self._server_tool_calls,
+                            ChatCompletionToolCallChunk(
+                                id=self._current_server_tool_id,
+                                type="function",
+                                function=ChatCompletionToolCallFunctionChunk(
+                                    name=self._current_server_tool_name,
+                                    arguments=args or json.dumps(server_tool_input),
+                                ),
+                                index=len(self._server_tool_calls),
+                            ),
+                        )
+                        provider_specific_fields["server_tool_calls"] = list(self._server_tool_calls)
                         self._current_server_tool_id = None
                 if self.is_response_format_tool:
                     text = self._flush_response_format_buffer()

litellm/llms/anthropic/chat/transformation.py

@@ -6,10 +6,11 @@ from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
 from functools import partial
 from itertools import chain
 from types import MappingProxyType
-from typing import TYPE_CHECKING, Any, ClassVar, Final, NoReturn, cast
+from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Final, NoReturn, cast
 
 import httpx
-from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
+from openai.types.responses.response_function_web_search import ResponseFunctionWebSearch
+from pydantic import BaseModel, BeforeValidator, ConfigDict, TypeAdapter, ValidationError
 from typing_extensions import ReadOnly, TypedDict
 
 import litellm
@@ -110,11 +111,11 @@ from litellm.types.llms.openai import (
     ChatCompletionToolCallChunk,
     ChatCompletionToolCallFunctionChunk,
     ChatCompletionToolParam,
-    OpenAIChatCompletionFinishReason,
     OpenAIMcpServerTool,
     OpenAIWebSearchOptions,
 )
 from litellm.types.responses.main import (
+    HOSTED_WEB_TOOL_NAMES,
     OutputCodeInterpreterCall,
     build_code_interpreter_log_outputs,
     build_web_search_call,
@@ -150,6 +151,7 @@ from ..common_utils import (
     stop_reason_provider_specific_fields,
     strip_advisor_blocks_from_messages,
     tools_marked_for_a_tool_config_point,
+    validated_object_mapping,
 )
 
 if TYPE_CHECKING:
@@ -187,9 +189,9 @@ _ANTHROPIC_TOOL_NAME_MAX_LEN: Final = 128
 _DEFER_LOADING_ADAPTER: Final = TypeAdapter(bool)
 _TOOL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
 _TOOLS_ADAPTER: Final = TypeAdapter(list[object])
-_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
+_CONTENT_BLOCKS_ADAPTER: Final = TypeAdapter(tuple[object, ...])
 _RESPONSE_BLOCKS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
-_PENDING_WEB_SEARCH_RESULT: Final = MappingProxyType({"content": ()})
+_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
 _REQUEST_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
 _NATIVE_OUTPUT_STRING_FORMATS: Final = frozenset(
     ("date-time", "time", "date", "duration", "email", "hostname", "uri", "ipv4", "ipv6", "uuid")
@@ -346,6 +348,51 @@ def _validated_defer_loading(value: object) -> bool:
         raise ValueError("defer_loading must be a boolean") from exc
 
 
+def _hosted_web_tool_use(block: object) -> tuple[str, str, object] | None:
+    match block:
+        case {
+            "type": "server_tool_use",
+            "name": str() as name,
+            "id": str() as call_id,
+            "input": object() as tool_input,
+        } if name in HOSTED_WEB_TOOL_NAMES:
+            return call_id, name, tool_input
+        case _:
+            return None
+
+
+def _server_tool_use_id(block: object) -> str | None:
+    match block:
+        case {"type": "server_tool_use", "id": str() as call_id}:
+            return call_id
+        case _:
+            return None
+
+
+def _dict_or_empty(value: object) -> object:
+    return validated_object_mapping(value) or {}
+
+
+_RESPONSE_BODY_ADAPTER: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(
+    Annotated[Mapping[str, object], BeforeValidator(_dict_or_empty)]
+)
+
+
+def _server_tool_use_ids(completion_response: Mapping[str, object]) -> frozenset[str]:
+    blocks: Final = _CONTENT_BLOCKS_ADAPTER.validate_python(completion_response.get("content") or ())
+    return frozenset(filter(None, map(_server_tool_use_id, blocks)))
+
+
+def _hosted_web_call(call_id: str, tool_name: str, tool_input: object, result: object) -> ResponseFunctionWebSearch:
+    return build_web_search_call(
+        tool_id=call_id,
+        tool_input=tool_input,
+        result=result,
+        status=None if result is not None else "in_progress",
+        tool_name=tool_name,
+    )
+
+
 class _AnthropicUsageIteration(TypedDict, total=False):
     """One entry of the ``usage.iterations`` array on an Anthropic response."""
 
@@ -3079,25 +3126,16 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         self,
         web_search_results: Sequence[object],
         completion_response: Mapping[str, object],
-    ) -> list[object]:
-        blocks: Final = _RESPONSE_BLOCKS.validate_python(completion_response.get("content") or ())
-        results_by_id: Final = {
-            tool_use_id: result
+    ) -> list[ResponseFunctionWebSearch]:
+        blocks: Final = _CONTENT_BLOCKS_ADAPTER.validate_python(completion_response.get("content") or ())
+        results_by_call: Final = {
+            (tool_use_id, result.get("type")): result
             for result in _RESPONSE_BLOCKS.validate_python(web_search_results)
-            if result.get("type") == "web_search_tool_result"
-            and isinstance((tool_use_id := result.get("tool_use_id")), str)
+            if isinstance((tool_use_id := result.get("tool_use_id")), str)
         }
         return [
-            build_web_search_call(
-                tool_id=call_id,
-                tool_input=block.get("input"),
-                result=results_by_id.get(call_id, _PENDING_WEB_SEARCH_RESULT),
-                status=None if call_id in results_by_id else "in_progress",
-            )
-            for block in blocks
-            if block.get("type") == "server_tool_use"
-            and block.get("name") == "web_search"
-            and isinstance((call_id := block.get("id")), str)
+            _hosted_web_call(call_id, tool_name, tool_input, results_by_call.get((call_id, f"{tool_name}_tool_result")))
+            for call_id, tool_name, tool_input in filter(None, map(_hosted_web_tool_use, blocks))
         ]
 
     def _build_provider_specific_fields(
@@ -3109,6 +3147,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         tool_results: Sequence[_AnthropicToolResultBlock] | None,
         compaction_blocks: Sequence[object] | None,
         tool_calls: list[ChatCompletionToolCallChunk],
+        server_tool_use_ids: frozenset[str],
     ) -> dict[str, object]:
         provider_specific_fields: Final[dict[str, object]] = {
             "citations": citations,
@@ -3148,6 +3187,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         if compaction_blocks is not None:
             provider_specific_fields["compaction_blocks"] = compaction_blocks
 
+        server_tool_calls: Final = [tc for tc in tool_calls if tc["id"] in server_tool_use_ids]
+        if server_tool_calls:
+            provider_specific_fields["server_tool_calls"] = server_tool_calls
+
         stop_reason_fields: Final = stop_reason_provider_specific_fields(
             STOP_REASON_FIELDS.validate_python(completion_response)
         )
@@ -3207,6 +3250,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         ):
             text_content = prefix_prompt + text_content
 
+        server_tool_use_ids: Final = _server_tool_use_ids(_RESPONSE_BODY_ADAPTER.validate_python(completion_response))
+        client_tool_calls: Final = [tc for tc in tool_calls if tc["id"] not in server_tool_use_ids]
         provider_specific_fields: Final = self._build_provider_specific_fields(
             completion_response,
             citations,
@@ -3215,11 +3260,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
             tool_results,
             compaction_blocks,
             tool_calls,
+            server_tool_use_ids,
         )
 
         json_mode_message, tool_calls_for_message, json_extra_content = self._resolve_json_mode_non_streaming(
             json_mode=json_mode,
-            tool_calls=tool_calls,
+            tool_calls=client_tool_calls,
             response_format_schema=response_format_schema,
         )
         merged_text = text_content or ""
@@ -3258,10 +3304,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
 
         model_response.choices[0].message = _message
         model_response.hidden_params["original_response"] = completion_response["content"]
-        model_response.choices[0].finish_reason = cast(
-            OpenAIChatCompletionFinishReason,
-            map_finish_reason(completion_response["stop_reason"]),
-        )
+        model_response.choices[0].finish_reason = map_finish_reason(completion_response["stop_reason"])
 
         usage: Final = self.calculate_usage(
             usage_object=completion_response["usage"],

litellm/llms/anthropic/common_utils.py

@@ -6,7 +6,7 @@ import copy
 import json
 import os
 import re
-from collections.abc import Container, Mapping, MutableMapping, Sequence
+from collections.abc import Callable, Container, Mapping, MutableMapping, Sequence
 from dataclasses import dataclass
 from datetime import datetime, timezone
 from itertools import chain
@@ -70,10 +70,26 @@ from litellm.types.llms.anthropic import (
     AnthropicThinkingParam,
 )
 from litellm.types.llms.base import LiteLLMBaseModel
-from litellm.types.llms.openai import AllMessageValues
+from litellm.types.llms.openai import (
+    AllMessageValues,
+    ChatCompletionAssistantToolCall,
+    ChatCompletionToolCallFunctionChunk,
+)
 from litellm.types.proxy.auth.special_headers import SpecialHeaders
 from litellm.types.proxy.model_listing import ModelInfoResponse
-from litellm.types.utils import LlmProviders
+from litellm.types.responses.main import (
+    ResponseFunctionWebSearch,
+    web_search_call_tool_id,
+    with_hosted_web_tool_arguments,
+)
+from litellm.types.utils import (
+    ChatCompletionMessageCustomToolCall,
+    ChatCompletionMessageToolCall,
+    Delta,
+    LlmProviders,
+    Message,
+    chat_completion_tool_call_from_dict,
+)
 
 _MessageT = TypeVar("_MessageT")
 
@@ -126,6 +142,11 @@ _DOTTED_VERSION_RE: Final = re.compile(r"(\d)\.(\d)")
 _CLAUDE_CODE_BILLING_HEADER_PREFIX: Final = "x-anthropic-billing-header:"
 OBJECT_MAPPING_ADAPTER: Final = TypeAdapter(dict[object, object])
 OBJECT_LIST_ADAPTER: Final = TypeAdapter(list[object])
+_SERVER_TOOL_CALLS_ADAPTER: Final = TypeAdapter(tuple[ChatCompletionAssistantToolCall, ...])
+_SERVER_TOOL_CALL_ADAPTER: Final = TypeAdapter(ChatCompletionAssistantToolCall)
+_SERVER_CALL_FIELD_KEYS: Final = ("server_tool_calls", "web_search_calls")
+_WEB_SEARCH_CALL_ADAPTER: Final = TypeAdapter(ResponseFunctionWebSearch)
+_TOOL_CALL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
 _DECODED_JSON: Final = TypeAdapter(object)
 _JSON_VALUE: Final = TypeAdapter(JsonValue)
 
@@ -2290,6 +2311,261 @@ def _replayed_server_tool_use(block: object) -> _ReplayedServerToolUse | None:
         return None
 
 
+def anthropic_server_tool_call_params(message: object) -> tuple[ChatCompletionAssistantToolCall, ...]:
+    fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields"))
+    try:
+        return _SERVER_TOOL_CALLS_ADAPTER.validate_python(fields.get("server_tool_calls", ()) if fields else ())
+    except ValidationError:
+        return ()
+
+
+def anthropic_server_tool_calls(message: object) -> tuple[ChatCompletionMessageToolCall, ...]:
+    return tuple(_message_tool_call_from_params(call) for call in anthropic_server_tool_call_params(message))
+
+
+def tool_calls_with_server_tool_calls(
+    message: object,
+) -> tuple[ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall, ...]:
+    client_tool_calls: Final = tuple(
+        call
+        for tool_call in parts_of(message_field(message, "tool_calls"))
+        if (call := _client_message_tool_call(tool_call)) is not None
+    )
+    return (*anthropic_server_tool_calls(message), *client_tool_calls)
+
+
+def anthropic_server_call_fields(holder: object) -> Mapping[str, object]:
+    fields: Final = validated_object_mapping(message_field(holder, "provider_specific_fields")) or {}
+    return MappingProxyType({key: fields[key] for key in _SERVER_CALL_FIELD_KEYS if fields.get(key)})
+
+
+def message_dict_with_server_tool_calls(message: object) -> object:
+    mapping: Final = validated_object_mapping(message)
+    server_tool_calls: Final = anthropic_server_tool_call_params(message)
+    if mapping is None or not server_tool_calls:
+        return message
+    return {**mapping, "tool_calls": [*server_tool_calls, *parts_of(mapping.get("tool_calls"))]}
+
+
+def _server_tool_call_param(tool_call: object) -> ChatCompletionAssistantToolCall | None:
+    try:
+        call: Final = _SERVER_TOOL_CALL_ADAPTER.validate_python(tool_call)
+    except ValidationError:
+        return None
+    return call if _server_tool_call_id(call).startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX) else None
+
+
+def message_dict_with_server_tool_calls_split_out(message: object) -> object:
+    mapping: Final = validated_object_mapping(message)
+    tool_calls: Final = parts_of(mapping.get("tool_calls")) if mapping is not None else ()
+    server_tool_calls: Final = tuple(
+        call for tool_call in tool_calls if (call := _server_tool_call_param(tool_call)) is not None
+    )
+    if mapping is None or not server_tool_calls:
+        return message
+    fields: Final = validated_object_mapping(mapping.get("provider_specific_fields")) or {}
+    changed: Final = changed_server_tool_call_arguments(anthropic_server_tool_call_params(message), server_tool_calls)
+    client_tool_calls: Final = [tool_call for tool_call in tool_calls if _server_tool_call_param(tool_call) is None]
+    return {
+        **mapping,
+        "tool_calls": client_tool_calls or None,
+        "provider_specific_fields": {
+            **fields,
+            "server_tool_calls": list(server_tool_calls),
+            **(
+                {
+                    "web_search_calls": _rewritten_web_search_calls(
+                        fields["web_search_calls"], changed, _with_hosted_web_call_arguments
+                    )
+                }
+                if changed and fields.get("web_search_calls") is not None
+                else {}
+            ),
+        },
+    }
+
+
+def with_server_tool_call_arguments(
+    call: ChatCompletionAssistantToolCall, arguments: str
+) -> ChatCompletionAssistantToolCall:
+    return ChatCompletionAssistantToolCall(
+        id=call["id"],
+        type=call["type"],
+        function=ChatCompletionToolCallFunctionChunk(name=call["function"].get("name"), arguments=arguments),
+    )
+
+
+def write_anthropic_server_tool_calls(
+    holder: Message | Delta, rewritten: tuple[ChatCompletionAssistantToolCall, ...]
+) -> None:
+    original: Final = anthropic_server_tool_call_params(holder)
+    if rewritten == original:
+        return
+    holder.provider_specific_fields = {  # rebind-ok: a guardrail rewrite must land on the message it scanned
+        **(holder.provider_specific_fields or {}),
+        "server_tool_calls": list(rewritten),
+    }
+    write_anthropic_hosted_web_call_arguments(holder, changed_server_tool_call_arguments(original, rewritten))
+
+
+def write_request_server_tool_calls(message: object, rewritten: tuple[ChatCompletionAssistantToolCall, ...]) -> None:
+    original: Final = anthropic_server_tool_call_params(message)
+    if rewritten == original:
+        return
+    write_request_hosted_web_call_arguments(message, changed_server_tool_call_arguments(original, rewritten))
+    fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields")) or {}
+    if not isinstance(message, dict):
+        return
+    message["provider_specific_fields"] = {  # rebind-ok: a guardrail rewrite must land on the request it scanned
+        **fields,
+        "server_tool_calls": list(rewritten),
+    }
+
+
+def write_request_hosted_web_call_arguments(message: object, arguments_by_tool_id: Mapping[str, str]) -> None:
+    fields: Final = validated_object_mapping(message_field(message, "provider_specific_fields"))
+    if not arguments_by_tool_id or not isinstance(message, dict) or fields is None:
+        return
+    if fields.get("web_search_calls") is None:
+        return
+    message["provider_specific_fields"] = {  # rebind-ok: a rewritten call's hosted copy must not keep its old input
+        **fields,
+        "web_search_calls": _rewritten_web_search_calls(
+            fields["web_search_calls"], arguments_by_tool_id, _jsonable_hosted_web_call
+        ),
+    }
+
+
+def _server_tool_call_id(call: ChatCompletionAssistantToolCall) -> str:
+    return call["id"] or ""
+
+
+def _server_tool_call_arguments(call: ChatCompletionAssistantToolCall) -> str:
+    return call["function"].get("arguments") or ""
+
+
+def changed_server_tool_call_arguments(
+    original: Sequence[ChatCompletionAssistantToolCall], rewritten: Sequence[ChatCompletionAssistantToolCall]
+) -> Mapping[str, str]:
+    original_arguments: Final = MappingProxyType(
+        {_server_tool_call_id(call): _server_tool_call_arguments(call) for call in original}
+    )
+    return MappingProxyType(
+        {
+            _server_tool_call_id(call): _server_tool_call_arguments(call)
+            for call in rewritten
+            if _server_tool_call_arguments(call) != original_arguments.get(_server_tool_call_id(call))
+        }
+    )
+
+
+def write_anthropic_hosted_web_call_arguments(holder: Message | Delta, arguments_by_tool_id: Mapping[str, str]) -> None:
+    fields: Final = holder.provider_specific_fields
+    if not arguments_by_tool_id or not fields or fields.get("web_search_calls") is None:
+        return
+    holder.provider_specific_fields = {  # rebind-ok: the hosted copy of a rewritten call must not keep its old input
+        **fields,
+        "web_search_calls": _rewritten_web_search_calls(
+            fields["web_search_calls"], arguments_by_tool_id, _with_hosted_web_call_arguments
+        ),
+    }
+
+
+def _rewritten_web_search_calls(
+    calls: object,
+    arguments_by_tool_id: Mapping[str, str],
+    rewrite: Callable[[object, Mapping[str, str]], object],
+) -> object:
+    mapping: Final = validated_object_mapping(calls)
+    if mapping is not None:
+        return {key: rewrite(call, arguments_by_tool_id) for key, call in mapping.items()}
+    if not isinstance(calls, list):
+        return calls
+    return [rewrite(call, arguments_by_tool_id) for call in parts_of(calls)]
+
+
+def _jsonable_hosted_web_call(item: object, arguments_by_tool_id: Mapping[str, str]) -> object:
+    call: Final = _with_hosted_web_call_arguments(item, arguments_by_tool_id)
+    if not isinstance(call, ResponseFunctionWebSearch):
+        return call
+    return _WEB_SEARCH_CALL_ADAPTER.dump_python(call, mode="json", exclude_none=True)
+
+
+def _with_hosted_web_call_arguments(item: object, arguments_by_tool_id: Mapping[str, str]) -> object:
+    try:
+        call: Final = _WEB_SEARCH_CALL_ADAPTER.validate_python(item)
+    except ValidationError:
+        return item
+    arguments: Final = arguments_by_tool_id.get(web_search_call_tool_id(call.id) or "")
+    if arguments is None:
+        return item
+    return call.model_copy(update={"action": with_hosted_web_tool_arguments(call.action, arguments)})
+
+
+def without_anthropic_server_tool_calls(holder: Message, tool_ids: frozenset[str]) -> None:
+    fields: Final = holder.provider_specific_fields or {}
+    holder.provider_specific_fields = {  # rebind-ok: a denied call must leave every copy of the message it was on
+        **fields,
+        "server_tool_calls": [
+            call for call in anthropic_server_tool_call_params(holder) if _server_tool_call_id(call) not in tool_ids
+        ],
+        **(
+            {"web_search_calls": _web_search_calls_without(fields["web_search_calls"], tool_ids)}
+            if fields.get("web_search_calls") is not None
+            else {}
+        ),
+    }
+
+
+def _web_search_calls_without(calls: object, tool_ids: frozenset[str]) -> object:
+    mapping: Final = validated_object_mapping(calls)
+    if mapping is not None:
+        return {key: call for key, call in mapping.items() if _hosted_web_call_tool_id(call) not in tool_ids}
+    items: Final = validated_object_list(calls)
+    if items is None:
+        return calls
+    return [call for call in items if _hosted_web_call_tool_id(call) not in tool_ids]
+
+
+def _hosted_web_call_tool_id(item: object) -> str | None:
+    try:
+        return web_search_call_tool_id(_WEB_SEARCH_CALL_ADAPTER.validate_python(item).id)
+    except ValidationError:
+        return None
+
+
+def write_anthropic_server_tool_call_arguments(holder: Message | Delta, index: int, arguments: str) -> None:
+    write_anthropic_server_tool_calls(
+        holder,
+        tuple(
+            with_server_tool_call_arguments(call, arguments) if call_index == index else call
+            for call_index, call in enumerate(anthropic_server_tool_call_params(holder))
+        ),
+    )
+
+
+def _message_tool_call_from_params(call: ChatCompletionAssistantToolCall) -> ChatCompletionMessageToolCall:
+    return ChatCompletionMessageToolCall(id=call["id"], type=call["type"], function=dict(call["function"]))
+
+
+def _client_message_tool_call(
+    tool_call: object,
+) -> ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall | None:
+    match tool_call:
+        case ChatCompletionMessageToolCall() | ChatCompletionMessageCustomToolCall():
+            return tool_call
+        case _:
+            try:
+                fields: Final = _TOOL_CALL_FIELDS_ADAPTER.validate_python(tool_call)
+                return chat_completion_tool_call_from_dict(
+                    MappingProxyType({key: value for key, value in fields.items() if key != "custom"})
+                    if fields.get("type") == "function"
+                    else fields
+                )
+            except (TypeError, ValidationError):
+                return None
+
+
 def tool_call_is_rebuilt_as_server_tool_use(
     tool_call_id: object, provider_specific_fields: object, answered_tool_ids: Container[str] = frozenset()
 ) -> bool:

litellm/llms/anthropic/messages/adapters/handler.py

@@ -13,7 +13,7 @@ from typing_extensions import TypedDict
 import litellm
 from litellm._logging import verbose_logger
 from litellm.litellm_core_utils.asyncify import run_async_function
-from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
+from litellm.litellm_core_utils.core_helpers import as_str_mapping, drops_unsupported_params
 from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObject
 from litellm.litellm_core_utils.tool_fields import (
     ROUTES_WITHOUT_MCP_SERVERS,
@@ -676,6 +676,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
             request_data,
             custom_llm_provider=target_provider,
             api_base=api_base if isinstance(api_base, str) else None,
+            deployment_model_info=as_str_mapping(provider_kwargs.get("model_info")),
             drop_params=drop_params,
         )
 

litellm/llms/anthropic/messages/adapters/transformation.py

@@ -46,6 +46,25 @@ def _thought_signature(provider_specific_fields: object) -> str | None:
     return signature if isinstance(signature, str) else None
 
 
+def _object_sequence(value: object) -> Sequence[object] | None:
+    return value if isinstance(value, (list, tuple)) else None
+
+
+def _server_tool_blocks(message: object) -> tuple[Mapping[str, object], ...]:
+    server_tool_calls: Final = anthropic_server_tool_call_params(message)
+    if not server_tool_calls:
+        return ()
+    fields: Final = _as_string_mapping(_optional_attr(message, "provider_specific_fields")) or {}
+    return tuple(
+        convert_to_anthropic_tool_invoke(
+            server_tool_calls,
+            web_search_results=_object_sequence(fields.get("web_search_results")),
+            tool_results=_object_sequence(fields.get("tool_results")),
+            server_tool_call_ids=frozenset(tool_call["id"] for tool_call in server_tool_calls),
+        )
+    )
+
+
 def _compaction_blocks(provider_specific_fields: object) -> tuple[Mapping[str, object], ...]:
     fields: Final = _as_string_mapping(provider_specific_fields)
     raw_blocks: Final = fields.get("compaction_blocks") if fields is not None else None
@@ -127,6 +146,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
 )
 from litellm.litellm_core_utils.prompt_templates.factory import (
     THOUGHT_SIGNATURE_SEPARATOR,
+    convert_to_anthropic_tool_invoke,
 )
 from litellm.litellm_core_utils.prompt_templates.prompt_cache_breakpoint_utils import (
     with_prompt_cache_breakpoint,
@@ -145,6 +165,7 @@ from litellm.litellm_core_utils.reasoning_effort_utils import (
     reasoning_effort_from_thinking_silently,
 )
 from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
     eager_input_streaming_flag,
     is_empty_unsigned_thinking_block,
     normalize_anthropic_tool_use_id,
@@ -237,6 +258,29 @@ def target_supports_mid_conversation_system(model: str | None, custom_llm_provid
     return supports_mid_conversation_system(model=model, custom_llm_provider=custom_llm_provider)
 
 
+def _is_azure_responses_target(
+    model: str | None,
+    custom_llm_provider: str | None,
+    deployment_model_info: Mapping[str, object] | None,
+) -> bool:
+    if not model:
+        return False
+    if custom_llm_provider not in (None, LlmProviders.AZURE.value):
+        return False
+    if custom_llm_provider is None and not model.startswith("azure/"):
+        return False
+
+    from litellm.responses.bridge_check import responses_api_bridge_check
+
+    resolved_model, provider, _, _ = litellm.get_llm_provider(model=model, custom_llm_provider=custom_llm_provider)
+    model_info, _ = responses_api_bridge_check(
+        model=resolved_model,
+        custom_llm_provider=provider,
+        deployment_model_info=deployment_model_info,
+    )
+    return model_info.get("mode") == "responses"
+
+
 def _chat_tool_param(function_chunk: ChatCompletionToolParamFunctionChunk, tool: object) -> ChatCompletionToolParam:
     eager_input_streaming: Final = eager_input_streaming_flag(tool)
     if eager_input_streaming is None:
@@ -272,6 +316,7 @@ class AnthropicAdapter:
         *,
         custom_llm_provider: str | None = None,
         api_base: str | None = None,
+        deployment_model_info: Mapping[str, object] | None = None,
         drop_params: bool = True,
     ) -> tuple[ChatCompletionRequest | None, dict[str, str]]:
         """
@@ -307,6 +352,7 @@ class AnthropicAdapter:
             anthropic_message_request=request_body,
             custom_llm_provider=custom_llm_provider,
             api_base=api_base,
+            deployment_model_info=deployment_model_info,
             drop_params=drop_params,
             report_reasoning_drops=True,
         )
@@ -502,10 +548,15 @@ class LiteLLMAnthropicMessagesAdapter:
         model: str | None = None,
         *,
         custom_llm_provider: str | None = None,
+        deployment_model_info: Mapping[str, object] | None = None,
         preserve_midturn_system: bool = False,
     ) -> list[AllMessageValues]:
         new_messages: Final[list[AllMessageValues]] = []
-        replayable_messages: Final = strip_encrypted_reasoning_blocks_from_anthropic_messages(messages)
+        replayable_messages: Final = (
+            messages
+            if _is_azure_responses_target(model, custom_llm_provider, deployment_model_info)
+            else strip_encrypted_reasoning_blocks_from_anthropic_messages(messages)
+        )
         leading_count: Final = next(
             (i for i, m in enumerate(replayable_messages) if not is_system_role_message(m)),
             len(replayable_messages),
@@ -1289,6 +1340,7 @@ class LiteLLMAnthropicMessagesAdapter:
         *,
         custom_llm_provider: str | None = None,
         api_base: str | None = None,
+        deployment_model_info: Mapping[str, object] | None = None,
         preserve_midturn_system: bool = False,
         drop_params: bool = True,
         report_reasoning_drops: bool = False,
@@ -1314,6 +1366,7 @@ class LiteLLMAnthropicMessagesAdapter:
             messages=messages_list,
             model=anthropic_message_request.get("model"),
             custom_llm_provider=custom_llm_provider,
+            deployment_model_info=deployment_model_info,
             preserve_midturn_system=preserve_midturn_system,
         )
         ## ADD SYSTEM MESSAGE TO MESSAGES
@@ -1476,6 +1529,7 @@ class LiteLLMAnthropicMessagesAdapter:
                     ).model_dump()
                 )
 
+            new_content.extend(dict(block) for block in _server_tool_blocks(choice.message))
             # Handle text content
             if choice.message.content is not None and (choice.message.content != "" or not compaction_blocks):
                 new_content.append(

litellm/llms/anthropic/messages/responses_adapters/transformation.py

@@ -584,6 +584,12 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
             if block.get("type") == "text" and isinstance(text := block.get("text"), str) and text
         )
 
+    @staticmethod
+    def _resolve_output_format_strict(strict: bool | None) -> bool:
+        if strict is None:
+            return True
+        return strict
+
     def translate_request(
         self,
         anthropic_request: AnthropicMessagesRequest,
@@ -693,7 +699,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
                         "type": "json_schema",
                         "name": "structured_output",
                         "schema": schema,
-                        "strict": output_format.get("strict", False),
+                        "strict": self._resolve_output_format_strict(output_format.get("strict")),
                     }
                 }
 

litellm/llms/base_llm/guardrail_translation/utils.py

@@ -4,11 +4,14 @@ import json
 from collections.abc import Callable, Iterator, Mapping, Sequence
 from typing import Final, TypeVar, cast  # noqa: TID251  # a rebuilt chat row has no typed constructor across roles
 
+from openai.types.responses import ResponseFunctionWebSearch
 from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
 
 from litellm.llms.custom_httpx.sse_lines import split_sse_lines
+from litellm.types.llms.anthropic import claude_hosted_web_tool_call
 from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicUsage
 from litellm.types.llms.openai import AllMessageValues, ResponseAPIUsage
+from litellm.types.responses.main import with_hosted_web_tool_arguments
 
 _DECODED_JSON: Final = TypeAdapter(object)
 
@@ -135,6 +138,36 @@ def stream_item_field(item: object, field: str) -> object | None:
     return getattr(item, field, None)
 
 
+def web_search_call(item: object) -> ResponseFunctionWebSearch | None:
+    if isinstance(item, ResponseFunctionWebSearch):
+        return item
+    if stream_item_field(item, "type") != "web_search_call":
+        return None
+    try:
+        return ResponseFunctionWebSearch.model_validate(item.model_dump() if isinstance(item, BaseModel) else item)
+    except ValidationError:
+        return None
+
+
+def claude_hosted_web_tool_arguments(item: object) -> str | None:
+    search: Final = web_search_call(item)
+    hosted_call: Final = None if search is None else claude_hosted_web_tool_call(search)
+    return None if hosted_call is None else hosted_call[2]
+
+
+def with_web_search_call_arguments(item: object, arguments: str) -> object:
+    search: Final = web_search_call(item)
+    if search is None:
+        return item
+    action: Final = with_hosted_web_tool_arguments(search.action, arguments)
+    if isinstance(item, ResponseFunctionWebSearch):
+        return item.model_copy(update={"action": action})
+    if isinstance(item, BaseModel):
+        return item.model_copy(update={"action": action.model_dump()})
+    fields: Final = cast("Mapping[str, object]", item)  # cast-ok: a validated web_search_call dict is str-keyed JSON
+    return {**fields, "action": action.model_dump()}
+
+
 def stream_item_fingerprint(item: object) -> str:
     plain: Final = item.model_dump() if isinstance(item, BaseModel) else item
     return json.dumps(plain, sort_keys=True, default=str)

litellm/llms/bedrock/chat/converse_transformation.py

@@ -960,7 +960,8 @@ class AmazonConverseConfig(BaseConfig):
     ) -> bool:
         """Check if the Bedrock model takes native structured outputs (outputConfig.textFormat) for this request.
 
-        ``supports_native_structured_output`` in ``litellm.model_cost`` turns it on for every request, and
+        ``supports_native_structured_output`` or ``supports_bedrock_native_structured_output_by_default`` in
+        ``litellm.model_cost`` turns it on for every request, and
         ``supports_bedrock_native_structured_output`` (the deployment's ``model_info`` first) also when Claude
         will think, which the json tool path would drop. When the request does not ask for thinking, a model
         that cannot force the json tool gets it unless that key is false.

litellm/llms/bedrock/chat/native_output_fallback.py

@@ -9,7 +9,9 @@ from litellm.llms.bedrock.common_utils import (
     NativeOutputFallback,
     bedrock_strict_tools_fallback,
     is_bedrock_native_output_rejection,
+    is_bedrock_stream_permission_denial,
     warn_bedrock_native_output_rejected,
+    warn_bedrock_native_output_stream_denied,
     warn_bedrock_strict_tools_dropped,
 )
 from litellm.types.utils import ModelResponse
@@ -25,22 +27,55 @@ NativeOutputFallbackFactory: TypeAlias = Callable[[], NativeOutputFallback | Non
 class _SelectedFallback:
     fallback: NativeOutputFallback
     native_output_retry: bool
+    buffered_native_output: bool = False
 
 
 _RetryFor: TypeAlias = Callable[[BaseLLMException], _SelectedFallback | None]
+_NextRetryFor: TypeAlias = Callable[[BaseLLMException, _SelectedFallback], _SelectedFallback | None]
 
 
-def _native_output_retry(
-    error: BaseLLMException, fallback: NativeOutputFallbackFactory | None, model: str
+def _native_output_rejection_retry(
+    error: BaseLLMException, fallback: NativeOutputFallbackFactory, model: str
 ) -> NativeOutputFallback | None:
-    if fallback is None or not is_bedrock_native_output_rejection(error.status_code, error.message):
-        return None
     retry: Final = fallback()
     if retry is not None:
         warn_bedrock_native_output_rejected(model, error.message)
     return retry
 
 
+def _buffered(retry: NativeOutputFallback) -> NativeOutputFallback:
+    return NativeOutputFallback(
+        optional_params={**retry.optional_params, "fake_stream": True}, litellm_params=retry.litellm_params
+    )
+
+
+def _buffered_stream_retry(fallback: NativeOutputFallbackFactory, model: str) -> NativeOutputFallback | None:
+    retry: Final = fallback()
+    if retry is None or retry.optional_params.get("fake_stream") is not True:
+        return None
+    warn_bedrock_native_output_stream_denied(model)
+    return retry
+
+
+def _native_output_retry(
+    error: BaseLLMException,
+    fallback: NativeOutputFallbackFactory | None,
+    buffered_stream_fallback: NativeOutputFallbackFactory | None,
+    model: str,
+) -> _SelectedFallback | None:
+    if fallback is not None and is_bedrock_native_output_rejection(error.status_code, error.message):
+        rejection_retry: Final = _native_output_rejection_retry(error, fallback, model)
+        return None if rejection_retry is None else _SelectedFallback(rejection_retry, native_output_retry=True)
+    if buffered_stream_fallback is not None and is_bedrock_stream_permission_denial(error.status_code, error.message):
+        stream_retry: Final = _buffered_stream_retry(buffered_stream_fallback, model)
+        return (
+            None
+            if stream_retry is None
+            else _SelectedFallback(stream_retry, native_output_retry=True, buffered_native_output=True)
+        )
+    return None
+
+
 def _strict_tools_retry(
     error: BaseLLMException, fallback: NativeOutputFallbackFactory | None, model: str
 ) -> NativeOutputFallback | None:
@@ -55,12 +90,13 @@ def _strict_tools_retry(
 def _fallback_for(
     error: BaseLLMException,
     fallback: NativeOutputFallbackFactory | None,
+    buffered_stream_fallback: NativeOutputFallbackFactory | None,
     strict_tools_fallback: NativeOutputFallbackFactory | None,
     model: str,
 ) -> _SelectedFallback | None:
-    native_output_retry: Final = _native_output_retry(error, fallback, model)
+    native_output_retry: Final = _native_output_retry(error, fallback, buffered_stream_fallback, model)
     if native_output_retry is not None:
-        return _SelectedFallback(native_output_retry, native_output_retry=True)
+        return native_output_retry
     strict_tools_retry: Final = _strict_tools_retry(error, strict_tools_fallback, model)
     if strict_tools_retry is None:
         return None
@@ -87,12 +123,81 @@ def _strict_tools_retry_after_native_output_retry(
     return composed_retry
 
 
+def _next_retry(
+    error: BaseLLMException,
+    retry: _SelectedFallback,
+    fallback: NativeOutputFallbackFactory | None,
+    strict_tools_fallback: NativeOutputFallbackFactory | None,
+    model: str,
+) -> _SelectedFallback | None:
+    if (
+        retry.buffered_native_output
+        and fallback is not None
+        and is_bedrock_native_output_rejection(error.status_code, error.message)
+    ):
+        rejection_retry: Final = _native_output_rejection_retry(error, fallback, model)
+        return (
+            None if rejection_retry is None else _SelectedFallback(_buffered(rejection_retry), native_output_retry=True)
+        )
+    composed_retry: Final = _strict_tools_retry_after_native_output_retry(error, retry, strict_tools_fallback, model)
+    return None if composed_retry is None else _SelectedFallback(composed_retry, native_output_retry=False)
+
+
+def _attempt_or_next_retry(
+    attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputAttemptResult | _SelectedFallback:
+    try:
+        return attempt(retry.fallback)
+    except BaseLLMException as error:
+        next_retry: Final = next_retry_for(error, retry)
+        if next_retry is None:
+            raise
+        return next_retry
+
+
+def _attempt_retries(
+    attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputAttemptResult:
+    after_first_retry: Final = _attempt_or_next_retry(attempt, retry, next_retry_for)
+    if not isinstance(after_first_retry, _SelectedFallback):
+        return after_first_retry
+    after_second_retry: Final = _attempt_or_next_retry(attempt, after_first_retry, next_retry_for)
+    if not isinstance(after_second_retry, _SelectedFallback):
+        return after_second_retry
+    return attempt(after_second_retry.fallback)
+
+
+async def _await_or_next_retry(
+    attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputResponse | _SelectedFallback:
+    try:
+        result: Final = attempt(retry.fallback)
+        return await result if isinstance(result, Coroutine) else result
+    except BaseLLMException as error:
+        next_retry: Final = next_retry_for(error, retry)
+        if next_retry is None:
+            raise
+        return next_retry
+
+
+async def _await_retries(
+    attempt: NativeOutputAttempt, retry: _SelectedFallback, next_retry_for: _NextRetryFor
+) -> NativeOutputResponse:
+    after_first_retry: Final = await _await_or_next_retry(attempt, retry, next_retry_for)
+    if not isinstance(after_first_retry, _SelectedFallback):
+        return after_first_retry
+    after_second_retry: Final = await _await_or_next_retry(attempt, after_first_retry, next_retry_for)
+    if not isinstance(after_second_retry, _SelectedFallback):
+        return after_second_retry
+    last_attempt: Final = attempt(after_second_retry.fallback)
+    return await last_attempt if isinstance(last_attempt, Coroutine) else last_attempt
+
+
 async def _await_with_native_output_fallback(
     first_attempt: Coroutine[object, object, NativeOutputResponse],
     attempt: NativeOutputAttempt,
     retry_for: _RetryFor,
-    strict_tools_fallback: NativeOutputFallbackFactory | None,
-    model: str,
+    next_retry_for: _NextRetryFor,
 ) -> NativeOutputResponse:
     try:
         return await first_attempt
@@ -100,17 +205,7 @@ async def _await_with_native_output_fallback(
         retry: Final = retry_for(error)
         if retry is None:
             raise
-    try:
-        second_attempt: Final = attempt(retry.fallback)
-        return await second_attempt if isinstance(second_attempt, Coroutine) else second_attempt
-    except BaseLLMException as error:
-        composed_retry: Final = _strict_tools_retry_after_native_output_retry(
-            error, retry, strict_tools_fallback, model
-        )
-        if composed_retry is None:
-            raise
-    third_attempt: Final = attempt(composed_retry)
-    return await third_attempt if isinstance(third_attempt, Coroutine) else third_attempt
+    return await _await_retries(attempt, retry, next_retry_for)
 
 
 def call_with_native_output_fallback(
@@ -118,11 +213,19 @@ def call_with_native_output_fallback(
     fallback: NativeOutputFallbackFactory | None,
     model: str,
     strict_tools_fallback: NativeOutputFallbackFactory | None = None,
+    buffered_stream_fallback: NativeOutputFallbackFactory | None = None,
 ) -> NativeOutputAttemptResult:
-    if fallback is None and strict_tools_fallback is None:
+    if fallback is None and strict_tools_fallback is None and buffered_stream_fallback is None:
         return attempt(None)
     retry_for: Final = partial(
-        _fallback_for, fallback=fallback, strict_tools_fallback=strict_tools_fallback, model=model
+        _fallback_for,
+        fallback=fallback,
+        buffered_stream_fallback=buffered_stream_fallback,
+        strict_tools_fallback=strict_tools_fallback,
+        model=model,
+    )
+    next_retry_for: Final = partial(
+        _next_retry, fallback=fallback, strict_tools_fallback=strict_tools_fallback, model=model
     )
     try:
         first_attempt: Final = attempt(None)
@@ -130,15 +233,7 @@ def call_with_native_output_fallback(
         retry: Final = retry_for(error)
         if retry is None:
             raise
-        try:
-            return attempt(retry.fallback)
-        except BaseLLMException as retry_error:
-            composed_retry: Final = _strict_tools_retry_after_native_output_retry(
-                retry_error, retry, strict_tools_fallback, model
-            )
-            if composed_retry is None:
-                raise
-            return attempt(composed_retry)
+        return _attempt_retries(attempt, retry, next_retry_for)
     if isinstance(first_attempt, Coroutine):
-        return _await_with_native_output_fallback(first_attempt, attempt, retry_for, strict_tools_fallback, model)
+        return _await_with_native_output_fallback(first_attempt, attempt, retry_for, next_retry_for)
     return first_attempt

litellm/llms/bedrock/common_utils.py

@@ -297,9 +297,16 @@ def _bedrock_model_supports(model: str, key: str, custom_llm_provider: str = "be
 def _bedrock_native_output_enabled(
     model: str, custom_llm_provider: str, model_info: Mapping[str, object] | None
 ) -> bool:
-    deployment_flag: Final = (model_info or {}).get("supports_native_structured_output")
+    deployment_info: Final = model_info or {}
+    deployment_flag: Final = deployment_info.get("supports_native_structured_output")
     if isinstance(deployment_flag, bool):
         return deployment_flag
+    return deployment_info.get("supports_bedrock_native_structured_output") is not False and (
+        bedrock_model_always_takes_native_output(model, custom_llm_provider)
+    )
+
+
+def bedrock_model_always_takes_native_output(model: str, custom_llm_provider: str = "bedrock") -> bool:
     return _bedrock_model_supports(model, "supports_native_structured_output", custom_llm_provider)
 
 
@@ -443,6 +450,18 @@ def warn_bedrock_native_output_rejected(model: str, error_text: str) -> None:
     )
 
 
+def is_bedrock_stream_permission_denial(status_code: int, error_text: str) -> bool:
+    return status_code == 403 and "bedrock:InvokeModelWithResponseStream" in error_text
+
+
+def warn_bedrock_native_output_stream_denied(model: str) -> None:
+    verbose_logger.warning(
+        "Bedrock denied bedrock:InvokeModelWithResponseStream for model=%s, so litellm sent the structured output "
+        "request again as a buffered stream. Allow that action to stream it.",
+        model,
+    )
+
+
 @dataclass(frozen=True, slots=True)
 class NativeOutputFallback:
     optional_params: Mapping[str, object]
@@ -543,6 +562,22 @@ def json_object_or_none(value: object) -> Mapping[str, object] | None:
 BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES: Final = 64
 
 
+def _response_format_json_schema(response_format: object) -> Mapping[str, object] | None:
+    json_schema: Final = json_object_or_none((json_object_or_none(response_format) or {}).get("json_schema"))
+    return json_object_or_none((json_schema or {}).get("schema"))
+
+
+def bedrock_batch_record_takes_native_output(
+    model: str, response_format: object, model_info: Mapping[str, object] | None = None
+) -> bool:
+    schema: Final = _response_format_json_schema(response_format)
+    return _bedrock_native_output_enabled(model, "bedrock", model_info) and (
+        schema is None
+        or schema_property_count(schema) <= BEDROCK_NATIVE_OUTPUT_MAX_PROPERTIES
+        or not _bedrock_model_supports(model, "supports_bedrock_native_structured_output_by_default")
+    )
+
+
 def bedrock_cleaned_output_schema(
     model: str, schema: Mapping[str, object], custom_llm_provider: str = "bedrock"
 ) -> Mapping[str, object] | None:
@@ -1560,15 +1595,29 @@ def bedrock_converse_accepts_redacted_thinking(model: str, base_model: str | Non
     its own. An ARN that hides its model, like an application inference profile, falls back to the
     deployment's ``base_model`` and counts as non-Claude without one.
     """
+    return _is_claude(_converse_target_model(model, base_model))
+
+
+def bedrock_converse_rejects_signed_thinking(model: str, base_model: str | None) -> bool:
+    if base_model is None and "." not in get_bedrock_base_model(model):
+        return False
+    return not _is_claude_behind_any_region_prefix(_converse_target_model(model, base_model))
+
+
+def _converse_target_model(model: str, base_model: str | None) -> str:
     if base_model is None or "." in get_bedrock_base_model(model):
-        return _is_claude(model)
-    return _is_claude(base_model)
+        return model
+    return base_model
 
 
 def _is_claude(model: str) -> bool:
     return get_bedrock_base_model(model).startswith(("anthropic", "claude"))
 
 
+def _is_claude_behind_any_region_prefix(model: str) -> bool:
+    return _is_claude(model) or get_bedrock_base_model(model).partition(".")[2].startswith("anthropic.")
+
+
 def bedrock_converse_supports_parallel_tool_use_config(model: str) -> bool:
     return any(
         (litellm.model_cost.get(candidate) or {}).get("supports_parallel_tool_use_config") is True

litellm/llms/bedrock/files/transformation.py

@@ -71,6 +71,7 @@ from litellm.utils import get_llm_provider, get_optional_params
 from ..base_aws_llm import BaseAWSLLM
 from ..common_utils import (
     BedrockError,
+    bedrock_batch_record_takes_native_output,
     deployment_model_info,
     merge_bedrock_aws_request_params,
     resolve_s3_encryption_key_id,
@@ -167,6 +168,20 @@ _LITELLM_PARAMS_THE_MAPPER_TAKES: Final = frozenset({"allowed_openai_params"})
 _MAPPED_PARAMS_THE_REQUEST_HANDLER_STRIPS: Final = frozenset({"json_mode"})
 
 
+_UPLOAD_PARAMS: Final = TypeAdapter(dict[str, object])
+
+
+def _upload_params(params: object) -> Mapping[str, object]:
+    return _UPLOAD_PARAMS.validate_python(params) if isinstance(params, Mapping) else {}
+
+
+def _upload_deployment_model_info(litellm_params: object) -> Mapping[str, object]:
+    upload_params: Final = _upload_params(litellm_params)
+    return deployment_model_info(upload_params) or deployment_model_info(
+        _upload_params(upload_params.get("litellm_metadata"))
+    )
+
+
 def _invoke_route_model(model: str) -> str:
     return f"invoke/{_strip_llm_routing_prefix(model).removeprefix('invoke/')}"
 
@@ -1034,6 +1049,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
         openai_request_body: Mapping[str, Any],
         model: str,
         provider: str | None = None,
+        model_info: Mapping[str, object] | None = None,
     ) -> dict[str, object]:
         """
         Transform OpenAI request body to Bedrock-compatible modelInput
@@ -1055,7 +1071,15 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
             )
 
             config: Final = AmazonAnthropicClaudeConfig()
-            litellm_params: Final = without_bedrock_native_output_opt_in({})
+            litellm_params: Final = without_bedrock_native_output_opt_in(
+                {
+                    "model_info": {
+                        "supports_native_structured_output": bedrock_batch_record_takes_native_output(
+                            _invoke_route_model(model), optional_params.get("response_format"), model_info
+                        )
+                    }
+                }
+            )
             mapped_params = get_optional_params(
                 model=_invoke_route_model(model),
                 custom_llm_provider="bedrock",
@@ -1150,25 +1174,32 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
         ]
 
     def transform_batch_input_lines(
-        self, batch_input_lines: Sequence[_BatchInputLine], target_model: str = ""
+        self,
+        batch_input_lines: Sequence[_BatchInputLine],
+        target_model: str = "",
+        model_info: Mapping[str, object] | None = None,
     ) -> tuple[_BedrockBatchRecord, ...]:
         return tuple(
-            self._transform_batch_input_line(idx, batch_input_line, target_model)
+            self._transform_batch_input_line(idx, batch_input_line, target_model, model_info)
             for idx, batch_input_line in enumerate(batch_input_lines)
         )
 
     def _transform_batch_input_line(
-        self, idx: int, batch_input_line: _BatchInputLine, target_model: str
+        self, idx: int, batch_input_line: _BatchInputLine, target_model: str, model_info: Mapping[str, object] | None
     ) -> _BedrockBatchRecord:
         try:
-            return self._transform_openai_batch_record(idx, batch_input_line.record, target_model)
+            return self._transform_openai_batch_record(idx, batch_input_line.record, target_model, model_info)
         except _BATCH_RECORD_CONVERSION_ERRORS as e:
             raise _batch_line_error(
                 batch_input_line.line_number, f"could not be converted to a Bedrock batch record: {e}"
             ) from e
 
     def _transform_openai_batch_record(
-        self, idx: int, openai_record: _OpenAIBatchRecord, target_model: str
+        self,
+        idx: int,
+        openai_record: _OpenAIBatchRecord,
+        target_model: str,
+        model_info: Mapping[str, object] | None = None,
     ) -> _BedrockBatchRecord:
         import litellm
 
@@ -1192,6 +1223,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
                 openai_request_body=self._transform_batch_body_to_chat_body(openai_body, record_kind),
                 model=model_for_transform,
                 provider=provider,
+                model_info=model_info,
             )
         )
         return _BedrockBatchRecord(
@@ -1211,7 +1243,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
         model: str,
         create_file_data: CreateFileRequest,
         optional_params: dict,
-        litellm_params: dict,
+        litellm_params: dict[str, object],
     ) -> bytes | str | dict:
         """
         Transform file request and return a pre-signed request for S3.
@@ -1237,7 +1269,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
             litellm_params_model: Final = litellm_params.get("model")
             target_model: Final = model or (litellm_params_model if isinstance(litellm_params_model, str) else "")
             bedrock_jsonl_content: Final = self.transform_batch_input_lines(
-                batch_input_lines, target_model=target_model
+                batch_input_lines, target_model=target_model, model_info=_upload_deployment_model_info(litellm_params)
             )
             file_content = "\n".join(json.dumps(item) for item in bedrock_jsonl_content)
         elif isinstance(extracted_file_data_content, bytes):

litellm/llms/chatgpt/responses/transformation.py

@@ -143,6 +143,8 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
             "reasoning",
             "previous_response_id",
             "truncation",
+            "text",
+            "max_output_tokens",
         }
 
         filtered: Final = {k: v for k, v in request.items() if k in allowed_keys}
@@ -218,7 +220,10 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
                 )
                 continue
 
-            if event_type == ResponsesAPIStreamEvents.RESPONSE_COMPLETED:
+            if event_type in (
+                ResponsesAPIStreamEvents.RESPONSE_COMPLETED,
+                ResponsesAPIStreamEvents.RESPONSE_INCOMPLETE,
+            ):
                 # Real OUTPUT_ITEM_DONE events take precedence at any given
                 # output_index, but text-only items at indices without a
                 # matching OUTPUT_ITEM_DONE must still be preserved (e.g.

litellm/llms/custom_httpx/llm_http_handler.py

@@ -897,7 +897,7 @@ class BaseLLMHTTPHandler:
 
         fake_stream = (
             fake_stream
-            or optional_params.pop("fake_stream", False)
+            or optional_params.pop("fake_stream", False) is True
             or provider_config.should_fake_stream(model=model, custom_llm_provider=custom_llm_provider, stream=stream)
         )
 
@@ -920,7 +920,7 @@ class BaseLLMHTTPHandler:
             api_key=api_key,
             model=model,
             optional_params=optional_params,
-            stream=stream,
+            stream=False if fake_stream else stream,
             litellm_params=litellm_params,
         )
 
@@ -1056,7 +1056,7 @@ class BaseLLMHTTPHandler:
                 provider_config=provider_config,
                 fake_stream=fake_stream,
             )
-            if provider_config.has_custom_stream_wrapper is True:
+            if provider_config.has_custom_stream_wrapper is True and not fake_stream:
                 return provider_config.get_sync_custom_stream_wrapper(
                     model=model,
                     custom_llm_provider=custom_llm_provider,
@@ -1225,7 +1225,7 @@ class BaseLLMHTTPHandler:
         json_mode: bool | None = None,
         signed_json_body: bytes | None = None,
     ):
-        if provider_config.has_custom_stream_wrapper is True:
+        if provider_config.has_custom_stream_wrapper is True and not fake_stream:
             return await provider_config.get_async_custom_stream_wrapper(
                 model=model,
                 custom_llm_provider=custom_llm_provider,

litellm/llms/fireworks_ai/responses/transformation.py

@@ -1,10 +1,11 @@
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
 from types import MappingProxyType
-from typing import TYPE_CHECKING, Final
+from typing import TYPE_CHECKING, Final, cast  # noqa: TID251  # the base signature leaves the params bag untyped
 from urllib.parse import unquote
 
 import httpx
 
+from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
 from litellm.litellm_core_utils.url_utils import join_api_base
 from litellm.llms.base_llm.responses.transformation import (
     STANDARD_MODE_WITHOUT_CONTEXT_REASONING_FIELD_VALUES,
@@ -20,8 +21,9 @@ from litellm.llms.fireworks_ai.common_utils import (
 )
 from litellm.llms.openai.responses.instruction_items import developer_items_as_system, instruction_items_folded
 from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
+from litellm.responses.additional_tools import HoistedAdditionalTools, hoist_additional_tools
 from litellm.secret_managers.main import get_secret_str
-from litellm.types.llms.openai import ResponseInputParam
+from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
 from litellm.types.responses.main import DeleteResponseResult
 from litellm.types.router import GenericLiteLLMParams
 from litellm.types.utils import LlmProviders
@@ -98,6 +100,23 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
         headers: dict[str, str],  # mutable-ok: overrides the base class signature
     ) -> dict[str, object]:  # mutable-ok: overrides the base class signature
         validated_input: Final = self._validate_input_param(input)
+        tools_param: Final = response_api_optional_request_params.get("tools")
+        hoisted: Final = (
+            hoist_additional_tools(
+                validated_input,
+                cast(  # cast-ok: the base signature leaves the params bag untyped
+                    "Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None", tools_param
+                ),
+                drop_params=drops_unsupported_params(litellm_params.drop_params),
+            )
+            if tools_param is None or isinstance(tools_param, (list, tuple))
+            else HoistedAdditionalTools(input=validated_input, tools=(), hoisted=())
+        )
+        hoisted_params: Final = (
+            {**response_api_optional_request_params, "tools": list(hoisted.tools)}
+            if hoisted.hoisted
+            else response_api_optional_request_params
+        )
         model_info: Final = deployment_model_info(litellm_params)
         declared: Final = declared_mid_conversation_system_support(model, self.custom_llm_provider.value, model_info)
         keeps_developer_items_in_place: Final = declared is True and (
@@ -107,14 +126,9 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
         folded: Final = (
             None
             if keeps_developer_items_in_place
-            else instruction_items_folded(
-                validated_input, response_api_optional_request_params, "user" if declared is False else "system"
-            )
-        )
-        placed_input, placed_params = folded or (
-            developer_items_as_system(validated_input),
-            response_api_optional_request_params,
+            else instruction_items_folded(hoisted.input, hoisted_params, "user" if declared is False else "system")
         )
+        placed_input, placed_params = folded or (developer_items_as_system(hoisted.input), hoisted_params)
         request: Final = super().transform_responses_api_request(
             model=resolve_fireworks_resource_name(model),
             input=placed_input,

litellm/llms/litellm_proxy/chat/transformation.py

@@ -4,9 +4,13 @@ Translate from OpenAI's `/v1/chat/completions` to VLLM's `/v1/chat/completions`
 
 from typing import TYPE_CHECKING, Final
 
-from litellm.constants import OPENAI_CHAT_COMPLETION_PARAMS
+from pydantic import BaseModel, ConfigDict
+
+import litellm
+from litellm.constants import NON_LLM_CONNECTION_TIMEOUT, OPENAI_CHAT_COMPLETION_PARAMS
 from litellm.secret_managers.main import get_secret_bool, get_secret_str
 from litellm.types.router import LiteLLM_Params
+from litellm.utils import add_path_to_api_base
 
 from ...openai.chat.gpt_transformation import OpenAIGPTConfig
 
@@ -14,6 +18,18 @@ if TYPE_CHECKING:
     from litellm.types.llms.openai import AllMessageValues
 
 
+class _ProxyModel(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    id: str
+
+
+class _ProxyModelList(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    data: tuple[_ProxyModel, ...]
+
+
 class LiteLLMProxyChatConfig(OpenAIGPTConfig):
     def get_supported_openai_params(self, model: str) -> list:
         params_list: Final = super().get_supported_openai_params(model)
@@ -53,8 +69,13 @@ class LiteLLMProxyChatConfig(OpenAIGPTConfig):
         api_base, api_key = self._get_openai_compatible_provider_info(api_base, api_key)
         if api_base is None:
             raise ValueError("api_base not set for LiteLLM Proxy route. Set in env via `LITELLM_PROXY_API_BASE`")
-        models: Final = super().get_models(api_key=api_key, api_base=api_base)
-        return [f"litellm_proxy/{model}" for model in models]
+        response: Final = litellm.module_level_client.client.get(
+            url=add_path_to_api_base(api_base, "/v1/models"),
+            headers={"Authorization": f"Bearer {api_key}"},
+            timeout=NON_LLM_CONNECTION_TIMEOUT,
+        )
+        response.raise_for_status()
+        return [f"litellm_proxy/{model.id}" for model in _ProxyModelList.model_validate_json(response.content).data]
 
     @staticmethod
     def get_api_key(api_key: str | None = None) -> str | None:

litellm/llms/openai/chat/guardrail_translation/handler.py

@@ -21,6 +21,7 @@ from collections.abc import Iterator, Mapping, Sequence
 from types import MappingProxyType
 from typing import TYPE_CHECKING, Any, Final, Union, cast
 
+from pydantic import TypeAdapter, ValidationError
 from typing_extensions import NotRequired, ReadOnly, TypedDict
 
 import litellm
@@ -29,6 +30,13 @@ from litellm.completion_extras.litellm_responses_transformation.transformation i
     replayed_tool_search_tool_names,
 )
 from litellm.litellm_core_utils.streaming_chunk_builder_utils import stream_chunk_builder
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    changed_server_tool_call_arguments,
+    write_anthropic_hosted_web_call_arguments,
+    write_anthropic_server_tool_calls,
+    write_request_server_tool_calls,
+)
 from litellm.llms.base_llm.guardrail_translation.base_translation import (
     BaseTranslation,
     StreamingScanKey,
@@ -50,7 +58,12 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
     stream_item_items,
     unappliable_request_rewrite,
 )
-from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
+from litellm.types.llms.openai import (
+    AllMessageValues,
+    ChatCompletionAssistantToolCall,
+    ChatCompletionToolCallFunctionChunk,
+    ChatCompletionToolParam,
+)
 from litellm.types.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
     coerce_stream_holdback_value,
 )
@@ -139,14 +152,27 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                 scan_only_tool_results=scan_only_tool_results,
             )
 
+        structured_messages: Final = self.get_structured_messages(data)
+        server_tool_call_tasks: Final = tuple(
+            _input_server_tool_call_tasks(
+                structured_messages or [],
+                skip_system_message=skip_system,
+                skip_tool_message=skip_tool,
+                scan_only_tool_results=scan_only_tool_results,
+            )
+        )
+        tool_calls_with_server_calls: Final = [
+            *tool_calls_to_check,
+            *(_server_tool_call_input(call) for _, call in server_tool_call_tasks),
+        ]
+
         # Step 2: Apply guardrail to all texts and tool calls in batch
-        if texts_to_check or tool_calls_to_check:
+        if texts_to_check or tool_calls_with_server_calls:
             inputs: Final = GenericGuardrailAPIInputs(texts=texts_to_check)
             if images_to_check:
                 inputs["images"] = images_to_check
-            if tool_calls_to_check:
-                inputs["tool_calls"] = tool_calls_to_check
-            structured_messages: Final = self.get_structured_messages(data)
+            if tool_calls_with_server_calls:
+                inputs["tool_calls"] = tool_calls_with_server_calls
             scoped_message_indices: Final = scoped_structured_message_indices(
                 structured_messages or [],
                 scan_only_tool_results=scan_only_tool_results,
@@ -218,6 +244,12 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                         tool_calls=guardrailed_tool_calls,
                         task_mappings=tool_call_task_mappings,
                     )
+                if len(guardrailed_tool_calls) == len(tool_calls_with_server_calls):
+                    self._apply_guardrail_responses_to_input_server_tool_calls(
+                        messages=structured_messages or [],
+                        guardrailed_tool_calls=guardrailed_tool_calls[len(tool_calls_to_check) :],
+                        task_mappings=tuple(mapping for mapping, _ in server_tool_call_tasks),
+                    )
 
         elif (
             not images_to_check
@@ -386,6 +418,22 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                         # Replace the tool call with the guardrailed version
                         message_tool_calls[tool_call_idx] = guardrailed_tool_call
 
+    @staticmethod
+    def _apply_guardrail_responses_to_input_server_tool_calls(
+        messages: Sequence[object],
+        guardrailed_tool_calls: Sequence[object],
+        task_mappings: Sequence[tuple[int, int]],
+    ) -> None:
+        guardrailed_by_task: Final = MappingProxyType(dict(zip(task_mappings, guardrailed_tool_calls)))
+        for msg_idx in sorted(frozenset(msg_idx for msg_idx, _ in task_mappings)):
+            write_request_server_tool_calls(
+                messages[msg_idx],
+                tuple(
+                    _rewritten_server_tool_call(server_tool_call, guardrailed_by_task.get((msg_idx, server_idx)))
+                    for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(messages[msg_idx]))
+                ),
+            )
+
     async def process_output_response(
         self,
         response: "ModelResponse",
@@ -436,9 +484,14 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                 text_task_mappings=text_task_mappings,
                 tool_call_task_mappings=tool_call_task_mappings,
             )
+        server_tool_call_tasks: Final = tuple(_server_tool_call_tasks(response.choices))
+        tool_calls_with_server_calls: Final = [
+            *tool_calls_to_check,
+            *(_server_tool_call_input(call) for _, call in server_tool_call_tasks),
+        ]
 
         # Step 2: Apply guardrail to all texts and tool calls in batch
-        if texts_to_check or tool_calls_to_check:
+        if texts_to_check or tool_calls_with_server_calls:
             # Use the real request_data if provided (proxy path), otherwise
             # create a standalone dict (SDK / direct-call path).
             if request_data is None:
@@ -452,8 +505,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
             inputs: Final = GenericGuardrailAPIInputs(texts=texts_to_check)
             if images_to_check:
                 inputs["images"] = images_to_check
-            if tool_calls_to_check:
-                inputs["tool_calls"] = tool_calls_to_check
+            if tool_calls_with_server_calls:
+                inputs["tool_calls"] = tool_calls_with_server_calls
             # Include model information from the response if available
             if hasattr(response, "model") and response.model:
                 inputs["model"] = response.model
@@ -469,8 +522,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
             returned_tool_calls: Final = guardrailed_inputs.get("tool_calls")
             guardrailed_tool_calls: Final[list[dict[str, object]]] = (
                 cast(list[dict[str, object]], returned_tool_calls)
-                if isinstance(returned_tool_calls, list) and len(returned_tool_calls) == len(tool_calls_to_check)
-                else tool_calls_to_check
+                if isinstance(returned_tool_calls, list)
+                and len(returned_tool_calls) == len(tool_calls_with_server_calls)
+                else tool_calls_with_server_calls
             )
 
             # Step 3: Map guardrail responses back to original response structure
@@ -488,6 +542,11 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     tool_calls=guardrailed_tool_calls,
                     task_mappings=tool_call_task_mappings,
                 )
+                self._apply_guardrail_responses_to_output_server_tool_calls(
+                    response=response,
+                    guardrailed_tool_calls=guardrailed_tool_calls[len(tool_calls_to_check) :],
+                    task_mappings=tuple(mapping for mapping, _ in server_tool_call_tasks),
+                )
 
         verbose_proxy_logger.debug("OpenAI Chat Completions: Processed output response: %s", response)
 
@@ -681,6 +740,10 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
             pre_guardrail_tool_calls=pre_guardrail_tool_calls,
             guardrail_name=guardrail_to_apply.guardrail_name or "unknown",
         )
+        self._write_ended_stream_server_tool_call_rewrites(
+            responses_so_far=responses_so_far,
+            guardrailed_response=model_response,
+        )
 
     @staticmethod
     def _rebuild_ended_stream_per_choice(
@@ -934,6 +997,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     if choice.message.tool_calls and isinstance(choice.message.tool_calls, list):
                         if len(choice.message.tool_calls) > 0:
                             return True
+                    if anthropic_server_tool_call_params(choice.message):
+                        return True
         elif isinstance(response, ModelResponseStream):
             for streaming_choice in response.choices:
                 if isinstance(streaming_choice, litellm.StreamingChoices):
@@ -944,6 +1009,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     if streaming_choice.delta.tool_calls and isinstance(streaming_choice.delta.tool_calls, list):
                         if len(streaming_choice.delta.tool_calls) > 0:
                             return True
+                    if anthropic_server_tool_call_params(streaming_choice.delta):
+                        return True
         return False
 
     def _extract_output_text_images_and_tool_calls(
@@ -1094,6 +1161,22 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                             if "name" in func_dict:
                                 existing_tool_call.function.name = func_dict["name"]
 
+    @staticmethod
+    def _apply_guardrail_responses_to_output_server_tool_calls(
+        response: "ModelResponse",
+        guardrailed_tool_calls: Sequence[object],
+        task_mappings: Sequence[tuple[int, int]],
+    ) -> None:
+        guardrailed_by_task: Final = MappingProxyType(dict(zip(task_mappings, guardrailed_tool_calls)))
+        for choice_idx, choice in enumerate(response.choices):
+            write_anthropic_server_tool_calls(
+                choice.message,
+                tuple(
+                    _rewritten_server_tool_call(server_tool_call, guardrailed_by_task.get((choice_idx, server_idx)))
+                    for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(choice.message))
+                ),
+            )
+
     @staticmethod
     def _string_choice_contents(response: "ModelResponse") -> tuple[str | None, ...]:
         return tuple(
@@ -1206,6 +1289,42 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
             for fragment in tail:
                 fragment.function.arguments = ""
 
+    @staticmethod
+    def _write_ended_stream_server_tool_call_rewrites(
+        responses_so_far: Sequence["ModelResponseStream"],
+        guardrailed_response: "ModelResponse",
+    ) -> None:
+        guardrailed_by_id: Final = MappingProxyType(
+            {
+                (guardrailed_response.choices[choice_position].index, server_tool_call["id"]): server_tool_call
+                for (choice_position, _), server_tool_call in _server_tool_call_tasks(guardrailed_response.choices)
+            }
+        )
+        if not guardrailed_by_id:
+            return
+        original_by_id: Final = MappingProxyType(dict(_streamed_server_tool_calls(responses_so_far)))
+        changed_by_choice: Final = MappingProxyType(
+            {
+                choice_index: changed_server_tool_call_arguments(
+                    _choice_server_tool_calls(original_by_id, choice_index),
+                    _choice_server_tool_calls(guardrailed_by_id, choice_index),
+                )
+                for choice_index in frozenset(index for index, _ in guardrailed_by_id)
+            }
+        )
+        for response in responses_so_far:
+            for choice in response.choices:
+                write_anthropic_server_tool_calls(
+                    choice.delta,
+                    tuple(
+                        guardrailed_by_id.get((choice.index, server_tool_call["id"]), server_tool_call)
+                        for server_tool_call in anthropic_server_tool_call_params(choice.delta)
+                    ),
+                )
+                write_anthropic_hosted_web_call_arguments(
+                    choice.delta, changed_by_choice.get(choice.index, MappingProxyType({}))
+                )
+
     async def _apply_guardrail_responses_to_output_streaming(
         self,
         responses: list["ModelResponseStream"],
@@ -1395,7 +1514,76 @@ def _stream_chunk_choices(item: object) -> Sequence[object]:
 def _streamed_delta_tool_calls(delta: object) -> tuple[object, ...]:
     function_call: Final = stream_item_field(delta, "function_call")
     legacy: Final = () if function_call is None else (function_call,)
-    return stream_item_items(delta, "tool_calls") + legacy
+    return anthropic_server_tool_call_params(delta) + stream_item_items(delta, "tool_calls") + legacy
+
+
+def _input_server_tool_call_tasks(
+    messages: Sequence[Mapping[str, object]],
+    *,
+    skip_system_message: bool,
+    skip_tool_message: bool,
+    scan_only_tool_results: bool,
+) -> Iterator[tuple[tuple[int, int], ChatCompletionAssistantToolCall]]:
+    for msg_idx, message in enumerate(messages):
+        if role_out_of_guardrail_scope(
+            str(message.get("role") or "").lower(),
+            skip_system_message=skip_system_message,
+            skip_tool_message=skip_tool_message,
+            scan_only_tool_results=scan_only_tool_results,
+        ):
+            continue
+        for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(message)):
+            yield (msg_idx, server_idx), server_tool_call
+
+
+def _server_tool_call_tasks(
+    choices: Sequence[object],
+) -> Iterator[tuple[tuple[int, int], ChatCompletionAssistantToolCall]]:
+    for choice_idx, choice in enumerate(choices):
+        if not isinstance(choice, Choices):
+            continue
+        for server_idx, server_tool_call in enumerate(anthropic_server_tool_call_params(choice.message)):
+            yield (choice_idx, server_idx), server_tool_call
+
+
+def _streamed_server_tool_calls(
+    responses: Sequence["ModelResponseStream"],
+) -> Iterator[tuple[tuple[int, str | None], ChatCompletionAssistantToolCall]]:
+    for response in responses:
+        for choice in response.choices:
+            for server_tool_call in anthropic_server_tool_call_params(choice.delta):
+                yield (choice.index, server_tool_call["id"]), server_tool_call
+
+
+def _choice_server_tool_calls(
+    calls_by_id: Mapping[tuple[int, str | None], ChatCompletionAssistantToolCall], choice_index: int
+) -> tuple[ChatCompletionAssistantToolCall, ...]:
+    return tuple(call for (index, _), call in calls_by_id.items() if index == choice_index)
+
+
+def _server_tool_call_input(server_tool_call: ChatCompletionAssistantToolCall) -> dict[str, object]:
+    return {
+        "id": server_tool_call["id"],
+        "type": server_tool_call["type"],
+        "function": dict(server_tool_call["function"]),
+    }
+
+
+_TOOL_CALL_FUNCTION_ADAPTER: Final = TypeAdapter(ChatCompletionToolCallFunctionChunk)
+
+
+def _rewritten_server_tool_call(
+    server_tool_call: ChatCompletionAssistantToolCall, guardrailed: object
+) -> ChatCompletionAssistantToolCall:
+    try:
+        function: Final = _TOOL_CALL_FUNCTION_ADAPTER.validate_python(stream_item_field(guardrailed, "function"))
+    except ValidationError:
+        return server_tool_call
+    return ChatCompletionAssistantToolCall(
+        id=server_tool_call["id"],
+        type=server_tool_call["type"],
+        function={**server_tool_call["function"], **function},
+    )
 
 
 def _released_choices(responses_so_far: Sequence[object]) -> Iterator[object]:

litellm/llms/openai/responses/guardrail_translation/handler.py

@@ -71,11 +71,14 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
     stream_item_fingerprint,
     stream_item_items,
     unappliable_request_rewrite,
+    web_search_call,
+    with_web_search_call_arguments,
 )
 from litellm.llms.openai.responses.guardrail_translation.tool_merge import (
     append_returned_tools,
     merge_guardrailed_tools,
 )
+from litellm.responses.additional_tools import additional_tools_in_input
 from litellm.responses.litellm_completion_transformation.compaction_items import (
     compaction_summary,
     decode_compaction_item,
@@ -86,12 +89,14 @@ from litellm.responses.litellm_completion_transformation.tool_search import is_s
 from litellm.responses.litellm_completion_transformation.transformation import (
     TOOL_CALL_OUTPUT_ITEM_TYPES,
     LiteLLMCompletionResponsesConfig,
+    claude_web_search_call_tool_call,
     tool_output_to_chat_content,
 )
 from litellm.types.llms.base import LiteLLMBaseModel
 from litellm.types.llms.openai import (
     AllMessageValues,
     BaseLiteLLMOpenAIResponseObject,
+    ChatCompletionAssistantToolCall,
     ChatCompletionToolCallChunk,
     ChatCompletionToolParam,
     ContentPartAddedEvent,
@@ -266,6 +271,8 @@ _TERMINAL_ENVELOPE_EVENT_TYPES: Final = frozenset(
 
 _TOOL_CALL_ITEM_TYPES: Final = frozenset({"function_call", "custom_tool_call"})
 _TOOL_SEARCH_CALL_TYPE: Final = "tool_search_call"
+_WEB_SEARCH_CALL_TYPE: Final = "web_search_call"
+_ITEM_ID_KEYED_TOOL_CALL_TYPES: Final = frozenset({_TOOL_SEARCH_CALL_TYPE, _WEB_SEARCH_CALL_TYPE})
 _OUTPUT_TOOL_CALL_ITEM_TYPES: Final = _TOOL_CALL_ITEM_TYPES | {_TOOL_SEARCH_CALL_TYPE}
 _SCANNED_TOOL_SEARCH_NAME: Final = "tool_search"
 _TOOL_SEARCH_ARGUMENTS_ADAPTER: Final = TypeAdapter(dict[str, object])
@@ -335,8 +342,27 @@ def _is_tool_call_item(item: object) -> bool:
     return isinstance(item, Mapping) and item.get("type") in _TOOL_CALL_ITEM_TYPES
 
 
+def _claude_web_search_tool_call(item: object) -> ChatCompletionAssistantToolCall | None:
+    search: Final = web_search_call(item)
+    return None if search is None else claude_web_search_call_tool_call(search)
+
+
+def _claude_web_tool_call(item: object, index: int) -> ChatCompletionToolCallChunk | None:
+    tool_call: Final = _claude_web_search_tool_call(item)
+    if tool_call is None:
+        return None
+    return ChatCompletionToolCallChunk(id=tool_call["id"], type="function", function=tool_call["function"], index=index)
+
+
+def _is_output_tool_call_item(item: object) -> bool:
+    return (
+        stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
+        or _claude_web_search_tool_call(item) is not None
+    )
+
+
 def _tool_call_output_item_mapping(item: object) -> Mapping[str, object] | None:
-    if stream_item_field(item, "type") not in _OUTPUT_TOOL_CALL_ITEM_TYPES:
+    if not _is_output_tool_call_item(item):
         return None
     if isinstance(item, Mapping):
         return cast("Mapping[str, object]", item)  # cast-ok: output items are str-keyed JSON objects
@@ -348,7 +374,7 @@ def _tool_call_item_key(item: object) -> str | None:
     if isinstance(call_id, str) and call_id:
         return call_id
     item_id: Final = stream_item_field(item, "id")
-    if stream_item_field(item, "type") == _TOOL_SEARCH_CALL_TYPE and isinstance(item_id, str) and item_id:
+    if stream_item_field(item, "type") in _ITEM_ID_KEYED_TOOL_CALL_TYPES and isinstance(item_id, str) and item_id:
         return item_id
     return None
 
@@ -368,6 +394,17 @@ def _scanned_tool_call_item(item: Mapping[str, object]) -> Mapping[str, object]:
     return {**item, "name": _SCANNED_TOOL_SEARCH_NAME, "arguments": json.dumps(item.get("arguments") or {})}
 
 
+def _scanned_tool_call(item: Mapping[str, object], index: int) -> ChatCompletionToolCallChunk:
+    web_tool_call: Final = _claude_web_tool_call(item, index)
+    if web_tool_call is not None:
+        return web_tool_call
+    return tool_call_dict_from_output_item(_scanned_tool_call_item(item), index)
+
+
+def _rewritten_web_search_call(item: object, payload: str | None) -> object:
+    return item if payload is None else with_web_search_call_arguments(item, payload)
+
+
 def _tool_search_rewrite(payload: str) -> Mapping[str, object]:
     try:
         return {"arguments": _TOOL_SEARCH_ARGUMENTS_ADAPTER.validate_json(payload)}
@@ -953,9 +990,14 @@ class OpenAIResponsesHandler(BaseTranslation):
 
     def extract_request_tool_names(self, data: Mapping[str, object]) -> list[str]:
         """Extract tool names from Responses API request (tools[].name for function
-        and custom, tools[].server_label for mcp)."""
+        and custom, tools[].server_label for mcp), including tools carried by
+        ``additional_tools`` input items."""
         names: Final[list[str]] = []
-        for tool in list_field(data, "tools"):
+        request_tools: Final[tuple[object, ...]] = (
+            *list_field(data, "tools"),
+            *additional_tools_in_input(data.get("input")),
+        )
+        for tool in request_tools:
             if not isinstance(tool, dict):
                 continue
             if tool.get("type") in ("function", "custom") and tool.get("name"):
@@ -1322,17 +1364,22 @@ class OpenAIResponsesHandler(BaseTranslation):
         # delivered, so a truncated buffer still reports text undeliverable. #
         # ------------------------------------------------------------------ #
         if final_chunk.get("type") == "response.output_item.done":
-            finished_summary: Final = _output_compaction_summary(stream_item_field(final_chunk, "item"))
+            finished_item: Final = stream_item_field(final_chunk, "item")
+            finished_summary: Final = _output_compaction_summary(finished_item)
             model_response_stream: Final = (
                 OpenAiResponsesToChatCompletionStreamIterator.translate_responses_chunk_to_openai_stream(final_chunk)
             )
-            tool_calls: Final = model_response_stream.choices[0].delta.tool_calls
+            finished_web_tool_call: Final = _claude_web_tool_call(finished_item, 0)
+            tool_calls: Final = [
+                *cast(list[ChatCompletionToolCallChunk], model_response_stream.choices[0].delta.tool_calls or []),
+                *(() if finished_web_tool_call is None else (finished_web_tool_call,)),
+            ]
             if tool_calls or finished_summary is not None:
                 inputs = GenericGuardrailAPIInputs()
                 if finished_summary is not None:
                     inputs["texts"] = [finished_summary]
                 if tool_calls:
-                    inputs["tool_calls"] = cast(list[ChatCompletionToolCallChunk], tool_calls)
+                    inputs["tool_calls"] = tool_calls
                 if hasattr(model_response_stream, "model") and model_response_stream.model:
                     inputs["model"] = model_response_stream.model
                 await guardrail_to_apply.apply_guardrail(
@@ -1629,7 +1676,7 @@ class OpenAIResponsesHandler(BaseTranslation):
         if event_type not in _OUTPUT_ITEM_EVENT_TYPES:
             return None
         item: Final = stream_item_field(event, "item")
-        return _tool_call_item_key(item) if stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES else None
+        return _tool_call_item_key(item) if _is_output_tool_call_item(item) else None
 
     @staticmethod
     def _write_tool_call_item(item: object, name: str | None, payload: str | None) -> object:
@@ -1643,6 +1690,8 @@ class OpenAIResponsesHandler(BaseTranslation):
             return item
         if item_type == _TOOL_SEARCH_CALL_TYPE:
             return _rewritten_tool_search_call(item, payload)
+        if item_type == _WEB_SEARCH_CALL_TYPE:
+            return _rewritten_web_search_call(item, payload)
         if name is not None:
             OpenAIResponsesHandler._write_event_field(item, "name", name)
         if payload is not None and isinstance(item_type, str) and item_type in _TOOL_CALL_PAYLOAD_FIELDS:
@@ -1698,7 +1747,7 @@ class OpenAIResponsesHandler(BaseTranslation):
             stream_item_field(event, "type") in _TOOL_CALL_PAYLOAD_EVENT_TYPES
             or (
                 stream_item_field(event, "type") in _OUTPUT_ITEM_EVENT_TYPES
-                and stream_item_field(stream_item_field(event, "item"), "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
+                and _is_output_tool_call_item(stream_item_field(event, "item"))
             )
             for event in responses_so_far
         )
@@ -1706,9 +1755,7 @@ class OpenAIResponsesHandler(BaseTranslation):
     @staticmethod
     def _completed_response_scan_key(response: object) -> StreamingScanKey:
         output_items: Final = stream_item_items(response, "output")
-        message_items: Final = tuple(
-            item for item in output_items if stream_item_field(item, "type") not in _OUTPUT_TOOL_CALL_ITEM_TYPES
-        )
+        message_items: Final = tuple(item for item in output_items if not _is_output_tool_call_item(item))
         message_parts: Final = chain.from_iterable(stream_item_items(item, "content") for item in message_items)
         part_texts: Final = tuple(
             text for part in message_parts if isinstance(text := stream_item_field(part, "text"), str) and text
@@ -1716,11 +1763,7 @@ class OpenAIResponsesHandler(BaseTranslation):
         summaries: Final = tuple(summary for item in output_items if (summary := _output_compaction_summary(item)))
         return StreamingScanKey(
             texts=(*part_texts, *summaries),
-            tool_calls=tuple(
-                stream_item_fingerprint(item)
-                for item in output_items
-                if stream_item_field(item, "type") in _OUTPUT_TOOL_CALL_ITEM_TYPES
-            ),
+            tool_calls=tuple(stream_item_fingerprint(item) for item in output_items if _is_output_tool_call_item(item)),
             stream_ended=True,
         )
 
@@ -1833,9 +1876,7 @@ class OpenAIResponsesHandler(BaseTranslation):
         tool_call_item: Final = _tool_call_output_item_mapping(output_item)
         if tool_call_item is not None:
             if tool_calls_to_check is not None:
-                tool_calls_to_check.append(
-                    tool_call_dict_from_output_item(_scanned_tool_call_item(tool_call_item), output_idx)
-                )
+                tool_calls_to_check.append(_scanned_tool_call(tool_call_item, output_idx))
             return
 
         summary: Final = _output_compaction_summary(output_item)

litellm/llms/openai/responses/transformation.py

@@ -77,7 +77,9 @@ _NO_TOOL_UPDATE: Final[Mapping[str, object]] = MappingProxyType({})
 _MODEL_FAMILIES_REJECTING_TOP_LEVEL_SCHEMA_COMBINATORS: Final = ("gpt-4", "gpt-3.5", "chatgpt-4o", "o1", "o3", "o4")
 _PROVIDERS_WITH_OPENAI_SCHEMA_VALIDATOR: Final = frozenset({LlmProviders.AZURE, LlmProviders.OPENAI})
 _PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS: Final = frozenset({LlmProviders.AZURE, LlmProviders.OPENAI})
-_PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING: Final = _PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS
+_PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING: Final = frozenset(
+    {LlmProviders.AZURE, LlmProviders.BEDROCK, LlmProviders.BEDROCK_MANTLE, LlmProviders.OPENAI}
+)
 _ROLES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"system", "developer", "user"})
 _PART_TYPES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"input_text", "input_image", "input_file"})
 _INPUT_PART_LIST_KEYS: Final = frozenset({"content", "output"})

litellm/llms/vertex_ai/text_to_speech/transformation.py

@@ -55,17 +55,45 @@ _LyriaVoice: TypeAlias = str | dict | None
 _JSON_OBJECT: Final = TypeAdapter(Mapping[str, object], config=ConfigDict(hide_input_in_errors=True))
 _JSON_ARRAY: Final = TypeAdapter(tuple[object, ...], config=ConfigDict(hide_input_in_errors=True))
 _STR: Final = TypeAdapter(str, config=ConfigDict(hide_input_in_errors=True))
-_GEMINI_CLOUD_TTS_LOCATIONS: Final = MappingProxyType(
+_CLOUD_TTS_LOCATIONS: Final = MappingProxyType(
     {
         **dict.fromkeys(
-            ("us", "us-central1", "us-east1", "us-east4", "us-east5", "us-south1", "us-west1", "us-west4"), "us"
+            (
+                "us",
+                "us-central1",
+                "us-east1",
+                "us-east4",
+                "us-east5",
+                "us-south1",
+                "us-west1",
+                "us-west2",
+                "us-west3",
+                "us-west4",
+                "us-west5",
+            ),
+            "us",
         ),
         **dict.fromkeys(
-            ("eu", "europe-central2", "europe-north1", "europe-southwest1", "europe-west1", "europe-west4"), "eu"
+            (
+                "eu",
+                "europe-central2",
+                "europe-north1",
+                "europe-north2",
+                "europe-southwest1",
+                "europe-west1",
+                "europe-west3",
+                "europe-west4",
+                "europe-west8",
+                "europe-west9",
+                "europe-west10",
+                "europe-west12",
+            ),
+            "eu",
         ),
         "northamerica-northeast1": "northamerica-northeast1",
     }
 )
+_CLOUD_TTS_MULTI_REGIONS: Final = frozenset(("us", "eu"))
 
 
 _DEFAULT_GEMINI_TTS_VOICE: Final = "Kore"
@@ -563,7 +591,11 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
         # Resolve Vertex AI credentials using VertexBase helpers
         vertex_credentials: Final = self.safe_get_vertex_ai_credentials(litellm_params_dict)
         vertex_project: Final = self.safe_get_vertex_ai_project(litellm_params_dict)
-        vertex_location: Final = self.safe_get_vertex_ai_location(litellm_params_dict)
+        vertex_location: Final = (
+            self.safe_get_vertex_ai_location(litellm_params_dict)
+            if self._is_gemini_tts_model(model)
+            else self.explicit_vertex_ai_location(litellm_params_dict)
+        )
 
         mapped_voice, mapped_params = (
             self.map_openai_params(
@@ -732,7 +764,11 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
         if api_base:
             return api_base
 
-        return self.TTS_API_URL
+        location: Final = _JSON_OBJECT.validate_python(litellm_params).get("vertex_location")
+        cloud_tts_location: Final = _CLOUD_TTS_LOCATIONS.get(location) if isinstance(location, str) else None
+        if cloud_tts_location not in _CLOUD_TTS_MULTI_REGIONS:
+            return self.TTS_API_URL
+        return f"https://{cloud_tts_location}-texttospeech.googleapis.com/v1/text:synthesize"
 
     def _get_gemini_tts_url(self, model: str, api_base: str | None, litellm_params: Mapping[str, object]) -> str:
         if api_base:
@@ -748,7 +784,7 @@ class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
         vertex_location: Final = location if isinstance(location, str) else self.get_default_vertex_location()
         if vertex_location == "global":
             return self.TTS_API_URL
-        cloud_tts_location: Final = _GEMINI_CLOUD_TTS_LOCATIONS.get(vertex_location)
+        cloud_tts_location: Final = _CLOUD_TTS_LOCATIONS.get(vertex_location)
         if cloud_tts_location is None:
             raise UnsupportedParamsError(
                 message=(

litellm/main.py

@@ -4393,6 +4393,14 @@ def _converse_native_output_fallback(
     return None if AmazonConverseConfig.sends_native_text_format(fallback.optional_params) else fallback
 
 
+def _buffered_stream_fallback(
+    first_attempt_params: Mapping[str, object], first_attempt_litellm_params: Mapping[str, object]
+) -> NativeOutputFallback:
+    return NativeOutputFallback(
+        optional_params={**first_attempt_params, "fake_stream": True}, litellm_params=first_attempt_litellm_params
+    )
+
+
 def _native_output_attempt_timeout(
     ctx: CompletionDispatchContext, fallback: NativeOutputFallback | None
 ) -> float | str | httpx.Timeout | None:
@@ -4496,11 +4504,14 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
             )
 
         converse_params: Final = _OPTIONAL_PARAMS_ADAPTER.validate_python(optional_params)
+        converse_uses_native_output: Final = AmazonConverseConfig().uses_native_output(
+            converse_model, converse_params, litellm_params
+        )
         response = call_with_native_output_fallback(
             converse_attempt,
             (
                 partial(_converse_native_output_fallback, ctx, converse_params, dict(litellm_params))
-                if AmazonConverseConfig().uses_native_output(converse_model, converse_params, litellm_params)
+                if converse_uses_native_output
                 else None
             ),
             converse_model,
@@ -4509,6 +4520,11 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
                 if drops_unsupported_params(litellm_params.get("drop_params"))
                 else None
             ),
+            buffered_stream_fallback=(
+                partial(_buffered_stream_fallback, converse_params, dict(litellm_params))
+                if converse_uses_native_output
+                else None
+            ),
         )
     elif bedrock_route == "converse_like":
         model = model.replace("converse_like/", "")
@@ -4551,15 +4567,20 @@ def _complete_bedrock(ctx: CompletionDispatchContext) -> _CompletionDispatchResu
             )
 
         invoke_params: Final = _OPTIONAL_PARAMS_ADAPTER.validate_python(optional_params)
+        invoke_uses_native_output: Final = isinstance(
+            provider_config, AmazonAnthropicClaudeConfig
+        ) and provider_config.uses_native_output(model, invoke_params, litellm_params)
         response = call_with_native_output_fallback(
             invoke_attempt,
-            (
-                partial(_native_output_fallback, ctx, invoke_params, dict(litellm_params))
-                if isinstance(provider_config, AmazonAnthropicClaudeConfig)
-                and provider_config.uses_native_output(model, invoke_params, litellm_params)
+            partial(_native_output_fallback, ctx, invoke_params, dict(litellm_params))
+            if invoke_uses_native_output
+            else None,
+            model,
+            buffered_stream_fallback=(
+                partial(_buffered_stream_fallback, invoke_params, dict(litellm_params))
+                if invoke_uses_native_output
                 else None
             ),
-            model,
         )
 
     return response

litellm/model_latest_aliases.json

@@ -0,0 +1,21 @@
+{
+    "openai": {
+        "gpt-luna-latest": "gpt-6-luna",
+        "gpt-sol-latest": "gpt-6.1-sol",
+        "gpt-terra-latest": "gpt-6.1-terra",
+        "gpt-astra-latest": "gpt-6-astra",
+        "gpt-mini-latest": "gpt-5.4-mini"
+    },
+    "anthropic": {
+        "claude-opus-latest": "claude-opus-5-5",
+        "claude-sonnet-latest": "claude-sonnet-5-5",
+        "claude-haiku-latest": "claude-haiku-5-5",
+        "claude-fable-latest": "claude-fable-5-1"
+    },
+    "xai": {"grok-latest": "grok-4.7"},
+    "moonshot": {"kimi-latest": "kimi-k3"},
+    "zai": {"glm-latest": "glm-5.3", "glm-flash-latest": "glm-5.3-flash"},
+    "deepseek": {"deepseek-pro-latest": "deepseek-v4-pro", "deepseek-flash-latest": "deepseek-v4-flash"},
+    "dashscope": {"qwen-max-latest": "qwen3.8-max", "qwen-flash-latest": "qwen3.8-flash"},
+    "minimax": {"minimax-latest": "MiniMax-M3"}
+}

litellm/model_prices_and_context_window_backup.json

@@ -1961,6 +1961,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "bedrock_output_config_effort_ceiling": "xhigh",
@@ -2041,6 +2042,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "bedrock_output_config_effort_ceiling": "xhigh",
@@ -2116,6 +2118,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "bedrock_output_config_effort_ceiling": "xhigh",
@@ -2242,7 +2245,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2287,6 +2291,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2331,7 +2336,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2376,6 +2382,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2420,7 +2427,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2465,6 +2473,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2504,7 +2513,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2549,6 +2559,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2593,7 +2604,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2633,6 +2645,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -2715,7 +2728,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -2759,7 +2773,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -2803,7 +2818,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -2847,7 +2863,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -2891,7 +2908,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -3023,6 +3041,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -3068,6 +3087,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -3113,6 +3133,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -3158,6 +3179,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -3203,6 +3225,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -12097,16 +12120,19 @@
     },
     "azure/tts-1": {
         "deprecation_date": "2026-12-15",
+        "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
         "input_cost_per_character": 1.5e-05,
         "litellm_provider": "azure",
-        "mode": "audio_speech"
+        "mode": "audio_speech",
+        "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
     },
     "azure/tts-1-hd": {
         "deprecation_date": "2026-12-15",
+        "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
         "input_cost_per_character": 3.3e-05,
         "litellm_provider": "azure",
         "mode": "audio_speech",
-        "source": "https://azure.microsoft.com/en-us/pricing/details/azure-openai/"
+        "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule"
     },
     "azure/us/gpt-4.1-2025-04-14": {
         "deprecation_date": "2027-04-14",
@@ -12859,11 +12885,13 @@
         "supports_vision": true
     },
     "azure/whisper-1": {
-        "deprecation_date": "2026-12-15",
+        "deprecation_date": "2027-02-26",
+        "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
         "input_cost_per_second": 0.0001,
         "litellm_provider": "azure",
         "mode": "audio_transcription",
-        "output_cost_per_second": 0.0001
+        "output_cost_per_second": 0.0001,
+        "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
     },
     "azure_ai/Cohere-embed-v3-english": {
         "deprecation_date": "2027-02-15",
@@ -31630,7 +31658,7 @@
         "input_cost_per_audio_token_batches": 2.5e-07
     },
     "gemini/gemini-embedding-2-preview": {
-        "deprecation_date": "2026-08-10",
+        "deprecation_date": "2026-11-06",
         "deprecation_source": "https://ai.google.dev/gemini-api/docs/deprecations",
         "input_cost_per_audio_token": 6.5e-06,
         "input_cost_per_audio_token_batches": 3.25e-06,
@@ -44089,6 +44117,7 @@
         "supports_vision": true
     },
     "mistral/mistral-large-2512": {
+        "deprecation_date": "2027-04-30",
         "cache_read_input_token_cost": 5e-08,
         "input_cost_per_token": 5e-07,
         "litellm_provider": "mistral",
@@ -44164,6 +44193,7 @@
         "supports_vision": true
     },
     "mistral/mistral-medium-3-5": {
+        "deprecation_date": "2027-04-30",
         "cache_read_input_token_cost": 1.5e-07,
         "input_cost_per_token": 1.5e-06,
         "litellm_provider": "mistral",
@@ -45679,6 +45709,7 @@
     "o1": {
         "cache_read_input_token_cost": 7.5e-06,
         "deprecation_date": "2026-10-23",
+        "deprecation_source": "https://developers.openai.com/api/docs/deprecations",
         "successor_model": "gpt-5.6-sol",
         "input_cost_per_token": 1.5e-05,
         "input_cost_per_token_batches": 7.5e-06,
@@ -64725,6 +64756,7 @@
         "source": "https://docs.mistral.ai/models/model-cards/mistral-moderation-26-03"
     },
     "mistral/voxtral-mini-2602": {
+        "deprecation_date": "2027-04-30",
         "input_cost_per_second": 5e-05,
         "litellm_provider": "mistral",
         "mode": "audio_transcription",
@@ -64741,6 +64773,7 @@
         "supports_audio_input": true
     },
     "mistral/voxtral-mini-transcribe-realtime-2602": {
+        "deprecation_date": "2027-04-30",
         "input_cost_per_second": 0.0001,
         "litellm_provider": "mistral",
         "mode": "audio_transcription",
@@ -64757,6 +64790,7 @@
         "supports_audio_input": true
     },
     "mistral/voxtral-mini-tts-2603": {
+        "deprecation_date": "2027-04-30",
         "input_cost_per_character": 1.6e-05,
         "litellm_provider": "mistral",
         "mode": "audio_speech",
@@ -75543,7 +75577,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -75583,7 +75618,8 @@
         "supports_tool_choice": true,
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
-        "supports_native_structured_output": true,
+        "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -75623,6 +75659,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
         "supports_parallel_tool_use_config": true,
@@ -75696,6 +75733,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -76994,13 +77032,13 @@
         "supports_vision": true
     },
     "azure/whisper": {
-        "deprecation_date": "2026-12-15",
-        "deprecation_source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule",
+        "deprecation_date": "2027-02-26",
+        "deprecation_source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01",
         "input_cost_per_second": 0.0001,
         "litellm_provider": "azure",
         "mode": "audio_transcription",
         "output_cost_per_second": 0.0001,
-        "source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
+        "source": "https://management.azure.com/subscriptions/a88257a5-3323-4ef9-95bf-5681a46c9624/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01"
     },
     "gemini/gemini-nano-banana-2.1": {
         "input_cost_per_token": 1.5e-06,
@@ -77411,6 +77449,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -77468,6 +77507,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -77645,6 +77685,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -77697,6 +77738,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -77744,6 +77786,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,
@@ -77857,6 +77900,7 @@
         "supports_vision": true,
         "supports_xhigh_reasoning_effort": true,
         "supports_native_structured_output": false,
+        "supports_bedrock_native_structured_output_by_default": true,
         "supports_bedrock_native_structured_output": true,
         "supports_max_reasoning_effort": true,
         "supports_output_config": true,

litellm/proxy/_lazy_openapi_snapshot.json

@@ -9633,7 +9633,7 @@
     "paths": {
       "/backend-api/codex/{endpoint}": {
         "get": {
-          "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
+          "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
           "operationId": "codex_passthrough_route_backend_api_codex__endpoint__get",
           "parameters": [
             {
@@ -9681,7 +9681,7 @@
           ]
         },
         "post": {
-          "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
+          "description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
           "operationId": "codex_passthrough_route_backend_api_codex__endpoint__post",
           "parameters": [
             {
@@ -9724,7 +9724,7 @@
                   }
                 }
               },
-              "description": "Unmodified Codex response or server-sent events"
+              "description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events"
             },
             "422": {
               "content": {

litellm/proxy/auth/auth_checks.py

@@ -180,6 +180,7 @@ from litellm.repositories.table_repositories import (
 )
 from litellm.repositories.team_repository import TeamRepository
 from litellm.repositories.user_repository import UserRepository
+from litellm.responses.additional_tools import additional_tools_in_input
 from litellm.router import Router
 from litellm.router_api import RegistryReader
 from litellm.types.proxy.auth.auth_checks import UserNotFoundError
@@ -1037,9 +1038,6 @@ async def check_tools_allowlist(
     call_types: Final = get_call_types_for_route(route)
     if not call_types or not any(ct.value in TOOL_CAPABLE_CALL_TYPES for ct in call_types):
         return
-    tool_names: Final = extract_request_tool_names(route, request_body)
-    if not tool_names:
-        return
     key_meta: Final = (valid_token.metadata or {}) if isinstance(valid_token.metadata, dict) else {}
     team_meta: Final = (valid_token.team_metadata or {}) if isinstance(valid_token.team_metadata, dict) else {}
     key_allowed: Final = key_meta.get("allowed_tools")
@@ -1047,6 +1045,9 @@ async def check_tools_allowlist(
     effective: Final = key_allowed if (isinstance(key_allowed, list) and len(key_allowed) > 0) else team_allowed
     if not isinstance(effective, list) or len(effective) == 0:
         return
+    tool_names: Final = extract_request_tool_names(route, request_body)
+    if not tool_names:
+        return
     allowed_set: Final = {str(t) for t in effective}
     disallowed: Final = [n for n in tool_names if n not in allowed_set]
     if disallowed:
@@ -7988,6 +7989,10 @@ def _tools(tools: object) -> tuple[object, ...]:
         return ()
 
 
+def _request_tools(request_body: Mapping[str, object]) -> tuple[object, ...]:
+    return (*_tools(request_body.get("tools")), *additional_tools_in_input(request_body.get("input")))
+
+
 def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
     try:
         tool_fields: Final = _TOOL_ADAPTER.validate_python(tool, strict=True)
@@ -7998,14 +8003,14 @@ def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
 
 def _strict_requested_vector_store_ids(request_body: Mapping[str, object]) -> tuple[str, ...]:
     """
-    Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, but a vector_store_ids that is
-    not a list of strings is a 400 instead of being skipped or iterated, and tools that are not
-    objects name no store.
+    Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, plus the tools of `additional_tools`
+    input items, but a vector_store_ids that is not a list of strings is a 400 instead of being skipped or
+    iterated, and tools that are not objects name no store.
     """
     return tuple(
         chain(
             _validated_vector_store_ids(request_body.get("vector_store_ids")),
-            chain.from_iterable(_tool_vector_store_ids(tool) for tool in _tools(request_body.get("tools"))),
+            chain.from_iterable(_tool_vector_store_ids(tool) for tool in _request_tools(request_body)),
         )
     )
 
@@ -8044,7 +8049,7 @@ def _legacy_requested_vector_store_ids(
     return tuple(
         chain(
             top_level,
-            chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _tools(request_body.get("tools"))),
+            chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _request_tools(request_body)),
         )
     )
 
@@ -8113,8 +8118,8 @@ async def vector_store_access_check(
     """
     Checks whether the caller may use every vector store the request names.
 
-    Requested stores come from `vector_store_ids`, `tools[].vector_store_ids` and the RAG
-    `retrieval_config.vector_store_id`. Grants come from each identity's
+    Requested stores come from `vector_store_ids`, `tools[].vector_store_ids`, the tools of
+    `additional_tools` input items and the RAG `retrieval_config.vector_store_id`. Grants come from each identity's
     `object_permission.vector_stores`.
 
     Both modes read the same request fields, with or without a vector store registry, and answer 400

litellm/proxy/auth/auth_utils.py

@@ -863,7 +863,7 @@ async def pre_db_read_auth_checks(
     Raises:
     - HTTPException if request fails initial auth checks
     """
-    from litellm.proxy.proxy_server import general_settings, llm_router, premium_user
+    from litellm.proxy.proxy_server import general_settings, llm_router
 
     # Check 1. request size
     await check_if_request_size_is_safe(request=request)
@@ -891,19 +891,12 @@ async def pre_db_read_auth_checks(
         )
 
     # Check 4. Check if request route is an allowed route on the proxy
-    if "allowed_routes" in general_settings:
-        _allowed_routes: Final = general_settings["allowed_routes"]
-        if premium_user is not True:
-            verbose_proxy_logger.error(
-                "Trying to set allowed_routes. This is an Enterprise feature. %s",
-                CommonProxyErrors.not_premium_user.value,
-            )
-        if route not in _allowed_routes:
-            verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, _allowed_routes)
-            raise HTTPException(
-                status_code=status.HTTP_403_FORBIDDEN,
-                detail=f"Access forbidden: Route {route} not allowed",
-            )
+    if not RouteChecks.is_allowed_on_proxy(route):
+        verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, general_settings["allowed_routes"])
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail=f"Access forbidden: Route {route} not allowed",
+        )
 
 
 def route_in_additonal_public_routes(current_route: str):

litellm/proxy/auth/login_throttle.py

@@ -431,6 +431,10 @@ class LoginAttempt:
     throttle: LoginThrottle
     username: str
 
+    async def refuse_if_blocked_while_waiting(self) -> None:
+        """Refuses a login that waited in line while failures ahead of it blocked this key."""
+        await self.throttle.attempt(self.username)
+
     async def succeeded(self) -> None:
         if not self.throttle.enabled:
             return

litellm/proxy/auth/login_utils.py

@@ -5,6 +5,7 @@ This module contains the core login logic that can be reused across different
 login endpoints (e.g., /login and /v2/login).
 """
 
+import asyncio
 import os
 import secrets
 from collections.abc import Mapping
@@ -31,6 +32,7 @@ from litellm.proxy._types import (
 from litellm.proxy.auth.auth_utils import is_sso_provider_fully_configured
 from litellm.proxy.auth.login_throttle import LoginAttempt, LoginThrottle
 from litellm.proxy.auth.password_policy import is_breach_check_enabled, is_password_breached
+from litellm.proxy.common_utils.fips import fips_mode_active
 from litellm.proxy.management_endpoints.internal_user_endpoints import user_update
 from litellm.proxy.management_endpoints.key_management_endpoints import (
     generate_key_helper_fn,
@@ -42,6 +44,7 @@ from litellm.proxy.utils import (
     PrismaClient,
     get_server_root_path,
     hash_password,
+    run_off_loop_if_fips,
     verify_password,
 )
 from litellm.repositories.user_repository import UserRepository
@@ -105,9 +108,21 @@ async def screen_login_password_for_breach(
     return breached
 
 
+_ONE_FIPS_LOGIN_AT_A_TIME: Final = asyncio.Lock()
+
+
+async def _password_matches(password: str, stored: str, attempt: LoginAttempt) -> bool:
+    """Checks inline, or in FIPS mode one login at a time, refusing a login whose user got blocked while it waited."""
+    if not fips_mode_active():
+        return verify_password(password, stored)
+    async with _ONE_FIPS_LOGIN_AT_A_TIME:
+        await attempt.refuse_if_blocked_while_waiting()
+        return await run_off_loop_if_fips(verify_password, password, stored)
+
+
 async def _rehash_password_if_needed(user_id: str, password: str, stored: str) -> None:
     """Rehash legacy password (SHA256) to scrypt on successful login."""
-    if stored.startswith("scrypt:"):
+    if stored.startswith(("scrypt:", "pbkdf2:")):
         return
     from litellm.proxy.proxy_server import prisma_client
 
@@ -411,7 +426,7 @@ async def _sign_in(
                 code=401,
             )
 
-        if verify_password(password, _password):
+        if await _password_matches(password, _password, attempt):
             await _rehash_password_if_needed(_user_row.user_id, password, _password)
             breached_now: Final = prisma_client is not None and await screen_login_password_for_breach(
                 user_id=_user_row.user_id,

litellm/proxy/auth/model_checks.py

@@ -41,7 +41,7 @@ _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields)
 _MODEL_LISTING_POOL: Final = ThreadPoolExecutor(thread_name_prefix="litellm-model-listing")
 
 _PROVIDERS_LISTED_ONLY_THROUGH_A_DEPLOYMENT: Final = frozenset(
-    {LlmProviders.OPENROUTER.value, LlmProviders.HOSTED_VLLM.value}
+    {LlmProviders.OPENROUTER.value, LlmProviders.HOSTED_VLLM.value, LlmProviders.LITELLM_PROXY.value}
 )
 
 _Listed: Final = TypeVar("_Listed")
@@ -83,7 +83,11 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N
         return None
     if ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is None:
         return None
-    return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
+    return get_valid_models(
+        check_provider_endpoint=llm_provider == LlmProviders.LITELLM_PROXY,
+        custom_llm_provider=provider,
+        litellm_params=litellm_params,
+    )
 
 
 def _get_models_from_access_groups(
@@ -188,34 +192,25 @@ def get_team_models(
     - Empty list if no models set
     - If model_access_groups is provided, only return models that are in the access groups
     """
-    all_models_set: Final[set[str]] = set()
-    if len(team_models) > 0:
-        all_models_set.update(team_models)
-        if SpecialModelNames.all_team_models.value in all_models_set:
-            all_models_set.update(team_models)
-            # GH#30619: expand all-team-models sentinel
-            # to the actual proxy model list
-            all_models_set.discard(SpecialModelNames.all_team_models.value)
-            all_models_set.update(proxy_model_list)
-            if include_model_access_groups:
-                all_models_set.update(model_access_groups.keys())
-        if SpecialModelNames.all_proxy_models.value in all_models_set:
-            all_models_set.update(proxy_model_list)
-            if include_model_access_groups:
-                all_models_set.update(model_access_groups.keys())
-
-    all_models = _get_models_from_access_groups(
+    expands_to_proxy_models: Final = (
+        SpecialModelNames.all_team_models.value in team_models
+        or SpecialModelNames.all_proxy_models.value in team_models
+    )
+    expansion: Final = (
+        (*proxy_model_list, *(model_access_groups if include_model_access_groups else ()))
+        if expands_to_proxy_models
+        else ()
+    )
+    named_models: Final = (model for model in team_models if model != SpecialModelNames.all_team_models.value)
+    all_models: Final = _get_models_from_access_groups(
         model_access_groups=model_access_groups,
-        all_models=list(all_models_set),
+        all_models=list(dict.fromkeys((*named_models, *expansion))),
         include_model_access_groups=include_model_access_groups,
         proxy_model_list=proxy_model_list,
     )
 
-    # deduplicate while preserving order
-    all_models = list(dict.fromkeys(all_models))
-
     verbose_proxy_logger.debug("ALL TEAM MODELS - %s", len(all_models))
-    return all_models
+    return list(dict.fromkeys(all_models))
 
 
 def get_complete_model_list(

litellm/proxy/auth/route_checks.py

@@ -439,6 +439,19 @@ class RouteChecks:
                     detail=f"user not allowed to access this route. Route={route} is an admin only route",
                 )
 
+    @staticmethod
+    def is_allowed_on_proxy(route: str) -> bool:
+        from litellm.proxy.proxy_server import general_settings, premium_user
+
+        if "allowed_routes" not in general_settings:
+            return True
+        if premium_user is not True:
+            verbose_proxy_logger.error(
+                "Trying to set allowed_routes. This is an Enterprise feature. %s",
+                CommonProxyErrors.not_premium_user.value,
+            )
+        return route in general_settings["allowed_routes"]
+
     @staticmethod
     def is_unbilled_route(route: str) -> bool:
         resource, _, action = route.rsplit("/", 1)[-1].partition(":")

litellm/proxy/auth/user_api_key_auth.py

@@ -309,6 +309,17 @@ async def _normalize_claude_model(
     rewrite_request_model(request_data, request, source)
 
 
+_CODEX_NATIVE_RESPONSES_ROUTE: Final = "/backend-api/codex/responses"
+
+
+def _is_codex_native_litellm_turn(route: str, model: object, llm_router: litellm.Router, team_id: str | None) -> bool:
+    if route != _CODEX_NATIVE_RESPONSES_ROUTE:
+        return False
+    from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import routes_to_litellm
+
+    return routes_to_litellm(model, llm_router, team_id)
+
+
 async def _resolve_router_settings_model_group_alias(
     request_data: dict[str, object],  # mutable-ok: the request body is rewritten in place for every downstream reader
     valid_token: UserAPIKeyAuth,
@@ -325,9 +336,12 @@ async def _resolve_router_settings_model_group_alias(
     if request.scope.get(MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY) is True:
         return
     request.scope[MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY] = True
-    if request_dispatched_to_pass_through_endpoint(request) or request_dispatched_to_provider_pass_through(request):
-        return
     requested: Final = request_data.get("model")
+    if request_dispatched_to_pass_through_endpoint(request) or (
+        request_dispatched_to_provider_pass_through(request)
+        and not _is_codex_native_litellm_turn(route, requested, llm_router, valid_token.team_id)
+    ):
+        return
     if not isinstance(requested, str) or await read_raw_json_body(request=request) is None:
         return
     settings: Final = await proxy_config.get_hierarchical_router_settings(
@@ -338,6 +352,11 @@ async def _resolve_router_settings_model_group_alias(
     target: Final = resolve_model_group_alias(settings.get("model_group_alias"), requested)
     if target is None or target == requested:
         return
+    if request_dispatched_to_provider_pass_through(request):
+        from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_turn_serves_through_litellm
+
+        if not await native_turn_serves_through_litellm(requested, llm_router, request, valid_token):
+            return
     verbose_proxy_logger.debug(
         "router_settings.model_group_alias resolved %s -> %s before auth",
         requested.replace("\r", "").replace("\n", ""),
@@ -347,6 +366,27 @@ async def _resolve_router_settings_model_group_alias(
     rewrite_request_model(request_data, request, target)
 
 
+def _resolve_codex_native_chatgpt_model(
+    request_data: dict[str, object],  # mutable-ok: the request body is rewritten in place for every downstream reader
+    valid_token: UserAPIKeyAuth,
+    request: Request | None,
+    route: str,
+) -> None:
+    requested: Final = request_data.get("model")
+    if route != _CODEX_NATIVE_RESPONSES_ROUTE or request is None or not isinstance(requested, str):
+        return
+    from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_chatgpt_model
+    from litellm.proxy.proxy_server import llm_router
+
+    if llm_router is None:
+        return
+    target: Final = native_chatgpt_model(requested, llm_router, valid_token)
+    if target is None:
+        return
+    request.scope.setdefault(CLIENT_REQUESTED_MODEL_SCOPE_KEY, requested)
+    rewrite_request_model(request_data, request, target)
+
+
 def _get_model_names_for_budget_checks(
     model: str | list[str] | None,
 ) -> list[str]:
@@ -4172,6 +4212,7 @@ async def _authorize_authenticated_request(
     RouteChecks.should_call_route(route=route, valid_token=user_api_key_auth_obj, request=request)
     await _normalize_claude_model(request_data, user_api_key_auth_obj, request, route)
     await _resolve_router_settings_model_group_alias(request_data, user_api_key_auth_obj, request, route)
+    _resolve_codex_native_chatgpt_model(request_data, user_api_key_auth_obj, request, route)
 
     # Single authorization point. Builder paths MUST NOT call common_checks.
     # Route through the same exception handler the builder uses so
@@ -4650,6 +4691,7 @@ async def enforce_key_and_fallback_model_access(
     """
     await _normalize_claude_model(request_data, valid_token, request, route)
     await _resolve_router_settings_model_group_alias(request_data, valid_token, request, route)
+    _resolve_codex_native_chatgpt_model(request_data, valid_token, request, route)
     config: Final = valid_token.config
 
     if config != {}:

litellm/proxy/common_utils/codex_model_catalog.py

@@ -24,8 +24,10 @@ a newer tag.
 
 from __future__ import annotations
 
+import hashlib
+import json
 from collections.abc import Callable, Mapping, Sequence
-from dataclasses import dataclass
+from dataclasses import asdict, dataclass
 from functools import cache
 from itertools import accumulate, pairwise
 from pathlib import Path
@@ -33,7 +35,7 @@ from types import MappingProxyType
 from typing import TYPE_CHECKING, Final, Literal, TypeAlias
 
 from packaging.version import Version
-from pydantic import ConfigDict, TypeAdapter, ValidationError
+from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
 
 from litellm._logging import verbose_proxy_logger
 from litellm.proxy.common_utils.codex_model_translation import (
@@ -61,6 +63,8 @@ CODEX_CHAT_MODES: Final = frozenset({"chat", "responses"})
 
 _BODY_PREFIX: Final = '{"models":['
 _BODY_SUFFIX: Final = "]}"
+_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
+_JSON_OBJECTS: Final = TypeAdapter(tuple[dict[str, object], ...])
 _NO_UPDATE: Final[Mapping[str, object]] = MappingProxyType({})
 
 
@@ -119,6 +123,7 @@ class CodexStockCatalog(LiteLLMBaseModel):
 
 CodexCatalogEntry = CodexFallbackModel | CodexStockModel
 CodexAliasDefaults: TypeAlias = Literal["codex", "deployment"]
+_NATIVE_ALIAS_DEFAULTS: Final[CodexAliasDefaults] = "deployment"
 
 
 @dataclass(frozen=True, slots=True)
@@ -408,6 +413,7 @@ def _entry(
         stock_model.upgrade if stock_model.upgrade is not None and stock_model.upgrade.model in served else None
     )
     is_alias: Final = row.id != stock_model.slug
+    deployment_decides: Final = is_alias and alias_defaults == "deployment"
     stock_name: Final = row.id if is_alias else stock_model.display_name
     listing: Final = {
         "slug": row.id,
@@ -417,12 +423,18 @@ def _entry(
         "supported_in_api": True,
         "upgrade": upgrade,
         **({} if search_tool is None else {"supports_search_tool": search_tool}),
-        **(_deployment_defaults(row, stock_model) if is_alias and alias_defaults == "deployment" else {}),
+        **(_deployment_defaults(row, stock_model) if deployment_decides else {}),
         **_configured_window(row, stock_model),
     }
     if configured is None:
         return stock_model.model_copy(update={**_priced_tiers(row, stock_model), **listing})
-    return stock_model.model_copy(update={**listing, **_tier_fields(stock_model, configured)})
+    return stock_model.model_copy(
+        update={
+            **listing,
+            **_tier_fields(stock_model, configured),
+            **({"default_service_tier": None} if deployment_decides else {}),
+        }
+    )
 
 
 def _serialized(entry: CodexCatalogEntry) -> str:
@@ -435,13 +447,12 @@ def _bytes_left_after(left: int, size: int) -> int:
     return left - size if size <= left else left
 
 
-def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]:
-    """Whether each entry, taken in order, fits in what is left of a body of at most `byte_limit`
-    bytes: an entry larger than the bytes left is passed over and a smaller one after it is still
-    taken, each entry counted with its separating comma and the last one's comma given back."""
-    if byte_limit is None:
+def _fits(serialized: Sequence[str], budget: int | None) -> tuple[bool, ...]:
+    """Whether each entry, taken in order, fits in what is left of `budget` bytes: an entry larger
+    than the bytes left is passed over and a smaller one after it is still taken, each entry
+    counted with its separating comma."""
+    if budget is None:
         return (True,) * len(serialized)
-    budget: Final = byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
     left: Final = tuple(
         accumulate((len(entry.encode()) + 1 for entry in serialized), _bytes_left_after, initial=budget)
     )
@@ -449,17 +460,34 @@ def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]
 
 
 def _kept_positions(
-    entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], byte_limit: int | None
+    entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], budget: int | None
 ) -> frozenset[int]:
     """The listing positions that survive the byte limit: an entry offering a service tier is taken
     ahead of one offering none, each group in listing order, since a model without a tier is driven
     by name just as well off Codex's bundled catalog; an entry too large for the bytes left never
     costs the smaller ones after it their place."""
     survival: Final = sorted(range(len(entries)), key=lambda position: (not entries[position].service_tiers, position))
-    fits: Final = _fits(tuple(serialized[position] for position in survival), byte_limit)
+    fits: Final = _fits(tuple(serialized[position] for position in survival), budget)
     return frozenset(position for position, fit in zip(survival, fits, strict=True) if fit)
 
 
+def _entries(
+    drivable: Sequence[CodexCatalogRow],
+    stock: Mapping[str, CodexStockModel] | None,
+    instructions: str | None,
+    first_priority: int,
+    alias_defaults: CodexAliasDefaults,
+    client_version: str | None,
+) -> tuple[CodexCatalogEntry, ...]:
+    known: Final = bundled_codex_models() if stock is None else stock
+    prompt: Final = codex_base_instructions() if instructions is None else instructions
+    served: Final = frozenset(row.id for row in drivable)
+    return tuple(
+        _entry(first_priority + index, row, known, served, prompt, alias_defaults, client_version)
+        for index, row in enumerate(drivable)
+    )
+
+
 @dataclass(frozen=True, slots=True)
 class CodexCatalogBody:
     """A `ModelsResponse` body, the ids it lists, and the chat-capable ids the byte limit left out."""
@@ -495,15 +523,12 @@ def codex_models_response_json(
     requesting Codex's version, which decides the reasoning levels a fallback entry may offer; None counts
     as a version too old for custom levels.
     """
-    known: Final = bundled_codex_models() if stock is None else stock
-    prompt: Final = codex_base_instructions() if instructions is None else instructions
     drivable: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode))
-    served: Final = frozenset(row.id for row in drivable)
-    entries: Final = tuple(
-        _entry(index, row, known, served, prompt, alias_defaults, client_version) for index, row in enumerate(drivable)
-    )
+    entries: Final = _entries(drivable, stock, instructions, 0, alias_defaults, client_version)
     serialized: Final = tuple(_serialized(entry) for entry in entries)
-    kept: Final = _kept_positions(entries, serialized, byte_limit)
+    kept: Final = _kept_positions(
+        entries, serialized, None if byte_limit is None else byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
+    )
     return CodexCatalogBody(
         json=f"{_BODY_PREFIX}{','.join(serialized[position] for position in sorted(kept))}{_BODY_SUFFIX}",
         listed=tuple(row.id for position, row in enumerate(drivable) if position in kept),
@@ -511,6 +536,74 @@ def codex_models_response_json(
     )
 
 
+def _priority(model: Mapping[str, object]) -> int:
+    priority: Final = model.get("priority")
+    return priority if isinstance(priority, int) else -1
+
+
+def _compact_json(value: object) -> str:
+    return json.dumps(value, separators=(",", ":"))
+
+
+def _entry_json(entry: CodexCatalogEntry) -> Mapping[str, object]:
+    return _JSON_OBJECT.validate_python(entry.model_dump(mode="json", exclude_unset=isinstance(entry, CodexStockModel)))
+
+
+def merged_codex_catalog(
+    upstream_body: bytes,
+    rows: Sequence[CodexCatalogRow],
+    *,
+    stock: Mapping[str, CodexStockModel] | None = None,
+    instructions: str | None = None,
+    byte_limit: int = CODEX_CATALOG_BYTE_LIMIT,
+    client_version: str | None = None,
+) -> str | None:
+    try:
+        top: Final = _JSON_OBJECT.validate_json(upstream_body)
+        upstream: Final = _JSON_OBJECTS.validate_python(top.get("models"))
+    except ValidationError:
+        return None
+    slugs: Final = frozenset(model.get("slug") for model in upstream)
+    added: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode) and row.id not in slugs)
+    first_priority: Final = max((_priority(model) for model in upstream), default=-1) + 1
+    entries: Final = _entries(added, stock, instructions, first_priority, _NATIVE_ALIAS_DEFAULTS, client_version)
+    added_models: Final = tuple(_entry_json(entry) for entry in entries)
+    serialized: Final = tuple(_compact_json(model) for model in added_models)
+    budget: Final = byte_limit - len(_compact_json({**top, "models": upstream}))
+    kept: Final = tuple(sorted(_kept_positions(entries, serialized, budget)))
+    if len(kept) < len(entries):
+        verbose_proxy_logger.warning(
+            "Codex native model catalog left out %d of %d LiteLLM models to stay within Codex's %d byte limit",
+            len(entries) - len(kept),
+            len(entries),
+            byte_limit,
+        )
+    if not kept:
+        return None
+    return _compact_json({**top, "models": [*upstream, *(added_models[position] for position in kept)]})
+
+
+def _canonical_value(value: object) -> object:
+    return value.model_dump(mode="json") if isinstance(value, BaseModel) else repr(value)
+
+
+def codex_catalog_etag(upstream_etag: str, rows: Sequence[CodexCatalogRow]) -> str:
+    canonical: Final = json.dumps(
+        [asdict(row) for row in rows], sort_keys=True, separators=(",", ":"), default=_canonical_value
+    )
+    digest: Final = hashlib.sha256(json.dumps([upstream_etag, canonical]).encode()).hexdigest()
+    return f'"{digest}"'
+
+
+def upstream_codex_slugs(upstream_body: bytes) -> frozenset[str] | None:
+    try:
+        upstream: Final = _JSON_OBJECTS.validate_python(_JSON_OBJECT.validate_json(upstream_body).get("models"))
+    except ValidationError:
+        return None
+    slugs: Final = (model.get("slug") for model in upstream)
+    return frozenset(slug for slug in slugs if isinstance(slug, str))
+
+
 def _group_reasoning_efforts(
     row: ModelInfoResponse, lookup_id: str, registry: RouterRegistry | None, team_id: str | None
 ) -> tuple[str, ...] | None:
@@ -520,6 +613,11 @@ def _group_reasoning_efforts(
     return group.supported_reasoning_efforts if group is not None else None
 
 
+def _shared_pinned_effort(pinned: tuple[object, ...]) -> str | None:
+    first: Final = next(iter(pinned), None)
+    return first if isinstance(first, str) and all(effort == first for effort in pinned) else None
+
+
 def deployment_capabilities(model_info: Mapping[str, object] | None, model_id: str) -> CodexDeploymentCapabilities:
     """One deployment's model info, merged with the cost map, as the capabilities a Codex entry describes.
     Model info that does not validate declares nothing, logged, so one bad value never fails the listing."""
@@ -562,6 +660,11 @@ def _catalog_row(
             registry.resolve(lookup_id, kind="upstream_provider", team_id=team_id) if registry is not None else None
         ),
         display_name=registry.group(label_id, field="display_name") if registry is not None else None,
+        reasoning_effort=(
+            _shared_pinned_effort(registry.group(lookup_id, field="pinned_reasoning_efforts", team_id=team_id))
+            if registry is not None
+            else None
+        ),
         service_tiers=registry.group(lookup_id, field="service_tiers", team_id=team_id) if registry is not None else (),
         supported_reasoning_efforts=_group_reasoning_efforts(row, lookup_id, registry, team_id),
         search_tool_support=(
@@ -619,3 +722,21 @@ def codex_model_list_body(
             ", ".join(body.left_out),
         )
     return body.json
+
+
+def codex_entry_names_default_reasoning_level(
+    model_id: str, llm_router: Router, team_id: str | None, client_version: str | None
+) -> bool:
+    """Whether the native catalog entry for `model_id` names a `default_reasoning_level`. Codex's model picker
+    sends reasoning effort `none` for a model whose entry names none, while showing that choice as
+    "default"."""
+    listed: Final[ModelInfoResponse] = {"id": model_id, "object": "model", "created": 0, "owned_by": ""}
+    (entry,) = _entries(
+        (_catalog_row(listed, model_id, model_id, llm_router, team_id),),
+        None,
+        "",
+        0,
+        _NATIVE_ALIAS_DEFAULTS,
+        client_version,
+    )
+    return _entry_json(entry).get("default_reasoning_level") is not None

litellm/proxy/common_utils/codex_model_translation.py

@@ -16,7 +16,7 @@ from types import NoneType
 from typing import Annotated, Final, Literal, TypeAlias
 
 from packaging.version import Version
-from pydantic import ConfigDict, Field, StringConstraints
+from pydantic import ConfigDict, Field, StringConstraints, field_serializer
 
 from litellm.litellm_core_utils.reasoning_effort_strength import nearest_declared_reasoning_effort
 from litellm.types.llms.base_model import LiteLLMBaseModel
@@ -117,6 +117,10 @@ class CodexDeploymentCapabilities(LiteLLMBaseModel):
     prices_tokens: bool = False
     priced_service_tier_keys: frozenset[str] = frozenset()
 
+    @field_serializer("priced_service_tier_keys")
+    def _sorted_priced_service_tier_keys(self, keys: frozenset[str]) -> tuple[str, ...]:
+        return tuple(sorted(keys))
+
 
 @cache
 def tier_price_keys() -> tuple[str, ...]:

litellm/proxy/common_utils/fips.py

@@ -1,3 +1,4 @@
+import threading
 from collections.abc import Mapping
 from typing import Final
 
@@ -5,6 +6,7 @@ from litellm.secret_managers.main import str_to_bool
 
 FIPS_MODE_SETTING: Final = "fips_mode"
 _SETTING: Final = f"general_settings.{FIPS_MODE_SETTING}"
+_FIPS_MODE_ACTIVE: Final = threading.Event()
 
 
 class FipsModeError(Exception):
@@ -87,3 +89,12 @@ def enforce_fips_mode(general_settings: Mapping[str, object]) -> None:
     refusal: Final = fips_mode_refusal(general_settings)
     if refusal is not None:
         raise FipsModeError(refusal)
+    if _fips_mode_setting(general_settings):
+        _FIPS_MODE_ACTIVE.set()
+    else:
+        _FIPS_MODE_ACTIVE.clear()
+
+
+def fips_mode_active() -> bool:
+    """Whether general_settings.fips_mode was on when enforce_fips_mode last accepted it."""
+    return _FIPS_MODE_ACTIVE.is_set()

litellm/proxy/db/spend_log_tool_index.py

@@ -63,11 +63,26 @@ def response_tool_call_names(completion_response: object) -> tuple[str, ...]:
     )
     return tuple(
         stripped
-        for name in chain(called_names, _auto_executed_mcp_tool_names(completion_response))
+        for name in chain(
+            _server_tool_call_names(completion_response),
+            called_names,
+            _auto_executed_mcp_tool_names(completion_response),
+        )
         if isinstance(name, str) and (stripped := name.strip())
     )
 
 
+def _server_tool_call_names(completion_response: object) -> tuple[str | None, ...]:
+    from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
+    from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
+
+    server_tool_calls: Final = chain.from_iterable(
+        anthropic_server_tool_call_params(message_field(choice, "message"))
+        for choice in parts_of(message_field(completion_response, "choices"))
+    )
+    return tuple(server_tool_call["function"].get("name") for server_tool_call in server_tool_calls)
+
+
 class _ExecutedTool(BaseModel):
     model_config = ConfigDict(frozen=True)
 

litellm/proxy/guardrails/_content_utils.py

@@ -18,6 +18,11 @@ from typing import Final
 from pydantic import JsonValue, TypeAdapter
 
 from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    changed_server_tool_call_arguments,
+    write_request_hosted_web_call_arguments,
+)
 from litellm.responses.litellm_completion_transformation.compaction_items import (
     compaction_summary,
     input_items,
@@ -407,18 +412,27 @@ def walk_user_text(
             return new_parts
         return content
 
-    messages: Final = data.get("messages")
-    for message in _object_list(messages) or ():
+    def _rewrite_call_arguments(calls: object) -> None:
+        for call in _object_list(calls) or ():
+            function = call.get("function") if isinstance(call, dict) else None
+            if isinstance(function, dict) and "arguments" in function:
+                function["arguments"] = _rewrite_arguments(function["arguments"])
+
+    messages: Final = _object_list(data.get("messages")) or ()
+    server_tool_calls_before: Final = tuple(anthropic_server_tool_call_params(message) for message in messages)
+    for message in messages:
         if (skip_tool_message and _is_tool_message(message)) or not isinstance(message, dict):
             continue
         if "content" in message:
             message["content"] = _rewrite_content(message["content"])
-        calls = message.get("tool_calls")
-        if isinstance(calls, list):
-            for call in calls:
-                function = call.get("function") if isinstance(call, dict) else None
-                if isinstance(function, dict) and "arguments" in function:
-                    function["arguments"] = _rewrite_arguments(function["arguments"])
+        _rewrite_call_arguments(message.get("tool_calls"))
+        provider_fields = message.get("provider_specific_fields")
+        if isinstance(provider_fields, dict):
+            _rewrite_call_arguments(provider_fields.get("server_tool_calls"))
+    for message, original in zip(messages, server_tool_calls_before, strict=True):
+        write_request_hosted_web_call_arguments(
+            message, changed_server_tool_call_arguments(original, anthropic_server_tool_call_params(message))
+        )
 
     if "system" in data:
         data["system"] = _rewrite_content(data["system"])  # rebind-ok: preserve the native hook's in-place contract

litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py

@@ -9,7 +9,7 @@ import contextlib
 import json
 import os
 import ssl
-from collections.abc import AsyncGenerator, AsyncIterable, Mapping, Sequence
+from collections.abc import AsyncGenerator, AsyncIterable, Iterator, Mapping, Sequence
 from ssl import SSLContext
 from typing import TYPE_CHECKING, Any, Final
 
@@ -23,7 +23,16 @@ from litellm import DualCache
 from litellm._logging import verbose_proxy_logger
 from litellm._version import version as litellm_version
 from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.litellm_core_utils.core_helpers import attribute_items, iter_items
+from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_calls,
+    write_anthropic_server_tool_call_arguments,
+)
+from litellm.llms.base_llm.guardrail_translation.utils import (
+    claude_hosted_web_tool_arguments,
+    with_web_search_call_arguments,
+)
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     get_ssl_configuration,
@@ -515,27 +524,37 @@ class CatoNetworksGuardrail(CustomGuardrail):
     @staticmethod
     def _output_fragments(message: Message) -> Sequence[tuple[tuple[str, int | None], str]]:
         """Assistant text the proxy returns to the caller: ``content`` plus every
-        ``tool_calls[].function.arguments`` string, each tagged with where a
-        redaction must be written back. ``content`` is only included when present
-        so a tool-call-only choice keeps its ``None`` content (the text-vs-tool-call
-        signal downstream consumers rely on) while its arguments are still inspected."""
-        fragments: Final[list] = []
-        if message.content is not None:
-            fragments.append((("content", None), message.content))
-        for idx, tool_call in enumerate(message.tool_calls or []):
-            function = getattr(tool_call, "function", None)
-            arguments = getattr(function, "arguments", None)
-            if isinstance(arguments, str) and arguments:
-                fragments.append((("tool_call", idx), arguments))
-        return fragments
+        tool call's ``function.arguments`` string, Claude's server tool calls first,
+        each tagged with where a redaction must be written back. ``content`` is only
+        included when present so a tool-call-only choice keeps its ``None`` content
+        (the text-vs-tool-call signal downstream consumers rely on) while its
+        arguments are still inspected."""
+        content: Final = () if message.content is None else ((("content", None), message.content),)
+        return (
+            *content,
+            *CatoNetworksGuardrail._tool_call_fragments("server_tool_call", anthropic_server_tool_calls(message)),
+            *CatoNetworksGuardrail._tool_call_fragments("tool_call", message.tool_calls or ()),
+        )
+
+    @staticmethod
+    def _tool_call_fragments(kind: str, tool_calls: Sequence[object]) -> tuple[tuple[tuple[str, int | None], str], ...]:
+        return tuple(
+            ((kind, idx), arguments)
+            for idx, arguments in enumerate(
+                getattr(getattr(tool_call, "function", None), "arguments", None) for tool_call in tool_calls
+            )
+            if isinstance(arguments, str) and arguments
+        )
 
     @staticmethod
     def _apply_output_fragment(message: Any, target: tuple[str, int | None], redacted: str) -> None:
-        kind, idx = target
-        if kind == "content":
-            message.content = redacted
-        else:
-            message.tool_calls[idx].function.arguments = redacted
+        match target:
+            case ("content", _):
+                message.content = redacted
+            case ("server_tool_call", int() as idx):
+                write_anthropic_server_tool_call_arguments(message, idx, redacted)
+            case (_, idx):
+                message.tool_calls[idx].function.arguments = redacted
 
     @staticmethod
     def _responses_output_field(item: object, key: str) -> str | Sequence[object] | None:
@@ -565,12 +584,19 @@ class CatoNetworksGuardrail(CustomGuardrail):
         return fragments
 
     @staticmethod
-    def _apply_responses_output_fragment(container: object, key: str, redacted: str) -> None:
+    def _apply_responses_output_fragment(container: object, key: str, redacted: object) -> None:
         if isinstance(container, dict):
             container[key] = redacted
         else:
             setattr(container, key, redacted)
 
+    @staticmethod
+    def _responses_hosted_web_calls(response: ResponsesAPIResponse) -> Iterator[tuple[object, str]]:
+        for item in attribute_items(response, "output"):
+            hosted_arguments = claude_hosted_web_tool_arguments(item)
+            if hosted_arguments:
+                yield item, hosted_arguments
+
     async def _inspect_output_text(
         self,
         data: dict,
@@ -612,6 +638,12 @@ class CatoNetworksGuardrail(CustomGuardrail):
                 redacted_output = await self._inspect_output_text(data, text, user_api_key_dict, user_email)
                 if redacted_output is not None:
                     self._apply_responses_output_fragment(container, key, redacted_output)
+            for item, hosted_arguments in self._responses_hosted_web_calls(response):
+                redacted_output = await self._inspect_output_text(data, hosted_arguments, user_api_key_dict, user_email)
+                if redacted_output is not None:
+                    self._apply_responses_output_fragment(
+                        item, "action", message_field(with_web_search_call_arguments(item, redacted_output), "action")
+                    )
         return response
 
     async def async_post_call_streaming_iterator_hook(

litellm/proxy/guardrails/guardrail_hooks/cisco_ai_defense/cisco_ai_defense.py

@@ -37,6 +37,16 @@ from litellm.integrations.custom_guardrail import (
     log_guardrail_information,
 )
 from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    tool_calls_with_server_tool_calls,
+    with_server_tool_call_arguments,
+    write_anthropic_server_tool_calls,
+)
+from litellm.llms.base_llm.guardrail_translation.utils import (
+    claude_hosted_web_tool_arguments,
+    with_web_search_call_arguments,
+)
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -48,7 +58,9 @@ from litellm.proxy.common_utils.callback_utils import (
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.utils import (
     Choices,
+    Delta,
     LLMResponseTypes,
+    Message,
     ModelResponse,
     ModelResponseStream,
     TextCompletionResponse,
@@ -1654,6 +1666,7 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
     @classmethod
     def _clear_tool_call_arguments(cls, message: object) -> None:
         """Clear tool-call / function-call arguments after Cisco redaction."""
+        cls._clear_server_tool_call_arguments(message)
         tool_calls = message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
         for tc in tool_calls or []:
             fn = tc.get("function") if isinstance(tc, dict) else getattr(tc, "function", None)
@@ -1663,9 +1676,26 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
         )
         cls._clear_arguments_field(function_call)
 
+    @staticmethod
+    def _clear_server_tool_call_arguments(message: object) -> None:
+        cleared: Final = tuple(
+            with_server_tool_call_arguments(server_tool_call, "{}")
+            for server_tool_call in anthropic_server_tool_call_params(message)
+        )
+        match message:
+            case Message() | Delta():
+                write_anthropic_server_tool_calls(message, cleared)
+            case dict() if cleared:
+                message["provider_specific_fields"] = {  # rebind-ok: a redaction must land on the message it scanned
+                    **(message.get("provider_specific_fields") or {}),
+                    "server_tool_calls": list(cleared),
+                }
+            case _:
+                pass
+
     def _redact_responses_api_output(
         self,
-        output_items: list,
+        output_items: list[object],
         sanitized_text: str | None,
         sanitized_messages: list[dict[str, object]] | None,
     ) -> bool:
@@ -1677,7 +1707,10 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
         if not replacement_text:
             return False
         applied = False
-        for item in output_items:
+        for index, item in enumerate(output_items):
+            if claude_hosted_web_tool_arguments(item):
+                output_items[index] = with_web_search_call_arguments(item, "{}")  # rebind-ok: in-place redaction
+                applied = True
             content = getattr(item, "content", None) or (item.get("content") if isinstance(item, dict) else None)
             if isinstance(content, list):
                 for part in content:
@@ -2038,8 +2071,9 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
         output_items: Final = getattr(response, "output", None)
         if not isinstance(output_items, list):
             return []
+        typed_output_items: Final[Sequence[object]] = output_items
         output_parts: Final[list[str]] = []
-        for item in output_items:
+        for item in typed_output_items:
             get = item.get if isinstance(item, dict) else (lambda k: getattr(item, k, None))
             for part in get("content") or []:
                 pget = part.get if isinstance(part, dict) else (lambda k: getattr(part, k, None))
@@ -2053,6 +2087,8 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
             direct = get("text")
             if isinstance(direct, str) and direct:
                 output_parts.append(direct)
+            if hosted_arguments := claude_hosted_web_tool_arguments(item):
+                output_parts.append(hosted_arguments)
         joined = " ".join(output_parts)
         return [{"role": "assistant", "content": joined}] if joined else []
 
@@ -2095,8 +2131,7 @@ class CiscoAIDefenseGuardrail(CiscoAIDefenseMcpMixin, CustomGuardrail):
     @classmethod
     def _extract_message_tool_argument_parts(cls, message: object) -> list[str]:
         parts: Final[list[str]] = []
-        tool_calls = message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
-        for tool_call in tool_calls or []:
+        for tool_call in tool_calls_with_server_tool_calls(message):
             args = cls._extract_tool_call_arguments(tool_call)
             if args:
                 parts.append(args)

litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py

@@ -22,6 +22,7 @@ from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -163,11 +164,11 @@ class lakeraAI_Moderation(CustomGuardrail):
             for message in data["messages"]:
                 role = message.get("role")
                 if role in stringified_roles:
-                    if "tool_calls" in message:
-                        tool_call_messages = [
-                            *tool_call_messages,
-                            *message["tool_calls"],
-                        ]
+                    tool_call_messages = [
+                        *tool_call_messages,
+                        *anthropic_server_tool_call_params(message),
+                        *(message.get("tool_calls") or ()),
+                    ]
                     if role == Role.SYSTEM.value:  # we need this for later
                         system_message = message
                         continue
@@ -185,10 +186,11 @@ class lakeraAI_Moderation(CustomGuardrail):
             if system_message is not None:
                 if not litellm.add_function_to_prompt:
                     content = system_message.get("content")
-                    function_input: Final = []
-                    for tool_call in tool_call_messages:
-                        if "function" in tool_call:
-                            function_input.append(tool_call["function"]["arguments"])
+                    function_input: Final[tuple[str, ...]] = tuple(
+                        tool_call["function"]["arguments"]
+                        for tool_call in tool_call_messages
+                        if "function" in tool_call
+                    )
 
                     if len(function_input) > 0:
                         content += " Function Input: " + " ".join(function_input)

litellm/proxy/guardrails/guardrail_hooks/lasso/lasso.py

@@ -40,6 +40,13 @@ from litellm.integrations.custom_guardrail import (
 )
 from litellm.integrations.custom_guardrail import dc as global_cache
 from litellm.litellm_core_utils.core_helpers import iter_items
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    anthropic_server_tool_calls,
+    with_server_tool_call_arguments,
+    write_anthropic_server_tool_calls,
+    write_request_server_tool_calls,
+)
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -50,6 +57,7 @@ from litellm.proxy.guardrails._content_utils import (
     has_non_string_content,
 )
 from litellm.types.guardrails import GuardrailEventHooks
+from litellm.types.llms.openai import ChatCompletionAssistantToolCall
 
 
 class LassoResponse(TypedDict):
@@ -259,7 +267,7 @@ class LassoGuardrail(CustomGuardrail):
                 msg = choice.message
                 if msg.content:
                     response_messages.append({"role": "assistant", "content": msg.content})
-                for call in getattr(msg, "tool_calls", None) or []:
+                for call in (*anthropic_server_tool_calls(msg), *(getattr(msg, "tool_calls", None) or ())):
                     call_id, name, input_data = self._extract_tool_call_fields(call)
                     if not call_id or not name:
                         continue
@@ -564,6 +572,15 @@ class LassoGuardrail(CustomGuardrail):
             elif role == "assistant" and not content and orig_msg.get("tool_calls"):
                 msg["tool_calls"] = self._update_tool_calls_from_masked(orig_msg["tool_calls"], masked_tool_use)
 
+            if role == "assistant":
+                write_request_server_tool_calls(
+                    msg,
+                    tuple(
+                        self._masked_server_tool_call(server_tool_call, masked_tool_use)
+                        for server_tool_call in anthropic_server_tool_call_params(orig_msg)
+                    ),
+                )
+
             result.append(msg)
 
         return result
@@ -704,7 +721,7 @@ class LassoGuardrail(CustomGuardrail):
                 expanded.append({"role": role, "content": content})
 
             if role == "assistant":
-                for call in iter_items(msg.get("tool_calls") or []):
+                for call in (*anthropic_server_tool_call_params(msg), *iter_items(msg.get("tool_calls") or [])):
                     call_id, name, input_data = self._extract_tool_call_fields(call)
                     if not call_id or not name:
                         verbose_proxy_logger.warning(
@@ -962,6 +979,14 @@ class LassoGuardrail(CustomGuardrail):
                 text_cursor += 1
                 verbose_proxy_logger.debug("Applied masked text content to choice %s", text_cursor)
 
+            write_anthropic_server_tool_calls(
+                msg,
+                tuple(
+                    self._masked_server_tool_call(server_tool_call, masked_tool_use)
+                    for server_tool_call in anthropic_server_tool_call_params(msg)
+                ),
+            )
+
             for call in getattr(msg, "tool_calls", None) or []:
                 call_id = self._get_field(call, "id")
                 if call_id and call_id in masked_tool_use:
@@ -977,6 +1002,16 @@ class LassoGuardrail(CustomGuardrail):
                                 func.arguments = json.dumps(masked_input)
                         verbose_proxy_logger.debug("Applied masked tool_call arguments for call_id=%s", call_id)
 
+    @staticmethod
+    def _masked_server_tool_call(
+        server_tool_call: ChatCompletionAssistantToolCall,
+        masked_tool_use: Mapping[object, Mapping[str, object]],
+    ) -> ChatCompletionAssistantToolCall:
+        masked_input: Final = masked_tool_use.get(server_tool_call["id"], {}).get("input")
+        if masked_input is None:
+            return server_tool_call
+        return with_server_tool_call_arguments(server_tool_call, json.dumps(masked_input))
+
     @staticmethod
     def get_config_model() -> type["GuardrailConfigModel"] | None:
         from litellm.types.proxy.guardrails.guardrail_hooks.lasso import (

litellm/proxy/guardrails/guardrail_hooks/llm_shield_proxy/llm_shield_proxy.py

@@ -24,13 +24,17 @@ from litellm.integrations.custom_guardrail import (
     log_guardrail_information,
 )
 from litellm.litellm_core_utils.core_helpers import attribute_items, attribute_or_none, iter_items, require_attributes
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    write_anthropic_server_tool_call_arguments,
+)
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
 )
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.types.guardrails import GuardrailEventHooks
-from litellm.types.utils import GenericGuardrailAPIInputs, TextChoices
+from litellm.types.utils import GenericGuardrailAPIInputs, Message, TextChoices
 
 if TYPE_CHECKING:
     from litellm.caching.caching import DualCache
@@ -87,6 +91,19 @@ _DEFAULT_TIMEOUT_SECONDS: Final = 10.0
 _SHIELD_BODY: Final = TypeAdapter(dict[str, object])
 
 
+def _server_tool_call_slots(message: object) -> tuple[Slot, ...]:
+    if not isinstance(message, Message):
+        return ()
+    return tuple(
+        (arguments, functools.partial(write_anthropic_server_tool_call_arguments, message, index))
+        for index, arguments in enumerate(
+            server_tool_call["function"].get("arguments")
+            for server_tool_call in anthropic_server_tool_call_params(message)
+        )
+        if isinstance(arguments, str) and arguments
+    )
+
+
 def _string_tool_arguments(calls: Sequence[object]) -> Iterator[tuple[object, str]]:
     for call in calls:
         function = read_field(call, "function")
@@ -382,6 +399,7 @@ class LLMShieldProxyGuardrail(CustomGuardrail):
             content = getattr(message, "content", None)
             if isinstance(content, str) and content:
                 pending.append((content, functools.partial(setattr, message, "content")))
+            pending.extend(_server_tool_call_slots(message))
             for tool_call in attribute_items(message, "tool_calls"):
                 function: object = getattr(tool_call, "function", None)
                 arguments = getattr(function, "arguments", None) if function is not None else None

litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/base.py

@@ -13,6 +13,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
     convert_content_list_to_str,
 )
 from litellm.litellm_core_utils.url_utils import encode_url_path_segment
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -434,9 +435,7 @@ class PurviewGuardrailBase:
         args: Final[list[str]] = []
 
         # tool_calls: [{"function": {"arguments": "..."}}]
-        tool_calls: Final[Sequence[object] | None] = (
-            message.get("tool_calls") if isinstance(message, dict) else getattr(message, "tool_calls", None)
-        )
+        tool_calls: Final = tool_calls_with_server_tool_calls(message)
         if tool_calls:
             for tc in tool_calls:
                 fn = tc.get("function") if isinstance(tc, dict) else getattr(tc, "function", None)

litellm/proxy/guardrails/guardrail_hooks/microsoft_purview/purview_dlp.py

@@ -22,6 +22,7 @@ from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
+from litellm.llms.base_llm.guardrail_translation.utils import claude_hosted_web_tool_arguments
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.utils import (
     Choices,
@@ -227,6 +228,9 @@ class MicrosoftPurviewDLPGuardrail(PurviewGuardrailBase, CustomGuardrail):
         if not output:
             return args
         for item in output:
+            hosted_arguments = claude_hosted_web_tool_arguments(item)
+            if hosted_arguments:
+                args.append(hosted_arguments)
             if isinstance(item, dict):
                 item_type = item.get("type")
                 arguments = item.get("arguments")

litellm/proxy/guardrails/guardrail_hooks/pangea/pangea.py

@@ -1,5 +1,6 @@
 # litellm/proxy/guardrails/guardrail_hooks/pangea.py
 import os
+from collections.abc import Callable
 from typing import TYPE_CHECKING, Final
 
 from fastapi import HTTPException
@@ -10,6 +11,11 @@ from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
+from litellm.llms.anthropic.common_utils import (
+    message_dict_with_server_tool_calls,
+    message_dict_with_server_tool_calls_split_out,
+    validated_object_mapping,
+)
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -30,6 +36,13 @@ if TYPE_CHECKING:
     from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel
 
 
+def _choice_with_message(choice: object, rewrite: Callable[[object], object]) -> object:
+    mapping: Final = validated_object_mapping(choice)
+    if mapping is None or "message" not in mapping:
+        return choice
+    return {**mapping, "message": rewrite(mapping["message"])}
+
+
 class PangeaGuardrailMissingSecrets(Exception):
     """Custom exception for missing Pangea secrets."""
 
@@ -242,9 +255,13 @@ class PangeaHandler(CustomGuardrail):
                 for c in choices:
                     if isinstance(c, Choices):
                         try:
-                            serialized_choices.append(c.model_dump())
+                            serialized_choices.append(
+                                _choice_with_message(c.model_dump(), message_dict_with_server_tool_calls)
+                            )
                         except Exception:
-                            serialized_choices.append(c.dict())
+                            serialized_choices.append(
+                                _choice_with_message(c.dict(), message_dict_with_server_tool_calls)
+                            )
                     else:
                         serialized_choices.append(c)
                 choices = serialized_choices
@@ -269,7 +286,9 @@ class PangeaHandler(CustomGuardrail):
             return
 
         output: Final = ai_guard_response.get("result", {}).get("output", {})
-        response.choices = output["choices"]
+        response.choices = [  # pyright: ignore[reportAttributeAccessIssue]  # Pangea returns untyped choice dicts
+            _choice_with_message(choice, message_dict_with_server_tool_calls_split_out) for choice in output["choices"]
+        ]
         return response
 
     @log_guardrail_information

litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py

@@ -27,6 +27,12 @@ from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    tool_calls_with_server_tool_calls,
+    with_server_tool_call_arguments,
+    write_anthropic_server_tool_calls,
+)
 from litellm.llms.base_llm.guardrail_translation.utils import (
     effective_scan_only_tool_results_for_guardrail,
     effective_skip_system_message_for_guardrail,
@@ -364,10 +370,9 @@ class PanwPrismaAirsHandler(CustomGuardrail):
                             text_parts.append(str(choice.message.content))
 
                         # Extract tool call arguments
-                        if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
-                            for tool_call in choice.message.tool_calls:
-                                if hasattr(tool_call, "function") and hasattr(tool_call.function, "arguments"):
-                                    text_parts.append(str(tool_call.function.arguments))
+                        for tool_call in tool_calls_with_server_tool_calls(choice.message):
+                            if hasattr(tool_call, "function") and hasattr(tool_call.function, "arguments"):
+                                text_parts.append(str(tool_call.function.arguments))
 
                         # Extract function call arguments (legacy)
                         if hasattr(choice.message, "function_call") and choice.message.function_call:
@@ -756,6 +761,14 @@ class PanwPrismaAirsHandler(CustomGuardrail):
                     elif isinstance(content, list):
                         choice.message.content = self._mask_content_list(content, masked_text)
 
+                write_anthropic_server_tool_calls(
+                    choice.message,
+                    tuple(
+                        with_server_tool_call_arguments(server_tool_call, masked_text)
+                        for server_tool_call in anthropic_server_tool_call_params(choice.message)
+                    ),
+                )
+
                 # Mask tool call arguments
                 if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
                     for tool_call in choice.message.tool_calls:

litellm/proxy/guardrails/guardrail_hooks/pillar/pillar.py

@@ -26,6 +26,7 @@ from litellm.integrations.custom_guardrail import (
     log_guardrail_information,
 )
 from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
+from litellm.llms.anthropic.common_utils import message_dict_with_server_tool_calls
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -205,6 +206,7 @@ class PillarGuardrail(CustomGuardrail):
         include_evidence: bool | None = None,
         fallback_on_error: str | None = None,
         timeout: float | None = None,
+        async_handler: _PillarProtectHTTPClient | None = None,
         **kwargs,
     ) -> None:
         """
@@ -223,8 +225,10 @@ class PillarGuardrail(CustomGuardrail):
             LiteLLM virtual key context (user_id, team_id, key_alias, etc.) is always
             automatically passed as X-LiteLLM-* headers to enable application/user tracking.
         """
-        self.async_handler: _PillarProtectHTTPClient = get_async_httpx_client(
-            llm_provider=httpxSpecialProvider.GuardrailCallback
+        self.async_handler: _PillarProtectHTTPClient = (
+            async_handler
+            if async_handler is not None
+            else get_async_httpx_client(llm_provider=httpxSpecialProvider.GuardrailCallback)
         )
         self.api_key = api_key or os.environ.get("PILLAR_API_KEY")
 
@@ -430,7 +434,9 @@ class PillarGuardrail(CustomGuardrail):
         # Extract response messages in the format Pillar expects
         response_dict = response.model_dump() if hasattr(response, "model_dump") else {}
         response_messages: Final = [
-            choice.get("message") for choice in response_dict.get("choices", []) if choice.get("message")
+            message_dict_with_server_tool_calls(choice["message"])
+            for choice in response_dict.get("choices", [])
+            if choice.get("message")
         ]
 
         if not response_messages:

litellm/proxy/guardrails/guardrail_hooks/presidio.py

@@ -12,7 +12,7 @@ import asyncio
 import json
 import re
 import threading
-from collections.abc import AsyncGenerator, AsyncIterable, AsyncIterator, Awaitable, Iterator, Sequence
+from collections.abc import AsyncGenerator, AsyncIterable, AsyncIterator, Awaitable, Callable, Iterator, Sequence
 from contextlib import asynccontextmanager
 from dataclasses import dataclass
 from datetime import datetime
@@ -43,6 +43,11 @@ from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    with_server_tool_call_arguments,
+    write_anthropic_server_tool_calls,
+)
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.proxy.common_utils.sse_keepalive import split_complete_sse_frames
 from litellm.proxy.guardrails.anthropic_sse import (
@@ -1404,6 +1409,18 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
             verbose_proxy_logger.debug("No pii_tokens found in request_data['metadata'] — nothing to unmask")
         presidio_config: Final = self.get_presidio_settings_from_request_data(request_data or {})
 
+        async def pii_processed_text(text: str) -> str:
+            match mode:
+                case "unmask":
+                    return self._unmask_pii_text(text, pii_tokens)
+                case "mask":
+                    return await self.check_pii(
+                        text=text,
+                        output_parse_pii=False,
+                        presidio_config=presidio_config,
+                        request_data=request_data,
+                    )
+
         for choice in response.choices:
             message: Message | None = getattr(choice, "message", None)
             if message is None:
@@ -1412,15 +1429,7 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
             # 1. Process content
             content = getattr(message, "content", None)
             if isinstance(content, str):
-                if mode == "unmask":
-                    message.content = self._unmask_pii_text(content, pii_tokens)
-                elif mode == "mask":
-                    message.content = await self.check_pii(
-                        text=content,
-                        output_parse_pii=False,
-                        presidio_config=presidio_config,
-                        request_data=request_data,
-                    )
+                message.content = await pii_processed_text(content)
             elif isinstance(content, list):
                 for item in content:
                     if not isinstance(item, dict):
@@ -1428,54 +1437,43 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
                     text_value = item.get("text")
                     if text_value is None:
                         continue
-                    if mode == "unmask":
-                        item["text"] = self._unmask_pii_text(text_value, pii_tokens)
-                    elif mode == "mask":
-                        item["text"] = await self.check_pii(
-                            text=text_value,
-                            output_parse_pii=False,
-                            presidio_config=presidio_config,
-                            request_data=request_data,
-                        )
+                    item["text"] = await pii_processed_text(text_value)
+
+            await self._process_server_tool_calls_for_pii(message, pii_processed_text)
 
             # 2. Process tool calls
             for tool_call in attribute_items(message, "tool_calls"):
                 function = attribute_or_none(tool_call, "function")
                 args = attribute_or_none(function, "arguments") if function else None
                 if isinstance(args, str):
-                    if mode == "unmask":
-                        _write_attribute(function, "arguments", self._unmask_pii_text(args, pii_tokens))
-                    elif mode == "mask":
-                        _write_attribute(
-                            function,
-                            "arguments",
-                            await self.check_pii(
-                                text=args,
-                                output_parse_pii=False,
-                                presidio_config=presidio_config,
-                                request_data=request_data,
-                            ),
-                        )
+                    _write_attribute(function, "arguments", await pii_processed_text(args))
 
             # 3. Process legacy function calls
             function_call = attribute_or_none(message, "function_call")
             legacy_args = attribute_or_none(function_call, "arguments") if function_call else None
             if isinstance(legacy_args, str):
-                if mode == "unmask":
-                    _write_attribute(function_call, "arguments", self._unmask_pii_text(legacy_args, pii_tokens))
-                elif mode == "mask":
-                    _write_attribute(
-                        function_call,
-                        "arguments",
-                        await self.check_pii(
-                            text=legacy_args,
-                            output_parse_pii=False,
-                            presidio_config=presidio_config,
-                            request_data=request_data,
-                        ),
-                    )
+                _write_attribute(function_call, "arguments", await pii_processed_text(legacy_args))
         return response
 
+    @staticmethod
+    async def _process_server_tool_calls_for_pii(
+        message: Message,
+        pii_processed_text: Callable[[str], Awaitable[str]],
+    ) -> None:
+        server_tool_calls: Final = anthropic_server_tool_call_params(message)
+        if not server_tool_calls:
+            return
+        processed_arguments: Final = tuple(
+            [
+                await pii_processed_text(server_tool_call["function"].get("arguments", ""))
+                for server_tool_call in server_tool_calls
+            ]
+        )
+        write_anthropic_server_tool_calls(
+            message,
+            tuple(map(with_server_tool_call_arguments, server_tool_calls, processed_arguments)),
+        )
+
     async def _mask_output_response(
         self,
         response: ModelResponse | EmbeddingResponse | ImageResponse,

litellm/proxy/guardrails/guardrail_hooks/qualifire/qualifire.py

@@ -18,6 +18,7 @@ from litellm.integrations.custom_guardrail import (
     log_guardrail_information,
 )
 from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.anthropic.common_utils import anthropic_server_tool_call_params
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
@@ -180,8 +181,8 @@ class QualifireGuardrail(CustomGuardrail):
                 api_message["tool_call_id"] = tool_call_id
 
             # Handle tool calls if present
-            tool_calls = msg.get("tool_calls")
-            if tool_calls and isinstance(tool_calls, list):
+            tool_calls = [*anthropic_server_tool_call_params(msg), *(msg.get("tool_calls") or [])]
+            if tool_calls:
                 api_tool_calls = []
                 for tc in tool_calls:
                     if isinstance(tc, dict):

litellm/proxy/guardrails/guardrail_hooks/repelloai/repelloai.py

@@ -12,6 +12,7 @@ from pydantic import BaseModel, TypeAdapter, ValidationError
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm.integrations.custom_guardrail import CustomGuardrail
+from litellm.llms.anthropic.common_utils import tool_calls_with_server_tool_calls
 from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,  # pyright: ignore[reportUnknownVariableType]
     httpxSpecialProvider,
@@ -80,13 +81,11 @@ class RepelloAIGuardrail(CustomGuardrail):
     def _extract_tool_call_args_from_message(cls, message: object) -> list[str]:
         args: Final[list[str]] = []
 
-        tool_calls: Final = cls._get_field(message, "tool_calls")
-        if _is_object_list(tool_calls):
-            for tool_call in tool_calls:
-                function = cls._get_field(tool_call, "function")
-                arguments = cls._get_field(function, "arguments")
-                if isinstance(arguments, str) and arguments.strip():
-                    args.append(arguments)
+        for tool_call in tool_calls_with_server_tool_calls(message):
+            function = cls._get_field(tool_call, "function")
+            arguments = cls._get_field(function, "arguments")
+            if isinstance(arguments, str) and arguments.strip():
+                args.append(arguments)
 
         function_call: Final = cls._get_field(message, "function_call")
         arguments = cls._get_field(function_call, "arguments")

litellm/proxy/guardrails/guardrail_hooks/tool_permission.py

@@ -24,6 +24,11 @@ from litellm.integrations.custom_guardrail import (
 )
 from litellm.litellm_core_utils.core_helpers import as_str_mapping, attribute_or_none
 from litellm.litellm_core_utils.tool_fields import declared_function_name, qualified_member_name
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    anthropic_server_tool_calls,
+    without_anthropic_server_tool_calls,
+)
 from litellm.llms.base_llm.guardrail_translation.utils import anthropic_tool_names
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.proxy.common_utils.callback_utils import (
@@ -983,7 +988,7 @@ class ToolPermissionGuardrail(CustomGuardrail):
 
         for choice_index, choice in enumerate(response.choices):
             if isinstance(choice, Choices):
-                for tool in choice.message.tool_calls or []:
+                for tool in (*(choice.message.tool_calls or ()), *anthropic_server_tool_calls(choice.message)):
                     tool_calls.append(tool)
                 legacy_tool_call = self._legacy_function_call_to_tool_call(
                     getattr(choice.message, "function_call", None), choice_index
@@ -1493,6 +1498,23 @@ class ToolPermissionGuardrail(CustomGuardrail):
 
                 choice.message.tool_calls = filtered_tool_calls if filtered_tool_calls else None
 
+                server_error_results = tuple(
+                    error_results.get(self._judged_tool_call(server_tool_call))
+                    for server_tool_call in anthropic_server_tool_calls(choice.message)
+                )
+                if any(server_error_results):
+                    error_messages.extend(result.content for result in server_error_results if result is not None)
+                    without_anthropic_server_tool_calls(
+                        choice.message,
+                        frozenset(
+                            server_tool_call["id"] or ""
+                            for server_tool_call, result in zip(
+                                anthropic_server_tool_call_params(choice.message), server_error_results
+                            )
+                            if result is not None
+                        ),
+                    )
+
                 legacy_tool_call = self._legacy_function_call_to_tool_call(
                     getattr(choice.message, "function_call", None), choice_index
                 )

litellm/proxy/guardrails/guardrail_hooks/tool_policy/tool_policy_guardrail.py

@@ -27,7 +27,10 @@ from pydantic import TypeAdapter
 from typing_extensions import TypedDict, Unpack
 
 from litellm._logging import verbose_proxy_logger
-from litellm.completion_extras.litellm_responses_transformation.transformation import replayed_tool_search_tool_names
+from litellm.completion_extras.litellm_responses_transformation.transformation import (
+    nested_tool_member_names,
+    replayed_tool_search_tool_names,
+)
 from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
@@ -47,6 +50,14 @@ class _CustomGuardrailOptions(TypedDict, total=False, extra_items=object):
     """Base-class constructor options this guardrail forwards untouched to CustomGuardrail."""
 
 
+def _function_tool_name(tool: object) -> str | None:
+    match tool:
+        case {"function": {"name": str(name)}}:
+            return name
+        case _:
+            return None
+
+
 def _attribute(source: object, name: str) -> object:
     return getattr(source, name, None)
 
@@ -140,17 +151,18 @@ class ToolPolicyGuardrail(CustomGuardrail):
         """
         if input_type == "request":
             tools: Final = inputs.get("tools") or []
-            function_tool_names: Final = [
-                t["function"]["name"]
-                for t in tools
-                if isinstance(t, dict) and isinstance(t.get("function"), dict) and t["function"].get("name")
-            ]
+            function_tool_names: Final = [name for name in map(_function_tool_name, tools) if name]
             route: Final = _get_request_route_from_data(request_data)
-            replayed_tool_names: Final = replayed_tool_search_tool_names(_REQUEST_DATA.validate_python(request_data))
-            tool_names = (
-                extract_request_tool_names(route, request_data)
-                if route and not function_tool_names
-                else [*function_tool_names, *replayed_tool_names]
+            request: Final = _REQUEST_DATA.validate_python(request_data)
+            tool_names = list(
+                dict.fromkeys(
+                    (
+                        *function_tool_names,
+                        *(extract_request_tool_names(route, request_data) if route else ()),
+                        *replayed_tool_search_tool_names(request),
+                        *nested_tool_member_names(request),
+                    )
+                )
             )
         else:
             tool_calls: Final = inputs.get("tool_calls") or []

litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py

@@ -25,6 +25,7 @@ from litellm.integrations.custom_logger import CustomLogger
 from litellm.litellm_core_utils.api_route_to_call_types import get_call_types_for_route
 from litellm.litellm_core_utils.core_helpers import get_or_create_metadata_bucket
 from litellm.llms import get_guardrail_translation_mapping, load_guardrail_translation_mappings
+from litellm.llms.anthropic.common_utils import anthropic_server_call_fields
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.utils import (
@@ -144,6 +145,15 @@ def _ends_stream(endpoint_translation: object, item: object) -> bool:
     return has_ended is not None and has_ended((item,))
 
 
+def _passthrough_delta(delta: object) -> Delta:
+    server_call_fields: Final = anthropic_server_call_fields(delta)
+    role: Final = getattr(delta, "role", None)
+    tool_calls: Final = getattr(delta, "tool_calls", None)
+    if not server_call_fields:
+        return Delta(content=None, role=role, tool_calls=tool_calls)
+    return Delta(content=None, role=role, tool_calls=tool_calls, provider_specific_fields=dict(server_call_fields))
+
+
 def _held_choices(held_chars_per_choice: Mapping[int, int]) -> frozenset[int]:
     return frozenset(idx for idx, held in held_chars_per_choice.items() if held > 0)
 
@@ -974,7 +984,7 @@ class UnifiedLLMGuardrails(CustomLogger):
     def _chunk_has_tool_calls(item: object) -> bool:
         for choice in _chunk_choices(item):
             delta = getattr(choice, "delta", None)
-            if getattr(delta, "tool_calls", None):
+            if getattr(delta, "tool_calls", None) or anthropic_server_call_fields(delta):
                 return True
         return False
 
@@ -1042,11 +1052,7 @@ class UnifiedLLMGuardrails(CustomLogger):
             synthetic_choices.append(
                 StreamingChoices(
                     index=idx,
-                    delta=Delta(
-                        content=None,
-                        role=getattr(delta, "role", None),
-                        tool_calls=getattr(delta, "tool_calls", None),
-                    ),
+                    delta=_passthrough_delta(delta),
                     finish_reason=passthrough_finish,
                 )
             )

litellm/proxy/hooks/model_max_budget_limiter.py

@@ -27,6 +27,7 @@ from litellm.litellm_core_utils.batch_utils import (
 )
 from litellm.litellm_core_utils.duration_parser import duration_in_seconds
 from litellm.litellm_core_utils.litellm_logging import StandardLoggingPayloadSetup
+from litellm.litellm_core_utils.model_latest_aliases import expand_model_latest_alias
 from litellm.llms.bedrock.common_utils import get_bedrock_base_model, get_bedrock_cross_region_inference_regions
 from litellm.proxy._types import Litellm_EntityType, UserAPIKeyAuth
 from litellm.proxy.hooks.model_budget_shared_windows import (
@@ -523,6 +524,25 @@ def _usable_entry(entry: str, raw_budget_config: object) -> ResolvedModelBudget
 
 def team_model_budget_entries(
     model: str, team_model_max_budget: Mapping[str, object], key_model_max_budget: Mapping[str, object] | None
+) -> Mapping[str, object]:
+    models: Final = expand_model_latest_alias(model)
+    if len(models) == 1:
+        return _single_team_model_budget_entries(model, team_model_max_budget, key_model_max_budget)
+    entries: Final = tuple(
+        _single_team_model_budget_entries(candidate, team_model_max_budget, key_model_max_budget)
+        for candidate in models
+    )
+    return MappingProxyType(
+        {
+            entry: config
+            for entry, config in team_model_max_budget.items()
+            if all(entry in mapping for mapping in entries)
+        }
+    )
+
+
+def _single_team_model_budget_entries(
+    model: str, team_model_max_budget: Mapping[str, object], key_model_max_budget: Mapping[str, object] | None
 ) -> Mapping[str, object]:
     """The team entries that budget `model` for this key.
 
@@ -780,6 +800,31 @@ def _as_float(value: object) -> float | None:
         return None
 
 
+def _entity_model_budget_entries(
+    model: str,
+    entity_budget: tuple[Litellm_EntityType, str | None, object],
+    model_access_groups: ModelAccessGroups,
+    team_id: str | None,
+) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]:
+    entity_type, entity_id, model_max_budget = entity_budget
+    if entity_id is None or not isinstance(model_max_budget, Mapping) or not model_max_budget:
+        return ()
+    return tuple(
+        (entity_type, entity_id, resolved)
+        for candidate in expand_model_latest_alias(model)
+        if (
+            resolved := resolve_model_budget(
+                model=candidate,
+                model_max_budget=model_max_budget,
+                model_access_groups=model_access_groups,
+                team_id=team_id,
+            )
+        )
+        is not None
+        and resolved.budget_config.budget_duration is not None
+    )
+
+
 def _resolve_entity_model_budgets(
     model: str,
     entity_budgets: Iterable[tuple[Litellm_EntityType, str | None, object]],
@@ -787,19 +832,15 @@ def _resolve_entity_model_budgets(
     team_id: str | None,
 ) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]:
     """Drop the scopes that do not budget `model`, keeping only what can be incremented."""
+    entries: Final = itertools.chain.from_iterable(
+        _entity_model_budget_entries(model, entity_budget, model_access_groups, team_id)
+        for entity_budget in entity_budgets
+    )
     return tuple(
-        (entity_type, entity_id, resolved)
-        for entity_type, entity_id, model_max_budget in entity_budgets
-        if entity_id is not None and isinstance(model_max_budget, Mapping) and model_max_budget
-        for resolved in (
-            resolve_model_budget(
-                model=model,
-                model_max_budget=model_max_budget,
-                model_access_groups=model_access_groups,
-                team_id=team_id,
-            ),
-        )
-        if resolved is not None and resolved.budget_config.budget_duration is not None
+        {
+            (entity_type, entity_id, resolved.budget_model): (entity_type, entity_id, resolved)
+            for entity_type, entity_id, resolved in entries
+        }.values()
     )
 
 
@@ -996,6 +1037,26 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting):
         model_max_budget: Mapping[str, object] | None,
         model: str,
         exceeded_message: str,
+    ) -> bool:
+        for candidate in expand_model_latest_alias(model):
+            await self._is_single_entity_within_model_budget(
+                entity_type=entity_type,
+                entity_id=entity_id,
+                team_id=team_id,
+                model_max_budget=model_max_budget,
+                model=candidate,
+                exceeded_message=exceeded_message,
+            )
+        return True
+
+    async def _is_single_entity_within_model_budget(
+        self,
+        entity_type: Litellm_EntityType,
+        entity_id: str | None,
+        team_id: str | None,
+        model_max_budget: Mapping[str, object] | None,
+        model: str,
+        exceeded_message: str,
     ) -> bool:
         if not model_max_budget:
             return True

litellm/proxy/hooks/parallel_request_limiter_v3.py

@@ -50,6 +50,7 @@ from litellm.litellm_core_utils.add_retry_fallback_headers import (
     response_has_hidden_params,
 )
 from litellm.litellm_core_utils.core_helpers import as_str_mapping
+from litellm.litellm_core_utils.model_latest_aliases import expand_model_latest_alias
 from litellm.litellm_core_utils.prompt_templates.common_utils import (
     get_str_from_messages,
 )
@@ -74,6 +75,7 @@ from litellm.proxy.auth.auth_utils import (
     get_model_rate_limit_from_metadata,
 )
 from litellm.proxy.auth.budget_throttle import throttled_limit
+from litellm.proxy.common_utils.callback_utils import get_model_group_from_litellm_kwargs
 from litellm.proxy.common_utils.http_parsing_utils import get_tags_from_request_body
 from litellm.proxy.common_utils.proxy_rate_limit_error import (
     ProxyRateLimitError,
@@ -348,6 +350,44 @@ class RateLimitDescriptor(TypedDict):
     rate_limit: RateLimitDescriptorRateLimitObject | None
 
 
+def _minimum_descriptor_limit(
+    descriptors: Sequence[RateLimitDescriptor],
+    metric: Literal["requests_per_unit", "tokens_per_unit", "max_parallel_requests"],
+) -> int | None:
+    return min(
+        (limit for descriptor in descriptors if (limit := (descriptor["rate_limit"] or {}).get(metric)) is not None),
+        default=None,
+    )
+
+
+def _merge_rate_limit_descriptors(descriptors: Sequence[RateLimitDescriptor]) -> RateLimitDescriptor:
+    first: Final = descriptors[0]
+    if len(descriptors) == 1:
+        return first
+    return RateLimitDescriptor(
+        key=first["key"],
+        value=first["value"],
+        rate_limit=RateLimitDescriptorRateLimitObject(
+            requests_per_unit=_minimum_descriptor_limit(descriptors, "requests_per_unit"),
+            tokens_per_unit=_minimum_descriptor_limit(descriptors, "tokens_per_unit"),
+            max_parallel_requests=_minimum_descriptor_limit(descriptors, "max_parallel_requests"),
+            window_size=(first["rate_limit"] or {}).get("window_size"),
+        ),
+    )
+
+
+def _coalesce_rate_limit_descriptors(descriptors: Sequence[RateLimitDescriptor]) -> Sequence[RateLimitDescriptor]:
+    identities: Final = tuple(dict.fromkeys((descriptor["key"], descriptor["value"]) for descriptor in descriptors))
+    if len(identities) == len(descriptors):
+        return descriptors
+    return tuple(
+        _merge_rate_limit_descriptors(
+            tuple(descriptor for descriptor in descriptors if (descriptor["key"], descriptor["value"]) == identity)
+        )
+        for identity in identities
+    )
+
+
 class ParallelRequestGauge(TypedDict):
     scope: RateLimitScope
     limit: int
@@ -2405,6 +2445,18 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             group=self._model_group_resolver(requested_model) or requested_model,
         )
 
+    def _rate_limited_models(self, model: RateLimitedModel | None) -> tuple[RateLimitedModel, ...]:
+        if model is None:
+            return ()
+        return (
+            model,
+            *(
+                target
+                for candidate in expand_model_latest_alias(model.requested)[1:]
+                if (target := self._rate_limited_model(candidate)) is not None
+            ),
+        )
+
     def create_organization_rate_limit_descriptor(
         self, user_api_key_dict: UserAPIKeyAuth, requested_model: str | None = None
     ) -> list[RateLimitDescriptor]:
@@ -2426,9 +2478,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
                 )
             )
 
-        model: Final = self._rate_limited_model(requested_model)
-        if model is None:
-            return descriptors
+        model_descriptors: Final = tuple(
+            descriptor
+            for model in self._rate_limited_models(self._rate_limited_model(requested_model))
+            if (descriptor := self._organization_model_rate_limit_descriptor(user_api_key_dict, model)) is not None
+        )
+        return [*descriptors, *_coalesce_rate_limit_descriptors(model_descriptors)]
+
+    def _organization_model_rate_limit_descriptor(
+        self, user_api_key_dict: UserAPIKeyAuth, model: RateLimitedModel
+    ) -> RateLimitDescriptor | None:
         model_specific_tpm_limit: Final = model.limit_in(
             get_model_rate_limit_from_metadata(user_api_key_dict, "organization_metadata", "model_tpm_limit")
         )
@@ -2436,19 +2495,16 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             get_model_rate_limit_from_metadata(user_api_key_dict, "organization_metadata", "model_rpm_limit")
         )
         if model_specific_tpm_limit is None and model_specific_rpm_limit is None:
-            return descriptors
-        descriptors.append(
-            RateLimitDescriptor(
-                key="model_per_organization",
-                value=f"{user_api_key_dict.org_id}:{model.group}",
-                rate_limit={
-                    "requests_per_unit": model_specific_rpm_limit,
-                    "tokens_per_unit": model_specific_tpm_limit,
-                    "window_size": self.window_size,
-                },
-            )
+            return None
+        return RateLimitDescriptor(
+            key="model_per_organization",
+            value=f"{user_api_key_dict.org_id}:{model.group}",
+            rate_limit={
+                "requests_per_unit": model_specific_rpm_limit,
+                "tokens_per_unit": model_specific_tpm_limit,
+                "window_size": self.window_size,
+            },
         )
-        return descriptors
 
     def _add_model_per_key_rate_limit_descriptor(
         self,
@@ -2935,11 +2991,13 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
 
         # Model rate limits
         requested_model: Final = data.get("model", None)
-        self._add_model_per_key_rate_limit_descriptor(
-            user_api_key_dict=user_api_key_dict,
-            requested_model=requested_model,
-            descriptors=descriptors,
-        )
+        model_candidates: Final = expand_model_latest_alias(requested_model if isinstance(requested_model, str) else "")
+        for candidate in model_candidates:
+            self._add_model_per_key_rate_limit_descriptor(
+                user_api_key_dict=user_api_key_dict,
+                requested_model=candidate,
+                descriptors=descriptors,
+            )
 
         # Per-request-tag rate limits scoped to this key
         self._add_tag_per_key_rate_limit_descriptor(
@@ -2948,11 +3006,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             descriptors=descriptors,
         )
 
-        self._add_team_model_rate_limit_descriptor_from_metadata(
-            user_api_key_dict=user_api_key_dict,
-            requested_model=requested_model if isinstance(requested_model, str) else None,
-            descriptors=descriptors,
-        )
+        for candidate in model_candidates:
+            self._add_team_model_rate_limit_descriptor_from_metadata(
+                user_api_key_dict=user_api_key_dict,
+                requested_model=candidate,
+                descriptors=descriptors,
+            )
         self._add_team_ptu_rate_limit_descriptor(
             user_api_key_dict=user_api_key_dict,
             requested_model=requested_model if isinstance(requested_model, str) else None,
@@ -2975,7 +3034,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
                 )
             )
 
-        return descriptors
+        return descriptors if len(model_candidates) == 1 else list(_coalesce_rate_limit_descriptors(descriptors))
 
     _create_rate_limit_descriptors = create_rate_limit_descriptors
 
@@ -3126,12 +3185,26 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         self,
         user_api_key_dict: UserAPIKeyAuth,
         requested_model: str | None,
-        descriptors: list[RateLimitDescriptor],
+        descriptors: _RateLimitDescriptorSink,
     ) -> None:
+        candidates: Final = expand_model_latest_alias(requested_model) if requested_model else (None,)
+        model_descriptors: Final = tuple(
+            descriptor
+            for candidate in candidates
+            if (descriptor := self._project_model_rate_limit_descriptor(user_api_key_dict, candidate)) is not None
+        )
+        for descriptor in _coalesce_rate_limit_descriptors(model_descriptors):
+            descriptors.append(descriptor)
+
+    def _project_model_rate_limit_descriptor(
+        self,
+        user_api_key_dict: UserAPIKeyAuth,
+        requested_model: str | None,
+    ) -> RateLimitDescriptor | None:
         """Add project model rate limit descriptor from project_metadata if applicable."""
         model: Final = self._rate_limited_model(requested_model)
         if model is None:
-            return
+            return None
         model_specific_tpm_limit: Final = model.limit_in(
             get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_tpm_limit")
         )
@@ -3139,17 +3212,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_rpm_limit")
         )
         if model_specific_tpm_limit is None and model_specific_rpm_limit is None:
-            return
-        descriptors.append(
-            RateLimitDescriptor(
-                key="model_per_project",
-                value=f"{user_api_key_dict.project_id}:{model.group}",
-                rate_limit={
-                    "requests_per_unit": model_specific_rpm_limit,
-                    "tokens_per_unit": model_specific_tpm_limit,
-                    "window_size": self.window_size,
-                },
-            )
+            return None
+        return RateLimitDescriptor(
+            key="model_per_project",
+            value=f"{user_api_key_dict.project_id}:{model.group}",
+            rate_limit={
+                "requests_per_unit": model_specific_rpm_limit,
+                "tokens_per_unit": model_specific_tpm_limit,
+                "window_size": self.window_size,
+            },
         )
 
     def add_project_io_token_rate_limit_descriptors_from_metadata(
@@ -3158,6 +3229,20 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         requested_model: str | None,
         descriptors: _RateLimitDescriptorSink,
     ) -> None:
+        candidates: Final = expand_model_latest_alias(requested_model) if requested_model else (None,)
+        model_descriptors: Final = tuple(
+            itertools.chain.from_iterable(
+                self._project_io_token_rate_limit_descriptors(user_api_key_dict, candidate) for candidate in candidates
+            )
+        )
+        for descriptor in _coalesce_rate_limit_descriptors(model_descriptors):
+            descriptors.append(descriptor)
+
+    def _project_io_token_rate_limit_descriptors(
+        self,
+        user_api_key_dict: UserAPIKeyAuth,
+        requested_model: str | None,
+    ) -> tuple[RateLimitDescriptor, ...]:
         """Add project-scoped ITPM/OTPM descriptors from project_metadata.
 
         Enforced independently of, and alongside, the combined ``model_per_project``
@@ -3166,7 +3251,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         """
         model: Final = self._rate_limited_model(requested_model)
         if model is None or user_api_key_dict.project_id is None:
-            return
+            return ()
 
         model_itpm_limit: Final = model.limit_in(
             get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_itpm_limit")
@@ -3175,34 +3260,23 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             get_model_rate_limit_from_metadata(user_api_key_dict, "project_metadata", "model_otpm_limit")
         )
 
-        if model_itpm_limit is None and model_otpm_limit is None:
-            return
-
         descriptor_value: Final = f"{user_api_key_dict.project_id}:{model.group}"
-        if model_itpm_limit is not None:
-            descriptors.append(
-                RateLimitDescriptor(
-                    key=PROJECT_ITPM_DESCRIPTOR_KEY,
-                    value=descriptor_value,
-                    rate_limit={
-                        "requests_per_unit": None,
-                        "tokens_per_unit": model_itpm_limit,
-                        "window_size": self.window_size,
-                    },
-                )
+        return tuple(
+            RateLimitDescriptor(
+                key=key,
+                value=descriptor_value,
+                rate_limit={
+                    "requests_per_unit": None,
+                    "tokens_per_unit": limit,
+                    "window_size": self.window_size,
+                },
             )
-        if model_otpm_limit is not None:
-            descriptors.append(
-                RateLimitDescriptor(
-                    key=PROJECT_OTPM_DESCRIPTOR_KEY,
-                    value=descriptor_value,
-                    rate_limit={
-                        "requests_per_unit": None,
-                        "tokens_per_unit": model_otpm_limit,
-                        "window_size": self.window_size,
-                    },
-                )
+            for key, limit in (
+                (PROJECT_ITPM_DESCRIPTOR_KEY, model_itpm_limit),
+                (PROJECT_OTPM_DESCRIPTOR_KEY, model_otpm_limit),
             )
+            if limit is not None
+        )
 
     def _handle_rate_limit_error(
         self,
@@ -4416,6 +4490,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         kwargs: object,
         model_group: str | None,
         tpm_limited_tags: Set[str] = frozenset(),
+        model_groups: tuple[str, ...] = (),
     ) -> list[tuple[str, str]]:
         """
         Enumerate every (scope_key, scope_value) pair that *might* carry a
@@ -4449,24 +4524,25 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         if user_api_key_organization_id:
             targets.append(("organization", user_api_key_organization_id))
         if model_group:
-            if user_api_key:
-                targets.append(("model_per_key", f"{user_api_key}:{model_group}"))
-            if user_api_key_team_id:
-                targets.append(("model_per_team", f"{user_api_key_team_id}:{model_group}"))
-            if user_api_key_organization_id:
-                targets.append(
-                    (
-                        "model_per_organization",
-                        f"{user_api_key_organization_id}:{model_group}",
+            for group in dict.fromkeys(model_groups or (model_group,)):
+                if user_api_key:
+                    targets.append(("model_per_key", f"{user_api_key}:{group}"))
+                if user_api_key_team_id:
+                    targets.append(("model_per_team", f"{user_api_key_team_id}:{group}"))
+                if user_api_key_organization_id:
+                    targets.append(
+                        (
+                            "model_per_organization",
+                            f"{user_api_key_organization_id}:{group}",
+                        )
                     )
-                )
-            if user_api_key_project_id:
-                targets.append(
-                    (
-                        "model_per_project",
-                        f"{user_api_key_project_id}:{model_group}",
+                if user_api_key_project_id:
+                    targets.append(
+                        (
+                            "model_per_project",
+                            f"{user_api_key_project_id}:{group}",
+                        )
                     )
-                )
         if agent_id:
             targets.append(("agent", agent_id))
             if session_id:
@@ -4565,9 +4641,6 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
     ) -> list[ReservationOperation]:
         """Build Redis pipeline increment ops for TPM / parallel-request counters."""
         from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs
-        from litellm.proxy.common_utils.callback_utils import (
-            get_model_group_from_litellm_kwargs,
-        )
 
         # Get metadata from standard_logging_object - this correctly handles both
         # 'metadata' and 'litellm_metadata' fields from litellm_params
@@ -4617,6 +4690,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
         # mid-flight cannot move the charge) is authoritative; otherwise fall
         # back to the router's ``model_group`` (the no-reservation charge path).
         reconcile_model: Final = reserved_model if reserved_model is not None else self._rate_limited_model(model_group)
+        reconcile_models: Final = self._rate_limited_models(reconcile_model)
 
         pipeline_operations: Final[list[ReservationOperation]] = []
 
@@ -4636,17 +4710,24 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
             kwargs=kwargs,
             tpm_limited_tags=_tpm_limited_tags(stash, kwargs),
             model_group=reconcile_model.group if reconcile_model is not None else None,
+            model_groups=tuple(model.group for model in reconcile_models),
         )
         targets.extend(
             scope
             for scope in sorted(reserved_scopes)
             if scope[0] in ("agent", "agent_session") and scope not in targets
         )
-        charged_targets: Final = (
-            [target for target in targets if target[0] != "model_per_team"]
-            if self._key_owns_model_tpm_limit_from_request_metadata(request_metadata, reconcile_model)
-            else targets
+        logging_metadata: Final = _callback_kwargs_mapping(standard_logging_metadata)
+        key_owned_team_scopes: Final = frozenset(
+            ("model_per_team", f"{logging_metadata.get('user_api_key_team_id')}:{model.group}")
+            for model in reconcile_models
+            if all(
+                self._key_owns_model_tpm_limit_from_request_metadata(request_metadata, candidate)
+                for candidate in reconcile_models
+                if candidate.group == model.group
+            )
         )
+        charged_targets: Final = [target for target in targets if target not in key_owned_team_scopes]
         if reserved_tokens > 0 and total_tokens < reserved_tokens:
             verbose_proxy_logger.debug(
                 "Releasing unused TPM budget on success: reserved=%s, actual=%s, release=%s",

litellm/proxy/management_endpoints/internal_user_endpoints.py

@@ -85,7 +85,7 @@ from litellm.proxy.management_helpers.object_permission_utils import (  # noqa:
 )
 from litellm.proxy.management_helpers.team_roster_sync import user_tx_db
 from litellm.proxy.management_helpers.utils import management_endpoint_wrapper
-from litellm.proxy.utils import handle_exception_on_proxy, hash_password
+from litellm.proxy.utils import handle_exception_on_proxy, hash_password, run_off_loop_if_fips
 from litellm.redis.stores.spend_counters import (
     user_counter,
 )
@@ -210,7 +210,7 @@ async def _hash_password_in_dict(
         if not password_prevalidated:
             validate_password_policy(data["password"], general_settings)
             await validate_password_not_breached(data["password"], general_settings, hibp_client)
-        data["password"] = hash_password(data["password"])
+        data["password"] = await run_off_loop_if_fips(hash_password, data["password"])
         data["password_reset_required"] = True
         data["last_breach_check_at"] = None
 

litellm/proxy/management_endpoints/key_management_endpoints.py

@@ -65,7 +65,6 @@ from litellm.proxy.auth.auth_utils import (
 from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
 from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
     evict_and_broadcast,
-    publish_auth_cache_invalidation,
 )
 from litellm.proxy.common_utils.callback_config_validation import logging_metadata_config_error
 from litellm.proxy.common_utils.callback_utils import (
@@ -171,7 +170,6 @@ from litellm.proxy.utils import (
     require_enterprise_license,
 )
 from litellm.redis.stores.spend_counters import (
-    SPEND_COUNTERS_TARGET,
     key_counter,
     key_window_counter,
 )
@@ -6809,38 +6807,6 @@ def _validate_reset_spend_value(reset_to: object, key_in_db: LiteLLM_Verificatio
     return reset_to
 
 
-@with_service_target(SPEND_COUNTERS_TARGET)
-async def _set_spend_counter_with_floor_and_broadcast(counter_key: str, value: float) -> None:
-    """
-    Set a Redis-backed spend counter to `value`, mirror it into the short-lived
-    spend_db_floor marker `_authoritative_floor_spend` reads, and broadcast both
-    to every worker (LIT-3803 pattern: setting, not deleting, means a worker's
-    own self-delivered broadcast still carries the reset value forward).
-
-    Without the floor marker, `_authoritative_floor_spend` can re-derive a
-    stale, pre-reset value from a marker another worker cached moments earlier
-    and raise the just-reset counter right back up via `_repair_stale_spend_counter`.
-    Without the broadcast, a worker that already cached the pre-reset key object
-    or floor marker keeps enforcing against it until its own TTL expires.
-    """
-    from litellm.proxy.proxy_server import SPEND_DB_FLOOR_CACHE_TTL_SECONDS, spend_counter_cache, spend_counter_store
-
-    try:
-        await spend_counter_store.set(counter_key, value, memory_ttl=60)
-    except Exception as redis_err:
-        verbose_proxy_logger.warning(
-            "Failed to update spend counter %s in Redis: %s. Budget checks may use stale value until counter expires.",
-            counter_key,
-            redis_err,
-        )
-
-    floor_key: Final = f"spend_db_floor:{counter_key}"
-    spend_counter_cache.in_memory_cache.set_cache(key=floor_key, value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
-
-    await publish_auth_cache_invalidation(cache_key=counter_key, new_value=value, ttl=60)
-    await publish_auth_cache_invalidation(cache_key=floor_key, new_value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
-
-
 def _budget_limit_windows(budget_limits: Sequence[object] | str | None) -> tuple[Mapping[str, object], ...]:
     """Coerce a key's stored `budget_limits` into a tuple of plain window dicts.
 
@@ -6925,6 +6891,8 @@ async def _reset_key_budget_windows(
     pre-reset boundary, re-sum the unchanged historical spend, and put the
     counter right back where it was before the write ever landed.
     """
+    from litellm.proxy.proxy_server import set_spend_counter_with_floor_and_broadcast
+
     windows: Final = _budget_limit_windows(budget_limits)
     if not windows:
         return
@@ -6943,7 +6911,7 @@ async def _reset_key_budget_windows(
         duration = window.get("budget_duration")
         if isinstance(duration, str) and duration:
             counter_key = key_window_counter(hashed_api_key, duration)
-            await _set_spend_counter_with_floor_and_broadcast(counter_key=counter_key, value=0.0)
+            await set_spend_counter_with_floor_and_broadcast(counter_key=counter_key, value=0.0)
 
 
 @router.post(
@@ -6966,6 +6934,7 @@ async def reset_key_spend_fn(
             hash_token,
             prisma_client,
             proxy_logging_obj,
+            set_spend_counter_with_floor_and_broadcast,
             user_api_key_cache,
         )
 
@@ -7017,7 +6986,7 @@ async def reset_key_spend_fn(
         # amount for every enforcement check immediately instead of the stale
         # pre-reset value.
         _counter_key: Final = key_counter(hashed_api_key)
-        await _set_spend_counter_with_floor_and_broadcast(counter_key=_counter_key, value=reset_to)
+        await set_spend_counter_with_floor_and_broadcast(counter_key=_counter_key, value=reset_to)
         await _reset_key_budget_windows(
             prisma_client=prisma_client,
             hashed_api_key=hashed_api_key,

litellm/proxy/management_endpoints/password_endpoints.py

@@ -33,7 +33,7 @@ from litellm.proxy.auth.password_policy import (
 from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
 from litellm.proxy.management_endpoints.session_endpoints import revoke_ui_session_keys
 from litellm.proxy.management_helpers.audit_logs import create_object_audit_log
-from litellm.proxy.utils import hash_password, verify_password
+from litellm.proxy.utils import hash_password, run_off_loop_if_fips, verify_password
 from litellm.repositories.prisma_protocols import TableActions
 from litellm.repositories.user_repository import UserRepository
 
@@ -129,7 +129,7 @@ async def change_password(
             ),
         )
 
-    if not verify_password(data.current_password, stored_password):
+    if not await run_off_loop_if_fips(verify_password, data.current_password, stored_password):
         raise HTTPException(status_code=400, detail=_error_detail("Current password is incorrect."))
 
     if data.new_password == data.current_password:
@@ -142,7 +142,7 @@ async def change_password(
     await validate_password_not_breached(data.new_password, general_settings, hibp_client)
 
     password_update: Final[prisma_types.LiteLLM_UserTableUpdateInput] = {
-        "password": hash_password(data.new_password),
+        "password": await run_off_loop_if_fips(hash_password, data.new_password),
         "password_reset_required": False,
         "last_breach_check_at": None,
     }

litellm/proxy/management_endpoints/tag_management_endpoints.py

@@ -30,6 +30,7 @@ from litellm.proxy.management_endpoints.common_daily_activity import (
     SpendAnalyticsPaginatedResponse,
     get_daily_activity,
 )
+from litellm.proxy.management_endpoints.common_utils import validate_finite_spend
 from litellm.proxy.management_helpers.utils import handle_budget_for_entity
 from litellm.redis.stores.spend_counters import (
     tag_counter,
@@ -415,6 +416,7 @@ async def update_tag(
 
     Parameters:
     - name: str - The name of the tag to update
+    - spend: Optional[float] - Updated spend for the tag
     - description: Optional[str] - Updated description
     - models: List[str] - Updated list of allowed LLM models
     - budget_id: Optional[str] - The id for a budget to associate with the tag
@@ -428,11 +430,13 @@ async def update_tag(
     - model_max_budget: Optional[dict] - Max budget for a specific model
     - budget_duration: Optional[str] - Frequency of resetting tag budget
     """
-    from litellm.proxy.proxy_server import prisma_client
+    from litellm.proxy.proxy_server import prisma_client, set_spend_counter_with_floor_and_broadcast
 
     if prisma_client is None:
         raise HTTPException(status_code=500, detail="Database not connected")
 
+    validate_finite_spend(tag.spend)
+
     try:
         # Check if tag exists
         existing_tag: Final = await _table(TagRepository(prisma_client)).find_unique(where={"tag_name": tag.name})
@@ -459,12 +463,10 @@ async def update_tag(
             "description": tag.description,
             "models": tag.models or [],
             "model_info": json.dumps(model_info),
+            **({"spend": tag.spend} if tag.spend is not None else {}),
+            **({"budget_id": budget_id} if budget_id != existing_tag.budget_id else {}),
         }
 
-        # Add budget_id if it changed
-        if budget_id != existing_tag.budget_id:
-            update_data["budget_id"] = budget_id
-
         # Update tag in database
         updated_tag_record: Final = await _table(TagRepository(prisma_client)).update(
             where={"tag_name": tag.name},
@@ -473,6 +475,9 @@ async def update_tag(
 
         await _evict_tag_cache_keys((tag_cache_key(tag.name),))
 
+        if tag.spend is not None:
+            await set_spend_counter_with_floor_and_broadcast(counter_key=f"spend:tag:{tag.name}", value=tag.spend)
+
         # Build response
         tag_config: Final = TagConfig(
             name=updated_tag_record.tag_name,

litellm/proxy/management_helpers/utils.py

@@ -217,7 +217,7 @@ async def handle_budget_for_entity(
     )
 
     # Get all budget field names
-    budget_params: Final = LiteLLM_BudgetTable.model_fields.keys()
+    budget_params: Final = LiteLLM_BudgetTable.model_fields.keys() - {"budget_id"}
 
     # Extract budget fields from data
     _json_data: Final = _BUDGET_DATA_MAPPING.validate_python(

litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py

@@ -1,15 +1,94 @@
+import asyncio
+import hashlib
 import hmac
+import json
+import re
+from collections.abc import Iterator, Mapping
+from dataclasses import dataclass
 from types import MappingProxyType
-from typing import Annotated, Final, Literal
+from typing import TYPE_CHECKING, Annotated, Final, Literal, Protocol
 
+import httpx
 from fastapi import APIRouter, Depends, HTTPException, Request, Response
+from fastapi.responses import StreamingResponse
+from pydantic import TypeAdapter, ValidationError
+from starlette.types import Message
 
+import litellm
+from litellm._internal_context import with_service_target
+from litellm._logging import verbose_proxy_logger
+from litellm.exceptions import BadRequestError
+from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, normalize_drop_params
+from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
 from litellm.llms.chatgpt.common_utils import CHATGPT_API_BASE
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.llms.custom_httpx.http_handler import (
+    get_async_httpx_client,  # pyright: ignore[reportUnknownVariableType]  # shared client factory has untyped params
+)
+from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
+from litellm.proxy.auth.auth_checks import check_model_access_helper, check_tools_allowlist
+from litellm.proxy.auth.handle_jwt import JWTAuthManager
+from litellm.proxy.auth.route_checks import RouteChecks
 from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
+from litellm.proxy.common_utils.codex_model_catalog import (
+    CodexCatalogRow,
+    bundled_codex_models,
+    codex_catalog_etag,
+    codex_entry_names_default_reasoning_level,
+    merged_codex_catalog,
+    upstream_codex_slugs,
+)
+from litellm.proxy.common_utils.http_parsing_utils import (
+    get_client_requested_model,
+    read_request_body,  # pyright: ignore[reportUnknownVariableType]  # returns a bare dict, validated at the call
+)
+from litellm.proxy.litellm_pre_call_utils import should_auto_drop_params_for_agentic_cli
 from litellm.proxy.pass_through_endpoints.pass_through_endpoints import pass_through_request
+from litellm.proxy.response_api_endpoints.endpoints import responses_api
+from litellm.responses.additional_tools import with_additional_tools_hoisted
+from litellm.responses.litellm_completion_transformation.reasoning_items import is_other_provider_reasoning_item
+from litellm.responses.utils import ResponsesAPIRequestUtils
+from litellm.types.llms.custom_http import httpxSpecialProvider
+from litellm.types.router import Deployment
+from litellm.types.utils import LlmProviders
+from litellm.utils import ProviderConfigManager, supports_none_reasoning_effort
+
+if TYPE_CHECKING:
+    from litellm.router import Router
 
 router: Final = APIRouter()
+_CHATGPT_PROVIDER: Final = "chatgpt"
+_CHATGPT_CREDENTIAL_HEADERS: Final = frozenset({b"authorization", b"chatgpt-account-id"})
+_MERGED_CATALOG_DROPPED_HEADERS: Final = frozenset({"etag", "content-length", "content-type"})
+_ETAG_HEADER: Final = "etag"
+_MODELS_ETAG_HEADER: Final = "x-models-etag"
+_ENCRYPTED_CONTENT_AFFINITY_FLAG: Final = "encrypted_content_affinity_enabled"
+_LITELLM_RESPONSES_PATH: Final = "/v1/responses"
+_CODEX_NO_DEFAULT_EFFORT: Final = "none"
+_NONE_EFFORT_SUPPORT_FLAG: Final = "supports_none_reasoning_effort"
+_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
+_INPUT_ITEMS: Final = TypeAdapter(tuple[object, ...])
+_CACHED_SLUGS: Final = TypeAdapter(frozenset[str])
+_CACHED_MODELS_ETAG: Final = TypeAdapter(dict[Literal["etag"], str])
+_MODEL_NAMES: Final = TypeAdapter(list[str])
+_HEADER_DERIVED_STATE: Final = frozenset({"_cached_headers"})
+_ACCOUNT_CATALOG_HEADERS: Final = frozenset({"authorization", "chatgpt-account-id", "originator", "user-agent"})
+_CODEX_USER_AGENT_PRODUCT: Final = re.compile(r"([^/]+)/(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.+-]+)?)")
+_ACCOUNT_CATALOG_TIMEOUT_SECONDS: Final = 5.0
+_SHARED_CACHE_TTL_SECONDS: Final = 3600
+_ACCOUNT_SLUGS_UNAVAILABLE: Final = "unavailable"
+_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS: Final = 60
+_ACCOUNT_SLUGS_CACHE_PREFIX: Final = "codex_account_slugs:"
+_ACCOUNT_SLUGS_TARGET: Final = "codex_account_slugs"
+_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX: Final = "codex_account_slugs_fetches:"
+_ACCOUNT_SLUGS_FETCHES_PER_CALLER: Final = 5
+_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS: Final = 60
+_MODELS_ETAG_CACHE_PREFIX: Final = "codex_models_etag:"
+_MODELS_ETAG_TTL_SECONDS: Final = 300
+_OPENAI_API_PROVIDERS: Final = frozenset({LlmProviders.OPENAI.value, LlmProviders.AZURE.value})
+_CODEX_TOOL_PROVIDERS: Final = frozenset(
+    {*_OPENAI_API_PROVIDERS, LlmProviders.FIREWORKS_AI.value, LlmProviders.BEDROCK_MANTLE.value}
+)
+_CHAT_COMPLETIONS_BRIDGE: Final = "use_chat_completions_api"
 _HEADERS: Final = frozenset(
     {
         "authorization",
@@ -29,10 +108,637 @@ _HEADERS: Final = frozenset(
 )
 
 
-@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"])
+def _is_chatgpt(deployment: Deployment) -> bool:
+    return (
+        deployment.litellm_params.custom_llm_provider == _CHATGPT_PROVIDER
+        or deployment.litellm_params.model.partition("/")[0] == _CHATGPT_PROVIDER
+    )
+
+
+def _serves_by_name(deployment: Deployment, names: frozenset[str]) -> bool:
+    return not _is_chatgpt(deployment) and (
+        deployment.model_name in names or deployment.model_info.team_public_model_name in names
+    )
+
+
+def _candidate_deployments(model: str, llm_router: "Router", team_id: str | None) -> tuple[Deployment, ...]:
+    registry: Final = llm_router.registry()
+    candidates: Final = tuple(
+        registry.deployment(model_id) for model_id in registry.names(model, kind="route_candidates", team_id=team_id)
+    )
+    return tuple(deployment for deployment in candidates if deployment is not None)
+
+
+def routes_to_litellm(model: object, llm_router: "Router | None", team_id: str | None) -> bool:
+    if (
+        not isinstance(model, str)
+        or llm_router is None
+        or model in bundled_codex_models()
+        or f"{_CHATGPT_PROVIDER}/{model}" in litellm.model_cost  # pyright: ignore[reportUnknownMemberType]  # registry is dynamically typed
+    ):
+        return False
+    deployments: Final = _candidate_deployments(model, llm_router, team_id)
+    names: Final = frozenset({model, llm_router.registry().resolve(model, kind="alias") or model})
+    return bool(deployments) and all(_serves_by_name(deployment, names) for deployment in deployments)
+
+
+def _key_and_team_allow(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> bool:
+    allowed_lists: Final = (
+        _MODEL_NAMES.validate_python(user_api_key_dict.models),
+        _MODEL_NAMES.validate_python(user_api_key_dict.team_models),
+    )
+    return all(
+        check_model_access_helper(model, llm_router, allowed, team_id=user_api_key_dict.team_id)
+        for allowed in allowed_lists
+    )
+
+
+def native_chatgpt_model(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> str | None:
+    if model.startswith(f"{_CHATGPT_PROVIDER}/") or _key_and_team_allow(model, llm_router, user_api_key_dict):
+        return None
+    chatgpt_model: Final = f"{_CHATGPT_PROVIDER}/{model}"
+    deployments: Final = _candidate_deployments(chatgpt_model, llm_router, user_api_key_dict.team_id)
+    if not deployments or not all(_is_chatgpt(deployment) for deployment in deployments):
+        return None
+    return chatgpt_model if _key_and_team_allow(chatgpt_model, llm_router, user_api_key_dict) else None
+
+
+class _SharedCache(Protocol):
+    async def async_get_cache(self, key: str) -> object: ...
+
+    async def async_set_cache(self, key: str, value: object, *, ttl: int) -> None: ...
+
+    async def async_increment_cache(self, key: str, value: float, *, ttl: int) -> float | None: ...
+
+
+def codex_shared_cache() -> _SharedCache:
+    from litellm.proxy.proxy_server import proxy_logging_obj
+
+    return proxy_logging_obj.internal_usage_cache.dual_cache
+
+
+def _key_may_call_litellm_responses(user_api_key_dict: UserAPIKeyAuth) -> bool:
+    if not RouteChecks.is_allowed_on_proxy(_LITELLM_RESPONSES_PATH):
+        return False
+    try:
+        _ = RouteChecks.should_call_route(route=_LITELLM_RESPONSES_PATH, valid_token=user_api_key_dict)
+        if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
+            RouteChecks.custom_admin_only_route_check(route=_LITELLM_RESPONSES_PATH)
+    except HTTPException:
+        return False
+    return True
+
+
+async def _jwt_policy_admits_litellm_responses(
+    gateway_token: str, model: str, request_headers: Mapping[str, str]
+) -> bool:
+    from litellm.proxy.proxy_server import (
+        general_settings_view,
+        jwt_handler,
+        prisma_client,
+        proxy_logging_obj,
+        user_api_key_cache,
+    )
+
+    try:
+        _ = await JWTAuthManager.auth_builder(
+            api_key=gateway_token,
+            jwt_handler=jwt_handler,
+            request_data={"model": model},
+            general_settings=dict(general_settings_view()),
+            route=_LITELLM_RESPONSES_PATH,
+            prisma_client=prisma_client,
+            user_api_key_cache=user_api_key_cache,
+            parent_otel_span=None,
+            proxy_logging_obj=proxy_logging_obj,
+            request_headers=request_headers,
+            request_method="POST",
+        )
+    except Exception:  # noqa: BLE001  # any JWT policy refusal keeps the caller on ChatGPT's models
+        return False
+    return True
+
+
+@dataclass(frozen=True, slots=True)
+class _LitellmResponsesAccess:
+    key_admits: bool
+    jwt_token: str | None
+    request_headers: Mapping[str, str]
+
+    async def admits(self, model: str) -> bool:
+        if not self.key_admits:
+            return False
+        if self.jwt_token is None:
+            return True
+        return await _jwt_policy_admits_litellm_responses(self.jwt_token, model, self.request_headers)
+
+
+def _litellm_responses_access(user_api_key_dict: UserAPIKeyAuth, request: Request) -> _LitellmResponsesAccess:
+    standard_jwt: Final = bool(user_api_key_dict.jwt_claims) and not user_api_key_dict.token
+    return _LitellmResponsesAccess(
+        key_admits=_key_may_call_litellm_responses(user_api_key_dict),
+        jwt_token=_gateway_token(request) if standard_jwt else None,
+        request_headers=request.headers,
+    )
+
+
+def _account_slugs_cache_key(headers: Mapping[str, str]) -> str | None:
+    account_id: Final = headers.get("chatgpt-account-id")
+    if not account_id:
+        return None
+    caller: Final = json.dumps([account_id, headers.get("authorization", "")])
+    return f"{_ACCOUNT_SLUGS_CACHE_PREFIX}{hashlib.sha256(caller.encode()).hexdigest()}"
+
+
+def _cached_slugs(cached: object) -> frozenset[str] | None:
+    if cached is None:
+        return None
+    try:
+        return _CACHED_SLUGS.validate_python(cached)
+    except ValidationError:
+        return None
+
+
+async def _remember_account_slugs(cache_key: str, slugs: frozenset[str] | None, cache: _SharedCache) -> None:
+    if slugs is None:
+        await cache.async_set_cache(cache_key, _ACCOUNT_SLUGS_UNAVAILABLE, ttl=_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS)
+        return
+    await cache.async_set_cache(cache_key, tuple(sorted(slugs)), ttl=_SHARED_CACHE_TTL_SECONDS)
+
+
+def _codex_client_version(headers: Mapping[str, str]) -> str | None:
+    product: Final = _CODEX_USER_AGENT_PRODUCT.fullmatch(headers.get("user-agent", "").partition(" ")[0])
+    if product is None or product.group(1) != headers.get("originator"):
+        return None
+    return product.group(2)
+
+
+async def _fetched_account_slugs(headers: Mapping[str, str], client_version: str) -> frozenset[str] | None:
+    client: Final = get_async_httpx_client(
+        llm_provider=httpxSpecialProvider.PassThroughEndpoint,
+        params={"timeout": _ACCOUNT_CATALOG_TIMEOUT_SECONDS},
+    )
+    try:
+        response: Final = await client.client.get(
+            f"{CHATGPT_API_BASE}/models",
+            params={"client_version": client_version},
+            headers={name: value for name, value in headers.items() if name in _ACCOUNT_CATALOG_HEADERS},
+        )
+    except httpx.HTTPError:
+        return None
+    return upstream_codex_slugs(response.content) if response.status_code == 200 else None
+
+
+async def _admits_account_slugs_fetch(caller: str, cache: _SharedCache) -> bool:
+    fetches: Final = await cache.async_increment_cache(
+        f"{_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX}{caller}", 1, ttl=_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS
+    )
+    return fetches is not None and fetches <= _ACCOUNT_SLUGS_FETCHES_PER_CALLER
+
+
+@with_service_target(_ACCOUNT_SLUGS_TARGET)
+async def _account_slugs(headers: Mapping[str, str], caller: str, cache: _SharedCache) -> frozenset[str] | None:
+    cache_key: Final = _account_slugs_cache_key(headers)
+    if cache_key is None:
+        return None
+    cached_value: Final = await cache.async_get_cache(cache_key)
+    if cached_value == _ACCOUNT_SLUGS_UNAVAILABLE:
+        return None
+    cached: Final = _cached_slugs(cached_value)
+    if cached is not None:
+        return cached
+    client_version: Final = _codex_client_version(headers)
+    if client_version is None or not await _admits_account_slugs_fetch(caller, cache):
+        return None
+    fetched: Final = await _fetched_account_slugs(headers, client_version)
+    await _remember_account_slugs(cache_key, fetched, cache)
+    return fetched
+
+
+async def _serves_through_litellm(
+    model: object,
+    llm_router: "Router | None",
+    user_api_key_dict: UserAPIKeyAuth,
+    access: _LitellmResponsesAccess,
+    caller: str,
+    headers: Mapping[str, str],
+    cache: _SharedCache,
+) -> bool:
+    if not isinstance(model, str) or not routes_to_litellm(model, llm_router, user_api_key_dict.team_id):
+        return False
+    if not await access.admits(model):
+        return False
+    account_slugs: Final = await _account_slugs(headers, caller, cache)
+    return account_slugs is not None and model not in account_slugs
+
+
+def _gateway_token(request: Request) -> str:
+    gateway_parts: Final = request.headers.get("x-litellm-api-key", "").split()
+    return (
+        gateway_parts[-1]
+        if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
+        else ""
+    )
+
+
+def _chatgpt_bearer(request: Request) -> str:
+    scheme, _, token = request.headers.get("authorization", "").partition(" ")
+    return token.strip() if scheme.lower() == "bearer" else ""
+
+
+def _forwarded_headers(request: Request) -> Mapping[str, str]:
+    return MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
+
+
+async def native_turn_serves_through_litellm(
+    model: str, llm_router: "Router", request: Request, user_api_key_dict: UserAPIKeyAuth
+) -> bool:
+    chatgpt_bearer: Final = _chatgpt_bearer(request)
+    gateway_token: Final = _gateway_token(request)
+    if not chatgpt_bearer or not gateway_token or hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
+        return False
+    return await _serves_through_litellm(
+        model,
+        llm_router,
+        user_api_key_dict,
+        _litellm_responses_access(user_api_key_dict, request),
+        _models_etag_caller(user_api_key_dict, gateway_token),
+        _forwarded_headers(request),
+        codex_shared_cache(),
+    )
+
+
+def _json_object(value: object) -> Mapping[str, object]:
+    try:
+        return _JSON_OBJECT.validate_python(value)
+    except ValidationError:
+        return MappingProxyType({})
+
+
+def _scope_state(request: Request) -> Mapping[str, object]:
+    return _json_object(request.scope.get("state"))
+
+
+def _as_litellm_responses_request(request: Request, body: bytes, parsed_body: Mapping[str, object]) -> Request:
+    replayed: Final[Iterator[Message]] = iter(({"type": "http.request", "body": body, "more_body": False},))
+
+    async def receive() -> Message:
+        return next(replayed, None) or await request.receive()
+
+    return Request(
+        {
+            **request.scope,
+            "path": _LITELLM_RESPONSES_PATH,
+            "raw_path": _LITELLM_RESPONSES_PATH.encode(),
+            "headers": [
+                (name, value) for name, value in request.headers.raw if name not in _CHATGPT_CREDENTIAL_HEADERS
+            ],
+            "parsed_body": (tuple(parsed_body), parsed_body),
+            "state": {
+                name: value for name, value in _scope_state(request).items() if name not in _HEADER_DERIVED_STATE
+            },
+        },
+        receive,
+    )
+
+
+def _drops_params(body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None) -> bool:
+    deployments: Final = _candidate_deployments(model, llm_router, team_id)
+    return drops_unsupported_params(body.get("drop_params")) or (
+        bool(deployments)
+        and all(normalize_drop_params(deployment.litellm_params.drop_params) is True for deployment in deployments)
+    )
+
+
+def _deployment_supports_no_effort(deployment: Deployment) -> bool:
+    configured: Final = _json_object(deployment.model_info.model_extra).get(_NONE_EFFORT_SUPPORT_FLAG)
+    if isinstance(configured, bool):
+        return configured
+    return supports_none_reasoning_effort(
+        model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
+    )
+
+
+def _supports_no_effort(model: str, llm_router: "Router", team_id: str | None) -> bool:
+    deployments: Final = _candidate_deployments(model, llm_router, team_id)
+    return bool(deployments) and all(_deployment_supports_no_effort(deployment) for deployment in deployments)
+
+
+def _with_provider_default_reasoning(
+    body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, client_version: str | None
+) -> Mapping[str, object]:
+    model: Final = body.get("model")
+    try:
+        reasoning: Final = _JSON_OBJECT.validate_python(body.get("reasoning"))
+    except ValidationError:
+        return body
+    if (
+        reasoning.get("effort") != _CODEX_NO_DEFAULT_EFFORT
+        or not isinstance(model, str)
+        or llm_router is None
+        or not _drops_params(body, model, llm_router, team_id)
+        or codex_entry_names_default_reasoning_level(model, llm_router, team_id, client_version)
+        or _supports_no_effort(model, llm_router, team_id)
+    ):
+        return body
+    return {**body, "reasoning": {key: value for key, value in reasoning.items() if key != "effort"}}
+
+
+def _with_wrapped_reasoning(body: Mapping[str, object]) -> Mapping[str, object]:
+    return {
+        **body,
+        "litellm_metadata": {**_json_object(body.get("litellm_metadata")), _ENCRYPTED_CONTENT_AFFINITY_FLAG: True},
+    }
+
+
+def _with_reasoning_the_targets_can_decrypt(
+    body: Mapping[str, object], model: str, llm_router: "Router", deployments: tuple[Deployment, ...]
+) -> Mapping[str, object]:
+    try:
+        items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
+    except ValidationError:
+        return body
+    model_group: Final = llm_router.registry().resolve(model, kind="alias") or model
+    kept: Final = llm_router.registry().decryptable_reasoning(model_group, items, deployments)
+    if body.get("store") is True:
+        return {**body, "input": list(kept)}
+    resolvable: Final = (_without_unpersisted_reasoning_reference(item) for item in kept)
+    return {**body, "input": [item for item in resolvable if item is not None]}
+
+
+def _without_unpersisted_reasoning_reference(item: object) -> object | None:
+    if not _is_unpersisted_reasoning_reference(item):
+        return item
+    return ResponsesAPIRequestUtils.readable_reasoning(_JSON_OBJECT.validate_python(item))
+
+
+def _model_and_provider(deployment: Deployment) -> tuple[str, str] | None:
+    try:
+        model, provider, _, _ = get_llm_provider(
+            model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
+        )
+    except BadRequestError:
+        return None
+    return model, provider
+
+
+def _is_openai_api(deployment: Deployment) -> bool:
+    resolved: Final = _model_and_provider(deployment)
+    return resolved is not None and resolved[1] in _OPENAI_API_PROVIDERS
+
+
+def _takes_codex_tools(deployment: Deployment) -> bool:
+    resolved: Final = _model_and_provider(deployment)
+    if resolved is None:
+        return False
+    model, provider = resolved
+    return (
+        provider in _CODEX_TOOL_PROVIDERS
+        or ProviderConfigManager.get_provider_responses_api_config(
+            provider=provider, model=model, api_base=deployment.litellm_params.api_base
+        )
+        is None
+    )
+
+
+def _with_tools_the_targets_accept(
+    body: Mapping[str, object], deployments: tuple[Deployment, ...], drop_params: bool
+) -> Mapping[str, object]:
+    if all(_is_openai_api(deployment) for deployment in deployments):
+        return body
+    if all(_takes_codex_tools(deployment) for deployment in deployments):
+        return with_additional_tools_hoisted(body, drop_params=drop_params)
+    return {**body, _CHAT_COMPLETIONS_BRIDGE: True}
+
+
+def _drops_params_for_client(
+    body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None, user_agent: str
+) -> bool:
+    from litellm.proxy.proxy_server import proxy_config
+
+    return _drops_params(body, model, llm_router, team_id) or should_auto_drop_params_for_agentic_cli(
+        user_agent, "drop_params" in body, proxy_config
+    )
+
+
+def _litellm_body(
+    body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, headers: Mapping[str, str]
+) -> Mapping[str, object]:
+    wrapped: Final = _with_wrapped_reasoning(
+        _with_provider_default_reasoning(body, llm_router, team_id, _codex_client_version(headers))
+    )
+    model: Final = body.get("model")
+    if not isinstance(model, str) or llm_router is None:
+        return wrapped
+    deployments: Final = _candidate_deployments(model, llm_router, team_id)
+    return _with_tools_the_targets_accept(
+        _with_reasoning_the_targets_can_decrypt(wrapped, model, llm_router, deployments),
+        deployments,
+        _drops_params_for_client(body, model, llm_router, team_id, headers.get("user-agent", "")),
+    )
+
+
+def _is_unpersisted_reasoning_reference(item: object) -> bool:
+    try:
+        fields: Final = _JSON_OBJECT.validate_python(item)
+    except ValidationError:
+        return False
+    item_id: Final = fields.get("id")
+    encrypted_content: Final = fields.get("encrypted_content")
+    return (
+        fields.get("type") == "reasoning"
+        and isinstance(item_id, str)
+        and bool(item_id)
+        and not (isinstance(encrypted_content, str) and bool(encrypted_content))
+    )
+
+
+def _rejected_by_chatgpt(item: object, stored: bool) -> bool:
+    return is_other_provider_reasoning_item(item) or (not stored and _is_unpersisted_reasoning_reference(item))
+
+
+def _input_chatgpt_accepts(body: Mapping[str, object]) -> tuple[object, ...] | None:
+    try:
+        items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
+    except ValidationError:
+        return None
+    stored: Final = body.get("store") is True
+    kept: Final = tuple(item for item in items if not _rejected_by_chatgpt(item, stored))
+    return kept if len(kept) < len(items) else None
+
+
+def _chatgpt_body(body: Mapping[str, object], client_model: str | None) -> Mapping[str, object] | None:
+    kept_input: Final = _input_chatgpt_accepts(body)
+    restored_model: Final = client_model is not None and body.get("model") == f"{_CHATGPT_PROVIDER}/{client_model}"
+    if kept_input is None and not restored_model:
+        return None
+    return {
+        **body,
+        **({"input": list(kept_input)} if kept_input is not None else {}),
+        **({"model": client_model} if restored_model else {}),
+    }
+
+
+async def _litellm_catalog_rows(
+    user_api_key_dict: UserAPIKeyAuth, access: _LitellmResponsesAccess, llm_router: "Router | None"
+) -> tuple[CodexCatalogRow, ...]:
+    from litellm.proxy.proxy_server import codex_catalog_rows_for_key
+
+    if llm_router is None or not access.key_admits:
+        return ()
+    try:
+        rows: Final = await codex_catalog_rows_for_key(user_api_key_dict)
+    except Exception as error:  # noqa: BLE001  # a failed listing must never fail the ChatGPT answer it extends
+        verbose_proxy_logger.warning(
+            "Codex catalog keeps only ChatGPT's models: listing the key's LiteLLM models raised %s",
+            type(error).__name__,
+        )
+        return ()
+    routed: Final = tuple(row for row in rows if routes_to_litellm(row.id, llm_router, user_api_key_dict.team_id))
+    admitted: Final = await asyncio.gather(*(access.admits(row.id) for row in routed))
+    return tuple(row for row, row_admitted in zip(routed, admitted, strict=True) if row_admitted)
+
+
+def _models_etag_caller(user_api_key_dict: UserAPIKeyAuth, gateway_token: str) -> str:
+    return user_api_key_dict.token or hashlib.sha256(gateway_token.encode()).hexdigest()
+
+
+def _models_etag_cache_key(caller: str, upstream_etag: str) -> str:
+    return f"{_MODELS_ETAG_CACHE_PREFIX}{hashlib.sha256(json.dumps([caller, upstream_etag]).encode()).hexdigest()}"
+
+
+async def _remembered_models_etag(
+    caller: str, upstream_etag: str, rows: tuple[CodexCatalogRow, ...], cache: _SharedCache
+) -> str:
+    etag: Final = codex_catalog_etag(upstream_etag, rows) if rows else upstream_etag
+    await cache.async_set_cache(
+        _models_etag_cache_key(caller, upstream_etag), {"etag": etag}, ttl=_MODELS_ETAG_TTL_SECONDS
+    )
+    return etag
+
+
+async def _cached_models_etag(caller: str, upstream_etag: str, cache: _SharedCache) -> str | None:
+    try:
+        return _CACHED_MODELS_ETAG.validate_python(
+            await cache.async_get_cache(_models_etag_cache_key(caller, upstream_etag))
+        )["etag"]
+    except ValidationError:
+        return None
+
+
+async def _key_models_etag(
+    upstream_etag: str,
+    user_api_key_dict: UserAPIKeyAuth,
+    access: _LitellmResponsesAccess,
+    caller: str,
+    llm_router: "Router | None",
+    cache: _SharedCache,
+) -> str:
+    cached: Final = await _cached_models_etag(caller, upstream_etag, cache)
+    if cached is not None:
+        return cached
+    rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
+    return await _remembered_models_etag(caller, upstream_etag, rows, cache)
+
+
+def _with_etag(response: Response, content: bytes | str, etag: str | None) -> Response:
+    kept_headers: Final = {
+        name: value for name, value in response.headers.items() if name not in _MERGED_CATALOG_DROPPED_HEADERS
+    }
+    return Response(
+        content=content,
+        status_code=response.status_code,
+        headers={**kept_headers, **({_ETAG_HEADER: etag} if etag is not None else {})},
+        media_type="application/json",
+    )
+
+
+async def _catalog(
+    request: Request,
+    user_api_key_dict: UserAPIKeyAuth,
+    access: _LitellmResponsesAccess,
+    caller: str,
+    headers: Mapping[str, str],
+    cache: _SharedCache,
+) -> Response:
+    from litellm.proxy.proxy_server import llm_router
+
+    upstream: Final = await pass_through_request(
+        request=request,
+        target=f"{CHATGPT_API_BASE}/models",
+        custom_headers=dict(headers),
+        user_api_key_dict=user_api_key_dict,
+        custom_llm_provider=_CHATGPT_PROVIDER,
+        upstream_headers_filter=lambda _: headers,
+    )
+    if upstream.status_code != 200 or isinstance(upstream, StreamingResponse):
+        return upstream
+    upstream_body: Final = bytes(upstream.body)
+    slugs: Final = upstream_codex_slugs(upstream_body)
+    if slugs is None:
+        return upstream
+    account_slugs_key: Final = _account_slugs_cache_key(headers)
+    if account_slugs_key is not None:
+        await _remember_account_slugs(account_slugs_key, slugs, cache)
+    rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
+    upstream_etag: Final = upstream.headers.get(_ETAG_HEADER)
+    etag: Final = None if upstream_etag is None else await _remembered_models_etag(caller, upstream_etag, rows, cache)
+    if not rows:
+        return upstream
+    merged: Final = await asyncio.to_thread(
+        merged_codex_catalog, upstream_body, rows, client_version=request.query_params.get("client_version")
+    )
+    return _with_etag(upstream, merged if merged is not None else upstream_body, etag)
+
+
+async def _responses(
+    request: Request,
+    fastapi_response: Response,
+    user_api_key_dict: UserAPIKeyAuth,
+    access: _LitellmResponsesAccess,
+    caller: str,
+    headers: Mapping[str, str],
+    cache: _SharedCache,
+) -> object:
+    from litellm.proxy.proxy_server import llm_router
+
+    body: Final = _JSON_OBJECT.validate_python(await read_request_body(request))
+    if await _serves_through_litellm(body.get("model"), llm_router, user_api_key_dict, access, caller, headers, cache):
+        litellm_body: Final = _litellm_body(body, llm_router, user_api_key_dict.team_id, headers)
+        await check_tools_allowlist(
+            request_body=dict(litellm_body),
+            valid_token=user_api_key_dict,
+            team_object=None,
+            route=_LITELLM_RESPONSES_PATH,
+        )
+        return await responses_api(
+            _as_litellm_responses_request(request, json.dumps(litellm_body).encode(), litellm_body),
+            fastapi_response,
+            user_api_key_dict.model_copy(update={"request_route": _LITELLM_RESPONSES_PATH}),
+        )
+    chatgpt_body: Final = _chatgpt_body(body, get_client_requested_model(request))
+    upstream: Final = await pass_through_request(
+        request=request,
+        target=f"{CHATGPT_API_BASE}/responses",
+        custom_headers=dict(headers),
+        user_api_key_dict=user_api_key_dict,
+        custom_body=dict(chatgpt_body) if chatgpt_body is not None else None,
+        custom_llm_provider=_CHATGPT_PROVIDER,
+        upstream_headers_filter=lambda _: headers,
+    )
+    upstream_models_etag: Final = upstream.headers.get(_MODELS_ETAG_HEADER)
+    if upstream_models_etag is not None:
+        upstream.headers[_MODELS_ETAG_HEADER] = await _key_models_etag(
+            upstream_models_etag, user_api_key_dict, access, caller, llm_router, cache
+        )
+    return upstream
+
+
+@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"], response_model=None)
 @router.post(
     "/backend-api/codex/{endpoint}",
     tags=["ChatGPT Pass-through", "pass-through"],
+    response_model=None,
     openapi_extra={
         "requestBody": {
             "required": True,
@@ -41,7 +747,7 @@ _HEADERS: Final = frozenset(
     },
     responses={
         200: {
-            "description": "Unmodified Codex response or server-sent events",
+            "description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events",
             "content": {
                 "application/json": {"schema": {"type": "object", "additionalProperties": True}},
                 "text/event-stream": {"schema": {"type": "string"}},
@@ -52,36 +758,29 @@ _HEADERS: Final = frozenset(
 async def codex_passthrough_route(
     endpoint: Literal["models", "responses"],
     request: Request,
+    fastapi_response: Response,
     user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
-) -> Response:
-    """Forward a client's ChatGPT subscription with separate LiteLLM authentication."""
+    cache: Annotated[_SharedCache, Depends(codex_shared_cache)],
+) -> object:
+    """Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
+    key's other chat models through LiteLLM."""
     expected_method: Final = "GET" if endpoint == "models" else "POST"
     if request.method != expected_method:
         raise HTTPException(status_code=405, detail="Method not allowed", headers={"Allow": expected_method})
 
-    authorization: Final = request.headers.get("authorization", "")
-    scheme, _, token = authorization.partition(" ")
-    gateway_key: Final = request.headers.get("x-litellm-api-key", "")
-    gateway_parts: Final = gateway_key.split()
-    gateway_token: Final = (
-        gateway_parts[-1]
-        if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
-        else ""
-    )
-    if scheme.lower() != "bearer" or not token.strip() or not gateway_token:
+    chatgpt_bearer: Final = _chatgpt_bearer(request)
+    gateway_token: Final = _gateway_token(request)
+    if not chatgpt_bearer or not gateway_token:
         raise HTTPException(
             status_code=401,
             detail="Send the ChatGPT bearer in Authorization and the LiteLLM key in x-litellm-api-key",
         )
-    if hmac.compare_digest(token.strip().encode(), gateway_token.encode()):
+    if hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
         raise HTTPException(status_code=401, detail="ChatGPT and LiteLLM credentials must be separate")
 
-    headers: Final = MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
-    return await pass_through_request(
-        request=request,
-        target=f"{CHATGPT_API_BASE}/{endpoint}",
-        custom_headers=dict(headers),
-        user_api_key_dict=user_api_key_dict,
-        custom_llm_provider="chatgpt",
-        upstream_headers_filter=lambda _: headers,
-    )
+    headers: Final = _forwarded_headers(request)
+    caller: Final = _models_etag_caller(user_api_key_dict, gateway_token)
+    access: Final = _litellm_responses_access(user_api_key_dict, request)
+    if endpoint == "models":
+        return await _catalog(request, user_api_key_dict, access, caller, headers, cache)
+    return await _responses(request, fastapi_response, user_api_key_dict, access, caller, headers, cache)

litellm/proxy/proxy_server.py

@@ -223,7 +223,12 @@ from litellm.types.utils import (
     is_server_derived_pricing_key,
     pricing_override_fields,
 )
-from litellm.utils import acredential_updates, cost_map_omits_token_price, with_named_credential
+from litellm.utils import (
+    acredential_updates,
+    cost_map_omits_token_price,
+    native_structured_output_support,
+    with_named_credential,
+)
 
 if TYPE_CHECKING:
     from aiohttp import ClientSession
@@ -233,6 +238,7 @@ if TYPE_CHECKING:
 
     from litellm.integrations.opentelemetry import OpenTelemetry
     from litellm.proxy.health_check_utils.shared_health_check_manager import SharedHealthCheckManager
+    from litellm.types.proxy.model_listing import ModelInfoResponse
 
     Span = _Span | Any
 else:
@@ -515,10 +521,11 @@ from litellm.proxy.common_utils.anthropic_error_payload import (
 )
 from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
     AuthCacheInvalidationSubscriber,
+    publish_auth_cache_invalidation,
     publish_auth_cache_invalidations,
 )
 from litellm.proxy.common_utils.callback_utils import initialize_callbacks_on_proxy
-from litellm.proxy.common_utils.codex_model_catalog import codex_model_list_body
+from litellm.proxy.common_utils.codex_model_catalog import CodexCatalogRow, codex_catalog_rows, codex_model_list_body
 from litellm.proxy.common_utils.config_includes import resolve_include_file_path, resolve_includes
 from litellm.proxy.common_utils.config_sync_pubsub import ConfigSyncSubscriber
 from litellm.proxy.common_utils.credential_hydration import decrypted_or_stored
@@ -557,6 +564,7 @@ from litellm.proxy.common_utils.http_parsing_utils import (
 from litellm.proxy.common_utils.load_config_utils import get_config_from_bucket
 from litellm.proxy.common_utils.model_deprecation import collect_model_deprecations
 from litellm.proxy.common_utils.model_listing_utils import (
+    CallerAliases,
     ClaudeCodeRoutingNames,
     DeployedModelGroups,
     TeamModelNameTranslator,
@@ -726,6 +734,7 @@ from litellm.proxy.litellm_pre_call_utils import (
     SCHEDULER_PRIORITY_HEADER,
     LiteLLMProxyRequestSetup,
     add_litellm_data_to_request,
+    sanitize_for_log,
 )
 from litellm.proxy.logging_endpoints.callback_logs_endpoints import (
     rust_control_plane_router,
@@ -1045,6 +1054,7 @@ from litellm.proxy.utils import (  # noqa: F401, RUF100  # legacy module exports
     migrate_passwords_to_scrypt_async,
     model_dump_with_preserved_fields,
     prefetch_config_params,
+    run_off_loop_if_fips,
     update_spend,
 )
 from litellm.proxy.video_endpoints.endpoints import router as video_router
@@ -4561,6 +4571,36 @@ def raise_if_spend_counters_unwritten(lost: Exception | None, saved: str) -> Non
     )
 
 
+@with_service_target(SPEND_COUNTERS_TARGET)
+async def set_spend_counter_with_floor_and_broadcast(counter_key: str, value: float) -> None:
+    """
+    Set a Redis-backed spend counter to `value`, mirror it into the short-lived
+    spend_db_floor marker `_authoritative_floor_spend` reads, and broadcast both
+    to every worker (LIT-3803 pattern: setting, not deleting, means a worker's
+    own self-delivered broadcast still carries the reset value forward).
+
+    Without the floor marker, `_authoritative_floor_spend` can re-derive a
+    stale, pre-reset value from a marker another worker cached moments earlier
+    and raise the just-reset counter right back up via `_repair_stale_spend_counter`.
+    Without the broadcast, a worker that already cached the pre-reset key object
+    or floor marker keeps enforcing against it until its own TTL expires.
+    """
+    try:
+        await spend_counter_store.set(counter_key, value, memory_ttl=60)
+    except Exception as redis_err:
+        verbose_proxy_logger.warning(
+            "Failed to update spend counter %s in Redis: %s. Budget checks may use stale value until counter expires.",
+            sanitize_for_log(counter_key),
+            redis_err,
+        )
+
+    floor_key: Final = f"spend_db_floor:{counter_key}"
+    spend_counter_cache.in_memory_cache.set_cache(key=floor_key, value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
+
+    await publish_auth_cache_invalidation(cache_key=counter_key, new_value=value, ttl=60)
+    await publish_auth_cache_invalidation(cache_key=floor_key, new_value=value, ttl=SPEND_DB_FLOOR_CACHE_TTL_SECONDS)
+
+
 async def _apply_spend_counter_increments(pending: Sequence[PendingSpendIncrement]) -> None:
     if _defer_spend_counter_increments(pending):
         return
@@ -12721,6 +12761,116 @@ async def _deployment_hidden_by_listing_callbacks(deployment: Deployment, user_a
     return listed_name in await _names_hidden_by_listing_callbacks(user_api_key_dict, (listed_name,))
 
 
+async def _caller_aliases(user_api_key_dict: UserAPIKeyAuth, team_id: str | None) -> CallerAliases:
+    caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
+        user_api_key_dict, prisma_client, proxy_logging_obj
+    )
+    return caller_alias_maps(
+        user_api_key_dict.aliases,
+        user_api_key_dict.team_model_aliases,
+        user_api_key_dict.team_id,
+        team_id,
+        caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
+    )
+
+
+async def _listing_hidden_names(healthy_only: bool | None) -> frozenset[str]:
+    blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
+    unhealthy_names: Final = await get_hidden_unhealthy_model_names(
+        healthy_only=healthy_only,
+        general_settings=general_settings_view(),
+        llm_router=llm_router,
+    )
+    return frozenset(blocked_names | unhealthy_names)
+
+
+@dataclass(frozen=True, slots=True)
+class _KeyModelListing:
+    rows: "tuple[ModelInfoResponse, ...]"
+    entries: tuple[tuple[str, str], ...]
+    labels: tuple[tuple[str, str], ...]
+
+
+def _listed_model_info(
+    response_id: str, lookup_id: str, include_metadata: bool, fallback_type: str | None
+) -> "ModelInfoResponse":
+    from litellm.proxy.utils import create_model_info_response
+
+    model_info: Final = create_model_info_response(
+        model_id=lookup_id,
+        provider="openai",
+        include_metadata=include_metadata,
+        fallback_type=fallback_type,
+        llm_router=llm_router,
+    )
+    model_info["id"] = response_id
+    return model_info
+
+
+async def _key_model_listing(
+    user_api_key_dict: UserAPIKeyAuth,
+    caller_aliases: CallerAliases,
+    hidden_names: frozenset[str],
+    *,
+    team_id: str | None = None,
+    include_model_access_groups: bool = False,
+    only_model_access_groups: bool = False,
+    include_wildcard_routes: bool = False,
+    include_metadata: bool = False,
+    fallback_type: str | None = None,
+    offered_model_names: Sequence[str] = (),
+) -> _KeyModelListing:
+    from litellm.proxy.utils import get_available_models_for_user
+
+    available: Final = await get_available_models_for_user(
+        user_api_key_dict=user_api_key_dict,
+        llm_router=llm_router,
+        general_settings=general_settings,
+        user_model=user_model,
+        prisma_client=prisma_client,
+        proxy_logging_obj=proxy_logging_obj,
+        team_id=team_id,
+        include_model_access_groups=include_model_access_groups,
+        only_model_access_groups=only_model_access_groups,
+        return_wildcard_routes=include_wildcard_routes,
+        user_api_key_cache=user_api_key_cache,
+    )
+    names: Final = (*available, *(name for name in offered_model_names if name not in available))
+    excluded: Final = hidden_names | undiscoverable_model_names(
+        names, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
+    )
+    listed_entries: Final = await _entries_kept_by_listing_callbacks(
+        TeamModelNameTranslator.listing_entries(
+            [name for name in names if name not in excluded], llm_router, general_settings_view()
+        ),
+        user_api_key_dict,
+    )
+    entries: Final = alias_listing_entries(
+        listed_entries,
+        caller_aliases,
+        _describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
+    )
+    return _KeyModelListing(
+        rows=tuple(
+            _listed_model_info(response_id, lookup_id, include_metadata, fallback_type)
+            for response_id, lookup_id in entries
+        ),
+        entries=entries,
+        labels=label_entries(listed_entries, entries),
+    )
+
+
+async def codex_catalog_rows_for_key(user_api_key_dict: UserAPIKeyAuth) -> tuple[CodexCatalogRow, ...]:
+    listing: Final = await _key_model_listing(
+        user_api_key_dict,
+        await _caller_aliases(user_api_key_dict, None),
+        await _listing_hidden_names(None),
+    )
+    return await asyncio.to_thread(
+        codex_catalog_rows, listing.rows, listing.entries, llm_router, user_api_key_dict.team_id, listing.labels
+    )
+
+
 @router.get("/v1/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"])
 @router.get(
     "/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"]
@@ -12788,10 +12938,7 @@ async def model_list(
     from litellm.proxy.management_endpoints.common_utils import (
         user_has_admin_privileges,
     )
-    from litellm.proxy.utils import (
-        create_model_info_response,
-        get_available_models_for_user,
-    )
+    from litellm.proxy.utils import create_model_info_response
     from litellm.types.proxy.model_listing import ModelInfoResponse
 
     http_request: Final = cast(Request | None, request)  # cast-ok: in-process callers pass no request
@@ -12799,16 +12946,7 @@ async def model_list(
         http_request is not None and http_request.headers.get("anthropic-version") is not None
     )
     client_headers: Final[Mapping[str, str]] = http_request.headers if http_request is not None else _EMPTY_HEADERS
-    caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
-        user_api_key_dict, prisma_client, proxy_logging_obj
-    )
-    caller_aliases: Final = caller_alias_maps(
-        user_api_key_dict.aliases,
-        user_api_key_dict.team_model_aliases,
-        user_api_key_dict.team_id,
-        team_id,
-        caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
-    )
+    caller_aliases: Final = await _caller_aliases(user_api_key_dict, team_id)
     routing_names: Final = ClaudeCodeRoutingNames(
         llm_router,
         team_id or user_api_key_dict.team_id,
@@ -12834,18 +12972,7 @@ async def model_list(
             proxy_logging_obj=proxy_logging_obj,
         )
 
-    # Compute once — used in both branches below to hide paused models from the listing.
-    blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
-
-    # Opt-in: also hide models whose deployments are all unhealthy per background
-    # health checks. Empty when health state is unavailable or stale (fail open).
-    unhealthy_names: Final = await get_hidden_unhealthy_model_names(
-        healthy_only=healthy_only,
-        general_settings=settings,
-        llm_router=llm_router,
-    )
-
-    hidden_names: Final = blocked_names | unhealthy_names
+    hidden_names: Final = await _listing_hidden_names(healthy_only)
 
     from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import typesafe_allowlist_model_names
 
@@ -12943,58 +13070,24 @@ async def model_list(
             object="list",
         )
 
-    # Otherwise, use the normal behavior (current implementation)
-    # Get available models for the user
-    available_models: Final = await get_available_models_for_user(
-        user_api_key_dict=user_api_key_dict,
-        llm_router=llm_router,
-        general_settings=general_settings,
-        user_model=user_model,
-        prisma_client=prisma_client,
-        proxy_logging_obj=proxy_logging_obj,
-        team_id=team_id,
-        include_model_access_groups=include_model_access_groups or False,
-        only_model_access_groups=only_model_access_groups or False,
-        return_wildcard_routes=include_wildcard_routes,
-        user_api_key_cache=user_api_key_cache,
-    )
     offers_typesafe_allowlist_names: Final = (
         asks_for_allowlist_names and team_id is None and user_api_key_has_admin_view(user_api_key_dict)
     )
-    typesafe_names_to_offer: Final = typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else ()
-    all_models = [*available_models, *(name for name in typesafe_names_to_offer if name not in available_models)]
-
-    undiscoverable_names: Final = undiscoverable_model_names(
-        all_models, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
-    )
-    if hidden_names or undiscoverable_names:
-        all_models = [m for m in all_models if m not in hidden_names and m not in undiscoverable_names]
-
-    # Surface the public team name by default; legacy internal keys via flag.
-    # The internal routing key drives the metadata/fallback lookup, while the
-    # public name is what the client sees as the model id.
-    model_data = []
-    listed_entries: Final = await _entries_kept_by_listing_callbacks(
-        TeamModelNameTranslator.listing_entries(all_models, llm_router, settings), user_api_key_dict
-    )
-    entries: Final = alias_listing_entries(
-        listed_entries,
+    key_listing: Final = await _key_model_listing(
+        user_api_key_dict,
         caller_aliases,
-        _describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
+        hidden_names,
+        team_id=team_id,
+        include_model_access_groups=include_model_access_groups or False,
+        only_model_access_groups=only_model_access_groups or False,
+        include_wildcard_routes=include_wildcard_routes,
+        include_metadata=include_metadata or False,
+        fallback_type=fallback_type,
+        offered_model_names=typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else (),
     )
-    labels: Final = label_entries(listed_entries, entries)
-    for response_id, lookup_id in entries:
-        model_info = create_model_info_response(
-            model_id=lookup_id,
-            provider="openai",
-            include_metadata=include_metadata or False,
-            fallback_type=fallback_type,
-            llm_router=llm_router,
-        )
-        model_info["id"] = response_id
-        model_data.append(model_info)
-
-    listing: Final = cast(Sequence[ModelInfoResponse], model_data)  # cast-ok: rows built above
+    listing: Final = key_listing.rows
+    entries: Final = key_listing.entries
+    labels: Final = key_listing.labels
     if client_version is not None:
         return Response(
             content=await asyncio.to_thread(
@@ -13021,7 +13114,7 @@ async def model_list(
         )
 
     return dict(
-        data=model_data,
+        data=list(listing),
         object="list",
     )
 
@@ -16875,7 +16968,7 @@ def _enrich_model_info_with_litellm_data(
     stamped_model_info: Final = MappingProxyType(
         {**model_info, **_pricing_override_stamps(model_info, model.get("litellm_params") or MappingProxyType({}))}
     )
-    model["model_info"] = {
+    merged_model_info: Final = {
         **stamped_model_info,
         **{
             k: None if unpriced and k in ("input_cost_per_token", "output_cost_per_token") else v
@@ -16883,6 +16976,18 @@ def _enrich_model_info_with_litellm_data(
             if k not in stamped_model_info or (stamped_model_info[k] is None and k in discovered_model_info)
         },
     }
+    native_output_info: Final = {
+        **merged_model_info,
+        "supports_bedrock_native_structured_output_by_default": litellm_model_info.get(
+            "supports_bedrock_native_structured_output_by_default"
+        ),
+    }
+    model["model_info"] = (
+        {**merged_model_info, "supports_native_structured_output": native_structured_output_support(native_output_info)}
+        if "supports_native_structured_output" in merged_model_info
+        and "supports_native_structured_output" not in stamped_model_info
+        else merged_model_info
+    )
     # don't return the api key / vertex credentials
     # don't return the llm credentials
     model = remove_sensitive_info_from_deployment(model, excluded_keys={"litellm_credential_name"})
@@ -19939,7 +20044,7 @@ async def claim_onboarding_link(data: InvitationClaim, request: Request):
 
     validate_password_policy(data.password, general_settings)
     await validate_password_not_breached(data.password, general_settings)
-    hashed_pw: Final = hash_password(data.password)
+    hashed_pw: Final = await run_off_loop_if_fips(hash_password, data.password)
     current_time = litellm.utils.get_utc_datetime()
     tx_manager: Final[_InvitationAcceptTxManager] = prisma_client.db.tx()
     async with tx_manager as tx:

litellm/proxy/public_endpoints/provider_create_fields.json

@@ -2267,7 +2267,7 @@
   },
   {
     "provider": "LITELLM_PROXY",
-    "provider_display_name": "Litellm Proxy",
+    "provider_display_name": "LiteLLM Gateway",
     "litellm_provider": "litellm_proxy",
     "credential_fields": [
       {

litellm/proxy/utils.py

@@ -6,6 +6,7 @@ import inspect
 import json
 import math
 import os
+import re
 import smtplib
 import ssl
 import sys
@@ -24,6 +25,7 @@ from collections.abc import (
     Mapping,
     Sequence,
 )
+from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, field
 from datetime import date, datetime, timedelta, timezone
 from email.mime.multipart import MIMEMultipart
@@ -85,7 +87,9 @@ from litellm.proxy._types import (
     SpendLogsMetadata,
     SpendLogsPayload,
 )
+from litellm.proxy.auth.password_policy import MIN_ALLOWED_LENGTH
 from litellm.proxy.bug_report_config import build_proxy_bug_report
+from litellm.proxy.common_utils.fips import fips_mode_active
 from litellm.proxy.common_utils.openai_error_payload import (
     attribute_of,
     error_status_code,
@@ -8399,23 +8403,42 @@ def hash_token(token: str):
     return hashed_token
 
 
+def _is_pbkdf2_row(stored: str) -> bool:
+    return re.fullmatch(r"pbkdf2:[A-Za-z0-9+/]{64}", stored) is not None
+
+
 def hash_password(password: str) -> str:
-    """Hash a password using scrypt with a random salt."""
+    """Hash a password using scrypt with a random salt, or PBKDF2 in FIPS mode."""
     import base64
     import hashlib
     import os
 
+    if fips_mode_active():
+        fips_salt: Final = ssl.RAND_bytes(16)
+        fips_dk: Final = hashlib.pbkdf2_hmac("sha256", password.encode(), fips_salt, iterations=600_000, dklen=32)
+        return "pbkdf2:" + base64.b64encode(fips_salt + fips_dk).decode()
     salt: Final = os.urandom(16)
     dk: Final = hashlib.scrypt(password.encode(), salt=salt, n=16384, r=8, p=1, dklen=32)
     return "scrypt:" + base64.b64encode(salt + dk).decode()
 
 
 def verify_password(password: str, stored: str) -> bool:
-    """Verify a password against a stored hash. Supports scrypt and SHA256."""
+    """Verify a password against a stored hash. Supports scrypt and SHA256, or only PBKDF2 in FIPS mode."""
     import base64
     import hashlib
     import secrets
 
+    if fips_mode_active():
+        if not _is_pbkdf2_row(stored):
+            verbose_proxy_logger.error("FIPS mode refuses a non-PBKDF2 password row. Reset it with /user/update.")
+            return False
+        if len(password) < MIN_ALLOWED_LENGTH:
+            return False
+        fips_row: Final = base64.b64decode(stored[7:])
+        fips_dk: Final = hashlib.pbkdf2_hmac(
+            "sha256", password.encode(errors="surrogatepass"), fips_row[:16], iterations=600_000, dklen=32
+        )
+        return secrets.compare_digest(fips_row[16:], fips_dk)
     if stored.startswith("scrypt:"):
         try:
             raw: Final = base64.b64decode(stored[7:])
@@ -8430,12 +8453,24 @@ def verify_password(password: str, stored: str) -> bool:
     return False
 
 
+_FIPS_KDF_POOL: Final = ThreadPoolExecutor(max_workers=1, thread_name_prefix="fips-password-kdf")
+
+
+async def run_off_loop_if_fips(kdf: Callable[..., _T], *args: str) -> _T:
+    """Run a password KDF inline, or on the single KDF thread in FIPS mode."""
+    if not fips_mode_active():
+        return kdf(*args)
+    return await asyncio.get_running_loop().run_in_executor(_FIPS_KDF_POOL, kdf, *args)
+
+
 async def migrate_passwords_to_scrypt_async(prisma_client) -> str:
     """
     Migrate plaintext passwords in the DB to scrypt. SHA256 passwords
     are left alone (they migrate on next login via the SHA256 fallback).
     Skips quickly if no plaintext passwords exist.
     """
+    if fips_mode_active():
+        return "Skipped, FIPS mode has no password migration"
     all_with_pw: Final = await UserRepository(prisma_client).table.find_many(
         where={"password": {"not": None}},
     )
@@ -8446,7 +8481,10 @@ async def migrate_passwords_to_scrypt_async(prisma_client) -> str:
     plaintext_users: Final = [
         (u.user_id, u.password)
         for u in all_with_pw
-        if u.password and not u.password.startswith("scrypt:") and not _is_sha256_hex(u.password)
+        if u.password
+        and not u.password.startswith("scrypt:")
+        and not _is_sha256_hex(u.password)
+        and not _is_pbkdf2_row(u.password)
     ]
     if not plaintext_users:
         return "No plaintext passwords found"

litellm/responses/additional_tools.py

@@ -1,21 +1,20 @@
 import json
 from collections.abc import Iterator, Mapping, Sequence
 from dataclasses import dataclass
-from itertools import groupby
+from itertools import chain, groupby
 from types import MappingProxyType
 from typing import Final, cast  # noqa: TID251  # validating the openai tool union strips vendor keys from raw tools
 
-from pydantic import ValidationError
+from pydantic import TypeAdapter, ValidationError
+from typing_extensions import TypeIs  # noqa: TID251  # narrows a client input item before its type is read
 
 from litellm._logging import verbose_logger
 from litellm.types.llms.base import LiteLLMBaseModel
 from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
 
 ADDITIONAL_TOOLS_INPUT_ITEM_TYPE: Final = "additional_tools"
-
-
-class _InputItemType(LiteLLMBaseModel):
-    type: str = ""
+_NAMESPACE_TOOL_TYPE: Final = "namespace"
+_ITEMS: Final = TypeAdapter(tuple[object, ...])
 
 
 class _ToolList(LiteLLMBaseModel):
@@ -29,11 +28,14 @@ class HoistedAdditionalTools:
     hoisted: tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]
 
 
+def _is_an_input_item(
+    item: object,
+) -> TypeIs[Mapping[str, object]]:  # guard-ok: auth scans every list input, and a pydantic error per item is slow
+    return isinstance(item, Mapping)
+
+
 def _is_additional_tools_item(item: object) -> bool:
-    try:
-        return _InputItemType.model_validate(item).type == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
-    except ValidationError:
-        return False
+    return _is_an_input_item(item) and item.get("type") == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
 
 
 def _tool_list_of(value: object) -> tuple[Mapping[str, object], ...]:
@@ -61,7 +63,7 @@ def _members_of_copies(copies: Sequence[Mapping[str, object]]) -> Iterator[Mappi
 
 def _merged(copies: Sequence[Mapping[str, object]]) -> Mapping[str, object]:
     newest: Final = copies[-1]
-    if newest.get("type") != "namespace" or len(copies) == 1:
+    if newest.get("type") != _NAMESPACE_TOOL_TYPE or len(copies) == 1:
         return newest
     members: Final = tuple(_members_of_copies(copies))
     return {**newest, "tools": list(_unique_tools(members))} if members else newest
@@ -86,6 +88,11 @@ def _unique_tools(tools: Sequence[Mapping[str, object]]) -> tuple[Mapping[str, o
     return tuple(_merged(copies_by_identity[identity]) for identity in first_seen_order)
 
 
+def additional_tools_in_input(input: object) -> tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]:
+    items: Final = _ITEMS.validate_python(input) if isinstance(input, list) else ()
+    return tuple(chain.from_iterable(_tools_of_item(item) for item in items if _is_additional_tools_item(item)))
+
+
 def hoist_additional_tools(
     input: str | ResponseInputParam,
     tools: Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None,
@@ -112,3 +119,26 @@ def hoist_additional_tools(
         for tool in _unique_tools((*existing, *hoisted))
     )
     return HoistedAdditionalTools(input=remaining_input, tools=unique, hoisted=hoisted)
+
+
+def _top_level_tools(request: Mapping[str, object]) -> tuple[object, ...] | None:
+    try:
+        return _ITEMS.validate_python(request.get("tools") or ())
+    except ValidationError:
+        return None
+
+
+def with_additional_tools_hoisted(request: Mapping[str, object], *, drop_params: bool) -> Mapping[str, object]:
+    request_input: Final = request.get("input")
+    top_level_tools: Final = _top_level_tools(request)
+    if not isinstance(request_input, list) or top_level_tools is None:
+        return request
+    input_items: Final = cast(ResponseInputParam, request_input)  # cast-ok: forwarded as sent, as inference does
+    hoisted: Final = hoist_additional_tools(
+        input_items,
+        cast("tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", top_level_tools),  # cast-ok: forwarded as sent
+        drop_params=drop_params,
+    )
+    if not hoisted.hoisted:
+        return request
+    return {**request, "input": hoisted.input, "tools": list(hoisted.tools)}

litellm/responses/litellm_completion_transformation/count_request.py

@@ -12,7 +12,7 @@ from litellm.litellm_core_utils.prompt_templates.file_id_utils import (
 )
 from litellm.litellm_core_utils.tool_fields import DropPolicy
 from litellm.llms.base_llm.files.batch_records import responses_batch_body_to_chat_body
-from litellm.responses.additional_tools import hoist_additional_tools
+from litellm.responses.additional_tools import with_additional_tools_hoisted
 from litellm.responses.file_search.emulated_handler import replace_file_search_tools
 from litellm.responses.litellm_completion_transformation.compaction_items import holds_compaction_summary
 from litellm.responses.litellm_completion_transformation.transformation import (
@@ -20,7 +20,7 @@ from litellm.responses.litellm_completion_transformation.transformation import (
     LiteLLMCompletionResponsesConfig,
 )
 from litellm.responses.main import previous_response_client_ids
-from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
+from litellm.types.llms.openai import ResponseInputParam
 from litellm.utils import ProviderConfigManager
 
 _COUNTED_CHAT_PARAMS: Final = (
@@ -114,20 +114,6 @@ def with_provider_file_ids(
     }
 
 
-def _with_hoisted_additional_tools(responses_request: Mapping[str, object], drop_params: bool) -> Mapping[str, object]:
-    responses_input: Final = responses_request.get("input")
-    if not isinstance(responses_input, list):
-        return responses_request
-    input_items: Final = cast(ResponseInputParam, responses_input)  # cast-ok: forwarded as sent, as inference does
-    top_level_tools: Final = cast(  # cast-ok: tools are forwarded as sent, as inference does
-        "tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", _ITEMS.validate_python(responses_request.get("tools") or ())
-    )
-    hoisted: Final = hoist_additional_tools(input_items, top_level_tools, drop_params=drop_params)
-    if not hoisted.hoisted:
-        return responses_request
-    return {**responses_request, "input": hoisted.input, "tools": list(hoisted.tools)}
-
-
 def _caller_input(responses_request: Mapping[str, object]) -> ResponseInputParam | None:
     responses_input: Final = responses_request.get("input")
     return (
@@ -205,7 +191,9 @@ async def chat_count_request(
         chat_body: Final = await _with_previous_turns(
             responses_batch_body_to_chat_body(
                 _with_compaction_as_summary_text(
-                    _with_emulated_file_search(_with_hoisted_additional_tools(responses_request, policy.drop_params))
+                    _with_emulated_file_search(
+                        with_additional_tools_hoisted(responses_request, drop_params=policy.drop_params)
+                    )
                 ),
                 custom_llm_provider,
                 policy.drop_params,

litellm/responses/litellm_completion_transformation/reasoning_items.py

@@ -6,6 +6,7 @@ from typing import Final
 from pydantic import BaseModel, TypeAdapter, ValidationError
 
 REASONING_ITEM_ID_PREFIX: Final = "rs_"
+WRAPPED_ENCRYPTED_CONTENT_PREFIX: Final = "litellm_enc:"
 _JSON_LIST: Final = TypeAdapter(list[object])
 _JSON_OBJECT: Final = TypeAdapter(dict[str, object])
 
@@ -58,16 +59,35 @@ def is_minted_reasoning_item_id(item_id: object) -> bool:
     return parsed.version == 4 and str(parsed) == suffix
 
 
-def is_litellm_minted_reasoning_item(item: object) -> bool:
+def _reasoning_fields(item: object) -> Mapping[str, object] | None:
     try:
         fields: Final = _JSON_OBJECT.validate_python(
             item.model_dump(exclude_none=True) if isinstance(item, BaseModel) else item
         )
     except ValidationError:
-        return False
-    if fields.get("type") != "reasoning":
-        return False
+        return None
+    return fields if fields.get("type") == "reasoning" else None
+
+
+def _is_minted(fields: Mapping[str, object]) -> bool:
     return (
         is_minted_reasoning_item_id(fields.get("id"))
         or decode_thinking_blocks(fields.get("encrypted_content")) is not None
     )
+
+
+def is_litellm_minted_reasoning_item(item: object) -> bool:
+    fields: Final = _reasoning_fields(item)
+    return fields is not None and _is_minted(fields)
+
+
+def is_other_provider_reasoning_item(item: object) -> bool:
+    fields: Final = _reasoning_fields(item)
+    if fields is None:
+        return False
+    encrypted_content: Final = fields.get("encrypted_content")
+    return (
+        _is_minted(fields)
+        or bool(fields.get("content"))
+        or (isinstance(encrypted_content, str) and encrypted_content.startswith(WRAPPED_ENCRYPTED_CONTENT_PREFIX))
+    )

litellm/responses/litellm_completion_transformation/streaming_iterator.py

@@ -9,6 +9,7 @@ import litellm
 from litellm.litellm_core_utils.core_helpers import as_str_mapping, attribute_items
 from litellm.litellm_core_utils.hidden_params import get_or_create_hidden_params, served_from_cache
 from litellm.litellm_core_utils.litellm_logging import BilledRoundsSpend
+from litellm.llms.anthropic.common_utils import anthropic_server_call_fields, anthropic_server_tool_calls
 from litellm.llms.anthropic.messages.native.utils import openai_chat_refusal_text
 from litellm.main import stream_chunk_builder
 from litellm.responses.litellm_completion_transformation.compaction_items import compaction_output_item
@@ -133,6 +134,7 @@ def _delta_carries_output(delta: ChatCompletionDelta) -> bool:
         getattr(delta, "images", None),
         getattr(delta, "audio", None),
         openai_chat_refusal_text(delta),
+        anthropic_server_call_fields(delta),
     )
     return any(fields) or _delta_has_signed_thinking_block(delta)
 
@@ -228,6 +230,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
         self._queued_web_search_call_ids: set[str] = set()  # mutable-ok: emitted call ids
         self._tool_search = tool_search_routing(request_tools)
         self._tool_search_executions: Mapping[str, ToolSearchExecution] = MappingProxyType({})
+        self._done_tool_search_call_ids: frozenset[str] = frozenset()
 
     async def aclose(self) -> None:
         await self.litellm_custom_stream_wrapper.aclose()
@@ -265,6 +268,9 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
     def _queue_tool_search_call_done(
         self, call_id: str, output_index: int, final_args: str, execution: ToolSearchExecution
     ) -> None:
+        if call_id in self._done_tool_search_call_ids:
+            return
+        self._done_tool_search_call_ids = self._done_tool_search_call_ids | {call_id}
         self._queue_tool_search_call_added(call_id, output_index, "", execution)
         item: Final = build_tool_search_call(
             call_id, final_args or self._tool_args_by_call_id.get(call_id, ""), "completed", execution
@@ -277,6 +283,18 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
             )
         )
 
+    def _queue_streamed_tool_search_calls(self, chunk: ModelResponseStream) -> None:
+        if not chunk.choices:
+            return
+        for call in anthropic_server_tool_calls(chunk.choices[0].delta):
+            if (execution := self._tool_search_execution(call.id, call.function.name or "")) is None:
+                continue
+            if call.id not in self._tool_output_index_by_call_id:
+                self._queue_held_annotations()
+            self._queue_tool_search_call_done(
+                call.id, self._get_or_assign_tool_output_index(call.id), call.function.arguments, execution
+            )
+
     def _get_or_assign_tool_output_index(self, call_id: str) -> int:
         existing: Final = self._tool_output_index_by_call_id.get(call_id)
         if existing is not None:
@@ -509,11 +527,17 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
 
         try:
             message: Final = litellm_complete_object.choices[0].message
-            tool_calls = getattr(message, "tool_calls", None)
+            server_tool_calls: Final = anthropic_server_tool_calls(message)
+            tool_calls: Final = (*server_tool_calls, *_tool_call_objects(getattr(message, "tool_calls", None)))
         except Exception:
-            tool_calls = None
+            return
 
-        for tc in _tool_call_objects(tool_calls):
+        if not tool_calls:
+            return
+
+        server_tool_call_ids: Final = frozenset(tool_call.id for tool_call in server_tool_calls)
+
+        for tc in tool_calls:
             call_id_raw = tc.get("id") if isinstance(tc, dict) else getattr(tc, "id", None)
             if not call_id_raw:
                 continue
@@ -538,6 +562,8 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
             if (search_execution := self._tool_search_execution(call_id, fn_name)) is not None:
                 self._queue_tool_search_call_done(call_id, output_index, fn_args, search_execution)
                 continue
+            if call_id in server_tool_call_ids:
+                continue
 
             # Track if this is a new tool call that wasn't streamed
             is_new_tool_call = call_id not in self._tool_item_id_by_call_id
@@ -1205,7 +1231,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
             return
 
         # Tool-first
-        if hasattr(delta, "tool_calls") and delta.tool_calls:
+        if (hasattr(delta, "tool_calls") and delta.tool_calls) or anthropic_server_call_fields(delta):
             # Tool calls already handled via _queue_tool_call_delta_events
             # DO NOT create message item
             return
@@ -1383,6 +1409,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):
 
     def _queue_chunk_events(self, chunk: ModelResponseStream) -> None:
         self._hold_chunk_annotations(chunk)
+        self._queue_streamed_tool_search_calls(chunk)
         response_api_chunk: Final = self._transform_chat_completion_chunk_to_response_api_chunk(chunk)
         if response_api_chunk:
             self._pending_response_events.append(response_api_chunk)

litellm/responses/litellm_completion_transformation/transformation.py

@@ -73,7 +73,12 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
     is_unhashable_value_behind,
 )
 from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import message_field, parts_of
-from litellm.llms.anthropic.common_utils import supports_anthropic_cache_control
+from litellm.llms.anthropic.common_utils import (
+    anthropic_server_tool_call_params,
+    anthropic_server_tool_calls,
+    supports_anthropic_cache_control,
+    validated_object_mapping,
+)
 from litellm.llms.anthropic.messages.native.utils import anthropic_stop_reason, openai_chat_refusal_text
 from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
 from litellm.llms.bedrock.common_utils import bedrock_declared_cache_point_support
@@ -84,11 +89,13 @@ from litellm.responses.litellm_completion_transformation.session_handler import
     ResponsesSessionHandler,
     StoredTurns,
 )
-from litellm.types.llms.anthropic import AnthropicToolSearchToolRegex
+from litellm.responses.utils import ResponsesAPIRequestUtils
+from litellm.types.llms.anthropic import AnthropicToolSearchToolRegex, claude_hosted_web_tool_call
 from litellm.types.llms.base import CachedTokensDetails
 from litellm.types.llms.openai import (
     AllMessageValues,
     ChatCompletionAssistantMessage,
+    ChatCompletionAssistantToolCall,
     ChatCompletionAudioObject,
     ChatCompletionCachedContent,
     ChatCompletionFileObject,
@@ -933,6 +940,22 @@ def _chain_session_metadata(chain_session: str, request_kwargs: Mapping[str, obj
     return {key: {**_STR_KEY_DICT_ADAPTER.validate_python(metadata), CALLER_SESSION_ID_METADATA_KEY: chain_session}}
 
 
+def claude_web_search_call_tool_call(search: ResponseFunctionWebSearch) -> ChatCompletionAssistantToolCall | None:
+    claude_call: Final = claude_hosted_web_tool_call(search)
+    if claude_call is None:
+        return None
+    tool_id, name, arguments = claude_call
+    return ChatCompletionAssistantToolCall(
+        id=tool_id,
+        type="function",
+        function=ChatCompletionToolCallFunctionChunk(name=name, arguments=arguments),
+    )
+
+
+def _pending_claude_server_tool_call(search: ResponseFunctionWebSearch) -> ChatCompletionAssistantToolCall | None:
+    return claude_web_search_call_tool_call(search) if search.status == "in_progress" else None
+
+
 class LiteLLMCompletionResponsesConfig:
     @staticmethod
     def get_supported_openai_params(model: str) -> list:
@@ -1889,10 +1912,22 @@ class LiteLLMCompletionResponsesConfig:
         message_cache_control: Final = (
             None if isinstance(content, list) else new_cache_control or previous_cache_control
         )
+        new_server_tool_calls: Final = anthropic_server_tool_call_params(new_message)
+        server_tool_call_fields: Final = (
+            {
+                "provider_specific_fields": {
+                    **(validated_object_mapping(last_message.get("provider_specific_fields")) or {}),
+                    "server_tool_calls": [*anthropic_server_tool_call_params(last_message), *new_server_tool_calls],
+                }
+            }
+            if new_server_tool_calls
+            else {}
+        )
         merged: Final = {
             **{key: value for key, value in last_message.items() if key != "cache_control"},
             "content": content,
             **({} if message_cache_control is None else {"cache_control": message_cache_control}),
+            **server_tool_call_fields,
         }
         return cast(  # cast-ok: preserves the assistant fields and content blocks
             ChatCompletionAssistantMessage, merged
@@ -2334,10 +2369,15 @@ class LiteLLMCompletionResponsesConfig:
                         content="Hosted web search: " + replay.model_dump_json(exclude_none=True),
                     )
                 ]
+            hosted_search_text: Final = "Hosted web search: " + search.model_dump_json(exclude_none=True)
+            pending_server_tool_call: Final = _pending_claude_server_tool_call(search)
+            if pending_server_tool_call is None:
+                return [GenericChatCompletionMessage(role="assistant", content=hosted_search_text)]
             return [
-                GenericChatCompletionMessage(
+                ChatCompletionResponseMessage(
                     role="assistant",
-                    content="Hosted web search: " + search.model_dump_json(exclude_none=True),
+                    content=hosted_search_text,
+                    provider_specific_fields={"server_tool_calls": [pending_server_tool_call]},
                 )
             ]
         if LiteLLMCompletionResponsesConfig._is_input_item_tool_call_output(input_item):
@@ -2517,7 +2557,12 @@ class LiteLLMCompletionResponsesConfig:
         Returns None for anything this deployment did not write, so a genuinely
         opaque blob is still skipped rather than forwarded as garbage.
         """
-        decoded: Final = decode_thinking_blocks(input_item.get("encrypted_content"))
+        encrypted_content: Final = input_item.get("encrypted_content")
+        decoded: Final = decode_thinking_blocks(
+            ResponsesAPIRequestUtils.unwrap_encrypted_content_with_model_id(encrypted_content)[1]
+            if isinstance(encrypted_content, str)
+            else encrypted_content
+        )
         if decoded is None:
             return None
         return tuple(
@@ -3085,13 +3130,12 @@ class LiteLLMCompletionResponsesConfig:
         all_chat_completion_tools: Final[list[ChatCompletionMessageToolCall]] = []
         for choice in chat_completion_response.choices:
             if isinstance(choice, Choices):
-                if choice.message.tool_calls:
-                    all_chat_completion_tools.extend(choice.message.tool_calls)
-                    for tool_call in choice.message.tool_calls:
-                        TOOL_CALLS_CACHE.set_cache(
-                            key=tool_call.id,
-                            value=tool_call,
-                        )
+                for tool_call in (*anthropic_server_tool_calls(choice.message), *(choice.message.tool_calls or ())):
+                    all_chat_completion_tools.append(tool_call)
+                    TOOL_CALLS_CACHE.set_cache(
+                        key=tool_call.id,
+                        value=tool_call,
+                    )
 
         request_tools: Final = responses_api_request.get("tools") if responses_api_request is not None else None
         custom_tool_names: Final = extract_custom_tool_names(request_tools)
@@ -3522,7 +3566,22 @@ class LiteLLMCompletionResponsesConfig:
                 for item in responses_output
             ]
 
-        return responses_output
+        server_tool_call_ids: Final = LiteLLMCompletionResponsesConfig._server_tool_call_ids(choices)
+        return [
+            item
+            for item in responses_output
+            if not (
+                getattr(item, "type", None) == "function_call"
+                and getattr(item, "call_id", None) in server_tool_call_ids
+            )
+        ]
+
+    @staticmethod
+    def _server_tool_call_ids(choices: Sequence[Choices]) -> frozenset[str]:
+        server_tool_calls: Final = chain.from_iterable(
+            anthropic_server_tool_calls(choice.message) for choice in choices
+        )
+        return frozenset(tool_call.id for tool_call in server_tool_calls)
 
     @staticmethod
     def _extract_tool_result_output_items(

litellm/responses/streaming_iterator.py

@@ -250,6 +250,8 @@ _ERROR_CODE_HTTP_STATUS: Final[Mapping[str, int]] = MappingProxyType(
         "server_error": 500,
         "rate_limit_exceeded": 429,
         "insufficient_quota": 429,
+        "resource_unavailable": 429,
+        "flex_unavailable": 429,
         "vector_store_timeout": 504,
         "invalid_prompt": 400,
         "invalid_image": 400,

litellm/responses/utils.py

@@ -50,6 +50,7 @@ from litellm.types.utils import (
 _PROVIDER_SPECIFIC_HEADER_ADAPTER: Final[
     TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]
 ] = TypeAdapter(ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None)
+_REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES: Final = frozenset({"encrypted_content", "id", "status"})
 
 
 class _UsageExtraFields(ExtraItemsTypedDict, total=False, extra_items=object):
@@ -678,7 +679,7 @@ class ResponsesAPIRequestUtils:
             return
         items: Final = cast(list[object], request_input)  # cast-ok: untyped client json
         stripped: Final = tuple(
-            ResponsesAPIRequestUtils._without_encrypted_reasoning(reasoning)
+            ResponsesAPIRequestUtils.readable_reasoning(reasoning)
             if (reasoning := ResponsesAPIRequestUtils._encrypted_reasoning_item(item)) is not None
             and (should_strip is None or should_strip(reasoning))
             else item
@@ -696,14 +697,14 @@ class ResponsesAPIRequestUtils:
         return reasoning
 
     @staticmethod
-    def _without_encrypted_reasoning(reasoning: Mapping[str, object]) -> object | None:
+    def readable_reasoning(reasoning: Mapping[str, object]) -> Mapping[str, object] | None:
         readable: Final = any(
             ResponsesAPIRequestUtils._has_readable_text(reasoning.get(key)) for key in ("summary", "content")
         )
         if not readable:
             return None
         kept: Final[dict[str, object]] = {  # mutable-ok: request item rebuilt without the undecryptable keys
-            key: value for key, value in reasoning.items() if key not in ("encrypted_content", "id")
+            key: value for key, value in reasoning.items() if key not in _REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES
         }
         return kept
 

litellm/router_api.py

@@ -158,6 +158,7 @@ GroupField: TypeAlias = Literal[
     "display_name",
     "mode",
     "service_tiers",
+    "pinned_reasoning_efforts",
     "max_input_tokens",
     "deployment_model_info",
     "token_limits",
@@ -939,7 +940,7 @@ class RouterRegistry:
         self,
         name: str,
         *,
-        field: Literal["service_tiers"],
+        field: Literal["service_tiers", "pinned_reasoning_efforts"],
         include_hidden: bool = False,
         team_id: str | None = None,
     ) -> tuple[object, ...]: ...
@@ -1001,6 +1002,7 @@ class RouterRegistry:
           (were ``get_configured_display_name``, ``get_configured_mode``,
           ``get_configured_token_limits``, ``get_model_listing_info``)
         - ``service_tiers``: (was ``get_configured_service_tiers``)
+        - ``pinned_reasoning_efforts``: the ``reasoning_effort`` each routable deployment pins
         - ``max_input_tokens``: each routable deployment's own ``model_info.max_input_tokens``
         - ``deployment_model_info``: each routable deployment's model info, merged with the pricing map
         """
@@ -1015,6 +1017,8 @@ class RouterRegistry:
             return r._core_get_configured_mode(name)
         if field == "service_tiers":
             return r._core_get_configured_service_tiers(name, team_id=team_id)
+        if field == "pinned_reasoning_efforts":
+            return r._core_get_pinned_reasoning_efforts(name, team_id=team_id)
         if field == "max_input_tokens":
             return r._core_get_configured_max_input_tokens(name, team_id=team_id)
         if field == "deployment_model_info":
@@ -1136,6 +1140,18 @@ class RouterRegistry:
             return r._core_prompt_caching_pin_cache
         raise ValueError(f"unknown plugin kind {kind!r}")
 
+    def decryptable_reasoning(
+        self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
+    ) -> tuple[object, ...]:
+        """
+        ``request_input`` without the encrypted reasoning that ``targets``, the deployments a
+        ``model_group`` request may go to, cannot decrypt, keeping each item's readable summary.
+        When encrypted content affinity is on for the group, the router sends the request to the
+        reasoning's origin, so reasoning one target can decrypt stays; otherwise every target must
+        decrypt it (was reading ``optional_callbacks`` for ``EncryptedContentAffinityCheck``).
+        """
+        return self._router._core_decryptable_reasoning(model_group, request_input, targets)
+
 
 # Settings a caller may read. Assigning one from outside the router goes through apply(ChangeSet(assign=...)).
 SETTING_FIELDS: Final[tuple[str, ...]] = (

litellm/router_rust/steps/callbacks.py

@@ -407,6 +407,28 @@ class OptionalChecks:
         bound: Final = request.get("litellm_logging_obj") in self._bound_requests
         return {"success": bound and not sync, "failure": bound, "make_call": make_call}
 
+    def decryptable_reasoning(
+        self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
+    ) -> tuple[object, ...]:
+        """``registry().decryptable_reasoning``: one target decrypting is enough only when encrypted
+        content affinity pins ``model_group`` to the reasoning's origin."""
+        target_dicts: Final = tuple(target.model_dump(exclude_none=True) for target in targets)
+        pinned: Final = any(
+            isinstance(callback, EncryptedContentAffinityCheck) and callback.is_enabled_for_model_group(model_group)
+            for callback in self.optional_callbacks or ()
+        )
+        target_groups: Final = (target_dicts,) if pinned else tuple((target,) for target in target_dicts)
+        kept: Final = list(request_input)
+        for group in target_groups:
+            EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
+                self._ctx,  # pyright: ignore[reportArgumentType]  # the core's RoutingContext serves registry()
+                kept,
+                None,
+                group,
+                unmarked_origin=None,
+            )
+        return tuple(kept)
+
     def routing_callbacks(self) -> Iterable[object]:
         """``_routing_callbacks``: the host's callbacks, or after ``observe(request_scoped=True)`` the
         host's callbacks with the router's own selectors and policies in place of theirs."""

litellm/router_rust/surface/model_info.py

@@ -46,6 +46,7 @@ from litellm.litellm_core_utils.add_retry_fallback_headers import (
 from litellm.litellm_core_utils.batch_utils import deployment_supported_endpoints
 from litellm.litellm_core_utils.core_helpers import as_str_mapping, coerce_token_limit
 from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
+from litellm.litellm_core_utils.litellm_logging import resolve_vertex_location_for_cost
 from litellm.litellm_core_utils.reasoning_effort_capability import (
     deployment_is_catalog_mapped,
     intersect_supported_reasoning_efforts,
@@ -56,11 +57,13 @@ from litellm.llms.anthropic.common_utils import AnthropicModelInfo
 from litellm.llms.bedrock.common_utils import get_bedrock_base_model
 from litellm.redis.stores.router_usage import ROUTER_USAGE_TARGET, RouterUsage, UsageCounter
 from litellm.router_python_host import python_host
+from litellm.router_utils.model_group_pricing import deployment_pricing
 from litellm.router_utils.pre_call_checks.io_token_rate_limit_check import build_io_token_rate_limit_headers
 from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
 from litellm.types.router import (
     Deployment,
     DeploymentModelListingInfo,
+    DeploymentPricing,
     DeploymentTypedDict,
     DiscoveredDeploymentModelInfo,
     LiteLLM_Params,
@@ -78,6 +81,7 @@ if TYPE_CHECKING:
     )
 
 _MODEL_INFO_ADAPTER: Final = TypeAdapter(Mapping[str, object])
+_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
 
 # The key router-core's merge reads as an opaque value (``router-core/src/model_info.rs``).
 _OPAQUE_KEY: Final = "__litellm_python_value__"
@@ -140,6 +144,11 @@ def _configured_model_info(deployment: Mapping[str, object]) -> Mapping[str, obj
     return _MODEL_INFO_ADAPTER.validate_python(deployment.get("model_info") or {})
 
 
+def _configured_deployment_id(deployment: Mapping[str, object]) -> str | None:
+    deployment_id: Final = _configured_model_info(deployment).get("id")
+    return deployment_id if isinstance(deployment_id, str) else None
+
+
 def _operator_max_input_tokens(deployment: Mapping[str, object]) -> int | None:
     """``_operator_max_input_tokens``: the deployment's own limit, skipping one a saved
     ``/model/info`` response copied from the cost map."""
@@ -187,6 +196,7 @@ class _GroupBuild:
     # Filled as the lookups arrive.
     params: dict[int, LiteLLM_Params] = dataclasses.field(default_factory=dict)  # mutable-ok: see above
     info: ModelGroupInfo | None = None
+    pricing: tuple[DeploymentPricing, ...] = ()
 
 
 def _usage_total(values: Sequence[object]) -> int | None:
@@ -358,6 +368,7 @@ class HostModelInfo:
             return None
         for field, value in (as_str_mapping(answer["assignments"]) or {}).items():
             setattr(group.info, field, _group_field(field, wire.from_wire(value)))
+        group.info.deployment_pricing = group.pricing
         return group.info
 
     def _answer_group_lookup(self, step: Mapping[str, object], group: "_GroupBuild", wire: "_Wire") -> object:
@@ -384,6 +395,17 @@ class HostModelInfo:
                 efforts: Final = resolve_supported_reasoning_efforts(
                     model_info, deployment_is_mapped=step["mapped"] is True
                 )
+                pricing: Final = deployment_pricing(
+                    llm_provider,
+                    _configured_deployment_id(deployment),
+                    group.params[int(str(step["index"]))].model,
+                    model_info,
+                    resolve_vertex_location_for_cost(
+                        llm_provider, deployment.get("litellm_params"), None, litellm_model
+                    ),
+                )
+                if pricing is not None and pricing not in group.pricing:
+                    group.pricing = (*group.pricing, pricing)  # rebind-ok: the merge's state
                 return {
                     "supported_endpoints": wire.to_wire(
                         list(
@@ -740,6 +762,20 @@ class HostModelInfo:
         )
         return tuple(wire.from_wire(tier) for tier in tiers)
 
+    def _core_get_pinned_reasoning_efforts(self, model_name: str, team_id: str | None = None) -> tuple[object, ...]:
+        """
+        Return the litellm_params.reasoning_effort each deployment a request
+        from team_id to model_name can route to (see _routable_deployments)
+        pins, unvalidated and in model_list order, None for a deployment that
+        pins none; the caller decides how the deployments combine.
+
+        Returns an empty tuple for wildcard-expanded or unknown names.
+        """
+        return tuple(
+            _LITELLM_PARAMS_ADAPTER.validate_python(deployment["litellm_params"]).get("reasoning_effort")
+            for deployment in self._routable_deployments(model_name, team_id)
+        )
+
     def _core_get_configured_max_input_tokens(
         self, model_name: str, team_id: str | None = None
     ) -> tuple[int | None, ...]:

litellm/router_rust/surface/registry.py

@@ -1510,6 +1510,14 @@ class HostRegistry(HostModelInfo):
             return self.__getattr__("_core_prompt_caching_pin_cache")  # pyright: ignore[reportReturnType]  # raises RustCoreUnsupported
         return self.checks.prompt_caching_pin_cache
 
+    def _core_decryptable_reasoning(
+        self, model_group: str, request_input: Sequence[object], targets: Sequence[Deployment]
+    ) -> tuple[object, ...]:
+        """The input ``targets`` can decrypt (``registry().decryptable_reasoning``)."""
+        if self.checks is None:
+            return self.__getattr__("_core_decryptable_reasoning")  # pyright: ignore[reportReturnType]  # raises RustCoreUnsupported
+        return self.checks.decryptable_reasoning(model_group, request_input, targets)
+
     def replay_model_cost_registrations(self) -> None:
         """Re-register every served deployment's cost-map entries (``replay_model_cost_registrations``)."""
         provider_unresolved: Final = self._provider_unresolved_deployments

litellm/router_utils/model_group_pricing.py

@@ -0,0 +1,69 @@
+import math
+from collections.abc import Mapping
+from typing import Final
+
+from pydantic import TypeAdapter, ValidationError
+
+from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info
+from litellm.litellm_core_utils.llm_cost_calc.utils import get_vertex_regional_endpoint_uplift
+from litellm.types.router import DeploymentPricing, PriceTier
+
+_RAW_PRICE_TIERS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
+
+
+def _numeric_text(value: str) -> float | None:
+    try:
+        parsed: Final = float(value)
+    except ValueError:
+        return None
+    return parsed if math.isfinite(parsed) else None
+
+
+def _cost_value(value: object) -> float | None:
+    match value:
+        case bool():
+            return None
+        case int() | float():
+            return float(value)
+        case str():
+            return _numeric_text(value)
+        case _:
+            return None
+
+
+def _cost_fields(fields: Mapping[str, object], token_uplift: float) -> Mapping[str, float]:
+    return {
+        name: cost * token_uplift if "token" in name else cost
+        for name, value in fields.items()
+        if "cost" in name and (cost := _cost_value(value)) is not None
+    }
+
+
+def _price_tiers(tiered_pricing: object, token_uplift: float) -> tuple[PriceTier, ...]:
+    try:
+        return tuple(
+            PriceTier.model_validate({"range": tier.get("range"), "costs": _cost_fields(tier, token_uplift)})
+            for tier in _RAW_PRICE_TIERS.validate_python(tiered_pricing)
+        )
+    except ValidationError:
+        return ()
+
+
+def deployment_pricing(
+    provider: str,
+    deployment_id: str | None,
+    deployment_model: str,
+    resolved_model_info: Mapping[str, object],
+    vertex_location: str | None,
+) -> DeploymentPricing | None:
+    """Prices from the entry billing charges this deployment at, so a cost-map variant its own pricing shadows
+    is never shown. A deployment that declares no pricing is billed at its resolved cost-map entry. Token prices
+    carry the Vertex regional-endpoint uplift billing applies at the deployment's location."""
+    billed_model_info: Final = deployment_pricing_model_info(model_id=deployment_id, deployment_model=deployment_model)
+    model_info: Final = resolved_model_info if billed_model_info is None else billed_model_info
+    token_uplift: Final = get_vertex_regional_endpoint_uplift(model_info, vertex_location)
+    costs: Final = _cost_fields(model_info, token_uplift)
+    tiers: Final = _price_tiers(model_info.get("tiered_pricing"), token_uplift)
+    if not costs and not tiers:
+        return None
+    return DeploymentPricing(provider=provider, costs=costs, tiered_pricing=tiers)

litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py

@@ -148,7 +148,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
 
         return any("encrypted_content_affinity" in checks for checks in model_group_affinity_config.values())
 
-    def _is_enabled_for_model_group(self, model_group: str) -> bool:
+    def is_enabled_for_model_group(self, model_group: str) -> bool:
         group_checks: Final = self.model_group_affinity_config.get(model_group)
         return self.enable_global_affinity or (
             group_checks is not None and "encrypted_content_affinity" in group_checks
@@ -530,7 +530,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
         """
         request_kwargs = request_kwargs or {}
         typed_healthy_deployments: Final = cast(list[dict[str, object]], healthy_deployments)
-        if not self._is_enabled_for_model_group(model):
+        if not self.is_enabled_for_model_group(model):
             return typed_healthy_deployments
         typed_request_kwargs: Final = _REQUEST_KWARGS_ADAPTER.validate_python(request_kwargs)
         target_order: Final = typed_request_kwargs.get("_target_order")

litellm/rust_bridge/catalog.py

@@ -110,6 +110,7 @@ class Feature(str, Enum):
     """A model_list that Python fans out across its deployments."""
     MODEL_ALIAS_MAP = "model_alias_map"
     """A model that litellm.model_alias_map rewrites before the call."""
+    LATEST_ALIAS = "latest_alias"
 
 
 @dataclass(frozen=True, slots=True)
@@ -153,6 +154,7 @@ COMMON_PYTHON_FEATURES: Final = frozenset(
         Feature.TIMEOUT_VARIANTS,
         Feature.AGENTIC_LOOP_HOOK,
         Feature.TLS_POLICY,
+        Feature.LATEST_ALIAS,
     }
 )