Merge 120fa5eaba into 2306816d40

A shadow eval whose judge_model is one of the router's tier models, the router's
default model, or a reverse job's baseline_model was accepted with no warning. An
LLM judge scores its own output higher than a rival's, so that tier's win rate
measures the judge instead of the models, and the job's whole budget buys a result
that has to be thrown away.

start_shadow_eval now rejects it with a 400 naming the colliding arm.

`judge_target` is the single answer to "where does a call to this name go for this
caller, and what answers it", and the resolvability gate, the collision gate and
the judge dispatch all read it. It has three outcomes and no others: the router
serves the name, the SDK serves it, or nothing does. Splitting that question is
what every bug here came from, so `router_resolves_model` and `answering_models`
are gone rather than joined by a third.

Two spellings of one model are one identity. A name is compared by what would
answer it, resolved through every channel `get_model_list` composes and then put
in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o`
collides with a tier deployment serving `openai/gpt-4o`, and a judge given as
`openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends
are normalised because an admin writes them at different times.

Answering is also per-caller. The shadow and judge calls carry the shadowed key's
`user_api_key_team_id`, which is what the router selects deployments with, so the
endpoint derives the job's teams once from the keys it already looks up and every
check runs under them, and the judge dispatch picks its arm under the same team.
A team's public model name resolves to nothing for everyone else and a team's own
deployment resolves for nobody else, so a check that omits the team answers for a
caller who does not exist. A collision under any one team fails the job, because
every key's verdicts land in the same win rates.

Three sites were separately re-deriving "the provider models this name resolves
to", with unexplained divergence in whether they fell back to the literal name.
`Router.resolved_litellm_models` is now the one owner; the routing-plugin
candidate list and the stream-options check both delegate to it, and
`_deployment_litellm_model` is gone.

The router's arms come from `strategy_router_dependencies`, the same enumeration
the health check reads. Only the roles that serve are arms: a classifier or
embedding model picks the tier and never produces a response anyone judges. A
semantic auto-router keeps its routes in an opaque config blob, so only its
default model is enumerable and the guard is incomplete there by design, able to
miss a collision but never to invent one

The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from
alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one
has to carry the base's drift to go green

.github/workflows/check-ui-api-types.yml

@@ -83,6 +83,24 @@ jobs:
         if: steps.changes.outputs.relevant == 'true'
         run: uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
 
+      - name: Regenerate the lazy OpenAPI snapshot
+        if: steps.changes.outputs.relevant == 'true'
+        run: uv run --no-sync python -m litellm.proxy._lazy_openapi_snapshot
+
+      - name: Fail if the lazy OpenAPI snapshot is stale
+        if: steps.changes.outputs.relevant == 'true'
+        run: |
+          if ! git diff --exit-code -- litellm/proxy/_lazy_openapi_snapshot.json; then
+            echo "::error file=litellm/proxy/_lazy_openapi_snapshot.json::The lazy OpenAPI snapshot is out of sync with the lazily loaded routes."
+            echo ""
+            echo "A lazily loaded route or model changed without regenerating the snapshot that /openapi.json serves for unloaded features."
+            echo "To fix, run from the repo root:"
+            echo "  uv run python -m litellm.proxy._lazy_openapi_snapshot"
+            echo "then run npm run gen:api from ui/litellm-dashboard and commit both files."
+            exit 1
+          fi
+          echo "_lazy_openapi_snapshot.json is in sync with the lazily loaded routes."
+
       - name: Set up Node.js
         if: steps.changes.outputs.relevant == 'true'
         uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0

basedpyright-code-budget.json

@@ -6,10 +6,10 @@
     "limit": 2564
   },
   "reportAssignmentType": {
-    "limit": 320
+    "limit": 319
   },
   "reportAttributeAccessIssue": {
-    "limit": 483
+    "limit": 480
   },
   "reportCallIssue": {
     "limit": 113
@@ -30,7 +30,7 @@
     "limit": 7
   },
   "reportGeneralTypeIssues": {
-    "limit": 154
+    "limit": 105
   },
   "reportIncompatibleMethodOverride": {
     "limit": 56
@@ -84,7 +84,7 @@
     "limit": 56
   },
   "reportPrivateUsage": {
-    "limit": 1810
+    "limit": 1808
   },
   "reportRedeclaration": {
     "limit": 8
@@ -99,19 +99,19 @@
     "limit": 0
   },
   "reportUnknownArgumentType": {
-    "limit": 44528
+    "limit": 44526
   },
   "reportUnknownLambdaType": {
     "limit": 109
   },
   "reportUnknownMemberType": {
-    "limit": 38804
+    "limit": 38782
   },
   "reportUnknownParameterType": {
     "limit": 19829
   },
   "reportUnknownVariableType": {
-    "limit": 30355
+    "limit": 30349
   },
   "reportUnnecessaryCast": {
     "limit": 117
@@ -123,7 +123,7 @@
     "limit": 5
   },
   "reportUnnecessaryIsInstance": {
-    "limit": 833
+    "limit": 831
   },
   "reportUntypedBaseClass": {
     "limit": 0
@@ -135,7 +135,7 @@
     "limit": 21
   },
   "reportUnusedFunction": {
-    "limit": 139
+    "limit": 138
   },
   "reportUnusedImport": {
     "limit": 544

ci_cd/generate_model_prices_schema.py

@@ -73,6 +73,11 @@ ARRAY_KEYS: dict[str, JsonSchema] = {
         "description": "Output modalities the model can produce.",
         "items": {"type": "string", "enum": ["text", "image", "audio", "video", "code"]},
     },
+    "reasoning_effort_levels": {
+        "type": "array",
+        "description": "Exact reasoning_effort levels this deployment accepts; wins over supports_* flags.",
+        "items": {"type": "string", "enum": ["none", "minimal", "low", "medium", "high", "xhigh", "max"]},
+    },
     "supported_regions": {
         "type": "array",
         "description": "Cloud regions the model is available in ('global' or region ids).",

db_scripts/partition_spend_logs.sql

@@ -10,6 +10,11 @@
 -- partitioned, so existing installs are unaffected until you run this.
 --
 -- IMPORTANT
+--   * After partitioning, `prisma db push` (including the proxy's
+--     --use_prisma_db_push startup mode) is NOT supported: it tries to rewrite
+--     the primary key back to ("request_id"), which Postgres rejects on a
+--     partitioned table. The proxy detects this and exits with guidance.
+--     Use the default startup path (`prisma migrate deploy`) instead.
 --   * Test on a staging copy first and take a backup.
 --   * Postgres cannot convert a populated table to partitioned in place, so this
 --     renames the old table aside and creates a fresh partitioned table.

enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py

@@ -1,6 +1,8 @@
 """
 Polls LiteLLM_ManagedObjectTable to check if the response is complete.
-Cost tracking is handled automatically by the get-responses call.
+Cost tracking is handled by the get-responses call, which prices normally only because the
+poll stamps itself with BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN; user-facing reads of the
+same route are non-inference and free.
 """
 
 from datetime import datetime, timedelta, timezone
@@ -9,12 +11,14 @@ from typing import TYPE_CHECKING, Dict, Optional, cast
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm.constants import (
+    INTERNAL_CALL_ORIGIN_METADATA_KEY,
     MANAGED_OBJECT_STALENESS_CUTOFF_DAYS,
     MAX_OBJECTS_PER_POLL_CYCLE,
     STALE_OBJECT_CLEANUP_BATCH_SIZE,
 )
 from litellm.responses.utils import ResponsesAPIRequestUtils
 from litellm.types.llms.openai import ResponsesAPIResponse
+from litellm.types.utils import BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN
 
 if TYPE_CHECKING:
     from litellm.proxy.utils import PrismaClient, ProxyLogging
@@ -113,7 +117,8 @@ class CheckResponsesCost:
         Check if background responses are complete and track their cost.
         - Get all status="queued" or "in_progress" and file_purpose="response" jobs
         - Query the provider to check if response is complete
-        - Cost is automatically tracked by the get-responses call
+        - Cost is tracked by the get-responses call, billed because the poll is stamped
+          with BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN
         - Mark responses in a terminal state as complete in the database
         """
         try:
@@ -153,6 +158,7 @@ class CheckResponsesCost:
                 # Prepare metadata with model information for cost tracking
                 litellm_metadata = {
                     "user_api_key_user_id": job.created_by or "default-user-id",
+                    INTERNAL_CALL_ORIGIN_METADATA_KEY: BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN,
                 }
                 
                 # Add model information if available

enterprise/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "litellm-enterprise"
-version = "0.1.60"
+version = "0.1.61"
 description = "Package for LiteLLM Enterprise features"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -26,7 +26,7 @@ required-version = ">=0.10.9"
 module-root = ""
 
 [tool.commitizen]
-version = "0.1.60"
+version = "0.1.61"
 version_files = [
     "pyproject.toml:^version",
     "../pyproject.toml:litellm-enterprise==",

litellm-proxy-extras/litellm_proxy_extras/utils.py

@@ -40,6 +40,65 @@ def _get_prisma_env() -> dict:
 
 _MIGRATION_TS_RE = re.compile(r"^(\d{14})_")
 
+_SPEND_LOGS_ALTER_RE = re.compile(r'^ALTER\s+TABLE\s+"LiteLLM_SpendLogs"\s', re.IGNORECASE)
+_SPEND_LOGS_ARTIFACT_DROP_RE = re.compile(
+    r'^DROP\s+TABLE\s+"LiteLLM_SpendLogs_[^"]*"', re.IGNORECASE
+)
+_SPEND_LOGS_PK_CLAUSE_RE = re.compile(
+    r'^(?:DROP\s+CONSTRAINT\s+"[^"]*_pkey"'
+    r'|ADD\s+(?:CONSTRAINT\s+"[^"]*"\s+)?PRIMARY\s+KEY\s*\([^)]*\))$',
+    re.IGNORECASE,
+)
+
+PARTITIONED_SPEND_LOGS_PUSH_ERROR = (
+    "LiteLLM_SpendLogs is a partitioned table (see db_scripts/partition_spend_logs.sql), "
+    "so its primary key must include the partition key (\"startTime\"). `prisma db push` "
+    "reconciles the database against schema.prisma, which declares the unpartitioned "
+    "primary key (\"request_id\"), and Postgres rejects that rewrite with: unique "
+    "constraint on partitioned table must include all partitioning columns. Start the "
+    "proxy without --use_prisma_db_push so it uses `prisma migrate deploy`, which only "
+    "applies shipped migrations and leaves the partitioned primary key alone."
+)
+
+
+def _without_sql_comments(statement: str) -> str:
+    return "\n".join(
+        line
+        for line in statement.splitlines()
+        if line.strip() and not line.strip().startswith("--")
+    ).strip()
+
+
+def _without_spend_logs_pk_clauses(statement: str) -> Optional[str]:
+    prefix_match = _SPEND_LOGS_ALTER_RE.match(statement)
+    if not prefix_match:
+        return statement
+    kept = tuple(
+        clause.strip()
+        for clause in statement[prefix_match.end():].split(",\n")
+        if not _SPEND_LOGS_PK_CLAUSE_RE.match(clause.strip())
+    )
+    if not kept:
+        return None
+    return statement[: prefix_match.end()] + ",\n".join(kept)
+
+
+def filter_partitioned_spend_logs_diff(diff_sql: str) -> str:
+    """Drop statements from a `prisma migrate diff` script that fight the
+    SpendLogs partitioning runbook (db_scripts/partition_spend_logs.sql): the
+    primary-key rewrite on "LiteLLM_SpendLogs", which Postgres rejects on a
+    partitioned table, and drops of runbook artifacts such as
+    "LiteLLM_SpendLogs_legacy"."""
+    kept = tuple(
+        filtered
+        for statement in diff_sql.split(";")
+        for bare in (_without_sql_comments(statement),)
+        if bare and not _SPEND_LOGS_ARTIFACT_DROP_RE.match(bare)
+        for filtered in (_without_spend_logs_pk_clauses(bare),)
+        if filtered is not None
+    )
+    return "".join(f"{statement};\n\n" for statement in kept)
+
 
 def _migration_timestamp(name: str) -> int:
     """Extract the leading `YYYYMMDDHHMMSS` timestamp from a migration name.
@@ -355,7 +414,24 @@ class ProxyExtrasDBManager:
             return
         logger.info(f"Migration diff created at {diff_sql_path}")
 
+        if ProxyExtrasDBManager.spend_logs_is_partitioned():
+            filtered_sql = filter_partitioned_spend_logs_diff(
+                diff_sql_path.read_text()
+            )
+            diff_sql_path.write_text(filtered_sql)
+            logger.info(
+                "LiteLLM_SpendLogs is partitioned; removed its primary-key "
+                "rewrite and partitioning artifacts from the drift script"
+            )
+            if not filtered_sql.strip():
+                logger.info("Drift script is empty after filtering; nothing to apply")
+                if not mark_all_applied:
+                    return
+                ProxyExtrasDBManager._mark_migrations_applied(migrations_dir)
+                return
+
         # 2. Run prisma db execute to apply the migration
+        applied_ok = False
         try:
             logger.info("Running prisma db execute to apply the migration diff...")
             result = subprocess.run(
@@ -376,6 +452,7 @@ class ProxyExtrasDBManager:
             )
             logger.info(f"prisma db execute stdout: {result.stdout}")
             logger.info("✅ Migration diff applied successfully")
+            applied_ok = True
         except subprocess.CalledProcessError as e:
             logger.warning(f"Failed to apply migration diff: {e.stderr}")
         except subprocess.TimeoutExpired:
@@ -384,6 +461,16 @@ class ProxyExtrasDBManager:
         # 3. Mark all migrations as applied
         if not mark_all_applied:
             return
+        if not applied_ok:
+            logger.warning(
+                "Drift script failed to apply; NOT marking migrations as "
+                "applied so a later migration run can retry them"
+            )
+            return
+        ProxyExtrasDBManager._mark_migrations_applied(migrations_dir)
+
+    @staticmethod
+    def _mark_migrations_applied(migrations_dir: str):
         migration_names = ProxyExtrasDBManager._get_migration_names(migrations_dir)
         logger.info(f"Resolving {len(migration_names)} migrations")
         for migration_name in migration_names:
@@ -410,6 +497,55 @@ class ProxyExtrasDBManager:
                         f"Failed to resolve migration {migration_name}: {e.stderr}"
                     )
 
+    @staticmethod
+    def spend_logs_is_partitioned() -> bool:
+        """True when the connected database's LiteLLM_SpendLogs is a
+        partitioned table in Prisma's target schema (the `schema` URL param,
+        falling back to Prisma's default target, public), i.e. the operator
+        ran db_scripts/partition_spend_logs.sql. Returns False when psycopg is
+        unavailable or the database cannot be reached, preserving the
+        pre-existing behavior in those cases."""
+        database_url = os.getenv("DATABASE_URL")
+        if not database_url:
+            return False
+
+        try:
+            import psycopg
+        except ImportError:
+            return False
+
+        cleaned_url = ProxyExtrasDBManager._strip_prisma_query_params(database_url)
+        try:
+            with psycopg.connect(
+                cleaned_url, connect_timeout=10, autocommit=True
+            ) as conn:
+                row = conn.execute(
+                    "SELECT 1 "
+                    "FROM pg_partitioned_table pt "
+                    "JOIN pg_class c ON c.oid = pt.partrelid "
+                    "JOIN pg_namespace n ON n.oid = c.relnamespace "
+                    "WHERE c.relname = 'LiteLLM_SpendLogs' "
+                    "  AND n.nspname = %s",
+                    (
+                        ProxyExtrasDBManager._prisma_schema_param(database_url)
+                        or "public",
+                    ),
+                ).fetchone()
+        except (psycopg.OperationalError, psycopg.DatabaseError):
+            return False
+        return row is not None
+
+    @staticmethod
+    def _prisma_schema_param(url: str) -> Optional[str]:
+        """The `schema` query param Prisma uses to pick its target schema,
+        or None when the URL does not set one."""
+        from urllib.parse import urlparse, parse_qsl
+
+        return next(
+            (v for k, v in parse_qsl(urlparse(url).query) if k == "schema"),
+            None,
+        )
+
     @staticmethod
     def _strip_prisma_query_params(url: str) -> str:
         """Remove Prisma-specific query params (connection_limit, pool_timeout,
@@ -528,7 +664,8 @@ class ProxyExtrasDBManager:
         migrations_dir = ProxyExtrasDBManager._get_prisma_dir()
 
         if not use_migrate:
-            # Preserve `prisma db push` path unchanged.
+            if ProxyExtrasDBManager.spend_logs_is_partitioned():
+                raise RuntimeError(PARTITIONED_SPEND_LOGS_PUSH_ERROR)
             original_dir = os.getcwd()
             os.chdir(migrations_dir)
             try:
@@ -972,6 +1109,8 @@ class ProxyExtrasDBManager:
                                 )
                                 raise
                 else:
+                    if ProxyExtrasDBManager.spend_logs_is_partitioned():
+                        raise RuntimeError(PARTITIONED_SPEND_LOGS_PUSH_ERROR)
                     # Use prisma db push with increased timeout
                     subprocess.run(
                         [_get_prisma_command(), "db", "push", "--accept-data-loss"],

litellm-proxy-extras/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "litellm-proxy-extras"
-version = "0.4.89"
+version = "0.4.90"
 description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -26,7 +26,7 @@ required-version = ">=0.10.9"
 module-root = ""
 
 [tool.commitizen]
-version = "0.4.89"
+version = "0.4.90"
 version_files = [
     "pyproject.toml:^version",
     "../pyproject.toml:litellm-proxy-extras==",

litellm/__init__.py

@@ -445,6 +445,7 @@ max_ui_session_budget: Optional[float] = (
     1.0  # USD budget for each dashboard login session (playground, test connection)
 )
 internal_user_budget_duration: Optional[str] = None
+budget_rollover: bool = False  # carry spend beyond max_budget into the next window instead of zeroing it
 tag_budget_config: Optional[Dict[str, "BudgetConfig"]] = None
 max_end_user_budget: Optional[float] = None
 max_end_user_budget_id: Optional[str] = None

litellm/_logging.py

@@ -5,7 +5,7 @@ import os
 import sys
 from datetime import datetime
 from logging import Formatter
-from typing import Any, Final
+from typing import Any, Final, TextIO
 
 import litellm
 from litellm.constants import (
@@ -234,11 +234,65 @@ class CorrelationContextFilter(logging.Filter):
 _correlation_filter: Final = CorrelationContextFilter()
 
 
-json_logs = bool(os.getenv("JSON_LOGS", False))
+_LOG_FORMAT_PREFIX: Final = "%(asctime)s - %(name)s:%(levelname)s"
+_LOG_FORMAT_SUFFIX: Final = ": %(filename)s:%(lineno)s - %(message)s"
+_PLAIN_LOG_FORMAT: Final = _LOG_FORMAT_PREFIX + _LOG_FORMAT_SUFFIX
+_COLOR_LOG_FORMAT: Final = f"\033[92m{_LOG_FORMAT_PREFIX}\033[0m{_LOG_FORMAT_SUFFIX}"
+
+
+def _stream_is_tty(stream: TextIO | None) -> bool:
+    """True when the stream is an open interactive terminal; never raises.
+
+    A stream can be None (pythonw/embedded interpreters), lack isatty entirely
+    (GUI log-redirect shims), or be closed; import must survive all three.
+    """
+    try:
+        return stream is not None and stream.isatty()
+    except (AttributeError, ValueError):
+        return False
+
+
+def _plain_log_format(stdout: TextIO | None, stderr: TextIO | None) -> str:
+    """The plain-text log format, colorized only when both streams are an interactive terminal.
+
+    Honors the NO_COLOR convention from no-color.org: color is disabled when
+    NO_COLOR is present with a non-empty value.
+    """
+    if os.environ.get("NO_COLOR"):
+        return _PLAIN_LOG_FORMAT
+    return _COLOR_LOG_FORMAT if _stream_is_tty(stdout) and _stream_is_tty(stderr) else _PLAIN_LOG_FORMAT
+
+
+class LevelRoutingStreamHandler(logging.StreamHandler):
+    """Writes records below WARNING to stdout and WARNING and above to stderr.
+
+    Collectors that derive severity from the stream report every stderr line as an error.
+    """
+
+    def emit(self, record: logging.LogRecord) -> None:
+        preferred: Final = sys.stdout if record.levelno < logging.WARNING else sys.stderr
+        if preferred is None or getattr(preferred, "closed", False):
+            self.stream = sys.stderr  # rebind-ok: fall back to the pre-fix stream rather than raising per record
+        else:
+            self.stream = preferred  # rebind-ok: StreamHandler.emit writes self.stream under the handler lock
+        super().emit(record)
+
+
+def _parse_json_logs_env(value: str | None) -> bool:
+    """Strict opt-in parse for the JSON_LOGS env var: only "true" (any case) enables JSON logs.
+
+    Matches the reader in litellm-proxy-extras/_logging.py. The previous
+    bool(os.getenv(...)) treated any non-empty value, including "false" and "0",
+    as enabled.
+    """
+    return (value or "").lower() == "true"
+
+
+json_logs: Final = _parse_json_logs_env(os.getenv("JSON_LOGS"))
 # Create a handler for the logger (you may need to adapt this based on your needs)
 log_level: Final = os.getenv("LITELLM_LOG", "DEBUG")
 numeric_level: Final[str] = getattr(logging, log_level.upper())
-handler: Final = logging.StreamHandler()
+handler: Final = LevelRoutingStreamHandler()
 handler.setLevel(numeric_level)
 handler.addFilter(_secret_filter)
 handler.addFilter(_correlation_filter)
@@ -447,7 +501,7 @@ if json_logs:
     _setup_json_exception_handlers(JsonFormatter())
 else:
     formatter: Final = CorrelationPlainFormatter(
-        "\033[92m%(asctime)s - %(name)s:%(levelname)s\033[0m: %(filename)s:%(lineno)s - %(message)s",
+        _plain_log_format(sys.stdout, sys.stderr),
         datefmt="%H:%M:%S",
     )
 
@@ -628,7 +682,7 @@ def _turn_on_json():
 
     - Adds a JSON formatter to all loggers
     """
-    handler: Final = logging.StreamHandler()
+    handler: Final = LevelRoutingStreamHandler()
     handler.setFormatter(JsonFormatter())
     _initialize_loggers_with_handler(handler)
     # Set up exception handlers

litellm/completion_extras/litellm_responses_transformation/transformation.py

@@ -59,9 +59,11 @@ if TYPE_CHECKING:
     from litellm.types.llms.openai import (
         ALL_RESPONSES_API_TOOL_PARAMS,
         AllMessageValues,
+        ChatCompletionFileObject,
         ChatCompletionImageObject,
         ChatCompletionRedactedThinkingBlock,
         ChatCompletionThinkingBlock,
+        ChatCompletionToolReferenceObject,
         OpenAIMessageContentListBlock,
     )
     from litellm.types.utils import Choices
@@ -175,6 +177,16 @@ def _map_incomplete_reason_to_finish_reason(incomplete_reason: str | None) -> Li
     return "length"
 
 
+def _input_file_from_file_value(file_value: object) -> dict[str, object]:
+    if not isinstance(file_value, dict):
+        return {"type": "input_file"}
+    file_dict: Final = cast("dict[str, object]", file_value)  # cast-ok: runtime dict checked
+    return {
+        "type": "input_file",
+        **{key: file_dict[key] for key in ("file_id", "file_data", "filename") if key in file_dict},
+    }
+
+
 def _incomplete_reason_from_response_payload(response_payload: object) -> str | None:
     if not isinstance(response_payload, Mapping):
         return None
@@ -957,7 +969,12 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
         content: str
         | list[object]
         | Iterable[
-            Union["OpenAIMessageContentListBlock", "ChatCompletionThinkingBlock", "ChatCompletionRedactedThinkingBlock"]
+            Union[
+                "OpenAIMessageContentListBlock",
+                "ChatCompletionThinkingBlock",
+                "ChatCompletionRedactedThinkingBlock",
+                "ChatCompletionToolReferenceObject",
+            ]
         ]
         | None,
         role: str,
@@ -1006,17 +1023,15 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
                             result.append(converted)
                             verbose_logger.debug("Chat provider:   image -> %s", converted)
                         elif item_type == "file":
-                            # Map Chat Completion file to Responses API input_file
-                            # {"type": "file", "file": {"file_data": "...", "filename": "..."}}
-                            # -> {"type": "input_file", "file_data": "...", "filename": "..."}
-                            file_data = item.get("file", {})
-                            converted = {"type": "input_file"}
-                            if isinstance(file_data, dict):
-                                for key in ["file_id", "file_data", "filename"]:
-                                    if key in file_data:
-                                        converted[key] = file_data[key]
+                            converted = _input_file_from_file_value(
+                                cast("ChatCompletionFileObject", item).get("file"),  # cast-ok: type tag checked
+                            )
                             result.append(converted)
                             verbose_logger.debug("Chat provider:   file -> %s", converted)
+                        elif item_type == "tool_reference":
+                            verbose_logger.debug(
+                                "Chat provider:   tool_reference has no responses API equivalent; skipped"
+                            )
                         elif item_type in [
                             "input_text",
                             "input_image",

litellm/constants.py

@@ -296,6 +296,9 @@ GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS: Final = int(
     os.getenv("GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS", 24 * 60 * 60)
 )
 BEDROCK_APPLY_GUARDRAIL_CHUNK_BUDGET_CHARS: Final = 25_000
+DEFAULT_PRESIDIO_ANALYZE_CHUNK_SIZE_BYTES: Final = 500_000
+PRESIDIO_ANALYZE_CHUNK_OVERLAP_CHARS: Final = 4096
+PRESIDIO_ANALYZE_CHUNK_CONCURRENCY: Final = 8
 # Aggregation threshold: default to 80% of the asyncio queue maxsize so the check can always trigger.
 # Must be < LITELLM_ASYNCIO_QUEUE_MAXSIZE; if set higher the aggregation logic will never fire.
 MAX_SIZE_IN_MEMORY_QUEUE: Final = int(os.getenv("MAX_SIZE_IN_MEMORY_QUEUE", int(LITELLM_ASYNCIO_QUEUE_MAXSIZE * 0.8)))
@@ -1364,8 +1367,6 @@ X_LITELLM_DISABLE_CALLBACKS: Final = "x-litellm-disable-callbacks"
 LITELLM_METADATA_FIELD: Final = "litellm_metadata"
 OLD_LITELLM_METADATA_FIELD: Final = "metadata"
 RETURN_RAW_MODEL_NAME_METADATA_KEY: Final = "_complexity_router_return_raw_model_name"
-AUTO_ROUTED_REQUEST_METADATA_KEY: Final = "_auto_routed_request"
-ROUTER_MODEL_NAME_RESPONSE_FIELD: Final = "router_model_name"
 SESSION_DEPLOYMENT_AFFINITY_TTL_METADATA_KEY: Final = "_session_deployment_affinity_ttl"
 CONSUMED_REQUEST_TAGS_METADATA_KEY: Final = "_consumed_request_tags"
 INTERNAL_CALL_ORIGIN_METADATA_KEY: Final = "internal_call_origin"
@@ -1475,6 +1476,12 @@ LITELLM_PROXY_MASTER_KEY_ALIAS: Final = "litellm_proxy_master_key"
 # ``ProxyLogging._handle_logging_proxy_only_error``.
 LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL: Final = "litellm_no_upstream_llm_call"
 
+# Key/team metadata fields naming the OTel Resource ``service.name``, highest
+# precedence first. Shared between the OTel v2 tenant router (which reads them
+# out of ``user_api_key_auth_metadata``) and proxy request setup (which re-applies
+# the key's values after the team metadata merge so a key outranks its team).
+OTEL_SERVICE_NAME_METADATA_KEYS: Final = ("otel_service_name_override", "otel_service_name")
+
 # Key Rotation Constants
 LITELLM_KEY_ROTATION_ENABLED: Final = os.getenv("LITELLM_KEY_ROTATION_ENABLED", "false")
 LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS: Final = int(
@@ -1648,6 +1655,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
     "enable_anthropic_prompt_caching",
     "anthropic_prompt_caching_ttl",
     "max_ui_session_budget",
+    "budget_rollover",
 ]
 SPECIAL_LITELLM_AUTH_TOKEN: Final = ["ui-token"]
 DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL = int(os.getenv("DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL", 60))
@@ -1814,6 +1822,43 @@ BROWSER_SECURITY_HEADERS: Final[frozenset[str]] = frozenset(
 
 UNSAFE_PROXY_RESPONSE_HEADERS: Final[frozenset[str]] = HTTP_FRAMING_HEADERS | BROWSER_SECURITY_HEADERS
 
+# A retrieved response replays the usage of the call that created it, so pricing these
+# read/management routes like inference bills the same tokens twice.
+NON_INFERENCE_CALL_TYPES: Final[frozenset[str]] = frozenset(
+    {
+        "get_responses",
+        "aget_responses",
+        "delete_responses",
+        "adelete_responses",
+        "cancel_responses",
+        "acancel_responses",
+        "list_input_items",
+        "alist_input_items",
+        "vector_store_create",
+        "avector_store_create",
+        "vector_store_retrieve",
+        "avector_store_retrieve",
+        "vector_store_list",
+        "avector_store_list",
+        "vector_store_update",
+        "avector_store_update",
+        "vector_store_delete",
+        "avector_store_delete",
+        "vector_store_file_create",
+        "avector_store_file_create",
+        "vector_store_file_list",
+        "avector_store_file_list",
+        "vector_store_file_retrieve",
+        "avector_store_file_retrieve",
+        "vector_store_file_content",
+        "avector_store_file_content",
+        "vector_store_file_update",
+        "avector_store_file_update",
+        "vector_store_file_delete",
+        "avector_store_file_delete",
+    }
+)
+
 # PTU reservation rollup writes rows to LiteLLM_DailyTeamSpend with this
 # sentinel api_key so PTU flat cost stays distinguishable from real per-request
 # spend under the table's composite unique constraint.

litellm/cost_calculator.py

@@ -76,7 +76,10 @@ from litellm.llms.perplexity.cost_calculator import (
 from litellm.llms.tencent.cost_calculator import (
     cost_per_token as tencent_cost_per_token,
 )
-from litellm.llms.together_ai.cost_calculator import get_model_params_and_category
+from litellm.llms.together_ai.cost_calculator import (
+    get_model_params_and_category,
+    has_together_registry_pricing,
+)
 from litellm.llms.vertex_ai.cost_calculator import (
     cost_per_character as google_cost_per_character,
 )
@@ -557,9 +560,10 @@ def cost_per_token(
         )
     elif call_type == "atranscription" or call_type == "transcription":
         if _transcription_usage_has_token_details(usage_block):
-            return openai_cost_per_token(
+            return generic_cost_per_token(
                 model=model_without_prefix,
                 usage=usage_block,
+                custom_llm_provider=custom_llm_provider,
                 service_tier=service_tier,
                 data_residency=data_residency,
             )
@@ -1568,10 +1572,9 @@ def completion_cost(
 
                     return MCPCostCalculator.calculate_mcp_tool_call_cost(litellm_logging_obj=litellm_logging_obj)
                 # Calculate cost based on prompt_tokens, completion_tokens
-                if "togethercomputer" in model or "together_ai" in model or custom_llm_provider == "together_ai":
-                    # together ai prices based on size of llm
-                    # get_model_params_and_category takes a model name and returns the category of LLM size it is in model_prices_and_context_window.json
-
+                if (
+                    "togethercomputer" in model or "together_ai" in model or custom_llm_provider == "together_ai"
+                ) and not has_together_registry_pricing(model, litellm.model_cost):
                     model = get_model_params_and_category(model, call_type=CallTypes(call_type))
 
                 # replicate llms are calculate based on time for request running

litellm/experimental_mcp_client/client.py

@@ -56,6 +56,9 @@ from litellm.types.mcp import (
     MCPStdioConfig,
     MCPTransport,
     MCPTransportType,
+    credential_redirect_hook,
+    has_header,
+    without_header,
 )
 
 
@@ -273,6 +276,7 @@ class MCPClient:
         transport_type: MCPTransportType = MCPTransport.http,
         auth_type: MCPAuthType = None,
         auth_value: str | dict[str, str] | None = None,
+        auth_header_name: str | None = None,
         timeout: float | None = None,
         stdio_config: MCPStdioConfig | None = None,
         extra_headers: dict[str, str] | None = None,
@@ -288,6 +292,11 @@ class MCPClient:
         self.auth_type: MCPAuthType = auth_type
         self.timeout: float = timeout if timeout is not None else MCP_CLIENT_TIMEOUT
         self._mcp_auth_value: str | dict[str, str] | None = None
+        # The one place this client decides which header its credential occupies: the operator's
+        # configured slot on the v1 path, or the slot the v2 resolver's auth object already owns.
+        # Every consumer reads this rather than re-deriving it, since each re-derivation so far
+        # picked up a different bug.
+        self._credential_slot: str | None = auth_header_name or getattr(resolved_auth, "header_name", None)
         self.stdio_config: MCPStdioConfig | None = stdio_config
         self.extra_headers: dict[str, str] | None = extra_headers
         self.ssl_verify: VerifyTypes | None = ssl_verify
@@ -501,26 +510,33 @@ class MCPClient:
         else:
             self._mcp_auth_value = mcp_auth_value
 
+    def _header_slot(self, default: str) -> str:
+        return self._credential_slot or default
+
     def _get_auth_headers(self) -> dict:
         """Generate authentication headers based on auth type."""
         headers: Final = {}
         if self._mcp_auth_value:
             if isinstance(self._mcp_auth_value, str):
                 if self.auth_type == MCPAuth.bearer_token:
-                    headers["Authorization"] = f"Bearer {strip_auth_scheme(self._mcp_auth_value, 'Bearer')}"
+                    static_bearer: Final = strip_auth_scheme(self._mcp_auth_value, "Bearer")
+                    headers[self._header_slot("Authorization")] = f"Bearer {static_bearer}"
                 elif self.auth_type == MCPAuth.basic:
-                    headers["Authorization"] = f"Basic {self._mcp_auth_value}"
+                    headers[self._header_slot("Authorization")] = f"Basic {self._mcp_auth_value}"
                 elif self.auth_type == MCPAuth.api_key:
-                    headers["X-API-Key"] = self._mcp_auth_value
+                    headers[self._header_slot("X-API-Key")] = self._mcp_auth_value
                 elif self.auth_type == MCPAuth.authorization:
                     # This auth type means the caller owns the whole header value.
-                    headers["Authorization"] = self._mcp_auth_value
+                    headers[self._header_slot("Authorization")] = self._mcp_auth_value
                 elif self.auth_type == MCPAuth.oauth2:
-                    headers["Authorization"] = f"Bearer {strip_auth_scheme(self._mcp_auth_value, 'Bearer')}"
+                    oauth2_bearer: Final = strip_auth_scheme(self._mcp_auth_value, "Bearer")
+                    headers[self._header_slot("Authorization")] = f"Bearer {oauth2_bearer}"
                 elif self.auth_type == MCPAuth.token:
-                    headers["Authorization"] = f"token {strip_auth_scheme(self._mcp_auth_value, 'token')}"
+                    scheme_token: Final = strip_auth_scheme(self._mcp_auth_value, "token")
+                    headers[self._header_slot("Authorization")] = f"token {scheme_token}"
                 elif self.auth_type == MCPAuth.oauth2_token_exchange:
-                    headers["Authorization"] = f"Bearer {strip_auth_scheme(self._mcp_auth_value, 'Bearer')}"
+                    exchanged_bearer: Final = strip_auth_scheme(self._mcp_auth_value, "Bearer")
+                    headers[self._header_slot("Authorization")] = f"Bearer {exchanged_bearer}"
             elif isinstance(self._mcp_auth_value, dict):
                 headers.update(self._mcp_auth_value)
         # Note: aws_sigv4 auth is not handled here — SigV4 requires per-request
@@ -528,7 +544,14 @@ class MCPClient:
         # of static headers. See MCPSigV4Auth and _create_httpx_client_factory().
         # update the headers with the extra headers
         if self.extra_headers:
-            headers.update(self.extra_headers)
+            # Mirrors _resolve_v2_auth: when the operator named a slot for the credential the
+            # gateway resolved, no injected header may shadow it, case-insensitively, since HTTP
+            # header names are. Without a configured slot the old precedence stands unchanged.
+            slot: Final = self._credential_slot
+            injected: Final = (
+                without_header(self.extra_headers, slot) if slot and has_header(headers, slot) else self.extra_headers
+            )
+            headers.update(injected or {})
         return _strip_header_whitespace(headers)
 
     def _create_httpx_client_factory(self) -> Callable[..., httpx.AsyncClient]:
@@ -556,12 +579,14 @@ class MCPClient:
             # SigV4 aws_auth. Both are None for the common case — no behavior change.
             fallback_auth: Final = self._resolved_auth if self._resolved_auth is not None else self._aws_auth
             effective_auth: Final = auth if auth is not None else fallback_auth
+            guard: Final = credential_redirect_hook(self.server_url, self._credential_slot)
             return httpx.AsyncClient(
                 headers=headers,
                 timeout=timeout,
                 auth=effective_auth,
                 verify=ssl_config,
                 follow_redirects=True,
+                event_hooks={"request": [guard]} if guard else {},
             )
 
         return factory

litellm/google_genai/streaming_iterator.py

@@ -2,6 +2,7 @@ import asyncio
 from datetime import datetime
 from typing import TYPE_CHECKING, Any, Final
 
+import litellm
 from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
 from litellm.proxy.pass_through_endpoints.success_handler import (
     PassThroughEndpointLogging,
@@ -65,6 +66,7 @@ class BaseGoogleGenAIGenerateContentStreamingIterator:
         litellm_logging_obj: LiteLLMLoggingObj,
         request_body: dict,
         model: str,
+        custom_llm_provider: str,
         hidden_params: dict[str, Any] | None = None,
     ):
         self.litellm_logging_obj = litellm_logging_obj
@@ -72,6 +74,10 @@ class BaseGoogleGenAIGenerateContentStreamingIterator:
         self.start_time = datetime.now()
         self.collected_chunks: list[bytes] = []
         self.model = model
+        self.custom_llm_provider = custom_llm_provider
+        self.endpoint_type: Final = (
+            EndpointType.GEMINI if custom_llm_provider == litellm.LlmProviders.GEMINI.value else EndpointType.VERTEX_AI
+        )
         self._hidden_params: dict[str, Any] = hidden_params or {}
 
     async def _handle_async_streaming_logging(
@@ -89,7 +95,7 @@ class BaseGoogleGenAIGenerateContentStreamingIterator:
                 passthrough_success_handler_obj=GLOBAL_PASS_THROUGH_SUCCESS_HANDLER_OBJ,
                 url_route="/v1/generateContent",
                 request_body=self.request_body or {},
-                endpoint_type=EndpointType.VERTEX_AI,
+                endpoint_type=self.endpoint_type,
                 start_time=self.start_time,
                 raw_bytes=self.collected_chunks,
                 end_time=end_time,
@@ -118,13 +124,13 @@ class GoogleGenAIGenerateContentStreamingIterator(BaseGoogleGenAIGenerateContent
             litellm_logging_obj=logging_obj,
             request_body=request_body or {},
             model=model,
+            custom_llm_provider=custom_llm_provider,
             hidden_params=hidden_params,
         )
         self.response = response
         self.model = model
         self.generate_content_provider_config = generate_content_provider_config
         self.litellm_metadata = litellm_metadata
-        self.custom_llm_provider = custom_llm_provider
         # Gemini streamGenerateContent uses SSE line framing; iter_lines keeps
         # large inlineData payloads (e.g. image/jpeg) intact within one event.
         self.stream_iterator = response.iter_lines()
@@ -169,13 +175,13 @@ class AsyncGoogleGenAIGenerateContentStreamingIterator(BaseGoogleGenAIGenerateCo
             litellm_logging_obj=logging_obj,
             request_body=request_body or {},
             model=model,
+            custom_llm_provider=custom_llm_provider,
             hidden_params=hidden_params,
         )
         self.response = response
         self.model = model
         self.generate_content_provider_config = generate_content_provider_config
         self.litellm_metadata = litellm_metadata
-        self.custom_llm_provider = custom_llm_provider
         # Gemini streamGenerateContent uses SSE line framing; aiter_lines keeps
         # large inlineData payloads (e.g. image/jpeg) intact within one event.
         self.stream_iterator = response.aiter_lines()

litellm/integrations/SlackAlerting/batching_handler.py

@@ -10,6 +10,8 @@ from typing import TYPE_CHECKING, Any, Final
 
 from litellm._logging import verbose_proxy_logger
 
+from .ms_teams import MS_TEAMS_ALERTING_DESTINATION, build_ms_teams_payload
+
 if TYPE_CHECKING:
     from .slack_alerting import SlackAlerting as _SlackAlerting
 
@@ -62,14 +64,17 @@ async def send_to_webhook(slackAlertingInstance: SlackAlertingType, item, count)
         if count > 1:
             payload["text"] = f"[Num Alerts: {count}]\n\n{payload['text']}"
 
+        request_body: Final = (
+            build_ms_teams_payload(payload["text"]) if item.get("format") == MS_TEAMS_ALERTING_DESTINATION else payload
+        )
         response: Final = await slackAlertingInstance.async_http_handler.post(
             url=item["url"],
             headers=item["headers"],
-            data=json.dumps(payload),
+            data=json.dumps(request_body),
         )
         if response.status_code != 200:
-            verbose_proxy_logger.debug("Error sending slack alert to url=%s. Error=%s", item["url"], response.text)
+            verbose_proxy_logger.debug("Error sending alert to url=%s. Error=%s", item["url"], response.text)
     except Exception as e:
-        verbose_proxy_logger.debug("Error sending slack alert: %s", e)
+        verbose_proxy_logger.debug("Error sending alert: %s", e)
     finally:
         _print_alerting_payload_warning(payload, slackAlertingInstance=slackAlertingInstance)

litellm/integrations/SlackAlerting/ms_teams.py

@@ -0,0 +1,75 @@
+"""Microsoft Teams alert delivery helpers.
+
+Teams incoming webhooks (Workflows and legacy connectors) accept an Adaptive
+Card wrapped in a message attachment, so alert text is delivered as a single
+wrapped TextBlock.
+"""
+
+import os
+from collections.abc import Mapping
+from types import MappingProxyType
+from typing import Final
+
+from typing_extensions import ReadOnly, TypedDict
+
+from litellm.types.integrations.slack_alerting import AlertType
+
+MS_TEAMS_WEBHOOK_URL_ENV: Final = "MS_TEAMS_WEBHOOK_URL"
+
+MS_TEAMS_ALERTING_DESTINATION: Final = "ms_teams"
+
+MS_TEAMS_ALERT_HEADERS: Final[Mapping[str, str]] = MappingProxyType({"Content-type": "application/json"})
+
+
+class MSTeamsTextBlock(TypedDict):
+    type: ReadOnly[str]
+    text: ReadOnly[str]
+    wrap: ReadOnly[bool]
+
+
+class MSTeamsAdaptiveCard(TypedDict):
+    type: ReadOnly[str]
+    version: ReadOnly[str]
+    body: ReadOnly[tuple[MSTeamsTextBlock, ...]]
+
+
+class MSTeamsAttachment(TypedDict):
+    contentType: ReadOnly[str]
+    content: ReadOnly[MSTeamsAdaptiveCard]
+
+
+class MSTeamsMessage(TypedDict):
+    type: ReadOnly[str]
+    attachments: ReadOnly[tuple[MSTeamsAttachment, ...]]
+
+
+class MSTeamsAlertText(TypedDict):
+    text: ReadOnly[str]
+
+
+class MSTeamsQueueItem(TypedDict):
+    url: ReadOnly[str]
+    headers: ReadOnly[Mapping[str, str]]
+    payload: ReadOnly[MSTeamsAlertText]
+    alert_type: ReadOnly[AlertType]
+    format: ReadOnly[str]
+
+
+def get_ms_teams_webhook_url() -> str | None:
+    return os.getenv(MS_TEAMS_WEBHOOK_URL_ENV)
+
+
+def build_ms_teams_payload(text: str) -> MSTeamsMessage:
+    return MSTeamsMessage(
+        type="message",
+        attachments=(
+            MSTeamsAttachment(
+                contentType="application/vnd.microsoft.card.adaptive",
+                content=MSTeamsAdaptiveCard(
+                    type="AdaptiveCard",
+                    version="1.4",
+                    body=(MSTeamsTextBlock(type="TextBlock", text=text, wrap=True),),
+                ),
+            ),
+        ),
+    )

litellm/integrations/SlackAlerting/slack_alerting.py

@@ -57,6 +57,13 @@ from litellm.types.proxy.model_deprecation import (
 
 from ..email_templates.templates import *
 from .batching_handler import send_to_webhook, squash_payloads
+from .ms_teams import (
+    MS_TEAMS_ALERT_HEADERS,
+    MS_TEAMS_ALERTING_DESTINATION,
+    MSTeamsAlertText,
+    MSTeamsQueueItem,
+    get_ms_teams_webhook_url,
+)
 from .utils import process_slack_alerting_variables
 
 if TYPE_CHECKING:
@@ -1431,13 +1438,45 @@ Model Info:
             # only send budget alerts over Email
             await self.send_email_alert_using_smtp(webhook_event=user_info, alert_type=alert_type)
 
-        if "slack" not in self.alerting:
+        send_to_slack: Final = "slack" in self.alerting
+        send_to_ms_teams: Final = MS_TEAMS_ALERTING_DESTINATION in self.alerting
+        if not send_to_slack and not send_to_ms_teams:
             return
         if alert_type not in self.alert_types:
             return
 
         from datetime import datetime
 
+        # Get the current timestamp
+        current_time: Final = datetime.now().strftime("%H:%M:%S")
+        _proxy_base_url: Final = os.getenv("PROXY_BASE_URL", None)
+        # Use .name if it's an enum, otherwise use as is
+        alert_type_name: Final = getattr(alert_type, "name", alert_type)
+        alert_type_formatted: Final = f"Alert type: `{alert_type_name}`"
+        if alert_type == "daily_reports" or alert_type == "new_model_added":
+            formatted_message = alert_type_formatted + message
+        else:
+            formatted_message = (
+                f"{alert_type_formatted}\nLevel: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
+            )
+
+        if kwargs:
+            for key, value in kwargs.items():
+                formatted_message += f"\n\n{key}: `{value}`\n\n"
+        if alerting_metadata:
+            for key, value in alerting_metadata.items():
+                formatted_message += f"\n\n*Alerting Metadata*: \n{key}: `{value}`\n\n"
+        if _proxy_base_url is not None:
+            formatted_message += f"\n\nProxy URL: `{_proxy_base_url}`"
+
+        if send_to_ms_teams:
+            self._enqueue_ms_teams_alert(formatted_message=formatted_message, alert_type=alert_type)
+
+        if not send_to_slack:
+            if len(self.log_queue) >= self.batch_size:
+                await self.flush_queue()
+            return
+
         # Check if digest mode is enabled for this alert type
         alert_type_name_str: Final = getattr(alert_type, "value", str(alert_type))
         _atc: Final = self.alert_type_config.get(alert_type_name_str)
@@ -1473,28 +1512,6 @@ Model Info:
                     )
             return  # Suppress immediate alert; will be emitted by _flush_digest_buckets
 
-        # Get the current timestamp
-        current_time: Final = datetime.now().strftime("%H:%M:%S")
-        _proxy_base_url: Final = os.getenv("PROXY_BASE_URL", None)
-        # Use .name if it's an enum, otherwise use as is
-        alert_type_name: Final = getattr(alert_type, "name", alert_type)
-        alert_type_formatted: Final = f"Alert type: `{alert_type_name}`"
-        if alert_type == "daily_reports" or alert_type == "new_model_added":
-            formatted_message = alert_type_formatted + message
-        else:
-            formatted_message = (
-                f"{alert_type_formatted}\nLevel: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
-            )
-
-        if kwargs:
-            for key, value in kwargs.items():
-                formatted_message += f"\n\n{key}: `{value}`\n\n"
-        if alerting_metadata:
-            for key, value in alerting_metadata.items():
-                formatted_message += f"\n\n*Alerting Metadata*: \n{key}: `{value}`\n\n"
-        if _proxy_base_url is not None:
-            formatted_message += f"\n\nProxy URL: `{_proxy_base_url}`"
-
         # check if we find the slack webhook url in self.alert_to_webhook_url
         if self.alert_to_webhook_url is not None and alert_type in self.alert_to_webhook_url:
             slack_webhook_url: str | list[str] | None = self.alert_to_webhook_url[alert_type]
@@ -1531,6 +1548,24 @@ Model Info:
         if len(self.log_queue) >= self.batch_size:
             await self.flush_queue()
 
+    def _enqueue_ms_teams_alert(self, formatted_message: str, alert_type: AlertType) -> None:
+        ms_teams_webhook_url: Final = get_ms_teams_webhook_url()
+        if ms_teams_webhook_url is None:
+            verbose_proxy_logger.error(
+                "MS Teams alerting is enabled but MS_TEAMS_WEBHOOK_URL is not set. Dropping alert type=%s",
+                alert_type,
+            )
+            return
+        payload: Final[MSTeamsAlertText] = {"text": formatted_message}
+        item: Final[MSTeamsQueueItem] = {
+            "url": ms_teams_webhook_url,
+            "headers": MS_TEAMS_ALERT_HEADERS,
+            "payload": payload,
+            "alert_type": alert_type,
+            "format": MS_TEAMS_ALERTING_DESTINATION,
+        }
+        self.log_queue.append(item)
+
     async def async_send_batch(self):
         if not self.log_queue:
             return

litellm/integrations/anthropic_cache_control_hook.py

@@ -376,7 +376,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
         # 2. list of objects - only apply to last item per Anthropic spec
         elif isinstance(message_content, list):
             if len(message_content) > 0 and isinstance(message_content[-1], dict):
-                message_content[-1]["cache_control"] = control
+                message_content[-1]["cache_control"] = control  # pyright: ignore[reportGeneralTypeIssues]  # loose runtime dict
         return message
 
     @staticmethod

litellm/integrations/custom_batch_logger.py

@@ -45,7 +45,7 @@ class CustomBatchLogger(CustomLogger):
 
         super().__init__(**kwargs)
 
-    async def periodic_flush(self):
+    async def periodic_flush(self) -> None:
         while True:
             await asyncio.sleep(self.flush_interval)
             verbose_logger.debug("CustomLogger periodic flush after %s seconds", self.flush_interval)

litellm/integrations/dotprompt/prompt_manager.py

@@ -149,7 +149,6 @@ class PromptManager:
                 )
                 self.prompts[template_id] = template
             except Exception:
-                # Optional: print(f"Error loading prompt from JSON: {template_id}")
                 pass
 
     def _load_prompt_file(self, file_path: str | Path, prompt_id: str) -> PromptTemplate:

litellm/integrations/langfuse/langfuse.py

@@ -5,6 +5,7 @@ import os
 import traceback
 from collections.abc import Callable, Iterable, Mapping
 from datetime import datetime
+from functools import lru_cache
 from types import MappingProxyType
 from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, cast
 
@@ -137,6 +138,16 @@ def resolve_langfuse_credentials(
     return public_key, secret_key, resolved_host
 
 
+@lru_cache(maxsize=8)
+def _warn_invalid_deployment_environment(raw_value: str, error: str) -> None:
+    verbose_logger.warning(
+        "Ignoring invalid LANGFUSE_TRACING_ENVIRONMENT=%r for the langfuse callback: %s. "
+        "Traces will be sent to Langfuse's default environment.",
+        raw_value,
+        error,
+    )
+
+
 class LangFuseLogger:
     # Class variables or attributes
     def __init__(
@@ -165,9 +176,11 @@ class LangFuseLogger:
             # add http:// if unset, assume communicating over private network - e.g. render
             self.langfuse_host = "http://" + self.langfuse_host
         _env_override: Final = str(langfuse_environment).strip() if langfuse_environment is not None else None
-        self.langfuse_environment = _env_override or os.getenv("LANGFUSE_TRACING_ENVIRONMENT")
-        if self.langfuse_environment:
-            validate_langfuse_environment_value(self.langfuse_environment)
+        if _env_override:
+            validate_langfuse_environment_value(_env_override)
+            self.langfuse_environment: str | None = _env_override
+        else:
+            self.langfuse_environment = self.resolve_deployment_environment()
         self.langfuse_release = os.getenv("LANGFUSE_RELEASE")
         self.langfuse_debug = os.getenv("LANGFUSE_DEBUG")
         self.langfuse_flush_interval = LangFuseLogger._get_langfuse_flush_interval(flush_interval)
@@ -953,6 +966,20 @@ class LangFuseLogger:
             verbose_logger.warning("Failed to apply masking function: %s. Returning original data.", e)
             return data
 
+    @staticmethod
+    def resolve_deployment_environment() -> str | None:
+        """Resolve LANGFUSE_TRACING_ENVIRONMENT: stripped value, "default" plus a warning when invalid, None when unset."""
+        raw: Final = os.getenv("LANGFUSE_TRACING_ENVIRONMENT")
+        if not raw:
+            return None
+        value: Final = raw.strip()
+        try:
+            validate_langfuse_environment_value(value)
+        except ValueError as e:
+            _warn_invalid_deployment_environment(raw, str(e))
+            return "default"
+        return value
+
     @staticmethod
     def _get_langfuse_flush_interval(flush_interval: int) -> int:
         """

litellm/integrations/langfuse/langfuse_handler.py

@@ -1,5 +1,3 @@
-import os
-
 """
 This file contains the LangFuseHandler class
 
@@ -8,6 +6,7 @@ Used to get the LangFuseLogger for a given request
 Handles Key/Team Based Langfuse Logging
 """
 
+import os
 from typing import TYPE_CHECKING, Any, Final
 
 from litellm.litellm_core_utils.litellm_logging import StandardCallbackDynamicParams
@@ -157,7 +156,11 @@ class LangFuseHandler:
         if raw is None:
             return None
         value = str(raw).strip()
-        if not value or value == os.getenv("LANGFUSE_TRACING_ENVIRONMENT"):
+        if (
+            not value
+            or value == os.getenv("LANGFUSE_TRACING_ENVIRONMENT")
+            or value == LangFuseLogger.resolve_deployment_environment()
+        ):
             return None
         return value
 

litellm/integrations/langfuse/langfuse_prompt_management.py

@@ -2,6 +2,7 @@
 Call Hook for LiteLLM Proxy which allows Langfuse prompt management.
 """
 
+import inspect
 import os
 from functools import lru_cache
 from typing import TYPE_CHECKING, Any, Final, Literal, TypeAlias, cast
@@ -109,6 +110,9 @@ def langfuse_client_init(
             cert=os.getenv("SSL_CERTIFICATE", litellm.ssl_certificate),
         )
 
+    if "environment" in inspect.signature(Langfuse.__init__).parameters:
+        parameters["environment"] = LangFuseLogger.resolve_deployment_environment()
+
     client: Final = Langfuse(**parameters)
 
     return client

litellm/integrations/newrelic/newrelic_metrics.py

@@ -0,0 +1,395 @@
+"""
+New Relic Metric API Integration - sends per-team cost/usage metrics to /metric/v1
+
+NR Reference API: https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/introduction-metric-api/
+
+`async_log_success_event` / `async_log_failure_event` queue one record per request;
+at flush the queue is aggregated by (team, model group, model, provider, status)
+into count/summary metrics. `interval.ms` is the real window between flushes,
+computed at flush time.
+
+Team-scoped by construction: the ingest key is injected explicitly and there is
+deliberately no environment-variable fallback, so a team's metrics are never sent
+with the proxy operator's credentials (mirrors ``allow_env_credentials=False`` on
+the Datadog team logger).
+
+Error policy on flush: 4xx drops the batch (a retry would fail identically; 403
+is a permanent credential failure), 5xx/network re-queues capped at
+``max_queue_size`` records with the oldest dropped.
+
+For batching specific details see CustomBatchLogger class
+"""
+
+import asyncio
+import gzip
+import time
+import traceback
+from collections.abc import Mapping
+from math import ceil
+from types import MappingProxyType
+from typing import Final
+
+from httpx import HTTPStatusError, Response
+
+from litellm._logging import verbose_logger
+from litellm.integrations.custom_batch_logger import CustomBatchLogger
+from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
+from litellm.llms.custom_httpx.http_handler import (
+    get_async_httpx_client,
+    httpxSpecialProvider,
+)
+from litellm.types.integrations.newrelic import (
+    NEWRELIC_DEFAULT_REGION,
+    NEWRELIC_METRIC_ATTRIBUTE_MAX_LEN,
+    NEWRELIC_METRIC_COMPLETION_TOKENS,
+    NEWRELIC_METRIC_COST_USD,
+    NEWRELIC_METRIC_ENDPOINT_BY_REGION,
+    NEWRELIC_METRIC_PROMPT_TOKENS,
+    NEWRELIC_METRIC_REQUEST_DURATION_MS,
+    NEWRELIC_METRIC_REQUESTS,
+    NEWRELIC_METRIC_TOTAL_TOKENS,
+    NEWRELIC_METRICS_MAX_BATCH_SIZE,
+    NEWRELIC_METRICS_MAX_DRAIN_PASSES,
+    NEWRELIC_METRICS_MAX_RETRY_QUEUE_SIZE,
+    NewRelicCountMetric,
+    NewRelicMetric,
+    NewRelicMetricCommon,
+    NewRelicMetricEnvelope,
+    NewRelicMetricRecord,
+    NewRelicSummaryMetric,
+    NewRelicSummaryValue,
+)
+from litellm.types.utils import StandardLoggingPayload
+
+# 408 (request timeout) and 429 (rate limit) are transient client errors the
+# Metric API expects a retry on, unlike 400/403 which a retry would only repeat.
+_RETRYABLE_CLIENT_STATUSES: Final = frozenset({408, 429})
+
+
+def resolve_newrelic_metric_endpoint(newrelic_region: str | None) -> str:
+    if not newrelic_region:
+        return NEWRELIC_METRIC_ENDPOINT_BY_REGION[NEWRELIC_DEFAULT_REGION]
+    endpoint: Final = NEWRELIC_METRIC_ENDPOINT_BY_REGION.get(newrelic_region.lower())
+    if endpoint is None:
+        verbose_logger.warning(
+            "New Relic: unknown newrelic_region %r; supported regions: %s. Using the default (US) endpoint.",
+            newrelic_region,
+            ", ".join(sorted(NEWRELIC_METRIC_ENDPOINT_BY_REGION)),
+        )
+        return NEWRELIC_METRIC_ENDPOINT_BY_REGION[NEWRELIC_DEFAULT_REGION]
+    return endpoint
+
+
+def _metric_record_from_payload(standard_logging_object: StandardLoggingPayload) -> NewRelicMetricRecord:
+    metadata: Final = standard_logging_object.get("metadata")
+    team_id: Final = ((metadata.get("user_api_key_team_id") or metadata.get("team_id")) if metadata else None) or ""
+    team_alias: Final = (
+        (metadata.get("user_api_key_team_alias") or metadata.get("team_alias")) if metadata else None
+    ) or ""
+    return NewRelicMetricRecord(
+        team_id=team_id,
+        team_alias=team_alias,
+        model_group=standard_logging_object.get("model_group") or "",
+        model=standard_logging_object.get("model") or "",
+        custom_llm_provider=standard_logging_object.get("custom_llm_provider") or "",
+        status=str(standard_logging_object.get("status") or "success"),
+        response_cost=float(standard_logging_object.get("response_cost") or 0.0),
+        prompt_tokens=int(standard_logging_object.get("prompt_tokens") or 0),
+        completion_tokens=int(standard_logging_object.get("completion_tokens") or 0),
+        total_tokens=int(standard_logging_object.get("total_tokens") or 0),
+        duration_ms=float(standard_logging_object.get("response_time") or 0.0) * 1000.0,
+    )
+
+
+def _bucket_metrics(bucket_records: tuple[NewRelicMetricRecord, ...]) -> tuple[NewRelicMetric, ...]:
+    first: Final = bucket_records[0]
+    attributes: Final[Mapping[str, str]] = {  # mutable-ok: JSON leaf; safe_dumps stringifies MappingProxyType
+        key: value[:NEWRELIC_METRIC_ATTRIBUTE_MAX_LEN]
+        for key, value in (
+            ("team_id", first.team_id),
+            ("team_alias", first.team_alias),
+            ("model_group", first.model_group),
+            ("model", first.model),
+            ("custom_llm_provider", first.custom_llm_provider),
+            ("status", first.status),
+        )
+        if value
+    }
+    durations: Final = tuple(record.duration_ms for record in bucket_records)
+    counts: Final[tuple[tuple[str, float], ...]] = (
+        (NEWRELIC_METRIC_REQUESTS, float(len(bucket_records))),
+        (NEWRELIC_METRIC_COST_USD, sum(record.response_cost for record in bucket_records)),
+        (NEWRELIC_METRIC_PROMPT_TOKENS, float(sum(record.prompt_tokens for record in bucket_records))),
+        (NEWRELIC_METRIC_COMPLETION_TOKENS, float(sum(record.completion_tokens for record in bucket_records))),
+        (NEWRELIC_METRIC_TOTAL_TOKENS, float(sum(record.total_tokens for record in bucket_records))),
+    )
+    count_metrics: Final[tuple[NewRelicMetric, ...]] = tuple(
+        NewRelicCountMetric(name=name, type="count", value=value, attributes=attributes) for name, value in counts
+    )
+    summary_metric: Final = NewRelicSummaryMetric(
+        name=NEWRELIC_METRIC_REQUEST_DURATION_MS,
+        type="summary",
+        value=NewRelicSummaryValue(
+            count=len(durations),
+            sum=sum(durations),
+            min=min(durations),
+            max=max(durations),
+        ),
+        attributes=attributes,
+    )
+    return (*count_metrics, summary_metric)
+
+
+def build_metric_payload(
+    records: tuple[NewRelicMetricRecord, ...],
+    *,
+    window_start: float,
+    now: float,
+) -> tuple[NewRelicMetricEnvelope, ...]:
+    """Aggregates records into one Metric API envelope for the flush window."""
+    interval_ms: Final = max(1, int((now - window_start) * 1000))
+    bucket_keys: Final = tuple(dict.fromkeys(record.bucket_key for record in records))
+    metrics: Final = tuple(
+        metric
+        for key in bucket_keys
+        for metric in _bucket_metrics(tuple(record for record in records if record.bucket_key == key))
+    )
+    common: Final[NewRelicMetricCommon] = {
+        "timestamp": int(window_start * 1000),
+        "interval.ms": interval_ms,
+    }
+    return (NewRelicMetricEnvelope(common=common, metrics=metrics),)
+
+
+class NewRelicMetricsLogger(CustomBatchLogger):
+    def __init__(
+        self,
+        newrelic_api_key: str,
+        newrelic_region: str | None = None,
+    ) -> None:
+        if not newrelic_api_key:
+            raise ValueError(
+                "newrelic_api_key is required for NewRelicMetricsLogger; "
+                "team-scoped metrics never fall back to environment credentials"
+            )
+        self.newrelic_api_key: Final = newrelic_api_key
+        self.metric_api_url: Final = resolve_newrelic_metric_endpoint(newrelic_region)
+        self.async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.LoggingCallback)
+        self._stopped: bool = False
+        self._drain_lock = asyncio.Lock()
+        asyncio.create_task(self.periodic_flush())
+        self.flush_lock = asyncio.Lock()
+        super().__init__(
+            flush_lock=self.flush_lock,
+            batch_size=NEWRELIC_METRICS_MAX_BATCH_SIZE,
+            max_queue_size=NEWRELIC_METRICS_MAX_RETRY_QUEUE_SIZE,
+        )
+
+    def stop(self) -> None:
+        """Ends the periodic flush loop; called on DynamicLoggingCache eviction.
+
+        Schedules one final drain of anything still queued, so eviction never
+        silently discards records. Guarded so it can never raise into the
+        cache's eviction path.
+        """
+        self._stopped = True
+        try:
+            asyncio.get_running_loop().create_task(self._final_drain())
+        except Exception:  # noqa: BLE001  # no running loop / shutdown; the periodic loop's final drain still runs
+            verbose_logger.debug("New Relic Metrics: could not schedule final drain on stop()", exc_info=True)
+
+    async def _drain_with_retry(self) -> None:
+        """Deliver everything queued on a stopped logger, or drop it with a log.
+
+        A stopped logger has no periodic loop left, so every post-stop path
+        funnels through here. ``_drain_lock`` serializes drains: a callback that
+        appends and starts its own drain queues behind the running one instead
+        of racing it. Each pass attempts the whole current queue in
+        ``batch_size`` chunks, unlike the periodic path it does not stop at the
+        first failing chunk, so a persistently failing head never starves the
+        tail. Only after ``_MAX_DRAIN_PASSES`` against a permanently failing
+        destination is the remainder dropped, and then only the records that were
+        queued when this drain began, so every dropped record got the full retry
+        budget: a record a callback appended mid-drain is not in that snapshot,
+        so it is left for its own serialized drain rather than dropped after
+        fewer attempts, and is never stranded.
+        """
+        async with self._drain_lock:
+            attempted: Final = tuple(self.log_queue)
+            for _pass in range(NEWRELIC_METRICS_MAX_DRAIN_PASSES):
+                await self._drain_flush_once()
+                if not self.log_queue:
+                    return
+                if _pass < NEWRELIC_METRICS_MAX_DRAIN_PASSES - 1:
+                    await asyncio.sleep(2**_pass)
+            async with self.flush_lock:
+                tried_ids: Final = frozenset(id(record) for record in attempted)
+                survivors: Final = tuple(record for record in self.log_queue if id(record) not in tried_ids)
+                dropped: Final = len(self.log_queue) - len(survivors)
+                if dropped:
+                    verbose_logger.warning(
+                        "New Relic Metrics: dropping %s records after %s drain passes",
+                        dropped,
+                        NEWRELIC_METRICS_MAX_DRAIN_PASSES,
+                    )
+                self.log_queue[:] = list(survivors)  # mutable-ok: leave late arrivals for the next serialized drain
+
+    async def _drain_flush_once(self) -> None:
+        """Attempt every queued record once, in ``batch_size`` chunks, without
+        stopping at the first failing chunk so a persistently failing head does
+        not starve the tail (the periodic ``flush_queue`` deliberately stops
+        instead). Takes the queue under ``flush_lock`` and re-queues only the
+        chunks a 5xx/network error left undelivered, so records a concurrent
+        request appends during the sends survive for the next pass."""
+        async with self.flush_lock:
+            pending: Final = tuple(self.log_queue)
+            window_start: Final = self.last_flush_time
+            self.last_flush_time = time.time()
+            del self.log_queue[:]
+        if not pending:
+            return
+        chunks: Final = tuple(
+            pending[start : start + self.batch_size] for start in range(0, len(pending), self.batch_size)
+        )
+        delivered: Final = tuple([await self._classify_and_send(chunk, window_start) for chunk in chunks])
+        failed: Final = tuple(record for chunk, ok in zip(chunks, delivered) for record in (() if ok else chunk))
+        if failed:
+            self._requeue(failed)
+
+    async def _final_drain(self) -> None:
+        await self._drain_with_retry()
+
+    async def periodic_flush(self) -> None:
+        while not self._stopped:
+            await asyncio.sleep(self.flush_interval)
+            if self._stopped:
+                break
+            await self.flush_queue()
+        await self._final_drain()
+
+    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time) -> None:
+        try:
+            await self._log_async_event(standard_logging_object=kwargs.get("standard_logging_object", None))
+        except Exception as e:  # noqa: BLE001  # logging must never break the request path
+            verbose_logger.exception("New Relic Metrics Layer Error - %s\n%s", e, traceback.format_exc())
+
+    async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time) -> None:
+        try:
+            await self._log_async_event(standard_logging_object=kwargs.get("standard_logging_object", None))
+        except Exception as e:  # noqa: BLE001  # logging must never break the request path
+            verbose_logger.exception("New Relic Metrics Layer Error - %s\n%s", e, traceback.format_exc())
+
+    async def _log_async_event(self, standard_logging_object: StandardLoggingPayload | None) -> None:
+        if standard_logging_object is None:
+            raise ValueError("standard_logging_object not found in kwargs")
+        self.log_queue.append(_metric_record_from_payload(standard_logging_object))
+        if self._stopped:
+            # A stopped logger has no periodic loop left; an in-flight callback
+            # that appends after the eviction drain delivers its own record.
+            await self._drain_with_retry()
+            return
+        if len(self.log_queue) >= self.batch_size:
+            await self.flush_queue()
+
+    async def flush_queue(self) -> None:
+        async with self.flush_lock:
+            window_start: Final = self.last_flush_time
+            self.last_flush_time = time.time()
+            queued: Final = len(self.log_queue)
+            if not queued:
+                return
+            verbose_logger.debug("New Relic Metrics: Flushing %s queued records", queued)
+            # Bounded by what is queued now: records appended mid-flush belong to
+            # the next window, and looping until empty would never end under load.
+            for _chunk in range(ceil(queued / self.batch_size)):
+                if not await self.async_send_batch(window_start=window_start):
+                    return
+
+    async def async_send_batch(self, window_start: float | None = None) -> bool:
+        """Sends the oldest ``batch_size`` records only, so a queue grown past that
+        by re-queues cannot breach the Metric API data point cap in one request.
+        Returns False once a chunk fails and is re-queued, so the caller stops."""
+        if not self.log_queue:
+            return False
+
+        batch_to_send: Final[tuple[NewRelicMetricRecord, ...]] = tuple(self.log_queue[: self.batch_size])
+        del self.log_queue[: len(batch_to_send)]
+
+        delivered: Final = await self._classify_and_send(
+            batch_to_send, window_start if window_start is not None else self.last_flush_time
+        )
+        if not delivered:
+            self._requeue(batch_to_send)
+        return delivered
+
+    async def _classify_and_send(self, batch: tuple[NewRelicMetricRecord, ...], window_start: float) -> bool:
+        """Send one chunk and classify the outcome, never touching the queue.
+        Returns True when the batch is done with (delivered on any 2xx, or a 4xx
+        a retry would only repeat, 403 being a permanent bad-key rejection), and
+        False when a 5xx or network error means the caller should re-queue it.
+
+        ``AsyncHTTPHandler.post`` raises ``HTTPStatusError`` on any non-2xx, so a
+        4xx never returns a response here; the status is read off the raised
+        error to keep the client-error path (drop) distinct from 5xx (retry)."""
+        payload: Final = build_metric_payload(records=batch, window_start=window_start, now=time.time())
+        try:
+            status = (
+                await self.async_send_compressed_data(payload)
+            ).status_code  # rebind-ok: reassigned from the raised HTTPStatusError below
+        except HTTPStatusError as e:
+            status = e.response.status_code
+        except Exception as e:  # noqa: BLE001  # transport/network failure re-queues the batch
+            verbose_logger.warning(
+                "New Relic Metrics: network error sending %s records, will retry - %s",
+                len(batch),
+                e,
+            )
+            return False
+
+        if 200 <= status < 300:
+            return True
+
+        if 400 <= status < 500 and status not in _RETRYABLE_CLIENT_STATUSES:
+            verbose_logger.warning(
+                "New Relic Metrics: %s from Metric API%s, dropping %s records.",
+                status,
+                " (permanent credential failure: invalid or revoked team ingest key)" if status == 403 else "",
+                len(batch),
+            )
+            return True
+
+        verbose_logger.warning(
+            "New Relic Metrics: %s from Metric API, will retry %s records",
+            status,
+            len(batch),
+        )
+        return False
+
+    def _requeue(self, batch: tuple[NewRelicMetricRecord, ...]) -> None:
+        """Prepends ``batch`` in place (never by assignment: records appended by
+        concurrent requests during the flush await must survive), keeping
+        chronological order so the cap drops the oldest records first."""
+        self.log_queue[:0] = batch
+        overflow: Final = len(self.log_queue) - self.max_queue_size
+        if overflow > 0:
+            del self.log_queue[:overflow]
+            verbose_logger.warning(
+                "New Relic Metrics: retry queue exceeded max_queue_size=%s; dropped %s oldest records.",
+                self.max_queue_size,
+                overflow,
+            )
+
+    async def async_send_compressed_data(self, payload: tuple[NewRelicMetricEnvelope, ...]) -> Response:
+        compressed_data: Final = gzip.compress(safe_dumps(payload).encode("utf-8"))
+        headers: Final[Mapping[str, str]] = MappingProxyType(
+            {
+                "Content-Type": "application/json",
+                "Content-Encoding": "gzip",
+                "Api-Key": self.newrelic_api_key,
+            }
+        )
+        return await self.async_client.post(
+            url=self.metric_api_url,
+            data=compressed_data,
+            headers=headers,
+        )

litellm/integrations/newrelic/newrelic_team_handler.py

@@ -0,0 +1,90 @@
+"""
+New Relic Team Handler
+
+Used to get the NewRelicMetricsLogger for a given request.
+Handles Key/Team Based New Relic metrics, following the same pattern as DataDogHandler.
+"""
+
+from typing import TYPE_CHECKING, Final
+
+from typing_extensions import ReadOnly, TypedDict
+
+from litellm._logging import verbose_logger
+from litellm.litellm_core_utils.litellm_logging import StandardCallbackDynamicParams
+
+from .newrelic_metrics import NewRelicMetricsLogger
+
+if TYPE_CHECKING:
+    from litellm.litellm_core_utils.litellm_logging import DynamicLoggingCache
+
+
+class NewRelicLoggingConfig(TypedDict):
+    newrelic_api_key: ReadOnly[str | None]
+    newrelic_region: ReadOnly[str | None]
+
+
+class NewRelicHandler:
+    @staticmethod
+    def get_newrelic_logger_for_request(
+        standard_callback_dynamic_params: StandardCallbackDynamicParams,
+        in_memory_dynamic_logger_cache: "DynamicLoggingCache",
+    ) -> NewRelicMetricsLogger:
+        """
+        Get a team-scoped NewRelicMetricsLogger for a given request.
+
+        Resolves and caches per-team NewRelicMetricsLogger instances using
+        DynamicLoggingCache, keyed by the team's New Relic credentials. Each unique
+        set of credentials gets its own logger instance with its own batch/flush loop.
+
+        Note: This handler is only called when a team-scoped newrelic_api_key is
+        present. The trace logger for the ``newrelic`` callback (OTel v2 / legacy
+        agent) is managed separately by _init_custom_logger_compatible_class via
+        _in_memory_loggers.
+        """
+        _credentials: Final = NewRelicHandler.get_dynamic_newrelic_logging_config(
+            standard_callback_dynamic_params=standard_callback_dynamic_params,
+        )
+
+        temp_newrelic_logger = in_memory_dynamic_logger_cache.get_cache(
+            credentials=_credentials, service_name="newrelic"
+        )
+
+        if temp_newrelic_logger is None:
+            temp_newrelic_logger = NewRelicHandler._create_newrelic_logger_from_credentials(
+                credentials=_credentials,
+                in_memory_dynamic_logger_cache=in_memory_dynamic_logger_cache,
+            )
+
+        return temp_newrelic_logger
+
+    @staticmethod
+    def _create_newrelic_logger_from_credentials(
+        credentials: NewRelicLoggingConfig,
+        in_memory_dynamic_logger_cache: "DynamicLoggingCache",
+    ) -> NewRelicMetricsLogger:
+        newrelic_logger: Final = NewRelicMetricsLogger(
+            newrelic_api_key=credentials.get("newrelic_api_key") or "",
+            newrelic_region=credentials.get("newrelic_region"),
+        )
+        in_memory_dynamic_logger_cache.set_cache(
+            credentials=credentials,
+            service_name="newrelic",
+            logging_obj=newrelic_logger,
+        )
+        verbose_logger.debug("New Relic: Created and cached new NewRelicMetricsLogger for team-scoped credentials")
+        return newrelic_logger
+
+    @staticmethod
+    def get_dynamic_newrelic_logging_config(
+        standard_callback_dynamic_params: StandardCallbackDynamicParams,
+    ) -> NewRelicLoggingConfig:
+        return NewRelicLoggingConfig(
+            newrelic_api_key=standard_callback_dynamic_params.get("newrelic_api_key"),
+            newrelic_region=standard_callback_dynamic_params.get("newrelic_region"),
+        )
+
+    @staticmethod
+    def _dynamic_newrelic_credentials_are_passed(
+        standard_callback_dynamic_params: StandardCallbackDynamicParams,
+    ) -> bool:
+        return standard_callback_dynamic_params.get("newrelic_api_key") is not None

litellm/integrations/opentelemetry.py

@@ -22,6 +22,7 @@ from litellm.integrations.opentelemetry_utils.gen_ai_semconv import (
 )
 from litellm.integrations.otel.model.db_endpoint import db_span_attributes
 from litellm.integrations.otel.model.semconv import Metric
+from litellm.litellm_core_utils.internal_call_metadata import is_unbilled_non_inference_call_from_params
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
 from litellm.litellm_core_utils.secret_redaction import redact_string
 from litellm.litellm_core_utils.service_tier_utils import (
@@ -1643,7 +1644,12 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
 
         if self._operation_duration_histogram:
             self._operation_duration_histogram.record(duration_s, attributes=common_attrs)
-            if response_obj and (usage := response_obj.get("usage")) and self._token_usage_histogram:
+            if (
+                self._token_usage_histogram
+                and response_obj
+                and not is_unbilled_non_inference_call_from_params(kwargs.get("call_type"), params, response_obj)
+                and (usage := response_obj.get("usage"))
+            ):
                 in_attrs: Final = {**common_attrs, TOKEN_TYPE_ATTRIBUTE: "input"}
                 out_attrs: Final = {**common_attrs, TOKEN_TYPE_ATTRIBUTE: "output"}
                 self._token_usage_histogram.record(usage.get("prompt_tokens", 0), attributes=in_attrs)
@@ -1719,6 +1725,11 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
         if not self._time_per_output_token_histogram:
             return
 
+        if is_unbilled_non_inference_call_from_params(
+            kwargs.get("call_type"), kwargs.get("litellm_params"), response_obj
+        ):
+            return
+
         # Get completion tokens from response_obj
         completion_tokens = None
         if response_obj and (usage := response_obj.get("usage")):
@@ -2488,7 +2499,14 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
 
             self._set_service_tier_attributes(span=span, standard_logging_payload=standard_logging_payload)
 
-            usage: Final = response_obj and response_obj.get("usage")
+            usage: Final = (
+                response_obj.get("usage")
+                if response_obj
+                and not is_unbilled_non_inference_call_from_params(
+                    kwargs.get("call_type"), litellm_params, response_obj
+                )
+                else None
+            )
             if usage:
                 self.safe_set_attribute(
                     span=span,

litellm/integrations/otel/emitter.py

@@ -146,7 +146,7 @@ class SpanEmitter:
         For callers that own and manage their own span lifecycle. ``tracer``
         overrides the bound tracer for this span only, used for per-request
         multi-tenant credential routing. ``links`` records related-but-not-parent
-        spans (e.g. the transport span of an MCP message, per MCP semconv).
+        spans (e.g. the trace context an MCP client propagated in ``params._meta``).
         """
         return (tracer or self._tracer).start_span(
             name,
@@ -196,8 +196,8 @@ class SpanEmitter:
 
         Return the span, or ``None`` if it was deduplicated away. ``tracer``
         overrides the bound tracer for this span, used for per-request routing.
-        ``links`` records related-but-not-parent spans (the transport span of an
-        MCP message).
+        ``links`` records related-but-not-parent spans (e.g. the trace context an
+        MCP client propagated in ``params._meta``).
         """
         # LLM-call and MCP tool-call spans carry a dedup key (their request's
         # call id), so a sync+async double-firing coalesces. ``isinstance`` narrows

litellm/integrations/otel/logger.py

@@ -390,10 +390,10 @@ class OpenTelemetryV2(CustomLogger):
 
         MCP tool calls reach the success/failure callbacks like any other request
         (with ``call_type`` ``call_mcp_tool``), but they are not LLM calls and have
-        no ``pre_call`` carrier — so they get their own CLIENT span here. Per the MCP
-        semconv it parents to the trace context the client propagated in
-        ``params._meta`` (or starts a new root) and links the transport span, rather
-        than nesting under the HTTP/session span. Returns whether it handled the
+        no ``pre_call`` carrier — so they get their own CLIENT span here. It nests
+        under the transport span of the request carrying this message, and trace
+        context the client propagated in ``params._meta`` is recorded as a span
+        link (see ``resolve_mcp_span_context``). Returns whether it handled the
         event, so the caller skips the LLM-call path. The whole span is emitted at
         once (there is no boundary to open it at), deduped on the call id.
         """
@@ -436,9 +436,9 @@ class OpenTelemetryV2(CustomLogger):
 
         Like a tool call, listing reaches the success/failure callbacks (here with
         ``call_type`` ``list_mcp_tools``) with no ``pre_call`` carrier, so it gets its
-        own CLIENT span. Per the MCP semconv it parents to the ``params._meta`` trace
-        context (or starts a new root) and links the transport span, rather than
-        nesting under the HTTP/session span. Returns whether it handled the event so
+        own CLIENT span, nested under the transport span of the request carrying
+        this message with any ``params._meta`` trace context recorded as a span
+        link (see ``resolve_mcp_span_context``). Returns whether it handled the event so
         the caller skips the LLM-call path.
         """
         raw_payload: Final = kwargs.get("standard_logging_object")

litellm/integrations/otel/model/semconv.py

@@ -32,6 +32,7 @@ class GenAIOperation(str, Enum):
     EXECUTE_TOOL = "execute_tool"  # MCP tool-call spans
     LITELLM_VECTOR_STORE_MANAGEMENT = "litellm.vector_store_management"
     LITELLM_VECTOR_STORE_FILE_MANAGEMENT = "litellm.vector_store_file_management"
+    LITELLM_RESPONSES_MANAGEMENT = "litellm.responses_management"
     LITELLM_MODERATION = "litellm.moderation"
 
 
@@ -383,6 +384,14 @@ _OPERATION_BY_CALL_TYPE: Final[dict[str, GenAIOperation]] = {
     "aembedding": GenAIOperation.EMBEDDINGS,
     "responses": GenAIOperation.CHAT,
     "aresponses": GenAIOperation.CHAT,
+    "get_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "aget_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "delete_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "adelete_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "cancel_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "acancel_responses": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "list_input_items": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
+    "alist_input_items": GenAIOperation.LITELLM_RESPONSES_MANAGEMENT,
     "image_generation": GenAIOperation.GENERATE_CONTENT,
     "aimage_generation": GenAIOperation.GENERATE_CONTENT,
     "moderation": GenAIOperation.LITELLM_MODERATION,

litellm/integrations/otel/model/spans.py

@@ -10,6 +10,8 @@ Canonical hierarchy::
     │   └── DB_CALL (CLIENT)          #   its key/user/team lookups nest here
     ├── GUARDRAIL  (INTERNAL)         # request-lifecycle hook, sibling of LLM_CALL
     ├── LLM_CALL   (CLIENT)
+    ├── MCP_TOOL_CALL  (CLIENT)       # nests under the POST carrying the message
+    ├── MCP_LIST_TOOLS (CLIENT)       #   (client-propagated context is a span link)
     └── DB_CALL    (CLIENT)           # e.g. the spend-log write
 
 Guardrails parent to PROXY_REQUEST, not LLM_CALL: pre/during/post-call guardrail
@@ -18,14 +20,14 @@ before the LLM call even starts), so a guardrail is a sibling of the LLM call,
 not a child of it. The emitter parents every span to the ambient OTel context
 (the active server span), which matches this.
 
-MCP spans (``MCP_TOOL_CALL``, ``MCP_LIST_TOOLS``) have two shapes, chosen at emit
-time by :func:`resolve_mcp_span_context`. When the client propagates trace context
-in ``params._meta`` MCP and the HTTP transport are independent contexts per the
-OTel GenAI MCP semconv, so the span parents to that propagated context and records
-the ``PROXY_REQUEST`` transport span as a span *link*, never a parent — the shape
-this registry's ``parent=None, links=PROXY_REQUEST`` entry encodes. When nothing is
-propagated (the common case) the span nests under the transport span of the request
-carrying that message, so the tool call stays in one trace.
+MCP spans (``MCP_TOOL_CALL``, ``MCP_LIST_TOOLS``) are parented at emit time by
+:func:`resolve_mcp_span_context`: they nest under the ``PROXY_REQUEST`` transport
+span of the request carrying that message, so the tool call stays in one trace.
+Trace context the client propagated in ``params._meta`` (SEP-414) is recorded as
+a span *link*, never the parent — a remote parent would root the span in a trace
+whose root never reaches the gateway's tracing backend. Links always target that
+remote client context, never a registry role, so ``SpanSpec`` declares no link
+field; the concrete transport parent is resolved per message at emit time.
 
 Not every service call becomes a span — :func:`span_role_for_service` decides:
 
@@ -85,25 +87,19 @@ class SpanSpec:
     role: SpanRole
     kind: LiteLLMSpanKind
     parent: SpanRole | None
-    links: SpanRole | None = None
 
 
 SPAN_REGISTRY: Final[dict[SpanRole, SpanSpec]] = {
     SpanRole.PROXY_REQUEST: SpanSpec(SpanRole.PROXY_REQUEST, LiteLLMSpanKind.SERVER, parent=None),
     SpanRole.LLM_CALL: SpanSpec(SpanRole.LLM_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
     # The proxy is an MCP client to the upstream server, so MCP spans are CLIENT
-    # spans. With trace context propagated in ``params._meta``, MCP and the HTTP
-    # transport are independent contexts (OTel GenAI MCP semconv): the span parents
-    # to the propagated context and records the PROXY_REQUEST transport span as a
-    # span *link*, never a parent — the shape ``parent=None, links=PROXY_REQUEST``
-    # encodes. With nothing propagated, ``resolve_mcp_span_context`` nests the span
-    # under that message's transport span instead, keeping the call in one trace.
-    SpanRole.MCP_TOOL_CALL: SpanSpec(
-        SpanRole.MCP_TOOL_CALL, LiteLLMSpanKind.CLIENT, parent=None, links=SpanRole.PROXY_REQUEST
-    ),
-    SpanRole.MCP_LIST_TOOLS: SpanSpec(
-        SpanRole.MCP_LIST_TOOLS, LiteLLMSpanKind.CLIENT, parent=None, links=SpanRole.PROXY_REQUEST
-    ),
+    # spans. ``resolve_mcp_span_context`` nests them under the PROXY_REQUEST
+    # transport span of the request carrying that message (resolved per message at
+    # emit time), keeping the call in one trace. Trace context the client
+    # propagated in ``params._meta`` becomes a span *link* to that remote context,
+    # which is not a registry role, so ``SpanSpec`` has no link field.
+    SpanRole.MCP_TOOL_CALL: SpanSpec(SpanRole.MCP_TOOL_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
+    SpanRole.MCP_LIST_TOOLS: SpanSpec(SpanRole.MCP_LIST_TOOLS, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
     SpanRole.GUARDRAIL: SpanSpec(SpanRole.GUARDRAIL, LiteLLMSpanKind.INTERNAL, parent=SpanRole.PROXY_REQUEST),
     SpanRole.DB_CALL: SpanSpec(SpanRole.DB_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
     SpanRole.SERVICE: SpanSpec(SpanRole.SERVICE, LiteLLMSpanKind.INTERNAL, parent=SpanRole.PROXY_REQUEST),
@@ -209,8 +205,8 @@ def service_span_name(data: "ServiceSpanData") -> str:
 
 
 def root_roles() -> list[SpanRole]:
-    """Roles with no in-process parent. They start a new trace unless they adopt a
-    remote parent (e.g. an MCP span joining the client's propagated context)."""
+    """Roles with no in-process parent, i.e. they start a new trace (only the
+    instrumentor-owned ``PROXY_REQUEST`` server span today)."""
     return [role for role, spec in SPAN_REGISTRY.items() if spec.parent is None]
 
 
@@ -227,8 +223,6 @@ def validate_registry(
             raise ValueError(f"SPAN_REGISTRY[{role}] has mismatched role {spec.role}")
         if spec.parent is not None and spec.parent not in reg:
             raise ValueError(f"span role {role} declares unknown parent {spec.parent}")
-        if spec.links is not None and spec.links not in reg:
-            raise ValueError(f"span role {role} declares unknown link target {spec.links}")
     missing: Final = [role for role in SpanRole if role not in reg]
     if missing:
         raise ValueError(f"SPAN_REGISTRY is missing roles: {missing}")

litellm/integrations/otel/plumbing/context.py

@@ -57,8 +57,8 @@ def request_root_span() -> "Span | None":
 
 # The W3C trace-context carrier (``traceparent``/``tracestate``/``baggage``) the
 # MCP client propagated in the current request's ``params._meta``. The MCP gateway
-# sets it per message so the MCP span can parent to the client's span rather than
-# to the transport. A ``ContextVar`` because, like the root-span anchor, it must
+# sets it per message so the MCP span can record the client's span as a span
+# link. A ``ContextVar`` because, like the root-span anchor, it must
 # ride the request task and be readable by the inline success-logging callback.
 _mcp_message_trace_carrier: Final["ContextVar[Mapping[str, str] | None]"] = ContextVar(
     "litellm_otel_mcp_message_trace_carrier", default=None
@@ -148,10 +148,10 @@ def _mcp_transport_span_context() -> "SpanContext | None":
 
     Prefers the transport the gateway published for this specific message; falls
     back to the ambient request anchor for paths that emit an MCP span on the
-    request task itself (the REST MCP endpoints, the SDK). Parenting and linking
-    only need the immutable context, and unlike ``mcp_message_transport_span`` they
-    stay correct against a transport that has already finished, so this does not
-    require the span to still be recording.
+    request task itself (the REST MCP endpoints). Parenting needs only the
+    immutable context, and unlike ``mcp_message_transport_span`` it stays correct
+    against a transport that has already finished, so this does not require the
+    span to still be recording.
     """
     published: Final = _mcp_message_transport_span.get()
     if published is not None:
@@ -222,25 +222,31 @@ def resolve_mcp_span_context(
 ) -> "tuple[Context, tuple[Link, ...]]":
     """Parent context + links for an MCP message span.
 
+    The span always nests under the transport span of the request carrying this
+    message, so a tool call and the ``POST`` that carried it stay in one trace.
+    The transport comes from :func:`_mcp_transport_span_context`, which is the
+    *current message's* POST rather than whatever request happened to open the
+    session, so a long-lived session does not glue every message under its first
+    request.
+
     When the client propagates W3C trace context in the request's ``params._meta``
-    (SEP-414), MCP and the underlying transport are independent lifecycles — one
-    streamable-HTTP session multiplexes many messages, and the client's own span is
-    the truthful parent. So, per the OTel GenAI MCP semconv:
+    (SEP-414), that remote context is recorded as a span *link*, never the parent.
+    The OTel GenAI MCP semconv prefers the inverse (remote parent, transport link),
+    but the gateway's tracing backend only ever receives the gateway's half of such
+    a trace: parenting into the client's trace id roots the span in a trace whose
+    root span never reaches the backend, so the span is unreachable from the trace
+    view and the transport transaction shows a dangling link (observed with
+    clients that propagate synthetic trace ids). Anchoring to the gateway's own
+    request and linking the client's context keeps every trace renderable while
+    preserving the client-side correlation.
 
-    * parent to the trace context the client propagated (a *remote* parent), and
-    * record the transport span as a *link*, never the parent.
-
-    Almost no client implements SEP-414 yet, so in practice nothing is propagated.
-    Rooting the span there splits a single tool call into two disconnected traces
-    joined only by a link, which is how it surfaces in APM: the ``POST`` transaction
-    and the ``tools/call`` span share no trace. With no remote parent to honor,
-    parent to the transport span of the request carrying this message instead, so
-    the call stays in one trace; no link is added since the transport is now the
-    real parent. The transport comes from :func:`_mcp_transport_span_context`, which
-    is the *current message's* POST rather than whatever request happened to open
-    the session, so a long-lived session does not glue every message under its
-    first request. With neither a remote parent nor a transport the returned context
-    carries no span and the span legitimately starts its own root trace.
+    With no transport at all the span starts its own root trace, still carrying
+    the link — the client context is only ever a link, so this event keeps one
+    shape everywhere. Both returned contexts are built on an explicitly empty
+    base, so ambient (stale session) state can never leak in, and the span
+    inherits the transport's sampling decision exactly like every other
+    request-level span — a client's sampled flag neither forces nor suppresses
+    recording.
 
     Only trace context (``traceparent``/``tracestate``) is extracted, never the
     client's W3C Baggage: ``params._meta`` is caller-controlled, and the otel
@@ -251,13 +257,12 @@ def resolve_mcp_span_context(
     never fall through to the ambient (stale session) span.
     """
     source: Final = carrier if carrier is not None else _mcp_message_trace_carrier.get()
-    parent: Final = _PROPAGATOR.extract(dict(source or {}), context=Context())
+    propagated: Final = get_current_span(_PROPAGATOR.extract(dict(source or {}), context=Context()))
+    links: Final = (Link(propagated.get_span_context()),) if is_recordable_span(propagated) else ()
     transport: Final = _mcp_transport_span_context()
-    if is_recordable_span(get_current_span(parent)):
-        return parent, (Link(transport),) if transport is not None else ()
-    if transport is not None:
-        return context_from_span(NonRecordingSpan(transport)), ()
-    return parent, ()
+    if transport is None:
+        return Context(), links
+    return context_from_span(NonRecordingSpan(transport), context=Context()), links
 
 
 def is_recordable_span(obj: object) -> bool:

litellm/integrations/otel/plumbing/metrics.py

@@ -32,6 +32,7 @@ from litellm.integrations.otel.model.semconv import (
     resolve_provider,
 )
 from litellm.integrations.otel.model.utils import to_seconds
+from litellm.litellm_core_utils.internal_call_metadata import is_unbilled_non_inference_call_from_params
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
 
 
@@ -198,16 +199,21 @@ class GenAIMetricRecorder:
     ) -> None:
         common_attrs: Final = self._filter_attributes(self._bounded_attributes(kwargs))
         duration_s: Final = (end_time - start_time).total_seconds()
+        usage_is_replayed: Final = is_unbilled_non_inference_call_from_params(
+            kwargs.get("call_type"), kwargs.get("litellm_params"), response_obj
+        )
 
         self._metrics.operation_duration.record(duration_s, attributes=common_attrs)
-        self._record_token_usage(response_obj, common_attrs)
+        if not usage_is_replayed:
+            self._record_token_usage(response_obj, common_attrs)
 
         cost: Final = kwargs.get("response_cost")
         if cost:
             self._metrics.token_cost.record(cost, attributes=common_attrs)
 
         self._record_time_to_first_token(kwargs, common_attrs)
-        self._record_time_per_output_token(kwargs, response_obj, end_time, duration_s, common_attrs)
+        if not usage_is_replayed:
+            self._record_time_per_output_token(kwargs, response_obj, end_time, duration_s, common_attrs)
         self._record_response_duration(kwargs, end_time, common_attrs)
 
     def record_failure(

litellm/integrations/otel/plumbing/routing.py

@@ -2,12 +2,13 @@
 
 When a request carries team/key vendor credentials in
 ``standard_callback_dynamic_params``, or the key/team config resolved at auth
-names a destination project, its spans must export through a
-``TracerProvider`` whose OTLP headers carry those credentials / that project.
-``TenantTracerCache`` builds and caches one provider per distinct
-(credentials, project) pair, and otherwise hands back the logger's default
-tracer. This lets a single logger fan requests out to many tenants without
-needing a logger per tenant.
+names a destination project or a service name, its spans must export through a
+``TracerProvider`` whose OTLP headers carry those credentials / that project,
+or whose Resource carries that ``service.name``. ``TenantTracerCache`` builds
+and caches one provider per distinct (credentials, project, service name)
+tuple, and otherwise hands back the logger's default tracer. This lets a
+single logger fan requests out to many tenants without needing a logger per
+tenant.
 """
 
 import threading
@@ -22,6 +23,7 @@ from opentelemetry.sdk.trace import TracerProvider
 from opentelemetry.trace import Tracer
 
 from litellm._logging import verbose_logger
+from litellm.constants import OTEL_SERVICE_NAME_METADATA_KEYS
 from litellm.integrations.otel.model.config import ExporterSpec, OpenTelemetryV2Config
 from litellm.integrations.otel.plumbing.providers import (
     build_tracer_provider,
@@ -65,8 +67,30 @@ _MAX_RETIRED_PROVIDERS: Final = 64
 
 _HeaderItems: TypeAlias = tuple[tuple[str, str], ...]
 
+_RouteKey: TypeAlias = tuple[_HeaderItems, _HeaderItems, str | None, str | None]
+
 _NO_HEADERS: Final[Mapping[str, str]] = MappingProxyType({})
 
+#: Key/team config fields naming the Resource ``service.name``, highest
+#: precedence first. Read only from ``user_api_key_auth_metadata`` (the config
+#: the proxy resolved at auth), never from client-supplied request metadata:
+#: the service name picks the dataset/service traces land in (Honeycomb routes
+#: datasets by it), so a caller must not be able to choose one.
+_SERVICE_NAME_KEYS: Final = OTEL_SERVICE_NAME_METADATA_KEYS
+
+
+def tenant_service_name(auth_metadata: Mapping[str, str] | None) -> str | None:
+    """The per-request ``service.name`` override for this key/team, if any.
+
+    ``None`` keeps the env-configured default (``OTEL_SERVICE_NAME``).
+    """
+    if not auth_metadata:
+        return None
+    return next(
+        (stripped for key in _SERVICE_NAME_KEYS if (stripped := (auth_metadata.get(key) or "").strip())),
+        None,
+    )
+
 
 def _shutdown_provider(provider: TracerProvider) -> None:
     """Flush + stop an evicted provider's processors (reclaims their threads).
@@ -116,7 +140,7 @@ class TenantRoute:
 
 
 class TenantTracerCache:
-    """Credential/project-scoped ``TracerProvider`` cache keyed by the routing headers."""
+    """Tenant-scoped ``TracerProvider`` cache keyed by routing headers and service name."""
 
     def __init__(
         self,
@@ -131,7 +155,7 @@ class TenantTracerCache:
         # thread-pool workers concurrently with the event loop, so cache
         # updates, span counts, and retirement must be atomic.
         self._lock: Final = threading.Lock()
-        self._providers: OrderedDict[tuple[_HeaderItems, _HeaderItems, str | None], TracerProvider] = (
+        self._providers: OrderedDict[_RouteKey, TracerProvider] = (
             OrderedDict()  # mutable-ok: bounded LRU; eviction needs in-place ordered mutation
         )
         self._open_span_counts: dict[TracerProvider, int] = {}  # mutable-ok: live refcount state
@@ -172,10 +196,11 @@ class TenantTracerCache:
     ) -> TenantRoute:
         """Return the tracer (and trace-detachment flag) for this request.
 
-        Use ``default`` unless the request's dynamic credentials or its key/team
-        project require a scoped tracer, in which case build (or reuse) one. The
-        cache is a bounded LRU: the least-recently-used provider is flushed and
-        shut down on overflow so its exporter threads don't accumulate.
+        Use ``default`` unless the request's dynamic credentials, its key/team
+        project, or its key/team service name require a scoped tracer, in
+        which case build (or reuse) one. The cache is a bounded LRU: the
+        least-recently-used provider is flushed and shut down on overflow so
+        its exporter threads don't accumulate.
 
         A routed provider is returned already held — its open-span count is
         incremented in the same critical section as the cache update — so a
@@ -184,7 +209,8 @@ class TenantTracerCache:
         """
         credential_headers: Final = dynamic_otlp_headers(self._callback_name, dynamic_params) or _NO_HEADERS
         project_headers: Final = self._project_headers(auth_metadata)
-        if not credential_headers and not project_headers:
+        service_name: Final = tenant_service_name(auth_metadata)
+        if not credential_headers and not project_headers and service_name is None:
             return TenantRoute(tracer=default, detached=False)
         # A fixed per-integration region endpoint (New Relic us/eu), never a
         # caller-supplied host; ``None`` keeps the preset's own endpoint.
@@ -193,9 +219,12 @@ class TenantTracerCache:
             tuple(sorted(credential_headers.items())),
             tuple(sorted(project_headers.items())),
             endpoint,
+            service_name,
         )
         with self._lock:
-            provider: Final = self._cached_provider_locked(cache_key, credential_headers, project_headers, endpoint)
+            provider: Final = self._cached_provider_locked(
+                cache_key, credential_headers, project_headers, endpoint, service_name
+            )
             self._open_span_counts[provider] = self._open_span_counts.get(provider, 0) + 1
             evicted: Final = self._evicted_on_overflow_locked()
         if evicted is not None:
@@ -208,16 +237,19 @@ class TenantTracerCache:
 
     def _cached_provider_locked(
         self,
-        cache_key: tuple[_HeaderItems, _HeaderItems, str | None],
+        cache_key: _RouteKey,
         credential_headers: Mapping[str, str],
         project_headers: Mapping[str, str],
         endpoint: str | None,
+        service_name: str | None,
     ) -> TracerProvider:
         cached: Final = self._providers.get(cache_key)
         if cached is not None:
             self._providers.move_to_end(cache_key)
             return cached
-        built: Final = build_tracer_provider(self._routed_config(credential_headers, project_headers, endpoint))
+        built: Final = build_tracer_provider(
+            self._routed_config(credential_headers, project_headers, endpoint, service_name)
+        )
         self._providers[cache_key] = built
         return built
 
@@ -267,6 +299,7 @@ class TenantTracerCache:
         credential_headers: Mapping[str, str],
         project_headers: Mapping[str, str],
         endpoint: str | None = None,
+        service_name: str | None = None,
     ) -> OpenTelemetryV2Config:
         """Clone the config, rewriting headers on the callback's own exporter.
 
@@ -285,7 +318,10 @@ class TenantTracerCache:
             self._routed_exporter(spec, credential_headers, project_headers, endpoint)
             for spec in self._config.exporters
         ]
-        return self._config.model_copy(update={"exporters": exporters})
+        update: Final = (
+            {"exporters": exporters} if service_name is None else {"exporters": exporters, "service_name": service_name}
+        )
+        return self._config.model_copy(update=update)
 
     def _routed_exporter(
         self,

litellm/integrations/shadow_eval_logger.py

@@ -452,6 +452,14 @@ async def _key_or_team_is_over_budget(metadata: Mapping[str, object]) -> bool:
     return False
 
 
+def _forwarded_team_id(metadata: Mapping[str, object]) -> str | None:
+    """The shadowed key's team, the identity the judge call already carries in its metadata
+    and the router already selects deployments with. Read here too so the arm choice, which
+    happens before the router sees the call, is made under the same team."""
+    team_id: Final = metadata.get("user_api_key_team_id")
+    return team_id if isinstance(team_id, str) and team_id else None
+
+
 def _routing_decision(metadata: Mapping[str, object]) -> Mapping[str, object]:
     """The routing decision a pre-routing strategy wrote to a call's metadata, empty when
     a plain model served it. Read off the sampled request for the control arm, and off the
@@ -915,6 +923,7 @@ class ShadowEvalLogger(CustomLogger):
                 self._router_provider(),
                 judge_model,
                 judge_messages,  # pyright: ignore[reportArgumentType]  # plain SDK message dicts
+                team_id=_forwarded_team_id(parent_metadata),
                 temperature=0,
                 max_tokens=JUDGE_MAX_OUTPUT_TOKENS,
                 response_format=PAIRWISE_JUDGE_RESPONSE_FORMAT,

litellm/litellm_core_utils/audio_utils/subtitle_utils.py

@@ -0,0 +1,193 @@
+"""Provider-agnostic SRT/WebVTT subtitle synthesis from timestamped transcription tokens."""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from itertools import accumulate, chain
+from typing import Final
+
+from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
+
+CUE_MAX_TOKENS: Final = 15
+CUE_MAX_DURATION_MS: Final = 5000
+
+SRT_RESPONSE_FORMAT: Final = "srt"
+VTT_RESPONSE_FORMAT: Final = "vtt"
+SUBTITLE_RESPONSE_FORMATS: Final = frozenset((SRT_RESPONSE_FORMAT, VTT_RESPONSE_FORMAT))
+
+
+@dataclass(frozen=True, slots=True)
+class SubtitleToken:
+    text: str
+    start_ms: int | None = None
+    end_ms: int | None = None
+    speaker: str | int | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class SubtitleCue:
+    start_ms: int
+    end_ms: int
+    text: str
+
+
+@dataclass(frozen=True, slots=True)
+class _CueAccumulator:
+    texts: tuple[str, ...] = ()
+    start_ms: int | None = None
+    end_ms: int | None = None
+    speaker: str | int | None = None
+
+
+def _completed_cue(accumulator: _CueAccumulator) -> tuple[SubtitleCue, ...]:
+    if not accumulator.texts or accumulator.start_ms is None:
+        return ()
+    text: Final = "".join(accumulator.texts).strip()
+    if not text:
+        return ()
+    end_ms: Final = accumulator.end_ms if accumulator.end_ms is not None else accumulator.start_ms
+    return (SubtitleCue(start_ms=accumulator.start_ms, end_ms=end_ms, text=text),)
+
+
+def _cue_break_reached(accumulator: _CueAccumulator, token: SubtitleToken) -> bool:
+    if len(accumulator.texts) >= CUE_MAX_TOKENS:
+        return True
+    return (
+        accumulator.start_ms is not None
+        and token.start_ms is not None
+        and token.start_ms - accumulator.start_ms >= CUE_MAX_DURATION_MS
+    )
+
+
+_AbsorbStep = tuple[tuple[SubtitleCue, ...], _CueAccumulator]
+
+
+def _absorb_token(accumulator: _CueAccumulator, token: SubtitleToken) -> _AbsorbStep:
+    if token.start_ms is None and accumulator.start_ms is None:
+        return (), accumulator
+    if token.speaker is not None and token.speaker != accumulator.speaker:
+        return _completed_cue(accumulator), _CueAccumulator(
+            texts=(token.text,),
+            start_ms=token.start_ms,
+            end_ms=token.end_ms,
+            speaker=token.speaker,
+        )
+    if _cue_break_reached(accumulator, token):
+        return _completed_cue(accumulator), _CueAccumulator(
+            texts=(token.text,),
+            start_ms=token.start_ms,
+            end_ms=token.end_ms,
+            speaker=accumulator.speaker,
+        )
+    return (), _CueAccumulator(
+        texts=(*accumulator.texts, token.text),
+        start_ms=accumulator.start_ms if accumulator.start_ms is not None else token.start_ms,
+        end_ms=token.end_ms if token.end_ms is not None else accumulator.end_ms,
+        speaker=accumulator.speaker,
+    )
+
+
+def _absorb_step(carry: _AbsorbStep, token: SubtitleToken) -> _AbsorbStep:
+    return _absorb_token(carry[1], token)
+
+
+def group_subtitle_tokens_into_cues(tokens: Sequence[SubtitleToken]) -> tuple[SubtitleCue, ...]:
+    steps: Final = tuple(accumulate(tokens, _absorb_step, initial=((), _CueAccumulator())))
+    completed: Final = chain.from_iterable(emitted for emitted, _ in steps)
+    return (*completed, *_completed_cue(steps[-1][1]))
+
+
+def _format_timestamp(total_ms: int, millis_separator: str) -> str:
+    clamped: Final = max(total_ms, 0)
+    hours, hour_remainder = divmod(clamped, 3_600_000)
+    minutes, minute_remainder = divmod(hour_remainder, 60_000)
+    seconds, millis = divmod(minute_remainder, 1_000)
+    return f"{hours:02d}:{minutes:02d}:{seconds:02d}{millis_separator}{millis:03d}"
+
+
+def _render_srt(cues: Sequence[SubtitleCue]) -> str:
+    lines: Final = tuple(
+        line
+        for index, cue in enumerate(cues, start=1)
+        for line in (
+            str(index),
+            f"{_format_timestamp(cue.start_ms, ',')} --> {_format_timestamp(cue.end_ms, ',')}",
+            cue.text,
+            "",
+        )
+    )
+    return "\n".join(lines)
+
+
+def _render_vtt(cues: Sequence[SubtitleCue]) -> str:
+    cue_lines: Final = tuple(
+        line
+        for cue in cues
+        for line in (
+            f"{_format_timestamp(cue.start_ms, '.')} --> {_format_timestamp(cue.end_ms, '.')}",
+            cue.text,
+            "",
+        )
+    )
+    return "\n".join(("WEBVTT", "", *cue_lines))
+
+
+def render_subtitle_tokens_as_srt(tokens: Sequence[SubtitleToken]) -> str:
+    """Render tokens as an SRT document; empty string when no token has timestamp data."""
+    cues: Final = group_subtitle_tokens_into_cues(tokens)
+    if not cues:
+        return ""
+    return _render_srt(cues)
+
+
+def render_subtitle_tokens_as_vtt(tokens: Sequence[SubtitleToken]) -> str:
+    """Render tokens as a WebVTT document; the WEBVTT header is emitted even without cues."""
+    return _render_vtt(group_subtitle_tokens_into_cues(tokens))
+
+
+class TranscriptionWordTiming(BaseModel):
+    model_config = ConfigDict(frozen=True, extra="ignore")
+
+    word: str = ""
+    start: float | None = None
+    end: float | None = None
+    speaker: str | None = None
+
+
+_WORD_TIMINGS_ADAPTER: Final = TypeAdapter(tuple[TranscriptionWordTiming, ...])
+
+
+def _seconds_to_ms(seconds: float | None) -> int | None:
+    if seconds is None:
+        return None
+    return round(seconds * 1000)
+
+
+def _word_to_subtitle_token(word: TranscriptionWordTiming) -> SubtitleToken:
+    return SubtitleToken(
+        text=f"{word.word} ",
+        start_ms=_seconds_to_ms(word.start),
+        end_ms=_seconds_to_ms(word.end),
+        speaker=word.speaker,
+    )
+
+
+def _parse_word_timings(words: object) -> tuple[TranscriptionWordTiming, ...]:
+    try:
+        return _WORD_TIMINGS_ADAPTER.validate_python(words)
+    except ValidationError:
+        return ()
+
+
+def synthesize_subtitle_document(words: object, response_format: str) -> str | None:
+    """
+    Build an SRT/VTT document from OpenAI verbose_json-style word dicts
+    (word/start/end in float seconds, optional speaker). Returns None when the
+    format is not a subtitle format or the words carry no usable timestamps.
+    """
+    if response_format not in SUBTITLE_RESPONSE_FORMATS:
+        return None
+    tokens: Final = tuple(_word_to_subtitle_token(word) for word in _parse_word_timings(words))
+    cues: Final = group_subtitle_tokens_into_cues(tokens)
+    if not cues:
+        return None
+    return _render_srt(cues) if response_format == SRT_RESPONSE_FORMAT else _render_vtt(cues)

litellm/litellm_core_utils/exception_mapping_utils.py

@@ -2341,6 +2341,7 @@ def exception_type(
     litellm_response_headers: Final = _get_response_headers(original_exception=original_exception)
     try:
         error_str = redact_string(str(original_exception)) if _ENABLE_SECRET_REDACTION else str(original_exception)
+        extra_information = ""
         if model or custom_llm_provider:
             if hasattr(original_exception, "message"):
                 error_str = (
@@ -2357,7 +2358,6 @@ def exception_type(
             # Common Extra information needed for all providers
             # We pass num retries, api_base, vertex_deployment etc to the exception here
             ################################################################################
-            extra_information = ""
             try:
                 _api_base: Final = litellm.get_api_base(model=model, optional_params=extra_kwargs)
                 messages: Final = litellm.get_first_chars_messages(kwargs=completion_kwargs)

litellm/litellm_core_utils/internal_call_metadata.py

@@ -20,8 +20,8 @@ from __future__ import annotations
 from collections.abc import Mapping
 from typing import Final
 
-from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
-from litellm.types.utils import InternalCallOrigin
+from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY, NON_INFERENCE_CALL_TYPES
+from litellm.types.utils import BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN, InternalCallOrigin
 
 BUDGET_RESERVATION_METADATA_KEYS: Final = frozenset({"user_api_key_budget_reservation"})
 
@@ -45,6 +45,60 @@ budget-checked like the request that spawned it. Everything else on the parent's
 be a lie on a sub-call that runs after it returned."""
 
 
+def is_background_response(response: object) -> bool:
+    """Whether a retrieved object is a response created with ``background=true``.
+
+    Such a create returns ``status="queued"`` and no usage at all, so nothing has billed the
+    job by the time anyone reads it back. Accepts the response as a mapping or a model,
+    because the callers hold it in both shapes.
+    """
+    if isinstance(response, Mapping):
+        return response.get("background") is True
+    return getattr(response, "background", None) is True
+
+
+def is_unbilled_non_inference_call(
+    call_type: str | None,
+    metadata: Mapping[str, object] | None,
+    response: object,
+) -> bool:
+    """A read/management route priced at zero, because the usage it reports belongs to the
+    call that created the object it just read.
+
+    Retrieving a background response is the exception, and the enterprise cost poller's read
+    is the same exception seen from the other side: that job's create billed nothing, so its
+    retrieval is the only place the spend is ever visible. Pricing those at zero would lose
+    the spend rather than deduplicate it.
+    """
+    if call_type not in NON_INFERENCE_CALL_TYPES:
+        return False
+    if is_background_response(response):
+        return False
+    if metadata is None:
+        return True
+    return metadata.get(INTERNAL_CALL_ORIGIN_METADATA_KEY) != BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN
+
+
+def is_unbilled_non_inference_call_from_params(
+    call_type: str | None,
+    litellm_params: Mapping[str, object] | None,
+    response: object,
+) -> bool:
+    """:func:`is_unbilled_non_inference_call` for callers holding raw ``litellm_params``.
+
+    The call-type membership test runs first so that inference traffic, which is every
+    request in a normal workload, never pays for the metadata merge behind it.
+    """
+    if call_type not in NON_INFERENCE_CALL_TYPES:
+        return False
+    from litellm.litellm_core_utils.litellm_logging import StandardLoggingPayloadSetup
+
+    metadata: Final = (
+        StandardLoggingPayloadSetup.merge_litellm_metadata(litellm_params) if litellm_params is not None else None
+    )
+    return is_unbilled_non_inference_call(call_type, metadata, response)
+
+
 def sanitize_user_api_key_auth(auth: object) -> object:
     """Copy of the auth object with its budget reservation removed; the cost callback
     falls back to reading the reservation from inside the auth object."""

litellm/litellm_core_utils/litellm_logging.py

@@ -64,6 +64,7 @@ from litellm.integrations.mlflow import MlflowLogger
 from litellm.integrations.sqs import SQSLogger
 from litellm.litellm_core_utils.core_helpers import is_expected_client_error, reconstruct_model_name
 from litellm.litellm_core_utils.get_litellm_params import get_litellm_params
+from litellm.litellm_core_utils.internal_call_metadata import is_unbilled_non_inference_call
 from litellm.litellm_core_utils.llm_cost_calc.guardrail_cost import (
     cost_breakdown_with_guardrail,
     guardrail_information_cost,
@@ -612,37 +613,60 @@ class Logging(LiteLLMLoggingBaseClass):
         processed_list: Final[list[str | Callable | CustomLogger]] = []
         for callback in callback_list:
             if isinstance(callback, str) and callback in litellm._known_custom_logger_compatible_callbacks:
-                # For callbacks that support team-scoped credentials (e.g. datadog),
-                # pass only the relevant dynamic params as custom_logger_init_args.
-                _custom_logger_init_args: dict | None = None
-                if callback == "datadog":
-                    # dd_* params are blocked from standard_callback_dynamic_params
-                    # (request-level security); only the proxy-stamped team/key
-                    # callback vars are admin-configured and trusted.
-                    _custom_logger_init_args = {k: v for k, v in self._trusted_callback_vars if k.startswith("dd_")}
-
-                callback_class = _init_custom_logger_compatible_class(
-                    callback,
-                    internal_usage_cache=None,
-                    llm_router=None,
-                    custom_logger_init_args=_custom_logger_init_args,
-                )
-                if callback_class is not None:
-                    processed_list.append(callback_class)
+                for callback_instance in self._resolve_dynamic_callback_string(callback):
+                    processed_list.append(callback_instance)
 
                     # If processing dynamic_success_callbacks, add to dynamic_async_success_callbacks
                     if dynamic_callbacks_type == "success":
                         if self.dynamic_async_success_callbacks is None:
                             self.dynamic_async_success_callbacks = []
-                        self.dynamic_async_success_callbacks.append(callback_class)
+                        self.dynamic_async_success_callbacks.append(callback_instance)
                     elif dynamic_callbacks_type == "failure":
                         if self.dynamic_async_failure_callbacks is None:
                             self.dynamic_async_failure_callbacks = []
-                        self.dynamic_async_failure_callbacks.append(callback_class)
+                        self.dynamic_async_failure_callbacks.append(callback_instance)
             else:
                 processed_list.append(callback)
         return processed_list
 
+    def _resolve_dynamic_callback_string(self, callback: str) -> "tuple[CustomLogger, ...]":
+        """
+        Resolve a known callback name to the logger instance(s) it dispatches to.
+
+        For callbacks that support team-scoped credentials (datadog, newrelic),
+        only the proxy-stamped team/key callback vars are passed as
+        custom_logger_init_args: dd_*/newrelic_* params are blocked from
+        standard_callback_dynamic_params (request-level security), so the
+        trusted-vars channel is the only way credentials reach a per-team logger.
+        """
+        _trusted_var_prefix: Final = "dd_" if callback == "datadog" else "newrelic_" if callback == "newrelic" else None
+        _custom_logger_init_args: Final[dict | None] = (
+            {k: v for k, v in self._trusted_callback_vars if k.startswith(_trusted_var_prefix)}
+            if _trusted_var_prefix is not None
+            else None
+        )
+
+        callback_class: Final = _init_custom_logger_compatible_class(
+            callback,
+            internal_usage_cache=None,
+            llm_router=None,
+            custom_logger_init_args=_custom_logger_init_args,
+        )
+        if callback_class is None:
+            return ()
+
+        # With team creds, "newrelic" resolves to the per-team METRICS logger;
+        # resolve the name again without creds so the trace logger (OTel v2 /
+        # legacy agent) keeps receiving this request.
+        _newrelic_trace_class: Final = (
+            _init_custom_logger_compatible_class(callback, internal_usage_cache=None, llm_router=None)
+            if callback == "newrelic" and _custom_logger_init_args and _custom_logger_init_args.get("newrelic_api_key")
+            else None
+        )
+        if _newrelic_trace_class is not None and _newrelic_trace_class is not callback_class:
+            return (callback_class, _newrelic_trace_class)
+        return (callback_class,)
+
     def initialize_standard_callback_dynamic_params(self, kwargs: dict | None = None) -> StandardCallbackDynamicParams:
         """
         Initialize the standard callback dynamic params from the kwargs
@@ -1586,6 +1610,11 @@ class Logging(LiteLLMLoggingBaseClass):
         if cache_hit is True:
             return 0.0
 
+        if is_unbilled_non_inference_call(
+            self.call_type, StandardLoggingPayloadSetup.merge_litellm_metadata(self.litellm_params), result
+        ):
+            return 0.0
+
         transformed_result: Final = self._generate_content_result_as_model_response(result)
         if transformed_result is not None:
             result = transformed_result
@@ -4636,6 +4665,19 @@ def _init_custom_logger_compatible_class(
             _in_memory_loggers.append(gitlab_logger)
             return gitlab_logger
         elif logging_integration == "newrelic":
+            if custom_logger_init_args.get("newrelic_api_key"):
+                # Team-scoped credentials: per-team METRICS logger, isolated per
+                # credential set via DynamicLoggingCache. The trace logger for
+                # this name stays on the global path below.
+                from litellm.integrations.newrelic.newrelic_team_handler import (
+                    NewRelicHandler,
+                )
+
+                return NewRelicHandler.get_newrelic_logger_for_request(
+                    standard_callback_dynamic_params=custom_logger_init_args,
+                    in_memory_dynamic_logger_cache=in_memory_dynamic_logger_cache,
+                )
+
             _v2 = _maybe_construct_otel_v2("newrelic", _in_memory_loggers)
             if _v2 is not None:
                 return _v2
@@ -5057,7 +5099,7 @@ class StandardLoggingPayloadSetup:
         return messages
 
     @staticmethod
-    def merge_litellm_metadata(litellm_params: dict) -> dict:
+    def merge_litellm_metadata(litellm_params: Mapping[str, object]) -> dict:
         """
         Merge both litellm_metadata and metadata from litellm_params.
 
@@ -5819,7 +5861,7 @@ def get_standard_logging_object_payload(
         cache_hit: Final = kwargs.get("cache_hit", False)
         # Extract usage as a plain dict, avoiding Pydantic round-trip
         raw_usage_dict: Final = StandardLoggingPayloadSetup.get_usage_as_dict(
-            response_obj=response_obj,
+            response_obj=None if is_unbilled_non_inference_call(call_type, metadata, response_obj) else response_obj,
             combined_usage_object=cast(Usage | None, kwargs.get("combined_usage_object")),
         )
         usage_dict: Final = (

litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py

@@ -7,7 +7,9 @@ from typing import Any, Final, Literal
 
 import litellm
 from litellm.constants import OPENAI_FILE_SEARCH_COST_PER_1K_CALLS
-from litellm.litellm_core_utils.llm_cost_calc.utils import _get_web_search_requests
+from litellm.litellm_core_utils.llm_cost_calc.utils import (
+    get_web_search_requests_from_usage,
+)
 from litellm.types.llms.openai import (
     FileSearchTool,
     ResponsesAPIResponse,
@@ -368,7 +370,7 @@ class StandardBuiltInToolCostTracking:
             get_anthropic_web_search_requests_from_response,
         )
 
-        if usage is not None and (_get_web_search_requests(getattr(usage, "server_tool_use", None)) is not None):
+        if usage is not None and (get_web_search_requests_from_usage(usage) is not None):
             return usage
         web_search_requests: Final = get_anthropic_web_search_requests_from_response(response_object)
         if web_search_requests is None:
@@ -416,7 +418,7 @@ class StandardBuiltInToolCostTracking:
                 # Anthropic Claude (direct API and Vertex AI) uses server_tool_use.web_search_requests.
                 # Without this check, Claude ModelResponse always falls through to return False
                 # and _handle_web_search_cost() is never called.
-                if hasattr(usage, "server_tool_use") and _get_web_search_requests(usage.server_tool_use) is not None:
+                if get_web_search_requests_from_usage(usage) is not None:
                     return True
                 # xAI reports usage.server_side_tool_usage_details.web_search_calls; a searched
                 # answer with no url_citation annotations has no other chat-path signal
@@ -429,16 +431,12 @@ class StandardBuiltInToolCostTracking:
                 response_object=response_object, output_type="web_search_call"
             )
         elif usage is not None:
-            if (
-                hasattr(usage, "server_tool_use")
-                and _get_web_search_requests(usage.server_tool_use) is not None
-                or (
-                    hasattr(usage, "prompt_tokens_details")
-                    and usage.prompt_tokens_details is not None
-                    and isinstance(usage.prompt_tokens_details, PromptTokensDetailsWrapper)
-                    and hasattr(usage.prompt_tokens_details, "web_search_requests")
-                    and usage.prompt_tokens_details.web_search_requests is not None
-                )
+            if get_web_search_requests_from_usage(usage) is not None or (
+                hasattr(usage, "prompt_tokens_details")
+                and usage.prompt_tokens_details is not None
+                and isinstance(usage.prompt_tokens_details, PromptTokensDetailsWrapper)
+                and hasattr(usage.prompt_tokens_details, "web_search_requests")
+                and usage.prompt_tokens_details.web_search_requests is not None
             ):
                 return True
             if _usage_reports_server_side_web_search_calls(usage):

litellm/litellm_core_utils/llm_cost_calc/utils.py

@@ -72,7 +72,7 @@ def _get_token_detail_value(details: object, key: str) -> int | None:
     return value if isinstance(value, int) else None
 
 
-def _get_web_search_requests(server_tool_use: Any) -> int | None:
+def get_web_search_requests(server_tool_use: Any) -> int | None:
     """
     Tolerantly read ``web_search_requests`` from a ``server_tool_use`` value
     that may be ``None``, a ``dict``, a ``ServerToolUse`` pydantic instance,
@@ -92,6 +92,16 @@ def _get_web_search_requests(server_tool_use: Any) -> int | None:
     return getattr(server_tool_use, "web_search_requests", None)
 
 
+def get_web_search_requests_from_usage(usage: Usage) -> int | None:
+    """Read ``web_search_requests`` from a ``Usage``'s ``server_tool_use``.
+
+    ``Usage`` deletes unset optional fields from ``__dict__`` (see
+    ``SafeAttributeModel``), so direct attribute access can raise
+    ``AttributeError``; ``getattr`` with a default is required here.
+    """
+    return get_web_search_requests(getattr(usage, "server_tool_use", None))
+
+
 def _is_above_128k(tokens: float) -> bool:
     if tokens > 128000:
         return True
@@ -889,11 +899,22 @@ def generic_cost_per_token(
     total_details: Final = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens + video_tokens
     has_double_counting: Final = (cache_hit > 0 or cache_creation > 0) and total_details > usage.prompt_tokens
 
-    if (text_tokens == 0 and prompt_tokens_details["image_count"] == 0) or has_double_counting:
-        text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens - video_tokens
+    if has_double_counting:
+        # cached and per-modality counts are both subsets of prompt_tokens and may overlap, so a
+        # modality can only bill what the cache did not already cover or the overlap is billed twice
+        uncached_budget: Final = max(usage.prompt_tokens - cache_hit - cache_creation, 0)
+        billable_audio: Final = min(audio_tokens, uncached_budget)
+        billable_image: Final = min(image_tokens, uncached_budget - billable_audio)
+        billable_video: Final = min(video_tokens, uncached_budget - billable_audio - billable_image)
+        prompt_tokens_details["audio_tokens"] = billable_audio
+        prompt_tokens_details["image_tokens"] = billable_image
+        prompt_tokens_details["video_tokens"] = billable_video
+        prompt_tokens_details["text_tokens"] = uncached_budget - billable_audio - billable_image - billable_video
+    elif text_tokens == 0 and prompt_tokens_details["image_count"] == 0:
         # Clamp to zero: inconsistent streaming usage
-        text_tokens = max(text_tokens, 0)
-        prompt_tokens_details["text_tokens"] = text_tokens
+        prompt_tokens_details["text_tokens"] = max(
+            usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens - video_tokens, 0
+        )
 
     (
         prompt_base_cost,

litellm/litellm_core_utils/llm_judge.py

@@ -4,7 +4,9 @@ from __future__ import annotations
 
 import json
 import re
-from typing import TYPE_CHECKING, Final
+from dataclasses import dataclass
+from functools import lru_cache
+from typing import TYPE_CHECKING, Final, Literal
 
 import litellm
 
@@ -56,17 +58,62 @@ def extract_text_from_content(content: object) -> str:
     return ""
 
 
-def router_resolves_model(router: Router | None, model: str) -> bool:
-    """Whether the model name resolves through the proxy's router (configured deployment
-    or model-group alias), the same check the judge dispatch itself makes, so start-time
-    validation cannot accept a name the call path then fails on."""
-    return router is not None and bool(model in router.model_group_alias or router.get_model_list(model_name=model))
+@lru_cache(maxsize=512)
+def _provider_qualified(model: str) -> str | None:
+    """`model` in the one spelling litellm itself resolves it to, or None if it maps to no
+    provider.
+
+    A deployment may be configured as `openai/gpt-4o` and a judge given as `gpt-4o`; both
+    reach the same model, so an identity that keeps them apart reports two models where
+    there is one. None is a different answer from "unchanged": a name that is already
+    provider-qualified normalises to itself, and reading that as a failure would call every
+    correctly-spelled public model unresolvable.
+    """
+    try:
+        stripped, provider, _, _ = litellm.get_llm_provider(model=model)
+    except Exception:  # noqa: BLE001  # an unmapped name has no provider, which is the answer
+        return None
+    return f"{provider}/{stripped}" if provider and stripped else None
+
+
+@dataclass(frozen=True, slots=True)
+class JudgeTarget:
+    """Where a call to one model name goes for one caller, and what answers it.
+
+    The single answer to that question: the resolvability gate, the judge-vs-candidate
+    gate and the dispatch all read it, so none of them can decide it differently. Splitting
+    it is what let start-time validation accept a team's own model while dispatch sent the
+    literal name to the SDK.
+    """
+
+    via: Literal["router", "sdk", "nothing"]
+    models: frozenset[str]
+
+
+def judge_target(router: Router | None, model: str, team_id: str | None = None) -> JudgeTarget:
+    """Resolve `model` the way a call from `team_id` would be.
+
+    Three outcomes and no others: the router serves it (a deployment, a team-public name,
+    an alias, a routing group or a wildcard, exactly the channels `get_model_list`
+    composes); the SDK serves it because litellm recognises the provider; or nothing does,
+    which is the only case a caller may refuse on.
+
+    `team_id` is part of the question, not a refinement of it. A team-public name resolves
+    only for its own team and a team's own deployment resolves for nobody else, so asking
+    without it answers for a caller who does not exist.
+    """
+    served: Final = router.resolved_litellm_models(model, team_id=team_id) if router is not None else ()
+    if served:
+        return JudgeTarget("router", frozenset(_provider_qualified(m) or m for m in served))
+    qualified: Final = _provider_qualified(model)
+    return JudgeTarget("sdk", frozenset({qualified})) if qualified is not None else JudgeTarget("nothing", frozenset())
 
 
 async def judge_acompletion(
     router: Router | None,
     judge_model: str,
     messages: list[AllMessageValues],  # mutable-ok: the SDK acompletion signature takes a list
+    team_id: str | None = None,
     **params: object,
 ) -> ModelResponse:
     """Dispatch a judge call through the proxy's router when the judge model is a
@@ -74,9 +121,13 @@ async def judge_acompletion(
     provider-qualified public names. The router path never retries or falls back:
     a failed judge call is the caller's counted failure, not a spend multiplier.
     Sampling preferences are advisory: models that removed sampling params (e.g.
-    claude-sonnet-5) drop them instead of rejecting the judge call."""
-    if router_resolves_model(router, judge_model):
-        return await router.acompletion(  # pyright: ignore[reportOptionalMemberAccess]  # router_resolves_model implies router is not None
+    claude-sonnet-5) drop them instead of rejecting the judge call.
+
+    The arm is chosen by `judge_target` under the caller's own team, the same call
+    start-time validation makes, so a judge a team can reach cannot be validated as a
+    deployment and then dispatched as a public name the SDK has never heard of."""
+    if judge_target(router, judge_model, team_id).via == "router":
+        return await router.acompletion(  # pyright: ignore[reportOptionalMemberAccess]  # a router target implies router is not None
             model=judge_model,
             messages=messages,
             num_retries=0,

litellm/litellm_core_utils/prompt_templates/common_utils.py

@@ -1747,6 +1747,46 @@ def hoist_images_from_tool_messages(
     ]
 
 
+def _is_tool_reference_part(part: object) -> bool:
+    return isinstance(part, dict) and part.get("type") == "tool_reference"
+
+
+def _tool_message_carries_tool_reference(message: AllMessageValues) -> bool:
+    if message.get("role") != "tool":
+        return False
+    content = message.get("content")
+    return isinstance(content, list) and any(_is_tool_reference_part(part) for part in content)
+
+
+def _drop_tool_reference_parts(message: AllMessageValues) -> AllMessageValues:
+    if not _tool_message_carries_tool_reference(message):
+        return message
+    content = cast(list, message.get("content"))  # cast-ok: shape checked by _tool_message_carries_tool_reference
+    remaining_parts = [  # mutable-ok: tool message content must stay a json list
+        part for part in content if not _is_tool_reference_part(part)
+    ]
+    new_content = remaining_parts if remaining_parts else ""
+    rewritten = {**message, "content": new_content}  # mutable-ok: chat messages are plain json dicts
+    return cast(AllMessageValues, rewritten)  # cast-ok: dict spread keeps keys like cache_control
+
+
+def drop_tool_reference_parts_from_tool_messages(
+    messages: list[AllMessageValues],  # mutable-ok: message pipelines type messages as mutable lists
+) -> list[AllMessageValues]:  # mutable-ok: message pipelines type messages as mutable lists
+    """
+    Remove tool_reference content parts from role:"tool" messages.
+
+    The OpenAI chat spec only accepts text in tool messages, so a tool_reference
+    part carried through the Anthropic adapter makes strict providers reject the
+    request. The reference names an already-declared tool rather than carrying
+    content, so it is dropped; a reference-only result keeps its tool message with
+    empty text so the preceding tool_call stays answered.
+    """
+    if not any(_tool_message_carries_tool_reference(message) for message in messages):
+        return messages
+    return [_drop_tool_reference_parts(message) for message in messages]  # mutable-ok: pipelines mutate message lists
+
+
 def _attempt_json_repair(s: str) -> Any | None:
     """
     Attempt to repair truncated JSON produced by LLM tool calls.

litellm/litellm_core_utils/prompt_templates/factory.py

@@ -1412,7 +1412,7 @@ def convert_to_gemini_tool_call_result(
                             )
                         except Exception as e:
                             verbose_logger.warning("Failed to process image in tool response: %s", e)
-                elif content_type in ("file", "input_file"):
+                elif content_type in ("file", "input_file"):  # pyright: ignore[reportUnnecessaryContains]  # loose runtime dict
                     # Extract file for inline_data (for tool results with PDF, audio, video, etc.)
                     file_data = content.get("file_data", "")
                     if not file_data:
@@ -1564,14 +1564,23 @@ def convert_to_anthropic_tool_result(
     }
     """
     anthropic_content: (
-        str | list[AnthropicMessagesToolResultContent | AnthropicMessagesImageParam | AnthropicMessagesDocumentParam]
+        str
+        | list[
+            AnthropicMessagesToolResultContent
+            | AnthropicMessagesImageParam
+            | AnthropicMessagesDocumentParam
+            | ToolReference
+        ]
     ) = ""
     if isinstance(message["content"], str):
         anthropic_content = message["content"]
     elif isinstance(message["content"], list):
         content_list: Final = message["content"]
         anthropic_content_list: list[
-            AnthropicMessagesToolResultContent | AnthropicMessagesImageParam | AnthropicMessagesDocumentParam
+            AnthropicMessagesToolResultContent
+            | AnthropicMessagesImageParam
+            | AnthropicMessagesDocumentParam
+            | ToolReference
         ] = []
         for content in content_list:
             if content["type"] == "text":
@@ -1614,6 +1623,8 @@ def convert_to_anthropic_tool_result(
                         original_content_element=content,
                     )
                     anthropic_content_list.append(cast(AnthropicMessagesImageParam, _anthropic_image_param))
+            elif content["type"] == "tool_reference":
+                anthropic_content_list.append(ToolReference(type="tool_reference", tool_name=content["tool_name"]))
             elif content["type"] == "file":
                 file_content = cast(ChatCompletionFileObject, content)
                 _file_block = anthropic_process_openai_file_message(file_content)

litellm/litellm_core_utils/realtime_streaming.py

@@ -330,6 +330,24 @@ class RealTimeStreaming:
         except (AttributeError, TypeError):
             pass
 
+    def _flush_unbilled_transcription_usage(self) -> None:
+        if self.provider_config is None:
+            return
+        usage: Final = self.provider_config.unbilled_usage_on_session_close(self.model)
+        if usage is None:
+            return
+        flush_event: Final = (
+            cast(  # cast-ok: usage-only partial event, the same shape _capture_transcription_usage logs
+                OpenAIRealtimeEvents,
+                {
+                    "type": "conversation.item.input_audio_transcription.completed",
+                    "usage": usage,
+                },
+            )
+        )
+        self.store_message(flush_event)
+        self._capture_transcription_usage(flush_event)
+
     def _collect_tool_calls_from_response_done(self, event_obj: dict | OpenAIRealtimeEvents) -> None:
         """Extract function_call items from response.done events for spend logging."""
         try:
@@ -955,6 +973,7 @@ class RealTimeStreaming:
                 transcript = event.get("transcript", "")
                 self._collect_user_input_from_backend_event(cast(dict, event))
                 self.store_message(event_str)
+                self._capture_transcription_usage(event)
                 await self._send_event_to_client(event, event_str)
                 blocked = await self.run_realtime_guardrails(
                     cast(str, transcript),
@@ -1068,6 +1087,7 @@ class RealTimeStreaming:
         except Exception as e:
             verbose_logger.exception("Error in backend to client send messages: %s", e)
         finally:
+            self._flush_unbilled_transcription_usage()
             await self.log_messages()
 
     @staticmethod

litellm/litellm_core_utils/specialty_caches/dynamic_logging_cache.py

@@ -13,6 +13,7 @@ import json
 from typing import Any, Final
 
 import litellm
+from litellm._logging import verbose_logger
 from litellm.constants import _DEFAULT_TTL_FOR_HTTPX_CLIENTS
 
 from ...caching import InMemoryCache
@@ -46,6 +47,15 @@ class LangfuseInMemoryCache(InMemoryCache):
             _created_langfuse_logger.Langfuse.flush()
             _created_langfuse_logger.Langfuse.shutdown()
 
+        # Loggers with a periodic flush task (e.g. NewRelicMetricsLogger) expose
+        # stop() so eviction actually ends the task instead of leaking it.
+        _evicted_stop: Final = getattr(self.cache_dict[key], "stop", None)
+        if callable(_evicted_stop):
+            try:
+                _evicted_stop()
+            except Exception:  # noqa: BLE001  # a failing stop() must not block eviction
+                verbose_logger.debug("DynamicLoggingCache: stop() raised during eviction", exc_info=True)
+
         #########################################################
         # Call parent class to remove key from cache
         #########################################################

litellm/litellm_core_utils/streaming_handler.py

@@ -8,11 +8,12 @@ import time
 import traceback
 from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping, Sequence
 from dataclasses import dataclass
+from types import MappingProxyType
 from typing import Any, Final, NoReturn, Protocol, TypeVar, cast
 
 import anyio
 import httpx
-from pydantic import BaseModel
+from pydantic import BaseModel, ValidationError
 from typing_extensions import NotRequired, TypedDict
 
 import litellm
@@ -182,6 +183,23 @@ class _VertexChunkLike(Protocol):
     candidates: Sequence[_VertexCandidateLike]
 
 
+class _ParsedChunkHiddenParams(BaseModel):
+    provider_specific_fields: Mapping[str, object] | None = None
+
+
+def _provider_hidden_params(chunk: object) -> Mapping[str, object] | None:
+    hidden: Final[object] = getattr(chunk, "_hidden_params", None)
+    if not isinstance(hidden, dict):
+        return None
+    try:
+        parsed: Final = _ParsedChunkHiddenParams.model_validate(hidden)
+    except ValidationError:
+        return None
+    if not parsed.provider_specific_fields:
+        return None
+    return MappingProxyType({"provider_specific_fields": dict(parsed.provider_specific_fields)})
+
+
 class CustomStreamWrapper:
     def __init__(
         self,
@@ -801,7 +819,7 @@ class CustomStreamWrapper:
         except Exception as e:
             raise e
 
-    def model_response_creator(self, chunk: dict | None = None, hidden_params: dict | None = None):
+    def model_response_creator(self, chunk: dict | None = None, hidden_params: Mapping[str, object] | None = None):
         _model: Final = self._cached_model_name
         _logging_obj_llm_provider: Final = self._cached_logging_llm_provider
 
@@ -1504,7 +1522,7 @@ class CustomStreamWrapper:
     def chunk_creator(self, chunk: Any):
         if hasattr(chunk, "id"):
             self.response_id = chunk.id
-        model_response = self.model_response_creator()
+        model_response = self.model_response_creator(hidden_params=_provider_hidden_params(chunk))
         response_obj: dict[str, Any] = {}
         try:
             # return this for all models

litellm/llms/anthropic/chat/guardrail_translation/handler.py

@@ -24,10 +24,12 @@ from litellm._logging import verbose_proxy_logger
 from litellm.llms.anthropic.chat.transformation import AnthropicConfig
 from litellm.llms.anthropic.experimental_pass_through.adapters.transformation import (
     LiteLLMAnthropicMessagesAdapter,
+    is_provider_native_tool_dict,
 )
 from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
 from litellm.llms.base_llm.guardrail_translation.utils import (
     anthropic_tool_name,
+    anthropic_tool_names,
     effective_scan_only_tool_results_for_guardrail,
     effective_skip_system_message_for_guardrail,
     effective_skip_tool_message_for_guardrail,
@@ -360,7 +362,13 @@ class AnthropicMessagesHandler(BaseTranslation):
         structured_messages: Final = [full_structured_messages[index] for index in scoped_message_indices]
 
         tools_to_check: Final[list[ChatCompletionToolParam]] = (
-            [] if scan_only_tool_results else chat_completion_compatible_request.get("tools", [])
+            []
+            if scan_only_tool_results
+            else [
+                tool
+                for tool in chat_completion_compatible_request.get("tools", [])
+                if not is_provider_native_tool_dict(tool)
+            ]
         )
 
         # Step 1: Extract all text content and images
@@ -419,7 +427,10 @@ class AnthropicMessagesHandler(BaseTranslation):
                         tool_name=anthropic_tool_name,
                     )
                     if scan_only_tool_results
-                    else anthropic_tools
+                    else [
+                        *(tool for tool in data.get("tools") or [] if is_provider_native_tool_dict(tool)),
+                        *anthropic_tools,
+                    ]
                 )
 
             guardrailed_structured_messages: Final = guardrailed_inputs.get("structured_messages")
@@ -677,12 +688,9 @@ class AnthropicMessagesHandler(BaseTranslation):
         )
 
     def extract_request_tool_names(self, data: dict) -> list[str]:
-        """Extract tool names from Anthropic messages request (tools[].name)."""
-        names: Final[list[str]] = []
-        for tool in data.get("tools") or []:
-            if isinstance(tool, dict) and tool.get("name"):
-                names.append(str(tool["name"]))
-        return names
+        """Extract every tool name in an Anthropic messages request: tools[].name, plus
+        tools[].function.name for OpenAI-format tools the bridge forwards verbatim."""
+        return [name for tool in data.get("tools") or [] for name in anthropic_tool_names(tool)]
 
     @classmethod
     def _extract_input_text_and_images(

litellm/llms/anthropic/cost_calculation.py

@@ -8,9 +8,9 @@ from typing import TYPE_CHECKING, Final, Optional
 from pydantic import BaseModel, ValidationError
 
 from litellm.litellm_core_utils.llm_cost_calc.utils import (
-    _get_web_search_requests,
     generic_cost_per_token,
     get_provider_specific_geo_multiplier,
+    get_web_search_requests_from_usage,
 )
 
 if TYPE_CHECKING:
@@ -104,7 +104,7 @@ def get_cost_for_anthropic_web_search(
 
     if usage is None:
         return 0.0
-    web_search_requests: Final = _get_web_search_requests(getattr(usage, "server_tool_use", None))
+    web_search_requests: Final = get_web_search_requests_from_usage(usage)
     if web_search_requests is None:
         return 0.0
 

litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py

@@ -1,8 +1,8 @@
 import copy
 import hashlib
 import json
-from collections.abc import AsyncIterator, Iterator, Mapping
-from typing import TYPE_CHECKING, Any, Final, Literal, TypeVar, cast
+from collections.abc import AsyncIterator, Iterator, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Final, Literal, TypeAlias, TypeVar, cast
 
 import litellm
 from litellm.llms.anthropic.experimental_pass_through.utils import (
@@ -18,6 +18,22 @@ TOOL_NAME_PREFIX_LENGTH: Final = OPENAI_MAX_TOOL_NAME_LENGTH - TOOL_NAME_HASH_LE
 PROVIDERS_PROXYING_AN_UNKNOWN_BACKEND: Final = frozenset({"litellm_proxy"})
 
 
+_ANTHROPIC_TOOL_SCHEMA_KEYS: Final = frozenset(
+    {"name", "type", "input_schema", "description", "cache_control", "strict"}
+)
+
+
+def _is_openai_function_tool(tool: Mapping[str, object]) -> bool:
+    return tool.get("type") == "function" and "function" in tool
+
+
+def is_provider_native_tool_dict(tool: Mapping[str, object]) -> bool:
+    if len(tool) != 1:
+        return False
+    key, value = next(iter(tool.items()))
+    return key not in _ANTHROPIC_TOOL_SCHEMA_KEYS and isinstance(value, dict)
+
+
 def truncate_tool_name(name: str) -> str:
     """
     Truncate tool names that exceed OpenAI's 64-character limit.
@@ -99,6 +115,7 @@ from litellm.types.llms.anthropic import (
     ContextManagementResponse,
     MessageBlockDelta,
     MessageDelta,
+    ServerToolUsage,
     StreamingContentBlockDeltaType,
     UsageDelta,
     UsageIteration,
@@ -125,7 +142,9 @@ from litellm.types.llms.openai import (
     ChatCompletionToolMessage,
     ChatCompletionToolParam,
     ChatCompletionToolParamFunctionChunk,
+    ChatCompletionToolReferenceObject,
     ChatCompletionUserMessage,
+    ToolMessageContentPart,
 )
 from litellm.types.utils import Choices, ModelResponse, StreamingChoices, Usage
 
@@ -134,6 +153,8 @@ from .streaming_iterator import AnthropicStreamWrapper
 if TYPE_CHECKING:
     from litellm.types.llms.anthropic import ContentBlockContentBlockDict
 
+ToolResultContent: TypeAlias = str | list[ToolMessageContentPart]
+
 
 class AnthropicAdapter:
     def __init__(self) -> None:
@@ -411,90 +432,13 @@ class LiteLLMAnthropicMessagesAdapter:
                                 self._add_cache_control_if_applicable(content, doc_obj, model)
                                 new_user_content_list.append(doc_obj)
                         elif content.get("type") == "tool_result":
-                            if "content" not in content:
-                                tool_result = ChatCompletionToolMessage(
-                                    role="tool",
-                                    tool_call_id=content.get("tool_use_id", ""),
-                                    content="",
-                                )
-                                self._add_cache_control_if_applicable(content, tool_result, model)
-                                tool_message_list.append(tool_result)
-                            elif isinstance(content.get("content"), str):
-                                tool_result = ChatCompletionToolMessage(
-                                    role="tool",
-                                    tool_call_id=content.get("tool_use_id", ""),
-                                    content=str(content.get("content", "")),
-                                )
-                                self._add_cache_control_if_applicable(content, tool_result, model)
-                                tool_message_list.append(tool_result)
-                            elif isinstance(content.get("content"), list):
-                                # Combine all content items into a single tool message
-                                # to avoid creating multiple tool_result blocks with the same ID
-                                # (each tool_use must have exactly one tool_result)
-                                content_items = list(content.get("content", []))
-
-                                # Single-item text keeps the backward-compatible string format; a single
-                                # image or document becomes a structured image_url part
-                                if len(content_items) == 1:
-                                    c = content_items[0]
-                                    if isinstance(c, str):
-                                        tool_result = ChatCompletionToolMessage(
-                                            role="tool",
-                                            tool_call_id=content.get("tool_use_id", ""),
-                                            content=c,
-                                        )
-                                        self._add_cache_control_if_applicable(content, tool_result, model)
-                                        tool_message_list.append(tool_result)
-                                    elif isinstance(c, dict):
-                                        if c.get("type") == "text":
-                                            tool_result = ChatCompletionToolMessage(
-                                                role="tool",
-                                                tool_call_id=content.get("tool_use_id", ""),
-                                                content=c.get("text", ""),
-                                            )
-                                            self._add_cache_control_if_applicable(content, tool_result, model)
-                                            tool_message_list.append(tool_result)
-                                        elif c.get("type") in ("image", "document"):
-                                            image_part = self._tool_result_image_part(c.get("source"))
-                                            tool_result = ChatCompletionToolMessage(
-                                                role="tool",
-                                                tool_call_id=content.get("tool_use_id", ""),
-                                                content=[image_part]  # mutable-ok: content must be a json list
-                                                if image_part
-                                                else "",
-                                            )
-                                            self._add_cache_control_if_applicable(content, tool_result, model)
-                                            tool_message_list.append(tool_result)
-                                else:
-                                    # For multiple content items, combine into a single tool message
-                                    # with list content to preserve all items while having one tool_use_id
-                                    combined_content_parts: list[
-                                        ChatCompletionTextObject | ChatCompletionImageObject
-                                    ] = []
-                                    for c in content_items:
-                                        if isinstance(c, str):
-                                            combined_content_parts.append(ChatCompletionTextObject(type="text", text=c))
-                                        elif isinstance(c, dict):
-                                            if c.get("type") == "text":
-                                                combined_content_parts.append(
-                                                    ChatCompletionTextObject(
-                                                        type="text",
-                                                        text=c.get("text", ""),
-                                                    )
-                                                )
-                                            elif c.get("type") in ("image", "document"):
-                                                image_part = self._tool_result_image_part(c.get("source"))
-                                                if image_part:
-                                                    combined_content_parts.append(image_part)
-                                    # Create a single tool message with combined content
-                                    if combined_content_parts:
-                                        tool_result = ChatCompletionToolMessage(
-                                            role="tool",
-                                            tool_call_id=content.get("tool_use_id", ""),
-                                            content=combined_content_parts,
-                                        )
-                                        self._add_cache_control_if_applicable(content, tool_result, model)
-                                        tool_message_list.append(tool_result)
+                            tool_result = ChatCompletionToolMessage(
+                                role="tool",
+                                tool_call_id=content.get("tool_use_id", ""),
+                                content=self._tool_result_content(content.get("content")),
+                            )
+                            self._add_cache_control_if_applicable(content, tool_result, model)
+                            tool_message_list.append(tool_result)
 
             if len(tool_message_list) > 0:
                 new_messages.extend(tool_message_list)
@@ -770,6 +714,10 @@ class LiteLLMAnthropicMessagesAdapter:
                 new_tools.append(tool)
                 continue
 
+            if _is_openai_function_tool(tool) or is_provider_native_tool_dict(tool):
+                new_tools.append(cast(ChatCompletionToolParam, tool))  # cast-ok: passed through verbatim to provider
+                continue
+
             raw_name = tool.get("name")
             if raw_name is None or (isinstance(raw_name, str) and not str(raw_name).strip()):
                 original_name = f"litellm_unnamed_tool_{idx}"
@@ -942,6 +890,31 @@ class LiteLLMAnthropicMessagesAdapter:
         )
         return "prompt_cache_key" in (supported_params or ())
 
+    @staticmethod
+    def _target_declares_reasoning_effort(model: str, custom_llm_provider: str | None) -> bool:
+        """Whether the target declares ``reasoning_effort`` among its supported params.
+
+        A Claude-family target is recognized by name, which says nothing about the carrier the
+        provider serving it accepts: Snowflake serves Claude over the Anthropic dialect and
+        declares ``thinking`` alone, so storing the tier there raises before the request reaches
+        the wire.
+
+        Without a resolved provider the tier stays behind, which is what this bridge sent before
+        it carried one at all. Reading the declaration from the model's own prefix instead would
+        resolve the provider through a lookup that runs an OAuth device flow for two of them, and
+        this runs inside a logging callback as well as on the request path.
+
+        Unlike ``_supports_prompt_cache_key`` this does not exclude a provider that proxies an
+        unknown backend, because that provider declares this param and forwards it to a proxy
+        that resolves the real target itself, where a derived cache key has no such guarantee.
+        """
+        if not model or not custom_llm_provider:
+            return False
+        supported_params: Final = litellm.get_supported_openai_params(
+            model=model, custom_llm_provider=custom_llm_provider
+        )
+        return "reasoning_effort" in (supported_params or ())
+
     def _translate_metadata_to_openai(
         self,
         anthropic_message_request: AnthropicMessagesRequest,
@@ -1030,8 +1003,32 @@ class LiteLLMAnthropicMessagesAdapter:
         self,
         anthropic_message_request: AnthropicMessagesRequest,
         new_kwargs: ChatCompletionRequest,
+        *,
+        custom_llm_provider: str | None = None,
     ) -> None:
-        """Translate Anthropic thinking to either thinking or reasoning_effort."""
+        """Translate Anthropic thinking to either thinking or reasoning_effort.
+
+        A Claude-family target keeps ``thinking`` verbatim, since every bridged provider serving one
+        speaks that param. Carrying its adaptive effort tier alongside takes two different params,
+        because the two are not interchangeable at the provider mapping below.
+
+        Bedrock takes ``output_config`` directly, which attaches the tier and leaves ``thinking``
+        alone. Another bridged Claude target takes ``reasoning_effort`` if it declares that param,
+        and used to be sent no tier at all, so an adaptive request arrived byte-identical whichever
+        effort the caller asked for. That tier stays a plain string there, since the summary it
+        would otherwise be wrapped with already travels inside the forwarded ``thinking`` block,
+        and the wrapped dict is rejected outright by some of these providers.
+
+        A target declaring neither carrier keeps its bare ``thinking`` block. Being Claude-family
+        is a fact about the model, not about the params the provider in front of it accepts, so
+        the tier is offered only where the target says it is taken.
+
+        ``reasoning_effort`` is not a substitute for ``output_config`` on the Bedrock side: an
+        application inference profile ARN resolves to neither, so the tier is dropped, and providers
+        that rebuild ``output_config`` from it overwrite a caller-set ``thinking.display`` doing so.
+        An adaptive request with no tier stays untouched either way, so the provider's own default
+        still applies.
+        """
         if "thinking" not in anthropic_message_request:
             return
 
@@ -1040,35 +1037,40 @@ class LiteLLMAnthropicMessagesAdapter:
             return
 
         model: Final = new_kwargs.get("model", "")
-        if self.is_anthropic_claude_model(model) or self.is_bedrock_arn_model(model):
+        is_bedrock_target: Final = model.startswith(("bedrock/", "converse/", "invoke/")) or self.is_bedrock_arn_model(
+            model
+        )
+        is_claude_target: Final = self.is_anthropic_claude_model(model) or self.is_bedrock_arn_model(model)
+        output_config: Final = anthropic_message_request.get("output_config")
+
+        if is_claude_target:
             new_kwargs["thinking"] = thinking
-            # Adaptive thinking without its effort tier makes Bedrock Converse
-            # return zero reasoning blocks, so forward output_config (minus
-            # `format`, already translated to response_format) for Bedrock
-            # targets only: other bridged providers reject the raw param, and
-            # get_llm_provider strips the `bedrock/` prefix before this runs.
-            if model.startswith(("bedrock/", "converse/", "invoke/")) or self.is_bedrock_arn_model(model):
-                claude_output_config: Final = anthropic_message_request.get("output_config")
-                if isinstance(claude_output_config, dict):
-                    effort_config: Final = {k: v for k, v in claude_output_config.items() if k != "format"}
+            if is_bedrock_target:
+                if isinstance(output_config, dict):
+                    effort_config: Final = {k: v for k, v in output_config.items() if k != "format"}
                     if effort_config:
                         new_kwargs["output_config"] = effort_config  # rebind-ok: out-param store like thinking above
+                return
+            if not self._target_declares_reasoning_effort(model, custom_llm_provider):
+                return
+
+        thinking_type: Final = thinking.get("type") if isinstance(thinking, dict) else None
+        declared_effort: Final = (
+            output_config.get("effort") if thinking_type == "adaptive" and isinstance(output_config, dict) else None
+        )
+        if is_claude_target and not declared_effort:
             return
 
-        reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(cast(AnthropicThinkingParam, thinking))
+        reasoning_effort: Final = declared_effort or self.translate_anthropic_thinking_to_reasoning_effort(
+            cast(AnthropicThinkingParam, thinking)
+        )
         if not reasoning_effort:
             return
 
-        thinking_type: Final = thinking.get("type") if isinstance(thinking, dict) else None
-
-        # For adaptive thinking, override with output_config.effort if available
-        if thinking_type == "adaptive":
-            output_config: Final = anthropic_message_request.get("output_config")
-            if isinstance(output_config, dict) and output_config.get("effort"):
-                reasoning_effort = output_config["effort"]
-
-        new_kwargs["reasoning_effort"] = self._apply_reasoning_summary_wrapping(
-            reasoning_effort, cast(dict[str, object], thinking)
+        new_kwargs["reasoning_effort"] = (
+            reasoning_effort
+            if is_claude_target
+            else self._apply_reasoning_summary_wrapping(reasoning_effort, cast(dict[str, object], thinking))
         )
 
     def _translate_output_format_to_openai(
@@ -1164,6 +1166,7 @@ class LiteLLMAnthropicMessagesAdapter:
         self._translate_thinking_to_openai(
             anthropic_message_request=anthropic_message_request,
             new_kwargs=new_kwargs,
+            custom_llm_provider=custom_llm_provider,
         )
         ## CONVERT STOP_SEQUENCES
         self._translate_stop_sequences_to_openai(
@@ -1209,6 +1212,39 @@ class LiteLLMAnthropicMessagesAdapter:
 
         return None
 
+    def _tool_result_content(self, raw_content: object) -> ToolResultContent:
+        if isinstance(raw_content, str):
+            return raw_content
+        if not isinstance(raw_content, list):
+            return ""
+        items: Final = cast(Sequence[object], raw_content)  # cast-ok: untrusted client payload
+        parts: Final = tuple(part for part in (self._tool_result_part(item) for item in items) if part is not None)
+        match parts:
+            case ():
+                return ""
+            case ({"type": "text", "text": str(text)},):
+                return text
+            case _:
+                return list(parts)  # mutable-ok: content must be a json list
+
+    def _tool_result_part(self, item: object) -> ToolMessageContentPart | None:
+        if isinstance(item, str):
+            return ChatCompletionTextObject(type="text", text=item)
+        if not isinstance(item, dict):
+            return None
+        block: Final = cast(Mapping[str, object], item)  # cast-ok: untrusted client payload
+        match block.get("type"):
+            case "text":
+                return ChatCompletionTextObject(type="text", text=str(block.get("text") or ""))
+            case "image" | "document":
+                return self._tool_result_image_part(block.get("source"))
+            case "tool_reference":
+                return ChatCompletionToolReferenceObject(
+                    type="tool_reference", tool_name=str(block.get("tool_name") or "")
+                )
+            case _:
+                return None
+
     def _tool_result_image_part(self, image_source: object) -> ChatCompletionImageObject | None:
         if not isinstance(image_source, dict):
             return None
@@ -1354,10 +1390,22 @@ class LiteLLMAnthropicMessagesAdapter:
             return explicit_value
         return cls._first_positive_prompt_tokens_detail_value(usage, ("cache_creation_tokens", "cache_write_tokens"))
 
+    @classmethod
+    def _get_web_search_request_count(cls, usage: Usage) -> int:
+        from litellm.litellm_core_utils.llm_cost_calc.utils import (
+            get_web_search_requests_from_usage,
+        )
+
+        from_server_tool_use: Final = cls._positive_int(get_web_search_requests_from_usage(usage))
+        if from_server_tool_use > 0:
+            return from_server_tool_use
+        return cls._first_positive_prompt_tokens_detail_value(usage, ("web_search_requests",))
+
     @classmethod
     def _translate_openai_usage_to_anthropic_usage_delta(cls, usage: Usage) -> UsageDelta:
         cache_read_input_tokens: Final = cls._get_cache_read_input_tokens(usage)
         cache_creation_input_tokens: Final = cls._get_cache_creation_input_tokens(usage)
+        web_search_requests: Final = cls._get_web_search_request_count(usage)
         input_tokens: Final = max(
             (usage.prompt_tokens or 0) - cache_read_input_tokens - cache_creation_input_tokens,
             0,
@@ -1371,6 +1419,11 @@ class LiteLLMAnthropicMessagesAdapter:
             usage_delta["cache_creation_input_tokens"] = cache_creation_input_tokens
         if cache_read_input_tokens > 0:
             usage_delta["cache_read_input_tokens"] = cache_read_input_tokens
+        if web_search_requests > 0:
+            return UsageDelta(
+                **usage_delta,
+                server_tool_use=ServerToolUsage(web_search_requests=web_search_requests),
+            )
         return usage_delta
 
     @classmethod

litellm/llms/anthropic/experimental_pass_through/utils.py

@@ -1,4 +1,6 @@
 import os
+from collections.abc import Mapping
+from types import MappingProxyType
 from typing import Final
 
 import litellm
@@ -23,6 +25,29 @@ def is_reasoning_auto_summary_enabled() -> bool:
     return litellm.reasoning_auto_summary or os.getenv("LITELLM_REASONING_AUTO_SUMMARY", "false").lower() == "true"
 
 
+_DECLARED_DEGRADATION_CHAINS: Final[Mapping[str, tuple[str, ...]]] = MappingProxyType(
+    {"max": ("max", "xhigh", "high"), "xhigh": ("xhigh", "high"), "minimal": ("minimal", "low")}
+)
+
+
+def _effort_from_declaration(model_info: ModelInfo, effort: str) -> str | None:
+    """A declared level set is the WHOLE answer for this gate, so a level it omits degrades even
+    where a per-level flag would have allowed it. Honoring both would let /model_group/info and
+    this path disagree about the same entry. None means the entry declares nothing, and the flag
+    chain below decides as before.
+
+    A declaration that omits every level in a chain still lands on that chain's terminal, which can
+    itself be undeclared. Picking a nearer declared level instead would need a strength ordering,
+    and the advertisement order is presentation only by design, so the terminal stays the answer."""
+    from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts
+
+    declared: Final = declared_reasoning_efforts(model_info)
+    if declared is None:
+        return None
+    chain: Final = _DECLARED_DEGRADATION_CHAINS[effort]
+    return next((level for level in chain if level in declared), chain[-1])
+
+
 def normalize_reasoning_effort_value(
     effort: str,
     model: str,
@@ -48,6 +73,10 @@ def normalize_reasoning_effort_value(
     except Exception:
         model_info = None
 
+    declared_effort: Final = _effort_from_declaration(model_info, effort) if model_info is not None else None
+    if declared_effort is not None:
+        return declared_effort
+
     if effort == "max":
         if model_info and model_info.get("supports_max_reasoning_effort"):
             return "max"

litellm/llms/azure/chat/gpt_transformation.py

@@ -4,6 +4,7 @@ from httpx._models import Headers, Response
 
 import litellm
 from litellm.litellm_core_utils.prompt_templates.common_utils import (
+    drop_tool_reference_parts_from_tool_messages,
     hoist_images_from_tool_messages,
 )
 from litellm.litellm_core_utils.prompt_templates.factory import (
@@ -252,7 +253,8 @@ class AzureOpenAIConfig(BaseConfig):
         litellm_params: dict,
         headers: dict,
     ) -> dict:
-        azure_messages: Final = convert_to_azure_openai_messages(hoist_images_from_tool_messages(messages))
+        stripped_messages: Final = drop_tool_reference_parts_from_tool_messages(messages)
+        azure_messages: Final = convert_to_azure_openai_messages(hoist_images_from_tool_messages(stripped_messages))
         return {
             "model": model,
             "messages": azure_messages,

litellm/llms/base_llm/audio_transcription/transformation.py

@@ -40,6 +40,16 @@ class BaseAudioTranscriptionConfig(BaseConfig, ABC):
     def get_supported_openai_params(self, model: str) -> list[OpenAIAudioTranscriptionOptionalParams]:
         pass
 
+    @property
+    def supports_subtitle_synthesis(self) -> bool:
+        """
+        Opt-in for providers without a native srt/vtt response body: when True
+        and the user asked for response_format srt/vtt, the http handler
+        synthesizes the subtitle document from the word timestamps the
+        provider's TranscriptionResponse carries in `words`.
+        """
+        return False
+
     def get_complete_url(
         self,
         api_base: str | None,

litellm/llms/base_llm/guardrail_translation/utils.py

@@ -209,9 +209,20 @@ def openai_tool_name(tool: object) -> str | None:
     return flat_name if isinstance(flat_name, str) else None
 
 
+def anthropic_tool_names(tool: object) -> tuple[str, ...]:
+    """Every name a /v1/messages tool dict can act under: the flat Anthropic ``name`` plus
+    ``function.name`` for OpenAI-format tools the bridge forwards verbatim. Allowlist checks
+    must see both, or a decoy flat name could smuggle a disallowed ``function.name`` through."""
+    if not isinstance(tool, dict):
+        return ()
+    function: Final = tool.get("function") if tool.get("type") == "function" else None
+    function_name: Final = function.get("name") if isinstance(function, dict) else None
+    return tuple(name for name in (tool.get("name"), function_name) if isinstance(name, str) and name)
+
+
 def anthropic_tool_name(tool: object) -> str | None:
-    name: Final = tool.get("name") if isinstance(tool, dict) else None
-    return name if isinstance(name, str) else None
+    names: Final = anthropic_tool_names(tool)
+    return names[0] if names else None
 
 
 def merge_returned_tools_into_request_tools(

litellm/llms/base_llm/realtime/transformation.py

@@ -5,6 +5,7 @@ import httpx
 
 from litellm.types.llms.openai import OpenAIRealtimeStreamSessionEvents
 from litellm.types.realtime import (
+    RealtimeInputAudioTranscriptionUsage,
     RealtimeResponseTransformInput,
     RealtimeResponseTypedDict,
 )
@@ -70,6 +71,9 @@ class BaseRealtimeConfig(ABC):
     def session_configuration_request(self, model: str) -> str | None:  # message sent to setup the realtime session
         return None
 
+    def unbilled_usage_on_session_close(self, model: str) -> RealtimeInputAudioTranscriptionUsage | None:
+        return None
+
     def transform_session_created_event(
         self,
         model: str,

litellm/llms/bedrock/base_aws_llm.py

@@ -1434,9 +1434,12 @@ class BaseAWSLLM:
         data: str | bytes,
         headers: dict,
         api_key: str | None = None,
+        supports_bearer_token: bool = True,
     ) -> AWSPreparedRequest:
-        if api_key is not None:
-            aws_bearer_token: str | None = api_key
+        if not supports_bearer_token:
+            aws_bearer_token: str | None = None
+        elif api_key is not None:
+            aws_bearer_token = api_key
         else:
             aws_bearer_token = get_secret_str("AWS_BEARER_TOKEN_BEDROCK")
 

litellm/llms/bedrock/rerank/handler.py

@@ -138,11 +138,6 @@ class BedrockRerankHandler(BaseAWSLLM):
         data: dict,
         optional_params: dict,
     ) -> BedrockPreparedRequest:
-        try:
-            from botocore.auth import SigV4Auth
-            from botocore.awsrequest import AWSRequest
-        except ImportError:
-            raise ImportError("Missing boto3 to call bedrock. Run 'pip install boto3'.")
         boto3_credentials_info: Final = self._get_boto_credentials_from_optional_params(optional_params, model)
 
         ### SET RUNTIME ENDPOINT ###
@@ -153,24 +148,21 @@ class BedrockRerankHandler(BaseAWSLLM):
         )
         proxy_endpoint_url = proxy_endpoint_url.replace("bedrock-runtime", "bedrock-agent-runtime")
         proxy_endpoint_url = f"{proxy_endpoint_url}/rerank"
-        sigv4: Final = SigV4Auth(
-            boto3_credentials_info.credentials,
-            "bedrock",
-            boto3_credentials_info.aws_region_name,
-        )
-        # Make POST Request
-        body: Final = json.dumps(data).encode("utf-8")
 
+        body: Final = json.dumps(data).encode("utf-8")
         headers = {"Content-Type": "application/json"}
         if extra_headers is not None:
             headers = {"Content-Type": "application/json", **extra_headers}
-        request: Final = AWSRequest(method="POST", url=proxy_endpoint_url, data=body, headers=headers)
-        sigv4.add_auth(request)
-        if (
-            extra_headers is not None and "Authorization" in extra_headers
-        ):  # prevent sigv4 from overwriting the auth header
-            request.headers["Authorization"] = extra_headers["Authorization"]
-        prepped: Final = request.prepare()
+
+        prepped: Final = self.get_request_headers(
+            credentials=boto3_credentials_info.credentials,
+            aws_region_name=boto3_credentials_info.aws_region_name,
+            extra_headers=extra_headers,
+            endpoint_url=proxy_endpoint_url,
+            data=body,
+            headers=headers,
+            supports_bearer_token=False,
+        )
 
         return BedrockPreparedRequest(
             endpoint_url=proxy_endpoint_url,

litellm/llms/custom_httpx/llm_http_handler.py

@@ -25,6 +25,10 @@ from litellm.litellm_core_utils.agentic_loop_settings import (
     validated_max_agentic_loops,
 )
 from litellm.litellm_core_utils.asyncify import run_async_function
+from litellm.litellm_core_utils.audio_utils.subtitle_utils import (
+    SUBTITLE_RESPONSE_FORMATS,
+    synthesize_subtitle_document,
+)
 from litellm.litellm_core_utils.llm_request_utils import serialize_multipart_form_fields
 from litellm.litellm_core_utils.realtime_errors import realtime_error_event, websocket_close_reason
 from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
@@ -1296,9 +1300,23 @@ class BaseLLMHTTPHandler:
         api_key: str | None,
     ) -> TranscriptionResponse:
         """Shared logic for transforming audio transcription responses."""
-        return provider_config.transform_audio_transcription_response(
+        transformed: Final = provider_config.transform_audio_transcription_response(
             raw_response=response,
         )
+        if not provider_config.supports_subtitle_synthesis:
+            return transformed
+        requested_format: Final = optional_params.get("response_format")
+        if not isinstance(requested_format, str) or requested_format not in SUBTITLE_RESPONSE_FORMATS:
+            return transformed
+        document: Final = synthesize_subtitle_document(
+            words=transformed.get("words"),
+            response_format=requested_format,
+        )
+        if document is not None:
+            transformed.text = document
+        if "words" in transformed:
+            delattr(transformed, "words")
+        return transformed
 
     def audio_transcriptions(
         self,

litellm/llms/dashscope/image_generation/transformation.py

@@ -11,7 +11,7 @@ Request format:
     "input": {
         "messages": [{"role": "user", "content": [{"text": "<prompt>"}]}]
     },
-    "parameters": {"size": "1024*1024", ...}
+    "parameters": {"size": "1024*1024", "n": 1, ...}
 }
 
 Response format:
@@ -19,7 +19,7 @@ Response format:
     "output": {
         "choices": [{"message": {"content": [{"image": "<url>"}]}}]
     },
-    "usage": {"input_tokens": 0, "output_tokens": 0, "width": 1024, "height": 1024, "image_count": 1}
+    "usage": {"output_width": 1024, "output_height": 1024, "output_image_count": 1}
 }
 """
 
@@ -46,6 +46,8 @@ else:
 
 DEFAULT_API_BASE: Final = "https://dashscope-intl.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation"
 
+CHAT_COMPATIBLE_MODE_PATH: Final = "/compatible-mode/v1"
+
 # Maps OpenAI size strings (WxH) to DashScope size strings (W*H)
 OPENAI_TO_DASHSCOPE_SIZE: Final[dict] = {
     "256x256": "256*256",
@@ -59,7 +61,8 @@ OPENAI_TO_DASHSCOPE_SIZE: Final[dict] = {
 
 class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
     """
-    Configuration for DashScope image generation (qwen-image-2.0, qwen-image-2.0-pro).
+    Configuration for DashScope image generation (qwen-image-2.0, qwen-image-2.0-pro,
+    qwen-image-3.0, qwen-image-3.0-pro).
     """
 
     def get_supported_openai_params(self, model: str) -> list[OpenAIImageGenerationOptionalParams]:
@@ -82,8 +85,8 @@ class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
             if k == "size":
                 # Convert "WxH" → "W*H"
                 mapped["size"] = OPENAI_TO_DASHSCOPE_SIZE.get(v, v.replace("x", "*"))
-            elif k == "n":
-                mapped["image_count"] = v
+            else:
+                mapped[k] = v
         return mapped
 
     def get_complete_url(
@@ -95,7 +98,10 @@ class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
         litellm_params: dict,
         stream: bool | None = None,
     ) -> str:
-        return api_base or get_secret_str("DASHSCOPE_API_BASE_IMAGE") or DEFAULT_API_BASE
+        image_api_base: Final = (
+            api_base if api_base and not api_base.rstrip("/").endswith(CHAT_COMPATIBLE_MODE_PATH) else None
+        )
+        return image_api_base or get_secret_str("DASHSCOPE_API_BASE_IMAGE") or DEFAULT_API_BASE
 
     def validate_environment(
         self,

litellm/llms/gemini/audio_transcription/transformation.py

@@ -0,0 +1,256 @@
+import base64
+from collections.abc import Mapping, Sequence
+from typing import Final
+
+from httpx import Headers, Response
+
+from litellm.litellm_core_utils.audio_utils.subtitle_utils import SUBTITLE_RESPONSE_FORMATS
+from litellm.litellm_core_utils.audio_utils.utils import (
+    normalize_transcription_language_to_bcp47,
+    process_audio_file,
+)
+from litellm.llms.base_llm.audio_transcription.transformation import (
+    AudioTranscriptionRequestData,
+    BaseAudioTranscriptionConfig,
+)
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.gemini.common_utils import GeminiError, GeminiModelInfo
+from litellm.types.llms.gemini_audio_transcription import (
+    GeminiTranscriptionAudioInput,
+    GeminiTranscriptionConfig,
+    GeminiTranscriptionInteractionRequest,
+    GeminiTranscriptionInteractionResponse,
+    GeminiTranscriptionWordAnnotation,
+)
+from litellm.types.llms.openai import (
+    AllMessageValues,
+    OpenAIAudioTranscriptionOptionalParams,
+)
+from litellm.types.utils import (
+    FileTypes,
+    TranscriptionResponse,
+    TranscriptionUsageInputTokenDetailsObject,
+    TranscriptionUsageTokensObject,
+)
+
+INTERACTIONS_API_REVISION: Final = "2026-05-20"
+WORD_INFO_ANNOTATION_TYPE: Final = "word_info"
+
+
+class GeminiAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
+    """
+    Maps OpenAI /v1/audio/transcriptions onto the Gemini Interactions API
+    (POST /v1beta/interactions) for transcription models like
+    gemini-3.5-transcribe. https://ai.google.dev/gemini-api/docs/transcribe
+    """
+
+    def get_supported_openai_params(
+        self, model: str
+    ) -> list[OpenAIAudioTranscriptionOptionalParams]:  # mutable-ok: BaseAudioTranscriptionConfig signature
+        return ["language", "response_format", "timestamp_granularities"]  # mutable-ok: base contract returns a list
+
+    @property
+    def supports_subtitle_synthesis(self) -> bool:
+        return True
+
+    def map_openai_params(
+        self,
+        non_default_params: Mapping[str, object],
+        optional_params: Mapping[str, object],
+        model: str,
+        drop_params: bool,
+    ) -> dict:  # mutable-ok: BaseAudioTranscriptionConfig signature
+        supported_params: Final = frozenset(self.get_supported_openai_params(model))
+        accepted: Final = tuple((k, v) for k, v in non_default_params.items() if k in supported_params)
+        return dict((*optional_params.items(), *accepted))  # mutable-ok: base contract returns a plain dict
+
+    def get_error_class(
+        self,
+        error_message: str,
+        status_code: int,
+        headers: dict | Headers,  # mutable-ok: base signature and BaseLLMException take dict | Headers
+    ) -> BaseLLMException:
+        return GeminiError(status_code=status_code, message=error_message, headers=headers)
+
+    def validate_environment(
+        self,
+        headers: Mapping[str, str],
+        model: str,
+        messages: Sequence[AllMessageValues],
+        optional_params: Mapping[str, object],
+        litellm_params: Mapping[str, object],
+        api_key: str | None = None,
+        api_base: str | None = None,
+    ) -> dict:  # mutable-ok: BaseAudioTranscriptionConfig signature
+        resolved_api_key: Final = GeminiModelInfo.get_api_key(api_key)
+        if not resolved_api_key:
+            raise GeminiError(
+                status_code=401,
+                message="Google API key is required. Set GOOGLE_API_KEY or GEMINI_API_KEY environment variable.",
+            )
+        return {  # mutable-ok: the http handler passes these headers straight to httpx
+            **headers,
+            "Content-Type": "application/json",
+            "x-goog-api-key": resolved_api_key,
+            "Api-Revision": INTERACTIONS_API_REVISION,
+        }
+
+    def get_complete_url(
+        self,
+        api_base: str | None,
+        api_key: str | None,
+        model: str,
+        optional_params: Mapping[str, object],
+        litellm_params: Mapping[str, object],
+        stream: bool | None = None,
+    ) -> str:
+        resolved_api_base: Final = GeminiModelInfo.get_api_base(api_base)
+        return f"{resolved_api_base}/v1beta/interactions"
+
+    def transform_audio_transcription_request(
+        self,
+        model: str,
+        audio_file: FileTypes,
+        optional_params: Mapping[str, object],
+        litellm_params: Mapping[str, object],
+    ) -> AudioTranscriptionRequestData:
+        processed_audio: Final = process_audio_file(audio_file)
+        audio_input: Final = GeminiTranscriptionAudioInput(
+            type="audio",
+            data=base64.b64encode(processed_audio.file_content).decode("utf-8"),
+            mime_type=processed_audio.content_type,
+        )
+        request: Final = _build_interaction_request(
+            model=model,
+            audio_input=audio_input,
+            transcription_config=_build_transcription_config(optional_params),
+        )
+        return AudioTranscriptionRequestData(data=dict(request))  # mutable-ok: AudioTranscriptionRequestData wants dict
+
+    def transform_audio_transcription_response(
+        self,
+        raw_response: Response,
+    ) -> TranscriptionResponse:
+        try:
+            response_json: Final = raw_response.json()
+        except ValueError:
+            raise GeminiError(
+                status_code=raw_response.status_code,
+                message=f"Received non-JSON response from Gemini Interactions API: {raw_response.text}",
+            )
+        parsed: Final = GeminiTranscriptionInteractionResponse.model_validate(response_json)
+        if parsed.status != "completed":
+            raise GeminiError(
+                status_code=raw_response.status_code,
+                message=f"Gemini transcription interaction did not complete (status={parsed.status}): {raw_response.text}",
+            )
+        text_contents: Final = tuple(
+            content
+            for step in parsed.steps
+            for content in step.content
+            if content.type == "text" and content.text is not None
+        )
+        response: Final = TranscriptionResponse(text=" ".join(content.text or "" for content in text_contents))
+        response["task"] = "transcribe"
+        words: Final = tuple(
+            word
+            for content in text_contents
+            for annotation in content.annotations
+            if (word := _annotation_to_word(annotation)) is not None
+        )
+        if words:
+            response["words"] = list(words)  # mutable-ok: verbose_json words is a JSON array
+            last_word_end: Final = words[-1].get("end")
+            if last_word_end is not None:
+                response["duration"] = last_word_end
+        if parsed.usage is not None:
+            audio_tokens: Final = sum(
+                by_modality.tokens
+                for by_modality in parsed.usage.input_tokens_by_modality
+                if by_modality.modality == "audio"
+            )
+            response.usage = TranscriptionUsageTokensObject(
+                type="tokens",
+                input_tokens=parsed.usage.total_input_tokens,
+                output_tokens=parsed.usage.total_output_tokens,
+                total_tokens=parsed.usage.total_tokens,
+                input_token_details=TranscriptionUsageInputTokenDetailsObject(
+                    audio_tokens=audio_tokens,
+                    text_tokens=parsed.usage.total_input_tokens - audio_tokens,
+                ),
+            )
+        return response
+
+
+_EMPTY_TRANSCRIPTION_CONFIG: Final[GeminiTranscriptionConfig] = {}
+_WORD_TIMESTAMP_CONFIG: Final[GeminiTranscriptionConfig] = {
+    "mode": {
+        "type": "verbatim",
+        "timestamp_granularities": ("word",),
+        "diarization_mode": "speaker",
+    },
+}
+
+
+def _build_interaction_request(
+    model: str,
+    audio_input: GeminiTranscriptionAudioInput,
+    transcription_config: GeminiTranscriptionConfig,
+) -> GeminiTranscriptionInteractionRequest:
+    if not transcription_config:
+        bare_request: Final[GeminiTranscriptionInteractionRequest] = {
+            "model": model.removeprefix("gemini/"),
+            "input": (audio_input,),
+        }
+        return bare_request
+    configured_request: Final[GeminiTranscriptionInteractionRequest] = {
+        "model": model.removeprefix("gemini/"),
+        "input": (audio_input,),
+        "generation_config": {"transcription_config": transcription_config},
+    }
+    return configured_request
+
+
+def _language_config(language: object) -> GeminiTranscriptionConfig:
+    if not isinstance(language, str) or not language:
+        return _EMPTY_TRANSCRIPTION_CONFIG
+    language_config: Final[GeminiTranscriptionConfig] = {
+        "language_codes": (normalize_transcription_language_to_bcp47(language),),
+    }
+    return language_config
+
+
+def _timestamp_config(timestamp_granularities: object, response_format: object) -> GeminiTranscriptionConfig:
+    wants_word_timestamps: Final = (
+        isinstance(timestamp_granularities, list) and "word" in timestamp_granularities
+    ) or (isinstance(response_format, str) and response_format in SUBTITLE_RESPONSE_FORMATS)
+    return _WORD_TIMESTAMP_CONFIG if wants_word_timestamps else _EMPTY_TRANSCRIPTION_CONFIG
+
+
+def _build_transcription_config(optional_params: Mapping[str, object]) -> GeminiTranscriptionConfig:
+    transcription_config: Final[GeminiTranscriptionConfig] = {
+        **_language_config(optional_params.get("language")),
+        **_timestamp_config(optional_params.get("timestamp_granularities"), optional_params.get("response_format")),
+    }
+    return transcription_config
+
+
+def _annotation_to_word(annotation: GeminiTranscriptionWordAnnotation) -> Mapping[str, str | float] | None:
+    if annotation.type != WORD_INFO_ANNOTATION_TYPE or annotation.text is None:
+        return None
+    entries: Final = (
+        ("word", annotation.text),
+        ("start", _parse_offset_seconds(annotation.start_offset)),
+        ("end", _parse_offset_seconds(annotation.end_offset)),
+        ("speaker", annotation.speaker),
+    )
+    return {key: value for key, value in entries if value is not None}  # mutable-ok: word entries serialize to JSON
+
+
+def _parse_offset_seconds(offset: str | None) -> float | None:
+    if offset is None or not offset.endswith("s"):
+        return None
+    try:
+        return float(offset[:-1])
+    except ValueError:
+        return None

litellm/llms/gemini/chat/transformation.py

@@ -8,7 +8,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
 from litellm.litellm_core_utils.prompt_templates.image_handling import (
     convert_url_to_base64,
 )
-from litellm.types.llms.openai import AllMessageValues, ChatCompletionFileObject
+from litellm.types.llms.openai import AllMessageValues, ChatCompletionFileObject, ChatCompletionImageObject
 from litellm.types.llms.vertex_ai import ContentType, PartType
 from litellm.utils import supports_reasoning
 
@@ -16,6 +16,13 @@ from ...vertex_ai.gemini.transformation import _gemini_convert_messages_with_his
 from ...vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
 
 
+def _image_url_fields(img_element: ChatCompletionImageObject) -> tuple[str | None, str | None, str | None]:
+    image_value: Final = img_element.get("image_url")
+    if isinstance(image_value, dict):
+        return image_value.get("url"), image_value.get("format"), image_value.get("detail")
+    return image_value, None, None
+
+
 class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
     """
     Reference: https://ai.google.dev/api/rest/v1beta/GenerationConfig
@@ -118,16 +125,8 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
                 _parts: list[PartType] = []
                 for element in _message_content:
                     if element.get("type") == "image_url":
-                        img_element = element
-                        _image_url: str | None = None
-                        format: str | None = None
-                        detail: str | None = None
-                        if isinstance(img_element.get("image_url"), dict):
-                            _image_url = img_element["image_url"].get("url")
-                            format = img_element["image_url"].get("format")
-                            detail = img_element["image_url"].get("detail")
-                        else:
-                            _image_url = img_element.get("image_url")
+                        img_element = cast(ChatCompletionImageObject, element)  # cast-ok: runtime type tag checked
+                        _image_url, format, detail = _image_url_fields(img_element)
                         if _image_url and "https://" in _image_url:
                             image_obj = convert_to_anthropic_image_obj(_image_url, format=format)
                             converted_image_url = convert_generic_image_chunk_to_openai_image_obj(image_obj)

litellm/llms/gemini/cost_calculator.py

@@ -39,28 +39,35 @@ def cost_per_web_search_request(usage: "Usage", model_info: "ModelInfo") -> floa
     ``model_info`` when available, falling back to $0.035 for models not
     yet updated in the pricing JSON.
     """
+    from litellm.litellm_core_utils.llm_cost_calc.utils import (
+        get_web_search_requests_from_usage,
+    )
     from litellm.types.utils import PromptTokensDetailsWrapper
 
     _DEFAULT_COST: Final = 35e-3
     search_costs: Final = model_info.get("search_context_cost_per_query") or {}
     _cost: Final = search_costs.get("search_context_size_medium", _DEFAULT_COST)
 
-    number_of_web_search_requests = 0
-    if (
-        usage is not None
-        and usage.prompt_tokens_details is not None
-        and isinstance(usage.prompt_tokens_details, PromptTokensDetailsWrapper)
-        and hasattr(usage.prompt_tokens_details, "web_search_requests")
-        and usage.prompt_tokens_details.web_search_requests is not None
-    ):
-        number_of_web_search_requests = usage.prompt_tokens_details.web_search_requests
+    requests_from_prompt_details: Final = (
+        usage.prompt_tokens_details.web_search_requests
+        if (
+            usage is not None
+            and usage.prompt_tokens_details is not None
+            and isinstance(usage.prompt_tokens_details, PromptTokensDetailsWrapper)
+            and hasattr(usage.prompt_tokens_details, "web_search_requests")
+            and usage.prompt_tokens_details.web_search_requests is not None
+        )
+        else None
+    )
+    requests_from_server_tool_use: Final = get_web_search_requests_from_usage(usage)
+    number_of_web_search_requests: Final = requests_from_prompt_details or requests_from_server_tool_use or 0
 
-    # per_prompt billing: clamp to 1 (flat fee per grounded API call)
     billing_mode: Final = model_info.get("web_search_billing_unit") or "per_prompt"
-    if number_of_web_search_requests > 0 and billing_mode == "per_prompt":
-        number_of_web_search_requests = 1
+    billable_requests: Final = (
+        1 if (number_of_web_search_requests > 0 and billing_mode == "per_prompt") else number_of_web_search_requests
+    )
 
-    return _cost * number_of_web_search_requests
+    return _cost * billable_requests
 
 
 GOOGLE_MAPS_GROUNDING_DEFAULT_COST_PER_QUERY: Final = 14e-3

litellm/llms/gemini/realtime/transformation.py

@@ -4,7 +4,7 @@ This file contains the transformation logic for the Gemini realtime API.
 
 import json
 from collections import OrderedDict
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
 from typing import Any, Final, cast
 
 import litellm
@@ -53,6 +53,7 @@ from litellm.types.llms.vertex_ai import (
 )
 from litellm.types.realtime import (
     ALL_DELTA_TYPES,
+    RealtimeInputAudioTranscriptionUsage,
     RealtimeModalityResponseTransformOutput,
     RealtimeResponseTransformInput,
     RealtimeResponseTypedDict,
@@ -95,6 +96,18 @@ def _gemini_live_speech_config(voice: object) -> Mapping[str, object] | None:
     return VertexGeminiConfig()._map_audio_params({"voice": voice})
 
 
+# Google bills Live transcription at an estimated 25 audio tokens/sec of input and
+# 175 text tokens/min of output (ai.google.dev/gemini-api/docs/pricing).
+GEMINI_LIVE_TRANSCRIBE_AUDIO_TOKENS_PER_SECOND: Final = 25
+GEMINI_LIVE_TRANSCRIBE_OUTPUT_TEXT_TOKENS_PER_MINUTE: Final = 175
+PCM16_INPUT_AUDIO_BYTES_PER_SECOND: Final = 48000
+
+
+def _base64_decoded_byte_count(data: str) -> int:
+    padding: Final = 2 if data.endswith("==") else 1 if data.endswith("=") else 0
+    return max(len(data) * 3 // 4 - padding, 0)
+
+
 class GeminiRealtimeConfig(BaseRealtimeConfig):
     _TOOL_CALL_ID_TO_NAME_MAX = 256  # LRU cap for call_id→name mapping
 
@@ -104,6 +117,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
         # Gemini Live sometimes emits usageMetadata in a standalone frame between
         # turns; buffer it here so the next response.done carries the token counts.
         self._pending_usage_metadata: dict | None = None
+        self._unbilled_input_audio_bytes: int = 0
 
     def is_setup_message(self, msg_obj: dict) -> bool:
         return "setup" in msg_obj
@@ -384,17 +398,25 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
         return bool(entry.get("gemini_native_audio") or entry.get("gemini_audio_only_live"))
 
     @staticmethod
-    def _coerce_response_modalities(model: str, modalities: list[Any]) -> list[str]:
-        """Map unsupported TEXT responseModalities to AUDIO for audio-only Live models."""
-        normalized: Final = [
+    def _is_text_only_live_model(model: str) -> bool:
+        return GeminiRealtimeConfig._model_cost_entry(model).get("mode") == "audio_transcription"
+
+    @staticmethod
+    def _default_response_modality(model: str) -> GeminiResponseModalities:
+        return "TEXT" if GeminiRealtimeConfig._is_text_only_live_model(model) else "AUDIO"
+
+    @staticmethod
+    def _coerce_response_modalities(model: str, modalities: Sequence[Any]) -> tuple[str, ...]:
+        """Swap responseModalities a Live model cannot produce: TEXT to AUDIO for
+        audio-only models, AUDIO to TEXT for text-only ones (e.g. transcribe-live)."""
+        normalized: Final = tuple(
             modality.upper() if isinstance(modality, str) else str(modality).upper() for modality in modalities
-        ]
-        if not GeminiRealtimeConfig._is_audio_only_live_model(model):
-            return normalized
-        if "TEXT" not in normalized:
-            return normalized
-        without_text: Final = [modality for modality in normalized if modality != "TEXT"]
-        return without_text if without_text else ["AUDIO"]
+        )
+        if GeminiRealtimeConfig._is_audio_only_live_model(model) and "TEXT" in normalized:
+            return tuple(modality for modality in normalized if modality != "TEXT") or ("AUDIO",)
+        if GeminiRealtimeConfig._is_text_only_live_model(model) and "AUDIO" in normalized:
+            return tuple(modality for modality in normalized if modality != "AUDIO") or ("TEXT",)
+        return normalized
 
     @staticmethod
     def _finalize_gemini_live_setup(model: str, setup: dict[str, Any]) -> dict[str, Any]:
@@ -436,7 +458,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
 
         if session_configuration_request is None:
             generation_config: Final = new_overrides.setdefault("generationConfig", {})
-            generation_config.setdefault("responseModalities", ["AUDIO"])
+            generation_config.setdefault("responseModalities", [GeminiRealtimeConfig._default_response_modality(model)])
             new_overrides.setdefault("inputAudioTranscription", {})
             new_overrides["model"] = f"models/{model}"
             verbose_logger.debug("Gemini Realtime: Sending initial setup with tools to backend")
@@ -558,9 +580,10 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
             return self._handle_conversation_item(json_message)
 
         if msg_type == "input_audio_buffer.append":
-            realtime_input_dict["audio"] = HttpxBlobType(
-                mimeType=self.get_audio_mime_type(), data=json_message["audio"]
-            )
+            audio_b64: Final = json_message["audio"]
+            if isinstance(audio_b64, str):
+                self._unbilled_input_audio_bytes += _base64_decoded_byte_count(audio_b64)
+            realtime_input_dict["audio"] = HttpxBlobType(mimeType=self.get_audio_mime_type(), data=audio_b64)
 
             realtime_input_dict = cast(
                 BidiGenerateContentRealtimeInput,
@@ -1151,6 +1174,26 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
             raise ValueError(f"Unknown openai event: {key}, value: {value}")
         return openai_event
 
+    def _consume_input_transcription_usage_estimate(self, model: str) -> RealtimeInputAudioTranscriptionUsage | None:
+        """Gemini Live sends no usageMetadata for transcribe sessions; estimate billing from streamed audio duration."""
+        if self._unbilled_input_audio_bytes <= 0 or not self._is_text_only_live_model(model):
+            return None
+        audio_seconds: Final = self._unbilled_input_audio_bytes / PCM16_INPUT_AUDIO_BYTES_PER_SECOND
+        self._unbilled_input_audio_bytes = 0
+        audio_tokens: Final = round(audio_seconds * GEMINI_LIVE_TRANSCRIBE_AUDIO_TOKENS_PER_SECOND)
+        output_tokens: Final = round(audio_seconds * GEMINI_LIVE_TRANSCRIBE_OUTPUT_TEXT_TOKENS_PER_MINUTE / 60)
+        usage: Final[RealtimeInputAudioTranscriptionUsage] = {
+            "type": "tokens",
+            "input_tokens": audio_tokens,
+            "output_tokens": output_tokens,
+            "total_tokens": audio_tokens + output_tokens,
+            "input_token_details": {"text_tokens": 0, "audio_tokens": audio_tokens},
+        }
+        return usage
+
+    def unbilled_usage_on_session_close(self, model: str) -> RealtimeInputAudioTranscriptionUsage | None:
+        return self._consume_input_transcription_usage_estimate(model)
+
     def transform_realtime_response(
         self,
         message: str | bytes,
@@ -1190,6 +1233,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
         if isinstance(server_content, dict):
             input_tx: Final = server_content.get("inputTranscription")
             if isinstance(input_tx, dict) and input_tx.get("text"):
+                transcription_usage: Final = self._consume_input_transcription_usage_estimate(model)
                 returned_message.append(
                     cast(
                         OpenAIRealtimeEvents,
@@ -1199,6 +1243,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
                             "transcript": input_tx["text"],
                             "item_id": f"item_{uuid.uuid4()}",
                             "content_index": 0,
+                            **({} if transcription_usage is None else {"usage": transcription_usage}),
                         },
                     )
                 )
@@ -1235,6 +1280,12 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
                     )
                 )
 
+            # Transcription-only models emit generationComplete with no prior
+            # modelTurn delta; there is no started OpenAI response to close, so
+            # drop it and let siblings (turnComplete, usageMetadata) process.
+            if current_delta_type is None and "modelTurn" not in server_content:
+                server_content.pop("generationComplete", None)
+
             # Mark transcription-only serverContent as handled so the main loop
             # skips it; sibling keys like toolCall are still processed below.
             _model_content_keys: Final = {
@@ -1583,7 +1634,9 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
         ```
         """
 
-        response_modalities: Final[list[GeminiResponseModalities]] = ["AUDIO"]
+        response_modalities: Final[list[GeminiResponseModalities]] = [
+            GeminiRealtimeConfig._default_response_modality(model)
+        ]
         output_audio_transcription: Final = False
         # if "audio" in model: ## UNCOMMENT THIS WHEN AUDIO IS SUPPORTED
         #     output_audio_transcription = True

litellm/llms/mistral/chat/transformation.py

@@ -292,7 +292,7 @@ class MistralConfig(OpenAIGPTConfig):
                         file_id = file_content.get("file", {}).get("file_id")
                         if file_id:
                             # Replace 'file' with 'file_id'
-                            file_content["file_id"] = file_id
+                            file_content["file_id"] = file_id  # pyright: ignore[reportGeneralTypeIssues]  # legacy in-place rewrite of the block shape
                             file_content.pop("file", None)
         return messages
 

litellm/llms/openai/chat/gpt_transformation.py

@@ -18,6 +18,7 @@ from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response impo
     _should_convert_tool_call_to_json_mode,
 )
 from litellm.litellm_core_utils.prompt_templates.common_utils import (
+    drop_tool_reference_parts_from_tool_messages,
     get_tool_call_names,
     hoist_images_from_tool_messages,
 )
@@ -336,7 +337,8 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
         self, messages: list[AllMessageValues], model: str, is_async: bool = False
     ) -> list[AllMessageValues] | Coroutine[Any, Any, list[AllMessageValues]]:
         """OpenAI no longer supports image_url as a string, so we need to convert it to a dict"""
-        hoisted_messages: Final = hoist_images_from_tool_messages(messages)
+        stripped_messages: Final = drop_tool_reference_parts_from_tool_messages(messages)
+        hoisted_messages: Final = hoist_images_from_tool_messages(stripped_messages)
 
         async def _async_transform():
             for message in hoisted_messages:

litellm/llms/openai_like/providers.json

@@ -200,5 +200,9 @@
       "temperature_max": 1.99
     },
     "supported_endpoints": ["/v1/chat/completions"]
+  },
+  "synthorai": {
+    "base_url": "https://synthorai.io/v1",
+    "api_key_env": "SYNTHORAI_API_KEY"
   }
 }

litellm/llms/soniox/common_utils.py

@@ -4,6 +4,11 @@ Shared utilities for the Soniox provider (https://soniox.com).
 
 from typing import Any, Final
 
+from litellm.litellm_core_utils.audio_utils.subtitle_utils import (
+    SubtitleToken,
+    render_subtitle_tokens_as_srt,
+    render_subtitle_tokens_as_vtt,
+)
 from litellm.llms.base_llm.chat.transformation import BaseLLMException
 
 # Soniox API base URL.
@@ -109,121 +114,13 @@ def render_soniox_tokens(tokens: list[dict[str, Any]]) -> str:
     return "".join(text_parts)
 
 
-# ---------------------------------------------------------------------------
-# SRT / VTT subtitle rendering
-# ---------------------------------------------------------------------------
-
-# Maximum number of tokens to group into a single subtitle cue.
-_CUE_MAX_TOKENS: Final[int] = 15
-
-# Maximum duration (in ms) for a single cue before forcing a break.
-_CUE_MAX_DURATION_MS: Final[int] = 5000
-
-
-def _format_timestamp_srt(ms: int) -> str:
-    """Format milliseconds as SRT timestamp: HH:MM:SS,mmm"""
-    ms = max(ms, 0)
-    hours: Final = ms // 3_600_000
-    ms %= 3_600_000
-    minutes: Final = ms // 60_000
-    ms %= 60_000
-    seconds: Final = ms // 1_000
-    millis: Final = ms % 1_000
-    return f"{hours:02d}:{minutes:02d}:{seconds:02d},{millis:03d}"
-
-
-def _format_timestamp_vtt(ms: int) -> str:
-    """Format milliseconds as VTT timestamp: HH:MM:SS.mmm"""
-    ms = max(ms, 0)
-    hours: Final = ms // 3_600_000
-    ms %= 3_600_000
-    minutes: Final = ms // 60_000
-    ms %= 60_000
-    seconds: Final = ms // 1_000
-    millis: Final = ms % 1_000
-    return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{millis:03d}"
-
-
-def _group_tokens_into_cues(
-    tokens: list[dict[str, Any]],
-) -> list[dict[str, Any]]:
-    """
-    Group Soniox tokens into subtitle cues.
-
-    Each cue has:
-      - start_ms: int
-      - end_ms: int
-      - text: str
-
-    Grouping heuristics:
-      - A new cue starts when token count exceeds _CUE_MAX_TOKENS.
-      - A new cue starts when duration exceeds _CUE_MAX_DURATION_MS.
-      - A new cue starts when the speaker changes (if diarization is on).
-      - Tokens without timestamps are appended to the current cue.
-    """
-    cues: Final[list[dict[str, Any]]] = []
-    current_tokens: list[str] = []
-    current_start: int | None = None
-    current_end: int | None = None
-    current_speaker: Any | None = None
-
-    def _flush() -> None:
-        if current_tokens and current_start is not None:
-            text: Final = "".join(current_tokens).strip()
-            if text:
-                cues.append(
-                    {
-                        "start_ms": current_start,
-                        "end_ms": (current_end if current_end is not None else current_start),
-                        "text": text,
-                    }
-                )
-
-    for token in tokens:
-        start_ms = token.get("start_ms")
-        end_ms = token.get("end_ms")
-        text = token.get("text", "")
-        speaker = token.get("speaker")
-
-        # Skip tokens with no timestamp data entirely if we have no cue started
-        if start_ms is None and current_start is None:
-            continue
-
-        # Speaker change forces a new cue
-        if speaker is not None and speaker != current_speaker:
-            _flush()
-            current_tokens = []
-            current_start = start_ms
-            current_end = end_ms
-            current_speaker = speaker
-            current_tokens.append(text)
-            continue
-
-        # Duration or token count exceeded -> flush
-        should_break = False
-        if (
-            len(current_tokens) >= _CUE_MAX_TOKENS
-            or current_start is not None
-            and start_ms is not None
-            and (start_ms - current_start) >= _CUE_MAX_DURATION_MS
-        ):
-            should_break = True
-
-        if should_break:
-            _flush()
-            current_tokens = []
-            current_start = start_ms
-            current_end = end_ms
-            current_tokens.append(text)
-        else:
-            if current_start is None:
-                current_start = start_ms
-            if end_ms is not None:
-                current_end = end_ms
-            current_tokens.append(text)
-
-    _flush()
-    return cues
+def _soniox_token_to_subtitle_token(token: dict[str, Any]) -> SubtitleToken:
+    return SubtitleToken(
+        text=token.get("text", ""),
+        start_ms=token.get("start_ms"),
+        end_ms=token.get("end_ms"),
+        speaker=token.get("speaker"),
+    )
 
 
 def render_soniox_tokens_as_srt(tokens: list[dict[str, Any]]) -> str:
@@ -232,20 +129,7 @@ def render_soniox_tokens_as_srt(tokens: list[dict[str, Any]]) -> str:
 
     Returns an empty string if no tokens have timestamp data.
     """
-    cues: Final = _group_tokens_into_cues(tokens)
-    if not cues:
-        return ""
-
-    lines: Final[list[str]] = []
-    for idx, cue in enumerate(cues, start=1):
-        start = _format_timestamp_srt(cue["start_ms"])
-        end = _format_timestamp_srt(cue["end_ms"])
-        lines.append(str(idx))
-        lines.append(f"{start} --> {end}")
-        lines.append(cue["text"])
-        lines.append("")  # blank line between cues
-
-    return "\n".join(lines)
+    return render_subtitle_tokens_as_srt(tuple(_soniox_token_to_subtitle_token(token) for token in tokens))
 
 
 def render_soniox_tokens_as_vtt(tokens: list[dict[str, Any]]) -> str:
@@ -254,14 +138,4 @@ def render_soniox_tokens_as_vtt(tokens: list[dict[str, Any]]) -> str:
 
     Returns the VTT header even if no cues are present.
     """
-    cues: Final = _group_tokens_into_cues(tokens)
-
-    lines: Final[list[str]] = ["WEBVTT", ""]
-    for cue in cues:
-        start = _format_timestamp_vtt(cue["start_ms"])
-        end = _format_timestamp_vtt(cue["end_ms"])
-        lines.append(f"{start} --> {end}")
-        lines.append(cue["text"])
-        lines.append("")  # blank line between cues
-
-    return "\n".join(lines)
+    return render_subtitle_tokens_as_vtt(tuple(_soniox_token_to_subtitle_token(token) for token in tokens))

litellm/llms/tencent/chat/transformation.py

@@ -3,14 +3,36 @@ Translates from OpenAI's `/v1/chat/completions` to Tencent TokenHub's
 OpenAI-compatible endpoint.
 """
 
-from typing import Final
+from collections.abc import Mapping
+from typing import Final, TypedDict
 
+from typing_extensions import ReadOnly
+
+import litellm
 from litellm.secret_managers.main import get_secret_str
 from litellm.utils import supports_reasoning
 
 from ...openai.chat.gpt_transformation import OpenAIGPTConfig
 
 
+class ThinkingPayload(TypedDict, total=False):
+    """Tencent TokenHub `thinking` object.
+
+    `type` ("enabled"/"disabled"/"adaptive") is required by TokenHub when the
+    object is passed; `budget_tokens` is auto-filled server-side when omitted.
+    Ref: https://www.tencentcloud.com/document/product/1300/82345
+    """
+
+    type: ReadOnly[str]
+    budget_tokens: ReadOnly[int]
+
+
+class ThinkingExtraBody(TypedDict, total=False):
+    """`extra_body` payload carrying TokenHub's `thinking` object."""
+
+    thinking: ReadOnly[Mapping[str, object]]
+
+
 class TencentChatConfig(OpenAIGPTConfig):
     def get_supported_openai_params(self, model: str) -> list:
         params: Final = super().get_supported_openai_params(model)
@@ -25,18 +47,71 @@ class TencentChatConfig(OpenAIGPTConfig):
         model: str,
         drop_params: bool,
     ) -> dict:
-        optional_params = super().map_openai_params(non_default_params, optional_params, model, drop_params)
+        mapped_params: Final = super().map_openai_params(non_default_params, optional_params, model, drop_params)
 
-        thinking_value: Final = optional_params.pop("thinking", None)
-        reasoning_effort: Final = optional_params.pop("reasoning_effort", None)
+        thinking_value: Final = mapped_params.pop("thinking", None)
+        reasoning_effort: Final = mapped_params.pop("reasoning_effort", None)
 
-        if thinking_value is not None:
-            if isinstance(thinking_value, dict):
-                optional_params["thinking"] = thinking_value
-        elif reasoning_effort is not None and reasoning_effort != "none":
-            optional_params["thinking"] = {"type": "enabled"}
+        thinking: Final = self._resolve_thinking_payload(
+            model=model,
+            thinking_value=thinking_value,  # pyright: ignore[reportUnknownArgumentType]  # value popped from the untyped provider params dict
+            reasoning_effort=reasoning_effort,  # pyright: ignore[reportUnknownArgumentType]  # value popped from the untyped provider params dict
+        )
+        if thinking is not None:
+            # TokenHub expects `thinking` in the request JSON body, but the
+            # OpenAI SDK's chat.completions.create() rejects unknown top-level
+            # kwargs, so it travels via `extra_body`, which the SDK merges into
+            # the payload. A plain assignment is merge-safe: get_optional_params
+            # spreads this dict into its own extra_body assembly downstream.
+            extra_body: Final[ThinkingExtraBody] = {"thinking": thinking}
+            mapped_params["extra_body"] = extra_body
+        return mapped_params
 
-        return optional_params
+    @classmethod
+    def _resolve_thinking_payload(
+        cls,
+        model: str,
+        thinking_value: object,
+        reasoning_effort: object,
+    ) -> Mapping[str, object] | None:
+        if isinstance(thinking_value, dict):
+            return cls._coerce_thinking_type_for_model(model=model, thinking=thinking_value)  # pyright: ignore[reportUnknownArgumentType]  # isinstance narrows to dict[Unknown, Unknown] out of the untyped provider params dict
+        if isinstance(reasoning_effort, str):
+            # TokenHub recommends explicitly disabling thinking rather than
+            # relying on per-model defaults (deepseek-v4-* default to enabled).
+            payload: Final[ThinkingPayload] = {"type": "disabled" if reasoning_effort == "none" else "enabled"}
+            return cls._coerce_thinking_type_for_model(model=model, thinking=payload)
+        return None
+
+    @staticmethod
+    def _coerce_thinking_type_for_model(model: str, thinking: Mapping[str, object]) -> Mapping[str, object]:
+        """Coerce `thinking.type` to a value the model accepts.
+
+        MiniMax models on TokenHub only accept "adaptive"/"disabled" and reject
+        "enabled" with a 400; "adaptive" (the model decides when to think) is
+        the closest semantic, so "enabled" is coerced for them. The capability
+        is read from the model map's `supports_adaptive_thinking` flag, so
+        aliases and newly onboarded adaptive-only models need no code change.
+        Ref: https://www.tencentcloud.com/document/product/1300/82345
+        """
+        if thinking.get("type") != "enabled" or not TencentChatConfig._is_adaptive_thinking_model(model):
+            return thinking
+
+        budget: Final[object] = thinking.get("budget_tokens")
+        if isinstance(budget, int):
+            coerced_with_budget: Final[ThinkingPayload] = {"type": "adaptive", "budget_tokens": budget}
+            return coerced_with_budget
+        coerced: Final[ThinkingPayload] = {"type": "adaptive"}
+        return coerced
+
+    @staticmethod
+    def _is_adaptive_thinking_model(model: str) -> bool:
+        """Read `supports_adaptive_thinking` from the model map under tencent."""
+        try:
+            model_info: Final[Mapping[str, object]] = litellm.get_model_info(model=model, custom_llm_provider="tencent")
+        except Exception:  # noqa: BLE001  # get_model_info raises a bare Exception for unmapped models
+            return False
+        return model_info.get("supports_adaptive_thinking") is True
 
     def _get_openai_compatible_provider_info(
         self, api_base: str | None, api_key: str | None

litellm/llms/together_ai/chat/transformation.py

@@ -4,7 +4,8 @@ Translates from OpenAI's `/v1/chat/completions` to Together AI's `/v1/chat/compl
 Docs: https://docs.together.ai/docs/chat-overview
 """
 
-from collections.abc import Callable, Container, Coroutine
+from collections.abc import Callable, Container, Coroutine, Mapping
+from types import MappingProxyType
 from typing import (
     Final,
     Literal,
@@ -12,11 +13,13 @@ from typing import (
     overload,
 )
 
+from typing_extensions import ReadOnly, TypedDict
+
 import litellm
 from litellm._logging import verbose_logger
 from litellm.exceptions import UnsupportedParamsError
 from litellm.types.llms.openai import AllMessageValues
-from litellm.utils import supports_function_calling, supports_response_schema
+from litellm.utils import supports_function_calling, supports_reasoning, supports_response_schema
 
 from ...openai.chat.gpt_transformation import OpenAIGPTConfig
 
@@ -38,6 +41,34 @@ def _registry_verdict(model: str, flag: str, check: Callable[[str], bool]) -> bo
     return None
 
 
+ADJUSTABLE_EFFORT_REASONING_MODELS: Final = frozenset(
+    {
+        "openai/gpt-oss-120b",
+        "openai/gpt-oss-20b",
+    }
+)
+HYBRID_REASONING_MODELS: Final = frozenset(
+    {
+        "MiniMaxAI/MiniMax-M3",
+        "Qwen/Qwen3.5-9B",
+        "Qwen/Qwen3.6-Plus",
+        "deepseek-ai/DeepSeek-V4-Pro",
+        "moonshotai/Kimi-K3",
+        "nvidia/nemotron-3-ultra-550b-a55b",
+        "zai-org/GLM-5.2",
+    }
+)
+HIGH_MAX_EFFORT_MODEL_PREFIX: Final = "deepseek-ai/DeepSeek-V4-Pro"
+EFFORT_TRANSLATION: Final = MappingProxyType({"minimal": "low", "xhigh": "high", "max": "high"})
+HIGH_MAX_EFFORT_TRANSLATION: Final = MappingProxyType(
+    {"minimal": "high", "low": "high", "medium": "high", "xhigh": "max"}
+)
+
+
+class TogetherReasoningToggle(TypedDict):
+    enabled: ReadOnly[bool]
+
+
 def _function_calling_verdict(model: str) -> bool | None:
     return _registry_verdict(
         model,
@@ -83,6 +114,36 @@ def _tool_params_to_drop(passed_params: Container[str], model: str, drop_params:
     )
 
 
+def _supports_together_reasoning(model: str) -> bool:
+    if model in ADJUSTABLE_EFFORT_REASONING_MODELS or model in HYBRID_REASONING_MODELS:
+        return True
+    if model.startswith(HIGH_MAX_EFFORT_MODEL_PREFIX):
+        return True
+    return supports_reasoning(model, custom_llm_provider="together_ai")
+
+
+def _adjustable_effort(effort: str, model: str) -> str:
+    if effort == "none":
+        verbose_logger.debug(
+            "together_ai model %s cannot disable reasoning; mapping reasoning_effort=none to low", model
+        )
+        return "low"
+    return EFFORT_TRANSLATION.get(effort, effort)
+
+
+def _reasoning_effort_payload(effort: str, model: str) -> Mapping[str, object]:
+    if effort == "default":
+        return MappingProxyType({})
+    if model in ADJUSTABLE_EFFORT_REASONING_MODELS:
+        return MappingProxyType({"reasoning_effort": _adjustable_effort(effort, model)})
+    if effort == "none":
+        disable_reasoning: Final[TogetherReasoningToggle] = {"enabled": False}
+        return MappingProxyType({"reasoning": disable_reasoning})
+    if model.startswith(HIGH_MAX_EFFORT_MODEL_PREFIX):
+        return MappingProxyType({"reasoning_effort": HIGH_MAX_EFFORT_TRANSLATION.get(effort, effort)})
+    return MappingProxyType({"reasoning_effort": EFFORT_TRANSLATION.get(effort, effort)})
+
+
 def _drop_response_format(passed_params: Container[str], model: str, drop_params: bool) -> bool:
     if "response_format" not in passed_params:
         return False
@@ -153,6 +214,15 @@ class TogetherAIChatConfig(OpenAIGPTConfig):
             return super()._transform_messages(stripped, model, is_async=True)
         return super()._transform_messages(stripped, model, is_async=False)
 
+    def get_supported_openai_params(self, model: str) -> list:  # mutable-ok: inherited contract
+        supported_params: Final = super().get_supported_openai_params(model)
+        if not _supports_together_reasoning(model):
+            return supported_params
+        return [  # mutable-ok: the inherited contract returns a plain list; building fresh avoids mutating the base class's value
+            *supported_params,
+            "reasoning_effort",
+        ]
+
     def map_openai_params(
         self,
         non_default_params: dict,
@@ -165,4 +235,10 @@ class TogetherAIChatConfig(OpenAIGPTConfig):
             mapped_openai_params.pop(param)
         if _drop_response_format(mapped_openai_params, model, drop_params):
             mapped_openai_params.pop("response_format")
+        effort: Final = mapped_openai_params.get("reasoning_effort")
+        if not isinstance(effort, str):
+            return mapped_openai_params
+        mapped_openai_params.pop("reasoning_effort")
+        for key, value in _reasoning_effort_payload(effort, model).items():
+            mapped_openai_params.setdefault(key, value)
         return mapped_openai_params

litellm/llms/together_ai/cost_calculator.py

@@ -3,6 +3,7 @@ Handles calculating cost for together ai models
 """
 
 import re
+from collections.abc import Mapping
 from typing import Final
 
 from litellm.constants import (
@@ -18,6 +19,12 @@ from litellm.constants import (
 from litellm.types.utils import CallTypes
 
 
+def has_together_registry_pricing(model: str, cost_map: Mapping[str, object]) -> bool:
+    stripped: Final = model.removeprefix("together_ai/")
+    entry: Final = cost_map.get(f"together_ai/{stripped}")
+    return isinstance(entry, Mapping) and "input_cost_per_token" in entry
+
+
 # Extract the number of billion parameters from the model name
 # only used for together_computer LLMs
 def get_model_params_and_category(model_name, call_type: CallTypes) -> str:

litellm/main.py

@@ -1219,6 +1219,7 @@ def _register_custom_pricing_for_request(
             shared_key: CustomPricingLiteLLMParams.strip_custom_pricing_fields(entry),
         },
         persist_across_reloads=False,
+        warning_display_name=shared_key,
     )
 
 

litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py

@@ -46,6 +46,7 @@ from litellm.proxy._types import (
 )
 from litellm.proxy.auth.ip_address_utils import IPAddressUtils
 from litellm.proxy.auth.user_api_key_auth import (
+    _get_bearer_token_or_received_api_key,  # pyright: ignore[reportPrivateUsage]  # shared x-litellm-api-key parser lives with user_api_key_auth
     _run_centralized_common_checks,
     user_api_key_auth,
 )
@@ -429,7 +430,10 @@ class MCPRequestHandler:
             # An explicit x-litellm-api-key is always a LiteLLM credential, even
             # for a delegated server, so validate it: identity / spend / rate
             # limits resolve and any stored upstream token can be forwarded.
-            validated_user_api_key_auth = await user_api_key_auth(api_key=litellm_api_key, request=request)
+            validated_user_api_key_auth = await user_api_key_auth(
+                api_key=f"Bearer {_get_bearer_token_or_received_api_key(litellm_api_key)}",
+                request=request,
+            )
         elif MCPRequestHandler._target_servers_delegate_auth_to_upstream(
             path=request_route,
             mcp_servers=mcp_servers,

litellm/proxy/_experimental/mcp_server/mcp_server_manager.py

@@ -34,6 +34,7 @@ from mcp.types import (
 )
 from mcp.types import Tool as MCPTool
 from pydantic import AnyUrl, BaseModel
+from typing_extensions import ReadOnly
 
 import litellm
 from litellm._logging import verbose_logger
@@ -72,6 +73,7 @@ from litellm.proxy._experimental.mcp_server.oauth2_token_cache import (
     MCPPerUserTokenCache,
     mcp_per_user_token_cache,
     resolve_mcp_auth,
+    resolved_token_header,
 )
 from litellm.proxy._experimental.mcp_server.oauth_utils import (
     _redact_mcp_resource_url,
@@ -99,6 +101,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.token_exchange_
     build_token_exchanger,
 )
 from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
+    DEFAULT_CREDENTIAL_HEADER,
     AuthorizationCodeConfig,
     ClientCredentialsConfig,
     CredError,
@@ -153,6 +156,8 @@ from litellm.types.mcp import (
     MCPAuth,
     MCPStdioConfig,
     MCPTokenEndpointAuthMethod,
+    has_header,
+    without_header,
 )
 from litellm.types.mcp_server.mcp_server_manager import (
     MCPInfo,
@@ -349,6 +354,7 @@ class MCPServerConfig(TypedDict, total=False):
     audience: str
     subject_token_type: str
     upstream_resource: str
+    upstream_token_header: ReadOnly[str]
     id_jag_resource_token_endpoint: str
     id_jag_resource: str
     client_private_key: str
@@ -828,18 +834,6 @@ def _should_strip_caller_authorization(
     )
 
 
-def _without_authorization(
-    headers: dict[str, str] | None,
-) -> dict[str, str] | None:
-    """A copy of ``headers`` with any ``Authorization`` key removed (case-insensitive), or
-    None if nothing remains. Drops only the credential, keeping other forwarded headers.
-    """
-    if not headers:
-        return None
-    filtered: Final = {k: v for k, v in headers.items() if k.lower() != "authorization"}
-    return filtered or None
-
-
 def _format_byok_openapi_auth_header(mcp_server: MCPServer, mcp_auth_header: str) -> str:
     """Format a raw BYOK credential for OpenAPI tool ``Authorization`` injection.
 
@@ -914,7 +908,9 @@ def _resolve_openapi_tool_auth(
 
     if isinstance(per_server, dict):
         authorization: Final = next((v for k, v in per_server.items() if k.lower() == "authorization"), None)
-        merged: Final = merge_mcp_headers(extra_headers=forwarded, static_headers=_without_authorization(per_server))
+        merged: Final = merge_mcp_headers(
+            extra_headers=forwarded, static_headers=without_header(per_server, DEFAULT_CREDENTIAL_HEADER)
+        )
         if authorization is None:
             byok: Final = _format_byok_openapi_auth_header(mcp_server, mcp_auth_header) if mcp_auth_header else None
             return byok, merged, mcp_auth_header
@@ -981,7 +977,7 @@ def _client_forwarded_authorization_headers(
         raw_headers=raw_headers,
         user_api_key_auth=user_api_key_auth,
     ):
-        return _without_authorization(extra_headers)
+        return without_header(extra_headers, DEFAULT_CREDENTIAL_HEADER)
     return extra_headers
 
 
@@ -994,7 +990,7 @@ def _take_forwarded_authorization(
     if not headers:
         return None, headers
     value: Final = next((v for k, v in headers.items() if k.lower() == "authorization"), None)
-    return value, _without_authorization(headers)
+    return value, without_header(headers, DEFAULT_CREDENTIAL_HEADER)
 
 
 def _passthrough_token_from_mcp_auth_header(
@@ -2166,6 +2162,7 @@ class MCPServerManager:
                     DEFAULT_SUBJECT_TOKEN_TYPE,
                 ),
                 upstream_resource=server_config.get("upstream_resource", None),
+                upstream_token_header=server_config.get("upstream_token_header", None),
                 # ID-JAG fields
                 id_jag_resource_token_endpoint=server_config.get("id_jag_resource_token_endpoint", None),
                 id_jag_resource=server_config.get("id_jag_resource", None),
@@ -2698,6 +2695,7 @@ class MCPServerManager:
             or (credentials_dict.get("subject_token_type") if credentials_dict else None)
             or DEFAULT_SUBJECT_TOKEN_TYPE,
             upstream_resource=(credentials_dict.get("upstream_resource") if credentials_dict else None),
+            upstream_token_header=(credentials_dict.get("upstream_token_header") if credentials_dict else None),
             # ID-JAG fields — read from credentials JSON blob
             id_jag_resource_token_endpoint=(
                 credentials_dict.get("id_jag_resource_token_endpoint") if credentials_dict else None
@@ -3525,10 +3523,9 @@ class MCPServerManager:
             case Ok(auth):
                 # NoOpAuth has no header_name and so never conflicts.
                 header_name: Final[str | None] = getattr(auth, "header_name", None)
-                conflicts: Final = bool(
-                    header_name and extra_headers and any(key.lower() == header_name.lower() for key in extra_headers)
-                )
-                if not conflicts:
+                if header_name is None or not extra_headers:
+                    return auth, extra_headers
+                if not has_header(extra_headers, header_name):
                     return auth, extra_headers
                 if isinstance(
                     spec.config,
@@ -3540,9 +3537,10 @@ class MCPServerManager:
                     # guardrail such as MCPJWTSigner, static_headers, or any other injected
                     # Authorization must NOT shadow it (otherwise the upstream gets e.g. the
                     # signer's JWT instead of the minted token and rejects it, and for M2M the
-                    # one-shot 401 refetch is lost with it). Drop the conflicting header so the
-                    # resolved token reaches upstream.
-                    return auth, _without_authorization(extra_headers)
+                    # one-shot 401 refetch is lost with it). Drop only the header the resolved
+                    # credential is about to occupy, so a static credential the operator aimed at a
+                    # DIFFERENT header still reaches upstream.
+                    return auth, without_header(extra_headers, header_name)
                 # Other modes: an Authorization already supplied via extra_headers (a forwarded caller
                 # header or static_headers) is intentional and wins; v1 applies those last.
                 return None, extra_headers
@@ -3650,6 +3648,7 @@ class MCPServerManager:
         ):
             spec = None
         auth_value: Final = await resolve_mcp_auth(resolved_server, mcp_auth_header) if spec is None else None
+        auth_header_name: Final = resolved_token_header(resolved_server, mcp_auth_header) if spec is None else None
 
         # Create sampling and elicitation callbacks for this client
         sampling_cb = (
@@ -3758,6 +3757,7 @@ class MCPServerManager:
                 transport_type=transport,
                 auth_type=resolved_server.auth_type,
                 auth_value=auth_value,
+                auth_header_name=auth_header_name,
                 timeout=(resolved_server.timeout if resolved_server.timeout is not None else MCP_CLIENT_TIMEOUT),
                 extra_headers=extra_headers,
                 aws_auth=aws_auth,
@@ -5256,7 +5256,9 @@ class MCPServerManager:
             proxy_logging_obj: Optional ProxyLogging object for hook integration
             host_progress_callback: Optional callback for progress updates
             hook_extra_headers: Optional headers injected by pre_mcp_call guardrail
-                hooks. Merged last (highest priority) into outbound request headers.
+                hooks. Merged last into outbound request headers, except a hook
+                Authorization header is dropped when an upstream credential already
+                occupies the Authorization slot.
 
         Returns:
             CallToolResult from the MCP server
@@ -5304,7 +5306,7 @@ class MCPServerManager:
                     raw_headers=raw_headers,
                     user_api_key_auth=user_api_key_auth,
                 ):
-                    extra_headers = _without_authorization(extra_headers)
+                    extra_headers = without_header(extra_headers, DEFAULT_CREDENTIAL_HEADER)
         elif mcp_server.is_client_forwarded_token:
             extra_headers = _client_forwarded_authorization_headers(
                 mcp_server=mcp_server,
@@ -5347,27 +5349,26 @@ class MCPServerManager:
         if hook_extra_headers:
             if extra_headers is None:
                 extra_headers = {}
-            if "Authorization" in hook_extra_headers:
-                if "Authorization" in extra_headers:
-                    verbose_logger.warning(
-                        "MCPServerManager: hook_extra_headers 'Authorization' will overwrite "
-                        "the existing Authorization header from static_headers. "
-                        "The hook JWT will take precedence."
-                    )
-                elif server_auth_header is not None:
-                    # server_auth_header is passed separately to _create_mcp_client as
-                    # auth_value.  Both will reach the upstream server — warn so admins
-                    # know two Authorization credentials are being sent.
-                    verbose_logger.warning(
-                        "MCPServerManager: hook_extra_headers injects 'Authorization' while "
-                        "server '%s' already has a configured authentication_token. "
-                        "Both credentials will be sent; the hook header is in extra_headers "
-                        "and the server token is in auth_value — the upstream server decides "
-                        "which one wins.  Consider unsetting authentication_token if you want "
-                        "the hook JWT to be the sole credential.",
-                        mcp_server.server_name or mcp_server.name,
-                    )
-            extra_headers.update(hook_extra_headers)
+            hook_has_authorization: Final = any(k.lower() == "authorization" for k in hook_extra_headers)
+            existing_has_authorization: Final = any(k.lower() == "authorization" for k in extra_headers)
+            server_auth_occupies_authorization: Final = (
+                any(k.lower() == "authorization" for k in server_auth_header)
+                if isinstance(server_auth_header, dict)
+                else server_auth_header is not None and mcp_server.auth_type != MCPAuth.api_key
+            )
+            if hook_has_authorization and (existing_has_authorization or server_auth_occupies_authorization):
+                # Mirror the tools/list signer guard: an upstream credential (user OAuth,
+                # static header, or configured authentication_token) already occupies the
+                # Authorization slot, so the hook must not replace it.
+                verbose_logger.warning(
+                    "MCPServerManager: dropping hook-injected 'Authorization' header for "
+                    "server '%s' because an upstream credential already occupies the "
+                    "Authorization slot; the existing credential is kept.",
+                    mcp_server.server_name or mcp_server.name,
+                )
+                extra_headers.update({k: v for k, v in hook_extra_headers.items() if k.lower() != "authorization"})
+            else:
+                extra_headers.update(hook_extra_headers)
 
         # Reset to None if no headers were actually added
         if extra_headers is not None and len(extra_headers) == 0:

litellm/proxy/_experimental/mcp_server/oauth2_token_cache.py

@@ -7,6 +7,7 @@ with ``client_id``, ``client_secret``, and ``token_url``.
 
 import asyncio
 import hashlib
+from collections.abc import Mapping
 from typing import TYPE_CHECKING, Final
 
 import httpx
@@ -313,9 +314,26 @@ async def resolve_mcp_auth(
     1. ``mcp_auth_header`` — per-request/per-user override
     2. OAuth2 client_credentials token — auto-fetched and cached
     3. ``server.authentication_token`` — static token from config/DB
+
+    ``resolved_token_header`` answers, for the same two inputs, which header the value belongs in.
     """
     if mcp_auth_header:
         return mcp_auth_header
     if server.has_client_credentials:
         return await mcp_oauth2_token_cache.async_get_token(server)
     return server.authentication_token
+
+
+def resolved_token_header(
+    server: "MCPServer",
+    mcp_auth_header: str | Mapping[str, str] | None = None,
+) -> str | None:
+    """Which upstream header the value ``resolve_mcp_auth`` just returned belongs in.
+
+    ``None`` means keep the auth_type default. A caller-supplied ``mcp_auth_header`` is the caller's
+    own credential aimed at the slot the upstream normally uses, so it never moves; only the values
+    the gateway resolved from its own config (the minted M2M token, the static token) follow
+    ``upstream_token_header``. Same inputs and same branch order as ``resolve_mcp_auth``, so the two
+    cannot disagree about which case they are in.
+    """
+    return None if mcp_auth_header else server.upstream_token_header

litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py

@@ -47,12 +47,14 @@ def sanitize_openapi_tool_name(raw_name: str) -> str:
 from litellm._logging import verbose_logger
 from litellm.litellm_core_utils.url_utils import async_safe_get
 from litellm.llms.custom_httpx.http_handler import (
+    AsyncHTTPHandler,
     get_async_httpx_client,
     httpxSpecialProvider,
 )
 from litellm.proxy._experimental.mcp_server.tool_registry import (
     global_mcp_tool_registry,
 )
+from litellm.types.mcp import credential_redirect_hook, custom_credential_slot
 
 
 class _OpenAPIJSONSchema(TypedDict, total=False):
@@ -119,6 +121,10 @@ _request_resolved_auth_headers: Final[contextvars.ContextVar[dict[str, str] | No
     "_request_resolved_auth_headers", default=None
 )
 
+_request_upstream_url: Final[contextvars.ContextVar[str | None]] = contextvars.ContextVar(
+    "_request_upstream_url", default=None
+)
+
 
 def _sanitize_path_parameter_value(param_value: object, param_name: str) -> str:
     """Ensure path params cannot introduce directory traversal."""
@@ -349,6 +355,35 @@ def build_input_schema(operation: _OpenAPIOperation) -> dict[str, object]:
     }
 
 
+async def _drop_credential_across_origin(request: httpx.Request) -> None:
+    """Apply this request's cross-origin credential guard, if it needs one.
+
+    Reads the per-request context rather than closing over it so the hook is one stable object, which
+    keeps the guarded client cacheable. A closure would key a new entry per call, and the handler it
+    built would never be closed.
+    """
+    guard: Final = credential_redirect_hook(
+        _request_upstream_url.get() or "", custom_credential_slot(_request_resolved_auth_headers.get())
+    )
+    if guard is not None:
+        await guard(request)
+
+
+def _upstream_client() -> AsyncHTTPHandler:
+    """The HTTP client for one upstream call, guarded when a credential rides a custom slot.
+
+    A resolved credential outside ``Authorization`` is not stripped across origins by the client
+    itself, so this arm installs the same hook the MCP client uses. Both variants come from the
+    shared cache, so a guarded call reuses its connection pool like any other.
+    """
+    if custom_credential_slot(_request_resolved_auth_headers.get()) is None:
+        return get_async_httpx_client(llm_provider=httpxSpecialProvider.MCP)
+    return get_async_httpx_client(
+        llm_provider=httpxSpecialProvider.MCP,
+        params={"event_hooks": {"request": [_drop_credential_across_origin]}},
+    )
+
+
 def _merge_openapi_tool_request_headers(
     static_headers: dict[str, str],
 ) -> dict[str, str]:
@@ -510,8 +545,9 @@ def create_tool_function(
                 except (json.JSONDecodeError, TypeError):
                     json_body = {"data": body_value}
 
-        client: Final = get_async_httpx_client(llm_provider=httpxSpecialProvider.MCP)
+        client: Final = _upstream_client()
         upstream: Final = server_label or f"{original_method.upper()} {path}"
+        url_token: Final = _request_upstream_url.set(url)
 
         try:
             if original_method == "get":
@@ -529,6 +565,8 @@ def create_tool_function(
         except MaskedHTTPStatusError as e:
             _raise_for_upstream_failure(e.response, upstream, relays_upstream_auth)
             raise
+        finally:
+            _request_upstream_url.reset(url_token)
 
         _raise_for_upstream_failure(response, upstream, relays_upstream_auth)
         return response.text

litellm/proxy/_experimental/mcp_server/outbound_credentials/__init__.py

@@ -21,6 +21,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.result import (
     Result,
 )
 from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
+    DEFAULT_CREDENTIAL_HEADER,
     Ambient,
     ApiKeyConfig,
     ApiKeySource,
@@ -35,6 +36,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
     ClientCredentialsConfig,
     ClientSecretAuth,
     CredError,
+    HeaderCarrier,
     IdJagConfig,
     NoneConfig,
     PassthroughConfig,
@@ -45,9 +47,11 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
     Subject,
     TokenExchangeConfig,
     parse_auth_spec_kind,
+    validate_header_name,
 )
 
 __all__ = [
+    "DEFAULT_CREDENTIAL_HEADER",
     "Ambient",
     "ApiKeyConfig",
     "ApiKeySource",
@@ -63,6 +67,7 @@ __all__ = [
     "ClientSecretAuth",
     "CredError",
     "Error",
+    "HeaderCarrier",
     "IdJagConfig",
     "NoOpAuth",
     "NoneConfig",
@@ -78,4 +83,5 @@ __all__ = [
     "TokenExchangeConfig",
     "UpstreamCredentialProvider",
     "parse_auth_spec_kind",
+    "validate_header_name",
 ]

litellm/proxy/_experimental/mcp_server/outbound_credentials/adapter.py

@@ -20,6 +20,7 @@ from typing_extensions import assert_never
 
 from litellm.proxy._experimental.mcp_server.oauth_utils import resolve_upstream_resource
 from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
+    DEFAULT_CREDENTIAL_HEADER,
     ApiKeyConfig,
     AuthorizationCodeConfig,
     ClientAuth,
@@ -45,6 +46,15 @@ _TOKEN_EXCHANGE_SUBJECT_TOKEN_DEFAULT: Final = "urn:ietf:params:oauth:token-type
 _ID_JAG_SUBJECT_TOKEN_DEFAULT: Final = "urn:ietf:params:oauth:token-type:id_token"
 
 
+def token_header(server: MCPServer) -> str:
+    """The upstream header this server's resolved credential occupies.
+
+    One owner for every arm, so no spec builder spells the default itself and a server can never
+    hand two arms different answers.
+    """
+    return server.upstream_token_header or DEFAULT_CREDENTIAL_HEADER
+
+
 def to_subject(user_api_key_auth: UserAPIKeyAuth | None, subject_token: str | None) -> Subject:
     """Map v1's authenticated principal onto the resolver's Subject.
 
@@ -122,7 +132,7 @@ def _oauth2_spec(server: MCPServer, resource: str) -> ServerSpec | None:
         return ServerSpec(
             server_id=server.server_id,
             resource=resource,
-            config=AuthorizationCodeConfig(),
+            config=AuthorizationCodeConfig(header_name=token_header(server)),
         )
     return None
 
@@ -140,6 +150,7 @@ def _client_credentials_spec(server: MCPServer, resource: str) -> ServerSpec:
         server_id=server.server_id,
         resource=resource,
         config=ClientCredentialsConfig(
+            header_name=token_header(server),
             client_id=server.client_id,
             client_secret=SecretStr(server.client_secret) if server.client_secret else None,
             token_url=server.effective_token_url,
@@ -173,6 +184,7 @@ def _token_exchange_spec(server: MCPServer, resource: str) -> ServerSpec | None:
         server_id=server.server_id,
         resource=resource,
         config=TokenExchangeConfig(
+            header_name=token_header(server),
             profile=profile,
             subject_token_type=server.subject_token_type or DEFAULT_SUBJECT_TOKEN_TYPE,
             token_exchange_endpoint=endpoint,
@@ -206,7 +218,7 @@ def _shared_key_spec(
         server_id=server.server_id,
         resource=resource,
         config=ApiKeyConfig(
-            header_name=header_name,
+            header_name=server.upstream_token_header or header_name,
             value_prefix=value_prefix,
             key_source=SharedKey(value=SecretStr(value)),
         ),
@@ -231,6 +243,7 @@ def _id_jag_spec(server: MCPServer, resource: str) -> ServerSpec | None:
         server_id=server.server_id,
         resource=resource,
         config=IdJagConfig(
+            header_name=token_header(server),
             org_token_endpoint=org_token_endpoint,
             resource_token_endpoint=resource_token_endpoint,
             client_id=client_id,

litellm/proxy/_experimental/mcp_server/outbound_credentials/bridge_credentials.py

@@ -239,5 +239,6 @@ def resolve_bridge_envelope(
     if opened.identity.server_id != expected_server_id:
         return BridgeEnvelopeInvalid()
     grant: Final = opened.grant
-    upstream_authorization: Final = f"{grant.token_type} {grant.access_token.get_secret_value()}"
+    authorization_scheme: Final = "Bearer" if grant.token_type.lower() == "bearer" else grant.token_type
+    upstream_authorization: Final = f"{authorization_scheme} {grant.access_token.get_secret_value()}"
     return BridgeEnvelopeAdmitted(identity=opened.identity, upstream_authorization=SecretStr(upstream_authorization))

litellm/proxy/_experimental/mcp_server/outbound_credentials/client_credentials.py

@@ -50,6 +50,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.result import (
 from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
     ClientCredentialsConfig,
     CredError,
+    HeaderCarrier,
 )
 
 
@@ -328,14 +329,21 @@ class ClientCredentialsBearerAuth(httpx.Auth):
     refetch fails, or the retried request 401s again, the upstream's response stands.
     """
 
-    def __init__(self, access_token: str, refetch: Callable[[str], Awaitable[str | None]]) -> None:
-        self.header_name = "Authorization"
+    def __init__(
+        self,
+        access_token: str,
+        refetch: Callable[[str], Awaitable[str | None]],
+        carrier: HeaderCarrier,
+    ) -> None:
+        self._carrier = carrier
+        self.header_name = carrier.header_name
         self._access_token = SecretStr(access_token)
         self._refetch = refetch
 
     async def async_auth_flow(self, request: httpx.Request) -> AsyncGenerator[httpx.Request, httpx.Response]:
         token: Final = self._access_token.get_secret_value()
-        request.headers[self.header_name] = f"Bearer {token}"
+        name, value = self._carrier.header(token)
+        request.headers[name] = value
         response: Final = yield request
         if response.status_code != 401:
             return
@@ -343,7 +351,8 @@ class ClientCredentialsBearerAuth(httpx.Auth):
         if fresh is None:
             return
         self._access_token = SecretStr(fresh)
-        request.headers[self.header_name] = f"Bearer {fresh}"
+        fresh_name, fresh_value = self._carrier.header(fresh)
+        request.headers[fresh_name] = fresh_value
         yield request
 
     def sync_auth_flow(self, request: httpx.Request) -> Generator[httpx.Request, httpx.Response, None]:

litellm/proxy/_experimental/mcp_server/outbound_credentials/resolver.py

@@ -145,8 +145,8 @@ class UpstreamCredentialProvider:
                 return await self._token_exchange(subject, server, config)
             case IdJagConfig() as config:
                 return await self._id_jag(subject, server, config)
-            case AuthorizationCodeConfig():
-                return await self._authorization_code(subject, server)
+            case AuthorizationCodeConfig() as config:
+                return await self._authorization_code(subject, server, config)
             case AwsSigV4Config():
                 return _not_implemented(AuthSpecKind.aws_sigv4)
         assert_never(server.config)
@@ -284,15 +284,19 @@ class UpstreamCredentialProvider:
 
         match await self._exchanged_tokens.get_or_compute(slot, _exchange, fingerprint=fingerprint):
             case Ok(access_token):
-                return Ok(StaticHeaderAuth(f"Bearer {access_token}"))
+                header_name, header_value = config.header(access_token)
+                return Ok(StaticHeaderAuth(header_value, header_name=header_name))
             case Error(err):
                 return Error(err)
 
-    async def _authorization_code(self, subject: Subject, server: ServerSpec) -> Result[StaticHeaderAuth, CredError]:
+    async def _authorization_code(
+        self, subject: Subject, server: ServerSpec, config: AuthorizationCodeConfig
+    ) -> Result[StaticHeaderAuth, CredError]:
         token: Final = await self._authz_token(subject, server)
         if token is None:
             return Error(CredError.of_unauthorized("Authorization required: complete the OAuth flow for this server."))
-        return Ok(StaticHeaderAuth(f"Bearer {token.access_token}", header_name="Authorization"))
+        header_name, header_value = config.header(token.access_token)
+        return Ok(StaticHeaderAuth(header_value, header_name=header_name))
 
     async def _client_credentials(
         self, server_id: str, config: ClientCredentialsConfig
@@ -307,7 +311,7 @@ class UpstreamCredentialProvider:
         match await self._client_credentials_source.get(server_id, config):
             case Ok(token):
                 refetch: Final = partial(self._client_credentials_source.refetch, server_id, config)
-                return Ok(ClientCredentialsBearerAuth(token.access_token, refetch))
+                return Ok(ClientCredentialsBearerAuth(token.access_token, refetch, config))
             case Error(err):
                 return Error(err)
 
@@ -332,7 +336,8 @@ class UpstreamCredentialProvider:
             inbound.get_secret_value(), server, config, tenant_id=subject.tenant_id
         ):
             case Ok(token):
-                return Ok(StaticHeaderAuth(f"Bearer {token.access_token}", header_name="Authorization"))
+                header_name, header_value = config.header(token.access_token)
+                return Ok(StaticHeaderAuth(header_value, header_name=header_name))
             case Error(err):
                 return Error(err)
 

litellm/proxy/_experimental/mcp_server/outbound_credentials/types.py

@@ -31,7 +31,7 @@ from enum import Enum
 from typing import Annotated, Final, Literal
 
 from expression import case, tag, tagged_union
-from pydantic import BaseModel, ConfigDict, Field, SecretStr
+from pydantic import BaseModel, ConfigDict, Field, SecretStr, field_validator
 from typing_extensions import assert_never
 
 from litellm.proxy._experimental.mcp_server.outbound_credentials.result import (
@@ -39,7 +39,11 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.result import (
     Ok,
     Result,
 )
-from litellm.types.mcp import DEFAULT_SUBJECT_TOKEN_TYPE
+from litellm.types.mcp import (
+    DEFAULT_CREDENTIAL_HEADER,
+    DEFAULT_SUBJECT_TOKEN_TYPE,
+    normalize_upstream_header_name,
+)
 
 
 class AuthSpecKind(str, Enum):
@@ -161,7 +165,52 @@ class CredError:
         assert_never(self.tag)
 
 
-class AuthorizationCodeConfig(BaseModel):
+def validate_header_name(raw: str) -> Result[str, CredError]:
+    """``normalize_upstream_header_name`` with this package's error-as-value policy.
+
+    The grammar itself lives in ``litellm.types.mcp`` so the v1 model, the management endpoint and
+    this vocabulary all judge a header name the same way while each keeps its own failure shape.
+    """
+    normalized: Final = normalize_upstream_header_name(raw)
+    if normalized is None:
+        return Error(CredError.of_misconfigured(f"invalid upstream header name: {raw!r}"))
+    return Ok(normalized)
+
+
+class HeaderCarrier(BaseModel):
+    """Where a resolved credential is written upstream, and how its value is formatted.
+
+    ``Authorization: Bearer`` is only OAuth's *default* conveyance (RFC 6750 section 2.1), not its
+    only one: an ESB or API gateway commonly terminates its own credential in a private header while
+    a second credential passes through to the origin, so a credential has to be able to say which
+    slot it owns. Modeled like OpenAPI's apiKey scheme, so any upstream convention is expressible
+    (Authorization + Bearer, a raw value on X-API-Key, Ocp-Apim-Subscription-Key, esb-oauth, ...).
+
+    Every config whose credential the gateway mints or holds inherits this, so no resolver arm names
+    a header itself and the conflict rule in ``_resolve_v2_auth`` can always ask the auth object
+    which slot it is about to occupy. ``passthrough`` deliberately does not: it forwards the
+    caller's own credential into the slot the caller used, and mints nothing to place.
+    """
+
+    model_config = ConfigDict(frozen=True)
+    header_name: str = DEFAULT_CREDENTIAL_HEADER
+    value_prefix: str = "Bearer"
+
+    @field_validator("header_name")
+    @classmethod
+    def _check_header_name(cls, value: str) -> str:
+        match validate_header_name(value):
+            case Ok(name):
+                return name
+            case Error(err):
+                raise ValueError(err.summary)
+
+    def header(self, value: str) -> tuple[str, str]:
+        formatted: Final = f"{self.value_prefix} {value}" if self.value_prefix else value
+        return self.header_name, formatted
+
+
+class AuthorizationCodeConfig(HeaderCarrier):
     """Per-user 3LO; the gateway is the OAuth client and stores the user's token.
 
     Endpoints are discovered (RFC 9728 -> RFC 8414) and the client is registered via DCR
@@ -179,7 +228,7 @@ class AuthorizationCodeConfig(BaseModel):
     token_url: str | None = None
 
 
-class ClientCredentialsConfig(BaseModel):
+class ClientCredentialsConfig(HeaderCarrier):
     """M2M service account; one upstream identity for every user.
 
     Fields are optional so the config can be built incomplete: a value may be supplied at
@@ -203,7 +252,7 @@ class ClientCredentialsConfig(BaseModel):
     token_endpoint_auth_method: Literal["client_secret_post", "client_secret_basic"] | None = None
 
 
-class TokenExchangeConfig(BaseModel):
+class TokenExchangeConfig(HeaderCarrier):
     """OBO: swap the caller's live inbound token for a token bound to the upstream's audience. The
     gateway authenticates to the exchange endpoint as an OAuth client (`client_id`/`client_secret`);
     the inbound token is sent only to that endpoint, never to the upstream.
@@ -255,7 +304,7 @@ class ClientSecretAuth(BaseModel):
 ClientAuth = Annotated[PrivateKeyJwtAuth | ClientSecretAuth, Field(discriminator="source")]
 
 
-class IdJagConfig(BaseModel):
+class IdJagConfig(HeaderCarrier):
     """draft-ietf-oauth-identity-assertion-authz-grant (Okta "AI agent token exchange").
 
     Two legs: leg 1 is an RFC 8693 token exchange at the IdP org AS (`org_token_endpoint`) that
@@ -297,23 +346,16 @@ class Byok(BaseModel):
 ApiKeySource = Annotated[SharedKey | Byok, Field(discriminator="source")]
 
 
-class ApiKeyConfig(BaseModel):
+class ApiKeyConfig(HeaderCarrier):
     """A fixed credential injected as a header. The value is shared (in config) or seeded
-    per-user (pulled from the store); `header_name` and `value_prefix` say where and how it is
-    written, modeled like OpenAPI's apiKey scheme so any upstream convention is expressible
-    (Authorization + Bearer, a raw value on X-API-Key, Ocp-Apim-Subscription-Key, etc.).
+    per-user (pulled from the store); the inherited `header_name` and `value_prefix` say where
+    and how it is written.
     """
 
     model_config = ConfigDict(frozen=True)
     kind: Literal[AuthSpecKind.api_key] = AuthSpecKind.api_key
-    header_name: str = "Authorization"
-    value_prefix: str = "Bearer"
     key_source: ApiKeySource
 
-    def header(self, value: str) -> tuple[str, str]:
-        formatted: Final = f"{self.value_prefix} {value}" if self.value_prefix else value
-        return self.header_name, formatted
-
 
 class PassthroughConfig(BaseModel):
     """Client-driven upstream OAuth; the gateway forwards the client's upstream token."""

litellm/proxy/_experimental/mcp_server/server.py

@@ -246,11 +246,12 @@ def _mcp_meta_trace_carrier(req_ctx: object) -> dict[str, str] | None:
     """The W3C trace context (``traceparent``/``tracestate``) the MCP client
     propagated in the request's ``params._meta`` (SEP-414), or ``None``.
 
-    When present, per the OTel MCP semconv the MCP span parents to this propagated
-    context rather than to the HTTP transport (which is recorded as a link instead).
-    When absent, the span nests under the transport span of the request carrying
-    this specific message, so a streamable-HTTP session that multiplexes many
-    messages still does not glue every message under the session's first request;
+    When present, the MCP span records this propagated context as a span *link*,
+    never the parent — a remote parent would root the span in a trace whose root
+    never reaches the gateway's tracing backend. The span itself nests under the
+    transport span of the request carrying this specific message, so a
+    streamable-HTTP session that multiplexes many messages still does not glue
+    every message under the session's first request;
     see ``resolve_mcp_span_context``. The client's W3C Baggage is
     deliberately excluded: it is caller-controlled, and the otel baggage processor
     stamps allowlisted baggage keys (``litellm.team.id``, ``litellm.metadata.*``,
@@ -432,7 +433,6 @@ if MCP_AVAILABLE:
         _client_forwarded_authorization_headers,
         _resolve_openapi_tool_auth,
         _should_strip_caller_authorization,
-        _without_authorization,
         global_mcp_server_manager,
     )
     from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
@@ -451,6 +451,7 @@ if MCP_AVAILABLE:
         split_server_prefix_from_name,
         strip_known_server_prefix,
     )
+    from litellm.types.mcp import DEFAULT_CREDENTIAL_HEADER, without_header
 
     ######################################################
     ############ MCP Tools List REST API Response Object #
@@ -1732,7 +1733,7 @@ if MCP_AVAILABLE:
                     raw_headers=raw_headers,
                     user_api_key_auth=user_api_key_auth,
                 ):
-                    extra_headers = _without_authorization(extra_headers)
+                    extra_headers = without_header(extra_headers, DEFAULT_CREDENTIAL_HEADER)
         elif is_client_forwarded_mode:
             if not withhold_forwarded_authorization:
                 extra_headers = _client_forwarded_authorization_headers(

litellm/proxy/_lazy_openapi_snapshot.py

@@ -3,18 +3,27 @@ Per-feature OpenAPI snapshot for lazy-loaded routers.
 
 The committed JSON is generated by `python -m litellm.proxy._lazy_openapi_snapshot`
 and consumed at runtime so /openapi.json can show full route info for unloaded
-features without importing them. No CI job regenerates this file; drift surfaces
-only indirectly through check-ui-api-types.yml, which rebuilds schema.d.ts from
-app.openapi() with the committed snapshot injected. After changing any lazily
-loaded route or this generator, rerun the module and commit the JSON, then run
-`npm run gen:api` in ui/litellm-dashboard and commit schema.d.ts.
+features without importing them. check-ui-api-types.yml (mirrored locally by
+`make check`) regenerates this file and fails when the committed copy differs,
+then rebuilds schema.d.ts from app.openapi() with the snapshot injected. After
+changing any lazily loaded route or this generator, rerun the module and commit
+the JSON, then run `npm run gen:api` in ui/litellm-dashboard and commit schema.d.ts.
 """
 
 import json
 import re
 import sys
+from collections.abc import Callable, Mapping
+from dataclasses import dataclass
 from pathlib import Path
-from typing import Final
+from typing import TYPE_CHECKING, Final
+
+from typing_extensions import ReadOnly, TypedDict
+
+if TYPE_CHECKING:
+    from fastapi import FastAPI
+
+    from litellm.proxy._lazy_features import LazyFeature
 
 SNAPSHOT_FILE: Final = Path(__file__).parent / "_lazy_openapi_snapshot.json"
 HTTP_METHOD_SUFFIXES: Final = {
@@ -83,51 +92,84 @@ def _normalize_operation_ids(paths: dict[str, dict]) -> None:
                     break
 
 
-def generate_snapshot() -> dict[str, dict]:
+class SnapshotFragment(TypedDict):
+    paths: ReadOnly[Mapping[str, Mapping[str, object]]]
+    components: ReadOnly[Mapping[str, Mapping[str, object]]]
+
+
+@dataclass(frozen=True, slots=True)
+class SnapshotResult:
+    fragments: Mapping[str, SnapshotFragment]
+    skipped: tuple[str, ...]
+
+
+def _register_feature(app: "FastAPI", feat: "LazyFeature") -> str | None:
     import importlib
 
+    try:
+        feat.register_fn(app, importlib.import_module(feat.module_path))
+    except Exception as exc:
+        sys.stderr.write(f"warning: skip {feat.name}: {exc}\n")
+        return feat.name
+    return None
+
+
+def _feature_fragment(app: "FastAPI", feat: "LazyFeature", used_operation_ids: set[str]) -> SnapshotFragment | None:
     from fastapi.openapi.utils import get_openapi
 
+    from litellm.proxy.proxy_server import ensure_unique_openapi_operation_ids
+
+    feat_routes: Final = [r for r in app.routes if feat.matches(getattr(r, "path", ""))]
+    if not feat_routes:
+        return None
+    _stabilize_multi_method_route_ids(feat_routes)
+    full: Final = get_openapi(title=app.title, version=app.version, routes=feat_routes)
+    paths: Final = full.get("paths", {})
+    _normalize_operation_ids(paths)
+    # Group all of a feature's routes under one tag.
+    for path_ops in paths.values():
+        for method, op in path_ops.items():
+            if isinstance(op, dict):
+                operation_id = op.get("operationId")
+                if isinstance(operation_id, str):
+                    for suffix in HTTP_METHOD_SUFFIXES:
+                        if operation_id.endswith(f"_{suffix}"):
+                            op["operationId"] = operation_id[: -len(suffix)] + method
+                            break
+                op["tags"] = [feat.name]
+    unique: Final = ensure_unique_openapi_operation_ids(full, used_operation_ids)
+    return {
+        "paths": paths,
+        "components": {"schemas": unique.get("components", {}).get("schemas", {})},
+    }
+
+
+def generate_snapshot() -> SnapshotResult:
     from litellm.proxy._lazy_features import LAZY_FEATURES
-    from litellm.proxy.proxy_server import app, ensure_unique_openapi_operation_ids
+    from litellm.proxy.proxy_server import app
 
-    for feat in LAZY_FEATURES:
-        try:
-            module = importlib.import_module(feat.module_path)
-            feat.register_fn(app, module)
-        except Exception as exc:
-            sys.stderr.write(f"warning: skip {feat.name}: {exc}\n")
-
-    fragments: Final[dict[str, dict]] = {}
+    skipped: Final = tuple(name for feat in LAZY_FEATURES if (name := _register_feature(app, feat)) is not None)
     used_operation_ids: Final[set[str]] = set()
-    for feat in LAZY_FEATURES:
-        feat_routes = [r for r in app.routes if feat.matches(getattr(r, "path", ""))]
-        if not feat_routes:
-            continue
-        _stabilize_multi_method_route_ids(feat_routes)
-        full = get_openapi(title=app.title, version=app.version, routes=feat_routes)
-        paths = full.get("paths", {})
-        _normalize_operation_ids(paths)
-        # Group all of a feature's routes under one tag.
-        for path_ops in full.get("paths", {}).values():
-            for method, op in path_ops.items():
-                if isinstance(op, dict):
-                    operation_id = op.get("operationId")
-                    if isinstance(operation_id, str):
-                        for suffix in HTTP_METHOD_SUFFIXES:
-                            if operation_id.endswith(f"_{suffix}"):
-                                op["operationId"] = operation_id[: -len(suffix)] + method
-                                break
-                    op["tags"] = [feat.name]
-        full = ensure_unique_openapi_operation_ids(full, used_operation_ids)
-        fragments[feat.name] = {
-            "paths": paths,
-            "components": {"schemas": full.get("components", {}).get("schemas", {})},
-        }
-    return fragments
+    fragments: Final = {
+        feat.name: fragment
+        for feat in LAZY_FEATURES
+        if (fragment := _feature_fragment(app, feat, used_operation_ids)) is not None
+    }
+    return SnapshotResult(fragments=fragments, skipped=skipped)
+
+
+def main(snapshot_file: Path = SNAPSHOT_FILE, generate: Callable[[], SnapshotResult] = generate_snapshot) -> int:
+    result: Final = generate()
+    if result.skipped:
+        sys.stderr.write(
+            f"error: {len(result.skipped)} feature(s) failed to import, so their fragments would vanish from the "
+            f"snapshot: {', '.join(result.skipped)}\n"
+        )
+        return 1
+    snapshot_file.write_text(json.dumps(result.fragments, indent=2, sort_keys=True) + "\n")
+    sys.stdout.write(f"wrote {len(result.fragments)} feature fragments to {snapshot_file}\n")
+    return 0
 
 
 if __name__ == "__main__":
-    fragments: Final = generate_snapshot()
-    SNAPSHOT_FILE.write_text(json.dumps(fragments, indent=2, sort_keys=True) + "\n")
-    sys.stdout.write(f"wrote {len(fragments)} feature fragments to {SNAPSHOT_FILE}\n")
+    sys.exit(main())

litellm/proxy/_types.py

@@ -2510,6 +2510,18 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase):
             "are skipped for on-demand GET /health as well as the background health loop."
         ),
     )
+    background_health_check_model_groups: tuple[str, ...] | None = Field(
+        None,
+        description=(
+            "Opt-in allowlist of model group names for background health checks and "
+            "health-check routing. When set, the background loop probes only deployments "
+            "whose model_name is listed, and enable_health_check_routing filters unhealthy "
+            "deployments only within the listed groups; every other group, including newly "
+            "added deployments, is skipped and keeps its configured routing strategy. "
+            "When unset, all deployments participate (opt out per deployment via "
+            "model_info.disable_background_health_check)."
+        ),
+    )
     model_list_healthy_only: bool | None = Field(
         None,
         description=(
@@ -2577,6 +2589,10 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase):
         None,
         description="By default, the user calling /team/new is automatically added to the new team as a team admin. If True, proxy admins are no longer auto-added; members explicitly listed in members_with_roles are unaffected. Default is False.",
     )
+    enforce_fallback_model_access: bool | None = Field(
+        None,
+        description="If True, router fallbacks configured in router_settings are only attempted when the calling key (and its team and project) is allowed to call the fallback model; unauthorized fallback targets are skipped and the primary model's error is returned. Default is False.",
+    )
     scheduled_job_stagger: ScheduledJobStaggerSettings | None = Field(
         None,
         description=(

litellm/proxy/auth/fallback_model_access.py

@@ -0,0 +1,90 @@
+"""
+Authorize router fallback targets against the caller's key, team and project model access.
+
+`_enforce_key_and_fallback_model_access` only sees fallbacks the client sends in the request body.
+Fallbacks configured on the router (`router_settings.fallbacks` and friends) are chosen after auth,
+inside the router, so this predicate is injected into the router to re-run the same model access
+checks for each fallback target before it is attempted. Opt-in via
+`general_settings.enforce_fallback_model_access: true`.
+"""
+
+from collections.abc import Callable, Mapping
+from dataclasses import dataclass
+from typing import Final
+
+from pydantic import BaseModel, ValidationError
+
+from litellm._logging import verbose_proxy_logger
+from litellm.proxy._types import ProxyException, UserAPIKeyAuth
+from litellm.proxy.auth.auth_checks import can_key_call_resolved_model
+from litellm.router import Router
+
+
+class _RequestMetadata(BaseModel):
+    user_api_key_auth: UserAPIKeyAuth | None = None
+
+
+class _FallbackAccessSettings(BaseModel):
+    enforce_fallback_model_access: bool = False
+
+
+async def is_model_authorized_for_token(*, model: str, valid_token: UserAPIKeyAuth, llm_router: Router) -> bool:
+    try:
+        await can_key_call_resolved_model(
+            model=model,
+            llm_model_list=None,
+            valid_token=valid_token,
+            llm_router=llm_router,
+        )
+    except ProxyException:
+        return False
+    except Exception as e:  # noqa: BLE001  # fail closed: a lookup failure must neither run the fallback nor replace the provider error
+        verbose_proxy_logger.warning("Skipping fallback to model=%s: authorization lookup failed: %s", model, e)
+        return False
+    return True
+
+
+def _token_in_metadata(metadata: object) -> UserAPIKeyAuth | None:
+    try:
+        return _RequestMetadata.model_validate(metadata).user_api_key_auth
+    except ValidationError:
+        return None
+
+
+def _user_api_key_auth_from_request(request_kwargs: Mapping[str, object]) -> UserAPIKeyAuth | None:
+    return next(
+        (
+            token
+            for field in ("metadata", "litellm_metadata")
+            if (token := _token_in_metadata(request_kwargs.get(field))) is not None
+        ),
+        None,
+    )
+
+
+def _enforced_by_general_settings() -> bool:
+    from litellm.proxy.proxy_server import general_settings
+
+    return _FallbackAccessSettings.model_validate(general_settings).enforce_fallback_model_access
+
+
+@dataclass(frozen=True, slots=True)
+class RouterFallbackAccessCheck:
+    """
+    `FallbackAccessCheck` for the proxy's router: while `is_enforced()` is true, a fallback target
+    is attempted only when the key behind the request could have requested it directly. Requests
+    that carry no key (for example internal health checks) are not restricted.
+    """
+
+    is_enforced: Callable[[], bool]
+
+    async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], llm_router: Router) -> bool:
+        if not self.is_enforced():
+            return True
+        valid_token: Final = _user_api_key_auth_from_request(request_kwargs)
+        if valid_token is None:
+            return True
+        return await is_model_authorized_for_token(model=model, valid_token=valid_token, llm_router=llm_router)
+
+
+router_fallback_access_check: Final = RouterFallbackAccessCheck(is_enforced=_enforced_by_general_settings)

litellm/proxy/auth/user_api_key_auth.py

@@ -1769,7 +1769,12 @@ async def _user_api_key_auth_builder(
 
             return valid_token
 
-        if valid_token is not None and isinstance(valid_token, UserAPIKeyAuth) and valid_token.team_id is not None:
+        if (
+            valid_token is not None
+            and isinstance(valid_token, UserAPIKeyAuth)
+            and valid_token.team_id is not None
+            and valid_token.team_id != UI_TEAM_ID
+        ):
             ## UPDATE TEAM VALUES BASED ON CACHED TEAM OBJECT - allows `/team/update` values to work for cached token
             try:
                 team_obj: Final[LiteLLM_TeamTableCachedObj] = await get_team_object(
@@ -2149,6 +2154,8 @@ async def _user_api_key_auth_builder(
             # Check 6: Additional Common Checks across jwt + key auth
             if valid_token.team_id is not None:
                 try:
+                    if valid_token.team_id == UI_TEAM_ID:
+                        raise TeamNotFoundError(team_id=UI_TEAM_ID)
                     with tracer.trace("litellm.proxy.auth.get_team_object"):
                         _team_obj = await get_team_object(
                             team_id=valid_token.team_id,
@@ -2443,7 +2450,7 @@ async def _run_centralized_common_checks(
         )
 
     fetch_coros: Final = []
-    if user_api_key_auth_obj.team_id is not None:
+    if user_api_key_auth_obj.team_id is not None and user_api_key_auth_obj.team_id != UI_TEAM_ID:
         fetch_coros.append(
             _safe_fetch(
                 "team",
@@ -2567,7 +2574,9 @@ async def _run_centralized_common_checks(
         else:
             raise team_result
     else:
-        team_object = team_result
+        team_object = (
+            _team_obj_from_token(user_api_key_auth_obj) if user_api_key_auth_obj.team_id == UI_TEAM_ID else team_result
+        )
 
     user_object: LiteLLM_UserTable | None = None if isinstance(user_result, BaseException) else user_result
     project_object: Final[LiteLLM_ProjectTableCachedObj | None] = (

litellm/proxy/common_request_processing.py

@@ -21,14 +21,13 @@ import litellm
 from litellm._logging import _redact_string, verbose_proxy_logger
 from litellm._uuid import uuid
 from litellm.constants import (
-    AUTO_ROUTED_REQUEST_METADATA_KEY,
     DD_TRACER_STREAMING_CHUNK_YIELD_RESOURCE,
     DEFAULT_MAX_RECURSE_DEPTH,
     LITELLM_DETAILED_TIMING,
     LITELLM_HTTP_STATUS_CLIENT_DISCONNECTED,
     MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG,
+    NON_INFERENCE_CALL_TYPES,
     RETURN_RAW_MODEL_NAME_METADATA_KEY,
-    ROUTER_MODEL_NAME_RESPONSE_FIELD,
     STREAM_SSE_DATA_PREFIX,
     STREAM_SSE_KEEPALIVE_PING_BYTES,
     UNSAFE_PROXY_RESPONSE_HEADERS,
@@ -39,6 +38,7 @@ from litellm.litellm_core_utils.dd_tracing import NullTracer, tracer
 from litellm.litellm_core_utils.get_supported_openai_params import (
     get_supported_openai_params,
 )
+from litellm.litellm_core_utils.internal_call_metadata import is_unbilled_non_inference_call_from_params
 from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
 from litellm.litellm_core_utils.llm_cost_calc.guardrail_cost import guardrail_information_cost
 from litellm.litellm_core_utils.llm_response_utils.get_headers import (
@@ -417,15 +417,6 @@ def _litellm_model_supports_stream_options(litellm_model: str) -> bool:
     return supported_params is not None and "stream_options" in supported_params
 
 
-def _deployment_litellm_model(deployment: Mapping[str, object]) -> str | None:
-    litellm_params: Final = deployment.get("litellm_params")
-    if isinstance(litellm_params, Mapping):
-        litellm_model = litellm_params.get("model")
-    else:
-        litellm_model = getattr(litellm_params, "model", None)
-    return litellm_model if isinstance(litellm_model, str) else None
-
-
 def _model_deployments_support_stream_options(
     model: object,
     llm_router: Router | None,
@@ -433,11 +424,8 @@ def _model_deployments_support_stream_options(
 ) -> bool:
     if not isinstance(model, str):
         return False
-    deployments = llm_router.get_model_list(model_name=model, team_id=team_id) if llm_router is not None else None
-    deployment_models: Final = tuple(
-        litellm_model
-        for deployment in deployments or ()
-        if (litellm_model := _deployment_litellm_model(deployment)) is not None
+    deployment_models: Final = (
+        llm_router.resolved_litellm_models(model, team_id=team_id) if llm_router is not None else ()
     )
     candidate_models: Final = deployment_models if deployment_models else (model,)
     return all(_litellm_model_supports_stream_options(m) for m in candidate_models)
@@ -1302,15 +1290,51 @@ def _uncached_input_cost(
     return input_cost - (cache_read_cost or 0.0) - (cache_creation_cost or 0.0)
 
 
+_ZERO_COST_BREAKDOWN: Final = CostBreakdownHeaderValues(
+    original_cost=0.0,
+    discount_amount=0.0,
+    margin_total_amount=0.0,
+    margin_percent=0.0,
+    input_cost=0.0,
+    output_cost=0.0,
+    tool_usage_cost=0.0,
+)
+"""The component split a call priced at zero advertises, so a client reading the cost headers off a
+read or management route still finds the whole family rather than a partially populated one."""
+
+
+def _totals_to_zero(response_cost: float | str | None) -> bool:
+    """Whether the total these headers carry is zero, counting a total no route ever priced as one.
+
+    A component split is only reported as zero alongside a total that agrees with it, so a read
+    that did price normally never advertises a real total beside an all-zero split.
+    """
+    if response_cost is None or response_cost == "":
+        return True
+    try:
+        return float(response_cost) == 0.0
+    except (TypeError, ValueError):
+        return False
+
+
 def _get_cost_breakdown_from_logging_obj(
     litellm_logging_obj: LiteLLMLoggingObj | None,
+    response_cost: float | str | None = None,
 ) -> CostBreakdownHeaderValues:
-    """Extract discount, margin, and per-component cost information from logging object's cost breakdown."""
+    """Extract discount, margin, and per-component cost information from logging object's cost breakdown.
+
+    A non-inference call that priced at zero never records a breakdown, so its components are
+    reported as zero here. Any such call that did price normally (retrieving a background response,
+    and the cost poller's read of one) reports the breakdown it stored, or nothing at all when the
+    breakdown has not landed yet.
+    """
     if not litellm_logging_obj or not hasattr(litellm_logging_obj, "cost_breakdown"):
         return CostBreakdownHeaderValues()
 
     cost_breakdown: Final = litellm_logging_obj.cost_breakdown
     if not cost_breakdown:
+        if litellm_logging_obj.call_type in NON_INFERENCE_CALL_TYPES and _totals_to_zero(response_cost):
+            return _ZERO_COST_BREAKDOWN
         return CostBreakdownHeaderValues()
 
     return CostBreakdownHeaderValues(
@@ -1459,7 +1483,9 @@ class ProxyBaseLLMRequestProcessing:
         exclude_values: Final = {"", None, "None"}
         hidden_params = hidden_params or {}
 
-        cost_breakdown: Final = _get_cost_breakdown_from_logging_obj(litellm_logging_obj=litellm_logging_obj)
+        cost_breakdown: Final = _get_cost_breakdown_from_logging_obj(
+            litellm_logging_obj=litellm_logging_obj, response_cost=response_cost
+        )
 
         # Calculate updated spend for header (include current response_cost)
         current_spend: Final = user_api_key_dict.spend or 0.0
@@ -2036,54 +2062,6 @@ class ProxyBaseLLMRequestProcessing:
                 return deployment
         return None
 
-    @staticmethod
-    def get_router_selected_model_name(
-        litellm_logging_obj: LiteLLMLoggingObj | None,
-    ) -> str | None:
-        """Model group an auto-routing strategy selected, or None if none fired.
-
-        The marker and ``deployment_model_name`` are written by different bucket
-        resolvers (``get_or_create_metadata_bucket`` vs
-        ``_get_router_metadata_variable_name``), so they can land in different
-        buckets on the same request. Resolve each across both.
-        """
-        litellm_params: Final = getattr(litellm_logging_obj, "litellm_params", None)
-        if not isinstance(litellm_params, dict):
-            return None
-        buckets: Final = tuple(
-            bucket for key in ("litellm_metadata", "metadata") if isinstance(bucket := litellm_params.get(key), dict)
-        )
-        if not any(bucket.get(AUTO_ROUTED_REQUEST_METADATA_KEY) is True for bucket in buckets):
-            return None
-        return next(
-            (
-                model_group
-                for bucket in buckets
-                if isinstance(model_group := bucket.get("deployment_model_name"), str) and model_group
-            ),
-            None,
-        )
-
-    @staticmethod
-    def set_router_selected_model_field(
-        *,
-        response_obj: object,
-        router_model_name: str | None,
-    ) -> None:
-        if not router_model_name:
-            return
-        if isinstance(response_obj, dict):
-            response_obj[ROUTER_MODEL_NAME_RESPONSE_FIELD] = router_model_name
-            return
-        try:
-            setattr(response_obj, ROUTER_MODEL_NAME_RESPONSE_FIELD, router_model_name)
-        except (AttributeError, TypeError, ValueError):
-            verbose_proxy_logger.debug(
-                "Could not set %s on response object of type %s",
-                ROUTER_MODEL_NAME_RESPONSE_FIELD,
-                type(response_obj),
-            )
-
     @staticmethod
     def _response_cost_from_logging_obj(
         *,
@@ -2582,20 +2560,21 @@ class ProxyBaseLLMRequestProcessing:
                 log_context=f"litellm_call_id={logging_obj.litellm_call_id}",
                 return_raw_model_name=_should_return_raw_model_name(self.data),
             )
-        self.set_router_selected_model_field(
-            response_obj=response,
-            router_model_name=self.get_router_selected_model_name(logging_obj),
-        )
 
         hidden_params = get_hidden_params_dict(response)  # get any updated response headers
         additional_headers = hidden_params.get("additional_headers", {}) or {}
 
         recover_response_cost: Final = not response_cost and hidden_params.get("response_cost") is None
-        llm_cost_for_headers: Final = (
+        computed_cost_for_headers: Final = (
             self._response_cost_from_logging_obj(response=response, logging_obj=logging_obj) or ""
             if recover_response_cost
             else response_cost
         )
+        llm_cost_for_headers: Final = (
+            0.0
+            if is_unbilled_non_inference_call_from_params(logging_obj.call_type, logging_obj.litellm_params, response)
+            else computed_cost_for_headers
+        )
         _, request_metadata_bucket = get_or_create_metadata_bucket(self.data)
         guardrail_cost_for_headers: Final = guardrail_information_cost(
             request_metadata_bucket.get("standard_logging_guardrail_information")

litellm/proxy/common_utils/reset_budget_job.py

@@ -1,5 +1,6 @@
 import asyncio
 import json
+import math
 import time
 from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
 from dataclasses import dataclass
@@ -45,6 +46,7 @@ from litellm.repositories.table_repositories import (
 )
 from litellm.repositories.team_repository import TeamRepository
 from litellm.repositories.unit_of_work import (
+    LinkedSpendResetWrites,
     budget_cascade_unit_of_work,
     spend_reset_unit_of_work,
 )
@@ -59,7 +61,15 @@ _LINKED_KEYS_WHERE: Final[Mapping[str, object]] = MappingProxyType({"budget_dura
 _SPENT_ROWS_WHERE: Final[Mapping[str, object]] = MappingProxyType({"spend": {"gt": 0}})
 
 
-class _TeamMembershipRow(Protocol):
+class _BudgetLinkedRow(Protocol):
+    @property
+    def spend(self) -> float | None: ...
+
+    @property
+    def budget_id(self) -> str | None: ...
+
+
+class _TeamMembershipRow(_BudgetLinkedRow, Protocol):
     @property
     def user_id(self) -> str: ...
 
@@ -67,26 +77,48 @@ class _TeamMembershipRow(Protocol):
     def team_id(self) -> str: ...
 
 
-class _KeyRow(Protocol):
+class _KeyRow(_BudgetLinkedRow, Protocol):
     @property
     def token(self) -> str: ...
 
 
-class _OrgRow(Protocol):
+class _OrgRow(_BudgetLinkedRow, Protocol):
     @property
     def organization_id(self) -> str: ...
 
 
-class _TagRow(Protocol):
+class _TagRow(_BudgetLinkedRow, Protocol):
     @property
     def tag_name(self) -> str: ...
 
 
-class _EndUserRow(Protocol):
+class _EndUserRow(_BudgetLinkedRow, Protocol):
     @property
     def user_id(self) -> str: ...
 
 
+def _rollover_enabled() -> bool:
+    return litellm.budget_rollover is True
+
+
+def _rollover_cap(max_budget: float | None) -> float | None:
+    if max_budget is None or not math.isfinite(max_budget):
+        return None
+    return max_budget
+
+
+def _carried_spend(spend: float | None, cap: float | None) -> float:
+    if cap is None:
+        return 0.0
+    return max(0.0, (spend or 0.0) - cap)
+
+
+def _row_carried_spend(row: _BudgetLinkedRow, caps: Mapping[str, float]) -> float:
+    if not caps:
+        return 0.0
+    return _carried_spend(row.spend, caps.get(row.budget_id) if row.budget_id is not None else None)
+
+
 def _team_membership_counter_key(row: _TeamMembershipRow) -> str:
     return f"spend:team_member:{row.user_id}:{row.team_id}"
 
@@ -129,6 +161,59 @@ def _budget_link_where(
     return {"budget_id": {"in": list(budget_ids)}, **extra}
 
 
+def _queue_budget_linked_resets(
+    writes: LinkedSpendResetWrites,
+    cascade: "_BudgetCascade",
+    extra: Mapping[str, object] = MappingProxyType({}),
+) -> None:
+    """Reset one linked table's spend for every expiring tier: tiers with a
+    rollover cap keep spend beyond the cap (decrement preserves writes racing
+    the reset), everything else is zeroed as before. Zero the under-cap rows
+    BEFORE decrementing the over-cap ones: the statements run sequentially in
+    one transaction, so the reverse order lets the zero re-match a row the
+    decrement just moved into the (0, cap] range and erase its carried spend."""
+    for budget_id, cap in cascade.rollover_caps.items():
+        writes.queue_spend_zero(
+            where={"budget_id": budget_id, **extra, "spend": {"gt": 0, "lte": cap}}
+        )  # mutable-ok: prisma where filter must be a dict
+        writes.queue_spend_decrement(
+            where={"budget_id": budget_id, **extra, "spend": {"gt": cap}}, amount=cap
+        )  # mutable-ok: prisma where filter must be a dict
+    plain_ids: Final = tuple(bid for bid in cascade.budget_ids if bid not in cascade.rollover_caps)
+    if plain_ids:
+        writes.queue_spend_zero(where=_budget_link_where(plain_ids, extra))
+
+
+def _queue_enduser_resets(writes: LinkedSpendResetWrites, cascade: "_BudgetCascade") -> None:
+    """End users are matched by id rather than budget link: rows with no
+    budget_id ride the default budget tier (litellm.max_end_user_budget_id).
+    Zero-before-decrement ordering matters here too (see
+    _queue_budget_linked_resets)."""
+    if not cascade.rollover_caps:
+        if cascade.endusers:
+            writes.queue_spend_zero(
+                where={"user_id": {"in": [row.user_id for row in cascade.endusers]}}
+            )  # mutable-ok: prisma where filter must be a dict
+        return
+    tiered: Final = tuple((row.budget_id or litellm.max_end_user_budget_id, row.user_id) for row in cascade.endusers)
+    for budget_id, cap in cascade.rollover_caps.items():
+        if not (
+            user_ids := [uid for bid, uid in tiered if bid == budget_id]
+        ):  # mutable-ok: prisma "in" filter takes a list
+            continue
+        writes.queue_spend_zero(
+            where={"user_id": {"in": user_ids}, "spend": {"lte": cap}}
+        )  # mutable-ok: prisma where filter must be a dict
+        writes.queue_spend_decrement(
+            where={"user_id": {"in": user_ids}, "spend": {"gt": cap}}, amount=cap
+        )  # mutable-ok: prisma where filter must be a dict
+    plain: Final = [
+        uid for bid, uid in tiered if bid is None or bid not in cascade.rollover_caps
+    ]  # mutable-ok: prisma "in" filter takes a list
+    if plain:
+        writes.queue_spend_zero(where={"user_id": {"in": plain}})  # mutable-ok: prisma where filter must be a dict
+
+
 @dataclass(frozen=True, slots=True)
 class _BudgetCascade:
     """Everything one budget-tier reset touches, resolved before any write."""
@@ -137,8 +222,9 @@ class _BudgetCascade:
     budget_ids: tuple[str, ...] = ()
     budget_resets: tuple[tuple[str, datetime], ...] = ()
     endusers: tuple[_EndUserRow, ...] = ()
-    counter_keys: tuple[str, ...] = ()
+    counter_resets: tuple[tuple[str, float], ...] = ()
     cache_keys: tuple[str, ...] = ()
+    rollover_caps: Mapping[str, float] = MappingProxyType({})
 
 
 @dataclass(frozen=True, slots=True)
@@ -404,8 +490,10 @@ class ResetBudgetJob:
         )
 
     @staticmethod
-    async def _invalidate_spend_counter(counter_key: str) -> None:
-        """Zero a spend counter so a DB-row reset takes effect immediately.
+    async def _invalidate_spend_counter(counter_key: str, new_spend: float = 0.0) -> None:
+        """Overwrite a spend counter with the post-reset value (0, or the carried
+        overage when budget rollover is enabled) so a DB-row reset takes effect
+        immediately.
 
         Call AFTER the DB write commits. Clearing Redis before the DB
         commit opens a window where get_current_spend reads 0 from Redis
@@ -414,10 +502,10 @@ class ResetBudgetJob:
         try:
             from litellm.proxy.proxy_server import spend_counter_cache
 
-            spend_counter_cache.in_memory_cache.set_cache(key=counter_key, value=0.0, ttl=60)
+            spend_counter_cache.in_memory_cache.set_cache(key=counter_key, value=new_spend, ttl=60)
             if spend_counter_cache.redis_cache is not None:
                 try:
-                    await spend_counter_cache.redis_cache.async_set_cache(key=counter_key, value=0.0, ttl=60)
+                    await spend_counter_cache.redis_cache.async_set_cache(key=counter_key, value=new_spend, ttl=60)
                 except Exception as redis_err:
                     verbose_proxy_logger.warning(
                         "Failed to reset spend counter %s in Redis: %s. "
@@ -522,6 +610,15 @@ class ResetBudgetJob:
             where=_budget_link_where(budget_ids, _SPENT_ROWS_WHERE),
             log_subject="tags",
         )
+        rollover_caps: Final[Mapping[str, float]] = MappingProxyType(
+            {  # mutable-ok: MappingProxyType wraps a one-shot dict comprehension
+                b.budget_id: cap
+                for b in budgets_to_reset
+                if b.budget_id is not None and (cap := _rollover_cap(b.max_budget)) is not None
+            }
+            if _rollover_enabled()
+            else {}  # mutable-ok: empty sentinel immediately frozen by MappingProxyType
+        )
         return _BudgetCascade(
             budgets=tuple(budgets_to_reset),
             budget_ids=budget_ids,
@@ -534,12 +631,16 @@ class ResetBudgetJob:
                 if b.budget_id is not None and b.budget_duration is not None
             ),
             endusers=await self._collect_endusers_to_reset(budget_ids),
-            counter_keys=(
-                *(_team_membership_counter_key(row) for row in team_memberships),
-                *(_key_counter_key(row) for row in keys),
-                *(_org_counter_key(row) for row in orgs),
-                *(_tag_counter_key(row) for row in tags),
+            counter_resets=(
+                *(
+                    (_team_membership_counter_key(row), _row_carried_spend(row, rollover_caps))
+                    for row in team_memberships
+                ),
+                *((_key_counter_key(row), _row_carried_spend(row, rollover_caps)) for row in keys),
+                *((_org_counter_key(row), _row_carried_spend(row, rollover_caps)) for row in orgs),
+                *((_tag_counter_key(row), _row_carried_spend(row, rollover_caps)) for row in tags),
             ),
+            rollover_caps=rollover_caps,
             cache_keys=(
                 *(key for row in team_memberships for key in _team_membership_cache_keys(row)),
                 *(key for row in keys for key in _key_cache_keys(row)),
@@ -565,20 +666,18 @@ class ResetBudgetJob:
         )
 
     async def _commit_budget_cascade_once(self, cascade: _BudgetCascade) -> None:
-        enduser_ids: Final = tuple(row.user_id for row in cascade.endusers)
         async with budget_cascade_unit_of_work(self.prisma_client.db.batch_) as uow:
-            uow.team_memberships.queue_spend_zero(where=_budget_link_where(cascade.budget_ids))
-            uow.keys.queue_spend_zero(where=_budget_link_where(cascade.budget_ids, _LINKED_KEYS_WHERE))
-            uow.organizations.queue_spend_zero(where=_budget_link_where(cascade.budget_ids, _SPENT_ROWS_WHERE))
-            uow.tags.queue_spend_zero(where=_budget_link_where(cascade.budget_ids, _SPENT_ROWS_WHERE))
-            if enduser_ids:
-                uow.endusers.queue_spend_zero(where={"user_id": {"in": list(enduser_ids)}})
+            _queue_budget_linked_resets(uow.team_memberships, cascade)
+            _queue_budget_linked_resets(uow.keys, cascade, extra=_LINKED_KEYS_WHERE)
+            _queue_budget_linked_resets(uow.organizations, cascade, extra=_SPENT_ROWS_WHERE)
+            _queue_budget_linked_resets(uow.tags, cascade, extra=_SPENT_ROWS_WHERE)
+            _queue_enduser_resets(uow.endusers, cascade)
             for budget_id, budget_reset_at in cascade.budget_resets:
                 uow.budgets.queue_window_advance(budget_id=budget_id, budget_reset_at=budget_reset_at)
 
     async def _invalidate_budget_cascade_caches(self, cascade: _BudgetCascade) -> None:
-        for counter_key in cascade.counter_keys:
-            await self._invalidate_spend_counter(counter_key)
+        for counter_key, new_spend in cascade.counter_resets:
+            await self._invalidate_spend_counter(counter_key, new_spend=new_spend)
         for cache_key in cascade.cache_keys:
             await self._invalidate_user_api_key_cache_entry(cache_key)
 
@@ -708,7 +807,11 @@ class ResetBudgetJob:
             for k in updated_keys:
                 if k.token is None:
                     continue
-                uow.keys.queue_spend_reset(token=k.token, budget_reset_at=k.budget_reset_at)
+                uow.keys.queue_spend_reset(
+                    token=k.token,
+                    budget_reset_at=k.budget_reset_at,
+                    spend_decrement=k.max_budget if (k.spend or 0.0) > 0.0 else None,
+                )
 
     async def _write_user_reset_updates(self, updated_users: list[LiteLLM_UserTable]) -> None:
         """
@@ -726,7 +829,11 @@ class ResetBudgetJob:
     async def _write_user_reset_updates_once(self, updated_users: list[LiteLLM_UserTable]) -> None:
         async with spend_reset_unit_of_work(self.prisma_client.db.batch_) as uow:
             for u in updated_users:
-                uow.users.queue_spend_reset(user_id=u.user_id, budget_reset_at=u.budget_reset_at)
+                uow.users.queue_spend_reset(
+                    user_id=u.user_id,
+                    budget_reset_at=u.budget_reset_at,
+                    spend_decrement=u.max_budget if (u.spend or 0.0) > 0.0 else None,
+                )
 
     async def _write_team_reset_updates(self, updated_teams: list[LiteLLM_TeamTable]) -> None:
         """
@@ -744,7 +851,11 @@ class ResetBudgetJob:
     async def _write_team_reset_updates_once(self, updated_teams: list[LiteLLM_TeamTable]) -> None:
         async with spend_reset_unit_of_work(self.prisma_client.db.batch_) as uow:
             for t in updated_teams:
-                uow.teams.queue_spend_reset(team_id=t.team_id, budget_reset_at=t.budget_reset_at)
+                uow.teams.queue_spend_reset(
+                    team_id=t.team_id,
+                    budget_reset_at=t.budget_reset_at,
+                    spend_decrement=t.max_budget if (t.spend or 0.0) > 0.0 else None,
+                )
 
     def _emit_phase_failure(
         self,
@@ -820,7 +931,7 @@ class ResetBudgetJob:
                     for k in updated_keys:
                         token = getattr(k, "token", None)
                         if token:
-                            await self._invalidate_spend_counter(f"spend:key:{token}")
+                            await self._invalidate_spend_counter(f"spend:key:{token}", new_spend=k.spend or 0.0)
 
             end_time = time.time()
             outcome: Final = _ChunkOutcome(
@@ -925,7 +1036,7 @@ class ResetBudgetJob:
                     for u in updated_users:
                         user_id = getattr(u, "user_id", None)
                         if user_id:
-                            await self._invalidate_spend_counter(f"spend:user:{user_id}")
+                            await self._invalidate_spend_counter(f"spend:user:{user_id}", new_spend=u.spend or 0.0)
                         if user_id == LITELLM_PROXY_BUDGET_NAME:
                             await self._invalidate_global_proxy_spend_cache()
 
@@ -1034,7 +1145,7 @@ class ResetBudgetJob:
                     for t in updated_teams:
                         team_id = getattr(t, "team_id", None)
                         if team_id:
-                            await self._invalidate_spend_counter(f"spend:team:{team_id}")
+                            await self._invalidate_spend_counter(f"spend:team:{team_id}", new_spend=t.spend or 0.0)
 
             end_time = time.time()
             outcome: Final = _ChunkOutcome(
@@ -1107,10 +1218,11 @@ class ResetBudgetJob:
         reset_at: Final = datetime.fromisoformat(reset_at_str.replace("Z", "+00:00")).replace(tzinfo=None)
         if reset_at > now:
             return False
-        spend_counter_cache.in_memory_cache.set_cache(key=counter_key, value=0.0)
+        new_value: Final = await ResetBudgetJob._window_carried_spend(window, counter_key, spend_counter_cache)
+        spend_counter_cache.in_memory_cache.set_cache(key=counter_key, value=new_value)
         if spend_counter_cache.redis_cache is not None:
             try:
-                await spend_counter_cache.redis_cache.async_set_cache(key=counter_key, value=0.0)
+                await spend_counter_cache.redis_cache.async_set_cache(key=counter_key, value=new_value)
             except Exception as redis_err:
                 verbose_proxy_logger.warning("Failed to reset Redis counter %s: %s", counter_key, redis_err)
         window["reset_at"] = compute_budget_reset_at(
@@ -1118,6 +1230,27 @@ class ResetBudgetJob:
         ).isoformat()
         return True
 
+    @staticmethod
+    async def _window_carried_spend(
+        window: Mapping[str, object], counter_key: str, spend_counter_cache: DualCache
+    ) -> float:
+        """Per-window spend lives only in the counter, so the carried overage is
+        read from it before the reset overwrites it."""
+        if not _rollover_enabled():
+            return 0.0
+        window_max: Final = window.get("max_budget")
+        cap: Final = _rollover_cap(window_max) if isinstance(window_max, (int, float)) else None
+        if cap is None:
+            return 0.0
+        try:
+            current: Final = await spend_counter_cache.async_get_cache(key=counter_key)
+        except Exception as e:  # noqa: BLE001  # an unreadable counter falls back to a plain zero reset
+            verbose_proxy_logger.warning("Failed to read spend counter %s for rollover: %s", counter_key, e)
+            return 0.0
+        if not isinstance(current, (int, float)):
+            return 0.0
+        return _carried_spend(float(current), cap)
+
     async def reset_budget_windows(self) -> None:
         """
         For keys and teams with budget_limits, reset any individual windows where
@@ -1222,7 +1355,7 @@ class ResetBudgetJob:
         still holds the pre-reset value, admitting requests past the cap.
         """
         try:
-            item.spend = 0.0
+            item.spend = _carried_spend(item.spend, _rollover_cap(item.max_budget)) if _rollover_enabled() else 0.0
             if hasattr(item, "budget_duration") and item.budget_duration is not None:
                 item.budget_reset_at = compute_budget_reset_at(
                     budget_duration=item.budget_duration, settings=reset_settings

litellm/proxy/config_resolvers/alerting.py

@@ -25,3 +25,7 @@ EMAIL_DESCRIPTORS: Final[tuple[FieldDescriptor, ...]] = (
 SLACK_DESCRIPTORS: Final[tuple[FieldDescriptor, ...]] = (
     FieldDescriptor("SLACK_WEBHOOK_URL", "SLACK_WEBHOOK_URL", "SLACK_WEBHOOK_URL", is_secret=True),
 )
+
+MS_TEAMS_DESCRIPTORS: Final[tuple[FieldDescriptor, ...]] = (
+    FieldDescriptor("MS_TEAMS_WEBHOOK_URL", "MS_TEAMS_WEBHOOK_URL", "MS_TEAMS_WEBHOOK_URL", is_secret=True),
+)

litellm/proxy/db/prisma_client.py

@@ -887,6 +887,22 @@ class PrismaManager:
             return
         ProxyExtrasDBManager.apply_replica_identity_full_if_requested()
 
+    @staticmethod
+    def _raise_if_partitioned_spend_logs() -> None:
+        """`prisma db push` rewrites a doc-partitioned LiteLLM_SpendLogs
+        primary key back to ("request_id"), which Postgres rejects. Fail fast
+        with guidance instead of retrying into that raw error. No-op when
+        litellm-proxy-extras is absent."""
+        try:
+            from litellm_proxy_extras.utils import (
+                PARTITIONED_SPEND_LOGS_PUSH_ERROR,
+                ProxyExtrasDBManager,
+            )
+        except ImportError:
+            return
+        if ProxyExtrasDBManager.spend_logs_is_partitioned():
+            raise RuntimeError(PARTITIONED_SPEND_LOGS_PUSH_ERROR)
+
     @staticmethod
     def setup_database(use_migrate: bool = False, use_v2_resolver: bool = False) -> bool:
         """
@@ -921,6 +937,7 @@ class PrismaManager:
                         use_v2_resolver=use_v2_resolver,
                     )
                 else:
+                    PrismaManager._raise_if_partitioned_spend_logs()
                     # Use prisma db push with increased timeout
                     subprocess.run(
                         [

litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py

@@ -686,6 +686,7 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
         aws_profile_name: Final = self.optional_params.get("aws_profile_name", None)
         aws_web_identity_token: Final = self.optional_params.get("aws_web_identity_token", None)
         aws_sts_endpoint: Final = self.optional_params.get("aws_sts_endpoint", None)
+        aws_external_id: Final = self.optional_params.get("aws_external_id", None)
 
         ### SET REGION NAME ###
         aws_region_name = self.get_aws_region_name_for_non_llm_api_calls(
@@ -702,6 +703,7 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
             aws_role_name=aws_role_name,
             aws_web_identity_token=aws_web_identity_token,
             aws_sts_endpoint=aws_sts_endpoint,
+            aws_external_id=aws_external_id,
         )
         return credentials, aws_region_name
 

litellm/proxy/guardrails/guardrail_hooks/crowdstrike_aidr/__init__.py

@@ -25,6 +25,7 @@ def initialize_guardrail(litellm_params: "LitellmParams", guardrail: "Guardrail"
             GuardrailEventHooks.post_call.value,
         ],
         default_on=litellm_params.default_on,
+        fail_on_error=litellm_params.fail_on_error,
     )
     litellm.logging_callback_manager.add_litellm_callback(_crowdstrike_aidr_callback)
 

litellm/proxy/guardrails/guardrail_hooks/crowdstrike_aidr/crowdstrike_aidr.py

@@ -1,10 +1,11 @@
 import json
 import os
+import time
 from collections.abc import Mapping, Sequence
 from typing import TYPE_CHECKING, Annotated, Final, Literal, NamedTuple, Optional, cast
 
 from fastapi import HTTPException
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
 from typing_extensions import Any, override
 
 from litellm._logging import verbose_proxy_logger
@@ -142,7 +143,7 @@ def _extract_text_from_message(message: _Message) -> str:
     return "\n".join(part.text for part in content if isinstance(part, _TextContentPart))
 
 
-def _merge_metadata_bags(request_data: Mapping[str, Any]) -> dict[str, Any] | None:
+def _merge_metadata_bags(request_data: Mapping[str, Any]) -> Mapping[str, Any] | None:
     merged: Final[dict[str, Any]] = {}
     present = False
     for bag in (request_data.get("metadata"), request_data.get("litellm_metadata")):
@@ -153,7 +154,7 @@ def _merge_metadata_bags(request_data: Mapping[str, Any]) -> dict[str, Any] | No
 
 
 def _messages_since_last_assistant(
-    messages: list[AllMessageValues],
+    messages: Sequence[AllMessageValues],
 ) -> _FilteredMessages:
     if not messages:
         return _FilteredMessages([], ())
@@ -239,6 +240,7 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
         guardrail_name: str,
         api_key: str | None = None,
         api_base: str | None = None,
+        fail_on_error: bool | None = True,
         **kwargs,
     ) -> None:
         """
@@ -251,6 +253,7 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
             **kwargs: Additional arguments passed to the CustomGuardrail base class.
         """
         self.async_handler = get_async_httpx_client(llm_provider=httpxSpecialProvider.GuardrailCallback)
+        self.fail_on_error = True if fail_on_error is None else fail_on_error
 
         self.api_key = api_key or os.environ.get("CS_AIDR_TOKEN")
         if not self.api_key:
@@ -306,11 +309,13 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
         assert response is not None
         response.raise_for_status()
 
-        result = _GuardChatCompletionsResponse.model_validate(response.json()).result or _GuardChatCompletionsResult()
+        response_body: Final[object] = response.json()
+        raw_result: Final[object] = response_body.get("result") if isinstance(response_body, dict) else None
+        blocked_signal: Final[object] = raw_result.get("blocked") if isinstance(raw_result, dict) else None
 
-        if result.blocked:
+        if blocked_signal:
             verbose_proxy_logger.warning(
-                "CrowdStrike AIDR Guardrail (%s): Request blocked. Response: %s", hook_name, result
+                "CrowdStrike AIDR Guardrail (%s): Request blocked. Verdict: %s", hook_name, blocked_signal
             )
             raise HTTPException(
                 status_code=400,  # Bad Request, indicating violation
@@ -319,6 +324,23 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
                     "guardrail_name": self.guardrail_name,
                 },
             )
+
+        try:
+            result: Final = (
+                _GuardChatCompletionsResponse.model_validate(response_body).result or _GuardChatCompletionsResult()
+            )
+        except ValidationError as validation_error:
+            transformed_signal: Final[object] = raw_result.get("transformed") if isinstance(raw_result, dict) else None
+            if transformed_signal:
+                raise HTTPException(
+                    status_code=500,
+                    detail={  # mutable-ok: one-shot HTTPException detail payload, never mutated after construction
+                        "error": "CrowdStrike AIDR returned a transformed response litellm could not parse; "
+                        "failing closed instead of dropping the delivered redactions",
+                        "guardrail_name": self.guardrail_name,
+                    },
+                ) from validation_error
+            raise
         verbose_proxy_logger.debug(
             "CrowdStrike AIDR Guardrail (%s): Request passed. Response: %s", hook_name, result.detectors
         )
@@ -362,6 +384,34 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
         tail: Final = guard_output.messages[-num_assistant_messages:] if num_assistant_messages > 0 else []
         return [_extract_text_from_message(msg) for msg in tail]
 
+    async def _call_or_fail_open(
+        self, payload: dict[str, Any], hook_name: str, request_data: dict
+    ) -> _GuardChatCompletionsResult:
+        start_time: Final = time.time()
+        try:
+            return await self._call_crowdstrike_aidr_guard(payload, hook_name)
+        except HTTPException:
+            raise
+        except Exception as error:
+            if self.fail_on_error:
+                raise
+            verbose_proxy_logger.error(
+                "CrowdStrike AIDR Guardrail failed open | hook_name: %s error: %s",
+                hook_name,
+                error,
+                exc_info=True,
+            )
+            end_time: Final = time.time()
+            self.add_standard_logging_guardrail_information_to_request_data(
+                guardrail_json_response=error,
+                request_data=request_data,
+                guardrail_status="guardrail_failed_to_respond",
+                start_time=start_time,
+                end_time=end_time,
+                duration=end_time - start_time,
+            )
+            return _GuardChatCompletionsResult()
+
     @override
     def structured_messages_cover_full_request(self) -> bool:
         return effective_skip_system_message_for_guardrail(self) or effective_skip_tool_message_for_guardrail(self)
@@ -439,7 +489,7 @@ class CrowdStrikeAIDRHandler(CustomGuardrail):
                 extra_info["user_name"] = user_email
             ai_guard_payload["extra_info"] = extra_info
 
-        result: Final = await self._call_crowdstrike_aidr_guard(ai_guard_payload, hook_name)
+        result: Final = await self._call_or_fail_open(ai_guard_payload, hook_name, request_data)
 
         if "body" in request_data or "messages" in request_data:
             add_guardrail_to_applied_guardrails_header(request_data=request_data, guardrail_name=self.guardrail_name)

litellm/proxy/guardrails/guardrail_hooks/presidio.py

@@ -11,7 +11,7 @@
 import asyncio
 import json
 import threading
-from collections.abc import AsyncGenerator
+from collections.abc import AsyncGenerator, Sequence
 from contextlib import asynccontextmanager
 from datetime import datetime
 from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypedDict, cast
@@ -22,6 +22,11 @@ from typing_extensions import NotRequired, ReadOnly
 import litellm
 from litellm import get_secret
 from litellm._logging import verbose_proxy_logger
+from litellm.constants import (
+    DEFAULT_PRESIDIO_ANALYZE_CHUNK_SIZE_BYTES,
+    PRESIDIO_ANALYZE_CHUNK_CONCURRENCY,
+    PRESIDIO_ANALYZE_CHUNK_OVERLAP_CHARS,
+)
 from litellm.types.utils import GenericGuardrailAPIInputs
 
 if TYPE_CHECKING:
@@ -63,6 +68,18 @@ class _PresidioAnonymizeResponse(TypedDict):
     items: ReadOnly[NotRequired[list[_PresidioAnonymizeItem]]]
 
 
+_LoopSemaphores = dict[asyncio.AbstractEventLoop, asyncio.Semaphore]
+
+
+def _json_escaped_len(text: str) -> int:
+    """
+    Byte length of ``text`` as it appears serialized inside the JSON request
+    body sent to Presidio (``json.dumps`` escapes non-ASCII characters, so a
+    3-byte UTF-8 character can occupy 6+ bytes on the wire).
+    """
+    return len(json.dumps(text).encode("utf-8")) - 2  # strip the surrounding quotes
+
+
 class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
     user_api_key_cache = None
     ad_hoc_recognizers: list[str] | None = None
@@ -93,6 +110,7 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
         presidio_language: str | None = None,
         presidio_score_thresholds: dict[PiiEntityType | str, float] | None = None,
         presidio_entities_deny_list: list[PiiEntityType | str] | None = None,
+        presidio_analyze_chunk_size_bytes: int | None = None,
         **kwargs,
     ):
         if logging_only is True:
@@ -121,6 +139,7 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
         self.presidio_score_thresholds: dict[PiiEntityType | str, float] = presidio_score_thresholds or {}
         self.presidio_entities_deny_list: list[PiiEntityType | str] = presidio_entities_deny_list or []
         self.presidio_language = presidio_language or "en"
+        self.presidio_analyze_chunk_size_bytes: int = self._coerce_analyze_chunk_size(presidio_analyze_chunk_size_bytes)
         # Shared HTTP session to prevent memory leaks (issue #14540)
         self._http_session: aiohttp.ClientSession | None = None
         # Lock to prevent race conditions when creating session under concurrent load
@@ -134,6 +153,10 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
         # Loop-bound session cache for background threads
         self._loop_sessions: dict[asyncio.AbstractEventLoop, aiohttp.ClientSession] = {}
 
+        # Per-loop semaphores bounding chunked-analyze fan-out across ALL
+        # concurrent oversized blocks/requests on this instance, not per call
+        self._loop_chunk_semaphores: _LoopSemaphores = {}  # mutable-ok: per-loop semaphore cache
+
         if mock_testing is True:  # for testing purposes only
             return
 
@@ -280,7 +303,28 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
     ) -> list[PresidioAnalyzeResponseItem] | _PresidioAnonymizeResponse:
         """
         Send text to the Presidio analyzer endpoint and get analysis results
+
+        Texts larger than ``presidio_analyze_chunk_size_bytes`` (UTF-8) are split
+        into overlapping chunks, analyzed per chunk, and the per-chunk results
+        are remapped onto the original text. Presidio analyzer deployments
+        commonly cap the /analyze request body size (e.g. at 1 MB), and analyzer
+        latency grows with payload size.
         """
+        # Chunk oversized texts before the try block so that a failing chunk
+        # keeps the same sanitized error message a single call would produce.
+        # A single-character text can never be split further, so it always
+        # takes the single-call path regardless of its encoded width.
+        if (
+            text
+            and len(text) > 1
+            and self.mock_redacted_text is None
+            and _json_escaped_len(text) > self.presidio_analyze_chunk_size_bytes
+        ):
+            return await self._analyze_text_chunked(
+                text=text,
+                presidio_config=presidio_config,
+                request_data=request_data,
+            )
         try:
             # Skip empty or whitespace-only text to avoid Presidio errors
             # Common in tool/function calling where assistant content is empty
@@ -397,6 +441,201 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
             # contain API keys or other secrets) in error responses.
             raise Exception(f"Presidio PII analysis failed: {type(e).__name__}") from e
 
+    async def _analyze_text_chunked(
+        self,
+        text: str,
+        presidio_config: PresidioPerRequestConfig | None,
+        request_data: dict,  # mutable-ok: shared per-request state dict, matching analyze_text's parameter
+    ) -> list[PresidioAnalyzeResponseItem]:  # mutable-ok: analyze_text's declared return type requires list
+        """
+        Analyze an oversized text by splitting it into overlapping chunks.
+
+        Each chunk serializes to at most ``presidio_analyze_chunk_size_bytes``
+        bytes inside the JSON request body, so every /analyze call stays below
+        the analyzer deployment's request body limit; per-chunk results are remapped onto the original text and
+        merged. Raises exactly like a single ``analyze_text`` call if any chunk
+        fails.
+
+        Only the analyzer side is chunked: the later anonymize call still
+        receives the full original text, so texts above the anonymizer's own
+        body limit that contain detections keep failing there.
+        """
+        text_chunks: Final = self._split_text_for_analysis(
+            text=text,
+            chunk_size_bytes=self.presidio_analyze_chunk_size_bytes,
+            overlap_chars=PRESIDIO_ANALYZE_CHUNK_OVERLAP_CHARS,
+        )
+        verbose_proxy_logger.debug(
+            "Presidio analyze: text exceeds %s bytes, analyzing in %s overlapping chunks",
+            self.presidio_analyze_chunk_size_bytes,
+            len(text_chunks),
+        )
+        # Bound the fan-out so oversized requests cannot saturate the analyzer.
+        # The semaphore is shared per event loop across every chunked call on
+        # this instance, so many oversized blocks in one request (or many
+        # concurrent requests) still hold at most this many analyzer calls in
+        # flight. On the proxy's main thread the shared-session lock in
+        # _get_session_iterator additionally serializes the HTTP calls; the
+        # bound matters for loop-bound sessions (background threads).
+        analyze_semaphore: Final = self._get_chunk_semaphore()
+
+        async def _analyze_chunk_bounded(
+            chunk_text: str,
+        ) -> Sequence[PresidioAnalyzeResponseItem] | _PresidioAnonymizeResponse:
+            async with analyze_semaphore:
+                return await self.analyze_text(
+                    text=chunk_text,
+                    presidio_config=presidio_config,
+                    request_data=request_data,
+                )
+
+        gathered: Final = await asyncio.gather(
+            *(_analyze_chunk_bounded(chunk_text) for _, chunk_text in text_chunks),
+            return_exceptions=True,
+        )
+        chunk_results: Final = []
+        for result in gathered:
+            if isinstance(result, BaseException):
+                raise result
+            # analyze_text only returns a non-list shape when mock_redacted_text
+            # is set, and the chunked path is never entered in that case.
+            typed_result = cast("list[PresidioAnalyzeResponseItem]", result)  # cast-ok: gather() erases element type
+            # Apply the configured score thresholds and deny list BEFORE the
+            # overlap merge: a below-threshold detection must not win overlap
+            # resolution against one the thresholds would keep. The same filter
+            # runs again downstream in check_pii, where it is a no-op for the
+            # already-filtered items.
+            filtered_result = self.filter_analyze_results_by_score(analyze_results=typed_result)
+            chunk_results.append(
+                cast("list[PresidioAnalyzeResponseItem]", filtered_result)  # cast-ok: list input yields list
+            )
+        return self._merge_chunked_analyze_results(text_chunks=text_chunks, chunk_results=chunk_results)
+
+    def _get_chunk_semaphore(self) -> asyncio.Semaphore:
+        """Per-event-loop semaphore shared by all chunked analyze calls on this instance."""
+        loop: Final = asyncio.get_running_loop()
+        existing: Final = self._loop_chunk_semaphores.get(loop)
+        if existing is not None:
+            return existing
+        created: Final = asyncio.Semaphore(PRESIDIO_ANALYZE_CHUNK_CONCURRENCY)
+        self._loop_chunk_semaphores[loop] = created
+        return created
+
+    @staticmethod
+    def _coerce_analyze_chunk_size(value: int | None) -> int:
+        """
+        Validate a configured chunk size, falling back to the default.
+
+        Non-positive values would either bypass chunking entirely or degenerate
+        it into per-character splits (silently disabling detection), so they are
+        replaced by the default; values below 4 bytes are floored to 4 and the
+        splitter always emits at least one character per chunk, so the chunked
+        path can never re-enter itself.
+        """
+        if not value or value <= 0:
+            return DEFAULT_PRESIDIO_ANALYZE_CHUNK_SIZE_BYTES
+        return max(value, 4)
+
+    @staticmethod
+    def _split_text_for_analysis(
+        text: str,
+        chunk_size_bytes: int,
+        overlap_chars: int,
+    ) -> Sequence[tuple[int, str]]:
+        """
+        Split ``text`` into chunks whose JSON-serialized form is at most
+        ``chunk_size_bytes`` bytes (the analyzer body limit applies to the
+        JSON request body, where non-ASCII characters are escaped and larger
+        than their raw UTF-8 encoding).
+
+        Consecutive chunks overlap by up to ``overlap_chars`` characters so a
+        PII entity up to that length lying across a chunk boundary is still
+        seen whole by one of the chunks (longer boundary-straddling entities
+        may be seen only truncated); ``_merge_chunked_analyze_results`` resolves
+        the duplicate and truncated detections this produces. Returns
+        ``(char_offset, chunk_text)`` pairs where ``char_offset`` is the
+        chunk's start position in the original text.
+        """
+        chunks: Final = []
+        text_len: Final = len(text)
+        start = 0  # rebind-ok: chunk cursor advances across the loop
+        while start < text_len:
+            # Serialized length of a character is at least 1 byte, so a slice
+            # of chunk_size_bytes characters is a sufficient search window.
+            candidate = text[start : start + chunk_size_bytes]
+            if _json_escaped_len(candidate) <= chunk_size_bytes:
+                chunk = candidate
+            else:
+                # Largest prefix whose serialized form fits the budget.
+                low, high = 1, len(candidate)
+                while low < high:
+                    mid = (low + high + 1) // 2
+                    if _json_escaped_len(candidate[:mid]) <= chunk_size_bytes:
+                        low = mid
+                    else:
+                        high = mid - 1
+                # low >= 1 keeps the loop advancing even when a single
+                # character serializes over a (floored, tiny) budget.
+                chunk = candidate[:low]
+            end = start + len(chunk)
+            chunks.append((start, chunk))
+            if end >= text_len:
+                break
+            # Cap the overlap so the next chunk always makes forward progress.
+            effective_overlap = min(overlap_chars, len(chunk) // 2)
+            start = max(start + 1, end - effective_overlap)
+        return chunks
+
+    @staticmethod
+    def _merge_chunked_analyze_results(
+        text_chunks: Sequence[tuple[int, str]],
+        chunk_results: Sequence[Sequence[PresidioAnalyzeResponseItem]],
+    ) -> list[PresidioAnalyzeResponseItem]:  # mutable-ok: analyze_text's declared return type requires list
+        """
+        Remap per-chunk analyzer offsets onto the original text and merge.
+
+        A detection in an overlap region is reported by both neighbouring
+        chunks, and a boundary entity can additionally be reported truncated by
+        the chunk that saw only its head or tail. Same-entity-type detections
+        with overlapping remapped spans are therefore resolved by keeping the
+        longest span (highest score on ties) — mirroring the same-type conflict
+        removal Presidio's AnalyzerEngine applies within a single call, and
+        keeping overlapping spans from corrupting the numbered-token rewriter.
+        Detections of DIFFERENT entity types may still overlap, exactly as in a
+        single-call response. The merged list is sorted by position.
+        """
+        remapped: Final = []
+        for (char_offset, _), results in zip(text_chunks, chunk_results, strict=True):
+            for item in results:
+                item_start = item.get("start")
+                item_end = item.get("end")
+                if item_start is not None:
+                    item["start"] = item_start + char_offset
+                if item_end is not None:
+                    item["end"] = item_end + char_offset
+                remapped.append(item)
+
+        def _priority(item: PresidioAnalyzeResponseItem) -> tuple[int, float]:
+            span_start: Final = item.get("start") or 0
+            span_end: Final = item.get("end") or 0
+            return (-(span_end - span_start), -(item.get("score") or 0.0))
+
+        merged: Final = []
+        kept_spans_by_type: Final = {}
+        for item in sorted(remapped, key=_priority):
+            item_start = item.get("start")
+            item_end = item.get("end")
+            if item_start is None or item_end is None:
+                merged.append(item)
+                continue
+            kept_spans = kept_spans_by_type.setdefault(str(item.get("entity_type")), [])
+            if any(item_start < kept_end and kept_start < item_end for kept_start, kept_end in kept_spans):
+                continue
+            kept_spans.append((item_start, item_end))
+            merged.append(item)
+        merged.sort(key=lambda r: (r.get("start") or 0, r.get("end") or 0))
+        return merged
+
     async def _post_presidio_anonymize(
         self,
         text: str,
@@ -1392,3 +1631,9 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
             self.presidio_score_thresholds = litellm_params.presidio_score_thresholds
         if litellm_params.presidio_entities_deny_list:
             self.presidio_entities_deny_list = litellm_params.presidio_entities_deny_list
+        if litellm_params.presidio_analyze_chunk_size_bytes is not None:
+            # Same validation as __init__: a non-positive value from a guardrail
+            # update must not silently disable detection via degenerate chunking.
+            self.presidio_analyze_chunk_size_bytes = self._coerce_analyze_chunk_size(
+                litellm_params.presidio_analyze_chunk_size_bytes
+            )

litellm/proxy/guardrails/guardrail_initializers.py

@@ -34,6 +34,7 @@ def initialize_bedrock(litellm_params: LitellmParams, guardrail: Guardrail):
         aws_role_name=litellm_params.aws_role_name,
         aws_web_identity_token=litellm_params.aws_web_identity_token,
         aws_sts_endpoint=litellm_params.aws_sts_endpoint,
+        aws_external_id=litellm_params.aws_external_id,
         aws_bedrock_runtime_endpoint=litellm_params.aws_bedrock_runtime_endpoint,
         experimental_use_latest_role_message_only=litellm_params.experimental_use_latest_role_message_only,
         only_scan_new_messages=litellm_params.only_scan_new_messages or False,
@@ -103,7 +104,12 @@ def initialize_presidio(litellm_params: LitellmParams, guardrail: Guardrail):
             apply_to_output=False,
         )
         params.update(overrides)
-        callback: Final = _OPTIONAL_PresidioPIIMasking(**params)
+        # Passed outside the heterogeneous params dict so the argument keeps
+        # its precise int | None type.
+        callback: Final = _OPTIONAL_PresidioPIIMasking(
+            presidio_analyze_chunk_size_bytes=litellm_params.presidio_analyze_chunk_size_bytes,
+            **params,
+        )
         litellm.logging_callback_manager.add_litellm_callback(callback)
         return callback
 

litellm/proxy/health_check.py

@@ -7,8 +7,11 @@ import sys
 import threading
 import time
 from collections.abc import Mapping, Sequence
+from collections.abc import Set as AbstractSet
 from types import MappingProxyType
-from typing import TYPE_CHECKING, Final
+from typing import TYPE_CHECKING, Final, TypeVar
+
+from pydantic import TypeAdapter, ValidationError
 
 import litellm
 
@@ -16,6 +19,7 @@ if TYPE_CHECKING:
     from litellm.router import Router
 
 logger: Final = logging.getLogger(__name__)
+_DeploymentT: Final = TypeVar("_DeploymentT", bound=Mapping[str, object])
 from litellm.constants import (
     BACKGROUND_HEALTH_CHECK_MAX_TOKENS,
     BACKGROUND_HEALTH_CHECK_MAX_TOKENS_REASONING,
@@ -167,6 +171,38 @@ def health_check_filter_kwargs_from_general_settings(
     }
 
 
+def parse_background_health_check_model_groups(
+    general_settings: Mapping[str, object] | None,
+) -> frozenset[str] | None:
+    """
+    Read ``general_settings.background_health_check_model_groups``.
+
+    ``None`` means the allowlist is unset and every deployment participates
+    (legacy behavior). A list scopes background health checks and health-check
+    routing to deployments whose ``model_name`` is listed. A malformed value
+    raises so the proxy fails at startup instead of silently probing everything.
+    """
+    raw: Final = (general_settings or {}).get("background_health_check_model_groups")
+    if raw is None:
+        return None
+    try:
+        return frozenset(TypeAdapter(list[str]).validate_python(raw))
+    except ValidationError as e:
+        raise ValueError(
+            "general_settings.background_health_check_model_groups must be a list of model group names"
+        ) from e
+
+
+def filter_deployments_to_model_groups(
+    model_list: Sequence[_DeploymentT],
+    model_groups: AbstractSet[str] | None,
+) -> tuple[_DeploymentT, ...]:
+    """Deployments whose ``model_name`` is in ``model_groups``; all of them when unset."""
+    if model_groups is None:
+        return tuple(model_list)
+    return tuple(x for x in model_list if x.get("model_name") in model_groups)
+
+
 def filter_deployments_by_id(
     model_list: Sequence[Mapping[str, object]],
 ) -> list:

litellm/proxy/health_endpoints/_health_endpoints.py

@@ -1,5 +1,6 @@
 import asyncio
 import copy
+import json
 import logging
 import os
 import secrets
@@ -11,10 +12,16 @@ from typing import Any, Final, Literal, TypedDict, cast
 
 import fastapi
 from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
+from typing_extensions import ReadOnly
 
 import litellm
 from litellm._logging import verbose_logger, verbose_proxy_logger
 from litellm.constants import HEALTH_CHECK_TIMEOUT_SECONDS
+from litellm.integrations.SlackAlerting.ms_teams import (
+    MS_TEAMS_ALERT_HEADERS,
+    build_ms_teams_payload,
+    get_ms_teams_webhook_url,
+)
 from litellm.litellm_core_utils.custom_logger_registry import CustomLoggerRegistry
 from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
 from litellm.proxy._types import (
@@ -164,6 +171,7 @@ services = (
         "langfuse",
         "langfuse_otel",
         "slack",
+        "ms_teams",
         "openmeter",
         "webhook",
         "email",
@@ -180,6 +188,15 @@ services = (
 )
 
 
+class _ServiceTestErrorDetail(TypedDict):
+    error: ReadOnly[str]
+
+
+class _ServiceTestSuccessResponse(TypedDict):
+    status: ReadOnly[str]
+    message: ReadOnly[str]
+
+
 @router.get(
     "/test",
     tags=["health"],
@@ -238,6 +255,7 @@ async def health_services_endpoint(
             "langfuse",
             "langfuse_otel",
             "slack",
+            "ms_teams",
             "openmeter",
             "webhook",
             "braintrust",
@@ -448,6 +466,38 @@ async def health_services_endpoint(
                     status_code=422,
                     detail={"error": f'"{service}" not in proxy config: general_settings. Unable to test this.'},
                 )
+        if service == "ms_teams":
+            if "ms_teams" not in general_settings.get("alerting", ()):
+                not_configured_detail: Final[_ServiceTestErrorDetail] = {
+                    "error": f'"{service}" not in proxy config: general_settings. Unable to test this.'
+                }
+                raise HTTPException(status_code=422, detail=not_configured_detail)
+            ms_teams_webhook_url: Final = get_ms_teams_webhook_url()
+            if ms_teams_webhook_url is None:
+                missing_webhook_detail: Final[_ServiceTestErrorDetail] = {
+                    "error": "MS_TEAMS_WEBHOOK_URL not set. Unable to test this."
+                }
+                raise HTTPException(status_code=422, detail=missing_webhook_detail)
+            ms_teams_test_message: Final = (
+                f"Alert type: `{AlertType.budget_alerts.value}`\nLevel: `Low`\n"
+                f"Timestamp: `{datetime.now().strftime('%H:%M:%S')}`\n\n"
+                "Message: This is a test MS Teams alert message"
+            )
+            ms_teams_response: Final = await proxy_logging_obj.slack_alerting_instance.async_http_handler.post(
+                url=ms_teams_webhook_url,
+                headers=dict(MS_TEAMS_ALERT_HEADERS),  # mutable-ok: async_http_handler.post only accepts dict headers
+                data=json.dumps(build_ms_teams_payload(ms_teams_test_message)),
+            )
+            if ms_teams_response.status_code >= 400:
+                delivery_failed_detail: Final[_ServiceTestErrorDetail] = {
+                    "error": f"MS Teams webhook returned status {ms_teams_response.status_code}: {ms_teams_response.text}"
+                }
+                raise HTTPException(status_code=500, detail=delivery_failed_detail)
+            ms_teams_success: Final[_ServiceTestSuccessResponse] = {
+                "status": "success",
+                "message": "Mock MS Teams Alert sent, verify MS Teams Alert Received in your channel",
+            }
+            return ms_teams_success
         if service == "email":
             webhook_event: Final = WebhookEvent(
                 event="key_created",