[Infra] Merging RC Branch with Main (#23786)

* fix(test): add missing mocks for test_streamable_http_mcp_handler_mock

The test was missing mocks for extract_mcp_auth_context and set_auth_context,
causing the handler to fail silently in the except block instead of reaching
session_manager.handle_request. This mirrors the fix already applied to the
sibling test_sse_mcp_handler_mock.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ci): route OpenAI models through chat completions in pass-through tests

The test_anthropic_messages_openai_model_streaming_cost_injection test fails
because the OpenAI Responses API returns 400 for requests routed through the
Anthropic Messages endpoint. Setting LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true
routes OpenAI models through the stable chat completions path instead.
Cost injection still works since it happens at the proxy level.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ci): fix assemblyai custom auth and router wildcard test flakiness

1. custom_auth_basic.py: Add user_role='proxy_admin' so the custom auth
   user can access management endpoints like /key/generate. The test
   test_assemblyai_transcribe_with_non_admin_key was hidden behind an
   earlier -x failure and was never reached before.

2. test_router_utils.py: Add flaky(retries=3) and increase sleep from 1s
   to 2s for test_router_get_model_group_usage_wildcard_routes. The async
   callback needs time to write usage to cache, and 1s is insufficient on
   slower CI hardware.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* ci: retrigger CI pipeline

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(mypy): use LitellmUserRoles enum instead of raw string in custom_auth_basic

Fixes mypy error: Argument 'user_role' has incompatible type 'str'; expected 'LitellmUserRoles | None'

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22926)

* fix: don't close HTTP/SDK clients on LLMClientCache eviction

Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:

  RuntimeError: Cannot send a request, as the client has been closed.

This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().

Fixes production crashes after the 1-hour cache TTL expires.

* test: update LLMClientCache unit tests for no-close-on-eviction behavior

Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.

Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.

* test: add e2e tests for OpenAI SDK client surviving cache eviction

Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
  doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
  eviction doesn't close the client

Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.

Also expand the module docstring to explain why the sleep is required.

* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction

* docs(CLAUDE.md): add HTTP client cache safety guideline

* [Fix] Install bsdmainutils for column command in security scans

The security_scans.sh script uses `column` to format vulnerability
output, but the package wasn't installed in the CI environment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle string callback values in prometheus multiproc setup

When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
  TypeError: can only concatenate str (not "list") to str

Normalize each callback setting to a list before concatenating.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* bump: version 1.82.2 → 1.82.3

* fix(test): update test_startup_fails_when_db_setup_fails for opt-in enforcement

The --enforce_prisma_migration_check flag is now required to trigger
sys.exit(1) on DB migration failure, after #23675 flipped the default
behavior to warn-and-continue.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(cost_calculator): use model name for per-request custom pricing when router_model_id has no pricing

When custom pricing is passed as per-request kwargs (input_cost_per_token/output_cost_per_token),
completion() registers pricing under the model name, but _select_model_name_for_cost_calc was
selecting the router deployment hash (which has no pricing data), causing response_cost to be 0.0.

Now checks whether the router_model_id entry actually has pricing before preferring it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

.circleci/config.yml

@@ -3218,6 +3218,7 @@ jobs:
               -e DD_API_KEY=$DD_API_KEY \
               -e DD_SITE=$DD_SITE \
               -e LITELLM_LICENSE=$LITELLM_LICENSE \
+              -e LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true \
               --add-host host.docker.internal:host-gateway \
               --name my-app \
               -v $(pwd)/litellm/proxy/example_config_yaml/pass_through_config.yaml:/app/config.yaml \

CLAUDE.md

@@ -156,4 +156,4 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
 **Fix options:**
 1. **Create a Prisma migration** (permanent) — run `prisma migrate dev --name <description>` in the worktree. The generated file will be picked up by `prisma migrate deploy` on next startup.
 2. **Apply manually for local dev** — `psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF NOT EXISTS ..."` after each proxy start. Fine for dev, not for production.
-3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.
+3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.

ci_cd/security_scans.sh

@@ -11,7 +11,7 @@ echo "Starting security scans for LiteLLM..."
 install_trivy() {
     echo "Installing Trivy and required tools..."
     sudo apt-get update
-    sudo apt-get install -y wget apt-transport-https gnupg lsb-release jq curl
+    sudo apt-get install -y wget apt-transport-https gnupg lsb-release jq curl bsdmainutils
     wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
     echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
     sudo apt-get update

litellm/cost_calculator.py

@@ -660,7 +660,14 @@ def _select_model_name_for_cost_calc(
 
     if custom_pricing is True:
         if router_model_id is not None and router_model_id in litellm.model_cost:
-            return_model = router_model_id
+            entry = litellm.model_cost[router_model_id]
+            if (
+                entry.get("input_cost_per_token") is not None
+                or entry.get("input_cost_per_second") is not None
+            ):
+                return_model = router_model_id
+            else:
+                return_model = model
         else:
             return_model = model
 

litellm/proxy/example_config_yaml/custom_auth_basic.py

@@ -1,6 +1,6 @@
 from fastapi import Request
 
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
 
 
 async def user_api_key_auth(request: Request, api_key: str) -> UserAPIKeyAuth:
@@ -9,6 +9,7 @@ async def user_api_key_auth(request: Request, api_key: str) -> UserAPIKeyAuth:
             api_key="best-api-key-ever",
             user_id="best-user-id-ever",
             team_id="best-team-id-ever",
+            user_role=LitellmUserRoles.PROXY_ADMIN,
         )
     except Exception:
         raise Exception

litellm/proxy/proxy_cli.py

@@ -340,9 +340,16 @@ class ProxyInitializationHelpers:
             return
 
         # Check if prometheus is in any callback list
+        # Each setting can be a list or a single string; normalize to list
         callbacks = litellm_settings.get("callbacks") or []
         success_callbacks = litellm_settings.get("success_callback") or []
         failure_callbacks = litellm_settings.get("failure_callback") or []
+        if isinstance(callbacks, str):
+            callbacks = [callbacks]
+        if isinstance(success_callbacks, str):
+            success_callbacks = [success_callbacks]
+        if isinstance(failure_callbacks, str):
+            failure_callbacks = [failure_callbacks]
         all_callbacks = callbacks + success_callbacks + failure_callbacks
         if "prometheus" not in all_callbacks:
             return

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "litellm"
-version = "1.82.2"
+version = "1.82.3"
 description = "Library to easily interface with LLM API providers"
 authors = ["BerriAI"]
 license = "MIT"
@@ -183,7 +183,7 @@ requires = ["poetry-core", "wheel"]
 build-backend = "poetry.core.masonry.api"
 
 [tool.commitizen]
-version = "1.82.2"
+version = "1.82.3"
 version_files = [
     "pyproject.toml:^version"
 ]

tests/local_testing/test_router_utils.py

@@ -199,6 +199,7 @@ def test_router_get_model_info_wildcard_routes():
 
 
 @pytest.mark.asyncio
+@pytest.mark.flaky(retries=3, delay=1)
 async def test_router_get_model_group_usage_wildcard_routes():
     os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
     litellm.model_cost = litellm.get_model_cost_map(url="")
@@ -219,7 +220,7 @@ async def test_router_get_model_group_usage_wildcard_routes():
     )
     print(resp)
 
-    await asyncio.sleep(1)
+    await asyncio.sleep(2)
 
     tpm, rpm = await router.get_model_group_usage(model_group="gemini/gemini-1.5-flash")
 

tests/mcp_tests/test_mcp_server.py

@@ -395,6 +395,7 @@ async def test_mcp_http_transport_tool_not_found():
 @pytest.mark.asyncio
 async def test_streamable_http_mcp_handler_mock():
     """Test the streamable HTTP MCP handler functionality"""
+    from litellm.proxy._types import UserAPIKeyAuth
 
     # Mock the session manager and its methods
     mock_session_manager = AsyncMock()
@@ -425,6 +426,8 @@ async def test_streamable_http_mcp_handler_mock():
     ), patch(
         "litellm.proxy._experimental.mcp_server.server.extract_mcp_auth_context",
         AsyncMock(return_value=mock_auth_context),
+    ), patch(
+        "litellm.proxy._experimental.mcp_server.server.set_auth_context",
     ):
         from litellm.proxy._experimental.mcp_server.server import (
             handle_streamable_http_mcp,

tests/test_litellm/proxy/test_prometheus_cleanup.py

@@ -67,6 +67,30 @@ class TestMaybeSetupPrometheusMultiprocDir:
             assert os.environ["PROMETHEUS_MULTIPROC_DIR"] == custom_dir
             assert os.path.isdir(custom_dir)
 
+    @pytest.mark.parametrize(
+        "litellm_settings",
+        [
+            {"callbacks": "prometheus"},
+            {"success_callback": "prometheus"},
+            {"failure_callback": "prometheus"},
+            {"callbacks": "custom_callback"},  # string but not prometheus
+        ],
+    )
+    def test_handles_string_callbacks(self, litellm_settings):
+        """When callbacks are specified as a string instead of a list, should not crash."""
+        with patch.dict(os.environ, {}, clear=False):
+            os.environ.pop("PROMETHEUS_MULTIPROC_DIR", None)
+            os.environ.pop("prometheus_multiproc_dir", None)
+
+            # Should not raise TypeError
+            ProxyInitializationHelpers._maybe_setup_prometheus_multiproc_dir(
+                num_workers=4,
+                litellm_settings=litellm_settings,
+            )
+
+            # Cleanup
+            os.environ.pop("PROMETHEUS_MULTIPROC_DIR", None)
+
     @pytest.mark.parametrize(
         "num_workers, litellm_settings",
         [

tests/test_litellm/proxy/test_proxy_cli.py

@@ -677,7 +677,7 @@ class TestHealthAppFactory:
         mock_atexit_register,
         mock_subprocess_run,
     ):
-        """Test that proxy exits with code 1 when PrismaManager.setup_database returns False"""
+        """Test that proxy exits with code 1 when PrismaManager.setup_database returns False and --enforce_prisma_migration_check is set"""
         from litellm.proxy.proxy_cli import run_server
 
         mock_subprocess_run.return_value = MagicMock(returncode=0)
@@ -717,7 +717,7 @@ class TestHealthAppFactory:
 
             with pytest.raises(SystemExit) as exc_info:
                 run_server.main(
-                    ["--local", "--skip_server_startup"], standalone_mode=False
+                    ["--local", "--skip_server_startup", "--enforce_prisma_migration_check"], standalone_mode=False
                 )
             assert exc_info.value.code == 1
             mock_setup_database.assert_called_once_with(use_migrate=True)

tests/test_litellm/test_cost_calculator.py

@@ -388,6 +388,65 @@ def test_custom_pricing_cost_calc_uses_router_model_id_from_litellm_metadata():
     assert custom_model_id not in (selected_model_no_custom or "")
 
 
+def test_per_request_custom_pricing_with_router():
+    """When custom pricing is passed as per-request kwargs (not in model_list),
+    _select_model_name_for_cost_calc should fall back to the model name
+    (where register_model stored the pricing) instead of the router_model_id
+    (which has no pricing data).
+
+    Regression test for the bug where response._hidden_params["response_cost"]
+    returned 0.0 for per-request custom pricing via Router.
+    """
+    from litellm import Router
+    from litellm.cost_calculator import _select_model_name_for_cost_calc
+
+    router = Router(
+        model_list=[
+            {
+                "model_name": "openai/gpt-3.5-turbo",
+                "litellm_params": {
+                    "model": "openai/gpt-3.5-turbo",
+                    "api_key": "test_api_key",
+                },
+            },
+        ]
+    )
+
+    # Get the deployment's model_id (hash) that the router registered
+    deployment = router.model_list[0]
+    router_model_id = deployment["model_info"]["id"]
+
+    # The router registered this hash in model_cost but without custom pricing
+    assert router_model_id in litellm.model_cost
+    entry = litellm.model_cost[router_model_id]
+    # No custom pricing was set in model_list, so these should be None
+    assert entry.get("input_cost_per_token") is None
+
+    # Now simulate what completion() does: register custom pricing under the model name
+    litellm.register_model(
+        {
+            "openai/gpt-3.5-turbo": {
+                "input_cost_per_token": 2.0,
+                "output_cost_per_token": 2.0,
+                "litellm_provider": "openai",
+            }
+        }
+    )
+
+    # _select_model_name_for_cost_calc should pick the model name (which has pricing),
+    # NOT the router_model_id (which has no pricing)
+    selected = _select_model_name_for_cost_calc(
+        model="openai/gpt-3.5-turbo",
+        completion_response=None,
+        custom_pricing=True,
+        custom_llm_provider="openai",
+        router_model_id=router_model_id,
+    )
+    assert selected is not None
+    assert router_model_id not in selected
+    assert "gpt-3.5-turbo" in selected
+
+
 def test_azure_realtime_cost_calculator():
     os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
     litellm.model_cost = litellm.get_model_cost_map(url="")