scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-06 02:48:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

Merge branch 'litellm_internal_staging' into litellm_e2e_haiku_model_refs
Some checks failed
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Details
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Details

Browse source
This commit is contained in:
Mateo Wang 2026-07-16 11:41:02 -07:00 • committed by GitHub
commit 318c5c0e6a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1463 changed files with 65968 additions and 16183 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.github/CODEOWNERS vendored Normal file
View file

@ -0,0 +1,3 @@
/ui/ @yuneng-jiang @ryan-crabbe-berri
/litellm/proxy/_experimental/out/ @yuneng-jiang @ryan-crabbe-berri
/ui/litellm-dashboard/src/lib/http/schema.d.ts

47
.github/actions/setup-uv-with-retries/action.yml vendored Normal file
View file

@ -0,0 +1,47 @@
name: "Set up uv with retries"
description: >-
Install uv via astral-sh/setup-uv, retrying on transient failures. Even with
an exact pinned version, the action resolves the artifact URL by fetching
https://raw.githubusercontent.com/astral-sh/versions/main/v1/uv.ndjson in a
single request with no retry, timeout, or fallback, so one connection-level
network error ("fetch failed") fails the whole job before any test runs.
Retrying the full step covers the manifest fetch and the binary download.
inputs:
version:
description: "uv version to install"
required: true
runs:
using: composite
steps:
- name: Set up uv (attempt 1)
id: attempt-1
continue-on-error: true
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}
- name: Wait before attempt 2
if: steps.attempt-1.outcome == 'failure'
shell: bash
run: sleep 15
- name: Set up uv (attempt 2)
id: attempt-2
if: steps.attempt-1.outcome == 'failure'
continue-on-error: true
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}
- name: Wait before attempt 3
if: steps.attempt-2.outcome == 'failure'
shell: bash
run: sleep 30
- name: Set up uv (attempt 3)
if: steps.attempt-2.outcome == 'failure'
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}

24
.github/pull_request_template.md vendored
View file

@ -41,3 +41,27 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
✅ Test
## Changes
## QA runbook
<!-- Only needed when your PR edits tests/e2e; delete this section otherwise
For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response) and the final item is the sanity-check step shown in the examples. Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
Example checklists:
- tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_blocks_over_limit - a key allowed 2 requests a minute serves exactly 2 and refuses the 3rd
- [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
- [ ] Send three /v1/chat/completions requests with that key inside one minute
- [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
- [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
- tests/e2e/management/test_management_e2e.py::TestModelRoutes::test_model_create_appears_in_ui - a deployment created through the API shows up on the Admin UI models page
- [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
- [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
- [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
-->
### Final Attestation
- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

2
.github/workflows/_test-unit-base.yml vendored
View file

@ -63,7 +63,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/auto_update_price_and_context_window.yml vendored
View file

@ -18,7 +18,7 @@ jobs:
with:
persist-credentials: false
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Update JSON Data

2
.github/workflows/check-ui-api-types.yml vendored
View file

@ -31,7 +31,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/codspeed.yml vendored
View file

@ -37,7 +37,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/mutation-test.yml vendored
View file

@ -39,7 +39,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/oss_daily_guardrails.yml vendored
View file

@ -35,7 +35,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-code-quality.yml vendored
View file

@ -38,7 +38,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

4
.github/workflows/test-linting.yml vendored
View file

@ -33,7 +33,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
@ -172,7 +172,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-litellm-ui-lint.yml vendored
View file

@ -89,4 +89,4 @@ jobs:
- name: Check for dead code (knip)
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: npm run knip
run: npm run knip:ci

2
.github/workflows/test-mcp.yml vendored
View file

@ -32,7 +32,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-semgrep.yml vendored
View file

@ -31,7 +31,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-terraform-provider.yml vendored
View file

@ -74,7 +74,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-unit-documentation.yml vendored
View file

@ -42,7 +42,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

2
.github/workflows/test-unit-proxy-legacy.yml vendored
View file

@ -59,7 +59,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"

7
.gitignore vendored
View file

@ -106,6 +106,13 @@ STABILIZATION_TODO.md
**/coverage
test-config
# Claude Code compatibility-matrix pytest artifact (CI-only output).
compat-results.json
compat-results.json.shards/
compat-rate-limit-summary.json
# Matrix JSON produced by the daily-cron publisher (pushed to litellm-docs).
compatibility-matrix.json
# ---------- Terraform ----------
# Provider binaries + module cache — regenerated by `terraform init`.
**/.terraform/

2
CLAUDE.md
View file

@ -39,6 +39,8 @@ Don't hesitate to use values in .env to get needed API keys and other secrets, a
Python max line length is 120, not 88
On a fresh worktree or clone, run `make bootstrap` before anything else. It provisions everything tests, `make pre-commit`, and a local proxy need
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. It only runs when there are staged frontend and/or backend changes and calculates violations, generates types, etc. based on the worktree, so stage what you need or stash/delete unwanted files in litellm/ or ui/ (where backend and frontend lint run, respectively) before running it. If it fails because dashboard api types are stale, it already regenerated them for you. You just need to stage the schema.d.ts, re-run `make pre-commit` to confirm it passes, and commit
When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It measures the working tree, so it must contain exactly the fixes you're committing

15
Makefile
View file

@ -9,11 +9,12 @@
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
install-dev install-proxy-dev install-test-deps install-hooks \
install-helm-unittest check-circular-imports check-import-safety pre-commit \
lint-install lint-fetch-base
lint-install lint-fetch-base bootstrap
# Default target
help:
@echo "Available commands:"
@echo " make bootstrap - Provision a fresh clone/worktree"
@echo " make install-dev - Install development dependencies"
@echo " make install-proxy-dev - Install proxy development dependencies"
@echo " make install-dev-ci - Install dev dependencies (CI-compatible, pins OpenAI)"
@ -71,6 +72,18 @@ info:
install-dev:
$(UV) sync --inexact --frozen
bootstrap:
$(UV) sync --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev
$(UV_RUN) python scripts/prisma_generate_if_needed.py
cd ui/litellm-dashboard && npm ci --no-audit --no-fund
@main_root=$$(git worktree list --porcelain | head -1 | sed 's/^worktree //'); \
if [ "$$main_root" != "$$(git rev-parse --show-toplevel)" ] && [ -f "$$main_root/.env" ] && [ ! -f .env ]; then \
cp "$$main_root/.env" .env && echo "bootstrap: copied .env from $$main_root"; \
else \
echo "bootstrap: .env left untouched"; \
fi
@echo "bootstrap: done"
install-proxy-dev:
$(UV) sync --frozen --group proxy-dev --extra proxy

13
README.md
View file

@ -552,17 +552,12 @@ The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws
2. Run dependent services `docker-compose up db prometheus`
#### Backend
1. (In root) create virtual environment `python -m venv .venv`
2. Activate virtual environment `source .venv/bin/activate`
3. Install dependencies `uv sync --all-extras --group proxy-dev`
4. `uv run prisma generate`
5. `prisma generate`
6. Start proxy backend `python litellm/proxy/proxy_cli.py`
1. Run `make bootstrap`
2. Start proxy backend: `uv run python litellm/proxy/proxy_cli.py`
#### Frontend
1. Navigate to `ui/litellm-dashboard`
2. Install dependencies `npm install`
3. Run `npm run dev` to start the dashboard
1. Navigate to `ui/litellm-dashboard` (dependencies were already installed w/ `make bootstrap`)
2. Start dashboard: `npm run dev`
### Verify Docker Image Signatures

2
docker/build_from_pip/Dockerfile.build_from_pip
View file

@ -36,7 +36,7 @@ RUN uv venv --python python && \
"opentelemetry-api==1.28.0" \
"opentelemetry-sdk==1.28.0" \
"opentelemetry-exporter-otlp==1.28.0" \
"ddtrace==2.19.0" \
"ddtrace==4.11.0" \
"sentry-sdk==2.21.0" \
"mangum==0.17.0" \
"azure-ai-contentsafety==1.0.0" \

141
enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py
View file

@ -7,7 +7,8 @@
# Thank you users! We ❤️ you! - Krrish & Ishaan
## This provides an LLM Guard Integration for content moderation on the proxy
from typing import Literal, Optional
import asyncio
from typing import Optional
import aiohttp
from fastapi import HTTPException
@ -18,7 +19,6 @@ from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import CallTypesLiteral
from litellm.utils import get_formatted_prompt
class _ENTERPRISE_LLMGuard(CustomLogger):
@ -46,45 +46,44 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
except Exception:
pass
async def moderation_check(self, text: str):
async def moderation_check(self, text: str) -> str:
"""
Runs the LLM Guard moderation check on ``text``.
Raises an HTTPException when the content violates the safety policy;
otherwise returns the sanitized prompt from LLM Guard, falling back to
the original text when the API does not provide one.
[TODO] make this more performant for high-throughput scenario
"""
try:
async with aiohttp.ClientSession() as session:
if self.mock_redacted_text is not None:
redacted_text = self.mock_redacted_text
else:
# Make the first request to /analyze
analyze_url = f"{self.llm_guard_api_base}analyze/prompt"
verbose_proxy_logger.debug("Making request to: %s", analyze_url)
analyze_payload = {"prompt": text}
redacted_text = None
if self.mock_redacted_text is not None:
redacted_text = self.mock_redacted_text
else:
analyze_url = f"{self.llm_guard_api_base}analyze/prompt"
verbose_proxy_logger.debug("Making request to: %s", analyze_url)
async with aiohttp.ClientSession() as session:
async with session.post(
analyze_url, json=analyze_payload
analyze_url, json={"prompt": text}
) as response:
redacted_text = await response.json()
verbose_proxy_logger.debug(
f"LLM Guard: Received response - {redacted_text}"
verbose_proxy_logger.debug(
f"LLM Guard: Received response - {redacted_text}"
)
if redacted_text is None:
raise HTTPException(
status_code=500,
detail={
"error": f"Invalid content moderation response: {redacted_text}"
},
)
if redacted_text is not None:
if (
redacted_text.get("is_valid", None) is not None
and redacted_text["is_valid"] is False
):
raise HTTPException(
status_code=400,
detail={"error": "Violated content safety policy"},
)
else:
pass
else:
raise HTTPException(
status_code=500,
detail={
"error": f"Invalid content moderation response: {redacted_text}"
},
)
if redacted_text.get("is_valid", None) is False:
raise HTTPException(
status_code=400,
detail={"error": "Violated content safety policy"},
)
sanitized_prompt = redacted_text.get("sanitized_prompt")
return sanitized_prompt if isinstance(sanitized_prompt, str) else text
except Exception as e:
verbose_proxy_logger.exception(
"litellm.enterprise.enterprise_hooks.llm_guard::moderation_check - Exception occurred - {}".format(
@ -138,23 +137,75 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
return
self.print_verbose("Makes LLM Guard Check")
try:
assert call_type in [
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
]
except Exception:
if call_type not in [
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
]:
self.print_verbose(
f"Call Type - {call_type}, not in accepted list - ['completion','embeddings','image_generation','moderation','audio_transcription']"
)
return data
formatted_prompt = get_formatted_prompt(data=data, call_type=call_type) # type: ignore
self.print_verbose(f"LLM Guard, formatted_prompt: {formatted_prompt}")
return await self.moderation_check(text=formatted_prompt)
return await self._moderate_request(data=data)
async def _moderate_request(self, data: dict) -> dict:
"""
Sanitizes the request in place using the prompt returned by LLM Guard so
the provider-bound request carries the redacted content, then returns it.
"""
messages = data.get("messages")
if messages is not None:
data["messages"] = list(
await asyncio.gather(
*(self._moderate_message(message) for message in messages)
)
)
return data
input_ = data.get("input")
if input_ is not None:
data["input"] = await self._moderate_input(input_)
return data
prompt = data.get("prompt")
if isinstance(prompt, str):
data["prompt"] = await self.moderation_check(text=prompt)
return data
async def _moderate_message(self, message: dict) -> dict:
content = message.get("content")
if isinstance(content, str):
return {**message, "content": await self.moderation_check(text=content)}
if isinstance(content, list):
return {
**message,
"content": list(
await asyncio.gather(
*(self._moderate_content_part(part) for part in content)
)
),
}
return message
async def _moderate_content_part(self, part: dict) -> dict:
if part.get("type") == "text" and isinstance(part.get("text"), str):
return {**part, "text": await self.moderation_check(text=part["text"])}
return part
async def _moderate_input(self, input_: object) -> object:
if isinstance(input_, str):
return await self.moderation_check(text=input_)
if isinstance(input_, list):
return [
await self.moderation_check(text=item)
if isinstance(item, str)
else item
for item in input_
]
return input_
async def async_post_call_streaming_hook(
self, user_api_key_dict: UserAPIKeyAuth, response: str

120
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -29,7 +29,7 @@ class CheckBatchCost:
proxy_logging_obj: "ProxyLogging",
prisma_client: "PrismaClient",
llm_router: "Router",
track_unmanaged_vertex_batch_cost: bool = False,
track_unmanaged_batch_cost: bool = False,
):
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -37,7 +37,7 @@ class CheckBatchCost:
self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
self._track_unmanaged_vertex_batch_cost = track_unmanaged_vertex_batch_cost
self._track_unmanaged_batch_cost = track_unmanaged_batch_cost
# Cached after the first poll cycle. Once we know the column is absent we skip
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
@ -118,11 +118,11 @@ class CheckBatchCost:
Resolve (model_id, batch_id) for a managed-object row, where model_id is a router
deployment id and batch_id is the raw provider batch id.
Managed batches encode both in a base64 unified id. Unmanaged Vertex batches, created with
a raw gs:// input_file_id, store the raw provider job id as unified_object_id; when
track_unmanaged_vertex_batch_cost is enabled the model is derived from the gs:// path and
mapped to a configured vertex_ai deployment. Returns None (recording a metric) when the row
can't be routed.
Managed batches encode both in a base64 unified id. Unmanaged batches (created outside
LiteLLM's own /v1/batches with a raw input_file_id) store the raw provider job id as
unified_object_id instead; when track_unmanaged_batch_cost is enabled the model is derived
from the provider-specific input_file_id layout (Vertex gs:// or Bedrock s3://) and mapped
to a matching deployment. Returns None (recording a metric) when the row can't be routed.
"""
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
@ -142,8 +142,43 @@ class CheckBatchCost:
return None
return model_id, get_batch_id_from_unified_batch_id(decoded)
if self._track_unmanaged_vertex_batch_cost:
return self._resolve_unmanaged_vertex_routing(job, prom_logger)
if self._track_unmanaged_batch_cost:
from litellm.llms.bedrock.batches.transformation import (
BedrockBatchesConfig,
)
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
return self._resolve_unmanaged_provider_routing(
job=job,
prom_logger=prom_logger,
llm_provider="vertex_ai",
bare_model_name=VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
),
)
if BedrockBatchesConfig.is_unmanaged_s3_batch_input_file_id(input_file_id):
assert input_file_id is not None # narrowed by is_unmanaged_s3_batch_input_file_id
return self._resolve_unmanaged_provider_routing(
job=job,
prom_logger=prom_logger,
llm_provider="bedrock",
bare_model_name=BedrockBatchesConfig.get_bare_model_name_from_s3_file(
input_file_id
),
)
verbose_proxy_logger.info(
f"Skipping job {unified_object_id}: not a recognized unmanaged batch "
"(no gs:// or s3:// input_file_id with an embedded model)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
@ -151,36 +186,17 @@ class CheckBatchCost:
self._record_error(prom_logger, "invalid_unified_id")
return None
def _resolve_unmanaged_vertex_routing(
def _resolve_unmanaged_provider_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
llm_provider: str,
bare_model_name: str,
) -> Optional[Tuple[str, str]]:
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if not VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
verbose_proxy_logger.info(
f"Skipping job {job.unified_object_id}: not an unmanaged vertex batch "
"(no gs:// input_file_id with a publishers/ model path)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
bare_model_name = VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
)
deployment_id = self._get_vertex_ai_deployment_id_for_bare_model(
bare_model_name
)
deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
if deployment_id is None:
verbose_proxy_logger.info(
f"Skipping unmanaged vertex batch {job.unified_object_id}: no vertex_ai "
f"Skipping unmanaged {llm_provider} batch {job.unified_object_id}: no {llm_provider} "
f"deployment configured for model {bare_model_name}"
)
self._record_error(prom_logger, "unmanaged_no_matching_deployment")
@ -188,22 +204,22 @@ class CheckBatchCost:
return deployment_id, job.unified_object_id
def _get_vertex_ai_deployment_id_for_bare_model(
self, bare_model_name: str
def _get_deployment_id_for_bare_model(
self, bare_model_name: str, llm_provider: str
) -> Optional[str]:
model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
deployment_id = (
self._get_vertex_ai_deployment_id(model_group) if model_group else None
self._get_deployment_id_for_provider(model_group, llm_provider) if model_group else None
)
if deployment_id is not None:
return deployment_id
return self._get_vertex_ai_deployment_id_from_matching_deployments(
bare_model_name
return self._get_deployment_id_from_matching_deployments(
bare_model_name, llm_provider
)
def _get_vertex_ai_deployment_id_from_matching_deployments(
self, bare_model_name: str
def _get_deployment_id_from_matching_deployments(
self, bare_model_name: str, llm_provider: str
) -> Optional[str]:
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
@ -215,13 +231,13 @@ class CheckBatchCost:
if not self._is_bare_model_match(actual_model, bare_model_name):
continue
try:
_, llm_provider, _, _ = get_llm_provider(
_, deployment_llm_provider, _, _ = get_llm_provider(
model=actual_model,
custom_llm_provider=litellm_params.get("custom_llm_provider"),
)
except Exception:
continue
if llm_provider != "vertex_ai":
if deployment_llm_provider != llm_provider:
continue
model_info = deployment.get("model_info") or {}
deployment_id = model_info.get("id")
@ -231,15 +247,21 @@ class CheckBatchCost:
@staticmethod
def _is_bare_model_match(actual_model: str, bare_model_name: str) -> bool:
# Bedrock model ids may have ":" replaced with "-" in the S3 object key (see
# BedrockBatchesConfig.get_bare_model_name_from_s3_file), so normalize both sides;
# a no-op for providers like vertex_ai whose model ids never contain a colon.
normalized_actual = actual_model.replace(":", "-")
normalized_bare = bare_model_name.replace(":", "-")
return (
actual_model == bare_model_name
or actual_model.endswith(f"/{bare_model_name}")
or actual_model.endswith(f":{bare_model_name}")
normalized_actual == normalized_bare
or normalized_actual.endswith(f"/{normalized_bare}")
)
def _get_vertex_ai_deployment_id(self, model_group: str) -> Optional[str]:
def _get_deployment_id_for_provider(
self, model_group: str, llm_provider: str
) -> Optional[str]:
"""
Returns the first deployment id for `model_group` whose provider is vertex_ai,
Returns the first deployment id for `model_group` whose provider is `llm_provider`,
skipping deployments from other providers that happen to share the model group name.
"""
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
@ -249,13 +271,13 @@ class CheckBatchCost:
if deployment_info is None:
continue
try:
_, llm_provider, _, _ = get_llm_provider(
_, deployment_llm_provider, _, _ = get_llm_provider(
model=deployment_info.litellm_params.model,
custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
)
except Exception:
continue
if llm_provider == "vertex_ai":
if deployment_llm_provider == llm_provider:
return deployment_id
return None

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.49"
version = "0.1.50"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.49"
version = "0.1.50"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

21
helm/litellm-helm/README.md
View file

@ -54,6 +54,12 @@ If `db.useStackgresOperator` is used (not yet implemented):
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
| `billingMetrics.enabled` | Enable enterprise billable-request metering. Requires an enterprise license. | `false` |
| `billingMetrics.endpoint` | Collector that the billable-request counter is pushed to. | `https://telemetry.litellm.ai` |
| `billingMetrics.secretName` | Name of an existing Secret holding the mTLS client certificate, under the keys `tls.crt` and `tls.key`. | `litellm-billing-metrics-mtls` |
| `billingMetrics.caSecretName` | Name of an existing Secret holding a CA bundle under the key `ca.crt`. Only needed for a private or test collector whose server certificate is not on the public web PKI. | `""` |
| `billingMetrics.exportIntervalMs` | How often the counter is pushed, in milliseconds. The proxy defaults to `60000` when unset. | `""` |
#### Example `proxy_config` ConfigMap from values (default):
```
@ -94,6 +100,21 @@ data:
type: Opaque
```
#### Enterprise billable-request metering
Enterprise licenses meter billable requests by pushing a counter to LiteLLM's collector over mutual TLS. The chart does not create the client certificate; it mounts one you already hold, read-only, so the private key is never exposed through the environment. Create the Secret under the name the chart expects, then turn the block on:
```
kubectl create secret tls litellm-billing-metrics-mtls --cert=client.crt --key=client.key
```
```
billingMetrics:
enabled: true
```
Set `billingMetrics.caSecretName` only when the collector is a private or test one whose server certificate is not on the public web PKI; the production collector needs no CA override. The chart fails the render rather than deploying a proxy that silently never exports, so a missing `secretName` or an emptied `endpoint` surfaces at `helm install` time.
### Database Settings
| Name | Description | Value |

47
helm/litellm-helm/templates/_helpers.tpl
View file

@ -50,6 +50,53 @@ app.kubernetes.io/name: {{ include "litellm.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Enterprise billable-request metering. The client certificate identifies the
deployment to LiteLLM's collector, so it is mounted read-only from an existing
Secret rather than passed through the environment.
*/}}
{{- define "litellm.billingMetrics.certDir" -}}/etc/litellm/billing-mtls{{- end -}}
{{- define "litellm.billingMetrics.caDir" -}}/etc/litellm/billing-mtls-ca{{- end -}}
{{- define "litellm.billingMetricsEnv" -}}
- name: LITELLM_BILLING_METRICS_ENDPOINT
value: {{ required "billingMetrics.endpoint is required when billingMetrics.enabled is true" .Values.billingMetrics.endpoint | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: {{ printf "%s/tls.crt" (include "litellm.billingMetrics.certDir" .) | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: {{ printf "%s/tls.key" (include "litellm.billingMetrics.certDir" .) | quote }}
{{- if .Values.billingMetrics.caSecretName }}
- name: LITELLM_BILLING_METRICS_CA_CERT
value: {{ printf "%s/ca.crt" (include "litellm.billingMetrics.caDir" .) | quote }}
{{- end }}
{{- with .Values.billingMetrics.exportIntervalMs }}
- name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: {{ . | quote }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumes" -}}
- name: billing-metrics-mtls
secret:
secretName: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)" .Values.billingMetrics.secretName }}
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
secret:
secretName: {{ .Values.billingMetrics.caSecretName }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumeMounts" -}}
- name: billing-metrics-mtls
mountPath: {{ include "litellm.billingMetrics.certDir" . }}
readOnly: true
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
mountPath: {{ include "litellm.billingMetrics.caDir" . }}
readOnly: true
{{- end }}
{{- end -}}
{{/*
Create the name of the service account to use
*/}}

9
helm/litellm-helm/templates/deployment.yaml
View file

@ -142,6 +142,9 @@ spec:
{{- with .Values.extraEnvVars }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsEnv" . | nindent 12 }}
{{- end }}
{{- if .Values.migrationJob.enabled }}
# Schema updates are owned by the dedicated migrations Job; skip
# the proxy's startup `prisma db push` so N replicas don't race
@ -220,6 +223,9 @@ spec:
- name: npm
mountPath: /.npm
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- with .Values.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
@ -252,6 +258,9 @@ spec:
items:
- key: {{ .Values.proxyConfigMap.key | default "config.yaml" }}
path: "config.yaml"
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}

297
helm/litellm-helm/tests/billing_metrics_tests.yaml Normal file
View file

@ -0,0 +1,297 @@
suite: test billingMetrics wiring on the proxy deployment
templates:
- deployment.yaml
- configmap-litellm.yaml
- migrations-job.yaml
tests:
- it: is off by default, adding no env, volume, or mount
template: deployment.yaml
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- it: renders the endpoint and the mounted cert paths when enabled
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: /etc/litellm/billing-mtls/tls.crt
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: /etc/litellm/billing-mtls/tls.key
# The conventional Secret name is the default, so enabling the block is enough.
- it: mounts the default cert secret read-only alongside the config volume
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- it: honours a secretName override
template: deployment.yaml
set:
billingMetrics:
enabled: true
secretName: my-billing-mtls
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: my-billing-mtls
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- it: honours an endpoint override
template: deployment.yaml
set:
billingMetrics:
enabled: true
endpoint: https://collector.internal:4318
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://collector.internal:4318
# The production collector presents a public web-PKI certificate, so the CA
# override must stay absent unless a private collector is configured.
- it: omits the CA env, volume, and mount when no caSecretName is set
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- it: mounts the CA secret when caSecretName is set
template: deployment.yaml
set:
billingMetrics:
enabled: true
caSecretName: billing-ca
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
- it: passes the export interval through only when set
template: deployment.yaml
set:
billingMetrics:
enabled: true
exportIntervalMs: 5000
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "5000"
- it: omits the export interval when unset
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "60000"
# Kubernetes resolves duplicate env names last-wins, so the chart-owned billing
# entries must render after .Values.envVars or a user could silently redirect
# the metering export. The three billing entries are the last ones emitted here
# (migrationJob, which appends DISABLE_SCHEMA_UPDATE, is off for this case).
- it: renders the billing endpoint after envVars so it cannot be shadowed
template: deployment.yaml
set:
migrationJob:
enabled: false
billingMetrics:
enabled: true
envVars:
LITELLM_BILLING_METRICS_ENDPOINT: https://shadowed.example
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://shadowed.example
- equal:
path: spec.template.spec.containers[0].env[-3]
value:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- equal:
path: spec.template.spec.containers[0].env[-2].name
value: LITELLM_BILLING_METRICS_CLIENT_CERT
- equal:
path: spec.template.spec.containers[0].env[-1].name
value: LITELLM_BILLING_METRICS_CLIENT_KEY
- it: keeps user-supplied volumes and mounts alongside the billing secret
template: deployment.yaml
set:
billingMetrics:
enabled: true
volumes:
- name: custom-callbacks
configMap:
name: my-callbacks
volumeMounts:
- name: custom-callbacks
mountPath: /app/callbacks
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: custom-callbacks
configMap:
name: my-callbacks
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: custom-callbacks
mountPath: /app/callbacks
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- it: still mounts the proxy config when enabled
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: litellm-config
mountPath: /etc/litellm/config.yaml
subPath: config.yaml
# Only the proxy serves billable traffic. The migrations Job must never mount
# the client certificate, and it renders its own env and volumes, so nothing
# stops a future edit from wiring the billing include into it by mistake.
- it: does not touch the migrations job when enabled
template: migrations-job.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- notExists:
path: spec.template.spec.containers[0].volumeMounts
- notExists:
path: spec.template.spec.volumes
- it: fails loudly when enabled with an emptied secretName
template: deployment.yaml
set:
billingMetrics:
enabled: true
secretName: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)
- it: fails loudly when enabled without an endpoint
template: deployment.yaml
set:
billingMetrics:
enabled: true
endpoint: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.endpoint is required when billingMetrics.enabled is true

14
helm/litellm-helm/values.yaml
View file

@ -139,6 +139,20 @@ masterkeySecretName: ""
# if set, use this secret key for the master key; otherwise, use the default key
masterkeySecretKey: ""
# Optional: enterprise billable-request metering. When enabled, the proxy counts
# successful requests to inference, MCP, and A2A endpoints and pushes them to
# LiteLLM's collector over mutual TLS. Requires an enterprise license.
# The client certificate identifies the deployment, so it is mounted read-only
# from an existing Secret and never passed through the environment.
billingMetrics:
enabled: false
endpoint: https://telemetry.litellm.ai # collector to push the counter to
secretName: litellm-billing-metrics-mtls # existing Secret holding tls.crt and tls.key
# Only for private or test collectors whose server certificate is not on the
# public web PKI. The production collector needs no CA override.
caSecretName: "" # existing Secret holding ca.crt
exportIntervalMs: "" # push cadence; the proxy defaults to 60000
proxyConfigMap:
# when true, creates a new configmap
create: true

5
helm/litellm/templates/NOTES.txt
View file

@ -46,4 +46,9 @@ Reminders:
- gateway.config.proxy_config (rendered into a ConfigMap and mounted at
/app/config/config.yaml; gateway reads it via
CONFIG_FILE_PATH)
- {component}.pdb.{enabled,minAvailable,maxUnavailable} (per-component PodDisruptionBudget; disabled by
default — with hpa.minReplicas of 1, minAvailable: 1
would block node drains)
- {component}.topologySpreadConstraints (standard k8s list, e.g. spread replicas across
topology.kubernetes.io/zone)
- Enable ingress.enabled=true to dispatch / → ui, gateway data-plane prefixes → gateway, and the catch-all → backend.

97
helm/litellm/templates/_helpers.tpl
View file

@ -34,6 +34,57 @@ app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
{{- end -}}
{{/*
Enterprise billable-request metering. Wired into gateway and backend, not the
migrations job. The gateway serves nearly all billable traffic, but the backend
keeps the named-server MCP transport (/{mcp_server_name}/mcp), which writes a
SpendLogs row, so metering only the gateway would silently drop that traffic.
The client certificate identifies the deployment to LiteLLM's collector, so it is
mounted read-only from an existing Secret rather than passed through the
environment.
*/}}
{{- define "litellm.billingMetrics.certDir" -}}/etc/litellm/billing-mtls{{- end -}}
{{- define "litellm.billingMetrics.caDir" -}}/etc/litellm/billing-mtls-ca{{- end -}}
{{- define "litellm.billingMetricsEnv" -}}
- name: LITELLM_BILLING_METRICS_ENDPOINT
value: {{ required "billingMetrics.endpoint is required when billingMetrics.enabled is true" .Values.billingMetrics.endpoint | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: {{ printf "%s/tls.crt" (include "litellm.billingMetrics.certDir" .) | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: {{ printf "%s/tls.key" (include "litellm.billingMetrics.certDir" .) | quote }}
{{- if .Values.billingMetrics.caSecretName }}
- name: LITELLM_BILLING_METRICS_CA_CERT
value: {{ printf "%s/ca.crt" (include "litellm.billingMetrics.caDir" .) | quote }}
{{- end }}
{{- with .Values.billingMetrics.exportIntervalMs }}
- name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: {{ . | quote }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumes" -}}
- name: billing-metrics-mtls
secret:
secretName: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)" .Values.billingMetrics.secretName }}
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
secret:
secretName: {{ .Values.billingMetrics.caSecretName }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumeMounts" -}}
- name: billing-metrics-mtls
mountPath: {{ include "litellm.billingMetrics.certDir" . }}
readOnly: true
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
mountPath: {{ include "litellm.billingMetrics.caDir" . }}
readOnly: true
{{- end }}
{{- end -}}
{{/*
Per-component selector labels — used in both Service selectors and Deployment matchLabels.
*/}}
@ -244,6 +295,52 @@ harmless no-op for the Job and authoritative for the app pods.
{{- end }}
{{- end -}}
{{/*
PodDisruptionBudget shared by gateway, backend, and ui.
Invoke with a dict:
(dict "root" $ "component" .Values.gateway "componentName" "gateway"
"fullname" (include "litellm.gateway.fullname" .)
"selectorLabels" (include "litellm.gateway.selectorLabels" .))
Renders nothing unless both the component and its `pdb.enabled` are on.
Only one of minAvailable / maxUnavailable should be set; if both are,
minAvailable wins. If neither is set, falls back to `maxUnavailable: 1` so
an enabled-but-unconfigured PDB still permits node drains.
"Set" means non-nil and non-empty-string, so an explicit 0 (e.g.
`maxUnavailable: 0` to forbid all voluntary disruptions) is honored rather
than silently replaced by the fallback.
*/}}
{{- define "litellm.pdb" -}}
{{- $root := .root -}}
{{- $component := .component -}}
{{- $min := $component.pdb.minAvailable -}}
{{- $max := $component.pdb.maxUnavailable -}}
{{- $minSet := not (or (kindIs "invalid" $min) (eq (printf "%v" $min) "")) -}}
{{- $maxSet := not (or (kindIs "invalid" $max) (eq (printf "%v" $max) "")) -}}
{{- if and $component.enabled $component.pdb $component.pdb.enabled }}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ .fullname }}
labels:
{{- include "litellm.commonLabels" $root | nindent 4 }}
app.kubernetes.io/component: {{ .componentName }}
spec:
selector:
matchLabels:
{{- .selectorLabels | nindent 6 }}
{{- if $minSet }}
minAvailable: {{ $min }}
{{- else if $maxSet }}
maxUnavailable: {{ $max }}
{{- else }}
maxUnavailable: 1
{{- end }}
{{- end }}
{{- end -}}
{{/*
Renders `envFrom:` block for a component's `envConfigMaps` / `envSecrets`
lists. Each entry is a resource name; the chart wires the whole ConfigMap /

17
helm/litellm/templates/backend/deployment.yaml
View file

@ -44,14 +44,20 @@ spec:
- name: CONFIG_FILE_PATH
value: /app/config/config.yaml
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsEnv" . | nindent 12 }}
{{- end }}
{{- include "litellm.envFrom" .Values.backend | nindent 10 }}
{{- if or .Values.gateway.config.create .Values.backend.volumeMounts }}
{{- if or .Values.gateway.config.create .Values.backend.volumeMounts .Values.billingMetrics.enabled }}
volumeMounts:
{{- if .Values.gateway.config.create }}
- name: gateway-config
mountPath: /app/config/config.yaml
subPath: config.yaml
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- with .Values.backend.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
@ -66,13 +72,16 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.backend.resources | nindent 12 }}
{{- if or .Values.gateway.config.create .Values.backend.volumes }}
{{- if or .Values.gateway.config.create .Values.backend.volumes .Values.billingMetrics.enabled }}
volumes:
{{- if .Values.gateway.config.create }}
- name: gateway-config
configMap:
name: {{ include "litellm.gateway.fullname" . }}-config
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- with .Values.backend.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
@ -89,4 +98,8 @@ spec:
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}

6
helm/litellm/templates/backend/poddisruptionbudget.yaml Normal file
View file

@ -0,0 +1,6 @@
{{- include "litellm.pdb" (dict
"root" $
"component" .Values.backend
"componentName" "backend"
"fullname" (include "litellm.backend.fullname" .)
"selectorLabels" (include "litellm.backend.selectorLabels" .)) }}

17
helm/litellm/templates/gateway/deployment.yaml
View file

@ -46,14 +46,20 @@ spec:
- name: NUM_WORKERS
value: {{ .Values.gateway.numWorkers | quote }}
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsEnv" . | nindent 12 }}
{{- end }}
{{- include "litellm.envFrom" .Values.gateway | nindent 10 }}
{{- if or .Values.gateway.config.create .Values.gateway.volumeMounts }}
{{- if or .Values.gateway.config.create .Values.gateway.volumeMounts .Values.billingMetrics.enabled }}
volumeMounts:
{{- if .Values.gateway.config.create }}
- name: gateway-config
mountPath: /app/config/config.yaml
subPath: config.yaml
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- with .Values.gateway.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
@ -68,13 +74,16 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.gateway.resources | nindent 12 }}
{{- if or .Values.gateway.config.create .Values.gateway.volumes }}
{{- if or .Values.gateway.config.create .Values.gateway.volumes .Values.billingMetrics.enabled }}
volumes:
{{- if .Values.gateway.config.create }}
- name: gateway-config
configMap:
name: {{ include "litellm.gateway.fullname" . }}-config
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- with .Values.gateway.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
@ -91,4 +100,8 @@ spec:
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.gateway.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}

6
helm/litellm/templates/gateway/poddisruptionbudget.yaml Normal file
View file

@ -0,0 +1,6 @@
{{- include "litellm.pdb" (dict
"root" $
"component" .Values.gateway
"componentName" "gateway"
"fullname" (include "litellm.gateway.fullname" .)
"selectorLabels" (include "litellm.gateway.selectorLabels" .)) }}

4
helm/litellm/templates/ui/deployment.yaml
View file

@ -76,4 +76,8 @@ spec:
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.ui.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}

6
helm/litellm/templates/ui/poddisruptionbudget.yaml Normal file
View file

@ -0,0 +1,6 @@
{{- include "litellm.pdb" (dict
"root" $
"component" .Values.ui
"componentName" "ui"
"fullname" (include "litellm.ui.fullname" .)
"selectorLabels" (include "litellm.ui.selectorLabels" .)) }}

249
helm/litellm/tests/billing_metrics_tests.yaml Normal file
View file

@ -0,0 +1,249 @@
suite: test billingMetrics wiring on gateway and backend
templates:
- gateway/deployment.yaml
- gateway/configmap.yaml
- backend/deployment.yaml
- migrations-job.yaml
values:
- ./values/required.yaml
tests:
- it: is off by default, adding no env, volume, or mount
template: gateway/deployment.yaml
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- equal:
path: spec.template.spec.containers[0].volumeMounts
value:
- name: gateway-config
mountPath: /app/config/config.yaml
subPath: config.yaml
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- it: renders the endpoint and the mounted cert paths when enabled
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: /etc/litellm/billing-mtls/tls.crt
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: /etc/litellm/billing-mtls/tls.key
- it: mounts the cert secret read-only alongside the config volume
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
# The production collector presents a public web-PKI certificate, so the CA
# override must stay absent unless a private collector is configured.
- it: omits the CA env, volume, and mount when no caSecretName is set
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- it: mounts the CA secret when caSecretName is set
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
caSecretName: billing-ca
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
- it: passes the export interval through only when set
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
exportIntervalMs: 5000
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "5000"
- it: keeps user-supplied gateway volumes alongside the billing secret
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
gateway.volumes:
- name: custom-callbacks
configMap:
name: my-callbacks
gateway.volumeMounts:
- name: custom-callbacks
mountPath: /app/callbacks
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: custom-callbacks
configMap:
name: my-callbacks
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
# The backend keeps the named-server MCP transport (/{mcp_server_name}/mcp),
# which writes a SpendLogs row, so it must meter too or that traffic is lost.
- it: meters the backend as well, since it serves the MCP transport
template: backend/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- it: leaves the backend alone when metering is off
template: backend/deployment.yaml
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
# The migrations job runs prisma and serves no traffic; it must never receive
# the client key.
- it: never mounts the billing cert on the migrations job
template: migrations-job.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- isNull:
path: spec.template.spec.volumes
# The conventional Secret name is the default, so enabling metering needs no
# secretName at all; the guard below only fires on an explicitly blanked one.
- it: uses the conventional secret name by default
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- it: fails loudly when the secretName is explicitly blanked
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)
- it: fails loudly when enabled without an endpoint
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
endpoint: ""
secretName: billing-mtls
asserts:
- failedTemplate:
errorMessage: billingMetrics.endpoint is required when billingMetrics.enabled is true

188
helm/litellm/tests/pdb_topology_spread_tests.yaml Normal file
View file

@ -0,0 +1,188 @@
suite: test pod disruption budgets and topology spread constraints
templates:
- gateway/poddisruptionbudget.yaml
- backend/poddisruptionbudget.yaml
- ui/poddisruptionbudget.yaml
- gateway/deployment.yaml
- gateway/configmap.yaml
- backend/deployment.yaml
- ui/deployment.yaml
values:
- ./values/required.yaml
tests:
- it: renders no PDB by default
templates:
- gateway/poddisruptionbudget.yaml
- backend/poddisruptionbudget.yaml
- ui/poddisruptionbudget.yaml
asserts:
- hasDocuments:
count: 0
- it: gateway PDB uses minAvailable and matches the gateway selector labels
template: gateway/poddisruptionbudget.yaml
set:
gateway.pdb.enabled: true
gateway.pdb.minAvailable: 1
asserts:
- isKind:
of: PodDisruptionBudget
- equal:
path: apiVersion
value: policy/v1
- equal:
path: metadata.name
value: RELEASE-NAME-litellm-gateway
- equal:
path: spec.minAvailable
value: 1
- notExists:
path: spec.maxUnavailable
- equal:
path: spec.selector.matchLabels
value:
app.kubernetes.io/name: litellm
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/component: gateway
- it: backend PDB uses maxUnavailable when minAvailable is unset
template: backend/poddisruptionbudget.yaml
set:
backend.pdb.enabled: true
backend.pdb.maxUnavailable: 25%
asserts:
- equal:
path: spec.maxUnavailable
value: 25%
- notExists:
path: spec.minAvailable
- equal:
path: spec.selector.matchLabels
value:
app.kubernetes.io/name: litellm
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/component: backend
- it: minAvailable wins when both minAvailable and maxUnavailable are set
template: gateway/poddisruptionbudget.yaml
set:
gateway.pdb.enabled: true
gateway.pdb.minAvailable: 2
gateway.pdb.maxUnavailable: 1
asserts:
- equal:
path: spec.minAvailable
value: 2
- notExists:
path: spec.maxUnavailable
- it: an explicit maxUnavailable 0 is honored instead of the fallback
template: backend/poddisruptionbudget.yaml
set:
backend.pdb.enabled: true
backend.pdb.maxUnavailable: 0
asserts:
- equal:
path: spec.maxUnavailable
value: 0
- notExists:
path: spec.minAvailable
- it: an explicit minAvailable 0 is honored and beats a set maxUnavailable
template: gateway/poddisruptionbudget.yaml
set:
gateway.pdb.enabled: true
gateway.pdb.minAvailable: 0
gateway.pdb.maxUnavailable: 1
asserts:
- equal:
path: spec.minAvailable
value: 0
- notExists:
path: spec.maxUnavailable
- it: enabled PDB with neither knob set falls back to maxUnavailable 1
template: ui/poddisruptionbudget.yaml
set:
ui.pdb.enabled: true
asserts:
- equal:
path: spec.maxUnavailable
value: 1
- notExists:
path: spec.minAvailable
- equal:
path: spec.selector.matchLabels
value:
app.kubernetes.io/name: litellm
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/component: ui
- it: renders no PDB for a disabled component even when its pdb is enabled
template: gateway/poddisruptionbudget.yaml
set:
gateway.enabled: false
gateway.pdb.enabled: true
asserts:
- hasDocuments:
count: 0
- it: deployments omit topologySpreadConstraints by default
templates:
- gateway/deployment.yaml
- backend/deployment.yaml
- ui/deployment.yaml
asserts:
- notExists:
path: spec.template.spec.topologySpreadConstraints
- it: gateway deployment renders configured topologySpreadConstraints
template: gateway/deployment.yaml
set:
gateway.topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/component: gateway
asserts:
- equal:
path: spec.template.spec.topologySpreadConstraints
value:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/component: gateway
- it: backend deployment renders configured topologySpreadConstraints
template: backend/deployment.yaml
set:
backend.topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/component: backend
asserts:
- equal:
path: spec.template.spec.topologySpreadConstraints[0].topologyKey
value: kubernetes.io/hostname
- equal:
path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable
value: DoNotSchedule
- it: ui deployment renders configured topologySpreadConstraints
template: ui/deployment.yaml
set:
ui.topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
asserts:
- equal:
path: spec.template.spec.topologySpreadConstraints[0].topologyKey
value: topology.kubernetes.io/zone

51
helm/litellm/values.yaml
View file

@ -73,6 +73,25 @@ masterKey:
secretName: litellm-master-key-secret # name of a Secret containing the master key
secretKey: master-key
# Optional: enterprise billable-request metering. When enabled, the gateway and
# backend count successful requests to inference, MCP, and A2A endpoints and push
# them to LiteLLM's collector over mutual TLS. Both components serve billable
# routes: the backend keeps the named-server MCP transport. Requires an
# enterprise license. The client certificate identifies the deployment, so it is
# mounted read-only from an existing Secret and never passed through the env.
billingMetrics:
enabled: false
endpoint: https://telemetry.litellm.ai # collector to push the counter to
# An existing Secret holding the client certificate under tls.crt and its key
# under tls.key, usually created from the onboarding artifact. The default is
# the conventional name, so the common path is to create that Secret and set
# enabled: true. Override only if yours is named differently.
secretName: litellm-billing-metrics-mtls
# Only for private or test collectors whose server certificate is not on the
# public web PKI. The production collector needs no CA override.
caSecretName: "" # existing Secret holding ca.crt
exportIntervalMs: "" # push cadence; the proxy defaults to 60000
# External Postgres connection.
database:
writer:
@ -171,10 +190,28 @@ gateway:
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
# PodDisruptionBudget for the gateway pods. Set exactly one of
# `minAvailable` / `maxUnavailable` (minAvailable wins if both are set;
# enabling without either falls back to `maxUnavailable: 1`). Disabled by
# default: with the default hpa.minReplicas of 1, a `minAvailable: 1` PDB
# would block node drains entirely.
pdb:
enabled: false
minAvailable: ""
maxUnavailable: ""
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
# Standard k8s topologySpreadConstraints for the gateway pods, e.g. to
# spread replicas across zones:
# - maxSkew: 1
# topologyKey: topology.kubernetes.io/zone
# whenUnsatisfiable: ScheduleAnyway
# labelSelector:
# matchLabels:
# app.kubernetes.io/component: gateway
topologySpreadConstraints: []
# ---------- backend (UI / management API) ----------
backend:
@ -214,10 +251,17 @@ backend:
minReplicas: 1
maxReplicas: 4
targetCPUUtilizationPercentage: 70
# Same shape as gateway.pdb.
pdb:
enabled: false
minAvailable: ""
maxUnavailable: ""
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
# Same shape as gateway.topologySpreadConstraints.
topologySpreadConstraints: []
# ---------- ui (Next.js static dashboard) ----------
ui:
@ -260,7 +304,14 @@ ui:
minReplicas: 1
maxReplicas: 3
targetCPUUtilizationPercentage: 80
# Same shape as gateway.pdb.
pdb:
enabled: false
minAvailable: ""
maxUnavailable: ""
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
# Same shape as gateway.topologySpreadConstraints.
topologySpreadConstraints: []

6
litellm-proxy-extras/litellm_proxy_extras/migrations/20260713230852_add_key_type_to_litellm_verification_token/migration.sql Normal file
View file

@ -0,0 +1,6 @@
-- AlterTable
ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN "key_type" TEXT;
-- AlterTable
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN "key_type" TEXT;

2
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -422,6 +422,7 @@ model LiteLLM_VerificationToken {
budget_reset_at DateTime?
allowed_cache_controls String[] @default([])
allowed_routes String[] @default([])
key_type String?
policies String[] @default([])
access_group_ids String[] @default([])
model_spend Json @default("{}")
@ -516,6 +517,7 @@ model LiteLLM_DeletedVerificationToken {
budget_reset_at DateTime?
allowed_cache_controls String[] @default([])
allowed_routes String[] @default([])
key_type String?
policies String[] @default([])
access_group_ids String[] @default([])
model_spend Json @default("{}")

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.75"
version = "0.4.77"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.75"
version = "0.4.77"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

21
litellm/batches/batch_utils.py
View file

@ -118,6 +118,7 @@ def _batch_cost_calculator(
total_cost = _get_batch_job_cost_from_file_content(
file_content_dictionary=file_content_dictionary,
custom_llm_provider=custom_llm_provider,
model_name=model_name,
model_info=model_info,
)
verbose_logger.debug("total_cost=%s", total_cost)
@ -363,6 +364,7 @@ def _count_entry_tokens(
def _get_batch_job_cost_from_file_content(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic"] = "openai",
model_name: Optional[str] = None,
model_info: Optional[ModelInfo] = None,
) -> float:
"""
@ -377,9 +379,15 @@ def _get_batch_job_cost_from_file_content(
for _item in file_content_dictionary:
if _batch_response_was_successful(_item, custom_llm_provider):
_response_body = _get_response_from_batch_job_output_file(_item, custom_llm_provider)
if model_info is not None or custom_llm_provider == "anthropic":
if model_info is not None or custom_llm_provider in ("anthropic", "bedrock"):
usage = _get_batch_job_usage_from_response_body(_response_body, custom_llm_provider)
model = _response_body.get("model", "")
# Bedrock batch output lines report a short internal model id
# (e.g. "claude-sonnet-4-6") that is not in the cost map; use the
# deployment model name for pricing when available.
if custom_llm_provider == "bedrock" and model_name:
model = model_name
else:
model = _response_body.get("model") or model_name or ""
prompt_cost, completion_cost = batch_cost_calculator(
usage=usage,
model=model,
@ -485,7 +493,7 @@ def _get_batch_job_usage_from_response_body(response_body: dict, custom_llm_prov
"""
Get the tokens of a batch job from the response body
"""
if custom_llm_provider == "anthropic":
if custom_llm_provider in ("anthropic", "bedrock"):
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
return AnthropicConfig().calculate_usage(
@ -513,6 +521,8 @@ def _get_response_from_batch_job_output_file(batch_job_output_file: dict, custom
"""
if custom_llm_provider == "anthropic":
return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("message", None) or {}
if custom_llm_provider == "bedrock":
return batch_job_output_file.get("modelOutput", None) or {}
_response: dict = batch_job_output_file.get("response", None) or {}
_response_body = _response.get("body", None) or {}
return _response_body
@ -523,9 +533,12 @@ def _batch_response_was_successful(batch_job_output_file: dict, custom_llm_provi
Check if the batch job response was successful
OpenAI-shaped output rows report ``response.status_code == 200``; Anthropic
message batch results lines report ``result.type == "succeeded"``.
message batch results lines report ``result.type == "succeeded"``; Bedrock
batch output lines report ``modelOutput`` (and no ``error``).
"""
if custom_llm_provider == "anthropic":
return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("type") == "succeeded"
if custom_llm_provider == "bedrock":
return batch_job_output_file.get("modelOutput") is not None and batch_job_output_file.get("error") is None
_response: dict = batch_job_output_file.get("response", None) or {}
return _response.get("status_code", None) == 200

22
litellm/caching/caching_handler.py
View file

@ -85,6 +85,22 @@ class CachingHandlerResponse(BaseModel):
in_memory_cache_obj = InMemoryCache()
def _drop_logging_obj_from_kwargs(request_kwargs: dict[str, object]) -> dict[str, object]:
"""
The caching handler is stored on the Logging object
(``logging_obj._llm_caching_handler``), so keeping ``litellm_logging_obj``
inside ``request_kwargs`` closes a reference cycle
(Logging -> LLMCachingHandler -> kwargs -> Logging) that keeps the full
request payload (messages included) alive until a generational GC pass
instead of being freed by refcount when the request ends. Nothing in the
caching layer reads the logging object from these kwargs; cache-key
generation ignores litellm-internal params.
"""
if "litellm_logging_obj" not in request_kwargs:
return request_kwargs
return {k: v for k, v in request_kwargs.items() if k != "litellm_logging_obj"}
def _is_chat_completion_cached_dict(cached_result: dict) -> bool:
cached_id = cached_result.get("id")
if isinstance(cached_id, str) and cached_id.startswith("chatcmpl"):
@ -118,7 +134,7 @@ class LLMCachingHandler:
self.async_streaming_chunks: List[ModelResponse] = []
self.sync_streaming_chunks: List[ModelResponse] = []
self.request_kwargs = request_kwargs
self.request_kwargs = _drop_logging_obj_from_kwargs(request_kwargs)
self.preset_cache_key: Optional[str] = None
self.original_function = original_function
self.start_time = start_time
@ -297,7 +313,7 @@ class LLMCachingHandler:
new_kwargs.pop("metadata", None)
if new_kwargs.get("stream") is True and "cache_key" not in new_kwargs:
new_kwargs["cache_key"] = litellm.cache.get_cache_key(**new_kwargs)
self.request_kwargs = new_kwargs
self.request_kwargs = _drop_logging_obj_from_kwargs(new_kwargs)
print_verbose("Checking Sync Cache")
cached_result = litellm.cache.get_cache(**new_kwargs)
if cached_result is not None:
@ -693,7 +709,7 @@ class LLMCachingHandler:
new_kwargs.pop("metadata", None)
if new_kwargs.get("stream") is True and "cache_key" not in new_kwargs:
new_kwargs["cache_key"] = litellm.cache.get_cache_key(**new_kwargs)
self.request_kwargs = new_kwargs
self.request_kwargs = _drop_logging_obj_from_kwargs(new_kwargs)
cached_result: Optional[Any] = None
if call_type == CallTypes.aembedding.value:
if isinstance(new_kwargs["input"], str):

1
litellm/constants.py
View file

@ -1496,6 +1496,7 @@ MAX_TEAM_LIST_LIMIT = int(os.getenv("MAX_TEAM_LIST_LIMIT", 20))
MAX_POLICY_ESTIMATE_IMPACT_ROWS = int(os.getenv("MAX_POLICY_ESTIMATE_IMPACT_ROWS", 1000))
DEFAULT_PROMPT_INJECTION_SIMILARITY_THRESHOLD = float(os.getenv("DEFAULT_PROMPT_INJECTION_SIMILARITY_THRESHOLD", 0.7))
LENGTH_OF_LITELLM_GENERATED_KEY = int(os.getenv("LENGTH_OF_LITELLM_GENERATED_KEY", 16))
MINIMUM_CUSTOM_KEY_LENGTH = int(os.getenv("MINIMUM_CUSTOM_KEY_LENGTH", 16))
SECRET_MANAGER_REFRESH_INTERVAL = int(os.getenv("SECRET_MANAGER_REFRESH_INTERVAL", 86400))
LITELLM_SETTINGS_SAFE_DB_OVERRIDES = [
"default_internal_user_params",

23
litellm/integrations/custom_guardrail.py
View file

@ -515,6 +515,22 @@ class CustomGuardrail(CustomLogger):
return True
return False
def uses_apply_guardrail_interface(self) -> bool:
return type(self).apply_guardrail is not CustomGuardrail.apply_guardrail
def _deployment_pre_call_target(self) -> "CustomLogger":
if not self.uses_apply_guardrail_interface():
return self
try:
from litellm.proxy.utils import unified_guardrail
except ImportError as e:
raise ImportError(
f"Guardrail {self.guardrail_name or type(self).__name__} implements apply_guardrail, which needs "
"the litellm proxy dependencies to run at the deployment level. "
"Install them with: pip install 'litellm[proxy]'"
) from e
return unified_guardrail
async def async_pre_call_deployment_hook(
self, kwargs: Dict[str, Any], call_type: Optional[CallTypes]
) -> Optional[dict]:
@ -533,7 +549,10 @@ class CustomGuardrail(CustomLogger):
# CHECK IF GUARDRAIL REJECTS THE REQUEST
if call_type == CallTypes.completion or call_type == CallTypes.acompletion:
result = await self.async_pre_call_hook(
target = self._deployment_pre_call_target()
if target is not self:
kwargs["guardrail_to_apply"] = self
result = await target.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(
user_id=kwargs.get("user_api_key_user_id"),
team_id=kwargs.get("user_api_key_team_id"),
@ -543,7 +562,7 @@ class CustomGuardrail(CustomLogger):
),
cache=dc,
data=kwargs,
call_type=call_type.value or "acompletion", # type: ignore
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
if result is not None and isinstance(result, dict):

117
litellm/integrations/prometheus.py
View file

@ -239,6 +239,18 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric("litellm_output_audio_tokens_metric"),
)
self.litellm_video_duration_seconds_metric = self._counter_factory(
"litellm_video_duration_seconds_metric",
"Seconds of video generated, from usage.duration_seconds on video generation calls",
labelnames=self.get_labels_for_metric("litellm_video_duration_seconds_metric"),
)
self.litellm_images_generated_metric = self._counter_factory(
"litellm_images_generated_metric",
"Number of images generated, from the image generation response",
labelnames=self.get_labels_for_metric("litellm_images_generated_metric"),
)
# Remaining Budget for Team
self.litellm_remaining_team_budget_metric = self._gauge_factory(
"litellm_remaining_team_budget_metric",
@ -1336,6 +1348,12 @@ class PrometheusLogger(CustomLogger):
label_context=label_context,
)
self._increment_media_generation_metrics(
standard_logging_payload=standard_logging_payload,
enum_values=enum_values,
label_context=label_context,
)
# MCP tool call metrics
self._increment_mcp_tool_call_metrics(
standard_logging_payload=standard_logging_payload,
@ -1459,8 +1477,65 @@ class PrometheusLogger(CustomLogger):
),
]
for counter, metric_name, value in detail_metrics:
if not isinstance(value, (int, float)) or value <= 0:
PrometheusLogger._inc_sparse_usage_counters(
self,
detail_metrics,
enum_values=enum_values,
label_context=label_context,
)
def _increment_media_generation_metrics(
self,
standard_logging_payload: StandardLoggingPayload,
enum_values: UserAPIKeyLabelValues,
label_context: PrometheusLabelFactoryContext | None = None,
) -> None:
"""
Increment video-seconds and images-generated counters from
``standard_logging_payload["metadata"]["usage_object"]``. Video
providers report ``duration_seconds`` there; image generation calls
report ``output_image_count``. Both are sparse: only emitted when the
value is present and > 0, so token-only call types are unaffected.
"""
metadata = standard_logging_payload.get("metadata") or {}
usage_object = metadata.get("usage_object") if isinstance(metadata, dict) else None
if not isinstance(usage_object, dict):
return
media_metrics: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]] = [
(
self.litellm_video_duration_seconds_metric,
"litellm_video_duration_seconds_metric",
usage_object.get("duration_seconds"),
),
(
self.litellm_images_generated_metric,
"litellm_images_generated_metric",
usage_object.get("output_image_count"),
),
]
PrometheusLogger._inc_sparse_usage_counters(
self,
media_metrics,
enum_values=enum_values,
label_context=label_context,
)
def _inc_sparse_usage_counters(
self,
counters_with_values: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]],
enum_values: UserAPIKeyLabelValues,
label_context: PrometheusLabelFactoryContext | None = None,
) -> None:
"""
Increment each ``(counter, metric_name, value)`` entry whose value is
a positive number. Non-numeric values (including booleans from
malformed provider usage dicts) and values <= 0 are skipped, keeping
scrape output sparse.
"""
for counter, metric_name, value in counters_with_values:
if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
continue
PrometheusLogger._inc_labeled_counter(
self,
@ -1716,6 +1791,35 @@ class PrometheusLogger(CustomLogger):
amount=float(response_cost),
)
@staticmethod
def _get_remaining_from_v3_rate_limit_headers(
standard_logging_payload: StandardLoggingPayload | None,
rate_limit_type: Literal["requests", "tokens"],
) -> int | None:
"""
Read the per-(key, model) remaining value emitted by the v3 rate
limiter (``parallel_request_limiter_v3.py``), which writes
``x-ratelimit-model_per_key-remaining-{requests,tokens}`` into
``standard_logging_object.hidden_params.additional_headers`` instead
of the ``litellm-key-remaining-*`` metadata keys the legacy limiter
sets. The header carries no model group; it always refers to this
request's model group, which is what the gauges are labeled with.
Values are written in-process as plain ints (never HTTP-serialized
strings), so anything else is rejected rather than coerced.
"""
if standard_logging_payload is None:
return None
hidden_params = standard_logging_payload.get("hidden_params")
if hidden_params is None:
return None
additional_headers = hidden_params.get("additional_headers")
if additional_headers is None:
return None
value = dict(additional_headers).get(f"x-ratelimit-model_per_key-remaining-{rate_limit_type}")
if isinstance(value, bool) or not isinstance(value, int):
return None
return value
def _set_virtual_key_rate_limit_metrics(
self,
user_api_key: Optional[str],
@ -1733,11 +1837,20 @@ class PrometheusLogger(CustomLogger):
model_group = get_model_group_from_litellm_kwargs(kwargs)
remaining_requests_variable_name = f"litellm-key-remaining-requests-{model_group}"
remaining_tokens_variable_name = f"litellm-key-remaining-tokens-{model_group}"
standard_logging_payload: StandardLoggingPayload | None = kwargs.get("standard_logging_object")
remaining_requests = metadata.get(remaining_requests_variable_name)
if remaining_requests is None:
remaining_requests = self._get_remaining_from_v3_rate_limit_headers(
standard_logging_payload=standard_logging_payload, rate_limit_type="requests"
)
if remaining_requests is None:
remaining_requests = sys.maxsize
remaining_tokens = metadata.get(remaining_tokens_variable_name)
if remaining_tokens is None:
remaining_tokens = self._get_remaining_from_v3_rate_limit_headers(
standard_logging_payload=standard_logging_payload, rate_limit_type="tokens"
)
if remaining_tokens is None:
remaining_tokens = sys.maxsize

7
litellm/integrations/s3.py
View file

@ -161,8 +161,13 @@ def get_s3_object_key(
start_time: datetime,
s3_file_name: str,
) -> str:
sanitized_s3_file_name = s3_file_name.replace("/", "_")
s3_object_key = (
(s3_path.rstrip("/") + "/" if s3_path else "") + prefix + start_time.strftime("%Y-%m-%d") + "/" + s3_file_name
(s3_path.rstrip("/") + "/" if s3_path else "")
+ prefix
+ start_time.strftime("%Y-%m-%d")
+ "/"
+ sanitized_s3_file_name
) # we need the s3 key to include the time, so we log cache hits too
s3_object_key += ".json"
return s3_object_key

226
litellm/integrations/websearch_interception/handler.py
View file

@ -19,9 +19,11 @@ from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.websearch_interception.tools import (
get_litellm_web_search_tool,
get_litellm_web_search_tool_openai,
get_litellm_web_search_tool_responses,
is_anthropic_native_web_search_tool,
is_web_search_tool,
is_web_search_tool_chat_completion,
is_web_search_tool_responses,
)
from litellm.integrations.websearch_interception.transformation import (
WebSearchTransformation,
@ -32,11 +34,12 @@ from litellm.types.integrations.websearch_interception import (
)
from litellm.types.integrations.custom_logger import (
CHAT_COMPLETION_AGENTIC_SURFACE,
RESPONSES_AGENTIC_SURFACE,
AgenticLoopPlan,
AgenticLoopRequestPatch,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import LlmProviders
from litellm.types.utils import CallTypes, LlmProviders
from litellm.utils import ProviderConfigManager
# Key used to flag, on per-request kwargs, that the originating client sent
@ -251,6 +254,9 @@ class WebSearchInterceptionLogger(CustomLogger):
if not tools:
return None
if call_type in (CallTypes.responses, CallTypes.aresponses):
return self._convert_responses_tools(kwargs=kwargs, tools=tools)
# Check if any tool is a web search tool (native or already LiteLLM standard)
has_websearch = any(is_web_search_tool(t) for t in tools)
@ -291,6 +297,26 @@ class WebSearchInterceptionLogger(CustomLogger):
return kwargs
def _convert_responses_tools(self, kwargs: dict[str, Any], tools: list[dict[str, Any]]) -> dict | None:
"""Convert Responses API web search tools to the LiteLLM standard function tool."""
if not any(is_web_search_tool_responses(tool) for tool in tools):
return None
verbose_logger.debug("WebSearchInterception: Converting Responses web_search tools to LiteLLM standard")
converted_tools = [
get_litellm_web_search_tool_responses() if is_web_search_tool_responses(tool) else tool for tool in tools
]
converted_kwargs = {**kwargs, "tools": converted_tools}
if kwargs.get("stream"):
verbose_logger.debug("WebSearchInterception: deployment hook converting stream=True to stream=False")
converted_kwargs["stream"] = False
converted_kwargs["_websearch_interception_converted_stream"] = True
return converted_kwargs
@classmethod
def from_config_yaml(cls, config: WebSearchInterceptionConfig) -> "WebSearchInterceptionLogger":
"""
@ -461,6 +487,17 @@ class WebSearchInterceptionLogger(CustomLogger):
kwargs=kwargs,
)
if kwargs.get("_agentic_loop_api_surface") == RESPONSES_AGENTIC_SURFACE:
return await self.async_should_run_responses_agentic_loop(
response=response,
model=model,
messages=messages,
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
)
verbose_logger.debug(f"WebSearchInterception: Hook called! provider={custom_llm_provider}, stream={stream}")
verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
@ -597,6 +634,54 @@ class WebSearchInterceptionLogger(CustomLogger):
}
return True, tools_dict
async def async_should_run_responses_agentic_loop(
self,
response: Any,
model: str,
messages: list[dict],
tools: list[dict] | None,
stream: bool,
custom_llm_provider: str,
kwargs: dict,
) -> tuple[bool, dict]:
"""Check if WebSearch interception is needed for the Responses API."""
verbose_logger.debug(
f"WebSearchInterception: Responses hook called! provider={custom_llm_provider}, stream={stream}"
)
if self.enabled_providers is not None and custom_llm_provider not in self.enabled_providers:
verbose_logger.debug(
f"WebSearchInterception: Skipping provider {custom_llm_provider} (not in enabled list: {self.enabled_providers})"
)
return False, {}
has_websearch_tool = any(is_web_search_tool_responses(t) for t in (tools or []))
if not has_websearch_tool:
verbose_logger.debug("WebSearchInterception: No litellm_web_search tool in responses request")
return False, {}
should_intercept, tool_calls = WebSearchTransformation.transform_request(
response=response,
stream=stream,
response_format="responses",
)
if not should_intercept:
verbose_logger.debug("WebSearchInterception: No WebSearch function_call detected in responses output")
return False, {}
verbose_logger.debug(
f"WebSearchInterception: Detected {len(tool_calls)} WebSearch function_call(s), executing agentic loop"
)
tools_dict = {
"tool_calls": tool_calls,
"tool_type": "websearch",
"provider": custom_llm_provider,
"response_format": "responses",
}
return True, tools_dict
async def async_run_agentic_loop(
self,
tools: Dict,
@ -655,6 +740,18 @@ class WebSearchInterceptionLogger(CustomLogger):
kwargs=kwargs,
)
if kwargs.get("_agentic_loop_api_surface") == RESPONSES_AGENTIC_SURFACE:
return await self.async_build_responses_agentic_loop_plan(
tools=tools,
model=model,
messages=messages,
response=response,
optional_params=anthropic_messages_optional_request_params,
logging_obj=logging_obj,
stream=stream,
kwargs=kwargs,
)
tool_calls = tools["tool_calls"]
thinking_blocks = tools.get("thinking_blocks", [])
request_patch, structured_results = await self._build_anthropic_request_patch(
@ -809,6 +906,133 @@ class WebSearchInterceptionLogger(CustomLogger):
metadata={"tool_type": "websearch", "response_format": response_format},
)
async def async_build_responses_agentic_loop_plan(
self,
tools: dict,
model: str,
messages: list[dict],
response: Any,
optional_params: dict,
logging_obj: Any,
stream: bool,
kwargs: dict,
) -> AgenticLoopPlan:
tool_calls = tools["tool_calls"]
request_patch = await self._build_responses_request_patch(
model=model,
messages=messages,
tool_calls=tool_calls,
optional_params=optional_params,
kwargs=kwargs,
)
return AgenticLoopPlan(
run_agentic_loop=True,
request_patch=request_patch,
metadata={"tool_type": "websearch", "response_format": "responses"},
)
async def _build_responses_request_patch(
self,
model: str,
messages: Union[str, list[dict]],
tool_calls: list[dict],
optional_params: dict,
kwargs: dict,
) -> AgenticLoopRequestPatch:
"""Execute litellm.asearch() and build a Responses API rerun patch."""
search_tasks = [
(
self._execute_search(tool_call["input"]["query"], kwargs=kwargs)
if isinstance(tool_call.get("input"), dict) and tool_call["input"].get("query")
else self._create_empty_search_result()
)
for tool_call in tool_calls
]
verbose_logger.debug(f"WebSearchInterception: Executing {len(search_tasks)} responses search(es) in parallel")
search_results = await asyncio.gather(*search_tasks, return_exceptions=True)
search_texts = [self._extract_search_text(result) for result in search_results]
followup_items = [
item
for tool_call, search_text in zip(tool_calls, search_texts)
for item in (
{
"type": "function_call",
"call_id": tool_call.get("call_id"),
"name": LITELLM_WEB_SEARCH_TOOL_NAME,
"arguments": tool_call.get("arguments", ""),
},
{
"type": "function_call_output",
"call_id": tool_call.get("call_id"),
"output": search_text,
},
)
]
input_list = self._normalize_responses_input(messages) + followup_items
tools_param = optional_params.get("tools")
optional_params_clean = {
k: v
for k, v in optional_params.items()
if k not in {"tools", "tool_choice", "stream", "model_alias_map", "stream_response", "custom_prompt_dict"}
}
kwargs_for_followup = {
k: v
for k, v in kwargs.items()
if not k.startswith("_websearch_interception")
and k
not in {
"_agentic_loop_api_surface",
"litellm_logging_obj",
"acompletion",
"custom_llm_provider",
"model_alias_map",
}
}
full_model_name = model
if "/" not in model and isinstance(kwargs.get("custom_llm_provider"), str):
full_model_name = f"{kwargs['custom_llm_provider']}/{model}"
verbose_logger.debug(
"WebSearchInterception: Built responses request patch model=%s input_items=%d searches=%d",
full_model_name,
len(input_list),
len(search_texts),
)
return AgenticLoopRequestPatch(
model=full_model_name,
messages=input_list,
tools=tools_param if isinstance(tools_param, list) else None,
optional_params=optional_params_clean,
kwargs=kwargs_for_followup,
)
@staticmethod
def _normalize_responses_input(messages: Union[str, list[dict]]) -> list[dict]:
if isinstance(messages, str):
return [{"role": "user", "content": messages}]
if isinstance(messages, list):
return list(messages)
return []
@staticmethod
def _extract_search_text(result: Any) -> str:
if isinstance(result, Exception):
verbose_logger.error(f"WebSearchInterception: Responses search failed with error: {str(result)}")
return f"Search failed: {str(result)}"
if isinstance(result, tuple) and len(result) == 2:
text_value, _ = result
return text_value if isinstance(text_value, str) else str(text_value)
verbose_logger.debug(f"WebSearchInterception: Unexpected search result type {type(result)}")
return str(result)
@staticmethod
def _resolve_max_tokens(
optional_params: Dict,

69
litellm/integrations/websearch_interception/tools.py
View file

@ -82,6 +82,75 @@ def get_litellm_web_search_tool_openai() -> Dict[str, Any]:
}
def get_litellm_web_search_tool_responses() -> dict[str, Any]:
"""
Get the standard LiteLLM web search tool definition in Responses API format.
Used by async_pre_call_deployment_hook on the Responses API path, where a
function tool is a flat object (``type: "function"`` with a top-level
``name`` and ``parameters``) rather than the nested ``function`` wrapper
used by Chat Completions.
Returns:
Dict containing the Responses-style function tool definition.
"""
return {
"type": "function",
"name": LITELLM_WEB_SEARCH_TOOL_NAME,
"description": (
"Search the web for information. Use this when you need current "
"information or answers to questions that require up-to-date data."
),
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query to execute",
}
},
"required": ["query"],
},
}
def is_web_search_tool_responses(tool: dict[str, Any]) -> bool:
"""
Check if a tool is a web search tool for the Responses API.
Detects:
- OpenAI native Responses web search tools, whose ``type`` is one of
``web_search``, ``web_search_2025_08_26``, ``web_search_preview``,
``web_search_preview_2025_03_11`` (matched by the ``web_search`` prefix)
- The LiteLLM standard function tool in Responses shape:
``{"type": "function", "name": "litellm_web_search"}``
Args:
tool: Tool dictionary to check
Returns:
True if tool is a Responses-API web search tool
Example:
>>> is_web_search_tool_responses({"type": "web_search"})
True
>>> is_web_search_tool_responses({"type": "web_search_preview"})
True
>>> is_web_search_tool_responses({"type": "function", "name": "litellm_web_search"})
True
>>> is_web_search_tool_responses({"type": "function", "name": "get_weather"})
False
"""
tool_type = tool.get("type", "")
if not isinstance(tool_type, str):
return False
if tool_type == "function":
return tool.get("name") == LITELLM_WEB_SEARCH_TOOL_NAME
return tool_type == "web_search" or tool_type.startswith("web_search_")
def is_web_search_tool_chat_completion(tool: Dict[str, Any]) -> bool:
"""
Check if a tool is a web search tool for Chat Completions API (strict check).

64
litellm/integrations/websearch_interception/transformation.py
View file

@ -59,9 +59,73 @@ class WebSearchTransformation:
# Parse non-streaming response based on format
if response_format == "openai":
return WebSearchTransformation._detect_from_openai_response(response)
elif response_format == "responses":
return WebSearchTransformation._detect_from_responses_response(response)
else:
return WebSearchTransformation._detect_from_non_streaming_response(response)
@staticmethod
def _detect_from_responses_response(
response: Any,
) -> tuple[bool, list[dict]]:
"""Parse a Responses API response for ``litellm_web_search`` function calls.
After pre-request conversion the native web search tool is replaced by a
``litellm_web_search`` function tool, so the model emits ``function_call``
items in ``response.output`` instead of a native ``web_search_call``.
"""
if isinstance(response, dict):
output = response.get("output", [])
else:
output = getattr(response, "output", None) or []
if not isinstance(output, list):
return False, []
tool_calls: list[dict] = []
for item in output:
if isinstance(item, dict):
item_type = item.get("type")
item_name = item.get("name")
call_id = item.get("call_id")
arguments = item.get("arguments", "")
else:
item_type = getattr(item, "type", None)
item_name = getattr(item, "name", None)
call_id = getattr(item, "call_id", None)
arguments = getattr(item, "arguments", "")
if item_type != "function_call" or item_name != LITELLM_WEB_SEARCH_TOOL_NAME:
continue
if isinstance(arguments, str):
try:
parsed_input = json.loads(arguments) if arguments else {}
except json.JSONDecodeError:
verbose_logger.warning(
f"WebSearchInterception: Failed to parse function_call arguments: {arguments}"
)
parsed_input = {}
elif isinstance(arguments, dict):
parsed_input = arguments
else:
parsed_input = {}
arguments_str = arguments if isinstance(arguments, str) else json.dumps(parsed_input)
tool_calls.append(
{
"id": call_id,
"call_id": call_id,
"type": "function_call",
"name": item_name,
"arguments": arguments_str,
"input": parsed_input,
}
)
verbose_logger.debug(f"WebSearchInterception: Found {item_name} function_call with call_id={call_id}")
return len(tool_calls) > 0, tool_calls
@staticmethod
def _detect_from_non_streaming_response(
response: Any,

2
litellm/litellm_core_utils/dd_tracing.py
View file

@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, Optional, Union
from litellm.secret_managers.main import get_secret_bool
if TYPE_CHECKING:
from ddtrace.tracer import Tracer as DD_TRACER
from ddtrace.trace import Tracer as DD_TRACER
else:
DD_TRACER = Any

56
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -2,26 +2,8 @@ from typing import Optional
from litellm.llms.openai.data_residency import infer_openai_data_residency
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
OPTIONAL_KWARGS_KEYS = frozenset(
AWS_CREDENTIAL_KWARGS_KEYS = frozenset(
{
"azure_ad_token",
"tenant_id",
"client_id",
"client_secret",
"azure_username",
"azure_password",
"azure_scope",
"timeout",
"gcs_bucket_name",
"bucket_name",
"vertex_credentials",
"vertex_project",
"vertex_location",
"vertex_ai_project",
"vertex_ai_location",
"vertex_ai_credentials",
"aws_region_name",
"aws_access_key_id",
"aws_secret_access_key",
@ -34,14 +16,40 @@ OPTIONAL_KWARGS_KEYS = frozenset(
"aws_external_id",
"aws_bedrock_runtime_endpoint",
"aws_bedrock_project_id",
"tpm",
"rpm",
"itpm",
"otpm",
"use_xai_oauth",
}
)
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
OPTIONAL_KWARGS_KEYS = (
frozenset(
{
"azure_ad_token",
"tenant_id",
"client_id",
"client_secret",
"azure_username",
"azure_password",
"azure_scope",
"timeout",
"gcs_bucket_name",
"bucket_name",
"vertex_credentials",
"vertex_project",
"vertex_location",
"vertex_ai_project",
"vertex_ai_location",
"vertex_ai_credentials",
"tpm",
"rpm",
"itpm",
"otpm",
"use_xai_oauth",
}
)
| AWS_CREDENTIAL_KWARGS_KEYS
)
# Backward-compatible alias for existing imports/tests.
_OPTIONAL_KWARGS_KEYS = OPTIONAL_KWARGS_KEYS

15
litellm/litellm_core_utils/litellm_logging.py
View file

@ -73,6 +73,7 @@ from litellm.litellm_core_utils.model_param_helper import ModelParamHelper
from litellm.litellm_core_utils.redact_messages import (
redact_message_input_output_from_custom_logger,
redact_message_input_output_from_logging,
redact_streaming_responses_for_custom_logger,
)
from litellm.llms.base_llm.ocr.transformation import OCRResponse
from litellm.llms.base_llm.search.transformation import SearchResponse
@ -924,7 +925,6 @@ class Logging(LiteLLMLoggingBaseClass):
def pre_call(self, input, api_key, model=None, additional_args={}):
# Log the exact input to the LLM API
litellm.error_logs["PRE_CALL"] = locals()
try:
self._pre_call(
input=input,
@ -1134,7 +1134,6 @@ class Logging(LiteLLMLoggingBaseClass):
def post_call(self, original_response, input=None, api_key=None, additional_args={}):
# Log the exact result from the LLM API, for streaming - log the type of response received
litellm.error_logs["POST_CALL"] = locals()
if isinstance(original_response, dict):
original_response = json.dumps(original_response, default=str)
try:
@ -2576,6 +2575,9 @@ class Logging(LiteLLMLoggingBaseClass):
model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
model_call_details=model_call_details
)
model_call_details = redact_streaming_responses_for_custom_logger(
model_call_details=model_call_details, custom_logger=callback
)
##################################
if self.stream is True:
if "async_complete_streaming_response" in model_call_details:
@ -3070,7 +3072,7 @@ class Logging(LiteLLMLoggingBaseClass):
def get_combined_callback_list(self, dynamic_success_callbacks: Optional[List], global_callbacks: List) -> List:
if dynamic_success_callbacks is None:
return list(global_callbacks)
return list(set(dynamic_success_callbacks + global_callbacks))
return list(dict.fromkeys(dynamic_success_callbacks + global_callbacks))
def _remove_internal_litellm_callbacks(self, callbacks: List) -> List:
"""
@ -5208,10 +5210,15 @@ def get_standard_logging_object_payload(
call_type = kwargs.get("call_type")
cache_hit = kwargs.get("cache_hit", False)
# Extract usage as a plain dict, avoiding Pydantic round-trip
usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
raw_usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
response_obj=response_obj,
combined_usage_object=cast(Optional[Usage], kwargs.get("combined_usage_object")),
)
usage_dict = (
{**raw_usage_dict, "output_image_count": len(init_response_obj.data)}
if isinstance(init_response_obj, ImageResponse) and init_response_obj.data
else raw_usage_dict
)
id = response_obj.get("id", kwargs.get("litellm_call_id"))

35
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -445,6 +445,7 @@ class PromptTokensDetailsResult(TypedDict):
text_tokens: int
audio_tokens: int
image_tokens: int
video_tokens: int
character_count: int
image_count: int
video_length_seconds: float
@ -473,6 +474,7 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
)
audio_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "audio_tokens", 0)) or 0
image_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "image_tokens", 0)) or 0
video_tokens = _coerce_token_count(getattr(usage.prompt_tokens_details, "video_tokens", 0))
character_count = (
cast(
Optional[int],
@ -503,6 +505,7 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
text_tokens=text_tokens,
audio_tokens=audio_tokens,
image_tokens=image_tokens,
video_tokens=video_tokens,
character_count=character_count,
image_count=image_count,
video_length_seconds=float(video_length_seconds),
@ -515,6 +518,7 @@ class CompletionTokensDetailsResult(TypedDict):
text_tokens: int
reasoning_tokens: int
image_tokens: int
video_tokens: int
def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsResult:
@ -546,12 +550,14 @@ def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsRes
)
or 0
)
video_tokens = _coerce_token_count(getattr(usage.completion_tokens_details, "video_tokens", 0))
return CompletionTokensDetailsResult(
audio_tokens=audio_tokens,
text_tokens=text_tokens,
reasoning_tokens=reasoning_tokens,
image_tokens=image_tokens,
video_tokens=video_tokens,
)
@ -586,6 +592,13 @@ def _calculate_input_cost(
image_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, image_token_cost_key, prompt_tokens_details["image_tokens"])
### VIDEO TOKEN COST
if prompt_tokens_details["video_tokens"]:
video_token_cost_key = "input_cost_per_video_token"
if model_info.get(video_token_cost_key) is None:
video_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, video_token_cost_key, prompt_tokens_details["video_tokens"])
### CACHE WRITING COST - Now uses tiered pricing
if (
prompt_tokens_details["cache_creation_tokens"]
@ -698,6 +711,7 @@ def generic_cost_per_token(
text_tokens=usage.prompt_tokens,
audio_tokens=0,
image_tokens=0,
video_tokens=0,
character_count=0,
image_count=0,
video_length_seconds=0.0,
@ -716,13 +730,14 @@ def generic_cost_per_token(
audio_tokens = prompt_tokens_details["audio_tokens"]
cache_creation = prompt_tokens_details["cache_creation_tokens"]
image_tokens = prompt_tokens_details["image_tokens"]
video_tokens = prompt_tokens_details["video_tokens"]
# Check for double-counting: sum of details > prompt_tokens means overlap
total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens
total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens + video_tokens
has_double_counting = cache_hit > 0 and total_details > usage.prompt_tokens
if (text_tokens == 0 and prompt_tokens_details["image_count"] == 0) or has_double_counting:
text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens
text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens - video_tokens
# Clamp to zero: inconsistent streaming usage
if text_tokens < 0:
text_tokens = 0
@ -751,6 +766,7 @@ def generic_cost_per_token(
audio_tokens = 0
reasoning_tokens = 0
image_tokens = 0
video_tokens = 0
is_text_tokens_total = False
if usage.completion_tokens_details is not None:
completion_tokens_details = _parse_completion_tokens_details(usage)
@ -758,19 +774,20 @@ def generic_cost_per_token(
text_tokens = completion_tokens_details["text_tokens"]
reasoning_tokens = completion_tokens_details["reasoning_tokens"]
image_tokens = completion_tokens_details["image_tokens"]
video_tokens = completion_tokens_details["video_tokens"]
# Handle text_tokens calculation:
# 1. If text_tokens is explicitly provided and > 0, use it
# 2. If there's a breakdown (reasoning/audio/image tokens), calculate text_tokens as the remainder
# 2. If there's a breakdown (reasoning/audio/image/video tokens), calculate text_tokens as the remainder
# 3. If no breakdown at all, assume all completion_tokens are text_tokens
has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0
has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0 or video_tokens > 0
if text_tokens == 0:
if has_token_breakdown:
# Calculate text tokens as remainder when we have a breakdown
# This handles cases like OpenAI's reasoning models where text_tokens isn't provided
text_tokens = max(
0,
usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens,
usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens - video_tokens,
)
else:
# No breakdown at all, all tokens are text tokens
@ -803,6 +820,14 @@ def generic_cost_per_token(
)
completion_cost += float(image_tokens) * _output_cost_per_image_token
## VIDEO COST
if not is_text_tokens_total and video_tokens and video_tokens > 0:
_output_cost_per_video_token = _get_cost_per_unit(model_info, "output_cost_per_video_token", None)
_output_cost_per_video_token = (
_output_cost_per_video_token if _output_cost_per_video_token is not None else completion_base_cost
)
completion_cost += float(video_tokens) * _output_cost_per_video_token
## REGIONAL DATA-RESIDENCY UPLIFT
# Applied as a flat multiplier across all token costs for the request
# when the upstream is a regionalized OpenAI host (eu./us.api.openai.com).

53
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -5494,3 +5494,56 @@ def has_tool_with_name(tools: Any, tool_name: str) -> bool:
elif tool.get("name") == tool_name:
return True
return False
def resolve_structured_messages(
messages: list[dict[str, Any]] | None,
request_kwargs: dict[str, Any],
) -> list[dict[str, Any]] | None:
"""
Normalize a request's messages to OpenAI-spec chat-completions shape,
regardless of which API surface produced them (chat completions,
Anthropic /v1/messages, Responses API ``input``, etc).
Returns ``messages`` unchanged if already present. Otherwise dispatches
through the guardrail translation handlers (the same per-surface
conversion logic guardrails use) to convert e.g. Responses API ``input``
into a message list. Returns ``None`` if no messages could be resolved.
"""
if messages:
return messages
from litellm.litellm_core_utils.api_route_to_call_types import (
get_call_types_for_route,
)
from litellm.llms import load_guardrail_translation_mappings
from litellm.types.utils import CallTypes
mappings = load_guardrail_translation_mappings()
call_type: CallTypes | None = None
# 1. Try route-based inference from proxy metadata
route = request_kwargs.get("litellm_metadata", {}).get("user_api_key_request_route")
if route:
call_types_list = get_call_types_for_route(route)
if call_types_list:
for ct in call_types_list:
if ct in mappings:
call_type = ct
break
# 2. Fallback: try each mapped handler until one produces messages
handlers_to_try: list[Any] = []
if call_type is not None and call_type in mappings:
handlers_to_try.append(mappings[call_type]())
else:
handlers_to_try.extend(handler_cls() for handler_cls in mappings.values())
for handler in handlers_to_try:
structured = handler.get_structured_messages(request_kwargs)
if structured:
return [
msg if isinstance(msg, dict) else msg.model_dump() # type: ignore
for msg in structured
]
return None

98
litellm/litellm_core_utils/redact_messages.py
View file

@ -38,10 +38,61 @@ def redact_message_input_output_from_custom_logger(
litellm_logging_obj: LiteLLMLoggingObject, result, custom_logger: CustomLogger
):
if hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True:
return perform_redaction(litellm_logging_obj.model_call_details, result)
return perform_redaction(litellm_logging_obj.model_call_details, result, redact_streaming_responses=False)
return result
def redact_streaming_responses_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
"""
Returns a copy of model_call_details whose streaming response entries are redacted deepcopies
when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks still receive the unredacted response.
"""
if not (hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True):
return model_call_details
redacted_entries = {
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
}
if not redacted_entries:
return model_call_details
return {**model_call_details, **redacted_entries}
def _redacted_streaming_response_copy(streaming_response):
redacted_response = copy.deepcopy(streaming_response)
_redact_streaming_response(redacted_response)
return redacted_response
def _redact_streaming_response(streaming_response):
if hasattr(streaming_response, "choices"):
for choice in streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(streaming_response)
elif hasattr(streaming_response, "output"):
_redact_responses_api_output(streaming_response.output)
if hasattr(streaming_response, "reasoning") and streaming_response.reasoning is not None:
streaming_response.reasoning = None
def _redact_tool_calls(tool_calls) -> None:
"""Redact tool call arguments (assistant tool calls carry prompt-derived data)."""
if not tool_calls:
return
for tool_call in tool_calls:
function = getattr(tool_call, "function", None)
if function is not None and hasattr(function, "arguments"):
function.arguments = "redacted-by-litellm"
def _redact_function_call(function_call) -> None:
"""Redact legacy assistant function_call arguments."""
if function_call is not None and hasattr(function_call, "arguments"):
function_call.arguments = "redacted-by-litellm"
def _redact_choice_content(choice):
"""Helper to redact content in a choice (message or delta)."""
if isinstance(choice, litellm.Choices):
@ -50,12 +101,16 @@ def _redact_choice_content(choice):
choice.message.reasoning_content = "redacted-by-litellm"
if hasattr(choice.message, "thinking_blocks"):
choice.message.thinking_blocks = None
_redact_tool_calls(getattr(choice.message, "tool_calls", None))
_redact_function_call(getattr(choice.message, "function_call", None))
elif isinstance(choice, litellm.utils.StreamingChoices):
choice.delta.content = "redacted-by-litellm"
if hasattr(choice.delta, "reasoning_content"):
choice.delta.reasoning_content = "redacted-by-litellm"
if hasattr(choice.delta, "thinking_blocks"):
choice.delta.thinking_blocks = None
_redact_tool_calls(getattr(choice.delta, "tool_calls", None))
_redact_function_call(getattr(choice.delta, "function_call", None))
def _redact_responses_api_output(output_items):
@ -76,6 +131,9 @@ def _redact_responses_api_output(output_items):
if hasattr(summary_item, "text"):
summary_item.text = "redacted-by-litellm"
if hasattr(output_item, "type") and output_item.type == "function_call" and hasattr(output_item, "arguments"):
output_item.arguments = "redacted-by-litellm"
def _redact_responses_api_output_dict(output_items, redacted_str: str):
"""Helper to redact ResponsesAPIResponse output items in dict form."""
@ -96,6 +154,9 @@ def _redact_responses_api_output_dict(output_items, redacted_str: str):
if isinstance(summary_item, dict) and "text" in summary_item:
summary_item["text"] = redacted_str
if output_item.get("type") == "function_call" and "arguments" in output_item:
output_item["arguments"] = redacted_str
def _redact_standard_logging_object(model_call_details: dict):
"""Redact messages and response inside standard_logging_object if present."""
@ -127,6 +188,19 @@ def _redact_standard_logging_object(model_call_details: dict):
standard_logging_object["response"] = {"text": redacted_str}
def _redact_tool_calls_dict(message: dict, redacted_str: str) -> None:
"""Redact tool call / function_call arguments in a dict-form message or delta."""
tool_calls = message.get("tool_calls")
if isinstance(tool_calls, list):
for tool_call in tool_calls:
if isinstance(tool_call, dict) and isinstance(tool_call.get("function"), dict):
tool_call["function"]["arguments"] = redacted_str
function_call = message.get("function_call")
if isinstance(function_call, dict) and "arguments" in function_call:
function_call["arguments"] = redacted_str
def _redact_model_response_dict_choices(choices, redacted_str: str):
for choice in choices:
if isinstance(choice, dict):
@ -138,6 +212,7 @@ def _redact_model_response_dict_choices(choices, redacted_str: str):
choice["message"]["thinking_blocks"] = None
if "audio" in choice["message"]:
choice["message"]["audio"] = None
_redact_tool_calls_dict(choice["message"], redacted_str)
elif "delta" in choice and isinstance(choice["delta"], dict):
choice["delta"]["content"] = redacted_str
if "reasoning_content" in choice["delta"]:
@ -146,13 +221,18 @@ def _redact_model_response_dict_choices(choices, redacted_str: str):
choice["delta"]["thinking_blocks"] = None
if "audio" in choice["delta"]:
choice["delta"]["audio"] = None
_redact_tool_calls_dict(choice["delta"], redacted_str)
else:
_redact_choice_content(choice)
def perform_redaction(model_call_details: dict, result):
def perform_redaction(model_call_details: dict, result, redact_streaming_responses: bool = True):
"""
Performs the actual redaction on the logging object and result.
redact_streaming_responses=False skips the in-place redaction of the shared streaming
response entries; per-callback redaction hands each opted-out callback its own redacted
copy via redact_streaming_responses_for_custom_logger instead.
"""
# Redact model_call_details
model_call_details["messages"] = [{"role": "user", "content": "redacted-by-litellm"}]
@ -162,17 +242,9 @@ def perform_redaction(model_call_details: dict, result):
redact_vertex_ai_metadata_from_litellm_params(model_call_details)
# Redact streaming response
if model_call_details.get("stream", False) is True and "complete_streaming_response" in model_call_details:
_streaming_response = model_call_details["complete_streaming_response"]
if hasattr(_streaming_response, "choices"):
for choice in _streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(_streaming_response)
elif hasattr(_streaming_response, "output"):
_redact_responses_api_output(_streaming_response.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_streaming_response, "reasoning") and _streaming_response.reasoning is not None:
_streaming_response.reasoning = None
if redact_streaming_responses and model_call_details.get("stream", False) is True:
for _streaming_key in ("complete_streaming_response", "async_complete_streaming_response"):
_redact_streaming_response(model_call_details.get(_streaming_key))
# Redact result
if result is not None:

4
litellm/litellm_core_utils/secret_redaction.py
View file

@ -9,6 +9,8 @@ secrets from strings without depending on the logging-configuration module.
import re
from typing import List
from litellm.constants import MINIMUM_CUSTOM_KEY_LENGTH
_REDACTED = "REDACTED"
@ -30,7 +32,7 @@ def _build_secret_patterns() -> "re.Pattern[str]":
# Basic auth headers
r"Basic\s+[A-Za-z0-9+/]{10,}={0,2}",
# OpenAI / Anthropic sk- prefixed keys
r"sk-[A-Za-z0-9\-_]{20,}",
rf"sk-[A-Za-z0-9\-_]{{{MINIMUM_CUSTOM_KEY_LENGTH - len('sk-')},}}",
# Generic api_key / api-key / apikey (handles 'key': 'value' dict repr)
r"(?:api[_-]?key)['\"]?\s*[:=]\s*['\"]?[^\s,'\"})\]{}>]{8,}",
# x-api-key / api-key header values (handles 'key': 'value' dict repr)

40
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -467,6 +467,7 @@ class ChunkProcessor:
cache_read_input_tokens: Optional[int] = None
completion_tokens_details: Optional[CompletionTokensDetails] = None
prompt_tokens_details: Optional[PromptTokensDetailsWrapper] = None
cost: Optional[float] = None
if "prompt_tokens" in usage_chunk:
prompt_tokens = usage_chunk.get("prompt_tokens", 0) or 0
@ -476,6 +477,8 @@ class ChunkProcessor:
cache_creation_input_tokens = usage_chunk.get("cache_creation_input_tokens")
if "cache_read_input_tokens" in usage_chunk:
cache_read_input_tokens = usage_chunk.get("cache_read_input_tokens")
if "cost" in usage_chunk:
cost = usage_chunk.get("cost")
if hasattr(usage_chunk, "completion_tokens_details"):
if isinstance(usage_chunk.completion_tokens_details, dict):
completion_tokens_details = CompletionTokensDetails(**usage_chunk.completion_tokens_details)
@ -494,6 +497,7 @@ class ChunkProcessor:
"cache_read_input_tokens": cache_read_input_tokens,
"completion_tokens_details": completion_tokens_details,
"prompt_tokens_details": prompt_tokens_details,
"cost": cost,
}
def count_reasoning_tokens(self, response: ModelResponse) -> Optional[int]:
@ -512,6 +516,22 @@ class ChunkProcessor:
return reasoning_tokens
@staticmethod
def _extract_usage_chunk(chunk: dict[str, Any] | ModelResponse | ModelResponseStream) -> Usage | None:
usage_chunk: Usage | dict[str, Any] | None = None
if hasattr(chunk, "usage") and chunk.usage is not None:
usage_chunk = chunk.usage
elif "usage" in chunk:
usage_chunk = chunk["usage"]
elif (isinstance(chunk, ModelResponse) or isinstance(chunk, ModelResponseStream)) and hasattr(
chunk, "_hidden_params"
):
usage_chunk = chunk._hidden_params.get("usage", None)
if isinstance(usage_chunk, dict):
return Usage(**usage_chunk)
return usage_chunk
def _calculate_usage_per_chunk(
self,
chunks: List[Union[Dict[str, Any], ModelResponse]],
@ -548,18 +568,12 @@ class ChunkProcessor:
# is last-wins, so without preserving this separately the 1h breakdown is
# lost and 1h cache writes get billed at the 5m rate.
cache_creation_token_details: Optional[CacheCreationTokenDetails] = None
cost: Optional[float] = None
for chunk in chunks:
usage_chunk: Optional[Usage] = None
if "usage" in chunk:
usage_chunk = chunk["usage"]
elif (isinstance(chunk, ModelResponse) or isinstance(chunk, ModelResponseStream)) and hasattr(
chunk, "_hidden_params"
):
usage_chunk = chunk._hidden_params.get("usage", None)
usage_chunk = self._extract_usage_chunk(chunk)
if usage_chunk is not None:
if isinstance(usage_chunk, dict):
usage_chunk = Usage(**usage_chunk)
usage_chunk_dict = self._usage_chunk_calculation_helper(usage_chunk)
if usage_chunk_dict["prompt_tokens"] is not None and usage_chunk_dict["prompt_tokens"] > 0:
prompt_tokens = usage_chunk_dict["prompt_tokens"]
@ -610,6 +624,9 @@ class ChunkProcessor:
prompt_tokens_details, cache_creation_token_details
)
if usage_chunk_dict["cost"] is not None:
cost = usage_chunk_dict["cost"]
prompt_tokens_details = self._attach_cache_creation_token_details(
prompt_tokens_details, cache_creation_token_details
)
@ -629,6 +646,7 @@ class ChunkProcessor:
web_search_requests=web_search_requests,
completion_tokens_details=completion_tokens_details,
prompt_tokens_details=prompt_tokens_details,
cost=cost,
)
@staticmethod
@ -727,6 +745,7 @@ class ChunkProcessor:
prompt_tokens_details: Optional[PromptTokensDetailsWrapper] = calculated_usage_per_chunk[
"prompt_tokens_details"
]
cost: Optional[float] = calculated_usage_per_chunk["cost"]
try:
returned_usage.prompt_tokens = prompt_tokens or token_counter(model=model, messages=messages)
@ -784,6 +803,9 @@ class ChunkProcessor:
else:
returned_usage.prompt_tokens_details.web_search_requests = web_search_requests
if cost is not None:
setattr(returned_usage, "cost", cost)
# Return a new usage object with the new values
returned_usage = Usage(**returned_usage.model_dump())

59
litellm/litellm_core_utils/streaming_handler.py
View file

@ -962,10 +962,11 @@ class CustomStreamWrapper:
if self.custom_llm_provider == "bedrock" and "trace" in model_response:
return model_response
# Default - return StopIteration
if hasattr(model_response, "usage"):
self.chunks.append(model_response)
raise StopIteration
# Don't raise StopIteration here - some providers (like OpenRouter)
# send usage/cost data in chunks after the finish_reason chunk
if hasattr(model_response, "usage") and model_response.usage is not None:
return model_response
return
# flush any remaining holding chunk
if len(self.holding_chunk) > 0:
if model_response.choices[0].delta.content is None:
@ -1474,12 +1475,16 @@ class CustomStreamWrapper:
self.tool_call = True
if hasattr(chunk, "usage") and chunk.usage is not None:
model_response.usage = chunk.usage
## RETURN ARG
return self.return_processed_chunk_logic(
result = self.return_processed_chunk_logic(
completion_obj=completion_obj,
model_response=model_response, # type: ignore
response_obj=response_obj,
)
return result
except StopIteration:
raise StopIteration
@ -1686,6 +1691,21 @@ class CustomStreamWrapper:
model_response.choices[0].finish_reason = "tool_calls"
return model_response
@staticmethod
def _propagate_usage_cost_to_hidden_params(
response: "ModelResponse",
) -> None:
"""
If the assembled response carries a provider-reported cost on
usage.cost, copy it into _hidden_params so litellm's cost
calculator uses it instead of a token-based estimate.
"""
_usage = getattr(response, "usage", None)
if _usage is not None and hasattr(_usage, "cost") and _usage.cost is not None:
if "additional_headers" not in response._hidden_params:
response._hidden_params["additional_headers"] = {}
response._hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] = float(_usage.cost)
def __next__(self) -> "ModelResponseStream":
cache_hit = False
if self.custom_llm_provider is not None and self.custom_llm_provider == "cached_response":
@ -1741,6 +1761,10 @@ class CustomStreamWrapper:
# hasattr(response, "usage") is always True — must check
# `is not None` to avoid running this path on every chunk.
if getattr(response, "usage", None) is not None:
usage_to_preserve = response.usage
if usage_to_preserve:
response._hidden_params["usage"] = usage_to_preserve
obj_dict = response.model_dump()
if "usage" in obj_dict:
@ -1789,6 +1813,8 @@ class CustomStreamWrapper:
response = self.model_response_creator()
if complete_streaming_response is not None:
self._propagate_usage_cost_to_hidden_params(complete_streaming_response)
setattr(
response,
"usage",
@ -1999,6 +2025,8 @@ class CustomStreamWrapper:
response = self.model_response_creator()
if complete_streaming_response is not None:
self._propagate_usage_cost_to_hidden_params(complete_streaming_response)
setattr(
response,
"usage",
@ -2228,12 +2256,16 @@ def calculate_total_usage(chunks: List[ModelResponse]) -> Usage:
"""Assume most recent usage chunk has total usage uptil then."""
prompt_tokens: int = 0
completion_tokens: int = 0
latest_usage_chunk = None
for chunk in chunks:
if "usage" in chunk and chunk["usage"] is not None:
if "prompt_tokens" in chunk["usage"]:
prompt_tokens = chunk["usage"].get("prompt_tokens", 0) or 0
if "completion_tokens" in chunk["usage"]:
completion_tokens = chunk["usage"].get("completion_tokens", 0) or 0
usage = chunk["usage"]
latest_usage_chunk = usage
if "prompt_tokens" in usage:
prompt_tokens = usage.get("prompt_tokens", 0) or 0
if "completion_tokens" in usage:
completion_tokens = usage.get("completion_tokens", 0) or 0
returned_usage_chunk = Usage(
prompt_tokens=prompt_tokens,
@ -2241,6 +2273,15 @@ def calculate_total_usage(chunks: List[ModelResponse]) -> Usage:
total_tokens=prompt_tokens + completion_tokens,
)
if latest_usage_chunk is not None:
latest_cost = (
latest_usage_chunk.get("cost")
if isinstance(latest_usage_chunk, dict)
else getattr(latest_usage_chunk, "cost", None)
)
if latest_cost is not None:
returned_usage_chunk.cost = latest_cost
return returned_usage_chunk

76
litellm/llms/anthropic/chat/transformation.py
View file

@ -227,6 +227,10 @@ DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING = (
"Sonnet 4.6+, and Mythos Preview."
)
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING = (
"Dropping adaptive `thinking` for model=%s: max_tokens is too small to fit the minimum thinking budget."
)
DROP_UNSUPPORTED_SPEED_WARNING = (
"Dropping unsupported `speed` for model=%s (drop_params=True). Fast mode is only supported on select Opus models."
)
@ -1220,6 +1224,23 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
llm_provider=llm_provider,
)
@staticmethod
def _cap_thinking_budget_to_max_tokens(
thinking: AnthropicThinkingParam, max_tokens: Optional[int]
) -> Optional[AnthropicThinkingParam]:
"""Cap a legacy ``thinking.budget_tokens`` below ``max_tokens`` (Anthropic
requires ``max_tokens > budget_tokens``). Returns the (possibly capped)
thinking dict, or ``None`` when ``max_tokens`` is too small to fit even the
minimum thinking budget and thinking should be dropped."""
budget = thinking.get("budget_tokens")
if max_tokens is None or not isinstance(budget, int):
return thinking
if max_tokens <= ANTHROPIC_MIN_THINKING_BUDGET_TOKENS:
return None
if budget < max_tokens:
return thinking
return AnthropicThinkingParam(type=thinking.get("type", "enabled"), budget_tokens=max_tokens - 1)
def _extract_json_schema_from_response_format(self, value: Optional[dict]) -> Optional[dict]:
if value is None:
return None
@ -1420,24 +1441,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
output_key=param,
)
elif param == "response_format" and isinstance(value, dict):
if any(
substring in model
for substring in {
"sonnet-4.5",
"sonnet-4-5",
"opus-4.1",
"opus-4-1",
"opus-4.5",
"opus-4-5",
"opus-4.6",
"opus-4-6",
"opus-4.7",
"opus-4-7",
"sonnet-4.6",
"sonnet-4-6",
"sonnet_4.6",
"sonnet_4_6",
}
if AnthropicConfig._supports_model_capability(
model,
"supports_native_structured_output",
self._resolved_provider,
):
_output_format = self.map_response_format_to_anthropic_output_format(value)
if _output_format is not None:
@ -1463,7 +1470,38 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
):
optional_params["metadata"] = {"user_id": value}
elif param == "thinking":
optional_params["thinking"] = value
if (
isinstance(value, dict)
and value.get("type") == "adaptive"
and not AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider)
):
# Callers (e.g. Claude Code) send adaptive thinking
# unconditionally; translate it down to the legacy
# `thinking={type: enabled, budget_tokens}` interface a
# pre-4.6 model actually supports instead of forwarding a
# shape the model will reject.
max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
legacy_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort="medium",
model=model,
custom_llm_provider=self._resolved_provider,
llm_provider=self._resolved_provider,
)
capped_thinking = (
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
if capped_thinking is not None:
optional_params["thinking"] = capped_thinking
else:
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
model,
)
optional_params.pop("thinking", None)
else:
optional_params["thinking"] = value
elif param == "reasoning_effort":
# Accept both string ("low") and dict ({"effort": "low",
# "summary": "concise"}). The Responses->Chat parser keeps the

12
litellm/llms/anthropic/common_utils.py
View file

@ -340,11 +340,15 @@ class AnthropicModelInfo(BaseLLMModelInfo):
def _get_model_capability(model: str, key: str) -> Optional[bool]:
"""Read boolean capability ``key`` from the model map, or None when
no entry declares it."""
from litellm.utils import _get_bundled_model_cost_map
try:
for cand in AnthropicModelInfo._model_map_lookup_candidates(model):
value = litellm.model_cost.get(cand, {}).get(key)
if isinstance(value, bool):
return value
candidates = AnthropicModelInfo._model_map_lookup_candidates(model)
for model_cost in (litellm.model_cost, _get_bundled_model_cost_map()):
for cand in candidates:
value = model_cost.get(cand, {}).get(key)
if isinstance(value, bool):
return value
except Exception:
pass
return None

74
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
View file

@ -13,14 +13,18 @@ from typing import (
List,
Literal,
Optional,
get_args,
)
from typing_extensions import assert_never
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.types.llms.anthropic import (
AppliedEdit,
CompactionBlock,
ContextManagementResponse,
StreamingContentBlockDeltaType,
UsageDelta,
UsageIteration,
)
@ -30,6 +34,23 @@ if TYPE_CHECKING:
from litellm.types.utils import ModelResponseStream
_STREAMING_DELTA_TYPES = frozenset(get_args(StreamingContentBlockDeltaType))
def _delta_payload_field(delta_type: StreamingContentBlockDeltaType) -> str:
match delta_type:
case "text_delta":
return "text"
case "input_json_delta":
return "partial_json"
case "thinking_delta":
return "thinking"
case "signature_delta":
return "signature"
case _:
assert_never(delta_type)
class _CombinedChunkSplitter:
"""
Splits a streaming chunk that carries BOTH response content and a
@ -458,12 +479,15 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
# 3. If the trigger chunk carries delta content, queue it
# so the first delta of the new block is not silently dropped.
if self._trigger_delta_has_content(processed_chunk):
if self._delta_has_content(processed_chunk):
self.chunk_queue.append(processed_chunk)
self.sent_content_block_finish = False
return self.chunk_queue.popleft()
if processed_chunk["type"] == "content_block_delta" and not self._delta_has_content(processed_chunk):
continue
if processed_chunk["type"] == "message_delta" and self.sent_content_block_finish is False:
# Queue both the content_block_stop and the message_delta
self.chunk_queue.append(
@ -670,13 +694,18 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
# 3. If the trigger chunk carries delta content, queue it
# so the first delta of the new block is not silently dropped.
if self._trigger_delta_has_content(processed_chunk):
if self._delta_has_content(processed_chunk):
self.chunk_queue.append(processed_chunk)
# Reset state for new block
self.sent_content_block_finish = False
return self.chunk_queue.popleft()
if processed_chunk["type"] == "content_block_delta" and not self._delta_has_content(
processed_chunk
):
continue
if processed_chunk["type"] == "message_delta" and self.sent_content_block_finish is False:
# Queue both the content_block_stop and the holding chunk
self.chunk_queue.append(
@ -808,20 +837,33 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
self.current_content_block_index += 1
@staticmethod
def _trigger_delta_has_content(processed_chunk: Dict[str, Any]) -> bool:
"""Return True if a translated trigger chunk carries a non-empty
``content_block_delta`` payload that must be re-emitted after a
block transition.
def _delta_has_content(processed_chunk: Dict[str, Any]) -> bool:
"""Return True if a translated chunk carries a non-empty
``content_block_delta`` payload.
When an upstream chunk both *triggers* a new content block (its type
differs from the active block) and *carries* delta content, that
content belongs to the new block. The synthesized
``content_block_start`` only ever carries an empty body — see
Gates every ``content_block_delta`` emission. An empty delta carries
no information, and the translate fallback types empty deltas as
``text_delta`` regardless of the active block's type — emitting one
into an open ``thinking`` block (e.g. Bedrock Converse sends an empty
reasoning delta mid-block) crashes strict Anthropic SDK clients with
"Content block is not a text block".
Also gates re-emission after a block transition: when an upstream
chunk both *triggers* a new content block (its type differs from the
active block) and *carries* delta content, that content belongs to
the new block. The synthesized ``content_block_start`` only ever
carries an empty body — see
``_translate_streaming_openai_chunk_to_anthropic_content_block``,
which returns an empty ``TextBlock``/``ToolUseBlock``/thinking block —
so the trigger chunk's delta must be re-queued or the first token of
the new block (the first non-empty text/thinking delta, or bundled
tool arguments) is silently dropped.
Delta types outside ``StreamingContentBlockDeltaType`` — the closed
set the translate layer can produce — are treated as empty. The
per-type payload lookup is exhaustively matched against that set in
``_delta_payload_field``, so extending the translate layer with a new
delta type fails type-checking here until it is handled.
"""
if processed_chunk.get("type") != "content_block_delta":
return False
@ -829,15 +871,9 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
if not isinstance(delta, dict):
return False
delta_type = delta.get("type")
if delta_type == "text_delta":
return bool(delta.get("text"))
if delta_type == "input_json_delta":
return bool(delta.get("partial_json"))
if delta_type == "thinking_delta":
return bool(delta.get("thinking"))
if delta_type == "signature_delta":
return bool(delta.get("signature"))
return False
if delta_type not in _STREAMING_DELTA_TYPES:
return False
return bool(delta.get(_delta_payload_field(delta_type)))
def _should_start_new_content_block(self, chunk: "ModelResponseStream") -> bool:
"""

15
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -104,6 +104,7 @@ from litellm.types.llms.anthropic import (
ContextManagementResponse,
MessageBlockDelta,
MessageDelta,
StreamingContentBlockDeltaType,
UsageDelta,
UsageIteration,
)
@ -1402,11 +1403,6 @@ class LiteLLMAnthropicMessagesAdapter:
assert isinstance(thinking, str)
assert isinstance(signature, str)
if thinking and signature:
raise ValueError(
"Both `thinking` and `signature` in a single streaming chunk isn't supported."
)
return "thinking", ChatCompletionThinkingBlock(
type="thinking", thinking=thinking, signature=signature
)
@ -1423,7 +1419,7 @@ class LiteLLMAnthropicMessagesAdapter:
def _translate_streaming_openai_chunk_to_anthropic(
self, choices: List[Union[OpenAIStreamingChoice, StreamingChoices]]
) -> Tuple[
Literal["text_delta", "input_json_delta", "thinking_delta", "signature_delta"],
StreamingContentBlockDeltaType,
Union[
ContentTextBlockDelta,
ContentJsonBlockDelta,
@ -1462,17 +1458,14 @@ class LiteLLMAnthropicMessagesAdapter:
if choice.delta.reasoning_content is not None:
reasoning_content += choice.delta.reasoning_content
if reasoning_content and reasoning_signature:
raise ValueError("Both `reasoning` and `signature` in a single streaming chunk isn't supported.")
if partial_json is not None:
return "input_json_delta", ContentJsonBlockDelta(type="input_json_delta", partial_json=partial_json)
elif reasoning_content:
return "thinking_delta", ContentThinkingBlockDelta(type="thinking_delta", thinking=reasoning_content)
elif reasoning_signature:
return "signature_delta", ContentThinkingSignatureBlockDelta(
type="signature_delta", signature=reasoning_signature
)
elif reasoning_content:
return "thinking_delta", ContentThinkingBlockDelta(type="thinking_delta", thinking=reasoning_content)
else:
return "text_delta", ContentTextBlockDelta(type="text_delta", text=text)

50
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -3,7 +3,6 @@ from typing import Any, AsyncIterator, Dict, List, Optional, Tuple
import httpx
from litellm.constants import (
ANTHROPIC_MIN_THINKING_BUDGET_TOKENS,
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_XHIGH_THINKING_BUDGET,
@ -358,7 +357,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
except _BadRequestError as e:
raise AnthropicError(message=str(e.message), status_code=400)
capped_thinking = (
AnthropicMessagesConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
@ -377,19 +376,34 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
optional_params.pop("output_config", None)
@staticmethod
def _cap_thinking_budget_to_max_tokens(thinking: Dict, max_tokens: Optional[int]) -> Optional[Dict]:
"""Cap a legacy ``thinking.budget_tokens`` below ``max_tokens`` (Anthropic
requires ``max_tokens > budget_tokens``). Returns the (possibly capped)
thinking dict, or ``None`` when ``max_tokens`` is too small to fit even the
minimum thinking budget and thinking should be dropped."""
budget = thinking.get("budget_tokens")
if max_tokens is None or not isinstance(budget, int):
return thinking
if max_tokens <= ANTHROPIC_MIN_THINKING_BUDGET_TOKENS:
return None
if budget < max_tokens:
return thinking
return {**thinking, "budget_tokens": max_tokens - 1}
def _drop_incompatible_temperature_for_thinking(
model: str, optional_params: dict, custom_llm_provider: str
) -> None:
"""Anthropic rejects any ``temperature`` other than 1 while extended thinking
is enabled ("temperature may only be set to 1 when thinking is enabled").
Clients like Claude Code send ``thinking``/``output_config.effort`` together
with a pinned ``temperature`` (e.g. the safety classifier uses ``temperature=0``
for determinism). When the request lands on a non-adaptive model, the effort
interface is reshaped above into legacy ``thinking={type: enabled}`` (or kept
as ``output_config.effort`` on Opus 4.5), and the leftover ``temperature`` would
400. Preserving the thinking the caller asked for wins over an unhonorable
sampling value (Anthropic forces ``temperature=1`` under thinking regardless),
so drop it and let the API default apply.
Adaptive models (4.6+) own this natively and are left untouched.
"""
if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
return
temperature = optional_params.get("temperature")
if temperature is None or temperature == 1:
return
thinking = optional_params.get("thinking")
output_config = optional_params.get("output_config")
thinking_enabled = isinstance(thinking, dict) and thinking.get("type") == "enabled"
effort_enabled = isinstance(output_config, dict) and output_config.get("effort") is not None
if thinking_enabled or effort_enabled:
optional_params.pop("temperature", None)
def transform_anthropic_messages_request(
self,
@ -431,6 +445,12 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
custom_llm_provider=self._resolved_provider,
)
self._drop_incompatible_temperature_for_thinking(
model=model,
optional_params=anthropic_messages_optional_request_params,
custom_llm_provider=self._resolved_provider,
)
system_param = anthropic_messages_optional_request_params.get("system")
if self.should_strip_billing_metadata() and system_param is not None:
filtered_system = self._filter_billing_headers_from_system(system_param)

8
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py
View file

@ -198,14 +198,16 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
@staticmethod
def translate_tool_choice_to_responses_api(
tool_choice: AnthropicMessagesToolChoice,
) -> Dict[str, Any]:
) -> Union[str, dict[str, Any]]:
"""Convert Anthropic tool_choice to Responses API tool_choice."""
tc_type = tool_choice.get("type")
if tc_type == "any":
return {"type": "required"}
return "required"
elif tc_type == "tool":
return {"type": "function", "name": tool_choice.get("name", "")}
return {"type": "auto"}
elif tc_type == "none":
return "none"
return "auto"
@staticmethod
def translate_context_management_to_responses_api(

38
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -1,4 +1,5 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
@ -11,10 +12,30 @@ if TYPE_CHECKING:
from litellm.types.llms.openai import AllMessageValues
@dataclass(slots=True)
class StreamTransformSink:
"""Out-parameter used by ``process_output_streaming_response`` to hand the
guardrailed streaming state back to the caller.
The streaming text-transform path must not mutate ``responses_so_far`` (it is
the raw accumulator the guardrail re-reads every round), so the guardrailed
accumulated text per choice (``mutated_text_per_choice``, keyed by
``StreamingChoices.index``) and the per-choice trailing holdback the guardrail
requested (``holdback_per_choice``, from ``stream_holdback_chars``) are
reported here instead of in place. Only the OpenAI chat handler populates this
today; the hook passes a fresh sink per round and reads it afterwards. A
mutable dataclass is deliberate: it is a write-once output parameter for a
single call, not shared state.
"""
mutated_text_per_choice: dict[int, str] = field(default_factory=dict)
holdback_per_choice: dict[int, int] = field(default_factory=dict)
class BaseTranslation(ABC):
@staticmethod
def transform_user_api_key_dict_to_metadata(
user_api_key_dict: Optional[Any],
user_api_key_dict: Any | None,
) -> Dict[str, Any]:
"""
Transform user_api_key_dict to a metadata dict with prefixed keys.
@ -73,7 +94,7 @@ class BaseTranslation(ABC):
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
request_data: Optional[dict] = None,
request_data: dict | None = None,
) -> Any:
"""
Process output response with guardrails.
@ -92,12 +113,15 @@ class BaseTranslation(ABC):
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
request_data: Optional[dict] = None,
request_data: dict | None = None,
stream_transform_sink: StreamTransformSink | None = None,
) -> Any:
"""
Process output streaming response with guardrails.
Optional to override in subclasses.
Optional to override in subclasses. ``stream_transform_sink`` is the
out-parameter used by handlers that support streaming text
transformations (see ``StreamTransformSink``); base handlers ignore it.
"""
return responses_so_far
@ -105,8 +129,8 @@ class BaseTranslation(ABC):
self,
exc: "ModifyResponseException",
stream_started: bool = False,
responses_so_far: Optional[list[Any]] = None,
) -> Optional[list[bytes]]:
responses_so_far: list[Any] | None = None,
) -> list[bytes] | None:
"""
Build the streaming chunks that deliver a guardrail block message and
cleanly terminate the stream in this provider's wire format.
@ -125,7 +149,7 @@ class BaseTranslation(ABC):
"""
return None
def get_structured_messages(self, data: dict) -> Optional[List["AllMessageValues"]]:
def get_structured_messages(self, data: dict) -> List["AllMessageValues"] | None:
"""
Convert request data to OpenAI-spec structured messages.

9
litellm/llms/bedrock/base_aws_llm.py
View file

@ -877,6 +877,15 @@ class BaseAWSLLM:
"Resource": "*",
"Condition": {"Bool": {"aws:SecureTransport": "true"}},
},
{
"Sid": "BedrockMantleLiteLLM",
"Effect": "Allow",
"Action": [
"bedrock-mantle:CreateInference",
],
"Resource": "*",
"Condition": {"Bool": {"aws:SecureTransport": "true"}},
},
],
}
assume_role_params = {

47
litellm/llms/bedrock/batches/transformation.py
View file

@ -5,6 +5,9 @@ from typing import Any, Dict, List, Literal, Optional, Union, cast
from httpx import Headers, Response
from litellm.litellm_core_utils.cloud_storage_security import (
BEDROCK_MANAGED_S3_BATCH_PREFIX,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.llms.base_llm.batches.transformation import BaseBatchesConfig
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -26,6 +29,15 @@ from litellm.types.utils import LiteLLMBatch, LlmProviders
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import CommonBatchFilesUtils
# Bedrock batch input files are uploaded as
# s3://bucket/litellm-bedrock-files-{model, ":" -> "-"}-{uuid4}.jsonl (see
# BedrockFilesTransformation._get_s3_object_name). A uuid4 is always 36 hex/dash
# characters, so it can be stripped off the end unambiguously even though the
# model name itself may contain dashes.
_S3_BATCH_FILE_UUID_SUFFIX_PATTERN = re.compile(
r"-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\.jsonl$"
)
class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
"""
@ -40,6 +52,41 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.BEDROCK
@classmethod
def _get_bare_model_name_from_s3_key(cls, object_key: str) -> Optional[str]:
if not object_key.startswith(BEDROCK_MANAGED_S3_BATCH_PREFIX):
return None
model_part = object_key[len(BEDROCK_MANAGED_S3_BATCH_PREFIX) :]
match = _S3_BATCH_FILE_UUID_SUFFIX_PATTERN.search(model_part)
if not match or match.start() == 0:
return None
return model_part[: match.start()]
@classmethod
def is_unmanaged_s3_batch_input_file_id(cls, input_file_id: Optional[str]) -> bool:
"""
Returns True if `input_file_id` is a raw s3:// Bedrock batch input file (i.e. not a
LiteLLM-managed unified file id) whose object key embeds the model name in the
`litellm-bedrock-files-{model}-{uuid}.jsonl` layout.
"""
if input_file_id is None or not input_file_id.startswith("s3://"):
return False
object_key = input_file_id.rsplit("/", 1)[-1]
return cls._get_bare_model_name_from_s3_key(object_key) is not None
@classmethod
def get_bare_model_name_from_s3_file(cls, input_file_id: str) -> str:
"""
Extracts the bare model name (e.g. "us.anthropic.claude-sonnet-4-20250514-v1-0") from
an unmanaged batch's s3:// input file id. Note any ":" in the original model id was
replaced with "-" at upload time, so callers must fuzzy-match against configured
deployments rather than expect an exact string match.
"""
object_key = input_file_id.rsplit("/", 1)[-1]
bare_model_name = cls._get_bare_model_name_from_s3_key(object_key)
assert bare_model_name is not None # narrowed by is_unmanaged_s3_batch_input_file_id
return bare_model_name
def validate_environment(
self,
headers: dict,

24
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -33,6 +33,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
make_valid_bedrock_tool_name,
)
from litellm.llms.anthropic.chat.transformation import (
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT,
AnthropicConfig,
@ -899,7 +900,28 @@ class AmazonConverseConfig(BaseConfig):
"tool_choice": {"disable_parallel_tool_use": disable_parallel}
}
if param == "thinking":
optional_params["thinking"] = value
if (
isinstance(value, dict)
and value.get("type") == "adaptive"
and not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock")
):
max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
legacy_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort="medium",
model=model,
custom_llm_provider="bedrock",
)
capped = (
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
if capped is not None:
optional_params["thinking"] = capped
else:
litellm.verbose_logger.warning(DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING, model)
else:
optional_params["thinking"] = value
elif param == "reasoning_effort" and isinstance(value, str):
self._handle_reasoning_effort_parameter(
model=model, reasoning_effort=value, optional_params=optional_params

46
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -1879,6 +1879,7 @@ class BaseLLMHTTPHandler:
litellm_params: GenericLiteLLMParams,
api_key: Optional[str],
model: str,
timeout: Optional[Union[float, httpx.Timeout]] = None,
) -> httpx.Response:
max_attempts = max(provider_config.max_retry_on_anthropic_messages_http_error, 1)
litellm_params_dict = dict(litellm_params)
@ -1891,6 +1892,7 @@ class BaseLLMHTTPHandler:
data=signed_json_body or json.dumps(request_body),
stream=stream or False,
logging_obj=logging_obj,
timeout=timeout,
)
response.raise_for_status()
return response
@ -1925,6 +1927,32 @@ class BaseLLMHTTPHandler:
raise RuntimeError("unreachable: anthropic messages HTTP retry loop exited without return")
@staticmethod
def _resolve_anthropic_messages_timeout(
litellm_params: GenericLiteLLMParams,
stream: bool,
custom_llm_provider: str,
) -> Optional[Union[float, httpx.Timeout]]:
from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
from litellm.utils import supports_httpx_timeout
stream_timeout = litellm_params.get("stream_timeout") if stream else None
model_timeout = stream_timeout if stream_timeout is not None else litellm_params.get("timeout")
request_timeout = litellm_params.get("request_timeout")
global_timeout = get_configured_request_timeout()
if model_timeout is None and request_timeout is None and global_timeout is None:
return None
return CompletionTimeout.resolve(
model_timeout,
{"request_timeout": request_timeout},
custom_llm_provider,
global_timeout=global_timeout,
supports_httpx_timeout=supports_httpx_timeout,
)
async def async_anthropic_messages_handler(
self,
model: str,
@ -2075,6 +2103,11 @@ class BaseLLMHTTPHandler:
litellm_params=litellm_params,
api_key=api_key,
model=model,
timeout=self._resolve_anthropic_messages_timeout(
litellm_params=litellm_params,
stream=stream or False,
custom_llm_provider=custom_llm_provider,
),
)
# used for logging + cost tracking
@ -2657,9 +2690,10 @@ class BaseLLMHTTPHandler:
)
result = final_response if final_response is not None else initial_response
if litellm_params.get("_code_interpreter_interception_converted_stream") and not litellm_params.get(
"_agentic_loop_depth"
):
interception_converted_stream = litellm_params.get(
"_code_interpreter_interception_converted_stream"
) or litellm_params.get("_websearch_interception_converted_stream")
if interception_converted_stream and not litellm_params.get("_agentic_loop_depth"):
return self._wrap_responses_response_as_fake_stream(
result=result,
model=model,
@ -5224,6 +5258,8 @@ class BaseLLMHTTPHandler:
tools = anthropic_messages_optional_request_params.get("tools", [])
depth, max_loops, fingerprints = self._get_agentic_loop_settings(kwargs=kwargs)
hook_kwargs = {**kwargs, "_agentic_loop_api_surface": api_surface}
for callback in callbacks:
if not isinstance(callback, CustomLogger):
continue
@ -5244,7 +5280,7 @@ class BaseLLMHTTPHandler:
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs=hook_kwargs,
)
except Exception as e:
_call_id = getattr(logging_obj, "litellm_call_id", "unknown")
@ -5270,7 +5306,7 @@ class BaseLLMHTTPHandler:
)
try:
kwargs_with_provider = kwargs.copy() if kwargs else {}
kwargs_with_provider = hook_kwargs.copy()
kwargs_with_provider["custom_llm_provider"] = custom_llm_provider
build_plan_overridden = (
callback.__class__.async_build_agentic_loop_plan is not CustomLogger.async_build_agentic_loop_plan

207
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -14,11 +14,14 @@ Pattern Overview:
This pattern can be replicated for other message formats (e.g., Anthropic).
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union, cast
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.llms.base_llm.guardrail_translation.base_translation import (
BaseTranslation,
StreamTransformSink,
)
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_skip_system_message_for_guardrail,
effective_skip_tool_message_for_guardrail,
@ -27,6 +30,9 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
)
from litellm.main import stream_chunk_builder
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
from litellm.types.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
coerce_stream_holdback_value,
)
from litellm.types.utils import (
Choices,
GenericGuardrailAPIInputs,
@ -50,7 +56,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Methods can be overridden to customize behavior for different message formats.
"""
def get_structured_messages(self, data: dict) -> Optional[List[AllMessageValues]]:
def get_structured_messages(self, data: dict) -> List[AllMessageValues] | None:
"""
Convert chat completions request data to OpenAI-spec structured messages.
@ -65,7 +71,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
data: dict,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
litellm_logging_obj: Any | None = None,
) -> Any:
"""
Process input messages by applying guardrails to text content.
@ -80,7 +86,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str] = []
images_to_check: List[str] = []
tool_calls_to_check: List[ChatCompletionToolParam] = []
text_task_mappings: List[Tuple[int, Optional[int]]] = []
text_task_mappings: List[Tuple[int, int | None]] = []
tool_call_task_mappings: List[Tuple[int, int]] = []
# Step 1: Extract all text content, images, and tool calls
@ -184,7 +190,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str],
images_to_check: List[str],
tool_calls_to_check: List[ChatCompletionToolParam],
text_task_mappings: List[Tuple[int, Optional[int]]],
text_task_mappings: List[Tuple[int, int | None]],
tool_call_task_mappings: List[Tuple[int, int]],
skip_system_message: bool = False,
skip_tool_message: bool = False,
@ -239,7 +245,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
messages: List[Dict[str, Any]],
responses: List[str],
task_mappings: List[Tuple[int, Optional[int]]],
task_mappings: List[Tuple[int, int | None]],
) -> None:
"""
Apply guardrail responses back to input message text content.
@ -249,7 +255,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for task_idx, guardrail_response in enumerate(responses):
mapping = task_mappings[task_idx]
msg_idx = cast(int, mapping[0])
content_idx_optional = cast(Optional[int], mapping[1])
content_idx_optional = cast(int | None, mapping[1])
# Handle content
content = messages[msg_idx].get("content", None)
@ -291,9 +297,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
response: "ModelResponse",
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
request_data: Optional[dict] = None,
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
request_data: dict | None = None,
) -> Any:
"""
Process output response by applying guardrails to text content.
@ -320,7 +326,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str] = []
images_to_check: List[str] = []
tool_calls_to_check: List[Dict[str, Any]] = []
text_task_mappings: List[Tuple[int, Optional[int]]] = []
text_task_mappings: List[Tuple[int, int | None]] = []
tool_call_task_mappings: List[Tuple[int, int]] = []
# text_task_mappings: Track (choice_index, content_index) for each text
# content_index is None for string content, int for list content
@ -402,9 +408,10 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
responses_so_far: List["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
request_data: Optional[dict] = None,
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
request_data: dict | None = None,
stream_transform_sink: StreamTransformSink | None = None,
) -> List["ModelResponseStream"]:
"""
Process output streaming responses by applying guardrails to text content.
@ -414,14 +421,50 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
user_api_key_dict: User API key metadata to pass to guardrails
stream_transform_sink: Optional out-parameter for the streaming text
transformation path. When provided, the guardrail runs over the raw
accumulated text (``responses_so_far`` is left untouched so it stays
a correct raw accumulator across rounds) and the guardrailed text
plus requested holdback are reported per choice on the sink.
Returns:
Modified list of responses with guardrail applied to content
The (unmodified) list of responses.
Response Format Support:
- String content: choice.message.content = "text here"
- List content: choice.message.content = [{"type": "text", "text": "text here"}, ...]
"""
if stream_transform_sink is not None:
await self._process_streaming_transform(
responses_so_far=responses_so_far,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=litellm_logging_obj,
user_api_key_dict=user_api_key_dict,
request_data=request_data,
sink=stream_transform_sink,
)
return responses_so_far
return await self._process_streaming_block_only(
responses_so_far=responses_so_far,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=litellm_logging_obj,
user_api_key_dict=user_api_key_dict,
request_data=request_data,
)
async def _process_streaming_block_only(
self,
*,
responses_so_far: list["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None,
user_api_key_dict: Any | None,
request_data: dict | None,
) -> list["ModelResponseStream"]:
"""Block-only streaming path: run the guardrail so an in-flight BLOCK can
terminate the stream. Text rewrites are not propagated to the client here
(see ``_process_streaming_transform`` for the incremental_diff path)."""
# check if the stream has ended
has_stream_ended = False
for chunk in responses_so_far:
@ -467,7 +510,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# Step 2: Create lists for guardrail processing
texts_to_check: List[str] = []
images_to_check: List[str] = []
task_mappings: List[Tuple[int, Optional[int]]] = []
task_mappings: List[Tuple[int, int | None]] = []
# Track (choice_index, content_index) for each combined text
for (map_choice_idx, map_content_idx), combined_text in combined_texts.items():
@ -520,9 +563,109 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
return responses_so_far
@staticmethod
def _accumulate_string_content_by_choice_index(
responses_so_far: list["ModelResponseStream"],
) -> dict[int, str]:
"""Accumulate raw string ``delta.content`` per choice, keyed by
``StreamingChoices.index`` (not enumerate position, which collapses to 0
when each chunk carries a single non-zero-indexed choice for ``n > 1``).
Only string content participates; list-of-blocks content is out of scope
for the incremental transform path. Reads ``responses_so_far`` without
mutating it so it stays a correct raw accumulator across rounds.
"""
accumulated: dict[int, str] = {}
for response in responses_so_far:
for choice in response.choices:
if isinstance(choice, litellm.StreamingChoices):
content = choice.delta.content
elif isinstance(choice, litellm.Choices):
content = choice.message.content
else:
continue
if isinstance(content, str) and content:
idx = getattr(choice, "index", 0) or 0
accumulated[idx] = accumulated.get(idx, "") + content
return accumulated
async def _process_streaming_transform(
self,
*,
responses_so_far: list["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None,
user_api_key_dict: Any | None,
request_data: dict | None,
sink: StreamTransformSink,
) -> None:
"""Run the guardrail over the raw accumulated text and report the
guardrailed text plus requested holdback per choice on ``sink``.
Unlike the block-only path this never mutates ``responses_so_far``: it
re-derives the raw accumulated text every round (so a rewrite guardrail
always sees consistent input) and hands the result back out of band.
"""
raw_by_index = self._accumulate_string_content_by_choice_index(responses_so_far)
if not raw_by_index:
sink.mutated_text_per_choice = {}
sink.holdback_per_choice = {}
return
# Fix #2 — sort by StreamingChoices.index so an n>1 stream that emits
# choice 1 before choice 0 still hands the guardrail texts in a
# deterministic index order. Without this, the guardrail's returned
# texts (aligned to the input order it received) would map back to the
# wrong choice indices when we rebuild the sink dicts by
# ``enumerate(indices)``.
indices = sorted(raw_by_index.keys())
texts_to_check = [raw_by_index[i] for i in indices]
if request_data is None:
request_data = {"responses": responses_so_far}
elif "responses" not in request_data:
request_data["responses"] = responses_so_far
if "litellm_metadata" not in request_data:
user_metadata = self.transform_user_api_key_dict_to_metadata(user_api_key_dict)
if user_metadata:
request_data["litellm_metadata"] = user_metadata
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
if responses_so_far and getattr(responses_so_far[0], "model", None):
inputs["model"] = responses_so_far[0].model
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=request_data,
input_type="response",
logging_obj=litellm_logging_obj,
)
returned_texts = guardrailed_inputs.get("texts")
# No "texts" key means the guardrail made no change (action NONE): the raw
# accumulated text is the guardrailed text. A present-but-shorter list is a
# guardrail contract violation; those choices are omitted below (withheld,
# not emitted raw) so a malformed response fails closed instead of leaking.
if returned_texts is None:
returned_texts = texts_to_check
elif len(returned_texts) < len(texts_to_check):
verbose_proxy_logger.warning(
"OpenAI Chat Completions: guardrail returned %s transformed texts for %s inputs on the "
"streaming transform path; withholding the unmatched choices to fail closed.",
len(returned_texts),
len(texts_to_check),
)
holdback = guardrailed_inputs.get("stream_holdback_chars") or []
sink.mutated_text_per_choice = {
idx: returned_texts[i] for i, idx in enumerate(indices) if i < len(returned_texts)
}
sink.holdback_per_choice = {
indices[i]: coerce_stream_holdback_value(holdback[i]) for i in range(len(indices)) if i < len(holdback)
}
def _combine_streaming_texts(
self, responses_so_far: List["ModelResponseStream"]
) -> Dict[Tuple[int, Optional[int]], str]:
) -> Dict[Tuple[int, int | None], str]:
"""
Combine all streaming chunks into complete text per choice.
@ -534,7 +677,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Returns:
Dict mapping (choice_idx, content_idx) to combined text string
"""
combined_texts: Dict[Tuple[int, Optional[int]], str] = {}
combined_texts: Dict[Tuple[int, int | None], str] = {}
for response_idx, response in enumerate(responses_so_far):
for choice_idx, choice in enumerate(response.choices):
@ -550,7 +693,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if isinstance(content, str):
# String content - accumulate for this choice
str_key: Tuple[int, Optional[int]] = (choice_idx, None)
str_key: Tuple[int, int | None] = (choice_idx, None)
if str_key not in combined_texts:
combined_texts[str_key] = ""
combined_texts[str_key] += content
@ -560,7 +703,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for content_idx, content_item in enumerate(content):
text_str = content_item.get("text")
if text_str:
list_key: Tuple[int, Optional[int]] = (
list_key: Tuple[int, int | None] = (
choice_idx,
content_idx,
)
@ -607,7 +750,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str],
images_to_check: List[str],
tool_calls_to_check: List[Dict[str, Any]],
text_task_mappings: List[Tuple[int, Optional[int]]],
text_task_mappings: List[Tuple[int, int | None]],
tool_call_task_mappings: List[Tuple[int, int]],
) -> None:
"""
@ -619,7 +762,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# Determine content source and tool calls based on choice type
content = None
tool_calls: Optional[List[Any]] = None
tool_calls: List[Any] | None = None
if isinstance(choice, litellm.Choices):
content = choice.message.content
tool_calls = choice.message.tool_calls
@ -662,7 +805,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls_to_check.append(tool_call_dict)
tool_call_task_mappings.append((choice_idx, int(tool_call_idx)))
def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Optional[Dict[str, Any]]:
def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Dict[str, Any] | None:
"""
Convert a tool call object to dictionary format.
@ -691,7 +834,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
response: "ModelResponse",
responses: List[str],
task_mappings: List[Tuple[int, Optional[int]]],
task_mappings: List[Tuple[int, int | None]],
) -> None:
"""
Apply guardrail text responses back to output response.
@ -701,7 +844,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for task_idx, guardrail_response in enumerate(responses):
mapping = task_mappings[task_idx]
choice_idx = cast(int, mapping[0])
content_idx_optional = cast(Optional[int], mapping[1])
content_idx_optional = cast(int | None, mapping[1])
choice = cast(Choices, response.choices[choice_idx])
@ -755,7 +898,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
responses: List["ModelResponseStream"],
guardrailed_texts: List[str],
task_mappings: List[Tuple[int, Optional[int]]],
task_mappings: List[Tuple[int, int | None]],
) -> None:
"""
Apply guardrail responses back to output streaming responses.
@ -771,16 +914,16 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Override this method to customize how responses are applied to streaming responses.
"""
# Build a mapping of what guardrailed text to use for each (choice_idx, content_idx)
guardrail_map: Dict[Tuple[int, Optional[int]], str] = {}
guardrail_map: Dict[Tuple[int, int | None], str] = {}
for task_idx, guardrail_response in enumerate(guardrailed_texts):
mapping = task_mappings[task_idx]
choice_idx = cast(int, mapping[0])
content_idx_optional = cast(Optional[int], mapping[1])
content_idx_optional = cast(int | None, mapping[1])
guardrail_map[(choice_idx, content_idx_optional)] = guardrail_response
# Track which choices we've already set the guardrailed text for
# Key: (choice_idx, content_idx), Value: boolean (True if already set)
already_set: Dict[Tuple[int, Optional[int]], bool] = {}
already_set: Dict[Tuple[int, int | None], bool] = {}
# Iterate through all responses and update content
for response_idx, response in enumerate(responses):
@ -797,7 +940,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if isinstance(content, str):
# String content
str_key: Tuple[int, Optional[int]] = (choice_idx_in_response, None)
str_key: Tuple[int, int | None] = (choice_idx_in_response, None)
if str_key in guardrail_map:
if str_key not in already_set:
# First chunk - set the complete guardrailed text
@ -817,7 +960,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# List content - handle each content item
for content_idx, content_item in enumerate(content):
if "text" in content_item:
list_key: Tuple[int, Optional[int]] = (
list_key: Tuple[int, int | None] = (
choice_idx_in_response,
content_idx,
)

4
litellm/llms/openai/responses/guardrail_translation/handler.py
View file

@ -247,6 +247,8 @@ class OpenAIResponsesHandler(BaseTranslation):
"""
Merge remapped guardrailed tools with original tools that were not sent
to the guardrail (e.g. web_search, web_search_preview), preserving order.
Tools a guardrail appended (``remapped`` longer than ``original_tools``)
have no original slot and are kept so an injected tool is not dropped.
"""
if not original_tools:
return remapped
@ -262,6 +264,8 @@ class OpenAIResponsesHandler(BaseTranslation):
if j < len(remapped):
result.append(remapped[j])
j += 1
# Keep guardrail-appended tools that matched no original slot above.
result.extend(remapped[j:])
return result
def _apply_guardrailed_tools_to_data(

18
litellm/llms/openai/responses/transformation.py
View file

@ -20,6 +20,8 @@ from litellm.types.utils import LlmProviders
from ..common_utils import OpenAIError
OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS = 16
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -59,6 +61,19 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
key="supports_none_reasoning_effort",
)
@staticmethod
def _enforce_min_max_output_tokens(max_output_tokens: "int | None") -> "int | None":
"""Raise sub-minimum max_output_tokens up to the OpenAI Responses API minimum.
OpenAI's Responses API rejects max_output_tokens below 16 for every model
(not gpt-5 specific), so a client like Claude Code that sends a max_tokens=1
warmup probe on model switch would otherwise 400. Values that are None or
already at/above the minimum are returned unchanged.
"""
if isinstance(max_output_tokens, int) and max_output_tokens < OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS:
return OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS
return max_output_tokens
def get_supported_openai_params(self, model: str) -> list:
"""
All OpenAI Responses API params are supported
@ -92,6 +107,9 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
"""
params = dict(response_api_optional_params)
if "max_output_tokens" in params:
params["max_output_tokens"] = self._enforce_min_max_output_tokens(params.get("max_output_tokens"))
if self._is_gpt_5_model(model=model):
temperature = params.get("temperature")
if temperature is not None and temperature != 1:

2
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -998,6 +998,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
response_modalities.append("IMAGE")
elif modality == "audio":
response_modalities.append("AUDIO")
elif modality == "video":
response_modalities.append("VIDEO")
else:
response_modalities.append("MODALITY_UNSPECIFIED")
return response_modalities

7
litellm/main.py
View file

@ -92,7 +92,10 @@ from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
from litellm.litellm_core_utils.get_litellm_params import OPTIONAL_KWARGS_KEYS
from litellm.litellm_core_utils.get_litellm_params import (
AWS_CREDENTIAL_KWARGS_KEYS,
OPTIONAL_KWARGS_KEYS,
)
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.get_provider_specific_headers import (
ProviderSpecificHeaderUtils,
@ -5322,7 +5325,7 @@ def completion( # type: ignore
tpm=kwargs.get("tpm"),
rpm=kwargs.get("rpm"),
use_xai_oauth=kwargs.get("use_xai_oauth", False),
aws_bedrock_project_id=kwargs.get("aws_bedrock_project_id"),
**{key: kwargs[key] for key in AWS_CREDENTIAL_KWARGS_KEYS if key in kwargs},
)
cast(LiteLLMLoggingObj, logging).update_environment_variables(
model=model,

285
litellm/model_prices_and_context_window_backup.json
View file

@ -11331,6 +11331,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true
},
@ -11362,6 +11363,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
@ -11424,6 +11426,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_tool_choice": true,
"supports_vision": true,
@ -11479,6 +11482,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true
},
@ -11506,6 +11510,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true
},
@ -11559,6 +11564,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_output_config": true
@ -11586,6 +11592,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_output_config": true
@ -11614,6 +11621,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true,
"provider_specific_entry": {
@ -11648,6 +11656,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_tool_choice": true,
"supports_vision": true,
"provider_specific_entry": {
@ -11682,6 +11691,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
@ -11718,6 +11728,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
@ -11788,6 +11799,7 @@
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_native_structured_output": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
@ -18747,6 +18759,49 @@
},
"supports_image_size": false
},
"gemini/gemini-3-pro-image": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "image_generation",
"output_cost_per_image": 0.134,
"output_cost_per_image_token": 0.00012,
"output_cost_per_token": 1.2e-05,
"rpm": 1000,
"tpm": 4000000,
"output_cost_per_token_batches": 6e-06,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3-pro-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
"supports_web_search": true,
"search_context_cost_per_query": {
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3-pro-image-preview": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -18778,6 +18833,49 @@
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
"supports_web_search": true,
"search_context_cost_per_query": {
"search_context_size_low": 0.014,
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.1-flash-image": {
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "image_generation",
"output_cost_per_image": 0.045,
"output_cost_per_image_token": 6e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"rpm": 1000,
"tpm": 4000000,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
@ -18819,6 +18917,7 @@
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
@ -19513,6 +19612,39 @@
},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-omni-flash-preview": {
"input_cost_per_audio_token": 1.5e-06,
"input_cost_per_token": 1.5e-06,
"litellm_provider": "gemini",
"max_input_tokens": 1048576,
"max_output_tokens": 65535,
"max_tokens": 65535,
"mode": "chat",
"output_cost_per_reasoning_token": 9e-06,
"output_cost_per_token": 9e-06,
"output_cost_per_video_token": 1.75e-05,
"rpm": 2000,
"source": "https://ai.google.dev/gemini-api/docs/pricing",
"supported_endpoints": [
"/v1/chat/completions"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text",
"video"
],
"supports_audio_input": true,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true,
"tpm": 800000
},
"gemini/gemini-3.1-pro-preview": {
"cache_read_input_token_cost": 2e-07,
"cache_read_input_token_cost_above_200k_tokens": 4e-07,
@ -19677,6 +19809,37 @@
},
"web_search_billing_unit": "per_query"
},
"gemini-omni-flash-preview": {
"input_cost_per_audio_token": 1.5e-06,
"input_cost_per_token": 1.5e-06,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 1048576,
"max_output_tokens": 65535,
"max_tokens": 65535,
"mode": "chat",
"output_cost_per_reasoning_token": 9e-06,
"output_cost_per_token": 9e-06,
"output_cost_per_video_token": 1.75e-05,
"source": "https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/omni-flash-preview",
"supported_endpoints": [
"/v1/chat/completions"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text",
"video"
],
"supports_audio_input": true,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true
},
"gemini-3.5-flash": {
"cache_read_input_token_cost": 1.5e-07,
"input_cost_per_audio_token": 1e-06,
@ -36777,6 +36940,7 @@
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
@ -36786,6 +36950,22 @@
"tpm": 8000000,
"supports_image_size": false
},
"vertex_ai/gemini-3-pro-image": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "image_generation",
"output_cost_per_image": 0.134,
"output_cost_per_image_token": 0.00012,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_reasoning": false,
"source": "https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-pro-image"
},
"vertex_ai/gemini-3-pro-image-preview": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -36799,8 +36979,23 @@
"output_cost_per_image_token": 0.00012,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_reasoning": false,
"source": "https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-pro-image"
},
"vertex_ai/gemini-3.1-flash-image": {
"input_cost_per_image": 0.00056,
"input_cost_per_token": 5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "image_generation",
"output_cost_per_image": 0.0672,
"output_cost_per_image_token": 6e-05,
"output_cost_per_token": 3e-06,
"supports_reasoning": false,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-image-preview": {
"input_cost_per_image": 0.00056,
"input_cost_per_token": 5e-07,
@ -36812,6 +37007,7 @@
"output_cost_per_image": 0.0672,
"output_cost_per_image_token": 6e-05,
"output_cost_per_token": 3e-06,
"supports_reasoning": false,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-lite-preview": {
@ -44065,6 +44261,90 @@
"supports_response_schema": true,
"supports_tool_choice": true
},
"bedrock_mantle/openai.gpt-5.6-sol": {
"input_cost_per_token": 5.5e-06,
"cache_creation_input_token_cost": 6.875e-06,
"cache_read_input_token_cost": 5.5e-07,
"output_cost_per_token": 3.3e-05,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": [
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/openai.gpt-5.6-terra": {
"input_cost_per_token": 2.75e-06,
"cache_creation_input_token_cost": 3.4375e-06,
"cache_read_input_token_cost": 2.75e-07,
"output_cost_per_token": 1.65e-05,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": [
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/openai.gpt-5.6-luna": {
"input_cost_per_token": 1.1e-06,
"cache_creation_input_token_cost": 1.375e-06,
"cache_read_input_token_cost": 1.1e-07,
"output_cost_per_token": 6.6e-06,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": [
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/openai.gpt-5.5": {
"input_cost_per_token": 5.5e-06,
"cache_read_input_token_cost": 5.5e-07,
@ -44177,6 +44457,7 @@
"supports_vision": true
},
"bedrock_mantle/xai.grok-4.3": {
"use_openai_responses_path": true,
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 2.5e-06,
"cache_read_input_token_cost": 2e-07,
@ -45051,8 +45332,8 @@
"rules": [
{
"name": "bedrock-claude-ids",
"pattern": "anthropic\\.claude-",
"description": "Any Bedrock-syntax Claude id: the dotted anthropic.claude- segment appears in bare (anthropic.claude-...), region-prefixed (us./eu./au./jp./apac.) and global.-prefixed ids, for every version. Routes these to bedrock before the bare-id Anthropic rule is consulted.",
"pattern": "^(?:[a-z-]+\\.)?anthropic\\.claude-",
"description": "A Bedrock-syntax Claude id, for every version: anthropic.claude- at the start of the name, optionally behind a single dotted geo segment (us./eu./au./jp./apac./global./us-gov.). Anchored to the start because routing rules see the raw request string and provider inference feeds the proxy's provider/* wildcard access checks: an id under an unrecognized namespace such as bedrockz/anthropic.claude-... must stay unroutable rather than resolve to bedrock and slip through a bedrock/* key. Routes to bedrock before the bare-id Anthropic rule is consulted.",
"model_info": {
"litellm_provider": "bedrock"
}

1
litellm/models/verification_token.py
View file

@ -36,6 +36,7 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
budget_reset_at: Optional[datetime] = None
allowed_cache_controls: Optional[list] = []
allowed_routes: Optional[list] = []
key_type: str | None = None
permissions: Dict = {}
model_spend: Dict = {}
model_max_budget: Dict = {}

96
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -18,6 +18,9 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credenti
is_bridge_envelope_shaped,
resolve_bridge_envelope,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
EnvelopeIdentity,
)
from litellm.proxy._types import (
UI_TEAM_ID,
LiteLLM_TeamTable,
@ -543,7 +546,7 @@ class MCPRequestHandler:
header_key = server.alias or server.server_name
if header_key is None:
raise HTTPException(status_code=500, detail="Server misconfigured: MCP server has no routable name")
admitted = await MCPRequestHandler._reload_admitted_key(result.identity.key_hash)
admitted = await MCPRequestHandler._reload_admitted_principal(result.identity)
await MCPRequestHandler._enforce_admitted_live_policy(admitted=admitted, request=request, route=route)
injected = {header_key: {"Authorization": result.upstream_authorization.get_secret_value()}}
new_headers = {**(mcp_server_auth_headers or {}), **injected}
@ -572,6 +575,89 @@ class MCPRequestHandler:
route=route,
)
@staticmethod
async def _reload_admitted_principal(identity: EnvelopeIdentity) -> UserAPIKeyAuth:
"""Reload the live litellm record the envelope's subject references.
Dispatches on the sealed subject type: a ``key_hash`` reloads the virtual key that
minted the envelope (the scripted two-header client that presents a litellm key at the
token endpoint), a ``user_id`` reloads the user that authenticated interactively (the
DCR client, whose SSO login at the bridged authorize yields a user, not a key). Both
return a ``UserAPIKeyAuth`` the caller runs through the centralized policy gate, so
team/project/org/budget/SCIM enforcement is identical to the principal presenting
itself directly."""
match identity.subject_type:
case "key_hash":
return await MCPRequestHandler._reload_admitted_key(identity.subject)
case "user_id":
return await MCPRequestHandler._reload_admitted_user(identity.subject)
case _:
assert_never(identity.subject_type)
@staticmethod
async def _reload_admitted_user(user_id: str) -> UserAPIKeyAuth:
"""Reload the live user an interactively-minted envelope references and admit them as
themselves.
The DCR client authenticates via SSO at the bridged authorize, which yields a user
subject rather than a virtual key, so the envelope admits under the user's own
identity: the reloaded ``user_id`` and the user's own MCP object permission ride on the
returned ``UserAPIKeyAuth``, and the SAME ``get_allowed_mcp_servers`` the key path uses then
computes which servers the user may reach, so the user's litellm MCP grants and access groups
gate the request exactly as a key's do. Only the user's OWN object permission is bound: a
``UserAPIKeyAuth`` carries a single ``team_id`` while a user may belong to many teams, so
team-inherited MCP grants for a user are a follow-up (they need a many-teams union
``get_allowed_mcp_servers`` does not do off one auth object). The caller's centralized policy
gate enforces the user's live budget and org state, and a SCIM-deactivated owner fails closed.
Error handling mirrors the key path's retryable-503 contract, but ``get_user_object`` defeats a
type-based check: where ``get_key_object`` raises a typed ``ProxyException`` for a missing key
and lets a DB outage propagate raw, ``get_user_object`` catches every DB failure and re-raises a
bare ``ValueError``, so a missing user and a real outage look identical and the original error
survives only as ``__context__``. ``_raise_503_if_db_unavailable`` therefore walks the cause
chain: a transient DB outage still surfaces as a retryable 503, while a missing user, or any
other non-outage resolution failure, fails closed as a 401 rather than an opaque 500. The
object-permission load shares this one boundary, so an outage there is classified the same
way (``get_object_permission`` itself swallows a failed load to ``None``, matching how
``get_key_object`` best-effort-loads a key's object permission)."""
from litellm.proxy.auth.auth_checks import get_object_permission, get_user_object
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
if prisma_client is None:
raise HTTPException(status_code=500, detail="Server misconfigured: no database connection")
try:
user_object = await get_user_object(
user_id=user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
# Resolve the user's own MCP object permission (get_user_object does not load it) so the shared
# get_allowed_mcp_servers can grant the user their litellm-granted servers. Reuses the same
# get_object_permission resolver the key and team paths use; no permission logic is duplicated.
object_permission = user_object.object_permission if user_object is not None else None
if user_object is not None and object_permission is None and user_object.object_permission_id:
object_permission = await get_object_permission(
object_permission_id=user_object.object_permission_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
except (ProxyException, HTTPException):
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
except Exception as e: # noqa: BLE001 # a DB outage anywhere in the resolution is a retryable 503, not an opaque 500; anything else fails closed as 401
MCPRequestHandler._raise_503_if_db_unavailable(e)
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
if user_object is None:
raise HTTPException(status_code=401, detail="Invalid or expired credential")
if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
raise HTTPException(status_code=401, detail="Invalid or expired credential")
return UserAPIKeyAuth(
user_id=user_object.user_id,
user_role=user_object.user_role,
object_permission=object_permission,
object_permission_id=user_object.object_permission_id,
)
@staticmethod
async def _reload_admitted_key(key_hash: str) -> UserAPIKeyAuth:
"""Reload the live key record an admitted envelope references and re-check live policy.
@ -615,10 +701,14 @@ class MCPRequestHandler:
"""Raise a retryable 503 when ``e`` means the auth database is unreachable, else return so the
caller applies its own fail-closed mapping. A DB outage must not masquerade as an auth failure
(401) or surface as an opaque 500; the caller retries. Mirrors ``UserAPIKeyAuthExceptionHandler``,
which renders a service-unavailable database error as 503 on the standard pipeline."""
which renders a service-unavailable database error as 503 on the standard pipeline.
Classifies across the ``__cause__``/``__context__`` chain, not just ``e`` itself: ``get_user_object``
re-raises every DB failure as a bare ``ValueError``, so a type-based check on the top exception
would miss a real outage wrapped inside it."""
from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler
if PrismaDBExceptionHandler.is_database_service_unavailable_error(e):
if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(e):
raise HTTPException(
status_code=503,
detail="Service Unavailable, the authentication database is temporarily unreachable. Please retry shortly.",

694
litellm/proxy/_experimental/mcp_server/bridge_token_flow.py Normal file
View file

@ -0,0 +1,694 @@
"""Bridge token flow: litellm identity resolution and the DCR-bridge oauth_delegate mint/refresh pipeline."""
import math
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Literal, Optional
from fastapi import HTTPException, Request
from fastapi.responses import JSONResponse
from pydantic import SecretStr
from typing_extensions import assert_never
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.oauth_utils import TOKEN_NO_CACHE_HEADERS
from litellm.types.mcp_server.mcp_server_manager import MCPServer
if TYPE_CHECKING:
from litellm.proxy._experimental.mcp_server.discoverable_endpoints import _BridgeAuthorizationCode
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
EnvelopeIdentity,
EnvelopeKeys,
RefreshCredential,
UpstreamTokenGrant,
)
from litellm.proxy._types import UserAPIKeyAuth
def _litellm_key_from_request(request: Request) -> Optional[str]:
"""Return the LiteLLM API key presented on the request, or ``None``.
Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
an OAuth/upstream bearer.
"""
for header_value in (
request.headers.get("x-litellm-api-key"),
request.headers.get("Authorization") or request.headers.get("authorization"),
):
if not header_value:
continue
value = header_value.strip()
if value.lower().startswith("bearer "):
value = value[7:].strip()
if value:
return value
return None
def _key_is_active(key_obj: "UserAPIKeyAuth") -> bool:
"""``True`` when the presented key is neither blocked nor past its expiry.
The OAuth token endpoint is unauthenticated, so the presented key is validated here before it is
trusted; a revoked or expired key must not mint a bridge envelope or write a stored credential.
``get_key_object`` resolves a row without these checks (the main ``user_api_key_auth`` pipeline
enforces them downstream, which this endpoint bypasses), so they are applied here. Deleted keys
are already rejected upstream, where ``get_key_object`` raises on a row that no longer exists.
This is an active-state gate only; it deliberately does not require a ``user_id``. A valid
team-scoped or service-account key has no ``user_id`` yet is a legitimate credential, so gating
on ``user_id`` presence would wrongly reject it. Callers that need the user (the per-user token
store) derive it separately via :func:`_active_key_user_id`.
Total by design: ``expires`` is typed ``str | datetime``, and an unparseable string would make
``datetime.fromisoformat`` raise. Since the callers run this outside their key-resolution
``try``, an uncaught parse error would surface as a 500 instead of the endpoint's fail-closed
behavior, so a malformed expiry is treated as inactive (return ``False``) rather than raising.
"""
if key_obj.blocked is True:
return False
expires = key_obj.expires
if expires is not None:
if isinstance(expires, datetime):
expiry = expires
else:
try:
expiry = datetime.fromisoformat(expires)
except (ValueError, TypeError):
return False
if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
expiry = expiry.replace(tzinfo=timezone.utc)
if expiry < datetime.now(timezone.utc):
return False
return True
def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> str | None:
"""The active key's ``user_id``, or ``None`` when the key is blocked/expired or simply has no
``user_id`` (a team-scoped or service-account key). Used only by the per-user token store, which
needs a user to key the stored credential; the bridge mint uses the key hash and does not."""
return key_obj.user_id if _key_is_active(key_obj) else None
@dataclass(frozen=True, slots=True)
class _ResolvedKey:
"""An active litellm key resolved from the token request: its hash (the value ``get_key_object``
and the cache/DB layer key the record by) and the live record."""
key_hash: str
key: "UserAPIKeyAuth"
_KeyResolutionFailure = Literal["no_active_key", "unavailable", "unresolvable"]
"""Why a token request yielded no active litellm key, kept distinct so a caller statuses each truthfully
instead of blaming the client for a gateway problem:
- ``no_active_key``: none was presented, or the presented key is unknown / blocked / expired (the
caller's request is at fault)
- ``unavailable``: the auth database was transiently unreachable while resolving (retryable)
- ``unresolvable``: the gateway cannot resolve identity right now (no DB connection, or an unexpected
error) -- a gateway fault, not the caller's
The classification mirrors admission's ``_reload_admitted_key`` so the mint (ingress) and admission
(egress) never disagree on the status of the same outage."""
async def _resolve_active_litellm_key(request: Request) -> "_ResolvedKey | _KeyResolutionFailure":
"""Resolve the presented litellm key to an active key record, or say precisely why not.
Single resolution path the OAuth token endpoint reuses, resolving authoritatively via
``get_key_object`` (cache first, then DB). The failure is a value, not a bare ``None``, so a caller
can tell "the client sent no usable credential" (a request error) apart from "the gateway could not
check" (an infrastructure error) and status each truthfully; collapsing both to ``None`` is what let
a DB outage read as a 400. A resolved key is still gated by ``_key_is_active``, so a blocked or
expired key is ``no_active_key`` while a valid team-scoped or service-account key (no ``user_id``)
resolves. Classification mirrors admission's ``_reload_admitted_key``: no DB connection is a gateway
fault, a ``ProxyException`` / ``HTTPException`` from ``get_key_object`` is an unknown or invalid key,
a database-service-unavailable error is a retryable outage, and anything else is an unexpected
gateway fault."""
token = _litellm_key_from_request(request)
if not token:
return "no_active_key"
from litellm.proxy._types import hash_token # noqa: PLC0415 # inline import avoids a module-load circular import
return await _reload_active_key_by_hash(hash_token(token))
async def _reload_active_key_by_hash(key_hash: str) -> "_ResolvedKey | _KeyResolutionFailure":
"""Reload the live key record for ``key_hash`` (cache first, then DB) and gate it on active state,
returning the resolved key or a precise failure. Shared by the token request's presented-key
resolution (:func:`_resolve_active_litellm_key`, which hashes the presented key) and the refresh
path (which already holds the hash sealed in the refresh envelope), so both re-validate identity
through one active-key gate and one failure classification. Classification mirrors admission's
``_reload_admitted_key``: no DB connection is a gateway fault, a ``ProxyException`` / ``HTTPException``
from ``get_key_object`` is an unknown or invalid key, a database-service-unavailable error is a
retryable outage, and anything else is an unexpected gateway fault. A blocked or expired key is
``no_active_key``, so a revoked key can neither mint nor refresh a bridge envelope."""
from litellm.proxy._types import (
ProxyException, # noqa: PLC0415 # inline import avoids a module-load circular import
)
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_key_object,
)
from litellm.proxy.db.exception_handler import ( # noqa: PLC0415 # inline import avoids a module-load circular import
PrismaDBExceptionHandler,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return "unresolvable"
try:
key_obj = await get_key_object(
hashed_token=key_hash,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
except (ProxyException, HTTPException):
return "no_active_key"
except Exception as exc: # noqa: BLE001 # classify: a DB outage is retryable, anything else is an opaque gateway fault
if PrismaDBExceptionHandler.is_database_service_unavailable_error(exc):
return "unavailable"
verbose_logger.debug(
"_reload_active_key_by_hash: unexpected key-resolution error (%s)",
type(exc).__name__,
)
return "unresolvable"
if not _key_is_active(key_obj):
return "no_active_key"
return _ResolvedKey(key_hash=key_hash, key=key_obj)
async def _reload_active_user_by_id(user_id: str) -> "_KeyResolutionFailure | None":
"""Re-validate a live litellm user by id, returning ``None`` when the user is active or a precise
failure otherwise. The interactive DCR client authenticates via SSO, so its refresh envelope seals a
user subject; renewing it must re-check the user is still live (present and not SCIM-deactivated) so a
deactivated user cannot keep refreshing, mirroring how admission re-validates the same user subject on
the egress side. No DB connection is a gateway fault (``unresolvable``) and a
database-service-unavailable error is a retryable outage (``unavailable``). Everything else fails
closed as ``no_active_key`` (the caller maps it to invalid_grant): a ``ProxyException`` /
``HTTPException``, a SCIM-deactivated user, and, unlike the key path, a missing user. ``get_user_object``
catches every DB failure and re-raises a bare ``ValueError`` (a deleted user and a real outage look
identical, the original error surviving only as ``__context__``), so the outage check walks the cause
chain, and a missing user falls through to ``no_active_key`` rather than an opaque gateway fault."""
from litellm.proxy._types import (
ProxyException, # noqa: PLC0415 # inline import avoids a module-load circular import
)
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_user_object,
)
from litellm.proxy.db.exception_handler import ( # noqa: PLC0415 # inline import avoids a module-load circular import
PrismaDBExceptionHandler,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return "unresolvable"
try:
user_object = await get_user_object(
user_id=user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
except (ProxyException, HTTPException):
return "no_active_key"
except Exception as exc: # noqa: BLE001 # a DB outage is retryable; a missing user (get_user_object's wrapped ValueError) or any other resolution failure fails closed as no_active_key, never a 500
if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):
return "unavailable"
verbose_logger.debug("_reload_active_user_by_id: user-resolution error (%s)", type(exc).__name__)
return "no_active_key"
if user_object is None:
return "no_active_key"
if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
return "no_active_key"
return None
async def _key_owner_scim_deactivated(key: "UserAPIKeyAuth") -> bool:
"""True only when the key's owning user was explicitly SCIM-deactivated, so a refresh revokes an
offboarded owner's key exactly as admission does via ``_reject_if_admitted_owner_scim_deactivated``.
A key with no owner, a missing owner record, or a failed lookup fails OPEN (returns ``False``),
matching admission and the standard builder: a key may outlive its owner record, and a transient DB
blip must not revoke a live key. Only an explicit ``scim_active`` of ``False`` gates renewal."""
if key.user_id is None:
return False
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_user_object,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return False
try:
owner = await get_user_object(
user_id=key.user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
except Exception as exc: # noqa: BLE001 # fail open: a missing owner (get_user_object's wrapped ValueError) or a DB blip must not revoke a live key
verbose_logger.debug("refresh: key-owner SCIM lookup failed, not revoking (%s)", type(exc).__name__)
return False
return owner is not None and isinstance(owner.metadata, dict) and owner.metadata.get("scim_active") is False
async def _revalidate_active_subject(identity: "EnvelopeIdentity") -> "_KeyResolutionFailure | None":
"""Re-validate that the subject sealed in a refresh envelope is still live, dispatching on its type:
a key_hash reloads the virtual key, a user_id reloads the user. Returns ``None`` when the subject is
active or a precise failure otherwise, so revocation gates renewal for either identity source the same
way admission gates the egress: a blocked or expired key, a SCIM-deactivated key owner (mirroring
admission's owner check, so an offboarded user cannot keep renewing a still-active key), and a
deactivated or deleted user all fail closed to ``no_active_key``."""
match identity.subject_type:
case "key_hash":
reloaded = await _reload_active_key_by_hash(identity.subject)
if not isinstance(reloaded, _ResolvedKey):
return reloaded
if await _key_owner_scim_deactivated(reloaded.key):
return "no_active_key"
return None
case "user_id":
return await _reload_active_user_by_id(identity.subject)
case _:
assert_never(identity.subject_type)
async def _extract_user_id_from_request(request: Request) -> str | None:
"""The litellm ``user_id`` for the token request, so a per-user token is stored under the same
identity the egress later reads it by. Storage is best-effort, so every non-resolved outcome
(including a transient DB outage) collapses to ``None`` here and the caller simply skips the store;
the bridge mint, which must status those outcomes differently, consumes
:func:`_resolve_active_litellm_key` directly."""
resolved = await _resolve_active_litellm_key(request)
if not isinstance(resolved, _ResolvedKey):
return None
return _active_key_user_id(resolved.key)
_UpstreamGrantRejection = Literal["no_access_token", "expired_lifetime"]
"""Why an upstream token response cannot back a bridge envelope:
- ``no_access_token``: the response carries no usable ``access_token``
- ``expired_lifetime``: the response reports a parseable, non-positive ``expires_in``, i.e. an upstream
token that is already dead, so sealing it would forward a bearer the edge cannot use
An absent or unparseable ``expires_in`` is NOT a rejection; the lifetime is merely unknown and the
envelope caps it, the by-design behaviour for an upstream that omits the field."""
def _classify_upstream_lifetime(raw_expires_in: object) -> "int | Literal['unspecified', 'expired']":
"""Classify an upstream ``expires_in`` into a positive number of seconds, ``"unspecified"`` (absent
or unparseable, so the envelope caps it), or ``"expired"`` (a non-positive value the upstream reports
as already elapsed). Telling "we do not know the lifetime" apart from "the upstream says it is
already dead" is what stops an explicitly-expired token from silently receiving the envelope's 1h
cap. The expired decision is made on the parsed numeric value, not on ``int(...)`` of it, so a
positive sub-second lifetime in ``(0, 1)`` is not truncated to ``0`` and misread as elapsed; the
envelope works in whole seconds, so such a lifetime clamps up to its 1s floor. ``bool`` is excluded
(an ``int`` subclass but never a real lifetime), and the conversions can raise on ``NaN`` /
``Infinity`` / oversized input, which reads as unparseable rather than surfacing as a 500."""
if raw_expires_in is None or isinstance(raw_expires_in, bool) or not isinstance(raw_expires_in, (int, float, str)):
return "unspecified"
try:
numeric = float(raw_expires_in)
seconds = int(numeric)
except (ValueError, TypeError, OverflowError):
return "unspecified"
if numeric <= 0:
return "expired"
return max(1, seconds)
def _bridge_grant_from_token_response(token_response: object) -> "UpstreamTokenGrant | _UpstreamGrantRejection":
"""Validate an upstream OAuth token response into a typed grant, or say why it cannot back an
envelope. Each field is isinstance-checked so nothing untyped from ``response.json()`` reaches the
grant. ``expires_in`` is read three ways (see :func:`_classify_upstream_lifetime`): an unknown
lifetime leaves the grant ``expires_in`` ``None`` for the envelope to cap, a positive value is
honoured, and an explicit already-elapsed value is a rejection rather than a silent fall-through to
the cap."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
UpstreamTokenGrant,
)
if not isinstance(token_response, dict):
return "no_access_token"
access = token_response.get("access_token")
if not isinstance(access, str) or not access:
return "no_access_token"
lifetime = _classify_upstream_lifetime(token_response.get("expires_in"))
if lifetime == "expired":
return "expired_lifetime"
token_type = token_response.get("token_type")
scope = token_response.get("scope")
return UpstreamTokenGrant(
access_token=SecretStr(access),
token_type=token_type if isinstance(token_type, str) and token_type else "Bearer",
# The upstream refresh_token is deliberately NOT sealed: the edge never consumes it (it forwards
# only token_type + access_token), so it would be dead weight embedding a long-lived upstream
# credential in the client-held bearer, and it enlarges the envelope. Refresh support is a
# follow-up (a dedicated refresh-envelope); the client re-runs authorization_code at the cap.
refresh_token=None,
scope=scope if isinstance(scope, str) and scope else None,
expires_in=lifetime if isinstance(lifetime, int) else None,
)
# ---------------------------------------------------------------------------
# DCR-bridge oauth_delegate mint: a three-phase pipeline whose failures are values.
#
# prepare (before the upstream exchange) -> validate every precondition and resolve identity+keys
# exchange (the single-use upstream code is consumed here, in exchange_token_with_server)
# finish (after the exchange) -> seal the upstream grant into the client-held envelope
#
# Every precondition lives in ``prepare``, which runs BEFORE the exchange, so no failure can burn the
# single-use code or rotate a refresh token, for either grant type -- that whole class of bug is gone
# by construction rather than guarded case by case. Failures are values mapped to an OAuth-shaped
# response in one place (``_bridge_mint_error_response``), so status codes and the RFC 6749 §5.2 body
# shape are uniform. Adding a failure mode is a new literal plus a match arm the type checker forces.
# ---------------------------------------------------------------------------
_BridgeMintError = Literal[
"no_identity",
"invalid_refresh",
"identity_unavailable",
"identity_unresolvable",
"not_configured",
"no_upstream_token",
"upstream_token_expired",
"too_large",
]
@dataclass(frozen=True, slots=True)
class _BridgeMintReady:
"""Everything the seal needs, resolved once before the exchange: the identity to bind the envelope
to and the master-key-derived envelope keys. The identity is a key_hash subject for the scripted
two-header client (resolved from the litellm key it presents) or a user_id subject for the
interactive SSO client (the user recovered from the gateway authorization code), so one phase-3 seal
serves both. Resolving identity here means ``_finish_bridge_mint`` has no preconditions left to
fail."""
identity: "EnvelopeIdentity"
keys: "EnvelopeKeys"
def _bridge_mint_error_response(error: _BridgeMintError) -> JSONResponse:
"""Map a bridge-mint failure value to its token-endpoint response: one place, RFC 6749 §5.2 shape
(top-level ``error``, no-store headers) for every case, with a status truthful about where the
failure is. The caller's request is 400, a transient gateway outage is 503, a gateway
misconfiguration is 500, and an upstream problem is 502. The identity-resolution statuses match how
admission statuses the same conditions on the egress side, so mint and admit never disagree under
one outage."""
match error:
case "no_identity":
status, code, desc = (
400,
"invalid_request",
"this server issues a gateway-bound credential; complete the interactive sign-in, or "
"send a litellm credential (x-litellm-api-key or Authorization) on the token request",
)
case "invalid_refresh":
status, code, desc = (
400,
"invalid_grant",
"the refresh credential is not a valid, live refresh envelope for this server; "
"re-run authorization_code to obtain a new one",
)
case "identity_unavailable":
status, code, desc = (
503,
"temporarily_unavailable",
"the authentication database is temporarily unreachable; retry shortly",
)
case "identity_unresolvable":
status, code, desc = (
500,
"server_error",
"the gateway could not resolve the litellm identity for this request",
)
case "not_configured":
status, code, desc = (
500,
"server_error",
"the gateway is not configured to mint a gateway-bound credential (master_key is not set)",
)
case "no_upstream_token":
status, code, desc = (
502,
"server_error",
"the upstream token response has no usable access_token",
)
case "upstream_token_expired":
status, code, desc = (
502,
"server_error",
"the upstream token response reports an already-expired lifetime",
)
case "too_large":
status, code, desc = (
502,
"server_error",
"the upstream token is too large to seal into a gateway-bound credential",
)
case _:
assert_never(error)
return JSONResponse(
status_code=status, content={"error": code, "error_description": desc}, headers=TOKEN_NO_CACHE_HEADERS
)
def _key_resolution_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
"""Lift an identity-resolution failure into the mint taxonomy, preserving origin so the status stays
truthful: the caller's missing credential is 400, a transient DB outage is 503, and a gateway that
cannot resolve identity is 500."""
match failure:
case "no_active_key":
return "no_identity"
case "unavailable":
return "identity_unavailable"
case "unresolvable":
return "identity_unresolvable"
case _:
assert_never(failure)
def _upstream_rejection_to_mint_error(rejection: _UpstreamGrantRejection) -> _BridgeMintError:
"""Lift an upstream-response rejection into the mint taxonomy; both are upstream faults (502)."""
match rejection:
case "no_access_token":
return "no_upstream_token"
case "expired_lifetime":
return "upstream_token_expired"
case _:
assert_never(rejection)
async def _prepare_bridge_mint(
request: Request,
mcp_server: MCPServer,
bridge_identity: "_BridgeAuthorizationCode | None" = None,
) -> "_BridgeMintReady | _BridgeMintError":
"""Phase 1 for the authorization_code grant, BEFORE the upstream exchange: confirm the gateway can
mint (master_key set), resolve the litellm identity, and derive the envelope keys. Returns a ready
context or a precise failure value. Running before the exchange is what makes every failure here fail
closed without consuming the single-use code.
Two identity sources, one envelope. The interactive DCR client authenticates via SSO at the bridged
authorize, so its identity arrives as ``bridge_identity`` (the user recovered from the gateway
authorization code) and mints a user subject. The scripted two-header client presents a litellm key
on the token request instead, so its identity is the active key's hash and mints a key_hash subject.
A missing or invalid presented key keeps its resolution origin so the mapper statuses it truthfully;
neither source present is ``no_identity``. The refresh_token grant has its own phase-1
(:func:`_prepare_bridge_refresh`), which recovers identity from the presented refresh envelope."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
envelope_keys_from_master_key,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
key_hash_identity,
user_identity,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
master_key,
)
if not master_key:
return "not_configured"
keys = envelope_keys_from_master_key(master_key)
if bridge_identity is not None:
identity = user_identity(server_id=mcp_server.server_id, user_id=bridge_identity.litellm_user_id)
return _BridgeMintReady(identity=identity, keys=keys)
resolved = await _resolve_active_litellm_key(request)
if not isinstance(resolved, _ResolvedKey):
return _key_resolution_failure_to_mint_error(resolved)
identity = key_hash_identity(server_id=mcp_server.server_id, key_hash=resolved.key_hash)
return _BridgeMintReady(identity=identity, keys=keys)
@dataclass(frozen=True, slots=True)
class _BridgeRefreshReady:
"""A validated refresh request: the identity+keys to mint the renewed pair under, the upstream refresh
token (unwrapped from the client's refresh envelope) to exchange with the upstream IdP, and the scope
sealed alongside it at mint. The upstream refresh token is a ``SecretStr`` like every other credential
in this layer, so a repr or a traceback that captures this value never exposes the raw upstream refresh
token in plaintext. ``upstream_scope`` carries the originally-granted scope so the renewal re-requests
it when the client (a DCR/MCP client that typically omits scope on refresh) sends none, keeping the
renewed token's scope stable against an upstream that would otherwise narrow or drop it."""
ready: "_BridgeMintReady"
upstream_refresh_token: SecretStr
upstream_scope: str | None = None
def _refresh_key_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
"""Lift an identity-resolution failure on the refresh path into the mint taxonomy. Unlike the mint
path, a resolved-but-inactive (or unknown) key is ``invalid_grant`` rather than ``invalid_request``:
the client did present an identity (sealed in the refresh envelope), but it is no longer live, so the
refresh is invalid and the client must re-authenticate. A transient outage is still 503 and a gateway
fault still 500, matching the mint path and admission."""
match failure:
case "no_active_key":
return "invalid_refresh"
case "unavailable":
return "identity_unavailable"
case "unresolvable":
return "identity_unresolvable"
case _:
assert_never(failure)
async def _prepare_bridge_refresh(
mcp_server: MCPServer, refresh_value: str | None
) -> "_BridgeRefreshReady | _BridgeMintError":
"""Phase 1 for the refresh_token grant, BEFORE the upstream exchange: open the client's refresh
envelope, re-validate the sealed litellm identity so a revoked key cannot keep refreshing, and
recover the upstream refresh token to exchange. Identity comes entirely from the sealed envelope, not
the HTTP request, so the request object is not needed here. The client presents a refresh envelope,
never a raw upstream refresh token, so a missing value, a non-envelope, an unopenable envelope, or one
minted for another server is ``invalid_grant``. Running before the exchange means a rejected refresh
never consumes or rotates the upstream refresh token."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
BridgeRefreshOpened,
envelope_keys_from_master_key,
open_bridge_refresh_envelope,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
master_key,
)
if not master_key:
return "not_configured"
if not refresh_value:
return "invalid_refresh"
keys = envelope_keys_from_master_key(master_key)
opened = open_bridge_refresh_envelope(refresh_value, keys, datetime.now(timezone.utc), mcp_server.server_id)
if not isinstance(opened, BridgeRefreshOpened):
return "invalid_refresh"
failure = await _revalidate_active_subject(opened.identity)
if failure is not None:
return _refresh_key_failure_to_mint_error(failure)
return _BridgeRefreshReady(
ready=_BridgeMintReady(identity=opened.identity, keys=keys),
upstream_refresh_token=opened.refresh.refresh_token,
upstream_scope=opened.refresh.scope,
)
def _finish_bridge_mint(
ready: "_BridgeMintReady", mcp_server: MCPServer, token_response: object, now: datetime
) -> "JSONResponse | _BridgeMintError":
"""Phase 3, AFTER the upstream exchange: seal the upstream grant into the client-held access envelope
using the pre-resolved identity and keys, and, when the upstream returned a refresh token, seal a
long-lived refresh envelope alongside it so the client can renew without re-authenticating. Shared by
the authorization_code and refresh_token paths, so a renewal that the upstream rotates re-issues a
fresh refresh envelope. The only hard failures here are properties of the upstream access token (no
usable token, an already-expired lifetime, or a token too large to seal); a refresh token that cannot
be sealed degrades to an access-only response rather than failing the whole exchange."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
build_bridge_token_response,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
SealedEnvelope,
UpstreamTokenGrant,
)
grant = _bridge_grant_from_token_response(token_response)
if not isinstance(grant, UpstreamTokenGrant):
return _upstream_rejection_to_mint_error(grant)
sealed = build_bridge_token_response(ready.identity, grant, ready.keys, now)
if not isinstance(sealed, SealedEnvelope):
return "too_large"
# Report expires_in from the JWT's own second-truncated exp, rounding the elapsed portion up, so the
# client is never told the bearer lives past the point admission (which uses that exp) rejects it.
expires_in = max(0, int(sealed.expires_at.timestamp()) - math.ceil(now.timestamp()))
refresh_envelope = _mint_refresh_envelope_value(ready.identity, token_response, ready.keys, now, mcp_server)
body = {
"access_token": sealed.token.get_secret_value(),
"token_type": "Bearer",
"expires_in": expires_in,
# A refresh envelope rides along only when the upstream returned a refresh token to seal; when it
# rotates on renewal, the client receives the new one and the old envelope's upstream token dies.
**({"refresh_token": refresh_envelope} if refresh_envelope is not None else {}),
}
return JSONResponse(body, headers=TOKEN_NO_CACHE_HEADERS)
def _upstream_refresh_credential(token_response: object) -> "RefreshCredential | None":
"""Extract the upstream refresh grant from a token response, or ``None`` when there is none to seal.
Each field is isinstance-checked so nothing untyped reaches the refresh envelope; ``refresh_expires_in``
(the refresh token's own lifetime, when the upstream reports it) is classified like ``expires_in`` and
bounds the refresh envelope's TTL. An upstream that reports the refresh token itself as already elapsed
(``refresh_expires_in`` non-positive) yields ``None`` rather than a refresh envelope: sealing a dead
token would hand the client a full-TTL-capped envelope the IdP will reject, so the exchange degrades to
an access-only response (the client re-authenticates at access expiry), mirroring how
:func:`_bridge_grant_from_token_response` refuses an already-elapsed access token instead of capping it."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
RefreshCredential,
)
if not isinstance(token_response, dict):
return None
refresh = token_response.get("refresh_token")
if not isinstance(refresh, str) or not refresh:
return None
lifetime = _classify_upstream_lifetime(token_response.get("refresh_expires_in"))
if lifetime == "expired":
return None
scope = token_response.get("scope")
return RefreshCredential(
refresh_token=SecretStr(refresh),
scope=scope if isinstance(scope, str) and scope else None,
expires_in=lifetime if isinstance(lifetime, int) else None,
)
def _mint_refresh_envelope_value(
identity: "EnvelopeIdentity", token_response: object, keys: "EnvelopeKeys", now: datetime, mcp_server: MCPServer
) -> str | None:
"""Seal the upstream refresh grant (if any) into a refresh envelope and return its bearer string, or
``None`` when the upstream returned no refresh token or the refresh token is too large to seal. A
too-large refresh token degrades to an access-only response (logged) rather than failing an exchange
that already succeeded upstream: the client simply re-authenticates when the access envelope expires."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
build_bridge_refresh_token_response,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
SealedEnvelope,
)
refresh_credential = _upstream_refresh_credential(token_response)
if refresh_credential is None:
return None
sealed = build_bridge_refresh_token_response(identity, refresh_credential, keys, now)
if isinstance(sealed, SealedEnvelope):
return sealed.token.get_secret_value()
verbose_logger.warning(
"bridge mint: the upstream refresh token is too large to seal into a refresh envelope for "
"server=%s; issuing an access-only response, so the client re-authenticates at access expiry",
mcp_server.server_id,
)
return None

370
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -10,7 +10,7 @@ from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
import httpx
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
from pydantic import BaseModel, ValidationError
from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationError
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
@ -21,6 +21,24 @@ from litellm.proxy._experimental.mcp_server.auth.token_endpoint_auth import (
TokenEndpointAuthConfigError,
build_token_endpoint_client_auth,
)
from litellm.proxy._experimental.mcp_server.bridge_token_flow import (
_bridge_mint_error_response,
_BridgeMintReady,
_BridgeRefreshReady,
_extract_user_id_from_request,
_finish_bridge_mint,
_prepare_bridge_mint,
_prepare_bridge_refresh,
)
from litellm.proxy._experimental.mcp_server.faults import (
CallerRejected,
CredentialSource,
UpstreamProtocolFault,
classify_upstream_dcr_rejection,
classify_upstream_token_rejection,
dcr_fault_detail,
render_token_fault,
)
from litellm.proxy._experimental.mcp_server.oauth_utils import (
TOKEN_NO_CACHE_HEADERS,
get_request_base_url,
@ -37,7 +55,7 @@ from litellm.types.mcp import MCPAuth, MCPCredentials
from litellm.types.mcp_server.mcp_server_manager import MCPServer
if TYPE_CHECKING:
from litellm.proxy._types import LiteLLM_MCPServerTable, UserAPIKeyAuth
from litellm.proxy._types import LiteLLM_MCPServerTable
# TTL cache for upstream OAuth metadata fetched from pass-through MCP servers.
# Keeps us from hammering the upstream IdP on each discovery request.
@ -91,6 +109,8 @@ def encode_state_with_base_url(
code_challenge: Optional[str] = None,
code_challenge_method: Optional[str] = None,
client_redirect_uri: Optional[str] = None,
litellm_user_id: str | None = None,
mcp_server_id: str | None = None,
) -> str:
"""
Encode the base_url, original state, and PKCE parameters using encryption.
@ -101,6 +121,11 @@ def encode_state_with_base_url(
code_challenge: PKCE code challenge from client
code_challenge_method: PKCE code challenge method from client
client_redirect_uri: Original redirect_uri from client
litellm_user_id: The SSO-authenticated litellm user captured at the bridge authorize
(interactive dcr_bridge oauth_delegate only); the callback seals it into the gateway
authorization code so the token mint can bind the envelope to this user
mcp_server_id: The bridge server the interactive flow targets, sealed alongside
litellm_user_id so the gateway code cannot be replayed against another server
Returns:
An encrypted string that encodes all values
@ -111,6 +136,8 @@ def encode_state_with_base_url(
"code_challenge": code_challenge,
"code_challenge_method": code_challenge_method,
"client_redirect_uri": client_redirect_uri,
"litellm_user_id": litellm_user_id,
"mcp_server_id": mcp_server_id,
}
state_json = json.dumps(state_data, sort_keys=True)
encrypted_state = encrypt_value_helper(state_json)
@ -138,6 +165,68 @@ def decode_state_hash(encrypted_state: str) -> dict:
return state_data
_BRIDGE_AUTH_CODE_PREFIX = "llm_bcode_"
class _BridgeAuthorizationCode(BaseModel):
"""The identity and upstream code the gateway seals into the authorization code it hands a DCR
client for an interactive dcr_bridge oauth_delegate sign-in, recovered at the token endpoint."""
model_config = ConfigDict(frozen=True)
upstream_code: str = Field(min_length=1)
litellm_user_id: str = Field(min_length=1)
mcp_server_id: str = Field(min_length=1)
def is_bridge_authorization_code(code: str) -> bool:
"""Cheap prefix check that ``code`` is a gateway-sealed bridge authorization code rather than a
raw upstream code, so the token endpoint can route without decrypting."""
return code.startswith(_BRIDGE_AUTH_CODE_PREFIX)
def seal_bridge_authorization_code(upstream_code: str, litellm_user_id: str, mcp_server_id: str) -> str:
"""Seal the upstream authorization code and the SSO-captured litellm user into a gateway
authorization code. The DCR client only echoes this opaque value back at the token endpoint; the
gateway decrypts it there to recover the user (to bind the envelope) and the upstream code (to
exchange with the upstream), so a litellm identity captured in the browser at authorize survives
to the back-channel token call with nothing stored server-side. Encrypted with the repo's
authenticated symmetric helper (the same family the OAuth state uses), so the client can neither
read nor forge it."""
payload = json.dumps(
{"upstream_code": upstream_code, "litellm_user_id": litellm_user_id, "mcp_server_id": mcp_server_id},
sort_keys=True,
)
return _BRIDGE_AUTH_CODE_PREFIX + encrypt_value_helper(payload)
def open_bridge_authorization_code(code: str) -> _BridgeAuthorizationCode | None:
"""Recover the sealed identity and upstream code, or ``None`` when ``code`` is not a gateway
bridge code or does not decrypt / validate. Total over hostile input: a raw upstream code (the
scripted two-header path) returns ``None`` and the caller falls through to the existing
behavior."""
if not is_bridge_authorization_code(code):
return None
decrypted = decrypt_value_helper(
code[len(_BRIDGE_AUTH_CODE_PREFIX) :], "bridge_authorization_code", return_original_value=False
)
if not isinstance(decrypted, str):
return None
try:
return _BridgeAuthorizationCode.model_validate_json(decrypted)
except ValidationError:
return None
def _redirect_to_litellm_login(request: Request) -> RedirectResponse:
"""Send an unauthenticated browser through litellm login before the interactive bridge authorize
can capture its identity. The bridge oauth_delegate flow seals the SSO user into the gateway code,
so a session is required; without one there is nothing to bind. After login the user re-initiates
the connection, which then finds the session cookie (the seamless return-to round-trip, which is
origin-validated against the control-plane URL, is a follow-up)."""
base_url = get_request_base_url(request)
return RedirectResponse(f"{base_url}/sso/key/generate")
# LIT-4197: some upstream authorization servers reject an over-long ``state``
# (the encrypted OAuth session blob routinely exceeds their limit). The upstream
# only needs an opaque value it echoes back on ``/callback``, so we forward a
@ -304,90 +393,6 @@ def _validate_token_response(
)
def _litellm_key_from_request(request: Request) -> Optional[str]:
"""Return the LiteLLM API key presented on the request, or ``None``.
Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
an OAuth/upstream bearer.
"""
for header_value in (
request.headers.get("x-litellm-api-key"),
request.headers.get("Authorization") or request.headers.get("authorization"),
):
if not header_value:
continue
value = header_value.strip()
if value.lower().startswith("bearer "):
value = value[7:].strip()
if value:
return value
return None
def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> Optional[str]:
"""The key's ``user_id``, or ``None`` if the key is blocked or expired.
The OAuth token endpoint is unauthenticated, so the presented key is validated here before its
identity is trusted to key a stored credential; a revoked or expired key must not be able to
write or overwrite the per-user OAuth token. ``get_key_object`` resolves a row without these
checks (the main ``user_api_key_auth`` pipeline enforces them downstream, which this endpoint
bypasses), so they are applied here. Deleted keys are already rejected upstream, where
``get_key_object`` raises on a row that no longer exists.
"""
if key_obj.blocked is True:
return None
expires = key_obj.expires
if expires is not None:
expiry = expires if isinstance(expires, datetime) else datetime.fromisoformat(expires)
if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
expiry = expiry.replace(tzinfo=timezone.utc)
if expiry < datetime.now(timezone.utc):
return None
return key_obj.user_id
async def _extract_user_id_from_request(request: Request) -> Optional[str]:
"""Resolve the LiteLLM ``user_id`` at the OAuth token endpoint so a per-user token is stored
under the same identity the egress later reads it by (``user_api_key_auth.user_id``).
Resolves authoritatively via ``get_key_object`` (cache first, then DB) instead of a raw cache
peek. On a multi-replica gateway the token-exchange request can land on a worker whose in-memory
cache never saw the key, and a cross-replica Redis hit deserializes to a plain ``dict`` rather
than a ``UserAPIKeyAuth``; the previous code read only ``Authorization`` and did
``getattr(cached, "user_id")`` with no ``model_type`` rehydration and no DB fallback, so it
silently returned ``None`` and the token was never persisted, which makes the egress 401 on every
reconnect. The resolved key is validated (``_active_key_user_id``) before its identity is trusted,
so a blocked or expired key cannot write. Returns ``None`` when no key is present, the key cannot
be resolved, or it is blocked/expired.
"""
token = _litellm_key_from_request(request)
if not token:
return None
try:
from litellm.proxy._types import hash_token # noqa: PLC0415
from litellm.proxy.auth.auth_checks import get_key_object # noqa: PLC0415
from litellm.proxy.proxy_server import ( # noqa: PLC0415
prisma_client,
user_api_key_cache,
)
key_obj = await get_key_object(
hashed_token=hash_token(token),
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
return _active_key_user_id(key_obj)
except Exception as exc:
verbose_logger.debug(
"_extract_user_id_from_request: could not resolve a LiteLLM user_id for the presented "
"key (%s); per-user token will not be stored server-side.",
type(exc).__name__,
)
return None
async def _store_per_user_token_server_side(
server: MCPServer,
user_id: str,
@ -620,12 +625,31 @@ async def authorize_with_server(
parsed = urlparse(redirect_uri)
base_url = urlunparse(parsed._replace(query=""))
request_base_url = get_request_base_url(request)
# Interactive dcr_bridge oauth_delegate sign-in: this arm runs the gateway /callback and /token in
# the loop, so the gateway can capture the litellm user here (from the browser's UI session) and
# carry it to the back-channel token mint. Seal the SSO user and the target server into the state;
# the callback reads them back to mint the gateway authorization code. A DCR client cannot present a
# litellm key, so the browser session is the only identity source; without one there is nothing to
# bind, so send the user through login first. Every other oauth2 server keeps the identity-less state.
litellm_user_id: str | None = None
if mcp_server.is_dcr_bridge and mcp_server.is_oauth_delegate:
from litellm.proxy._experimental.mcp_server.byok_oauth_endpoints import ( # noqa: PLC0415 # inline import avoids a module-load circular import
_user_id_from_session_cookie,
)
litellm_user_id = _user_id_from_session_cookie(request)
if litellm_user_id is None:
return _redirect_to_litellm_login(request)
encoded_state = encode_state_with_base_url(
base_url=base_url,
original_state=state,
code_challenge=code_challenge,
code_challenge_method=code_challenge_method,
client_redirect_uri=redirect_uri,
litellm_user_id=litellm_user_id,
mcp_server_id=mcp_server.server_id if litellm_user_id else None,
)
relay_state = secrets.token_urlsafe(_OAUTH_STATE_HANDLE_BYTES)
@ -654,6 +678,13 @@ async def authorize_with_server(
return response
def _token_credential_source(mcp_server: MCPServer) -> CredentialSource:
"""Mirrors the resolved-client rule in :func:`exchange_token_with_server`: when the server has a
stored client_id the gateway presents its own credentials upstream, so a credential rejection is
the operator's fault, not the caller's."""
return "gateway_stored" if mcp_server.client_id else "caller_supplied"
async def exchange_token_with_server(
request: Request,
mcp_server: MCPServer,
@ -688,25 +719,61 @@ async def exchange_token_with_server(
except TokenEndpointAuthConfigError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
bridge_identity: _BridgeAuthorizationCode | None = None
bridge_mint_ready: _BridgeMintReady | None = None
bridge_upstream_refresh: SecretStr | None = None
bridge_upstream_scope: str | None = None
refresh_request_scope: str | None = None
is_bridge = mcp_server.is_oauth_delegate and mcp_server.is_dcr_bridge
if grant_type == "refresh_token":
if not refresh_token:
# Phase 1 for a bridge refresh: open the client's refresh envelope, re-validate the sealed
# identity, and unwrap the real upstream refresh token BEFORE building token_data, so the exchange
# sends the upstream token and never the envelope. A failure returns without touching the upstream.
if is_bridge:
prepared_refresh = await _prepare_bridge_refresh(mcp_server, refresh_token)
if not isinstance(prepared_refresh, _BridgeRefreshReady):
return _bridge_mint_error_response(prepared_refresh)
bridge_mint_ready = prepared_refresh.ready
bridge_upstream_refresh = prepared_refresh.upstream_refresh_token
bridge_upstream_scope = prepared_refresh.upstream_scope
# A bridge server sends the unwrapped upstream refresh token recovered from the client's refresh
# envelope above; every other server sends the client's own refresh token verbatim.
upstream_refresh_token = (
bridge_upstream_refresh.get_secret_value() if bridge_upstream_refresh is not None else refresh_token
)
if not upstream_refresh_token:
raise HTTPException(
status_code=400,
detail="refresh_token is required for refresh_token grant",
)
token_data: dict = {
"grant_type": "refresh_token",
"refresh_token": refresh_token,
"refresh_token": upstream_refresh_token,
**client_auth.body,
}
if scope:
token_data["scope"] = scope
refresh_request_scope = scope or bridge_upstream_scope
if refresh_request_scope:
token_data["scope"] = refresh_request_scope
else:
if not code:
raise HTTPException(
status_code=400,
detail="code is required for authorization_code grant",
)
# Interactive dcr_bridge oauth_delegate: the client presents the gateway authorization code the
# callback sealed. Recover the SSO user and the real upstream code from it; the upstream exchange
# below uses the upstream code, and the mint binds the envelope to the recovered user. Bind the
# sealed server to this request so a code minted for one bridge server cannot be spent at another.
# A raw upstream code (scripted path) opens to None and the code is used as-is.
bridge_identity = open_bridge_authorization_code(code)
if bridge_identity is not None:
if bridge_identity.mcp_server_id != mcp_server.server_id:
raise HTTPException(
status_code=400,
detail="Authorization code was issued for a different MCP server",
)
code = bridge_identity.upstream_code
bridge_token_relay = _dcr_bridge_relays_client_registration(mcp_server)
if bridge_token_relay and not redirect_uri:
raise HTTPException(
@ -726,32 +793,49 @@ async def exchange_token_with_server(
}
if code_verifier:
token_data["code_verifier"] = code_verifier
# Phase 1 for a bridge authorization_code mint: resolve identity (the SSO user recovered above, or
# the presented litellm key) and the envelope keys BEFORE the exchange consumes the single-use code.
if is_bridge:
prepared = await _prepare_bridge_mint(request, mcp_server, bridge_identity)
if not isinstance(prepared, _BridgeMintReady):
return _bridge_mint_error_response(prepared)
bridge_mint_ready = prepared
async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Check)
response = await async_client.post(
mcp_server.token_url,
headers={"Accept": "application/json", **client_auth.headers},
data=token_data,
)
try:
response = await async_client.post(
mcp_server.token_url,
headers={"Accept": "application/json", **client_auth.headers},
data=token_data,
)
if response is not None:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
fault = classify_upstream_token_rejection(
exc.response,
credential_source=_token_credential_source(mcp_server),
log_context=mcp_server.server_id,
)
upstream_rejected_bridge_refresh = (
is_bridge
and grant_type == "refresh_token"
and isinstance(fault, CallerRejected)
and fault.code == "invalid_grant"
)
if upstream_rejected_bridge_refresh:
verbose_logger.info(
"bridge refresh: the upstream rejected the sealed refresh token for server=%s with "
"invalid_grant (revoked or expired at the IdP); returning invalid_grant so the client "
"re-runs authorization_code rather than an opaque upstream error",
mcp_server.server_id,
)
return _bridge_mint_error_response("invalid_refresh")
return render_token_fault(fault)
if response is None:
raise HTTPException(
status_code=502,
detail="MCP upstream token endpoint returned no response",
)
try:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
if "invalid_target" in exc.response.text:
verbose_logger.warning(
"MCP server %s: the upstream authorization server rejected the token request with "
"invalid_target; it may require RFC 8707 resource indicators, which the gateway "
"does not send yet (tracked as LIT-4339)",
mcp_server.server_id,
)
raise
token_response = response.json()
access_token = token_response["access_token"]
# Validate token response against server-configured rules before any storage.
# This rejects tokens from wrong Slack workspaces, Atlassian orgs, etc.
@ -791,8 +875,23 @@ async def exchange_token_with_server(
mcp_server.server_id,
)
# A DCR-bridge oauth_delegate server hands the client a gateway-bound envelope (identity plus the
# upstream token) instead of the raw upstream token, so the one bearer both admits the caller and
# forwards the upstream credential. Only this mode mints; every other server returns the raw token.
if bridge_mint_ready is not None:
if refresh_request_scope and isinstance(token_response, dict) and not token_response.get("scope"):
token_response = {**token_response, "scope": refresh_request_scope}
# Phase 3: seal the upstream grant into the client-held envelope; failures map through the same
# OAuth-shaped response as the phase-1 preconditions.
minted = _finish_bridge_mint(bridge_mint_ready, mcp_server, token_response, datetime.now(timezone.utc))
return minted if isinstance(minted, JSONResponse) else _bridge_mint_error_response(minted)
raw_access_token = token_response.get("access_token") if isinstance(token_response, dict) else None
if not isinstance(raw_access_token, str) or not raw_access_token:
return render_token_fault(UpstreamProtocolFault(note="the upstream token response has no usable access_token"))
result = {
"access_token": access_token,
"access_token": raw_access_token,
"token_type": token_response.get("token_type", "Bearer"),
}
@ -1048,21 +1147,6 @@ async def _persist_dcr_client_registration(
return "failed"
_MAX_UPSTREAM_ERROR_CHARS = 500
def _safe_upstream_error_detail(response: httpx.Response) -> str:
"""Bounded plaintext summary of an upstream registration failure for the client.
RFC 7591 error bodies are small JSON objects (``error`` / ``error_description``); relaying the
text lets the client read the real reason instead of a bare 500, and the length bound keeps a
hostile or oversized upstream body from bloating the gateway response."""
body = response.text
if not body:
return response.reason_phrase or "upstream registration failed"
return body[:_MAX_UPSTREAM_ERROR_CHARS]
async def register_client_with_server(
request: Request,
mcp_server: MCPServer,
@ -1122,19 +1206,24 @@ async def register_client_with_server(
}
async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Register)
response = await async_client.post(
mcp_server.registration_url,
headers=headers,
json=register_data,
)
try:
response = await async_client.post(
mcp_server.registration_url,
headers=headers,
json=register_data,
)
if response is not None:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
status_code, detail = dcr_fault_detail(
classify_upstream_dcr_rejection(exc.response, log_context=mcp_server.server_id)
)
raise HTTPException(status_code=status_code, detail=detail) from exc
if response is None:
raise HTTPException(
status_code=502,
detail="MCP upstream registration endpoint returned no response",
)
if bridge_relay and response.status_code >= 400:
raise HTTPException(status_code=response.status_code, detail=_safe_upstream_error_detail(response))
response.raise_for_status()
token_response = response.json()
@ -1362,7 +1451,20 @@ async def callback(
# states while permitting same-origin / allowlisted clients.
redirect_uri = _get_validated_client_redirect_uri(request, state_data)
params = {"code": code, "state": original_state}
# Interactive dcr_bridge oauth_delegate: the state carries the litellm user the authorize step
# captured. Instead of forwarding the raw upstream code (which the client would present at the
# token endpoint with no way to prove who signed in), seal the user and the upstream code into a
# gateway authorization code and forward THAT. The token endpoint decrypts it to bind the
# envelope to this user. Every other flow forwards the raw code unchanged.
litellm_user_id = state_data.get("litellm_user_id")
mcp_server_id = state_data.get("mcp_server_id")
forwarded_code = code
if isinstance(litellm_user_id, str) and litellm_user_id and isinstance(mcp_server_id, str) and mcp_server_id:
forwarded_code = seal_bridge_authorization_code(
upstream_code=code, litellm_user_id=litellm_user_id, mcp_server_id=mcp_server_id
)
params = {"code": forwarded_code, "state": original_state}
complete_returned_url = _append_query_params(redirect_uri, params)
response = RedirectResponse(url=complete_returned_url, status_code=302)
_clear_oauth_state_cookie(response, request, state)

38
litellm/proxy/_experimental/mcp_server/faults/__init__.py Normal file
View file

@ -0,0 +1,38 @@
"""Typed fault values for upstream OAuth/DCR failures (phase 1 of the MCP error-handling framework).
The invariant this package exists to enforce: an upstream failure is classified ONCE into a single
fault value, and the response status, wire error code, and prose are all derived from that value.
Deriving all three from one classification makes contradictory pairings (a caller-fault error code on
a server-fault status) unrepresentable, and gives the trust-boundary rule one enforcement point:
spec-defined machine fields may cross to callers, upstream prose and raw bodies go to server logs.
"""
from litellm.proxy._experimental.mcp_server.faults.classify import (
classify_upstream_dcr_rejection,
classify_upstream_token_rejection,
)
from litellm.proxy._experimental.mcp_server.faults.render_oauth import (
dcr_fault_detail,
render_token_fault,
)
from litellm.proxy._experimental.mcp_server.faults.types import (
CallerRejected,
CredentialSource,
GatewayRejected,
UpstreamOAuthFault,
UpstreamProtocolFault,
UpstreamReportedFault,
)
__all__ = [
"CallerRejected",
"CredentialSource",
"GatewayRejected",
"UpstreamOAuthFault",
"UpstreamProtocolFault",
"UpstreamReportedFault",
"classify_upstream_dcr_rejection",
"classify_upstream_token_rejection",
"dcr_fault_detail",
"render_token_fault",
]

133
litellm/proxy/_experimental/mcp_server/faults/classify.py Normal file
View file

@ -0,0 +1,133 @@
"""The single place that reads upstream OAuth/DCR failure responses.
Every accessor here is total: an upstream that lies about its content encoding, sends an undecodable
body, or omits the spec fields yields a classified fault, never an exception. Nothing outside this
module should touch a failed upstream response's body.
"""
from __future__ import annotations
import httpx
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.faults.types import (
GATEWAY_CAPABILITY_CODES,
GATEWAY_CREDENTIAL_CODES,
MAX_WIRE_FIELD_CHARS,
CallerRejected,
CredentialSource,
GatewayRejected,
UpstreamOAuthFault,
UpstreamProtocolFault,
UpstreamReportedFault,
)
def _safe_text(response: httpx.Response) -> str:
try:
return response.text
except Exception:
return ""
def _safe_json(response: httpx.Response) -> object:
try:
return response.json()
except Exception:
return None
def _bounded_field(value: object) -> str | None:
if not isinstance(value, str) or not value:
return None
return value[:MAX_WIRE_FIELD_CHARS]
def _log_out_of_contract(endpoint_kind: str, response: httpx.Response, log_context: str) -> None:
verbose_logger.warning(
"MCP upstream %s endpoint (%s) returned HTTP %s outside the OAuth error contract (first %s chars): %s",
endpoint_kind,
log_context,
response.status_code,
MAX_WIRE_FIELD_CHARS,
_safe_text(response)[:MAX_WIRE_FIELD_CHARS],
)
def _classify_oauth_error_code(
code: str,
description: str | None,
error_uri: str | None,
credential_source: CredentialSource,
log_context: str,
) -> UpstreamOAuthFault:
"""Blame assignment for a contract-conformant OAuth error code, shared by the token and DCR
classifiers. Codes by which the upstream blames itself keep that blame; ``invalid_target`` is a
gateway capability gap (RFC 8707 resource indicators, LIT-4339) no matter whose credentials were
presented; credential-indicting codes follow the credential source; everything else, including
codes we do not recognize, is the caller's to act on. The upstream's HTTP status is deliberately
never consulted: status derives from this classification at render time, which is what keeps
status and code from contradicting each other."""
if code == "server_error" or code == "temporarily_unavailable":
return UpstreamReportedFault(code=code)
if code in GATEWAY_CAPABILITY_CODES:
verbose_logger.warning(
"MCP server %s: the upstream authorization server rejected the request with "
"invalid_target; it may require RFC 8707 resource indicators, which the gateway "
"does not send yet (tracked as LIT-4339)",
log_context,
)
return GatewayRejected(code=code)
if credential_source == "gateway_stored" and code in GATEWAY_CREDENTIAL_CODES:
verbose_logger.warning(
"MCP server %s: upstream authorization server rejected the gateway's configured client "
"credentials (%s): %s",
log_context,
code,
description or "<no description>",
)
return GatewayRejected(code=code)
return CallerRejected(code=code, description=description, error_uri=error_uri)
def classify_upstream_token_rejection(
response: httpx.Response,
credential_source: CredentialSource,
log_context: str,
) -> UpstreamOAuthFault:
"""Classify a token-endpoint rejection into exactly one fault: a body with an RFC 6749 §5.2
``error`` field goes through blame assignment (:func:`_classify_oauth_error_code`); anything
without a usable ``error`` field is an upstream protocol fault."""
parsed = _safe_json(response)
fields = parsed if isinstance(parsed, dict) else {}
code = _bounded_field(fields.get("error"))
if code is None:
_log_out_of_contract("token", response, log_context)
return UpstreamProtocolFault(note=f"upstream token endpoint returned HTTP {response.status_code}")
return _classify_oauth_error_code(
code,
description=_bounded_field(fields.get("error_description")),
error_uri=_bounded_field(fields.get("error_uri")),
credential_source=credential_source,
log_context=log_context,
)
def classify_upstream_dcr_rejection(response: httpx.Response, log_context: str) -> UpstreamOAuthFault:
"""Classify a dynamic-client-registration rejection. RFC 7591 §3.2.2 errors carry
``error`` / ``error_description`` and go through the same blame assignment as token errors
(registration sends no client credentials, so credential codes stay caller-actionable); anything
without a usable ``error`` field is an upstream protocol fault."""
parsed = _safe_json(response)
fields = parsed if isinstance(parsed, dict) else {}
code = _bounded_field(fields.get("error"))
if code is None:
_log_out_of_contract("registration", response, log_context)
return UpstreamProtocolFault(note=f"upstream registration failed with HTTP {response.status_code}")
return _classify_oauth_error_code(
code,
description=_bounded_field(fields.get("error_description")),
error_uri=None,
credential_source="caller_supplied",
log_context=log_context,
)

89
litellm/proxy/_experimental/mcp_server/faults/render_oauth.py Normal file
View file

@ -0,0 +1,89 @@
"""Render upstream OAuth/DCR faults onto the wire. The only place that chooses statuses and bodies
for these faults, so every consumer emits the same contract: RFC 6749 §5.2-shaped JSON with the §5.1
no-store headers on token endpoints, HTTPException details on registration. Status, code, and prose
all derive from the fault tag; exhaustive matches keep a new fault arm from shipping unrendered.
"""
from __future__ import annotations
from fastapi.responses import JSONResponse
from typing_extensions import assert_never
from litellm.proxy._experimental.mcp_server.faults.types import UpstreamOAuthFault
from litellm.proxy._experimental.mcp_server.oauth_utils import TOKEN_NO_CACHE_HEADERS
def _gateway_rejected_description(code: str) -> str:
if code == "invalid_target":
return (
"the upstream authorization server rejected the request (invalid_target); "
"it may require RFC 8707 resource indicators, which the gateway does not send yet"
)
return (
f"the upstream authorization server rejected the gateway's configured client credentials "
f"({code}); verify the MCP server's client_id and client_secret"
)
def _upstream_reported_status_and_description(code: str) -> tuple[int, str]:
if code == "temporarily_unavailable":
return 503, "the upstream authorization server is temporarily unavailable; retry shortly"
return 502, "the upstream authorization server reported an internal error"
def render_token_fault(fault: UpstreamOAuthFault) -> JSONResponse:
"""RFC 6749 §5.2 response for a token-endpoint fault. Caller-actionable rejections relay the
upstream's code on the status that code implies (401 for invalid_client per §5.2, else 400);
gateway-side faults are 502 ``server_error`` with gateway-authored prose so a caller is never
blamed for, or shown the internals of, a failure only the operator can fix."""
match fault.tag:
case "caller_rejected":
content = {
"error": fault.code,
**({"error_description": fault.description} if fault.description else {}),
**({"error_uri": fault.error_uri} if fault.error_uri else {}),
}
status_code = 401 if fault.code == "invalid_client" else 400
return JSONResponse(status_code=status_code, content=content, headers=TOKEN_NO_CACHE_HEADERS)
case "gateway_rejected":
return JSONResponse(
status_code=502,
content={
"error": "server_error",
"error_description": _gateway_rejected_description(fault.code),
},
headers=TOKEN_NO_CACHE_HEADERS,
)
case "upstream_reported_fault":
status_code, description = _upstream_reported_status_and_description(fault.code)
return JSONResponse(
status_code=status_code,
content={"error": fault.code, "error_description": description},
headers=TOKEN_NO_CACHE_HEADERS,
)
case "upstream_protocol_fault":
return JSONResponse(
status_code=502,
content={"error": "server_error", "error_description": fault.note},
headers=TOKEN_NO_CACHE_HEADERS,
)
case _:
assert_never(fault.tag)
def dcr_fault_detail(fault: UpstreamOAuthFault) -> tuple[int, str]:
"""Status and detail string for a registration fault, raised as HTTPException by the caller.
RFC 7591 §3.2.2 defines registration errors as 400, so a contract-conformant rejection is 400
regardless of the status the upstream chose; everything else is a 502 upstream fault."""
match fault.tag:
case "caller_rejected":
detail = f"{fault.code}: {fault.description}" if fault.description else fault.code
return 400, detail
case "gateway_rejected":
return 502, _gateway_rejected_description(fault.code)
case "upstream_reported_fault":
return _upstream_reported_status_and_description(fault.code)
case "upstream_protocol_fault":
return 502, fault.note
case _:
assert_never(fault.tag)

79
litellm/proxy/_experimental/mcp_server/faults/types.py Normal file
View file

@ -0,0 +1,79 @@
"""Fault taxonomy for upstream OAuth token and DCR registration failures.
Each fault is a frozen model on a ``tag`` literal. The tag alone decides the HTTP status, the wire
error code, and whose prose the caller sees, so those three facts can never disagree the way they can
when an upstream's status and error code are relayed independently.
"""
from __future__ import annotations
from typing import Literal, TypeAlias
from pydantic import BaseModel, ConfigDict
MAX_WIRE_FIELD_CHARS = 500
"""Bound on every upstream-derived string that crosses to a caller or into a log line."""
CredentialSource: TypeAlias = Literal["gateway_stored", "caller_supplied"]
"""Whose client credentials the gateway presented upstream: the MCP server's stored configuration or
credentials the caller supplied on the request. Decides whether a credential rejection is the
caller's problem to fix or the gateway operator's."""
GATEWAY_CREDENTIAL_CODES: frozenset[str] = frozenset({"invalid_client", "unauthorized_client"})
"""RFC 6749 error codes that indict the OAuth client's credentials or grant authorization. When the
gateway presented its own stored credentials, these are gateway-side faults the caller cannot act on;
when the caller supplied the credentials, they are the caller's to fix."""
GATEWAY_CAPABILITY_CODES: frozenset[str] = frozenset({"invalid_target"})
"""Codes that indict a gateway capability regardless of whose credentials were presented:
``invalid_target`` means the upstream wants RFC 8707 resource indicators, which the gateway does not
send yet (LIT-4339). Never the caller's fault."""
UPSTREAM_FAULT_CODES: frozenset[str] = frozenset({"server_error", "temporarily_unavailable"})
"""Codes by which the upstream blames itself. Relaying them as caller faults would invert blame, so
they classify as upstream-reported faults and render on the 5xx their meaning implies."""
class CallerRejected(BaseModel):
"""The upstream spoke the OAuth error contract and the failure is actionable by our caller
(e.g. ``invalid_grant``: re-run authorization). The code and its bounded prose relay on the
4xx status the code itself implies."""
model_config = ConfigDict(frozen=True)
tag: Literal["caller_rejected"] = "caller_rejected"
code: str
description: str | None = None
error_uri: str | None = None
class GatewayRejected(BaseModel):
"""The upstream rejected the request for a cause only the gateway operator can address: the
server's stored client credentials or a gateway capability gap. Not actionable by the caller:
rendered as 502 with gateway-authored prose naming the code; the upstream's prose goes to
server logs only."""
model_config = ConfigDict(frozen=True)
tag: Literal["gateway_rejected"] = "gateway_rejected"
code: str
class UpstreamReportedFault(BaseModel):
"""The upstream blamed itself in the OAuth vocabulary. Rendered on the 5xx the code implies
(``server_error`` 502, ``temporarily_unavailable`` 503) so blame and status agree."""
model_config = ConfigDict(frozen=True)
tag: Literal["upstream_reported_fault"] = "upstream_reported_fault"
code: Literal["server_error", "temporarily_unavailable"]
class UpstreamProtocolFault(BaseModel):
"""The upstream broke the error contract: no JSON ``error`` field, an undecodable body, or a
success response without a usable token. Rendered as 502 with a gateway-authored note; the
upstream body never crosses to the caller."""
model_config = ConfigDict(frozen=True)
tag: Literal["upstream_protocol_fault"] = "upstream_protocol_fault"
note: str
UpstreamOAuthFault: TypeAlias = CallerRejected | GatewayRejected | UpstreamReportedFault | UpstreamProtocolFault

295
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -186,6 +186,144 @@ _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES: tuple[MCPAuth, ...] = (
)
def _blank_to_none(value: str | None) -> str | None:
"""Collapse an absent, empty, or whitespace-only string to ``None``.
OAuth endpoint fields are consumed by truthiness-based merges (``row or discovered``) and by the
corroboration gate. A whitespace-only value is truthy to ``or`` but is not a usable endpoint, so
without this the merge would keep the blank value for redirects while the gate treats it as
unpinned and backfills the other fields, yielding a broken half-discovered config. Normalizing
the pinned fields once, at each build entry point, gives every downstream consumer a single
notion of "blank" so those code paths cannot disagree.
"""
if not isinstance(value, str):
return None
return value.strip() or None
def _normalized_authorize_endpoint(url: str) -> str:
"""Compare authorize endpoints on scheme, host, and path only. The default port is elided and
the host is lowercased so ``https://IDP.example.com:443/authorize/`` and
``https://idp.example.com/authorize`` are the same identity; query and trailing slash are not."""
parsed = urlparse(url)
scheme = parsed.scheme.lower()
host = (parsed.hostname or "").lower()
default_port = {"https": 443, "http": 80}.get(scheme)
try:
port = parsed.port
except ValueError:
port = None
authority = host if port is None or port == default_port else f"{host}:{port}"
return f"{scheme}://{authority}{parsed.path.rstrip('/')}"
def _endpoints_corroborate_authorization_url(
source_authorization_url: str | None,
trusted_authorization_url: str | None,
) -> bool:
"""Whether a source's ``token_url``/``registration_url`` may be paired with a trusted authorize
endpoint. This is the single trust rule for adopting OAuth endpoints from any non-manual source.
Discovery is rooted at the MCP resource (RFC 9728), so a compromised upstream can advertise an
attacker-run authorization server. When ``authorization_url`` is admin-pinned, pairing it with a
``token_url`` from a different source is the RFC 9700 authorization-server mix-up: the user signs
in at the trusted authorize endpoint while the gateway redeems the code, with the stored client
secret and PKCE verifier, at the attacker's token endpoint. Endpoints are trustworthy together
only when they share an authorization server, so a source's endpoints are adopted only when the
same source advertised an ``authorization_endpoint`` matching the pinned value. With no pinned
value (``trusted_authorization_url is None``) there is nothing to protect: the authorize endpoint
comes from the same source as the token endpoint, so they corroborate each other by construction.
"""
if not (trusted_authorization_url and trusted_authorization_url.strip()):
return True
return bool(source_authorization_url) and _normalized_authorize_endpoint(
source_authorization_url
) == _normalized_authorize_endpoint(trusted_authorization_url)
def _carry_forward_resolved_oauth_endpoints(new_server: MCPServer, previous_server: MCPServer | None) -> None:
"""Keep the last known good OAuth endpoints when a rebuild's re-discovery comes back empty.
A rebuild wholesale-replaces the registry entry, so without this a transient upstream outage
during re-discovery downgrades a working server (``authorization_url`` set) to a broken one
(``None``, /authorize 400s) with no configuration change. Mirrors the ``short_prefix``
carry-forward. Skipped when the server's ``url`` or ``auth_type`` changed, since the previous
endpoints may then belong to a different upstream. ``registration_url`` IS carried even though
``_persist_discovered_oauth_endpoints`` refuses to write it to the row: carrying only restores
the same in-memory value the previous build already ran with, while persisting it would flip
``_dcr_bridge_relays_client_registration`` (which keys off the stored column) for dcr_bridge
servers that never had one configured.
Carry-forward is a non-manual endpoint source, so the same trust rule as discovery applies: the
previous ``token_url``/``registration_url``/``scopes`` are carried only when the previous
``authorization_url`` corroborates the authorize endpoint this build will use, i.e. when the
incoming build has no pinned authorize endpoint (``None`` -> we adopt the previous one too, a
consistent group) or pins the same one. An admin re-pointing ``authorization_url`` to a different
server must not keep serving the old server's token endpoint or granted scopes.
"""
if previous_server is None:
return
if previous_server.url != new_server.url or previous_server.auth_type != new_server.auth_type:
return
may_carry = _endpoints_corroborate_authorization_url(
previous_server.authorization_url, new_server.authorization_url
)
if new_server.authorization_url is None and previous_server.authorization_url:
new_server.authorization_url = previous_server.authorization_url
if may_carry and new_server.token_url is None and previous_server.token_url:
new_server.token_url = previous_server.token_url
if may_carry and new_server.registration_url is None and previous_server.registration_url:
new_server.registration_url = previous_server.registration_url
if may_carry and not new_server.scopes and previous_server.scopes:
new_server.scopes = previous_server.scopes
def _restrict_discovery_to_corroborated_authorization_server(
metadata: MCPOAuthMetadata | None,
manual_authorization_url: str | None,
server_identifier: str,
is_dcr_bridge: bool,
) -> MCPOAuthMetadata | None:
"""Reject discovered token/registration endpoints a manually pinned authorize endpoint cannot
vouch for (the RFC 9700 authorization-server mix-up).
Discovery is rooted at the MCP resource, so a compromised upstream can advertise an attacker
``token_endpoint``: with ``authorization_url`` admin-pinned but ``token_url`` blank, the merge
would pair the trusted authorize endpoint with that attacker token endpoint, and the gateway would
post the authorization code and client secret there. So the discovered ``token_url`` and
``registration_url`` are kept only if the document corroborates the pin (its
``authorization_endpoint`` matches). ``scopes`` are deliberately NOT gated here: per the MCP
authorization spec Scope Selection Strategy and RFC 9700 §2.3, the scopes a client requests are
resource-driven (the WWW-Authenticate challenge or the RFC 9728 protected-resource
``scopes_supported``), and scope inflation by a compromised resource is bounded by the
authorization server and user consent (RFC 6749 §3.3), not by the client second-guessing the
request. With no pin there is no trust anchor to protect, so discovery is returned as-is.
"""
if metadata is None or not (manual_authorization_url and manual_authorization_url.strip()):
return metadata
if _endpoints_corroborate_authorization_url(metadata.authorization_url, manual_authorization_url):
return metadata
if not metadata.token_url and not metadata.registration_url:
return metadata
bridge_note = (
" The discovered registration_url is rejected with it, so this dcr_bridge server stays on the"
" short-circuit registration arm."
if is_dcr_bridge and metadata.registration_url
else ""
)
verbose_logger.warning(
"MCP OAuth discovery for server %s advertised authorization_endpoint %s, which does not match the "
"manually configured authorization_url %s; rejecting the discovered token_url/registration_url so "
"authorization codes and client credentials only follow the configured authorization server. "
"Configure Token URL manually if the mismatch is intentional.%s",
server_identifier,
_normalized_authorize_endpoint(metadata.authorization_url) if metadata.authorization_url else "<absent>",
_normalized_authorize_endpoint(manual_authorization_url),
bridge_note,
)
return metadata.model_copy(update={"token_url": None, "registration_url": None})
def invalidate_user_env_vars_cache(user_id: str, server_id: str) -> None:
"""Drop a cached entry after the user stores or clears their env var values
so the next request reads the fresh value instead of a stale one."""
@ -999,12 +1137,15 @@ class MCPServerManager:
)
auth_type = server_config.get("auth_type", None)
manual_authorization_url = _blank_to_none(server_config.get("authorization_url"))
manual_token_url = _blank_to_none(server_config.get("token_url"))
manual_registration_url = _blank_to_none(server_config.get("registration_url"))
if server_url and (
auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES
or self._obo_needs_endpoint_discovery(
auth_type,
server_config.get("token_exchange_endpoint"),
server_config.get("token_url"),
manual_token_url,
)
):
mcp_oauth_metadata = await self._descovery_metadata(
@ -1014,20 +1155,29 @@ class MCPServerManager:
else:
mcp_oauth_metadata = None
gated_oauth_metadata = (
_restrict_discovery_to_corroborated_authorization_server(
mcp_oauth_metadata,
manual_authorization_url,
server_name or server_id,
bool(server_config.get("dcr_bridge")),
)
if auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES
else mcp_oauth_metadata
)
# Filter blank scopes (e.g. YAML ``scopes: [""]``) the same way the DB-build path does, so
# an all-blank list normalizes to None rather than a ``("",)`` tuple that skips the
# entra_obo fail-closed scope precondition and POSTs an empty scope to the IdP.
resolved_scopes = self._extract_scopes(server_config.get("scopes")) or (
mcp_oauth_metadata.scopes if mcp_oauth_metadata else None
gated_oauth_metadata.scopes if gated_oauth_metadata else None
)
resolved_authorization_url = server_config.get("authorization_url") or (
mcp_oauth_metadata.authorization_url if mcp_oauth_metadata else None
resolved_authorization_url = manual_authorization_url or (
gated_oauth_metadata.authorization_url if gated_oauth_metadata else None
)
resolved_token_url = server_config.get("token_url") or (
mcp_oauth_metadata.token_url if mcp_oauth_metadata else None
)
resolved_registration_url = server_config.get("registration_url") or (
mcp_oauth_metadata.registration_url if mcp_oauth_metadata else None
resolved_token_url = manual_token_url or (gated_oauth_metadata.token_url if gated_oauth_metadata else None)
resolved_registration_url = manual_registration_url or (
gated_oauth_metadata.registration_url if gated_oauth_metadata else None
)
config_oauth2_flow = server_config.get("oauth2_flow", None)
@ -1343,6 +1493,7 @@ class MCPServerManager:
*,
credentials_are_encrypted: bool = True,
env_vars_are_encrypted: Optional[bool] = None,
persist_discovered_endpoints: bool = True,
) -> MCPServer:
_mcp_info: MCPInfo = mcp_server.mcp_info or {}
env_dict = _deserialize_json_dict(getattr(mcp_server, "env", None))
@ -1419,13 +1570,17 @@ class MCPServerManager:
auth_type = cast(MCPAuthType, mcp_server.auth_type)
server_url = mcp_server.url
manual_authorization_url = _blank_to_none(mcp_server.authorization_url)
manual_token_url = _blank_to_none(mcp_server.token_url)
manual_registration_url = _blank_to_none(mcp_server.registration_url)
has_all_upstream_oauth_fields = bool(manual_authorization_url and manual_token_url and scopes)
needs_discovery = bool(server_url) and (
(auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES and not mcp_server.authorization_url)
(auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES and not has_all_upstream_oauth_fields)
or self._obo_needs_endpoint_discovery(
auth_type,
mcp_server.token_exchange_endpoint
or (credentials_dict.get("token_exchange_endpoint") if credentials_dict else None),
mcp_server.token_url,
manual_token_url,
)
)
mcp_oauth_metadata = (
@ -1436,8 +1591,25 @@ class MCPServerManager:
if needs_discovery
else None
)
if needs_discovery and mcp_oauth_metadata is None:
verbose_logger.warning(
"MCP OAuth discovery yielded no metadata for server %s (%s); "
"OAuth endpoints/scopes stay unresolved until a rebuild succeeds",
mcp_server.server_id,
server_url,
)
gated_oauth_metadata = (
_restrict_discovery_to_corroborated_authorization_server(
mcp_oauth_metadata,
manual_authorization_url,
mcp_server.server_id,
bool(getattr(mcp_server, "dcr_bridge", None)),
)
if auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES
else mcp_oauth_metadata
)
resolved_scopes = scopes or (mcp_oauth_metadata.scopes if mcp_oauth_metadata else None)
resolved_scopes = scopes or (gated_oauth_metadata.scopes if gated_oauth_metadata else None)
new_server = MCPServer(
server_id=mcp_server.server_id,
@ -1457,9 +1629,9 @@ class MCPServerManager:
client_secret=client_secret_value or getattr(mcp_server, "client_secret", None),
oauth2_flow=self._explicit_oauth2_flow(getattr(mcp_server, "oauth2_flow", None)),
scopes=resolved_scopes,
authorization_url=mcp_server.authorization_url or getattr(mcp_oauth_metadata, "authorization_url", None),
token_url=mcp_server.token_url or getattr(mcp_oauth_metadata, "token_url", None),
registration_url=mcp_server.registration_url or getattr(mcp_oauth_metadata, "registration_url", None),
authorization_url=manual_authorization_url or getattr(gated_oauth_metadata, "authorization_url", None),
token_url=manual_token_url or getattr(gated_oauth_metadata, "token_url", None),
registration_url=manual_registration_url or getattr(gated_oauth_metadata, "registration_url", None),
token_endpoint_auth_method=(
credentials_dict.get("token_endpoint_auth_method") if credentials_dict else None
),
@ -1506,12 +1678,21 @@ class MCPServerManager:
max_concurrent_requests=getattr(mcp_server, "max_concurrent_requests", None),
)
_warn_internal_delegate_pkce_if_applicable(new_server, source="database")
await self._persist_discovered_obo_token_url(
server_id=mcp_server.server_id,
auth_type=auth_type,
existing_token_url=mcp_server.token_url,
discovered_token_url=new_server.token_url,
)
if persist_discovered_endpoints:
await self._persist_discovered_obo_token_url(
server_id=mcp_server.server_id,
auth_type=auth_type,
existing_token_url=manual_token_url,
discovered_token_url=new_server.token_url,
)
await self._persist_discovered_oauth_endpoints(
server_id=mcp_server.server_id,
auth_type=auth_type,
existing_authorization_url=manual_authorization_url,
existing_token_url=manual_token_url,
existing_scopes=scopes,
metadata=gated_oauth_metadata,
)
return new_server
async def _persist_discovered_obo_token_url(
@ -1549,6 +1730,69 @@ class MCPServerManager:
except Exception as exc: # noqa: BLE001 - best-effort; a failed write re-discovers next build
verbose_logger.warning("Failed to persist discovered OBO token_url for MCP server %s: %s", server_id, exc)
async def _persist_discovered_oauth_endpoints(
self,
*,
server_id: str,
auth_type: MCPAuthType | None,
existing_authorization_url: str | None,
existing_token_url: str | None,
existing_scopes: list[str] | None,
metadata: MCPOAuthMetadata | None,
) -> None:
"""Write freshly discovered OAuth endpoints back onto the DB row.
Same rationale as ``_persist_discovered_obo_token_url`` but for the interactive oauth2
family: discovered ``authorization_url``/``token_url``/``scopes`` otherwise live only on
the in-memory registry entry, which is rebuilt on every client connect (the DCR reuse path
calls ``update_server``) and on every post-write DB reload, so one failed re-discovery
serves 400 "authorization url is not set" from /authorize until a later rebuild succeeds.
Only fills row fields that are currently empty, never persists origin-fallback guesses
(RFC 9728/8414-advertised metadata only), and deliberately skips ``registration_url``
because ``_dcr_bridge_relays_client_registration`` keys off that column. Best-effort: a
failed write re-discovers on the next build. Scopes go through ``update_mcp_server`` so
they merge into the credentials blob without touching the stored client credentials.
"""
if auth_type not in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES:
return
if metadata is None or metadata.from_origin_fallback:
return
authorization_url_update = (
{"authorization_url": metadata.authorization_url}
if metadata.authorization_url and not existing_authorization_url
else {}
)
token_url_update = {"token_url": metadata.token_url} if metadata.token_url and not existing_token_url else {}
scopes_update = {"credentials": {"scopes": metadata.scopes}} if metadata.scopes and not existing_scopes else {}
updates: dict[str, object] = {**authorization_url_update, **token_url_update, **scopes_update}
if not updates:
return
from litellm.proxy._experimental.mcp_server.db import ( # noqa: PLC0415 # db.py imports this module at load
update_mcp_server,
)
from litellm.proxy._types import UpdateMCPServerRequest # noqa: PLC0415 # heavy module; import at call time
from litellm.proxy.proxy_server import prisma_client # noqa: PLC0415 # runtime value, set after startup
if prisma_client is None:
return
try:
await update_mcp_server(
prisma_client=prisma_client,
data=UpdateMCPServerRequest.model_validate({"server_id": server_id, **updates}),
touched_by="mcp_oauth_discovery",
)
verbose_logger.info(
"Persisted discovered OAuth endpoints for MCP server %s: %s",
server_id,
sorted(updates),
)
except Exception as exc: # noqa: BLE001 - best-effort; a failed write re-discovers next build
verbose_logger.warning(
"Failed to persist discovered OAuth endpoints for MCP server %s: %s",
server_id,
exc,
)
async def _maybe_register_openapi_tools(self, server: MCPServer, *, initialize_mapping: bool = True):
"""Register OpenAPI tools if the server has a spec_path configured."""
if server.spec_path:
@ -1607,6 +1851,10 @@ class MCPServerManager:
existing_prefix = self.registry[mcp_server.server_id].short_prefix
if existing_prefix and not new_server.short_prefix:
new_server.short_prefix = existing_prefix
_carry_forward_resolved_oauth_endpoints(
new_server=new_server,
previous_server=self.registry[mcp_server.server_id],
)
self._assign_unique_short_prefix(new_server)
self.registry[mcp_server.server_id] = new_server
await self._maybe_register_openapi_tools(new_server)
@ -2969,16 +3217,20 @@ class MCPServerManager:
) = await self._attempt_well_known_discovery(server_url)
metadata = None
used_origin_fallback = False
if allow_origin_fallback and not authorization_servers:
try:
parsed_url = urlparse(server_url)
if parsed_url.scheme and parsed_url.netloc:
authorization_servers = [f"{parsed_url.scheme}://{parsed_url.netloc}"]
used_origin_fallback = True
except Exception:
authorization_servers = []
if authorization_servers:
metadata = await self._fetch_authorization_server_metadata(authorization_servers, server_url)
if metadata is not None and used_origin_fallback:
metadata.from_origin_fallback = True
preferred_scopes = scopes or resource_scopes
if metadata is None and preferred_scopes:
@ -4489,6 +4741,7 @@ class MCPServerManager:
# (if any) so the prefix is stable across reloads.
if existing_server is not None and existing_server.short_prefix:
new_server.short_prefix = existing_server.short_prefix
_carry_forward_resolved_oauth_endpoints(new_server=new_server, previous_server=existing_server)
new_registry[server.server_id] = new_server
except Exception as e:
verbose_logger.exception(

18
litellm/proxy/_experimental/mcp_server/oauth2_token_cache.py
View file

@ -175,16 +175,18 @@ mcp_oauth2_token_cache = MCPOAuth2TokenCache()
def _compute_per_user_token_ttl(server: "MCPServer", expires_in: Optional[int]) -> int:
"""Compute Redis TTL for a per-user token.
Uses server.token_storage_ttl_seconds when configured; otherwise derives
TTL from expires_in minus the expiry buffer; falls back to the default TTL.
Uses server.token_storage_ttl_seconds when configured, capped at the token's
remaining lifetime (expires_in minus the expiry buffer) so a cached entry never
outlives the token itself; otherwise derives TTL from expires_in minus the
expiry buffer; falls back to the default TTL.
"""
lifetime_bound = expires_in - MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS if expires_in is not None else None
if server.token_storage_ttl_seconds is not None:
return max(server.token_storage_ttl_seconds, 1)
if expires_in is not None:
return max(
expires_in - MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS,
1,
)
if lifetime_bound is None:
return max(server.token_storage_ttl_seconds, 1)
return max(min(server.token_storage_ttl_seconds, lifetime_bound), 1)
if lifetime_bound is not None:
return max(lifetime_bound, 1)
return MCP_PER_USER_TOKEN_DEFAULT_TTL

82
litellm/proxy/_experimental/mcp_server/outbound_credentials/bridge_credentials.py
View file

@ -21,11 +21,16 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import
EnvelopeKeys,
EnvelopeMintError,
OpenedEnvelope,
OpenedRefreshEnvelope,
RefreshCredential,
SealedEnvelope,
UpstreamTokenGrant,
is_envelope,
is_refresh_envelope,
mint_envelope,
mint_refresh_envelope,
open_envelope,
open_refresh_envelope,
)
_SIGNING_KEY_DOMAIN = b"litellm-mcp-bridge:envelope-signing:"
@ -92,6 +97,67 @@ def build_bridge_token_response(
return mint_envelope(identity, grant, keys, now)
def build_bridge_refresh_token_response(
identity: EnvelopeIdentity,
refresh: RefreshCredential,
keys: EnvelopeKeys,
now: datetime,
) -> SealedEnvelope | EnvelopeMintError:
"""Seal ``refresh`` for ``identity`` into the long-lived refresh envelope the token endpoint returns
alongside the access envelope, so the client can renew without re-authenticating. A thin, pure
wrapper over :func:`mint_refresh_envelope`; returns the mint error as a value for the caller to map.
"""
return mint_refresh_envelope(identity, refresh, keys, now)
class BridgeRefreshOpened(BaseModel):
"""A valid refresh envelope presented to the token endpoint: the identity to re-validate and renew
under, and the upstream refresh grant to exchange."""
model_config = ConfigDict(frozen=True)
tag: Literal["opened"] = "opened"
identity: EnvelopeIdentity
refresh: RefreshCredential
class BridgeRefreshInvalid(BaseModel):
"""The presented refresh grant is not a valid refresh envelope for this server (not refresh-shaped,
will not open, or minted for a different server); the token endpoint fails the refresh closed."""
model_config = ConfigDict(frozen=True)
tag: Literal["invalid"] = "invalid"
BridgeRefreshResult: TypeAlias = BridgeRefreshOpened | BridgeRefreshInvalid
def open_bridge_refresh_envelope(
refresh_value: str,
keys: EnvelopeKeys,
now: datetime,
expected_server_id: str,
) -> BridgeRefreshResult:
"""Open a refresh envelope a bridge ``oauth_delegate`` client presented on a refresh_token grant.
The token-endpoint mirror of :func:`resolve_bridge_envelope`: strips an optional ``Bearer`` scheme,
then returns ``BridgeRefreshOpened`` with the recovered identity and upstream refresh grant, or
``BridgeRefreshInvalid`` for anything that is not a valid refresh envelope for this server. Never
raises; total over hostile input via :func:`open_refresh_envelope`. ``expected_server_id`` binds the
envelope to the server the request targets, so a refresh envelope minted for one server cannot renew
against another. A raw upstream refresh token (not envelope-shaped) is ``BridgeRefreshInvalid``: this
mode never hands the client a bare upstream refresh token, so it must never accept one.
"""
candidate = _strip_bearer(refresh_value)
if not is_refresh_envelope(candidate):
return BridgeRefreshInvalid()
opened = open_refresh_envelope(candidate, keys, now)
if not isinstance(opened, OpenedRefreshEnvelope):
return BridgeRefreshInvalid()
if opened.identity.server_id != expected_server_id:
return BridgeRefreshInvalid()
return BridgeRefreshOpened(identity=opened.identity, refresh=opened.refresh)
class NotBridgeEnvelope(BaseModel):
"""The bearer is not an envelope; admission continues on its normal path."""
@ -128,10 +194,12 @@ def _strip_bearer(value: str) -> str:
def is_bridge_envelope_shaped(authorization_value: str) -> bool:
"""Cheap, keyless test that an ``Authorization`` value carries an envelope (optional
``Bearer`` scheme stripped). The admission edge engages the bridge arm only for an
envelope, so a plain upstream bearer falls through to normal oauth2 admission."""
return is_envelope(_strip_bearer(authorization_value))
"""Cheap, keyless test that an ``Authorization`` value carries an envelope of either kind (optional
``Bearer`` scheme stripped). The admission edge engages the bridge arm for an access envelope (to
admit) and for a refresh envelope (to reject it explicitly, since a refresh credential is never
usable at the tool-call edge); a plain upstream bearer falls through to normal oauth2 admission."""
candidate = _strip_bearer(authorization_value)
return is_envelope(candidate) or is_refresh_envelope(candidate)
def resolve_bridge_envelope(
@ -148,6 +216,10 @@ def resolve_bridge_envelope(
envelope, and ``BridgeEnvelopeInvalid`` for an envelope-shaped bearer that will not
open. Never raises: it is total over hostile input via :func:`open_envelope`.
A refresh envelope is ``BridgeEnvelopeInvalid`` here: it is a valid gateway credential but only ever
presented back to the token endpoint, never usable to authenticate a tool call, so admission must
fail it closed rather than let it fall through to another arm.
``expected_server_id`` is the ``server_id`` of the MCP server the request targets; an
opened envelope whose sealed ``server_id`` does not match is rejected as
``BridgeEnvelopeInvalid``. Binding here (rather than leaving it to the caller) prevents
@ -157,6 +229,8 @@ def resolve_bridge_envelope(
unlike ``hmac.compare_digest`` on ``str``, does not raise on a non-ASCII server_id.
"""
candidate = _strip_bearer(authorization_value)
if is_refresh_envelope(candidate):
return BridgeEnvelopeInvalid()
if not is_envelope(candidate):
return NotBridgeEnvelope()
opened = open_envelope(candidate, keys, now)

286
litellm/proxy/_experimental/mcp_server/outbound_credentials/envelope.py
View file

@ -44,18 +44,33 @@ from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationError
from litellm.proxy.common_utils.encrypt_decrypt_utils import decrypt_value, encrypt_value
ENVELOPE_PREFIX = "llm_env_"
"""Marker prefix on every serialized envelope so the edge can cheaply tell an envelope
"""Marker prefix on every serialized ACCESS envelope so the edge can cheaply tell an envelope
from a raw upstream token before doing any cryptography."""
REFRESH_ENVELOPE_PREFIX = "llm_refresh_"
"""Marker prefix on every serialized REFRESH envelope. A distinct prefix keeps the two credentials
routable without crypto and, together with the signed ``kind`` claim, stops one from being presented
where the other is expected: a refresh envelope carries a long-lived upstream refresh token and is only
ever presented back to the token endpoint, never forwarded upstream on a tool call."""
ENVELOPE_ISSUER = "litellm-mcp-bridge"
"""``iss`` claim stamped into every envelope and required back on open."""
MAX_ENVELOPE_TTL_SECONDS = 3600
"""Hard ceiling on envelope lifetime. ``exp`` is ``min(upstream expires_in, this cap)``
"""Hard ceiling on ACCESS envelope lifetime. ``exp`` is ``min(upstream expires_in, this cap)``
(the cap alone when the upstream omits ``expires_in``), matching the 1h lifetime of the
BYOK session bearer this module's signing approach is borrowed from: a client-held
credential should never outlive a bounded window even when the upstream token does."""
MAX_REFRESH_ENVELOPE_TTL_SECONDS = 1209600
"""Hard ceiling on REFRESH envelope lifetime (14 days). A refresh envelope only renews the short-lived
access envelope, and each renewal re-validates the sealed litellm key (revocation gates it) and is
re-minted with a fresh window, so the practical bound is idle time, not a fixed session. ``exp`` is
``min(upstream refresh_expires_in, this cap)`` (the cap alone when the upstream omits it); if the
upstream refresh token dies first, the next renewal simply fails at the upstream and the client
re-authenticates. The value is deliberately far shorter than a typical upstream refresh-token lifetime
so a leaked refresh envelope is bounded even if the upstream would have honoured it for longer."""
MAX_ENVELOPE_BYTES = 12288
"""Size cap on the final serialized envelope (prefix + JWT, in bytes). Upstream JWTs
commonly run 2-4KB; base64 plus encryption overhead roughly doubles that inside the
@ -66,21 +81,48 @@ typed error, never truncated."""
_ENVELOPE_JWT_ALGORITHM = "HS256"
EnvelopeKind = Literal["access", "refresh"]
"""Which credential an envelope is. Stamped into the signed claims and required to match on open, so a
signature-valid envelope of one kind cannot be replayed as the other even if its wire prefix is swapped
(the prefix is not part of the signed payload; this claim is)."""
EnvelopeSubjectType: TypeAlias = Literal["key_hash", "user_id"]
"""Discriminator for what litellm principal the envelope binds the grant to.
``key_hash`` is a hashed virtual key (the scripted two-header client mints under the key it
presents at the token endpoint); ``user_id`` is a litellm user subject (the interactive DCR
client mints under the SSO-authenticated user, which is the only identity that browser login
yields). Admission reloads a key record for the first and a user record for the second, then
runs both through the same live-policy gate, so team/org/budget/revocation enforcement is
identical either way."""
class EnvelopeIdentity(BaseModel):
"""The litellm identity the envelope binds the inner grant to.
"""The litellm principal the envelope binds the inner grant to.
``key_hash`` is the hashed litellm key that authorized the mint, never a raw
credential (and the edge rejects a bare hash presented as a bearer). Admission
reloads the live key record by it, so the key's current team/org/object-permission
restrictions and its revocation state are enforced at use time rather than frozen at
mint time. ``server_id`` binds the envelope to one MCP server so it cannot be replayed
across a server boundary.
``subject`` is the principal identifier and ``subject_type`` says how to resolve it: a
hashed litellm key (``key_hash``) or a litellm user id (``user_id``), never a raw
credential (and the edge rejects a bare hash or id presented as a bearer). Admission
reloads the live record by it, so the principal's current team/org restrictions and its
revocation state are enforced at use time rather than frozen at mint time. ``server_id``
binds the envelope to one MCP server so it cannot be replayed across a server boundary.
"""
model_config = ConfigDict(frozen=True)
server_id: str = Field(min_length=1)
key_hash: str = Field(min_length=1)
subject_type: EnvelopeSubjectType
subject: str = Field(min_length=1)
def key_hash_identity(server_id: str, key_hash: str) -> EnvelopeIdentity:
"""The identity for the scripted client that mints under a presented virtual key."""
return EnvelopeIdentity(server_id=server_id, subject_type="key_hash", subject=key_hash)
def user_identity(server_id: str, user_id: str) -> EnvelopeIdentity:
"""The identity for the interactive DCR client that mints under its SSO user subject."""
return EnvelopeIdentity(server_id=server_id, subject_type="user_id", subject=user_id)
class UpstreamTokenGrant(BaseModel):
@ -99,6 +141,21 @@ class UpstreamTokenGrant(BaseModel):
expires_in: int | None = Field(default=None, gt=0)
class RefreshCredential(BaseModel):
"""The upstream refresh grant sealed inside a refresh envelope.
Only the refresh token (plus the scope to re-request and the refresh token's own lifetime, when the
upstream reports it) is sealed; the access token is never in a refresh envelope. ``refresh_token`` is
a ``SecretStr`` so reprs never leak it, and ``expires_in`` (the refresh token's lifetime, not the
access token's) must be positive when present.
"""
model_config = ConfigDict(frozen=True)
refresh_token: SecretStr = Field(min_length=1)
scope: str | None = None
expires_in: int | None = Field(default=None, gt=0)
class EnvelopeKeys(BaseModel):
"""Injected key material: the HS256 signing key and the symmetric encryption key.
@ -121,13 +178,21 @@ class SealedEnvelope(BaseModel):
class OpenedEnvelope(BaseModel):
"""A validated envelope: the identity it was minted for and the recovered grant."""
"""A validated access envelope: the identity it was minted for and the recovered grant."""
model_config = ConfigDict(frozen=True)
identity: EnvelopeIdentity
grant: UpstreamTokenGrant
class OpenedRefreshEnvelope(BaseModel):
"""A validated refresh envelope: the identity it was minted for and the recovered refresh grant."""
model_config = ConfigDict(frozen=True)
identity: EnvelopeIdentity
refresh: RefreshCredential
class EnvelopeTooLarge(BaseModel):
"""The serialized envelope exceeded ``MAX_ENVELOPE_BYTES``; carries sizes only."""
@ -199,8 +264,10 @@ class _EnvelopeClaims(BaseModel):
iss: str
iat: int
exp: int
kind: EnvelopeKind
server_id: str = Field(min_length=1)
key_hash: str = Field(min_length=1)
subject_type: EnvelopeSubjectType
subject: str = Field(min_length=1)
grant: str = Field(min_length=1)
@ -213,11 +280,25 @@ class _GrantWire(BaseModel):
expires_in: int | None = None
class _RefreshWire(BaseModel):
model_config = ConfigDict(frozen=True)
refresh_token: str
scope: str | None = None
expires_in: int | None = None
def is_envelope(candidate: str) -> bool:
"""Cheap prefix check so the edge can route envelopes vs raw tokens without crypto."""
"""Cheap prefix check for an ACCESS envelope so the edge can route envelopes vs raw tokens without
crypto. A refresh envelope has a different prefix and is not an access envelope."""
return candidate.startswith(ENVELOPE_PREFIX)
def is_refresh_envelope(candidate: str) -> bool:
"""Cheap prefix check for a REFRESH envelope so the token endpoint can route a refresh grant that
carries an envelope vs a raw upstream refresh token without crypto."""
return candidate.startswith(REFRESH_ENVELOPE_PREFIX)
def mint_envelope(
identity: EnvelopeIdentity,
grant: UpstreamTokenGrant,
@ -231,23 +312,15 @@ def mint_envelope(
serialized envelope exceeds ``MAX_ENVELOPE_BYTES``.
"""
expires_at = now + timedelta(seconds=_envelope_ttl_seconds(grant.expires_in))
claims = _EnvelopeClaims(
iss=ENVELOPE_ISSUER,
iat=int(now.timestamp()),
exp=int(expires_at.timestamp()),
server_id=identity.server_id,
key_hash=identity.key_hash,
grant=_encrypt_grant_blob(_grant_plaintext(grant), keys.encryption_key),
return _seal(
kind="access",
prefix=ENVELOPE_PREFIX,
identity=identity,
grant_blob=_encrypt_grant_blob(_grant_plaintext(grant), keys.encryption_key),
expires_at=expires_at,
signing_key=keys.signing_key,
now=now,
)
token = ENVELOPE_PREFIX + jwt.encode(
claims.model_dump(),
keys.signing_key.get_secret_value(),
algorithm=_ENVELOPE_JWT_ALGORITHM,
)
size_bytes = len(token.encode("utf-8"))
if size_bytes > MAX_ENVELOPE_BYTES:
return EnvelopeTooLarge(size_bytes=size_bytes, max_bytes=MAX_ENVELOPE_BYTES)
return SealedEnvelope(token=SecretStr(token), expires_at=expires_at)
def open_envelope(
@ -263,35 +336,136 @@ def open_envelope(
re-derived, so it is stale by up to the envelope's lifetime; callers that need a
live remaining lifetime should use ``now`` against the upstream, not this field.
"""
if not is_envelope(candidate):
return NotAnEnvelope()
# UTF-8 byte length is never below character length, so a character count already over the
# cap rejects an oversize candidate in O(1) without encoding it; the exact byte check then
# runs only on candidates already bounded to <= MAX_ENVELOPE_BYTES characters.
if len(candidate) > MAX_ENVELOPE_BYTES:
return MalformedPayload()
if len(candidate.encode("utf-8", "surrogatepass")) > MAX_ENVELOPE_BYTES:
return MalformedPayload()
claims = _decode_claims(candidate.removeprefix(ENVELOPE_PREFIX), keys.signing_key)
claims = _open_claims(candidate, prefix=ENVELOPE_PREFIX, expected_kind="access", keys=keys, now=now)
if not isinstance(claims, _EnvelopeClaims):
return claims
if now.timestamp() >= claims.exp:
return Expired()
grant = _decrypt_grant(claims.grant, keys.encryption_key)
if not isinstance(grant, UpstreamTokenGrant):
return grant
return OpenedEnvelope(
identity=EnvelopeIdentity(server_id=claims.server_id, key_hash=claims.key_hash),
identity=EnvelopeIdentity(server_id=claims.server_id, subject_type=claims.subject_type, subject=claims.subject),
grant=grant,
)
def mint_refresh_envelope(
identity: EnvelopeIdentity,
refresh: RefreshCredential,
keys: EnvelopeKeys,
now: datetime,
) -> SealedEnvelope | EnvelopeMintError:
"""Seal ``refresh`` for ``identity`` into a long-lived, client-held refresh envelope.
``exp`` is ``min(refresh.expires_in, MAX_REFRESH_ENVELOPE_TTL_SECONDS)`` seconds from ``now`` (the
cap alone when the upstream omits the refresh lifetime). Sealing a distinct ``kind="refresh"`` claim
is what keeps a refresh envelope from ever opening as an access credential at the MCP edge. Returns
``EnvelopeTooLarge`` when the serialized envelope exceeds ``MAX_ENVELOPE_BYTES``.
"""
expires_at = now + timedelta(seconds=_refresh_ttl_seconds(refresh.expires_in))
return _seal(
kind="refresh",
prefix=REFRESH_ENVELOPE_PREFIX,
identity=identity,
grant_blob=_encrypt_grant_blob(_refresh_plaintext(refresh), keys.encryption_key),
expires_at=expires_at,
signing_key=keys.signing_key,
now=now,
)
def open_refresh_envelope(
candidate: str,
keys: EnvelopeKeys,
now: datetime,
) -> OpenedRefreshEnvelope | EnvelopeOpenError:
"""Validate a refresh ``candidate`` and recover the identity and inner refresh grant.
Total over hostile input exactly like :func:`open_envelope`: every invalid, expired, tampered,
wrong-kind, or undecryptable candidate maps to a distinct ``EnvelopeOpenError`` variant, never a
raise. The ``kind="refresh"`` claim is required, so an access envelope re-prefixed as a refresh one
is rejected as ``MalformedPayload``.
"""
claims = _open_claims(candidate, prefix=REFRESH_ENVELOPE_PREFIX, expected_kind="refresh", keys=keys, now=now)
if not isinstance(claims, _EnvelopeClaims):
return claims
refresh = _decrypt_refresh(claims.grant, keys.encryption_key)
if not isinstance(refresh, RefreshCredential):
return refresh
return OpenedRefreshEnvelope(
identity=EnvelopeIdentity(server_id=claims.server_id, subject_type=claims.subject_type, subject=claims.subject),
refresh=refresh,
)
def _seal(
kind: EnvelopeKind,
prefix: str,
identity: EnvelopeIdentity,
grant_blob: str,
expires_at: datetime,
signing_key: SecretStr,
now: datetime,
) -> SealedEnvelope | EnvelopeTooLarge:
"""Sign the claims for either envelope kind and enforce the size cap. Shared by both mints so the
JWT shape, issuer, and size guard cannot drift between access and refresh envelopes."""
claims = _EnvelopeClaims(
iss=ENVELOPE_ISSUER,
iat=int(now.timestamp()),
exp=int(expires_at.timestamp()),
kind=kind,
server_id=identity.server_id,
subject_type=identity.subject_type,
subject=identity.subject,
grant=grant_blob,
)
token = prefix + jwt.encode(claims.model_dump(), signing_key.get_secret_value(), algorithm=_ENVELOPE_JWT_ALGORITHM)
size_bytes = len(token.encode("utf-8"))
if size_bytes > MAX_ENVELOPE_BYTES:
return EnvelopeTooLarge(size_bytes=size_bytes, max_bytes=MAX_ENVELOPE_BYTES)
return SealedEnvelope(token=SecretStr(token), expires_at=expires_at)
def _open_claims(
candidate: str,
prefix: str,
expected_kind: EnvelopeKind,
keys: EnvelopeKeys,
now: datetime,
) -> _EnvelopeClaims | EnvelopeOpenError:
"""Prefix-route, size-bound, signature-verify, kind-check, and expiry-check an attacker-controlled
candidate, shared by both openers so the security gate is identical for access and refresh. Returns
the validated claims or a distinct ``EnvelopeOpenError``; never raises."""
if not candidate.startswith(prefix):
return NotAnEnvelope()
# UTF-8 byte length is never below character length, so a character count already over the cap
# rejects an oversize candidate in O(1) without encoding it; the exact byte check then runs only on
# candidates already bounded to <= MAX_ENVELOPE_BYTES characters.
if len(candidate) > MAX_ENVELOPE_BYTES:
return MalformedPayload()
if len(candidate.encode("utf-8", "surrogatepass")) > MAX_ENVELOPE_BYTES:
return MalformedPayload()
claims = _decode_claims(candidate.removeprefix(prefix), keys.signing_key)
if not isinstance(claims, _EnvelopeClaims):
return claims
if claims.kind != expected_kind:
return MalformedPayload()
if now.timestamp() >= claims.exp:
return Expired()
return claims
def _envelope_ttl_seconds(upstream_expires_in: int | None) -> int:
if upstream_expires_in is None:
return MAX_ENVELOPE_TTL_SECONDS
return min(upstream_expires_in, MAX_ENVELOPE_TTL_SECONDS)
def _refresh_ttl_seconds(upstream_refresh_expires_in: int | None) -> int:
if upstream_refresh_expires_in is None:
return MAX_REFRESH_ENVELOPE_TTL_SECONDS
return min(upstream_refresh_expires_in, MAX_REFRESH_ENVELOPE_TTL_SECONDS)
def _grant_plaintext(grant: UpstreamTokenGrant) -> str:
wire = _GrantWire(
access_token=grant.access_token.get_secret_value(),
@ -303,6 +477,15 @@ def _grant_plaintext(grant: UpstreamTokenGrant) -> str:
return wire.model_dump_json(exclude_none=True)
def _refresh_plaintext(refresh: RefreshCredential) -> str:
wire = _RefreshWire(
refresh_token=refresh.refresh_token.get_secret_value(),
scope=refresh.scope,
expires_in=refresh.expires_in,
)
return wire.model_dump_json(exclude_none=True)
def _decode_claims(
compact: str,
signing_key: SecretStr,
@ -364,3 +547,22 @@ def _decrypt_grant(
return UpstreamTokenGrant.model_validate_json(plaintext)
except ValidationError:
return MalformedPayload()
def _decrypt_refresh(
blob: str,
encryption_key: SecretStr,
) -> RefreshCredential | DecryptFailed | MalformedPayload:
from nacl.exceptions import CryptoError
try:
plaintext = decrypt_value(
value=base64.urlsafe_b64decode(blob),
signing_key=encryption_key.get_secret_value(),
)
except (CryptoError, ValueError):
return DecryptFailed()
try:
return RefreshCredential.model_validate_json(plaintext)
except ValidationError:
return MalformedPayload()

105
litellm/proxy/_experimental/mcp_server/semantic_tool_filter.py
View file

@ -4,6 +4,7 @@ Semantic MCP Tool Filtering using semantic-router
Filters MCP tools semantically for /chat/completions and /responses endpoints.
"""
import asyncio
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from litellm._logging import verbose_logger
@ -76,6 +77,7 @@ class SemanticMCPToolFilter:
self.tool_router: Optional["SemanticRouter"] = None
self.context_window_error: Optional[str] = None
self._tool_map: Dict[str, Any] = {} # MCPTool objects or OpenAI function dicts
self._index_sync_lock = asyncio.Lock()
async def build_router_from_mcp_registry(self) -> None:
"""Build semantic router from all MCP tools in the registry (no auth checks)."""
@ -182,6 +184,81 @@ class SemanticMCPToolFilter:
return
raise
def _has_tools_missing_from_index(self, tools: list[Any]) -> bool:
"""Allocation-free check for any named tool not yet in the semantic index."""
return any(name and name not in self._tool_map for name in (self._extract_tool_info(t)[0] for t in tools))
def _tools_missing_from_index(self, tools: list[Any]) -> dict[str, Any]:
"""Map name -> tool for every named tool not yet in the semantic index."""
return {
name: tool
for name, tool in ((self._extract_tool_info(t)[0], t) for t in tools)
if name and name not in self._tool_map
}
async def _ensure_tools_indexed(self, available_tools: list[Any]) -> None:
"""
Index request-time tools the startup build never saw.
The startup index lists every registered MCP server WITHOUT per-user
credentials, so servers requiring per-user auth (interactive OAuth
tokens, user-scoped env vars) contribute zero routes. Tools reaching
the filter came through an authenticated expansion; without indexing
them here they can never be selected, so requests either bypass
filtering entirely (N->N) or lose every tool to unrelated matches.
Runs async-only (no synchronous embedding on the request path) and
never writes shared error state: an embedding failure here raises and
is scoped to the requesting call, so one request's oversized tool
description cannot poison the filter for other users on the worker.
"""
from semantic_router.routers import SemanticRouter
from semantic_router.routers.base import Route
from litellm.router_strategy.auto_router.litellm_encoder import (
LiteLLMRouterEncoder,
)
if not self._has_tools_missing_from_index(available_tools):
return
async with self._index_sync_lock:
missing = self._tools_missing_from_index(available_tools)
if not missing:
return
descriptions = {name: self._extract_tool_info(tool)[1] for name, tool in missing.items()}
routes = [
Route(
name=name,
description=description,
utterances=[description],
score_threshold=self.similarity_threshold,
)
for name, description in descriptions.items()
]
if self.tool_router is None:
router = SemanticRouter(
routes=[],
encoder=LiteLLMRouterEncoder(
litellm_router_instance=self.router_instance,
model_name=self.embedding_model,
score_threshold=self.similarity_threshold,
),
auto_sync="local",
top_k=self.top_k,
)
await router.aadd(routes)
self.tool_router = router
else:
await self.tool_router.aadd(routes)
self._tool_map.update(missing)
verbose_logger.info(
f"Semantic tool filter indexed {len(routes)} request-time tools missing from the startup index"
)
async def filter_tools(
self,
query: str,
@ -216,22 +293,34 @@ class SemanticMCPToolFilter:
if not query or not query.strip():
return available_tools
# Router should be built on startup - if not, something went wrong
if self.tool_router is None:
verbose_logger.warning("Router not initialized - was build_router_from_mcp_registry() called on startup?")
return available_tools
# Run semantic filtering
try:
await self._ensure_tools_indexed(available_tools)
if self.tool_router is None:
verbose_logger.warning("Semantic router could not be built from the request's tools")
return available_tools
available_names = [name for name in (self._extract_tool_info(t)[0] for t in available_tools) if name]
if not available_names:
return available_tools
limit = top_k or self.top_k
matches = self.tool_router(text=query, limit=limit)
if self.tool_router.top_k < limit:
self.tool_router.top_k = limit
matches = self.tool_router(text=query, limit=limit, route_filter=available_names)
matched_tool_names = self._extract_tool_names_from_matches(matches)
if not matched_tool_names:
return available_tools
return self._get_tools_by_names(matched_tool_names, available_tools)
filtered_tools = self._get_tools_by_names(matched_tool_names, available_tools)
if not filtered_tools:
return available_tools
return filtered_tools
except SemanticToolFilterContextWindowError:
raise
except Exception as e:
if _is_context_window_error(e):
verbose_logger.error(
@ -240,7 +329,7 @@ class SemanticMCPToolFilter:
)
raise SemanticToolFilterContextWindowError(
embedding_model=self.embedding_model,
stage="the user query",
stage="the user query or the MCP tool descriptions being indexed",
original_error=str(e),
) from e
verbose_logger.error(f"Semantic tool filter failed: {e}", exc_info=True)

121
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -3719,8 +3719,15 @@ if MCP_AVAILABLE:
headers={"www-authenticate": upstream_www_authenticate},
)
def _get_authorization_header_from_scope(scope: Scope) -> Optional[str]:
"""First ``Authorization`` header value in the ASGI scope, or None."""
for key, value in scope.get("headers", []):
if key.lower() == b"authorization":
return value.decode("latin-1")
return None
def _scope_has_authorization_header(scope: Scope) -> bool:
return any(key.lower() == b"authorization" for key, _ in scope.get("headers", []))
return _get_authorization_header_from_scope(scope) is not None
def _get_forwarded_auth_from_scope(scope: Scope) -> Optional[str]:
"""Return the upstream-bound ``Authorization`` header value, or None.
@ -3733,17 +3740,24 @@ if MCP_AVAILABLE:
``MCPRequestHandler.process_mcp_request``), and forwarding it upstream
would leak the proxy key to a third-party MCP server.
"""
authorization = None
has_litellm_key_header = False
for key, value in scope.get("headers", []):
key_lower = key.lower()
if key_lower == b"authorization":
authorization = value.decode("latin-1")
elif key_lower == b"x-litellm-api-key":
has_litellm_key_header = True
has_litellm_key_header = any(key.lower() == b"x-litellm-api-key" for key, _ in scope.get("headers", []))
if not has_litellm_key_header:
return None
return authorization
return _get_authorization_header_from_scope(scope)
def _is_delegate_upstream_probe_target(server: MCPServer) -> bool:
"""Whether ``server`` is an interactive delegate-auth server whose client-supplied
token should be preflighted upstream.
Mirrors the anonymous-delegate gate in ``get_allowed_mcp_servers``: the flow is
resolved via ``effective_oauth2_flow`` so an unstamped M2M-shape row fails closed
(its stored client credentials drive egress; the caller's bearer is irrelevant).
"""
return (
server.auth_type == MCPAuth.oauth2
and server.delegate_auth_to_upstream is True
and MCPServerManager.effective_oauth2_flow(server) != "client_credentials"
)
async def _probe_upstream_auth(
url: str,
@ -3805,7 +3819,7 @@ if MCP_AVAILABLE:
mcp_servers: Optional[List[str]],
client_ip: Optional[str],
) -> None:
"""Probe pass-through upstream servers in parallel before the MCP session starts.
"""Probe pass-through and delegate-auth upstream servers in parallel before the MCP session starts.
Only servers the caller's key is already authorized to reach are probed —
the list is derived from _get_allowed_mcp_servers so that a user cannot
@ -3813,11 +3827,42 @@ if MCP_AVAILABLE:
The MCP SDK commits HTTP 200 headers before invoking handlers, so a 401
can only be returned before that point. This function raises HTTPException(401)
with a WWW-Authenticate header if any upstream rejects the client token.
with a WWW-Authenticate header if any upstream rejects the client token, or 403
if the upstream accepts it but forbids the caller.
Fails-open: network errors are logged and the request is allowed through.
Delegate-auth servers (``auth_type=oauth2`` + ``delegate_auth_to_upstream``)
are probed with the caller's bare ``Authorization`` bearer. That bearer is only
an upstream token (never a LiteLLM key) when admission took the delegate bypass,
so the delegate target is resolved through ``get_mcp_server_by_name`` -- the same
resolver admission used -- rather than the wider allowed-server prefix/access-group
matching. A name that only reaches a delegate server via server_id or an access
group would have been admitted as a real LiteLLM key, so probing it would leak that
key upstream; requiring the admission-resolver match closes that gap. Without the
probe a rejected token is absorbed by the tools/list handler and masked as an empty
tool list. Gated to single-server routes so one rejected token cannot 401 a
multi-server aggregate connect, matching the OBO preflight gating; the challenge
echoes the requested name so aliased routes get the same resource_metadata URL as
the tokenless preemptive challenge.
"""
forwarded_auth = _get_forwarded_auth_from_scope(scope)
if not forwarded_auth:
requested_single_target = mcp_servers[0] if mcp_servers is not None and len(mcp_servers) == 1 else None
# The bare Authorization header (no x-litellm-api-key) is a valid upstream token
# only when admission classified it as one, i.e. the single requested name resolves
# to a delegate server under admission's own resolver. Resolve it the same way here
# so a server_id- or access-group-named delegate (which admission would have treated
# as a LiteLLM key) is never probed with that key.
delegate_server = (
global_mcp_server_manager.get_mcp_server_by_name(requested_single_target, client_ip=client_ip)
if requested_single_target
else None
)
delegate_auth = (
_get_authorization_header_from_scope(scope)
if delegate_server is not None and _is_delegate_upstream_probe_target(delegate_server)
else None
)
if not forwarded_auth and not delegate_auth:
return
# Use the authorized server set, not the raw user-supplied names, so that
@ -3827,33 +3872,49 @@ if MCP_AVAILABLE:
mcp_servers=mcp_servers,
client_ip=client_ip,
)
passthrough_servers = [
srv
for srv in allowed_servers
# Restrict to genuine OAuth pass-through servers (auth_type none +
# Authorization in extra_headers). Gateway-managed OAuth2 servers
# must not receive the ``resource_metadata=`` challenge emitted
# below — they require ``authorization_uri=`` pointing at the
# gateway AS metadata. ``is_oauth_passthrough`` already requires
# ``auth_type in (None, MCPAuth.none)``, which is mutually
# exclusive with ``has_client_credentials`` (oauth2 + M2M flow),
# so M2M servers are implicitly excluded here.
if srv.is_oauth_passthrough
]
if not passthrough_servers:
passthrough_targets: Tuple[Tuple[MCPServer, str, str], ...] = (
tuple(
(srv, forwarded_auth, srv.name)
for srv in allowed_servers
# Restrict to genuine OAuth pass-through servers (auth_type none +
# Authorization in extra_headers). Gateway-managed OAuth2 servers
# must not receive the ``resource_metadata=`` challenge emitted
# below — they require ``authorization_uri=`` pointing at the
# gateway AS metadata. ``is_oauth_passthrough`` already requires
# ``auth_type in (None, MCPAuth.none)``, which is mutually
# exclusive with ``has_client_credentials`` (oauth2 + M2M flow),
# so M2M servers are implicitly excluded here.
if srv.is_oauth_passthrough
)
if forwarded_auth
else ()
)
# Probe the admission-resolved delegate server only when the caller is actually
# authorized for it (present in the IP-filtered allowed set), keyed by server_id.
delegate_targets: Tuple[Tuple[MCPServer, str, str], ...] = (
tuple(
(srv, delegate_auth, requested_single_target)
for srv in allowed_servers
if delegate_server is not None and srv.server_id == delegate_server.server_id
)
if delegate_auth and requested_single_target
else ()
)
probe_targets = passthrough_targets + delegate_targets
if not probe_targets:
return
probe_results = await asyncio.gather(
*[_probe_upstream_auth(srv.url or "", forwarded_auth) for srv in passthrough_servers]
*[_probe_upstream_auth(srv.url or "", auth_header) for srv, auth_header, _ in probe_targets]
)
for srv, (probe_status, _) in zip(passthrough_servers, probe_results):
for (srv, _, challenge_server_name), (probe_status, _) in zip(probe_targets, probe_results):
if probe_status == 401:
# Token is missing or expired: keep pass-through clients on the
# protected-resource discovery flow so they re-authorize against
# the upstream IdP metadata proxied by LiteLLM.
www_authenticate = _get_passthrough_www_authenticate(
scope=scope,
server_name=srv.name,
server_name=challenge_server_name,
invalid_token=True,
)
raise HTTPException(

2
litellm/proxy/_experimental/out/404.html
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/404/index.html
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.__PAGE__.txt
View file

@ -1,9 +1,9 @@
1:"$Sreact.fragment"
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0uuigwiz-in3~.js","/litellm-asset-prefix/_next/static/chunks/0-0c4mv4-mc9n.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0p3v32gvsxp6h.js","/litellm-asset-prefix/_next/static/chunks/0x09ws363q4_0.js","/litellm-asset-prefix/_next/static/chunks/08o64zaid_juv.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/0g9k1~ppf2hw3.js","/litellm-asset-prefix/_next/static/chunks/151r-5htw45m~.js","/litellm-asset-prefix/_next/static/chunks/0v85l0arelm41.js","/litellm-asset-prefix/_next/static/chunks/0zbgu4ogb6mba.js","/litellm-asset-prefix/_next/static/chunks/0ae3np_qb52e-.js","/litellm-asset-prefix/_next/static/chunks/0zam.8alu6_vj.js","/litellm-asset-prefix/_next/static/chunks/0uu6lckpr0s15.js","/litellm-asset-prefix/_next/static/chunks/0.bx44y-6~tug.js","/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","/litellm-asset-prefix/_next/static/chunks/0-s2am3eulbyd.js","/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","/litellm-asset-prefix/_next/static/chunks/0.w8~sa9q0n_s.js","/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js","/litellm-asset-prefix/_next/static/chunks/0efmbzvj03niy.js","/litellm-asset-prefix/_next/static/chunks/055egae-ggkjh.js","/litellm-asset-prefix/_next/static/chunks/0hwip5a7qsmis.js","/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","/litellm-asset-prefix/_next/static/chunks/0mh1wnrvmv_y7.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"OutletBoundary"]
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js","/litellm-asset-prefix/_next/static/chunks/0kc37~1yrtr2p.js","/litellm-asset-prefix/_next/static/chunks/04s-iyzsr4cq~.js","/litellm-asset-prefix/_next/static/chunks/16zj68af4snfa.js","/litellm-asset-prefix/_next/static/chunks/003_1s9xbht43.js","/litellm-asset-prefix/_next/static/chunks/0tmaomqtwbi33.js","/litellm-asset-prefix/_next/static/chunks/02-2~p5k.ielz.js","/litellm-asset-prefix/_next/static/chunks/0op63kdo3uwng.js","/litellm-asset-prefix/_next/static/chunks/112_-0alpxot8.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/12yfh0_n50ojz.js","/litellm-asset-prefix/_next/static/chunks/06w8_.601z7_i.js","/litellm-asset-prefix/_next/static/chunks/0nht59ws0elww.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/18187o3gb9vc5.js","/litellm-asset-prefix/_next/static/chunks/0._ir~nvcseg7.js","/litellm-asset-prefix/_next/static/chunks/0vffq7buvlg04.js","/litellm-asset-prefix/_next/static/chunks/122djf0bncn-8.js","/litellm-asset-prefix/_next/static/chunks/0axk76owb7jv..js","/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","/litellm-asset-prefix/_next/static/chunks/0sqw622fcvsv4.js","/litellm-asset-prefix/_next/static/chunks/0.bx44y-6~tug.js","/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","/litellm-asset-prefix/_next/static/chunks/14g~hmf3h_efw.js","/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","/litellm-asset-prefix/_next/static/chunks/0cmepm.jkel-i.js","/litellm-asset-prefix/_next/static/chunks/0onea0n77pqw1.js","/litellm-asset-prefix/_next/static/chunks/02-u6qtmsnqn0.js","/litellm-asset-prefix/_next/static/chunks/0mh1wnrvmv_y7.js","/litellm-asset-prefix/_next/static/chunks/05q6y.kb.q2s..js","/litellm-asset-prefix/_next/static/chunks/0rehsq9xe1kde.js","/litellm-asset-prefix/_next/static/chunks/0nk7-_~gcxbz0.js","/litellm-asset-prefix/_next/static/chunks/0~r95y0t-0dlp.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"OutletBoundary"]
7:"$Sreact.suspense"
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0ae3np_qb52e-.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0zam.8alu6_vj.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0uu6lckpr0s15.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0.bx44y-6~tug.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/0-s2am3eulbyd.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/0.w8~sa9q0n_s.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/0efmbzvj03niy.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/055egae-ggkjh.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/0hwip5a7qsmis.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/0mh1wnrvmv_y7.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"KYqiq5stbD-H4YcZ-6OuP"}
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0vffq7buvlg04.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/122djf0bncn-8.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0axk76owb7jv..js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0sqw622fcvsv4.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/0.bx44y-6~tug.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/14g~hmf3h_efw.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/0cmepm.jkel-i.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/0onea0n77pqw1.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/02-u6qtmsnqn0.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/0mh1wnrvmv_y7.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/05q6y.kb.q2s..js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/0rehsq9xe1kde.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/0nk7-_~gcxbz0.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/0~r95y0t-0dlp.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"N7WCdfNd30Hp6HEF5tFIL"}
4:{}
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
8:null

10
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.txt
View file

@ -1,7 +1,7 @@
1:"$Sreact.fragment"
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0uuigwiz-in3~.js","/litellm-asset-prefix/_next/static/chunks/0-0c4mv4-mc9n.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0p3v32gvsxp6h.js","/litellm-asset-prefix/_next/static/chunks/0x09ws363q4_0.js","/litellm-asset-prefix/_next/static/chunks/08o64zaid_juv.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/0g9k1~ppf2hw3.js","/litellm-asset-prefix/_next/static/chunks/151r-5htw45m~.js","/litellm-asset-prefix/_next/static/chunks/0v85l0arelm41.js","/litellm-asset-prefix/_next/static/chunks/0zbgu4ogb6mba.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0uuigwiz-in3~.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0-0c4mv4-mc9n.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0p3v32gvsxp6h.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0x09ws363q4_0.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/08o64zaid_juv.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0g9k1~ppf2hw3.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/151r-5htw45m~.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/0v85l0arelm41.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/0zbgu4ogb6mba.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"KYqiq5stbD-H4YcZ-6OuP"}
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js","/litellm-asset-prefix/_next/static/chunks/0kc37~1yrtr2p.js","/litellm-asset-prefix/_next/static/chunks/04s-iyzsr4cq~.js","/litellm-asset-prefix/_next/static/chunks/16zj68af4snfa.js","/litellm-asset-prefix/_next/static/chunks/003_1s9xbht43.js","/litellm-asset-prefix/_next/static/chunks/0tmaomqtwbi33.js","/litellm-asset-prefix/_next/static/chunks/02-2~p5k.ielz.js","/litellm-asset-prefix/_next/static/chunks/0op63kdo3uwng.js","/litellm-asset-prefix/_next/static/chunks/112_-0alpxot8.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/12yfh0_n50ojz.js","/litellm-asset-prefix/_next/static/chunks/06w8_.601z7_i.js","/litellm-asset-prefix/_next/static/chunks/0nht59ws0elww.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/18187o3gb9vc5.js","/litellm-asset-prefix/_next/static/chunks/0._ir~nvcseg7.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0kc37~1yrtr2p.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/04s-iyzsr4cq~.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/16zj68af4snfa.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/003_1s9xbht43.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0tmaomqtwbi33.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/02-2~p5k.ielz.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/0op63kdo3uwng.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/112_-0alpxot8.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/12yfh0_n50ojz.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/06w8_.601z7_i.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/0nht59ws0elww.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/18187o3gb9vc5.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/0._ir~nvcseg7.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"N7WCdfNd30Hp6HEF5tFIL"}
6:"$0:rsc:props:children:1:props:serverProvidedParams:params"

55
litellm/proxy/_experimental/out/__next._full.txt
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/__next._head.txt
View file

@ -1,6 +1,6 @@
1:"$Sreact.fragment"
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"MetadataBoundary"]
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"MetadataBoundary"]
4:"$Sreact.suspense"
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.0~dgapwhi~75y.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"KYqiq5stbD-H4YcZ-6OuP"}
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.0~dgapwhi~75y.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"N7WCdfNd30Hp6HEF5tFIL"}

14
litellm/proxy/_experimental/out/__next._index.txt
View file

@ -1,9 +1,9 @@
1:"$Sreact.fragment"
2:I[867271,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
3:I[71195,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
4:I[557951,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"AuthProvider"]
5:I[339756,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
6:I[837457,["/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
2:I[867271,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
3:I[71195,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
4:I[557951,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"AuthProvider"]
5:I[339756,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
6:I[837457,["/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js"],"default"]
:HL["/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/075sund.-mh4~.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/075sund.-mh4~.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/08yy42xvwaak6.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0e9hs7onyj28m.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","template":["$","$L6",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"KYqiq5stbD-H4YcZ-6OuP"}
:HL["/litellm-asset-prefix/_next/static/chunks/15j3hwz2dxrik.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/15j3hwz2dxrik.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/09qhs_ev_bxr8.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0a6iga_s7xld1.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0hpxif-db_y5-.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","template":["$","$L6",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"N7WCdfNd30Hp6HEF5tFIL"}

Some files were not shown because too many files have changed in this diff Show more