scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-17 04:34:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 265

Merge remote-tracking branch 'upstream/litellm_internal_staging' into feat/router-ttft-timeout

Browse source
This commit is contained in:
Nathan Price 2026-07-10 08:06:14 -05:00
commit 2b8b488b67
477 changed files with 15870 additions and 3808 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

48
.github/actions/detect-backend-changes/action.yml vendored Normal file
View file

@ -0,0 +1,48 @@
name: "Detect backend-relevant changes"
description: >-
Classify the pull request's changed files with .circleci/scripts/classify_changes.sh
and expose decision=run|skip. decision=skip means only ui/**, **.md or **.mdx files
changed, so callers can short-circuit expensive steps while the job still completes
successfully and satisfies its required status check. The decision defaults to run for
any non pull_request event or whenever the changed set cannot be resolved, so tests are
never skipped when the classification is uncertain.
outputs:
decision:
description: "run when backend-relevant files changed, otherwise skip"
value: ${{ steps.classify.outputs.decision }}
runs:
using: composite
steps:
- id: classify
shell: bash
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -uo pipefail
if [ -z "${BASE_SHA:-}" ]; then
echo "detect-backend-changes: not a pull_request event; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
fi
if ! git fetch --no-tags --depth=1 origin "${BASE_SHA}" >/dev/null 2>&1; then
echo "detect-backend-changes: could not fetch base ${BASE_SHA}; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
fi
changed="$(git diff --name-only "${BASE_SHA}" HEAD 2>/dev/null)" || {
echo "detect-backend-changes: git diff failed; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
}
if [ -z "${changed}" ]; then
echo "detect-backend-changes: no changed files vs ${BASE_SHA}; skipping job"
echo "decision=skip" >> "${GITHUB_OUTPUT}"
exit 0
fi
echo "detect-backend-changes: changed files vs ${BASE_SHA}:"
printf '%s\n' "${changed}" | sed 's/^/ /'
decision="$(printf '%s\n' "${changed}" | bash .circleci/scripts/classify_changes.sh backend)" || decision="run"
echo "detect-backend-changes: decision=${decision}"
echo "decision=${decision}" >> "${GITHUB_OUTPUT}"

13
.github/workflows/_test-unit-base.yml vendored
View file

@ -45,12 +45,18 @@ jobs:
name: Run tests
runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.timeout-minutes }}
outputs:
decision: ${{ steps.changes.outputs.decision }}
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
@ -72,16 +78,19 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Run tests
if: steps.changes.outputs.decision != 'skip'
env:
TEST_PATH: ${{ inputs.test-path }}
MAX_FAILURES: ${{ inputs.max-failures }}
@ -114,7 +123,7 @@ jobs:
fi
- name: Save coverage report
if: always()
if: always() && steps.changes.outputs.decision != 'skip'
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: coverage-${{ inputs.artifact-name }}-${{ github.run_id }}-${{ github.run_attempt }}
@ -124,7 +133,7 @@ jobs:
upload-coverage:
name: Upload coverage to Codecov
needs: run
if: always()
if: always() && needs.run.outputs.decision != 'skip'
runs-on: ubuntu-latest
permissions:
contents: read

61
.github/workflows/create_daily_oss_branch.yml vendored Normal file
View file

@ -0,0 +1,61 @@
name: Create Daily OSS Branch
on:
schedule:
- cron: "0 16 * * 1-5" # 9am PT during daylight saving time, weekdays.
workflow_dispatch:
inputs:
date:
description: "Branch date in YYYY_MM_DD format. Defaults to today's UTC date."
required: false
type: string
permissions:
contents: write
jobs:
create-oss-branch:
if: github.repository == 'BerriAI/litellm'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
persist-credentials: false
- name: Create dated OSS branch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REQUESTED_DATE: ${{ inputs.date }}
run: |
set -euo pipefail
if [ -n "${REQUESTED_DATE}" ]; then
if ! echo "${REQUESTED_DATE}" | grep -Eq '^[0-9]{4}_[0-9]{2}_[0-9]{2}$'; then
echo "::error::date must use YYYY_MM_DD format, got '${REQUESTED_DATE}'"
exit 1
fi
BRANCH_DATE="${REQUESTED_DATE}"
else
BRANCH_DATE="$(date -u +'%Y_%m_%d')"
fi
BRANCH_NAME="litellm_oss_daily_${BRANCH_DATE}"
echo "Creating branch: ${BRANCH_NAME}"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git fetch origin main "${BRANCH_NAME}" || true
if git show-ref --verify --quiet "refs/remotes/origin/${BRANCH_NAME}"; then
echo "Branch ${BRANCH_NAME} already exists. Skipping creation."
exit 0
fi
git checkout -b "${BRANCH_NAME}" origin/main
git push "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "${BRANCH_NAME}"
echo "Successfully created and pushed branch: ${BRANCH_NAME}"

50
.github/workflows/oss_daily_guardrails.yml vendored Normal file
View file

@ -0,0 +1,50 @@
name: OSS Daily Guardrails
on:
push:
branches:
- "litellm_oss_daily_20*"
pull_request:
branches:
- "litellm_oss_daily_20*"
- litellm_internal_staging
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
oss-safe-checks:
name: Run OSS daily safe checks
if: startsWith(github.ref_name, 'litellm_oss_daily_20') || startsWith(github.head_ref, 'litellm_oss_daily_20') || startsWith(github.base_ref, 'litellm_oss_daily_20')
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
- name: Run secret scan test
run: |
uv run --frozen --with 'pytest==9.0.2' pytest tests/litellm/test_no_hardcoded_secrets.py -v
- name: Run Ruff
run: |
uv sync --frozen
cd litellm
uv run --no-sync ruff check .

4
.github/workflows/test-unit-core-utils.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

11
.github/workflows/test-unit-documentation.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read
@ -36,6 +32,10 @@ jobs:
path: docs/my-website
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
@ -57,10 +57,12 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
@ -68,6 +70,7 @@ jobs:
# Run the same documentation tests that CircleCI ran (as direct Python scripts)
- name: Run documentation validation tests
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync python ./tests/documentation_tests/test_env_keys.py
uv run --no-sync python ./tests/documentation_tests/test_router_settings.py

4
.github/workflows/test-unit-enterprise-routing.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-integrations.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-llm-providers.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-misc.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-proxy-auth.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-proxy-db.yml vendored
View file

@ -5,10 +5,6 @@ on:
branches:
- main
- litellm_internal_staging
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

4
.github/workflows/test-unit-proxy-endpoints.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
workflow_dispatch:
permissions:

4
.github/workflows/test-unit-proxy-infra.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

11
.github/workflows/test-unit-proxy-legacy.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read
@ -53,6 +49,10 @@ jobs:
with:
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
@ -74,16 +74,19 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Run tests - ${{ matrix.test-group.name }}
if: steps.changes.outputs.decision != 'skip'
env:
TEST_PATH: ${{ matrix.test-group.path }}
run: |

4
.github/workflows/test-unit-responses-caching-types.yml vendored
View file

@ -7,10 +7,6 @@ on:
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths-ignore:
- "ui/**"
- "**.md"
- "**.mdx"
permissions:
contents: read

2
enterprise/README.md
View file

@ -6,4 +6,4 @@ Code in this folder is licensed under a commercial license. Please review the [L
👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://enterprise.litellm.ai/demo?month=2024-02)
See all Enterprise Features here 👉 [Docs](https://docs.litellm.ai/docs/proxy/enterprise)
See all Enterprise Features here 👉 [Docs](https://docs.litellm.ai/docs/enterprise)

4
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
View file

@ -919,9 +919,9 @@ class BaseEmailLogger(CustomLogger):
"""
Construct invitation link for the user
# http://localhost:4000/ui?invitation_id=7a096b3a-37c6-440f-9dd1-ba22e8043f6b
# http://localhost:4000/ui/onboarding?invitation_id=7a096b3a-37c6-440f-9dd1-ba22e8043f6b
"""
return f"{base_url}/ui?invitation_id={invitation_id}"
return f"{base_url}/ui/onboarding?invitation_id={invitation_id}"
async def send_email(
self,

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.48"
version = "0.1.49"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.48"
version = "0.1.49"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

2
litellm/anthropic_beta_headers_config.json
View file

@ -102,7 +102,7 @@
"computer-use-2025-01-24": "computer-use-2025-01-24",
"computer-use-2025-11-24": "computer-use-2025-11-24",
"context-1m-2025-08-07": "context-1m-2025-08-07",
"context-management-2025-06-27": null,
"context-management-2025-06-27": "context-management-2025-06-27",
"effort-2025-11-24": "effort-2025-11-24",
"fast-mode-2026-02-01": null,
"files-api-2025-04-14": null,

2
litellm/constants.py
View file

@ -715,6 +715,7 @@ openai_compatible_endpoints: List = [
"https://api.clarifai.com/v2/ext/openai/v1",
"https://api.libertai.io/v1",
"https://pinstripes.io/v1",
"https://api.meta.ai/v1",
]
@ -781,6 +782,7 @@ openai_compatible_providers: List = [
"ragflow",
"pinstripes", # Pinstripes - JSON-configured provider
"darkbloom",
"meta", # Meta Model API (Muse Spark) - JSON-configured provider
]
openai_text_completion_compatible_providers: List = [ # providers that support `/v1/completions`
"together_ai",

14
litellm/exceptions.py
View file

@ -1180,20 +1180,6 @@ class ModifyResponseException(Exception):
super().__init__(message)
class GuardrailInterventionNormalStringError(
Exception
): # custom exception to raise when a guardrail intervenes, but we want to return a normal string to the user
def __init__(self, message: str):
self.message = message
super().__init__(self.message)
def __str__(self):
return self.message
def __repr__(self):
return self.__str__()
class SensitiveDataRouteException(Exception):
"""
Exception raised when a guardrail detects sensitive data and wants to reroute the request.

6
litellm/integrations/custom_guardrail.py
View file

@ -757,6 +757,12 @@ class CustomGuardrail(CustomLogger):
# raw provider JSON so redaction is not duplicated upstream).
clean_guardrail_response = redact_nested_match_and_regex_keys(clean_guardrail_response)
from litellm.litellm_core_utils.sensitive_data_masker import (
mask_credentials_in_payload,
)
clean_guardrail_response = mask_credentials_in_payload(clean_guardrail_response)
slg = StandardLoggingGuardrailInformation(
guardrail_name=self.guardrail_name,
guardrail_provider=guardrail_provider,

2
litellm/integrations/otel/__init__.py
View file

@ -52,6 +52,7 @@ from litellm.integrations.otel.model.semconv import (
GenAIProvider,
JsonRpc,
LiteLLM,
LiteLLMError,
MCPMethod,
Metric,
Network,
@ -87,6 +88,7 @@ __all__ = [
"HTTP",
"JsonRpc",
"LiteLLM",
"LiteLLMError",
"MCP",
"MCPMethod",
"Metric",

35
litellm/integrations/otel/emitter.py
View file

@ -16,9 +16,10 @@ from litellm.integrations.otel.model.payloads import (
MCPListToolsSpanData,
MCPToolCallSpanData,
ServiceSpanData,
SpanError,
)
from litellm.integrations.otel.plumbing.providers import to_otel_span_kind
from litellm.integrations.otel.model.semconv import Error, ExceptionEvent
from litellm.integrations.otel.model.semconv import Error, ExceptionEvent, LiteLLMError
from litellm.integrations.otel.model.spans import (
SPAN_REGISTRY,
SpanRole,
@ -49,6 +50,27 @@ _NAME_BUILDERS: dict[SpanRole, Callable[..., str]] = {
_DEDUP_CACHE_MAX = 10_000
def _stamp_otel_error_attributes(span: Span, error_type: str, resolved_message: str) -> None:
"""Stamp the OTel-semconv error attributes (``error.type`` + ``error.message``).
``error_type`` and ``resolved_message`` are ``finish_span``'s already-computed
fallback chains, so the pair on the status, event, and attributes stays in
lockstep."""
span.set_attribute(Error.TYPE, error_type)
span.set_attribute(Error.MESSAGE, resolved_message)
def _stamp_litellm_error_attributes(span: Span, error: SpanError) -> None:
"""Stamp litellm-specific error detail attributes. Emitted only when the
corresponding field is populated so guardrail-shape errors carrying only a
message aren't polluted with empty detail keys."""
if error.code:
span.set_attribute(LiteLLMError.CODE, error.code)
if error.stack_trace:
span.set_attribute(LiteLLMError.STACK_TRACE, error.stack_trace)
if error.llm_provider:
span.set_attribute(LiteLLMError.LLM_PROVIDER, error.llm_provider)
class SpanEmitter:
def __init__(
self,
@ -190,12 +212,13 @@ class SpanEmitter:
if error and (error.error_type or error.message):
error_type = error.error_type or "error"
message = error.message or error.error_type or "error"
span.set_attribute(Error.TYPE, error_type)
_stamp_otel_error_attributes(span, error_type, message)
_stamp_litellm_error_attributes(span, error)
span.set_status(Status(StatusCode.ERROR, message))
# Carry the full message on the standard ``exception`` event so backends
# map it as full text under ``exception.message``. Setting it as a bare
# string attribute instead lets backends like Elasticsearch dynamic-map
# it to a ``keyword`` capped at 1024 chars, truncating the message.
# Also emit the semconv ``exception`` event so backends that
# dynamic-map unknown string span attrs to ``keyword`` (e.g.
# Elasticsearch with a 1024-char ``ignore_above``) still see the
# full untruncated message on the recognized event field.
span.add_event(
ExceptionEvent.NAME,
{ExceptionEvent.TYPE: error_type, ExceptionEvent.MESSAGE: message},

6
litellm/integrations/otel/model/payloads.py
View file

@ -141,6 +141,9 @@ class LLMCost:
class SpanError:
error_type: str | None = None
message: str | None = None
code: str | None = None
stack_trace: str | None = None
llm_provider: str | None = None
@dataclass(frozen=True)
@ -571,6 +574,9 @@ def _parse_error(payload: "StandardLoggingPayload") -> SpanError | None:
return SpanError(
error_type=as_str(info.get("error_class")) or as_str(info.get("error_code")),
message=as_str(info.get("error_message")) or as_str(payload.get("error_str")),
code=as_str(info.get("error_code")),
stack_trace=as_str(info.get("traceback")),
llm_provider=as_str(info.get("llm_provider")),
)

17
litellm/integrations/otel/model/semconv.py
View file

@ -144,7 +144,24 @@ class Client:
class Error:
"""OTel-defined error attribute keys, from the semconv ``error.*`` registry.
``MESSAGE`` is marked *Deprecated* upstream in favor of domain-specific
error message keys plus ``exception.message`` on the exception event, but
litellm still stamps it."""
TYPE: Final = "error.type"
MESSAGE: Final = "error.message"
class LiteLLMError:
"""Detail keys for the mapped provider exception of a failed LLM call.
OTel semconv does not define these, so they live under the ``litellm.*``
vendor namespace rather than squatting on the semconv-owned ``error.*``
namespace."""
CODE: Final = "litellm.provider.error.code"
STACK_TRACE: Final = "litellm.provider.error.stack_trace"
LLM_PROVIDER: Final = "litellm.provider.error.llm_provider"
class ExceptionEvent:

3
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -346,6 +346,9 @@ def get_llm_provider(
elif endpoint == "https://pinstripes.io/v1":
custom_llm_provider = "pinstripes"
dynamic_api_key = get_secret_str("PINSTRIPES_API_KEY")
elif endpoint == "https://api.meta.ai/v1":
custom_llm_provider = "meta"
dynamic_api_key = get_secret_str("META_API_KEY")
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))

12
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -95,6 +95,17 @@ class HealthCheckHelpers:
"""
import litellm
logging_obj = filtered_model_params.get("litellm_logging_obj")
if logging_obj is not None:
api_base = filtered_model_params.get("api_base")
logging_obj.update_from_kwargs(
kwargs=filtered_model_params,
model=filtered_model_params.get("model"),
user=None,
optional_params={},
litellm_params={"api_base": api_base} if api_base else None,
)
if custom_llm_provider in LIST_BATCHES_SUPPORTED_PROVIDERS:
return await litellm.alist_batches(**filtered_model_params)
else:
@ -188,6 +199,7 @@ class HealthCheckHelpers:
api_base=model_params.get("api_base", None),
api_key=model_params.get("api_key", None),
api_version=model_params.get("api_version", None),
model_params=model_params,
),
"batch": lambda: HealthCheckHelpers._batch_health_check(
custom_llm_provider=custom_llm_provider,

68
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -3626,6 +3626,7 @@ class BedrockImageProcessor:
def _convert_to_bedrock_tool_call_invoke(
tool_calls: list,
model: Optional[str] = None,
) -> List[BedrockContentBlock]:
"""
OpenAI tool invokes:
@ -3701,7 +3702,13 @@ def _convert_to_bedrock_tool_call_invoke(
# cache_control applies to the whole original
# tool call; attach after the last split block.
if tool.get("cache_control", None) is not None:
_parts_list.append(BedrockContentBlock(cachePoint=CachePointBlock(type="default")))
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
{"cache_control": tool["cache_control"]},
block_type="content_block",
model=model,
)
if _cache_point_block is not None:
_parts_list.append(_cache_point_block)
continue
# Fallback: no objects extracted — use empty dict.
arguments_dict = {}
@ -3712,8 +3719,13 @@ def _convert_to_bedrock_tool_call_invoke(
# Check for cache_control and add a separate cachePoint block
if tool.get("cache_control", None) is not None:
cache_point_block = BedrockContentBlock(cachePoint=CachePointBlock(type="default"))
_parts_list.append(cache_point_block)
cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
{"cache_control": tool["cache_control"]},
block_type="content_block",
model=model,
)
if cache_point_block is not None:
_parts_list.append(cache_point_block)
return _parts_list
except Exception as e:
raise Exception(
@ -4377,6 +4389,7 @@ class BedrockConverseMessagesProcessor:
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block=cast(OpenAIMessageContentListBlock, element),
block_type="content_block",
model=model,
)
if _cache_point_block is not None:
_parts.append(_cache_point_block)
@ -4384,7 +4397,7 @@ class BedrockConverseMessagesProcessor:
elif message_block["content"] and isinstance(message_block["content"], str):
_part = BedrockContentBlock(text=messages[msg_i]["content"])
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block, block_type="content_block"
message_block, block_type="content_block", model=model
)
user_content.append(_part)
if _cache_point_block is not None:
@ -4416,22 +4429,27 @@ class BedrockConverseMessagesProcessor:
tool_content.append(tool_call_result)
# Check if we need to add a separate cachePoint block
has_cache_control = False
tool_msg_cache_control = None
# Check for message-level cache_control
if current_message.get("cache_control", None) is not None:
has_cache_control = True
tool_msg_cache_control = current_message["cache_control"]
# Check for content-level cache_control in list content
elif isinstance(current_message.get("content"), list):
for content_element in current_message["content"]:
if isinstance(content_element, dict) and content_element.get("cache_control", None) is not None:
has_cache_control = True
tool_msg_cache_control = content_element["cache_control"]
break
# Add a separate cachePoint block if cache_control is present
if has_cache_control:
cache_point_block = BedrockContentBlock(cachePoint=CachePointBlock(type="default"))
tool_content.append(cache_point_block)
if tool_msg_cache_control is not None:
cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
{"cache_control": tool_msg_cache_control},
block_type="content_block",
model=model,
)
if cache_point_block is not None:
tool_content.append(cache_point_block)
msg_i += 1
# Deduplicate toolResult blocks with the same toolUseId
@ -4509,6 +4527,7 @@ class BedrockConverseMessagesProcessor:
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block=cast(OpenAIMessageContentListBlock, element),
block_type="content_block",
model=model,
)
if _cache_point_block is not None:
assistants_parts.append(_cache_point_block)
@ -4520,14 +4539,14 @@ class BedrockConverseMessagesProcessor:
# If content is empty/whitespace, skip it (don't add a placeholder)
# Add cache point block for assistant string content
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
assistant_message_block, block_type="content_block"
assistant_message_block, block_type="content_block", model=model
)
if _cache_point_block is not None:
assistant_content.append(_cache_point_block)
_tool_calls = assistant_message_block.get("tool_calls", [])
if _tool_calls:
assistant_content.extend(_convert_to_bedrock_tool_call_invoke(_tool_calls))
assistant_content.extend(_convert_to_bedrock_tool_call_invoke(_tool_calls, model=model))
msg_i += 1
@ -4745,6 +4764,7 @@ def _bedrock_converse_messages_pt(
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block=cast(OpenAIMessageContentListBlock, element),
block_type="content_block",
model=model,
)
if _cache_point_block is not None:
_parts.append(_cache_point_block)
@ -4752,7 +4772,7 @@ def _bedrock_converse_messages_pt(
elif message_block["content"] and isinstance(message_block["content"], str):
_part = BedrockContentBlock(text=messages[msg_i]["content"])
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block, block_type="content_block"
message_block, block_type="content_block", model=model
)
user_content.append(_part)
if _cache_point_block is not None:
@ -4786,22 +4806,27 @@ def _bedrock_converse_messages_pt(
tool_content.append(tool_call_result)
# Check if we need to add a separate cachePoint block
has_cache_control = False
tool_msg_cache_control = None
# Check for message-level cache_control
if current_message.get("cache_control", None) is not None:
has_cache_control = True
tool_msg_cache_control = current_message["cache_control"]
# Check for content-level cache_control in list content
elif isinstance(current_message.get("content"), list):
for content_element in current_message["content"]:
if isinstance(content_element, dict) and content_element.get("cache_control", None) is not None:
has_cache_control = True
tool_msg_cache_control = content_element["cache_control"]
break
# Add a separate cachePoint block if cache_control is present
if has_cache_control:
cache_point_block = BedrockContentBlock(cachePoint=CachePointBlock(type="default"))
tool_content.append(cache_point_block)
if tool_msg_cache_control is not None:
cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
{"cache_control": tool_msg_cache_control},
block_type="content_block",
model=model,
)
if cache_point_block is not None:
tool_content.append(cache_point_block)
msg_i += 1
# Deduplicate toolResult blocks with the same toolUseId
@ -4882,6 +4907,7 @@ def _bedrock_converse_messages_pt(
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
message_block=cast(OpenAIMessageContentListBlock, element),
block_type="content_block",
model=model,
)
if _cache_point_block is not None:
assistants_parts.append(_cache_point_block)
@ -4892,13 +4918,13 @@ def _bedrock_converse_messages_pt(
assistant_content.append(BedrockContentBlock(text=_assistant_content))
# Add cache point block for assistant string content
_cache_point_block = litellm.AmazonConverseConfig()._get_cache_point_block(
assistant_message_block, block_type="content_block"
assistant_message_block, block_type="content_block", model=model
)
if _cache_point_block is not None:
assistant_content.append(_cache_point_block)
_tool_calls = assistant_message_block.get("tool_calls", [])
if _tool_calls:
assistant_content.extend(_convert_to_bedrock_tool_call_invoke(_tool_calls))
assistant_content.extend(_convert_to_bedrock_tool_call_invoke(_tool_calls, model=model))
msg_i += 1

35
litellm/litellm_core_utils/sensitive_data_masker.py
View file

@ -1,6 +1,8 @@
from collections.abc import Mapping
from typing import Any, Dict, List, Optional, Set
from pydantic import BaseModel
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
@ -153,6 +155,39 @@ def mask_sensitive_structure(data: object) -> object:
return _error_masker.mask(data)
def mask_credentials_in_payload(data: object) -> object:
"""Return a copy of ``data`` where string values under sensitive-named keys
are masked but every other value (``None``, ``int``, ``float``, ``bool``,
``bytes``, ``datetime``, tuples, sets, typed objects) is preserved by
identity, and dicts/lists are rebuilt structurally.
Use this for logging payloads that carry response data through to
SpendLogs / OTel / Langfuse, where :meth:`SensitiveDataMasker.mask`'s
config-dump semantics (``None`` -> ``"None"``, tuples stringified,
objects flattened via ``__dict__``) would silently distort the record.
Sensitive-key detection is delegated to the shared
:class:`SensitiveDataMasker` so pattern updates stay in one place.
"""
return _walk_payload(data, key_is_sensitive=False, depth=0)
def _walk_payload(node: object, key_is_sensitive: bool, depth: int) -> object:
if depth >= DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER:
return node
if isinstance(node, Mapping):
return {k: _walk_payload(v, _default_masker.is_sensitive_key(k), depth + 1) for k, v in node.items()}
if isinstance(node, list):
return [_walk_payload(item, key_is_sensitive, depth + 1) for item in node]
if isinstance(node, tuple):
return tuple(_walk_payload(item, key_is_sensitive, depth + 1) for item in node)
if isinstance(node, BaseModel):
return _walk_payload(node.model_dump(), key_is_sensitive, depth)
if key_is_sensitive and isinstance(node, str) and node:
return _default_masker._mask_value(node)
return node
def mask_sensitive_keys(data: Dict[str, Any], sensitive_fields: Set[str]) -> Dict[str, Any]:
"""Return a new dict with values masked for keys listed in ``sensitive_fields``.

13
litellm/llms/azure_ai/ocr/common_utils.py
View file

@ -13,6 +13,17 @@ if TYPE_CHECKING:
from litellm.llms.base_llm.ocr.transformation import BaseOCRConfig
def is_azure_document_intelligence_model(model: str) -> bool:
"""Whether an azure_ai OCR model routes to Azure Document Intelligence.
Azure AI exposes two OCR services on the same provider; the sub-route in the
model name (`azure_ai/doc-intelligence/<model>`) selects Document Intelligence
over Mistral OCR. This is the single source of truth for that routing decision.
"""
lowered = model.lower()
return "doc-intelligence" in lowered or "documentintelligence" in lowered
def get_azure_ai_ocr_config(model: str) -> Optional["BaseOCRConfig"]:
"""
Determine which Azure AI OCR configuration to use based on the model name.
@ -41,7 +52,7 @@ def get_azure_ai_ocr_config(model: str) -> Optional["BaseOCRConfig"]:
from litellm.llms.azure_ai.ocr.transformation import AzureAIOCRConfig
# Check for Azure Document Intelligence models
if "doc-intelligence" in model or "documentintelligence" in model:
if is_azure_document_intelligence_model(model):
verbose_logger.debug(f"Routing {model} to Azure Document Intelligence OCR config")
return AzureDocumentIntelligenceOCRConfig()

57
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -489,24 +489,43 @@ class AmazonAnthropicClaudeMessagesConfig(
if self._supports_tool_search_on_bedrock(model):
beta_set.add("tool-search-tool-2025-10-19")
# Bedrock-InvokeModel-supported ``context_management.edits`` types and the
# ``anthropic-beta`` header that each one requires. ``clear_thinking_20251015``
# is intentionally absent — it is LiteLLM-internal, consumed via
# ``_ensure_thinking_for_clear_thinking_context_management``, and forwarding
# the raw edit trips Bedrock's
# ``"context_management: Extra inputs are not permitted"`` 400.
#
# Bedrock InvokeModel DOES support ``clear_tool_uses_20250919`` under the
# ``context-management-2025-06-27`` beta. AWS docs:
# https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-tool-use.md
_BEDROCK_INVOKE_SUPPORTED_CONTEXT_MANAGEMENT_EDITS: Dict[str, str] = {
"compact_20260112": ANTHROPIC_BETA_HEADER_VALUES.COMPACT_2026_01_12.value,
"clear_tool_uses_20250919": ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value,
}
@staticmethod
def _filter_context_management_for_bedrock_invoke(
anthropic_messages_request: Dict,
beta_set: set,
) -> None:
"""
Bedrock InvokeModel accepts ``context_management`` only when it carries
``compact_20260112`` edits paired with the ``compact-2026-01-12``
anthropic-beta header. Other edit types (notably ``clear_thinking_20251015``,
which Claude Code sends on every request) are LiteLLM-internal and would
cause Bedrock to 400 with ``"context_management: Extra inputs are not
permitted"``.
Filter ``context_management.edits`` to the subset that Bedrock InvokeModel
accepts and add the matching ``anthropic-beta`` header for each surviving
edit type.
Filter the edits list to the supported subset, add the beta header when
compact edits remain, and drop ``context_management`` entirely when no
supported edits are left so the safety-net allowlist can pass it through.
- ``compact_20260112`` -> ``compact-2026-01-12``
- ``clear_tool_uses_20250919`` -> ``context-management-2025-06-27``
Ref: https://github.com/BerriAI/litellm/issues/27532
Other edit types (notably ``clear_thinking_20251015``, which Claude Code
sends on every request) are LiteLLM-internal: thinking is injected
separately via ``_ensure_thinking_for_clear_thinking_context_management``,
and forwarding the raw edit would trip Bedrock's
``"context_management: Extra inputs are not permitted"`` 400.
Refs:
* https://github.com/BerriAI/litellm/issues/27532
* https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-tool-use.md
"""
cm = anthropic_messages_request.get("context_management")
if not isinstance(cm, dict):
@ -516,15 +535,17 @@ class AmazonAnthropicClaudeMessagesConfig(
anthropic_messages_request.pop("context_management", None)
return
compact_edits = [e for e in edits if isinstance(e, dict) and e.get("type") == "compact_20260112"]
if compact_edits:
beta_set.add(ANTHROPIC_BETA_HEADER_VALUES.COMPACT_2026_01_12.value)
anthropic_messages_request["context_management"] = {
**cm,
"edits": compact_edits,
}
else:
supported = AmazonAnthropicClaudeMessagesConfig._BEDROCK_INVOKE_SUPPORTED_CONTEXT_MANAGEMENT_EDITS
retained_edits = [e for e in edits if isinstance(e, dict) and e.get("type") in supported]
if not retained_edits:
anthropic_messages_request.pop("context_management", None)
return
beta_set.update(supported[e["type"]] for e in retained_edits)
anthropic_messages_request["context_management"] = {
**cm,
"edits": retained_edits,
}
def _get_bedrock_invoke_anthropic_beta_headers(
self,

18
litellm/llms/deepseek/chat/transformation.py
View file

@ -35,7 +35,9 @@ class DeepSeekChatConfig(OpenAIGPTConfig):
Map OpenAI params to DeepSeek params.
Handles `thinking` and `reasoning_effort` parameters for DeepSeek reasoner models.
DeepSeek only supports `{"type": "enabled"}` - no budget_tokens like Anthropic.
DeepSeek supports `{"type": "enabled"}` and `{"type": "disabled"}` - no budget_tokens
like Anthropic. `reasoning_effort="none"` is the OpenAI-style way to ask for thinking
off, so it maps to `{"type": "disabled"}`; any other effort keeps thinking on.
Reference: https://api-docs.deepseek.com/guides/thinking_mode
"""
@ -47,15 +49,13 @@ class DeepSeekChatConfig(OpenAIGPTConfig):
thinking_value = optional_params.pop("thinking", None)
reasoning_effort = optional_params.pop("reasoning_effort", None)
# Handle thinking parameter - only accept {"type": "enabled"}
if thinking_value is not None:
if isinstance(thinking_value, dict) and thinking_value.get("type") == "enabled":
# DeepSeek only accepts {"type": "enabled"}, ignore budget_tokens
optional_params["thinking"] = {"type": "enabled"}
# Handle thinking parameter - accept both enabled and disabled, ignore budget_tokens
if isinstance(thinking_value, dict) and thinking_value.get("type") in ("enabled", "disabled"):
optional_params["thinking"] = {"type": thinking_value["type"]}
# Handle reasoning_effort - map to thinking enabled
elif reasoning_effort is not None and reasoning_effort != "none":
optional_params["thinking"] = {"type": "enabled"}
# Otherwise fall back to reasoning_effort: "none" disables, anything else enables
elif reasoning_effort is not None:
optional_params["thinking"] = {"type": "disabled" if reasoning_effort == "none" else "enabled"}
return optional_params

6
litellm/llms/openai_like/dynamic_config.py
View file

@ -91,7 +91,7 @@ def create_config_class(provider: SimpleProviderConfig):
def get_supported_openai_params(self, model: str) -> list:
"""Get supported OpenAI params, excluding tool-related params for models
that don't support function calling."""
from litellm.utils import supports_function_calling
from litellm.utils import supports_function_calling, supports_reasoning
supported_params = super().get_supported_openai_params(model=model)
@ -113,6 +113,10 @@ def create_config_class(provider: SimpleProviderConfig):
f"function calling — removed tool-related params from supported params."
)
_supports_reasoning = supports_reasoning(model=model, custom_llm_provider=provider.slug)
if _supports_reasoning and "reasoning_effort" not in supported_params:
supported_params.append("reasoning_effort")
return supported_params
def map_openai_params(

65
litellm/llms/openai_like/messages/transformation.py
View file

@ -1,8 +1,11 @@
from typing import Any, Optional
import litellm
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from litellm.llms.openai_like.json_loader import SimpleProviderConfig
from litellm.secret_managers.main import get_secret_str
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
@ -67,3 +70,65 @@ class OpenAILikeAnthropicMessagesConfig(AnthropicMessagesConfig):
if base.endswith("/v1"):
base = base[: -len("/v1")]
return f"{base}/v1/messages"
class JSONProviderAnthropicMessagesConfig(OpenAILikeAnthropicMessagesConfig):
"""
Provider-level native Anthropic Messages passthrough for JSON-configured
OpenAI-compatible providers whose ``supported_endpoints`` in providers.json
includes ``"/v1/messages"``. Resolves the api key and api base from the
provider's configured env vars, then forwards the Anthropic payload
untranslated like ``OpenAILikeAnthropicMessagesConfig``.
"""
def __init__(self, provider: SimpleProviderConfig):
super().__init__()
self._provider = provider
def should_strip_billing_metadata(self) -> bool:
return True
def _resolve_api_key(self, api_key: Optional[str]) -> Optional[str]:
return api_key or get_secret_str(self._provider.api_key_env) or litellm.api_key
def _resolve_api_base(self, api_base: Optional[str]) -> str:
env_api_base = get_secret_str(self._provider.api_base_env) if self._provider.api_base_env else None
return api_base or env_api_base or self._provider.base_url
def validate_anthropic_messages_environment(
self,
headers: dict[str, str],
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict[str, str], Optional[str]]:
return super().validate_anthropic_messages_environment(
headers=headers,
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
api_key=self._resolve_api_key(api_key),
api_base=api_base,
)
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
return super().get_complete_url(
api_base=self._resolve_api_base(api_base),
api_key=api_key,
model=model,
optional_params=optional_params,
litellm_params=litellm_params,
stream=stream,
)

7
litellm/llms/openai_like/providers.json
View file

@ -168,6 +168,13 @@
},
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
},
"meta": {
"base_url": "https://api.meta.ai/v1",
"api_key_env": "META_API_KEY",
"api_base_env": "META_API_BASE",
"base_class": "openai_gpt",
"supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/messages"]
},
"pinstripes": {
"base_url": "https://pinstripes.io/v1",
"api_key_env": "PINSTRIPES_API_KEY",

806
litellm/model_prices_and_context_window_backup.json
View file

@ -6012,6 +6012,522 @@
"supports_vision": true,
"supports_web_search": true
},
"azure/gpt-5.6": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_priority": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_priority": 2e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_priority": 1e-05,
"input_cost_per_token_above_272k_tokens_priority": 2e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token_above_272k_tokens_priority": 9e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-sol": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_priority": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_priority": 2e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_priority": 1e-05,
"input_cost_per_token_above_272k_tokens_priority": 2e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token_above_272k_tokens_priority": 9e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost_priority": 5e-07,
"cache_read_input_token_cost_above_272k_tokens_priority": 1e-06,
"input_cost_per_token": 2.5e-06,
"input_cost_per_token_above_272k_tokens": 5e-06,
"input_cost_per_token_priority": 5e-06,
"input_cost_per_token_above_272k_tokens_priority": 1e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_272k_tokens": 2.25e-05,
"output_cost_per_token_priority": 3e-05,
"output_cost_per_token_above_272k_tokens_priority": 4.5e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-luna": {
"cache_read_input_token_cost": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-07,
"cache_read_input_token_cost_priority": 2e-07,
"cache_read_input_token_cost_above_272k_tokens_priority": 4e-07,
"input_cost_per_token": 1e-06,
"input_cost_per_token_above_272k_tokens": 2e-06,
"input_cost_per_token_priority": 2e-06,
"input_cost_per_token_above_272k_tokens_priority": 4e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_272k_tokens": 9e-06,
"output_cost_per_token_priority": 1.2e-05,
"output_cost_per_token_above_272k_tokens_priority": 1.8e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-sol": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.75e-07,
"cache_read_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost_priority": 6.875e-07,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_above_272k_tokens": 5.5e-06,
"input_cost_per_token_priority": 6.875e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"output_cost_per_token_above_272k_tokens": 2.475e-05,
"output_cost_per_token_priority": 4.125e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-luna": {
"cache_read_input_token_cost": 1.1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2.2e-07,
"cache_read_input_token_cost_priority": 2.75e-07,
"input_cost_per_token": 1.1e-06,
"input_cost_per_token_above_272k_tokens": 2.2e-06,
"input_cost_per_token_priority": 2.75e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.6e-06,
"output_cost_per_token_above_272k_tokens": 9.9e-06,
"output_cost_per_token_priority": 1.65e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-sol": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.75e-07,
"cache_read_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost_priority": 6.875e-07,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_above_272k_tokens": 5.5e-06,
"input_cost_per_token_priority": 6.875e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"output_cost_per_token_above_272k_tokens": 2.475e-05,
"output_cost_per_token_priority": 4.125e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-luna": {
"cache_read_input_token_cost": 1.1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2.2e-07,
"cache_read_input_token_cost_priority": 2.75e-07,
"input_cost_per_token": 1.1e-06,
"input_cost_per_token_above_272k_tokens": 2.2e-06,
"input_cost_per_token_priority": 2.75e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.6e-06,
"output_cost_per_token_above_272k_tokens": 9.9e-06,
"output_cost_per_token_priority": 1.65e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
@ -22273,6 +22789,218 @@
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": true
},
"gpt-5.6": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-sol": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-terra": {
"cache_creation_input_token_cost": 3.125e-06,
"cache_creation_input_token_cost_above_272k_tokens": 6.25e-06,
"cache_creation_input_token_cost_flex": 1.5625e-06,
"cache_creation_input_token_cost_priority": 6.25e-06,
"cache_read_input_token_cost": 2.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost_flex": 1.25e-07,
"cache_read_input_token_cost_priority": 5e-07,
"input_cost_per_token": 2.5e-06,
"input_cost_per_token_above_272k_tokens": 5e-06,
"input_cost_per_token_batches": 1.25e-06,
"input_cost_per_token_flex": 1.25e-06,
"input_cost_per_token_priority": 5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_272k_tokens": 2.25e-05,
"output_cost_per_token_batches": 7.5e-06,
"output_cost_per_token_flex": 7.5e-06,
"output_cost_per_token_priority": 3e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-luna": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 2.5e-06,
"cache_creation_input_token_cost_flex": 6.25e-07,
"cache_creation_input_token_cost_priority": 2.5e-06,
"cache_read_input_token_cost": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-07,
"cache_read_input_token_cost_flex": 5e-08,
"cache_read_input_token_cost_priority": 2e-07,
"input_cost_per_token": 1e-06,
"input_cost_per_token_above_272k_tokens": 2e-06,
"input_cost_per_token_batches": 5e-07,
"input_cost_per_token_flex": 5e-07,
"input_cost_per_token_priority": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_272k_tokens": 9e-06,
"output_cost_per_token_batches": 3e-06,
"output_cost_per_token_flex": 3e-06,
"output_cost_per_token_priority": 1.2e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
@ -25289,6 +26017,42 @@
"supports_function_calling": true,
"supports_tool_choice": false
},
"meta/muse-spark-1.1": {
"cache_read_input_token_cost": 1.5e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "meta",
"max_input_tokens": 1048576,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 4.25e-06,
"source": "https://dev.meta.ai/docs/getting-started/pricing-rate-limits",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/messages"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"meta_llama/Llama-3.3-70B-Instruct": {
"litellm_provider": "meta_llama",
"max_input_tokens": 128000,
@ -38101,6 +38865,48 @@
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-4.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"litellm_provider": "xai",
"max_input_tokens": 500000,
"max_output_tokens": 500000,
"max_tokens": 500000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_200k_tokens": 1.2e-05,
"source": "https://docs.x.ai/docs/models",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-4.5-latest": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"litellm_provider": "xai",
"max_input_tokens": 500000,
"max_output_tokens": 500000,
"max_tokens": 500000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_200k_tokens": 1.2e-05,
"source": "https://docs.x.ai/docs/models",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-beta": {
"input_cost_per_token": 5e-06,
"litellm_provider": "xai",

15
litellm/ocr/main.py
View file

@ -17,6 +17,9 @@ import litellm
from litellm._logging import verbose_logger
from litellm.constants import request_timeout
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure_ai.ocr.common_utils import (
is_azure_document_intelligence_model,
)
from litellm.llms.base_llm.ocr.transformation import BaseOCRConfig, OCRResponse
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.rust_bridge import ocr as rust_ocr_bridge
@ -83,6 +86,8 @@ def _prepare_ocr_request(
if doc_type not in ["document_url", "image_url"]:
raise ValueError(f"Invalid document type: {doc_type}. Must be 'document_url', 'image_url', or 'file'")
caller_supplied_api_base = api_base is not None
(
model,
custom_llm_provider,
@ -95,9 +100,14 @@ def _prepare_ocr_request(
api_key=api_key,
)
suppress_dynamic_api_base = (
not caller_supplied_api_base
and custom_llm_provider == "azure_ai"
and is_azure_document_intelligence_model(model)
)
if dynamic_api_key:
api_key = dynamic_api_key
if dynamic_api_base:
if dynamic_api_base and not suppress_dynamic_api_base:
api_base = dynamic_api_base
ocr_provider_config = ProviderConfigManager.get_provider_ocr_config(
@ -191,8 +201,7 @@ def _rust_bridge_api_base(
if prepared_request.api_base is not None:
return prepared_request.api_base
if prepared_request.custom_llm_provider == "azure_ai":
model = prepared_request.model.lower()
if "doc-intelligence" in model or "documentintelligence" in model:
if is_azure_document_intelligence_model(prepared_request.model):
return resolve_secret("AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT")
return resolve_secret("AZURE_AI_API_BASE")
return None

64
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -220,6 +220,12 @@ class MCPRequestHandler:
# when EVERY target is auth_type=oauth2 with delegate_auth_to_upstream
# set; fails closed otherwise.
validated_user_api_key_auth = UserAPIKeyAuth()
elif MCPRequestHandler._target_servers_are_true_passthrough(
path=request_route,
mcp_servers=mcp_servers,
client_ip=IPAddressUtils.get_mcp_client_ip(request),
):
validated_user_api_key_auth = UserAPIKeyAuth()
elif oauth2_headers:
# Authorization on a non-delegated server: the bearer must be a real
# LiteLLM credential, so a failed validation is a genuine 401/403 and
@ -399,6 +405,33 @@ class MCPRequestHandler:
return False
return True
@staticmethod
def _target_servers_are_true_passthrough(
path: str, mcp_servers: Optional[list[str]], client_ip: Optional[str]
) -> bool:
"""
True only when EVERY MCP server the request targets is ``auth_type == true_passthrough``.
Fails closed when any target does not opt in or cannot be resolved.
Used by :meth:`process_mcp_request` to skip LiteLLM admission auth entirely: the gateway is a
transparent proxy and the caller's ``Authorization`` is an upstream token, never a LiteLLM key.
Mirrors :meth:`_target_servers_delegate_auth_to_upstream`; a mixed-target request keeps normal auth.
"""
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.types.mcp import MCPAuth
target_names = MCPRequestHandler._resolve_target_server_names(path=path, mcp_servers_header=mcp_servers)
if not target_names:
return False
for name in target_names:
server = global_mcp_server_manager.get_mcp_server_by_name(name, client_ip=client_ip)
if server is None or server.auth_type != MCPAuth.true_passthrough:
return False
return True
@staticmethod
def _resolve_target_server_names(path: str, mcp_servers_header: Optional[List[str]]) -> List[str]:
"""
@ -558,10 +591,19 @@ class MCPRequestHandler:
ASGI headers are in format: List[List[bytes, bytes]]
We need to convert them to the format Headers expects.
Collapsing the ASGI list into a dict keeps the last value for a duplicated
header name, so a request carrying more than one ``Authorization`` is
rejected first: for the client-forwarded token modes the gateway relays the
caller's ``Authorization`` upstream, so a duplicate would make which token is
forwarded ambiguous (and diverge from what admission inspected). Multiple
``Authorization`` headers is malformed for bearer auth anyway (RFC 9110: not
a comma-combinable field), so fail closed with a 400.
"""
raw_headers = scope.get("headers", [])
MCPRequestHandler._reject_duplicate_authorization(raw_headers)
try:
# ASGI headers are list of [name: bytes, value: bytes] pairs
raw_headers = scope.get("headers", [])
# Convert bytes to strings and create dict for Headers constructor
headers_dict = {name.decode("latin-1"): value.decode("latin-1") for name, value in raw_headers}
return Headers(headers_dict)
@ -570,6 +612,26 @@ class MCPRequestHandler:
# Return empty Headers object with empty dict
return Headers({})
@staticmethod
def _reject_duplicate_authorization(raw_headers: object) -> None:
"""Raise 400 when the raw ASGI headers carry more than one ``Authorization`` header."""
if not isinstance(raw_headers, (list, tuple)):
return
count = 0
for entry in raw_headers:
if not isinstance(entry, (list, tuple)) or len(entry) < 1:
continue
name = entry[0]
if isinstance(name, (bytes, bytearray)) and bytes(name).lower() == b"authorization":
count += 1
elif isinstance(name, str) and name.lower() == "authorization":
count += 1
if count > 1:
raise HTTPException(
status_code=400,
detail="Multiple Authorization headers are not allowed",
)
@staticmethod
async def get_allowed_mcp_servers(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,

132
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -3,7 +3,7 @@ import binascii
import hashlib
import json
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Set, Union, cast
from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, Iterable, List, Optional, Set, Union, cast
from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
@ -558,7 +558,11 @@ async def delete_mcp_server_from_virtualkey():
pass
async def delete_mcp_server(prisma_client: PrismaClient, server_id: str) -> Optional[LiteLLM_MCPServerTable]:
async def delete_mcp_server(
prisma_client: PrismaClient,
server_id: str,
invalidate_token_cache: Optional[Callable[[str, str], Awaitable[None]]] = None,
) -> Optional[LiteLLM_MCPServerTable]:
"""
Delete the mcp server from the db by server_id
@ -569,6 +573,12 @@ async def delete_mcp_server(prisma_client: PrismaClient, server_id: str) -> Opti
caller-visible error. Each table is cleaned independently so a failure on one
still attempts the other.
Each enumerated credential row's user also gets their cached per-user token
invalidated (legacy cache + v2 store, via invalidate_token_cache, defaulting
to the manager's shared invalidation): the caches are keyed by
(user_id, server_id), so without this a re-created server reusing the same
server_id would serve tokens minted for the deleted server until TTL.
Returns the deleted mcp server record if it exists, otherwise None
"""
deleted_server = await MCPServerRepository(prisma_client).table.delete(
@ -577,6 +587,18 @@ async def delete_mcp_server(prisma_client: PrismaClient, server_id: str) -> Opti
},
)
if deleted_server is not None:
credential_user_ids: List[str] = []
try:
credential_rows = await prisma_client.db.litellm_mcpusercredentials.find_many(
where={"server_id": server_id}
)
credential_user_ids = [row.user_id for row in credential_rows]
except Exception as e: # noqa: BLE001 - enumeration is best-effort; cached tokens expire by TTL
verbose_proxy_logger.warning(
"MCP server %s deleted but per-user credential enumeration failed; cached tokens expire by TTL: %s",
server_id,
e,
)
for model, label in (
(prisma_client.db.litellm_mcpusercredentials, "credential"),
(prisma_client.db.litellm_mcpuserenvvars, "env var"),
@ -591,6 +613,15 @@ async def delete_mcp_server(prisma_client: PrismaClient, server_id: str) -> Opti
label,
e,
)
if credential_user_ids:
if invalidate_token_cache is None:
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
invalidate_token_cache = global_mcp_server_manager.invalidate_user_oauth_token_cache
for user_id in credential_user_ids:
await invalidate_token_cache(user_id, server_id)
return deleted_server
@ -1070,6 +1101,103 @@ async def list_user_oauth_credentials(
return results
def _decrypted_credential_field(creds: Dict[str, object], field: str) -> object:
"""Return one credential field decrypted with the global salt key; non-string and legacy
plaintext values come back unchanged (decrypt_value_helper returns the original on failure)."""
value = creds.get(field)
if not isinstance(value, str):
return value
return decrypt_value_helper(
value=value,
key=field,
exception_type="debug",
return_original_value=True,
)
def mcp_oauth_token_identity(server: object) -> tuple[object, ...]:
"""The upstream-OAuth-token-determining fields of an MCP server: the resource/audience (url, or
spec_path for OpenAPI servers), the OAuth mode/grant (auth_type, oauth2_flow), the
authorization-server endpoints, and the OAuth client + scopes. Mirrors the dashboard's
getOAuthAuthorizationIdentity. When any of these change on a server update, previously stored
per-user tokens were minted for the old identity and are stale. Excludes transport and
delegate_auth_to_upstream, which do not affect what token is minted (RFC 8707/8693).
client_id/client_secret are compared decrypted: stored values are NaCl-encrypted with a fresh
nonce on every write, so comparing ciphertext would flag every routine save as an identity
change and purge tokens that are still valid."""
creds = getattr(server, "credentials", None)
if isinstance(creds, str):
try:
parsed: object = json.loads(creds)
except ValueError:
parsed = None
else:
parsed = creds
creds_dict: Dict[str, object] = parsed if isinstance(parsed, dict) else {}
return (
getattr(server, "url", None),
getattr(server, "spec_path", None),
getattr(server, "auth_type", None),
getattr(server, "oauth2_flow", None),
getattr(server, "authorization_url", None),
getattr(server, "token_url", None),
getattr(server, "registration_url", None),
_decrypted_credential_field(creds_dict, "client_id"),
_decrypted_credential_field(creds_dict, "client_secret"),
creds_dict.get("scopes"),
)
async def purge_user_oauth_credentials_for_server(
prisma_client: PrismaClient,
server_id: str,
invalidate_token_cache: Optional[Callable[[str, str], Awaitable[None]]] = None,
) -> int:
"""Delete every stored per-user OAuth token for a server and invalidate each user's cached
token everywhere it can be served from (the legacy per-user token cache and the v2 per-user OAuth
token store), so no user keeps a token minted for a superseded configuration. Called when a server
update changes a mint-relevant field (see mcp_oauth_token_identity). Returns the number of rows
removed.
LiteLLM_MCPUserCredentials also stores BYOK API keys in the same column; only rows whose payload
decodes as an OAuth2 credential (see _decode_oauth_payload) are deleted, because a config change
only invalidates minted tokens, never a user's own stored key. Rows are therefore deleted per
(user_id, server_id) pair rather than by a blanket server_id filter. An OAuth row inserted while
the purge runs for a user not yet enumerated survives; a re-auth completing in the window for an
already-enumerated user is deleted along with the stale row (the pair delete cannot tell them
apart), which costs that user one extra re-auth and nothing else.
invalidate_token_cache is injectable for tests; it defaults to the manager's shared
invalidate_user_oauth_token_cache, the single invalidation point for per-user tokens."""
repo = MCPUserCredentialsRepository(prisma_client)
rows = await repo.table.find_many(where={"server_id": server_id})
oauth_rows = [row for row in rows if _decode_oauth_payload(row.credential_b64) is not None]
if not oauth_rows:
return 0
deleted_count = await repo.table.delete_many(
where={"server_id": server_id, "user_id": {"in": [row.user_id for row in oauth_rows]}}
)
if invalidate_token_cache is None:
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
invalidate_token_cache = global_mcp_server_manager.invalidate_user_oauth_token_cache
for row in oauth_rows:
await invalidate_token_cache(row.user_id, server_id)
if deleted_count != len(oauth_rows):
verbose_proxy_logger.warning(
"MCP server %s: purge removed %d OAuth credential row(s) but %d were enumerated; "
"row(s) were deleted concurrently during the purge",
server_id,
deleted_count,
len(oauth_rows),
)
return deleted_count
async def refresh_user_oauth_token(
prisma_client: PrismaClient,
user_id: str,

47
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -465,8 +465,19 @@ async def _store_per_user_token_server_side(
def _raise_if_not_oauth2(mcp_server: MCPServer) -> None:
"""Reject a non-oauth2 server from the gateway's OAuth authorize/token/register flow."""
if mcp_server.auth_type == MCPAuth.oauth2:
"""Reject a server without upstream OAuth from the gateway's authorize/token/register flow.
The client-forwarded token modes (``true_passthrough`` / ``oauth_delegate``) are allowed
through: the caller owns the upstream token, and this relayed flow is how a browser obtains
one against the upstream IdP (the admin UI's browser-only Authorize uses it). The minted
token is upstream-audienced and held by the caller; the gateway persists nothing for these
modes (DCR persistence is opt-in and never enabled on this path).
"""
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415 # circular import with mcp_server_manager at module load
_UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
)
if mcp_server.auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES:
return
raise HTTPException(
status_code=400,
@ -515,8 +526,7 @@ async def authorize_with_server(
response_type: Optional[str] = None,
scope: Optional[str] = None,
):
if mcp_server.auth_type != "oauth2":
raise HTTPException(status_code=400, detail="MCP server is not OAuth2")
_raise_if_not_oauth2(mcp_server)
if mcp_server.authorization_url is None:
raise HTTPException(status_code=400, detail="MCP server authorization url is not set")
@ -581,8 +591,12 @@ async def exchange_token_with_server(
if mcp_server.token_url is None:
raise HTTPException(status_code=400, detail="MCP server token url is not set")
# The id and secret must come from the same source. When the server-side client_id wins,
# falling back to the caller's secret pairs the persisted client with a foreign secret; the
# register short-circuit hands clients a placeholder secret ("dummy"), so a re-auth against a
# persisted public PKCE client (no stored secret) would send that placeholder and the IdP 401s.
resolved_client_id = mcp_server.client_id if mcp_server.client_id else client_id
resolved_client_secret = mcp_server.client_secret if mcp_server.client_secret else client_secret
resolved_client_secret = mcp_server.client_secret if mcp_server.client_id else client_secret
try:
client_auth = build_token_endpoint_client_auth(
auth_method=mcp_server.token_endpoint_auth_method,
@ -1298,11 +1312,15 @@ async def _build_oauth_protected_resource_response(
"""
Build OAuth protected resource response with the appropriate URL pattern.
For pass-through MCP servers (``MCPServer.is_oauth_passthrough``), the
gateway proxies the upstream's own ``oauth-protected-resource`` metadata
so that standards-compliant MCP clients discover the **upstream** IdP
instead of the gateway. The ``resource`` field is rewritten to the
gateway's own URL so clients present the bearer token back to the gateway.
For pass-through MCP servers, the gateway proxies the upstream's own
``oauth-protected-resource`` metadata so standards-compliant MCP clients
discover the **upstream** IdP instead of the gateway. For ``true_passthrough``
and ``oauth_delegate`` the metadata is returned verbatim (``resource`` stays
the upstream): the caller's token is forwarded to and validated by the
upstream, so its audience must be the upstream — rewriting it to the gateway
would make a strict IdP (e.g. Entra) refuse to mint it or the upstream reject
it. Only the legacy ``is_oauth_passthrough`` opt-in rewrites ``resource`` to
the gateway's own URL so clients present the bearer token back to the gateway.
Args:
request: FastAPI Request object
@ -1343,7 +1361,9 @@ async def _build_oauth_protected_resource_response(
# Pass-through branch: proxy the upstream's own metadata so discovery
# directs the client at the real IdP (Okta, Keycloak, …) instead of us.
if mcp_server is not None and mcp_server.is_oauth_passthrough:
if mcp_server is not None and (
mcp_server.is_oauth_passthrough or mcp_server.is_oauth_delegate or mcp_server.is_true_passthrough
):
try:
upstream_metadata = await fetch_upstream_oauth_protected_resource(mcp_server)
except Exception as exc:
@ -1359,8 +1379,9 @@ async def _build_oauth_protected_resource_response(
)
if upstream_metadata is not None:
response = {**upstream_metadata, "resource": resource_url}
return response
if mcp_server.is_true_passthrough or mcp_server.is_oauth_delegate:
return upstream_metadata
return {**upstream_metadata, "resource": resource_url}
# Upstream responded but with non-200 or non-dict payload. For
# pass-through servers the gateway is NOT the authorization server,

15
litellm/proxy/_experimental/mcp_server/exceptions.py
View file

@ -73,3 +73,18 @@ class MCPUpstreamAuthError(Exception):
detail=detail,
headers={"www-authenticate": challenge} if challenge else None,
)
class MCPToolResultError(Exception):
"""An MCP tool call completed with ``isError=True`` in its result.
Never raised on the wire path: streamable HTTP MCP correctly returns tool
failures as HTTP 200 with ``result.isError: true`` per the MCP spec. This
exception only drives the standard failure logging (``status="failure"``
payload, OTel ERROR span) for such results.
Lives here rather than ``utils.py`` deliberately: tests reload ``utils``
to re-read its env-derived constants, and a reload would fork this class
into two identities, breaking ``isinstance`` checks against instances
created before the reload.
"""

189
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -57,7 +57,11 @@ from litellm.proxy._experimental.mcp_server.elicitation_handler import (
from litellm.proxy._experimental.mcp_server.sampling_handler import (
MCP_SAMPLING_AVAILABLE,
)
from litellm.proxy._experimental.mcp_server.oauth2_token_cache import resolve_mcp_auth
from litellm.proxy._experimental.mcp_server.oauth2_token_cache import (
MCPPerUserTokenCache,
mcp_per_user_token_cache,
resolve_mcp_auth,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials import (
Error,
Ok,
@ -81,6 +85,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.token_exchange_
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
AuthorizationCodeConfig,
PassthroughConfig,
ServerSpec,
TokenExchangeConfig,
)
@ -170,6 +175,16 @@ _user_env_vars_cache: dict[tuple[str, str], tuple[dict[str, str], float]] = {}
_USER_ENV_VARS_CACHE_TTL = 60 # seconds
_USER_ENV_VARS_CACHE_MAX_SIZE = 4096 # cap to prevent unbounded growth
# Auth types whose upstream OAuth endpoints (protected-resource + authorization-server metadata) the
# gateway discovers from the upstream itself: interactive oauth2 and the two client-forwarded modes.
# OBO/M2M endpoint discovery is decided separately via _obo_needs_endpoint_discovery. Shared by the
# config-YAML and DB server loaders so the two paths cannot drift on which modes trigger discovery.
_UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES: tuple[MCPAuth, ...] = (
MCPAuth.oauth2,
MCPAuth.true_passthrough,
MCPAuth.oauth_delegate,
)
def invalidate_user_env_vars_cache(user_id: str, server_id: str) -> None:
"""Drop a cached entry after the user stores or clears their env var values
@ -216,6 +231,13 @@ def _should_strip_caller_authorization(
pass-through cold-start case (RFC 9728) the bearer in
``Authorization`` is the upstream OAuth token and must be
forwarded, so we keep it.
- **oauth_delegate servers**: admission always runs and there is no
anonymous path, so the caller's separate ``Authorization`` is
forwarded only when a distinct ``x-litellm-api-key`` carried
admission. Without that header the ``Authorization`` *was* the
admission credential — a virtual key, an IdP JWT, or an SSO / OIDC /
session token whose ``api_key`` is ``None`` — and must never reach
the upstream, so it is stripped regardless of the ``api_key`` value.
"""
if mcp_server.auth_type == MCPAuth.oauth2_token_exchange:
# OBO: the inbound Authorization is the subject token. It is exchanged at the IdP and only the
@ -229,11 +251,13 @@ def _should_strip_caller_authorization(
# upstream — it would override another user's stored credential. Delegate and
# pass-through return None from to_server_spec and keep forwarding the bearer.
return True
if not mcp_server.is_oauth_passthrough:
if not (mcp_server.is_oauth_passthrough or mcp_server.is_oauth_delegate):
return False
normalized_raw_headers = {str(k).lower(): v for k, v in (raw_headers or {}).items() if isinstance(k, str)}
has_explicit_litellm_admission_header = normalized_raw_headers.get("x-litellm-api-key") is not None
if mcp_server.is_oauth_delegate:
return not has_explicit_litellm_admission_header
admission_consumed_authorization_as_litellm_key = (
user_api_key_auth is not None
and bool(getattr(user_api_key_auth, "api_key", None))
@ -326,6 +350,89 @@ async def _resolve_byok_mcp_auth_header(
return mcp_auth_header
def _client_forwarded_authorization_headers(
mcp_server: MCPServer,
oauth2_headers: Optional[dict[str, str]],
raw_headers: Optional[dict[str, str]],
user_api_key_auth: Optional[UserAPIKeyAuth],
) -> Optional[dict[str, str]]:
"""Egress headers for the client-forwarded-token modes (``true_passthrough`` / ``oauth_delegate``).
Forwards the caller's ``Authorization`` to the upstream, stripped when
``_should_strip_caller_authorization`` says it was consumed as the LiteLLM admission key. Shared by
``_call_regular_mcp_tool`` and ``server.py``'s ``_prepare_mcp_server_headers`` so the two egress
paths cannot drift, mirroring the ``_should_strip_caller_authorization`` split.
"""
extra_headers = oauth2_headers.copy() if oauth2_headers else None
if extra_headers and _should_strip_caller_authorization(
mcp_server=mcp_server,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
):
return _without_authorization(extra_headers)
return extra_headers
def _take_forwarded_authorization(
headers: Optional[dict[str, str]],
) -> tuple[Optional[str], Optional[dict[str, str]]]:
"""Pop the ``Authorization`` value out of ``headers`` (case-insensitive), returning it with the
remaining headers, so the passthrough resolver arm is the single Authorization source rather than
the header also riding in ``extra_headers`` (which the resolved auth would then defer to)."""
if not headers:
return None, headers
value = next((v for k, v in headers.items() if k.lower() == "authorization"), None)
return value, _without_authorization(headers)
def _passthrough_token_from_mcp_auth_header(
mcp_auth_header: Optional[Union[str, dict[str, str]]],
) -> Optional[str]:
"""The caller's per-server upstream credential for a passthrough-mode server, or None.
Sourced from ``x-mcp-{alias}-authorization`` (string or per-header dict form) or the deprecated
global ``x-mcp-auth`` fallback. Per-server headers are the multi-server shape: they bind one
token to one server, so an aggregate scope with several passthrough-mode servers never replays
a single credential across upstreams. The value is forwarded verbatim, so it must be the full
header value (e.g. ``Bearer <upstream-token>``)."""
if isinstance(mcp_auth_header, str):
return mcp_auth_header or None
if isinstance(mcp_auth_header, dict):
return next((v for k, v in mcp_auth_header.items() if k.lower() == "authorization"), None)
return None
def _consumes_caller_authorization(server: MCPServer) -> bool:
"""True when this server's egress forwards the caller's request-wide ``Authorization`` upstream:
the client-forwarded token modes, legacy OAuth pass-through, and legacy upstream-delegated
interactive oauth2. An unstamped oauth2 row (flow column not yet backfilled) reads as a consumer,
which errs toward suppression — the fail-safe direction."""
if server.is_true_passthrough or server.is_oauth_delegate or server.is_oauth_passthrough:
return True
return (
server.auth_type == MCPAuth.oauth2
and getattr(server, "delegate_auth_to_upstream", False) is True
and not server.has_client_credentials
)
def _caller_authorization_fans_out(
server: MCPServer,
scope_servers: Optional[list[MCPServer]],
) -> bool:
"""True when forwarding the caller's request-wide ``Authorization`` to ``server`` inside a
listing fan-out would replay one credential against multiple upstreams: another server in the
scope also consumes it (RFC 9700 cross-resource replay). ``scope_servers`` is None for
explicitly-addressed operations (tool call, get_prompt, read_resource, single-server routes),
where the client named the one target and the gateway is not choosing recipients."""
if scope_servers is None:
return False
return any(
other is not None and other.server_id != server.server_id and _consumes_caller_authorization(other)
for other in scope_servers
)
def _extract_upstream_auth_failure(
exc: BaseException,
) -> Optional[tuple[int, Optional[str]]]:
@ -696,10 +803,12 @@ class MCPServerManager:
self,
cred_provider: Optional[UpstreamCredentialProvider] = None,
per_user_oauth_token_store: Optional[InvalidatableOAuthTokenStore] = None,
per_user_token_cache: Optional[MCPPerUserTokenCache] = None,
):
self._per_user_oauth_token_store = per_user_oauth_token_store or LazyPerUserOAuthTokenStore(
self.get_mcp_server_by_id
)
self._per_user_token_cache = per_user_token_cache or mcp_per_user_token_cache
self._cred_provider = cred_provider or UpstreamCredentialProvider(
oauth_token_store=self._per_user_oauth_token_store,
token_exchanger=build_token_exchanger(),
@ -891,7 +1000,7 @@ class MCPServerManager:
auth_type = server_config.get("auth_type", None)
if server_url and (
auth_type == MCPAuth.oauth2
auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES
or self._obo_needs_endpoint_discovery(
auth_type,
server_config.get("token_exchange_endpoint"),
@ -900,7 +1009,7 @@ class MCPServerManager:
):
mcp_oauth_metadata = await self._descovery_metadata(
server_url=server_url,
allow_origin_fallback=auth_type == MCPAuth.oauth2,
allow_origin_fallback=auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
)
else:
mcp_oauth_metadata = None
@ -1292,7 +1401,7 @@ class MCPServerManager:
auth_type = cast(MCPAuthType, mcp_server.auth_type)
server_url = mcp_server.url
needs_discovery = bool(server_url) and (
(auth_type == MCPAuth.oauth2 and not mcp_server.authorization_url)
(auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES and not mcp_server.authorization_url)
or self._obo_needs_endpoint_discovery(
auth_type,
mcp_server.token_exchange_endpoint
@ -1303,7 +1412,7 @@ class MCPServerManager:
mcp_oauth_metadata = (
await self._descovery_metadata(
server_url=server_url, # type: ignore[arg-type]
allow_origin_fallback=auth_type == MCPAuth.oauth2,
allow_origin_fallback=auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
)
if needs_discovery
else None
@ -1635,15 +1744,18 @@ class MCPServerManager:
delegate_server_ids = [
server.server_id
for server in self.get_registry().values()
if getattr(server, "auth_type", None) == MCPAuth.oauth2
and getattr(server, "delegate_auth_to_upstream", False) is True
# M2M servers must not be exposed anonymously: an
# unauthenticated caller would get LiteLLM to proxy tool
# calls using its stored client_credentials. Resolve the flow
# rather than reading has_client_credentials so an unstamped
# M2M-shape row (null column, verbatim-read as non-M2M) still
# fails closed here, matching the anonymous-delegate auth gate.
and MCPServerManager.effective_oauth2_flow(server) != "client_credentials"
if (
getattr(server, "auth_type", None) == MCPAuth.oauth2
and getattr(server, "delegate_auth_to_upstream", False) is True
# M2M servers must not be exposed anonymously: an
# unauthenticated caller would get LiteLLM to proxy tool
# calls using its stored client_credentials. Resolve the flow
# rather than reading has_client_credentials so an unstamped
# M2M-shape row (null column, verbatim-read as non-M2M) still
# fails closed here, matching the anonymous-delegate auth gate.
and MCPServerManager.effective_oauth2_flow(server) != "client_credentials"
)
or getattr(server, "auth_type", None) == MCPAuth.true_passthrough
]
combined_servers.update(delegate_server_ids)
@ -2241,16 +2353,17 @@ class MCPServerManager:
spec = None if transport == MCPTransport.stdio else to_server_spec(server)
provider = cred_provider or self._cred_provider
# A caller-supplied per-request override (mcp_auth_header / x-mcp-*) defers to the v1 path
# so it wins - except for the per-user modes the v2 resolver owns (authorization_code's
# stored token and token_exchange's RFC 8693 minted token). A caller must not be able to
# substitute another user's stored credential, nor silently disable the OBO exchange and
# forward an arbitrary bearer upstream, so we keep the v2 spec and ignore the override for
# both; the REST tools preview supplies its not-yet-persisted token through the resolver
# (cred_provider), never this path.
# so it wins - except for the modes the v2 resolver owns per-caller (authorization_code's
# stored token, token_exchange's RFC 8693 minted token, and the passthrough modes'
# forwarded caller token). A caller must not be able to substitute another user's stored
# credential, nor silently disable the OBO exchange and forward an arbitrary bearer
# upstream, so we keep the v2 spec and ignore the override for these; the REST tools
# preview supplies its not-yet-persisted token through the resolver (cred_provider),
# never this path.
if (
spec is not None
and mcp_auth_header
and not isinstance(spec.config, (AuthorizationCodeConfig, TokenExchangeConfig))
and not isinstance(spec.config, (AuthorizationCodeConfig, PassthroughConfig, TokenExchangeConfig))
):
spec = None
auth_value = (
@ -2317,11 +2430,17 @@ class MCPServerManager:
server_url = server.url or ""
if spec is not None:
inbound_token = subject_token
if isinstance(spec.config, PassthroughConfig):
inbound_token, extra_headers = _take_forwarded_authorization(extra_headers)
per_server_token = _passthrough_token_from_mcp_auth_header(mcp_auth_header)
if per_server_token is not None:
inbound_token = per_server_token
resolved_auth, extra_headers = await self._resolve_v2_auth(
server=server,
spec=spec,
provider=provider,
subject_token=subject_token,
subject_token=inbound_token,
user_api_key_auth=user_api_key_auth,
extra_headers=extra_headers,
)
@ -3743,6 +3862,13 @@ class MCPServerManager:
user_api_key_auth=user_api_key_auth,
):
extra_headers = _without_authorization(extra_headers)
elif mcp_server.is_true_passthrough or mcp_server.is_oauth_delegate:
extra_headers = _client_forwarded_authorization_headers(
mcp_server=mcp_server,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
)
if mcp_server.extra_headers and raw_headers:
if extra_headers is None:
@ -3933,10 +4059,13 @@ class MCPServerManager:
return await self._cred_provider.has_user_token(to_subject(user_api_key_auth, None), spec)
async def invalidate_user_oauth_token_cache(self, user_id: str, server_id: str) -> None:
"""Drop the v2 chain's cached token for ``(user_id, server_id)`` after the credential row
changes (re-auth, revoke), so the next resolve reads the new row instead of serving the
replaced token until its cache TTL. Best-effort: a cache-drop failure is logged, never
raised, because the DB write already succeeded and the TTL remains the backstop.
"""Drop every cached token for ``(user_id, server_id)`` after the credential row changes
(re-auth, revoke, config-change purge): the v2 chain's cache and the legacy per-user token
cache, so the next resolve reads the new row instead of serving the replaced token until its
cache TTL, whichever path resolves it. This is the single invalidation point for per-user
OAuth tokens; callers must not evict individual caches directly. Best-effort: a cache-drop
failure is logged, never raised, because the DB write already succeeded and the TTL remains
the backstop.
"""
try:
await self._per_user_oauth_token_store.invalidate(user_id, server_id)
@ -3944,6 +4073,12 @@ class MCPServerManager:
verbose_logger.warning(
"Failed to invalidate cached MCP OAuth token for user=%s server=%s: %s", user_id, server_id, exc
)
try:
await self._per_user_token_cache.delete(user_id, server_id)
except Exception as exc: # noqa: BLE001 - cache drop is best-effort; TTL is the backstop
verbose_logger.warning(
"Failed to drop legacy cached MCP OAuth token for user=%s server=%s: %s", user_id, server_id, exc
)
async def _resolve_oauth2_headers_for_tool_call(
self,

10
litellm/proxy/_experimental/mcp_server/outbound_credentials/adapter.py
View file

@ -23,6 +23,7 @@ from litellm.proxy._experimental.mcp_server.outbound_credentials.types import (
AuthorizationCodeConfig,
CredError,
NoneConfig,
PassthroughConfig,
ServerSpec,
SharedKey,
Subject,
@ -62,9 +63,10 @@ def to_server_spec(server: MCPServer) -> Optional[ServerSpec]:
an ``assert_never`` tail, so a newly added auth mode fails the type gate here until it is
explicitly mapped or explicitly deferred, rather than silently falling through to v1. Live
modes: ``none``, the static-header family (``api_key`` plus the Authorization schemes,
all shared-key), ``oauth2`` per-user tokens (``authorization_code``), and
``oauth2_token_exchange`` (OBO); client_credentials (M2M), delegated/passthrough
oauth2, and SigV4 return None and stay on v1.
all shared-key), ``oauth2`` per-user tokens (``authorization_code``), ``oauth2_token_exchange``
(OBO), and the client-forwarded token modes ``true_passthrough`` / ``oauth_delegate``
(``PassthroughConfig``); client_credentials (M2M), delegated/passthrough oauth2, and SigV4
return None and stay on v1.
"""
if server.is_byok:
return None # per-user BYOK source not migrated yet -> defer to v1 (any auth_type)
@ -94,6 +96,8 @@ def to_server_spec(server: MCPServer) -> Optional[ServerSpec]:
)
# client_credentials (M2M) and delegate/passthrough oauth2 stay on v1
return None
case MCPAuth.true_passthrough | MCPAuth.oauth_delegate:
return ServerSpec(server_id=server.server_id, resource=resource, config=PassthroughConfig())
case MCPAuth.oauth2_token_exchange:
return _token_exchange_spec(server, resource)
case MCPAuth.aws_sigv4:

23
litellm/proxy/_experimental/mcp_server/outbound_credentials/resolver.py
View file

@ -7,10 +7,11 @@ no precedence cascade. It is wildcard-free with an `assert_never` tail, so addin
an arm fails the type gate (basedpyright `reportMatchNotExhaustive`); a bypassed gate fails loudly
at runtime instead of returning `None`.
`none` and `api_key` (shared-key source) are live, as is `authorization_code`, which reads the
user's token from the injected `OAuthTokenStore`, and `token_exchange`, which swaps the caller's
inbound token through the injected `TokenExchanger`. The remaining arms are `not_implemented` stubs
that each land in a follow-up PR with their seam. Pure v2: no imports from v1.
`none`, `api_key` (shared-key source), and `passthrough` (forwards the caller's own inbound token)
are live, as is `authorization_code`, which reads the user's token from the injected
`OAuthTokenStore`, and `token_exchange`, which swaps the caller's inbound token through the
injected `TokenExchanger`. The remaining arms are `not_implemented` stubs that each land in a
follow-up PR with their seam. Pure v2: no imports from v1.
"""
from __future__ import annotations
@ -97,7 +98,7 @@ class UpstreamCredentialProvider:
case ApiKeyConfig() as config:
return self._api_key(config)
case PassthroughConfig():
return _not_implemented(AuthSpecKind.passthrough)
return self._passthrough(subject)
case ClientCredentialsConfig():
return _not_implemented(AuthSpecKind.client_credentials)
case TokenExchangeConfig() as config:
@ -118,6 +119,18 @@ class UpstreamCredentialProvider:
"""
return await self._authz_token(subject, server) is not None
def _passthrough(self, subject: Subject) -> Result[httpx.Auth, CredError]:
"""Forward the caller's own upstream credential verbatim; the gateway mints nothing.
The inbound token is the caller's already-disambiguated ``Authorization`` (never the LiteLLM
admission credential; the edge adapter drops that before building the ``Subject``). When it is
absent the request is sent unauthenticated so the upstream's own 401 surfaces, rather than the
gateway challenging on the upstream's behalf.
"""
if subject.inbound_token is None:
return Ok(NoOpAuth())
return Ok(StaticHeaderAuth(subject.inbound_token.get_secret_value(), header_name="Authorization"))
def _api_key(self, config: ApiKeyConfig) -> Result[httpx.Auth, CredError]:
match config.key_source:
case SharedKey() as source:

44
litellm/proxy/_experimental/mcp_server/rest_endpoints.py
View file

@ -8,6 +8,7 @@ from typing import (
Dict,
List,
Literal,
Mapping,
Optional,
Set,
Tuple,
@ -68,6 +69,7 @@ if MCP_AVAILABLE:
from mcp.types import Tool as MCPTool
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
_UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
global_mcp_server_manager,
)
from litellm.proxy._experimental.mcp_server.oauth_utils import (
@ -78,7 +80,7 @@ if MCP_AVAILABLE:
MCPInfo,
MCPServer,
_apply_toolset_scope,
_fire_mcp_success_logging,
_fire_mcp_tool_call_logging,
_tool_name_matches,
execute_mcp_tool,
filter_tools_by_allowed_tools,
@ -86,23 +88,32 @@ if MCP_AVAILABLE:
########################################################
############ MCP Server REST API Routes #################
async def _safe_fire_mcp_success_logging(
async def _safe_fire_mcp_tool_call_logging(
logging_obj: Optional[Any],
result: Any,
start_time: datetime,
end_time: datetime,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
request_data: Optional[Mapping[str, object]] = None,
) -> None:
if logging_obj is None:
return
logging_results = await asyncio.gather(
_fire_mcp_success_logging(logging_obj, result, start_time, end_time),
_fire_mcp_tool_call_logging(
logging_obj,
result,
start_time,
end_time,
user_api_key_auth=user_api_key_auth,
request_data=request_data,
),
return_exceptions=True,
)
logging_error = logging_results[0]
if isinstance(logging_error, asyncio.CancelledError):
raise logging_error
if isinstance(logging_error, BaseException):
verbose_logger.warning("MCP tool success logging failed (continuing): %s", logging_error)
verbose_logger.warning("MCP tool call logging failed (continuing): %s", logging_error)
def _get_server_auth_header(
server,
@ -872,7 +883,14 @@ if MCP_AVAILABLE:
raw_headers=virtual_raw_headers,
litellm_logging_obj=virtual_logging_obj,
)
await _safe_fire_mcp_success_logging(virtual_logging_obj, result, _tool_start_time, datetime.now())
await _safe_fire_mcp_tool_call_logging(
virtual_logging_obj,
result,
_tool_start_time,
datetime.now(),
user_api_key_auth=user_api_key_dict,
request_data=data,
)
return result
# Validate required parameters early
@ -955,7 +973,14 @@ if MCP_AVAILABLE:
litellm_logging_obj=data.get("litellm_logging_obj"),
requested_server_id=canonical_server_id,
)
await _safe_fire_mcp_success_logging(logging_obj, result, _tool_start_time, datetime.now())
await _safe_fire_mcp_tool_call_logging(
logging_obj,
result,
_tool_start_time,
datetime.now(),
user_api_key_auth=user_api_key_dict,
request_data=data,
)
return result
except MCPMissingUserEnvVarsError as e:
verbose_logger.info(
@ -1297,8 +1322,13 @@ if MCP_AVAILABLE:
if isinstance(credentials, dict):
mcp_auth_header = credentials.get("auth_value")
# Authorization doubles as the admission fallback (LITELLM_API_KEY_HEADER_NAME_SECONDARY):
# when the primary x-litellm-api-key header is absent, the Authorization value is the
# caller's LiteLLM key, not an upstream token, and must never be forwarded upstream.
oauth2_headers: Optional[Dict[str, str]] = None
if new_mcp_server_request.auth_type == MCPAuth.oauth2:
if new_mcp_server_request.auth_type in _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES and headers.get(
MCPRequestHandler.LITELLM_API_KEY_HEADER_NAME_PRIMARY
):
oauth2_headers = MCPRequestHandler._get_oauth2_headers_from_headers(headers)
async def _list_tools_operation(client):

221
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -20,12 +20,14 @@ from typing import (
Callable,
Dict,
List,
Mapping,
Optional,
Set,
Tuple,
Union,
cast,
)
from urllib.parse import urlsplit, urlunsplit
import httpx
from fastapi import FastAPI, HTTPException
@ -47,7 +49,10 @@ from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
from litellm.proxy._experimental.mcp_server.discoverable_endpoints import (
get_request_base_url,
)
from litellm.proxy._experimental.mcp_server.exceptions import MCPUpstreamAuthError
from litellm.proxy._experimental.mcp_server.exceptions import (
MCPToolResultError,
MCPUpstreamAuthError,
)
from litellm.proxy._experimental.mcp_server.mcp_context import (
_mcp_active_toolset_id,
_mcp_gateway_initialize_instructions,
@ -60,6 +65,7 @@ from litellm.proxy._experimental.mcp_server.utils import (
LITELLM_MCP_SERVER_VERSION,
MCPMissingUserEnvVarsError,
add_server_prefix_to_name,
extract_mcp_tool_result_error_message,
get_server_prefix,
iter_known_server_prefixes,
)
@ -100,6 +106,27 @@ _MAX_STATEFUL_SESSIONS_PER_OWNER = 100
_MCP_ROUTING_PEEK_MAX_BYTES = 4096
def _redact_mcp_resource_url(url: Optional[str]) -> Optional[str]:
"""Reduce an MCP server URL to its origin (scheme + host + port) for logging.
Everything else is dropped: userinfo (``user:pass@``), the query string, the
fragment, and the path, because hosted MCP servers routinely embed the
credential in the path (e.g. ``/mcp/s/<token>``) and this value is persisted
in spend-log metadata that a caller who can invoke the tool can read back.
Returns None when the URL has no host to identify (nothing safe to log).
"""
if not isinstance(url, str) or not url:
return None
try:
parts = urlsplit(url)
except ValueError:
return None
if not parts.hostname:
return None
netloc = f"{parts.hostname}:{parts.port}" if parts.port else parts.hostname
return urlunsplit((parts.scheme, netloc, "", "", "")) or None
def _invalidate_byok_cred_cache(user_id: str, server_id: str) -> None:
"""Remove a (user_id, server_id) entry from the BYOK credential cache.
@ -331,6 +358,8 @@ if MCP_AVAILABLE:
)
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
MCPServerManager,
_caller_authorization_fans_out,
_client_forwarded_authorization_headers,
_should_strip_caller_authorization,
_without_authorization,
global_mcp_server_manager,
@ -1439,6 +1468,35 @@ if MCP_AVAILABLE:
return allowed_mcp_servers
def _client_has_per_server_auth_header(
server: MCPServer,
mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
) -> bool:
"""True if the request carries a per-server ``x-mcp-{alias}-authorization``
header for this server. This is the multi-server binding: it names one
upstream, so it is unambiguously the caller's upstream token regardless of
auth mode (never the LiteLLM admission credential).
Resolves through the same ``lookup_mcp_server_auth_in_headers`` egress uses, so
the connect gate and egress agree on which per-server header names match: a
dashboard client sends ``x-mcp-{sanitize_mcp_alias_for_header(alias)}-authorization``,
and matching only the raw alias here would 401 a token egress would forward.
"""
if not mcp_server_auth_headers:
return False
from litellm.proxy._experimental.mcp_server.utils import (
lookup_mcp_server_auth_in_headers,
)
server_headers = lookup_mcp_server_auth_in_headers(
mcp_server_auth_headers, alias=server.alias, server_name=server.server_name
)
if isinstance(server_headers, str):
return bool(server_headers.strip())
if isinstance(server_headers, dict):
return any(isinstance(hk, str) and hk.lower() == "authorization" for hk in server_headers)
return False
def _client_has_passthrough_authorization(
server: MCPServer,
oauth2_headers: Optional[Dict[str, str]],
@ -1456,24 +1514,7 @@ if MCP_AVAILABLE:
for k in oauth2_headers.keys():
if k.lower() == "authorization":
return True
if mcp_server_auth_headers:
for key in (server.alias, server.server_name, server.name):
if not key:
continue
server_headers = None
for k, v in mcp_server_auth_headers.items():
if k.lower() == key.lower():
server_headers = v
break
if server_headers is None:
continue
if isinstance(server_headers, str) and server_headers.strip():
return True
if isinstance(server_headers, dict):
for hk in server_headers.keys():
if hk.lower() == "authorization":
return True
return False
return _client_has_per_server_auth_header(server, mcp_server_auth_headers)
async def _get_user_oauth_extra_headers_from_db(
server: MCPServer,
@ -1528,8 +1569,16 @@ if MCP_AVAILABLE:
oauth2_headers: Optional[Dict[str, str]],
raw_headers: Optional[Dict[str, str]],
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
scope_servers: Optional[list[MCPServer]] = None,
) -> Tuple[Optional[Union[Dict[str, str], str]], Optional[Dict[str, str]]]:
"""Build auth and extra headers for a server."""
"""Build auth and extra headers for a server.
``scope_servers`` is the full server list a fan-out handler iterates. Passing it lets the
client-forwarded token modes withhold the caller's request-wide ``Authorization`` when
another server in the scope would also receive it (``_caller_authorization_fans_out``);
explicitly-addressed operations leave it None. Per-server ``x-mcp-{alias}-authorization``
headers are unaffected — they bind one token to one server and are the multi-server shape.
"""
server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers:
from litellm.proxy._experimental.mcp_server.utils import (
@ -1543,6 +1592,16 @@ if MCP_AVAILABLE:
)
extra_headers: Optional[Dict[str, str]] = None
is_client_forwarded_mode = server.is_true_passthrough or server.is_oauth_delegate
# In a multi-server listing scope the request-wide Authorization can only carry one token,
# so it is withheld from a client-forwarded server when another server in scope also consumes
# it (RFC 9700 cross-resource replay); such scopes must bind per-server via
# x-mcp-{alias}-authorization. The decision is computed once so BOTH the forwarding branch and
# the extra_headers copy loop below honor it — otherwise a server that lists Authorization in
# extra_headers would re-copy the withheld bearer from raw_headers and replay it anyway.
withhold_forwarded_authorization = is_client_forwarded_mode and _caller_authorization_fans_out(
server, scope_servers
)
if server.auth_type == MCPAuth.oauth2:
# For OAuth2 M2M servers, upstream Authorization must come from
# client_credentials token fetch, never from caller headers.
@ -1561,6 +1620,14 @@ if MCP_AVAILABLE:
user_api_key_auth=user_api_key_auth,
):
extra_headers = _without_authorization(extra_headers)
elif is_client_forwarded_mode:
if not withhold_forwarded_authorization:
extra_headers = _client_forwarded_authorization_headers(
mcp_server=server,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
)
if server.extra_headers and raw_headers:
if extra_headers is None:
@ -1581,7 +1648,9 @@ if MCP_AVAILABLE:
for header in server.extra_headers:
if not isinstance(header, str):
continue
if header.lower() == "authorization" and strip_caller_authorization:
if header.lower() == "authorization" and (
strip_caller_authorization or withhold_forwarded_authorization
):
continue
header_value = normalized_raw_headers.get(header.lower())
if header_value is None:
@ -1785,6 +1854,7 @@ if MCP_AVAILABLE:
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
scope_servers=allowed_mcp_servers,
)
# Prefer server-stored per-user OAuth when configured, so a stale
@ -1971,6 +2041,7 @@ if MCP_AVAILABLE:
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
scope_servers=allowed_mcp_servers,
)
try:
@ -2023,6 +2094,7 @@ if MCP_AVAILABLE:
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
scope_servers=allowed_mcp_servers,
)
try:
@ -2073,6 +2145,7 @@ if MCP_AVAILABLE:
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
user_api_key_auth=user_api_key_auth,
scope_servers=allowed_mcp_servers,
)
try:
@ -2743,12 +2816,40 @@ if MCP_AVAILABLE:
return response
async def _fire_mcp_success_logging(
_MCP_CREDENTIAL_REQUEST_FIELDS = frozenset(
{
"raw_headers",
"mcp_auth_header",
"mcp_server_auth_headers",
"oauth2_headers",
"user_api_key_auth",
}
)
async def _fire_mcp_tool_call_logging(
logging_obj: LiteLLMLoggingObj,
result: Any,
start_time: datetime,
end_time: datetime,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
request_data: Optional[Mapping[str, object]] = None,
) -> None:
"""Fire post-call logging for an executed MCP tool call.
A result with ``isError=True`` is logged as a failure (``status="failure"``
payload, so OTel marks the span ERROR) while the HTTP wire behavior stays
200 + ``isError: true`` per the MCP spec. The error check runs after
``async_post_mcp_tool_call_hook`` because guardrails may flip the result
to ``isError=True`` in that hook. Raised exceptions never reach here (the
``@client`` wrapper and ``call_mcp_tool``'s except path log those), so
this cannot double-log a failure.
``request_data`` may carry credential-bearing fields (the REST path puts
``raw_headers``, ``mcp_auth_header``, ``mcp_server_auth_headers``, and
``oauth2_headers`` at the top level of its data dict), so those are
stripped before the dict is handed to ``post_call_failure_hook``
callbacks.
"""
logging_obj.post_call(original_response=result)
await logging_obj.async_post_mcp_tool_call_hook(
kwargs=logging_obj.model_call_details,
@ -2757,7 +2858,31 @@ if MCP_AVAILABLE:
end_time=end_time,
)
logging_obj.call_type = CallTypes.call_mcp_tool.value
await logging_obj.async_success_handler(result=result, start_time=start_time, end_time=end_time)
error_message = extract_mcp_tool_result_error_message(result)
if error_message is None:
await logging_obj.async_success_handler(result=result, start_time=start_time, end_time=end_time)
return
logging_obj.has_run_logging(event_type="sync_success")
logging_obj.has_run_logging(event_type="async_success")
tool_error = MCPToolResultError(error_message)
logging_obj.failure_handler(tool_error, "", start_time, end_time)
await logging_obj.async_failure_handler(tool_error, "", start_time, end_time)
if user_api_key_auth is None:
return
from litellm.proxy.proxy_server import proxy_logging_obj
if proxy_logging_obj:
sanitized_request_data = {
key: value for key, value in (request_data or {}).items() if key not in _MCP_CREDENTIAL_REQUEST_FIELDS
}
await proxy_logging_obj.post_call_failure_hook(
request_data=sanitized_request_data,
original_exception=tool_error,
user_api_key_dict=user_api_key_auth,
route="/mcp/call_tool",
)
@client
async def call_mcp_tool(
@ -2833,7 +2958,14 @@ if MCP_AVAILABLE:
raise
if litellm_logging_obj:
await _fire_mcp_success_logging(litellm_logging_obj, response, start_time, datetime.now())
await _fire_mcp_tool_call_logging(
logging_obj=litellm_logging_obj,
result=response,
start_time=start_time,
end_time=datetime.now(),
user_api_key_auth=user_api_key_auth,
request_data=kwargs,
)
return response
async def mcp_get_prompt(
@ -2954,6 +3086,8 @@ if MCP_AVAILABLE:
mcp_server_logo_url=mcp_info.get("logo_url"),
namespaced_tool_name=namespaced_tool_name,
mcp_session_id=session_id,
mcp_auth_mode=mcp_server.auth_type,
mcp_server_resource=_redact_mcp_resource_url(mcp_server.url),
)
else:
return StandardLoggingMCPToolCall(
@ -3518,6 +3652,41 @@ if MCP_AVAILABLE:
headers={"www-authenticate": www_authenticate},
)
if (
server
and server.is_oauth_delegate
and len(mcp_servers or []) == 1
and _get_forwarded_auth_from_scope(scope) is None
and not _client_has_per_server_auth_header(server, mcp_server_auth_headers)
):
www_authenticate = _get_passthrough_www_authenticate(
scope=scope,
server_name=server_name,
)
raise HTTPException(
status_code=401,
detail="Unauthorized",
headers={"www-authenticate": www_authenticate},
)
if (
server
and server.is_true_passthrough
and len(mcp_servers or []) == 1
and not _scope_has_authorization_header(scope)
and not _client_has_per_server_auth_header(server, mcp_server_auth_headers)
):
upstream_status, upstream_www_authenticate = await _probe_upstream_auth(server.url or "", "")
if upstream_status == 401 and upstream_www_authenticate:
raise HTTPException(
status_code=401,
detail="Unauthorized",
headers={"www-authenticate": upstream_www_authenticate},
)
def _scope_has_authorization_header(scope: Scope) -> bool:
return any(key.lower() == b"authorization" for key, _ in scope.get("headers", []))
def _get_forwarded_auth_from_scope(scope: Scope) -> Optional[str]:
"""Return the upstream-bound ``Authorization`` header value, or None.
@ -3545,7 +3714,7 @@ if MCP_AVAILABLE:
url: str,
auth_header: str,
timeout: float = 5.0,
) -> tuple:
) -> tuple[int, Optional[str]]:
"""JSON-RPC initialize-probe the upstream URL to check whether the token is accepted.
Uses POST so StreamableHTTP MCP servers run the same auth path as a
@ -3575,8 +3744,8 @@ if MCP_AVAILABLE:
},
}
probe_headers = {
"Authorization": auth_header,
"Accept": "application/json, text/event-stream",
**({"Authorization": auth_header} if auth_header else {}),
}
try:
resp = await client.post(

19
litellm/proxy/_experimental/mcp_server/utils.py
View file

@ -415,6 +415,25 @@ def validate_mcp_server_name(server_name: str, raise_http_exception: bool = Fals
raise Exception(error_message)
def extract_mcp_tool_result_error_message(result: object) -> Optional[str]:
"""The first text content of an ``isError=True`` tool result, or ``None``
when the result is not an error.
Accepts both ``mcp.types.CallToolResult`` objects and their dict
equivalents, duck-typed so the ``mcp`` package is not required.
"""
is_error: object = result.get("isError") if isinstance(result, Mapping) else getattr(result, "isError", None)
if is_error is not True:
return None
content: object = result.get("content") if isinstance(result, Mapping) else getattr(result, "content", None)
if isinstance(content, (list, tuple)):
for item in content:
text: object = item.get("text") if isinstance(item, Mapping) else getattr(item, "text", None)
if isinstance(text, str) and text:
return text
return "MCP tool call returned isError=true"
TOOL_DISPLAY_NAME_PATTERN = re.compile(r"^[a-zA-Z0-9_-]+$")

25
litellm/proxy/auth/auth_checks.py
View file

@ -98,7 +98,10 @@ from litellm.repositories.user_repository import UserRepository
from litellm.router import Router
from litellm.utils import get_utc_datetime
from .auth_checks_organization import organization_role_based_access_check
from .auth_checks_organization import (
add_team_org_context_to_request_body,
organization_role_based_access_check,
)
from .auth_utils import get_model_from_request
if TYPE_CHECKING:
@ -707,10 +710,28 @@ async def common_checks(
# 10 [OPTIONAL] Organization RBAC checks
organization_role_based_access_check(user_object=user_object, route=route, request_body=request_body)
async def _fetch_team_org_id(team_id: str) -> Optional[str]:
try:
team = await get_team_object(
team_id=team_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
except HTTPException:
return None
return team.organization_id
request_body_for_route_check = await add_team_org_context_to_request_body(
route=route,
request_body=request_body,
fetch_team_org_id=_fetch_team_org_id,
)
_is_route_allowed = _is_api_route_allowed(
route=route,
request=request,
request_data=request_body,
request_data=request_body_for_route_check,
valid_token=valid_token,
user_obj=user_object,
)

32
litellm/proxy/auth/auth_checks_organization.py
View file

@ -2,7 +2,7 @@
Auth Checks for Organizations
"""
from typing import Dict, List, Optional, Tuple
from typing import Awaitable, Callable, Dict, List, Optional, Tuple
from fastapi import status
@ -170,3 +170,33 @@ def _user_is_org_admin(
# User must be admin of ALL requested orgs, not just any one
return all(org_id in admin_org_ids for org_id in candidate_org_ids)
TEAM_ORG_CONTEXT_ROUTES = frozenset({"/team/update"})
async def add_team_org_context_to_request_body(
route: str,
request_body: dict,
fetch_team_org_id: Callable[[str], Awaitable[Optional[str]]],
) -> dict:
"""
Return a copy of request_body with organization_id resolved from the target
team when the route identifies the team by team_id and the caller did not
pass organization_id. This lets an org admin of the team's own org reach the
org-scoped branch of the route gate (which keys off organization_id) without
the client having to send it. Returns request_body unchanged when it does
not apply, so callers that already pass organization_id and non-team routes
are untouched.
"""
if route not in TEAM_ORG_CONTEXT_ROUTES:
return request_body
if request_body.get("organization_id"):
return request_body
team_id = request_body.get("team_id")
if not isinstance(team_id, str) or not team_id:
return request_body
org_id = await fetch_team_org_id(team_id)
if not org_id:
return request_body
return {**request_body, "organization_id": org_id}

60
litellm/proxy/guardrails/_content_utils.py
View file

@ -32,6 +32,9 @@ def is_text_content_call_type(call_type: str) -> bool:
return call_type in TEXT_CONTENT_CALL_TYPES
TEXT_PART_TYPES: FrozenSet[str] = frozenset({"text", "input_text", "output_text"})
def _iter_text_parts_in_content(content: Any) -> Iterator[str]:
"""Yield text fragments from a ``message.content`` value (string or
multimodal list). Non-text parts (images, audio, …) are skipped."""
@ -48,7 +51,7 @@ def _iter_text_parts_in_content(content: Any) -> Iterator[str]:
continue
if not isinstance(part, dict):
continue
if part.get("type") == "text":
if part.get("type") in TEXT_PART_TYPES:
text = part.get("text")
if isinstance(text, str) and text:
yield text
@ -58,14 +61,20 @@ def _coerce_input_to_messages(input_value: Any) -> List[Dict[str, Any]]:
"""Coerce a Responses-API ``data["input"]`` value into chat-style messages."""
if isinstance(input_value, str):
return [{"role": "user", "content": input_value}]
if isinstance(input_value, list):
if input_value and all(isinstance(item, dict) and "role" in item for item in input_value):
return list(input_value)
# Mixed lists (content-part dicts + bare strings) and pure
# string/dict lists all become a single user message; the content
# iterator below handles each element type uniformly.
return [{"role": "user", "content": input_value}]
return []
if not isinstance(input_value, list):
return []
messages: List[Dict[str, Any]] = []
for item in input_value:
if isinstance(item, str):
messages.append({"role": "user", "content": item})
elif isinstance(item, dict):
if item.get("type") in TEXT_PART_TYPES:
messages.append({"role": item.get("role") or "user", "content": [item]})
elif "content" in item:
messages.append({"role": item.get("role") or "user", "content": item["content"]})
elif item.get("type") == "function_call_output" and "output" in item:
messages.append({"role": item.get("role") or "tool", "content": item["output"]})
return messages
def _iter_inspection_messages(data: Dict[str, Any]) -> Iterator[Dict[str, Any]]:
@ -112,7 +121,7 @@ def walk_user_text(data: Dict[str, Any], visit: Callable[[str], str]) -> int:
new_parts.append(visit(part))
elif (
isinstance(part, dict)
and part.get("type") == "text"
and part.get("type") in TEXT_PART_TYPES
and isinstance(part.get("text"), str)
and part["text"]
):
@ -136,25 +145,20 @@ def walk_user_text(data: Dict[str, Any], visit: Callable[[str], str]) -> int:
data["input"] = visit(input_value)
return visited
if isinstance(input_value, list):
# List of full messages: rewrite each message's content.
if input_value and all(isinstance(item, dict) and "role" in item for item in input_value):
for item in input_value:
if "content" in item:
item["content"] = _rewrite_content(item["content"])
return visited
# List of content parts and/or bare strings: rewrite in place.
for idx, item in enumerate(input_value):
if isinstance(item, str) and item:
visited += 1
input_value[idx] = visit(item)
elif (
isinstance(item, dict)
and item.get("type") == "text"
and isinstance(item.get("text"), str)
and item["text"]
):
visited += 1
input_value[idx] = {**item, "text": visit(item["text"])}
if isinstance(item, str):
if item:
visited += 1
input_value[idx] = visit(item)
elif isinstance(item, dict):
if item.get("type") in TEXT_PART_TYPES:
if isinstance(item.get("text"), str) and item["text"]:
visited += 1
input_value[idx] = {**item, "text": visit(item["text"])}
elif "content" in item:
item["content"] = _rewrite_content(item["content"])
elif item.get("type") == "function_call_output" and "output" in item:
item["output"] = _rewrite_content(item["output"])
return visited
return visited

17
litellm/proxy/guardrails/guardrail_hooks/aim/aim.py
View file

@ -93,11 +93,10 @@ class AimGuardrail(CustomGuardrail):
user_email=user_email,
litellm_call_id=call_id,
)
# Covers multimodal list content + Responses-API input.
response = await self.async_handler.post(
f"{self.api_base}/fw/v1/analyze",
headers=headers,
json={"messages": build_inspection_messages(data)},
json={"messages": self._build_aim_inspection_messages(data)},
)
response.raise_for_status()
res = response.json()
@ -116,6 +115,15 @@ class AimGuardrail(CustomGuardrail):
verbose_proxy_logger.error(f"Aim: {action_type} action")
return data
@staticmethod
def _build_aim_inspection_messages(data: dict) -> list[dict[str, str]]:
"""AIM validates against the OpenAI chat schema. Bare ``role: "tool"``
without ``tool_call_id`` and bare ``role: "function"`` without ``name``
are rejected; the flatten drops those fields, so any role outside
``{system, user, assistant}`` collapses to ``user`` for the AIM POST."""
safe_roles = {"system", "user", "assistant"}
return [{**m, "role": "user"} if m["role"] not in safe_roles else m for m in build_inspection_messages(data)]
@staticmethod
def _rejection(message: str, *, openai_code: str | None = None) -> ProxyException:
return ProxyException(
@ -177,7 +185,10 @@ class AimGuardrail(CustomGuardrail):
user_email=user_email,
litellm_call_id=call_id,
),
json={"messages": build_inspection_messages(request_data) + [{"role": "assistant", "content": output}]},
json={
"messages": self._build_aim_inspection_messages(request_data)
+ [{"role": "assistant", "content": output}]
},
)
response.raise_for_status()
res = response.json()

145
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py
View file

@ -33,7 +33,7 @@ import litellm
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.core_helpers import redact_nested_match_and_regex_keys
from litellm.caching import DualCache
from litellm.exceptions import GuardrailInterventionNormalStringError
from litellm.exceptions import ModifyResponseException
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.custom_httpx.http_handler import (
@ -754,7 +754,9 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
)
bedrock_guardrail_response = BedrockGuardrailResponse(**_json_response)
if self._should_raise_guardrail_blocked_exception(bedrock_guardrail_response):
raise self._get_http_exception_for_blocked_guardrail(bedrock_guardrail_response)
raise self._get_http_exception_for_blocked_guardrail(
bedrock_guardrail_response, request_data=request_data
)
else:
status_code, detail_message = self._parse_bedrock_guardrail_error_response(httpx_response)
verbose_proxy_logger.error(
@ -1027,8 +1029,8 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
return blocked
def _get_http_exception_for_blocked_guardrail(
self, response: BedrockGuardrailResponse
) -> Union[HTTPException, GuardrailInterventionNormalStringError]:
self, response: BedrockGuardrailResponse, request_data: Optional[dict] = None
) -> Union[HTTPException, ModifyResponseException]:
"""
Get the HTTP exception for a blocked guardrail.
"""
@ -1040,7 +1042,13 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
bedrock_guardrail_output_text += output.get("text") or ""
if self.disable_exception_on_block is True:
return GuardrailInterventionNormalStringError(message=bedrock_guardrail_output_text)
_request_data = request_data or {}
return ModifyResponseException(
message=bedrock_guardrail_output_text,
model=_request_data.get("model", "bedrock-guardrail"),
request_data=_request_data,
guardrail_name=self.guardrail_name,
)
detail: Dict[str, Any] = {
"error": "Violated guardrail policy",
@ -1134,18 +1142,6 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
# This means all actions were ANONYMIZED or NONE, so don't raise exception
return False
def create_guardrail_blocked_response(self, response: str) -> ModelResponse:
from litellm.types.utils import Choices, Message, ModelResponse
return ModelResponse(
choices=[
Choices(
message=Message(content=response),
)
],
model="bedrock-guardrail",
)
async def async_pre_call_hook(
self,
user_api_key_dict: UserAPIKeyAuth,
@ -1183,16 +1179,15 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
#########################################################
########## 1. Make the Bedrock API request ##########
#########################################################
bedrock_guardrail_response: Optional[Union[BedrockGuardrailResponse, str]] = None
try:
bedrock_guardrail_response = await self.make_bedrock_api_request(
source="INPUT",
messages=filtered_messages,
request_data=data,
logging_event_type=GuardrailEventHooks.pre_call,
)
except GuardrailInterventionNormalStringError as e:
bedrock_guardrail_response = e.message
# A block with disable_exception_on_block=True raises ModifyResponseException
# from make_bedrock_api_request; that propagates to the endpoint handler,
# which returns a 200 whose message is the guardrail's blockedInputMessaging.
bedrock_guardrail_response = await self.make_bedrock_api_request(
source="INPUT",
messages=filtered_messages,
request_data=data,
logging_event_type=GuardrailEventHooks.pre_call,
)
#########################################################
#########################################################
@ -1207,8 +1202,6 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
updated_target_messages=updated_subset,
target_indices=filter_result.target_indices,
)
if isinstance(bedrock_guardrail_response, str):
data["mock_response"] = self.create_guardrail_blocked_response(response=bedrock_guardrail_response)
#########################################################
########## 3. Add the guardrail to the applied guardrails header ##########
@ -1248,16 +1241,19 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
#########################################################
########## 1. Make the Bedrock API request ##########
#########################################################
bedrock_guardrail_response: Optional[Union[BedrockGuardrailResponse, str]] = None
try:
bedrock_guardrail_response = await self.make_bedrock_api_request(
source="INPUT",
messages=filtered_messages,
request_data=data,
logging_event_type=GuardrailEventHooks.during_call,
)
except GuardrailInterventionNormalStringError as e:
bedrock_guardrail_response = e.message
# A block with disable_exception_on_block=True raises ModifyResponseException
# from make_bedrock_api_request. Because during_call runs in an asyncio.gather
# alongside the LLM call (common_request_processing.py), swallowing the
# exception here to set data["mock_response"] was ineffective: route_request
# unpacked kwargs before this hook ran, and the LLM task's response was taken
# unconditionally. Letting the exception propagate cancels the LLM task and
# the endpoint handler returns the block response.
bedrock_guardrail_response = await self.make_bedrock_api_request(
source="INPUT",
messages=filtered_messages,
request_data=data,
logging_event_type=GuardrailEventHooks.during_call,
)
#########################################################
#########################################################
@ -1272,8 +1268,6 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
updated_target_messages=updated_subset,
target_indices=filter_result.target_indices,
)
if isinstance(bedrock_guardrail_response, str):
data["mock_response"] = self.create_guardrail_blocked_response(response=bedrock_guardrail_response)
#########################################################
########## 3. Add the guardrail to the applied guardrails header ##########
@ -1323,7 +1317,11 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
# users should configure if they want input validation. Running an
# extra INPUT scan here produced a duplicate post-call entry in the
# trace and made no semantic sense for a "post-call" event.
output_content_bedrock: Optional[Union[BedrockGuardrailResponse, str]] = None
# A block with disable_exception_on_block=True raises ModifyResponseException
# from make_bedrock_api_request; that propagates to the endpoint handler,
# which returns a 200 whose message is the guardrail's blockedInputMessaging.
# Attach the LLM response to original_response so the synthetic block reply
# reports the real token usage the upstream call consumed instead of zero.
try:
output_content_bedrock = await self.make_bedrock_api_request(
source="OUTPUT",
@ -1332,15 +1330,15 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
request_data=data,
logging_event_type=GuardrailEventHooks.post_call,
)
except GuardrailInterventionNormalStringError as e:
output_content_bedrock = e.message
except ModifyResponseException as e:
if e.original_response is None:
e.original_response = response
raise
#########################################################
########## 2. Apply masking to response with output guardrail response ##########
#########################################################
if isinstance(output_content_bedrock, str):
response = self.create_guardrail_blocked_response(response=output_content_bedrock)
elif output_content_bedrock is not None:
if output_content_bedrock is not None:
self._apply_masking_to_response(
response=response,
bedrock_guardrail_response=output_content_bedrock,
@ -1357,7 +1355,7 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
def _update_messages_with_updated_bedrock_guardrail_response(
self,
messages: List[AllMessageValues],
bedrock_guardrail_response: Union[BedrockGuardrailResponse, str],
bedrock_guardrail_response: BedrockGuardrailResponse,
) -> List[AllMessageValues]:
"""
Use the output from the bedrock guardrail to mask sensitive content in messages.
@ -1369,8 +1367,6 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
Returns:
List of messages with content masked according to guardrail response
"""
if isinstance(bedrock_guardrail_response, str):
return messages
# Get masked texts from guardrail response
masked_texts = self._extract_masked_texts_from_response(bedrock_guardrail_response)
@ -1422,7 +1418,14 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
# pre_call / during_call. Bedrock will raise if the response
# violates the guardrail policy.
###################################################################
output_guardrail_response: Optional[Union[BedrockGuardrailResponse, str]] = None
# A block with disable_exception_on_block=True raises ModifyResponseException
# from make_bedrock_api_request. Non-streaming paths let it propagate so
# the endpoint handler turns it into a 200. Streaming can't do that: the
# SSE response headers are already flushed, so a raise would be serialized
# as an error frame by async_streaming_data_generator. Instead, replace
# the assembled response with the synthetic block content in-place and
# yield it as a normal stream, matching the shape a non-streaming block
# produces.
try:
output_guardrail_response = await self.make_bedrock_api_request(
source="OUTPUT",
@ -1431,15 +1434,31 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
request_data=request_data,
logging_event_type=GuardrailEventHooks.post_call,
)
except GuardrailInterventionNormalStringError as e:
output_guardrail_response = e.message
except ModifyResponseException as e:
# Preserve upstream usage from the LLM call we already
# consumed. Non-streaming blocks carry it via
# ModifyResponseException.original_response +
# _blocked_response_usage; streaming has to do the copy
# itself since the exception can't escape this generator.
_original_usage = getattr(assembled_model_response, "usage", None)
assembled_model_response = ModelResponse(
choices=[
Choices(
index=0,
message=Message(role="assistant", content=e.message),
finish_reason="content_filter",
)
],
model=e.model,
)
if _original_usage is not None:
assembled_model_response.usage = _original_usage
output_guardrail_response = None
#########################################################################
########## 2. Apply masking to response with output guardrail response ##########
#########################################################################
if isinstance(output_guardrail_response, str):
assembled_model_response = self.create_guardrail_blocked_response(response=output_guardrail_response)
elif output_guardrail_response is not None:
if output_guardrail_response is not None:
self._apply_masking_to_response(
response=assembled_model_response,
bedrock_guardrail_response=output_guardrail_response,
@ -1732,13 +1751,13 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
inputs["texts"] = masked_texts
return inputs
except (HTTPException, GuardrailInterventionNormalStringError):
# Let guardrail blocking exceptions propagate as-is so the proxy
# can return the correct HTTP status (400) or handle the
# GuardrailInterventionNormalStringError for disable_exception_on_block mode.
# Without this, the generic except below wraps them into a plain
# Exception, losing the HTTP semantics and preventing the proxy
# from properly blocking the call.
except (HTTPException, ModifyResponseException):
# Let guardrail blocking exceptions propagate as-is so the proxy can
# return the correct HTTP status (400 for HTTPException, 200 with the
# block message for ModifyResponseException in disable_exception_on_block
# mode). Without this, the generic except below wraps them into a plain
# Exception, losing the semantics and preventing the proxy from
# properly blocking the call.
raise
except Exception as e:
verbose_proxy_logger.error("Bedrock Guardrail: Failed to apply guardrail: %s", str(e))

48
litellm/proxy/guardrails/guardrail_hooks/grayswan/grayswan.py
View file

@ -213,7 +213,7 @@ class GraySwanGuardrail(CustomGuardrail):
verbose_proxy_logger.debug("Gray Swan Guardrail: dynamic extra_body=%s", safe_dumps(dynamic_body))
# Prepare and send payload
payload = self._prepare_payload(messages, dynamic_body, request_data)
payload = self._prepare_payload(messages, dynamic_body, request_data, logging_obj)
if payload is None:
return inputs
@ -502,10 +502,38 @@ class GraySwanGuardrail(CustomGuardrail):
"grayswan-api-key": self.api_key,
}
def _extract_inbound_headers(
self,
request_data: dict,
logging_obj: Optional["LiteLLMLoggingObj"] = None,
) -> Optional[dict[str, str]]:
headers = (request_data.get("proxy_server_request") or {}).get("headers")
if not headers:
headers = request_data.get("headers")
if not headers:
headers = (request_data.get("metadata") or {}).get("headers")
if not headers and logging_obj and getattr(logging_obj, "model_call_details", None):
headers = (
(logging_obj.model_call_details or {}).get("litellm_params", {}).get("metadata", {}).get("headers")
)
if not isinstance(headers, dict):
return None
forwarded_header_names = ("shade_scan_id",)
forwarded_headers = {}
for key, value in headers.items():
if str(key).lower() in forwarded_header_names:
forwarded_headers[str(key)] = str(value)
return forwarded_headers or None
def _prepare_payload(
self, messages: List[Dict[str, str]], dynamic_body: dict, request_data: dict
) -> Optional[Dict[str, Any]]:
payload: Dict[str, Any] = {"messages": messages}
self,
messages: list[dict[str, str]],
dynamic_body: dict,
request_data: dict,
logging_obj: Optional["LiteLLMLoggingObj"] = None,
) -> Optional[dict[str, Any]]:
payload: dict[str, Any] = {"messages": messages}
categories = dynamic_body.get("categories") or self.categories
if categories:
@ -523,10 +551,16 @@ class GraySwanGuardrail(CustomGuardrail):
if "metadata" in dynamic_body:
payload["metadata"] = dynamic_body["metadata"]
inbound_headers = self._extract_inbound_headers(request_data, logging_obj)
litellm_metadata = request_data.get("litellm_metadata")
if isinstance(litellm_metadata, dict) and litellm_metadata:
cleaned_litellm_metadata = dict(litellm_metadata)
# cleaned_litellm_metadata.pop("user_api_key_auth", None)
cleaned_litellm_metadata = dict(litellm_metadata) if isinstance(litellm_metadata, dict) else {}
if inbound_headers:
existing_headers = cleaned_litellm_metadata.get("headers")
cleaned_litellm_metadata["headers"] = (
{**existing_headers, **inbound_headers} if isinstance(existing_headers, dict) else inbound_headers
)
if cleaned_litellm_metadata:
sanitized = safe_json_loads(safe_dumps(cleaned_litellm_metadata), default={})
if isinstance(sanitized, dict) and sanitized:
payload["litellm_metadata"] = sanitized

24
litellm/proxy/management_endpoints/key_management_endpoints.py
View file

@ -111,6 +111,7 @@ from litellm.repositories.verification_token_repository import (
VerificationTokenRepository,
)
from litellm.router import Router
from litellm.secret_managers.base_secret_manager import raise_if_unsafe_secret_name
from litellm.secret_managers.main import get_secret
from litellm.types.proxy.management_endpoints.key_management_endpoints import (
BulkUpdateKeyRequest,
@ -6291,8 +6292,13 @@ def _validate_key_alias_format(key_alias: Optional[str]) -> None:
"""
Validate the format of the key_alias.
Gated behind ``litellm.enable_key_alias_format_validation`` (default **False**).
When disabled, no validation is performed so existing workflows are not broken.
A baseline validation always runs, regardless of
``litellm.enable_key_alias_format_validation``.
The remaining charset/length rules are gated behind
``litellm.enable_key_alias_format_validation`` (default **False**). When disabled,
only the baseline validation above is performed, so existing workflows are not
broken.
Rules (when enabled):
- None is OK (no alias).
@ -6300,10 +6306,20 @@ def _validate_key_alias_format(key_alias: Optional[str]) -> None:
- start/end with alphanumeric
- only allow a-zA-Z0-9_-/.@
"""
if not litellm.enable_key_alias_format_validation:
if key_alias is None:
return
if key_alias is None:
try:
raise_if_unsafe_secret_name(key_alias)
except ValueError:
raise ProxyException(
message="Invalid key_alias",
type=ProxyErrorTypes.bad_request_error,
param="key_alias",
code=400,
)
if not litellm.enable_key_alias_format_validation:
return
if not _KEY_ALIAS_PATTERN.match(key_alias):

39
litellm/proxy/management_endpoints/mcp_management_endpoints.py
View file

@ -125,7 +125,9 @@ if MCP_AVAILABLE:
get_user_env_vars_bulk,
get_user_oauth_credential,
list_user_oauth_credentials,
mcp_oauth_token_identity,
merge_user_env_vars,
purge_user_oauth_credentials_for_server,
reject_mcp_server,
store_user_credential,
store_user_oauth_credential,
@ -2318,6 +2320,19 @@ if MCP_AVAILABLE:
},
)
# Snapshot the pre-update identity so we can detect a mint-relevant change below. The read is
# advisory (it only feeds the stale-token purge decision), so a failure skips the purge with a
# warning instead of failing the edit, whose primary job is the update itself.
try:
old_server_record = await get_mcp_server(prisma_client, payload.server_id)
except Exception as exc: # noqa: BLE001 - advisory read; invalidation is best-effort end-to-end
verbose_logger.warning(
"MCP server %s: could not snapshot the pre-update record; skipping the stale-token check: %s",
payload.server_id,
exc,
)
old_server_record = None
# try to update the mcp server
mcp_server_record_updated = await update_mcp_server(
prisma_client,
@ -2336,6 +2351,30 @@ if MCP_AVAILABLE:
# Ensure registry is up to date by reloading from database
await global_mcp_server_manager.reload_servers_from_database()
# If a field that determines which upstream OAuth token gets minted changed (url/audience, OAuth
# mode/grant, authorization-server endpoints, or the OAuth client + scopes), every stored per-user
# token was minted for the old configuration and is stale. Purge them (DB + cache) so the next
# tool call re-authorizes instead of forwarding a token for a resource/AS/client that no longer
# matches. Best-effort: a purge failure must not fail the update, whose primary job already
# succeeded.
if old_server_record is not None and mcp_oauth_token_identity(old_server_record) != mcp_oauth_token_identity(
mcp_server_record_updated
):
try:
purged = await purge_user_oauth_credentials_for_server(prisma_client, payload.server_id)
if purged:
verbose_logger.info(
"MCP server %s: purged %d stale per-user OAuth token(s) after a mint-relevant config change",
payload.server_id,
purged,
)
except Exception as exc: # noqa: BLE001 - purge is best-effort; the server update already succeeded
verbose_logger.warning(
"MCP server %s: failed to purge stale per-user OAuth tokens after config change: %s",
payload.server_id,
exc,
)
# TODO: Enterprise: Finish audit log trail
if litellm.store_audit_logs:
pass

34
litellm/proxy/management_endpoints/ui_sso.py
View file

@ -113,7 +113,7 @@ from litellm.proxy.utils import (
from litellm.repositories.table_repositories import SSOConfigRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.secret_managers.main import get_secret_bool, str_to_bool
from litellm.secret_managers.main import get_secret_bool, get_secret_str, str_to_bool
from litellm.types.proxy.management_endpoints.ui_sso import * # noqa: F403
from litellm.types.proxy.management_endpoints.ui_sso import (
DefaultTeamSSOParams,
@ -1746,6 +1746,18 @@ async def auth_callback(request: Request, state: Optional[str] = None):
"""Verify login"""
verbose_proxy_logger.info(f"Starting SSO callback with state: {state}")
oauth_error = request.query_params.get("error")
if oauth_error:
oauth_error_description = request.query_params.get("error_description")
verbose_proxy_logger.warning(
f"SSO callback received OAuth error: {oauth_error}, description: {oauth_error_description}"
)
raise HTTPException(
status_code=401,
detail=f"OAuth error: {oauth_error}"
+ (f", error_description: {oauth_error_description}" if oauth_error_description else ""),
)
# Check if this is a CLI login (state starts with our CLI prefix)
from litellm.constants import LITELLM_CLI_SESSION_TOKEN_PREFIX
from litellm.proxy._types import LiteLLM_JWTAuth
@ -3737,8 +3749,7 @@ class MicrosoftSSOHandler:
Handles Microsoft SSO callback response and returns a CustomOpenID object
"""
graph_api_base_url = "https://graph.microsoft.com/v1.0"
graph_api_user_groups_endpoint = f"{graph_api_base_url}/me/memberOf"
DEFAULT_GRAPH_API_BASE_URL = "https://graph.microsoft.com/v1.0"
"""
Constants
@ -3748,6 +3759,19 @@ class MicrosoftSSOHandler:
# used for debugging to show the user groups litellm found from Graph API
GRAPH_API_RESPONSE_KEY = "graph_api_user_groups"
@staticmethod
def get_graph_api_base_url() -> str:
"""
Returns the Microsoft Graph API base URL, configurable via the
`MICROSOFT_GRAPH_ENDPOINT` env var so non-default clouds such as Azure
Government (GCC High) can point at `https://graph.microsoft.us/v1.0`
"""
return get_secret_str("MICROSOFT_GRAPH_ENDPOINT") or MicrosoftSSOHandler.DEFAULT_GRAPH_API_BASE_URL
@staticmethod
def get_graph_api_user_groups_endpoint() -> str:
return f"{MicrosoftSSOHandler.get_graph_api_base_url()}/me/memberOf"
@staticmethod
async def get_microsoft_callback_response(
request: Request,
@ -3924,7 +3948,7 @@ class MicrosoftSSOHandler:
# Fetch user membership from Microsoft Graph API
all_group_ids = []
next_link: Optional[str] = MicrosoftSSOHandler.graph_api_user_groups_endpoint
next_link: Optional[str] = MicrosoftSSOHandler.get_graph_api_user_groups_endpoint()
auth_headers = {"Authorization": f"Bearer {access_token}"}
page_count = 0
@ -4007,7 +4031,7 @@ class MicrosoftSSOHandler:
Users use Enterprise Applications to manage Groups and Users on Microsoft Entra ID
"""
base_url = "https://graph.microsoft.com/v1.0"
base_url = MicrosoftSSOHandler.get_graph_api_base_url()
# Endpoint to get app role assignments for the given service principal
endpoint = f"/servicePrincipals/{service_principal_id}/appRoleAssignedTo"
url = base_url + endpoint

4
litellm/proxy/proxy_server.py
View file

@ -8544,6 +8544,8 @@ async def chat_completion(
except ModifyResponseException as e:
# Guardrail flagged content in passthrough mode - return 200 with violation message
_data = e.request_data
# Capture logging_obj before post_call_failure_hook pops it from _data.
_logging_obj = _data.get("litellm_logging_obj")
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
@ -8563,7 +8565,7 @@ async def chat_completion(
completion_stream=_iterator,
model=e.model,
custom_llm_provider="cached_response",
logging_obj=_data.get("litellm_logging_obj", None),
logging_obj=_logging_obj,
)
selected_data_generator = select_data_generator(
response=_streaming_response,

10
litellm/proxy/spend_tracking/spend_management_endpoints.py
View file

@ -1622,6 +1622,10 @@ async def ui_view_spend_logs(
default=None,
description="request_id to get spend logs for specific request_id",
),
session_id: str | None = fastapi.Query(
default=None,
description="Filter spend logs by session_id (partial string match)",
),
team_id: str | None = fastapi.Query(
default=None,
description="Filter spend logs by team_id",
@ -1906,6 +1910,12 @@ async def ui_view_spend_logs(
p += 2
sql_conditions.append(or_clause)
if session_id is not None and isinstance(session_id, str):
like_escaped_session_id = session_id.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
sql_conditions.append(f"session_id LIKE ${p}")
sql_params.append(f"%{like_escaped_session_id}%")
p += 1
# Status filter
if status_filter is not None:
if status_filter == "success":

12
litellm/realtime_api/main.py
View file

@ -515,6 +515,7 @@ async def _realtime_health_check(
api_base: Optional[str] = None,
api_version: Optional[str] = None,
realtime_protocol: Optional[str] = None,
model_params: Optional[dict] = None,
):
"""
Health check for realtime API - tries connection to the realtime API websocket
@ -550,14 +551,17 @@ async def _realtime_health_check(
elif custom_llm_provider == "xai":
url = xai_realtime._construct_url(api_base=api_base or "https://api.x.ai/v1", query_params={"model": model})
elif custom_llm_provider == "vertex_ai":
vertex_location = litellm.vertex_location or get_secret_str("VERTEXAI_LOCATION")
resolved_location = vertex_llm_base.get_vertex_region(vertex_region=vertex_location, model=model)
vertex_model_params = model_params or {}
resolved_location = vertex_llm_base.get_vertex_region(
vertex_region=VertexBase.safe_get_vertex_ai_location(vertex_model_params),
model=model,
)
(
access_token,
resolved_project,
) = await vertex_llm_base._ensure_access_token_async(
credentials=None,
project_id=litellm.vertex_project or get_secret_str("VERTEXAI_PROJECT"),
credentials=VertexBase.safe_get_vertex_ai_credentials(vertex_model_params),
project_id=VertexBase.safe_get_vertex_ai_project(vertex_model_params),
custom_llm_provider="vertex_ai",
)
vertex_realtime_config = VertexAIRealtimeConfig(

2
litellm/rerank_api/main.py
View file

@ -158,7 +158,7 @@ def rerank(
instruction=instruction,
non_default_params=kwargs,
)
verbose_logger.info(f"optional_rerank_params: {optional_rerank_params}")
verbose_logger.debug(f"optional_rerank_params: {optional_rerank_params}")
if isinstance(optional_params.timeout, str):
optional_params.timeout = float(optional_params.timeout)

15
litellm/secret_managers/base_secret_manager.py
View file

@ -1,3 +1,4 @@
import re
from abc import ABC, abstractmethod
from typing import Any, Dict, Optional, Union
@ -5,6 +6,20 @@ import httpx
from litellm import verbose_logger
_UNSAFE_SECRET_NAME_PATTERN = re.compile(r"(^|/)\.\.(/|$)|[\x00-\x1f\x7f-\x9f…

]")
def raise_if_unsafe_secret_name(secret_name: str) -> None:
"""
Validate a secret name before it is used by a secret manager integration.
Rejects ".." only as a path segment (bounded by "/" or the start/end of the
string, e.g. "../x", "x/..", or exactly ".."), not as a plain substring, so
names like "release-1.0..2" are not rejected.
"""
if _UNSAFE_SECRET_NAME_PATTERN.search(secret_name):
raise ValueError(f"Invalid secret_name {secret_name!r}")
class BaseSecretManager(ABC):
"""

8
litellm/secret_managers/cyberark_secret_manager.py
View file

@ -4,6 +4,7 @@ from typing import Any, Dict, Optional, Union
from urllib.parse import quote
import httpx
import yaml
import litellm
from litellm._logging import verbose_logger
@ -15,7 +16,7 @@ from litellm.llms.custom_httpx.http_handler import (
)
from litellm.proxy._types import KeyManagementSystem
from .base_secret_manager import BaseSecretManager
from .base_secret_manager import BaseSecretManager, raise_if_unsafe_secret_name
from .main import str_to_bool
@ -125,8 +126,11 @@ class CyberArkSecretManager(BaseSecretManager):
"""
# In production, we'd check if the variable exists first
# For now, we'll attempt to create it and ignore if it already exists
raise_if_unsafe_secret_name(secret_name)
policy_url = f"{self.conjur_addr}/policies/{self.conjur_account}/policy/root"
policy_yaml = f"- !variable {secret_name}\n"
# Use a real YAML serializer to build the scalar safely.
quoted_name = yaml.safe_dump(secret_name, default_style='"').strip()
policy_yaml = f"- !variable {quoted_name}\n"
try:
client = _get_httpx_client(params={"ssl_verify": self.ssl_verify})

3
litellm/secret_managers/hashicorp_secret_manager.py
View file

@ -14,7 +14,7 @@ from litellm.llms.custom_httpx.http_handler import (
)
from litellm.proxy._types import KeyManagementSystem
from .base_secret_manager import BaseSecretManager
from .base_secret_manager import BaseSecretManager, raise_if_unsafe_secret_name
class HashicorpSecretManager(BaseSecretManager):
@ -220,6 +220,7 @@ class HashicorpSecretManager(BaseSecretManager):
- With custom mount: http://127.0.0.1:8200/v1/kv/data/mykey
- With path prefix: http://127.0.0.1:8200/v1/secret/data/myapp/mykey
"""
raise_if_unsafe_secret_name(secret_name)
resolved_namespace = self._sanitize_path_component(namespace if namespace is not None else self.vault_namespace)
resolved_mount = self._sanitize_path_component(mount_name if mount_name is not None else self.vault_mount_name)
if resolved_mount is None:

4
litellm/types/mcp.py
View file

@ -38,6 +38,8 @@ class MCPAuth(str, enum.Enum):
aws_sigv4 = "aws_sigv4"
token = "token"
oauth2_token_exchange = "oauth2_token_exchange"
true_passthrough = "true_passthrough"
oauth_delegate = "oauth_delegate"
# RFC 8693 default subject_token_type. A NULL column / omitted config key means
@ -60,6 +62,8 @@ MCPAuthType = Optional[
MCPAuth.aws_sigv4,
MCPAuth.token,
MCPAuth.oauth2_token_exchange,
MCPAuth.true_passthrough,
MCPAuth.oauth_delegate,
]
]

15
litellm/types/mcp_server/mcp_server_manager.py
View file

@ -152,6 +152,18 @@ class MCPServer(BaseModel):
"""True if this is an OAuth2 server that relies on per-user tokens (no client_credentials)."""
return self.auth_type == MCPAuth.oauth2 and not self.has_client_credentials
@property
def is_true_passthrough(self) -> bool:
"""True for the transparent-proxy mode: LiteLLM performs no admission auth and forwards the
client's ``Authorization`` to the upstream unchanged."""
return self.auth_type == MCPAuth.true_passthrough
@property
def is_oauth_delegate(self) -> bool:
"""True for the delegated-upstream-OAuth mode: LiteLLM still admits the caller (API key / SSO /
JWT) but forwards the caller's separate upstream ``Authorization`` unchanged, minting nothing."""
return self.auth_type == MCPAuth.oauth_delegate
@property
def requires_per_user_auth(self) -> bool:
"""
@ -167,6 +179,9 @@ class MCPServer(BaseModel):
if self.needs_user_oauth_token:
return True
if self.is_true_passthrough or self.is_oauth_delegate:
return True
# PAT passthrough: auth_type is none but extra_headers includes auth headers
if self.auth_type == MCPAuth.none and self.extra_headers:
auth_header_names = {"authorization", "x-api-key", "api-key", "apikey"}

18
litellm/types/utils.py
View file

@ -2523,6 +2523,23 @@ class StandardLoggingMCPToolCall(TypedDict, total=False):
the client is driving a stateful session. Absent for stateless calls.
"""
mcp_auth_mode: Optional[str]
"""
The server's auth_type for this call (e.g. `true_passthrough`, `oauth_delegate`,
`oauth2`). For the client-forwarded token modes this records that the caller's own
upstream token was relayed, so an audit can attribute a relayed request to its mode
without logging any credential.
"""
mcp_server_resource: Optional[str]
"""
The origin (scheme + host + port) of the upstream MCP server the tool call was forwarded
to. Redacted for logging: userinfo, the path, the query string, and the fragment are all
stripped, because hosted MCP servers routinely embed the credential in the URL path and
this value is readable by callers via request logs.
Records which upstream received a relayed request; never a credential.
"""
class StandardLoggingVectorStoreRequest(TypedDict, total=False):
"""
@ -3383,6 +3400,7 @@ class LlmProviders(str, Enum):
LIBERTAI = "libertai"
PINSTRIPES = "pinstripes"
DARKBLOOM = "darkbloom"
META = "meta"
LITELLM_AGENT = "litellm_agent"
CURSOR = "cursor"
BEDROCK_MANTLE = "bedrock_mantle"

10
litellm/utils.py
View file

@ -8028,6 +8028,16 @@ class ProviderConfigManager:
)
return GithubCopilotAnthropicMessagesConfig()
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
json_provider = JSONProviderRegistry.get(provider.value)
if json_provider is not None and "/v1/messages" in json_provider.supported_endpoints:
from litellm.llms.openai_like.messages.transformation import (
JSONProviderAnthropicMessagesConfig,
)
return JSONProviderAnthropicMessagesConfig(json_provider)
return None
@staticmethod

806
model_prices_and_context_window.json
View file

@ -6012,6 +6012,522 @@
"supports_vision": true,
"supports_web_search": true
},
"azure/gpt-5.6": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_priority": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_priority": 2e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_priority": 1e-05,
"input_cost_per_token_above_272k_tokens_priority": 2e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token_above_272k_tokens_priority": 9e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-sol": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_priority": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_priority": 2e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_priority": 1e-05,
"input_cost_per_token_above_272k_tokens_priority": 2e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token_above_272k_tokens_priority": 9e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost_priority": 5e-07,
"cache_read_input_token_cost_above_272k_tokens_priority": 1e-06,
"input_cost_per_token": 2.5e-06,
"input_cost_per_token_above_272k_tokens": 5e-06,
"input_cost_per_token_priority": 5e-06,
"input_cost_per_token_above_272k_tokens_priority": 1e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_272k_tokens": 2.25e-05,
"output_cost_per_token_priority": 3e-05,
"output_cost_per_token_above_272k_tokens_priority": 4.5e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.6-luna": {
"cache_read_input_token_cost": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-07,
"cache_read_input_token_cost_priority": 2e-07,
"cache_read_input_token_cost_above_272k_tokens_priority": 4e-07,
"input_cost_per_token": 1e-06,
"input_cost_per_token_above_272k_tokens": 2e-06,
"input_cost_per_token_priority": 2e-06,
"input_cost_per_token_above_272k_tokens_priority": 4e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_272k_tokens": 9e-06,
"output_cost_per_token_priority": 1.2e-05,
"output_cost_per_token_above_272k_tokens_priority": 1.8e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-sol": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.75e-07,
"cache_read_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost_priority": 6.875e-07,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_above_272k_tokens": 5.5e-06,
"input_cost_per_token_priority": 6.875e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"output_cost_per_token_above_272k_tokens": 2.475e-05,
"output_cost_per_token_priority": 4.125e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/us/gpt-5.6-luna": {
"cache_read_input_token_cost": 1.1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2.2e-07,
"cache_read_input_token_cost_priority": 2.75e-07,
"input_cost_per_token": 1.1e-06,
"input_cost_per_token_above_272k_tokens": 2.2e-06,
"input_cost_per_token_priority": 2.75e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.6e-06,
"output_cost_per_token_above_272k_tokens": 9.9e-06,
"output_cost_per_token_priority": 1.65e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-sol": {
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"cache_read_input_token_cost_priority": 1.375e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"input_cost_per_token_priority": 1.375e-05,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"output_cost_per_token_priority": 8.25e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-terra": {
"cache_read_input_token_cost": 2.75e-07,
"cache_read_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost_priority": 6.875e-07,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_above_272k_tokens": 5.5e-06,
"input_cost_per_token_priority": 6.875e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"output_cost_per_token_above_272k_tokens": 2.475e-05,
"output_cost_per_token_priority": 4.125e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/eu/gpt-5.6-luna": {
"cache_read_input_token_cost": 1.1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2.2e-07,
"cache_read_input_token_cost_priority": 2.75e-07,
"input_cost_per_token": 1.1e-06,
"input_cost_per_token_above_272k_tokens": 2.2e-06,
"input_cost_per_token_priority": 2.75e-06,
"litellm_provider": "azure",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6.6e-06,
"output_cost_per_token_above_272k_tokens": 9.9e-06,
"output_cost_per_token_priority": 1.65e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_none_reasoning_effort": true,
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": false
},
"azure/gpt-5.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
@ -22431,6 +22947,218 @@
"supports_xhigh_reasoning_effort": true,
"supports_minimal_reasoning_effort": true
},
"gpt-5.6": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-sol": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-terra": {
"cache_creation_input_token_cost": 3.125e-06,
"cache_creation_input_token_cost_above_272k_tokens": 6.25e-06,
"cache_creation_input_token_cost_flex": 1.5625e-06,
"cache_creation_input_token_cost_priority": 6.25e-06,
"cache_read_input_token_cost": 2.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost_flex": 1.25e-07,
"cache_read_input_token_cost_priority": 5e-07,
"input_cost_per_token": 2.5e-06,
"input_cost_per_token_above_272k_tokens": 5e-06,
"input_cost_per_token_batches": 1.25e-06,
"input_cost_per_token_flex": 1.25e-06,
"input_cost_per_token_priority": 5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_272k_tokens": 2.25e-05,
"output_cost_per_token_batches": 7.5e-06,
"output_cost_per_token_flex": 7.5e-06,
"output_cost_per_token_priority": 3e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-luna": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 2.5e-06,
"cache_creation_input_token_cost_flex": 6.25e-07,
"cache_creation_input_token_cost_priority": 2.5e-06,
"cache_read_input_token_cost": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-07,
"cache_read_input_token_cost_flex": 5e-08,
"cache_read_input_token_cost_priority": 2e-07,
"input_cost_per_token": 1e-06,
"input_cost_per_token_above_272k_tokens": 2e-06,
"input_cost_per_token_batches": 5e-07,
"input_cost_per_token_flex": 5e-07,
"input_cost_per_token_priority": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_272k_tokens": 9e-06,
"output_cost_per_token_batches": 3e-06,
"output_cost_per_token_flex": 3e-06,
"output_cost_per_token_priority": 1.2e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
"supports_none_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"gpt-5.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
@ -25447,6 +26175,42 @@
"supports_function_calling": true,
"supports_tool_choice": false
},
"meta/muse-spark-1.1": {
"cache_read_input_token_cost": 1.5e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "meta",
"max_input_tokens": 1048576,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 4.25e-06,
"source": "https://dev.meta.ai/docs/getting-started/pricing-rate-limits",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/messages"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_minimal_reasoning_effort": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true,
"supports_xhigh_reasoning_effort": true
},
"meta_llama/Llama-3.3-70B-Instruct": {
"litellm_provider": "meta_llama",
"max_input_tokens": 128000,
@ -38303,6 +39067,48 @@
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-4.5": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"litellm_provider": "xai",
"max_input_tokens": 500000,
"max_output_tokens": 500000,
"max_tokens": 500000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_200k_tokens": 1.2e-05,
"source": "https://docs.x.ai/docs/models",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-4.5-latest": {
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"litellm_provider": "xai",
"max_input_tokens": 500000,
"max_output_tokens": 500000,
"max_tokens": 500000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"output_cost_per_token_above_200k_tokens": 1.2e-05,
"source": "https://docs.x.ai/docs/models",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"xai/grok-beta": {
"input_cost_per_token": 5e-06,
"litellm_provider": "xai",

17
provider_endpoints_support.json
View file

@ -1984,6 +1984,23 @@
"interactions": true
}
},
"meta": {
"display_name": "Meta Model API (`meta`)",
"url": "https://docs.litellm.ai/docs/providers/meta",
"endpoints": {
"chat_completions": true,
"messages": true,
"responses": true,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"a2a": false
}
},
"pinstripes": {
"display_name": "Pinstripes (`pinstripes`)",
"url": "https://docs.litellm.ai/docs/providers/pinstripes",

3
pyproject.toml
View file

@ -63,7 +63,7 @@ proxy = [
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.26.0,<2.0",
"litellm-proxy-extras==0.4.75",
"litellm-enterprise==0.1.48",
"litellm-enterprise==0.1.49",
"RestrictedPython>=8.1,<9.0",
"rich>=13.9.4,<14.0",
"polars>=1.38.1,<2.0",
@ -259,6 +259,7 @@ constraint-dependencies = [
"tornado>=6.5.6",
"aiohttp>=3.14.1,<4.0",
"packaging>=24.0",
"soupsieve>=2.8.4",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.

36
tests/benchmarks/conftest.py Normal file
View file

@ -0,0 +1,36 @@
"""Shared setup keeping CodSpeed measurements hermetic.
CodSpeed's callgrind instrumentation counts instructions from every thread while
a measurement window is open, and valgrind serializes all threads onto one
virtual CPU. Work deferred to litellm's shared logging executor would therefore
be attributed to whichever benchmark the valgrind scheduler resumes it under,
flipping results between runs. Running the executor inline keeps each
benchmark's cost self-contained and deterministic.
"""
from collections.abc import Callable, Iterator
from concurrent.futures import Future
from typing import ParamSpec, TypeVar
import pytest
from litellm.litellm_core_utils.thread_pool_executor import executor
P = ParamSpec("P")
R = TypeVar("R")
def _submit_inline(fn: Callable[P, R], /, *args: P.args, **kwargs: P.kwargs) -> Future[R]:
future: Future[R] = Future()
try:
future.set_result(fn(*args, **kwargs))
except BaseException as exc:
future.set_exception(exc)
return future
@pytest.fixture(autouse=True, scope="session")
def inline_logging_executor() -> Iterator[None]:
executor.submit = _submit_inline
yield
del executor.submit

21
tests/benchmarks/test_benchmarks.py
View file

@ -6,10 +6,13 @@ in the litellm hot path: token counting, model info lookup, provider
resolution, and cost calculation.
"""
import threading
import pytest
import litellm
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.thread_pool_executor import executor
from litellm.litellm_core_utils.token_counter import token_counter
@ -205,3 +208,21 @@ def test_get_model_cost_key_exact_match():
def test_get_model_cost_key_case_insensitive():
"""Benchmark model cost key lookup with case-insensitive fallback."""
litellm.utils._get_model_cost_key("GPT-4o")
# ---------------------------------------------------------------------------
# Measurement hermeticity guard
# ---------------------------------------------------------------------------
@pytest.mark.benchmark
def test_logging_executor_runs_inline():
"""Guard that the shared logging executor runs submissions inline.
Deferred submissions execute on worker threads, and callgrind attributes
their instructions to whichever benchmark's measurement window is open when
the valgrind scheduler resumes them, making results nondeterministic.
"""
future = executor.submit(threading.get_ident)
assert future.done()
assert future.result() == threading.get_ident()

1
tests/code_coverage_tests/recursive_detector.py
View file

@ -36,6 +36,7 @@ IGNORE_FUNCTIONS = [
"_collect_argument_paths", # max depth set.
"_split_text", # max depth set.
"_mask_sequence", # max depth set.
"_walk_payload", # max depth set (DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER).
"_delete_nested_value_custom", # max depth set (bounded by number of path segments).
"filter_exceptions_from_params", # max depth set (default 20) to prevent infinite recursion.
"__getattr__", # lazy loading pattern in litellm/__init__.py with proper caching to prevent infinite recursion.

4
tests/e2e/CLAUDE.md
View file

@ -63,7 +63,7 @@ The harness is fully typed and new code must not add `Any` or widen the basedpyr
The set of tests we want is a registry checked into this repo, one row per behavior; that file is the definition of done and the denominator. Each e2e test declares what it covers with `@pytest.mark.covers("...")`, and a small collector diffs the registry against the tests and ships coverage to the existing Grafana. No Allure, no new dependencies
Coverage is organized as module > feature > test. Dashboard modules are Core LLMs, Non-Core LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, and Other. A feature is either an endpoint (`/chat/completions`) or a behavior (fallbacks, rate limits; config-driven, with no route of its own). A cell reads like `llm.chat_completions.bedrock_converse.tool_use.stream.works`
Coverage is organized as module > feature > test. Dashboard modules are `Core LLMs`, `Non-Core LLMs`, `MCPs`, `Management/UI`, `Reliability & Performance`, `Logging & Guardrails`, and `Other`. The Loki stdout formatter maps those display modules to log-safe labels (`core_llms`, `non_core_llms`, `mcp`, `management_ui`, `reliability_performance`, `logging_guardrails`, and `other`) without changing JSON or Prometheus labels. A feature is either an endpoint (`/chat/completions`) or a behavior (fallbacks, rate limits; config-driven, with no route of its own). A cell reads like `llm.chat_completions.bedrock_converse.tool_use.stream.works`
The metric is coverage: the share of registry rows that have a passing covering test, reported to Grafana per module so a gap surfaces as an uncovered row rather than a silent absence
@ -71,7 +71,7 @@ Tests do not declare a dashboard module directly. They only declare the registry
### Naming grammar per module
LLMs - endpoint features (subject = the route), seeded from the Claude Code compat matrix. `chat_completions`, `messages`, and `responses` are Core LLMs. Other LLM endpoints, including `batches` and `realtime`, roll up as Non-Core LLMs.
LLMs - endpoint features (subject = the route), seeded from the Claude Code compat matrix. `chat_completions`, `messages`, and `responses` roll up to `Core LLMs`. Other LLM endpoints, including `batches` and `realtime`, roll up to `Non-Core LLMs`.
```
llm.<endpoint>.<route>.<capability>.<streaming>.<assertion>

6
tests/e2e/batches/capabilities.py
View file

@ -44,10 +44,10 @@ class Provider:
)
case "azure":
return LiteLLMParamsBody(
model="azure/gpt-4.1-mini-batch",
model="azure/gpt-5.4-mini-batch",
api_base="os.environ/AZURE_API_BASE",
api_key="os.environ/AZURE_API_KEY",
api_version="2024-07-01-preview",
api_version="2025-04-01-preview",
)
case "vertex_ai":
return LiteLLMParamsBody(
@ -97,7 +97,7 @@ class Capability:
PROVIDERS: tuple[Provider, ...] = (
Provider("openai", "openai-batch", "gpt-4o-mini", can_cancel=True, can_list=True),
Provider("azure", "azure-batch", "gpt-4.1-mini-batch", can_cancel=True, can_list=True),
Provider("azure", "azure-batch", "gpt-5.4-mini-batch", can_cancel=True, can_list=True),
Provider(
"vertex_ai", "vertex-batch", "gemini-2.5-flash", can_cancel=True, can_list=True
),

247
tests/e2e/bob_the_builder.py Normal file
View file

@ -0,0 +1,247 @@
"""Bob the builder: on a red e2e run, ask Devin to fix the failing tests.
Wired as a ``pytest_sessionfinish`` step (see ``conftest.py``). When the run went
red and remediation is enabled, it hands the failing tests plus their captured
tracebacks to Devin *through the LiteLLM proxy's own MCP gateway* -- the same
gateway + master key the suite already uses -- so Devin files a Linear ticket per
failure and opens fix PRs. Nothing new ships in the runner pod: the proxy already
registers the ``devin`` MCP server and holds ``DEVIN_API_KEY``, injecting it
upstream, so this process only needs the proxy key it always has.
Opt-in via ``E2E_DEVIN_REMEDIATION=1`` so a normal local ``pytest tests/e2e`` run
never spawns a Devin session. ``DEVIN_DRY_RUN=1`` prints the prompt it would send
and makes no call. Everything is best-effort: any error here is logged and
swallowed so the run's exit status still reflects the tests, not remediation.
"""
from __future__ import annotations
import hashlib
import os
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol, cast
import pytest
from pydantic import BaseModel, ConfigDict
from e2e_config import MASTER_KEY, PROXY_BASE_URL
from e2e_http import Success
from transport import HttpTransport
REMEDIATION_ENV = "E2E_DEVIN_REMEDIATION"
_LIST_PATH = "/mcp-rest/tools/list"
_CALL_PATH = "/mcp-rest/tools/call"
@dataclass(frozen=True, slots=True)
class Failure:
"""One failed test: its pytest node id and the captured failure text."""
nodeid: str
detail: str
@dataclass(frozen=True, slots=True)
class Config:
server: str
create_tool: str
linear_team: str
target_repo: str
target_ref: str
max_failures: int
max_detail_chars: int
tags: tuple[str, ...]
dry_run: bool
class _NoParams(BaseModel):
pass
class _McpToolInfo(BaseModel):
model_config = ConfigDict(extra="allow")
server_name: str | None = None
alias: str | None = None
class _McpTool(BaseModel):
model_config = ConfigDict(extra="allow")
name: str
mcp_info: _McpToolInfo | None = None
class _McpToolsList(BaseModel):
model_config = ConfigDict(extra="allow")
tools: tuple[_McpTool, ...] = ()
class _DevinSessionArgs(BaseModel):
prompt: str
title: str
tags: list[str]
class _ToolCallBody(BaseModel):
name: str
arguments: _DevinSessionArgs
class _ToolCallResult(BaseModel):
model_config = ConfigDict(extra="allow")
class _Report(Protocol):
@property
def nodeid(self) -> str: ...
@property
def longreprtext(self) -> str: ...
class _TerminalReporter(Protocol):
stats: Mapping[str, Sequence[_Report]]
def _env(name: str, default: str) -> str:
value = os.environ.get(name, "").strip()
return value or default
def load_config() -> Config:
raw_tags = _env("DEVIN_TAGS", "e2e,stage")
return Config(
server=_env("DEVIN_MCP_SERVER", "devin"),
create_tool=_env("DEVIN_SESSION_TOOL", "devin_session_create"),
linear_team=_env("DEVIN_LINEAR_TEAM", "LIT"),
target_repo=_env("DEVIN_TARGET_REPO", "BerriAI/litellm"),
target_ref=_env("DEVIN_TARGET_REF", "litellm_internal_staging"),
max_failures=int(_env("DEVIN_MAX_FAILURES", "50")),
max_detail_chars=int(_env("DEVIN_MAX_DETAIL_CHARS", "3000")),
tags=tuple(t.strip() for t in raw_tags.split(",") if t.strip()),
dry_run=_env("DEVIN_DRY_RUN", "0") == "1",
)
def collect_failures(session: pytest.Session, max_detail_chars: int) -> tuple[Failure, ...]:
"""Pull the failed and errored tests (with their tracebacks) off the run's
terminal reporter. Returns empty when nothing failed or the reporter is
absent (e.g. a skipped, proxy-less session)."""
plugin: object = session.config.pluginmanager.getplugin("terminalreporter")
if plugin is None:
return ()
reporter = cast(_TerminalReporter, plugin)
reports = (*reporter.stats.get("failed", ()), *reporter.stats.get("error", ()))
return tuple(
Failure(nodeid=r.nodeid, detail=r.longreprtext.strip()[-max_detail_chars:]) for r in reports
)
def dedup_tag(failures: tuple[Failure, ...]) -> str:
"""Stable short tag identifying this exact set of failing tests, so repeated
nightly runs on the same failures reference one body of work."""
joined = "\n".join(sorted(f.nodeid for f in failures))
return "e2e-fail-" + hashlib.sha256(joined.encode()).hexdigest()[:12]
def _revision() -> str:
for candidate in (Path(__file__).parent / ".litellm-revision", Path("/app/e2e/.litellm-revision")):
try:
return candidate.read_text(encoding="utf-8").strip()
except OSError:
continue
return _env("E2E_REVISION", "unknown")
def build_prompt(cfg: Config, failures: tuple[Failure, ...], tag: str) -> str:
shown = failures[: cfg.max_failures]
header = (
f"The LiteLLM end-to-end suite failed on the "
f"{_env('E2E_ENVIRONMENT', 'stage')} proxy. Source repo {cfg.target_repo} "
f"at revision {_revision()} (branch {cfg.target_ref}). {len(failures)} "
f"test(s) failed"
+ (f"; the first {len(shown)} are shown" if len(shown) < len(failures) else "")
+ ".\n\n"
)
task = (
"For each failing test below:\n"
f"1. Open a Linear ticket under the {cfg.linear_team} team describing the "
"failure (test id, the assertion/error, likely cause), unless an open "
"ticket for that same test already exists -- do not create duplicates.\n"
f"2. Fix it in {cfg.target_repo}, branching off {cfg.target_ref} and "
"following the repo's CONTRIBUTING and CLAUDE.md conventions (meaningful "
"regression coverage, conventional commits, run the suite locally), then "
"open a PR that references the Linear ticket.\n"
"3. Prefer one focused PR per failing test; if several share a root cause, "
"group them and say so.\n"
f"Before starting, search existing sessions/PRs tagged '{tag}' or "
"referencing these test ids and continue that work instead of restarting.\n\n"
"Failing tests and their captured output:\n"
)
blocks = [f"### {i}. {f.nodeid}\n```\n{f.detail}\n```\n" for i, f in enumerate(shown, start=1)]
return header + task + "\n".join(blocks)
def _resolve_tool_name(transport: HttpTransport, cfg: Config) -> str | None:
"""Find Devin's create-session tool on the gateway. The proxy prefixes tools
with the server alias, so match by suffix and (when present) the owning
server."""
result = transport.get(
_LIST_PATH, headers=transport.master, params=_NoParams(), response_type=_McpToolsList
)
if not isinstance(result, Success):
print(f"bob_the_builder: could not list gateway MCP tools: {result}")
return None
for tool in result.data.tools:
owner = tool.mcp_info.server_name or tool.mcp_info.alias if tool.mcp_info else None
if (owner is None or owner == cfg.server) and (
tool.name == cfg.create_tool or tool.name.endswith(cfg.create_tool)
):
return tool.name
print(
f"bob_the_builder: no '{cfg.create_tool}' tool for server '{cfg.server}' on the gateway; "
f"saw {[t.name for t in result.data.tools]}"
)
return None
def remediate(session: pytest.Session) -> None:
"""Entry point called from ``pytest_sessionfinish``. No-op unless remediation
is enabled and the run actually had failures."""
if os.environ.get(REMEDIATION_ENV) != "1":
return
cfg = load_config()
failures = collect_failures(session, cfg.max_detail_chars)
if not failures:
return
tag = dedup_tag(failures)
title = f"Fix {len(failures)} failing LiteLLM e2e test(s) [{tag}]"
prompt = build_prompt(cfg, failures, tag)
args = _DevinSessionArgs(prompt=prompt, title=title, tags=[*cfg.tags, tag])
if cfg.dry_run:
print("bob_the_builder: DRY RUN -- would create a Devin session:")
print(f" server : {cfg.server}\n tool : {cfg.create_tool}\n title : {title}")
print(f" tags : {args.tags}\n---- prompt ----\n{prompt}")
return
try:
transport = HttpTransport(base_url=PROXY_BASE_URL, master_key=MASTER_KEY)
tool_name = _resolve_tool_name(transport, cfg)
if tool_name is None:
return
result = transport.post(
_CALL_PATH,
headers=transport.master,
json=_ToolCallBody(name=tool_name, arguments=args),
response_type=_ToolCallResult,
)
if isinstance(result, Success):
print(f"bob_the_builder: created Devin session for {len(failures)} failure(s) [{tag}]")
print(result.data.model_dump_json())
else:
print(f"bob_the_builder: Devin session call failed: {result}")
except Exception as exc: # noqa: BLE001 - remediation must never fail the run
print(f"bob_the_builder: remediation error (ignored): {exc}")

7
tests/e2e/conftest.py
View file

@ -107,6 +107,13 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
if spend_dir in sys.path:
sys.path.remove(spend_dir)
try:
from bob_the_builder import remediate
remediate(session)
except Exception as exc: # noqa: BLE001 - remediation is best-effort
print(f"devin remediation skipped: {exc}")
@pytest.fixture
def resources(client: GatewayProvider) -> Iterator[ResourceManager]:

25
tests/e2e/coverage_registry/README.md
View file

@ -9,18 +9,18 @@ note; the naming grammar lives in `tests/e2e/CLAUDE.md`.
A **cell** is one customer-noticeable behavior a single e2e test can assert pass/fail
on, for example `llm.chat_completions.bedrock_converse.tool_use.stream.works`. Cells are
grouped `module > feature > test`, with LLM cells split into Core LLMs and Non-Core
LLMs for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
grouped `module > feature > test`, with LLM cells split into `Core LLMs` and
`Non-Core LLMs` for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
`fail_before_fix` flag.
The rows live in per-prefix YAML files (`llm_*.yaml`, `mgmt.yaml`, `mcp.yaml`,
`reliability.yaml`, `logging.yaml`, `guardrail.yaml`, `other.yaml`) and validate against
the discriminated union in `schema.py`, so an LLM row cannot carry a guardrail field and
vice versa. `llm` rows with `subject_endpoint` of `chat_completions`, `messages`, or
`responses` roll up to "Core LLMs"; all other LLM endpoints roll up to "Non-Core
LLMs". LLM endpoint, route, and capability values are typed in `schema.py`, so new
taxonomy values require an explicit schema change. `logging` and `guardrail` are two
id-prefixes that roll up into the single "Logging & Guardrails" dashboard module.
`responses` roll up to `Core LLMs`; all other LLM endpoints roll up to `Non-Core LLMs`.
LLM endpoint, route, and capability values are typed in `schema.py`, so new taxonomy
values require an explicit schema change. `logging` and `guardrail` are two id-prefixes
that roll up into the single `Logging & Guardrails` dashboard module.
A test declares what it covers with a marker:
@ -40,8 +40,17 @@ proxy. Whether a covered cell currently passes or fails is a separate, live conc
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
```
Use `--format prometheus` or `--format json` for CI jobs that publish coverage to
Grafana.
Use `--format loki` after the e2e pytest run in the same Kubernetes job/pod to print
structured stdout lines for Loki:
```
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict
```
This emits exactly one `COVERAGE_TOTAL` line and one `COVERAGE_MODULE` line per module
in `MODULE_ORDER`, in that order. Loki uses log-safe `module=` labels from
`LOKI_MODULE_LABELS` (`core_llms`, `management_ui`, etc.) so existing JSON and
Prometheus consumers keep their human-readable module names unchanged.
The headline is overall coverage. The collector also lists markers that point at ids
not in the registry, so a typo or an unenumerated behavior surfaces instead of being

29
tests/e2e/coverage_registry/collector.py
View file

@ -21,7 +21,7 @@ from pathlib import Path
import pytest
from .registry import load_registry
from .schema import MODULE_ORDER, Cell, Tier, dashboard_module
from .schema import MODULE_ORDER, Cell, Tier, dashboard_module, loki_module_label
E2E_DIR = Path(__file__).resolve().parent.parent
@ -239,13 +239,31 @@ def render_prometheus(report: CoverageReport) -> str:
return "\n".join(lines)
def render_loki(report: CoverageReport) -> str:
lines = [
(
f"COVERAGE_TOTAL percent={report.coverage_percent:.1f} "
f"covered={report.covered} total={report.total}"
)
]
lines.extend(
(
f"COVERAGE_MODULE module={loki_module_label(module.module)} "
f"percent={module.coverage_percent:.1f} "
f"covered={module.covered} total={module.total}"
)
for module in report.modules
)
return "\n".join(lines)
def main() -> int:
parser = ArgumentParser()
parser.add_argument(
"--format",
choices=("text", "json", "prometheus"),
choices=("text", "json", "prometheus", "loki"),
default="text",
help="Output format. Use prometheus or json for Grafana ingestion jobs.",
help="Output format. Use loki for structured stdout lines in the e2e job.",
)
parser.add_argument(
"--strict",
@ -265,9 +283,8 @@ def main() -> int:
"text": render,
"json": render_json,
"prometheus": render_prometheus,
}[
args.format
](report)
"loki": render_loki,
}[args.format](report)
print(output) # noqa: T201 # CLI entrypoint output
if args.strict and report.orphan_markers:
return 1

15
tests/e2e/coverage_registry/schema.py
View file

@ -157,6 +157,16 @@ MODULE_ORDER: tuple[str, ...] = (
"Other",
)
LOKI_MODULE_LABELS: dict[str, str] = {
"Core LLMs": "core_llms",
"Non-Core LLMs": "non_core_llms",
"MCPs": "mcp",
"Management/UI": "management_ui",
"Reliability & Performance": "reliability_performance",
"Logging & Guardrails": "logging_guardrails",
"Other": "other",
}
def dashboard_module(cell: Cell) -> str:
"""Return the Grafana/reporting module for a registry cell."""
@ -165,3 +175,8 @@ def dashboard_module(cell: Cell) -> str:
return "Core LLMs"
return "Non-Core LLMs"
return PREFIX_ROLLUP[cell.module]
def loki_module_label(module: str) -> str:
"""Return the log-safe Loki label for a dashboard module."""
return LOKI_MODULE_LABELS[module]

26
tests/e2e/coverage_registry/test_collector.py
View file

@ -15,6 +15,7 @@ from coverage_registry.collector import (
compute_coverage,
render,
render_json,
render_loki,
render_prometheus,
)
from coverage_registry.registry import load_registry
@ -24,6 +25,7 @@ from coverage_registry.schema import (
LlmEndpoint,
LoggingCell,
Tier,
loki_module_label,
)
@ -149,6 +151,30 @@ def test_prometheus_render_exposes_module_coverage_timeseries() -> None:
assert "litellm_e2e_coverage_orphan_markers 0" in metrics
def test_loki_render_exposes_exact_stdout_lines_for_loki() -> None:
report = compute_coverage(
(_llm("llm.chat", Tier.P0), _llm("llm.batches", Tier.P0, "batches")),
frozenset({"llm.chat"}),
)
lines = render_loki(report).splitlines()
assert len(lines) == 1 + len(report.modules)
assert lines[0] == "COVERAGE_TOTAL percent=50.0 covered=1 total=2"
assert (
lines[1] == "COVERAGE_MODULE module=core_llms percent=100.0 covered=1 total=1"
)
assert (
lines[2] == "COVERAGE_MODULE module=non_core_llms percent=0.0 covered=0 total=1"
)
assert [line.split("module=", 1)[1].split(" ", 1)[0] for line in lines[1:]] == [
loki_module_label(module.module) for module in report.modules
]
assert all(
" " not in line.split("module=", 1)[1].split(" ", 1)[0] for line in lines[1:]
)
def test_real_registry_loads_and_ids_are_unique() -> None:
cells = load_registry()
ids = [c.id for c in cells]

4
tests/e2e/docker-compose.yml
View file

@ -25,6 +25,10 @@ configs:
fallbacks:
- gemini-2.5-flash: ["gpt-5.5", "claude-haiku-4-5"]
finetune_settings:
- custom_llm_provider: openai
api_key: os.environ/OPENAI_API_KEY
model_list:
- model_name: gpt-5.5
litellm_params:

70
tests/e2e/e2e_gateway.py
View file

@ -28,6 +28,9 @@ from models import (
CustomerDeleteBody,
EmbedBody,
EmbedResponse,
FileListResponse,
FineTuningJobsParams,
FineTuningJobsResponse,
KeyDeleteBody,
KeyGenerateBody,
KeyGenerateResponse,
@ -42,6 +45,7 @@ from models import (
ModelMode,
ModelNewBody,
ModelNewResponse,
ModelsListResponse,
OcrBody,
OcrResponse,
SpendLogRow,
@ -119,27 +123,83 @@ class Gateway:
)
).data
def list_files(self, key: str) -> Result[FileListResponse]:
return self.transport.get(
"/v1/files",
headers=self.transport.bearer(key),
params=NoBody(),
response_type=FileListResponse,
)
def list_fine_tuning_jobs(
self, key: str, params: FineTuningJobsParams
) -> Result[FineTuningJobsResponse]:
return self.transport.get(
"/v1/fine_tuning/jobs",
headers=self.transport.bearer(key),
params=params,
response_type=FineTuningJobsResponse,
)
def create_model(
self,
model_name: str,
litellm_params: LiteLLMParamsBody,
mode: ModelMode | None = None,
) -> str:
"""Register a deployment under `model_name` (id == model_name) and return the
model_id. add_deployment runs synchronously in /model/new, so the model is
callable as soon as this returns."""
return unwrap(
"""Register a deployment under `model_name` and return its proxy-assigned
model_id, once the model is actually servable on the data plane.
/model/new is a control-plane route; in a split control/data-plane
deployment the gateway (data plane, which serves /chat, /ocr, ...) only
picks the new model up on its next DB reload, so a call issued the instant
this returns can race the reload and 400 with "Invalid model name passed".
We therefore poll the data-plane /v1/models until the model appears before
handing back, so callers can invoke it immediately. In the monolithic case
it is already present on the first poll, so this adds one request."""
model_id = unwrap(
self.transport.post(
"/model/new",
headers=self.transport.master,
json=ModelNewBody(
model_name=model_name,
litellm_params=litellm_params,
model_info=ModelInfoBody(id=model_name, mode=mode),
model_info=ModelInfoBody(mode=mode),
),
response_type=ModelNewResponse,
)
).model_id
self._await_model_servable(model_name)
return model_id
def _await_model_servable(self, model_name: str) -> None:
"""Block until the data plane lists `model_name`, or fail loudly if it does
not within poll_timeout (a real propagation/config problem, surfaced here
instead of as a downstream "Invalid model name passed")."""
deadline = time.monotonic() + self.poll_timeout
last_result: Result[ModelsListResponse] | None = None
while time.monotonic() < deadline:
last_result = self.transport.get(
"/v1/models",
headers=self.transport.master,
params=NoBody(),
response_type=ModelsListResponse,
)
if isinstance(last_result, Success) and any(
entry.id == model_name for entry in last_result.data.data
):
return
time.sleep(self.poll_interval)
last_error = (
f"; last /v1/models poll did not succeed: {last_result}"
if last_result is not None and not isinstance(last_result, Success)
else ""
)
raise AssertionError(
f"model {model_name!r} was created but never became servable on the data "
f"plane within {self.poll_timeout}s of /model/new (control/data-plane "
f"propagation or STORE_MODEL_IN_DB reload issue){last_error}"
)
def delete_model(self, model_id: str) -> None:
result = self.transport.post(

66
tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md Normal file
View file

@ -0,0 +1,66 @@
# Realtime e2e coverage
Live tests for the proxy realtime websocket endpoint (`/v1/realtime`). One
GA-speaking websocket client drives every provider; the proxy normalizes each
provider's stream into the OpenAI GA event schema, so the same assertions hold
across providers and only the model alias changes.
## What is asserted
For each configured provider, `test_text_conversation` checks the session
lifecycle (`session.created`, then `session.update` echoed by `session.updated`),
the canonical response sequence (`response.created`, `response.output_item.added`,
through `response.done`), that the streamed deltas reconstruct a non-empty
transcript, and that `response.done` carries normalized usage.
`test_tool_call_round_trip` checks the full tool path: the model emits a
normalized `response.function_call_arguments.done` with valid JSON arguments and
a matching `function_call` output item, the test sends a `function_call_output`
back, and the follow-up response incorporates the result (the temperature 72
appears).
`test_realtime_pipecat_e2e` is a realism layer that drives the same providers
through pipecat's GA `OpenAIRealtimeLLMService` (base_url pointed at the proxy)
rather than speaking the protocol by hand. Its assertions are coarse (the tool
callback fired, assistant text was produced); the raw-websocket suite is the
source of truth. It skips unless `pipecat-ai` is installed
(`uv pip install "pipecat-ai[openai]"`).
## Provisioning
The suite registers every provider's realtime deployment through `/model/new` at
session start (the `realtime_models` fixture) and deletes them on teardown, so it
never depends on a static or misconfigured gateway `model_list`. Each deployment
is created with `model_info.mode: realtime` and marker-unique names, and its
`litellm_params` point the credentials at `os.environ/*` refs the gateway resolves
at call time. The provider table below is the source of truth; edit `PROVIDERS` in
`realtime_client.py` to change a model or add one.
| provider | model alias | upstream model |
|----------|-------------|----------------|
| openai | `openai-realtime` | `openai/gpt-realtime-2` |
| azure | `azure-realtime` | `azure/gpt-realtime-2` (GA protocol) |
| gemini | `gemini-realtime` | `gemini/gemini-3.1-flash-live-preview` |
| vertex_ai | `vertex-realtime` | `vertex_ai/gemini-live-2.5-flash-preview-native-audio-09-2025` |
Bedrock and xai (`xai/grok-4-1-fast`) are supported by the proxy but
kept commented out in `PROVIDERS` until they pass end-to-end here; re-enable them by
uncommenting their entry.
Every provider is provisioned and asserted; the suite never skips a provider. Per
`tests/e2e/CLAUDE.md` the only sanctioned skip is the whole-suite proxy-liveness
skip, so a provider whose credentials or upstream realtime model are missing on the
gateway is a hard failure, not a skip. Give the gateway each provider's credentials
to turn its tests green.
## Running
Start a proxy with the provider keys set in its environment (the suite registers
the deployments itself), then
```
uv run pytest tests/e2e/llm_translation/realtime/ -v
```
The whole suite skips only when no proxy answers `GET /health/liveliness` at
`LITELLM_PROXY_URL` (default `http://localhost:4000`).

37
tests/e2e/llm_translation/realtime/conftest.py Normal file
View file

@ -0,0 +1,37 @@
"""Realtime suite's `client` and `realtime_models` fixtures.
The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker
live in the parent tests/e2e/conftest.py. RealtimeClient holds the shared Gateway,
so the `resources` fixture cleans up keys this suite creates.
`realtime_models` registers every provider's realtime deployment through /model/new
at session start and deletes them at teardown, so the suite provisions the models it
uses through the management endpoints instead of depending on a static (or
misconfigured) gateway model_list.
"""
from collections.abc import Iterator
import pytest
from realtime_client import PROVIDERS, RealtimeClient, build_client
@pytest.fixture(scope="session")
def client() -> RealtimeClient:
return build_client()
@pytest.fixture(scope="session")
def realtime_models(client: RealtimeClient) -> Iterator[dict[str, str]]:
"""Provision each provider's realtime deployment via /model/new and yield a
provider-id -> model-name map the tests connect with; delete them on teardown.
Every provider is provisioned (never skipped): a provider whose credentials or
upstream model are missing on the gateway hard-fails its test, per the suite's
fail-on-behavior contract in tests/e2e/CLAUDE.md."""
records = tuple((provider.id, *client.provision(provider)) for provider in PROVIDERS)
try:
yield {provider_id: model_name for provider_id, model_name, _ in records}
finally:
for _, _, model_id in records:
client.gateway.delete_model(model_id)

0
tests/e2e/realtime/fixtures/weather_question_24k.wav → tests/e2e/llm_translation/realtime/fixtures/weather_question_24k.wav
View file

0
tests/e2e/realtime/pipecat_service.py → tests/e2e/llm_translation/realtime/pipecat_service.py
View file

96
tests/e2e/realtime/realtime_client.py → tests/e2e/llm_translation/realtime/realtime_client.py
View file

@ -11,19 +11,19 @@ models, matching the suite's no-raw-dicts rule.
from __future__ import annotations
import time
from collections.abc import Generator
from collections.abc import Generator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any, TypeVar
from urllib.parse import urlencode
import pytest
from pydantic import BaseModel, ConfigDict
from websockets.sync.client import connect
from websockets.sync.connection import Connection
from e2e_config import PROXY_BASE_URL
from e2e_config import PROXY_BASE_URL, unique_marker
from e2e_gateway import Gateway, build_gateway
from models import LiteLLMParamsBody
_M = TypeVar("_M", bound=BaseModel)
@ -41,25 +41,76 @@ def realtime_ws_url(model: str) -> str:
@dataclass(frozen=True, slots=True)
class RealtimeProvider:
"""A realtime provider the suite exercises. `litellm_params` is the deployment
the suite registers through /model/new (the gateway resolves the os.environ/*
credential refs), so the suite is self-contained and never depends on a static
gateway model_list. Every provider here is provisioned and asserted: per
tests/e2e/CLAUDE.md the suite never skips a provider, so a provider whose
credentials or upstream realtime model are missing on the gateway is a hard
failure, not a skip."""
id: str
model: str
alias: str
litellm_params: LiteLLMParamsBody
PROVIDERS = (
RealtimeProvider("openai", "openai-realtime"),
RealtimeProvider("azure", "azure-realtime"),
RealtimeProvider("gemini", "gemini-realtime"),
RealtimeProvider("vertex_ai", "vertex-realtime"),
# RealtimeProvider("bedrock", "bedrock-realtime"), # TODO: Enable this when Bedrock is passing
RealtimeProvider("xai", "xai-realtime"),
RealtimeProvider(
"openai",
"openai-realtime",
LiteLLMParamsBody(
model="openai/gpt-realtime-2",
api_key="os.environ/OPENAI_API_KEY",
),
),
RealtimeProvider(
"azure",
"azure-realtime",
LiteLLMParamsBody(
model="azure/gpt-realtime",
api_key="os.environ/AZURE_API_KEY",
api_version="2025-08-28",
realtime_protocol="GA",
),
),
RealtimeProvider(
"gemini",
"gemini-realtime",
LiteLLMParamsBody(
model="gemini/gemini-3.1-flash-live-preview",
api_key="os.environ/GEMINI_API_KEY",
),
),
RealtimeProvider(
"vertex_ai",
"vertex-realtime",
LiteLLMParamsBody(
model="vertex_ai/gemini-live-2.5-flash-preview-native-audio-09-2025",
vertex_location="us-central1",
vertex_credentials="os.environ/VERTEXAI_CREDENTIALS",
),
),
# RealtimeProvider("bedrock", "bedrock-realtime", ...) # TODO: Enable when Bedrock is passing
# RealtimeProvider(
# "xai",
# "xai-realtime",
# LiteLLMParamsBody(
# model="xai/grok-4-1-fast",
# api_key="os.environ/XAI_API_KEY",
# ),
# ), # TODO: Enable once xai Grok Voice realtime is passing end-to-end here
)
def skip_if_unconfigured(
provider: RealtimeProvider, configured: frozenset[str]
) -> None:
if provider.model not in configured:
pytest.skip(f"{provider.model} not configured on proxy")
def realtime_model(provider: RealtimeProvider, provisioned: Mapping[str, str]) -> str:
"""Return the provisioned deployment name for this provider. Every provider in
PROVIDERS is provisioned at session start, so a missing entry is a harness bug,
never an environment skip - the suite hard-fails instead (see tests/e2e/CLAUDE.md)."""
model = provisioned.get(provider.id)
assert model is not None, (
f"{provider.id} was not provisioned; the realtime_models fixture is broken"
)
return model
# ---- sent events -------------------------------------------------------
@ -280,12 +331,17 @@ class RealtimeSession:
class RealtimeClient:
gateway: Gateway
def configured_models(self) -> frozenset[str]:
return frozenset(
entry.model_name
for entry in self.gateway.model_info()
if entry.model_info.mode == "realtime"
def provision(self, provider: RealtimeProvider) -> tuple[str, str]:
"""Register this provider's realtime deployment through /model/new and return
(model_name, model_id). The name is marker-unique so it never collides with a
same-named deployment already on the shared proxy, and mode=realtime makes it
show up as a realtime model on /model/info. add_deployment runs synchronously,
so the deployment is connectable as soon as this returns."""
model_name = f"{provider.alias}-{unique_marker()}"
model_id = self.gateway.create_model(
model_name, provider.litellm_params, mode="realtime"
)
return model_name, model_id
@contextmanager
def connect(

14
tests/e2e/realtime/test_realtime_e2e.py → tests/e2e/llm_translation/realtime/test_realtime_e2e.py
View file

@ -31,7 +31,7 @@ from realtime_client import (
SessionUpdate,
function_call_item,
parse_last,
skip_if_unconfigured,
realtime_model,
transcript,
user_message,
)
@ -62,12 +62,12 @@ class WeatherResult(BaseModel):
def test_text_conversation(
client: RealtimeClient,
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
skip_if_unconfigured(provider, configured_models)
model = realtime_model(provider, realtime_models)
with client.connect(key=scoped_key, model=provider.model) as session:
with client.connect(key=scoped_key, model=model) as session:
created = session.collect_until("session.created", timeout=20)
assert created[-1].type == "session.created"
@ -99,12 +99,12 @@ def test_text_conversation(
def test_tool_call_round_trip(
client: RealtimeClient,
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
skip_if_unconfigured(provider, configured_models)
model = realtime_model(provider, realtime_models)
with client.connect(key=scoped_key, model=provider.model) as session:
with client.connect(key=scoped_key, model=model) as session:
session.collect_until("session.created", timeout=20)
session.send(
SessionUpdate(

25
tests/e2e/realtime/test_realtime_pipecat_audio_e2e.py → tests/e2e/llm_translation/realtime/test_realtime_pipecat_audio_e2e.py
View file

@ -31,7 +31,7 @@ from realtime_client import (
PROVIDERS,
RealtimeProvider,
_ws_base_url,
skip_if_unconfigured,
realtime_model,
)
pytestmark = pytest.mark.e2e
@ -96,7 +96,7 @@ SERVER_VAD_SETTINGS = rt_events.SessionProperties(
noise_reduction=rt_events.InputAudioNoiseReduction(type="near_field"),
turn_detection=rt_events.TurnDetection(
type="server_vad",
threshold=0.8,
threshold=0.5,
prefix_padding_ms=300,
silence_duration_ms=700,
),
@ -189,13 +189,13 @@ async def _run_pipeline(
@pytest.mark.parametrize("provider", PROVIDER_PARAMS)
def test_pipecat_server_vad(
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
"""Session is configured with server-VAD; bot must respond to a text prompt."""
skip_if_unconfigured(provider, configured_models)
model = realtime_model(provider, realtime_models)
tool_called, got_text, _ = asyncio.run(_run_pipeline(scoped_key, provider.model))
tool_called, got_text, _ = asyncio.run(_run_pipeline(scoped_key, model))
assert tool_called, "get_weather tool was not invoked"
assert got_text, "no assistant text frames produced"
@ -204,16 +204,16 @@ def test_pipecat_server_vad(
@pytest.mark.parametrize("provider", PROVIDER_PARAMS)
def test_pipecat_audio_output(
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
"""Bot must produce at least one non-empty TTS audio frame."""
skip_if_unconfigured(provider, configured_models)
model = realtime_model(provider, realtime_models)
_, got_text, audio_bytes = asyncio.run(
_run_pipeline(
scoped_key,
provider.model,
model,
prompt="Say hello in one short sentence.",
timeout=30.0,
)
@ -328,7 +328,7 @@ async def _run_audio_input_pipeline(
@pytest.mark.parametrize("provider", PROVIDER_PARAMS)
def test_pipecat_server_vad_audio_input(
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
"""Stream a real PCM16 WAV fixture; server VAD must detect speech end and respond.
@ -337,12 +337,11 @@ def test_pipecat_server_vad_audio_input(
→ server-VAD turn detection → response.create (auto) → assistant reply.
No LLMRunFrame is sent — the response must be triggered entirely by VAD.
"""
if not WEATHER_WAV.exists():
pytest.skip(f"audio fixture not found: {WEATHER_WAV}")
skip_if_unconfigured(provider, configured_models)
assert WEATHER_WAV.exists(), f"audio fixture not found: {WEATHER_WAV}"
model = realtime_model(provider, realtime_models)
got_text, audio_bytes = asyncio.run(
_run_audio_input_pipeline(scoped_key, provider.model)
_run_audio_input_pipeline(scoped_key, model)
)
assert got_text, "server VAD did not trigger a response (no assistant text)"

8
tests/e2e/realtime/test_realtime_pipecat_e2e.py → tests/e2e/llm_translation/realtime/test_realtime_pipecat_e2e.py
View file

@ -29,7 +29,7 @@ from realtime_client import (
PROVIDERS,
RealtimeProvider,
_ws_base_url,
skip_if_unconfigured,
realtime_model,
)
pytestmark = pytest.mark.e2e
@ -123,12 +123,12 @@ async def _run_pipeline(key: str, model: str) -> tuple[bool, bool]:
@pytest.mark.parametrize("provider", PROVIDER_PARAMS)
def test_pipecat_tool_smoke(
scoped_key: str,
configured_models: frozenset[str],
realtime_models: dict[str, str],
provider: RealtimeProvider,
) -> None:
skip_if_unconfigured(provider, configured_models)
model = realtime_model(provider, realtime_models)
tool_called, produced_text = asyncio.run(_run_pipeline(scoped_key, provider.model))
tool_called, produced_text = asyncio.run(_run_pipeline(scoped_key, model))
assert tool_called, "pipecat did not invoke the get_weather callback"
assert produced_text, "pipecat produced no assistant text frames"

163
tests/e2e/llm_translation/test_cache_control.py Normal file
View file

@ -0,0 +1,163 @@
"""Live e2e: provider-specific /chat/completions features take real effect.
Each case asserts the feature actually happened, not just a 200. Coverage matrix
(register-on-demand deployments, deleted on teardown):
- Bedrock (anthropic claude-haiku-4-5): prompt caching. A large cacheable prefix
marked with ``cache_control`` is sent twice; the second call must report
cache-read usage tokens > 0. service_tier is out of scope for Bedrock; AWS
Bedrock does not expose an OpenAI-style request service tier, so that cell is
intentionally not covered here.
- Vertex (gemini-2.5-flash): prompt caching via ``cache_control`` context
caching; the second identical call must report cached prompt tokens > 0.
service_tier lives in test_provider_features_e2e.py.
The provider-native cache_control request shape is not expressible with the
shared ``ChatBody`` (whose content is a plain string), so the cacheable body is
modelled locally with typed content blocks.
"""
from __future__ import annotations
import time
import pytest
from pydantic import BaseModel
from e2e_config import unique_marker
from e2e_http import Result, unwrap
from lifecycle import ResourceManager
from models import ChatResponse, LiteLLMParamsBody, Usage
from passthrough_client import PassthroughClient
import os
pytestmark = pytest.mark.e2e
BEDROCK_MODEL = "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"
VERTEX_MODEL = "vertex_ai/gemini-2.5-flash"
class CacheControl(BaseModel):
type: str = "ephemeral"
class TextBlock(BaseModel):
type: str = "text"
text: str
cache_control: CacheControl | None = None
class RichMessage(BaseModel):
role: str
content: list[TextBlock]
class CacheChatBody(BaseModel):
model: str
messages: list[RichMessage]
max_tokens: int = 64
cache: dict[str, bool] = {"no-cache": True}
def _cacheable_prefix() -> str:
"""A prefix long enough to clear provider minimum cacheable sizes (Haiku is
2048 tokens), unique per run so the first call writes and the second reads."""
marker = unique_marker()
body = " ".join(
f"Cacheable reference paragraph {index} for run {marker}." for index in range(600)
)
return f"{body}\nEnd of reference material {marker}."
def _cached_read_tokens(usage: Usage | None) -> int:
"""Cache-read tokens however the provider reports them: Anthropic-style
``cache_read_input_tokens`` or OpenAI-style ``prompt_tokens_details.cached_tokens``."""
if usage is None:
return 0
if usage.cache_read_input_tokens:
return usage.cache_read_input_tokens
if usage.prompt_tokens_details and usage.prompt_tokens_details.cached_tokens:
return usage.prompt_tokens_details.cached_tokens
return 0
def _cache_chat(
client: PassthroughClient, key: str, model: str, prefix: str
) -> Result[ChatResponse]:
body = CacheChatBody(
model=model,
messages=[
RichMessage(
role="system",
content=[TextBlock(text=prefix, cache_control=CacheControl())],
),
RichMessage(role="user", content=[TextBlock(text="Reply with one word.")]),
],
)
return client.gateway.transport.post(
"/chat/completions",
headers=client.gateway.transport.bearer(key),
json=body,
response_type=ChatResponse,
)
def _assert_cache_read_on_second_call(
client: PassthroughClient, key: str, model: str
) -> None:
prefix = _cacheable_prefix()
first = unwrap(_cache_chat(client, key, model, prefix))
assert first.choices, f"{model}: first cache-priming call returned no choices: {first}"
read_tokens = 0
deadline = time.monotonic() + 30.0
while time.monotonic() < deadline:
second = unwrap(_cache_chat(client, key, model, prefix))
read_tokens = _cached_read_tokens(second.usage)
if read_tokens > 0:
break
time.sleep(3.0)
assert read_tokens > 0, (
f"{model}: second identical call reported no cache-read tokens "
f"({second.usage}); prompt caching did not take effect"
)
class TestCacheControl:
@pytest.mark.covers(
"llm.chat_completions.bedrock_converse.prompt_cache_5m.nonstream.works",
exercised_on=[],
)
def test_bedrock_prompt_caching_reads_cache(
self, client: PassthroughClient, resources: ResourceManager
) -> None:
model = f"e2e-bedrock-cache-{unique_marker()}"
model_id = client.gateway.create_model(
model,
LiteLLMParamsBody(model=BEDROCK_MODEL, aws_region_name="us-east-1"),
)
resources.defer(lambda: client.gateway.delete_model(model_id))
_assert_cache_read_on_second_call(client, resources.key(), model)
@pytest.mark.covers(
"llm.chat_completions.vertex.prompt_cache_5m.nonstream.works",
exercised_on=[],
)
def test_vertex_prompt_caching_reads_cache(
self, client: PassthroughClient, resources: ResourceManager
) -> None:
model = f"e2e-vertex-cache-{unique_marker()}"
model_id = client.gateway.create_model(
model,
LiteLLMParamsBody(
model=VERTEX_MODEL,
vertex_project=os.environ.get("VERTEXAI_PROJECT"),
vertex_location="us-central1",
vertex_credentials=os.environ.get("VERTEXAI_CREDENTIALS"),
),
)
resources.defer(lambda: client.gateway.delete_model(model_id))
_assert_cache_read_on_second_call(client, resources.key(), model)

29
tests/e2e/llm_translation/test_deepseek_reasoning_e2e.py
View file

@ -2,17 +2,14 @@
DeepSeek's reasoner defaults thinking ON and surfaces the chain as
``message.reasoning_content``. Two documented ways to disable it are
``reasoning_effort="none"`` and ``thinking={"type": "disabled"}``. Today the
DeepSeek param mapper (``litellm/llms/deepseek/chat/transformation.py``
``map_openai_params``) drops both without forwarding any disable signal, so the
outbound body carries no ``thinking`` key and DeepSeek keeps thinking on; the
response still comes back with ``reasoning_content``. That is the product gap
tracked by LIT-3686 / GH #27453.
``reasoning_effort="none"`` and ``thinking={"type": "disabled"}``. The DeepSeek
param mapper (``litellm/llms/deepseek/chat/transformation.py``
``map_openai_params``) forwards both as ``thinking={"type": "disabled"}`` so the
outbound body carries a real disable signal and ``deepseek-reasoner`` returns no
``reasoning_content``. This is the behavior tracked by LIT-3686 / GH #27453.
The control case proves the model and path work (reasoning is returned when
nothing asks to disable it), so the two disable assertions are meaningful. Those
two are marked xfail(strict) until the mapper forwards a real disable signal; an
xpass then alerts that the fix landed.
nothing asks to disable it), so the two disable assertions are meaningful.
Requires DEEPSEEK_API_KEY on the proxy (tests/e2e/.env). No skip gate: once the
proxy is up, a failure here is real, per the suite's hard-fail contract.
@ -74,13 +71,6 @@ class TestDeepSeekReasoningDisable:
f"disable param, so the disable assertions below can't be trusted: {response}"
)
@pytest.mark.xfail(
strict=True,
reason=(
"LIT-3686 / GH #27453: DeepSeek reasoning_effort='none' and "
"thinking type='disabled' are silently dropped; reasoning not disabled"
),
)
def test_reasoning_effort_none_disables_reasoning(
self, client: PassthroughClient, resources: ResourceManager
) -> None:
@ -103,13 +93,6 @@ class TestDeepSeekReasoningDisable:
f"is still present: {response}"
)
@pytest.mark.xfail(
strict=True,
reason=(
"LIT-3686 / GH #27453: DeepSeek reasoning_effort='none' and "
"thinking type='disabled' are silently dropped; reasoning not disabled"
),
)
def test_thinking_disabled_disables_reasoning(
self, client: PassthroughClient, resources: ResourceManager
) -> None:

23
tests/e2e/llm_translation/test_ocr_rust_e2e.py
View file

@ -86,16 +86,18 @@ class AzureDocIntelligenceOcr:
@dataclass(frozen=True, slots=True)
class VertexOcr:
"""Vertex AI OCR (Mistral publisher). Only the location (not a secret) is set;
the project and credentials are left unset so the gateway resolves VERTEXAI_PROJECT
and VERTEXAI_CREDENTIALS from its own environment by name, keeping every secret on
the gateway like the azure_ai cases above. This is deliberate: the OCR path reads
vertex_project verbatim from litellm_params and never unwraps an `os.environ/*`
ref, so passing one would put the literal string in the request URL."""
model: str
location: str
def litellm_params(self) -> LiteLLMParamsBody:
return LiteLLMParamsBody(
model=self.model,
vertex_project="os.environ/VERTEXAI_PROJECT",
vertex_location=self.location,
vertex_credentials="os.environ/VERTEXAI_CREDENTIALS",
)
return LiteLLMParamsBody(model=self.model, vertex_location=self.location)
@dataclass(frozen=True, slots=True)
@ -113,7 +115,7 @@ RUST_OCR_CASES: tuple[_OcrCase, ...] = (
),
_OcrCase(
"azure-ai",
AzureAiOcr("azure_ai/mistral-document-ai-2505"),
AzureAiOcr("azure_ai/mistral-document-ai-2512"),
OcrDocument(type="document_url", document_url=TEST_PDF_URL),
),
_OcrCase(
@ -126,11 +128,6 @@ RUST_OCR_CASES: tuple[_OcrCase, ...] = (
VertexOcr("vertex_ai/mistral-ocr-2505", "us-central1"),
OcrDocument(type="document_url", document_url=TEST_PDF_URL),
),
_OcrCase(
"vertex-deepseek",
VertexOcr("vertex_ai/deepseek-ocr-maas", "global"),
OcrDocument(type="image_url", image_url=TEST_IMAGE_URL),
),
)
_CASE_IDS = tuple(case.suffix for case in RUST_OCR_CASES)
@ -155,3 +152,5 @@ class TestRustOcrGateway:
response = unwrap(endpoints_client.gateway.ocr(key, OcrBody(model=model, document=case.document)))
_assert_ocr_document(response)

115
tests/e2e/llm_translation/test_provider_features_e2e.py
View file

@ -1,78 +1,32 @@
"""Live e2e for model-specific request features: service_tier and prompt caching.
"""Live e2e for model-specific request features: service_tier.
Each case asserts the feature took effect, not just a 200.
service_tier is an OpenAI concept. The proxy forwards it and the provider echoes
the tier back on the response, so sending a non-default tier ("flex") and reading
it back off ``service_tier`` proves the param was honored end to end; litellm's own
default injection would report "default", so a "flex" echo can only come from the
request being forwarded. Bedrock and Vertex do not accept service_tier, so that
cell is OpenAI-only by design.
the tier back on the response, so sending a non-default tier ("priority") and
reading it back off ``service_tier`` proves the param was honored end to end;
litellm's own default injection (and service_tier="auto") both report "default",
so a "priority" echo can only come from the request being forwarded. "flex" is
avoided here because it is capacity-constrained and returns a transient 429 when
flex resources are unavailable. Bedrock and Vertex do not accept service_tier, so
that cell is OpenAI-only by design.
Prompt caching is asserted through provider prompt-cache usage tokens. The
deterministic path is explicit ``cache_control`` on an Anthropic-family model
(here Bedrock's Claude): a large cacheable prefix is sent twice and the second
call must report ``cache_read_input_tokens > 0``. OpenAI and Gemini only offer
implicit automatic caching, which does not deterministically produce a cache read
within a test window (verified: repeated >3k-token prompts kept
``prompt_tokens_details.cached_tokens`` at 0), so those caching cells are out of
scope here and covered only by the explicit-cache-control Bedrock case.
Prompt caching lives in test_cache_control.py.
"""
from __future__ import annotations
import pytest
from pydantic import BaseModel
from e2e_config import unique_marker
from e2e_http import unwrap
from lifecycle import ResourceManager
from models import ChatBody, ChatMessage, ChatResponse, LiteLLMParamsBody
from models import ChatBody, ChatMessage, LiteLLMParamsBody
from passthrough_client import PassthroughClient
pytestmark = pytest.mark.e2e
SERVICE_TIER = "flex"
CACHE_MIN_READ_TOKENS = 1
class CacheControl(BaseModel):
type: str = "ephemeral"
class CacheTextBlock(BaseModel):
type: str = "text"
text: str
cache_control: CacheControl | None = None
class RichMessage(BaseModel):
role: str
content: list[CacheTextBlock]
class CacheChatBody(BaseModel):
model: str
messages: list[RichMessage]
max_tokens: int
def cacheable_prefix() -> str:
return (
"You are a policy compliance auditor. The following corpus is the immutable "
"reference the assistant must consult on every turn. "
) + ("Clause: obey all safety, formatting, and citation rules exactly. " * 400)
def post_chat(client: PassthroughClient, key: str, body: BaseModel) -> ChatResponse:
return unwrap(
client.gateway.transport.post(
"/chat/completions",
headers=client.gateway.transport.bearer(key),
json=body,
response_type=ChatResponse,
)
)
SERVICE_TIER = "priority"
class TestServiceTier:
@ -107,50 +61,3 @@ class TestServiceTier:
f"service_tier not honored: sent {SERVICE_TIER!r}, response reported "
f"{response.service_tier!r} ({response})"
)
class TestPromptCaching:
@pytest.mark.covers(
"llm.chat_completions.bedrock_converse.prompt_cache_5m.nonstream.works",
exercised_on=[],
)
def test_bedrock_cache_control_produces_cache_read(
self, client: PassthroughClient, resources: ResourceManager
) -> None:
model = f"e2e-bedrock-cache-{unique_marker()}"
model_id = client.gateway.create_model(
model,
LiteLLMParamsBody(
model="bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0",
aws_region_name="us-east-1",
),
)
resources.defer(lambda: client.gateway.delete_model(model_id))
key = resources.key()
body = CacheChatBody(
model=model,
max_tokens=32,
messages=[
RichMessage(
role="user",
content=[
CacheTextBlock(
text=cacheable_prefix(), cache_control=CacheControl()
),
CacheTextBlock(text="Answer in one word: acknowledged?"),
],
)
],
)
first = post_chat(client, key, body)
assert first.usage is not None, f"first call reported no usage: {first}"
second = post_chat(client, key, body)
assert second.usage is not None, f"second call reported no usage: {second}"
cache_read = second.usage.cache_read_input_tokens
assert cache_read is not None and cache_read >= CACHE_MIN_READ_TOKENS, (
"second identical request did not read the prompt cache: "
f"cache_read_input_tokens={cache_read!r} (usage={second.usage})"
)

19
tests/e2e/models.py
View file

@ -376,6 +376,7 @@ class LiteLLMParamsBody(BaseModel):
api_key: str | None = None
api_base: str | None = None
api_version: str | None = None
realtime_protocol: str | None = None
aws_region_name: str | None = None
vertex_project: str | None = None
vertex_location: str | None = None
@ -394,7 +395,11 @@ ModelMode = Literal["batch", "realtime", "image_generation"]
class ModelInfoBody(BaseModel):
id: str
# id is left unset so the proxy assigns a unique model_id per deployment.
# Pinning it to the model_name made re-registrations of a fixed-name model
# (e.g. the batch suite's openai-batch) collide on the model_id unique
# constraint when a prior run's teardown had not removed the row.
id: str | None = None
mode: ModelMode | None = None
@ -410,6 +415,18 @@ class ModelNewResponse(BaseModel):
model_id: str
class ModelListEntry(BaseModel):
id: str
class ModelsListResponse(BaseModel):
"""GET /v1/models on the data plane: the deployments the gateway can actually
serve right now. Used to confirm a freshly created model has propagated from
the control plane before a test calls it."""
data: tuple[ModelListEntry, ...] = ()
class ModelDeleteBody(BaseModel):
id: str

55
tests/e2e/realtime/REALTIME_COVERAGE_MATRIX.md
View file

@ -1,55 +0,0 @@
# Realtime e2e coverage
Live tests for the proxy realtime websocket endpoint (`/v1/realtime`). One
GA-speaking websocket client drives every provider; the proxy normalizes each
provider's stream into the OpenAI GA event schema, so the same assertions hold
across providers and only the model alias changes.
## What is asserted
For each configured provider, `test_text_conversation` checks the session
lifecycle (`session.created`, then `session.update` echoed by `session.updated`),
the canonical response sequence (`response.created`, `response.output_item.added`,
through `response.done`), that the streamed deltas reconstruct a non-empty
transcript, and that `response.done` carries normalized usage.
`test_tool_call_round_trip` checks the full tool path: the model emits a
normalized `response.function_call_arguments.done` with valid JSON arguments and
a matching `function_call` output item, the test sends a `function_call_output`
back, and the follow-up response incorporates the result (the temperature 72
appears).
`test_realtime_pipecat_e2e` is a realism layer that drives the same providers
through pipecat's GA `OpenAIRealtimeLLMService` (base_url pointed at the proxy)
rather than speaking the protocol by hand. Its assertions are coarse (the tool
callback fired, assistant text was produced); the raw-websocket suite is the
source of truth. It skips unless `pipecat-ai` is installed
(`uv pip install "pipecat-ai[openai]"`).
## Provider status
| provider | model alias | status |
|----------|-------------|--------|
| openai | `openai-realtime` | covered (in gateway config) |
| gemini | `gemini-realtime` | covered (in gateway config; needs Gemini Live API access) |
| azure | `azure-realtime` | gap: add to gateway config + AZURE creds |
| vertex_ai | `vertex-realtime` | gap: add to gateway config + Vertex creds |
| bedrock | `bedrock-realtime` | gap: add to gateway config + AWS creds |
| xai | `xai-realtime` | gap: add to gateway config + XAI_API_KEY |
A provider whose alias is not present in the proxy's `/model/info` skips (skip on
environment). To enable one, add a `model_info.mode: realtime` entry under that
alias to `tests/e2e/gateway/litellm-config.yml` and give the proxy the
provider's credentials; the test then runs with no code change.
## Running
Start a proxy with the gateway config and the provider keys set in its
environment, then
```
uv run pytest tests/e2e/realtime/ -v
```
Tests skip when no proxy answers `GET /health/liveliness` at `LITELLM_PROXY_URL`
(default `http://localhost:4000`).

20
tests/e2e/realtime/conftest.py
View file

@ -1,20 +0,0 @@
"""Realtime suite's `client` fixture.
The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker
live in the parent tests/e2e/conftest.py. RealtimeClient holds the shared
Gateway, so the `resources` fixture cleans up keys this suite creates.
"""
import pytest
from realtime_client import RealtimeClient, build_client
@pytest.fixture(scope="session")
def client() -> RealtimeClient:
return build_client()
@pytest.fixture(scope="session")
def configured_models(client: RealtimeClient) -> frozenset[str]:
return client.configured_models()

Some files were not shown because too many files have changed in this diff Show more