scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-12 23:01:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 181

Merge pull request #29243 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
Details
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
Details
CodeQL / Analyze (python) (push) Has been cancelled
Details
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Details
Helm unit test / unit-test (push) Has been cancelled
Details
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Details
Unit Tests: Security / security (push) Has been cancelled
Details
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / schema-migration (push) Has been cancelled
Details
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
Details

Browse source 
chore(ci): promote internal staging to main
This commit is contained in:
yuneng-jiang 2026-05-28 19:10:24 -07:00 • committed by GitHub
commit a021a5be86
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
295 changed files with 19057 additions and 5305 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

47
.github/workflows/create_daily_oss_agent_shin_branch.yml vendored Normal file
View file

@ -0,0 +1,47 @@
name: Create Daily oss-agent-shin Branch
on:
schedule:
- cron: "0 0 * * *" # Runs every day at midnight UTC
workflow_dispatch: # Allow manual trigger
jobs:
create-oss-agent-shin-branch:
if: github.repository == 'BerriAI/litellm'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
persist-credentials: false
- name: Create daily oss-agent-shin branch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Configure Git user
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
# Generate branch name with MM_DD_YYYY format
BRANCH_NAME="litellm_oss_agent_shin_$(date +'%m_%d_%Y')"
echo "Creating branch: $BRANCH_NAME"
# Fetch all branches
git fetch --all
# Check if the branch already exists
if git show-ref --verify --quiet refs/remotes/origin/$BRANCH_NAME; then
echo "Branch $BRANCH_NAME already exists. Skipping creation."
else
echo "Creating new branch: $BRANCH_NAME"
# Create the new branch from main
git checkout -b $BRANCH_NAME origin/main
# Push the new branch
git push origin $BRANCH_NAME
echo "Successfully created and pushed branch: $BRANCH_NAME"
fi

14
.github/workflows/test-unit-proxy-endpoints.yml vendored
View file

@ -7,6 +7,7 @@ on:
- litellm_internal_staging
- litellm_oss_branch
- "litellm_**"
workflow_dispatch:
permissions:
contents: read
@ -42,3 +43,16 @@ jobs:
workers: 2
reruns: 2
artifact-name: proxy-endpoints
# Behavior-pinning tests for litellm/proxy/proxy_server.py. Owns its
# own job (not a path on the proxy-endpoints job above) so its budget
# is independent and its coverage artifact is uploaded separately.
# See: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc
proxy-server:
uses: ./.github/workflows/_test-unit-base.yml
with:
test-path: tests/test_litellm/proxy/proxy_server
workers: 4
reruns: 2
timeout-minutes: 60
artifact-name: proxy-server

25
.github/workflows/test_server_root_path.yml vendored
View file

@ -101,6 +101,31 @@ jobs:
docker logs litellm-test
exit 1
- name: Setup Node for Playwright
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "20"
- name: Install UI deps and Chromium
working-directory: ui/litellm-dashboard
run: |
npm ci
npx playwright install --with-deps chromium
- name: Run SERVER_ROOT_PATH redirect e2e
working-directory: ui/litellm-dashboard
env:
SERVER_ROOT_PATH: ${{ matrix.root_path }}
run: npx playwright test --config=e2e_tests/serverRootPath.config.ts
- name: Upload Playwright artifacts on failure
if: failure()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: playwright-trace-${{ strategy.job-index }}
path: ui/litellm-dashboard/test-results/
retention-days: 7
- name: Cleanup
if: always()
run: |

13
AGENTS.md
View file

@ -2,6 +2,19 @@
This document provides comprehensive instructions for AI agents working in the LiteLLM repository.
## Confidentiality: Customer and Company Names in Code
The codebase is public. Before writing **any** third-party organization name into this repository — in source code, file or directory names, docstrings, comments, tests, fixtures, mock payloads, error messages, log lines, commit messages, or PR descriptions — pause and check:
**Already in the codebase** (OpenAI, Anthropic, Google, Azure, Bedrock, Fireworks, and other established LLM providers / integrations) — fine to use. Quick check: `git grep -i "<name>"` — if it returns hits in real code (not just your current diff), the name is established.
**Anything else** — customers, prospects, partners, new vendor integrations, observability tools, infra vendors, or any organization name that does not already appear in the repo. STOP and surface it to the user. Ask for explicit consent before writing the name into any file, commit message, or PR description. Do not write it speculatively and clean up later. Do not substitute a placeholder and proceed. Do not assume it is safe because it "looks like" a public company. The user must approve first.
**What to do instead of a customer-specific reference:**
- If you find yourself reaching for a customer name — real or fake — step back. The code shouldn't be customer-specific in the first place. Generalize the feature, or capture the customer motivation in internal docs (Notion / Linear / the internal staging PR description), never in the repo.
- Frame changes by the capability they add, not the customer who asked for it ("add per-team Bedrock guardrail routing", not "add routing for $CUSTOMER").
- Standard "fake value" markers (`example.com`, `localhost`, `127.0.0.1`, `test@example.com`) and abstract identifiers (`team_a`, `user_1`, `tenant_x`) are fine — those are not customer stand-ins.
## OVERVIEW
LiteLLM is a unified interface for 100+ LLMs that:

13
CLAUDE.md
View file

@ -2,6 +2,19 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Confidentiality: Customer and Company Names in Code
The codebase is public. Before writing **any** third-party organization name into this repository — in source code, file or directory names, docstrings, comments, tests, fixtures, mock payloads, error messages, log lines, commit messages, or PR descriptions — pause and check:
**Already in the codebase** (OpenAI, Anthropic, Google, Azure, Bedrock, Fireworks, and other established LLM providers / integrations) — fine to use. Quick check: `git grep -i "<name>"` — if it returns hits in real code (not just your current diff), the name is established.
**Anything else** — customers, prospects, partners, new vendor integrations, observability tools, infra vendors, or any organization name that does not already appear in the repo. STOP and surface it to the user. Ask for explicit consent before writing the name into any file, commit message, or PR description. Do not write it speculatively and clean up later. Do not substitute a placeholder and proceed. Do not assume it is safe because it "looks like" a public company. The user must approve first.
**What to do instead of a customer-specific reference:**
- If you find yourself reaching for a customer name — real or fake — step back. The code shouldn't be customer-specific in the first place. Generalize the feature, or capture the customer motivation in internal docs (Notion / Linear / the internal staging PR description), never in the repo.
- Frame changes by the capability they add, not the customer who asked for it ("add per-team Bedrock guardrail routing", not "add routing for $CUSTOMER").
- Standard "fake value" markers (`example.com`, `localhost`, `127.0.0.1`, `test@example.com`) and abstract identifiers (`team_a`, `user_1`, `tenant_x`) are fine — those are not customer stand-ins.
## Documentation
Documentation lives in a separate repository: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs). It is served at [docs.litellm.ai](https://docs.litellm.ai). Do not create or edit documentation files in this repository — open doc PRs against `BerriAI/litellm-docs` instead.

18
backend/Dockerfile
View file

@ -12,17 +12,27 @@ USER root
COPY --from=uvbin /uv /uvx /usr/local/bin/
RUN apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile
# nodejs/npm so `prisma generate` uses Wolfi's Node via PRISMA_USE_GLOBAL_NODE
# instead of nodeenv downloading one whose dynamic deps may not be in Wolfi
# (e.g. Node 26.2.0 needs libatomic). Retry for transient apk.cgr.dev flakes.
RUN for i in 1 2 3; do \
apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile nodejs npm && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
# UV_COMPILE_BYTECODE=1 precompiles .pyc at install time → faster cold start.
# UV_LINK_MODE=copy avoids hardlink warnings when uv installs from a
# BuildKit cache mount (different filesystem).
# UV_PYTHON_DOWNLOADS=0 force uv to use the apk-installed CPython instead of
# silently pulling a managed interpreter.
# PRISMA_USE_GLOBAL_NODE explicit (matches default) so an env override can't
# silently re-enable nodeenv's Node download.
ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=0 \
PRISMA_USE_GLOBAL_NODE=true \
PATH="/app/.venv/bin:${PATH}"
# Stage 1 — install dependencies only.
@ -58,7 +68,11 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
RUN apk add --no-cache bash openssl tzdata python3 libsndfile libatomic
RUN for i in 1 2 3; do \
apk add --no-cache bash openssl tzdata python3 libsndfile libatomic && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532) with
# /home/nonroot. We run the backend as that user

2
cookbook/gollem_go_agent_framework/go.mod
View file

@ -1,5 +1,5 @@
module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
go 1.25.1
go 1.26.3
require github.com/fugue-labs/gollem v0.1.0

2
deploy/charts/litellm-helm/templates/deployment.yaml
View file

@ -30,7 +30,7 @@ spec:
checksum/config: {{ include (print $.Template.BasePath "/configmap-litellm.yaml") . | sha256sum }}
{{- end }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
labels:
{{- include "litellm.labels" . | nindent 8 }}

25
deploy/charts/litellm-helm/tests/deployment_tests.yaml
View file

@ -377,3 +377,28 @@ tests:
content:
name: sidecar-tpl
image: "ghcr.io/berriai/litellm-database:test"
- it: should support tpl in podAnnotations
template: deployment.yaml
set:
image:
repository: ghcr.io/berriai/litellm-database
tag: test
# Mirrors the real-world scenario this feature unblocks:
# user disables the built-in ConfigMap (and its built-in checksum/config
# annotation) and re-implements checksum/config themselves via tpl.
proxyConfigMap:
create: false
podAnnotations:
checksum/config: "{{ .Values.image.tag }}"
example.com/some-key: "{{ .Values.image.repository }}"
example.com/literal: "plain-string-value"
asserts:
- equal:
path: spec.template.metadata.annotations["checksum/config"]
value: "test"
- equal:
path: spec.template.metadata.annotations["example.com/some-key"]
value: "ghcr.io/berriai/litellm-database"
- equal:
path: spec.template.metadata.annotations["example.com/literal"]
value: "plain-string-value"

14
docker/Dockerfile.non_root
View file

@ -55,22 +55,10 @@ COPY . .
# Set non-root flag for build time consistency
ENV LITELLM_NON_ROOT=true
# Stage the pre-built Admin UI from the checked-in Next.js static export.
# _experimental/out/ is regenerated as part of the release runbook.
# Restructure extensionless routes (foo.html -> foo/index.html) to match the layout
# proxy_server.py expects, and drop a readiness marker.
RUN mkdir -p /var/lib/litellm/ui /var/lib/litellm/assets && \
cp -r /app/litellm/proxy/_experimental/out/. /var/lib/litellm/ui/ && \
cp /app/litellm/proxy/logo.jpg /var/lib/litellm/assets/logo.jpg && \
( cd /var/lib/litellm/ui && \
for html_file in *.html; do \
if [ "$html_file" != "index.html" ] && [ -f "$html_file" ]; then \
folder_name="${html_file%.html}" && \
mkdir -p "$folder_name" && \
mv "$html_file" "$folder_name/index.html"; \
fi; \
done && \
touch .litellm_ui_ready )
touch /var/lib/litellm/ui/.litellm_ui_ready
RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
if [ "$PROXY_EXTRAS_SOURCE" = "published" ]; then \

18
gateway/Dockerfile
View file

@ -12,17 +12,27 @@ USER root
COPY --from=uvbin /uv /uvx /usr/local/bin/
RUN apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile
# nodejs/npm so `prisma generate` uses Wolfi's Node via PRISMA_USE_GLOBAL_NODE
# instead of nodeenv downloading one whose dynamic deps may not be in Wolfi
# (e.g. Node 26.2.0 needs libatomic). Retry for transient apk.cgr.dev flakes.
RUN for i in 1 2 3; do \
apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile nodejs npm && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
# UV_COMPILE_BYTECODE=1 precompiles .pyc at install time → faster cold start.
# UV_LINK_MODE=copy avoids hardlink warnings when uv installs from a
# BuildKit cache mount (different filesystem).
# UV_PYTHON_DOWNLOADS=0 force uv to use the apk-installed CPython instead of
# silently pulling a managed interpreter.
# PRISMA_USE_GLOBAL_NODE explicit (matches default) so an env override can't
# silently re-enable nodeenv's Node download.
ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=0 \
PRISMA_USE_GLOBAL_NODE=true \
PATH="/app/.venv/bin:${PATH}"
# Stage 1 — install dependencies only.
@ -58,7 +68,11 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
RUN apk add --no-cache bash openssl tzdata python3 libsndfile libatomic
RUN for i in 1 2 3; do \
apk add --no-cache bash openssl tzdata python3 libsndfile libatomic && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532) with
# /home/nonroot. We run the proxy as that user.

34
helm/litellm/templates/_helpers.tpl
View file

@ -56,16 +56,34 @@ app.kubernetes.io/component: ui
{{- end -}}
{{/*
Shared ServiceAccount name used by all three component Deployments. When
`serviceAccount.create` is true and `serviceAccount.name` is empty, default
to the chart fullname. When `create` is false, fall back to the provided
name or the namespace's `default` SA.
Per-component ServiceAccount name helpers.
Each component (gateway, backend, ui) has its own SA config under
.Values.serviceAccounts.<component>. When `create` is true and `name` is
empty the chart defaults to "<release>-litellm-<component>". When `create`
is false the chart uses the provided name, or the namespace `default` SA.
*/}}
{{- define "litellm.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
{{ default (include "litellm.fullname" .) .Values.serviceAccount.name }}
{{- define "litellm.gateway.serviceAccountName" -}}
{{- if .Values.serviceAccounts.gateway.create -}}
{{ default (include "litellm.gateway.fullname" .) .Values.serviceAccounts.gateway.name }}
{{- else -}}
{{ default "default" .Values.serviceAccount.name }}
{{ default "default" .Values.serviceAccounts.gateway.name }}
{{- end -}}
{{- end -}}
{{- define "litellm.backend.serviceAccountName" -}}
{{- if .Values.serviceAccounts.backend.create -}}
{{ default (include "litellm.backend.fullname" .) .Values.serviceAccounts.backend.name }}
{{- else -}}
{{ default "default" .Values.serviceAccounts.backend.name }}
{{- end -}}
{{- end -}}
{{- define "litellm.ui.serviceAccountName" -}}
{{- if .Values.serviceAccounts.ui.create -}}
{{ default (include "litellm.ui.fullname" .) .Values.serviceAccounts.ui.name }}
{{- else -}}
{{ default "default" .Values.serviceAccounts.ui.name }}
{{- end -}}
{{- end -}}

3
helm/litellm/templates/backend/deployment.yaml
View file

@ -19,7 +19,8 @@ spec:
labels:
{{- include "litellm.backend.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "litellm.serviceAccountName" . }}
serviceAccountName: {{ include "litellm.backend.serviceAccountName" . }}
automountServiceAccountToken: {{ .Values.serviceAccounts.backend.automount }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}

3
helm/litellm/templates/gateway/deployment.yaml
View file

@ -22,7 +22,8 @@ spec:
labels:
{{- include "litellm.gateway.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "litellm.serviceAccountName" . }}
serviceAccountName: {{ include "litellm.gateway.serviceAccountName" . }}
automountServiceAccountToken: {{ .Values.serviceAccounts.gateway.automount }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}

2
helm/litellm/templates/migrations-job.yaml
View file

@ -28,7 +28,7 @@ spec:
app.kubernetes.io/component: migrations
spec:
restartPolicy: Never
serviceAccountName: {{ include "litellm.serviceAccountName" . }}
serviceAccountName: {{ include "litellm.backend.serviceAccountName" . }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}

46
helm/litellm/templates/serviceaccount.yaml
View file

@ -1,13 +1,51 @@
{{- if .Values.serviceAccount.create -}}
{{- $prev := false -}}
{{- if .Values.serviceAccounts.gateway.create -}}
{{- $prev = true }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "litellm.serviceAccountName" . }}
name: {{ include "litellm.gateway.serviceAccountName" . }}
labels:
{{- include "litellm.commonLabels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
app.kubernetes.io/component: gateway
{{- with .Values.serviceAccounts.gateway.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
automountServiceAccountToken: {{ .Values.serviceAccounts.gateway.automount }}
{{- end }}
{{- if .Values.serviceAccounts.backend.create }}
{{- if $prev }}
---
{{- end }}
{{- $prev = true }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "litellm.backend.serviceAccountName" . }}
labels:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: backend
{{- with .Values.serviceAccounts.backend.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccounts.backend.automount }}
{{- end }}
{{- if .Values.serviceAccounts.ui.create }}
{{- if $prev }}
---
{{- end }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "litellm.ui.serviceAccountName" . }}
labels:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: ui
{{- with .Values.serviceAccounts.ui.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccounts.ui.automount }}
{{- end }}

3
helm/litellm/templates/ui/deployment.yaml
View file

@ -19,7 +19,8 @@ spec:
labels:
{{- include "litellm.ui.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "litellm.serviceAccountName" . }}
serviceAccountName: {{ include "litellm.ui.serviceAccountName" . }}
automountServiceAccountToken: {{ .Values.serviceAccounts.ui.automount }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}

37
helm/litellm/values.yaml
View file

@ -14,16 +14,33 @@ ingress:
host: "" # optional; if set, becomes the rule's host
tls: []
# Shared ServiceAccount used by all three component Deployments. Set
# `create: true` to have the chart provision it (e.g. when wiring an EKS
# Pod Identity association by SA name). Set `name` to use an existing SA
# (chart-created or out-of-band). When both are empty / false, pods run
# with the namespace's `default` SA.
serviceAccount:
create: false
automount: true
annotations: {}
name: ""
# Per-component ServiceAccounts for gateway, backend, and ui.
#
# Each section mirrors the old shared serviceAccount shape. Set `create:
# true` to have the chart provision the SA (useful for EKS Pod Identity /
# GKE Workload Identity annotations). Set `name` to bind an existing SA.
# When both are unset the component pod runs with the namespace `default` SA.
#
# The UI SA deliberately defaults to `automount: false` — the static nginx
# container does not need the K8s API and should not carry a projected
# ServiceAccount token that a compromised container could use to call the
# cloud-provider metadata service or the K8s API.
serviceAccounts:
gateway:
create: false
automount: true
annotations: {}
name: ""
backend:
create: false
automount: true
annotations: {}
name: ""
ui:
create: false
automount: false
annotations: {}
name: ""
# Pre-install / pre-upgrade Helm hook that runs `prisma migrate deploy`
# against the writer database, creating the LiteLLM schema (tables that

5
litellm/__init__.py
View file

@ -225,6 +225,11 @@ use_chat_completions_url_for_anthropic_messages: bool = bool(
route_all_chat_openai_to_responses: bool = (
os.getenv("LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES", "false").lower() == "true"
) # When True, routes all OpenAI /chat/completions requests through the Responses API bridge
# When True, Gemini/Vertex Live setup is deferred until client `session.update`.
# Default False preserves historical behavior (auto-send setup on connect).
gemini_live_defer_setup: bool = (
os.getenv("LITELLM_GEMINI_LIVE_DEFER_SETUP", "false").lower() == "true"
)
use_legacy_interactions_schema: bool = (
os.getenv("LITELLM_USE_LEGACY_INTERACTIONS_SCHEMA", "false").lower() == "true"
) # When True, sends Api-Revision: 2026-05-07 to Google so responses use the legacy `outputs`

1
litellm/constants.py
View file

@ -1147,6 +1147,7 @@ BEDROCK_CONVERSE_MODELS = [
"openai.gpt-oss-120b-1:0",
"anthropic.claude-haiku-4-5-20251001-v1:0",
"anthropic.claude-sonnet-4-5-20250929-v1:0",
"anthropic.claude-opus-4-8",
"anthropic.claude-opus-4-7",
"anthropic.claude-opus-4-6-v1:0",
"anthropic.claude-opus-4-6-v1",

73
litellm/cost_calculator.py
View file

@ -24,6 +24,7 @@ from litellm.litellm_core_utils.llm_cost_calc.usage_object_transformation import
from litellm.litellm_core_utils.llm_cost_calc.utils import (
CostCalculatorUtils,
_generic_cost_per_character,
_get_regional_uplift_multiplier,
_get_service_tier_cost_key,
_parse_prompt_tokens_details,
calculate_cost_component,
@ -132,6 +133,8 @@ _VIDEO_CALL_TYPES = frozenset(
{
CallTypes.create_video.value,
CallTypes.acreate_video.value,
CallTypes.video_edit.value,
CallTypes.avideo_edit.value,
CallTypes.video_remix.value,
CallTypes.avideo_remix.value,
}
@ -312,6 +315,10 @@ def cost_per_token( # noqa: PLR0915
audio_transcription_file_duration: float = 0.0, # for audio transcription calls - the file time in seconds
### SERVICE TIER ###
service_tier: Optional[str] = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
data_residency: Optional[
str
] = None, # for OpenAI regional-processing uplift (e.g. "eu", "us")
response: Optional[Any] = None,
### REQUEST MODEL ###
request_model: Optional[str] = None, # original request model for router detection
@ -412,9 +419,36 @@ def cost_per_token( # noqa: PLR0915
prompt_tokens_cost_usd_dollar: float = 0
completion_tokens_cost_usd_dollar: float = 0
model_cost_ref = litellm.model_cost
# Only callers that explicitly pass `custom_llm_provider` get the
# dedup/prefix-join treatment. When provider is omitted, preserve legacy
# behavior: `model_with_provider` stays equal to the raw `model` string
# (provider is detected below for downstream use only).
caller_supplied_provider = custom_llm_provider is not None
# `model` is normally a string, but callers that mock the transport can pass
# non-string objects. Only run the string-based dedup/prefix-join when it is
# actually a string — e.g. a MagicMock's `.startswith()` is always truthy and
# its slices return new mocks, which would spin the dedup loop forever.
model_is_str = isinstance(model, str)
# Router/proxy deployments may repeat the provider segment (e.g. model_name
# "openai/openai/gpt-5.5"). Strip duplicated `{provider}/` chains before joining.
if caller_supplied_provider and model_is_str:
_dup_prefix = f"{custom_llm_provider}/"
while model.startswith(_dup_prefix):
_remainder = model[len(_dup_prefix) :]
if _remainder.startswith(_dup_prefix):
model = _remainder
else:
break
model_with_provider = model
if custom_llm_provider is not None:
model_with_provider = custom_llm_provider + "/" + model
if caller_supplied_provider:
_prov_prefix = f"{custom_llm_provider}/"
if model_is_str and model.startswith(_prov_prefix):
model_with_provider = model
else:
model_with_provider = f"{custom_llm_provider}/{model}"
if region_name is not None:
model_with_provider_and_region = (
f"{custom_llm_provider}/{region_name}/{model}"
@ -425,6 +459,9 @@ def cost_per_token( # noqa: PLR0915
model_with_provider = model_with_provider_and_region
else:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(model=model)
assert custom_llm_provider is not None # caller-supplied or get_llm_provider
model_without_prefix = model
model_parts = model.split("/", 1)
if len(model_parts) > 1:
@ -493,6 +530,7 @@ def cost_per_token( # noqa: PLR0915
usage=usage_block,
custom_llm_provider=custom_llm_provider,
service_tier=service_tier,
data_residency=data_residency,
)
return prompt_cost, completion_cost
@ -521,7 +559,10 @@ def cost_per_token( # noqa: PLR0915
or call_type == CallTypes.retrieve_batch
):
return batch_cost_calculator(
usage=usage_block, model=model, custom_llm_provider=custom_llm_provider
usage=usage_block,
model=model,
custom_llm_provider=custom_llm_provider,
data_residency=data_residency,
)
elif call_type == "atranscription" or call_type == "transcription":
if _transcription_usage_has_token_details(usage_block):
@ -529,6 +570,7 @@ def cost_per_token( # noqa: PLR0915
model=model_without_prefix,
usage=usage_block,
service_tier=service_tier,
data_residency=data_residency,
)
return openai_cost_per_second(
@ -579,7 +621,10 @@ def cost_per_token( # noqa: PLR0915
)
elif custom_llm_provider == "openai":
return openai_cost_per_token(
model=model, usage=usage_block, service_tier=service_tier
model=model,
usage=usage_block,
service_tier=service_tier,
data_residency=data_residency,
)
elif custom_llm_provider == "databricks":
return databricks_cost_per_token(model=model, usage=usage_block)
@ -631,6 +676,7 @@ def cost_per_token( # noqa: PLR0915
usage=usage_block,
custom_llm_provider=custom_llm_provider,
service_tier=service_tier,
data_residency=data_residency,
)
if (
@ -1117,6 +1163,10 @@ def completion_cost( # noqa: PLR0915
litellm_logging_obj: Optional[LitellmLoggingObject] = None,
### SERVICE TIER ###
service_tier: Optional[str] = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
data_residency: Optional[
str
] = None, # for OpenAI regional-processing uplift (e.g. "eu", "us")
) -> float:
"""
Calculate the cost of a given completion call fot GPT-3.5-turbo, llama2, any litellm supported llm.
@ -1516,6 +1566,7 @@ def completion_cost( # noqa: PLR0915
combined_usage_object=cost_per_token_usage_object,
custom_llm_provider=custom_llm_provider,
litellm_model_name=model,
data_residency=data_residency,
)
elif call_type == _MCP_CALL_TYPE:
from litellm.proxy._experimental.mcp_server.cost_calculator import (
@ -1600,6 +1651,7 @@ def completion_cost( # noqa: PLR0915
audio_transcription_file_duration=audio_transcription_file_duration,
rerank_billed_units=rerank_billed_units,
service_tier=service_tier,
data_residency=data_residency,
response=completion_response,
request_model=request_model_for_cost,
)
@ -1811,6 +1863,10 @@ def response_cost_calculator(
litellm_logging_obj: Optional[LitellmLoggingObject] = None,
### SERVICE TIER ###
service_tier: Optional[str] = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
data_residency: Optional[
str
] = None, # for OpenAI regional-processing uplift (e.g. "eu", "us")
) -> float:
"""
Returns
@ -1844,6 +1900,7 @@ def response_cost_calculator(
router_model_id=router_model_id,
litellm_logging_obj=litellm_logging_obj,
service_tier=service_tier,
data_residency=data_residency,
)
return response_cost
except Exception as e:
@ -2202,6 +2259,7 @@ def batch_cost_calculator(
model: str,
custom_llm_provider: Optional[str] = None,
model_info: Optional[ModelInfo] = None,
data_residency: Optional[str] = None,
) -> Tuple[float, float]:
"""
Calculate the cost of a batch job.
@ -2286,6 +2344,11 @@ def batch_cost_calculator(
usage.completion_tokens * (output_cost_per_token) / 2
) # batch cost is usually half of the regular token cost
uplift = _get_regional_uplift_multiplier(model_info, data_residency)
if uplift != 1.0:
total_prompt_cost *= uplift
total_completion_cost *= uplift
return total_prompt_cost, total_completion_cost
@ -2431,6 +2494,7 @@ def handle_realtime_stream_cost_calculation(
combined_usage_object: Usage,
custom_llm_provider: str,
litellm_model_name: str,
data_residency: Optional[str] = None,
) -> float:
"""
Handles the cost calculation for realtime stream responses.
@ -2461,6 +2525,7 @@ def handle_realtime_stream_cost_calculation(
model=model_name,
usage=combined_usage_object,
custom_llm_provider=custom_llm_provider,
data_residency=data_residency,
)
except Exception:
continue

397
litellm/integrations/arize/arize_phoenix.py
View file

@ -1,5 +1,7 @@
import os
from typing import TYPE_CHECKING, Any, Optional, Union
import threading
from collections import OrderedDict
from typing import TYPE_CHECKING, Any, Optional, Tuple, Union
from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
@ -8,8 +10,10 @@ from litellm.types.integrations.arize_phoenix import ArizePhoenixConfig
if TYPE_CHECKING:
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SpanProcessor
from opentelemetry.trace import Span as _Span
from opentelemetry.trace import SpanKind
from opentelemetry.trace import Tracer
from litellm.integrations.opentelemetry import OpenTelemetry as _OpenTelemetry
from litellm.integrations.opentelemetry import (
@ -21,20 +25,27 @@ if TYPE_CHECKING:
OpenTelemetryConfig = _OpenTelemetryConfig
Span = Union[_Span, Any]
OpenTelemetry = _OpenTelemetry
LITELLM_TRACER_NAME: str
else:
Protocol = Any
OpenTelemetryConfig = Any
Span = Any
Tracer = Any
TracerProvider = Any
SpanKind = Any
# Import OpenTelemetry at runtime
SpanProcessor = Any
try:
from litellm.integrations.opentelemetry import OpenTelemetry
from litellm.integrations.opentelemetry import (
LITELLM_TRACER_NAME,
OpenTelemetry,
)
except ImportError:
LITELLM_TRACER_NAME = "litellm"
OpenTelemetry = None # type: ignore
ARIZE_HOSTED_PHOENIX_ENDPOINT = "https://otlp.arize.com/v1/traces"
_MAX_PROJECT_PROVIDERS = 64
class ArizePhoenixLogger(OpenTelemetry): # type: ignore
@ -48,37 +59,142 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
def _init_tracing(self, tracer_provider):
"""
Override to always create a *private* TracerProvider for Arize Phoenix.
Override to create per-project TracerProviders (LRU-cached) for Arize Phoenix.
The base ``OpenTelemetry._init_tracing`` falls back to the global
TracerProvider when one already exists. That causes whichever
integration initialises second to silently reuse the first one's
exporter, so spans only reach one destination.
By creating our own provider we guarantee Arize Phoenix always gets
its own exporter pipeline, regardless of initialisation order.
"""
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import SpanKind
if tracer_provider is not None:
# Explicitly supplied (e.g. in tests) — honour it.
self.tracer = tracer_provider.get_tracer("litellm")
self._use_injected_tracer_provider = True
self._shared_span_processor = None
self.tracer = tracer_provider.get_tracer(LITELLM_TRACER_NAME)
self.span_kind = SpanKind
return
# Always create a dedicated provider — never touch the global one.
provider = TracerProvider(resource=self._get_litellm_resource(self.config))
provider.add_span_processor(self._get_span_processor())
self.tracer = provider.get_tracer("litellm")
self._use_injected_tracer_provider = False
self._project_providers: OrderedDict[str, TracerProvider] = OrderedDict()
self._project_providers_lock = threading.Lock()
self._shared_span_processor = self._get_span_processor()
self.span_kind = SpanKind
default_project = self._resolve_project_name({})
self.tracer = self._get_tracer_for(default_project)
verbose_logger.debug(
"ArizePhoenixLogger: Created dedicated TracerProvider "
"(endpoint=%s, exporter=%s)",
"ArizePhoenixLogger: Initialized per-project TracerProvider cache "
"(default_project=%s, endpoint=%s, exporter=%s)",
default_project,
self.config.endpoint,
self.config.exporter,
)
def flush_tracer_providers(self) -> None:
"""
Flush all cached per-project providers and the shared span processor.
Call on graceful proxy shutdown. Do not call on LRU eviction — in-flight
spans may still reference evicted providers.
"""
if getattr(self, "_use_injected_tracer_provider", False):
return
shared_processor = getattr(self, "_shared_span_processor", None)
if shared_processor is not None:
try:
shared_processor.force_flush()
except Exception as e:
verbose_logger.debug(
"ArizePhoenixLogger: shared span processor force_flush failed: %s",
e,
)
with getattr(self, "_project_providers_lock", threading.Lock()):
providers = list(getattr(self, "_project_providers", {}).values())
for provider in providers:
try:
provider.force_flush()
except Exception as e:
verbose_logger.debug(
"ArizePhoenixLogger: TracerProvider force_flush failed: %s", e
)
def _get_litellm_resource_for_project(self, project_name: str):
"""
Build an OTEL Resource with project routing attrs that win over env detector.
Phoenix uses ``openinference.project.name``; Arize AX uses ``model_id`` and
``service.name``. Project attrs are merged last so OTEL_RESOURCE_ATTRIBUTES
from init does not pin every provider to one project.
"""
from opentelemetry.sdk.resources import OTELResourceDetector, Resource
project_attributes: dict[str, str] = {
"openinference.project.name": project_name,
"model_id": project_name,
"service.name": project_name,
}
deployment_environment = getattr(self.config, "deployment_environment", None)
if deployment_environment is not None:
project_attributes["deployment.environment"] = deployment_environment
env_resource = OTELResourceDetector().detect()
project_resource = Resource.create(project_attributes) # type: ignore[arg-type]
return env_resource.merge(project_resource)
def _build_tracer_provider_for_project(self, project_name: str) -> TracerProvider:
"""Create a TracerProvider for *project_name* (caller holds no cache lock)."""
from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider(
resource=self._get_litellm_resource_for_project(project_name)
)
provider.add_span_processor(self._shared_span_processor)
return provider
def _get_tracer_for(self, project_name: str) -> Tracer:
"""Return a tracer for *project_name*, creating/caching a provider on miss."""
if getattr(self, "_use_injected_tracer_provider", False):
return self.tracer
with self._project_providers_lock:
if project_name in self._project_providers:
self._project_providers.move_to_end(project_name)
return self._project_providers[project_name].get_tracer(
LITELLM_TRACER_NAME
)
# OTELResourceDetector().detect() is synchronous; build outside the lock so
# concurrent requests for other projects are not blocked on cache misses.
new_provider = self._build_tracer_provider_for_project(project_name)
with self._project_providers_lock:
if project_name in self._project_providers:
self._project_providers.move_to_end(project_name)
return self._project_providers[project_name].get_tracer(
LITELLM_TRACER_NAME
)
if len(self._project_providers) >= _MAX_PROJECT_PROVIDERS:
self._project_providers.popitem(last=False)
self._project_providers[project_name] = new_provider
return new_provider.get_tracer(LITELLM_TRACER_NAME)
def _resolve_tracer_for_kwargs(self, kwargs: dict) -> Tuple[str, Tracer]:
"""Resolve project name once and return the matching tracer."""
project_name = self._resolve_project_name(kwargs)
return project_name, self._get_tracer_for(project_name)
def get_tracer_to_use_for_request(self, kwargs: dict) -> Tracer:
"""Route guardrail/raw-request spans to the same per-project tracer as the request."""
if getattr(self, "_use_injected_tracer_provider", False):
return self.tracer
return self._resolve_tracer_for_kwargs(kwargs)[1]
def _init_otel_logger_on_litellm_proxy(self):
"""
Override: Arize Phoenix should NOT overwrite the proxy's
@ -93,56 +209,109 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
@staticmethod
def set_arize_phoenix_attributes(span: Span, kwargs, response_obj):
from litellm.integrations.opentelemetry_utils.base_otel_llm_obs_attributes import (
safe_set_attribute,
)
_utils.set_attributes(span, kwargs, response_obj, ArizeOTELAttributes)
# Dynamic project name: check metadata first, then fall back to env var config
dynamic_project_name = ArizePhoenixLogger._get_dynamic_project_name(kwargs)
if dynamic_project_name:
safe_set_attribute(span, "openinference.project.name", dynamic_project_name)
else:
# Fall back to static config from env var
config = ArizePhoenixLogger.get_arize_phoenix_config()
if config.project_name:
safe_set_attribute(
span, "openinference.project.name", config.project_name
)
return
@staticmethod
def _get_dynamic_project_name(kwargs) -> Optional[str]:
"""
Retrieve dynamic Phoenix project name from request metadata.
def _normalize_project_name(name: Optional[str]) -> Optional[str]:
if name is None:
return None
normalized = str(name).strip()
return normalized if normalized else None
Users can set `metadata.phoenix_project_name` in their request to route
traces to different Phoenix projects dynamically.
"""
standard_logging_payload = kwargs.get("standard_logging_object")
if isinstance(standard_logging_payload, dict):
metadata = standard_logging_payload.get("metadata")
@staticmethod
def _iter_metadata_dicts_from_kwargs(kwargs: dict):
"""Yield request metadata dicts; standard_logging_object before litellm_params."""
for key in ("standard_logging_object", "litellm_params"):
found_key = kwargs.get(key)
if not isinstance(found_key, dict):
continue
metadata = found_key.get("metadata")
if isinstance(metadata, dict):
project_name = metadata.get("phoenix_project_name")
if project_name:
return str(project_name)
yield metadata
# Also check litellm_params.metadata for SDK usage
@staticmethod
def _is_proxy_request(kwargs: dict) -> bool:
"""True when the call is routed through the LiteLLM proxy.
Proxy mode is determined solely by the server-set ``proxy_server_request``
field in ``litellm_params``. Checking request metadata for
``user_api_key_auth_metadata`` is intentionally avoided: that field is
user-supplied and would let an authenticated caller fake proxy-mode
detection to route their telemetry into arbitrary Arize/Phoenix projects.
"""
litellm_params = kwargs.get("litellm_params")
if isinstance(litellm_params, dict):
metadata = litellm_params.get("metadata") or {}
else:
metadata = {}
if isinstance(metadata, dict):
project_name = metadata.get("phoenix_project_name")
if project_name:
return str(project_name)
return isinstance(litellm_params, dict) and bool(
litellm_params.get("proxy_server_request")
)
@staticmethod
def _project_from_metadata_dict(
metadata: dict, metadata_key: str, *, proxy_mode: bool
) -> Optional[str]:
"""
Read a Phoenix project field from proxy/SDK metadata.
On the proxy, only ``user_api_key_auth_metadata`` (team/key config) may
select the project. SDK callers may still set project fields directly on
``metadata``.
"""
auth_metadata = metadata.get("user_api_key_auth_metadata")
if isinstance(auth_metadata, dict):
project = ArizePhoenixLogger._normalize_project_name(
auth_metadata.get(metadata_key)
)
if project:
return project
if not proxy_mode:
return ArizePhoenixLogger._normalize_project_name(
metadata.get(metadata_key)
)
return None
def _get_phoenix_context(self, kwargs):
@staticmethod
def _metadata_project_from_kwargs(kwargs: dict, metadata_key: str) -> Optional[str]:
proxy_mode = ArizePhoenixLogger._is_proxy_request(kwargs)
for metadata in ArizePhoenixLogger._iter_metadata_dicts_from_kwargs(kwargs):
project = ArizePhoenixLogger._project_from_metadata_dict(
metadata, metadata_key, proxy_mode=proxy_mode
)
if project:
return project
return None
@staticmethod
def _resolve_project_name(kwargs: dict) -> str:
"""
Resolve the target Phoenix/Arize project for this request.
Proxy priority: ``user_api_key_auth_metadata.phoenix_project_name_override``,
``user_api_key_auth_metadata.phoenix_project_name``, env, then ``default``.
SDK priority: request metadata fields, then env, then ``default``.
"""
override = ArizePhoenixLogger._metadata_project_from_kwargs(
kwargs, "phoenix_project_name_override"
)
if override:
return override
phoenix_name = ArizePhoenixLogger._metadata_project_from_kwargs(
kwargs, "phoenix_project_name"
)
if phoenix_name:
return phoenix_name
env_name = ArizePhoenixLogger._normalize_project_name(
os.environ.get("PHOENIX_PROJECT_NAME")
or os.environ.get("ARIZE_PROJECT_NAME")
)
if env_name:
return env_name
return "default"
def _get_phoenix_context(self, kwargs, tracer: Optional[Tracer] = None):
"""
Build a trace context for Phoenix's dedicated TracerProvider.
@ -159,11 +328,13 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
"""
from opentelemetry import trace
if tracer is None:
tracer = self._resolve_tracer_for_kwargs(kwargs)[1]
litellm_params = kwargs.get("litellm_params", {}) or {}
proxy_server_request = litellm_params.get("proxy_server_request", {}) or {}
headers = proxy_server_request.get("headers", {}) or {}
# Propagate distributed trace context if the caller sent a traceparent
traceparent_ctx = (
self.get_traceparent_from_header(headers=headers)
if headers.get("traceparent")
@ -173,10 +344,8 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
is_proxy_mode = bool(proxy_server_request)
if is_proxy_mode:
# Create a parent span on Phoenix's own tracer so both parent
# and child are exported to Phoenix.
start_time_val = kwargs.get("start_time", kwargs.get("api_call_start_time"))
parent_span = self.tracer.start_span(
parent_span = tracer.start_span(
name="litellm_proxy_request",
start_time=(
self._to_ns(start_time_val) if start_time_val is not None else None
@ -187,100 +356,77 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
ctx = trace.set_span_in_context(parent_span)
return ctx, parent_span
# SDK mode — no parent span needed
return traceparent_ctx, None
def _handle_success(self, kwargs, response_obj, start_time, end_time):
"""
Override to always create spans on ArizePhoenixLogger's dedicated TracerProvider.
The base class's ``_get_span_context`` would find the parent span created by
the ``otel`` callback on the *global* TracerProvider. That span is invisible
in Phoenix (different exporter pipeline), so we ignore it and build our own
hierarchy via ``_get_phoenix_context``.
"""
from opentelemetry.trace import Status, StatusCode
verbose_logger.debug(
"ArizePhoenixLogger: Logging kwargs: %s, OTEL config settings=%s",
kwargs,
self.config,
self._handle_phoenix_trace(
kwargs, response_obj, start_time, end_time, success=True
)
ctx, parent_span = self._get_phoenix_context(kwargs)
# Create litellm_request span (child of our parent when in proxy mode)
span = self.tracer.start_span(
name=self._get_span_name(kwargs),
start_time=self._to_ns(start_time),
context=ctx,
)
span.set_status(Status(StatusCode.OK))
self.set_attributes(span, kwargs, response_obj)
# Raw-request sub-span (if enabled) — must be created before
# ending the parent span so the hierarchy is valid.
self._maybe_log_raw_request(kwargs, response_obj, start_time, end_time, span)
span.end(end_time=self._to_ns(end_time))
# Guardrail span
self._create_guardrail_span(kwargs=kwargs, context=ctx)
# Annotate and close our proxy parent span
if parent_span is not None:
parent_span.set_status(Status(StatusCode.OK))
self.set_attributes(parent_span, kwargs, response_obj)
parent_span.end(end_time=self._to_ns(end_time))
# Metrics & cost recording
self._record_metrics(kwargs, response_obj, start_time, end_time)
# Semantic logs
if self.config.enable_events:
self._emit_semantic_logs(kwargs, response_obj, span)
def _handle_failure(self, kwargs, response_obj, start_time, end_time):
"""
Override to always create failure spans on ArizePhoenixLogger's dedicated
TracerProvider. Mirrors ``_handle_success`` but sets ERROR status.
"""
self._handle_phoenix_trace(
kwargs, response_obj, start_time, end_time, success=False
)
def _handle_phoenix_trace(
self,
kwargs,
response_obj,
start_time,
end_time,
*,
success: bool,
):
from opentelemetry.trace import Status, StatusCode
verbose_logger.debug(
"ArizePhoenixLogger: Failure - Logging kwargs: %s, OTEL config settings=%s",
"ArizePhoenixLogger: %s - kwargs: %s, OTEL config settings=%s",
"success" if success else "failure",
kwargs,
self.config,
)
ctx, parent_span = self._get_phoenix_context(kwargs)
_project_name, tracer = self._resolve_tracer_for_kwargs(kwargs)
ctx, parent_span = self._get_phoenix_context(kwargs, tracer=tracer)
# Create litellm_request span (child of our parent when in proxy mode)
span = self.tracer.start_span(
status = Status(StatusCode.OK if success else StatusCode.ERROR)
span = tracer.start_span(
name=self._get_span_name(kwargs),
start_time=self._to_ns(start_time),
context=ctx,
)
span.set_status(Status(StatusCode.ERROR))
span.set_status(status)
self.set_attributes(span, kwargs, response_obj)
self._record_exception_on_span(span=span, kwargs=kwargs)
if not success:
self._record_exception_on_span(span=span, kwargs=kwargs)
if success:
self._maybe_log_raw_request(
kwargs, response_obj, start_time, end_time, span
)
span.end(end_time=self._to_ns(end_time))
# Guardrail span
self._create_guardrail_span(kwargs=kwargs, context=ctx)
# Annotate and close our proxy parent span
if parent_span is not None:
parent_span.set_status(Status(StatusCode.ERROR))
parent_span.set_status(status)
self.set_attributes(parent_span, kwargs, response_obj)
self._record_exception_on_span(span=parent_span, kwargs=kwargs)
if not success:
self._record_exception_on_span(span=parent_span, kwargs=kwargs)
parent_span.end(end_time=self._to_ns(end_time))
if success:
self._record_metrics(kwargs, response_obj, start_time, end_time)
if self.config.enable_events:
self._emit_semantic_logs(kwargs, response_obj, span)
@staticmethod
def get_arize_phoenix_config() -> ArizePhoenixConfig:
"""
Retrieves the Arize Phoenix configuration based on environment variables.
Returns:
ArizePhoenixConfig: A Pydantic model containing Arize Phoenix configuration.
"""
api_key = os.environ.get("PHOENIX_API_KEY", None)
@ -295,18 +441,15 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
protocol: Protocol = "otlp_http"
if collector_endpoint:
# Parse the endpoint to determine protocol
if collector_endpoint.startswith("grpc://") or (
":4317" in collector_endpoint and "/v1/traces" not in collector_endpoint
):
endpoint = collector_endpoint
protocol = "otlp_grpc"
else:
# Phoenix Cloud endpoints (app.phoenix.arize.com) include the space in the URL
if "app.phoenix.arize.com" in collector_endpoint:
endpoint = collector_endpoint
protocol = "otlp_http"
# For other HTTP endpoints, ensure they have the correct path
elif "/v1/traces" not in collector_endpoint:
if collector_endpoint.endswith("/v1"):
endpoint = collector_endpoint + "/traces"
@ -318,7 +461,6 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
endpoint = collector_endpoint
protocol = "otlp_http"
else:
# If no endpoint specified, self hosted phoenix
endpoint = "http://localhost:6006/v1/traces"
protocol = "otlp_http"
verbose_logger.debug(
@ -329,12 +471,11 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
if api_key is not None:
otlp_auth_headers = f"Authorization=Bearer {api_key}"
elif "app.phoenix.arize.com" in endpoint:
# Phoenix Cloud requires an API key
raise ValueError(
"PHOENIX_API_KEY must be set when using Phoenix Cloud (app.phoenix.arize.com)."
)
project_name = os.environ.get("PHOENIX_PROJECT_NAME", "default")
project_name = os.environ.get("PHOENIX_PROJECT_NAME") or "default"
return ArizePhoenixConfig(
otlp_auth_headers=otlp_auth_headers,
@ -343,8 +484,6 @@ class ArizePhoenixLogger(OpenTelemetry): # type: ignore
project_name=project_name,
)
## cannot suppress additional proxy server spans, removed previous methods.
async def async_health_check(self):
config = self.get_arize_phoenix_config()

145
litellm/integrations/datadog/datadog_cost_management.py
View file

@ -2,10 +2,17 @@ import asyncio
import os
import time
from datetime import datetime
from typing import Dict, List, Optional, Tuple
from typing import Any, Dict, List, Optional, Tuple, cast
from litellm._logging import verbose_logger
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.integrations.datadog.datadog_handler import (
get_datadog_env,
get_datadog_hostname,
get_datadog_pod_name,
get_datadog_service,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@ -15,9 +22,30 @@ from litellm.types.integrations.datadog_cost_management import (
)
from litellm.types.utils import StandardLoggingPayload
# Reserved tag keys whose values come from trusted sources (infra env, LiteLLM
# core payload fields, or proxy-controlled auth metadata). User-supplied
# request_tags / metadata cannot overwrite these, even when the key is
# allowlisted via cost_tag_keys, because that would let an authenticated caller
# spoof cost attribution (e.g. request_tags=["team:victim-team"]).
_RESERVED_TAG_KEYS: frozenset = frozenset(
{
"env",
"service",
"host",
"pod_name",
"provider",
"model",
"model_id",
"team",
"user",
"model_group",
}
)
class DatadogCostManagementLogger(CustomBatchLogger):
def __init__(self, **kwargs):
def __init__(self, cost_tag_keys: Optional[List[str]] = None, **kwargs):
self.cost_tag_keys: List[str] = list(cost_tag_keys) if cost_tag_keys else []
self.dd_api_key = os.getenv("DD_API_KEY")
self.dd_app_key = os.getenv("DD_APP_KEY")
self.dd_site = os.getenv("DD_SITE", "datadoghq.com")
@ -68,20 +96,21 @@ class DatadogCostManagementLogger(CustomBatchLogger):
if not self.log_queue:
return
batch_to_send = self.log_queue[:]
self.log_queue = []
try:
# Aggregate costs from the batch
aggregated_entries = self._aggregate_costs(self.log_queue)
aggregated_entries = self._aggregate_costs(batch_to_send)
if not aggregated_entries:
verbose_logger.debug(
"Datadog Cost Management: batch produced no aggregable entries; "
"dropping %d log(s) from queue.",
len(batch_to_send),
)
return
# Send to Datadog
await self._upload_to_datadog(aggregated_entries)
# Clear queue only on success (or if we decide to drop on failure)
# CustomBatchLogger clears queue in flush_queue, so we just process here
except Exception as e:
self.log_queue = batch_to_send + self.log_queue
verbose_logger.exception(
f"Datadog Cost Management: Error in async_send_batch: {str(e)}"
)
@ -151,45 +180,81 @@ class DatadogCostManagementLogger(CustomBatchLogger):
return list(aggregator.values())
def _extract_tags(self, log: StandardLoggingPayload) -> Dict[str, str]:
from litellm.integrations.datadog.datadog_handler import (
get_datadog_env,
get_datadog_hostname,
get_datadog_pod_name,
get_datadog_service,
)
tags = {
tags: Dict[str, str] = {
"env": get_datadog_env(),
"service": get_datadog_service(),
"host": get_datadog_hostname(),
"pod_name": get_datadog_pod_name(),
}
# Add metadata as tags
metadata = log.get("metadata", {})
if metadata:
# Add user info
# Add user info
if metadata.get("user_api_key_alias"):
tags["user"] = str(metadata["user_api_key_alias"])
# Always-on canonical FOCUS dimensions from top-level payload fields.
# Non-sensitive and required for Datadog Custom Costs per-model attribution.
self._add_tag(tags, "provider", log.get("custom_llm_provider"))
self._add_tag(tags, "model", log.get("model"))
self._add_tag(tags, "model_id", log.get("model_id"))
# Add Team Tag
team_tag = (
metadata.get("user_api_key_team_alias")
or metadata.get("team_alias") # type: ignore
or metadata.get("user_api_key_team_id")
or metadata.get("team_id") # type: ignore
)
# cast because StandardLoggingMetadata is a TypedDict; we iterate it
# as a generic mapping below.
metadata: Dict[str, Any] = cast(Dict[str, Any], log.get("metadata") or {})
if team_tag:
tags["team"] = str(team_tag)
# model_group is not in StandardLoggingMetadata TypedDict, so we need to access it via dict.get()
model_group = metadata.get("model_group") # type: ignore[misc]
if model_group:
tags["model_group"] = str(model_group)
# Backwards-compat: team/user/model_group preserved regardless of allowlist.
if metadata.get("user_api_key_alias"):
tags["user"] = str(metadata["user_api_key_alias"])
team_tag = (
metadata.get("user_api_key_team_alias")
or metadata.get("team_alias")
or metadata.get("user_api_key_team_id")
or metadata.get("team_id")
)
if team_tag:
tags["team"] = str(team_tag)
if metadata.get("model_group"):
tags["model_group"] = str(metadata["model_group"])
# Allowlist-gated: request_tags (split on `:`) and arbitrary metadata.*.
# Reserved keys are hard-blocked here regardless of allowlist membership —
# see _RESERVED_TAG_KEYS for the rationale.
if self.cost_tag_keys:
allow = set(self.cost_tag_keys)
for rt in log.get("request_tags") or []:
if not isinstance(rt, str) or ":" not in rt:
continue
k, _, v = rt.partition(":")
if k in allow and v:
self._set_custom_tag(tags, k, v)
for k, v in metadata.items():
if k in allow and v is not None and not isinstance(v, (dict, list)):
self._set_custom_tag(tags, k, str(v))
for nested_key in ("spend_logs_metadata", "requester_metadata"):
nested = metadata.get(nested_key)
if isinstance(nested, dict):
for k, v in nested.items():
if (
k in allow
and v is not None
and not isinstance(v, (dict, list))
):
self._set_custom_tag(tags, k, str(v))
return tags
@staticmethod
def _set_custom_tag(tags: Dict[str, str], key: str, value: str) -> None:
if key in _RESERVED_TAG_KEYS:
verbose_logger.debug(
"Datadog Cost Management: dropping user-supplied tag %r=%r — "
"key is reserved for trusted cost attribution.",
key,
value,
)
return
tags[key] = value
@staticmethod
def _add_tag(tags: Dict[str, str], key: str, value: Any) -> None:
if value:
tags[key] = str(value)
async def _upload_to_datadog(self, payload: List[Dict]):
if not self.dd_api_key or not self.dd_app_key:
return
@ -201,8 +266,6 @@ class DatadogCostManagementLogger(CustomBatchLogger):
}
# The API endpoint expects a list of objects directly in the body (file content behavior)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
data_json = safe_dumps(payload)
response = await self.async_client.put(

21
litellm/integrations/datadog/datadog_metrics.py
View file

@ -144,7 +144,26 @@ class DatadogMetricsLogger(CustomBatchLogger):
}
self.log_queue.append(series_llm_latency)
# 3. Request Count / Status Code
# 3. LiteLLM Overhead Latency Metric (total - llm_api time)
hidden_params = log.get("hidden_params", {}) or {}
litellm_overhead_time_ms = hidden_params.get("litellm_overhead_time_ms")
if litellm_overhead_time_ms is not None:
overhead_tags = self._extract_tags(log) # no status_code on latency metric
series_overhead: DatadogMetricSeries = {
"metric": "litellm.overhead.latency",
"type": 3, # gauge
"points": [
{
"timestamp": timestamp,
"value": litellm_overhead_time_ms
/ 1000, # convert ms → seconds
}
],
"tags": overhead_tags,
}
self.log_queue.append(series_overhead)
# 4. Request Count / Status Code
series_count: DatadogMetricSeries = {
"metric": "litellm.llm_api.request_count",
"type": 1, # count

280
litellm/integrations/galileo.py
View file

@ -1,18 +1,29 @@
import json
import os
from typing import Any, Dict, List, Optional
import re
from typing import Any, Dict, List, Optional, Tuple, cast
from pydantic import BaseModel, Field
import litellm
from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.prompt_templates.common_utils import (
convert_content_list_to_str,
get_content_from_model_response,
)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.types.llms.openai import AllMessageValues
GALILEO_CLOUD_API_BASE_URL = "https://api.galileo.ai"
# Cap the in-memory buffer so persistent flush failures (e.g. Galileo
# unavailable, invalid credentials) cannot leak memory unboundedly.
GALILEO_MAX_IN_MEMORY_RECORDS = 1000
# from here: https://docs.rungalileo.io/galileo/gen-ai-studio-products/galileo-observe/how-to/logging-data-via-restful-apis#structuring-your-records
class LLMResponse(BaseModel):
latency_ms: int
status_code: int
@ -37,65 +48,190 @@ class GalileoObserve(CustomLogger):
def __init__(self) -> None:
self.in_memory_records: List[dict] = []
self.batch_size = 1
self.base_url = os.getenv("GALILEO_BASE_URL", None)
self.project_id = os.getenv("GALILEO_PROJECT_ID", None)
self.api_key = os.getenv("GALILEO_API_KEY")
self.project_id = os.getenv("GALILEO_PROJECT_ID")
self.log_stream_id = os.getenv("GALILEO_LOG_STREAM_ID")
self.username = os.getenv("GALILEO_USERNAME")
self.password = os.getenv("GALILEO_PASSWORD")
self.base_url = self._normalize_base_url(os.getenv("GALILEO_BASE_URL"))
if self.api_key and not self.base_url:
self.base_url = GALILEO_CLOUD_API_BASE_URL
self.use_v2_api = bool(self.api_key)
self.headers: Optional[Dict[str, str]] = None
self.async_httpx_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)
pass
def set_galileo_headers(self):
# following https://docs.rungalileo.io/galileo/gen-ai-studio-products/galileo-observe/how-to/logging-data-via-restful-apis#logging-your-records
@staticmethod
def _normalize_base_url(base_url: Optional[str]) -> Optional[str]:
if base_url:
return base_url.rstrip("/")
return None
headers = {
"accept": "application/json",
"Content-Type": "application/x-www-form-urlencoded",
}
galileo_login_response = litellm.module_level_client.post(
def _is_configured(self) -> bool:
if not self.project_id or not self.base_url:
return False
if self.use_v2_api:
return bool(self.api_key)
return bool(self.username and self.password)
async def async_set_galileo_headers(self) -> None:
galileo_login_response = await self.async_httpx_handler.post(
url=f"{self.base_url}/login",
headers=headers,
headers={
"accept": "application/json",
"Content-Type": "application/x-www-form-urlencoded",
},
data={
"username": os.getenv("GALILEO_USERNAME"),
"password": os.getenv("GALILEO_PASSWORD"),
"username": self.username,
"password": self.password,
},
)
galileo_login_response.raise_for_status()
access_token = galileo_login_response.json()["access_token"]
self.headers = {
"accept": "application/json",
"Content-Type": "application/json",
"Authorization": f"Bearer {access_token}",
}
def get_output_str_from_response(self, response_obj, kwargs):
output = None
if response_obj is not None and (
kwargs.get("call_type", None) == "embedding"
or isinstance(response_obj, litellm.EmbeddingResponse)
):
output = None
elif response_obj is not None and isinstance(
response_obj, litellm.ModelResponse
):
output = response_obj["choices"][0]["message"].json()
elif response_obj is not None and isinstance(
response_obj, litellm.TextCompletionResponse
):
output = response_obj.choices[0].text
elif response_obj is not None and isinstance(
response_obj, litellm.ImageResponse
):
output = response_obj["data"]
async def _ensure_headers(self) -> bool:
if self.headers is not None:
return True
return output
if self.use_v2_api:
if not self.api_key:
return False
self.headers = {
"accept": "application/json",
"Content-Type": "application/json",
"Galileo-API-Key": self.api_key,
}
return True
if not (self.username and self.password and self.base_url):
return False
try:
await self.async_set_galileo_headers()
return True
except Exception as e:
verbose_logger.debug("Galileo Logger: failed to authenticate: %s", e)
return False
@staticmethod
def _galileo_input_messages(
messages: Optional[List[Any]], input_text: str
) -> List[Dict[str, str]]:
if not messages:
return [{"role": "user", "content": input_text}]
galileo_messages: List[Dict[str, str]] = []
for message in messages:
if not isinstance(message, dict):
continue
role = message.get("role")
if not role:
continue
galileo_messages.append(
{
"role": str(role),
"content": convert_content_list_to_str(
message=cast(AllMessageValues, message)
),
}
)
if galileo_messages:
return galileo_messages
return [{"role": "user", "content": input_text}]
@staticmethod
def _record_to_v2_span(record: Dict[str, Any]) -> Dict[str, Any]:
created_at = record.get("created_at", "")
if created_at and not re.search(r"(Z|[+-]\d{2}:?\d{2})$", created_at):
created_at = f"{created_at}Z"
span: Dict[str, Any] = {
"type": "llm",
"name": record.get("node_type", "litellm"),
"created_at": created_at,
"input": GalileoObserve._galileo_input_messages(
record.get("messages"), record.get("input_text", "")
),
"output": {
"role": "assistant",
"content": record.get("output_text", ""),
},
"status_code": record.get("status_code", 200),
"model": record.get("model"),
"metrics": {
"duration_ns": int(record.get("latency_ms", 0)) * 1_000_000,
"num_input_tokens": record.get("num_input_tokens"),
"num_output_tokens": record.get("num_output_tokens"),
},
}
if record.get("tags"):
span["tags"] = record["tags"]
return span
def _get_ingest_request(self) -> Optional[Tuple[str, Dict[str, Any]]]:
if not self.base_url or not self.project_id:
return None
# Snapshot the records to be sent into a new list so concurrent appends
# during the network round-trip (across the await points in
# flush_in_memory_records) aren't silently dropped when we later clear
# the in-memory buffer.
records = list(self.in_memory_records)
if self.use_v2_api:
payload: Dict[str, Any] = {
"spans": [self._record_to_v2_span(record) for record in records],
"reliable": False,
}
if self.log_stream_id:
payload["log_stream_id"] = self.log_stream_id
return (
f"{self.base_url}/v2/projects/{self.project_id}/spans",
payload,
)
return (
f"{self.base_url}/projects/{self.project_id}/observe/ingest",
{"records": records},
)
def get_output_str_from_response(
self, response_obj: Any, kwargs: Dict[str, Any]
) -> Optional[str]:
if response_obj is None:
return None
if kwargs.get("call_type", None) == "embedding" or isinstance(
response_obj, litellm.EmbeddingResponse
):
return None
if isinstance(response_obj, litellm.TextCompletionResponse):
return response_obj.choices[0].text
if isinstance(response_obj, litellm.ImageResponse):
return json.dumps(response_obj["data"], default=str)
if isinstance(response_obj, (litellm.ModelResponse, dict)):
return get_content_from_model_response(response_obj)
return None
async def async_log_success_event(
self, kwargs: Any, response_obj: Any, start_time: Any, end_time: Any
):
verbose_logger.debug("On Async Success")
if not self._is_configured():
verbose_logger.debug(
"Galileo Logger: skipping flush — set GALILEO_PROJECT_ID and "
"either GALILEO_API_KEY (hosted) or GALILEO_USERNAME/GALILEO_PASSWORD "
"(enterprise Observe)."
)
return
_latency_ms = int((end_time - start_time).total_seconds() * 1000)
_call_type = kwargs.get("call_type", "litellm")
input_text = litellm.utils.get_formatted_prompt(
@ -125,26 +261,69 @@ class GalileoObserve(CustomLogger):
), # timestamp str constructed in "%Y-%m-%dT%H:%M:%S" format
)
# dump to dict
request_dict = request_record.model_dump()
messages = kwargs.get("messages")
if messages:
request_dict["messages"] = messages
self.in_memory_records.append(request_dict)
# Bound the buffer so persistent flush failures cannot grow it
# without limit. Drop the oldest records once we exceed the cap.
if len(self.in_memory_records) > GALILEO_MAX_IN_MEMORY_RECORDS:
dropped = len(self.in_memory_records) - GALILEO_MAX_IN_MEMORY_RECORDS
self.in_memory_records = self.in_memory_records[
-GALILEO_MAX_IN_MEMORY_RECORDS:
]
verbose_logger.warning(
"Galileo Logger: in-memory buffer exceeded %s records; "
"dropped %s oldest record(s). Check Galileo connectivity/credentials.",
GALILEO_MAX_IN_MEMORY_RECORDS,
dropped,
)
if len(self.in_memory_records) >= self.batch_size:
await self.flush_in_memory_records()
async def flush_in_memory_records(self):
verbose_logger.debug("flushing in memory records")
response = await self.async_httpx_handler.post(
url=f"{self.base_url}/projects/{self.project_id}/observe/ingest",
headers=self.headers,
json={"records": self.in_memory_records},
)
if not self.in_memory_records:
return
if response.status_code == 200:
# Capture the number of records that will be sent BEFORE any await so
# that concurrent appends made by other asyncio tasks during the
# network round-trip aren't silently dropped on the success-clear.
records_in_payload = len(self.in_memory_records)
ingest_request = self._get_ingest_request()
if ingest_request is None:
verbose_logger.debug(
"Galileo Logger:successfully flushed in memory records"
"Galileo Logger: missing GALILEO_BASE_URL or GALILEO_PROJECT_ID"
)
self.in_memory_records = []
return
if not await self._ensure_headers():
verbose_logger.debug("Galileo Logger: could not set request headers")
return
url, payload = ingest_request
verbose_logger.debug("flushing in memory records to %s", url)
try:
response = await self.async_httpx_handler.post(
url=url,
headers=self.headers,
json=payload,
)
except Exception as e:
verbose_logger.debug(
"Galileo Logger: failed to flush in memory records: %s", e
)
return
if response.is_success:
verbose_logger.debug(
"Galileo Logger: successfully flushed in memory records"
)
del self.in_memory_records[:records_in_payload]
else:
verbose_logger.debug("Galileo Logger: failed to flush in memory records")
verbose_logger.debug(
@ -152,6 +331,13 @@ class GalileoObserve(CustomLogger):
response.text,
response.status_code,
)
# Legacy enterprise auth caches a bearer token obtained from
# /login. If the request was rejected for auth reasons, drop the
# cached headers so the next flush re-authenticates instead of
# silently failing forever on a stale token. The v2 API key path
# uses a long-lived static key, so leave its headers in place.
if not self.use_v2_api and response.status_code in (401, 403):
self.headers = None
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
verbose_logger.debug("On Async Failure")

7
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -1,5 +1,7 @@
from typing import Optional
from litellm.llms.openai.data_residency import infer_openai_data_residency
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
_OPTIONAL_KWARGS_KEYS = frozenset(
@ -103,6 +105,10 @@ def get_litellm_params(
if litellm_trace_id is None:
litellm_trace_id = _meta.get("trace_id") or _meta.get("session_id")
data_residency: Optional[str] = infer_openai_data_residency(
custom_llm_provider, api_base
)
# Build base dict with explicit parameters (always included)
litellm_params = {
"acompletion": acompletion,
@ -112,6 +118,7 @@ def get_litellm_params(
"verbose": verbose,
"custom_llm_provider": custom_llm_provider,
"api_base": api_base,
"data_residency": data_residency,
"litellm_call_id": litellm_call_id,
"model_alias_map": model_alias_map,
"completion_call_id": completion_call_id,

30
litellm/litellm_core_utils/litellm_logging.py
View file

@ -1546,6 +1546,11 @@ class Logging(LiteLLMLoggingBaseClass):
if self.optional_params
else None
),
"data_residency": (
self.litellm_params.get("data_residency")
if hasattr(self, "litellm_params") and self.litellm_params
else None
),
}
except Exception as e: # error creating kwargs for cost calculation
debug_info = StandardLoggingModelCostFailureDebugInformation(
@ -3905,31 +3910,6 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
endpoint=arize_phoenix_config.endpoint,
headers=arize_phoenix_config.otlp_auth_headers,
)
if arize_phoenix_config.project_name:
existing_attrs = os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "")
# Add openinference.project.name attribute
if existing_attrs:
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = (
f"{existing_attrs},openinference.project.name={arize_phoenix_config.project_name}"
)
else:
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = (
f"openinference.project.name={arize_phoenix_config.project_name}"
)
# Set Phoenix project name from environment variable
phoenix_project_name = os.environ.get("PHOENIX_PROJECT_NAME", None)
if phoenix_project_name:
existing_attrs = os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "")
# Add openinference.project.name attribute
if existing_attrs:
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = (
f"{existing_attrs},openinference.project.name={phoenix_project_name}"
)
else:
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = (
f"openinference.project.name={phoenix_project_name}"
)
# auth can be disabled on local deployments of arize phoenix
if arize_phoenix_config.otlp_auth_headers is not None:

49
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -9,6 +9,7 @@ from litellm.types.utils import (
CacheCreationTokenDetails,
CallTypes,
CompletionTokensDetailsWrapper,
DataResidency,
ImageResponse,
ModelInfo,
PassthroughCallTypes,
@ -29,6 +30,9 @@ _IMAGE_RESPONSE_CALL_TYPES = frozenset(
}
)
# Pre-resolved DataResidency enum values for fast membership checks
_VALID_DATA_RESIDENCIES = frozenset(r.value for r in DataResidency)
def _is_above_128k(tokens: float) -> bool:
if tokens > 128000:
@ -617,11 +621,46 @@ def _calculate_input_cost(
return prompt_cost
def _get_regional_uplift_multiplier(
model_info: ModelInfo, data_residency: Optional[str]
) -> float:
"""
Resolve the per-model regional-processing uplift multiplier for a given
data-residency region.
OpenAI applies a flat percentage uplift (e.g. +10%) on all token costs for
requests served from a regionalized hostname (eu./us.api.openai.com). The
multiplier is stored on the model entry as
``regional_processing_uplift_multiplier_<region>`` (e.g. 1.10).
Returns 1.0 (no uplift) when ``data_residency`` is ``None`` or when the
model has no multiplier configured for the given region.
"""
if data_residency is None:
return 1.0
residency = data_residency.lower()
if residency not in _VALID_DATA_RESIDENCIES:
return 1.0
multiplier = model_info.get(f"regional_processing_uplift_multiplier_{residency}")
if multiplier is None:
return 1.0
try:
return float(cast(float, multiplier))
except (TypeError, ValueError):
verbose_logger.exception(
"Invalid regional_processing_uplift_multiplier_%s for model; "
"defaulting to 1.0",
residency,
)
return 1.0
def generic_cost_per_token( # noqa: PLR0915
model: str,
usage: Usage,
custom_llm_provider: str,
service_tier: Optional[str] = None,
data_residency: Optional[str] = None,
) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -631,6 +670,8 @@ def generic_cost_per_token( # noqa: PLR0915
Input:
- model: str, the model name without provider prefix
- usage: LiteLLM Usage block, containing anthropic caching information
- data_residency: optional OpenAI data-residency region (e.g. "eu", "us"),
used to apply the per-model regional-processing uplift multiplier.
Returns:
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
@ -781,6 +822,14 @@ def generic_cost_per_token( # noqa: PLR0915
)
completion_cost += float(image_tokens) * _output_cost_per_image_token
## REGIONAL DATA-RESIDENCY UPLIFT
# Applied as a flat multiplier across all token costs for the request
# when the upstream is a regionalized OpenAI host (eu./us.api.openai.com).
uplift = _get_regional_uplift_multiplier(model_info, data_residency)
if uplift != 1.0:
prompt_cost *= uplift
completion_cost *= uplift
return prompt_cost, completion_cost

14
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -3997,7 +3997,7 @@ def _convert_to_bedrock_tool_call_invoke(
for tool in tool_calls:
if "function" in tool:
tool_id = tool["id"]
name = tool["function"].get("name", "")
name = make_valid_bedrock_tool_name(tool["function"].get("name", ""))
arguments = tool["function"].get("arguments", "")
if not arguments or not arguments.strip():
@ -5323,16 +5323,10 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
def make_valid_bedrock_tool_name(input_tool_name: str) -> str:
"""
Replaces any invalid characters in the input tool name with underscores
and ensures the resulting string is a valid identifier for Bedrock tools
"""
"""Normalize tool names to Bedrock pattern [a-zA-Z][a-zA-Z0-9_-]*."""
def replace_invalid(char):
"""
Bedrock tool names only supports alpha-numeric characters and underscores
"""
if char.isalnum() or char == "_":
if char.isalnum() or char in ("_", "-"):
return char
return "_"
@ -5492,7 +5486,7 @@ def _bedrock_tools_pt(
raw_name = f"litellm_unnamed_tool_{tool_idx}"
# related issue: https://github.com/BerriAI/litellm/issues/5007
# Bedrock tool names must satisfy regular expression pattern: [a-zA-Z][a-zA-Z0-9_]* ensure this is true
# Bedrock tool names must satisfy pattern: [a-zA-Z][a-zA-Z0-9_-]*
name = make_valid_bedrock_tool_name(input_tool_name=raw_name)
if _tool_description: # bedrock doesn't accept empty "" or None descriptions
description = _tool_description

298
litellm/litellm_core_utils/realtime_streaming.py
View file

@ -86,6 +86,12 @@ class RealTimeStreaming:
# When a text message is blocked, hold the guardrail reason so the next
# response.create can be rewritten to include the failure context.
self._pending_guardrail_message: Optional[str] = None
# Track whether session.created has already been sent to the client
# (e.g. synthetic event in deferred setup mode).
self._session_created_sent_to_client: bool = False
# Track whether we have already sent the guardrail turn-detection update
# that disables provider auto-response for transcription guardrails.
self._guardrail_turn_detection_update_sent: bool = False
_SESSION_EVENT_TYPES = frozenset(["session.created", "session.updated"])
_AUDIO_FORMAT_MAP: Dict[str, Dict[str, Any]] = {
@ -248,40 +254,82 @@ class RealTimeStreaming:
## SYNC LOGGING
executor.submit(self.logging_obj.success_handler(self.messages))
async def _send_to_backend(self, message: str) -> None:
async def _send_to_backend(self, message: str) -> bool:
"""Send a message to the backend WebSocket.
If a provider_config is set the message is first passed through
transform_realtime_request so that provider-specific translation
(e.g. dropping session.update for Vertex AI) is applied even for
guardrail-injected messages.
Returns True if at least one message was actually delivered to the
backend, False if the provider transformation produced no output and
the message was effectively dropped.
"""
if self.provider_config:
transformed = self.provider_config.transform_realtime_request(
message, self.model, self.session_configuration_request
)
sent = False
for msg in transformed:
# Send first; only cache the setup payload once the backend
# has actually accepted it. Caching before send would leave
# ``session_configuration_request`` populated after a failed
# send, causing subsequent client session.update messages to
# be treated as "subsequent" and dropped even though the
# backend never received the original setup.
await self.backend_ws.send(msg) # type: ignore[union-attr, attr-defined]
else:
await self.backend_ws.send(message) # type: ignore[union-attr, attr-defined]
self._cache_session_configuration_request(msg)
sent = True
return sent
await self.backend_ws.send(message) # type: ignore[union-attr, attr-defined]
return True
def _cache_session_configuration_request(self, transformed_message: str) -> None:
"""Store setup payload once sent to backend.
Updates the cached setup on every successful setup send so follow-up
``session.update`` messages (which produce a merged setup with new
``generationConfig`` / ``systemInstruction`` / etc.) are reflected in
the cache used by downstream readers (``transform_session_created_event``,
``return_new_content_delta_events`` modality lookup, ...).
"""
try:
message_obj = json.loads(transformed_message)
if "setup" in message_obj:
self.session_configuration_request = transformed_message
except (json.JSONDecodeError, TypeError):
return
def _make_disable_auto_response_message(self) -> str:
"""Return a session.update that disables VAD auto-response."""
turn_detection: Dict[str, Any] = {
"type": "server_vad",
"create_response": False,
}
if self._backend_uses_beta_protocol:
session: Dict[str, Any] = {
"turn_detection": {"create_response": False},
}
session: Dict[str, Any] = {"turn_detection": turn_detection}
else:
session = {
"type": "realtime",
"audio": {
"input": {
"turn_detection": {"create_response": False},
}
},
"audio": {"input": {"turn_detection": turn_detection}},
}
return json.dumps({"type": "session.update", "session": session})
async def _maybe_send_guardrail_turn_detection_update(self) -> None:
"""Disable provider auto-response once when transcription guardrails are enabled."""
if self._guardrail_turn_detection_update_sent:
return
if not self._has_audio_transcription_guardrails():
return
sent = await self._send_to_backend(self._make_disable_auto_response_message())
# Only mark as sent when the provider transformation actually delivered
# the update to the backend. Otherwise (e.g. Gemini drops session.update
# after the initial setup), leave the flag unset so future opportunities
# — such as a duplicate session.created — can retry.
if sent:
self._guardrail_turn_detection_update_sent = True
def _has_realtime_guardrails(self) -> bool:
"""Return True if any callback is registered for realtime guardrail event types."""
from litellm.integrations.custom_guardrail import CustomGuardrail
@ -320,12 +368,20 @@ class RealTimeStreaming:
self,
transcript: str,
item_id: Optional[str] = None,
pre_block_backend_message: Optional[str] = None,
) -> bool:
"""
Run registered guardrails on a completed speech transcription.
Returns True if blocked (synthetic warning already sent to client).
Returns False if clean (caller should send response.create to the backend).
``pre_block_backend_message`` (if provided) is sent to the backend
BEFORE any of the guardrail's own backend messages when a block is
triggered. This is needed for protocol contracts that require a
specific message to be sent first — e.g. Gemini Live requires a
matching ``toolResponse`` immediately after a ``toolCall`` before any
other client messages can be accepted.
"""
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.guardrails import GuardrailEventHooks
@ -385,6 +441,13 @@ class RealTimeStreaming:
getattr(callback, "realtime_violation_message", None) or safe_msg
)
# Deliver any caller-supplied backend message FIRST so that
# protocol contracts requiring a specific ordering (e.g.
# Gemini Live's mandatory ``toolResponse`` after a
# ``toolCall``) are honored before the guardrail's own
# clientContent / cancel messages are sent.
if pre_block_backend_message is not None:
await self._send_to_backend(pre_block_backend_message)
# Cancel any in-progress LLM response (e.g. VAD auto-response).
await self._send_to_backend(json.dumps({"type": "response.cancel"}))
# Send the policy violation hint (shows as small gray status text in UI).
@ -480,16 +543,34 @@ class RealTimeStreaming:
else [transformed_response]
)
for event in events:
is_session_created_event = (
isinstance(event, dict) and event.get("type") == "session.created"
)
if is_session_created_event:
if self._session_created_sent_to_client:
# A synthetic session.created (with placeholder defaults) was
# already forwarded to the client when we connected. The
# provider's real session.created (e.g. emitted from Gemini
# `setupComplete`) carries the authoritative modalities/model
# from the client's session.update. Re-emit it as
# `session.updated` so the client learns the corrected
# configuration without seeing two `session.created` events.
event = {**event, "type": "session.updated"}
else:
self._session_created_sent_to_client = True
event_str = json.dumps(event)
## For audio/VAD guardrail path: forward session.created first, then inject.
if (
isinstance(event, dict)
and event.get("type") == "session.created"
and self._has_audio_transcription_guardrails()
):
## For audio/VAD guardrail path: forward the (possibly retyped)
## session.created first, then invoke the one-time guardrail
## turn-detection update. ``_maybe_send_guardrail_turn_detection_update``
## is idempotent (gated by ``_guardrail_turn_detection_update_sent``),
## so duplicate session.created events — including those emitted
## after a synthetic session.created from ``llm_http_handler`` in
## deferred-setup mode — still get a single chance to inject the
## update if a prior attempt was dropped by the provider transform.
if is_session_created_event and self._has_audio_transcription_guardrails():
self.store_message(event_str)
await self.websocket.send_text(event_str)
await self._send_to_backend(self._make_disable_auto_response_message())
await self._maybe_send_guardrail_turn_detection_update()
continue
## GUARDRAIL: run on transcription events in provider_config path too
if (
@ -564,10 +645,19 @@ class RealTimeStreaming:
try:
raw_response = await self.backend_ws.recv( # type: ignore[union-attr]
decode=False
) # improves performance
)
except TypeError:
raw_response = await self.backend_ws.recv() # type: ignore[union-attr, assignment]
if isinstance(raw_response, bytes):
try:
raw_response = raw_response.decode("utf-8")
except UnicodeDecodeError:
verbose_logger.warning(
"Received non-UTF-8 binary frame from backend, skipping."
)
continue
if self.provider_config:
try:
await self._handle_provider_config_message(raw_response)
@ -783,12 +873,13 @@ class RealTimeStreaming:
item["content"] = new_content
return item
async def client_ack_messages(self):
async def client_ack_messages(self): # noqa: PLR0915
try:
while True:
message = await self.websocket.receive_text()
## GUARDRAIL: intercept conversation.item.create for text-based injection.
guardrail_turn_detection_injected = False
try:
msg_obj = json.loads(message)
msg_type = msg_obj.get("type")
@ -796,7 +887,68 @@ class RealTimeStreaming:
if msg_type == "conversation.item.create":
# Check user text messages for prompt injection
item = msg_obj.get("item", {})
if item.get("role") == "user":
# Check function_call_output first so a client cannot
# bypass the tool-result guardrail by also setting
# role="user" on a function_call_output item.
if item.get("type") == "function_call_output":
# Tool results are client-controlled and fed to the
# model; check them with the same guardrail used for
# user text so an attacker cannot smuggle blocked
# content into a function_call_output.
output = item.get("output", "")
output_text = (
output
if isinstance(output, str)
else json.dumps(output)
)
if output_text:
# Build the sanitized function_call_output up
# front so we can hand it to the guardrail
# runner as the pre-block message. Providers
# that pair every toolCall with a toolResponse
# (e.g. Gemini/Vertex Live) require the
# toolResponse to arrive BEFORE any other
# client message — otherwise the guardrail's
# own clientContent would violate the
# pending-tool-call protocol contract and the
# backend could close the connection before
# the sanitized response ever lands. Dropping
# the blocked item outright would similarly
# leave such providers waiting indefinitely.
# The sanitized payload carries no blocked
# content — only a generic policy marker.
sanitized_msg = json.dumps(
{
**msg_obj,
"item": {
**item,
"output": json.dumps(
{
"error": "Tool output blocked by content policy",
}
),
},
}
)
blocked = await self.run_realtime_guardrails(
output_text,
pre_block_backend_message=sanitized_msg,
)
if blocked:
# ``_pending_guardrail_message`` is
# intentionally NOT set here. That flag
# exists to swallow the reflexive
# ``response.create`` an OpenAI client
# sends immediately after a user text
# message. In a tool-calling flow the
# client may not send a ``response.create``
# at all (e.g. Gemini SDKs auto-respond),
# so leaving the flag set would
# incorrectly drop an unrelated
# ``response.create`` from a later
# interaction turn.
continue
elif item.get("role") == "user":
content_list = item.get("content", [])
texts = [
c.get("text", "")
@ -824,6 +976,89 @@ class RealTimeStreaming:
self._pending_guardrail_message = None
continue
## GUARDRAIL: Inject turn_detection into first session.update
# if needed. Done BEFORE the GA remap so the injected
# ``create_response`` rides along with any client-provided
# turn_detection fields (e.g. silence_duration_ms) into the
# nested ``audio.input.turn_detection`` path produced by the
# remap. Doing this after the remap would create a separate
# minimal root-level ``turn_detection`` and silently drop
# the client's nested settings.
if (
msg_type == "session.update"
and self.session_configuration_request is None
and not self._guardrail_turn_detection_update_sent
and self._has_audio_transcription_guardrails()
):
session = msg_obj.setdefault("session", {})
if isinstance(session, dict):
existing_td = session.get("turn_detection")
if not isinstance(existing_td, dict):
existing_td = {}
existing_td["create_response"] = False
session["turn_detection"] = existing_td
message = json.dumps(msg_obj)
guardrail_turn_detection_injected = True
verbose_logger.debug(
"Injected turn_detection into first session.update for audio transcription guardrails"
)
## GUARDRAIL: Force ``create_response`` to False in any
# client-provided ``turn_detection`` so a later
# ``session.update`` cannot re-enable VAD auto-response
# and bypass the transcription guardrail after the
# initial disable. Covers both the flat beta key and the
# nested GA ``audio.input.turn_detection`` shape, since
# the GA remap below also accepts either form. Skipped
# when the injection block above already ran for this
# message, to avoid redundant double-serialization.
if (
msg_type == "session.update"
and not guardrail_turn_detection_injected
and self._has_audio_transcription_guardrails()
):
session = msg_obj.get("session")
if isinstance(session, dict):
td_overridden = False
flat_td = session.get("turn_detection")
flat_td_present = flat_td is not None
if flat_td_present:
if not isinstance(flat_td, dict):
flat_td = {}
if flat_td.get("create_response") is not False:
flat_td["create_response"] = False
session["turn_detection"] = flat_td
td_overridden = True
nested_td_present = False
audio = session.get("audio")
if isinstance(audio, dict):
audio_input = audio.get("input")
if isinstance(audio_input, dict):
nested_td = audio_input.get("turn_detection")
if nested_td is not None:
nested_td_present = True
if not isinstance(nested_td, dict):
nested_td = {}
if (
nested_td.get("create_response")
is not False
):
nested_td["create_response"] = False
audio_input["turn_detection"] = nested_td
td_overridden = True
# Symmetric with the first-update injection block:
# if the client omitted turn_detection entirely on
# a subsequent session.update, still inject the
# ``create_response: False`` override so the
# transcription guardrail cannot be re-enabled by
# any downstream merge that drops the original
# disable.
if not flat_td_present and not nested_td_present:
session["turn_detection"] = {"create_response": False}
td_overridden = True
if td_overridden:
message = json.dumps(msg_obj)
# GA compatibility: remap beta-style session fields only when
# the upstream is in GA mode. Beta upstreams expect the flat
# session shape unchanged.
@ -841,17 +1076,20 @@ class RealTimeStreaming:
pass
## LOGGING
# Log after any in-place modifications (GA remap, guardrail
# turn_detection injection) so audit logs reflect what we
# actually forward to the backend.
self.store_input(message=message)
## FORWARD TO BACKEND
if self.provider_config:
message = self.provider_config.transform_realtime_request(
message, self.model
)
for msg in message:
await self.backend_ws.send(msg) # type: ignore[union-attr]
else:
await self.backend_ws.send(message) # type: ignore[union-attr]
## FORWARD TO BACKEND
# Only mark the guardrail turn_detection update as sent after the
# backend actually accepted the message. Setting the flag earlier
# would permanently disable the injection if ``_send_to_backend``
# raised — neither this loop nor
# ``_maybe_send_guardrail_turn_detection_update`` would retry.
sent = await self._send_to_backend(message)
if guardrail_turn_detection_injected and sent:
self._guardrail_turn_detection_update_sent = True
except Exception as e:
verbose_logger.debug(f"Error in client ack messages: {e}")

31
litellm/litellm_core_utils/sensitive_data_masker.py
View file

@ -146,6 +146,37 @@ class SensitiveDataMasker:
return masked_data
_default_masker = SensitiveDataMasker()
def mask_sensitive_keys(
data: Dict[str, Any], sensitive_fields: Set[str]
) -> Dict[str, Any]:
"""Return a new dict with values masked for keys listed in ``sensitive_fields``.
Unlike :meth:`SensitiveDataMasker.mask_dict`, this does exact key-name
matching (not segment matching), so callers explicitly enumerate which
fields to mask. Non-string and None values are passed through unchanged.
Values shorter than ``visible_prefix + visible_suffix`` (8 by default)
fall outside :meth:`SensitiveDataMasker._mask_value`'s partial-reveal
range and are replaced with a fixed-length all-mask string, so a short
credential is never returned verbatim.
"""
masked: Dict[str, Any] = {}
mask_char = _default_masker.mask_char
min_visible = _default_masker.visible_prefix + _default_masker.visible_suffix
for key, value in data.items():
if value is not None and key in sensitive_fields and isinstance(value, str):
if len(value) < min_visible:
masked[key] = mask_char * len(value) if value else value
else:
masked[key] = _default_masker._mask_value(value)
else:
masked[key] = value
return masked
# Usage example:
"""
masker = SensitiveDataMasker()

124
litellm/litellm_core_utils/streaming_handler.py
View file

@ -59,6 +59,8 @@ FUNCTION_CALL_ATTRIBUTE = "function_call"
_SYNC_ITER_EXHAUSTED = object()
_GCHUNK_FIELDS: frozenset = frozenset(GChunk.__annotations__)
def _next_sync_or_exhausted(it: Any) -> Any:
"""
@ -181,6 +183,30 @@ class CustomStreamWrapper:
self.created: Optional[int] = None
self._last_returned_hidden_params: Optional[dict] = None
_cached_logging_provider = self.logging_obj.model_call_details.get(
"custom_llm_provider", None
)
self._cached_logging_llm_provider: Optional[str] = _cached_logging_provider
_effective_model = model or ""
if (
custom_llm_provider == "openai"
and custom_llm_provider != _cached_logging_provider
):
_effective_model = "{}/{}".format(
_cached_logging_provider, _effective_model
)
self._cached_model_name: str = _effective_model
# Snapshot assumes self._hidden_params is populated from litellm_params
# at init and never mutated during the stream. If that ever changes,
# this cache must be removed.
self._base_hidden_params: Dict[str, Any] = {
**self._hidden_params,
"response_cost": None,
}
self._post_streaming_hooks: Optional[List] = None
def _check_max_streaming_duration(self) -> None:
"""Raise litellm.Timeout if the stream has exceeded LITELLM_MAX_STREAMING_DURATION_SECONDS."""
from litellm.constants import LITELLM_MAX_STREAMING_DURATION_SECONDS
@ -681,29 +707,16 @@ class CustomStreamWrapper:
def model_response_creator(
self, chunk: Optional[dict] = None, hidden_params: Optional[dict] = None
):
_model = self.model
_received_llm_provider = self.custom_llm_provider
_logging_obj_llm_provider = self.logging_obj.model_call_details.get("custom_llm_provider", None) # type: ignore
if (
_received_llm_provider == "openai"
and _received_llm_provider != _logging_obj_llm_provider
):
_model = "{}/{}".format(_logging_obj_llm_provider, _model)
_model = self._cached_model_name
_logging_obj_llm_provider = self._cached_logging_llm_provider
if chunk is None:
chunk = {}
args: Dict[str, Any] = {"model": _model}
else:
# pop model keyword
chunk.pop("model", None)
chunk_dict = {}
for key, value in chunk.items():
if key != "stream":
chunk_dict[key] = value
args = {
"model": _model,
**chunk_dict,
}
args = {"model": _model}
if chunk:
args.update({k: v for k, v in chunk.items() if k != "stream"})
model_response = ModelResponseStream(**args)
if self.response_id is not None:
@ -717,15 +730,23 @@ class CustomStreamWrapper:
model_response.created = self.created
else:
self.created = model_response.created
# Spread order is load-bearing: _base_hidden_params (model_id, api_base, ...)
# must win over both caller-supplied hidden_params and the computed
# custom_llm_provider/created_at values, so it comes last.
if hidden_params is not None:
model_response._hidden_params = hidden_params
model_response._hidden_params["custom_llm_provider"] = _logging_obj_llm_provider
model_response._hidden_params["created_at"] = time.time()
model_response._hidden_params = {
**model_response._hidden_params,
**self._hidden_params,
"response_cost": None,
}
model_response._hidden_params = {
**hidden_params,
"custom_llm_provider": _logging_obj_llm_provider,
"created_at": time.time(),
**self._base_hidden_params,
}
else:
model_response._hidden_params = {
"custom_llm_provider": _logging_obj_llm_provider,
"created_at": time.time(),
**self._base_hidden_params,
}
if (
len(model_response.choices) > 0
@ -1627,7 +1648,17 @@ class CustomStreamWrapper:
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import CallTypes
# Get request kwargs from logging object
if self._post_streaming_hooks is None:
self._post_streaming_hooks = [
cb
for cb in litellm.callbacks
if isinstance(cb, CustomLogger)
and hasattr(cb, "async_post_call_streaming_deployment_hook")
]
if not self._post_streaming_hooks:
return chunk
request_data = self.logging_obj.model_call_details
call_type_str = self.logging_obj.call_type
@ -1636,18 +1667,14 @@ class CustomStreamWrapper:
except ValueError:
typed_call_type = None
# Call hooks for all callbacks
for callback in litellm.callbacks:
if isinstance(callback, CustomLogger) and hasattr(
callback, "async_post_call_streaming_deployment_hook"
):
result = await callback.async_post_call_streaming_deployment_hook(
request_data=request_data,
response_chunk=chunk,
call_type=typed_call_type,
)
if result is not None:
chunk = result
for callback in self._post_streaming_hooks:
result = await callback.async_post_call_streaming_deployment_hook(
request_data=request_data,
response_chunk=chunk,
call_type=typed_call_type,
)
if result is not None:
chunk = result
return chunk
except Exception as e:
@ -1888,17 +1915,15 @@ class CustomStreamWrapper:
response = self._add_mcp_list_tools_to_first_chunk(response)
self.sent_first_chunk = True
if hasattr(
response, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
# ModelResponseStream declares `usage` as a field, so
# hasattr(response, "usage") is always True — must check
# `is not None` to avoid running this path on every chunk.
if getattr(response, "usage", None) is not None:
obj_dict = response.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
del obj_dict["usage"]
# Create a new object without the removed attribute
response = self.model_response_creator(
chunk=obj_dict, hidden_params=response._hidden_params
)
@ -2398,10 +2423,7 @@ def generic_chunk_has_all_required_fields(chunk: dict) -> bool:
:param chunk: The dictionary to check.
:return: True if all required fields are present, False otherwise.
"""
_all_fields = GChunk.__annotations__
decision = all(key in _all_fields for key in chunk)
return decision
return all(key in _GCHUNK_FIELDS for key in chunk)
def convert_generic_chunk_to_model_response_stream(

29
litellm/llms/anthropic/chat/transformation.py
View file

@ -337,13 +337,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
@staticmethod
def _supports_effort_level(model: str, level: str) -> bool:
"""Check ``supports_{level}_reasoning_effort`` in the model map.
def _supports_model_capability(model: str, key: str) -> bool:
"""Check a boolean capability ``key`` in the model map.
Strips bedrock/vertex prefixes so a provider-routed Claude still
resolves to the Anthropic model-map entry.
"""
key = f"supports_{level}_reasoning_effort"
try:
if _supports_factory(
model=model,
@ -372,8 +371,6 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
except Exception:
pass
try:
import litellm
for cand in candidates:
if cand in litellm.model_cost and (
litellm.model_cost[cand].get(key) is True
@ -383,6 +380,13 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
pass
return False
@staticmethod
def _supports_effort_level(model: str, level: str) -> bool:
"""Check ``supports_{level}_reasoning_effort`` in the model map."""
return AnthropicConfig._supports_model_capability(
model, f"supports_{level}_reasoning_effort"
)
@staticmethod
def _validate_effort_for_model(model: str, effort: Optional[str]) -> Optional[str]:
"""Return ``None`` if ``effort`` is allowed on ``model``, else an error message."""
@ -400,7 +404,15 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
@staticmethod
def _model_supports_effort_param(model: str) -> bool:
"""Whether the model accepts ``output_config.effort`` at all."""
"""Whether the model accepts ``output_config.effort`` at all.
A model qualifies if its map entry advertises ``supports_output_config``
or any ``supports_*_reasoning_effort`` flag. The two are independent
signals: e.g. Claude Opus 4.5 supports ``output_config`` without
advertising a non-default (max/xhigh) effort level.
"""
if AnthropicConfig._supports_model_capability(model, "supports_output_config"):
return True
return any(
AnthropicConfig._supports_effort_level(model, level)
for level in ("low", "minimal", "medium", "high", "xhigh", "max")
@ -1793,7 +1805,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
self._ensure_context_management_beta_header(
headers, optional_params["context_management"]
)
if optional_params.get("output_format") is not None:
output_config = optional_params.get("output_config")
if optional_params.get("output_format") is not None or (
isinstance(output_config, dict) and output_config.get("format") is not None
):
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value
)

9
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -427,8 +427,13 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
)
# Check for structured outputs
if optional_params.get("output_format") is not None:
# Check for structured outputs. Anthropic's newer request shape nests
# the schema under output_config.format; the older top-level
# output_format remains supported for backwards compatibility.
output_config = optional_params.get("output_config")
if optional_params.get("output_format") is not None or (
isinstance(output_config, dict) and output_config.get("format") is not None
):
beta_values.add(
ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value
)

46
litellm/llms/azure/common_utils.py
View file

@ -1,3 +1,5 @@
import asyncio
import hashlib
import json
import os
from typing import Any, Callable, Dict, Literal, NamedTuple, Optional, Union, cast
@ -449,6 +451,25 @@ class BaseAzureLLM(BaseOpenAILLM):
] = None
client_initialization_params: dict = locals()
client_initialization_params["is_async"] = _is_async
_lp = litellm_params or {}
_ad_provider = _lp.get("azure_ad_token_provider")
_ad_token = _lp.get("azure_ad_token")
_client_secret = _lp.get("client_secret")
_azure_password = _lp.get("azure_password")
client_initialization_params["azure_ad_token"] = (
hashlib.sha256(_ad_token.encode()).hexdigest()
if isinstance(_ad_token, str)
else None
)
client_initialization_params["azure_ad_token_provider"] = (
f"provider_id={id(_ad_provider) if callable(_ad_provider) else None}"
f"|tenant_id={_lp.get('tenant_id')}"
f"|client_id={_lp.get('client_id')}"
f"|client_secret={hashlib.sha256(_client_secret.encode()).hexdigest() if isinstance(_client_secret, str) else None}"
f"|azure_username={_lp.get('azure_username')}"
f"|azure_password={hashlib.sha256(_azure_password.encode()).hexdigest() if isinstance(_azure_password, str) else None}"
f"|azure_scope={_lp.get('azure_scope')}"
)
if client is None:
cached_client = self.get_cached_openai_client(
client_initialization_params=client_initialization_params,
@ -474,8 +495,29 @@ class BaseAzureLLM(BaseOpenAILLM):
if self._is_azure_v1_api_version(api_version):
# Extract only params that OpenAI client accepts
# Always use /openai/v1/ regardless of whether user passed "v1", "latest", or "preview"
v1_params = {
"api_key": azure_client_params.get("api_key"),
# The OpenAI client accepts a callable for `api_key` and re-invokes it
# on every request (via `_refresh_api_key`), so passing
# `azure_ad_token_provider` directly preserves Azure AD token refresh
# behavior that the regular AzureOpenAI client provides.
v1_api_key: Optional[Union[str, Callable[[], Any]]] = (
azure_client_params.get("api_key")
or azure_client_params.get("azure_ad_token_provider")
or azure_client_params.get("azure_ad_token")
)
if _is_async is True and callable(v1_api_key):
# AsyncOpenAI expects an async provider; wrap the sync provider
# returned by azure-identity. Offload to a thread so a token
# refresh (blocking HTTP call to AAD on cache miss) does not
# stall the event loop.
_sync_provider = v1_api_key
async def _async_v1_api_key() -> str:
return await asyncio.to_thread(_sync_provider)
v1_api_key = _async_v1_api_key
v1_params: Dict[str, Any] = {
"api_key": v1_api_key,
"base_url": f"{api_base}/openai/v1/",
}
if "timeout" in azure_client_params:

7
litellm/llms/base_llm/chat/transformation.py
View file

@ -108,10 +108,9 @@ class BaseConfig(ABC):
return type_to_response_format_param(response_format=response_format)
def is_thinking_enabled(self, non_default_params: dict) -> bool:
return (
non_default_params.get("thinking", {}).get("type") == "enabled"
or non_default_params.get("reasoning_effort") is not None
)
return (non_default_params.get("thinking") or {}).get(
"type"
) == "enabled" or non_default_params.get("reasoning_effort") is not None
def is_max_tokens_in_request(self, non_default_params: dict) -> bool:
"""

10
litellm/llms/base_llm/managed_resources/utils.py
View file

@ -177,8 +177,14 @@ def extract_model_id_from_unified_id(
if decoded_id:
unified_id = decoded_id
# Extract model ID
match = re.search(r"model_id,([^;]+)", unified_id)
# Extract model ID. Anchor to a field boundary (start of string or
# after `;`) so this regex doesn't substring-match the `model_id,`
# inside file_id encodings' `llm_output_file_model_id,<deployment_uuid>`
# field — that would feed the deployment UUID as a model candidate
# into the team-access check and 403 every team-BYOK file attach
# with `Tried to access <uuid>` (LIT-3244 patch/1.86.0 second-order
# finding).
match = re.search(r"(?:^|;)model_id,([^;]+)", unified_id)
if match:
return match.group(1).strip()

15
litellm/llms/base_llm/realtime/transformation.py
View file

@ -3,6 +3,7 @@ from typing import TYPE_CHECKING, Any, List, Optional, Union
import httpx
from litellm.types.llms.openai import OpenAIRealtimeStreamSessionEvents
from litellm.types.realtime import (
RealtimeResponseTransformInput,
RealtimeResponseTypedDict,
@ -69,6 +70,20 @@ class BaseRealtimeConfig(ABC):
) -> Optional[str]: # message sent to setup the realtime session
return None
def transform_session_created_event(
self,
model: str,
logging_session_id: str,
session_configuration_request: Optional[str] = None,
) -> Optional[Union[dict, OpenAIRealtimeStreamSessionEvents]]:
"""
Optional hook for providers that defer session setup until client `session.update`.
Return an OpenAI-compatible `session.created` payload when the proxy should
emit a synthetic event immediately after backend websocket connection.
"""
return None
@abstractmethod
def transform_realtime_response(
self,

19
litellm/llms/base_llm/videos/transformation.py
View file

@ -321,6 +321,23 @@ class BaseVideoConfig(ABC):
"video get character is not supported for this provider"
)
def get_video_edit_prefetch_params(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Optional[Tuple[str, Dict]]:
"""
Return (url, body) for a pre-fetch HTTP call that must be made before
transform_video_edit_request, or None if no pre-fetch is required.
Providers that need to retrieve the source video before constructing the
edit request (e.g. Vertex AI) should override this method. The handler
uses the existing shared httpx client so the call is properly async.
"""
return None
def transform_video_edit_request(
self,
prompt: str,
@ -329,6 +346,7 @@ class BaseVideoConfig(ABC):
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: Optional[Dict[str, Any]] = None,
prefetched_source_data: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Transform the video edit request into a URL and JSON data.
@ -343,6 +361,7 @@ class BaseVideoConfig(ABC):
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
raise NotImplementedError("video edit is not supported for this provider")

6
litellm/llms/bedrock/chat/agentcore/transformation.py
View file

@ -157,8 +157,8 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
def _get_agent_runtime_arn(self, model: str) -> str:
"""
Extract ARN from model string
model = "agentcore/arn:aws:bedrock-agentcore:us-west-2:888602223428:runtime/hosted_agent_r9jvp-3ySZuRHjLC"
returns: "arn:aws:bedrock-agentcore:us-west-2:888602223428:runtime/hosted_agent_r9jvp-3ySZuRHjLC"
model = "agentcore/arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp"
returns: "arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp"
"""
parts = model.split("/", 1)
if len(parts) != 2 or parts[0] != "agentcore":
@ -170,7 +170,7 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
def _extract_region_from_arn(self, arn: str) -> str:
"""
Extract region from ARN
arn:aws:bedrock-agentcore:us-west-2:888602223428:runtime/hosted_agent_r9jvp-3ySZuRHjLC
arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp
returns: us-west-2
"""
parts = arn.split(":")

65
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -30,6 +30,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
BedrockConverseMessagesProcessor,
_bedrock_converse_messages_pt,
_bedrock_tools_pt,
make_valid_bedrock_tool_name,
)
from litellm.llms.anthropic.chat.transformation import (
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
@ -77,6 +78,7 @@ from ..common_utils import (
get_anthropic_beta_from_headers,
get_bedrock_tool_name,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
)
# Computer use tool prefixes supported by Bedrock
@ -447,10 +449,20 @@ class AmazonConverseConfig(BaseConfig):
value=reasoning_effort,
llm_provider="bedrock_converse",
)
existing_output_config = optional_params.get("output_config")
if not isinstance(existing_output_config, dict):
existing_output_config = {}
existing_output_config.setdefault("effort", mapped_effort)
normalize_bedrock_opus_output_config_effort(
model=model,
output_config=existing_output_config,
)
mapped_effort = existing_output_config["effort"]
self._validate_anthropic_adaptive_effort(
model=model, effort=mapped_effort
)
optional_params["output_config"] = {"effort": mapped_effort}
optional_params["output_config"] = existing_output_config
optional_params["_output_config_normalized"] = True
@staticmethod
def _validate_anthropic_adaptive_effort(model: str, effort: str) -> None:
@ -595,7 +607,9 @@ class AmazonConverseConfig(BaseConfig):
elif isinstance(tool_choice, dict):
# only supported for anthropic + mistral models - https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ToolChoice.html
specific_tool = SpecificToolChoiceBlock(
name=tool_choice.get("function", {}).get("name", "")
name=make_valid_bedrock_tool_name(
tool_choice.get("function", {}).get("name", "")
)
)
return ToolChoiceValuesBlock(tool=specific_tool)
else:
@ -1198,6 +1212,12 @@ class AmazonConverseConfig(BaseConfig):
self, optional_params: dict, model: str
) -> Tuple[dict, dict, dict, Optional[OutputConfigBlock]]:
"""Prepare and separate request parameters."""
# Consume the internal ``_output_config_normalized`` marker set by
# ``_handle_reasoning_effort_parameter`` so it does not linger on the
# caller's ``optional_params`` after the transformation returns.
anthropic_output_config_already_normalized = bool(
optional_params.pop("_output_config_normalized", False)
)
# Filter out exception objects before deepcopy to prevent deepcopy failures
# Exceptions should not be stored in optional_params (this is a defensive fix)
cleaned_params = filter_exceptions_from_params(optional_params)
@ -1216,8 +1236,17 @@ class AmazonConverseConfig(BaseConfig):
# Anthropic-only ``output_config`` (snake_case) — re-attached to
# ``additionalModelRequestFields`` for Anthropic models below. The
# Bedrock-native ``outputConfig`` (camelCase) is handled separately.
# structured-output ``format`` subfield is consumed into Bedrock's
# native ``outputConfig`` (camelCase), which is handled separately.
anthropic_output_config = inference_params.pop("output_config", None)
output_config_format = None
if isinstance(anthropic_output_config, dict):
anthropic_output_config = dict(anthropic_output_config)
candidate_output_config_format = anthropic_output_config.pop("format", None)
if isinstance(candidate_output_config_format, dict):
output_config_format = candidate_output_config_format
if not anthropic_output_config:
anthropic_output_config = None
# Extract requestMetadata before processing other parameters
request_metadata = inference_params.pop("requestMetadata", None)
@ -1227,6 +1256,30 @@ class AmazonConverseConfig(BaseConfig):
output_config: Optional[OutputConfigBlock] = inference_params.pop(
"outputConfig", None
)
base_model = BedrockModelInfo.get_base_model(model)
if (
output_config is None
and output_config_format is not None
and output_config_format.get("type") == "json_schema"
and base_model.startswith("anthropic")
and self._supports_native_structured_outputs(
model, self.custom_llm_provider
)
):
output_config = self._create_output_config_for_response_format(
json_schema=output_config_format.get("schema"),
name=output_config_format.get("name"),
description=output_config_format.get("description"),
)
elif output_config is None and output_config_format is not None:
litellm.verbose_logger.warning(
"Bedrock Converse: dropping `output_config.format` for model=%s — "
"model does not advertise `supports_native_structured_output` in "
"model_prices_and_context_window.json. The schema will not be "
"enforced; pass `response_format` to use the synthetic tool-call "
"fallback.",
model,
)
# keep supported params in 'inference_params', and set all model-specific params in 'additional_request_params'
additional_request_params = {
@ -1272,7 +1325,6 @@ class AmazonConverseConfig(BaseConfig):
if anthropic_output_config is not None and isinstance(
anthropic_output_config, dict
):
base_model = BedrockModelInfo.get_base_model(model)
if base_model.startswith("anthropic"):
if (
litellm.drop_params is True
@ -1283,6 +1335,11 @@ class AmazonConverseConfig(BaseConfig):
model,
)
else:
if not anthropic_output_config_already_normalized:
normalize_bedrock_opus_output_config_effort(
model=model,
output_config=anthropic_output_config,
)
effort = anthropic_output_config.get("effort")
if effort is not None:
self._validate_anthropic_adaptive_effort(

57
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -16,8 +16,11 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import (
convert_bedrock_invoke_output_format_to_inline_schema,
get_anthropic_beta_from_headers,
normalize_bedrock_opus_output_config_effort,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
@ -75,6 +78,17 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
# Use a model name that forces tool-based approach
model = "claude-3-sonnet-20240229"
# Clamp ``reasoning_effort`` to the Bedrock effort ceiling before the
# parent mapping converts it to ``output_config.effort`` and the
# downstream effort gate runs. Mirrors the converse path's
# ``_handle_reasoning_effort_parameter`` and the messages path's
# ``_clamp_adaptive_reasoning_effort_for_bedrock`` so adaptive Claude
# requests degrade ``xhigh`` -> ``max`` rather than 400-ing on
# models like Opus 4.6 that don't natively advertise xhigh.
self._clamp_adaptive_reasoning_effort_for_bedrock(
model=original_model, params=non_default_params
)
optional_params = AnthropicConfig.map_openai_params(
self,
non_default_params,
@ -88,6 +102,27 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
return optional_params
@staticmethod
def _clamp_adaptive_reasoning_effort_for_bedrock(model: str, params: dict) -> None:
"""Lower ``reasoning_effort`` to the Bedrock effort ceiling before mapping.
Bedrock's adaptive Claude models accept the OpenAI-style
``reasoning_effort`` tier, but the request validator can reject tiers
the model does not natively advertise (e.g. ``xhigh`` on Opus 4.6).
Clamp the raw tier to the model's
``bedrock_output_config_effort_ceiling`` so Claude Code "goal mode"
keeps working. Non-adaptive models and models without a ceiling are
left untouched.
"""
if not AnthropicConfig._is_adaptive_thinking_model(model):
return
effort = params.get("reasoning_effort")
if not isinstance(effort, str):
return
clamped = {"effort": effort}
normalize_bedrock_opus_output_config_effort(model=model, output_config=clamped)
params["reasoning_effort"] = clamped["effort"]
def transform_request(
self,
model: str,
@ -157,6 +192,13 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
for k, v in optional_params.items()
if k not in self.aws_authentication_params
}
output_config = filtered_params.get("output_config")
if isinstance(output_config, dict):
filtered_params["output_config"] = dict(output_config)
normalize_bedrock_opus_output_config_effort(
model=model,
output_config=filtered_params["output_config"],
)
filtered_params = self._normalize_bedrock_tool_search_tools(filtered_params)
anthropic_request = AnthropicConfig.transform_request(
@ -170,7 +212,20 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
anthropic_request.pop("model", None)
anthropic_request.pop("stream", None)
anthropic_request.pop("output_format", None)
output_format = anthropic_request.pop("output_format", None)
output_config_format = pop_bedrock_invoke_output_config_format(
anthropic_request
)
if output_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_format,
request_body=anthropic_request,
)
elif output_config_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_config_format,
request_body=anthropic_request,
)
if not (
_supports_factory(
model=model,

138
litellm/llms/bedrock/common_utils.py
View file

@ -34,6 +34,15 @@ class BedrockError(BaseLLMException):
# Lazy import cache to avoid circular imports and performance impact
_get_model_info = None
BedrockOutputConfigEffort = Literal["low", "medium", "high", "max", "xhigh"]
_BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER: Dict[BedrockOutputConfigEffort, int] = {
"low": 0,
"medium": 1,
"high": 2,
"max": 3,
"xhigh": 4,
}
def get_cached_model_info():
"""
@ -51,6 +60,79 @@ def get_cached_model_info():
return _get_model_info
@functools.lru_cache(maxsize=1)
def _get_local_model_cost_map() -> Dict:
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
return GetModelCostMap.load_local_model_cost_map()
def pop_bedrock_invoke_output_config_format(request_body: Dict) -> Optional[Dict]:
"""
Remove and return Anthropic's nested ``output_config.format`` field.
Bedrock Invoke paths convert the schema to inline message text. Any remaining
``output_config`` keys, such as ``effort``, are left in place.
"""
output_config = request_body.get("output_config")
if not isinstance(output_config, dict):
return None
output_format = output_config.pop("format", None)
if not output_config:
request_body.pop("output_config", None)
if isinstance(output_format, dict):
return output_format
return None
def convert_bedrock_invoke_output_format_to_inline_schema(
output_format: Dict,
request_body: Dict,
) -> None:
"""
Embed an Anthropic structured-output schema into the last user message.
Bedrock Invoke does not support ``output_format`` directly, so the schema is
appended to the final user message for prompt-engineered structured output.
The caller's ``messages`` list, message dict, and content list are not
mutated; a fresh ``messages`` list with a copied final user message is
written back to ``request_body``.
"""
schema = output_format.get("schema")
if not schema:
return
messages = request_body.get("messages")
if not isinstance(messages, list) or not messages:
return
last_user_idx = None
for i in range(len(messages) - 1, -1, -1):
message = messages[i]
if isinstance(message, dict) and message.get("role") == "user":
last_user_idx = i
break
if last_user_idx is None:
return
original = messages[last_user_idx]
content = original.get("content", [])
schema_block = {"type": "text", "text": json.dumps(schema)}
if isinstance(content, str):
new_content = [{"type": "text", "text": content}, schema_block]
elif isinstance(content, list):
new_content = [*content, schema_block]
else:
return
new_messages = list(messages)
new_messages[last_user_idx] = {**original, "content": new_content}
request_body["messages"] = new_messages
def remove_custom_field_from_tools(request_body: dict) -> None:
"""
Remove ``custom`` field from each tool in the request body.
@ -603,6 +685,62 @@ def is_claude_4_5_on_bedrock(model: str) -> bool:
return any(pattern in model_lower for pattern in claude_4_5_patterns)
def normalize_bedrock_opus_output_config_effort(model: str, output_config: Any) -> None:
"""
Normalize Anthropic ``output_config.effort`` values for Bedrock Opus ids.
Bedrock's Claude Opus request validator can accept a narrower effort
vocabulary than Anthropic's compatibility surface. The Bedrock ceiling is
read from ``model_prices_and_context_window.json`` via
``bedrock_output_config_effort_ceiling``.
Mutates ``output_config`` in place so callers can accept Claude Code's
``xhigh`` input without forwarding a provider-invalid value.
"""
if not isinstance(output_config, dict):
return
effort = output_config.get("effort")
if effort not in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER:
return
ceiling = _get_bedrock_output_config_effort_ceiling(model)
if ceiling is None:
return
if (
_BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER[effort]
> _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER[ceiling]
):
output_config["effort"] = ceiling
def _get_bedrock_output_config_effort_ceiling(
model: str,
) -> Optional[BedrockOutputConfigEffort]:
try:
model_info = get_cached_model_info()(
model=model,
custom_llm_provider="bedrock",
)
except Exception:
return None
ceiling = model_info.get("bedrock_output_config_effort_ceiling")
if isinstance(ceiling, str) and ceiling in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER:
return ceiling # type: ignore[return-value]
model_cost_key = model_info.get("key")
if not isinstance(model_cost_key, str):
return None
local_model_info = _get_local_model_cost_map().get(model_cost_key, {})
ceiling = local_model_info.get("bedrock_output_config_effort_ceiling")
if isinstance(ceiling, str) and ceiling in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER:
return ceiling # type: ignore[return-value]
return None
# Import after standalone functions to avoid circular imports
from litellm.llms.bedrock.count_tokens.bedrock_token_counter import BedrockTokenCounter

308
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -32,10 +32,13 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import (
convert_bedrock_invoke_output_format_to_inline_schema,
ensure_bedrock_anthropic_messages_tool_names,
get_anthropic_beta_from_headers,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
@ -450,145 +453,15 @@ class AmazonAnthropicClaudeMessagesConfig(
else:
anthropic_messages_request.pop("context_management", None)
def _convert_output_format_to_inline_schema(
self,
output_format: Dict,
anthropic_messages_request: Dict,
) -> None:
"""
Convert Anthropic output_format to inline schema in message content.
Bedrock Invoke doesn't support the output_format parameter, so we embed
the schema directly into the user message content as text instructions.
This approach adds the schema to the last user message, instructing the model
to respond in the specified JSON format.
Args:
output_format: The output_format dict with 'type' and 'schema'
anthropic_messages_request: The request dict to modify in-place
Ref: https://aws.amazon.com/blogs/machine-learning/structured-data-response-with-amazon-bedrock-prompt-engineering-and-tool-use/
"""
import json
# Extract schema from output_format
schema = output_format.get("schema")
if not schema:
return
# Get messages from the request
messages = anthropic_messages_request.get("messages", [])
if not messages:
return
# Find the last user message
last_user_message_idx = None
for idx in range(len(messages) - 1, -1, -1):
if messages[idx].get("role") == "user":
last_user_message_idx = idx
break
if last_user_message_idx is None:
return
last_user_message = messages[last_user_message_idx]
content = last_user_message.get("content", [])
# Ensure content is a list
if isinstance(content, str):
content = [{"type": "text", "text": content}]
last_user_message["content"] = content
# Add schema as text content to the message
schema_text = {"type": "text", "text": json.dumps(schema)}
content.append(schema_text)
def transform_anthropic_messages_request(
def _get_bedrock_invoke_anthropic_beta_headers(
self,
model: str,
messages: List[Dict],
anthropic_messages_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Dict:
anthropic_messages_request = AnthropicMessagesConfig.transform_anthropic_messages_request(
self=self,
model=model,
messages=messages,
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
litellm_params=litellm_params,
headers=headers,
)
#########################################################
############## BEDROCK Invoke SPECIFIC TRANSFORMATION ###
#########################################################
# 1. anthropic_version is required for all claude models
if "anthropic_version" not in anthropic_messages_request:
anthropic_messages_request["anthropic_version"] = (
self.DEFAULT_BEDROCK_ANTHROPIC_API_VERSION
)
# 2. `stream` is not allowed in request body for bedrock invoke
if "stream" in anthropic_messages_request:
anthropic_messages_request.pop("stream", None)
# 3. `model` is not allowed in request body for bedrock invoke
if "model" in anthropic_messages_request:
anthropic_messages_request.pop("model", None)
injected_thinking_for_clear_thinking = (
self._ensure_thinking_for_clear_thinking_context_management(
anthropic_messages_request=anthropic_messages_request,
model=model,
)
)
# 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it for older models)
self._remove_ttl_from_cache_control(
anthropic_messages_request=anthropic_messages_request, model=model
)
# 5. Convert `output_format` to inline schema (Bedrock invoke doesn't support output_format)
output_format = anthropic_messages_request.pop("output_format", None)
if output_format:
self._convert_output_format_to_inline_schema(
output_format=output_format,
anthropic_messages_request=anthropic_messages_request,
)
# 5a. Bedrock Invoke supports output_config (effort) for Claude 4.6+ models,
# but older models do not — strip it to avoid request rejection.
# Ref: https://github.com/BerriAI/litellm/issues/22797
if not (
_supports_factory(
model=model,
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model)
):
if anthropic_messages_request.pop("output_config", None) is not None:
verbose_logger.warning(
"Bedrock Invoke: stripping unsupported `output_config` for "
"model=%s — neither `supports_output_config` nor any "
"`supports_*_reasoning_effort` flag is set in "
"model_prices_and_context_window.json. Add the capability "
"flag to the model JSON entry if this model accepts "
"`output_config`.",
model,
)
# 5b. Remove `custom` field from tools (Bedrock doesn't support it)
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
# Ref: https://github.com/BerriAI/litellm/issues/22847
remove_custom_field_from_tools(anthropic_messages_request)
normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request)
ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request)
# 6. AUTO-INJECT beta headers based on features used
anthropic_messages_request: Dict,
injected_thinking_for_clear_thinking: bool,
) -> List[str]:
anthropic_model_info = AnthropicModelInfo()
tools = anthropic_messages_optional_request_params.get("tools")
messages_typed = cast(List[AllMessageValues], messages)
@ -651,6 +524,160 @@ class AmazonAnthropicClaudeMessagesConfig(
dropped_user_betas,
)
return filtered_betas
def _strip_unsupported_bedrock_invoke_fields(
self,
anthropic_messages_request: Dict,
) -> Dict:
allowed = self.BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS
stripped = sorted(k for k in anthropic_messages_request if k not in allowed)
if stripped:
verbose_logger.debug(
"Bedrock Invoke: stripping unsupported top-level request fields: %s",
stripped,
)
return {k: v for k, v in anthropic_messages_request.items() if k in allowed}
@staticmethod
def _clamp_adaptive_reasoning_effort_for_bedrock(
model: str, optional_params: Dict
) -> None:
"""Lower ``reasoning_effort`` to the Bedrock effort ceiling before validation.
The shared ``/v1/messages`` effort gate rejects tiers a model does not
natively support (e.g. ``xhigh`` on Opus 4.6). Bedrock's chat paths instead
clamp the tier to the model's ``bedrock_output_config_effort_ceiling`` so
Claude Code "goal mode" keeps working; mirror that here so the messages
path degrades ``xhigh`` -> ``max`` rather than 400-ing. Non-adaptive models
and models without a ceiling are left untouched.
"""
if not AnthropicModelInfo._is_adaptive_thinking_model(model):
return
effort = optional_params.get("reasoning_effort")
if not isinstance(effort, str):
return
clamped = {"effort": effort}
normalize_bedrock_opus_output_config_effort(model=model, output_config=clamped)
optional_params["reasoning_effort"] = clamped["effort"]
def transform_anthropic_messages_request(
self,
model: str,
messages: List[Dict],
anthropic_messages_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Dict:
self._clamp_adaptive_reasoning_effort_for_bedrock(
model=model,
optional_params=anthropic_messages_optional_request_params,
)
anthropic_messages_request = AnthropicMessagesConfig.transform_anthropic_messages_request(
self=self,
model=model,
messages=messages,
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
litellm_params=litellm_params,
headers=headers,
)
#########################################################
############## BEDROCK Invoke SPECIFIC TRANSFORMATION ###
#########################################################
# 1. anthropic_version is required for all claude models
if "anthropic_version" not in anthropic_messages_request:
anthropic_messages_request["anthropic_version"] = (
self.DEFAULT_BEDROCK_ANTHROPIC_API_VERSION
)
# 2. `stream` is not allowed in request body for bedrock invoke
if "stream" in anthropic_messages_request:
anthropic_messages_request.pop("stream", None)
# 3. `model` is not allowed in request body for bedrock invoke
if "model" in anthropic_messages_request:
anthropic_messages_request.pop("model", None)
injected_thinking_for_clear_thinking = (
self._ensure_thinking_for_clear_thinking_context_management(
anthropic_messages_request=anthropic_messages_request,
model=model,
)
)
# 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it for older models)
self._remove_ttl_from_cache_control(
anthropic_messages_request=anthropic_messages_request, model=model
)
# 5. Convert structured-output params to inline schema.
# Bedrock Invoke doesn't support top-level `output_format`; its
# accepted `output_config` subset is also narrower than Anthropic's, so
# consume the newer `output_config.format` shape here instead of
# forwarding it as an unknown nested key.
existing_output_config = anthropic_messages_request.get("output_config")
if isinstance(existing_output_config, dict):
anthropic_messages_request["output_config"] = dict(existing_output_config)
output_format = anthropic_messages_request.pop("output_format", None)
output_config_format = pop_bedrock_invoke_output_config_format(
anthropic_messages_request
)
if output_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_format,
request_body=anthropic_messages_request,
)
elif output_config_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_config_format,
request_body=anthropic_messages_request,
)
normalize_bedrock_opus_output_config_effort(
model=model,
output_config=anthropic_messages_request.get("output_config"),
)
# 5a. Bedrock Invoke supports output_config (effort) for Claude 4.6+ models,
# but older models do not — strip it to avoid request rejection.
# Ref: https://github.com/BerriAI/litellm/issues/22797
if not (
_supports_factory(
model=model,
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model)
):
if anthropic_messages_request.pop("output_config", None) is not None:
verbose_logger.warning(
"Bedrock Invoke: stripping unsupported `output_config` for "
"model=%s — neither `supports_output_config` nor any "
"`supports_*_reasoning_effort` flag is set in "
"model_prices_and_context_window.json. Add the capability "
"flag to the model JSON entry if this model accepts "
"`output_config`.",
model,
)
# 5b. Remove `custom` field from tools (Bedrock doesn't support it)
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
# Ref: https://github.com/BerriAI/litellm/issues/22847
remove_custom_field_from_tools(anthropic_messages_request)
normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request)
ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request)
# 6. AUTO-INJECT beta headers based on features used
filtered_betas = self._get_bedrock_invoke_anthropic_beta_headers(
model=model,
messages=messages,
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
headers=headers,
anthropic_messages_request=anthropic_messages_request,
injected_thinking_for_clear_thinking=injected_thinking_for_clear_thinking,
)
if filtered_betas:
anthropic_messages_request["anthropic_beta"] = filtered_betas
@ -669,16 +696,9 @@ class AmazonAnthropicClaudeMessagesConfig(
# Catches Anthropic-only extensions (output_config, speed, mcp_servers, ...)
# and any future additions Claude Code may start sending. ``context_management``
# has already been pre-filtered to its Bedrock-supported subset above.
allowed = self.BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS
stripped = sorted(k for k in anthropic_messages_request if k not in allowed)
if stripped:
verbose_logger.debug(
"Bedrock Invoke: stripping unsupported top-level request fields: %s",
stripped,
)
anthropic_messages_request = {
k: v for k, v in anthropic_messages_request.items() if k in allowed
}
anthropic_messages_request = self._strip_unsupported_bedrock_invoke_fields(
anthropic_messages_request
)
return anthropic_messages_request

135
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -5316,6 +5316,28 @@ class BaseLLMHTTPHandler:
)
if _session_config:
realtime_streaming.session_configuration_request = _session_config
# For providers that defer setup until client session.update, optionally
# send synthetic session.created to unblock clients waiting on connect.
if not provider_config.requires_session_configuration():
synthetic_session = provider_config.transform_session_created_event(
model=model,
logging_session_id=logging_obj.litellm_trace_id,
session_configuration_request=None,
)
if synthetic_session is not None:
synthetic_session_str = json.dumps(synthetic_session)
# Record before sending so the synthetic session.created is
# captured in the session log alongside provider-driven
# events; without this it would be silently absent from
# success_handler / async_success_handler payloads.
realtime_streaming.store_message(synthetic_session_str)
await websocket.send_text(synthetic_session_str)
realtime_streaming._session_created_sent_to_client = True
verbose_logger.debug(
"Sent synthetic session.created to client to unblock connection"
)
await realtime_streaming.bidirectional_forward()
except websockets.exceptions.InvalidStatusCode as e: # type: ignore
@ -6538,6 +6560,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
@ -6620,6 +6643,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
@ -6712,6 +6736,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -6783,6 +6808,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -6866,6 +6892,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -6923,6 +6950,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -6999,6 +7027,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -7009,27 +7038,49 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
url, data = video_provider_config.transform_video_edit_request(
prompt=prompt,
prefetched_source_data = None
prefetch_params = video_provider_config.get_video_edit_prefetch_params(
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
headers=headers,
extra_body=extra_body,
)
logging_obj.pre_call(
input=prompt,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": url,
"headers": headers,
"video_id": video_id,
},
)
if prefetch_params is not None:
prefetch_url, prefetch_body = prefetch_params
try:
prefetch_resp = sync_httpx_client.post(
url=prefetch_url,
headers=headers,
json=prefetch_body,
timeout=timeout,
)
prefetch_resp.raise_for_status()
except Exception as e:
raise self._handle_error(e=e, provider_config=video_provider_config)
prefetched_source_data = prefetch_resp.json()
try:
url, data = video_provider_config.transform_video_edit_request(
prompt=prompt,
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
headers=headers,
extra_body=extra_body,
prefetched_source_data=prefetched_source_data,
)
logging_obj.pre_call(
input=prompt,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": url,
"headers": headers,
"video_id": video_id,
},
)
response = sync_httpx_client.post(
url=url,
headers=headers,
@ -7041,6 +7092,7 @@ class BaseLLMHTTPHandler:
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
request_data=data,
)
except Exception as e:
raise self._handle_error(e=e, provider_config=video_provider_config)
@ -7071,6 +7123,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -7081,27 +7134,49 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
url, data = video_provider_config.transform_video_edit_request(
prompt=prompt,
prefetched_source_data = None
prefetch_params = video_provider_config.get_video_edit_prefetch_params(
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
headers=headers,
extra_body=extra_body,
)
logging_obj.pre_call(
input=prompt,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": url,
"headers": headers,
"video_id": video_id,
},
)
if prefetch_params is not None:
prefetch_url, prefetch_body = prefetch_params
try:
prefetch_resp = await async_httpx_client.post(
url=prefetch_url,
headers=headers,
json=prefetch_body,
timeout=timeout,
)
prefetch_resp.raise_for_status()
except Exception as e:
raise self._handle_error(e=e, provider_config=video_provider_config)
prefetched_source_data = prefetch_resp.json()
try:
url, data = video_provider_config.transform_video_edit_request(
prompt=prompt,
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
headers=headers,
extra_body=extra_body,
prefetched_source_data=prefetched_source_data,
)
logging_obj.pre_call(
input=prompt,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": url,
"headers": headers,
"video_id": video_id,
},
)
response = await async_httpx_client.post(
url=url,
headers=headers,
@ -7113,6 +7188,7 @@ class BaseLLMHTTPHandler:
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
request_data=data,
)
except Exception as e:
raise self._handle_error(e=e, provider_config=video_provider_config)
@ -7160,6 +7236,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -7234,6 +7311,7 @@ class BaseLLMHTTPHandler:
api_key=api_key or litellm_params.get("api_key", None),
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
@ -7445,6 +7523,7 @@ class BaseLLMHTTPHandler:
api_key=api_key,
headers=extra_headers or {},
model="",
litellm_params=litellm_params,
)
if extra_headers:

867
litellm/llms/gemini/realtime/transformation.py
View file

File diff suppressed because it is too large Load diff

15
litellm/llms/gemini/videos/transformation.py
View file

@ -581,12 +581,23 @@ class GeminiVideoConfig(BaseVideoConfig):
raise NotImplementedError("video get character is not supported for Gemini")
def transform_video_edit_request(
self, prompt, video_id, api_base, litellm_params, headers, extra_body=None
self,
prompt,
video_id,
api_base,
litellm_params,
headers,
extra_body=None,
prefetched_source_data=None,
):
raise NotImplementedError("video edit is not supported for Gemini")
def transform_video_edit_response(
self, raw_response, logging_obj, custom_llm_provider=None
self,
raw_response,
logging_obj,
custom_llm_provider=None,
request_data=None,
):
raise NotImplementedError("video edit is not supported for Gemini")

9
litellm/llms/openai/cost_calculation.py
View file

@ -19,7 +19,10 @@ def cost_router(call_type: CallTypes) -> Literal["cost_per_token", "cost_per_sec
def cost_per_token(
model: str, usage: Usage, service_tier: Optional[str] = None
model: str,
usage: Usage,
service_tier: Optional[str] = None,
data_residency: Optional[str] = None,
) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -27,6 +30,9 @@ def cost_per_token(
Input:
- model: str, the model name without provider prefix
- usage: LiteLLM Usage block, containing anthropic caching information
- data_residency: optional OpenAI data-residency region (e.g. "eu", "us"),
inferred from api_base. Applies the model's regional-processing
uplift multiplier when set.
Returns:
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
@ -37,6 +43,7 @@ def cost_per_token(
usage=usage,
custom_llm_provider="openai",
service_tier=service_tier,
data_residency=data_residency,
)
# ### Non-cached text tokens
# non_cached_text_tokens = usage.prompt_tokens

41
litellm/llms/openai/data_residency.py Normal file
View file

@ -0,0 +1,41 @@
"""
Helpers for resolving OpenAI data-residency (regional processing) from an
api_base URL.
OpenAI enforces hostname-per-region for projects with geography restrictions
enabled and rejects requests sent to the wrong host, so the api_base hostname
is the authoritative signal of which region a request was processed in.
"""
from typing import Dict, Optional
from urllib.parse import urlparse
# Mapping of OpenAI regional hostnames to the corresponding data-residency
# value used by the cost calculator. See
# https://developers.openai.com/api/docs/pricing for the regional-processing
# uplift these hostnames trigger.
_OPENAI_REGIONAL_HOSTS: Dict[str, str] = {
"eu.api.openai.com": "eu",
"us.api.openai.com": "us",
}
def infer_openai_data_residency(
custom_llm_provider: Optional[str], api_base: Optional[str]
) -> Optional[str]:
"""
Derive the OpenAI data-residency region from an api_base URL.
Returns ``"eu"`` for the EU regional host, ``"us"`` for the US regional
host, and ``None`` for the default global host, any non-OpenAI provider,
or any non-OpenAI URL.
"""
if custom_llm_provider != "openai" or not api_base:
return None
try:
host = urlparse(api_base).hostname
except (TypeError, ValueError):
return None
if not host:
return None
return _OPENAI_REGIONAL_HOSTS.get(host.lower())

2
litellm/llms/openai/videos/transformation.py
View file

@ -534,6 +534,7 @@ class OpenAIVideoConfig(BaseVideoConfig):
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: Optional[Dict[str, Any]] = None,
prefetched_source_data: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
original_video_id = extract_original_video_id(video_id)
url = f"{api_base.rstrip('/')}/edits"
@ -547,6 +548,7 @@ class OpenAIVideoConfig(BaseVideoConfig):
raw_response: httpx.Response,
logging_obj: Any,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
video_obj = VideoObject(**raw_response.json())
if custom_llm_provider and video_obj.id:

15
litellm/llms/runwayml/videos/transformation.py
View file

@ -623,12 +623,23 @@ class RunwayMLVideoConfig(BaseVideoConfig):
raise NotImplementedError("video get character is not supported for RunwayML")
def transform_video_edit_request(
self, prompt, video_id, api_base, litellm_params, headers, extra_body=None
self,
prompt,
video_id,
api_base,
litellm_params,
headers,
extra_body=None,
prefetched_source_data=None,
):
raise NotImplementedError("video edit is not supported for RunwayML")
def transform_video_edit_response(
self, raw_response, logging_obj, custom_llm_provider=None
self,
raw_response,
logging_obj,
custom_llm_provider=None,
request_data=None,
):
raise NotImplementedError("video edit is not supported for RunwayML")

108
litellm/llms/vertex_ai/realtime/transformation.py
View file

@ -14,6 +14,7 @@ Auth: OAuth2 Bearer token (not an API key).
import json
from typing import List, Optional
from litellm import verbose_logger
from litellm.llms.gemini.realtime.transformation import GeminiRealtimeConfig
@ -26,6 +27,7 @@ class VertexAIRealtimeConfig(GeminiRealtimeConfig):
"""
def __init__(self, access_token: str, project: str, location: str) -> None:
super().__init__()
self._access_token = access_token
self._project = project
self._location = location
@ -138,6 +140,62 @@ class VertexAIRealtimeConfig(GeminiRealtimeConfig):
# Request translation
# ------------------------------------------------------------------
def _vertex_model_path(self, model: str) -> str:
"""Return the fully-qualified Vertex AI model resource path."""
return (
f"projects/{self._project}"
f"/locations/{self._location}"
f"/publishers/google/models/{model}"
)
def _build_vertex_ai_setup_config(self, model: str, session_params: dict) -> dict:
"""Build Vertex AI setup configuration with proper model path and defaults."""
# Normalize GA-remapped fields (``output_modalities``, nested
# ``audio.input.transcription``, ``audio.input.turn_detection``) back to
# their flat beta keys so ``map_openai_params`` picks them up. Without
# this, GA clients' explicit modality / transcription / turn-detection
# settings would be silently dropped because ``map_openai_params`` only
# recognises the flat OpenAI-beta key names.
session_params = self._normalize_session_payload_for_mapping(session_params)
setup_config = self.map_openai_params(
optional_params={}, non_default_params=session_params
)
# Use full Vertex AI model path
setup_config["model"] = self._vertex_model_path(model)
# Add Vertex AI specific defaults if not provided
generation_config = setup_config.setdefault("generationConfig", {})
generation_config.setdefault("responseModalities", ["AUDIO"])
# Ensure Vertex defaults for realtimeInputConfig apply even when
# the client provided a partial ``turn_detection`` (e.g. only
# ``silence_duration_ms``). ``map_automatic_turn_detection`` sets
# ``disabled=True`` whenever ``create_response`` is absent or
# ``False``. Force ``disabled=False`` only when the client did
# not explicitly request ``create_response: False`` — that path
# is how transcription guardrails suppress automatic responses,
# and overriding it here would silently bypass the guardrail.
# Vertex Live has no "VAD on, no auto-response" mode, so callers
# that need that behaviour must accept that VAD is off.
client_turn_detection = session_params.get("turn_detection")
client_disabled_auto_response = (
isinstance(client_turn_detection, dict)
and client_turn_detection.get("create_response") is False
)
realtime_input_config = setup_config.setdefault("realtimeInputConfig", {})
automatic_detection = realtime_input_config.setdefault(
"automaticActivityDetection", {}
)
if not client_disabled_auto_response:
automatic_detection["disabled"] = False
automatic_detection.setdefault("silenceDurationMs", 800)
setup_config.setdefault("inputAudioTranscription", {})
setup_config.setdefault("outputAudioTranscription", {})
return setup_config
def transform_realtime_request(
self,
message: str,
@ -147,16 +205,50 @@ class VertexAIRealtimeConfig(GeminiRealtimeConfig):
"""
Translate OpenAI realtime client messages to Vertex AI format.
``session.update`` is intentionally ignored (returns []) because
Vertex AI only accepts a single ``setup`` message at the start of
the connection — sending a second one causes a 1007 close error.
The initial setup (sent automatically before bidirectional_forward)
already includes AUDIO modality and server VAD, so there is nothing
more to configure.
On the first ``session.update`` (when no setup has been sent yet) the
full ``BidiGenerateContentSetup`` is built with Vertex AI's model path
and forwarded. Any later ``session.update`` is dropped: Vertex AI
documents ``setup`` as the first-and-only client message, and a second
``setup`` closes the connection with a 1007 policy error.
"""
json_message = json.loads(message)
if json_message.get("type") == "session.update":
# Do not forward as a second setup — Vertex AI rejects it.
msg_type = json_message.get("type")
if msg_type == "session.update":
if session_configuration_request is None:
setup_config = self._build_vertex_ai_setup_config(
model, json_message.get("session") or {}
)
gemini_setup_msg = json.dumps({"setup": setup_config})
verbose_logger.debug(
"Vertex AI Realtime: Sending initial setup with tools to backend"
)
return [gemini_setup_msg]
# A follow-up session.update can't be forwarded as a second setup
# (Vertex Live closes the WebSocket with 1007). If this drop is
# silencing the audio-transcription guardrail's create_response
# disable, surface a warning so operators know the model will
# auto-respond before the guardrail can gate it on Vertex AI.
client_turn_detection = GeminiRealtimeConfig._extract_turn_detection(
json_message.get("session") or {}
)
if (
isinstance(client_turn_detection, dict)
and client_turn_detection.get("create_response") is False
):
verbose_logger.warning(
"Vertex AI Realtime: Dropping subsequent session.update "
"(turn_detection.create_response=False) — Vertex Live "
"rejects a second setup message. Audio-transcription "
"guardrails cannot suppress the model's auto-response on "
"Vertex AI in non-deferred mode."
)
else:
verbose_logger.debug(
"Vertex AI Realtime: Ignoring session.update (setup already sent)"
)
return []
return super().transform_realtime_request(

161
litellm/llms/vertex_ai/videos/transformation.py
View file

@ -40,6 +40,29 @@ else:
BaseLLMException = Any
def _build_vertex_video_usage_from_request_data(
request_data: Optional[Dict[str, Any]],
) -> Dict[str, Any]:
"""Build usage metadata (duration, resolution) for video cost calculation."""
usage_data: Dict[str, Any] = {}
if not request_data:
return usage_data
parameters = request_data.get("parameters", {})
duration = (
parameters.get("durationSeconds") or DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
)
if duration is not None:
try:
usage_data["duration_seconds"] = float(duration)
except (ValueError, TypeError):
pass
res = parameters.get("resolution")
if res is not None and str(res).strip() != "":
usage_data["video_resolution"] = str(res).strip().lower()
return usage_data
def _convert_image_to_vertex_format(image_file) -> Dict[str, str]:
"""
Convert image file to Vertex AI format with base64 encoding and MIME type.
@ -363,23 +386,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
id=video_id, object="video", status="processing", model=model
)
usage_data: Dict[str, Any] = {}
if request_data:
parameters = request_data.get("parameters", {})
duration = (
parameters.get("durationSeconds")
or DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
)
if duration is not None:
try:
usage_data["duration_seconds"] = float(duration)
except (ValueError, TypeError):
pass
res = parameters.get("resolution")
if res is not None and str(res).strip() != "":
usage_data["video_resolution"] = str(res).strip().lower()
video_obj.usage = usage_data
video_obj.usage = _build_vertex_video_usage_from_request_data(request_data)
return video_obj
def transform_video_status_retrieve_request(
@ -647,15 +654,123 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
def transform_video_get_character_response(self, raw_response, logging_obj):
raise NotImplementedError("video get character is not supported for Vertex AI")
def get_video_edit_prefetch_params(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""Return the fetchPredictOperation URL and body needed to retrieve the source video."""
return self.transform_video_status_retrieve_request(
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
headers=headers,
)
def transform_video_edit_request(
self, prompt, video_id, api_base, litellm_params, headers, extra_body=None
):
raise NotImplementedError("video edit is not supported for Vertex AI")
self,
prompt: str,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: Optional[Dict[str, Any]] = None,
prefetched_source_data: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Build a predictLongRunning edit request from the pre-fetched source video.
The actual fetchPredictOperation HTTP call is hoisted into the handler so
it can use the shared async/sync httpx client instead of blocking the loop.
"""
if prefetched_source_data is None:
raise ValueError(
"prefetched_source_data is required for Vertex AI video edit. "
"Ensure get_video_edit_prefetch_params is called by the handler."
)
if not prefetched_source_data.get("done", False):
raise ValueError(
"Source video generation is not complete yet. "
"Check the video status before editing."
)
videos = prefetched_source_data.get("response", {}).get("videos", [])
if not videos:
raise ValueError("No videos found in the completed operation. Cannot edit.")
source_video = videos[0]
video_input: Dict[str, Any] = {}
if "gcsUri" in source_video:
video_input["gcsUri"] = source_video["gcsUri"]
elif "bytesBase64Encoded" in source_video:
video_input["bytesBase64Encoded"] = source_video["bytesBase64Encoded"]
video_input["mimeType"] = source_video.get("mimeType", "video/mp4")
else:
raise ValueError(
"Source video has neither gcsUri nor bytesBase64Encoded. Cannot edit."
)
operation_name = extract_original_video_id(video_id)
model = self.extract_model_from_operation_name(operation_name) or ""
instance_dict: Dict[str, Any] = {"prompt": prompt, "video": video_input}
request_data: Dict[str, Any] = {"instances": [instance_dict]}
if extra_body:
extra_body_copy = dict(extra_body)
nested_params = extra_body_copy.pop("parameters", None)
vertex_params: Dict[str, Any] = {}
if isinstance(nested_params, dict):
vertex_params.update(nested_params)
vertex_params.update(extra_body_copy)
if vertex_params:
request_data["parameters"] = vertex_params
edit_url = f"{api_base.rstrip('/')}/{model}:predictLongRunning"
return edit_url, request_data
def transform_video_edit_response(
self, raw_response, logging_obj, custom_llm_provider=None
):
raise NotImplementedError("video edit is not supported for Vertex AI")
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
"""
Transform the Veo video edit response.
Veo returns the same operation response as video generation:
{"name": "projects/.../operations/OPERATION_ID"}
usage includes duration_seconds and optional video_resolution from the
edit request parameters for cost calculation.
"""
response_data = raw_response.json()
operation_name = response_data.get("name")
if not operation_name:
raise ValueError(f"No operation name in Veo edit response: {response_data}")
model = self.extract_model_from_operation_name(operation_name) or ""
if custom_llm_provider:
video_id = encode_video_id_with_provider(
operation_name, custom_llm_provider, model
)
else:
video_id = operation_name
video_obj = VideoObject(
id=video_id,
object="video",
status="processing",
model=model,
)
video_obj.usage = _build_vertex_video_usage_from_request_data(request_data)
return video_obj
def transform_video_extension_request(
self,

982
litellm/model_prices_and_context_window_backup.json
View file

File diff suppressed because it is too large Load diff

5
litellm/passthrough/utils.py
View file

@ -71,6 +71,11 @@ class BasePassthroughUtils:
request_headers.pop("content-length", None)
request_headers.pop("host", None)
custom_header_names = {header_name.lower() for header_name in headers}
for header_name in list(request_headers.keys()):
if header_name.lower() in custom_header_names:
request_headers.pop(header_name, None)
# Combine request headers with custom headers
headers = {**request_headers, **headers}

154
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -118,15 +118,19 @@ class MCPRequestHandler:
return b"{}"
request.body = mock_body # type: ignore
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import ( # noqa: PLC0415
get_request_route,
)
request_route = get_request_route(request)
# Only OAuth metadata routes registered under /.well-known/ are public.
# Match on request.url.path (path-only, exact prefix) so the substring
# cannot be smuggled via query string, hostname, or a deeper URL segment.
if request.url.path.startswith("/.well-known/"):
if request_route.startswith("/.well-known/"):
validated_user_api_key_auth = UserAPIKeyAuth()
elif (
not litellm_api_key
and MCPRequestHandler._target_servers_delegate_auth_to_upstream( # noqa: E501
path=request.url.path, mcp_servers=mcp_servers
path=request_route, mcp_servers=mcp_servers
)
):
# Operator opted this oauth2 server into upstream-delegated auth
@ -174,7 +178,7 @@ class MCPRequestHandler:
"401",
"403",
) and MCPRequestHandler._target_servers_use_oauth2(
path=request.url.path, mcp_servers=mcp_servers
path=request_route, mcp_servers=mcp_servers
):
verbose_logger.debug(
"MCP OAuth2: target server is OAuth2-mode, treating "
@ -562,25 +566,32 @@ class MCPRequestHandler:
)
)
key_access_group_extras = (
await MCPRequestHandler._get_key_access_group_mcp_server_extras(
user_api_key_auth
)
)
#########################################################
# Calculate key/team allowed servers using inheritance and intersection logic
#########################################################
allowed_mcp_servers: List[str] = []
has_lower_level_mcp_restrictions = (
len(allowed_mcp_servers_for_key) > 0
or len(allowed_mcp_servers_for_team) > 0
)
if len(allowed_mcp_servers_for_team) > 0:
if len(allowed_mcp_servers_for_key) > 0:
# Key has its own MCP permissions - use intersection with team permissions
for _mcp_server in allowed_mcp_servers_for_key:
if _mcp_server in allowed_mcp_servers_for_team:
allowed_mcp_servers.append(_mcp_server)
else:
# Key has no MCP permissions - inherit from team
allowed_mcp_servers = allowed_mcp_servers_for_team
key_set = set(allowed_mcp_servers_for_key)
team_set = set(allowed_mcp_servers_for_team)
extras_set = set(key_access_group_extras)
has_lower_level_mcp_restrictions = bool(key_set or team_set or extras_set)
# 1. Team-gated base scope.
if not team_set:
base = key_set # no team restriction
elif not key_set:
base = team_set # key has no own perms → inherits team
else:
allowed_mcp_servers = allowed_mcp_servers_for_key
base = key_set & team_set # both restrict → intersect
# 2. Extend with access-group extras (LIT-3189 — bypasses team
# ceiling, gated by group's assigned_team_ids / assigned_key_ids).
allowed_mcp_servers: List[str] = list(base | extras_set)
#########################################################
# Check end_user permissions if end_user_id is set
@ -873,6 +884,43 @@ class MCPRequestHandler:
return True
return False
@staticmethod
async def _get_key_access_group_mcp_server_extras(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> List[str]:
"""
Resolve the key's unified `access_group_ids` (LiteLLM_AccessGroupTable) to
MCP server IDs, gated by the access group's `assigned_team_ids` /
`assigned_key_ids`. These servers extend the team's MCP scope rather
than being capped by it. Tag-style `mcp_access_groups` (per-server tags)
are intentionally not handled here — they have no assignment fields and
remain subject to the team ceiling.
"""
if user_api_key_auth is None:
return []
try:
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.proxy.auth.auth_checks import (
get_authorized_resources_from_key_access_groups,
)
raw_server_ids = await get_authorized_resources_from_key_access_groups(
valid_token=user_api_key_auth,
team_object=None,
resource_field="access_mcp_server_ids",
)
if not raw_server_ids:
return []
# Permission entries may be server_ids OR names/aliases — expand to ids.
return global_mcp_server_manager.expand_permission_list(raw_server_ids)
except Exception as e:
verbose_logger.warning(
f"Failed to get key access group MCP server extras: {str(e)}"
)
return []
@staticmethod
async def _get_allowed_mcp_servers_for_key(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
@ -944,42 +992,78 @@ class MCPRequestHandler:
"""
Get allowed MCP servers for a team.
Note: object_permission is automatically loaded by get_team_object() in main auth flow.
Unions two sources:
- Legacy team.object_permission (mcp_servers, mcp_access_groups,
mcp_tool_permissions).
- Unified team.access_group_ids → access_group.access_mcp_server_ids.
Mirrors the model-side pattern in can_team_access_model — the group
is already attached to the team, so the team relationship is itself
the gate (no assigned_team_ids check needed here).
"""
try:
# Get team object permission (already loaded in main auth flow)
object_permissions = await MCPRequestHandler._get_team_object_permission(
user_api_key_auth
)
if object_permissions is None:
return []
# Permission entries may be server_ids OR names/aliases — expand to ids.
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.proxy.auth.auth_checks import (
_get_mcp_server_ids_from_access_groups,
get_team_object,
)
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if (
user_api_key_auth is None
or not user_api_key_auth.team_id
or prisma_client is None
):
return []
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
team_id=user_api_key_auth.team_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
if team_obj is None:
return []
team_access_group_servers = await _get_mcp_server_ids_from_access_groups(
access_group_ids=team_obj.access_group_ids or [],
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
object_permissions = team_obj.object_permission
if object_permissions is None:
return list(set(team_access_group_servers))
direct_mcp_servers = global_mcp_server_manager.expand_permission_list(
object_permissions.mcp_servers or []
)
# Get MCP servers from access groups
access_group_servers = (
legacy_access_group_servers = (
await MCPRequestHandler._get_mcp_servers_from_access_groups(
object_permissions.mcp_access_groups or []
)
)
# servers referenced in tool permissions should also be accessible
tool_perm_servers = list(
global_mcp_server_manager.expand_tool_permissions(
object_permissions.mcp_tool_permissions
).keys()
)
# Combine all lists
all_servers = direct_mcp_servers + access_group_servers + tool_perm_servers
all_servers = (
direct_mcp_servers
+ legacy_access_group_servers
+ tool_perm_servers
+ team_access_group_servers
)
return list(set(all_servers))
except Exception as e:
verbose_logger.warning(

100
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -1,3 +1,4 @@
import html as _html
import json
from typing import Any, Dict, Optional
from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
@ -618,8 +619,105 @@ async def token_endpoint(
)
# Per RFC 6749 §4.1.2.1, an IdP that rejects an OAuth authorization request
# redirects back to the configured redirect URI with ``error`` /
# ``error_description`` / ``error_uri`` query params and no ``code``. The MCP
# loopback flow funnels that response through this /callback endpoint, so
# the endpoint must accept either a successful (``code``+``state``) or an
# error response. Declaring ``code``/``state`` as required would cause
# FastAPI to reject the error response with a 422 before the handler runs,
# which strands the MCP client waiting on the loopback (see LIT-2750).
def _render_oauth_error_html(error: str, description: Optional[str]) -> HTMLResponse:
"""Render an actionable HTML page for an IdP-reported OAuth error.
Used when we cannot propagate the error back to the registered
``redirect_uri`` (state missing or undecryptable). Returned with a 400
status so the failure is observable to operators while still being a
human-readable page for the end user.
"""
safe_error = _html.escape(error or "unknown_error")
safe_description = _html.escape(description) if description else ""
description_html = f"<p>{safe_description}</p>" if safe_description else ""
body = (
"<html><body>"
"<h2>Authentication failed</h2>"
f"<p><strong>Error:</strong> {safe_error}</p>"
f"{description_html}"
"<p>You can close this window and try again.</p>"
"</body></html>"
)
return HTMLResponse(body, status_code=400)
@router.get("/callback")
async def callback(request: Request, code: str, state: str):
async def callback(
request: Request,
code: Optional[str] = None,
state: Optional[str] = None,
error: Optional[str] = None,
error_description: Optional[str] = None,
error_uri: Optional[str] = None,
):
"""OAuth 2.0 authorization response handler for MCP loopback clients.
Accepts either:
- A successful authorization response (``code`` + ``state``), which is
forwarded back to the validated client ``redirect_uri`` with the
original (un-wrapped) ``state``.
- An error response (``error``[+``error_description``/``error_uri``]), per
RFC 6749 §4.1.2.1. When ``state`` is present and decodes to a trusted
``redirect_uri``, the error params are propagated back to the client so
its OAuth library can surface them. Otherwise we render an HTML error
page so the user is not left on an opaque 422 / blank screen.
"""
# 1. IdP-reported error path (e.g. ``?error=access_denied``).
if error:
verbose_logger.info(
"MCP /callback received IdP error: error=%s, error_description=%s",
error,
error_description,
)
if state:
try:
state_data = decode_state_hash(state)
original_state = state_data.get("original_state")
redirect_uri = _get_validated_client_redirect_uri(request, state_data)
except HTTPException:
# Untrusted/invalid client redirect_uri — surface inline rather
# than blindly forwarding the error to an attacker-controlled URL.
return _render_oauth_error_html(error, error_description)
except Exception:
# State could not be decrypted (expired key, tampered, etc.).
return _render_oauth_error_html(error, error_description)
params: Dict[str, str] = {"error": error}
if error_description:
params["error_description"] = error_description
if error_uri:
params["error_uri"] = error_uri
if original_state is not None:
params["state"] = original_state
complete_returned_url = _append_query_params(redirect_uri, params)
return RedirectResponse(url=complete_returned_url, status_code=302)
# No state — nothing to round-trip to. Show the user the error.
return _render_oauth_error_html(error, error_description)
# 2. Neither success nor error parameters present — most likely a stray
# GET / dropped SSO redirect chain. Surface a 400 instead of 422.
if not code or not state:
missing = [
name for name, value in (("code", code), ("state", state)) if not value
]
return _render_oauth_error_html(
"invalid_request",
f"Missing authorization {' and '.join(repr(m) for m in missing)} parameter(s).",
)
# 3. Successful authorization response.
try:
state_data = decode_state_hash(state)
original_state = state_data["original_state"]

3
litellm/proxy/_types.py
View file

@ -255,6 +255,7 @@ class KeyManagementRoutes(str, enum.Enum):
# team spend-log viewing
SPEND_LOGS = "/spend/logs"
SPEND_LOGS_V2 = "/spend/logs/v2"
class LiteLLMRoutes(enum.Enum):
@ -548,6 +549,7 @@ class LiteLLMRoutes(enum.Enum):
KeyManagementRoutes.TEAM_KEY_BULK_UPDATE.value,
KeyManagementRoutes.TEAM_DAILY_ACTIVITY.value,
KeyManagementRoutes.SPEND_LOGS.value,
KeyManagementRoutes.SPEND_LOGS_V2.value,
KeyManagementRoutes.KEY_RESET_SPEND.value,
KeyManagementRoutes.KEY_ALIASES.value,
]
@ -599,6 +601,7 @@ class LiteLLMRoutes(enum.Enum):
"/spend/tags",
"/spend/calculate",
"/spend/logs",
"/spend/logs/v2",
"/spend/logs/ui",
"/spend/logs/session/ui",
"/cost/estimate",

99
litellm/proxy/auth/auth_checks.py
View file

@ -619,6 +619,9 @@ async def common_checks( # noqa: PLR0915
proxy_logging_obj=proxy_logging_obj,
)
# Run before apply_key_tags_pre_auth injects key metadata.tags into request_body.
_reject_clientside_metadata_tags_check(general_settings, request_body, route)
# If this is a free model, skip all budget checks
if not skip_budget_checks:
# 3. If team is in budget
@ -660,6 +663,14 @@ async def common_checks( # noqa: PLR0915
proxy_logging_obj=proxy_logging_obj,
)
if valid_token is not None:
from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
LiteLLMProxyRequestSetup.apply_key_tags_pre_auth(
request_data=request_body,
user_api_key_dict=valid_token,
)
with tracer.trace("litellm.proxy.auth.common_checks.tag_max_budget_check"):
await _tag_max_budget_check(
request_body=request_body,
@ -709,7 +720,6 @@ async def common_checks( # noqa: PLR0915
await _check_end_user_budget(end_user_obj=end_user_object, route=route)
_enforce_user_param_check(general_settings, request, request_body, route)
_reject_clientside_metadata_tags_check(general_settings, request_body, route)
_global_proxy_budget_check(global_proxy_spend, skip_budget_checks, route)
_guardrail_modification_check(request_body, team_object)
@ -1765,19 +1775,39 @@ async def _cache_team_object(
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: Optional[ProxyLogging],
):
key = "team_id:{}".format(team_id)
## CACHE REFRESH TIME!
team_table.last_refreshed_at = time.time()
# team_id is the table primary key — guaranteed unique, safe to write.
await _cache_management_object(
key=key,
key="team_id:{}".format(team_id),
value=team_table,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
model_type=LiteLLM_TeamTableCachedObj,
)
# Invalidate the alias-keyed cache so the JWT auth path with
# `team_alias_jwt_field` (which reads via `get_team_object_by_alias`)
# doesn't keep serving the pre-mutation team after every team-write
# endpoint (team_model_add, team_model_delete, update_team, etc.).
#
# Why DELETE and not WRITE: `team_alias` has no UNIQUE constraint in
# schema.prisma. Writing this cache from the generic refresh path
# would let a team admin who renamed their team to collide with
# another team's alias silently overwrite the cached team for
# JWT-by-alias auth (veria-ai review on #28739). Deleting forces the
# next reader through `get_team_object_by_alias`, which DOES enforce
# uniqueness (len(teams) > 1 raises HTTPException) before populating
# the cache from a verified single row.
if team_table.team_alias:
alias_key = "team_alias:{}".format(team_table.team_alias)
user_api_key_cache.delete_cache(key=alias_key)
if proxy_logging_obj is not None:
await proxy_logging_obj.internal_usage_cache.dual_cache.async_delete_cache(
key=alias_key
)
async def _cache_key_object(
hashed_token: str,
@ -3143,44 +3173,40 @@ async def can_team_access_model(
raise
async def _key_access_group_grants_model(
model: Union[str, List[str]],
async def get_authorized_resources_from_key_access_groups(
valid_token: Optional[UserAPIKeyAuth],
team_object: Optional[LiteLLM_TeamTable],
llm_router: Optional[Router],
) -> bool:
resource_field: Literal[
"access_model_names", "access_mcp_server_ids", "access_agent_ids"
],
) -> List[str]:
"""
Returns True if the key's `access_group_ids` expand to models that grant
access to `model`. Used to let a key's access group override a team's
model restriction in `common_checks`.
A key's access group only counts if the access group itself authorizes the
caller as an owner — that is, the group's `assigned_team_ids` includes the
key's `team_id`, or the group's `assigned_key_ids` includes the key's
token. This preserves the team-as-owner boundary (a team member cannot
escalate by naming a group assigned to a different team) while still
letting a group reach the key without first being added to the team's
`access_group_ids` list.
For each access_group_id on the key, fetch the LiteLLM_AccessGroupTable row
and contribute its `resource_field` only if the group authorizes the caller
as an owner — that is, the group's `assigned_team_ids` includes the key's
`team_id`, or the group's `assigned_key_ids` includes the key's token. This
preserves the team-as-owner boundary while still letting a group reach the
key without first being added to the team's `access_group_ids` list.
"""
if valid_token is None:
return False
return []
key_access_group_ids = list(valid_token.access_group_ids or [])
if not key_access_group_ids:
return False
return []
from litellm.proxy.proxy_server import prisma_client as _prisma_client
from litellm.proxy.proxy_server import proxy_logging_obj as _proxy_logging_obj
from litellm.proxy.proxy_server import user_api_key_cache as _user_api_key_cache
if _prisma_client is None or _user_api_key_cache is None:
return False
return []
key_team_id = valid_token.team_id or (
team_object.team_id if team_object is not None else None
)
key_token = valid_token.token
authorized_models: List[str] = []
authorized_resources: List[str] = []
for ag_id in key_access_group_ids:
try:
ag = await get_access_object(
@ -3196,17 +3222,36 @@ async def _key_access_group_grants_model(
)
key_authorized = bool(key_token and key_token in (ag.assigned_key_ids or []))
if team_authorized or key_authorized:
authorized_models.extend(ag.access_model_names or [])
authorized_resources.extend(getattr(ag, resource_field, []) or [])
return list(set(authorized_resources))
async def _key_access_group_grants_model(
model: Union[str, List[str]],
valid_token: Optional[UserAPIKeyAuth],
team_object: Optional[LiteLLM_TeamTable],
llm_router: Optional[Router],
) -> bool:
"""
Returns True if the key's `access_group_ids` expand to models that grant
access to `model`. Used to let a key's access group override a team's
model restriction in `common_checks`.
"""
authorized_models = await get_authorized_resources_from_key_access_groups(
valid_token=valid_token,
team_object=team_object,
resource_field="access_model_names",
)
if not authorized_models:
return False
try:
_can_object_call_model(
model=model,
llm_router=llm_router,
models=list(set(authorized_models)),
team_model_aliases=valid_token.team_model_aliases,
team_id=valid_token.team_id,
models=authorized_models,
team_model_aliases=valid_token.team_model_aliases if valid_token else None,
team_id=valid_token.team_id if valid_token else None,
object_type="key",
)
return True

19
litellm/proxy/auth/auth_utils.py
View file

@ -213,6 +213,12 @@ _EXTRA_BANNED_OBSERVABILITY_PARAMS: FrozenSet[str] = frozenset(
{
"posthog_api_url",
"phoenix_project_name",
"phoenix_project_name_override",
# Server-reserved: written exclusively by add_user_api_key_auth_to_request_metadata
# from the authenticated key's database record. A caller-supplied value
# would survive the server merge and let an authenticated user redirect
# their Arize/Phoenix telemetry into arbitrary projects.
"user_api_key_auth_metadata",
"wandb_api_key",
"weave_project_id",
}
@ -498,9 +504,18 @@ def route_in_additonal_public_routes(current_route: str):
def get_request_route(request: Request) -> str:
"""
Helper to get the route from the request
Resolve the request route from the ASGI scope, with ``root_path`` stripped.
remove base url from path if set e.g. `/genai/chat/completions` -> `/chat/completions
Prefer this over ``request.url.path`` for any auth, ACL, routing, or
audit-log decision: Starlette reconstructs ``url.path`` by interpolating
the Host header into a URL string and re-parsing with ``urlsplit``, so a
malformed Host (e.g. ``localhost/?x=1``) collapses ``url.path`` to ``"/"``
while FastAPI continues to dispatch on ``scope["path"]``. ``scope["path"]``
is uvicorn's parse of the HTTP request line and matches the actual
handler, so it's the authoritative route.
Also normalizes sub-path deployments by stripping ``scope["root_path"]``
e.g. ``/genai/chat/completions`` -> ``/chat/completions``.
"""
try:
scope = request.scope

3
litellm/proxy/auth/model_checks.py
View file

@ -14,6 +14,9 @@ from litellm.utils import get_valid_models
_CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields)
_CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields)
def _check_wildcard_routing(model: str) -> bool:
"""
Returns True if a model is a provider wildcard.

58
litellm/proxy/auth/route_checks.py
View file

@ -62,7 +62,11 @@ _PROXY_ADMIN_VIEW_ONLY_BLOCKED_KEY_SUFFIXES = ("/regenerate", "/reset_spend")
class RouteChecks:
@staticmethod
def should_call_route(route: str, valid_token: UserAPIKeyAuth):
def should_call_route(
route: str,
valid_token: UserAPIKeyAuth,
request: Optional[Request] = None,
):
"""
Check if management route is disabled and raise exception
"""
@ -77,13 +81,15 @@ class RouteChecks:
# Check if Virtual Key is allowed to call the route - Applies to all Roles
RouteChecks.is_virtual_key_allowed_to_call_route(
route=route, valid_token=valid_token
route=route, valid_token=valid_token, request=request
)
return True
@staticmethod
def is_virtual_key_allowed_to_call_route(
route: str, valid_token: UserAPIKeyAuth
route: str,
valid_token: UserAPIKeyAuth,
request: Optional[Request] = None,
) -> bool:
"""
Raises Exception if Virtual Key is not allowed to call the route
@ -130,6 +136,21 @@ class RouteChecks:
):
return True
# Method-aware carve-out: allow GET on the two
# read-only MCP-server discovery endpoints
# (`/v1/mcp/server` and `/v1/mcp/server/{server_id}`)
# so virtual keys with allowed_routes=["llm_api_routes"]
# can list/inspect MCP servers. The GET handlers in
# mcp_management_endpoints.py sanitize the response
# for restricted virtual keys (stripping url,
# headers, env, credentials). POST/PUT/DELETE on
# these paths are admin-only management writes and
# are intentionally not covered.
if RouteChecks._is_get_mcp_server_discovery_route(
route=route, request=request
):
return True
# check if wildcard pattern is allowed
for allowed_route in valid_token.allowed_routes:
if RouteChecks._route_matches_wildcard_pattern(
@ -401,6 +422,31 @@ class RouteChecks:
return True
return False
@staticmethod
def _is_get_mcp_server_discovery_route(
route: str, request: Optional[Request]
) -> bool:
"""
Returns True if `request` is a GET against one of the two read-only
MCP-server discovery paths:
- GET `/v1/mcp/server` (list)
- GET `/v1/mcp/server/{server_id}` (single server, single segment)
Multi-segment paths (`/v1/mcp/server/{id}/approve`, etc.) and any
non-GET method return False, so admin-only management writes on the
same path prefix are not reachable through this carve-out.
"""
if request is None or request.method.upper() != "GET":
return False
if route == "/v1/mcp/server":
return True
prefix = "/v1/mcp/server/"
if not route.startswith(prefix):
return False
remainder = route[len(prefix) :]
return bool(remainder) and "/" not in remainder
@staticmethod
def is_management_route(route: str) -> bool:
"""
@ -627,7 +673,11 @@ class RouteChecks:
Returns:
bool: True if `thread` or `assistant` is in the request path, False otherwise
"""
if "thread" in request.url.path or "assistant" in request.url.path:
# Inline import — auth_utils participates in a proxy import cycle.
from .auth_utils import get_request_route # noqa: PLC0415
route = get_request_route(request)
if "thread" in route or "assistant" in route:
return True
return False

4
litellm/proxy/auth/user_api_key_auth.py
View file

@ -2200,7 +2200,9 @@ async def user_api_key_auth(
user_api_key_auth_obj.budget_reservation = None
## ENSURE DISABLE ROUTE WORKS ACROSS ALL USER AUTH FLOWS ##
RouteChecks.should_call_route(route=route, valid_token=user_api_key_auth_obj)
RouteChecks.should_call_route(
route=route, valid_token=user_api_key_auth_obj, request=request
)
# Single authorization point. Builder paths MUST NOT call common_checks.
# Route through the same exception handler the builder uses so

6
litellm/proxy/batches_endpoints/endpoints.py
View file

@ -15,6 +15,9 @@ from litellm.batches.main import CancelBatchRequest, RetrieveBatchRequest
from litellm.proxy._types import *
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
from litellm.proxy.common_utils.callback_utils import (
sanitize_openai_provider_metadata,
)
from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
from litellm.proxy.common_utils.openai_endpoint_utils import (
get_custom_llm_provider_from_request_headers,
@ -120,6 +123,9 @@ async def create_batch( # noqa: PLR0915
or get_custom_llm_provider_from_request_headers(request=request)
or "openai"
)
if isinstance(data.get("metadata"), dict):
data["metadata"] = sanitize_openai_provider_metadata(data["metadata"])
_create_batch_data = LiteLLMBatchCreateRequest(**data)
# Apply team-level batch output expiry enforcement

101
litellm/proxy/common_request_processing.py
View file

@ -839,6 +839,7 @@ class ProxyBaseLLMRequestProcessing:
"aget_run",
"acancel_run",
"adelete_run",
"apply_guardrail",
],
version: Optional[str] = None,
user_model: Optional[str] = None,
@ -1368,6 +1369,21 @@ class ProxyBaseLLMRequestProcessing:
user_api_key_dict=user_api_key_dict,
request_data=self.data,
)
if route_type == "aresponses":
# Streaming /v1/responses returns here without
# reaching the non-streaming ownership tail below.
# Wrap the SSE generator so container ownership is
# written once the upstream iterator finishes
# assembling ``completed_response`` — otherwise
# code-interpreter containers created during the
# stream stay unregistered and follow-up file API
# calls 403. Covers the background-polling path
# too, which loops ``body_iterator`` end-to-end.
selected_data_generator = ProxyBaseLLMRequestProcessing._wrap_responses_stream_for_container_ownership(
original_stream_response=response,
wrapped_generator=selected_data_generator,
user_api_key_dict=user_api_key_dict,
)
return await create_response(
generator=selected_data_generator,
media_type="text/event-stream",
@ -1483,8 +1499,93 @@ class ProxyBaseLLMRequestProcessing:
await check_response_size_is_safe(response=response)
if route_type in {"aresponses", "aget_responses"}:
await ProxyBaseLLMRequestProcessing._record_container_owners_from_responses_if_needed(
response=response,
user_api_key_dict=user_api_key_dict,
)
return response
@staticmethod
async def _record_container_owners_from_responses_if_needed(
response: Any,
user_api_key_dict: UserAPIKeyAuth,
) -> None:
"""Register code-interpreter containers so follow-up file APIs pass ownership checks."""
from litellm.proxy.container_endpoints.ownership import (
record_container_owners_from_responses_response,
)
if response is None:
return
try:
await record_container_owners_from_responses_response(
response=response,
user_api_key_dict=user_api_key_dict,
)
except Exception as e:
verbose_proxy_logger.exception(
"Container ownership recording failed after responses call: %s",
e,
)
@staticmethod
def _extract_completed_responses_response(stream_response: Any) -> Any:
"""Pull the assembled ``ResponsesAPIResponse`` off a streaming iterator.
``ResponsesAPIStreamingIterator`` stores the terminal stream event
(``response.completed`` / ``response.incomplete`` / ``response.failed``)
in ``completed_response``; the actual response body hangs off
that event's ``.response`` attribute. Some iterators store the
``ResponsesAPIResponse`` directly. Handle both shapes so the
container-ownership recording path can walk ``.output`` either way.
"""
completed = getattr(stream_response, "completed_response", None)
if completed is None:
return None
response_obj = getattr(completed, "response", None)
if response_obj is not None:
return response_obj
return completed
@staticmethod
async def _wrap_responses_stream_for_container_ownership(
original_stream_response: Any,
wrapped_generator: Any,
user_api_key_dict: UserAPIKeyAuth,
):
"""Forward SSE chunks, then record container ownership at stream end.
Streaming ``/v1/responses`` short-circuits out of
``base_process_llm_request`` before the non-streaming ownership
tail runs, so without this wrap the
``LiteLLM_ManagedObjectTable`` row for any container created
during the stream is never written and follow-up file API calls
return 403.
"""
try:
async for chunk in wrapped_generator:
yield chunk
finally:
try:
completed_obj = (
ProxyBaseLLMRequestProcessing._extract_completed_responses_response(
original_stream_response
)
)
if completed_obj is not None:
await ProxyBaseLLMRequestProcessing._record_container_owners_from_responses_if_needed(
response=completed_obj,
user_api_key_dict=user_api_key_dict,
)
except Exception as e:
verbose_proxy_logger.exception(
"Container ownership recording failed after streaming responses call: %s",
e,
)
async def base_passthrough_process_llm_request(
self,
request: Request,

138
litellm/proxy/common_utils/callback_utils.py
View file

@ -317,7 +317,15 @@ def initialize_callbacks_on_proxy( # noqa: PLR0915
DatadogCostManagementLogger,
)
datadog_cost_management_obj = DatadogCostManagementLogger()
init_params = {}
if (
"datadog_cost_management" in callback_specific_params
and isinstance(
callback_specific_params["datadog_cost_management"], dict
)
):
init_params = callback_specific_params["datadog_cost_management"]
datadog_cost_management_obj = DatadogCostManagementLogger(**init_params)
imported_list.append(datadog_cost_management_obj)
elif isinstance(callback, CustomLogger):
imported_list.append(callback)
@ -409,11 +417,15 @@ def get_remaining_tokens_and_requests_from_request_data(data: Dict) -> Dict[str,
def get_logging_caching_headers(request_data: Dict) -> Optional[Dict]:
_metadata = request_data.get("metadata", None)
if not _metadata:
_metadata = request_data.get("litellm_metadata", None)
if not isinstance(_metadata, dict):
_metadata = {}
_metadata: Dict = {}
metadata_bucket = request_data.get("metadata")
litellm_metadata_bucket = request_data.get("litellm_metadata")
if isinstance(metadata_bucket, dict):
_metadata.update(metadata_bucket)
if isinstance(litellm_metadata_bucket, dict):
# Batch/file routes store proxy tracking in litellm_metadata while
# user-facing metadata stays in metadata; merge both for headers.
_metadata.update(litellm_metadata_bucket)
headers = {}
if "applied_guardrails" in _metadata:
headers["x-litellm-applied-guardrails"] = ",".join(
@ -452,19 +464,103 @@ def get_logging_caching_headers(request_data: Dict) -> Optional[Dict]:
return headers
def get_metadata_variable_name_from_kwargs(
kwargs: dict,
) -> Literal["metadata", "litellm_metadata"]:
"""
Helper to return what the "metadata" field should be called in the request data
- New endpoints return `litellm_metadata`
- Old endpoints return `metadata`
Context:
- LiteLLM used `metadata` as an internal field for storing metadata
- OpenAI then started using this field for their metadata
- LiteLLM is now moving to using `litellm_metadata` for our metadata
"""
return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata"
LITELLM_PROXY_INTERNAL_METADATA_KEYS = frozenset(
{
"applied_policies",
"applied_guardrails",
"policy_sources",
"guardrails",
"guardrail_config",
"_guardrail_pipelines",
"_pipeline_managed_guardrails",
"disable_global_guardrails",
"disable_global_guardrail",
"opted_out_global_guardrails",
"pillar_response_headers",
"_pillar_response_headers_trusted",
"pillar_flagged",
"pillar_scanners",
"pillar_evidence",
"pillar_evidence_truncated",
"pillar_session_id_response",
"standard_logging_object",
"proxy_server_request",
"secret_fields",
}
)
def _get_or_create_proxy_metadata_bucket(
request_data: Dict,
) -> tuple[Literal["metadata", "litellm_metadata"], dict]:
"""
Return the proxy-internal metadata bucket for this request.
Batch/file routes store proxy state in ``litellm_metadata`` so the OpenAI
``metadata`` field can remain provider-safe (string values only).
"""
metadata_key = get_metadata_variable_name_from_kwargs(request_data)
metadata_bucket = request_data.get(metadata_key)
if not isinstance(metadata_bucket, dict):
metadata_bucket = {}
request_data[metadata_key] = metadata_bucket
return metadata_key, metadata_bucket
def sanitize_openai_provider_metadata(
metadata: Optional[Dict[str, Any]],
) -> Optional[Dict[str, str]]:
"""
Keep only provider-safe OpenAI metadata entries (string keys -> string values).
Strips LiteLLM proxy-internal tracking fields that must not be forwarded to
OpenAI batch/file APIs.
"""
if not metadata:
return metadata
sanitized: Dict[str, str] = {}
for key, value in metadata.items():
if key in LITELLM_PROXY_INTERNAL_METADATA_KEYS:
continue
if isinstance(value, str):
sanitized[key] = value
else:
verbose_proxy_logger.debug(
"sanitize_openai_provider_metadata: dropping key %r with non-string value of type %s",
key,
type(value).__name__,
)
return sanitized or None
def add_guardrail_to_applied_guardrails_header(
request_data: Dict, guardrail_name: Optional[str]
):
if guardrail_name is None:
return
_metadata = request_data.get("metadata", None) or {}
_, _metadata = _get_or_create_proxy_metadata_bucket(request_data)
if "applied_guardrails" in _metadata:
if guardrail_name not in _metadata["applied_guardrails"]:
_metadata["applied_guardrails"].append(guardrail_name)
else:
_metadata["applied_guardrails"] = [guardrail_name]
# Ensure metadata is set back to request_data (important when metadata didn't exist)
request_data["metadata"] = _metadata
def add_policy_to_applied_policies_header(
@ -478,14 +574,12 @@ def add_policy_to_applied_policies_header(
"""
if policy_name is None:
return
_metadata = request_data.get("metadata", None) or {}
_, _metadata = _get_or_create_proxy_metadata_bucket(request_data)
if "applied_policies" in _metadata:
if policy_name not in _metadata["applied_policies"]:
_metadata["applied_policies"].append(policy_name)
else:
_metadata["applied_policies"] = [policy_name]
# Ensure metadata is set back to request_data (important when metadata didn't exist)
request_data["metadata"] = _metadata
def add_policy_sources_to_metadata(request_data: Dict, policy_sources: Dict[str, str]):
@ -498,13 +592,12 @@ def add_policy_sources_to_metadata(request_data: Dict, policy_sources: Dict[str,
"""
if not policy_sources:
return
_metadata = request_data.get("metadata", None) or {}
_, _metadata = _get_or_create_proxy_metadata_bucket(request_data)
existing = _metadata.get("policy_sources", {})
if not isinstance(existing, dict):
existing = {}
existing.update(policy_sources)
_metadata["policy_sources"] = existing
request_data["metadata"] = _metadata
def add_guardrail_response_to_standard_logging_object(
@ -527,23 +620,6 @@ def add_guardrail_response_to_standard_logging_object(
return standard_logging_object
def get_metadata_variable_name_from_kwargs(
kwargs: dict,
) -> Literal["metadata", "litellm_metadata"]:
"""
Helper to return what the "metadata" field should be called in the request data
- New endpoints return `litellm_metadata`
- Old endpoints return `metadata`
Context:
- LiteLLM used `metadata` as an internal field for storing metadata
- OpenAI then started using this field for their metadata
- LiteLLM is now moving to using `litellm_metadata` for our metadata
"""
return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata"
def process_callback(
_callback: str, callback_type: str, environment_variables: dict
) -> dict:

5
litellm/proxy/common_utils/http_parsing_utils.py
View file

@ -546,7 +546,10 @@ def _add_vector_store_id_from_path(request_data: dict, request: Request) -> None
request_data: The request data dictionary to populate
request: The FastAPI Request object
"""
path = request.url.path
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
path = get_request_route(request)
vector_store_match = re.search(r"/vector_stores/([^/]+)/", path)
if vector_store_match:
vector_store_id = vector_store_match.group(1)

58
litellm/proxy/container_endpoints/ownership.py
View file

@ -117,6 +117,58 @@ async def _get_prisma_client():
return prisma_client
def _custom_llm_provider_from_responses_response(
response: Any,
default: str = "openai",
) -> str:
hidden_params: Dict[str, Any] = {}
if isinstance(response, dict):
hidden_params = response.get("_hidden_params") or {}
else:
hidden_params = getattr(response, "_hidden_params", None) or {}
provider = hidden_params.get("custom_llm_provider")
if isinstance(provider, str) and provider:
return provider
return default
async def record_container_owners_from_responses_response(
response: Any,
user_api_key_dict: UserAPIKeyAuth,
custom_llm_provider: Optional[str] = None,
) -> None:
"""Track containers created implicitly by code interpreter in /v1/responses."""
container_ids = (
ResponsesAPIRequestUtils.collect_container_ids_from_responses_response(response)
)
if not container_ids:
return
resolved_provider = (
custom_llm_provider or _custom_llm_provider_from_responses_response(response)
)
for container_id in container_ids:
try:
await record_container_owner(
response={"id": container_id, "object": "container"},
user_api_key_dict=user_api_key_dict,
custom_llm_provider=resolved_provider,
)
except Exception as e:
# Per-container errors (including ``HTTPException`` from
# conflicting/forbidden ownership rows) must not abort the
# batch — other containers in the same response should still
# get recorded so their follow-up file API calls don't 403.
verbose_proxy_logger.exception(
"Failed to record container ownership from responses output "
"for container_id=%s: %s",
container_id,
e,
)
async def record_container_owner(
response: Any,
user_api_key_dict: UserAPIKeyAuth,
@ -151,6 +203,8 @@ async def record_container_owner(
file_object = _dump_response(response)
file_object["custom_llm_provider"] = resolved_provider
file_object["provider_container_id"] = original_container_id
# Prisma Python requires Json fields to be serialized as a JSON string.
file_object_json: str = json.dumps(file_object)
prisma_client = await _get_prisma_client()
if prisma_client is None:
@ -172,7 +226,7 @@ async def record_container_owner(
where={"model_object_id": model_object_id},
data={
"unified_object_id": container_id,
"file_object": file_object,
"file_object": file_object_json,
"updated_by": owner,
},
)
@ -181,7 +235,7 @@ async def record_container_owner(
data={
"unified_object_id": container_id,
"model_object_id": model_object_id,
"file_object": file_object,
"file_object": file_object_json,
"file_purpose": CONTAINER_OBJECT_PURPOSE,
"created_by": owner,
"updated_by": owner,

4
litellm/proxy/example_config_yaml/oai_misc_config.yaml
View file

@ -23,11 +23,11 @@ model_list:
model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
#########################################################
########## batch specific params ########################
s3_bucket_name: litellm-proxy
s3_bucket_name: litellm-proxy-123456789012
s3_region_name: us-west-2
s3_access_key_id: os.environ/AWS_ACCESS_KEY_ID
s3_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_batch_role_arn: arn:aws:iam::888602223428:role/service-role/AmazonBedrockExecutionRoleForAgents_BB9HNW6V4CV
aws_batch_role_arn: arn:aws:iam::123456789012:role/service-role/AmazonBedrockExecutionRoleForAgents_EXAMPLE
model_info:
mode: batch

2
litellm/proxy/example_config_yaml/otel_test_config.yaml
View file

@ -55,7 +55,7 @@ guardrails:
litellm_params:
guardrail: bedrock # supported values: "bedrock", "lakera"
mode: "during_call"
guardrailIdentifier: ff6ujrregl1q
guardrailIdentifier: 4w3d1di3snt5
guardrailVersion: "DRAFT"
- guardrail_name: "custom-pre-guard"
litellm_params:

189
litellm/proxy/guardrails/guardrail_endpoints.py
View file

@ -10,7 +10,7 @@ from datetime import datetime, timezone
from typing import Any, Dict, List, Literal, Optional, Type, TypeVar, Union, cast
from urllib.parse import urlparse
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from litellm.proxy.common_utils.path_utils import safe_join
@ -2187,9 +2187,97 @@ async def test_custom_code_guardrail(
)
def _resolve_guardrail_input_type(
active_guardrail: CustomGuardrail, input_type: str
) -> Literal["request", "response"]:
"""Return the effective input_type, auto-upgrading to 'response' for post_call guardrails."""
if input_type == "request":
hook = getattr(active_guardrail, "event_hook", None)
if hook == GuardrailEventHooks.post_call or hook == "post_call":
return "response"
return "response" if input_type == "response" else "request"
def _patch_logging_obj_for_guardrail(
litellm_logging_obj: Any, request: ApplyGuardrailRequest
) -> None:
"""Configure the logging object so Langfuse/OTEL extract input and output correctly."""
litellm_logging_obj.call_type = "pass_through_endpoint"
litellm_logging_obj.model_call_details["call_type"] = "pass_through_endpoint"
litellm_logging_obj.update_messages(
request.messages
if request.messages
else [{"role": "user", "content": request.text}]
)
async def _emit_guardrail_success_logs(
proxy_logging_obj: Any,
litellm_logging_obj: Any,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
response: ApplyGuardrailResponse,
start_time: datetime,
) -> ApplyGuardrailResponse:
"""Fire proxy and LiteLLM success hooks after a successful guardrail run.
Each hook is wrapped defensively so a callback failure never prevents the
caller from receiving the guardrail response. Returns the (possibly
hook-modified) response.
"""
from litellm.litellm_core_utils.thread_pool_executor import (
executor as thread_pool_executor,
)
try:
modified = await proxy_logging_obj.post_call_success_hook(
data=data,
user_api_key_dict=user_api_key_dict,
response=response,
)
if isinstance(modified, ApplyGuardrailResponse):
response = modified
except Exception:
verbose_proxy_logger.exception("apply_guardrail: post_call_success_hook failed")
# Build the logging payload after post_call_success_hook so that logged
# data matches what the caller actually receives if the hook modified
# the response.
response_for_logging = {"response": response.model_dump(exclude_none=True)}
if litellm_logging_obj is not None:
end_time = datetime.now(timezone.utc)
try:
await litellm_logging_obj.async_success_handler(
result=response_for_logging,
start_time=start_time,
end_time=end_time,
cache_hit=False,
)
except Exception:
verbose_proxy_logger.exception(
"apply_guardrail: async_success_handler failed"
)
try:
thread_pool_executor.submit(
litellm_logging_obj.success_handler,
response_for_logging,
start_time,
end_time,
False,
)
except Exception:
verbose_proxy_logger.exception(
"apply_guardrail: success_handler submit failed"
)
return response
@router.post("/guardrails/apply_guardrail", response_model=ApplyGuardrailResponse)
@router.post("/apply_guardrail", response_model=ApplyGuardrailResponse)
async def apply_guardrail(
fastapi_request: Request,
request: ApplyGuardrailRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
@ -2198,8 +2286,29 @@ async def apply_guardrail(
This endpoint allows testing guardrails by applying them to custom text inputs.
"""
import traceback
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
from litellm.litellm_core_utils.thread_pool_executor import (
executor as thread_pool_executor,
)
from litellm.proxy.proxy_server import (
general_settings,
proxy_config,
proxy_logging_obj,
version,
)
from litellm.proxy.utils import handle_exception_on_proxy
data: dict = {
"guardrail_name": request.guardrail_name,
"input": [request.text],
"messages": request.messages or [],
"metadata": {"route": "/apply_guardrail"},
}
litellm_logging_obj = None
start_time = datetime.now(timezone.utc)
try:
active_guardrail: Optional[CustomGuardrail] = (
GUARDRAIL_REGISTRY.get_initialized_guardrail_callback(
@ -2212,23 +2321,25 @@ async def apply_guardrail(
detail=f"Guardrail '{request.guardrail_name}' not found. Please ensure the guardrail is configured in your LiteLLM proxy.",
)
request_data: dict = {}
if request.messages:
request_data["messages"] = request.messages
request_processor = ProxyBaseLLMRequestProcessing(data=data)
data, litellm_logging_obj = (
await request_processor.common_processing_pre_call_logic(
request=fastapi_request,
general_settings=general_settings,
user_api_key_dict=user_api_key_dict,
version=version,
proxy_logging_obj=proxy_logging_obj,
proxy_config=proxy_config,
route_type="apply_guardrail",
)
)
# Auto-detect input_type: if the caller didn't specify "response" but the
# guardrail only runs post_call (e.g. LLM-as-a-judge), use "response" so
# the test actually exercises the guardrail logic.
from litellm.types.guardrails import GuardrailEventHooks
if litellm_logging_obj is not None:
_patch_logging_obj_for_guardrail(litellm_logging_obj, request)
resolved_input_type = request.input_type
if resolved_input_type == "request":
hook = getattr(active_guardrail, "event_hook", None)
if hook == GuardrailEventHooks.post_call or hook == "post_call":
resolved_input_type = "response"
_input_type: Literal["request", "response"] = (
"response" if resolved_input_type == "response" else "request"
request_data: dict = {"messages": request.messages} if request.messages else {}
_input_type = _resolve_guardrail_input_type(
active_guardrail, request.input_type
)
guardrailed_inputs = await active_guardrail.apply_guardrail(
inputs={"texts": [request.text]},
@ -2236,13 +2347,55 @@ async def apply_guardrail(
input_type=_input_type,
)
response_text = guardrailed_inputs.get("texts", [])
return ApplyGuardrailResponse(
response = ApplyGuardrailResponse(
response_text=response_text[0] if response_text else request.text
)
except Exception as e:
if litellm_logging_obj is not None and not isinstance(e, HTTPException):
try:
await litellm_logging_obj.async_failure_handler(
exception=e,
traceback_exception=traceback.format_exc(),
)
except Exception:
verbose_proxy_logger.exception(
"apply_guardrail: async_failure_handler failed"
)
try:
thread_pool_executor.submit(
litellm_logging_obj.failure_handler,
e,
traceback.format_exc(),
)
except Exception:
verbose_proxy_logger.exception(
"apply_guardrail: failure_handler submit failed"
)
try:
transformed_exception = await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
request_data=data,
)
if isinstance(transformed_exception, Exception):
e = transformed_exception
except Exception:
verbose_proxy_logger.exception(
"apply_guardrail: post_call_failure_hook failed"
)
raise handle_exception_on_proxy(e)
# Success logging outside except so a hook error never triggers failure handlers.
response = await _emit_guardrail_success_logs(
proxy_logging_obj=proxy_logging_obj,
litellm_logging_obj=litellm_logging_obj,
data=data,
user_api_key_dict=user_api_key_dict,
response=response,
start_time=start_time,
)
return response
# Usage (dashboard) endpoints: overview, detail, logs
router.include_router(guardrails_usage_router)

5
litellm/proxy/health_endpoints/_health_endpoints.py
View file

@ -151,7 +151,10 @@ async def test_endpoint(request: Request):
dict: A dictionary containing the route of the request URL.
"""
# ping the proxy server to check if its healthy
return {"route": request.url.path}
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
return {"route": get_request_route(request)}
@router.get(

48
litellm/proxy/litellm_pre_call_utils.py
View file

@ -333,8 +333,10 @@ def _get_metadata_variable_name(request: Request) -> str:
For ALL other endpoints we call this "metadata"
"""
path = request.url.path
# Inline imports — auth_utils/route_checks participate in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
path = get_request_route(request)
if "thread" in path or "assistant" in path:
return "litellm_metadata"
@ -1191,6 +1193,36 @@ class LiteLLMProxyRequestSetup:
return tags
@staticmethod
def apply_key_tags_pre_auth(
request_data: dict,
user_api_key_dict: UserAPIKeyAuth,
) -> None:
"""Merge key metadata tags into request_data before _tag_max_budget_check."""
key_metadata = user_api_key_dict.metadata
if not key_metadata:
return
key_tags = key_metadata.get("tags")
if not key_tags or not isinstance(key_tags, list):
return
_metadata_variable_name = get_metadata_variable_name_from_kwargs(request_data)
metadata = request_data.get(_metadata_variable_name)
if isinstance(metadata, str):
parsed = safe_json_loads(metadata)
metadata = parsed if isinstance(parsed, dict) else {}
request_data[_metadata_variable_name] = metadata
elif not isinstance(metadata, dict):
metadata = {}
request_data[_metadata_variable_name] = metadata
existing_tags = metadata.get("tags")
metadata["tags"] = LiteLLMProxyRequestSetup._merge_tags(
request_tags=existing_tags if isinstance(existing_tags, list) else None,
tags_to_add=key_tags,
)
@staticmethod
def apply_client_tag_policy_pre_auth(
request: Request,
@ -1511,10 +1543,16 @@ async def add_litellm_data_to_request( # noqa: PLR0915
# spend_tracking_utils, streaming_iterator) read `body` to audit the
# request; taking the snapshot here ensures they see cleaned metadata.
#
# Exclude secret_fields (which contains raw_headers with Authorization
# tokens) from the snapshot — they must never be persisted in spend logs
# or any other audit trail.
_body_snapshot = {k: v for k, v in data.items() if k != "secret_fields"}
# Exclude:
# - secret_fields: contains raw_headers with Authorization tokens; must
# never be persisted in spend logs or any other audit trail.
# - proxy_server_request: already a key on `data` at this point (set
# earlier in this function); including it would make the snapshot
# self-reference — body.proxy_server_request.body would be the same
# dict as body, producing an infinite traversal loop for any consumer
# that walks the structure.
_body_snapshot_exclude = {"secret_fields", "proxy_server_request"}
_body_snapshot = {k: v for k, v in data.items() if k not in _body_snapshot_exclude}
data["proxy_server_request"]["body"] = _body_snapshot
# Snapshot the requester-supplied metadata for downstream consumers.

11
litellm/proxy/management_endpoints/cache_settings_endpoints.py
View file

@ -19,6 +19,7 @@ from pydantic import BaseModel, Field
import litellm
from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.sensitive_data_masker import mask_sensitive_keys
from litellm.proxy._types import (
AUDIT_ACTIONS,
LiteLLM_AuditLogs,
@ -34,6 +35,10 @@ from litellm.types.management_endpoints import (
router = APIRouter()
# Cache fields holding credentials. Masked on read so plaintext Redis /
# Sentinel passwords never leave the server in a GET response.
_CACHE_SENSITIVE_FIELDS: set = {"password", "sentinel_password"}
_REDACTED_VALUE = "***REDACTED***"
@ -295,7 +300,11 @@ async def get_cache_settings(
else:
decrypted_settings["redis_type"] = "node"
current_values = decrypted_settings
# Mask credential fields so the GET response never carries
# plaintext Redis / Sentinel passwords off the server.
current_values = mask_sensitive_keys(
decrypted_settings, _CACHE_SENSITIVE_FIELDS
)
# Update field values with current values
for field in cache_fields:

5
litellm/proxy/management_endpoints/mcp_management_endpoints.py
View file

@ -1568,6 +1568,9 @@ if MCP_AVAILABLE:
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415
global_mcp_server_manager,
)
from litellm.proxy.auth.auth_utils import ( # noqa: PLC0415
get_request_route,
)
server_id = request.path_params.get("server_id", "")
if server_id:
@ -1584,7 +1587,7 @@ if MCP_AVAILABLE:
):
# For /token, require PKCE authorization_code; refresh_token
# grants must NOT bypass auth (see comment above).
path_lower = (request.url.path or "").rstrip("/").lower()
path_lower = get_request_route(request).rstrip("/").lower()
if path_lower.endswith("/token"):
body_data = await _read_request_body(request=request)
grant_type = (body_data or {}).get("grant_type", "")

27
litellm/proxy/management_endpoints/model_management_endpoints.py
View file

@ -51,6 +51,7 @@ from litellm.types.proxy.management_endpoints.model_management_endpoints import
UpdateUsefulLinksRequest,
)
from litellm.types.router import (
SPECIAL_MODEL_INFO_PARAMS,
Deployment,
DeploymentTypedDict,
LiteLLMParamsTypedDict,
@ -130,6 +131,32 @@ def update_db_model(
updated_patch.model_info.model_dump(exclude_none=True)
)
# Honor explicit-null clears LAST, after both merges, so a model_info blob the UI
# passes through (which today re-sends the OLD pricing on every save) cannot
# silently undo a litellm_params clear via .update().
#
# Restricted to SPECIAL_MODEL_INFO_PARAMS (input/output cost per token/character
# and cache read/write costs) so this path cannot be used to null out privileged
# model_info fields like team_id or access groups. SPECIAL_MODEL_INFO_PARAMS are
# mirrored between litellm_params and model_info by Deployment.__init__, so the
# clear propagates to both blobs.
if updated_patch.litellm_params:
for field in updated_patch.litellm_params.model_fields_set:
if (
field in SPECIAL_MODEL_INFO_PARAMS
and getattr(updated_patch.litellm_params, field) is None
):
merged_deployment_dict["litellm_params"].pop(field, None) # type: ignore
merged_deployment_dict.get("model_info", {}).pop(field, None)
if updated_patch.model_info:
for field in updated_patch.model_info.model_fields_set:
if (
field in SPECIAL_MODEL_INFO_PARAMS
and getattr(updated_patch.model_info, field) is None
):
merged_deployment_dict["model_info"].pop(field, None) # type: ignore
merged_deployment_dict.get("litellm_params", {}).pop(field, None) # type: ignore
# convert to prisma compatible format
prisma_compatible_model_dict = PrismaCompatibleUpdateDBModel()

82
litellm/proxy/management_endpoints/team_endpoints.py
View file

@ -64,6 +64,7 @@ from litellm.proxy._types import (
UserAPIKeyAuth,
)
from litellm.proxy.auth.auth_checks import (
_cache_team_object,
allowed_route_check_inside_route,
can_org_access_model,
get_org_object,
@ -130,6 +131,33 @@ def _sanitize_for_log(value: Any) -> str:
return text.replace("\r", "").replace("\n", "")
async def _refresh_cached_team(
team_row: Any,
user_api_key_cache: Any,
proxy_logging_obj: Any,
) -> None:
"""
Refresh the in-memory cached team object after a DB write.
Every endpoint that mutates `litellm_teamtable` must call this so the
cached `LiteLLM_TeamTableCachedObj` used by `common_checks` stays in
sync. Without this, subsequent auth checks read a stale team and can
403 on permissions the DB has already granted (or, symmetrically,
keep granting permissions the DB has already revoked).
`team_row` is the Prisma row returned by `update`/`find_unique` on
`litellm_teamtable`. It is converted to `LiteLLM_TeamTableCachedObj`
via `model_dump()` to match the cache shape `_cache_team_object`
expects.
"""
await _cache_team_object(
team_id=team_row.team_id,
team_table=LiteLLM_TeamTableCachedObj(**team_row.model_dump()),
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
async def _verify_team_access(
team_obj: LiteLLM_TeamTable,
user_api_key_dict: UserAPIKeyAuth,
@ -1591,7 +1619,6 @@ async def update_team( # noqa: PLR0915
```
"""
try:
from litellm.proxy.auth.auth_checks import _cache_team_object
from litellm.proxy.proxy_server import (
litellm_proxy_admin_name,
llm_router,
@ -1861,7 +1888,13 @@ async def update_team( # noqa: PLR0915
await prisma_client.db.litellm_teamtable.update(
where={"team_id": data.team_id},
data=updated_kv,
include={"litellm_model_table": True}, # type: ignore
# `object_permission` is included so `_refresh_cached_team`
# doesn't write a cached team with the relation nulled out —
# see team_model_add for the full rationale.
include={
"litellm_model_table": True,
"object_permission": True,
}, # type: ignore
)
)
@ -1874,9 +1907,8 @@ async def update_team( # noqa: PLR0915
verbose_proxy_logger.info(
"Successfully updated team - %s, info", team_row.team_id
)
await _cache_team_object(
team_id=team_row.team_id,
team_table=LiteLLM_TeamTableCachedObj(**team_row.model_dump()),
await _refresh_cached_team(
team_row=team_row,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
@ -4569,7 +4601,11 @@ async def team_model_add(
}'
```
"""
from litellm.proxy.proxy_server import prisma_client
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if prisma_client is None:
raise HTTPException(status_code=500, detail={"error": "No db connected"})
@ -4603,9 +4639,21 @@ async def team_model_add(
)
updated_models = add_new_models_to_team(team_obj=team_obj, new_models=data.models)
# Update team
# Update team. `include` mirrors the relations the auth path consumes
# off the cached team object so that `_refresh_cached_team` doesn't
# null them out — see object_permission_utils.validate_key_search_tools_against_team
# and the MCP/agent authz paths, which treat a missing object_permission
# as "no team-level restriction".
updated_team = await prisma_client.db.litellm_teamtable.update(
where={"team_id": data.team_id}, data={"models": updated_models}
where={"team_id": data.team_id},
data={"models": updated_models},
include={"object_permission": True}, # type: ignore
)
await _refresh_cached_team(
team_row=updated_team,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
return updated_team
@ -4640,7 +4688,11 @@ async def team_model_delete(
}'
```
"""
from litellm.proxy.proxy_server import prisma_client
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if prisma_client is None:
raise HTTPException(status_code=500, detail={"error": "No db connected"})
@ -4679,9 +4731,17 @@ async def team_model_delete(
# Remove specified models
updated_models = [m for m in current_models if m not in data.models]
# Update team
# Update team. See team_model_add for the rationale on `include`.
updated_team = await prisma_client.db.litellm_teamtable.update(
where={"team_id": data.team_id}, data={"models": updated_models}
where={"team_id": data.team_id},
data={"models": updated_models},
include={"object_permission": True}, # type: ignore
)
await _refresh_cached_team(
team_row=updated_team,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
return updated_team

149
litellm/proxy/management_helpers/utils.py
View file

@ -2,7 +2,7 @@
## Helper utils for the management endpoints (keys/users/teams)
from datetime import datetime
from functools import wraps
from typing import List, Optional, Tuple
from typing import Any, Callable, List, Optional, Tuple
from fastapi import HTTPException, Request
@ -435,6 +435,85 @@ async def send_management_endpoint_alert(
)
async def _emit_management_endpoint_otel_span(
func: Callable,
kwargs: dict,
parent_otel_span: Any,
start_time: datetime,
end_time: datetime,
result: Any = None,
exception: Optional[Exception] = None,
) -> None:
"""Stamp + end the parent OTEL SERVER span for a management endpoint.
Routes the request/response (or exception) through the OTEL success/failure
hook. Falls back to ``func.__name__`` for the route when the handler has no
``http_request`` param — endpoints like ``/key/generate`` never receive one,
and gating the hook on it leaked their SERVER span (created in auth, never
ended → never exported). Always emitting keeps both success and failure
paths consistent.
"""
from litellm.proxy.proxy_server import open_telemetry_logger
if open_telemetry_logger is None:
return
http_request: Optional[Request] = kwargs.get("http_request")
if http_request is not None:
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import ( # noqa: PLC0415
get_request_route,
)
route = get_request_route(http_request)
request_body: dict = await _read_request_body(request=http_request)
else:
route = func.__name__
request_body = {}
_CREDENTIAL_FIELDS = frozenset(
{
"key",
"token",
"api_key",
"secret",
"password",
"access_token",
"refresh_token",
"private_key",
"service_account_key",
}
)
_response: Optional[dict] = None
if exception is None and result is not None:
try:
raw = dict(result)
_response = {k: v for k, v in raw.items() if k not in _CREDENTIAL_FIELDS}
except Exception:
_response = None
logging_payload = ManagementEndpointLoggingPayload(
route=route,
request_data=request_body,
response=_response,
start_time=start_time,
end_time=end_time,
exception=exception,
)
if exception is None:
await open_telemetry_logger.async_management_endpoint_success_hook(
logging_payload=logging_payload,
parent_otel_span=parent_otel_span,
)
else:
await open_telemetry_logger.async_management_endpoint_failure_hook(
logging_payload=logging_payload,
parent_otel_span=parent_otel_span,
)
def management_endpoint_wrapper(func):
"""
This wrapper does the following:
@ -446,13 +525,10 @@ def management_endpoint_wrapper(func):
@wraps(func)
async def wrapper(*args, **kwargs):
start_time = datetime.now()
_http_request: Optional[Request] = None
try:
result = await func(*args, **kwargs)
end_time = datetime.now()
try:
if kwargs is None:
kwargs = {}
user_api_key_dict: UserAPIKeyAuth = (
kwargs.get("user_api_key_dict") or UserAPIKeyAuth()
)
@ -462,31 +538,16 @@ def management_endpoint_wrapper(func):
user_api_key_dict=user_api_key_dict,
function_name=func.__name__,
)
_http_request = kwargs.get("http_request", None)
parent_otel_span = getattr(user_api_key_dict, "parent_otel_span", None)
if parent_otel_span is not None:
from litellm.proxy.proxy_server import open_telemetry_logger
if open_telemetry_logger is not None:
if _http_request:
_route = _http_request.url.path
_request_body: dict = await _read_request_body(
request=_http_request
)
_response = dict(result) if result is not None else None
logging_payload = ManagementEndpointLoggingPayload(
route=_route,
request_data=_request_body,
response=_response,
start_time=start_time,
end_time=end_time,
)
await open_telemetry_logger.async_management_endpoint_success_hook( # type: ignore
logging_payload=logging_payload,
parent_otel_span=parent_otel_span,
)
await _emit_management_endpoint_otel_span(
func=func,
kwargs=kwargs,
parent_otel_span=parent_otel_span,
start_time=start_time,
end_time=end_time,
result=result,
)
# Delete updated/deleted info from cache
_delete_api_key_from_cache(kwargs=kwargs)
@ -502,38 +563,26 @@ def management_endpoint_wrapper(func):
except Exception as e:
end_time = datetime.now()
if kwargs is None:
kwargs = {}
user_api_key_dict: UserAPIKeyAuth = (
kwargs.get("user_api_key_dict") or UserAPIKeyAuth()
)
parent_otel_span = getattr(user_api_key_dict, "parent_otel_span", None)
if parent_otel_span is not None:
from litellm.proxy.proxy_server import open_telemetry_logger
if open_telemetry_logger is not None:
_http_request = kwargs.get("http_request")
if _http_request:
_route = _http_request.url.path
_request_body: dict = await _read_request_body(
request=_http_request
)
else:
_route = func.__name__
_request_body = {}
logging_payload = ManagementEndpointLoggingPayload(
route=_route,
request_data=_request_body,
response=None,
try:
await _emit_management_endpoint_otel_span(
func=func,
kwargs=kwargs,
parent_otel_span=parent_otel_span,
start_time=start_time,
end_time=end_time,
exception=e,
)
await open_telemetry_logger.async_management_endpoint_failure_hook( # type: ignore
logging_payload=logging_payload,
parent_otel_span=parent_otel_span,
except Exception as otel_exc:
# Non-Blocking Exception - never let OTEL failures swallow
# the original management-endpoint exception.
verbose_logger.debug(
"Error emitting OTEL span in management endpoint wrapper failure path: %s",
str(otel_exc),
)
raise e

4
litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
View file

@ -44,6 +44,7 @@ from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
)
from litellm.types.passthrough_endpoints.pass_through_endpoints import (
LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY,
LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY,
)
from litellm.proxy.utils import is_known_model
from litellm.proxy.vector_store_endpoints.utils import (
@ -1123,6 +1124,9 @@ async def bedrock_proxy_route(
_forward_headers=True,
) # dynamically construct pass-through endpoint based on incoming path
setattr(request.state, LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY, data)
# SigV4 signs an exact payload; pass-through must send prepped.body, not json.dumps
# of a dict that hooks may mutate (logging_obj, metadata, etc.).
setattr(request.state, LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY, prepped.body)
received_value = await endpoint_func(
request,
fastapi_response,

73
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file

@ -6,7 +6,7 @@ import posixpath
import traceback
from base64 import b64encode
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple, Union, cast
from typing import Any, Dict, List, Mapping, Optional, Tuple, Union, cast
from urllib.parse import urlencode, urlparse
import httpx
@ -62,6 +62,7 @@ from litellm.types.llms.custom_http import httpxSpecialProvider
from litellm.types.passthrough_endpoints.pass_through_endpoints import (
EndpointType,
LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY,
LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY,
PassthroughStandardLoggingPayload,
)
@ -735,6 +736,22 @@ async def pass_through_request( # noqa: PLR0915
str(url)
)
# SigV4-signed callers (e.g. Bedrock) attach the exact bytes that were
# signed via request.state; we must send those instead of re-encoding the
# parsed dict (hooks mutate it, breaking the signature / Content-Length).
# Tolerate request objects without `state` (test fixtures) and only honor
# values httpx accepts for `content=`.
_request_state = getattr(request, "state", None)
state_raw_body: Optional[Union[str, bytes]] = (
getattr(_request_state, LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY, None)
if _request_state is not None
else None
)
if state_raw_body is not None and not isinstance(
state_raw_body, (str, bytes, bytearray)
):
state_raw_body = None
# Skip body parsing for multipart requests - make_multipart_http_request will handle it
# But if custom_body is provided (e.g., JSON parsed despite multipart content-type), use it
is_multipart = (
@ -883,12 +900,19 @@ async def pass_through_request( # noqa: PLR0915
)
)
else:
# SigV4-signed callers (Bedrock) supply the exact pre-signed bytes;
# otherwise httpx encodes the parsed JSON dict as before.
body_kwargs: Dict[str, Any] = (
{"content": state_raw_body}
if state_raw_body is not None
else {"json": _parsed_body}
)
req = async_client.build_request(
"POST",
request.method,
url,
json=_parsed_body,
params=requested_query_params,
headers=headers,
**body_kwargs,
)
response = await async_client.send(req, stream=stream)
@ -917,17 +941,28 @@ async def pass_through_request( # noqa: PLR0915
status_code=response.status_code,
)
response = (
await HttpPassThroughEndpointHelpers.non_streaming_http_request_handler(
request=request,
async_client=async_client,
if state_raw_body is not None:
# SigV4-signed callers (Bedrock) require the exact pre-signed bytes
# to be forwarded so the signature/Content-Length stay valid.
response = await async_client.request(
method=request.method,
url=url,
headers=headers,
requested_query_params=requested_query_params,
_parsed_body=_parsed_body,
forward_multipart=is_multipart,
params=requested_query_params,
content=state_raw_body,
)
else:
response = (
await HttpPassThroughEndpointHelpers.non_streaming_http_request_handler(
request=request,
async_client=async_client,
url=url,
headers=headers,
requested_query_params=requested_query_params,
_parsed_body=_parsed_body,
forward_multipart=is_multipart,
)
)
)
verbose_proxy_logger.debug("response.headers= %s", response.headers)
if _is_streaming_response(response) is True:
@ -1225,7 +1260,7 @@ async def _parse_request_data_by_content_type(
def create_pass_through_route(
endpoint,
target: str,
custom_headers: Optional[dict] = None,
custom_headers: Optional[Mapping[str, Any]] = None,
_forward_headers: Optional[bool] = False,
_merge_query_params: Optional[bool] = False,
dependencies: Optional[List] = None,
@ -1272,11 +1307,14 @@ def create_pass_through_route(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
subpath: str = "", # captures sub-paths when include_subpath=True
):
from litellm.proxy.auth.auth_utils import ( # noqa: PLC0415
get_request_route,
)
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
InitPassThroughEndpointHelpers,
)
path = request.url.path
path = get_request_route(request)
# Parse request data based on content type
(
@ -1335,9 +1373,12 @@ def create_pass_through_route(
)
)
# Ensure custom_headers is a dict
# Ensure custom_headers is a dict. Botocore returns a HeadersDict
# for SigV4-prepared requests, which is a Mapping but not a dict.
headers_dict = (
param_custom_headers if isinstance(param_custom_headers, dict) else {}
dict(param_custom_headers)
if isinstance(param_custom_headers, Mapping)
else {}
)
# Ensure query_params and custom_body are dicts or None
@ -1380,6 +1421,8 @@ def create_pass_through_route(
finally:
if hasattr(request.state, LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY):
delattr(request.state, LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY)
if hasattr(request.state, LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY):
delattr(request.state, LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY)
return endpoint_func

19
litellm/proxy/proxy_server.py
View file

@ -241,7 +241,10 @@ from litellm.litellm_core_utils.core_helpers import (
)
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.sensitive_data_masker import SensitiveDataMasker
from litellm.litellm_core_utils.sensitive_data_masker import (
SensitiveDataMasker,
mask_sensitive_keys,
)
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
from litellm.proxy._types import *
@ -990,6 +993,15 @@ _OPENAPI_HTTP_METHODS = {
}
# Credentials surfaced by `/get/config/callbacks` in the alerting block: the
# full Slack incoming-webhook URL is itself a credential, and the SMTP
# password is a service password. Masked on read so plaintext never reaches
# the UI. Kept here at module scope to match the analogous
# `_SSO_SENSITIVE_FIELDS` / `_CACHE_SENSITIVE_FIELDS` constants in the SSO
# and cache endpoint files.
_ALERTING_SENSITIVE_VARS: Set[str] = {"SLACK_WEBHOOK_URL", "SMTP_PASSWORD"}
def _strip_operation_id_method_suffix(operation_id: str) -> str:
base, separator, suffix = operation_id.rpartition("_")
if separator and suffix in _OPENAPI_HTTP_METHODS:
@ -1059,6 +1071,7 @@ app = FastAPI(
root_path=server_root_path,
lifespan=proxy_startup_event, # type: ignore[reportGeneralTypeIssues]
generate_unique_id_function=_generate_stable_operation_id,
strict_content_type=False,
)
vertex_live_passthrough_vertex_base = VertexBase()
@ -14708,6 +14721,9 @@ async def get_config(): # noqa: PLR0915
value=env_variable, key=_var
)
_slack_env_vars[_var] = _decrypted_value
_slack_env_vars = mask_sensitive_keys(
_slack_env_vars, _ALERTING_SENSITIVE_VARS
)
_alerting_types = proxy_logging_obj.slack_alerting_instance.alert_types
_all_alert_types = (
@ -14744,6 +14760,7 @@ async def get_config(): # noqa: PLR0915
# decode + decrypt the value
_decrypted_value = decrypt_value_helper(value=env_variable, key=_var)
_email_env_vars[_var] = _decrypted_value
_email_env_vars = mask_sensitive_keys(_email_env_vars, _ALERTING_SENSITIVE_VARS)
alerting_data.append(
{

5
litellm/proxy/spend_tracking/spend_management_endpoints.py
View file

@ -1817,7 +1817,10 @@ async def ui_view_spend_logs( # noqa: PLR0915
)
try:
is_v2 = "/spend/logs/v2" in request.url.path
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
is_v2 = "/spend/logs/v2" in get_request_route(request)
formats = ["%Y-%m-%d %H:%M:%S", "%Y-%m-%d"] if is_v2 else ["%Y-%m-%d %H:%M:%S"]
def parse_date(date_str: str) -> datetime:

16
litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py
View file

@ -9,6 +9,7 @@ from pydantic.fields import FieldInfo
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.sensitive_data_masker import mask_sensitive_keys
from litellm.proxy._types import *
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.types.proxy.management_endpoints.ui_sso import (
@ -19,6 +20,16 @@ from litellm.types.proxy.management_endpoints.ui_sso import (
router = APIRouter()
# SSO secret fields returned by /get/sso_settings. These are masked on read so
# the UI can show "(set)" without ever transporting the plaintext OAuth secret
# off the server, matching the write-once + masked-on-read contract used for
# the HashiCorp Vault config override.
_SSO_SENSITIVE_FIELDS: Set[str] = {
"google_client_secret",
"microsoft_client_secret",
"generic_client_secret",
}
class IPAddress(BaseModel):
ip: str
@ -728,8 +739,9 @@ async def get_sso_settings():
schema = TypeAdapter(SSOConfig).json_schema(by_alias=True)
# Convert to dict for response
sso_dict = sso_config.model_dump()
# Convert to dict for response, masking OAuth client secrets so plaintext
# is never sent to the UI.
sso_dict = mask_sensitive_keys(sso_config.model_dump(), _SSO_SENSITIVE_FIELDS)
# Add descriptions to the response
result = {

18
litellm/proxy/vector_store_endpoints/utils.py
View file

@ -330,11 +330,16 @@ def is_allowed_to_call_vector_store_endpoint(
provider_config.get_vector_store_endpoints_by_type()
)
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
request_route = get_request_route(request)
# Determine the permission type based on the request
permission_type = None
for endpoint in provider_vector_store_endpoints["read"]:
if request.method == endpoint[0] and _does_endpoint_match(
endpoint[1], request.url.path
endpoint[1], request_route
):
permission_type = "read"
break
@ -342,7 +347,7 @@ def is_allowed_to_call_vector_store_endpoint(
if permission_type is None:
for endpoint in provider_vector_store_endpoints["write"]:
if request.method == endpoint[0] and _does_endpoint_match(
endpoint[1], request.url.path
endpoint[1], request_route
):
permission_type = "write"
break
@ -392,10 +397,15 @@ def is_allowed_to_call_vector_store_files_endpoint(
provider_config.get_vector_store_file_endpoints_by_type()
)
# Inline import — auth_utils participates in a proxy import cycle.
from litellm.proxy.auth.auth_utils import get_request_route # noqa: PLC0415
request_route = get_request_route(request)
permission_type: Optional[str] = None
for endpoint in provider_vector_store_endpoints.get("read", ()):
if request.method == endpoint[0] and _does_endpoint_match(
endpoint[1], request.url.path
endpoint[1], request_route
):
permission_type = "read"
break
@ -403,7 +413,7 @@ def is_allowed_to_call_vector_store_files_endpoint(
if permission_type is None:
for endpoint in provider_vector_store_endpoints.get("write", ()):
if request.method == endpoint[0] and _does_endpoint_match(
endpoint[1], request.url.path
endpoint[1], request_route
):
permission_type = "write"
break

12
litellm/responses/main.py
View file

@ -54,6 +54,7 @@ if TYPE_CHECKING:
else:
ResponseText = str # Fallback for ResponseText import
from litellm.litellm_core_utils.get_litellm_params import get_litellm_params
from litellm.llms.openai.data_residency import infer_openai_data_residency
from litellm.secret_managers.main import get_secret_str
from litellm.types.responses.main import *
from litellm.types.router import GenericLiteLLMParams
@ -1139,6 +1140,9 @@ def responses(
"aresponses": _is_async,
"litellm_call_id": litellm_call_id,
"model_info": kwargs.get("model_info"),
"data_residency": infer_openai_data_residency(
custom_llm_provider, litellm_params.api_base
),
"metadata": (
kwargs["litellm_metadata"]
if "litellm_metadata" in kwargs
@ -2032,6 +2036,9 @@ def compact_responses(
litellm_params={
**responses_api_request_params,
"litellm_call_id": litellm_call_id,
"data_residency": infer_openai_data_residency(
custom_llm_provider, litellm_params.api_base
),
},
custom_llm_provider=custom_llm_provider,
)
@ -2129,6 +2136,11 @@ async def _aresponses_websocket(
api_key=api_key,
)
litellm_params_dict["data_residency"] = infer_openai_data_residency(
_custom_llm_provider,
dynamic_api_base or litellm_params.api_base or litellm.api_base,
)
litellm_logging_obj.update_from_kwargs(
kwargs=kwargs,
model=model,

92
litellm/responses/utils.py
View file

@ -738,6 +738,98 @@ class ResponsesAPIRequestUtils:
model_id,
)
@staticmethod
def _collect_container_ids_from_annotations(
annotations: Any,
collected: set[str],
) -> None:
if not annotations or not isinstance(annotations, list):
return
for ann in annotations:
ResponsesAPIRequestUtils._collect_container_ids_from_output_item(
ann, collected
)
@staticmethod
def _collect_container_ids_from_message_content(
content: Any,
collected: set[str],
) -> None:
if not content:
return
if isinstance(content, list):
for part in content:
if isinstance(part, dict):
ResponsesAPIRequestUtils._collect_container_ids_from_annotations(
part.get("annotations"),
collected,
)
else:
ResponsesAPIRequestUtils._collect_container_ids_from_annotations(
getattr(part, "annotations", None),
collected,
)
@staticmethod
def _collect_container_ids_from_output_item(
item: Any,
collected: set[str],
) -> None:
"""Collect managed or raw ``container_id`` values from one output item."""
if item is None:
return
if isinstance(item, dict):
cid = item.get("container_id")
if isinstance(cid, str) and cid:
collected.add(cid)
nested = item.get("code_interpreter_call")
if isinstance(nested, dict):
nc = nested.get("container_id")
if isinstance(nc, str) and nc:
collected.add(nc)
if item.get("type") == "message":
ResponsesAPIRequestUtils._collect_container_ids_from_message_content(
item.get("content"),
collected,
)
return
cid_attr = getattr(item, "container_id", None)
if isinstance(cid_attr, str) and cid_attr:
collected.add(cid_attr)
nested_obj = getattr(item, "code_interpreter_call", None)
if nested_obj is not None:
ResponsesAPIRequestUtils._collect_container_ids_from_output_item(
nested_obj, collected
)
if getattr(item, "type", None) == "message":
ResponsesAPIRequestUtils._collect_container_ids_from_message_content(
getattr(item, "content", None),
collected,
)
@staticmethod
def collect_container_ids_from_responses_response(response: Any) -> list[str]:
"""Return unique container IDs referenced in a Responses API payload."""
if response is None:
return []
if isinstance(response, dict):
output = response.get("output", [])
else:
output = getattr(response, "output", []) or []
collected: set[str] = set()
if output:
for item in output:
ResponsesAPIRequestUtils._collect_container_ids_from_output_item(
item, collected
)
return list(collected)
@staticmethod
def _update_container_ids_in_response(
responses_api_response: Union[ResponsesAPIResponse, Dict[str, Any]],

3
litellm/setup_wizard.py
View file

@ -52,11 +52,12 @@ PROVIDERS: List[Dict] = [
{
"id": "anthropic",
"name": "Anthropic",
"description": "Claude Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5",
"description": "Claude Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5",
"env_key": "ANTHROPIC_API_KEY",
"key_hint": "sk-ant-...",
"test_model": "claude-haiku-4-5-20251001",
"models": [
"claude-opus-4-8",
"claude-opus-4-7",
"claude-opus-4-6",
"claude-sonnet-4-6",

4
litellm/types/integrations/datadog_cost_management.py
View file

@ -1,4 +1,4 @@
from typing import Dict, Optional, TypedDict
from typing import Dict, List, Optional, TypedDict
from litellm.types.integrations.custom_logger import StandardCustomLoggerInitParams
@ -9,7 +9,7 @@ class DatadogCostManagementInitParams(StandardCustomLoggerInitParams):
Init params for Datadog Cost Management
"""
datadog_cost_management_params: Optional[Dict] = None
cost_tag_keys: Optional[List[str]] = None
class DatadogFOCUSCostEntry(TypedDict):

3
litellm/types/llms/anthropic.py
View file

@ -39,7 +39,8 @@ class AnthropicOutputSchema(TypedDict, total=False):
class AnthropicOutputConfig(TypedDict, total=False):
"""Configuration for controlling Claude's output behavior."""
effort: Literal["high", "medium", "low"]
effort: Literal["high", "medium", "low", "xhigh", "max"]
format: AnthropicOutputSchema
class AnthropicMessagesTool(TypedDict, total=False):

2
litellm/types/llms/gemini.py
View file

@ -133,7 +133,7 @@ class BidiGenerateContentSetup(TypedDict, total=False):
tools: List[Tools]
"""The tools to be used for the realtime session."""
realtimeInputConfig: dict
realtimeInputConfig: BidiGenerateContentRealtimeInputConfig
"""The realtime config to be used for the realtime session."""
sessionResumption: dict

22
litellm/types/llms/openai.py
View file

@ -79,7 +79,14 @@ from pydantic import (
field_serializer,
field_validator,
)
from typing_extensions import Annotated, Dict, Required, TypedDict, override
from typing_extensions import (
Annotated,
Dict,
NotRequired,
Required,
TypedDict,
override,
)
from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
from litellm.types.responses.main import (
@ -1935,6 +1942,7 @@ class OpenAIRealtimeStreamResponseOutputItemAdded(TypedDict):
response_id: str
output_index: int
item: OpenAIRealtimeStreamResponseOutputItem
event_id: NotRequired[str]
class OpenAIRealtimeStreamResponseBaseObject(TypedDict):
@ -2061,6 +2069,17 @@ class OpenAIRealtimeContentPartDone(TypedDict):
type: Literal["response.content_part.done"]
class OpenAIRealtimeFunctionCallArgumentsDone(TypedDict):
type: Literal["response.function_call_arguments.done"]
event_id: str
response_id: str
item_id: str
output_index: int
call_id: str
name: str
arguments: str
class OpenAIRealtimeOutputItemDone(TypedDict):
event_id: str
item: OpenAIRealtimeStreamResponseOutputItem
@ -2126,6 +2145,7 @@ OpenAIRealtimeEvents = Union[
OpenAIRealtimeResponseAudioDone,
OpenAIRealtimeContentPartDone,
OpenAIRealtimeOutputItemDone,
OpenAIRealtimeFunctionCallArgumentsDone,
OpenAIRealtimeDoneEvent,
]

4
litellm/types/passthrough_endpoints/pass_through_endpoints.py
View file

@ -7,6 +7,10 @@ from typing_extensions import TypedDict
# JSON without a FastAPI `custom_body` parameter (which would consume the HTTP body).
LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY = "litellm_pass_through_custom_body"
# Request.state key for programmatic pass-through callers that must preserve an
# exact byte/string body, such as AWS SigV4-signed requests.
LITELLM_PASS_THROUGH_RAW_BODY_STATE_KEY = "litellm_pass_through_raw_body"
class EndpointType(str, Enum):
VERTEX_AI = "vertex-ai"

2
litellm/types/router.py
View file

@ -398,6 +398,8 @@ SPECIAL_MODEL_INFO_PARAMS = [
"output_cost_per_token",
"input_cost_per_character",
"output_cost_per_character",
"cache_read_input_token_cost",
"cache_creation_input_token_cost",
]

23
litellm/types/utils.py
View file

@ -148,6 +148,9 @@ class ProviderSpecificModelInfo(TypedDict, total=False):
supports_xhigh_reasoning_effort: Optional[bool]
supports_max_reasoning_effort: Optional[bool]
supports_output_config: Optional[bool]
bedrock_output_config_effort_ceiling: Optional[
Literal["low", "medium", "high", "max", "xhigh"]
]
class SearchContextCostPerQuery(TypedDict, total=False):
@ -219,6 +222,12 @@ class ModelInfoBase(ProviderSpecificModelInfo, total=False):
output_cost_per_token_priority: Optional[
float
] # OpenAI priority service tier pricing
regional_processing_uplift_multiplier_eu: Optional[
float
] # OpenAI EU data-residency uplift multiplier applied to all token costs (e.g. 1.10 = +10%)
regional_processing_uplift_multiplier_us: Optional[
float
] # OpenAI US data-residency uplift multiplier applied to all token costs (e.g. 1.10 = +10%)
output_cost_per_character: Optional[float] # only for vertex ai models
output_cost_per_audio_token: Optional[float]
output_cost_per_token_above_128k_tokens: Optional[
@ -3601,6 +3610,20 @@ class ServiceTier(Enum):
PRIORITY = "priority"
class DataResidency(Enum):
"""
OpenAI data-residency / regional-processing regions.
Inferred from the OpenAI api_base host (eu.api.openai.com -> EU,
us.api.openai.com -> US). Used to apply the regional-processing
cost uplift (see ``regional_processing_uplift_multiplier_<region>``
on ModelInfo).
"""
US = "us"
EU = "eu"
LLMResponseTypes = Union[
ModelResponse,
EmbeddingResponse,

9
litellm/utils.py
View file

@ -5942,6 +5942,12 @@ def _get_model_info_helper( # noqa: PLR0915
output_cost_per_token_priority=_model_info.get(
"output_cost_per_token_priority", None
),
regional_processing_uplift_multiplier_eu=_model_info.get(
"regional_processing_uplift_multiplier_eu", None
),
regional_processing_uplift_multiplier_us=_model_info.get(
"regional_processing_uplift_multiplier_us", None
),
output_cost_per_audio_token=_model_info.get(
"output_cost_per_audio_token", None
),
@ -6030,6 +6036,9 @@ def _get_model_info_helper( # noqa: PLR0915
supports_max_reasoning_effort=_model_info.get(
"supports_max_reasoning_effort", None
),
bedrock_output_config_effort_ceiling=_model_info.get(
"bedrock_output_config_effort_ceiling", None
),
supports_computer_use=_model_info.get("supports_computer_use", None),
search_context_cost_per_query=_model_info.get(
"search_context_cost_per_query", None

16
migrations/Dockerfile
View file

@ -31,12 +31,20 @@ USER root
COPY --from=uvbin /uv /uvx /usr/local/bin/
RUN apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile
# nodejs/npm so `prisma generate` uses Wolfi's Node via PRISMA_USE_GLOBAL_NODE
# instead of nodeenv downloading one whose dynamic deps may not be in Wolfi
# (e.g. Node 26.2.0 needs libatomic). Retry for transient apk.cgr.dev flakes.
RUN for i in 1 2 3; do \
apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile nodejs npm && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=0 \
PRISMA_USE_GLOBAL_NODE=true \
PATH="/app/.venv/bin:${PATH}"
# Stage 1 — install third-party deps only (cached by pyproject.toml/uv.lock).
@ -78,7 +86,11 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
RUN apk add --no-cache bash openssl tzdata python3 libsndfile libatomic
RUN for i in 1 2 3; do \
apk add --no-cache bash openssl tzdata python3 libsndfile libatomic && break; \
[ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
sleep 5; \
done
# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532). The
# Prisma engine binaries are dynamically linked against libssl/libcrypto, so

602
model_prices_and_context_window.json
View file

File diff suppressed because it is too large Load diff

122
pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm"
version = "1.87.0"
version = "1.88.0"
description = "Library to easily interface with LLM API providers"
readme = "README.md"
requires-python = ">=3.10, <3.14"
@ -33,62 +33,66 @@ Homepage = "https://litellm.ai"
Repository = "https://github.com/BerriAI/litellm"
Documentation = "https://docs.litellm.ai"
# Optional extras retain exact pins because they are consumed by Docker images
# where exact reproducibility matters. The core SDK uses ranges so downstream
# consumers can coexist with other packages without forced downgrades.
# Optional extras use compatible ranges (like the core SDK above) so downstream
# consumers can coexist with other packages and pick up security patches without
# forking. Reproducibility for our Docker/CI comes from `uv.lock` (images install
# via `uv sync --frozen`). A few deps stay exact-pinned: litellm's own
# sub-packages and the opentelemetry trio move in lockstep, and grpcio is
# supply-chain-pinned to a vetted, aged release.
[project.optional-dependencies]
proxy = [
"gunicorn==23.0.0",
"uvicorn==0.33.0",
"granian==2.5.7",
"uvloop==0.21.0; sys_platform != 'win32'",
"fastapi==0.124.4",
"backoff==2.2.1",
"pyyaml==6.0.3",
"rq==2.7.0",
"orjson==3.11.6",
"apscheduler==3.11.2",
"fastapi-sso==0.19.0",
"PyJWT==2.12.0",
"python-multipart==0.0.27",
"cryptography==46.0.7",
"pynacl==1.6.2",
"websockets==15.0.1",
"boto3==1.43.1",
"azure-identity==1.25.2",
"azure-storage-blob==12.28.0",
"mcp==1.26.0",
"gunicorn>=23.0.0,<24.0",
"uvicorn>=0.33.0,<1.0",
"granian>=2.7.4,<3.0",
"uvloop>=0.21.0,<1.0; sys_platform != 'win32'",
"fastapi>=0.136.3,<1.0",
"starlette>=1.0.1,<2.0",
"backoff>=2.2.1,<3.0",
"pyyaml>=6.0.3,<7.0",
"rq>=2.7.0,<3.0",
"orjson>=3.11.6,<4.0",
"apscheduler>=3.11.2,<4.0",
"fastapi-sso>=0.19.0,<1.0",
"PyJWT>=2.12.0,<3.0",
"python-multipart>=0.0.27,<1.0",
"cryptography>=46.0.7,<47.0",
"pynacl>=1.6.2,<2.0",
"websockets>=15.0.1,<16.0",
"boto3>=1.43.1,<2.0",
"azure-identity>=1.25.2,<2.0",
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.26.0,<2.0",
"litellm-proxy-extras==0.4.73",
"litellm-enterprise==0.1.41",
"RestrictedPython==8.1",
"rich==13.9.4",
"polars==1.38.1",
"soundfile==0.12.1",
"pyroscope-io==0.8.16; sys_platform != 'win32'",
"pydantic-settings>=2.14.1",
"RestrictedPython>=8.1,<9.0",
"rich>=13.9.4,<14.0",
"polars>=1.38.1,<2.0",
"soundfile>=0.12.1,<1.0",
"pyroscope-io>=0.8.16,<1.0; sys_platform != 'win32'",
"pydantic-settings>=2.14.1,<3.0",
]
extra_proxy = [
"prisma==0.11.0",
"azure-identity==1.25.2",
"azure-keyvault-secrets==4.10.0",
"prisma>=0.11.0,<1.0",
"azure-identity>=1.25.2,<2.0",
"azure-keyvault-secrets>=4.10.0,<5.0",
# Not in PyPI proxy extra.
"google-cloud-kms==2.24.2",
"google-cloud-iam==2.19.1",
"google-cloud-kms>=2.24.2,<3.0",
"google-cloud-iam>=2.19.1,<3.0",
# Not in PyPI proxy extra.
"resend==2.23.0",
"redisvl==0.4.1; python_version < '3.14'",
"a2a-sdk==0.3.24",
"resend>=2.23.0,<3.0",
"redisvl>=0.4.1,<1.0; python_version < '3.14'",
"a2a-sdk>=0.3.24,<1.0",
]
utils = [
# Not in Docker or PyPI proxy extra.
"numpydoc==1.8.0",
"numpydoc>=1.8.0,<2.0",
]
caching = ["diskcache==5.6.3"]
caching = ["diskcache>=5.6.3,<6.0"]
semantic-router = [
"semantic-router==0.1.12; python_version < '3.14'",
"aurelio-sdk==0.0.19; python_version < '3.14'",
"semantic-router>=0.1.15,<1.0; python_version < '3.14'",
"aurelio-sdk>=0.0.19,<1.0; python_version < '3.14'",
]
mlflow = ["mlflow==3.11.1"]
mlflow = ["mlflow>=3.11.1,<4.0"]
grpc = [
# Newest non-yanked release older than the 30-day cutoff.
"grpcio==1.78.0",
@ -101,28 +105,28 @@ stt-nvidia-riva = [
"audioread>=3.0.1",
"numpy>=1.26.0",
]
google = ["google-cloud-aiplatform==1.133.0"]
google = ["google-cloud-aiplatform>=1.133.0,<2.0"]
proxy-runtime = [
# Historically bundled in the proxy Docker images via requirements.txt.
# Keep these in a dedicated extra so uv-based images preserve the same
# feature surface without forcing the base SDK install to grow.
"google-cloud-aiplatform==1.133.0",
"google-genai==1.37.0",
"anthropic[vertex]==0.84.0",
"google-cloud-aiplatform>=1.133.0,<2.0",
"google-genai>=1.37.0,<2.0",
"anthropic[vertex]>=0.84.0,<1.0",
"grpcio==1.78.0",
"prometheus-client==0.20.0",
"langfuse==2.59.7",
"prometheus-client>=0.20.0,<1.0",
"langfuse>=2.59.7,<3.0",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"ddtrace==2.19.0",
"sentry-sdk==2.21.0",
"mangum==0.17.0",
"azure-ai-contentsafety==1.0.0",
"azure-storage-file-datalake==12.20.0",
"pypdf==6.10.2; python_version < '3.14'",
"llm-sandbox==0.3.39",
"detect-secrets==1.5.0",
"ddtrace>=2.19.0,<3.0",
"sentry-sdk>=2.21.0,<3.0",
"mangum>=0.17.0,<1.0",
"azure-ai-contentsafety>=1.0.0,<2.0",
"azure-storage-file-datalake>=12.20.0,<13.0",
"pypdf>=6.10.2,<7.0; python_version < '3.14'",
"llm-sandbox>=0.3.39,<1.0",
"detect-secrets>=1.5.0,<2.0",
]
[project.scripts]
@ -188,7 +192,7 @@ ci = [
"psycopg2-binary==2.9.11",
"pytest-codspeed==4.3.0",
"pytest-retry==1.7.0",
"pyarrow==22.0.0",
"pyarrow==23.0.1",
"langchain==1.2.10",
"lunary==1.4.36; python_version == '3.10'",
"lunary==1.4.37; python_version >= '3.11'",
@ -253,7 +257,7 @@ source-exclude = [
profile = "black"
[tool.commitizen]
version = "1.87.0"
version = "1.88.0"
version_files = [
"pyproject.toml:^version",
]

191
scripts/benchmark_model_response_creator.py Normal file
View file

@ -0,0 +1,191 @@
#!/usr/bin/env python3
"""Tight microbenchmark for CustomStreamWrapper.model_response_creator.
Calls model_response_creator() in a tight loop on a pre-built wrapper to
isolate per-call cost. Driving the full wrapper adds threadpool logging,
gc, and other noise that swamps microsecond-scale changes here.
Example:
uv run python scripts/benchmark_model_response_creator.py --label baseline
uv run python scripts/benchmark_model_response_creator.py --label optimized
"""
from __future__ import annotations
import argparse
import gc
import json
import logging
import os
import statistics
import time
from dataclasses import asdict, dataclass
from typing import List
from unittest.mock import MagicMock
os.environ.setdefault("LITELLM_LOG", "ERROR")
logging.getLogger("LiteLLM").setLevel(logging.ERROR)
import litellm # noqa: E402
litellm.suppress_debug_info = True
from litellm.litellm_core_utils.streaming_handler import (
CustomStreamWrapper,
) # noqa: E402
def _make_logging_obj(provider: str) -> MagicMock:
logging_obj = MagicMock()
logging_obj.model_call_details = {
"custom_llm_provider": provider,
"litellm_params": {},
}
logging_obj.call_type = "completion"
logging_obj.stream_options = None
logging_obj.messages = [{"role": "user", "content": "hi"}]
logging_obj.completion_start_time = None
logging_obj._llm_caching_handler = None
return logging_obj
def _make_wrapper(provider: str, model: str) -> CustomStreamWrapper:
return CustomStreamWrapper(
completion_stream=iter([]),
model=model,
logging_obj=_make_logging_obj(provider),
custom_llm_provider=provider,
)
@dataclass
class Result:
label: str
scenario: str
iterations: int
elapsed_min_s: float
elapsed_median_s: float
per_call_us: float
calls_per_sec: float
SCENARIOS = {
"no_chunk": {
"description": "model_response_creator() — no chunk arg (most common path)",
"chunk_factory": lambda i: None,
},
"text_chunk": {
"description": "model_response_creator(chunk={'text': '...'}) — text delta path",
"chunk_factory": lambda i: {"text": f"token{i}"},
},
"rich_chunk": {
"description": "model_response_creator(chunk={...}) — full chunk dict path",
"chunk_factory": lambda i: {
"id": f"id-{i}",
"object": "chat.completion.chunk",
"created": 1234567890,
},
},
}
def bench_no_chunk(wrapper: CustomStreamWrapper, iterations: int) -> float:
gc.collect()
gc.disable()
try:
start = time.perf_counter()
for _ in range(iterations):
wrapper.model_response_creator()
elapsed = time.perf_counter() - start
finally:
gc.enable()
return elapsed
def bench_with_chunk(wrapper: CustomStreamWrapper, factory, iterations: int) -> float:
# Pre-build chunks so we don't measure their construction cost.
chunks = [factory(i) for i in range(iterations)]
gc.collect()
gc.disable()
try:
start = time.perf_counter()
for chunk in chunks:
wrapper.model_response_creator(chunk=dict(chunk)) # copy because mutated
elapsed = time.perf_counter() - start
finally:
gc.enable()
return elapsed
def run_scenario(
label: str,
scenario_key: str,
iterations: int,
repeats: int,
warmup: int,
) -> Result:
spec = SCENARIOS[scenario_key]
wrapper = _make_wrapper(provider="anthropic", model="claude-3-5-sonnet")
if scenario_key == "no_chunk":
runner = lambda: bench_no_chunk(wrapper, iterations) # noqa: E731
else:
runner = lambda: bench_with_chunk(
wrapper, spec["chunk_factory"], iterations
) # noqa: E731
for _ in range(warmup):
runner()
samples = [runner() for _ in range(repeats)]
elapsed_min = min(samples)
elapsed_median = statistics.median(samples)
per_call_us = (elapsed_min * 1_000_000) / iterations
calls_per_sec = iterations / elapsed_min if elapsed_min > 0 else 0.0
return Result(
label=label,
scenario=scenario_key,
iterations=iterations,
elapsed_min_s=elapsed_min,
elapsed_median_s=elapsed_median,
per_call_us=per_call_us,
calls_per_sec=calls_per_sec,
)
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--label", required=True)
ap.add_argument("--iterations", type=int, default=200_000)
ap.add_argument("--warmup", type=int, default=2)
ap.add_argument("--repeats", type=int, default=8)
ap.add_argument("--json", dest="json_out")
args = ap.parse_args()
print(
f"\n=== label={args.label} iterations={args.iterations:,} "
f"warmup={args.warmup} repeats={args.repeats} (min reported) ==="
)
results: List[Result] = []
for scenario in SCENARIOS:
r = run_scenario(
args.label, scenario, args.iterations, args.repeats, args.warmup
)
results.append(r)
print(
f" {r.scenario:12s}: "
f"min={r.elapsed_min_s*1000:8.2f} ms "
f"median={r.elapsed_median_s*1000:8.2f} ms "
f"per-call={r.per_call_us:7.3f} μs "
f"calls/s={r.calls_per_sec:>12,.0f}"
)
if args.json_out:
with open(args.json_out, "w", encoding="utf-8") as f:
json.dump([asdict(r) for r in results], f, indent=2)
print(f"\nWrote {len(results)} results to {args.json_out}")
if __name__ == "__main__":
main()

369
scripts/benchmark_streaming_chunk_overhead.py Normal file
View file

@ -0,0 +1,369 @@
#!/usr/bin/env python3
"""Benchmark CustomStreamWrapper per-chunk overhead.
Drives CustomStreamWrapper directly with synthetic in-memory chunks for
Anthropic (GenericStreamingChunk), Bedrock Invoke (GenericStreamingChunk),
and Bedrock Converse (ModelResponseStream). A full proxy benchmark adds
FastAPI, HTTP, and TCP latency, which dilutes the per-chunk CPU signal.
Example:
uv run python scripts/benchmark_streaming_chunk_overhead.py \\
--streams 500 --chunks 200 --warmup 50 --repeats 5
"""
from __future__ import annotations
import argparse
import asyncio
import gc
import json
import logging
import os
import statistics
import time
from dataclasses import asdict, dataclass
from typing import Callable, List, Optional
from unittest.mock import MagicMock
# Silence litellm's "Provider List" warnings emitted by get_llm_provider
# when it sees synthetic model names — we're not exercising provider
# routing, only the per-chunk wrapper hot path.
os.environ.setdefault("LITELLM_LOG", "ERROR")
logging.getLogger("LiteLLM").setLevel(logging.ERROR)
import litellm # noqa: E402
litellm.suppress_debug_info = True
from litellm.litellm_core_utils.streaming_handler import (
CustomStreamWrapper,
) # noqa: E402
from litellm.types.utils import ( # noqa: E402
Delta,
GenericStreamingChunk as GChunk,
ModelResponseStream,
StreamingChoices,
Usage,
)
# ---------------------------------------------------------------------------
# Synthetic chunk fixtures
# ---------------------------------------------------------------------------
def _make_logging_obj(provider: str) -> MagicMock:
logging_obj = MagicMock()
logging_obj.model_call_details = {
"custom_llm_provider": provider,
"litellm_params": {},
}
logging_obj.call_type = "completion"
logging_obj.stream_options = None
logging_obj.messages = [{"role": "user", "content": "hi"}]
logging_obj.completion_start_time = None
logging_obj._llm_caching_handler = None
return logging_obj
def _make_generic_chunk(
text: str,
is_finished: bool = False,
finish_reason: str = "",
usage: Optional[dict] = None,
) -> GChunk:
return GChunk(
text=text,
is_finished=is_finished,
finish_reason=finish_reason,
usage=usage,
index=0,
tool_use=None,
)
def _make_converse_chunk(
text: str = "",
finish_reason: str = "",
usage: Optional[Usage] = None,
) -> ModelResponseStream:
return ModelResponseStream(
choices=[
StreamingChoices(
finish_reason=finish_reason or None,
index=0,
delta=Delta(content=text, role="assistant"),
)
],
id="msg-bench",
model="anthropic.claude-3-5-sonnet",
usage=usage,
)
# ---------------------------------------------------------------------------
# Provider stream factories
# ---------------------------------------------------------------------------
def anthropic_chunks(n: int) -> List[GChunk]:
out: List[GChunk] = [_make_generic_chunk(f"tok{i} ") for i in range(n)]
out.append(
_make_generic_chunk(
"",
is_finished=True,
finish_reason="stop",
usage={"prompt_tokens": 10, "completion_tokens": n, "total_tokens": 10 + n},
)
)
return out
def bedrock_invoke_chunks(n: int) -> List[GChunk]:
# Bedrock Invoke surfaces GChunk-shaped dicts, same shape as Anthropic.
return anthropic_chunks(n)
def bedrock_converse_chunks(n: int) -> List[ModelResponseStream]:
out: List[ModelResponseStream] = [
_make_converse_chunk(f"tok{i} ") for i in range(n)
]
out.append(
_make_converse_chunk(
text="",
finish_reason="stop",
usage=Usage(prompt_tokens=10, completion_tokens=n, total_tokens=10 + n),
)
)
return out
PROVIDERS: dict[str, tuple[str, Callable[[int], list]]] = {
"anthropic": ("anthropic", anthropic_chunks),
"bedrock_invoke": ("bedrock", bedrock_invoke_chunks),
"bedrock_converse": ("bedrock", bedrock_converse_chunks),
}
# ---------------------------------------------------------------------------
# Drive a single stream end-to-end
# ---------------------------------------------------------------------------
def _make_wrapper(
chunks: list, provider: str, async_stream: bool
) -> CustomStreamWrapper:
logging_obj = _make_logging_obj(provider)
if async_stream:
async def _agen():
for c in chunks:
yield c
stream = _agen()
else:
stream = iter(chunks)
return CustomStreamWrapper(
completion_stream=stream,
model="claude-3-5-sonnet",
logging_obj=logging_obj,
custom_llm_provider=provider,
)
def drive_sync(provider_key: str, chunks_per_stream: int, n_streams: int) -> float:
provider, factory = PROVIDERS[provider_key]
# Pre-build the chunk lists; we only measure wrapper iteration cost.
chunk_lists = [factory(chunks_per_stream) for _ in range(n_streams)]
gc.collect()
gc.disable()
try:
start = time.perf_counter()
for chunks in chunk_lists:
wrapper = _make_wrapper(chunks, provider, async_stream=False)
for _ in wrapper:
pass
elapsed = time.perf_counter() - start
finally:
gc.enable()
return elapsed
async def drive_async(
provider_key: str, chunks_per_stream: int, n_streams: int
) -> float:
provider, factory = PROVIDERS[provider_key]
chunk_lists = [factory(chunks_per_stream) for _ in range(n_streams)]
gc.collect()
gc.disable()
try:
start = time.perf_counter()
for chunks in chunk_lists:
wrapper = _make_wrapper(chunks, provider, async_stream=True)
async for _ in wrapper:
pass
elapsed = time.perf_counter() - start
finally:
gc.enable()
return elapsed
# ---------------------------------------------------------------------------
# Repeat × take-min runner
# ---------------------------------------------------------------------------
@dataclass
class Result:
label: str
provider: str
mode: str
streams: int
chunks_per_stream: int
total_chunks: int
elapsed_min_s: float
elapsed_median_s: float
per_chunk_us: float
chunks_per_sec: float
streams_per_sec: float
def run_case(
label: str,
provider_key: str,
mode: str,
chunks_per_stream: int,
n_streams: int,
repeats: int,
warmup: int,
) -> Result:
if mode == "sync":
# Warmup runs amortize import-time and JIT-y caches.
for _ in range(warmup):
drive_sync(provider_key, chunks_per_stream, max(1, n_streams // 10))
samples = [
drive_sync(provider_key, chunks_per_stream, n_streams)
for _ in range(repeats)
]
elif mode == "async":
async def _warm():
for _ in range(warmup):
await drive_async(
provider_key, chunks_per_stream, max(1, n_streams // 10)
)
asyncio.run(_warm())
samples = [
asyncio.run(drive_async(provider_key, chunks_per_stream, n_streams))
for _ in range(repeats)
]
else:
raise ValueError(f"unknown mode {mode!r}")
elapsed_min = min(samples)
elapsed_median = statistics.median(samples)
# Each stream emits chunks_per_stream text chunks + 1 finish/usage chunk.
total_chunks = n_streams * (chunks_per_stream + 1)
per_chunk_us = (elapsed_min * 1_000_000) / total_chunks
chunks_per_sec = total_chunks / elapsed_min if elapsed_min > 0 else 0.0
streams_per_sec = n_streams / elapsed_min if elapsed_min > 0 else 0.0
return Result(
label=label,
provider=provider_key,
mode=mode,
streams=n_streams,
chunks_per_stream=chunks_per_stream,
total_chunks=total_chunks,
elapsed_min_s=elapsed_min,
elapsed_median_s=elapsed_median,
per_chunk_us=per_chunk_us,
chunks_per_sec=chunks_per_sec,
streams_per_sec=streams_per_sec,
)
def format_result(r: Result) -> str:
return (
f" {r.provider:18s} {r.mode:5s}: "
f"min={r.elapsed_min_s*1000:8.2f} ms "
f"median={r.elapsed_median_s*1000:8.2f} ms "
f"per-chunk={r.per_chunk_us:7.2f} μs "
f"chunks/s={r.chunks_per_sec:>10,.0f} "
f"streams/s={r.streams_per_sec:>8,.1f}"
)
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument(
"--label", required=True, help="Run label (e.g. baseline / optimized)"
)
ap.add_argument("--streams", type=int, default=500, help="Streams per run")
ap.add_argument(
"--chunks",
type=int,
default=200,
help="Text chunks per stream (excl. finish chunk)",
)
ap.add_argument("--warmup", type=int, default=2, help="Warmup runs")
ap.add_argument(
"--repeats", type=int, default=5, help="Measured runs (we report min)"
)
ap.add_argument(
"--providers",
default="anthropic,bedrock_invoke,bedrock_converse",
help="Comma-separated provider list",
)
ap.add_argument(
"--modes",
default="sync,async",
help="Comma-separated iteration modes (sync/async)",
)
ap.add_argument(
"--json", dest="json_out", help="Write results as JSON to this path"
)
args = ap.parse_args()
providers = [p.strip() for p in args.providers.split(",") if p.strip()]
modes = [m.strip() for m in args.modes.split(",") if m.strip()]
for p in providers:
if p not in PROVIDERS:
raise SystemExit(f"unknown provider {p!r}; choose from {list(PROVIDERS)}")
for m in modes:
if m not in {"sync", "async"}:
raise SystemExit(f"unknown mode {m!r}; choose from sync/async")
print(
f"\n=== label={args.label} streams={args.streams} chunks/stream={args.chunks} "
f"warmup={args.warmup} repeats={args.repeats} (min reported) ==="
)
results: List[Result] = []
for provider_key in providers:
for mode in modes:
r = run_case(
label=args.label,
provider_key=provider_key,
mode=mode,
chunks_per_stream=args.chunks,
n_streams=args.streams,
repeats=args.repeats,
warmup=args.warmup,
)
results.append(r)
print(format_result(r))
if args.json_out:
with open(args.json_out, "w", encoding="utf-8") as f:
json.dump([asdict(r) for r in results], f, indent=2)
print(f"\nWrote {len(results)} results to {args.json_out}")
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show more