scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 136

Merge branch 'main' into litellm_/gifted-spence

Browse source
This commit is contained in:
yuneng-jiang 2026-03-20 12:44:20 -07:00 • committed by GitHub
commit 4d198558c8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1984 changed files with 73367 additions and 33052 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1098
.circleci/config.yml
View file

File diff suppressed because it is too large Load diff

3
.circleci/requirements.txt
View file

@ -17,4 +17,5 @@ mcp==1.25.0 # for MCP server
semantic_router==0.1.10 # for auto-routing with litellm
fastuuid==0.12.0
responses==0.25.7 # for proxy client tests
pytest-retry==1.6.3 # for automatic test retries
pytest-retry==1.6.3 # for automatic test retries
litellm-proxy-extras # for prisma migrations

4
.github/pull_request_template.md vendored
View file

@ -11,6 +11,10 @@
- [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
- [ ] I have requested a Greptile review by commenting `@greptileai` and received a **Confidence Score of at least 4/5** before requesting a maintainer review
## Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slack (#pr-review)](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA).
## CI (LiteLLM team)
> **CI status guideline:**

44
.github/workflows/codspeed.yml vendored Normal file
View file

@ -0,0 +1,44 @@
name: CodSpeed Benchmarks
on:
push:
branches:
- main
pull_request:
branches:
- main
# Allow CodSpeed to trigger backtest performance analysis
# in order to generate initial data
workflow_dispatch:
permissions:
contents: read
id-token: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
benchmarks:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dependencies
run: |
pip install -e "."
pip install pytest pytest-codspeed==4.3.0
- name: Run benchmarks
uses: CodSpeedHQ/action@v4
with:
mode: simulation
run: pytest tests/benchmarks/ --codspeed

3
.github/workflows/ghcr_deploy.yml vendored
View file

@ -369,7 +369,8 @@ jobs:
release:
name: "New LiteLLM Release"
needs: [docker-hub-deploy, build-and-push-image, build-and-push-image-database]
permissions:
contents: write
runs-on: "ubuntu-latest"
steps:

9
.github/workflows/test-linting.yml vendored
View file

@ -28,15 +28,18 @@ jobs:
find . -type d -name "__pycache__" -exec rm -rf {} + || true
find . -name "*.pyc" -delete || true
- name: Check poetry.lock is up to date
run: |
poetry check --lock || (echo "❌ poetry.lock is out of sync with pyproject.toml. Run 'poetry lock' locally and commit the result." && exit 1)
- name: Install dependencies
run: |
poetry lock
poetry install --with dev
- name: Run Black formatting
- name: Check Black formatting
run: |
cd litellm
poetry run black .
poetry run black --check --exclude '/enterprise/' .
cd ..
- name: Debug - Check file state

12
.pre-commit-config.yaml
View file

@ -14,12 +14,12 @@ repos:
types: [python]
files: (litellm/|litellm_proxy_extras/|enterprise/).*\.py
exclude: ^litellm/__init__.py$
# - id: black
# name: black
# entry: poetry run black
# language: system
# types: [python]
# files: (litellm/|litellm_proxy_extras/|enterprise/).*\.py
- id: black
name: black
entry: poetry run black
language: system
types: [python]
files: (litellm/|litellm_proxy_extras/).*\.py
- repo: https://github.com/pycqa/flake8
rev: 7.0.0 # The version of flake8 to use
hooks:

13
CLAUDE.md
View file

@ -91,6 +91,10 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
- Async/await patterns throughout
- Type hints required for all public APIs
- **Avoid imports within methods** — place all imports at the top of the file (module-level). Inline imports inside functions/methods make dependencies harder to trace and hurt readability. The only exception is avoiding circular imports where absolutely necessary.
- **Use dict spread for immutable copies** — prefer `{**original, "key": new_value}` over `dict(obj)` + mutation. The spread produces the final dict in one step and makes intent clear.
- **Guard at resolution time** — when resolving an optional value through a fallback chain (`a or b or ""`), raise immediately if the resolved result being empty is an error. Don't pass empty strings or sentinel values downstream for the callee to deal with.
- **Extract complex comprehensions to named helpers** — a set/dict comprehension that calls into the DB or manager (e.g. "which of these server IDs are OAuth2?") belongs in a named helper function, not inline in the caller.
- **FastAPI parameter declarations** — mark required query/form params with `= Query(...)` / `= Form(...)` explicitly when other params in the same handler are optional. Mixing `str` (required) with `Optional[str] = None` in the same signature causes silent 422s when the required param is missing.
### Testing Strategy
- Unit tests in `tests/test_litellm/`
@ -98,6 +102,8 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
- Proxy tests in `tests/proxy_unit_tests/`
- Load tests in `tests/load_tests/`
- **Always add tests when adding new entity types or features** — if the existing test file covers other entity types, add corresponding tests for the new one
- **Keep monkeypatch stubs in sync with real signatures** — when a function gains a new optional parameter, update every `fake_*` / `stub_*` in tests that patch it to also accept that kwarg (even as `**kwargs`). Stale stubs fail with `unexpected keyword argument` and mask real bugs.
- **Test all branches of name→ID resolution** — when adding server/resource lookup that resolves names to UUIDs, test: (1) name resolves and UUID is allowed, (2) name resolves but UUID is not allowed, (3) name does not resolve at all. The silent-fallback path is where access-control bugs hide.
### UI / Backend Consistency
- When wiring a new UI entity type to an existing backend endpoint, verify the backend API contract (single value vs. array, required vs. optional params) and ensure the UI controls match — e.g., use a single-select dropdown when the backend accepts a single value, not a multi-select
@ -134,6 +140,11 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
- **Check index coverage.** For new or modified queries, check `schema.prisma` for a supporting index. Prefer extending an existing index (e.g. `@@index([a])` → `@@index([a, b])`) over adding a new one, unless it's a `@@unique`. Only add indexes for large/frequent queries.
- **Keep schema files in sync.** Apply schema changes to all `schema.prisma` copies (`schema.prisma`, `litellm/proxy/`, `litellm-proxy-extras/`, `litellm-js/spend-logs/` for SpendLogs) with a migration under `litellm-proxy-extras/litellm_proxy_extras/migrations/`.
### Setup Wizard (`litellm/setup_wizard.py`)
- The wizard is implemented as a single `SetupWizard` class with `@staticmethod` methods — keep it that way. No module-level functions except `run_setup_wizard()` (the public entrypoint) and pure helpers (color, ANSI).
- Use `litellm.utils.check_valid_key(model, api_key)` for credential validation — never roll a custom completion call.
- Do not hardcode provider env-key names or model lists that already exist in the codebase. Add a `test_model` field to each provider entry to drive `check_valid_key`; set it to `None` for providers that can't be validated with a single API key (Azure, Bedrock, Ollama).
### Enterprise Features
- Enterprise-specific code in `enterprise/` directory
- Optional features enabled via environment variables
@ -150,4 +161,4 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
**Fix options:**
1. **Create a Prisma migration** (permanent) — run `prisma migrate dev --name <description>` in the worktree. The generated file will be picked up by `prisma migrate deploy` on next startup.
2. **Apply manually for local dev** — `psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF NOT EXISTS ..."` after each proxy start. Fine for dev, not for production.
3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.
3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.

4
Dockerfile
View file

@ -39,7 +39,7 @@ RUN pip wheel --no-cache-dir --wheel-dir=/wheels/ -r requirements.txt
# ensure pyjwt is used, not jwt
RUN pip uninstall jwt -y
RUN pip uninstall PyJWT -y
RUN pip install PyJWT==2.9.0 --no-cache-dir
RUN pip install PyJWT==2.12.0 --no-cache-dir
# Runtime stage
FROM $LITELLM_RUNTIME_IMAGE AS runtime
@ -49,7 +49,7 @@ USER root
# Install runtime dependencies (libsndfile needed for audio processing on ARM64)
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
# SECURITY FIX: npm bundles tar, glob, and brace-expansion at multiple nested
# levels inside its dependency tree. `npm install -g <pkg>` only creates a
# SEPARATE global package, it does NOT replace npm's internal copies.

3
README.md
View file

@ -28,6 +28,9 @@
<a href="https://www.litellm.ai/support">
<img src="https://img.shields.io/static/v1?label=Chat%20on&message=Slack&color=black&logo=Slack&style=flat-square" alt="Slack">
</a>
<a href="https://codspeed.io/BerriAI/litellm?utm_source=badge">
<img src="https://img.shields.io/endpoint?url=https://codspeed.io/badge.json" alt="CodSpeed"/>
</a>
</h4>
<img width="2688" height="1600" alt="Group 7154 (1)" src="https://github.com/user-attachments/assets/c5ee0412-6fb5-4fb6-ab5b-bafae4209ca6" />

5
ci_cd/security_scans.sh
View file

@ -11,7 +11,7 @@ echo "Starting security scans for LiteLLM..."
install_trivy() {
echo "Installing Trivy and required tools..."
sudo apt-get update
sudo apt-get install -y wget apt-transport-https gnupg lsb-release jq curl
sudo apt-get install -y wget apt-transport-https gnupg lsb-release jq curl bsdmainutils
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
@ -163,6 +163,9 @@ run_grype_scans() {
"CVE-2026-25639" # axios - full fix requires 1.x major version bump; pinned to >=0.30.2 to clear other axios CVEs, upgrade to 1.x in follow-up
"CVE-2026-2297" # Python 3.13 SourcelessFileLoader audit hook bypass - no fix available in base image
"GHSA-qffp-2rhf-9h96" # tar hardlink path traversal - from nodejs_wheel bundled npm, not used in application runtime code
"CVE-2026-2673" # OpenSSL 3.6.1 TLS 1.3 key exchange group negotiation issue - no fix available yet
"CVE-2026-3644" # Python 3.13 vulnerability - no fix available in base image
"CVE-2026-4224" # Python 3.13 Expat parser stack overflow in ElementDeclHandler - no fix available in base image
)
# Build JSON array of allowlisted CVE IDs for jq

3
deploy/charts/litellm-helm/templates/deployment.yaml
View file

@ -20,6 +20,9 @@ spec:
selector:
matchLabels:
{{- include "litellm.selectorLabels" . | nindent 6 }}
{{- if .Values.deploymentMinReadySeconds }}
minReadySeconds: {{ .Values.deploymentMinReadySeconds }}
{{- end }}
template:
metadata:
annotations:

13
deploy/charts/litellm-helm/tests/deployment_tests.yaml
View file

@ -306,3 +306,16 @@ tests:
- equal:
path: spec.template.spec.containers[0].resources
value: {}
- it: should be able to set minReadySeconds
template: deployment.yaml
set:
deploymentMinReadySeconds: 5
asserts:
- equal:
path: spec.minReadySeconds
value: 5
- it: should have minReadySeconds absent when deploymentMinReadySeconds is not set
template: deployment.yaml
asserts:
- notExists:
path: spec.minReadySeconds

2
deploy/charts/litellm-helm/values.yaml
View file

@ -31,6 +31,8 @@ serviceAccount:
# annotations for litellm deployment
deploymentAnnotations: {}
deploymentLabels: {}
deploymentMinReadySeconds: 0
# annotations for litellm pods
podAnnotations: {}
podLabels: {}

2
docker/Dockerfile.custom_ui
View file

@ -19,7 +19,7 @@ RUN apt-get update && apt-get upgrade -y \
libgnutls30 \
libc6 && \
apt-get install -y nodejs npm && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \

4
docker/Dockerfile.database
View file

@ -50,7 +50,7 @@ USER root
# Install runtime dependencies
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -112,7 +112,7 @@ RUN sed -i 's/\r$//' docker/install_auto_router.sh && chmod +x docker/install_au
# ensure pyjwt is used, not jwt
RUN pip uninstall jwt -y
RUN pip uninstall PyJWT -y
RUN pip install PyJWT==2.9.0 --no-cache-dir
RUN pip install PyJWT==2.12.0 --no-cache-dir
# Build Admin UI (runtime stage)
# Convert Windows line endings to Unix and make executable

4
docker/Dockerfile.dev
View file

@ -31,7 +31,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# Fix JWT dependency conflicts early
RUN pip uninstall jwt -y || true && \
pip uninstall PyJWT -y || true && \
pip install PyJWT==2.9.0 --no-cache-dir
pip install PyJWT==2.12.0 --no-cache-dir
# Copy only necessary files for build
COPY pyproject.toml README.md schema.prisma poetry.lock ./
@ -75,7 +75,7 @@ RUN apt-get update && apt-get upgrade -y \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/* \
&& npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& npm install -g npm@latest tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \

6
docker/Dockerfile.non_root
View file

@ -32,7 +32,7 @@ RUN for i in 1 2 3; do \
# Cache Python dependencies
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir=/wheels/ -r requirements.txt \
&& pip wheel --no-cache-dir --wheel-dir=/wheels/ "semantic_router==0.1.11" "aurelio-sdk==0.0.19" "PyJWT==2.9.0"
&& pip wheel --no-cache-dir --wheel-dir=/wheels/ "semantic_router==0.1.11" "aurelio-sdk==0.0.19" "PyJWT==2.12.0"
# Copy source after dependency layers
COPY . .
@ -106,7 +106,7 @@ RUN for i in 1 2 3; do \
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor && break || sleep 5; \
done \
&& apk upgrade --no-cache nodejs \
&& npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& npm install -g npm@latest tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -198,7 +198,7 @@ RUN sed -i 's/\r$//' docker/entrypoint.sh && \
chown -R nobody:nogroup /app /var/lib/litellm/ui /var/lib/litellm/assets /nonexistent /.npm && \
pip uninstall jwt -y || true && \
pip uninstall PyJWT -y || true && \
pip install --no-index --find-links=/wheels/ PyJWT==2.10.1 --no-cache-dir && \
pip install --no-index --find-links=/wheels/ PyJWT==2.12.0 --no-cache-dir && \
rm -rf /wheels && \
PRISMA_PATH=$(python -c "import os, prisma; print(os.path.dirname(prisma.__file__))") && \
chown -R nobody:nogroup $PRISMA_PATH && \

17
docs/my-website/blog/anthropic_opus_4_5_and_advanced_features/index.md
View file

@ -3,18 +3,9 @@ slug: anthropic_advanced_features
title: "Day 0 Support: Claude 4.5 Opus (+Advanced Features)"
date: 2025-11-25T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to Claude Opus 4.5 and advanced features in LiteLLM: Tool Search, Programmatic Tool Calling, and Effort Parameter."
tags: [anthropic, claude, tool search, programmatic tool calling, effort, advanced features]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
This guide covers Anthropic's latest model (Claude Opus 4.5) and its advanced features now available in LiteLLM: Tool Search, Programmatic Tool Calling, Tool Input Examples, and the Effort Parameter.
{/* truncate */}
---
| Feature | Supported Models |

15
docs/my-website/blog/anthropic_wildcard_model_access_incident/index.md
View file

@ -3,18 +3,9 @@ slug: anthropic-wildcard-model-access-incident
title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
date: 2026-02-23T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, proxy, auth, model-access]
hide_table_of_contents: false
---

24
docs/my-website/blog/authors.yml
View file

@ -4,6 +4,12 @@ litellm:
url: https://github.com/BerriAI/litellm
image_url: https://github.com/BerriAI.png
sameer:
name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
krrish:
name: Krrish Dholakia
title: CEO, LiteLLM
@ -22,3 +28,21 @@ ishaan-alt:
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
ryan:
name: Ryan Crabbe
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
alexsander:
name: Alexsander Hamir
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
yuneng:
name: Yuneng Jiang
title: SWE @ LiteLLM (Full Stack)
url: https://www.linkedin.com/in/yuneng-david-jiang-455676139/
image_url: https://avatars.githubusercontent.com/u/171294688?v=4

17
docs/my-website/blog/claude_code_beta_headers/index.md
View file

@ -3,18 +3,9 @@ slug: claude-code-beta-headers-incident
title: "Incident Report: Invalid beta headers with Claude Code"
date: 2026-02-16T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- sameer
- ishaan-alt
- krrish
tags: [incident-report, anthropic, stability]
hide_table_of_contents: false
---
@ -173,5 +164,5 @@ curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
## Related documentation
- [Managing Anthropic Beta Headers](../proxy/sync_anthropic_beta_headers.md) - Complete configuration guide
- [Managing Anthropic Beta Headers](../../docs/proxy/sync_anthropic_beta_headers) - Complete configuration guide
- [`anthropic_beta_headers_config.json`](https://github.com/BerriAI/litellm/blob/main/litellm/anthropic_beta_headers_config.json) - Current configuration file

17
docs/my-website/blog/claude_opus_4_6/index.md
View file

@ -3,18 +3,9 @@ slug: claude_opus_4_6
title: "Day 0 Support: Claude Opus 4.6"
date: 2026-02-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- sameer
- ishaan-alt
- krrish
description: "Day 0 support for Claude Opus 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
tags: [anthropic, claude, opus 4.6]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports Claude Opus 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
{/* truncate */}
## Docker Image
```bash

12
docs/my-website/blog/claude_sonnet_4_6/index.md
View file

@ -3,14 +3,8 @@ slug: claude_sonnet_4_6
title: "Day 0 Support: Claude Sonnet 4.6"
date: 2026-02-17T10:00:00
authors:
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- ishaan-alt
- krrish
description: "Day 0 support for Claude Sonnet 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
tags: [anthropic, claude, sonnet 4.6]
hide_table_of_contents: false
@ -21,6 +15,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports Claude Sonnet 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
{/* truncate */}
## Docker Image
```bash

15
docs/my-website/blog/fastapi_middleware_performance/index.mdx
View file

@ -3,18 +3,9 @@ slug: fastapi-middleware-performance
title: "Your Middleware Could Be a Bottleneck"
date: 2026-02-07T10:00:00
authors:
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Ryan Crabbe
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
- krrish
- ishaan-alt
- ryan
description: "How we improved LiteLLM proxy latency and throughput by replacing a single middleware base class"
tags: [performance, fastapi, middleware]
hide_table_of_contents: false

20
docs/my-website/blog/gemin_3.1/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_1_pro
title: "DAY 0 Support: Gemini 3.1 Pro on LiteLLM"
date: 2026-02-19T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3.1 Pro on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -28,6 +19,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports `gemini-3.1-pro-preview` and all the new API changes along with it.
{/* truncate */}
## Deploy this version
<Tabs>
@ -67,7 +60,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3.1 Pro on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
@ -147,4 +140,3 @@ curl -X POST http://localhost:4000/v1/chat/completions \
| `high` | `high` |
| `disable` | `minimal` |
| `none` | `minimal` |

26
docs/my-website/blog/gemini_3/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3
title: "DAY 0 Support: Gemini 3 on LiteLLM"
date: 2025-11-19T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Common questions and best practices for using gemini-3-pro-preview with LiteLLM Proxy and SDK."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -29,6 +20,8 @@ This guide covers common questions and best practices for using `gemini-3-pro-pr
:::
{/* truncate */}
## Quick Start
<Tabs>
@ -976,8 +969,7 @@ messages.append(response.choices[0].message) # ✅ Includes thought signatures
## Additional Resources
- [Gemini Provider Documentation](../gemini.md)
- [Thought Signatures Guide](../gemini.md#thought-signatures)
- [Reasoning Content Documentation](../../reasoning_content.md)
- [Function Calling Guide](../../function_calling.md)
- [Gemini Provider Documentation](../../docs/providers/gemini)
- [Thought Signatures Guide](../../docs/providers/gemini#thought-signatures)
- [Reasoning Content Documentation](../../docs/reasoning_content)
- [Function Calling Guide](../../docs/completion/function_call)

21
docs/my-website/blog/gemini_3_1_flash_lite/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_1_flash_lite_preview
title: "DAY 0 Support: Gemini 3.1 Flash Lite Preview on LiteLLM"
date: 2026-03-03T08:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3.1 Flash Lite Preview on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms, supernova]
hide_table_of_contents: false
@ -32,6 +23,8 @@ LiteLLM now supports `gemini-3.1-flash-lite-preview` with full day 0 support!
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
{/* truncate */}
## Deploy this version
<Tabs>
@ -150,7 +143,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3.1 Flash Lite Preview o
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
@ -172,4 +165,4 @@ LiteLLM automatically maps OpenAI's `reasoning_effort` parameter to Gemini's `th
| `medium` | `medium` | Balanced reasoning for moderate complexity |
| `high` | `high` | Maximum reasoning depth, complex problems |
| `disable` | `minimal` | Disable extended reasoning |
| `none` | `minimal` | No extended reasoning |
| `none` | `minimal` | No extended reasoning |

20
docs/my-website/blog/gemini_3_flash/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_flash
title: "DAY 0 Support: Gemini 3 Flash on LiteLLM"
date: 2025-12-17T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3 Flash on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -32,6 +23,8 @@ LiteLLM now supports `gemini-3-flash-preview` and all the new API changes along
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
{/* truncate */}
## Deploy this version
<Tabs>
@ -80,7 +73,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3 Flash on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
- Function calling with thought signatures
@ -252,4 +245,3 @@ If using this model via vertex_ai, keep the location as global as this is the on
| `high` | `high` |
| `disable` | `minimal` |
| `none` | `minimal` |

7
docs/my-website/blog/gemini_embedding_2_multimodal/index.md
View file

@ -3,10 +3,7 @@ slug: gemini_embedding_2_multimodal
title: "Gemini Embedding 2 Preview: Multimodal Embeddings on LiteLLM"
date: 2025-03-11T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- sameer
description: "Generate embeddings from text, images, audio, video, and PDFs with gemini-embedding-2-preview on LiteLLM via Gemini API and Vertex AI."
tags: [gemini, embeddings, multimodal, vertex ai]
hide_table_of_contents: false
@ -19,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports **multimodal embeddings** with `gemini-embedding-2-preview`—generating a single embedding from a mix of text, images, audio, video, and PDF content. Available via both the **Gemini API** (API key) and **Vertex AI** (GCP credentials).
{/* truncate */}
## Supported Input Types
| Modality | Supported Formats |

17
docs/my-website/blog/gpt_5_3_codex/index.md
View file

@ -3,18 +3,9 @@ slug: gpt_5_3_codex
title: "Day 0 Support: GPT-5.3-Codex"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Day 0 support for GPT-5.3-Codex on LiteLLM, including phase parameter handling for Responses API."
tags: [openai, gpt-5.3-codex, codex, day 0 support]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports GPT-5.3-Codex on Day 0, including support for the new assistant `phase` metadata on Responses API output items.
{/* truncate */}
## Why `phase` matters for GPT-5.3-Codex
`phase` appears on assistant output items and helps distinguish preamble/commentary turns from final closeout responses.

17
docs/my-website/blog/gpt_5_4/index.md
View file

@ -3,18 +3,9 @@ slug: gpt_5_4
title: "Day 0 Support: GPT-5.4"
date: 2026-03-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "GPT-5.4 model support in LiteLLM"
tags: [openai, gpt-5.4, completion]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports fully GPT-5.4!
{/* truncate */}
## Docker Image
```bash

78
docs/my-website/blog/guardrail_logging_secret_exposure_incident/index.md Normal file
View file

@ -0,0 +1,78 @@
---
slug: guardrail-logging-secret-exposure-incident
title: "Incident Report: Guardrail logging exposed secret headers in spend logs and traces"
date: 2026-03-18T10:00:00
authors:
- litellm
tags: [incident-report, security, guardrails]
hide_table_of_contents: false
---
**Date:** March 18, 2026
**Duration:** Unknown
**Severity:** High
**Status:** Resolved
## Summary
When a custom guardrail returned the full LiteLLM request/data dictionary, the guardrail response logged by LiteLLM could include `secret_fields.raw_headers`, including plaintext `Authorization` headers containing API keys or other credentials.
This information could then propagate to logging and observability surfaces that consume guardrail metadata, including:
- **Spend logs in the LiteLLM UI:** visible to admins with access to spend-log data
- **OpenTelemetry traces:** visible to anyone with access to the relevant telemetry backend
LLM calls, proxy routing, and provider execution were not blocked by this bug. The impact was exposure of sensitive request headers in observability and logging paths.
{/* truncate */}
---
## Background
LiteLLM keeps internal request data (including request headers) for use during the call. That data is not meant to be written to logs or telemetry.
When custom guardrails run, their outcomes are logged so they can appear in spend logs, OpenTelemetry traces, and other observability backends. If a guardrail returned the full request payload instead of a minimal result, that internal request data could be included in what was logged. Before the fix, the guardrail logging path did not strip that data before sending it to those systems.
```mermaid
flowchart TD
inboundRequest["1. Incoming proxy request"] --> storeSecrets["2. Store internal request data"]
storeSecrets --> guardrailRuns["3. Custom guardrail runs"]
guardrailRuns --> fullDataReturn["4. Guardrail returns full request payload"]
fullDataReturn --> loggingBuild["5. Build guardrail log payload"]
loggingBuild --> spendLogs["6a. Persist to spend logs / UI"]
loggingBuild --> otelTraces["6b. Attach to OTEL guardrail spans"]
```
---
## Root Cause
The root cause was incomplete sanitization in the guardrail logging path. When building the payload that gets sent to spend logs and traces, LiteLLM prepared guardrail responses for logging but did not strip internal request data (such as headers) from them. If a guardrail returned a response that included that data, it was passed through to the logging and observability systems unchanged.
---
## Impact
This issue required all of the following:
1. A custom guardrail returned the full LiteLLM request/data dictionary, or another response object containing `secret_fields`.
2. LiteLLM logged that guardrail response through the standard guardrail logging path.
3. An operator, admin, or telemetry consumer had access to the resulting logs or traces.
When those conditions were met, sensitive values could become visible through:
- **Spend logs / UI responses:** guardrail metadata could be included in spend-log payloads rendered in the admin UI.
- **OpenTelemetry traces:** `guardrail_response` could be written as a span attribute on guardrail spans.
- **Other downstream observability backends:** any integration consuming the same guardrail metadata could receive the leaked values.
This was a logging and telemetry exposure bug. It did not let callers bypass auth, access other tenants directly, or change model behavior, but it could expose plaintext credentials to people with access to those observability systems.
---
## Guidance For Users
- Upgrade to LiteLLM 1.82.3+.
- If you operated custom guardrails that return the full request/data dict, review whether spend logs or telemetry traces were retained during the affected period.
- Rotate any credentials that may have appeared in `Authorization` or other forwarded request headers in those systems.
- Apply least-privilege access controls to spend-log views and telemetry backends that may contain request-derived metadata.

16
docs/my-website/blog/httpx_cache_eviction_incident/index.md
View file

@ -3,17 +3,9 @@ slug: httpx-cache-eviction-incident
title: "Incident Report: Cache Eviction Closes In-Use httpx Clients"
date: 2026-02-27T10:00:00
authors:
- name: Ryan Crabbe
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- ryan
- ishaan-alt
- krrish
tags: [incident-report, caching, stability]
hide_table_of_contents: false
---
@ -31,6 +23,8 @@ A change to improve Redis connection pool cleanup introduced a regression that c
**Impact:** Any proxy instance that hit the cache TTL (default 10 minutes) or capacity limit (200 entries) would have its httpx clients closed out from under it, causing requests to LLM providers to fail with connection errors.
{/* truncate */}
---
## Background

18
docs/my-website/blog/litellm_observatory/index.md
View file

@ -3,18 +3,9 @@ slug: litellm-observatory
title: "Improve release stability with 24 hour load tests"
date: 2026-02-06T10:00:00
authors:
- name: Alexsander Hamir
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- alexsander
- krrish
- ishaan-alt
description: "How we built a long-running, release-validation system to catch regressions before they reach users."
tags: [testing, observability, reliability, releases]
hide_table_of_contents: false
@ -28,6 +19,8 @@ As LiteLLM adoption has grown, so have expectations around reliability, performa
This post introduces **LiteLLM Observatory**, a long-running release-validation system we built to catch regressions before they reach users.
{/* truncate */}
---
## Why We Built the Observatory
@ -133,4 +126,3 @@ Reliability is an ongoing investment.
LiteLLM Observatory is one of several systems we’re building to continuously raise the bar on release quality and operational safety. As LiteLLM evolves, so will our validation tooling, informed by real-world usage and lessons learned.
We’ll continue to share those improvements openly as we go.

17
docs/my-website/blog/minimax_m2_5/index.md
View file

@ -3,18 +3,9 @@ slug: minimax_m2_5
title: "Day 0 Support: MiniMax-M2.5"
date: 2026-02-12T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Day 0 support for MiniMax-M2.5 on LiteLLM"
tags: [minimax, M2.5, llm]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports MiniMax-M2.5 on Day 0. Use it across OpenAI-compatible and Anthropic-compatible APIs through the LiteLLM AI Gateway.
{/* truncate */}
## Supported Models
LiteLLM supports the following MiniMax models:

5
docs/my-website/blog/model_cost_map_incident/index.md
View file

@ -3,10 +3,7 @@ slug: model-cost-map-incident
title: "Incident Report: Invalid model cost map on main"
date: 2026-02-10T10:00:00
authors:
- name: Ishaan Jaffer
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/ishaanjaffer/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- ishaan
tags: [incident-report, stability]
hide_table_of_contents: false
---

111
docs/my-website/blog/realtime_webrtc_http_endpoints/index.md Normal file
View file

@ -0,0 +1,111 @@
---
slug: realtime_webrtc_http_endpoints
title: "Realtime WebRTC HTTP Endpoints"
date: 2026-03-12T10:00:00
authors:
- sameer
- krrish
- ishaan-alt
description: "Use the LiteLLM proxy to route OpenAI-style WebRTC realtime via HTTP: client_secrets and SDP exchange."
tags: [realtime, webrtc, proxy, openai]
hide_table_of_contents: false
---
import WebRTCTester from '@site/src/components/WebRTCTester';
Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth and key management.
{/* truncate */}
## How it works
![WebRTC flow: Browser, LiteLLM Proxy, and OpenAI/Azure](../../img/webrtc_flow.png)
**Flow of generating ephemeral token**
![Ephemeral token flow: Browser requests token, LiteLLM gets real token from OpenAI, returns encrypted token](../../img/ephemeral_token.png)
## Proxy Setup
```yaml
model_list:
- model_name: gpt-4o-realtime
litellm_params:
model: openai/gpt-4o-realtime-preview-2024-12-17
api_key: os.environ/OPENAI_API_KEY
model_info:
mode: realtime
```
**Azure:** use `model: azure/gpt-4o-realtime-preview`, `api_key`, `api_base`.
```bash
litellm --config /path/to/config.yaml
```
## Try it live
<WebRTCTester />
## Client Usage
**1. Get token** - `POST /v1/realtime/client_secrets` with LiteLLM API key and `{ model }`.
**2. WebRTC handshake** - Create `RTCPeerConnection`, add mic track, create data channel `oai-events`, send SDP offer to `POST /v1/realtime/calls` with `Authorization: Bearer <encrypted_token>` and `Content-Type: application/sdp`.
**3. Events** - Use the data channel for `session.update` and other events.
<details>
<summary>Full code example</summary>
```javascript
// 1. Token
const r = await fetch("http://proxy:4000/v1/realtime/client_secrets", {
method: "POST",
headers: { "Authorization": "Bearer sk-litellm-key", "Content-Type": "application/json" },
body: JSON.stringify({ model: "gpt-4o-realtime" }),
});
const { client_secret } = await r.json();
const token = client_secret.value;
// 2. WebRTC
const pc = new RTCPeerConnection();
const audio = document.createElement("audio");
audio.autoplay = true;
pc.ontrack = (e) => (audio.srcObject = e.streams[0]);
const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(ms.getTracks()[0]);
const dc = pc.createDataChannel("oai-events");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
const sdpRes = await fetch("http://proxy:4000/v1/realtime/calls", {
method: "POST",
headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/sdp" },
body: offer.sdp,
});
await pc.setRemoteDescription({ type: "answer", sdp: await sdpRes.text() });
// 3. Events
dc.send(JSON.stringify({ type: "session.update", session: { instructions: "..." } }));
```
</details>
## FAQ
**Q: What do I do if I get a 401 Token expired error?**
A: Tokens are short-lived. Get a fresh token right before creating the WebRTC offer.
**Q: Which key should I use for `/v1/realtime/calls`?**
A: Use the **encrypted token** from `client_secrets`, not your raw API key.
**Q: Should I pass the `model` parameter when making the call?**
A: No, the encrypted token already encodes all routing information including model.
**Q: How do I resolve Azure `api-version` errors?**
A: Set the correct `api_version` in `litellm_params` (or via the `AZURE_API_VERSION` environment variable), along with the right `api_base` and deployment values.
**Q: What if I get no audio?**
A: Make sure you grant microphone permission, ensure `pc.ontrack` assigns the audio element with `autoplay` enabled, check your network/firewall for WebRTC traffic, and inspect the browser console for ICE or SDP errors.

15
docs/my-website/blog/responses_api_encrypted_content_incident/index.md
View file

@ -3,18 +3,9 @@ slug: responses-api-encrypted-content-incident
title: "Incident Report: Encrypted Content Failures in Multi-Region Responses API Load Balancing"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, proxy, responses-api, load-balancing]
hide_table_of_contents: false
---

14
docs/my-website/blog/server_root_path/index.md
View file

@ -3,17 +3,9 @@ slug: server-root-path-incident
title: "Incident Report: SERVER_ROOT_PATH regression broke UI routing"
date: 2026-02-21T10:00:00
authors:
- name: Yuneng Jiang
title: SWE @ LiteLLM (Full Stack)
url: https://www.linkedin.com/in/yunengjiang/
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- yuneng
- ishaan-alt
- krrish
tags: [incident-report, ui, stability]
hide_table_of_contents: false
---

17
docs/my-website/blog/sub_millisecond_proxy_overhead/index.md
View file

@ -3,18 +3,9 @@ slug: sub-millisecond-proxy-overhead
title: "Achieving Sub-Millisecond Proxy Overhead"
date: 2026-02-02T10:00:00
authors:
- name: Alexsander Hamir
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- alexsander
- krrish
- ishaan-alt
description: "Our Q1 performance target and architectural direction for achieving sub-millisecond proxy overhead on modest hardware."
tags: [performance, architecture]
hide_table_of_contents: false
@ -32,6 +23,8 @@ Proxy overhead refers to the latency introduced by LiteLLM itself, independent o
To measure it, we run the same workload directly against the provider and through LiteLLM at identical QPS (for example, 1,000 QPS) and compare the latency delta. To reduce noise, the load generator, LiteLLM, and a mock LLM endpoint all run on the same machine, ensuring the difference reflects proxy overhead rather than network latency.
{/* truncate */}
---
## Where We're Coming From

121
docs/my-website/blog/video_characters_litellm/index.md Normal file
View file

@ -0,0 +1,121 @@
---
slug: video_characters_api
title: "New Video Characters, Edit and Extension API support"
date: 2026-03-16T10:00:00
authors:
- sameer
- krrish
- ishaan-alt
description: "LiteLLM now supports creating, retrieving, and managing reusable video characters across multiple video generations."
tags: [videos, characters, proxy, routing]
hide_table_of_contents: false
---
LiteLLM now supoports videos character, edit and extension apis.
{/* truncate */}
## What's New
Four new endpoints for video character operations:
- **Create character** - Upload a video to create a reusable asset
- **Get character** - Retrieve character metadata
- **Edit video** - Modify generated videos
- **Extend video** - Continue clips with character consistency
**Available from:** LiteLLM v1.83.0+
## Quick Example
```python
import litellm
# Create character from video
character = litellm.avideo_create_character(
name="Luna",
video=open("luna.mp4", "rb"),
custom_llm_provider="openai",
model="sora-2"
)
print(f"Character: {character.id}")
# Use in generation
video = litellm.avideo(
model="sora-2",
prompt="Luna dances through a magical forest.",
characters=[{"id": character.id}],
seconds="8"
)
# Get character info
fetched = litellm.avideo_get_character(
character_id=character.id,
custom_llm_provider="openai"
)
# Edit with character preserved
edited = litellm.avideo_edit(
video_id=video.id,
prompt="Add warm golden lighting"
)
# Extend sequence
extended = litellm.avideo_extension(
video_id=video.id,
prompt="Luna waves goodbye",
seconds="5"
)
```
## Via Proxy
```bash
# Create character
curl -X POST "http://localhost:4000/v1/videos/characters" \
-H "Authorization: Bearer sk-litellm-key" \
-F "video=@luna.mp4" \
-F "name=Luna"
# Get character
curl -X GET "http://localhost:4000/v1/videos/characters/char_abc123def456" \
-H "Authorization: Bearer sk-litellm-key"
# Edit video
curl -X POST "http://localhost:4000/v1/videos/edits" \
-H "Authorization: Bearer sk-litellm-key" \
-H "Content-Type: application/json" \
-d '{
"video": {"id": "video_xyz789"},
"prompt": "Add warm golden lighting and enhance colors"
}'
# Extend video
curl -X POST "http://localhost:4000/v1/videos/extensions" \
-H "Authorization: Bearer sk-litellm-key" \
-H "Content-Type: application/json" \
-d '{
"video": {"id": "video_xyz789"},
"prompt": "Luna waves goodbye and walks into the sunset",
"seconds": "5"
}'
```
## Managed Character IDs
LiteLLM automatically encodes provider and model metadata into character IDs:
**What happens:**
```
Upload character "Luna" with model "sora-2" on OpenAI
↓
LiteLLM creates: char_abc123def456 (contains provider + model_id)
↓
When you reference it later, LiteLLM decodes automatically
↓
Router knows exactly which deployment to use
```
**Behind the scenes:**
- Character ID format: `character_<base64_encoded_metadata>`
- Metadata includes: provider, model_id, original_character_id
- Transparent to you - just use the ID, LiteLLM handles routing

15
docs/my-website/blog/vllm_embeddings_incident/index.md
View file

@ -3,18 +3,9 @@ slug: vllm-embeddings-incident
title: "Incident Report: vLLM Embeddings Broken by encoding_format Parameter"
date: 2026-02-18T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, embeddings, vllm]
hide_table_of_contents: false
---

6
docs/my-website/docs/files_endpoints.md
View file

@ -326,4 +326,10 @@ print("file content=", content.text)
### [Bedrock](./providers/bedrock_batches#4-retrieve-batch-results)
### [Anthropic](./providers/anthropic#files-api)
:::note
Anthropic Files API has a different purpose than OpenAI's. It's **not** for Batches or Fine-tuning—it's for uploading files once and referencing them by `file_id` in multiple messages, avoiding re-uploads. File API operations are free — file content used in Messages requests is priced as input tokens.
:::
## [Swagger API Reference](https://litellm-api.up.railway.app/#/files)

78
docs/my-website/docs/guides/index.md Normal file
View file

@ -0,0 +1,78 @@
---
title: Guides
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
**Guides** are focused references organized by the job you are trying to do with LiteLLM: make requests, use tools, handle media, manage context, or operate the gateway safely.
> New to LiteLLM or not sure whether you need the SDK or Gateway path first? Start at [Learn →](/docs/learn)
---
## Build With LiteLLM
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Core Requests",
description: "Streaming, batching, structured outputs, and reasoning behavior.",
to: "/docs/guides/core_request_response_patterns",
},
{
icon: "🛠️",
title: "Tool Calling",
description: "Function calling, web tools, interception patterns, computer use, code interpreter, and tool-call hygiene.",
to: "/docs/guides/tools_integrations",
},
{
icon: "🖼️",
title: "Multimodal I/O",
description: "Vision, audio, PDFs, image generation, and video generation.",
to: "/docs/guides/multimodal_io",
},
{
icon: "📚",
title: "Retrieval & Knowledge",
description: "Vector stores, file search, citations, and knowledge-base routing.",
to: "/docs/guides/retrieval_knowledge",
},
{
icon: "🧠",
title: "Prompts & Context",
description: "Prompt caching, trimming, formatting, assistant prefill, and predicted outputs.",
to: "/docs/guides/prompts_context",
},
]}
/>
---
## Operate & Extend
<NavigationCards
columns={3}
items={[
{
icon: "🎛️",
title: "Compatibility & Extensibility",
description: "Provider-specific params, model aliases, fine-tuned models, and adapters.",
to: "/docs/guides/compatibility_extensibility",
},
{
icon: "🧪",
title: "Reliability, Testing & Spend",
description: "Retries, fallbacks, mock responses, and budget controls.",
to: "/docs/guides/reliability_testing_spend",
},
{
icon: "🔒",
title: "Security & Network",
description: "SSL, custom CA bundles, HTTP proxy settings, and per-service verification.",
to: "/docs/guides/security_network",
},
]}
/>

1051
docs/my-website/docs/index.md
View file

File diff suppressed because it is too large Load diff

340
docs/my-website/docs/integrations/index.md
View file

@ -1,18 +1,336 @@
# Integrations
---
title: Integrations
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
This section covers integrations with various tools and services that can be used with LiteLLM (either Proxy or SDK).
## AI Agent Frameworks
- **[Letta](./letta.md)** - Build stateful LLM agents with persistent memory using LiteLLM Proxy
---
## Development Tools
- **[OpenWebUI](../tutorials/openweb_ui.md)** - Self-hosted ChatGPT-style interface
## Observability
## Observability & Monitoring
- **[Langfuse](../observability/langfuse_integration.md)** - LLM observability and analytics
- **[Prometheus](../proxy/prometheus.md)** - Metrics collection and monitoring
- **[PagerDuty](../proxy/pagerduty.md)** - Incident response and alerting
- **[Datadog](../observability/datadog.md)**
Track, debug, and analyze LLM calls with observability platforms.
<NavigationCards
columns={3}
items={[
{
icon: "🪢",
title: "Langfuse",
description: "LLM observability and analytics.",
to: "/docs/observability/langfuse_integration",
},
{
icon: "🐶",
title: "Datadog",
description: "Metrics, traces, and dashboards.",
to: "/docs/observability/datadog",
},
{
icon: "📡",
title: "OpenTelemetry",
description: "Vendor-neutral tracing.",
to: "/docs/observability/opentelemetry_integration",
},
{
icon: "🔗",
title: "LangSmith",
description: "LLM debugging and evaluation.",
to: "/docs/observability/langsmith_integration",
},
{
icon: "🔥",
title: "Arize / Phoenix",
description: "ML observability and evaluation.",
to: "/docs/observability/arize_integration",
},
{
icon: "🌀",
title: "Helicone",
description: "LLM request logging and analytics.",
to: "/docs/observability/helicone_integration",
},
{
icon: "📊",
title: "MLflow",
description: "Experiment tracking.",
to: "/docs/observability/mlflow",
},
{
icon: "🏋️",
title: "Weights & Biases",
description: "ML experiment tracking.",
to: "/docs/observability/wandb_integration",
},
{
icon: "📉",
title: "PostHog",
description: "Product analytics.",
to: "/docs/observability/posthog_integration",
},
]}
/>
Click into each section to learn more about the integrations.
[View all observability integrations →](/docs/integrations/observability_integrations)
---
## Alerting & Monitoring
Set up alerts, metrics collection, and infrastructure monitoring.
<NavigationCards
columns={2}
items={[
{
icon: "📈",
title: "Prometheus",
description: "Metrics collection and monitoring.",
to: "../proxy/prometheus",
},
{
icon: "🚨",
title: "PagerDuty",
description: "Incident response and alerting.",
to: "../proxy/pagerduty",
},
{
icon: "🔔",
title: "Alerting",
description: "Slack, Teams, and webhook alerts.",
to: "../proxy/alerting",
},
{
icon: "🔍",
title: "Pyroscope",
description: "Continuous profiling.",
to: "../proxy/pyroscope_profiling",
},
]}
/>
---
## Guardrail Providers
Add safety and content filtering to LLM calls.
<NavigationCards
columns={3}
items={[
{
icon: "🛡️",
title: "Lakera AI",
description: "Prompt injection detection.",
to: "/docs/proxy/guardrails/lakera_ai",
},
{
icon: "☁️",
title: "Azure Content Safety",
description: "Content moderation.",
to: "/docs/proxy/guardrails/azure_content_guardrail",
},
{
icon: "🛏️",
title: "Bedrock Guardrails",
description: "AWS Bedrock safety.",
to: "/docs/proxy/guardrails/bedrock",
},
{
icon: "🤖",
title: "OpenAI Moderation",
description: "OpenAI content policy.",
to: "/docs/proxy/guardrails/openai_moderation",
},
{
icon: "🔐",
title: "Secret Detection",
description: "Prevent credential leaks.",
to: "/docs/proxy/guardrails/secret_detection",
},
{
icon: "🕵️",
title: "PII Masking",
description: "Mask sensitive data.",
to: "/docs/proxy/guardrails/pii_masking_v2",
},
]}
/>
[View all guardrail providers →](/docs/guardrail_providers)
---
## Policies
Define and enforce usage policies across your LLM deployment.
<NavigationCards
columns={3}
items={[
{
icon: "📋",
title: "Guardrail Policies",
description: "Policy-based guardrail rules.",
to: "../proxy/guardrails/guardrail_policies",
},
{
icon: "🔀",
title: "Policy Flow Builder",
description: "Visual policy configuration.",
to: "../proxy/guardrails/policy_flow_builder",
},
{
icon: "📄",
title: "Policy Templates",
description: "Pre-built policy templates.",
to: "../proxy/guardrails/policy_templates",
},
]}
/>
---
## AI Tools
Connect LiteLLM to AI-powered coding and productivity tools.
<NavigationCards
columns={3}
items={[
{
icon: "💬",
title: "OpenWebUI",
description: "Self-hosted ChatGPT-style interface.",
to: "../tutorials/openweb_ui",
},
{
icon: "🤖",
title: "Claude Code",
description: "Use LiteLLM with Claude Code.",
to: "../tutorials/claude_responses_api",
},
{
icon: "🖱️",
title: "Cursor",
description: "AI code editor integration.",
to: "../tutorials/cursor_integration",
},
{
icon: "🐙",
title: "GitHub Copilot",
description: "GitHub Copilot integration.",
to: "../tutorials/github_copilot_integration",
},
{
icon: "💻",
title: "OpenCode",
description: "Open source coding assistant.",
to: "../tutorials/opencode_integration",
},
{
icon: "🔧",
title: "Retool Assist",
description: "Retool AI assistant.",
to: "../tutorials/retool_assist",
},
]}
/>
---
## Agent SDKs
Use LiteLLM with agent frameworks and SDKs.
<NavigationCards
columns={3}
items={[
{
icon: "🤖",
title: "OpenAI Agents SDK",
description: "Build agents with OpenAI's SDK.",
to: "../tutorials/openai_agents_sdk",
},
{
icon: "🧠",
title: "Claude Agent SDK",
description: "Build agents with Anthropic's SDK.",
to: "../tutorials/claude_agent_sdk",
},
{
icon: "🌐",
title: "Google ADK",
description: "Google Agent Development Kit.",
to: "../tutorials/google_adk",
},
{
icon: "🚀",
title: "CopilotKit",
description: "In-app AI copilots.",
to: "../tutorials/copilotkit_sdk",
},
{
icon: "🧬",
title: "Letta",
description: "Build stateful LLM agents with persistent memory.",
to: "./letta",
},
{
icon: "🎙️",
title: "LiveKit",
description: "Real-time voice and video AI agents.",
to: "../tutorials/livekit_xai_realtime",
},
]}
/>
---
## Prompt Management
Manage, version, and deploy prompts.
<NavigationCards
columns={3}
items={[
{
icon: "📝",
title: "LiteLLM Prompt Management",
description: "Built-in prompt management.",
to: "../proxy/litellm_prompt_management",
},
{
icon: "🔌",
title: "Custom Prompt Management",
description: "Bring your own prompt store.",
to: "../proxy/custom_prompt_management",
},
{
icon: "🔥",
title: "Arize Phoenix Prompts",
description: "Prompt management with Phoenix.",
to: "../proxy/arize_phoenix_prompts",
},
]}
/>
---
## Manage with AI Agents
Use AI agents to manage your LiteLLM deployment — create users, teams, keys, models, and more via natural language.
<NavigationCards
columns={1}
items={[
{
icon: "🤖",
title: "LiteLLM Skills",
description: "Manage LiteLLM via Claude Code — create keys, teams, models, and more using natural language commands.",
to: "../tutorials/claude_code_skills",
},
]}
/>

12
docs/my-website/docs/integrations/letta.md
View file

@ -920,9 +920,9 @@ for model in models:
## Resources
- [Letta Documentation](https://docs.letta.ai/)
- [LiteLLM Proxy Documentation](../proxy/quick_start.md)
- [LiteLLM SDK Documentation](../completion/input.md)
- [Function Calling Guide](../completion/function_call.md)
- [Observability Setup](../observability/langfuse_integration.md)
- [Router Configuration](../routing.md)
- [Letta Documentation](https://docs.letta.com/)
- [LiteLLM Proxy Documentation](/docs/simple_proxy)
- [LiteLLM SDK Documentation](/docs/#litellm-python-sdk)
- [Function Calling Guide](/docs/completion/function_call)
- [Observability Setup](/docs/integrations/observability_integrations)
- [Router Configuration](/docs/routing)

28
docs/my-website/docs/integrations/observability_index.md Normal file
View file

@ -0,0 +1,28 @@
---
title: Observability
sidebar_label: Overview
slug: observability_integrations
---
Track, debug, and analyze LLM calls with observability platforms.
import NavigationCards from '@site/src/components/NavigationCards';
## Observability Integrations
<NavigationCards
columns={3}
items={[
{ icon: "🪢", title: "Langfuse", description: "LLM observability and analytics.", to: "/docs/observability/langfuse_integration" },
{ icon: "🐶", title: "Datadog", description: "Metrics, traces, and dashboards.", to: "/docs/observability/datadog" },
{ icon: "📡", title: "OpenTelemetry", description: "Vendor-neutral tracing.", to: "/docs/observability/opentelemetry_integration" },
{ icon: "🔗", title: "LangSmith", description: "LLM debugging and evaluation.", to: "/docs/observability/langsmith_integration" },
{ icon: "🔥", title: "Arize / Phoenix", description: "ML observability and evaluation.", to: "/docs/observability/arize_integration" },
{ icon: "🌀", title: "Helicone", description: "LLM request logging and analytics.", to: "/docs/observability/helicone_integration" },
{ icon: "📊", title: "MLflow", description: "Experiment tracking.", to: "/docs/observability/mlflow" },
{ icon: "🏋️", title: "Weights & Biases", description: "ML experiment tracking.", to: "/docs/observability/wandb_integration" },
{ icon: "📉", title: "PostHog", description: "Product analytics.", to: "/docs/observability/posthog_integration" },
]}
/>
[View all observability integrations →](/docs/observability/callbacks)

2
docs/my-website/docs/integrations/websearch_interception.md
View file

@ -375,7 +375,7 @@ search_tools:
- [Search Providers](../search/index.md) - Detailed search provider setup
- [Claude Code WebSearch](../tutorials/claude_code_websearch.md) - Using with Claude Code
- [Tool Calling](../completion/function_call.md) - General tool calling documentation
- [Callbacks](./custom_callback.md) - Custom callback documentation
- [Callbacks](../observability/custom_callback.md) - Custom callback documentation
## Technical Details

174
docs/my-website/docs/learn/gateway_quickstart.md Normal file
View file

@ -0,0 +1,174 @@
---
title: Gateway Quickstart
sidebar_label: Gateway Quickstart
description: Start LiteLLM Gateway, add models and keys, then connect applications and SDKs to one shared endpoint.
---
import NavigationCards from '@site/src/components/NavigationCards';
Use this path if you need one shared OpenAI-compatible endpoint for a team or platform.
If you need a Docker or database-first setup, use the [Docker + Database tutorial](/docs/proxy/docker_quick_start). Otherwise, use the steps below to get to a working request fast.
## 1. Install The Gateway
```bash
pip install 'litellm[proxy]'
```
## 2. Set One Provider Key
```bash
export OPENAI_API_KEY="your-api-key"
```
## 3. Create `config.yaml`
```yaml
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
general_settings:
master_key: sk-1234
```
## 4. Start The Gateway
```bash
litellm --config config.yaml
```
You should see the proxy start on `http://0.0.0.0:4000`.
## 5. Send Your First Request
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Hello from LiteLLM Gateway"}
]
}'
```
## 6. Check The Response
If the request succeeds, the proxy returns `200 OK` with an OpenAI-style response.
The assistant text will be in:
```json
choices[0].message.content
```
If your gateway is routing to OpenAI, a real response can look like this:
```json
{
"id": "chatcmpl-abc123",
"created": 1677858242,
"model": "gpt-4o-mini-2024-07-18",
"object": "chat.completion",
"system_fingerprint": "fp_406d6473f8",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! How can I assist you today?",
"tool_calls": null,
"function_call": null,
"annotations": []
}
}
],
"usage": {
"completion_tokens": 9,
"prompt_tokens": 13,
"total_tokens": 22,
"completion_tokens_details": {
"accepted_prediction_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0,
"rejected_prediction_tokens": 0
},
"prompt_tokens_details": {
"audio_tokens": 0,
"cached_tokens": 0
}
},
"service_tier": "default"
}
```
`id`, `created`, the resolved model version, token counts, and message text will vary by request. Other providers may return a smaller or slightly different set of fields, but `choices[0].message.content` is the main field to read.
## 7. Add Keys And The UI
If you need virtual keys, spend tracking, or the admin UI, add a database next.
- Add `database_url` under `general_settings`
- Use [Virtual keys](/docs/proxy/virtual_keys) for key creation and budgets
- Use [Admin UI](/docs/proxy/ui) to manage models and keys
- Use the [Docker + Database tutorial](/docs/proxy/docker_quick_start) if you want a fuller setup
## 8. Pick Your Next Step
<NavigationCards
columns={3}
items={[
{
icon: "🖥️",
title: "Make LLM Requests",
description: "Point LiteLLM or OpenAI-compatible clients to the gateway.",
to: "/docs/proxy/user_keys",
},
{
icon: "🎛️",
title: "Model Config",
description: "Add more models and gateway settings.",
to: "/docs/proxy/configs",
},
{
icon: "🔑",
title: "Virtual Keys",
description: "Create keys, budgets, and access controls.",
to: "/docs/proxy/virtual_keys",
},
{
icon: "📈",
title: "Add Logging",
description: "Capture logs, spend, and traces.",
to: "/docs/proxy/logging",
},
{
icon: "🔀",
title: "Load Balance",
description: "Route across deployments, regions, or providers.",
to: "/docs/proxy/load_balancing",
},
{
icon: "🛡️",
title: "Add Guardrails",
description: "Add safety checks and policy enforcement.",
to: "/docs/proxy/guardrails/quick_start",
},
{
icon: "📊",
title: "Reliability",
description: "Configure retries, fallbacks, and timeouts.",
to: "/docs/proxy/reliability",
},
]}
/>
## When To Use The SDK Path Instead
If you only need to call models from one application and do not need centralized auth or shared infrastructure, start with the [SDK Quickstart](/docs/learn/sdk_quickstart) instead.

117
docs/my-website/docs/learn/index.md Normal file
View file

@ -0,0 +1,117 @@
---
title: Learn LiteLLM
sidebar_label: Learn
slug: /learn
---
import NavigationCards from '@site/src/components/NavigationCards';
LiteLLM gives you one OpenAI-compatible interface for 100+ LLM providers. Start with the path that matches your setup.
---
## Start Here
Pick one path first.
<NavigationCards
columns={2}
items={[
{
icon: "🐍",
title: "SDK Quickstart",
description: "Use LiteLLM directly in application code.",
listDescription: [
"Install",
"First request",
"Next SDK features",
],
to: "/docs/learn/sdk_quickstart",
},
{
icon: "🖥️",
title: "Gateway Quickstart",
description: "Run LiteLLM as a shared gateway.",
listDescription: [
"Start proxy",
"Add models and keys",
"Connect clients",
],
to: "/docs/learn/gateway_quickstart",
},
]}
/>
---
## Common Tasks
Jump to a specific task.
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Stream Responses",
description: "Return tokens as they are generated.",
to: "/docs/guides/core_request_response_patterns",
},
{
icon: "🧰",
title: "Use Tools",
description: "Add function calling to your app.",
to: "/docs/guides/tools_integrations",
},
{
icon: "🔀",
title: "Add Routing",
description: "Retries, fallbacks, and load balancing.",
to: "/docs/routing-load-balancing",
},
{
icon: "🔑",
title: "Set Up Keys",
description: "Gateway auth, virtual keys, and access control.",
to: "/docs/proxy/virtual_keys",
},
{
icon: "📈",
title: "Add Logging",
description: "Capture request logs and spend data.",
to: "/docs/proxy/logging",
},
{
icon: "🌐",
title: "Choose A Provider",
description: "Find provider-specific auth and params.",
to: "/docs/providers",
},
]}
/>
---
## Docs Map
Use these when you already know the type of doc you want.
<NavigationCards
columns={2}
items={[
{
icon: "📚",
title: "Guides",
description: "Feature reference.",
to: "/docs/guides",
},
{
icon: "🛠️",
title: "Tutorials",
description: "Step-by-step integrations.",
to: "/docs/tutorials",
},
]}
/>
Not sure where to start? Use [SDK Quickstart](/docs/learn/sdk_quickstart) for app code or [Gateway Quickstart](/docs/learn/gateway_quickstart) for shared infrastructure.

174
docs/my-website/docs/learn/sdk_quickstart.md Normal file
View file

@ -0,0 +1,174 @@
---
title: SDK Quickstart
sidebar_label: SDK Quickstart
description: Make your first LiteLLM SDK call, then jump to the right docs for the next feature you need.
---
import NavigationCards from '@site/src/components/NavigationCards';
Use this path if you are integrating LiteLLM directly into application code.
## 1. Install LiteLLM
```bash
pip install litellm
```
## 2. Set Provider Credentials
Start with one provider and set its environment variables.
- OpenAI: `OPENAI_API_KEY`
- Anthropic: `ANTHROPIC_API_KEY`
- Azure OpenAI: `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`
- Bedrock: standard AWS credentials
- Vertex AI: `VERTEXAI_PROJECT`, `VERTEXAI_LOCATION`
If you have not picked a provider yet, browse [all supported providers](/docs/providers).
## 3. Make Your First Call
```python
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
response = completion(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Hello, how are you?"}],
)
print(response.choices[0].message.content)
```
## 4. Check The Response
The line below:
```python
print(response.choices[0].message.content)
```
prints the assistant text, for example:
```text
Hello! I'm doing well, thanks for asking.
```
If you print the full object with:
```python
print(response)
```
you will see a Python `ModelResponse(...)` object. For an OpenAI-backed model, it can look like this:
```python
ModelResponse(
id='chatcmpl-abc123',
created=1773782130,
model='gpt-4o-2024-08-06',
object='chat.completion',
system_fingerprint='fp_4ff89bf575',
choices=[
Choices(
finish_reason='stop',
index=0,
message=Message(
content="Hello! I'm just a program, but I'm here to help you. How can I assist you today?",
role='assistant',
tool_calls=None,
function_call=None,
provider_specific_fields={'refusal': None},
annotations=[]
),
provider_specific_fields={}
)
],
usage=Usage(
completion_tokens=21,
prompt_tokens=13,
total_tokens=34,
completion_tokens_details=CompletionTokensDetailsWrapper(...),
prompt_tokens_details=PromptTokensDetailsWrapper(...)
),
service_tier='default'
)
```
The same response follows an OpenAI-style shape. Conceptually, it looks like this:
```json
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1677858242,
"model": "gpt-4o",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! I'm doing well, thanks for asking."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 13,
"completion_tokens": 12,
"total_tokens": 25
}
}
```
`id`, `created`, token counts, and message text will vary by request.
If you call an OpenAI-backed model, you may also see extra fields such as `system_fingerprint`, `service_tier`, `tool_calls`, `function_call`, `annotations`, `provider_specific_fields`, and detailed token usage. For the full output reference, see [completion output](/docs/completion/output).
Need more provider examples? See the main [Getting Started](/docs/#quick-start) page.
## 5. Pick Your Next Step
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Stream Responses",
description: "Receive tokens incrementally with stream=True.",
to: "/docs/completion/stream",
},
{
icon: "🧰",
title: "Use Tools",
description: "Add function calling in a provider-agnostic way.",
to: "/docs/completion/function_call",
},
{
icon: "📦",
title: "Return JSON",
description: "Constrain responses to structured JSON output.",
to: "/docs/completion/json_mode",
},
{
icon: "🔀",
title: "Add Routing",
description: "Use retries, fallbacks, and load balancing in app code.",
to: "/docs/routing",
},
{
icon: "🌐",
title: "Choose A Provider",
description: "Find provider-specific auth, model naming, and params.",
to: "/docs/providers",
},
]}
/>
## When To Use Gateway Instead
Use LiteLLM Gateway if you need centralized auth, virtual keys, spend tracking, shared logging, or one OpenAI-compatible endpoint for multiple apps.
[Go to Gateway Quickstart →](/docs/learn/gateway_quickstart)

4
docs/my-website/docs/load_test_advanced.md
View file

@ -11,7 +11,7 @@ Tutorial on how to get to 1K+ RPS with LiteLLM Proxy on locust
- [Github releases](https://github.com/BerriAI/litellm/releases)
- [litellm docker containers](https://github.com/BerriAI/litellm/pkgs/container/litellm)
- [litellm database docker container](https://github.com/BerriAI/litellm/pkgs/container/litellm-database)
- [ ] Ensure you're following **ALL** [best practices for production](./proxy/production_setup.md)
- [ ] Ensure you're following **ALL** [best practices for production](./proxy/prod.md)
- [ ] Locust - Ensure you're Locust instance can create 1K+ requests per second
- 👉 You can use our **[maintained locust instance here](https://locust-load-tester-production.up.railway.app/)**
- If you're self hosting locust
@ -222,4 +222,4 @@ class MyUser(HttpUser):
def on_start(self):
self.api_key = os.getenv('API_KEY', 'sk-1234')
self.client.headers.update({'Authorization': f'Bearer {self.api_key}'})
```
```

294
docs/my-website/docs/mcp_zero_trust.md Normal file
View file

@ -0,0 +1,294 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# MCP Zero Trust Auth (JWT Signer)
![Zero Trust MCP Gateway](/img/mcp_zero_trust_gateway.png)
MCP servers have no built-in way to verify that a request actually came through LiteLLM. Without this guardrail, any client that can reach your MCP server directly can call tools — bypassing your access controls entirely.
`MCPJWTSigner` fixes this. It signs every outbound tool call with a short-lived RS256 JWT. Your MCP server verifies the signature against LiteLLM's public key. Requests that didn't go through LiteLLM have no valid signature and are rejected.
---
## Basic setup
Add the guardrail to your config and point your MCP server at LiteLLM's JWKS endpoint. Every tool call gets a signed JWT automatically — no changes needed on the client side.
```yaml title="config.yaml"
mcp_servers:
- server_name: weather
url: http://localhost:8000/mcp
transport: http
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
issuer: "https://my-litellm.example.com" # defaults to request base URL
audience: "mcp" # default: "mcp"
ttl_seconds: 300 # default: 300
```
**Bring your own signing key** — recommended for production. Auto-generated keys are lost on restart.
```bash
export MCP_JWT_SIGNING_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
# or point to a file
export MCP_JWT_SIGNING_KEY="file:///secrets/mcp-signing-key.pem"
```
**Build a verified MCP server with [FastMCP](https://gofastmcp.com):**
```python title="weather_server.py"
from fastmcp import FastMCP, Context
from fastmcp.server.auth.providers.jwt import JWTVerifier
auth = JWTVerifier(
jwks_uri="https://my-litellm.example.com/.well-known/jwks.json",
issuer="https://my-litellm.example.com",
audience="mcp",
algorithm="RS256",
)
mcp = FastMCP("weather-server", auth=auth)
@mcp.tool()
async def get_weather(city: str, ctx: Context) -> str:
caller = ctx.client_id # JWT `sub` — the verified user identity
return f"Weather in {city}: sunny, 72°F (requested by {caller})"
if __name__ == "__main__":
mcp.run(transport="http", host="0.0.0.0", port=8000)
```
FastMCP fetches the JWKS automatically and re-fetches when the signing key changes.
LiteLLM publishes OIDC discovery so MCP servers find the key without any manual configuration:
```
GET /.well-known/openid-configuration → { "jwks_uri": "https://<litellm>/.well-known/jwks.json" }
GET /.well-known/jwks.json → { "keys": [{ "kty": "RSA", "alg": "RS256", ... }] }
```
> **Read further only if you need to:** thread a corporate IdP identity into the JWT, enforce specific claims on callers, add custom metadata, use AWS Bedrock AgentCore Gateway, or debug JWT rejections.
---
## Thread IdP identity into MCP JWTs
By default the outbound JWT `sub` is LiteLLM's internal `user_id`. If your users authenticate with Okta, Azure AD, or another IdP, the MCP server sees a LiteLLM-internal ID — not the user's email or employee ID.
With verify+re-sign, LiteLLM validates the incoming IdP token first, then builds the outbound JWT using the real identity claims from that token. The MCP server gets the user's actual identity without ever having to trust the original IdP directly.
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
issuer: "https://my-litellm.example.com"
# Validate the incoming Bearer token against the IdP
access_token_discovery_uri: "https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration"
verify_issuer: "https://login.microsoftonline.com/{tenant}/v2.0"
verify_audience: "api://my-app"
# Which claim to use for `sub` in the outbound JWT — first non-empty value wins
end_user_claim_sources:
- "token:sub" # from the verified incoming JWT
- "token:email" # fallback to email
- "litellm:user_id" # last resort: LiteLLM's internal user_id
```
If the incoming token is **opaque** (not a JWT — some IdPs issue these), add an introspection endpoint. LiteLLM will POST the token to it (RFC 7662) and use the returned claims:
```yaml
token_introspection_endpoint: "https://idp.example.com/oauth2/introspect"
```
**Supported `end_user_claim_sources` values:**
| Source | Resolves to |
|--------|-------------|
| `token:<claim>` | Any claim from the verified incoming JWT (e.g. `token:sub`, `token:email`, `token:oid`) |
| `litellm:user_id` | LiteLLM's internal user ID |
| `litellm:email` | User email from LiteLLM auth context |
| `litellm:end_user_id` | End-user ID if set separately |
| `litellm:team_id` | Team ID from LiteLLM auth context |
---
## Block callers missing required attributes
Some MCP servers expose sensitive operations that should only be reachable by verified employees — not service accounts, not external API keys. You can enforce this at the LiteLLM layer so the MCP server never receives the request at all.
`required_claims` rejects with `403` if the incoming token is missing any listed claim. `optional_claims` forwards claims that are useful but not mandatory.
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
access_token_discovery_uri: "https://idp.example.com/.well-known/openid-configuration"
# Service accounts without `employee_id` are blocked before the tool runs
required_claims:
- "sub"
- "employee_id"
# Forward these into the outbound JWT when present — skipped silently if absent
optional_claims:
- "groups"
- "department"
```
**What the client sees when blocked:**
```json
HTTP 403
{ "error": "MCPJWTSigner: incoming token is missing required claims: ['employee_id']. Configure the IdP to include these claims." }
```
---
## Add custom metadata to every JWT
Your MCP server may need context that LiteLLM doesn't carry natively — which deployment sent the request, a tenant ID, an environment tag. Use claim operations to inject, override, or strip claims from the outbound JWT.
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
# add: insert only when the key is not already in the JWT
add_claims:
deployment_id: "prod-us-east-1"
tenant_id: "acme-corp"
# set: always override — even if the claim came from the incoming token
set_claims:
env: "production"
# remove: strip claims the MCP server shouldn't see
remove_claims:
- "nbf" # some validators reject nbf; remove it if yours does
```
Operations run in order — `add_claims` → `set_claims` → `remove_claims`. `set_claims` always wins over `add_claims`; `remove_claims` beats both.
---
## AWS Bedrock AgentCore Gateway
Bedrock AgentCore Gateway uses two separate JWTs: one to authenticate the transport connection and another to authorize tool calls. They need different `aud` values and TTLs — a single JWT won't work for both.
LiteLLM can issue both in one hook and inject them into separate headers:
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
issuer: "https://my-litellm.example.com"
audience: "mcp-resource" # for the MCP resource layer
ttl_seconds: 300
# Second JWT for the transport channel — same sub/act/scope, different aud + TTL
channel_token_audience: "bedrock-agentcore-gateway"
channel_token_ttl: 60 # transport tokens should be short-lived
```
LiteLLM injects two headers on every tool call:
- `Authorization: Bearer <resource-token>` — audience `mcp-resource`, TTL 300s
- `x-mcp-channel-token: Bearer <channel-token>` — audience `bedrock-agentcore-gateway`, TTL 60s
Both tokens are signed with the same LiteLLM key, so your MCP server only needs to trust one JWKS endpoint.
---
## Control which scopes go into the JWT
By default LiteLLM generates least-privilege scopes per request:
- Tool call → `mcp:tools/call mcp:tools/{name}:call`
- List tools → `mcp:tools/call mcp:tools/list`
If your MCP server does its own scope enforcement and needs a specific format, set `allowed_scopes` to replace auto-generation entirely:
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
allowed_scopes:
- "mcp:tools/call"
- "mcp:tools/list"
- "mcp:admin"
```
Every JWT carries exactly those scopes regardless of which tool is being called.
---
## Debug JWT rejections
Your MCP server is returning 401 and you're not sure what's in the JWT. Enable `debug_headers` and LiteLLM adds a `x-litellm-mcp-debug` response header with the key claims that were signed:
```yaml title="config.yaml"
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
debug_headers: true
```
Response header:
```
x-litellm-mcp-debug: v=1; kid=a3f1b2c4d5e6f708; sub=alice@corp.com; iss=https://my-litellm.example.com; exp=1712345678; scope=mcp:tools/call mcp:tools/get_weather:call
```
Check that `kid` matches what the MCP server fetched from JWKS, `iss`/`aud` match your server's expected values, and `exp` hasn't passed. Disable in production — the header leaks claim metadata.
---
## JWT claims reference
| Claim | Value |
|-------|-------|
| `iss` | `issuer` config value (or request base URL) |
| `aud` | `audience` config value (default: `"mcp"`) |
| `sub` | Resolved via `end_user_claim_sources` (default: `user_id` → api-key hash → `"litellm-proxy"`) |
| `act.sub` | `team_id` → `org_id` → `"litellm-proxy"` (RFC 8693 delegation) |
| `email` | `user_email` from LiteLLM auth context (when available) |
| `scope` | Auto-generated per tool call, or `allowed_scopes` when set |
| `iat`, `exp`, `nbf` | Standard timing claims (RFC 7519) |
---
## Limitations
- **OpenAPI-backed MCP servers** (`spec_path` set) do not support JWT injection. LiteLLM logs a warning and skips the header. Use SSE/HTTP transport servers to get full JWT injection.
- The keypair is **in-memory by default** and rotated on each restart unless `MCP_JWT_SIGNING_KEY` is set. FastMCP's `JWTVerifier` handles key rotation transparently via JWKS key ID matching.
---
## Related
- [MCP Guardrails](./mcp_guardrail) — PII masking and blocking for MCP calls
- [MCP OAuth](./mcp_oauth) — upstream OAuth2 for MCP server access
- [MCP AWS SigV4](./mcp_aws_sigv4) — AWS-signed requests to MCP servers

6
docs/my-website/docs/observability/langfuse_otel_integration.md
View file

@ -83,6 +83,9 @@ os.environ["LANGFUSE_OTEL_HOST"] = "https://cloud.langfuse.com" # EU region
# Or use self-hosted instance
# os.environ["LANGFUSE_OTEL_HOST"] = "https://my-langfuse.company.com"
# Optional: Ignore otel context propagation to prevent parent-child relationships with spans from other providers
# os.environ["OTEL_IGNORE_CONTEXT_PROPAGATION"] = "true"
litellm.callbacks = ["langfuse_otel"]
```
@ -124,6 +127,9 @@ export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_OTEL_HOST="https://us.cloud.langfuse.com" # Default US region
# export LANGFUSE_OTEL_HOST="https://otel.my-langfuse.company.com" # custom OTEL endpoint
# Optional: Ignore otel context propagation to prevent parent-child relationships with spans from other providers
# export OTEL_IGNORE_CONTEXT_PROPAGATION="true"
```
2. Setup config.yaml

148
docs/my-website/docs/observability/vantage.md Normal file
View file

@ -0,0 +1,148 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Vantage Integration
LiteLLM can export proxy spend data to [Vantage](https://vantage.sh) as [FOCUS 1.2](https://focus.finops.org/) formatted cost reports. This lets you visualize LLM spend alongside your cloud infrastructure costs in the Vantage dashboard.
## Overview
| Property | Details |
|----------|---------|
| Destination | Export LiteLLM usage data to Vantage Custom Provider |
| Data format | FOCUS CSV (automatically transformed from LiteLLM spend data) |
| Supported operations | Manual export, automatic scheduled export (hourly/daily/interval) |
| Authentication | Vantage API key + Custom Provider token |
## Prerequisites
You need two credentials from the [Vantage console](https://console.vantage.sh):
1. **API Key** — Go to **Settings → API Access Tokens** → Create a token with **Write** scope. The token looks like `vntg_tkn_...`.
2. **Custom Provider Token** — Go to **Settings → Integrations** → Create a **Custom Provider** integration → Copy the Provider ID (looks like `accss_crdntl_...`).
## Setup via API
The recommended setup uses the proxy admin endpoints. No config file changes needed.
### 1. Initialize credentials
```bash
curl -X POST http://localhost:4000/vantage/init \
-H "Authorization: Bearer $LITELLM_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"api_key": "vntg_tkn_YOUR_VANTAGE_API_KEY",
"integration_token": "accss_crdntl_YOUR_PROVIDER_TOKEN"
}'
```
Credentials are encrypted and stored in the proxy database.
### 2. Preview data (dry run)
```bash
curl -X POST http://localhost:4000/vantage/dry-run \
-H "Authorization: Bearer $LITELLM_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{"limit": 10}'
```
This returns FOCUS-transformed data without sending anything to Vantage. Use it to verify the pipeline works and inspect the data mapping.
### 3. Export to Vantage
```bash
curl -X POST http://localhost:4000/vantage/export \
-H "Authorization: Bearer $LITELLM_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{}'
```
Optional parameters:
- `limit` — Max number of records to export
- `start_time_utc` / `end_time_utc` — Filter by time range (must be provided together)
### 4. Verify in Vantage
Go to **Settings → Integrations → your Custom Provider → Import Costs** tab to see uploaded CSVs. Once the status changes from "Importing and Processing" to "Stable", costs appear in **Cost Reporting → All Resources**.
## Setup via Environment Variables
For automatic scheduled exports, configure via environment variables and proxy config:
### Environment variables
| Variable | Required | Description |
|----------|----------|-------------|
| `VANTAGE_API_KEY` | Yes | Vantage API access token |
| `VANTAGE_INTEGRATION_TOKEN` | Yes | Custom Provider token from Vantage dashboard |
| `VANTAGE_BASE_URL` | No | API URL override (default: `https://api.vantage.sh`) |
| `VANTAGE_EXPORT_FREQUENCY` | No | `hourly` (default), `daily`, or `interval` |
| `VANTAGE_EXPORT_INTERVAL_SECONDS` | No | Seconds between exports when frequency is `interval` |
### Proxy config
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-your-key
litellm_settings:
callbacks: ["vantage"]
```
```bash
export VANTAGE_API_KEY="vntg_tkn_..."
export VANTAGE_INTEGRATION_TOKEN="accss_crdntl_..."
litellm --config /path/to/config.yaml
```
The proxy registers a background job that exports data on the configured schedule.
## API Endpoints
All endpoints require admin authentication.
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/vantage/init` | Store Vantage credentials (encrypted) |
| `GET` | `/vantage/settings` | View current config (credentials masked) |
| `PUT` | `/vantage/settings` | Update credentials or base URL |
| `POST` | `/vantage/dry-run` | Preview FOCUS data without uploading |
| `POST` | `/vantage/export` | Upload cost data to Vantage |
| `DELETE` | `/vantage/delete` | Remove credentials and stop scheduled exports |
## FOCUS Field Mapping
LiteLLM spend data is transformed into the FOCUS 1.2 schema:
| LiteLLM Field | FOCUS Column | Description |
|---------------|-------------|-------------|
| `spend` | BilledCost, EffectiveCost | Cost of the usage |
| `model` | ChargeDescription, ResourceId | Model identifier |
| `model_group` | ServiceName | Model group / deployment |
| `custom_llm_provider` | ProviderName, PublisherName | Provider (openai, anthropic, etc.) |
| `api_key` | BillingAccountId | Hashed API key |
| `api_key_alias` | BillingAccountName | Human-readable key alias |
| `team_id` | SubAccountId | Team identifier |
| `team_alias` | SubAccountName | Team name |
Additional metadata (user_id, model_group, etc.) is included in the `Tags` column as JSON.
## Upload Limits
Vantage enforces per-upload limits. LiteLLM handles these automatically:
- **10,000 rows** per upload — large exports are split into batches
- **2 MB** per upload — oversized batches are further split by size
- **Unsupported columns** are stripped before upload
## Related Links
- [Vantage](https://vantage.sh)
- [Vantage Custom Providers](https://docs.vantage.sh/connecting_custom_providers)
- [FOCUS Specification](https://focus.finops.org/)
- [Focus Export (S3/Parquet)](./focus.md)

92
docs/my-website/docs/providers/anthropic.md
View file

@ -1965,6 +1965,98 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
</TabItem>
</Tabs>
## Files API
Upload files once and reference them by `file_id` in multiple requests—no need to re-upload content each time.
:::info
The `file_id` obtained from Anthropic only works with Anthropic Claude models. You cannot use it with other providers (OpenAI, Bedrock, etc.).
:::
- **Max file size:** 500 MB | **Total storage:** 100 GB per org
- **Pricing:** File API operations are free. File content used in Messages requests is priced as input tokens.
**Supported models by file type:**
- **Images:** All Claude 3+ models
- **PDFs:** All Claude 3.5+ models
- **Other file types** (for code execution): Claude 3.5 Haiku + all Claude 3.7+ models
### Quick Start
```python
import litellm
import os
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."
# 1. Upload a file once
file = litellm.create_file(
file=open("document.pdf", "rb"),
purpose="messages",
custom_llm_provider="anthropic",
)
# 2. Use file_id in messages (no re-upload needed)
response = litellm.completion(
model="anthropic/claude-sonnet-4-5-20250929",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Summarize this document"},
{"type": "file", "file": {"file_id": file.id, "format": "application/pdf"}}
]
}]
)
```
### File Operations
| Operation | Function |
|-----------|----------|
| Upload | `litellm.create_file(file, purpose="messages", custom_llm_provider="anthropic")` |
| List | `litellm.file_list(custom_llm_provider="anthropic")` |
| Retrieve | `litellm.file_retrieve(file_id, custom_llm_provider="anthropic")` |
| Delete | `litellm.file_delete(file_id, custom_llm_provider="anthropic")` |
| Download | `litellm.file_content(file_id, custom_llm_provider="anthropic")` |
:::note
Download only works for files created by the [code execution tool](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool), not uploaded files.
:::
### Supported Formats
| File Type | Format Value |
|-----------|-------------|
| PDF | `application/pdf` |
| Plain text | `text/plain` |
| JPEG | `image/jpeg` |
| PNG | `image/png` |
| GIF | `image/gif` |
| WebP | `image/webp` |
### Using Images
```python
# Upload image
image = litellm.create_file(
file=open("photo.jpg", "rb"),
purpose="messages",
custom_llm_provider="anthropic",
)
# Use in message
response = litellm.completion(
model="anthropic/claude-sonnet-4-5-20250929",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "file", "file": {"file_id": image.id, "format": "image/jpeg"}}
]
}]
)
```
## Usage - passing 'user_id' to Anthropic
LiteLLM translates the OpenAI `user` param to Anthropic's `metadata[user_id]` param.

95
docs/my-website/docs/providers/aws_sagemaker.md
View file

@ -526,3 +526,98 @@ print(f"response: {response}")
```
## Nova Models on SageMaker
LiteLLM supports Amazon Nova models (Nova Micro, Nova Lite, Nova 2 Lite) deployed on SageMaker Inference real-time endpoints. These custom/fine-tuned Nova models use an OpenAI-compatible API format.
**Reference:** [AWS Blog - Amazon SageMaker Inference for Custom Amazon Nova Models](https://aws.amazon.com/blogs/aws/announcing-amazon-sagemaker-inference-for-custom-amazon-nova-models/)
### Usage
Use the `sagemaker_nova/` prefix with your SageMaker endpoint name:
```python
import litellm
import os
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = "us-east-1"
# Basic chat completion
response = litellm.completion(
model="sagemaker_nova/my-nova-endpoint",
messages=[{"role": "user", "content": "Hello, how are you?"}],
temperature=0.7,
max_tokens=512,
)
print(response.choices[0].message.content)
```
### Streaming
```python
response = litellm.completion(
model="sagemaker_nova/my-nova-endpoint",
messages=[{"role": "user", "content": "Write a short poem"}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
### Multimodal (Images)
Nova models on SageMaker support image inputs using base64 data URIs:
```python
response = litellm.completion(
model="sagemaker_nova/my-nova-endpoint",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
]
}
],
)
```
### Proxy Config
```yaml
model_list:
- model_name: nova-micro
litellm_params:
model: sagemaker_nova/my-nova-micro-endpoint
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-east-1
```
### Supported Parameters
All standard OpenAI parameters are supported, plus these Nova-specific parameters:
| Parameter | Type | Description |
|-----------|------|-------------|
| `top_k` | integer | Limits token selection to top K most likely tokens |
| `reasoning_effort` | `"low"` \| `"high"` | Reasoning effort level (Nova 2 Lite custom models only) |
| `allowed_token_ids` | array[int] | Restrict output to specified token IDs |
| `truncate_prompt_tokens` | integer | Truncate prompt to N tokens if it exceeds limit |
```python
response = litellm.completion(
model="sagemaker_nova/my-nova-endpoint",
messages=[{"role": "user", "content": "Think step by step: what is 2+2?"}],
top_k=40,
reasoning_effort="low",
logprobs=True,
top_logprobs=2,
)
```

2
docs/my-website/docs/providers/azure/azure.md
View file

@ -1032,7 +1032,7 @@ print("list_batches_response=", list_batches_response)
</TabItem>
</Tabs>
### [Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only)
### [Health Check Azure Batch models](../../proxy/health.md#batch-models-azure-only)
### [BETA] Loadbalance Multiple Azure Deployments

5
docs/my-website/docs/providers/azure/azure_anthropic.md
View file

@ -372,7 +372,6 @@ response = completion(
## Related Documentation
- [Anthropic Provider Documentation](./anthropic.md) - For standard Anthropic API usage
- [Anthropic Provider Documentation](../anthropic.md) - For standard Anthropic API usage
- [Azure OpenAI Documentation](./azure.md) - For Azure OpenAI models
- [Azure Authentication Guide](../secret_managers/azure_key_vault.md) - For Azure AD token setup
- [Azure Authentication Guide](../../secret_managers/azure_key_vault.md) - For Azure AD token setup

6
docs/my-website/docs/providers/openai.md
View file

@ -638,7 +638,9 @@ This is useful when you want to use [Responses API](https://platform.openai.com/
:::tip gpt-5.4 + reasoning_effort + function tools
OpenAI does not support `reasoning_effort` with function tools for `gpt-5.4` in `/v1/chat/completions`. Use the responses bridge instead:
LiteLLM drops `reasoning_effort` from `gpt-5.4` requests to `litellm.completion()` that include tools, since that combination is supported in the Responses API.
If you need reasoning **and** tools together, use the responses bridge instead:
```python
response = litellm.completion(
@ -1151,4 +1153,4 @@ response = completion(
LiteLLM supports OpenAI's video generation models including Sora.
For detailed documentation on video generation, see [OpenAI Video Generation →](./openai/video_generation.md)
For detailed documentation on video generation, see [OpenAI Video Generation →](./openai/videos.md)

75
docs/my-website/docs/providers/openai/videos.md
View file

@ -135,6 +135,81 @@ curl --location --request POST 'http://localhost:4000/v1/videos/video_id/remix'
}'
```
### Character, Edit, and Extension Routes
OpenAI video routes supported by LiteLLM proxy:
- `POST /v1/videos/characters`
- `GET /v1/videos/characters/{character_id}`
- `POST /v1/videos/edits`
- `POST /v1/videos/extensions`
#### `target_model_names` support on character creation
`POST /v1/videos/characters` supports `target_model_names` for model-based routing (same behavior as video create).
```bash
curl --location 'http://localhost:4000/v1/videos/characters' \
--header 'Authorization: Bearer sk-1234' \
-F 'name=hero' \
-F 'target_model_names=gpt-4' \
-F 'video=@/path/to/character.mp4'
```
When `target_model_names` is used, LiteLLM returns an encoded character ID:
```json
{
"id": "character_...",
"object": "character",
"created_at": 1712697600,
"name": "hero"
}
```
Use that encoded ID directly on get:
```bash
curl --location 'http://localhost:4000/v1/videos/characters/character_...' \
--header 'Authorization: Bearer sk-1234'
```
#### Encoded and non-encoded video IDs for edit/extension
Both routes accept either plain or encoded `video.id`:
- `POST /v1/videos/edits`
- `POST /v1/videos/extensions`
```bash
curl --location 'http://localhost:4000/v1/videos/edits' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"prompt": "Make this brighter",
"video": { "id": "video_..." }
}'
```
```bash
curl --location 'http://localhost:4000/v1/videos/extensions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"prompt": "Continue this scene",
"seconds": "4",
"video": { "id": "video_..." }
}'
```
#### `custom_llm_provider` input sources
For these routes, `custom_llm_provider` may be supplied via:
- header: `custom-llm-provider`
- query: `?custom_llm_provider=...`
- body: `custom_llm_provider` (and `extra_body.custom_llm_provider` where supported)
Test OpenAI video generation request
```bash

74
docs/my-website/docs/providers/vertex_partner.md
View file

@ -11,6 +11,7 @@ import TabItem from '@theme/TabItem';
|----------|---------------|---------------|
| Anthropic (Claude) | `vertex_ai/claude-*` | [Vertex AI - Anthropic Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude) |
| DeepSeek | `vertex_ai/deepseek-ai/{MODEL}` | [Vertex AI - DeepSeek Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/deepseek) |
| ZAI (GLM) | `vertex_ai/zai-org/{MODEL}` | [Vertex AI - GLM Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/zaiorg/glm-47) |
| Meta/Llama | `vertex_ai/meta/{MODEL}` | [Vertex AI - Meta Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/llama) |
| Mistral | `vertex_ai/mistral-*` | [Vertex AI - Mistral Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/mistral) |
| AI21 (Jamba) | `vertex_ai/jamba-*` | [Vertex AI - AI21 Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/ai21) |
@ -226,6 +227,79 @@ ModelResponse(
|------------------|------------------------------|
| vertex_ai/deepseek-ai/deepseek-r1-0528-maas | `completion('vertex_ai/deepseek-ai/deepseek-r1-0528-maas', messages)` |
## VertexAI ZAI (GLM)
| Property | Details |
|----------|---------|
| Provider Route | `vertex_ai/zai-org/{MODEL}` |
| Vertex Documentation | [Vertex AI - GLM Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/zaiorg/glm-47) |
**LiteLLM Supports all Vertex AI GLM Models.** Ensure you use the `vertex_ai/zai-org/` prefix for all Vertex AI GLM models.
| Model Name | Usage |
|------------|-------|
| vertex_ai/zai-org/glm-4.7-maas | `completion('vertex_ai/zai-org/glm-4.7-maas', messages)` |
#### Usage
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""
response = completion(
model="vertex_ai/zai-org/glm-4.7-maas",
messages=[{"role": "user", "content": "hi"}],
vertex_project="your-vertex-project",
# vertex_location routes to "global"
)
print("\nModel Response", response)
```
</TabItem>
<TabItem value="proxy" label="Proxy">
**1. Add to config**
```yaml
model_list:
- model_name: glm-4.7
litellm_params:
model: vertex_ai/zai-org/glm-4.7-maas
vertex_project: "my-project"
# vertex_location routes to "global"
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000
```
**3. Test it!**
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"model": "glm-4.7",
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
}'
```
</TabItem>
</Tabs>
## VertexAI Meta/Llama API

2
docs/my-website/docs/proxy/call_hooks.md
View file

@ -7,7 +7,7 @@ import Image from '@theme/IdealImage';
- Enforce 'user' param for all openai endpoint calls
:::tip
**Understanding Callback Hooks?** Check out our [Callback Management Guide](../observability/callback_management.md) to understand the differences between proxy-specific hooks like `async_pre_call_hook` and general logging hooks like `async_log_success_event`.
**Understanding Callback Hooks?** Check out our [Callback Guide](../observability/callbacks.md) to understand the differences between proxy-specific hooks like `async_pre_call_hook` and general logging hooks like `async_log_success_event`.
:::
## Which Hook Should I Use?

17
docs/my-website/docs/proxy/config_settings.md
View file

@ -364,7 +364,7 @@ router_settings:
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `encrypted_content_affinity`, `deployment_affinity`, `session_affinity`, `forward_client_headers_by_model_group` |
| deployment_affinity_ttl_seconds | int | TTL (seconds) for user-key → deployment affinity mapping when `deployment_affinity` is enabled (configured at Router init / proxy startup). Defaults to `3600` (1 hour). |
| ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search/index.md) |
| guardrail_list | List[GuardrailTypedDict] | List of guardrail configurations for guardrail load balancing. Enables load balancing across multiple guardrail deployments with the same guardrail_name. [Further Docs](./guardrails/guardrail_load_balancing.md) |
@ -401,8 +401,10 @@ router_settings:
| AUTH_STRATEGY | Strategy used for authentication (e.g., OAuth, API key)
| AUTO_REDIRECT_UI_LOGIN_TO_SSO | Flag to enable automatic redirect of UI login page to SSO when SSO is configured. Default is **false**
| AUDIO_SPEECH_CHUNK_SIZE | Chunk size for audio speech processing. Default is 1024
| ANTHROPIC_API_KEY | API key for Anthropic service
| ANTHROPIC_API_KEY | API key for Anthropic service. Uses `x-api-key` header for authentication.
| ANTHROPIC_AUTH_TOKEN | Alternative auth token for Anthropic service. Uses `Authorization: Bearer` header instead of `x-api-key`. Used as fallback when `ANTHROPIC_API_KEY` is not set.
| ANTHROPIC_API_BASE | Base URL for Anthropic API. Default is https://api.anthropic.com
| ANTHROPIC_BASE_URL | Alternative to `ANTHROPIC_API_BASE` for setting the Anthropic API base URL. Used as fallback when `ANTHROPIC_API_BASE` is not set.
| ANTHROPIC_TOKEN_COUNTING_BETA_VERSION | Beta version header for Anthropic token counting API. Default is `token-counting-2024-11-01`
| AWS_ACCESS_KEY_ID | Access Key ID for AWS services
| AWS_BATCH_ROLE_ARN | ARN of the AWS IAM role for batch operations
@ -778,6 +780,7 @@ router_settings:
| LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRIES | Maximum retries for parallel requests in LiteLLM
| LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRY_TIMEOUT | Timeout for retries of parallel requests in LiteLLM
| LITELLM_DISABLE_LAZY_LOADING | When set to "1", "true", "yes", or "on", disables lazy loading of attributes (currently only affects encoding/tiktoken). This ensures encoding is initialized before VCR starts recording HTTP requests, fixing VCR cassette creation issues. See [issue #18659](https://github.com/BerriAI/litellm/issues/18659)
| LITELLM_DISABLE_REDACT_SECRETS | When set to "true", disables automatic redaction of secrets (API keys, tokens, credentials) from proxy log output. Secret redaction is enabled by default.
| LITELLM_MIGRATION_DIR | Custom migrations directory for prisma migrations, used for baselining db in read-only file systems.
| LITELLM_HOSTED_UI | URL of the hosted UI for LiteLLM
| LITELLM_UI_API_DOC_BASE_URL | Optional override for the API Reference base URL (used in sample code/docs) when the admin UI runs on a different host than the proxy. Defaults to `PROXY_BASE_URL` when unset.
@ -901,6 +904,7 @@ router_settings:
| OTEL_SERVICE_NAME | Service name identifier for OpenTelemetry
| OTEL_TRACER_NAME | Tracer name for OpenTelemetry tracing
| OTEL_LOGS_EXPORTER | Exporter type for OpenTelemetry logs (e.g., console)
| OTEL_IGNORE_CONTEXT_PROPAGATION | When true, ignore parent span context propagation in OpenTelemetry callbacks
| PAGERDUTY_API_KEY | API key for PagerDuty Alerting
| PANW_PRISMA_AIRS_API_KEY | API key for PANW Prisma AIRS service
| PANW_PRISMA_AIRS_API_BASE | Base URL for PANW Prisma AIRS service
@ -910,6 +914,7 @@ router_settings:
| PILLAR_API_BASE | Base URL for Pillar API Guardrails
| PILLAR_API_KEY | API key for Pillar API Guardrails
| PILLAR_ON_FLAGGED_ACTION | Action to take when content is flagged ('block' or 'monitor')
| PKCE_STRICT_CACHE_MISS | When set to `true`, the SSO callback will return a 401 error if the PKCE code_verifier is not found in the cache (e.g. due to a cache miss across pods). When `false` (default), it logs a warning and continues without the code_verifier.
| POD_NAME | Pod name for the server, this will be [emitted to `datadog` logs](https://docs.litellm.ai/docs/proxy/logging#datadog) as `POD_NAME`
| POSTHOG_API_KEY | API key for PostHog analytics integration
| POSTHOG_API_URL | Base URL for PostHog API (defaults to https://us.i.posthog.com)
@ -934,6 +939,9 @@ router_settings:
| PROXY_BASE_URL | Base URL for proxy service
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 10
| PROXY_BATCH_POLLING_INTERVAL | Time in seconds to wait before polling a batch, to check if it's completed. Default is 6000s (1 hour)
| PROXY_BATCH_POLLING_ENABLED | Set to `false` to disable the `CheckBatchCost` and `CheckResponsesCost` background polling jobs entirely. Useful for emergency mitigation on installs with large numbers of stale managed objects. Default is `true`
| MAX_OBJECTS_PER_POLL_CYCLE | Maximum number of managed objects (batches / responses) fetched per polling cycle. Prevents OOM on installs with many stale rows. Default is `50`
| MANAGED_OBJECT_STALENESS_CUTOFF_DAYS | Managed objects older than this many days in a non-terminal state are marked `stale_expired` at the start of each poll cycle and skipped. Default is `7`
| PROXY_BUDGET_RESCHEDULER_MAX_TIME | Maximum time in seconds to wait before checking database for budget resets. Default is 605
| PROXY_BUDGET_RESCHEDULER_MIN_TIME | Minimum time in seconds to wait before checking database for budget resets. Default is 597
| PYTHON_GC_THRESHOLD | GC thresholds ('gen0,gen1,gen2', e.g. '1000,50,50'); defaults to Python’s values.
@ -1017,6 +1025,11 @@ router_settings:
| UPSTREAM_LANGFUSE_SECRET_KEY | Secret key for upstream Langfuse authentication
| USE_AWS_KMS | Flag to enable AWS Key Management Service for encryption
| USE_PRISMA_MIGRATE | Flag to use prisma migrate instead of prisma db push. Recommended for production environments.
| VANTAGE_API_KEY | API key for Vantage cost-import integration
| VANTAGE_BASE_URL | Base URL for Vantage API. Default is `https://api.vantage.sh`
| VANTAGE_EXPORT_FREQUENCY | Export frequency for Vantage — `hourly` (default), `daily`, or `interval`
| VANTAGE_EXPORT_INTERVAL_SECONDS | Interval in seconds when VANTAGE_EXPORT_FREQUENCY is `interval`
| VANTAGE_INTEGRATION_TOKEN | Vantage integration token for the cost-import endpoint
| WANDB_API_KEY | API key for Weights & Biases (W&B) logging integration
| WANDB_HOST | Host URL for Weights & Biases (W&B) service
| WANDB_PROJECT_ID | Project ID for Weights & Biases (W&B) logging integration

16
docs/my-website/docs/proxy/configs.md
View file

@ -602,6 +602,22 @@ Since you shouldn't use 12.5, round down to **10** to leave a safety buffer. Thi
- Total maximum connections: 8 workers × 10 connections = 80 connections
- This stays safely under your database's 100 connection limit
## LiteLLM License Key (Enterprise)
To enable [LiteLLM Enterprise features](https://docs.litellm.ai/docs/proxy/enterprise), set your license key as an environment variable:
```bash
export LITELLM_LICENSE="eyJ..."
```
The license key is a JWT token provided when you purchase a LiteLLM Enterprise license. Once set, LiteLLM will automatically detect and activate enterprise features.
You can also add it to your `.env` file:
```env
LITELLM_LICENSE="eyJ..."
```
## Extras

307
docs/my-website/docs/proxy/docker_quick_start.md
View file

@ -1,25 +1,90 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# Getting Started Tutorial
End-to-End tutorial for LiteLLM Proxy to:
- Add an Azure OpenAI model
- Make a successful /chat/completion call
- Generate a virtual key
- Set RPM limit on virtual key
- Add an Azure OpenAI model
- Make a successful /chat/completion call
- Generate a virtual key
- Set RPM limit on virtual key
## Quick Install (Recommended for local / beginners)
New to LiteLLM? This is the easiest way to get started locally. One command installs LiteLLM and walks you through setup interactively — no config files to write by hand.
### 1. Install
```bash
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
```
This detects your OS, installs `litellm[proxy]`, and drops you straight into the setup wizard.
### 2. Follow the wizard
```
$ litellm --setup
Welcome to LiteLLM
Choose your LLM providers
○ 1. OpenAI GPT-4o, GPT-4o-mini, o1
○ 2. Anthropic Claude Opus, Sonnet, Haiku
○ 3. Azure OpenAI GPT-4o via Azure
○ 4. Google Gemini Gemini 2.0 Flash, 1.5 Pro
○ 5. AWS Bedrock Claude, Llama via AWS
○ 6. Ollama Local models
❯ Provider(s): 1,2
❯ OpenAI API key: sk-...
❯ Anthropic API key: sk-ant-...
❯ Port [4000]:
❯ Master key [auto-generate]:
✔ Config saved → ./litellm_config.yaml
❯ Start the proxy now? (Y/n):
```
The wizard walks you through:
1. Pick your LLM providers (OpenAI, Anthropic, Azure, Bedrock, Gemini, Ollama)
2. Enter API keys for each provider
3. Set a port and master key (or accept the defaults)
4. Config is saved to `./litellm_config.yaml` and the proxy starts immediately
### 3. Make a call
Your proxy is running on `http://0.0.0.0:4000`. Test it:
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <your-master-key>' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
:::tip Already have pip installed?
You can skip the curl install and run `litellm --setup` directly after `pip install 'litellm[proxy]'`.
:::
---
## Pre-Requisites
- Install LiteLLM Docker Image **OR** LiteLLM CLI (pip package)
Choose your install method. **Docker Compose** users complete their full setup inside the tab and are done. **Docker** and **pip** users continue with the steps below the tabs.
<Tabs>
<TabItem value="docker" label="Docker">
```
```bash
docker pull docker.litellm.ai/berriai/litellm:main-latest
```
@ -37,7 +102,25 @@ $ pip install 'litellm[proxy]'
<TabItem value="docker-compose" label="Docker Compose (Proxy + DB)">
Use this docker compose to spin up the proxy with a postgres database running locally.
Docker Compose bundles LiteLLM with a Postgres database. Follow the steps below — the proxy will be fully running by the end.
### Step 1 — Pull the LiteLLM database image
LiteLLM provides a dedicated `litellm-database` image for proxy deployments that connect to Postgres.
```bash
docker pull ghcr.io/berriai/litellm-database:main-latest
```
See all available tags on the [GitHub Container Registry](https://github.com/BerriAI/litellm/pkgs/container/litellm-database).
---
### Step 2 — Set up a database
Complete all three config files **before** running `docker compose up`. The proxy server will not start correctly if any of these are missing.
#### 2.1 — Get `docker-compose.yml` and create `.env`
```bash
# Get the docker compose file
@ -46,26 +129,154 @@ curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/docker-compose.ym
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
# Add the litellm salt key — cannot be changed after adding a model
# Used to encrypt/decrypt your LLM API key credentials
# Generate a strong random value: https://1password.com/password-generator/
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env
# Start
# Add your model credentials
echo 'AZURE_API_BASE="https://openai-***********/"' >> .env
echo 'AZURE_API_KEY="your-azure-api-key"' >> .env
```
#### 2.2 — Create `config.yaml`
The default `docker-compose.yml` starts a Postgres container at `db:5432`. Your `config.yaml` must include `database_url` pointing to it:
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: azure/my_azure_deployment
api_base: os.environ/AZURE_API_BASE
api_key: os.environ/AZURE_API_KEY
api_version: "2025-01-01-preview"
general_settings:
master_key: sk-1234 # 🔑 your proxy admin key (must start with sk-)
database_url: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
```
:::tip
`database_url` enables virtual keys, spend tracking, and the UI. Replace it with your [Supabase](https://supabase.com/) or [Neon](https://neon.tech/) connection string if you prefer a managed database.
:::
#### 2.3 — Create `prometheus.yml`
This file **must exist as a file** before `docker compose up`. If it is missing, Docker auto-creates it as an empty directory and the Prometheus container fails to start.
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "litellm"
static_configs:
- targets: ["litellm:4000"]
```
Also verify that the `config.yaml` volume mount and `--config` flag are **not commented out** in `docker-compose.yml`:
```yaml
services:
litellm:
volumes:
- ./config.yaml:/app/config.yaml # ✅ must be uncommented
command:
- "--config=/app/config.yaml" # ✅ must be uncommented
```
:::warning
All three files (`.env`, `config.yaml`, `prometheus.yml`) must be present before running `docker compose up`. See [Troubleshooting](#troubleshooting) if you run into issues.
:::
---
### Step 3 — Start the proxy server and test it
After `config.yaml`, `prometheus.yml`, and `.env` are complete, start the proxy:
```bash
docker compose up
```
Once running, test it with a curl request:
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
**Expected response:**
```json
{
"id": "chatcmpl-abcd",
"created": 1773817678,
"model": "gpt-4o",
"object": "chat.completion",
"system_fingerprint": "fp_6b1ef07cda",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Hello! How can I assist you today?",
"role": "assistant",
"annotations": []
}
}
],
"usage": {
"completion_tokens": 9,
"prompt_tokens": 9,
"total_tokens": 18,
"completion_tokens_details": {
"accepted_prediction_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0,
"rejected_prediction_tokens": 0
},
"prompt_tokens_details": {
"audio_tokens": 0,
"cached_tokens": 0
}
},
"service_tier": "default"
}
```
---
### Optional — Navigate to the LiteLLM UI and generate a virtual key
Open [http://localhost:4000/ui](http://localhost:4000/ui) in your browser and log in with your master key (`sk-1234`).
Navigate to **Virtual Keys** and click **+ Create New Key**:
<Image img={require('../../img/litellm_ui_create_key.png')} alt="LiteLLM UI — Create Virtual Key" />
Virtual keys let you track spend, set rate limits, and control model access per user or team.
</TabItem>
</Tabs>
## 1. Add a model
:::note Docker Compose users
Your setup is complete — the steps below are for **Docker** and **pip** users only.
:::
Control LiteLLM Proxy with a config.yaml file.
---
Setup your config.yaml with your azure model.
## Step 1 — Add a model
Note: When using the proxy with a database, you can also **just add models via UI** (UI is available on `/ui` route).
Control LiteLLM Proxy with a `config.yaml` file. Create one with your Azure model:
```yaml
model_list:
@ -89,8 +300,6 @@ You can read more about how model resolution works in the [Model Configuration](
- **`api_base`** (`str`) - The API base for your azure deployment.
- **`api_version`** (`str`) - The API Version to use when calling Azure's OpenAI API. Get the latest Inference API version [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation?source=recommendations#latest-preview-api-releases).
---
---
@ -138,19 +347,19 @@ $ litellm --config /app/config.yaml --detailed_debug
</Tabs>
Confirm your config was loaded correctly — you should see this in the logs:
Confirm your config.yaml got mounted correctly
```bash
```
Loaded config YAML (api_key and environment_variables are not shown):
{
"model_list": [
{
"model_name ...
"model_list": [
{
"model_name": ...
```
### 2.2 Make Call
LiteLLM Proxy is 100% OpenAI-compatible. Test your model via `/chat/completions`:
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
@ -244,15 +453,17 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
- [Other/Non-Chat Completion Endpoints](../embedding/supported_embedding.md)
- [Pass-through for VertexAI, Bedrock, etc.](../pass_through/vertex_ai.md)
## 3. Generate a virtual key
## Optional: Generate a virtual key
Track Spend, and control model access via virtual keys for the proxy
Track spend and control model access via virtual keys for the proxy.
### 3.1 Set up a Database
### Prerequisite — Set up a database
**Requirements**
- Need a postgres database (e.g. [Supabase](https://supabase.com/), [Neon](https://neon.tech/), etc)
:::note Docker Compose users
Your Postgres container is already running — skip ahead to [Create Key w/ RPM Limit](#create-key-w-rpm-limit) below.
:::
**Docker / pip users** — you need a Postgres database (e.g. [Supabase](https://supabase.com/), [Neon](https://neon.tech/), or self-hosted). Add `general_settings` to your `config.yaml`:
```yaml
model_list:
@ -268,7 +479,9 @@ general_settings:
database_url: "postgresql://<user>:<password>@<host>:<port>/<dbname>" # 👈 KEY CHANGE
```
Save config.yaml as `litellm_config.yaml` (used in 3.2).
Save config.yaml as `litellm_config.yaml` before continuing.
You must finish this setup before starting the proxy server.
---
@ -294,7 +507,7 @@ See All General Settings [here](http://localhost:3000/docs/proxy/configs#all-set
`database_url: "postgresql://..."`
- Set `DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname>` in your env
### 3.2 Start Proxy
### Start Proxy
```bash
docker run \
@ -302,12 +515,11 @@ docker run \
-e AZURE_API_KEY=d6*********** \
-e AZURE_API_BASE=https://openai-***********/ \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:main-latest \
ghcr.io/berriai/litellm-database:main-latest \
--config /app/config.yaml --detailed_debug
```
### 3.3 Create Key w/ RPM Limit
### Create Key w/ RPM Limit
Create a key with `rpm_limit: 1`. This will only allow 1 request per minute for calls to proxy with this key.
@ -330,9 +542,9 @@ curl -L -X POST 'http://0.0.0.0:4000/key/generate' \
}
```
### 3.4 Test it!
### Test it!
**Use your virtual key from step 3.3**
**Use the virtual key you just created.**
1st call - Expect to work!
@ -546,6 +758,24 @@ model_list:
## Troubleshooting
### `prometheus.yml` mount error — "not a directory"
If you see:
```bash
Error: cannot create subdirectories in ".../prometheus.yml": not a directory
```
Docker created `prometheus.yml` as an **empty directory** instead of a file. This happens when the file is missing at `docker compose up` time.
Fix it:
Then create the file (see [Step 2.3 — Create `prometheus.yml`](#23--create-prometheusyml)) and run `docker compose up` again.
```bash
rm -rf prometheus.yml
```
Then create the file (see [Step 2.4](#step-24--create-prometheusyml)) and run `docker compose up` again.
### Non-root docker image?
If you need to run the docker image as a non-root user, use [this](https://github.com/BerriAI/litellm/pkgs/container/litellm-non_root).
@ -645,6 +875,3 @@ LiteLLM Proxy uses the [LiteLLM Python SDK](https://docs.litellm.ai/docs/routing
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
[![Chat on WhatsApp](https://img.shields.io/static/v1?label=Chat%20on&message=WhatsApp&color=success&logo=WhatsApp&style=flat-square)](https://wa.link/huol9n) [![Chat on Discord](https://img.shields.io/static/v1?label=Chat%20on&message=Discord&color=blue&logo=Discord&style=flat-square)](https://discord.gg/wuPM9dRgDw)

2
docs/my-website/docs/proxy/endpoint_activity.md
View file

@ -114,4 +114,4 @@ Understand spend distribution across endpoints:
- [Customer Usage](./customer_usage.md) - Track spend and usage for individual customers
- [Cost Tracking](./cost_tracking.md) - Comprehensive cost tracking and analytics
- [Spend Logs](./spend_logs.md) - Detailed request-level spend logs
- [Spend Logs](./cost_tracking.md#-spend-logs-api---individual-transaction-logs) - Detailed request-level spend logs

4
docs/my-website/docs/proxy/keys_teams_router_settings.md
View file

@ -146,5 +146,5 @@ Test new router settings on specific keys or teams before applying globally:
- [Router Settings Reference](./config_settings.md#router_settings---reference) - Complete reference of all router settings
- [Load Balancing](./load_balancing.md) - Learn about routing strategies and load balancing
- [Reliability](./reliability.md) - Configure fallbacks, retries, and error handling
- [Keys](./keys.md) - Manage API keys and their settings
- [Teams](./teams.md) - Organize keys into teams
- [Keys](./virtual_keys.md) - Manage API keys and their settings
- [Teams](./multi_tenant_architecture.md) - Organize keys into teams

4
docs/my-website/docs/proxy/model_compare_ui.md
View file

@ -187,7 +187,7 @@ Use tags and multiple comparisons to run structured A/B tests:
## Related Features
- [Playground Chat UI](./playground.md) - Single model testing interface
- [Playground Chat UI](./ui.md) - Single model testing interface
- [Model Management](./model_management.md) - Configure and manage models
- [Guardrails](./guardrails.md) - Set up safety filters
- [Guardrails](./guardrails/quick_start.md) - Set up safety filters
- [AI Hub](./ai_hub.md) - Share models and agents with your organization

84
docs/my-website/docs/proxy/realtime_webrtc.md Normal file
View file

@ -0,0 +1,84 @@
# /realtime - WebRTC Support
Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth; audio streams directly to OpenAI/Azure.
**Providers:** OpenAI · Azure
:::info **WebRTC vs WebSocket**
- **WebSocket** (`/v1/realtime`) — server-to-server
- **WebRTC** (`/v1/realtime/client_secrets` + `/v1/realtime/calls`) — browser/mobile, lower latency
:::
## How it works
LiteLLM issues tokens and relays SDP; audio never passes through the proxy.
```
Browser LiteLLM Proxy OpenAI/Azure
| | |
|-- POST client_secrets --->|-- POST sessions -------->|
|<-- encrypted_token -------|<-- ek_... ---------------|
|-- POST calls [SDP+token] ->|-- POST calls ----------->|
|<-- SDP answer ------------|<-- SDP answer -----------|
|===== audio P2P direct ===============================>|
```
## Proxy Setup
```yaml
model_list:
- model_name: gpt-4o-realtime
litellm_params:
model: openai/gpt-4o-realtime-preview-2024-12-17
api_key: os.environ/OPENAI_API_KEY
model_info:
mode: realtime
```
**Azure:** `model: azure/gpt-4o-realtime-preview`, `api_key`, `api_base`.
```bash
litellm --config /path/to/config.yaml
```
## Client Usage
1. **Token** — `POST /v1/realtime/client_secrets` with LiteLLM key and `{ model }`.
2. **WebRTC** — Create `RTCPeerConnection`, add mic, data channel `oai-events`, send SDP offer to `POST /v1/realtime/calls` with `Authorization: Bearer <token>`, `Content-Type: application/sdp`.
3. **Events** — Use data channel for `session.update` and other events.
```javascript
const r = await fetch("http://proxy:4000/v1/realtime/client_secrets", {
method: "POST",
headers: { "Authorization": "Bearer sk-litellm-key", "Content-Type": "application/json" },
body: JSON.stringify({ model: "gpt-4o-realtime" }),
});
const token = (await r.json()).client_secret.value;
const pc = new RTCPeerConnection();
const audio = document.createElement("audio");
audio.autoplay = true;
pc.ontrack = (e) => (audio.srcObject = e.streams[0]);
const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(ms.getTracks()[0]);
const dc = pc.createDataChannel("oai-events");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
const sdpRes = await fetch("http://proxy:4000/v1/realtime/calls", {
method: "POST",
headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/sdp" },
body: offer.sdp,
});
await pc.setRemoteDescription({ type: "answer", sdp: await sdpRes.text() });
dc.send(JSON.stringify({ type: "session.update", session: { instructions: "..." } }));
```
## FAQ
- **401 Token expired** — Get a fresh token right before creating the WebRTC offer.
- **Which key for `/calls`?** — Encrypted token from `client_secrets`, not raw key.
- **Pass `model`?** — No. Token encodes routing.
- **Azure `api-version`** — Set `api_version` in `litellm_params` and correct `api_base`.
- **No audio** — Grant mic; ensure `pc.ontrack` sets autoplay audio; check firewall/WebRTC; inspect console.

2
docs/my-website/docs/proxy/sync_anthropic_beta_headers.md
View file

@ -125,4 +125,4 @@ curl -X DELETE "https://your-proxy-url/schedule/anthropic_beta_headers_reload" \
## Related
- [Model Cost Map Sync](./sync_models_github.md) - Auto-sync model pricing data
- [Anthropic Beta Headers](../completion/anthropic.md#beta-features) - Using Anthropic beta features
- [Anthropic Beta Headers](../providers/anthropic.md) - Using Anthropic beta features

100
docs/my-website/docs/proxy/tag_routing.md
View file

@ -209,6 +209,106 @@ Expect to see the following response header when this works
x-litellm-model-id: default-model
```
## Regex-based tag routing (`tag_regex`)
Use `tag_regex` to route requests based on regex patterns matched against request headers, without requiring clients to pass a tag explicitly. This is useful when clients already send a recognisable header, such as `User-Agent`.
**Use case: route all Claude Code traffic to dedicated AWS accounts**
Claude Code always sends `User-Agent: claude-code/<version>`. With `tag_regex` you can route that traffic to a dedicated deployment automatically — no per-developer configuration needed.
### 1. Config
```yaml
model_list:
# Claude Code traffic → dedicated deployment, matched by User-Agent
- model_name: claude-sonnet
litellm_params:
model: bedrock/converse/anthropic-claude-sonnet-4-6
aws_region_name: us-east-1
aws_role_name: arn:aws:iam::111122223333:role/LiteLLMClaudeCode
tag_regex:
- "^User-Agent: claude-code\\/" # matches claude-code/1.x, 2.x, etc.
model_info:
id: claude-code-deployment
# All other traffic falls back to the default deployment
- model_name: claude-sonnet
litellm_params:
model: bedrock/converse/anthropic-claude-sonnet-4-6
aws_region_name: us-east-1
aws_role_name: arn:aws:iam::444455556666:role/LiteLLMDefault
tags:
- default
model_info:
id: regular-deployment
router_settings:
enable_tag_filtering: true
tag_filtering_match_any: true
general_settings:
master_key: sk-1234
```
### 2. Verify routing
Claude Code sets `User-Agent: claude-code/<version>` automatically — no client config needed:
```shell
# Claude Code request (User-Agent set automatically by Claude Code)
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-1234" \
-H "User-Agent: claude-code/1.2.3" \
-d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}'
# → x-litellm-model-id: claude-code-deployment
# Any other client (no matching User-Agent) → default deployment
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-1234" \
-d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}'
# → x-litellm-model-id: regular-deployment
```
### How matching works
| Priority | Condition | Result |
|----------|-----------|--------|
| 1 | Request has `tags` AND deployment has `tags` | Exact tag match (respects `match_any` setting) |
| 2 | Deployment has `tag_regex` AND request has a `User-Agent` | Regex match (always OR logic — any pattern match suffices) |
| 3 | Deployment has `tags: [default]` | Default fallback |
| 4 | No default set | All healthy deployments returned |
`tag_regex` always uses OR semantics — `tag_filtering_match_any=False` applies only to exact tag matching, not to regex patterns.
### Observability
When a regex matches, `tag_routing` is written into request metadata and flows to SpendLogs:
```json
{
"tag_routing": {
"matched_via": "tag_regex",
"matched_value": "^User-Agent: claude-code\\/",
"user_agent": "claude-code/1.2.3",
"request_tags": []
}
}
```
### Security note
:::caution
**`User-Agent` is a client-supplied header and can be set to any value.** Any API consumer can send `User-Agent: claude-code/1.0` regardless of whether they are actually using Claude Code.
Do not rely on `tag_regex` routing to enforce access controls or spend limits — use [team/key-based routing](./users) for that. `tag_regex` is a **traffic classification hint** (useful for billing visibility, capacity planning, and routing convenience), not a security boundary.
:::
---
## ✨ Team based tag routing (Enterprise)
LiteLLM Proxy supports team-based tag routing, allowing you to associate specific tags with teams and route requests accordingly. Example **Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B** (LLM Access Control For Teams)

4
docs/my-website/docs/proxy/team_budgets.md
View file

@ -177,3 +177,7 @@ Expect to see this metric on prometheus to track the Remaining Budget for the te
```shell
litellm_remaining_team_budget_metric{team_alias="QA Prod Bot",team_id="de35b29e-6ca8-4f47-b804-2b79d07aa99a"} 9.699999999999992e-06
```
## See Also
- [Per-model TPM/RPM for teams](./users.md#per-team-model) - Set rate limits per model for all keys in a team

138
docs/my-website/docs/proxy/ui/ui_edit_logo.md Normal file
View file

@ -0,0 +1,138 @@
import Image from '@theme/IdealImage';
# Customize UI Logo
Personalize your LiteLLM dashboard by replacing the default logo with your own company branding. You can set a custom logo via the UI or the API.
## Via the UI
### 1. Navigate to Settings
Click the **Settings** icon in the sidebar.
![Navigate to Settings](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/57a15404-51f7-481e-9db2-cea94566d3ce/ascreenshot_7a348567c839448bb806fd71cf4abca0_text_export.jpeg)
### 2. Open UI Theme Settings
Click **UI Theme** from the settings menu.
![Open UI Theme](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/30663fe1-9f78-4496-96d4-c53513cbaf82/ascreenshot_ac1eb59eda0e423fbd0e7d3a6cabd4c7_text_export.jpeg)
### 3. Click the Logo URL Field
Click the **Logo URL** text field to start editing.
![Click Logo URL Field](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/069e8412-8ec1-4d36-ba38-6b2e2858a45a/ascreenshot_8fc7fb4a3af74815bc1b69a8554bc110_text_export.jpeg)
### 4. Find Your Logo Image
Open a new browser tab and find the logo image you want to use (e.g., search Google Images for your company logo).
![Find Logo Image](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/d9b55dac-bc4e-4728-b422-4afbc21f9034/ascreenshot_2a805f39c83d4b5e95f43495a6ea4e79_text_export.jpeg)
### 5. Right-Click on the Logo Image
Right-click the image you want to use as your logo.
![Right-Click Image](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/9d42d13e-6028-4710-acb2-c6af04a855c7/ascreenshot_0f21f29ba0e44132afe483a4b88e8b70_text_export.jpeg)
### 6. Copy the Image Address
Select **Copy Image Address** from the context menu to copy the URL.
![Copy Image Address](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/c25637be-383a-498b-ad11-eb1761d52757/ascreenshot_b237ee800979462189a02c1e1942ebf1_text_export.jpeg)
### 7. Switch Back to LiteLLM
Navigate back to the LiteLLM UI tab (e.g., press **Cmd + Left** or click the tab).
![Switch Back](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/f0647856-679c-4591-9ff7-7fd3cfbc70b4/ascreenshot_3ce46dae64c94891ac0983f5ed8f085a_text_export.jpeg)
### 8. Paste the Logo URL
Paste the copied image URL into the **Logo URL** field with **Cmd + V**.
![Paste URL](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/54dd30d9-7a88-41e8-a580-a6acf707c7fa/ascreenshot_8a772218ac0743d9ae8ffd3311eccd5a_text_export.jpeg)
### 9. Save Changes
Click **Save Changes** to apply your new logo.
![Save Changes](https://colony-recorder.s3.amazonaws.com/files/2026-03-13/4baf6494-d146-4600-b6f2-ef667338d580/ascreenshot_722cbcd568ec4267af5122b3958bb248_text_export.jpeg)
Your custom logo will now appear in the LiteLLM dashboard sidebar and login page.
## Via the API
### Set a Custom Logo
```bash
curl -X PATCH 'http://localhost:4000/settings/update/ui_theme_settings' \
-H 'Authorization: Bearer <your-admin-key>' \
-H 'Content-Type: application/json' \
-d '{
"logo_url": "https://example.com/your-company-logo.png"
}'
```
### Set a Custom Favicon
You can also customize the browser tab favicon:
```bash
curl -X PATCH 'http://localhost:4000/settings/update/ui_theme_settings' \
-H 'Authorization: Bearer <your-admin-key>' \
-H 'Content-Type: application/json' \
-d '{
"logo_url": "https://example.com/your-company-logo.png",
"favicon_url": "https://example.com/your-favicon.ico"
}'
```
### Get Current Theme Settings
```bash
curl -X GET 'http://localhost:4000/settings/get/ui_theme_settings'
```
### Reset to Default Logo
Send an empty `logo_url` to restore the default LiteLLM logo:
```bash
curl -X PATCH 'http://localhost:4000/settings/update/ui_theme_settings' \
-H 'Authorization: Bearer <your-admin-key>' \
-H 'Content-Type: application/json' \
-d '{
"logo_url": ""
}'
```
## Via `proxy_config.yaml`
You can also set the logo URL in your proxy configuration file:
```yaml
litellm_settings:
ui_theme_config:
logo_url: "https://example.com/your-company-logo.png"
favicon_url: "https://example.com/your-favicon.ico" # optional
```
Or set it as an environment variable:
```yaml
environment_variables:
UI_LOGO_PATH: "https://example.com/your-company-logo.png"
```
## Supported Logo Formats
| Format | Supported |
|--------|-----------|
| JPEG / JPG | Yes |
| PNG | Yes |
| SVG | Yes |
| ICO (favicon only) | Yes |
| HTTP/HTTPS URL | Yes |
| Local file path | Yes |

4
docs/my-website/docs/proxy/ui_store_model_db_setting.md
View file

@ -87,6 +87,6 @@ Change the setting from the UI and have it take effect immediately—perfect for
## Related Documentation
- [Admin UI Overview](./ui_overview.md) – General guide to the LiteLLM Admin UI
- [Models and Endpoints](./models_and_endpoints.md) – Managing models and API endpoints
- [Admin UI Overview](./ui.md) – General guide to the LiteLLM Admin UI
- [Models and Endpoints](./model_management.md) – Managing models and API endpoints
- [Config Settings](./config_settings.md) – `store_model_in_db` in `general_settings`

2
docs/my-website/docs/proxy/user_onboarding.md
View file

@ -79,4 +79,4 @@ curl -X POST http://localhost:4000/v1/chat/completions \
## See Also
- [Proxy Quick Start](./quick_start.md)
- [User Management](./users.md)
- [Key Management](./key_management.md)
- [Key Management](./virtual_keys.md)

58
docs/my-website/docs/proxy/users.md
View file

@ -641,7 +641,7 @@ You can set:
- tpm limits (tokens per minute)
- rpm limits (requests per minute)
- max parallel requests
- rpm / tpm limits per model for a given key
- rpm / tpm limits per model for a given key or team
### TPM Rate Limit Type (Input/Output/Total)
@ -689,6 +689,62 @@ curl --location 'http://0.0.0.0:4000/team/new' \
}
```
</TabItem>
<TabItem value="per-team-model" label="Per Team Per Model">
**Set rate limits per model for a team**
Use `model_rpm_limit` and `model_tpm_limit` to set rate limits per model for all keys belonging to a team. These limits apply across all keys in the team and are inherited by keys unless overridden at the key level.
Use `/team/new` or `/team/update` with `model_rpm_limit` and `model_tpm_limit` as dictionaries mapping model names to their limits:
```shell
curl --location 'http://0.0.0.0:4000/team/new' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"team_id": "my-prod-team",
"model_rpm_limit": {"gpt-4": 100, "gpt-3.5-turbo": 200},
"model_tpm_limit": {"gpt-4": 10000, "gpt-3.5-turbo": 20000}
}'
```
**Update existing team with per-model limits:**
```shell
curl --location 'http://0.0.0.0:4000/team/update' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"team_id": "my-prod-team",
"model_rpm_limit": {"gpt-4": 100, "gpt-3.5-turbo": 200},
"model_tpm_limit": {"gpt-4": 10000, "gpt-3.5-turbo": 20000}
}'
```
**Alternative: Use metadata**
You can also pass per-model limits via the `metadata` field:
```shell
curl --location 'http://0.0.0.0:4000/team/update' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"team_id": "my-prod-team",
"metadata": {
"model_rpm_limit": {"gpt-4": 100, "gpt-3.5-turbo": 200},
"model_tpm_limit": {"gpt-4": 10000, "gpt-3.5-turbo": 20000}
}
}'
```
**Resolution order:** When a key belongs to a team, rate limits are resolved as: **Key metadata > Key model_max_budget > Team metadata**. Keys can override team-level per-model limits with their own `model_rpm_limit` or `model_tpm_limit`.
**Verify:** Make a `/chat/completions` request and check response headers `x-litellm-key-remaining-requests-{model}` and `x-litellm-key-remaining-tokens-{model}` for the model-specific limits.
[**See Swagger**](https://litellm-api.up.railway.app/#/team%20management/new_team_team_new_post)
</TabItem>
<TabItem value="per-user" label="Per Internal User">

4
docs/my-website/docs/reasoning_content.md
View file

@ -594,7 +594,9 @@ Expected Response
:::tip gpt-5.4: reasoning_effort + function tools
OpenAI does not support `reasoning_effort` with function tools for `gpt-5.4` in `/v1/chat/completions`. Use `openai/responses/gpt-5.4` to route through the Responses API instead. See [Responses API Bridge](/docs/providers/openai#openai-chat-completion-to-responses-api-bridge) for details.
LiteLLM drops `reasoning_effort` from `gpt-5.4` requests to `litellm.completion()` that include tools, since that combination is supported in the Responses API.
If you need reasoning **and** tools together, use `openai/responses/gpt-5.4` to route through the Responses API instead. See [Responses API Bridge](/docs/providers/openai#openai-chat-completion-to-responses-api-bridge) for details.
:::

99
docs/my-website/docs/tutorials/claude_code_skills.md Normal file
View file

@ -0,0 +1,99 @@
# LiteLLM Skills
[litellm-skills](https://github.com/BerriAI/litellm-skills) is a collection of [Agent Skills](https://agentskills.io) for managing a live LiteLLM proxy. Install them once and any agent that supports the Agent Skills standard (Claude Code, OpenCode, OpenClaw, etc.) can create users, teams, keys, models, MCP servers, agents, and query usage — all by running `curl` commands against your proxy.
## Install
```bash
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm-skills/main/install.sh | sh
```
## Requirements
- `curl` installed
- A running LiteLLM proxy (local or remote)
- A proxy admin key — not a virtual key scoped to `llm_api_routes`
## Available Skills
### Users
| Skill | What it does |
|-------|-------------|
| `/add-user` | Create a user — email, role, budget, model access |
| `/update-user` | Update budget, role, or models for an existing user |
| `/delete-user` | Delete one or more users |
### Teams
| Skill | What it does |
|-------|-------------|
| `/add-team` | Create a team with budget and model limits |
| `/update-team` | Update budget, models, or rate limits |
| `/delete-team` | Delete one or more teams |
### API Keys
| Skill | What it does |
|-------|-------------|
| `/add-key` | Generate a key scoped to a user, team, budget, and expiry |
| `/update-key` | Update budget, models, or expiry |
| `/delete-key` | Delete by key value or alias |
### Organizations
| Skill | What it does |
|-------|-------------|
| `/add-org` | Create an org with budget and model access |
| `/delete-org` | Delete one or more orgs |
### Models
| Skill | What it does |
|-------|-------------|
| `/add-model` | Add any provider (OpenAI, Azure, Anthropic, Bedrock, Ollama…) and test it |
| `/update-model` | Rotate credentials or swap the underlying deployment |
| `/delete-model` | Remove a model |
### MCP Servers
| Skill | What it does |
|-------|-------------|
| `/add-mcp` | Register an MCP server (SSE, HTTP, or stdio) |
| `/update-mcp` | Update URL, credentials, or allowed tools |
| `/delete-mcp` | Remove an MCP server |
### Agents
| Skill | What it does |
|-------|-------------|
| `/add-agent` | Create an agent backed by a model and optional MCP servers |
| `/update-agent` | Swap the model or update description and limits |
| `/delete-agent` | Remove an agent |
### Usage
| Skill | What it does |
|-------|-------------|
| `/view-usage` | Daily spend and token activity — by user, team, org, or model |
## How it works
When you invoke a skill, the agent asks for your `LITELLM_BASE_URL` and admin key, collects the fields needed for that operation, runs the `curl`, and shows the result. For example:
```
/add-model
```
→ Agent asks: provider, public name, credentials. Adds the model, runs a test completion, reports pass/fail.
```
/view-usage
```
→ Agent asks: date range (defaults to current month), optional team/model filter. Prints a table of daily requests, tokens, and spend.
## Related
- [litellm-skills on GitHub](https://github.com/BerriAI/litellm-skills)
- [Virtual Keys](../proxy/virtual_keys.md) — managing API keys on the proxy
- [Team-based routing](../proxy/team_based_routing.md) — setting up teams
- [Model Management](../proxy/model_management.md) — adding models via config or API

98
docs/my-website/docs/tutorials/index.md Normal file
View file

@ -0,0 +1,98 @@
---
title: Tutorials
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
**Tutorials** are step-by-step walkthroughs for integrating LiteLLM with external tools, frameworks, and services — or building complete end-to-end workflows.
> Need help choosing the right path before you start? See [Learn →](/docs/learn)
---
## Getting Started
<NavigationCards
columns={2}
items={[
{
icon: "⚡",
title: "Getting Started",
description: "Installation, playground, text completion, and mock completions.",
to: "/docs/tutorials/getting_started",
},
]}
/>
---
## Integrations
<NavigationCards
columns={2}
items={[
{
icon: "🤖",
title: "Agent SDKs & Frameworks",
description: "OpenAI Agents SDK, Claude Agent SDK, Google ADK, CopilotKit, Letta, LiveKit, Instructor.",
to: "/docs/agent_sdks",
},
{
icon: "🛠️",
title: "AI Coding Tools",
description: "Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Qwen Code, OpenAI Codex.",
to: "/docs/ai_tools",
},
{
icon: "🐍",
title: "Python SDK",
description: "Gradio, fallbacks, provider-specific params — no proxy required.",
to: "/docs/tutorials/python_sdk",
},
{
icon: "🔌",
title: "Provider Setup",
description: "Azure OpenAI, HuggingFace, TogetherAI, local models, and more.",
to: "/docs/tutorials/provider_tutorials",
},
]}
/>
---
## Proxy
<NavigationCards
columns={2}
items={[
{
icon: "👥",
title: "Proxy: Admin & Access",
description: "User and team management, SSO, SCIM, and routing rules.",
to: "/docs/tutorials/proxy_admin_access",
},
{
icon: "🛡️",
title: "Proxy: Features & Safety",
description: "Prompt caching, passthrough APIs, realtime, guardrails, and PII masking.",
to: "/docs/tutorials/proxy_features_safety",
},
]}
/>
---
## Observability & Evaluation
<NavigationCards
columns={2}
items={[
{
icon: "🔍",
title: "Observability & Evaluation",
description: "Logging to Elasticsearch, benchmarking, and evaluation suites.",
to: "/docs/tutorials/observability_evaluation",
},
]}
/>

8
docs/my-website/docs/tutorials/installation.md
View file

@ -1,7 +1,3 @@
---
displayed_sidebar: tutorialSidebar
---
# Set up environment
Let's get the necessary keys to set up our demo environment.
@ -11,7 +7,5 @@ Every LLM provider needs API keys (e.g. `OPENAI_API_KEY`). You can get API keys
Let's get them for our demo!
**OpenAI**: https://platform.openai.com/account/api-keys
**Cohere**: https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys (no credit card required)
**Cohere**: https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys (no credit card required)
**AI21**: https://studio.ai21.com/account/api-key (no credit card required)

6
docs/my-website/docs/vertex_batch_passthrough.md
View file

@ -155,6 +155,6 @@ Common error scenarios and their solutions:
## Related Documentation
- [Vertex AI Provider Documentation](./vertex.md)
- [General Batches API Documentation](../batches.md)
- [Cost Tracking and Monitoring](../observability/telemetry.md)
- [Vertex AI Provider Documentation](./providers/vertex.md)
- [General Batches API Documentation](./batches.md)
- [Cost Tracking and Monitoring](./observability/telemetry.md)

76
docs/my-website/docs/videos.md
View file

@ -290,6 +290,82 @@ curl --location 'http://localhost:4000/v1/videos' \
--header 'custom-llm-provider: azure'
```
### Character, Edit, and Extension Endpoints
LiteLLM proxy also supports these OpenAI-compatible video routes:
- `POST /v1/videos/characters`
- `GET /v1/videos/characters/{character_id}`
- `POST /v1/videos/edits`
- `POST /v1/videos/extensions`
#### Routing Behavior (`target_model_names`, encoded IDs, and provider overrides)
- `POST /v1/videos/characters` supports `target_model_names` like `POST /v1/videos`.
- When `target_model_names` is provided on character creation, LiteLLM encodes the returned `character_id` with routing metadata.
- `GET /v1/videos/characters/{character_id}` accepts encoded character IDs directly. LiteLLM decodes the ID internally and routes with the correct model/provider metadata.
- `POST /v1/videos/edits` and `POST /v1/videos/extensions` support both:
- plain `video.id`
- encoded `video.id` values returned by LiteLLM
- `custom_llm_provider` can be supplied using the same patterns as other proxy endpoints:
- header: `custom-llm-provider`
- query: `?custom_llm_provider=...`
- body: `custom_llm_provider` (or `extra_body.custom_llm_provider` where applicable)
#### Character create with `target_model_names`
```bash
curl --location 'http://localhost:4000/v1/videos/characters' \
--header 'Authorization: Bearer sk-1234' \
-F 'name=hero' \
-F 'target_model_names=gpt-4' \
-F 'video=@/path/to/character.mp4'
```
Example response (encoded `id`):
```json
{
"id": "character_...",
"object": "character",
"created_at": 1712697600,
"name": "hero"
}
```
#### Get character using encoded `character_id`
```bash
curl --location 'http://localhost:4000/v1/videos/characters/character_...' \
--header 'Authorization: Bearer sk-1234'
```
#### Video edit with encoded `video.id`
```bash
curl --location 'http://localhost:4000/v1/videos/edits' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"prompt": "Make this brighter",
"video": { "id": "video_..." }
}'
```
#### Video extension with provider override from `extra_body`
```bash
curl --location 'http://localhost:4000/v1/videos/extensions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"prompt": "Continue this scene",
"seconds": "4",
"video": { "id": "video_..." },
"extra_body": { "custom_llm_provider": "openai" }
}'
```
Test Azure video generation request
```bash

144
docs/my-website/docusaurus.config.js
View file

@ -2,9 +2,9 @@
// Note: type annotations allow type checking and IDEs autocompletion
// @ts-ignore
const lightCodeTheme = require('prism-react-renderer/themes/github');
const lightCodeTheme = require('prism-react-renderer/themes/vsLight');
// @ts-ignore
const darkCodeTheme = require('prism-react-renderer/themes/dracula');
const darkCodeTheme = require('prism-react-renderer/themes/nightOwl');
const inkeepConfig = {
baseSettings: {
@ -87,18 +87,88 @@ const config = {
},
],
[
'@docusaurus/plugin-content-blog',
'@docusaurus/plugin-content-docs',
{
id: 'release_notes',
id: 'release-notes',
path: './release_notes',
routeBasePath: 'release_notes',
blogTitle: 'Release Notes',
blogSidebarTitle: 'Releases',
blogSidebarCount: 'ALL',
postsPerPage: 'ALL',
showReadingTime: false,
sortPosts: 'descending',
include: ['**/*.{md,mdx}'],
sidebarPath: require.resolve('./sidebars-release-notes.js'),
async sidebarItemsGenerator({defaultSidebarItemsGenerator, docs, ...args}) {
const items = await defaultSidebarItemsGenerator({docs, ...args});
// Build map of doc id -> year from frontmatter date
const docYearMap = {};
for (const doc of docs) {
const date = doc.frontMatter && doc.frontMatter.date;
if (date) {
const year = new Date(date).getFullYear();
docYearMap[doc.id] = year;
}
}
function parseVersion(str) {
const match = (str || '').match(/v?(\d+)\.(\d+)\.(\d+)/);
if (!match) return [0, 0, 0];
return [parseInt(match[1]), parseInt(match[2]), parseInt(match[3])];
}
function compareVersionsDesc(a, b) {
const [aMaj, aMin, aPatch] = parseVersion(a.label || a.id || '');
const [bMaj, bMin, bPatch] = parseVersion(b.label || b.id || '');
if (bMaj !== aMaj) return bMaj - aMaj;
if (bMin !== aMin) return bMin - aMin;
return bPatch - aPatch;
}
// Flatten and transform doc items (filter index, shorten labels)
function flattenDocs(list) {
const result = [];
for (const item of list) {
if (item.type === 'doc' && item.id === 'index') continue;
if (item.type === 'doc') {
const label = item.id.replace(/\/index$/, '');
result.push({...item, label});
} else if (item.type === 'category') {
if (item.link && item.link.type === 'doc' && item.link.id !== 'index') {
const id = item.link.id;
const label = id.replace(/\/index$/, '');
result.push({type: 'doc', id, label});
} else {
result.push(...flattenDocs(item.items));
}
}
}
return result;
}
const docItems = flattenDocs(items);
// Group by year
const byYear = {};
for (const item of docItems) {
const year = docYearMap[item.id] || 'Other';
if (!byYear[year]) byYear[year] = [];
byYear[year].push(item);
}
// Sort each year's items by version descending
for (const year of Object.keys(byYear)) {
byYear[year].sort(compareVersionsDesc);
}
// Build categories sorted by year descending
const years = Object.keys(byYear).sort((a, b) => {
// Object.keys() returns strings; avoid numeric subtraction type errors.
const na = Number.parseInt(a, 10);
const nb = Number.parseInt(b, 10);
return nb - na;
});
return years.map(year => ({
type: 'category',
label: String(year),
collapsed: year !== String(years[0]),
items: byYear[year],
}));
},
},
],
[
@ -130,6 +200,20 @@ const config = {
};
},
}),
// Ensure gtag exists before the GA script loads.
() => ({
name: 'gtag-shim',
injectHtmlTags() {
return {
headTags: [
{
tagName: 'script',
innerHTML: `window.dataLayer=window.dataLayer||[];function gtag(){dataLayer.push(arguments);}if(!window.gtag){window.gtag=gtag;}`,
},
],
};
},
}),
],
presets: [
@ -137,10 +221,13 @@ const config = {
'classic',
/** @type {import('@docusaurus/preset-classic').Options} */
({
gtag: {
trackingID: 'G-K7K215ZVNC',
anonymizeIP: true,
},
gtag:
process.env.NODE_ENV === 'production'
? {
trackingID: 'G-K7K215ZVNC',
anonymizeIP: true,
}
: undefined,
docs: {
sidebarPath: require.resolve('./sidebars.js'),
},
@ -181,34 +268,39 @@ const config = {
label: 'Docs',
},
{
type: 'docSidebar',
sidebarId: 'learnSidebar',
position: 'left',
label: 'Learn',
},
{
type: 'docSidebar',
sidebarId: 'integrationsSidebar',
position: 'left',
label: 'Integrations',
to: "docs/integrations"
},
{
sidebarId: 'tutorialSidebar',
position: 'left',
label: 'Enterprise',
to: "docs/enterprise"
},
{ to: '/release_notes', label: 'Release Notes', position: 'left' },
{ to: '/blog', label: 'Blog', position: 'left' },
{
href: 'https://models.litellm.ai/',
label: '💸 LLM Model Cost Map',
position: 'right',
},
{
href: 'https://github.com/BerriAI/litellm',
label: 'GitHub',
position: 'right',
className: 'header-github-link',
'aria-label': 'GitHub repository',
},
{
href: 'https://www.litellm.ai/support',
label: 'Slack/Discord',
position: 'right',
}
className: 'header-discord-link',
'aria-label': 'Discord / Slack community',
},
{
type: 'search',
position: 'right',
},
],
},
footer: {

BIN
docs/my-website/img/ephemeral_token.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 290 KiB

BIN
docs/my-website/img/hero.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 6.2 MiB

BIN
docs/my-website/img/mcp_zero_trust_gateway.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 294 KiB

BIN
docs/my-website/img/webrtc_flow.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 238 KiB

52
docs/my-website/release_notes/index.md Normal file
View file

@ -0,0 +1,52 @@
---
title: Release Notes
sidebar_label: Overview
slug: /
---
# Release Notes
LiteLLM ships new releases regularly with new provider support, performance improvements, and enterprise features. Use the sidebar to browse all releases.
## Latest Release
### [v1.82.3 — Nebius AI, gpt-5.4, Gemini 3.x, FLUX Kontext, and 116 New Models](/release_notes/v1.82.3/v1-82-3)
_March 16, 2026_
116 new models including Nebius AI, gpt-5.4, Gemini 3.x, and FLUX Kontext.
---
## Recent Releases
| Version | Date | Highlights |
| ----------------------------------- | ------------ | ---------------------------------------------------------- |
| [v1.82.0](/release_notes/v1.82.0/v1-82-0) | Feb 28, 2026 | Realtime Guardrails, Projects Management, and 10+ Performance Optimizations |
| [v1.81.14](/release_notes/v1.81.14/v1-81-14) | Feb 21, 2026 | New Gateway Level Guardrails & Compliance Playground |
| [v1.81.12](/release_notes/v1.81.12/v1-81-12) | Feb 14, 2026 | Guardrail Policy Templates & Action Builder |
| [v1.81.9](/release_notes/v1.81.9/v1-81-9) | Feb 7, 2026 | Control which MCP Servers are exposed on the Internet |
| [v1.81.6](/release_notes/v1.81.6/v1-81-6) | Jan 31, 2026 | Logs v2 with Tool Call Tracing |
| [v1.81.3](/release_notes/v1.81.3-stable/v1-81-3) | Jan 26, 2026 | Performance — 25% CPU Usage Reduction |
| [v1.81.0](/release_notes/v1.81.0/v1-81-0) | Jan 18, 2026 | Claude Code — Web Search Across All Providers |
| [v1.80.15](/release_notes/v1.80.15/v1-80-15) | Jan 10, 2026 | Manus API Support |
| [v1.80.8](/release_notes/v1.80.8-stable/v1-80-8) | Dec 6, 2025 | Introducing A2A Agent Gateway |
| [v1.80.5](/release_notes/v1.80.5-stable/v1-80-5) | Nov 22, 2025 | Gemini 3.0 Support |
| [v1.80.0](/release_notes/v1.80.0-stable/v1-80-0) | Nov 15, 2025 | Introducing Agent Hub: Register, Publish, and Share Agents |
| [v1.79.3](/release_notes/v1.79.3-stable/v1-79-3) | Nov 8, 2025 | Built-in Guardrails on AI Gateway |
| [v1.79.0](/release_notes/v1.79.0-stable/v1-79-0) | Oct 26, 2025 | Search APIs |
| [v1.78.5](/release_notes/v1.78.5-stable/v1-78-5) | Oct 18, 2025 | Native OCR Support |
| [v1.78.0](/release_notes/v1.78.0-stable/v1-78-0) | Oct 11, 2025 | MCP Gateway: Control Tool Access by Team, Key |
| [v1.77.7](/release_notes/v1.77.7-stable/v1-77-7) | Oct 4, 2025 | 2.9x Lower Median Latency |
| [v1.77.5](/release_notes/v1.77.5-stable/v1-77-5) | Sep 29, 2025 | MCP OAuth 2.0 Support |
| [v1.77.3](/release_notes/v1.77.3-stable/v1-77-3) | Sep 21, 2025 | Priority Based Rate Limiting |
---
## Stay Updated
- **GitHub**: Watch the [BerriAI/litellm](https://github.com/BerriAI/litellm) repository for release notifications
- **Discord**: Join our [community](https://discord.com/invite/wuPM9dRgDw) for announcements
- **Twitter**: Follow [@LiteLLM](https://twitter.com/LiteLLM)
Use the sidebar to browse the full release history.

8
docs/my-website/release_notes/v1.63.14/index.md
View file

@ -62,10 +62,10 @@ Here's a Demo Instance to test changes:
- Infer aws region from bedrock application profile id - (`arn:aws:bedrock:us-east-1:...`)
- Ollama - support calling via `/v1/completions` [Get Started](../../docs/providers/ollama#using-ollama-fim-on-v1completions)
- Bedrock - support `us.deepseek.r1-v1:0` model name [Docs](../../docs/providers/bedrock#supported-aws-bedrock-models)
- OpenRouter - `OPENROUTER_API_BASE` env var support [Docs](../../docs/providers/openrouter.md)
- OpenRouter - `OPENROUTER_API_BASE` env var support [Docs](../../docs/providers/openrouter)
- Azure - add audio model parameter support - [Docs](../../docs/providers/azure#azure-audio-model)
- OpenAI - PDF File support [Docs](../../docs/completion/document_understanding#openai-file-message-type)
- OpenAI - o1-pro Responses API streaming support [Docs](../../docs/response_api.md#streaming)
- OpenAI - o1-pro Responses API streaming support [Docs](../../docs/response_api#streaming)
- [BETA] MCP - Use MCP Tools with LiteLLM SDK [Docs](../../docs/mcp)
2. **Bug Fixes**
@ -102,7 +102,7 @@ Here's a Demo Instance to test changes:
- fix logging to just log the LLM I/O [PR](https://github.com/BerriAI/litellm/pull/9353)
- Dynamic API Key/Space param support [Get Started](../../docs/observability/arize_integration#pass-arize-spacekey-per-request)
- StandardLoggingPayload - Log litellm_model_name in payload. Allows knowing what the model sent to API provider was [Get Started](../../docs/proxy/logging_spec#standardlogginghiddenparams)
- Prompt Management - Allow building custom prompt management integration [Get Started](../../docs/proxy/custom_prompt_management.md)
- Prompt Management - Allow building custom prompt management integration [Get Started](../../docs/proxy/custom_prompt_management)
## Performance / Reliability improvements
@ -128,4 +128,4 @@ Here's a Demo Instance to test changes:
## Complete Git Diff
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.11-stable...v1.63.14.rc)
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.11-stable...v1.63.14.rc)

8
docs/my-website/release_notes/v1.63.2-stable/index.md
View file

@ -64,15 +64,15 @@ Here's a Demo Instance to test changes:
9. Bedrock - handle thinking blocks in assistant message. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content)
10. Anthropic - Return `signature` on streaming. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content)
- Note: We've also migrated from `signature_delta` to `signature`. [Read more](https://docs.litellm.ai/release_notes/v1.63.0)
11. Support format param for specifying image type. [Get Started](../../docs/completion/vision.md#explicitly-specify-image-type)
12. Anthropic - `/v1/messages` endpoint - `thinking` param support. [Get Started](../../docs/anthropic_unified.md)
11. Support format param for specifying image type. [Get Started](../../docs/completion/vision#explicitly-specify-image-type)
12. Anthropic - `/v1/messages` endpoint - `thinking` param support. [Get Started](../../docs/anthropic_unified)
- Note: this refactors the [BETA] unified `/v1/messages` endpoint, to just work for the Anthropic API.
13. Vertex AI - handle $id in response schema when calling vertex ai. [Get Started](https://docs.litellm.ai/docs/providers/vertex#json-schema)
## Spend Tracking Improvements
1. Batches API - Fix cost calculation to run on retrieve_batch. [Get Started](https://docs.litellm.ai/docs/batches)
2. Batches API - Log batch models in spend logs / standard logging payload. [Get Started](../../docs/proxy/logging_spec.md#standardlogginghiddenparams)
2. Batches API - Log batch models in spend logs / standard logging payload. [Get Started](../../docs/proxy/logging_spec#standardlogginghiddenparams)
## Management Endpoints / UI
@ -109,4 +109,4 @@ Here's a Demo Instance to test changes:
## Complete Git Diff
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.61.20-stable...v1.63.2-stable)
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.61.20-stable...v1.63.2-stable)

3
docs/my-website/release_notes/v1.80.15/index.md
View file

@ -53,7 +53,7 @@ pip install litellm==1.80.15
- **MCP Global Mode** - [Configure MCP servers globally with visibility controls](../../docs/mcp)
- **Interactions API Bridge** - [Use all LiteLLM providers with the Interactions API](../../docs/interactions)
- **RAG Query Endpoint** - [New RAG Search/Query endpoint for retrieval-augmented generation](../../docs/search/index)
- **UI Usage - Endpoint Activity** - [Users can now see Endpoint Activity Metrics in the UI](../../docs/proxy/endpoint_activity.md)
- **UI Usage - Endpoint Activity** - [Users can now see Endpoint Activity Metrics in the UI](../../docs/proxy/endpoint_activity)
- **50% Overhead Reduction** - LiteLLM now sends 2.5× more requests to LLM providers
@ -640,4 +640,3 @@ Users can now see Endpoint Activity Metrics in the UI.
**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.11.rc.1...v1.80.15-stable.1)**

4
docs/my-website/release_notes/v1.81.0/index.md
View file

@ -48,7 +48,7 @@ pip install litellm==1.81.0
- **Claude Code** - Support for using web search across Bedrock, Vertex AI, and all LiteLLM providers
- **Major Change** - [50MB limit on image URL downloads](#major-change---chatcompletions-image-url-download-size-limit) to improve reliability
- **Performance** - [25% CPU Usage Reduction](#performance---25-cpu-usage-reduction) by removing premature model.dump() calls from the hot path
- **Deleted Keys Audit Table on UI** - [View deleted keys and teams for audit purposes](../../docs/proxy/deleted_keys_teams.md) with spend and budget information at the time of deletion
- **Deleted Keys Audit Table on UI** - [View deleted keys and teams for audit purposes](../../docs/proxy/deleted_keys_teams) with spend and budget information at the time of deletion
---
@ -166,7 +166,7 @@ LiteLLM now reduces CPU usage by removing premature `model.dump()` calls from th
<Image img={require('../../img/ui_deleted_keys_table.png')} />
LiteLLM now provides a comprehensive audit table for deleted API keys and teams directly in the UI. This feature allows you to easily track the spend of deleted keys, view their associated team information, and maintain accurate financial records for auditing and compliance purposes. The table displays key details including key aliases, team associations, and spend information captured at the time of deletion. For more information on how to use this feature, see the [Deleted Keys & Teams documentation](../../docs/proxy/deleted_keys_teams.md).
LiteLLM now provides a comprehensive audit table for deleted API keys and teams directly in the UI. This feature allows you to easily track the spend of deleted keys, view their associated team information, and maintain accurate financial records for auditing and compliance purposes. The table displays key details including key aliases, team associations, and spend information captured at the time of deletion. For more information on how to use this feature, see the [Deleted Keys & Teams documentation](../../docs/proxy/deleted_keys_teams).
---

4
docs/my-website/release_notes/v1.81.12.md → docs/my-website/release_notes/v1.81.12/index.md
View file

@ -62,13 +62,13 @@ This release fixes out-of-memory (OOM) risks from unbounded `asyncio.Queue()` us
This release adds a visual action builder for guardrail policies with conditional execution support. You can now chain guardrails into multi-step pipelines — if a simple guardrail fails, route to an advanced one instead of immediately blocking. Each step has configurable ON PASS and ON FAIL actions (Next Step, Block, or Allow), and you can test the full pipeline with a sample message before saving.
![Guardrail Action Builder](../img/release_notes/guard_actions.png)
![Guardrail Action Builder](../../img/release_notes/guard_actions.png)
### Access Groups
Access Groups simplify defining resource access across your organization. One group can grant access to models, MCP servers, and agents—simply attach it to a key or team. Create groups in the Admin UI, define which resources each group includes, then assign the group when creating keys or teams. Updates to a group apply automatically to all attached keys and teams.
<Image img={require('../img/ui_access_groups.png')} />
<Image img={require('../../img/ui_access_groups.png')} />
## New Providers and Endpoints

8
docs/my-website/release_notes/v1.81.14.md → docs/my-website/release_notes/v1.81.14/index.md
View file

@ -56,7 +56,7 @@ pip install litellm==1.81.14
AI Platform Admins can now browse built-in and partner guardrails from the Guardrail Garden. Guardrails are organized by use case — blocking financial advice, filtering insults, detecting competitor mentions, and more — so you can find the right one and deploy it in a few clicks.
![Guardrail Garden](../img/release_notes/guardrail_garden.png)
![Guardrail Garden](../../img/release_notes/guardrail_garden.png)
### 3 New Built-in Guardrails
@ -72,7 +72,7 @@ These guardrails are built for production and on our benchmarks had a 100% Recal
Previously, the `store_model_in_db` setting could only be configured in `proxy_config.yaml` under `general_settings`, requiring a proxy restart to take effect. Now you can enable or disable this setting directly from the Admin UI without any restarts. This is especially useful for cloud deployments where you don't have direct access to config files or want to avoid downtime. Enable `store_model_in_db` to move model definitions from your YAML into the database—reducing config complexity, improving scalability, and enabling dynamic model management across multiple proxy instances.
![Store model in DB Setting](../img/ui_store_model_in_db.png)
![Store model in DB Setting](../../img/ui_store_model_in_db.png)
#### Eval results
@ -91,14 +91,14 @@ We benchmarked our new built-in guardrails against labeled datasets before shipp
The Compliance Playground lets you test any guardrail against our pre-built eval datasets or your own custom datasets, so you can see precision, recall, and false positive rate before rolling it out to production.
![Compliance Playground](../img/release_notes/compliance_playground.png)
![Compliance Playground](../../img/release_notes/compliance_playground.png)
---
## Performance & Reliability — Up to 13% Lower Latency
<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
<Image img={require('../../img/release_notes/v1_81_14_perf.png')} />
This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself.

0
docs/my-website/release_notes/v1.81.6.md → docs/my-website/release_notes/v1.81.6/index.md
View file

4
docs/my-website/release_notes/v1.81.9.md → docs/my-website/release_notes/v1.81.9/index.md
View file

@ -81,7 +81,7 @@ This release makes it safe to expose MCP servers on the public internet by addin
[Get started](../../docs/mcp_public_internet)
<Image
img={require('../img/release_notes/mcp_internet.png')}
img={require('../../img/release_notes/mcp_internet.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>
@ -92,7 +92,7 @@ Set a soft budget on any team to receive email alerts when spending crosses the
[Get started](../../docs/proxy/ui_team_soft_budget_alerts)
<Image
img={require('../img/ui_team_soft_budget_alerts.png')}
img={require('../../img/ui_team_soft_budget_alerts.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>

0
docs/my-website/release_notes/v1.82.0.md → docs/my-website/release_notes/v1.82.0/index.md
View file

Some files were not shown because too many files have changed in this diff Show more