scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 154

Merge branch 'BerriAI:main' into patch-1

Browse source
This commit is contained in:
Zenwoh 2026-02-10 13:44:40 +01:00 • committed by GitHub
commit 3cbe5aa25b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1029 changed files with 36902 additions and 8328 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

53
.circleci/config.yml
View file

@ -1372,6 +1372,51 @@ jobs:
paths:
- mcp_coverage.xml
- mcp_coverage
agent_testing:
docker:
- image: cimg/python:3.11
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
working_directory: ~/project
steps:
- checkout
- setup_google_dns
- run:
name: Install Dependencies
command: |
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
pip install "pytest==7.3.1"
pip install "pytest-retry==1.6.3"
pip install "pytest-cov==5.0.0"
pip install "pytest-asyncio==0.21.1"
pip install "respx==0.22.0"
pip install "pydantic==2.11.0"
pip install "a2a-sdk"
# Run pytest and generate JUnit XML report
- run:
name: Run tests
command: |
pwd
ls
python -m pytest -vv tests/agent_tests --ignore=tests/agent_tests/local_only_agent_tests --cov=litellm --cov-report=xml -x -s -v --junitxml=test-results/junit.xml --durations=5
no_output_timeout: 120m
- run:
name: Rename the coverage files
command: |
mv coverage.xml agent_coverage.xml
mv .coverage agent_coverage
# Store test results
- store_test_results:
path: test-results
- persist_to_workspace:
root: .
paths:
- agent_coverage.xml
- agent_coverage
guardrails_testing:
docker:
- image: cimg/python:3.11
@ -4264,6 +4309,12 @@ workflows:
only:
- main
- /litellm_.*/
- agent_testing:
filters:
branches:
only:
- main
- /litellm_.*/
- guardrails_testing:
filters:
branches:
@ -4371,6 +4422,7 @@ workflows:
- llm_translation_testing
- realtime_translation_testing
- mcp_testing
- agent_testing
- google_generate_content_endpoint_testing
- guardrails_testing
- llm_responses_api_testing
@ -4449,6 +4501,7 @@ workflows:
- llm_translation_testing
- realtime_translation_testing
- mcp_testing
- agent_testing
- google_generate_content_endpoint_testing
- llm_responses_api_testing
- ocr_testing

2
.dockerignore
View file

@ -48,7 +48,7 @@ dist/
build/
*.egg-info/
.DS_Store
node_modules/
**/node_modules
*.log
.env
.env.local

1
CLAUDE.md
View file

@ -90,6 +90,7 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
- Pydantic v2 for data validation
- Async/await patterns throughout
- Type hints required for all public APIs
- **Avoid imports within methods** — place all imports at the top of the file (module-level). Inline imports inside functions/methods make dependencies harder to trace and hurt readability. The only exception is avoiding circular imports where absolutely necessary.
### Testing Strategy
- Unit tests in `tests/test_litellm/`

48
CONTRIBUTING.md
View file

@ -7,11 +7,20 @@ Thank you for your interest in contributing to LiteLLM! We welcome contributions
Here are the core requirements for any PR submitted to LiteLLM:
- [ ] **Sign the Contributor License Agreement (CLA)** - [see details](#contributor-license-agreement-cla)
- [ ] **Keep scope isolated** - Your changes should address 1 specific problem at a time
#### Proxy (Backend) PRs
- [ ] **Add testing** - Adding at least 1 test is a hard requirement - [see details](#adding-testing)
- [ ] **Ensure your PR passes all checks**:
- [ ] [Unit Tests](#running-unit-tests) - `make test-unit`
- [ ] [Linting / Formatting](#running-linting-and-formatting-checks) - `make lint`
- [ ] **Keep scope isolated** - Your changes should address 1 specific problem at a time
#### UI PRs
- [ ] **Ensure the UI builds successfully** - `npm run build`
- [ ] **Ensure all UI unit tests pass** - `npm run test`
- [ ] **Add tests for new components or logic** - If you are adding a new component or new logic, add corresponding tests
## **Contributor License Agreement (CLA)**
@ -245,6 +254,43 @@ docker run \
--config /app/config.yaml --detailed_debug
```
## UI Development
### 1. Setup Your Local UI Development Environment
```bash
# Clone the repo (if you haven't already)
git clone https://github.com/YOUR_USERNAME/litellm.git
cd litellm
# Navigate to the UI dashboard directory
cd ui/litellm-dashboard
# Install dependencies
npm install
# Start the development server
npm run dev
```
### 2. Adding UI Tests
If you are adding a **new component** or **new logic**, you must add corresponding tests.
### 3. Running UI Unit Tests
```bash
npm run test
```
### 4. Building the UI
Ensure the UI builds successfully before submitting your PR:
```bash
npm run build
```
## Submitting Your PR
1. **Push your branch**: `git push origin your-feature-branch`

36
Dockerfile
View file

@ -3,6 +3,7 @@ ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -48,7 +49,22 @@ USER root
# Install runtime dependencies (libsndfile needed for audio processing on ARM64)
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@latest
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
# SECURITY FIX: npm bundles tar, glob, and brace-expansion at multiple nested
# levels inside its dependency tree. `npm install -g <pkg>` only creates a
# SEPARATE global package, it does NOT replace npm's internal copies.
# We must find and replace EVERY copy inside npm's directory.
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
npm cache clean --force
WORKDIR /app
# Copy the current directory contents into the container at /app
@ -62,10 +78,28 @@ COPY --from=builder /wheels/ /wheels/
# Install the built wheel using pip; again using a wildcard if it's the only file
RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ && rm -f *.whl && rm -rf /wheels
# Replace the nodejs-wheel-binaries bundled node with the system node (fixes CVE-2025-55130)
RUN NODEJS_WHEEL_NODE=$(find /usr/lib -path "*/nodejs_wheel/bin/node" 2>/dev/null) && \
if [ -n "$NODEJS_WHEEL_NODE" ]; then cp /usr/bin/node "$NODEJS_WHEEL_NODE"; fi
# Remove test files and keys from dependencies
RUN find /usr/lib -type f -path "*/tornado/test/*" -delete && \
find /usr/lib -type d -path "*/tornado/test" -delete
# SECURITY FIX: nodejs-wheel-binaries (pip package used by Prisma) bundles a complete
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done
# Install semantic_router and aurelio-sdk using script
# Convert Windows line endings to Unix and make executable
RUN sed -i 's/\r$//' docker/install_auto_router.sh && chmod +x docker/install_auto_router.sh && ./docker/install_auto_router.sh

2
README.md
View file

@ -309,7 +309,7 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
| [Deepgram (`deepgram`)](https://docs.litellm.ai/docs/providers/deepgram) | ✅ | ✅ | ✅ | | | ✅ | | | | |
| [DeepInfra (`deepinfra`)](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | | | | | | | |
| [Deepseek (`deepseek`)](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ | | | | | | | |
| [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ | | | | ✅ | | | |
| [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ | | | ✅ | ✅ | | | |
| [Empower (`empower`)](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | | | | | | | |
| [Fal AI (`fal_ai`)](https://docs.litellm.ai/docs/providers/fal_ai) | ✅ | ✅ | ✅ | | ✅ | | | | | |
| [Featherless AI (`featherless_ai`)](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | | | | | | | |

1
ci_cd/security_scans.sh
View file

@ -155,6 +155,7 @@ run_grype_scans() {
"CVE-2025-12781" # No fix available yet
"CVE-2025-11468" # No fix available yet
"CVE-2026-1299" # Python 3.13 email module header injection - not applicable, LiteLLM doesn't use BytesGenerator for email serialization
"CVE-2026-0775" # npm cli incorrect permission assignment - no fix available yet, npm is only used at build/prisma-generate time
)
# Build JSON array of allowlisted CVE IDs for jq

13
docker/Dockerfile.custom_ui
View file

@ -6,7 +6,18 @@ WORKDIR /app
# Install Node.js and npm (adjust version as needed)
RUN apt-get update && apt-get install -y nodejs npm && \
npm install -g npm@latest tar@latest
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
npm cache clean --force
# Copy the UI source into the container
COPY ./ui/litellm-dashboard /app/ui/litellm-dashboard

27
docker/Dockerfile.database
View file

@ -50,7 +50,18 @@ USER root
# Install runtime dependencies
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@latest
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
npm cache clean --force
WORKDIR /app
# Copy the current directory contents into the container at /app
@ -64,6 +75,20 @@ COPY --from=builder /wheels/ /wheels/
# Install the built wheel using pip; again using a wildcard if it's the only file
RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ && rm -f *.whl && rm -rf /wheels
# SECURITY FIX: nodejs-wheel-binaries (pip package used by Prisma) bundles a complete
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done
# Install semantic_router and aurelio-sdk using script
# Convert Windows line endings to Unix and make executable
RUN sed -i 's/\r$//' docker/install_auto_router.sh && chmod +x docker/install_auto_router.sh && ./docker/install_auto_router.sh

27
docker/Dockerfile.dev
View file

@ -62,7 +62,18 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/* \
&& npm install -g npm@latest tar@latest
&& npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done \
&& npm cache clean --force
WORKDIR /app
@ -80,6 +91,20 @@ RUN pip install --no-cache-dir *.whl /wheels/* --no-index --find-links=/wheels/
rm -f *.whl && \
rm -rf /wheels
# SECURITY FIX: nodejs-wheel-binaries (pip package used by Prisma) bundles a complete
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done
# Generate prisma client and set permissions
# Convert Windows line endings to Unix for entrypoint scripts
RUN prisma generate && \

27
docker/Dockerfile.non_root
View file

@ -104,7 +104,18 @@ RUN for i in 1 2 3; do \
&& for i in 1 2 3; do \
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor && break || sleep 5; \
done \
&& npm install -g npm@latest tar@latest
&& npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done \
&& npm cache clean --force
# Copy artifacts from builder
COPY --from=builder /app/requirements.txt /app/requirements.txt
@ -146,6 +157,20 @@ RUN pip install --no-index --find-links=/wheels/ -r requirements.txt && \
fi; \
fi
# SECURITY FIX: nodejs-wheel-binaries (pip package used by Prisma) bundles a complete
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done
# Permissions, cleanup, and Prisma prep
# Convert Windows line endings to Unix for entrypoint scripts
RUN sed -i 's/\r$//' docker/entrypoint.sh && \

491
docs/my-website/blog/claude_opus_4_6/index.md
View file

@ -3,6 +3,10 @@ slug: claude_opus_4_6
title: "Day 0 Support: Claude Opus 4.6"
date: 2026-02-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
@ -219,6 +223,489 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
</TabItem>
</Tabs>
## More Features Coming Soon
## Advanced Features
We're actively working on supporting new features for Claude Opus 4.6. Stay tuned for updates!
### Compaction
<Tabs>
<TabItem value="completions" label="/chat/completions">
Litellm supports enabling compaction for the new claude-opus-4-6.
**Enabling Compaction**
To enable compaction, add the `context_management` parameter with the `compact_20260112` edit type:
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "What is the weather in San Francisco?"
}
],
"context_management": {
"edits": [
{
"type": "compact_20260112"
}
]
},
"max_tokens": 100
}'
```
All the parameters supported for context_management by anthropic are supported and can be directly added. Litellm automatically adds the `compact-2026-01-12` beta header in the request.
</TabItem>
<TabItem value="messages" label="/v1/messages">
Enable compaction to reduce context size while preserving key information. LiteLLM automatically adds the `compact-2026-01-12` beta header when compaction is enabled.
:::info
**Provider Support:** Compaction is supported on Anthropic, Azure AI, and Vertex AI. It is **not supported** on Bedrock (Invoke or Converse APIs).
:::
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 4096,
"messages": [
{
"role": "user",
"content": "Hi"
}
],
"context_management": {
"edits": [
{
"type": "compact_20260112"
}
]
}
}'
```
</TabItem>
</Tabs>
**Response with Compaction Block**
The response will include the compaction summary in `provider_specific_fields.compaction_blocks`:
```json
{
"id": "chatcmpl-a6c105a3-4b25-419e-9551-c800633b6cb2",
"created": 1770357619,
"model": "claude-opus-4-6",
"object": "chat.completion",
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"content": "I don't have access to real-time data, so I can't provide the current weather in San Francisco. To get up-to-date weather information, I'd recommend checking:\n\n- **Weather websites** like weather.com, accuweather.com, or wunderground.com\n- **Search engines** – just Google \"San Francisco weather\"\n- **Weather apps** on your phone (e.g., Apple Weather, Google Weather)\n- **National",
"role": "assistant",
"provider_specific_fields": {
"compaction_blocks": [
{
"type": "compaction",
"content": "Summary of the conversation: The user requested help building a web scraper..."
}
]
}
}
}
],
"usage": {
"completion_tokens": 100,
"prompt_tokens": 86,
"total_tokens": 186
}
}
```
**Using Compaction Blocks in Follow-up Requests**
To continue the conversation with compaction, include the compaction block in the assistant message's `provider_specific_fields`:
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "How can I build a web scraper?"
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "Certainly! To build a basic web scraper, you'll typically use a programming language like Python along with libraries such as `requests` (for fetching web pages) and `BeautifulSoup` (for parsing HTML). Here's a basic example:\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\n\nurl = 'https://example.com'\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.text, 'html.parser')\n\n# Extract and print all text\ntext = soup.get_text()\nprint(text)\n```\n\nLet me know what you're interested in scraping or if you need help with a specific website!"
}
],
"provider_specific_fields": {
"compaction_blocks": [
{
"type": "compaction",
"content": "Summary of the conversation: The user asked how to build a web scraper, and the assistant gave an overview using Python with requests and BeautifulSoup."
}
]
}
},
{
"role": "user",
"content": "How do I use it to scrape product prices?"
}
],
"context_management": {
"edits": [
{
"type": "compact_20260112"
}
]
},
"max_tokens": 100
}'
```
**Streaming Support**
Compaction blocks are also supported in streaming mode. You'll receive:
- `compaction_start` event when a compaction block begins
- `compaction_delta` events with the compaction content
- The accumulated `compaction_blocks` in `provider_specific_fields`
### Adaptive Thinking
<Tabs>
<TabItem value="completions" label="/chat/completions">
LiteLLM supports adaptive thinking through the `reasoning_effort` parameter:
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "Solve this complex problem: What is the optimal strategy for..."
}
],
"reasoning_effort": "high"
}'
```
</TabItem>
<TabItem value="messages" label="/v1/messages">
Use the `thinking` parameter with `type: "adaptive"` to enable adaptive thinking mode:
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 16000,
"thinking": {
"type": "adaptive"
},
"messages": [
{
"role": "user",
"content": "Explain why the sum of two even numbers is always even."
}
]
}'
```
</TabItem>
</Tabs>
### Effort Levels
<Tabs>
<TabItem value="completions" label="/chat/completions">
Four effort levels available: `low`, `medium`, `high` (default), and `max`. Pass directly via the `output_config` parameter:
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "Explain quantum computing"
}
],
"output_config": {
"effort": "medium"
}
}'
```
You can use reasoning effort plus output_config to have more control on the model.
</TabItem>
<TabItem value="messages" label="/v1/messages">
Four effort levels available: `low`, `medium`, `high` (default), and `max`. Pass directly via the `output_config` parameter:
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 4096,
"messages": [
{
"role": "user",
"content": "Explain quantum computing"
}
],
"output_config": {
"effort": "medium"
}
}'
```
</TabItem>
</Tabs>
### 1M Token Context (Beta)
Opus 4.6 supports 1M token context. Premium pricing applies for prompts exceeding 200k tokens ($10/$37.50 per million input/output tokens). LiteLLM supports cost calculations for 1M token contexts.
<Tabs>
<TabItem value="completions" label="/chat/completions">
To use the 1M token context window, you need to forward the `anthropic-beta` header from your client to the LLM provider.
**Step 1: Enable header forwarding in your config**
```yaml
general_settings:
forward_client_headers_to_llm_api: true
```
**Step 2: Send requests with the beta header**
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--header 'anthropic-beta: context-1m-2025-08-07' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "Analyze this large document..."
}
]
}'
```
</TabItem>
<TabItem value="messages" label="/v1/messages">
To use the 1M token context window, you need to forward the `anthropic-beta` header from your client to the LLM provider.
**Step 1: Enable header forwarding in your config**
```yaml
general_settings:
forward_client_headers_to_llm_api: true
```
**Step 2: Send requests with the beta header**
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'anthropic-beta: context-1m-2025-08-07' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 16000,
"messages": [
{
"role": "user",
"content": "Analyze this large document..."
}
]
}'
```
:::tip
You can combine multiple beta headers by separating them with commas:
```bash
--header 'anthropic-beta: context-1m-2025-08-07,compact-2026-01-12'
```
:::
</TabItem>
</Tabs>
### US-Only Inference
Available at 1.1× token pricing. LiteLLM automatically tracks costs for US-only inference.
<Tabs>
<TabItem value="completions" label="/chat/completions">
Use the `inference_geo` parameter to specify US-only inference:
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"inference_geo": "us"
}'
```
LiteLLM will automatically apply the 1.1× pricing multiplier for US-only inference in cost tracking.
</TabItem>
<TabItem value="messages" label="/v1/messages">
Use the `inference_geo` parameter to specify US-only inference:
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 4096,
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"inference_geo": "us"
}'
```
LiteLLM will automatically apply the 1.1× pricing multiplier for US-only inference in cost tracking.
</TabItem>
</Tabs>
### Fast Mode
:::info
Fast mode is **only supported on the Anthropic provider** (`anthropic/claude-opus-4-6`). It is not available on Azure AI, Vertex AI, or Bedrock.
:::
**Pricing:**
- Standard: $5 input / $25 output per MTok
- Fast: $30 input / $150 output per MTok (6× premium)
<Tabs>
<TabItem value="completions" label="/chat/completions">
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
"model": "claude-opus-4-6",
"messages": [
{
"role": "user",
"content": "Refactor this module..."
}
],
"max_tokens": 4096,
"speed": "fast"
}'
```
**Using OpenAI SDK:**
```python
import openai
client = openai.OpenAI(
api_key="your-litellm-key",
base_url="http://0.0.0.0:4000"
)
response = client.chat.completions.create(
model="claude-opus-4-6",
messages=[{"role": "user", "content": "Refactor this module..."}],
max_tokens=4096,
extra_body={"speed": "fast"}
)
```
**Using LiteLLM SDK:**
```python
from litellm import completion
response = completion(
model="anthropic/claude-opus-4-6",
messages=[{"role": "user", "content": "Refactor this module..."}],
max_tokens=4096,
speed="fast"
)
```
LiteLLM automatically tracks the higher costs for fast mode in usage and cost calculations.
</TabItem>
<TabItem value="messages" label="/v1/messages">
```bash
curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
"model": "claude-opus-4-6",
"max_tokens": 4096,
"speed": "fast",
"messages": [
{
"role": "user",
"content": "Refactor this module..."
}
]
}'
```
LiteLLM automatically:
- Adds the `fast-mode-2026-02-01` beta header
- Tracks the 6× premium pricing in cost calculations
</TabItem>
</Tabs>

220
docs/my-website/blog/fastapi_middleware_performance/index.mdx Normal file
View file

@ -0,0 +1,220 @@
---
slug: fastapi-middleware-performance
title: "Your Middleware Could Be a Bottleneck"
date: 2026-02-07T10:00:00
authors:
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Ryan Crabbe
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
description: "How we improved LiteLLM proxy latency and throughput by replacing a single middleware base class"
tags: [performance, fastapi, middleware]
hide_table_of_contents: false
---
import { BaseHTTPMiddlewareAnimation, PureASGIAnimation, BenchmarkVisualization } from '@site/src/components/MiddlewareDiagrams';
> How we improved LiteLLM proxy latency and throughput by replacing a single, simple middleware base class
---
## Our Setup
The LiteLLM proxy server has two middleware layers. The first is Starlette's `CORSMiddleware` (re-exported by FastAPI), which is a pure ASGI middleware. Then we have a simple BaseHTTPMiddleware called PrometheusAuthMiddleware.
The job of `PrometheusAuthMiddleware` is to authenticate requests to the `/metrics` endpoint. It's not on by default, you enable it with a flag in your proxy config:
<details>
<summary>Proxy config flag</summary>
```yaml
litellm_settings:
require_auth_for_metrics_endpoint: true
```
</details>
The middleware checks two things: is the request hitting `/metrics`, and is auth even enabled? If both checks fail, which they do for the vast majority of requests, it just passes the request through unchanged.
<details>
<summary>PrometheusAuthMiddleware source</summary>
```python
class PrometheusAuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
if self._is_prometheus_metrics_endpoint(request):
if self._should_run_auth_on_metrics_endpoint() is True:
try:
await user_api_key_auth(request=request, api_key=...)
except Exception as e:
return JSONResponse(status_code=401, content=...)
response = await call_next(request)
return response
@staticmethod
def _is_prometheus_metrics_endpoint(request: Request):
if "/metrics" in request.url.path:
return True
return False
```
</details>
Looks harmless. Subclass `BaseHTTPMiddleware`, implement `dispatch()`, done. This is what you will see in Starlette's documentation<sup>[1](#footnote-1)</sup>.
{/* truncate */}
---
## What BaseHTTPMiddleware Actually Does
When you write a `dispatch()` method, you'd expect the request to flow straight through your function and out the other side. What actually happens is much more involved.
On every request, even a pure passthrough (meaning nothing happens), `BaseHTTPMiddleware` creates **7 intermediate objects and tasks**:
<BaseHTTPMiddlewareAnimation />
It wraps the request in a new object to track body state, creates a synchronization event, allocates an in-memory channel to pass messages between your middleware and the inner app, sets up a task group to manage the lifecycle, and then runs your actual route handler in a *separate background task* when you call `call_next()`. The response body then flows back through that in-memory channel, gets re-wrapped in a streaming response object, and finally reaches the caller. That's a lot.
For a middleware that for us, does nothing on 99.9% of requests, paying this cost doesn't make sense.
Compare that to a pure ASGI middleware, which we can have just check the request path and continue along.
<PureASGIAnimation />
Our middleware is doing something really simple. For the vast majority of requests it doesn't need to do anything at all but just let the request pass through. It doesn't need task groups, memory streams, or cancel scopes. It needs a function call.
---
## Comparing Both
We replaced the `BaseHTTPMiddleware` subclass with a pure ASGI middleware. To benchmark the difference, we used Apache Bench<sup>[2](#footnote-2)</sup> to compare both configurations of LiteLLM's middleware stack: the old setup (1 pure ASGI + 1 `BaseHTTPMiddleware`) against the new setup (2 pure ASGI).
A minimal FastAPI app serves `GET /health` → `PlainTextResponse("ok")`. The endpoint does zero work to isolate the middleware overhead: any difference between configs is purely the cost of the middleware plumbing itself. Both middlewares are just calling the next layer. Same work, different base class.
Apache Bench (`ab`) fires requests at the server with 1,000 concurrent connections and a single uvicorn worker. One worker means one event loop, so the benchmark directly measures how each middleware design handles concurrent load on a single thread.
<BenchmarkVisualization />
<details>
<summary>Try it yourself</summary>
Save the script below as `benchmark_middleware.py`, then run:
```bash
# Terminal 1 — start the "before" server (1 ASGI + 1 BaseHTTPMiddleware)
python benchmark_middleware.py --middleware mixed
# Terminal 2 — benchmark it
ab -n 50000 -c 1000 http://localhost:8000/health
# Stop the server, then start the "after" server (2x pure ASGI)
python benchmark_middleware.py --middleware asgi
# Terminal 2 — benchmark again
ab -n 50000 -c 1000 http://localhost:8000/health
```
```python
import argparse
import uvicorn
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.types import ASGIApp, Receive, Scope, Send
class NoOpBaseHTTPMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
return await call_next(request)
class NoOpPureASGIMiddleware:
def __init__(self, app: ASGIApp) -> None:
self.app = app
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
await self.app(scope, receive, send)
def create_app(middleware_type: str | None = None, layers: int = 2) -> FastAPI:
app = FastAPI()
@app.get("/health")
async def health():
return PlainTextResponse("ok")
if middleware_type == "mixed":
app.add_middleware(NoOpBaseHTTPMiddleware)
app.add_middleware(NoOpPureASGIMiddleware)
elif middleware_type == "asgi":
for _ in range(layers):
app.add_middleware(NoOpPureASGIMiddleware)
return app
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--middleware", choices=["asgi", "mixed"], default=None)
parser.add_argument("--layers", type=int, default=2)
parser.add_argument("--port", type=int, default=8000)
args = parser.parse_args()
app = create_app(middleware_type=args.middleware, layers=args.layers)
uvicorn.run(app, host="0.0.0.0", port=args.port, workers=1, log_level="warning")
```
</details>
---
## Our Change
Here's what we replaced it with:
```python
class PrometheusAuthMiddleware:
def __init__(self, app: ASGIApp) -> None:
self.app = app
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http" or "/metrics" not in scope.get("path", ""):
await self.app(scope, receive, send)
return
if litellm.require_auth_for_metrics_endpoint is True:
request = Request(scope, receive)
api_key = request.headers.get("Authorization") or ""
try:
await user_api_key_auth(request=request, api_key=api_key)
except Exception as e:
# send 401 directly via ASGI protocol
...
return
await self.app(scope, receive, send)
```
For the 99.9% of requests that aren't hitting `/metrics`, the middleware is now one dict lookup, one string check, and one function call. No objects allocated, no tasks spawned.
It's important to evaluate if the tools you're using are the right fit for the job as your software grows and handles more responsiblity. We're now putting in a static analysis check to prevent this from happening again with any newly introduced middlewares. If we find the use case is necessary then that's okay and we'll reevalute but for everything LiteLLM needs to do at the moment it's not.
This middleware change was one part of a broader optimization effort on the LiteLLM proxy. Across all optimizations combined, we've measured about a **30% reduction in proxy overhead** over the past two weeks.
---
<a id="footnote-1"></a>
<sup>1</sup> [Starlette Middleware — BaseHTTPMiddleware](https://starlette.dev/middleware/#basehttpmiddleware)
<a id="footnote-2"></a>
<sup>2</sup> [Apache HTTP server benchmarking tool (`ab`)](https://httpd.apache.org/docs/2.4/programs/ab.html)

136
docs/my-website/blog/litellm_observatory/index.md Normal file
View file

@ -0,0 +1,136 @@
---
slug: litellm-observatory
title: "Improve release stability with 24 hour load tests"
date: 2026-02-06T10:00:00
authors:
- name: Alexsander Hamir
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
description: "How we built a long-running, release-validation system to catch regressions before they reach users."
tags: [testing, observability, reliability, releases]
hide_table_of_contents: false
---
![LiteLLM Observatory](https://raw.githubusercontent.com/AlexsanderHamir/assets/main/Screenshot%202026-01-31%20175355.png)
# Improve release stability with 24 hour load tests
As LiteLLM adoption has grown, so have expectations around reliability, performance, and operational safety. Meeting those expectations requires more than correctness-focused tests, it requires validating how the system behaves over time, under real-world conditions.
This post introduces **LiteLLM Observatory**, a long-running release-validation system we built to catch regressions before they reach users.
---
## Why We Built the Observatory
LiteLLM operates at the intersection of external providers, long-lived network connections, and high-throughput workloads. While our unit and integration tests do an excellent job validating correctness, they are not designed to surface issues that only appear after extended operation.
A subtle lifecycle edge case discovered in v1.81.3 reinforced the need for stronger release validation in this area.
---
## A Real-World Lifecycle Edge Case
In v1.81.3, we shipped a fix for an HTTP client memory leak. The change passed unit and integration tests and behaved correctly in short-lived runs.
The issue that surfaced was not caused by a single incorrect line of logic, but by how multiple components interacted over time:
- A cached `httpx` client was configured with a 1-hour TTL
- When the cache expired, the underlying HTTP connection was closed as expected
- A higher-level client continued to hold a reference to that connection
- Subsequent requests failed with:
```
Cannot send a request, as the client has been closed
```
**Before (with bug):**
| Provider | Requests | Success | Failures | Fail % |
|----------|----------|---------|----------|--------|
| OpenAI | 720,000 | 432,000 | 288,000 | 40% |
| Azure | 692,000 | 415,200 | 276,800 | 40% |
**After (fixed):**
| Provider | Requests | Success | Failures | Fail % |
|----------|------------|-----------|----------|---------|
| OpenAI | 1,200,000 | 1,199,988 | 12 | 0.001% |
| Azure | 1,150,000 | 1,149,982 | 18 | 0.002% |
Our focus moving forward is on being the first to detect issues, even when they aren’t covered by unit tests. LiteLLM Observatory is designed to surface latency regressions, OOMs, and failure modes that only appear under real traffic patterns in **our own production deployments** during release validation.
---
### How the Observatory Works
[LiteLLM Observatory](https://github.com/BerriAI/litellm-observatory) is a testing service that runs long-running tests against our LiteLLM deployments. We trigger tests by sending API requests, and results are automatically sent to Slack when tests complete.
#### How Tests Run
1. **Start a Test**: We send a request to the Observatory API with:
- Which LiteLLM deployment to test (URL and API key)
- Which test to run (e.g., `TestOAIAzureRelease`)
- Test settings (which models to test, how long to run, failure thresholds)
2. **Smart Queueing**:
- The system checks whether we are attempting to run the exact same test more than once
- If a duplicate test is already running or queued, we receive an error to avoid wasting resources
- Otherwise, the test is added to a queue and runs when capacity is available (up to 5 tests can run concurrently by default)
3. **Instant Response**: The API responds immediately—we do not wait for the test to finish. Tests may run for hours, but the request itself completes in milliseconds.
4. **Background Execution**:
- The test runs in the background, issuing requests against our LiteLLM deployment
- It tracks request success and failure rates over time
- When the test completes, results are automatically posted to our Slack channel
#### Example: The OpenAI / Azure Reliability Test
The `TestOAIAzureRelease` test is designed to catch a class of bugs that only surface after sustained runtime:
- **Duration**: Runs continuously for 3 hours
- **Behavior**: Cycles through specified models (such as `gpt-4` and `gpt-3.5-turbo`), issuing requests continuously
- **Why 3 Hours**: This helps catch issues where HTTP clients degrade or fail after extended use (for example, a bug observed in LiteLLM v1.81.3)
- **Pass / Fail Criteria**: The test passes if fewer than 1% of requests fail. If the failure rate exceeds 1%, the test fails and we are notified in Slack
- **Key Detail**: The same HTTP client is reused for the entire run, allowing us to detect lifecycle-related bugs that only appear under prolonged reuse
#### When We Use It
- **Before Deployments**: Run tests before promoting a new LiteLLM version to production
- **Routine Validation**: Schedule regular runs (daily or weekly) to catch regressions early
- **Issue Investigation**: Run tests on demand when we suspect a deployment issue
- **Long-Running Failure Detection**: Identify bugs that only appear under sustained load, beyond what short smoke tests can reveal
### Complementing Unit Tests
Unit tests remain a foundational part of our development process. They are fast and precise, but they don’t cover:
- Real provider behavior
- Long-lived network interactions
- Resource lifecycle edge cases
- Time-dependent regressions
LiteLLM Observatory complements unit tests by validating the system as it actually runs in production-like environments.
---
### Looking Ahead
Reliability is an ongoing investment.
LiteLLM Observatory is one of several systems we’re building to continuously raise the bar on release quality and operational safety. As LiteLLM evolves, so will our validation tooling, informed by real-world usage and lessons learned.
We’ll continue to share those improvements openly as we go.

150
docs/my-website/docs/extras/contributing_code.md
View file

@ -1,27 +1,36 @@
# Contributing Code
## **Checklist before submitting a PR**
## Checklist before submitting a PR
Here are the core requirements for any PR submitted to LiteLLM
Here are the core requirements for any PR submitted to LiteLLM:
- [ ] Sign the Contributor License Agreement (CLA) - [see details](#contributor-license-agreement-cla)
- [ ] Add testing, **Adding at least 1 test is a hard requirement** - [see details](#2-adding-testing-to-your-pr)
- [ ] Ensure your PR passes the following tests:
- [ ] [Unit Tests](#3-running-unit-tests)
- [ ] [Formatting / Linting Tests](#35-running-linting-tests)
- [ ] Keep scope as isolated as possible. As a general rule, your changes should address 1 specific problem at a time
- [ ] Sign the [Contributor License Agreement (CLA)](#contributor-license-agreement-cla)
- [ ] Keep scope as isolated as possible — your changes should address **one specific problem** at a time
## **Contributor License Agreement (CLA)**
### Proxy (Backend) PRs
- [ ] Add testing — **at least 1 test is a hard requirement** ([details](#2-adding-tests))
- [ ] Ensure your PR passes:
- [ ] [Unit Tests](#3-running-unit-tests) — `make test-unit`
- [ ] [Formatting / Linting Tests](#4-running-linting-tests) — `make lint`
### UI PRs
- [ ] Ensure the UI builds successfully — `npm run build`
- [ ] Ensure all UI unit tests pass — `npm run test`
- [ ] If you are adding a **new component** or **new logic**, add corresponding tests
## Contributor License Agreement (CLA)
Before contributing code to LiteLLM, you must sign our [Contributor License Agreement (CLA)](https://cla-assistant.io/BerriAI/litellm). This is a legal requirement for all contributions to be merged into the main repository. The CLA helps protect both you and the project by clearly defining the terms under which your contributions are made.
**Important:** We strongly recommend reviewing and signing the CLA before starting work on your contribution to avoid any delays in the PR process. You can find the CLA [here](https://cla-assistant.io/BerriAI/litellm) and sign it through our CLA management system when you submit your first PR.
**Important:** We strongly recommend signing the CLA **before** starting work on your contribution to avoid delays in the review process. You can find and sign the CLA [here](https://cla-assistant.io/BerriAI/litellm).
## Quick start
---
## 1. Setup your local dev environment
## Proxy (Backend)
Here's how to modify the repo locally:
### 1. Setting up your local dev environment
Step 1: Clone the repo
@ -29,56 +38,53 @@ Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
```
Step 2: Install dev dependencies:
Step 2: Install dev dependencies
```shell
poetry install --with dev --extras proxy
```
That's it, your local dev environment is ready!
### 2. Adding tests
## 2. Adding Testing to your PR
- Add your tests to the [`tests/test_litellm/` directory](https://github.com/BerriAI/litellm/tree/main/tests/litellm).
- This directory mirrors the `litellm/` directory 1:1 and should **only** contain mocked tests.
- **Do not** add real LLM API calls to this directory.
- Add your test to the [`tests/test_litellm/` directory](https://github.com/BerriAI/litellm/tree/main/tests/litellm)
#### File naming convention for `tests/test_litellm/`
- This directory 1:1 maps the the `litellm/` directory, and can only contain mocked tests.
- Do not add real llm api calls to this directory.
The test directory follows the same structure as `litellm/`:
### 2.1 File Naming Convention for `tests/test_litellm/`
The `tests/test_litellm/` directory follows the same directory structure as `litellm/`.
- `litellm/proxy/test_caching_routes.py` maps to `litellm/proxy/caching_routes.py`
- `test_{filename}.py` maps to `litellm/{filename}.py`
- `litellm/proxy/test_caching_routes.py` maps to `litellm/proxy/caching_routes.py`
## 3. Running Unit Tests
### 3. Running unit tests
run the following command on the root of the litellm directory
Run the following command from the root of the `litellm` directory:
```shell
make test-unit
```
## 3.5 Running Linting Tests
### 4. Running linting tests
run the following command on the root of the litellm directory
Run the following command from the root of the `litellm` directory:
```shell
make lint
```
LiteLLM uses mypy for linting. On ci/cd we also run `black` for formatting.
LiteLLM uses `mypy` for type checking. CI/CD also runs `black` for formatting.
## 4. Submit a PR with your changes!
### 5. Submit a PR
- push your fork to your GitHub repo
- submit a PR from there
- Push your changes to your fork on GitHub
- Open a Pull Request from your fork
## Advanced
---
### Building LiteLLM Docker Image
## UI
Some people might want to build the LiteLLM docker image themselves. Follow these instructions if you want to build / run the LiteLLM Docker Image yourself.
### 1. Setting up your local dev environment
Step 1: Clone the repo
@ -86,17 +92,72 @@ Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
```
Step 2: Build the Docker Image
Step 2: Navigate to the UI dashboard directory
Build using Dockerfile.non_root
```shell
cd ui/litellm-dashboard
```
Step 3: Install dependencies
```shell
npm install
```
Step 4: Start the development server
```shell
npm run dev
```
### 2. Adding tests
If you are adding a **new component** or **new logic**, you must add corresponding tests.
### 3. Running UI unit tests
```shell
npm run test
```
### 4. Building the UI
Ensure the UI builds successfully before submitting your PR:
```shell
npm run build
```
### 5. Submit a PR
- Push your changes to your fork on GitHub
- Open a Pull Request from your fork
---
## Advanced
### Building the LiteLLM Docker Image
Follow these instructions if you want to build and run the LiteLLM Docker image yourself.
Step 1: Clone the repo
```shell
git clone https://github.com/BerriAI/litellm.git
```
Step 2: Build the Docker image
Build using `Dockerfile.non_root`:
```shell
docker build -f docker/Dockerfile.non_root -t litellm_test_image .
```
Step 3: Run the Docker Image
Step 3: Run the Docker image
Make sure config.yaml is present in the root directory. This is your litellm proxy config file.
Make sure `config.yaml` is present in the root directory. This is your LiteLLM proxy config file.
```shell
docker run \
@ -107,18 +168,19 @@ docker run \
litellm_test_image \
--config /app/config.yaml --detailed_debug
```
### Running LiteLLM Proxy Locally
1. cd into the `proxy/` directory
### Running the LiteLLM Proxy Locally
```
1. Navigate to the `proxy/` directory:
```shell
cd litellm/litellm/proxy
```
2. Run the proxy
2. Run the proxy:
```shell
python3 proxy_cli.py --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
```

411
docs/my-website/docs/integrations/websearch_interception.md Normal file
View file

@ -0,0 +1,411 @@
# Web Search Integration
Enable transparent server-side web search execution for any LLM provider. LiteLLM automatically intercepts web search tool calls and executes them using your configured search provider (Perplexity, Tavily, etc.).
## Quick Start
### 1. Configure Web Search Interception
Add to your `config.yaml`:
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
callbacks:
- websearch_interception:
enabled_providers:
- openai
- minimax
- anthropic
search_tool_name: perplexity-search # Optional
search_tools:
- search_tool_name: perplexity-search
litellm_params:
search_provider: perplexity
api_key: os.environ/PERPLEXITY_API_KEY
```
### 2. Use with Any Provider
```python
import litellm
response = await litellm.acompletion(
model="gpt-4o",
messages=[
{"role": "user", "content": "What's the weather in San Francisco today?"}
],
tools=[
{
"type": "function",
"function": {
"name": "litellm_web_search",
"description": "Search the web for information",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"}
},
"required": ["query"]
}
}
}
]
)
# Response includes search results automatically!
print(response.choices[0].message.content)
```
## How It Works
When a model makes a web search tool call, LiteLLM:
1. **Detects** the `litellm_web_search` tool call in the response
2. **Executes** the search using your configured search provider
3. **Makes a follow-up request** with the search results
4. **Returns** the final answer to the user
```mermaid
sequenceDiagram
participant User
participant LiteLLM
participant LLM as LLM Provider
participant Search as Search Provider
User->>LiteLLM: Request with web_search tool
LiteLLM->>LLM: Forward request
LLM-->>LiteLLM: Response with tool_call
Note over LiteLLM: Detect web search<br/>tool call
LiteLLM->>Search: Execute search
Search-->>LiteLLM: Search results
LiteLLM->>LLM: Follow-up with results
LLM-->>LiteLLM: Final answer
LiteLLM-->>User: Final answer with search results
```
**Result**: One API call from user → Complete answer with search results
## Supported Providers
Web search integration works with **all providers** that use:
- ✅ **Base HTTP Handler** (`BaseLLMHTTPHandler`)
- ✅ **OpenAI Completion Handler** (`OpenAIChatCompletion`)
### Providers Using Base HTTP Handler
| Provider | Status | Notes |
|----------|--------|-------|
| **OpenAI** | ✅ Supported | GPT-4, GPT-3.5, etc. |
| **Anthropic** | ✅ Supported | Claude models via HTTP handler |
| **MiniMax** | ✅ Supported | All MiniMax models |
| **Mistral** | ✅ Supported | Mistral AI models |
| **Cohere** | ✅ Supported | Command models |
| **Fireworks AI** | ✅ Supported | All Fireworks models |
| **Together AI** | ✅ Supported | All Together AI models |
| **Groq** | ✅ Supported | All Groq models |
| **Perplexity** | ✅ Supported | Perplexity models |
| **DeepSeek** | ✅ Supported | DeepSeek models |
| **xAI** | ✅ Supported | Grok models |
| **Hugging Face** | ✅ Supported | Inference API models |
| **OCI** | ✅ Supported | Oracle Cloud models |
| **Vertex AI** | ✅ Supported | Google Vertex AI models |
| **Bedrock** | ✅ Supported | AWS Bedrock models (converse_like route) |
| **Azure OpenAI** | ✅ Supported | Azure-hosted OpenAI models |
| **Sagemaker** | ✅ Supported | AWS Sagemaker models |
| **Databricks** | ✅ Supported | Databricks models |
| **DataRobot** | ✅ Supported | DataRobot models |
| **Hosted VLLM** | ✅ Supported | Self-hosted VLLM |
| **Heroku** | ✅ Supported | Heroku-hosted models |
| **RAGFlow** | ✅ Supported | RAGFlow models |
| **Compactif** | ✅ Supported | Compactif models |
| **Cometapi** | ✅ Supported | Comet API models |
| **A2A** | ✅ Supported | Agent-to-Agent models |
| **Bytez** | ✅ Supported | Bytez models |
### Providers Using OpenAI Handler
| Provider | Status | Notes |
|----------|--------|-------|
| **OpenAI** | ✅ Supported | Native OpenAI API |
| **Azure OpenAI** | ✅ Supported | Azure-hosted OpenAI |
| **OpenAI-Compatible** | ✅ Supported | Any OpenAI-compatible API |
## Configuration
### WebSearch Interception Parameters
| Parameter | Type | Required | Description | Example |
|-----------|------|----------|-------------|---------|
| `enabled_providers` | List[String] | Yes | List of providers to enable web search for | `[openai, minimax, anthropic]` |
| `search_tool_name` | String | No | Specific search tool from `search_tools` config. If not set, uses first available. | `perplexity-search` |
### Provider Values
Use these values in `enabled_providers`:
| Provider | Value | Provider | Value |
|----------|-------|----------|-------|
| OpenAI | `openai` | Anthropic | `anthropic` |
| MiniMax | `minimax` | Mistral | `mistral` |
| Cohere | `cohere` | Fireworks AI | `fireworks_ai` |
| Together AI | `together_ai` | Groq | `groq` |
| Perplexity | `perplexity` | DeepSeek | `deepseek` |
| xAI | `xai` | Hugging Face | `huggingface` |
| OCI | `oci` | Vertex AI | `vertex_ai` |
| Bedrock | `bedrock` | Azure | `azure` |
| Sagemaker | `sagemaker_chat` | Databricks | `databricks` |
| DataRobot | `datarobot` | VLLM | `hosted_vllm` |
| Heroku | `heroku` | RAGFlow | `ragflow` |
| Compactif | `compactif` | Cometapi | `cometapi` |
| A2A | `a2a` | Bytez | `bytez` |
## Search Providers
Configure which search provider to use. LiteLLM supports multiple search providers:
| Provider | `search_provider` Value | Environment Variable |
|----------|------------------------|----------------------|
| **Perplexity AI** | `perplexity` | `PERPLEXITYAI_API_KEY` |
| **Tavily** | `tavily` | `TAVILY_API_KEY` |
| **Exa AI** | `exa_ai` | `EXA_API_KEY` |
| **Parallel AI** | `parallel_ai` | `PARALLEL_AI_API_KEY` |
| **Google PSE** | `google_pse` | `GOOGLE_PSE_API_KEY`, `GOOGLE_PSE_ENGINE_ID` |
| **DataForSEO** | `dataforseo` | `DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD` |
| **Firecrawl** | `firecrawl` | `FIRECRAWL_API_KEY` |
| **SearXNG** | `searxng` | `SEARXNG_API_BASE` (required) |
| **Linkup** | `linkup` | `LINKUP_API_KEY` |
See [Search Providers Documentation](../search/index.md) for detailed setup instructions.
## Complete Configuration Example
```yaml
model_list:
# OpenAI
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
# MiniMax
- model_name: minimax
litellm_params:
model: minimax/MiniMax-M2.1
api_key: os.environ/MINIMAX_API_KEY
# Anthropic
- model_name: claude
litellm_params:
model: anthropic/claude-sonnet-4-5
api_key: os.environ/ANTHROPIC_API_KEY
# Azure OpenAI
- model_name: azure-gpt4
litellm_params:
model: azure/gpt-4
api_base: https://my-azure.openai.azure.com
api_key: os.environ/AZURE_API_KEY
litellm_settings:
callbacks:
- websearch_interception:
enabled_providers:
- openai
- minimax
- anthropic
- azure
search_tool_name: perplexity-search
search_tools:
- search_tool_name: perplexity-search
litellm_params:
search_provider: perplexity
api_key: os.environ/PERPLEXITY_API_KEY
- search_tool_name: tavily-search
litellm_params:
search_provider: tavily
api_key: os.environ/TAVILY_API_KEY
```
## Usage Examples
### Python SDK
```python
import litellm
# Configure callbacks
litellm.callbacks = ["websearch_interception"]
# Make completion with web search tool
response = await litellm.acompletion(
model="gpt-4o",
messages=[
{"role": "user", "content": "What are the latest AI news?"}
],
tools=[
{
"type": "function",
"function": {
"name": "litellm_web_search",
"description": "Search the web for current information",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query"
}
},
"required": ["query"]
}
}
}
]
)
print(response.choices[0].message.content)
```
### Proxy Server
```bash
# Start proxy with config
litellm --config config.yaml
# Make request
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-4o",
"messages": [
{"role": "user", "content": "What is the weather in San Francisco?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "litellm_web_search",
"description": "Search the web",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"}
},
"required": ["query"]
}
}
}
]
}'
```
## How Search Tool Selection Works
1. **If `search_tool_name` is specified** → Uses that specific search tool
2. **If `search_tool_name` is not specified** → Uses first search tool in `search_tools` list
```yaml
search_tools:
- search_tool_name: perplexity-search # ← This will be used if no search_tool_name specified
litellm_params:
search_provider: perplexity
api_key: os.environ/PERPLEXITY_API_KEY
- search_tool_name: tavily-search
litellm_params:
search_provider: tavily
api_key: os.environ/TAVILY_API_KEY
```
## Troubleshooting
### Web Search Not Working
1. **Check provider is enabled**:
```yaml
enabled_providers:
- openai # Make sure your provider is in this list
```
2. **Verify search tool is configured**:
```yaml
search_tools:
- search_tool_name: perplexity-search
litellm_params:
search_provider: perplexity
api_key: os.environ/PERPLEXITY_API_KEY
```
3. **Check API keys are set**:
```bash
export PERPLEXITY_API_KEY=your-key
```
4. **Enable debug logging**:
```python
litellm.set_verbose = True
```
### Common Issues
**Issue**: Model returns tool_calls instead of final answer
- **Cause**: Provider not in `enabled_providers` list
- **Solution**: Add provider to `enabled_providers`
**Issue**: "No search tool configured" error
- **Cause**: No search tools in `search_tools` config
- **Solution**: Add at least one search tool configuration
**Issue**: "Invalid function arguments json string" error (MiniMax)
- **Cause**: Fixed in latest version - arguments weren't properly JSON serialized
- **Solution**: Update to latest LiteLLM version
## Related Documentation
- [Search Providers](../search/index.md) - Detailed search provider setup
- [Claude Code WebSearch](../tutorials/claude_code_websearch.md) - Using with Claude Code
- [Tool Calling](../completion/function_call.md) - General tool calling documentation
- [Callbacks](./custom_callback.md) - Custom callback documentation
## Technical Details
### Architecture
Web search integration is implemented as a custom callback (`WebSearchInterceptionLogger`) that:
1. **Pre-request Hook**: Converts native web search tools to LiteLLM standard format
2. **Post-response Hook**: Detects web search tool calls in responses
3. **Agentic Loop**: Executes searches and makes follow-up requests automatically
### Supported APIs
- ✅ **Chat Completions API** (OpenAI format)
- ✅ **Anthropic Messages API** (Anthropic format)
- ✅ **Streaming** (automatically converted)
- ✅ **Non-streaming**
### Response Format Detection
The handler automatically detects response format:
- **OpenAI format**: `tool_calls` in assistant message
- **Anthropic format**: `tool_use` blocks in content
### Performance
- **Latency**: Adds one additional LLM call (follow-up request with search results)
- **Caching**: Search results can be cached (depends on search provider)
- **Parallel Searches**: Multiple search queries executed in parallel
## Contributing
Found a bug or want to add support for a new provider? See our [Contributing Guide](https://github.com/BerriAI/litellm/blob/main/CONTRIBUTING.md).

13
docs/my-website/docs/mcp.md
View file

@ -506,7 +506,14 @@ Your OpenAPI specification should follow standard OpenAPI/Swagger conventions:
- **Operation IDs**: Each operation should have a unique `operationId` (this becomes the tool name)
- **Parameters**: Request parameters should be properly documented with types and descriptions
## MCP Oauth
## MCP OAuth
LiteLLM supports OAuth 2.0 for MCP servers -- both interactive (PKCE) flows for user-facing clients and machine-to-machine (M2M) `client_credentials` for backend services.
See the **[MCP OAuth guide](./mcp_oauth.md)** for setup instructions, sequence diagrams, and a test server.
<details>
<summary>Detailed OAuth reference (click to expand)</summary>
LiteLLM v 1.77.6 added support for OAuth 2.0 Client Credentials for MCP servers.
@ -588,6 +595,8 @@ sequenceDiagram
See the official [MCP Authorization Flow](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-flow-steps) for additional reference.
</details>
## Forwarding Custom Headers to MCP Servers
@ -1486,7 +1495,7 @@ async with stdio_client(server_params) as (read, write):
**Q: How do I use OAuth2 client_credentials (machine-to-machine) with MCP servers behind LiteLLM?**
At the moment LiteLLM only forwards whatever `Authorization` header/value you configure for the MCP server; it does not issue OAuth2 tokens by itself. If your MCP requires the Client Credentials grant, obtain the access token directly from the authorization server and set that bearer token as the MCP server’s Authorization header value. LiteLLM does not yet fetch or refresh those machine-to-machine tokens on your behalf, but we plan to add first-class client_credentials support in a future release so the proxy can manage those tokens automatically.
LiteLLM supports automatic token management for the `client_credentials` grant. Configure `client_id`, `client_secret`, and `token_url` on your MCP server and LiteLLM will fetch, cache, and refresh tokens automatically. See the [MCP OAuth M2M guide](./mcp_oauth.md#machine-to-machine-m2m-auth) for setup instructions.
**Q: When I fetch an OAuth token from the LiteLLM UI, where is it stored?**

244
docs/my-website/docs/mcp_oauth.md Normal file
View file

@ -0,0 +1,244 @@
# MCP OAuth
LiteLLM supports two OAuth 2.0 flows for MCP servers:
| Flow | Use Case | How It Works |
|------|----------|--------------|
| **Interactive (PKCE)** | User-facing apps (Claude Code, Cursor) | Browser-based consent, per-user tokens |
| **Machine-to-Machine (M2M)** | Backend services, CI/CD, automated agents | `client_credentials` grant, proxy-managed tokens |
## Interactive OAuth (PKCE)
For user-facing MCP clients (Claude Code, Cursor), LiteLLM supports the full OAuth 2.0 authorization code flow with PKCE.
### Setup
```yaml title="config.yaml" showLineNumbers
mcp_servers:
github_mcp:
url: "https://api.githubcopilot.com/mcp"
auth_type: oauth2
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
```
[**See Claude Code Tutorial**](./tutorials/claude_responses_api#connecting-mcp-servers)
### How It Works
```mermaid
sequenceDiagram
participant Browser as User-Agent (Browser)
participant Client as Client
participant LiteLLM as LiteLLM Proxy
participant MCP as MCP Server (Resource Server)
participant Auth as Authorization Server
Note over Client,LiteLLM: Step 1 – Resource discovery
Client->>LiteLLM: GET /.well-known/oauth-protected-resource/{mcp_server_name}/mcp
LiteLLM->>Client: Return resource metadata
Note over Client,LiteLLM: Step 2 – Authorization server discovery
Client->>LiteLLM: GET /.well-known/oauth-authorization-server/{mcp_server_name}
LiteLLM->>Client: Return authorization server metadata
Note over Client,Auth: Step 3 – Dynamic client registration
Client->>LiteLLM: POST /{mcp_server_name}/register
LiteLLM->>Auth: Forward registration request
Auth->>LiteLLM: Issue client credentials
LiteLLM->>Client: Return client credentials
Note over Client,Browser: Step 4 – User authorization (PKCE)
Client->>Browser: Open authorization URL + code_challenge + resource
Browser->>Auth: Authorization request
Note over Auth: User authorizes
Auth->>Browser: Redirect with authorization code
Browser->>LiteLLM: Callback to LiteLLM with code
LiteLLM->>Browser: Redirect back with authorization code
Browser->>Client: Callback with authorization code
Note over Client,Auth: Step 5 – Token exchange
Client->>LiteLLM: Token request + code_verifier + resource
LiteLLM->>Auth: Forward token request
Auth->>LiteLLM: Access (and refresh) token
LiteLLM->>Client: Return tokens
Note over Client,MCP: Step 6 – Authenticated MCP call
Client->>LiteLLM: MCP request with access token + LiteLLM API key
LiteLLM->>MCP: MCP request with Bearer token
MCP-->>LiteLLM: MCP response
LiteLLM-->>Client: Return MCP response
```
**Participants**
- **Client** -- The MCP-capable AI agent (e.g., Claude Code, Cursor, or another IDE/agent) that initiates OAuth discovery, authorization, and tool invocations on behalf of the user.
- **LiteLLM Proxy** -- Mediates all OAuth discovery, registration, token exchange, and MCP traffic while protecting stored credentials.
- **Authorization Server** -- Issues OAuth 2.0 tokens via dynamic client registration, PKCE authorization, and token endpoints.
- **MCP Server (Resource Server)** -- The protected MCP endpoint that receives LiteLLM's authenticated JSON-RPC requests.
- **User-Agent (Browser)** -- Temporarily involved so the end user can grant consent during the authorization step.
**Flow Steps**
1. **Resource Discovery**: The client fetches MCP resource metadata from LiteLLM's `.well-known/oauth-protected-resource` endpoint to understand scopes and capabilities.
2. **Authorization Server Discovery**: The client retrieves the OAuth server metadata (token endpoint, authorization endpoint, supported PKCE methods) through LiteLLM's `.well-known/oauth-authorization-server` endpoint.
3. **Dynamic Client Registration**: The client registers through LiteLLM, which forwards the request to the authorization server (RFC 7591). If the provider doesn't support dynamic registration, you can pre-store `client_id`/`client_secret` in LiteLLM (e.g., GitHub MCP) and the flow proceeds the same way.
4. **User Authorization**: The client launches a browser session (with code challenge and resource hints). The user approves access, the authorization server sends the code through LiteLLM back to the client.
5. **Token Exchange**: The client calls LiteLLM with the authorization code, code verifier, and resource. LiteLLM exchanges them with the authorization server and returns the issued access/refresh tokens.
6. **MCP Invocation**: With a valid token, the client sends the MCP JSON-RPC request (plus LiteLLM API key) to LiteLLM, which forwards it to the MCP server and relays the tool response.
See the official [MCP Authorization Flow](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-flow-steps) for additional reference.
## Machine-to-Machine (M2M) Auth
LiteLLM automatically fetches, caches, and refreshes OAuth2 tokens using the `client_credentials` grant. No manual token management required.
### Setup
You can configure M2M OAuth via the LiteLLM UI or `config.yaml`.
### UI Setup
Navigate to the **MCP Servers** page and click **+ Add New MCP Server**.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/d1f1e89c-a789-4975-8846-b15d9821984a/ascreenshot_630800e00a2e4b598baabfc25efbabd3_text_export.jpeg)
Enter a name for your server and select **HTTP** as the transport type.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/2008c9d6-6093-4121-beab-1e52c71376aa/ascreenshot_516ffd6c7b524465a253a56048c3d228_text_export.jpeg)
Paste the MCP server URL.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/b0ee8b7d-6de8-492b-8962-287987feec29/ascreenshot_b3efca82078a4c6bb1453c58161909f9_text_export.jpeg)
Under **Authentication**, select **OAuth**.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/e1597814-ff8e-40b9-9d7b-864dcdbe0910/ascreenshot_2097612712264d8f9e553f7ca9175fb0_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/f6ea5694-f28a-4bc3-9c9a-bb79f199bd65/ascreenshot_9be839f55b1b4f96bfe24030ba2c7f8d_text_export.jpeg)
Choose **Machine-to-Machine (M2M)** as the OAuth flow type. This is for server-to-server authentication using the `client_credentials` grant — no browser interaction required.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/9853310c-1d86-4628-bad1-7a391eca0e4d/ascreenshot_f302a286fa264fdd8d56db53b8f9395c_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/df64dc65-ef86-475d-adaf-12e227d5e873/ascreenshot_9e2f41d43a76435f918a00b52ffcc639_text_export.jpeg)
Fill in the **Client ID** and **Client Secret** provided by your OAuth provider.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/0de5a7bd-9898-4fc7-8843-b23dd5aac47f/ascreenshot_b9087aaa81a14b5b9c199929efc4a563_text_export.jpeg)
Enter the **Token URL** — this is the endpoint LiteLLM will call to fetch access tokens using `client_credentials`.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/0aea70f1-558c-4dca-91bc-1175fe1ddc89/ascreenshot_b3fcf8a1287e4e2d9a3d67c4a29f7bff_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/e842ef09-1fd7-47a6-909b-252d389f0abc/ascreenshot_2a87dad3624847e7ac370591d1d1aedd_text_export.jpeg)
Scroll down and review the server URL and all fields, then click **Create MCP Server**.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/0857712b-4b53-40f8-8c1f-a4c72edaa644/ascreenshot_47be3fcd5de64ed391f70c1fb74a8bfc_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/9d961765-955f-4905-a3dc-1a446aa3b2cc/ascreenshot_43fd39d014224564bc6b35aced1fb6d3_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/3825d5fa-8fd1-4e71-b090-77ff0259c3f6/ascreenshot_2509a7ebd9bf421eb0e82f2553566745_text_export.jpeg)
Once created, open the server and navigate to the **MCP Tools** tab to verify that LiteLLM can connect and list available tools.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/8107e27b-5072-4675-8fd6-89b47692b1bd/ascreenshot_f774bc76138f430d808fb4482ebfcdca_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/ce94bb7b-c81b-4396-9939-178efb2cdfce/ascreenshot_28b838ab6ae34c76858454555c4c1d79_text_export.jpeg)
Select a tool (e.g. **echo**) to test it. Fill in the required parameters and click **Call Tool**.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/c459c1d3-ec29-4211-9c28-37fbe7783bbc/ascreenshot_e9b138b3c2cc4440bb1a6f42ac7ae861_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/5438ac60-e0ac-4a79-bf6f-5594f160d3b5/ascreenshot_9133a17d26204c46bce497e74685c483_text_export.jpeg)
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/a8f6821b-3982-4b4d-9b25-70c8aff5ac31/ascreenshot_28d474d0e62545a482cff6128527883a_text_export.jpeg)
LiteLLM automatically fetches an OAuth token behind the scenes and calls the tool. The result confirms the M2M OAuth flow is working end-to-end.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-10/c3924549-a949-48d1-ac67-ab4c30475859/ascreenshot_8f6eca9d717f45478d50a881bd244bb3_text_export.jpeg)
### Config.yaml Setup
```yaml title="config.yaml" showLineNumbers
mcp_servers:
my_mcp_server:
url: "https://my-mcp-server.com/mcp"
auth_type: oauth2
client_id: os.environ/MCP_CLIENT_ID
client_secret: os.environ/MCP_CLIENT_SECRET
token_url: "https://auth.example.com/oauth/token"
scopes: ["mcp:read", "mcp:write"] # optional
```
### How It Works
1. On first MCP request, LiteLLM POSTs to `token_url` with `grant_type=client_credentials`
2. The access token is cached in-memory with TTL = `expires_in - 60s`
3. Subsequent requests reuse the cached token
4. When the token expires, LiteLLM fetches a new one automatically
```mermaid
sequenceDiagram
participant Client as Client
participant LiteLLM as LiteLLM Proxy
participant Auth as Authorization Server
participant MCP as MCP Server
Client->>LiteLLM: MCP request + LiteLLM API key
LiteLLM->>Auth: POST /oauth/token (client_credentials)
Auth->>LiteLLM: access_token (expires_in: 3600)
LiteLLM->>MCP: MCP request + Bearer token
MCP-->>LiteLLM: MCP response
LiteLLM-->>Client: MCP response
Note over LiteLLM: Token cached for subsequent requests
Client->>LiteLLM: Next MCP request
LiteLLM->>MCP: MCP request + cached Bearer token
MCP-->>LiteLLM: MCP response
LiteLLM-->>Client: MCP response
```
### Test with Mock Server
Use [BerriAI/mock-oauth2-mcp-server](https://github.com/BerriAI/mock-oauth2-mcp-server) to test locally:
```bash title="Terminal 1 - Start mock server" showLineNumbers
pip install fastapi uvicorn
python mock_oauth2_mcp_server.py # starts on :8765
```
```yaml title="config.yaml" showLineNumbers
mcp_servers:
test_oauth2:
url: "http://localhost:8765/mcp"
auth_type: oauth2
client_id: "test-client"
client_secret: "test-secret"
token_url: "http://localhost:8765/oauth/token"
```
```bash title="Terminal 2 - Start proxy and test" showLineNumbers
litellm --config config.yaml --port 4000
# List tools
curl http://localhost:4000/mcp-rest/tools/list \
-H "Authorization: Bearer sk-1234"
# Call a tool
curl http://localhost:4000/mcp-rest/tools/call \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{"name": "echo", "arguments": {"message": "hello"}}'
```
### Config Reference
| Field | Required | Description |
|-------|----------|-------------|
| `auth_type` | Yes | Must be `oauth2` |
| `client_id` | Yes | OAuth2 client ID. Supports `os.environ/VAR_NAME` |
| `client_secret` | Yes | OAuth2 client secret. Supports `os.environ/VAR_NAME` |
| `token_url` | Yes | Token endpoint URL |
| `scopes` | No | List of scopes to request |

251
docs/my-website/docs/mcp_public_internet.md Normal file
View file

@ -0,0 +1,251 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Exposing MCPs on the Public Internet
Control which MCP servers are visible to external callers (e.g., ChatGPT, Claude Desktop) vs. internal-only callers. This is useful when you want a subset of your MCP servers available publicly while keeping sensitive servers restricted to your private network.
## Overview
| Property | Details |
|-------|-------|
| Description | IP-based access control for MCP servers — external callers only see servers marked as public |
| Setting | `available_on_public_internet` on each MCP server |
| Network Config | `mcp_internal_ip_ranges` in `general_settings` |
| Supported Clients | ChatGPT, Claude Desktop, Cursor, OpenAI API, or any MCP client |
## How It Works
When a request arrives at LiteLLM's MCP endpoints, LiteLLM checks the caller's IP address to determine whether they are an **internal** or **external** caller:
1. **Extract the client IP** from the incoming request (supports `X-Forwarded-For` when configured behind a reverse proxy).
2. **Classify the IP** as internal or external by checking it against the configured private IP ranges (defaults to RFC 1918: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`).
3. **Filter the server list**:
- **Internal callers** see all MCP servers (public and private).
- **External callers** only see servers with `available_on_public_internet: true`.
This filtering is applied at every MCP access point: the MCP registry, tool listing, tool calling, dynamic server routes, and OAuth discovery endpoints.
```mermaid
flowchart TD
A[Incoming MCP Request] --> B[Extract Client IP Address]
B --> C{Is IP in private ranges?}
C -->|Yes - Internal caller| D[Return ALL MCP servers]
C -->|No - External caller| E[Return ONLY servers with<br/>available_on_public_internet = true]
```
## Walkthrough
This walkthrough covers two flows:
1. **Adding a public MCP server** (DeepWiki) and connecting to it from ChatGPT
2. **Making an existing server private** (Exa) and verifying ChatGPT no longer sees it
### Flow 1: Add a Public MCP Server (DeepWiki)
DeepWiki is a free MCP server — a good candidate to expose publicly so AI gateway users can access it from ChatGPT.
#### Step 1: Create the MCP Server
Navigate to the MCP Servers page and click **"+ Add New MCP Server"**.
![Click Add New MCP Server](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/28cc27c2-d980-4255-b552-ebf542ef95be/ascreenshot_30a7e3c043834f1c87b69e6ffc5bba4f_text_export.jpeg)
The create dialog opens. Enter **"DeepWiki"** as the server name.
![Enter server name](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/8c733c38-310a-40ef-8a5b-7af91cc7f74f/ascreenshot_16df83fed5bd4683a22a042e07063cec_text_export.jpeg)
For the transport type dropdown, select **HTTP** since DeepWiki uses the Streamable HTTP transport.
![Select transport type](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/e473f603-d692-40c7-a218-866c2e1cb554/ascreenshot_e93997971f2f44beac6152786889addf_text_export.jpeg)
Now scroll down to the MCP Server URL field.
![Configure server](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/b08d3c1f-9279-45b6-8efb-f73008901da6/ascreenshot_ce0de66f230a41b0a454e76653429021_text_export.jpeg)
Enter the DeepWiki MCP URL: `https://mcp.deepwiki.com/mcp`.
![Enter MCP server URL](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/e59f8285-cfde-4c57-aa79-24244acc9160/ascreenshot_8d575c66dc614a4183212ba282d22b41_text_export.jpeg)
With the name, transport, and URL filled in, the basic server configuration is complete.
![Server URL configured](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/0f1af7ed-760d-4445-bdec-3da706d4eef4/ascreenshot_d7d6db69bc254ded871d14a71188a212_text_export.jpeg)
#### Step 2: Enable "Available on Public Internet"
Before creating, scroll down and expand the **Permission Management / Access Control** section. This is where you control who can see this server.
![Expand Permission Management](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/cc10dea2-6028-4a27-a33b-1b1b7212efb5/ascreenshot_0fdd152b862a4bf39973bc805ce64c57_text_export.jpeg)
Toggle **"Available on Public Internet"** on. This is the key setting — it tells LiteLLM that external callers (like ChatGPT connecting from the public internet) should be able to discover and use this server.
![Toggle Available on Public Internet](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/39c14543-c5ae-4189-8f85-9efc87135820/ascreenshot_9991f54910c24e21bba5c05ea4fa8e28_text_export.jpeg)
With the toggle enabled, click **"Create"** to save the server.
![Click Create](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/843be209-aade-44f4-98da-e55d1644854c/ascreenshot_8cfc90345a5f4d069b397e80d0a6e449_text_export.jpeg)
#### Step 3: Connect from ChatGPT
Now let's verify it works. Open ChatGPT and look for the MCP server icon to add a new connection. The endpoint to use is `<your-litellm-url>/mcp`.
![ChatGPT add MCP server](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/58b5f674-edf4-4156-a5fa-5fdc8ed5d7b9/ascreenshot_36735f7c37394e919793968794614126_text_export.jpeg)
In the dropdown, select **"Add an MCP server"** to configure a new connection.
![ChatGPT MCP server option](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/f89da8af-bc61-44a7-a765-f52733f4970d/ascreenshot_6410a917b782437eb558de3bfcd35ffd_text_export.jpeg)
ChatGPT asks for a server label. Give it a recognizable name like "LiteLLM".
![Enter server label](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/88505afe-07c1-4674-a89c-8035a5d05eb6/ascreenshot_143aefc38ddd4d3f9f5823ca2cc09bc2_text_export.jpeg)
Next, enter the Server URL. This should be your LiteLLM proxy's MCP endpoint — `<your-litellm-url>/mcp`.
![Enter LiteLLM MCP URL](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/9048be4a-7e40-43e7-9789-059fed2741a6/ascreenshot_e81232c17fd148f48f0ae552e9dc2a10_text_export.jpeg)
Paste your LiteLLM URL and confirm it looks correct.
![URL pasted](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/7707e796-e146-47c8-bce0-58e6f4076272/ascreenshot_0710dc58b8ed4d6887856b1388d59329_text_export.jpeg)
ChatGPT also needs authentication. Enter your LiteLLM API key in the authentication field so it can connect to the proxy.
![Enter API key](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/f6cfcb81-021d-4a41-94d7-d4eaf449d025/ascreenshot_d635865abfb64732a7278922f08dbcaa_text_export.jpeg)
Click **"Connect"** to establish the connection.
![Click Connect](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/1146b326-6f0c-4050-9729-af5c88e1bc81/ascreenshot_e19fb857e5394b9a9bf77b075b4fb620_text_export.jpeg)
ChatGPT connects and shows the available tools. Since both DeepWiki and Exa are currently marked as public, ChatGPT can see tools from both servers.
![ChatGPT shows available MCP tools](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/43ac56b7-9933-4762-903a-370fc52c79b5/ascreenshot_39073d6dc3bc4bb6a79d93365a26a4f8_text_export.jpeg)
---
### Flow 2: Make an Existing Server Private (Exa)
Now let's do the reverse — take an existing MCP server (Exa) that's currently public and restrict it to internal access only. After this change, ChatGPT should no longer see Exa's tools.
#### Step 1: Edit the Server
Go to the MCP Servers table and click on the Exa server to open its detail view.
![Exa server overview](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/65844f13-b1ec-4092-b3fd-b1cae3c0c833/ascreenshot_cc8ea435c5e14761a1394ca80fe817c0_text_export.jpeg)
Switch to the **"Settings"** tab to access the edit form.
![Click Settings](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/d5b65271-561e-4d2a-b832-96d32611f6e4/ascreenshot_a200942b17264c1eb7a3ffdb2c2141f5_text_export.jpeg)
The edit form loads with Exa's current configuration.
![Edit server](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/119184f6-f3cd-45b7-9cfa-0ea08de27020/ascreenshot_c39a793da03a4f0fb84b5ee829af9034_text_export.jpeg)
#### Step 2: Toggle Off "Available on Public Internet"
Scroll down and expand the **Permission Management / Access Control** section to find the public internet toggle.
![Expand permissions](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/bf7114cc-8741-4fa0-a39a-fe625482e88a/ascreenshot_8a987649c03e46558a2ec9a6f2f539a4_text_export.jpeg)
Toggle **"Available on Public Internet"** off. This will hide Exa from any caller outside your private network.
![Toggle off public internet](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/f36af5ad-028f-4bb1-aed1-43e38ff9b733/ascreenshot_9128364a049f489bb8483e18e5c88015_text_export.jpeg)
Click **"Save Changes"** to apply. The change takes effect immediately — no proxy restart needed.
![Save changes](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/126a71b3-02e1-4d61-a208-942b92e9ef25/ascreenshot_f349ef69e08044dd8e4903f4286b7b97_text_export.jpeg)
#### Step 3: Verify in ChatGPT
Go back to ChatGPT to confirm Exa is no longer visible. You'll need to reconnect for ChatGPT to re-fetch the tool list.
![ChatGPT verify](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/15518882-8b19-44d3-9bba-245aeb62b4b1/ascreenshot_f98f59c51e6543e1be4f3960ba375fc9_text_export.jpeg)
Open the MCP server settings and select to add or reconnect a server.
![Reconnect to server](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/784d3174-77c0-42e6-a059-4c906db8f72a/ascreenshot_d77db951b83e4b15a00373222712f6b5_text_export.jpeg)
Enter the same LiteLLM MCP URL as before.
![Reconnect URL](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/17ef5fb0-b240-4556-8d20-753d359b7fcf/ascreenshot_583466ce9e8f40d1ba0af8b1e7d04413_text_export.jpeg)
Set the server label.
![Reconnect name](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/d7907637-c957-4a3c-ab4f-1600ca9a70a0/ascreenshot_e429eea43f3f4b3ca4d3ac5a77fbde2d_text_export.jpeg)
Enter your API key for authentication.
![Reconnect key](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/9cfff77a-37aa-4ca6-8032-0b46c50f37e3/ascreenshot_250664183399496b8f5c9f86f576fc0b_text_export.jpeg)
Click **"Connect"** to re-establish the connection.
![Click Connect](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/686f6307-b4ae-448b-ac6c-2c9d7b4f6b57/ascreenshot_3f499d0812af42ab89fed103cc21c249_text_export.jpeg)
This time, only DeepWiki's tools appear — Exa is gone. LiteLLM detected that ChatGPT is calling from a public IP and filtered out Exa since it's no longer marked as public. Internal users on your private network would still see both servers.
![Only DeepWiki tools visible](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/667d79b6-75f9-4799-9315-0c176e7a5e34/ascreenshot_efa43050ac0b4445a09e542fa8f270ff_text_export.jpeg)
## Configuration Reference
### Per-Server Setting
<Tabs>
<TabItem value="ui" label="UI">
Toggle **"Available on Public Internet"** in the Permission Management section when creating or editing an MCP server.
</TabItem>
<TabItem value="config" label="config.yaml">
```yaml title="config.yaml" showLineNumbers
mcp_servers:
deepwiki:
url: https://mcp.deepwiki.com/mcp
available_on_public_internet: true # visible to external callers
exa:
url: https://exa.ai/mcp
auth_type: api_key
auth_value: os.environ/EXA_API_KEY
available_on_public_internet: false # internal only (default)
```
</TabItem>
<TabItem value="api" label="API">
```bash title="Create a public MCP server" showLineNumbers
curl -X POST <your-litellm-url>/v1/mcp/server \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"server_name": "DeepWiki",
"url": "https://mcp.deepwiki.com/mcp",
"transport": "http",
"available_on_public_internet": true
}'
```
```bash title="Update an existing server" showLineNumbers
curl -X PUT <your-litellm-url>/v1/mcp/server \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"server_id": "<server-id>",
"available_on_public_internet": false
}'
```
</TabItem>
</Tabs>
### Custom Private IP Ranges
By default, LiteLLM treats RFC 1918 private ranges as internal. You can customize this in the **Network Settings** tab under MCP Servers, or via config:
```yaml title="config.yaml" showLineNumbers
general_settings:
mcp_internal_ip_ranges:
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- "100.64.0.0/10" # Add your VPN/Tailscale range
```
When empty, the standard private ranges are used (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`).

144
docs/my-website/docs/pass_through/bedrock.md
View file

@ -556,3 +556,147 @@ for event in response.get("completion"):
print(completion)
```
## Using LangChain AWS SDK with LiteLLM
You can use the [LangChain AWS SDK](https://python.langchain.com/docs/integrations/chat/bedrock/) with LiteLLM Proxy to get cost tracking, load balancing, and other LiteLLM features.
### Quick Start
**1. Install LangChain AWS**:
```bash showLineNumbers
pip install langchain-aws
```
**2. Setup LiteLLM Proxy**:
Create a `config.yaml`:
```yaml showLineNumbers
model_list:
- model_name: claude-sonnet
litellm_params:
model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0
aws_region_name: us-east-1
custom_llm_provider: bedrock
```
Start the proxy:
```bash showLineNumbers
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000
```
**3. Use LangChain with LiteLLM**:
```python showLineNumbers
from langchain_aws import ChatBedrockConverse
from langchain_core.messages import HumanMessage
# Your LiteLLM API key
API_KEY = "Bearer sk-1234"
# Initialize ChatBedrockConverse pointing to LiteLLM proxy
llm = ChatBedrockConverse(
model_id="us.anthropic.claude-3-7-sonnet-20250219-v1:0",
endpoint_url="http://localhost:4000/bedrock",
region_name="us-east-1",
aws_access_key_id=API_KEY,
aws_secret_access_key="bedrock" # Any non-empty value works
)
# Invoke the model
messages = [HumanMessage(content="Hello, how are you?")]
response = llm.invoke(messages)
print(response.content)
```
### Advanced Example: PDF Document Processing with Citations
LangChain AWS SDK supports Bedrock's document processing features. Here's how to use it with LiteLLM:
```python showLineNumbers
import os
import json
from langchain_aws import ChatBedrockConverse
from langchain_core.messages import HumanMessage
# Your LiteLLM API key
API_KEY = "Bearer sk-1234"
def get_llm() -> ChatBedrockConverse:
"""Initialize LLM pointing to LiteLLM proxy"""
llm = ChatBedrockConverse(
model_id="us.anthropic.claude-3-7-sonnet-20250219-v1:0",
base_model_id="anthropic.claude-3-7-sonnet-20250219-v1:0",
endpoint_url="http://localhost:4000/bedrock",
region_name="us-east-1",
aws_access_key_id=API_KEY,
aws_secret_access_key="bedrock"
)
return llm
if __name__ == "__main__":
# Initialize the LLM
llm = get_llm()
# Read PDF file as bytes (Converse API requires raw bytes)
with open("your-document.pdf", "rb") as file:
file_bytes = file.read()
# Prepare messages with document attachment
messages = [
HumanMessage(content=[
{"text": "What is the policy number in this document?"},
{
"document": {
"format": "pdf",
"name": "PolicyDocument",
"source": {"bytes": file_bytes},
"citations": {"enabled": True}
}
}
])
]
# Invoke the LLM
response = llm.invoke(messages)
# Print response with citations
print(json.dumps(response.content, indent=4))
```
### Supported LangChain Features
All LangChain AWS features work with LiteLLM:
| Feature | Supported | Notes |
|---------|-----------|-------|
| Text Generation | ✅ | Full support |
| Streaming | ✅ | Use `stream()` method |
| Document Processing | ✅ | PDF, images, etc. |
| Citations | ✅ | Enable in document config |
| Tool Use | ✅ | Function calling support |
| Multi-modal | ✅ | Text + images + documents |
### Troubleshooting
**Issue**: `UnknownOperationException` error
**Solution**: Make sure you're using the correct endpoint URL format:
- ✅ Correct: `http://localhost:4000/bedrock`
- ❌ Wrong: `http://localhost:4000/bedrock/v2`
**Issue**: Authentication errors
**Solution**: Ensure your API key is in the correct format:
```python
aws_access_key_id="Bearer sk-1234" # Include "Bearer " prefix
```

22
docs/my-website/docs/providers/litellm_proxy.md
View file

@ -227,6 +227,28 @@ response = litellm.completion(
)
```
## OAuth2/JWT Authentication
If your LiteLLM Proxy requires OAuth2/JWT authentication (e.g., Azure AD, Keycloak, Okta), the SDK can automatically obtain and refresh tokens for you.
```python
import litellm
from litellm.proxy_auth import AzureADCredential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=AzureADCredential(),
scope="api://my-litellm-proxy/.default"
)
litellm.api_base = "https://my-proxy.example.com"
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}]
)
```
[Learn more about SDK Proxy Authentication (OAuth2/JWT Auto-Refresh) →](../proxy_auth)
## Sending `tags` to LiteLLM Proxy
Tags allow you to categorize and track your API requests for monitoring, debugging, and analytics purposes. You can send tags as a list of strings to the LiteLLM Proxy using the `extra_body` parameter.

35
docs/my-website/docs/proxy/admin_ui_sso.md
View file

@ -223,6 +223,7 @@ GENERIC_USER_FIRST_NAME_ATTRIBUTE = "first_name"
GENERIC_USER_LAST_NAME_ATTRIBUTE = "last_name"
GENERIC_USER_ROLE_ATTRIBUTE = "given_role"
GENERIC_USER_PROVIDER_ATTRIBUTE = "provider"
GENERIC_USER_EXTRA_ATTRIBUTES = "department,employee_id,manager" # comma-separated list of additional fields to extract from SSO response
GENERIC_CLIENT_STATE = "some-state" # if the provider needs a state parameter
GENERIC_INCLUDE_CLIENT_ID = "false" # some providers enforce that the client_id is not in the body
GENERIC_SCOPE = "openid profile email" # default scope openid is sometimes not enough to retrieve basic user info like first_name and last_name located in profile scope
@ -239,6 +240,40 @@ Use `GENERIC_USER_ROLE_ATTRIBUTE` to specify which attribute in the SSO token co
Nested attribute paths are supported (e.g., `claims.role` or `attributes.litellm_role`).
**Capturing Additional SSO Fields**
Use `GENERIC_USER_EXTRA_ATTRIBUTES` to extract additional fields from the SSO provider response beyond the standard user attributes (id, email, name, etc.). This is useful when you need to access custom organization-specific data (e.g., department, employee ID, groups) in your [custom SSO handler](./custom_sso.md).
```shell
# Comma-separated list of field names to extract
GENERIC_USER_EXTRA_ATTRIBUTES="department,employee_id,manager,groups"
```
**Accessing Extra Fields in Custom SSO Handler:**
```python
from litellm.proxy.management_endpoints.types import CustomOpenID
async def custom_sso_handler(userIDPInfo: CustomOpenID):
# Access the extra fields
extra_fields = getattr(userIDPInfo, 'extra_fields', None) or {}
user_department = extra_fields.get("department")
employee_id = extra_fields.get("employee_id")
user_groups = extra_fields.get("groups", [])
# Use these fields for custom logic (e.g., team assignment, access control)
# ...
```
**Nested Field Paths:**
Dot notation is supported for nested fields:
```shell
GENERIC_USER_EXTRA_ATTRIBUTES="org_info.department,org_info.cost_center,metadata.employee_type"
```
- Set Redirect URI, if your provider requires it
- Set a redirect url = `<your proxy base url>/sso/callback`
```shell

1
docs/my-website/docs/proxy/config_settings.md
View file

@ -640,6 +640,7 @@ router_settings:
| GENERIC_TOKEN_ENDPOINT | Token endpoint for generic OAuth providers
| GENERIC_USER_DISPLAY_NAME_ATTRIBUTE | Attribute for user's display name in generic auth
| GENERIC_USER_EMAIL_ATTRIBUTE | Attribute for user's email in generic auth
| GENERIC_USER_EXTRA_ATTRIBUTES | Comma-separated list of additional fields to extract from generic SSO provider response (e.g., "department,employee_id,groups"). Accessible via `CustomOpenID.extra_fields` in custom SSO handlers. Supports dot notation for nested fields
| GENERIC_USER_FIRST_NAME_ATTRIBUTE | Attribute for user's first name in generic auth
| GENERIC_USER_ID_ATTRIBUTE | Attribute for user ID in generic auth
| GENERIC_USER_LAST_NAME_ATTRIBUTE | Attribute for user's last name in generic auth

12
docs/my-website/docs/proxy/custom_sso.md
View file

@ -142,6 +142,18 @@ async def custom_sso_handler(userIDPInfo: OpenID) -> SSOUserDefinedValues:
f"No ID found for user. userIDPInfo.id is None {userIDPInfo}"
)
#################################################
# Access extra fields from SSO provider (requires GENERIC_USER_EXTRA_ATTRIBUTES env var)
# Example: Set GENERIC_USER_EXTRA_ATTRIBUTES="department,employee_id,groups"
extra_fields = getattr(userIDPInfo, 'extra_fields', None) or {}
user_department = extra_fields.get("department")
employee_id = extra_fields.get("employee_id")
user_groups = extra_fields.get("groups", [])
print(f"User department: {user_department}") # noqa
print(f"Employee ID: {employee_id}") # noqa
print(f"User groups: {user_groups}") # noqa
#################################################
#################################################
# Run your custom code / logic here

46
docs/my-website/docs/proxy/forward_client_headers.md
View file

@ -6,6 +6,52 @@ Control which model groups can forward client headers to the underlying LLM prov
By default, LiteLLM does not forward client headers to LLM provider APIs for security reasons. However, you can selectively enable header forwarding for specific model groups using the `forward_client_headers_to_llm_api` setting.
## How it Works
LiteLLM does **not** forward all client headers to the LLM provider. Instead, it uses an **allowlist** approach — only headers matching specific rules are forwarded. This ensures sensitive headers (like your LiteLLM API key) are never accidentally sent to upstream providers.
```mermaid
sequenceDiagram
participant Client as Client (SDK / curl)
participant Proxy as LiteLLM Proxy
participant Filter as Header Filter (Allowlist)
participant LLM as LLM Provider (OpenAI, Anthropic, etc.)
Client->>Proxy: Request with all headers<br/>(Authorization, x-trace-id,<br/>x-custom-header, anthropic-beta, etc.)
Proxy->>Filter: Check forward_client_headers_to_llm_api<br/>setting for this model group
Note over Filter: Allowlist rules:<br/>1. Headers starting with "x-" ✅<br/>2. "anthropic-beta" ✅<br/>3. "x-stainless-*" ❌ (blocked)<br/>4. All other headers ❌ (blocked)
Filter-->>Proxy: Return only allowed headers
Proxy->>LLM: Request with filtered headers<br/>(x-trace-id, x-custom-header,<br/>anthropic-beta)
LLM-->>Proxy: Response
Proxy-->>Client: Response
```
### Header Allowlist Rules
The following rules determine which headers are forwarded (see [`_get_forwardable_headers`](https://github.com/litellm/litellm/blob/main/litellm/proxy/litellm_pre_call_utils.py) in `litellm/proxy/litellm_pre_call_utils.py`):
| Rule | Example | Forwarded? |
|---|---|---|
| Headers starting with `x-` | `x-trace-id`, `x-custom-header`, `x-request-source` | ✅ Yes |
| `anthropic-beta` header | `anthropic-beta: prompt-caching-2024-07-31` | ✅ Yes |
| Headers starting with `x-stainless-*` | `x-stainless-lang`, `x-stainless-arch` | ❌ No (causes OpenAI SDK issues) |
| Standard HTTP headers | `Authorization`, `Content-Type`, `Host` | ❌ No |
| Other provider headers | `Accept`, `User-Agent` | ❌ No |
### Additional Header Mechanisms
| Mechanism | Description | Reference |
|---|---|---|
| **`x-pass-` prefix** | Headers prefixed with `x-pass-` are always forwarded with the prefix stripped, regardless of settings. E.g., `x-pass-anthropic-beta: value` → `anthropic-beta: value`. Works for all pass-through endpoints. | [Source code](https://github.com/litellm/litellm/blob/main/litellm/passthrough/utils.py) |
| **`openai-organization`** | Forwarded only when `forward_openai_org_id: true` is set in `general_settings`. | [Forward OpenAI Org ID](#enable-globally) |
| **User information headers** | When `add_user_information_to_llm_headers: true`, LiteLLM adds `x-litellm-user-id`, `x-litellm-org-id`, etc. | [User Information Headers](#user-information-headers-optional) |
| **Vertex AI pass-through** | Uses a separate, stricter allowlist: only `anthropic-beta` and `content-type`. | [Source code](https://github.com/litellm/litellm/blob/main/litellm/constants.py) |
## Configuration
## Enable Globally

58
docs/my-website/docs/proxy/guardrails/custom_code_guardrail.md
View file

@ -61,15 +61,23 @@ curl -X POST http://localhost:4000/chat/completions \
### Function Signature
Your code must define an `apply_guardrail` function:
Your code must define an `apply_guardrail` function. It can be either sync or async:
```python
# Sync version
def apply_guardrail(inputs, request_data, input_type):
# inputs: see table below
# request_data: {"model": "...", "user_id": "...", "team_id": "...", "metadata": {...}}
# input_type: "request" or "response"
return allow() # or block() or modify()
# Async version (recommended when using HTTP primitives)
async def apply_guardrail(inputs, request_data, input_type):
response = await http_post("https://api.example.com/check", body={"text": inputs["texts"][0]})
if response["success"] and response["body"].get("flagged"):
return block("Content flagged")
return allow()
```
### `inputs` Parameter
@ -145,6 +153,29 @@ def apply_guardrail(inputs, request_data, input_type):
| `char_count(text)` | Count characters |
| `lower(text)` / `upper(text)` / `trim(text)` | String transforms |
### HTTP Requests (Async)
Make async HTTP requests to external APIs for additional validation or content moderation.
| Function | Description |
|----------|-------------|
| `await http_request(url, method, headers, body, timeout)` | General async HTTP request |
| `await http_get(url, headers, timeout)` | Async GET request |
| `await http_post(url, body, headers, timeout)` | Async POST request |
**Response format:**
```python
{
"status_code": 200, # HTTP status code
"body": {...}, # Response body (parsed JSON or string)
"headers": {...}, # Response headers
"success": True, # True if status code is 2xx
"error": None # Error message if request failed
}
```
**Note:** When using HTTP primitives, define your function as `async def apply_guardrail(...)` for non-blocking execution.
## Examples
### Block PII (SSN)
@ -213,6 +244,29 @@ def apply_guardrail(inputs, request_data, input_type):
return allow()
```
### Call External Moderation API (Async)
```python
async def apply_guardrail(inputs, request_data, input_type):
# Call an external moderation API
for text in inputs["texts"]:
response = await http_post(
"https://api.example.com/moderate",
body={"text": text, "user_id": request_data["user_id"]},
headers={"Authorization": "Bearer YOUR_API_KEY"},
timeout=10
)
if not response["success"]:
# API call failed - decide whether to allow or block
return allow()
if response["body"].get("flagged"):
return block(response["body"].get("reason", "Content flagged"))
return allow()
```
### Combine Multiple Checks
```python
@ -241,8 +295,8 @@ Custom code runs in a restricted environment:
- ❌ No `import` statements
- ❌ No file I/O
- ❌ No network access
- ❌ No `exec()` or `eval()`
- ✅ HTTP requests via built-in `http_request`, `http_get`, `http_post` primitives
- ✅ Only LiteLLM-provided primitives available
## Per-Request Usage

28
docs/my-website/docs/proxy/guardrails/zscaler_ai_guard.md
View file

@ -100,7 +100,7 @@ In cases where encounter other errors when apply Zscaler AI Guard, return exampl
}
}
```
## 6. Sending User Information to Zscaler AI Guard for Analysis (Optional)
## 6. Sending User Information to Zscaler AI Guard (Optional)
If you need to send end-user information to Zscaler AI Guard for analysis, you can set the configuration in the environment variables to True and include the relevant information in custom_headers on Zscaler AI Guard.
- To send user_api_key_alias:
@ -133,4 +133,30 @@ curl -i http://localhost:8165/v1/chat/completions \
"zguard_policy_id": <the custom policy id>
}
}'
```
## 8. Set Custom Zscaler AI Guard Policy on Litellm Team OR Key Metadata (Optional)
In addition to setting `zguard_policy_id` in a request or the configuration file, you can also set it in the metadata for LiteLLM Team or Key. The `zguard_policy_id` is determined using the following order of precedence: request, Key, Team, config file. This logic is illustrated below:
```
user_api_key_metadata = metadata.get("user_api_key_metadata", {}) or {}
team_metadata = metadata.get("team_metadata", {}) or {}
policy_id = (
metadata.get("zguard_policy_id")
if "zguard_policy_id" in metadata
else (
user_api_key_metadata.get("zguard_policy_id")
if "zguard_policy_id" in user_api_key_metadata
else (
team_metadata.get("zguard_policy_id")
if "zguard_policy_id" in team_metadata
else self.policy_id
)
)
)
```
You can leverage this feature to apply multiple policies configured on the Zscaler AI Guard (ZGuard) to traffic from different applications. (Note: It is recommended to map policies using either Team or Key metadata, but not a mix of both.)
Example set in Team/Key Metadata, you can set From UI:
```
{"zguard_policy_id": 100}
```

130
docs/my-website/docs/proxy/ui_team_soft_budget_alerts.md Normal file
View file

@ -0,0 +1,130 @@
import Image from '@theme/IdealImage';
# Team Soft Budget Alerts
Set a soft budget on a team and get email alerts when spending crosses the threshold — without blocking any requests.
## Overview
A **soft budget** is a spending threshold that triggers email notifications when exceeded, but **does not block requests**. This is different from a hard budget (`max_budget`), which rejects requests once the limit is reached.
<Image img={require('../../img/ui_team_soft_budget_alerts.png')} />
Team soft budget alerts let you:
- **Get notified early** — receive email alerts when a team's spend crosses the soft budget threshold
- **Keep requests flowing** — unlike hard budgets, soft budgets never block API calls
- **Target specific recipients** — send alerts to specific email addresses (e.g. team leads, finance), not just the team members
- **Work without global alerting** — team soft budget alerts are sent via email independently of Slack or other global alerting configuration
:::warning Email integration required
Team soft budget alerts are sent via email. You must have an active email integration (SendGrid, Resend, or SMTP) configured on your proxy for alerts to be delivered. See [Email Notifications](./email.md) for setup instructions.
:::
:::info Automatically active
Team soft budget alerts are **automatically active** once you configure a soft budget and at least one alerting email on a team. No additional proxy configuration or restart is needed — alerts are checked on every request.
:::
## How It Works
On every API request made with a key belonging to a team, the proxy checks:
1. Does the team have a `soft_budget` set?
2. Is the team's current `spend` >= the `soft_budget`?
3. Are there any emails configured in `soft_budget_alerting_emails`?
If all three conditions are met, an email alert is sent to the configured recipients. Alerts are **deduplicated** so the same alert is only sent once within a 24-hour window.
## How to Set Up Team Soft Budget Alerts
### 1. Navigate to the Admin UI
Go to the Admin UI (e.g. `http://localhost:4000/ui` or your `PROXY_BASE_URL/ui`).
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/f06d75ad-25ef-4ee8-90c3-9604f8e46a1c/ascreenshot_1a6defaed1494d6da0001459511ecfd5_text_export.jpeg)
### 2. Go to Teams
Click **Teams** in the sidebar.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/f06d75ad-25ef-4ee8-90c3-9604f8e46a1c/ascreenshot_2d258fa280f6463b966bf7a05bb102d5_text_export.jpeg)
### 3. Select a team
Click on the team you want to configure soft budget alerts for.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/490f09fb-6bf5-45a8-a384-676889f34c88/ascreenshot_15cceb22abe64df0bf7d7c742ecb5b2f_text_export.jpeg)
### 4. Open team Settings
Click the **Settings** tab to view the team's configuration.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/28dd1bc5-7d07-462f-b277-33f885bdc07e/ascreenshot_12f2b762b5d24686801d93ad5b067e06_text_export.jpeg)
### 5. Edit Settings
Click **Edit Settings** to modify the team's budget configuration.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/30a483ea-7e01-4fdc-ac5f-a5572388d138/ascreenshot_0915eadd9e754a798489853b82de3cb5_text_export.jpeg)
### 6. Set the Soft Budget
Click the **Soft Budget (USD)** field and enter your desired threshold. For example, enter `0.01` for testing or a higher value like `500` for production.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/8b306d80-4943-4ad0-a51a-94b5ebdd6680/ascreenshot_5bb6e65c6428473fac2607f6a7f4b98a_text_export.jpeg)
### 7. Add alerting emails
Click the **Soft Budget Alerting Emails** field and enter one or more comma-separated email addresses that should receive the alert.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/a97c6efa-cc93-45d7-979e-d2a533f423b9/ascreenshot_2d8223ce8e934aa1bfadfb2f78aee5fc_text_export.jpeg)
### 8. Save Changes
Click **Save Changes**. The soft budget alert is now active — no proxy restart required.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-07/865ba6f1-3fc6-4c19-8e08-433561d6c3f7/ascreenshot_b2f0503ada3a479a83dc8b7d01c1f8da_text_export.jpeg)
### 9. Verify: email alert received
Once the team's spend crosses the soft budget, an email alert is sent to the configured recipients. Below is an example of the alert email:
<Image img={require('../../img/ui_team_soft_budget_email_example.png')} />
## Settings Reference
| Setting | Description |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Soft Budget (USD)** | The spending threshold that triggers an email alert. Requests are **not** blocked when this limit is exceeded. |
| **Soft Budget Alerting Emails** | Comma-separated email addresses that receive the alert when the soft budget is crossed. At least one email is required for alerts to be sent. |
:::tip Soft Budget vs. Max Budget
- **Soft Budget**: Advisory threshold — sends email alerts but does **not** block requests.
- **Max Budget**: Hard limit — blocks requests once the budget is exceeded.
You can set both on the same team to get early warnings (soft) and a hard stop (max).
:::
## API Configuration
You can also configure team soft budgets via the API when creating or updating a team:
```bash
curl -X POST 'http://localhost:4000/team/update' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"team_id": "your-team-id",
"soft_budget": 500.00,
"metadata": {
"soft_budget_alerting_emails": ["lead@example.com", "finance@example.com"]
}
}'
```
## Related Documentation
- [Email Notifications](./email.md) – Configure email integrations (Resend, SMTP) for LiteLLM Proxy
- [Alerting](./alerting.md) – Set up Slack and other alerting channels
- [Cost Tracking](./cost_tracking.md) – Track and manage spend across teams, keys, and users

333
docs/my-website/docs/proxy_auth.md Normal file
View file

@ -0,0 +1,333 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# SDK Proxy Authentication (OAuth2/JWT Auto-Refresh)
Automatically obtain and refresh OAuth2/JWT tokens when using the LiteLLM Python SDK with a LiteLLM Proxy that requires JWT authentication.
## Overview
When your LiteLLM Proxy is protected by an OAuth2/OIDC provider (Azure AD, Keycloak, Okta, Auth0, etc.), your SDK clients need valid JWT tokens for every request. Instead of manually managing token lifecycle, `litellm.proxy_auth` handles this automatically:
- Obtains tokens from your identity provider
- Caches tokens to avoid unnecessary requests
- Refreshes tokens before they expire (60-second buffer)
- Injects `Authorization: Bearer <token>` headers into every request
## Quick Start
### Azure AD
<Tabs>
<TabItem value="default" label="DefaultAzureCredential">
Uses the [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential) chain (environment variables, managed identity, Azure CLI, etc.):
```python
import litellm
from litellm.proxy_auth import AzureADCredential, ProxyAuthHandler
# One-time setup
litellm.proxy_auth = ProxyAuthHandler(
credential=AzureADCredential(), # uses DefaultAzureCredential
scope="api://my-litellm-proxy/.default"
)
litellm.api_base = "https://my-proxy.example.com"
# All requests now include Authorization headers automatically
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}]
)
```
</TabItem>
<TabItem value="client-secret" label="ClientSecretCredential">
Use a specific Azure AD app registration:
```python
import litellm
from azure.identity import ClientSecretCredential
from litellm.proxy_auth import AzureADCredential, ProxyAuthHandler
azure_cred = ClientSecretCredential(
tenant_id="your-tenant-id",
client_id="your-client-id",
client_secret="your-client-secret"
)
litellm.proxy_auth = ProxyAuthHandler(
credential=AzureADCredential(credential=azure_cred),
scope="api://my-litellm-proxy/.default"
)
litellm.api_base = "https://my-proxy.example.com"
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}]
)
```
</TabItem>
</Tabs>
**Required package:** `pip install azure-identity`
### Generic OAuth2 (Okta, Auth0, Keycloak, etc.)
Works with any OAuth2 provider that supports the `client_credentials` grant type:
```python
import litellm
from litellm.proxy_auth import GenericOAuth2Credential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=GenericOAuth2Credential(
client_id="your-client-id",
client_secret="your-client-secret",
token_url="https://your-idp.example.com/oauth2/token"
),
scope="litellm_proxy_api"
)
litellm.api_base = "https://my-proxy.example.com"
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}]
)
```
### Custom Credential Provider
Implement the `TokenCredential` protocol to use any authentication mechanism:
```python
import time
import litellm
from litellm.proxy_auth import AccessToken, ProxyAuthHandler
class MyCustomCredential:
"""Any class with a get_token(scope) -> AccessToken method works."""
def get_token(self, scope: str) -> AccessToken:
# Your custom logic to obtain a token
token = my_auth_system.get_jwt(scope=scope)
return AccessToken(
token=token,
expires_on=int(time.time()) + 3600
)
litellm.proxy_auth = ProxyAuthHandler(
credential=MyCustomCredential(),
scope="my-scope"
)
```
## Supported Endpoints
Auth headers are automatically injected for:
| Endpoint | Function |
|----------|----------|
| Chat Completions | `litellm.completion()` / `litellm.acompletion()` |
| Embeddings | `litellm.embedding()` / `litellm.aembedding()` |
## How It Works
```
┌──────────┐ ┌──────────────────┐ ┌──────────────┐ ┌──────────────┐
│ Your │ │ ProxyAuthHandler │ │ Identity │ │ LiteLLM │
│ Code │────▶│ (token cache) │────▶│ Provider │ │ Proxy │
│ │ │ │◀────│ (Azure AD, │ │ │
│ │ │ │ │ Okta, etc) │ │ │
│ │ └────────┬─────────┘ └──────────────┘ │ │
│ │ │ Authorization: Bearer <token> │ │
│ │──────────────┼───────────────────────────────────▶│ │
│ │◀─────────────┼────────────────────────────────────│ │
└──────────┘ │ └──────────────┘
```
1. You set `litellm.proxy_auth` once at startup
2. On each SDK call (`completion()`, `embedding()`), the handler checks its cached token
3. If the token is missing or expires within 60 seconds, it requests a new one from your identity provider
4. The `Authorization: Bearer <token>` header is injected into the request
5. If token retrieval fails, a warning is logged and the request proceeds without auth headers
## API Reference
### ProxyAuthHandler
The main handler that manages the token lifecycle.
```python
from litellm.proxy_auth import ProxyAuthHandler
handler = ProxyAuthHandler(
credential=<TokenCredential>, # required - credential provider
scope="<oauth2-scope>" # required - OAuth2 scope to request
)
```
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `credential` | `TokenCredential` | Yes | A credential provider (AzureADCredential, GenericOAuth2Credential, or custom) |
| `scope` | `str` | Yes | The OAuth2 scope to request tokens for |
**Methods:**
| Method | Returns | Description |
|--------|---------|-------------|
| `get_token()` | `AccessToken` | Get a valid token, refreshing if needed |
| `get_auth_headers()` | `dict` | Get `{"Authorization": "Bearer <token>"}` headers |
### AzureADCredential
Wraps any `azure-identity` credential with lazy initialization.
```python
from litellm.proxy_auth import AzureADCredential
# Uses DefaultAzureCredential (recommended)
cred = AzureADCredential()
# Or wrap a specific azure-identity credential
from azure.identity import ManagedIdentityCredential
cred = AzureADCredential(credential=ManagedIdentityCredential())
```
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `credential` | Azure `TokenCredential` | No | An azure-identity credential. If `None`, uses `DefaultAzureCredential` |
### GenericOAuth2Credential
Standard OAuth2 client credentials flow for any provider.
```python
from litellm.proxy_auth import GenericOAuth2Credential
cred = GenericOAuth2Credential(
client_id="your-client-id",
client_secret="your-client-secret",
token_url="https://your-idp.com/oauth2/token"
)
```
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `client_id` | `str` | Yes | OAuth2 client ID |
| `client_secret` | `str` | Yes | OAuth2 client secret |
| `token_url` | `str` | Yes | Token endpoint URL |
### AccessToken
Dataclass representing an OAuth2 access token.
```python
from litellm.proxy_auth import AccessToken
token = AccessToken(
token="eyJhbG...", # JWT string
expires_on=1234567890 # Unix timestamp
)
```
### TokenCredential Protocol
Any class implementing this protocol can be used as a credential provider:
```python
from litellm.proxy_auth import AccessToken
class MyCredential:
def get_token(self, scope: str) -> AccessToken:
...
```
## Provider-Specific Examples
### Keycloak
```python
from litellm.proxy_auth import GenericOAuth2Credential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=GenericOAuth2Credential(
client_id="litellm-client",
client_secret="your-keycloak-client-secret",
token_url="https://keycloak.example.com/realms/your-realm/protocol/openid-connect/token"
),
scope="openid"
)
```
### Okta
```python
from litellm.proxy_auth import GenericOAuth2Credential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=GenericOAuth2Credential(
client_id="your-okta-client-id",
client_secret="your-okta-client-secret",
token_url="https://your-org.okta.com/oauth2/default/v1/token"
),
scope="litellm_api"
)
```
### Auth0
```python
from litellm.proxy_auth import GenericOAuth2Credential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=GenericOAuth2Credential(
client_id="your-auth0-client-id",
client_secret="your-auth0-client-secret",
token_url="https://your-tenant.auth0.com/oauth/token"
),
scope="https://my-proxy.example.com/api"
)
```
### Azure AD with Managed Identity
```python
from azure.identity import ManagedIdentityCredential
from litellm.proxy_auth import AzureADCredential, ProxyAuthHandler
litellm.proxy_auth = ProxyAuthHandler(
credential=AzureADCredential(
credential=ManagedIdentityCredential()
),
scope="api://my-litellm-proxy/.default"
)
```
## Combining with `use_litellm_proxy`
You can use `proxy_auth` together with [`use_litellm_proxy`](./providers/litellm_proxy#send-all-sdk-requests-to-litellm-proxy) to route all SDK requests through an authenticated proxy:
```python
import os
import litellm
from litellm.proxy_auth import AzureADCredential, ProxyAuthHandler
# Route all requests through the proxy
os.environ["LITELLM_PROXY_API_BASE"] = "https://my-proxy.example.com"
litellm.use_litellm_proxy = True
# Authenticate with OAuth2/JWT
litellm.proxy_auth = ProxyAuthHandler(
credential=AzureADCredential(),
scope="api://my-litellm-proxy/.default"
)
# This request goes through the proxy with automatic JWT auth
response = litellm.completion(
model="vertex_ai/gemini-2.0-flash-001",
messages=[{"role": "user", "content": "Hello!"}]
)
```

52
docs/my-website/docs/troubleshoot.md
View file

@ -45,6 +45,58 @@ Full error logs, stack traces, and any images from service metrics (CPU, memory,
---
## UI Issues
If you're experiencing issues with the LiteLLM Admin UI, please include the following information in addition to the general details above.
### 1. Steps to Reproduce
A clear, step-by-step description of how to trigger the issue (e.g., "Navigate to Settings → Team, click 'Create Team', fill in fields, click submit → error appears").
### 2. LiteLLM Version
The current version of LiteLLM you're running. Check via `litellm --version` or the UI's settings page.
### 3. Architecture & Deployment Setup
Distributed environments are a known source of UI issues. Please describe:
- **Number of LiteLLM instances/replicas** and how they are deployed (e.g., Kubernetes, Docker Compose, ECS)
- **Load balancer** type and configuration (e.g., ALB, Nginx, Cloudflare Tunnel) — include whether sticky sessions are enabled
- **How the UI is accessed** — directly via LiteLLM, through a reverse proxy, or behind an ingress controller
- **Any CDN or caching layers** between the user and the LiteLLM server
### 4. Network Tab Requests
Open your browser's Developer Tools (F12 → Network tab), reproduce the issue, and share:
- The **failing request(s)** — URL, method, status code, and response body
- **Screenshots or HAR export** of the relevant network activity
- Any **CORS or mixed-content errors** shown in the Console tab
### 5. Environment Variables
Non-sensitive environment variables related to the UI and proxy setup, such as:
- `LITELLM_MASTER_KEY`
- `PROXY_BASE_URL` / `LITELLM_PROXY_BASE_URL`
- `UI_BASE_PATH`
- Any SSO-related variables (e.g., `GOOGLE_CLIENT_ID`, `MICROSOFT_TENANT`)
Do **not** include passwords, secrets, or API keys.
### 6. Browser & Access Details
- **Browser** and version (e.g., Chrome 120, Firefox 121)
- **Access URL** used to reach the UI (redact sensitive parts)
- Whether the issue occurs for **all users or specific roles** (Admin, Internal User, etc.)
### 7. Screenshots or Screen Recordings
A screenshot or short screen recording of the issue is extremely helpful. Include any visible error messages, toasts, or unexpected behavior.
---
## Support Channels
[Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)

68
docs/my-website/docs/troubleshoot/max_callbacks.md Normal file
View file

@ -0,0 +1,68 @@
# MAX_CALLBACKS Limit
## Error Message
```
Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
```
## What This Means
LiteLLM limits the number of callbacks that can be registered to prevent performance degradation. Each callback runs on every LLM request, so having too many callbacks can cause exponential CPU usage and slow down your proxy.
The default limit is **30 callbacks**.
## When You Might Hit This Limit
- **Large enterprise deployments** with many teams, each having their own guardrails
- **Multiple logging integrations** combined with custom callbacks
- **Per-team callback configurations** that add up across your organization
## How to Override
Set the `LITELLM_MAX_CALLBACKS` environment variable to increase the limit:
```bash
# Docker
docker run -e LITELLM_MAX_CALLBACKS=100 ...
# Docker Compose
environment:
- LITELLM_MAX_CALLBACKS=100
# Kubernetes
env:
- name: LITELLM_MAX_CALLBACKS
value: "100"
# Direct
export LITELLM_MAX_CALLBACKS=100
litellm --config config.yaml
```
## Recommendations
1. **Start conservative** - Only increase as much as you need. If you have 60 teams with guardrails, try `LITELLM_MAX_CALLBACKS=75` to leave headroom.
2. **Monitor performance** - More callbacks means more processing per request. Watch your CPU usage and response latency after increasing the limit.
3. **Consolidate where possible** - If multiple teams use identical guardrails, consider using shared callback configurations rather than per-team duplicates.
## Example: Large Enterprise Setup
For an organization with 60+ teams, each with a guardrail callback:
```yaml
# config.yaml
litellm_settings:
callbacks: ["prometheus", "langfuse"] # 2 global callbacks
# Each team adds 1 guardrail callback = 60+ callbacks
# Total: 62+ callbacks needed
```
Set the environment variable:
```bash
export LITELLM_MAX_CALLBACKS=100
```

129
docs/my-website/docs/tutorials/claude_code_beta_headers.md Normal file
View file

@ -0,0 +1,129 @@
import Image from '@theme/IdealImage';
# Claude Code - Fixing Invalid Beta Header Errors
When using Claude Code with LiteLLM and non-Anthropic providers (Bedrock, Azure AI, Vertex AI), you may encounter "invalid beta header" errors. This guide explains how to fix these errors locally or contribute a fix to LiteLLM.
## What Are Beta Headers?
Anthropic uses beta headers to enable experimental features in Claude. When you use Claude Code, it may send beta headers like:
```
anthropic-beta: prompt-caching-scope-2026-01-05,advanced-tool-use-2025-11-20
```
However, not all providers support all Anthropic beta features. When an unsupported beta header is sent to a provider, you'll see an error.
## Common Error Message
```bash
Error: The model returned the following errors: invalid beta flag
```
## How LiteLLM Handles Beta Headers
LiteLLM automatically filters out unsupported beta headers using a configuration file:
```
litellm/litellm/anthropic_beta_headers_config.json
```
This JSON file lists which beta headers are **unsupported** for each provider. Headers not in the unsupported list are passed through to the provider.
## Quick Fix: Update Config Locally
If you encounter an invalid beta header error, you can fix it immediately by updating the config file locally.
### Step 1: Locate the Config File
Find the file in your LiteLLM installation:
```bash
# If installed via pip
cd $(python -c "import litellm; import os; print(os.path.dirname(litellm.__file__))")
# The config file is at:
# litellm/anthropic_beta_headers_config.json
```
### Step 2: Add the Unsupported Header
Open `anthropic_beta_headers_config.json` and add the problematic header to the appropriate provider's list:
```json title="anthropic_beta_headers_config.json"
{
"description": "Unsupported Anthropic beta headers for each provider. Headers listed here will be dropped. Headers not listed are passed through as-is.",
"anthropic": [],
"azure_ai": [],
"bedrock_converse": [
"prompt-caching-scope-2026-01-05",
"bash_20250124",
"bash_20241022",
"text_editor_20250124",
"text_editor_20241022",
"compact-2026-01-12",
"advanced-tool-use-2025-11-20",
"web-fetch-2025-09-10",
"code-execution-2025-08-25",
"skills-2025-10-02",
"files-api-2025-04-14"
],
"bedrock": [
"advanced-tool-use-2025-11-20",
"prompt-caching-scope-2026-01-05",
"structured-outputs-2025-11-13",
"web-fetch-2025-09-10",
"code-execution-2025-08-25",
"skills-2025-10-02",
"files-api-2025-04-14"
],
"vertex_ai": [
"prompt-caching-scope-2026-01-05"
]
}
```
### Step 3: Restart Your Application
After updating the config file, restart your LiteLLM proxy or application:
```bash
# If using LiteLLM proxy
litellm --config config.yaml
# If using Python SDK
# Just restart your Python application
```
The updated configuration will be loaded automatically.
## Contributing a Fix to LiteLLM
Help the community by contributing your fix! If your local changes work, please raise a PR with the addition of the header and we will merge it.
## How Beta Header Filtering Works
When you make a request through LiteLLM:
```mermaid
sequenceDiagram
participant CC as Claude Code
participant LP as LiteLLM
participant Config as Beta Headers Config
participant Provider as Provider (Bedrock/Azure/etc)
CC->>LP: Request with beta headers
Note over CC,LP: anthropic-beta: header1,header2,header3
LP->>Config: Load unsupported headers for provider
Config-->>LP: Returns unsupported list
Note over LP: Filter headers:<br/>- Remove unsupported<br/>- Keep supported
LP->>Provider: Request with filtered headers
Note over LP,Provider: anthropic-beta: header2<br/>(header1, header3 removed)
Provider-->>LP: Success response
LP-->>CC: Response
```

43
docs/my-website/docs/tutorials/claude_code_prompt_cache_routing.md Normal file
View file

@ -0,0 +1,43 @@
# Claude Code - Prompt Cache Routing
Claude's [Prompt Caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) feature helps to optimize API usage through attempting to cache prompts and re-use cached prompts during subsequent API calls. This feature is used by Claude Code.
When LiteLLM [load balancing](../proxy/load_balancing.md) is enabled, to ensure this prompt caching feature still works with Claude Code, LiteLLM needs to be configured to use the `PromptCachingDeploymentCheck` pre-call check. This pre-call check will ensure that API calls that used prompt caching are remembered and that subsequent API calls that try to use that prompt caching are routed to the same model deployment where a cache write occurred.
## Set Up
1. Configure the router so that it uses the `PromptCachingDeploymentCheck` (via setting the `optional_pre_call_checks` property), and configure the models so that they can access multiple deployments of Claude; below, we show an example for multiple AWS accounts (referred to as `account-1` and `account-2`, using the `aws_profile_name` property):
```yaml
router_settings:
optional_pre_call_checks: ["prompt_caching"]
model_list:
- litellm_params:
model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_profile_name: account-1
aws_region_name: us-west-2
model_info:
litellm_provider: bedrock
model_name: us.anthropic.claude-sonnet-4-5-20250929-v1:0
- litellm_params:
model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_profile_name: account-2
aws_region_name: us-west-2
model_info:
litellm_provider: bedrock
model_name: us.anthropic.claude-sonnet-4-5-20250929-v1:0
```
2. Utilize Claude Code:
1. Launch Claude Code, which will do a warm-up API call that tries to cache its warm-up prompt and its system prompt.
2. Wait a few seconds, then quit Claude Code and re-open it.
3. You'll notice that the warm-up API call successfully gets a cache hit (if using Claude Code in an IDE like VS Code, ensure that you don't do anything between step 2.1 and 2.2 here, otherwise there may not be a cache hit):
1. Go to the [LiteLLM Request Logs page](../proxy/ui_logs.md) in the Admin UI
2. Click on the individual requests to see (a) the cache creation and cache read tokens; and (b) the Model ID. In particular, the API call from step 2.1 should show a cache write, and the API call from step 2.2 should show a cache read; in addition, the Model ID should be equal (meaning the API call is getting forwarded to the same AWS account).
## Related
- [Claude Code - Quickstart](./claude_responses_api.md)
- [Claude Code - Customer Tracking](./claude_code_customer_tracking.md)
- [Claude Code - Plugin Marketplace](./claude_code_plugin_marketplace.md)
- [Claude Code - WebSearch](./claude_code_websearch.md)
- [Proxy - Load Balancing](../proxy/load_balancing.md)

BIN
docs/my-website/img/release_notes/mcp_internet.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 724 KiB

BIN
docs/my-website/img/ui_team_soft_budget_alerts.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 328 KiB

BIN
docs/my-website/img/ui_team_soft_budget_email_example.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 50 KiB

2
docs/my-website/package.json
View file

@ -61,6 +61,8 @@
"mermaid": ">=11.10.0",
"gray-matter": "4.0.3",
"glob": ">=11.1.0",
"tar": ">=7.5.7",
"@isaacs/brace-expansion": ">=5.0.1",
"node-forge": ">=1.3.2",
"mdast-util-to-hast": ">=13.2.1",
"lodash-es": ">=4.17.23"

2
docs/my-website/release_notes/v1.81.3-stable/index.md
View file

@ -27,7 +27,7 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.81.3.rc.2
docker.litellm.ai/berriai/litellm:v1.81.3-stable
```
</TabItem>

2
docs/my-website/release_notes/v1.81.6.md
View file

@ -1,5 +1,5 @@
---
title: "v1.81.6 - Logs v2 with Tool Call Tracing"
title: "[Preview] v1.81.6 - Logs v2 with Tool Call Tracing"
slug: "v1-81-6"
date: 2026-01-31T00:00:00
authors:

372
docs/my-website/release_notes/v1.81.9.md Normal file
View file

@ -0,0 +1,372 @@
---
title: "[Preview] v1.81.9 - Control which MCP Servers are exposed on the Internet"
slug: "v1-81-9"
date: 2026-02-07T00:00:00
authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false
---
## Deploy this version
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.81.9.rc.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.81.9
```
</TabItem>
</Tabs>
## Key Highlights
- **Claude Opus 4.6** - [Full support across Anthropic, AWS Bedrock, Azure AI, and Vertex AI with adaptive thinking and 1M context window](../../blog/claude_opus_4_6)
- **A2A Agent Gateway** - [Call A2A (Agent-to-Agent) registered agents through the standard `/chat/completions` API](../../docs/a2a_invoking_agents)
- **Expose MCP servers on the public internet** - [Launch MCP servers with public/private visibility and IP-based access control for internet-facing deployments](../../docs/mcp_public_internet)
- **UI Team Soft Budget Alerts** - [Set soft budgets on teams and receive email alerts when spending crosses the threshold — without blocking requests](../../docs/proxy/ui_team_soft_budget_alerts)
- **Performance Optimizations** - Multiple performance improvements including ~40% Prometheus CPU reduction, LRU caching, and optimized logging paths
- **LiteLLM Observatory** - [Automated 24-hour load tests](../../blog/litellm-observatory)
- **30% Faster Request Processing for Callback-Heavy Deployments** - [Performance improvement for callback heavy deployments][PR #20354](https://github.com/BerriAI/litellm/pull/20354)
---
## 30% Faster Request Processing for Callback-Heavy Deployments
If you use logging callbacks like Langfuse, Datadog, or Prometheus, every request was paying an unnecessary cost: three loops that re-sorted your callbacks on every single request, even though the callback list hadn't changed. The more callbacks you had configured, the more time was wasted. We moved this work to happen once at startup instead of on every request. For deployments with the default callback set, this is a ~30% speedup in request setup. For deployments with many callbacks configured, the improvement is even larger.
---
## LiteLLM Observatory
LiteLLM Observatory is a long-running release-validation system we built to catch regressions before they reach users. The system is built to be extensible—you can add new tests, configure models and failure thresholds, and queue runs against any deployment. Our goal is to achieve 100% coverage of LiteLLM functionality through these tests. We run 24-hour load tests against our production deployments before all releases, surfacing issues like resource lifecycle bugs, OOMs, and CPU regressions that only appear under sustained load.
---
## MCP Servers on the Public Internet
This release makes it safe to expose MCP servers on the public internet by adding public/private visibility and IP-based access control. You can now run internet-facing MCP services while restricting access to trusted networks and keeping internal tools private.
[Get started](../../docs/mcp_public_internet)
<Image
img={require('../img/release_notes/mcp_internet.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>
## UI Team Soft Budget Alerts
Set a soft budget on any team to receive email alerts when spending crosses the threshold — without blocking any requests. Configure the threshold and alerting emails directly from the Admin UI, with no proxy restart needed.
[Get started](../../docs/proxy/ui_team_soft_budget_alerts)
<Image
img={require('../img/ui_team_soft_budget_alerts.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>
Let's dive in.
---
## New Models / Updated Models
#### New Model Support (13 new models)
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) |
| -------- | ----- | -------------- | ------------------- | -------------------- |
| Anthropic | `claude-opus-4-6` | 1M | $5.00 | $25.00 |
| AWS Bedrock | `anthropic.claude-opus-4-6-v1` | 1M | $5.00 | $25.00 |
| Azure AI | `azure_ai/claude-opus-4-6` | 200K | $5.00 | $25.00 |
| Vertex AI | `vertex_ai/claude-opus-4-6` | 1M | $5.00 | $25.00 |
| Google Gemini | `gemini/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 |
| Vertex AI | `vertex_ai/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 |
| Moonshot | `moonshot/kimi-k2.5` | 262K | $0.60 | $3.00 |
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-2507` | 262K | $0.07 | $0.10 |
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-thinking-2507` | 262K | $0.11 | $0.60 |
| Together AI | `together_ai/zai-org/GLM-4.7` | 200K | $0.45 | $2.00 |
| Together AI | `together_ai/moonshotai/Kimi-K2.5` | 256K | $0.50 | $2.80 |
| ElevenLabs | `elevenlabs/eleven_v3` | - | $0.18/1K chars | - |
| ElevenLabs | `elevenlabs/eleven_multilingual_v2` | - | $0.18/1K chars | - |
#### Features
- **[Anthropic](../../docs/providers/anthropic)**
- Full Claude Opus 4.6 support with adaptive thinking across all regions (us, eu, apac, au) - [PR #20506](https://github.com/BerriAI/litellm/pull/20506), [PR #20508](https://github.com/BerriAI/litellm/pull/20508), [PR #20514](https://github.com/BerriAI/litellm/pull/20514), [PR #20551](https://github.com/BerriAI/litellm/pull/20551)
- Map reasoning content to anthropic thinking block (streaming + non-streaming) - [PR #20254](https://github.com/BerriAI/litellm/pull/20254)
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Add 1hr tiered caching costs for long-context models - [PR #20214](https://github.com/BerriAI/litellm/pull/20214)
- Support TTL (1h) field in prompt caching for Bedrock Claude 4.5 models - [PR #20338](https://github.com/BerriAI/litellm/pull/20338)
- Add Nova Sonic speech-to-speech model support - [PR #20244](https://github.com/BerriAI/litellm/pull/20244)
- Fix empty assistant message for Converse API - [PR #20390](https://github.com/BerriAI/litellm/pull/20390)
- Fix content blocked handling - [PR #20606](https://github.com/BerriAI/litellm/pull/20606)
- **[Google Gemini / Vertex AI](../../docs/providers/gemini)**
- Add Gemini Deep Research model support - [PR #20406](https://github.com/BerriAI/litellm/pull/20406)
- Fix Vertex AI Gemini streaming content_filter handling - [PR #20105](https://github.com/BerriAI/litellm/pull/20105)
- Allow using OpenAI-style tools for `web_search` with Vertex AI/Gemini models - [PR #20280](https://github.com/BerriAI/litellm/pull/20280)
- Fix `supports_native_streaming` for Gemini and Vertex AI models - [PR #20408](https://github.com/BerriAI/litellm/pull/20408)
- Add mapping for responses tools in file IDs - [PR #20402](https://github.com/BerriAI/litellm/pull/20402)
- **[Cohere](../../docs/providers/cohere)**
- Support `dimensions` param for Cohere embed v4 - [PR #20235](https://github.com/BerriAI/litellm/pull/20235)
- **[Cerebras](../../docs/providers/cerebras)**
- Add reasoning param support for GPT OSS Cerebras - [PR #20258](https://github.com/BerriAI/litellm/pull/20258)
- **[Moonshot](../../docs/providers/moonshot)**
- Add Kimi K2.5 model entries - [PR #20273](https://github.com/BerriAI/litellm/pull/20273)
- **[OpenRouter](../../docs/providers/openrouter)**
- Add Qwen3-235B models - [PR #20455](https://github.com/BerriAI/litellm/pull/20455)
- **[Together AI](../../docs/providers/togetherai)**
- Add GLM-4.7 and Kimi-K2.5 models - [PR #20319](https://github.com/BerriAI/litellm/pull/20319)
- **[ElevenLabs](../../docs/providers/elevenlabs)**
- Add `eleven_v3` and `eleven_multilingual_v2` TTS models - [PR #20522](https://github.com/BerriAI/litellm/pull/20522)
- **[Vercel AI Gateway](../../docs/providers/vercel_ai_gateway)**
- Add missing capability flags to models - [PR #20276](https://github.com/BerriAI/litellm/pull/20276)
- **[GitHub Copilot](../../docs/providers/github_copilot)**
- Fix system prompts being dropped and auto-add required Copilot headers - [PR #20113](https://github.com/BerriAI/litellm/pull/20113)
- **[GigaChat](../../docs/providers/gigachat)**
- Fix incorrect merging of consecutive user messages for GigaChat provider - [PR #20341](https://github.com/BerriAI/litellm/pull/20341)
- **[xAI](../../docs/providers/xai_realtime)**
- Add xAI `/realtime` API support - works with LiveKit SDK - [PR #20381](https://github.com/BerriAI/litellm/pull/20381)
- **[OpenAI](../../docs/providers/openai)**
- Add `gpt-5-search-api` model and docs clarifications - [PR #20512](https://github.com/BerriAI/litellm/pull/20512)
### Bug Fixes
- **[Anthropic](../../docs/providers/anthropic)**
- Fix extra inputs not permitted error for `provider_specific_fields` - [PR #20334](https://github.com/BerriAI/litellm/pull/20334)
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Fix: Managed Batches inconsistent state management for list and cancel batches - [PR #20331](https://github.com/BerriAI/litellm/pull/20331)
- **[OpenAI Embeddings](../../docs/providers/openai)**
- Fix `open_ai_embedding_models` to have `custom_llm_provider` None - [PR #20253](https://github.com/BerriAI/litellm/pull/20253)
---
## LLM API Endpoints
#### Features
- **[Messages API](../../docs/providers/anthropic)**
- Filter unsupported Claude Code beta headers for non-Anthropic providers - [PR #20578](https://github.com/BerriAI/litellm/pull/20578)
- Fix inconsistent response format in `anthropic.messages.acreate()` when using non-Anthropic providers - [PR #20442](https://github.com/BerriAI/litellm/pull/20442)
- Fix 404 on `/api/event_logging/batch` endpoint that caused Claude Code "route not found" errors - [PR #20504](https://github.com/BerriAI/litellm/pull/20504)
- **[A2A Agent Gateway](../../docs/a2a)**
- Allow calling A2A agents through LiteLLM `/chat/completions` API - [PR #20358](https://github.com/BerriAI/litellm/pull/20358)
- Use A2A registered agents with `/chat/completions` - [PR #20362](https://github.com/BerriAI/litellm/pull/20362)
- Fix A2A agents deployed with localhost/internal URLs in their agent cards - [PR #20604](https://github.com/BerriAI/litellm/pull/20604)
- **[Files API](../../docs/providers/gemini)**
- Add support for delete and GET via file_id for Gemini - [PR #20329](https://github.com/BerriAI/litellm/pull/20329)
- **General**
- Add User-Agent customization support - [PR #19881](https://github.com/BerriAI/litellm/pull/19881)
- Fix search tools not found when using per-request routers - [PR #19818](https://github.com/BerriAI/litellm/pull/19818)
- Forward extra headers in chat - [PR #20386](https://github.com/BerriAI/litellm/pull/20386)
---
## Management Endpoints / UI
#### Features
- **SSO Configuration**
- SSO Config Team Mappings - [PR #20111](https://github.com/BerriAI/litellm/pull/20111)
- UI - SSO: Add Team Mappings - [PR #20299](https://github.com/BerriAI/litellm/pull/20299)
- Extract user roles from JWT access token for Keycloak compatibility - [PR #20591](https://github.com/BerriAI/litellm/pull/20591)
- **Auth / SDK**
- Add `proxy_auth` for auto OAuth2/JWT token management in SDK - [PR #20238](https://github.com/BerriAI/litellm/pull/20238)
- **Virtual Keys**
- Key `reset_spend` endpoint - [PR #20305](https://github.com/BerriAI/litellm/pull/20305)
- UI - Keys: Allowed Routes to Key Info and Edit Pages - [PR #20369](https://github.com/BerriAI/litellm/pull/20369)
- Add Key info endpoint object permission data - [PR #20407](https://github.com/BerriAI/litellm/pull/20407)
- Keys and Teams Router Setting + Allow Override of Router Settings - [PR #20205](https://github.com/BerriAI/litellm/pull/20205)
- **Teams & Budgets**
- Add `soft_budget` to Team Table + Create/Update Endpoints - [PR #20530](https://github.com/BerriAI/litellm/pull/20530)
- Team Soft Budget Email Alerts - [PR #20553](https://github.com/BerriAI/litellm/pull/20553)
- UI - Team Settings: Soft Budget + Alerting Emails - [PR #20634](https://github.com/BerriAI/litellm/pull/20634)
- UI - User Budget Page: Unlimited Budget Checkbox - [PR #20380](https://github.com/BerriAI/litellm/pull/20380)
- `/user/update` allow for `max_budget` resets - [PR #20375](https://github.com/BerriAI/litellm/pull/20375)
- **UI Improvements**
- Default Team Settings: Migrate to use Reusable Model Select - [PR #20310](https://github.com/BerriAI/litellm/pull/20310)
- Navbar: Option to Hide Community Engagement Buttons - [PR #20308](https://github.com/BerriAI/litellm/pull/20308)
- Show team alias on Models health page - [PR #20359](https://github.com/BerriAI/litellm/pull/20359)
- Admin Settings: Add option for Authentication for public AI Hub - [PR #20444](https://github.com/BerriAI/litellm/pull/20444)
- Adjust daily spend date filtering for user timezone - [PR #20472](https://github.com/BerriAI/litellm/pull/20472)
- **SCIM**
- Add base `/scim/v2` endpoint for SCIM resource discovery - [PR #20301](https://github.com/BerriAI/litellm/pull/20301)
- **Proxy CLI**
- CLI arguments for RDS IAM auth - [PR #20437](https://github.com/BerriAI/litellm/pull/20437)
#### Bugs
- Fix: Remove unnecessary key blocking on UI login that prevented access - [PR #20210](https://github.com/BerriAI/litellm/pull/20210)
- UI - Team Settings: Disable Global Guardrail Persistence - [PR #20307](https://github.com/BerriAI/litellm/pull/20307)
- UI - Model Info Page: Fix Input and Output Labels - [PR #20462](https://github.com/BerriAI/litellm/pull/20462)
- UI - Model Page: Column Resizing on Smaller Screens - [PR #20599](https://github.com/BerriAI/litellm/pull/20599)
- Fix `/key/list` `user_id` Empty String Edge Case - [PR #20623](https://github.com/BerriAI/litellm/pull/20623)
- Add array type checks for model, agent, and MCP hub data to prevent UI crashes - [PR #20469](https://github.com/BerriAI/litellm/pull/20469)
- Fix unique constraint on daily tables + logging when updates fail - [PR #20394](https://github.com/BerriAI/litellm/pull/20394)
---
## Logging / Guardrail / Prompt Management Integrations
#### Bug Fixes (3 fixes)
- **[Langfuse](../../docs/proxy/logging#langfuse)**
- Fix Langfuse OTEL trace export failing when spans contain null attributes - [PR #20382](https://github.com/BerriAI/litellm/pull/20382)
- **[Prometheus](../../docs/proxy/logging#prometheus)**
- Fix incorrect failure metrics labels causing miscounted error rates - [PR #20152](https://github.com/BerriAI/litellm/pull/20152)
- **[Slack Alerts](../../docs/proxy/alerting)**
- Fix Slack alert delivery failing for certain budget threshold configurations - [PR #20257](https://github.com/BerriAI/litellm/pull/20257)
#### Guardrails (7 updates)
- **Custom Code Guardrails**
- Add HTTP support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support - [PR #20619](https://github.com/BerriAI/litellm/pull/20619)
- Custom Code Guardrails UI Playground - [PR #20377](https://github.com/BerriAI/litellm/pull/20377)
- **Team-Based Guardrails**
- Implement team-based isolation guardrails management - [PR #20318](https://github.com/BerriAI/litellm/pull/20318)
- **[OpenAI Moderations](../../docs/apply_guardrail)**
- Ensure OpenAI Moderations Guard works with OpenAI Embeddings - [PR #20523](https://github.com/BerriAI/litellm/pull/20523)
- **[GraySwan / Cygnal](../../docs/apply_guardrail)**
- Fix fail-open for GraySwan and pass metadata to Cygnal API endpoint - [PR #19837](https://github.com/BerriAI/litellm/pull/19837)
- **General**
- Check for `model_response_choices` before guardrail input - [PR #19784](https://github.com/BerriAI/litellm/pull/19784)
- Preserve streaming content on guardrail-sampled chunks - [PR #20027](https://github.com/BerriAI/litellm/pull/20027)
---
## Spend Tracking, Budgets and Rate Limiting
- **Support 0 cost models** - Allow zero-cost model entries for internal/free-tier models - [PR #20249](https://github.com/BerriAI/litellm/pull/20249)
---
## MCP Gateway (9 updates)
- **MCP Semantic Filtering** - Filter MCP tools using semantic similarity to reduce tool sprawl for LLM calls - [PR #20296](https://github.com/BerriAI/litellm/pull/20296), [PR #20316](https://github.com/BerriAI/litellm/pull/20316)
- **UI - MCP Semantic Filtering** - Add support for MCP Semantic Filtering configuration on UI - [PR #20454](https://github.com/BerriAI/litellm/pull/20454)
- **MCP IP-Based Access Control** - Set MCP servers as private/public available on internet with IP-based restrictions - [PR #20607](https://github.com/BerriAI/litellm/pull/20607), [PR #20620](https://github.com/BerriAI/litellm/pull/20620)
- **Fix MCP "Session not found" error** on VSCode reconnect - [PR #20298](https://github.com/BerriAI/litellm/pull/20298)
- **Fix OAuth2 'Capabilities: none' bug** for upstream MCP servers - [PR #20602](https://github.com/BerriAI/litellm/pull/20602)
- **Include Config Defined Search Tools** in `/search_tools/list` - [PR #20371](https://github.com/BerriAI/litellm/pull/20371)
- **UI - Search Tools**: Show Config Defined Search Tools - [PR #20436](https://github.com/BerriAI/litellm/pull/20436)
- **Ensure MCP permissions are enforced** when using JWT Auth - [PR #20383](https://github.com/BerriAI/litellm/pull/20383)
- **Fix `gcs_bucket_name` not being passed** correctly for MCP server storage configuration - [PR #20491](https://github.com/BerriAI/litellm/pull/20491)
---
## Performance / Loadbalancing / Reliability improvements (14 improvements)
- **Prometheus ~40% CPU reduction** - Parallelize budget metrics, fix caching bug, reduce CPU usage - [PR #20544](https://github.com/BerriAI/litellm/pull/20544)
- **Prevent closed client errors** by reverting httpx client caching - [PR #20025](https://github.com/BerriAI/litellm/pull/20025)
- **Avoid unnecessary Router creation** when no models or search tools are configured - [PR #20661](https://github.com/BerriAI/litellm/pull/20661)
- **Optimize `wrapper_async`** with `CallTypes` caching and reduced lookups - [PR #20204](https://github.com/BerriAI/litellm/pull/20204)
- **Cache `_get_relevant_args_to_use_for_logging()`** at module level - [PR #20077](https://github.com/BerriAI/litellm/pull/20077)
- **LRU cache for `normalize_request_route`** - [PR #19812](https://github.com/BerriAI/litellm/pull/19812)
- **Optimize `get_standard_logging_metadata`** with set intersection - [PR #19685](https://github.com/BerriAI/litellm/pull/19685)
- **Early-exit guards in `completion_cost`** for unused features - [PR #20020](https://github.com/BerriAI/litellm/pull/20020)
- **Optimize `get_litellm_params`** with sparse kwargs extraction - [PR #19884](https://github.com/BerriAI/litellm/pull/19884)
- **Guard debug log f-strings** and remove redundant dict copies - [PR #19961](https://github.com/BerriAI/litellm/pull/19961)
- **Replace enum construction with frozenset lookup** - [PR #20302](https://github.com/BerriAI/litellm/pull/20302)
- **Guard debug f-string in `update_environment_variables`** - [PR #20360](https://github.com/BerriAI/litellm/pull/20360)
- **Warn when budget lookup fails** to surface silent caching misses - [PR #20545](https://github.com/BerriAI/litellm/pull/20545)
- **Add INFO-level session reuse logging** per request for better observability - [PR #20597](https://github.com/BerriAI/litellm/pull/20597)
---
## Database Changes
### Schema Updates
| Table | Change Type | Description | PR | Migration |
| ----- | ----------- | ----------- | -- | --------- |
| `LiteLLM_TeamTable` | New Column | Added `allow_team_guardrail_config` boolean field for team-based guardrail isolation | [PR #20318](https://github.com/BerriAI/litellm/pull/20318) | [Migration](https://github.com/BerriAI/litellm/blob/main/litellm-proxy-extras/litellm_proxy_extras/migrations/20260205091235_allow_team_guardrail_config/migration.sql) |
| `LiteLLM_DeletedTeamTable` | New Column | Added `allow_team_guardrail_config` boolean field | [PR #20318](https://github.com/BerriAI/litellm/pull/20318) | [Migration](https://github.com/BerriAI/litellm/blob/main/litellm-proxy-extras/litellm_proxy_extras/migrations/20260205091235_allow_team_guardrail_config/migration.sql) |
| `LiteLLM_TeamTable` | New Column | Added `soft_budget` (double precision) for soft budget alerting | [PR #20530](https://github.com/BerriAI/litellm/pull/20530) | [Migration](https://github.com/BerriAI/litellm/blob/main/litellm-proxy-extras/litellm_proxy_extras/migrations/20260205144610_add_soft_budget_to_team_table/migration.sql) |
| `LiteLLM_DeletedTeamTable` | New Column | Added `soft_budget` (double precision) | [PR #20653](https://github.com/BerriAI/litellm/pull/20653) | [Migration](https://github.com/BerriAI/litellm/blob/main/litellm-proxy-extras/litellm_proxy_extras/migrations/20260207110613_add_soft_budget_to_deleted_teams_table/migration.sql) |
| `LiteLLM_MCPServerTable` | New Column | Added `available_on_public_internet` boolean for MCP IP-based access control | [PR #20607](https://github.com/BerriAI/litellm/pull/20607) | [Migration](https://github.com/BerriAI/litellm/blob/main/litellm-proxy-extras/litellm_proxy_extras/migrations/20260207093506_add_available_on_public_internet_to_mcp_servers/migration.sql) |
---
## Documentation Updates (14 updates)
- Add FAQ for setting up and verifying LITELLM_LICENSE - [PR #20284](https://github.com/BerriAI/litellm/pull/20284)
- Model request tags documentation - [PR #20290](https://github.com/BerriAI/litellm/pull/20290)
- Add Prisma migration troubleshooting guide - [PR #20300](https://github.com/BerriAI/litellm/pull/20300)
- MCP Semantic Filtering documentation - [PR #20316](https://github.com/BerriAI/litellm/pull/20316)
- Add CopilotKit SDK doc as supported agents SDK - [PR #20396](https://github.com/BerriAI/litellm/pull/20396)
- Add documentation for Nova Sonic - [PR #20320](https://github.com/BerriAI/litellm/pull/20320)
- Update Vertex AI Text to Speech doc to show use of audio - [PR #20255](https://github.com/BerriAI/litellm/pull/20255)
- Improve Okta SSO setup guide with step-by-step instructions - [PR #20353](https://github.com/BerriAI/litellm/pull/20353)
- Langfuse doc update - [PR #20443](https://github.com/BerriAI/litellm/pull/20443)
- Expose MCPs on public internet documentation - [PR #20626](https://github.com/BerriAI/litellm/pull/20626)
- Add blog post: Achieving Sub-Millisecond Proxy Overhead - [PR #20309](https://github.com/BerriAI/litellm/pull/20309)
- Add blog post about litellm-observatory - [PR #20622](https://github.com/BerriAI/litellm/pull/20622)
- Update Opus 4.6 blog with adaptive thinking - [PR #20637](https://github.com/BerriAI/litellm/pull/20637)
- `gpt-5-search-api` docs clarifications - [PR #20512](https://github.com/BerriAI/litellm/pull/20512)
---
## New Contributors
* @Quentin-M made their first contribution in [PR #19818](https://github.com/BerriAI/litellm/pull/19818)
* @amirzaushnizer made their first contribution in [PR #20235](https://github.com/BerriAI/litellm/pull/20235)
* @cscguochang made their first contribution in [PR #20214](https://github.com/BerriAI/litellm/pull/20214)
* @krauckbot made their first contribution in [PR #20273](https://github.com/BerriAI/litellm/pull/20273)
* @agrattan0820 made their first contribution in [PR #19784](https://github.com/BerriAI/litellm/pull/19784)
* @nina-hu made their first contribution in [PR #20472](https://github.com/BerriAI/litellm/pull/20472)
* @swayambhu94 made their first contribution in [PR #20469](https://github.com/BerriAI/litellm/pull/20469)
* @ssadedin made their first contribution in [PR #20566](https://github.com/BerriAI/litellm/pull/20566)
---
## Full Changelog
[v1.81.6-nightly...v1.81.9](https://github.com/BerriAI/litellm/compare/v1.81.6-nightly...v1.81.9)

52
docs/my-website/sidebars.js
View file

@ -96,6 +96,11 @@ const sidebars = {
"proxy/prometheus"
]
},
{
type: "doc",
id: "integrations/websearch_interception",
label: "Web Search Integration"
},
{
type: "category",
label: "[Beta] Prompt Management",
@ -125,10 +130,12 @@ const sidebars = {
"tutorials/claude_responses_api",
"tutorials/claude_code_max_subscription",
"tutorials/claude_code_customer_tracking",
"tutorials/claude_code_prompt_cache_routing",
"tutorials/claude_code_websearch",
"tutorials/claude_mcp",
"tutorials/claude_non_anthropic_models",
"tutorials/claude_code_plugin_marketplace",
"tutorials/claude_code_beta_headers",
]
},
"tutorials/opencode_integration",
@ -222,6 +229,7 @@ const sidebars = {
label: "Configuration",
items: [
"set_keys",
"proxy_auth",
"caching/all_caches",
],
},
@ -286,40 +294,52 @@ const sidebars = {
label: "Admin UI",
items: [
"proxy/ui",
"proxy/admin_ui_sso",
"proxy/custom_root_ui",
"proxy/custom_sso",
"proxy/ai_hub",
"proxy/model_compare_ui",
"proxy/ui_credentials",
"tutorials/scim_litellm",
{
type: "category",
label: "UI User/Team Management",
label: "Setup & SSO",
items: [
"proxy/access_control",
"proxy/public_teams",
"proxy/admin_ui_sso",
"proxy/custom_sso",
"proxy/custom_root_ui",
"tutorials/scim_litellm",
]
},
{
type: "category",
label: "Models",
items: [
"proxy/ui_credentials",
"proxy/ai_hub",
"proxy/model_compare_ui",
]
},
{
type: "category",
label: "Teams & Organizations",
items: [
"proxy/access_control",
"proxy/self_serve",
"proxy/public_teams",
"proxy/ui/bulk_edit_users",
"proxy/ui/page_visibility",
]
},
{
type: "category",
label: "UI Usage Tracking",
label: "Observability: Usage",
items: [
"proxy/customer_usage",
"proxy/endpoint_activity"
"proxy/endpoint_activity",
]
},
{
type: "category",
label: "UI Logs",
label: "Logs",
items: [
"proxy/ui_logs",
"proxy/ui_spend_log_settings",
"proxy/ui_logs_sessions",
"proxy/deleted_keys_teams"
"proxy/deleted_keys_teams",
]
}
],
@ -367,6 +387,7 @@ const sidebars = {
items: [
"proxy/users",
"proxy/team_budgets",
"proxy/ui_team_soft_budget_alerts",
"proxy/tag_budgets",
"proxy/customers",
"proxy/dynamic_rate_limit",
@ -542,6 +563,8 @@ const sidebars = {
items: [
"mcp",
"mcp_usage",
"mcp_oauth",
"mcp_public_internet",
"mcp_semantic_filter",
"mcp_control",
"mcp_cost",
@ -1059,6 +1082,7 @@ const sidebars = {
"troubleshoot/cpu_issues",
"troubleshoot/memory_issues",
"troubleshoot/spend_queue_warnings",
"troubleshoot/max_callbacks",
],
},
],

133
docs/my-website/src/components/MiddlewareDiagrams/BaseHTTPMiddlewareAnimation.tsx Normal file
View file

@ -0,0 +1,133 @@
import React, { useState, useEffect, useCallback, useRef } from 'react';
import styles from './styles.module.css';
interface Stage {
label: string;
subtitle: string;
code: string;
}
const STAGES: Stage[] = [
{
label: 'Request Wrapping',
subtitle: '_CachedRequest',
code: 'request = _CachedRequest(scope, receive)',
},
{
label: 'Sync Event',
subtitle: 'anyio.Event()',
code: 'response_sent = anyio.Event()',
},
{
label: 'Memory Stream',
subtitle: 'create_memory_object_stream()',
code: 'send_stream, recv_stream = anyio.create_memory_object_stream()',
},
{
label: 'Task Group',
subtitle: 'create_task_group()',
code: 'async with anyio.create_task_group() as task_group:',
},
{
label: 'Background Task',
subtitle: 'task_group.start_soon(coro)',
code: 'task_group.start_soon(coro) # app runs in separate task',
},
{
label: 'Nested Task Group',
subtitle: 'receive_or_disconnect()',
code: 'async with anyio.create_task_group() as task_group: ...',
},
{
label: 'Response Wrapping',
subtitle: '_StreamingResponse',
code: 'response = _StreamingResponse(status_code=..., content=body_stream())',
},
];
const INTERVAL_MS = 1200;
const PAUSE_MS = 600;
export default function BaseHTTPMiddlewareAnimation() {
const [activeStage, setActiveStage] = useState(0);
const [paused, setPaused] = useState(false);
const [expandedStage, setExpandedStage] = useState<number | null>(null);
const timerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const clearTimer = useCallback(() => {
if (timerRef.current !== null) {
clearTimeout(timerRef.current);
timerRef.current = null;
}
}, []);
useEffect(() => {
if (paused) return;
const advance = () => {
setActiveStage((prev) => {
const next = (prev + 1) % STAGES.length;
// If wrapping around, add extra pause
if (next === 0) {
timerRef.current = setTimeout(() => {
timerRef.current = setTimeout(advance, INTERVAL_MS);
}, PAUSE_MS);
return next;
}
timerRef.current = setTimeout(advance, INTERVAL_MS);
return next;
});
};
timerRef.current = setTimeout(advance, INTERVAL_MS);
return clearTimer;
}, [paused, clearTimer]);
const handleStageClick = (index: number) => {
clearTimer();
setPaused(true);
setActiveStage(index);
if (expandedStage === index) {
// Close panel and resume
setExpandedStage(null);
setPaused(false);
} else {
setExpandedStage(index);
}
};
return (
<div className={styles.pipelineWrapper}>
<div className={styles.pipelineLabel}>7 steps per request</div>
<div className={styles.pipeline}>
{STAGES.map((stage, i) => (
<div className={styles.stageWrapper} key={i}>
<div
className={`${styles.stage} ${activeStage === i ? styles.stageActive : ''}`}
onClick={() => handleStageClick(i)}
role="button"
tabIndex={0}
onKeyDown={(e) => {
if (e.key === 'Enter' || e.key === ' ') handleStageClick(i);
}}
>
<div className={styles.stageNumber}>{i + 1}</div>
<div className={styles.stageLabel}>{stage.label}</div>
<div className={styles.stageSubtitle}>{stage.subtitle}</div>
</div>
</div>
))}
</div>
<div
className={`${styles.codePanel} ${expandedStage !== null ? styles.codePanelOpen : ''}`}
>
{expandedStage !== null && (
<pre className={styles.codePanelCode}>
<code>{STAGES[expandedStage].code}</code>
</pre>
)}
</div>
</div>
);
}

337
docs/my-website/src/components/MiddlewareDiagrams/BenchmarkVisualization.tsx Normal file
View file

@ -0,0 +1,337 @@
import React, { useState, useEffect, useRef, useCallback } from 'react';
import styles from './styles.module.css';
/* ── Constants ── */
const TOTAL_REQUESTS = 50_000;
const DURATION_AFTER_MS = 8_000; // "After" column finishes in 8s
const DURATION_BEFORE_MS = 13_920; // 74% slower → 8000 * 1.74
const TICK_MS = 50;
const RESET_PAUSE_MS = 2_000;
const MAX_DOTS = 14;
const BEFORE_RPS = 3_785;
const AFTER_RPS = 6_577;
const BEFORE_P50 = 21;
const AFTER_P50 = 13;
const BEFORE_LAYERS = [
{ label: 'ab client', warning: false },
{ label: 'uvicorn \u00B7 1 worker', warning: false },
{ label: 'ASGI Middleware', warning: false },
{ label: 'BaseHTTPMiddleware', warning: true },
{ label: 'GET /health \u2192 "ok"', warning: false },
];
const AFTER_LAYERS = [
{ label: 'ab client', warning: false },
{ label: 'uvicorn \u00B7 1 worker', warning: false },
{ label: 'ASGI Middleware', warning: false },
{ label: 'ASGI Middleware', warning: false },
{ label: 'GET /health \u2192 "ok"', warning: false },
];
const BENCHMARK_RUNS = [
{ config: 'Before (1 ASGI + 1 BaseHTTP)', run: 1, rps: 3596, p50: 21 },
{ config: 'Before (1 ASGI + 1 BaseHTTP)', run: 2, rps: 3599, p50: 21 },
{ config: 'Before (1 ASGI + 1 BaseHTTP)', run: 3, rps: 4161, p50: 21 },
{ config: 'After (2x Pure ASGI)', run: 1, rps: 6504, p50: 13 },
{ config: 'After (2x Pure ASGI)', run: 2, rps: 6631, p50: 13 },
{ config: 'After (2x Pure ASGI)', run: 3, rps: 6595, p50: 13 },
];
/* ── Dot type ── */
interface Dot {
id: number;
progress: number; // 0..1 (top to bottom)
}
/* ── Component ── */
export default function BenchmarkVisualization() {
const [elapsed, setElapsed] = useState(0);
const [running, setRunning] = useState(false);
const [afterDone, setAfterDone] = useState(false);
const [beforeDone, setBeforeDone] = useState(false);
const [tableOpen, setTableOpen] = useState(false);
const [beforeDots, setBeforeDots] = useState<Dot[]>([]);
const [afterDots, setAfterDots] = useState<Dot[]>([]);
const dotIdRef = useRef(0);
const observerRef = useRef<IntersectionObserver | null>(null);
const wrapperRef = useRef<HTMLDivElement | null>(null);
const timerRef = useRef<ReturnType<typeof setInterval> | null>(null);
const hasStartedRef = useRef(false);
const beforeProgress = Math.min(elapsed / DURATION_BEFORE_MS, 1);
const afterProgress = Math.min(elapsed / DURATION_AFTER_MS, 1);
const beforeCompleted = Math.round(beforeProgress * TOTAL_REQUESTS);
const afterCompleted = Math.round(afterProgress * TOTAL_REQUESTS);
const beforeCurrentRPS = running && !beforeDone
? Math.round(BEFORE_RPS * (0.9 + Math.random() * 0.2))
: beforeDone ? 0 : 0;
const afterCurrentRPS = running && !afterDone
? Math.round(AFTER_RPS * (0.9 + Math.random() * 0.2))
: afterDone ? 0 : 0;
const reset = useCallback(() => {
setElapsed(0);
setAfterDone(false);
setBeforeDone(false);
setBeforeDots([]);
setAfterDots([]);
dotIdRef.current = 0;
}, []);
// Start/restart loop
const startSimulation = useCallback(() => {
reset();
setRunning(true);
}, [reset]);
// IntersectionObserver to auto-start on scroll
useEffect(() => {
observerRef.current = new IntersectionObserver(
([entry]) => {
if (entry.isIntersecting && !hasStartedRef.current) {
hasStartedRef.current = true;
startSimulation();
}
},
{ threshold: 0.3 }
);
if (wrapperRef.current) {
observerRef.current.observe(wrapperRef.current);
}
return () => {
observerRef.current?.disconnect();
};
}, [startSimulation]);
// Main tick
useEffect(() => {
if (!running) return;
timerRef.current = setInterval(() => {
setElapsed((prev) => {
const next = prev + TICK_MS;
if (next >= DURATION_AFTER_MS) setAfterDone(true);
if (next >= DURATION_BEFORE_MS) setBeforeDone(true);
// Both done → schedule reset
if (next >= DURATION_BEFORE_MS) {
setTimeout(() => {
startSimulation();
}, RESET_PAUSE_MS);
setRunning(false);
return next;
}
return next;
});
}, TICK_MS);
return () => {
if (timerRef.current) clearInterval(timerRef.current);
};
}, [running, startSimulation]);
// Dot animation
useEffect(() => {
if (!running) return;
const dotInterval = setInterval(() => {
const spawnBefore = !beforeDone && Math.random() < 0.4;
const spawnAfter = !afterDone && Math.random() < 0.65;
if (spawnBefore) {
setBeforeDots((prev) => {
const dots = [...prev, { id: dotIdRef.current++, progress: 0 }];
return dots.slice(-MAX_DOTS);
});
}
if (spawnAfter) {
setAfterDots((prev) => {
const dots = [...prev, { id: dotIdRef.current++, progress: 0 }];
return dots.slice(-MAX_DOTS);
});
}
// Advance existing dots
setBeforeDots((prev) =>
prev
.map((d) => ({ ...d, progress: d.progress + 0.08 }))
.filter((d) => d.progress <= 1)
);
setAfterDots((prev) =>
prev
.map((d) => ({ ...d, progress: d.progress + 0.14 }))
.filter((d) => d.progress <= 1)
);
}, 100);
return () => clearInterval(dotInterval);
}, [running, beforeDone, afterDone]);
const renderFlowStack = (
layers: { label: string; warning: boolean }[],
dots: Dot[],
isBefore: boolean
) => (
<div className={styles.flowStack}>
<div className={styles.dotsCanvas}>
{dots.map((dot) => (
<div
key={dot.id}
className={`${styles.dot} ${isBefore ? styles.dotSlow : styles.dotFast}`}
style={{
top: `${dot.progress * 92}%`,
left: `${48 + Math.sin(dot.id * 1.7) * 12}%`,
opacity: dot.progress > 0.85 ? (1 - dot.progress) * 6 : 0.8,
}}
/>
))}
</div>
{layers.map((layer, i) => (
<React.Fragment key={i}>
{i > 0 && <div className={styles.flowArrow}>&darr;</div>}
<div
className={`${styles.flowLayer} ${layer.warning ? styles.flowLayerWarning : ''}`}
>
{layer.label}
{layer.warning && <span className={styles.overheadTag}>&larr; overhead</span>}
</div>
</React.Fragment>
))}
</div>
);
const formatNum = (n: number) => n.toLocaleString();
return (
<div className={styles.benchmarkWrapper} ref={wrapperRef}>
<div className={styles.benchmarkConfig}>
50,000 requests &middot; 1,000 concurrent &middot; 1 worker
</div>
<div className={styles.benchmarkColumns}>
{/* Before column */}
<div className={styles.benchmarkColumn}>
<div className={`${styles.columnTitle} ${styles.columnTitleBefore}`}>
Before (1 ASGI + 1 BaseHTTP)
{beforeDone && (
<span className={`${styles.doneBadge} ${styles.doneBadgeBefore}`}>done</span>
)}
</div>
{renderFlowStack(BEFORE_LAYERS, beforeDots, true)}
<div className={styles.statsRow}>
<div className={styles.stat}>
<div className={styles.statValue}>{formatNum(beforeCurrentRPS)}</div>
<div className={styles.statLabel}>RPS</div>
</div>
<div className={styles.stat}>
<div className={styles.statValue}>{formatNum(beforeCompleted)}</div>
<div className={styles.statLabel}>Completed</div>
</div>
<div className={styles.stat}>
<div className={styles.statValue}>{BEFORE_P50}ms</div>
<div className={styles.statLabel}>P50</div>
</div>
</div>
<div className={styles.progressBar}>
<div
className={`${styles.progressFill} ${styles.progressFillBefore}`}
style={{ width: `${beforeProgress * 100}%` }}
/>
</div>
</div>
{/* After column */}
<div className={styles.benchmarkColumn}>
<div className={`${styles.columnTitle} ${styles.columnTitleAfter}`}>
After (2x Pure ASGI)
{afterDone && (
<span className={`${styles.doneBadge} ${styles.doneBadgeAfter}`}>done</span>
)}
</div>
{renderFlowStack(AFTER_LAYERS, afterDots, false)}
<div className={styles.statsRow}>
<div className={styles.stat}>
<div className={styles.statValue}>{formatNum(afterCurrentRPS)}</div>
<div className={styles.statLabel}>RPS</div>
</div>
<div className={styles.stat}>
<div className={styles.statValue}>{formatNum(afterCompleted)}</div>
<div className={styles.statLabel}>Completed</div>
</div>
<div className={styles.stat}>
<div className={styles.statValue}>{AFTER_P50}ms</div>
<div className={styles.statLabel}>P50</div>
</div>
</div>
<div className={styles.progressBar}>
<div
className={`${styles.progressFill} ${styles.progressFillAfter}`}
style={{ width: `${afterProgress * 100}%` }}
/>
</div>
</div>
</div>
{/* Summary stats */}
<div className={styles.summaryStats}>
<div className={styles.summaryItem}>
<div className={styles.summaryValue}>+74%</div>
<div className={styles.summaryLabel}>Throughput (RPS)</div>
</div>
<div className={styles.summaryItem}>
<div className={styles.summaryValue}>-38%</div>
<div className={styles.summaryLabel}>Median Latency (P50)</div>
</div>
</div>
{/* Collapsible per-run data */}
<div className={styles.collapsible}>
<button
className={styles.collapsibleToggle}
onClick={() => setTableOpen(!tableOpen)}
>
<span
className={`${styles.collapsibleChevron} ${
tableOpen ? styles.collapsibleChevronOpen : ''
}`}
>
&#9654;
</span>
Per-run data (3 runs each)
</button>
<div
className={`${styles.collapsibleContent} ${
tableOpen ? styles.collapsibleContentOpen : ''
}`}
>
<table className={styles.dataTable}>
<thead>
<tr>
<th>Config</th>
<th>Run</th>
<th>RPS</th>
<th>P50 (ms)</th>
</tr>
</thead>
<tbody>
{BENCHMARK_RUNS.map((row, i) => (
<tr key={i}>
<td>{row.config}</td>
<td>{row.run}</td>
<td>{formatNum(row.rps)}</td>
<td>{row.p50}</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
</div>
);
}

67
docs/my-website/src/components/MiddlewareDiagrams/PureASGIAnimation.tsx Normal file
View file

@ -0,0 +1,67 @@
import React, { useState, useEffect, useRef, useCallback } from 'react';
import styles from './styles.module.css';
interface Stage {
label: string;
subtitle: string;
}
const STAGES: Stage[] = [
{ label: 'Scope Check', subtitle: 'scope["type"] != "http"' },
{ label: 'Direct Call', subtitle: 'await self.app(scope, receive, send)' },
];
const INTERVAL_MS = 1200;
const PAUSE_MS = 600;
export default function PureASGIAnimation() {
const [activeStage, setActiveStage] = useState(0);
const timerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const clearTimer = useCallback(() => {
if (timerRef.current !== null) {
clearTimeout(timerRef.current);
timerRef.current = null;
}
}, []);
useEffect(() => {
const advance = () => {
setActiveStage((prev) => {
const next = (prev + 1) % STAGES.length;
if (next === 0) {
timerRef.current = setTimeout(() => {
timerRef.current = setTimeout(advance, INTERVAL_MS);
}, PAUSE_MS);
return next;
}
timerRef.current = setTimeout(advance, INTERVAL_MS);
return next;
});
};
timerRef.current = setTimeout(advance, INTERVAL_MS);
return clearTimer;
}, [clearTimer]);
return (
<div className={styles.pipelineWrapper}>
<div className={styles.pipelineLabel}>2 steps per request</div>
<div className={`${styles.pipeline} ${styles.pipelineTwoCol}`}>
{STAGES.map((stage, i) => (
<div className={styles.stageWrapper} key={i}>
<div
className={`${styles.stage} ${styles.stageNoClick} ${
activeStage === i ? styles.stageActiveGreen : ''
}`}
>
<div className={styles.stageNumber}>{i + 1}</div>
<div className={styles.stageLabel}>{stage.label}</div>
<div className={styles.stageSubtitle}>{stage.subtitle}</div>
</div>
</div>
))}
</div>
</div>
);
}

3
docs/my-website/src/components/MiddlewareDiagrams/index.tsx Normal file
View file

@ -0,0 +1,3 @@
export { default as BaseHTTPMiddlewareAnimation } from './BaseHTTPMiddlewareAnimation';
export { default as PureASGIAnimation } from './PureASGIAnimation';
export { default as BenchmarkVisualization } from './BenchmarkVisualization';

494
docs/my-website/src/components/MiddlewareDiagrams/styles.module.css Normal file
View file

@ -0,0 +1,494 @@
/* ── Shared custom properties ── */
:root {
--mw-stage-bg: #f8f9fa;
--mw-stage-border: #dee2e6;
--mw-stage-active-bg: #e8f4fd;
--mw-stage-active-border: #3b82f6;
--mw-stage-green-active-bg: #ecfdf5;
--mw-stage-green-active-border: #10b981;
--mw-dot-color: #3b82f6;
--mw-warning-accent: #ef4444;
--mw-success-accent: #10b981;
--mw-text-primary: #1a1a2e;
--mw-text-secondary: #6b7280;
--mw-code-bg: #f1f5f9;
--mw-panel-bg: #ffffff;
--mw-panel-border: #e5e7eb;
--mw-bar-bg: #e5e7eb;
--mw-arrow-color: #9ca3af;
--mw-column-bg: #fafafa;
--mw-column-border: #e5e7eb;
--mw-layer-bg: #f3f4f6;
--mw-layer-border: #d1d5db;
--mw-layer-warning-bg: #fef2f2;
--mw-layer-warning-border: #fca5a5;
--mw-progress-bg: #e5e7eb;
}
[data-theme='dark'] {
--mw-stage-bg: #1e1e2e;
--mw-stage-border: #374151;
--mw-stage-active-bg: #1e3a5f;
--mw-stage-active-border: #60a5fa;
--mw-stage-green-active-bg: #064e3b;
--mw-stage-green-active-border: #34d399;
--mw-dot-color: #60a5fa;
--mw-warning-accent: #f87171;
--mw-success-accent: #34d399;
--mw-text-primary: #e5e7eb;
--mw-text-secondary: #9ca3af;
--mw-code-bg: #1e293b;
--mw-panel-bg: #111827;
--mw-panel-border: #374151;
--mw-bar-bg: #374151;
--mw-arrow-color: #6b7280;
--mw-column-bg: #111827;
--mw-column-border: #374151;
--mw-layer-bg: #1f2937;
--mw-layer-border: #4b5563;
--mw-layer-warning-bg: #451a1a;
--mw-layer-warning-border: #b91c1c;
--mw-progress-bg: #374151;
}
/* ── Pipeline (shared between BaseHTTP and PureASGI) ── */
.pipelineWrapper {
margin: 1.5rem 0;
}
.pipelineLabel {
text-align: center;
font-size: 0.85rem;
font-weight: 600;
color: var(--mw-text-secondary);
margin-bottom: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.05em;
}
.pipeline {
display: flex;
flex-wrap: wrap;
justify-content: center;
align-items: stretch;
gap: 0.75rem;
padding: 0.5rem 0;
}
.pipelineTwoCol {
max-width: 480px;
margin: 0 auto;
}
.stageWrapper {
display: flex;
align-items: center;
width: 160px;
flex-shrink: 0;
}
.pipelineTwoCol .stageWrapper {
width: 200px;
}
.arrow {
display: none;
}
.stage {
flex: 1;
padding: 0.85rem 0.75rem;
min-height: 100px;
display: flex;
flex-direction: column;
justify-content: center;
background: var(--mw-stage-bg);
border: 2px solid var(--mw-stage-border);
border-radius: 8px;
text-align: center;
cursor: pointer;
transition: background 0.4s ease, border-color 0.4s ease, box-shadow 0.4s ease;
user-select: none;
}
.stage:hover {
border-color: var(--mw-stage-active-border);
}
.stageActive {
background: var(--mw-stage-active-bg);
border-color: var(--mw-stage-active-border);
box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.15);
}
.stageActiveGreen {
background: var(--mw-stage-green-active-bg);
border-color: var(--mw-stage-green-active-border);
box-shadow: 0 0 0 3px rgba(16, 185, 129, 0.15);
}
.stageNoClick {
cursor: default;
}
.stageNumber {
font-size: 0.7rem;
font-weight: 700;
color: var(--mw-text-secondary);
margin-bottom: 0.3rem;
}
.stageLabel {
font-size: 0.85rem;
font-weight: 600;
color: var(--mw-text-primary);
margin-bottom: 0.25rem;
line-height: 1.3;
}
.stageSubtitle {
font-size: 0.72rem;
color: var(--mw-text-secondary);
font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, monospace;
word-break: break-word;
line-height: 1.3;
}
/* ── Code panel (accordion) ── */
.codePanel {
max-height: 0;
overflow: hidden;
transition: max-height 0.35s ease, padding 0.35s ease;
background: var(--mw-code-bg);
border-radius: 0 0 8px 8px;
margin-top: 0.5rem;
}
.codePanelOpen {
max-height: 120px;
padding: 0.75rem 1rem;
}
.codePanelCode {
font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, monospace;
font-size: 0.8rem;
color: var(--mw-text-primary);
white-space: pre;
margin: 0;
line-height: 1.5;
}
/* ── Benchmark Visualization ── */
.benchmarkWrapper {
margin: 1.5rem 0;
}
.benchmarkConfig {
text-align: center;
font-size: 0.85rem;
color: var(--mw-text-secondary);
margin-bottom: 1rem;
font-weight: 500;
}
.benchmarkColumns {
display: flex;
gap: 1.5rem;
}
.benchmarkColumn {
flex: 1;
background: var(--mw-column-bg);
border: 1px solid var(--mw-column-border);
border-radius: 12px;
padding: 1.25rem;
position: relative;
overflow: hidden;
}
.columnTitle {
font-size: 0.9rem;
font-weight: 700;
color: var(--mw-text-primary);
text-align: center;
margin-bottom: 1rem;
}
.columnTitleBefore {
color: var(--mw-warning-accent);
}
.columnTitleAfter {
color: var(--mw-success-accent);
}
/* ── Request flow stack ── */
.flowStack {
display: flex;
flex-direction: column;
align-items: center;
gap: 0;
position: relative;
min-height: 280px;
}
.flowLayer {
width: 100%;
max-width: 260px;
padding: 0.6rem 0.75rem;
background: var(--mw-layer-bg);
border: 1px solid var(--mw-layer-border);
border-radius: 6px;
text-align: center;
font-size: 0.78rem;
font-weight: 500;
color: var(--mw-text-primary);
position: relative;
z-index: 1;
}
.flowLayerWarning {
background: var(--mw-layer-warning-bg);
border-color: var(--mw-layer-warning-border);
font-weight: 700;
}
.flowArrow {
display: flex;
justify-content: center;
color: var(--mw-arrow-color);
font-size: 0.9rem;
padding: 0.15rem 0;
position: relative;
z-index: 0;
min-height: 20px;
}
.overheadTag {
font-size: 0.65rem;
color: var(--mw-warning-accent);
margin-left: 0.4rem;
}
/* ── Dots layer (canvas for flowing dots) ── */
.dotsCanvas {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
pointer-events: none;
z-index: 2;
}
.dot {
position: absolute;
width: 6px;
height: 6px;
border-radius: 50%;
background: var(--mw-dot-color);
opacity: 0.8;
}
.dotSlow {
background: var(--mw-warning-accent);
}
.dotFast {
background: var(--mw-success-accent);
}
/* ── Stats & progress ── */
.statsRow {
display: flex;
justify-content: space-around;
margin-top: 1rem;
padding-top: 0.75rem;
border-top: 1px solid var(--mw-panel-border);
}
.stat {
text-align: center;
}
.statValue {
font-size: 1.1rem;
font-weight: 700;
color: var(--mw-text-primary);
font-variant-numeric: tabular-nums;
}
.statLabel {
font-size: 0.7rem;
color: var(--mw-text-secondary);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.progressBar {
width: 100%;
height: 6px;
background: var(--mw-progress-bg);
border-radius: 3px;
margin-top: 0.75rem;
overflow: hidden;
}
.progressFill {
height: 100%;
border-radius: 3px;
transition: width 0.1s linear;
}
.progressFillBefore {
background: var(--mw-warning-accent);
}
.progressFillAfter {
background: var(--mw-success-accent);
}
/* ── Summary stats below simulation ── */
.summaryStats {
display: flex;
justify-content: center;
gap: 2rem;
margin-top: 1.5rem;
flex-wrap: wrap;
}
.summaryItem {
text-align: center;
padding: 0.75rem 1.25rem;
background: var(--mw-stage-bg);
border-radius: 8px;
border: 1px solid var(--mw-panel-border);
}
.summaryValue {
font-size: 1.5rem;
font-weight: 800;
color: var(--mw-success-accent);
}
.summaryLabel {
font-size: 0.8rem;
color: var(--mw-text-secondary);
margin-top: 0.2rem;
}
/* ── Collapsible table ── */
.collapsible {
margin-top: 1.5rem;
}
.collapsibleToggle {
background: none;
border: 1px solid var(--mw-panel-border);
border-radius: 6px;
padding: 0.5rem 1rem;
cursor: pointer;
font-size: 0.85rem;
color: var(--mw-text-primary);
width: 100%;
text-align: left;
display: flex;
align-items: center;
gap: 0.5rem;
transition: background 0.2s;
}
.collapsibleToggle:hover {
background: var(--mw-stage-bg);
}
.collapsibleChevron {
transition: transform 0.3s ease;
font-size: 0.7rem;
}
.collapsibleChevronOpen {
transform: rotate(90deg);
}
.collapsibleContent {
max-height: 0;
overflow: hidden;
transition: max-height 0.35s ease;
}
.collapsibleContentOpen {
max-height: 600px;
}
.dataTable {
width: 100%;
border-collapse: collapse;
margin-top: 0.75rem;
font-size: 0.85rem;
}
.dataTable th,
.dataTable td {
padding: 0.5rem 0.75rem;
text-align: left;
border-bottom: 1px solid var(--mw-panel-border);
}
.dataTable th {
font-weight: 600;
color: var(--mw-text-secondary);
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.04em;
}
.dataTable td {
color: var(--mw-text-primary);
font-variant-numeric: tabular-nums;
}
/* ── Reproduce section ── */
.reproduceSection {
margin-top: 1rem;
}
/* ── Done badge ── */
.doneBadge {
display: inline-block;
font-size: 0.75rem;
font-weight: 600;
padding: 0.2rem 0.6rem;
border-radius: 4px;
margin-left: 0.5rem;
}
.doneBadgeBefore {
color: var(--mw-warning-accent);
background: var(--mw-layer-warning-bg);
}
.doneBadgeAfter {
color: var(--mw-success-accent);
background: var(--mw-stage-green-active-bg);
}
/* ── Responsive ── */
@media (max-width: 768px) {
.stageWrapper {
width: 140px;
}
.pipelineTwoCol .stageWrapper {
width: 160px;
}
.benchmarkColumns {
flex-direction: column;
}
.summaryStats {
flex-direction: column;
align-items: center;
}
}

BIN
enterprise/dist/litellm_enterprise-0.1.30-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.30.tar.gz vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.31-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.31.tar.gz vendored Normal file
View file

Binary file not shown.

119
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
View file

@ -30,8 +30,15 @@ from litellm.integrations.email_templates.user_invitation_email import (
from litellm.integrations.email_templates.templates import (
MAX_BUDGET_ALERT_EMAIL_TEMPLATE,
SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
)
from litellm.proxy._types import (
CallInfo,
InvitationNew,
Litellm_EntityType,
UserAPIKeyAuth,
WebhookEvent,
)
from litellm.proxy._types import CallInfo, InvitationNew, UserAPIKeyAuth, WebhookEvent
from litellm.secret_managers.main import get_secret_bool
from litellm.types.integrations.slack_alerting import LITELLM_LOGO_URL
from litellm.constants import (
@ -217,6 +224,78 @@ class BaseEmailLogger(CustomLogger):
)
pass
async def send_team_soft_budget_alert_email(self, event: WebhookEvent):
"""
Send email to team members when team soft budget is crossed
Supports multiple recipients via alert_emails field from team metadata
"""
# Collect all recipient emails
recipient_emails: List[str] = []
# Add additional alert emails from team metadata.soft_budget_alert_emails
if hasattr(event, "alert_emails") and event.alert_emails:
for email in event.alert_emails:
if email and email not in recipient_emails: # Avoid duplicates
recipient_emails.append(email)
# If no recipients found, skip sending
if not recipient_emails:
verbose_proxy_logger.warning(
f"No recipient emails found for team soft budget alert. event={event.model_dump(exclude_none=True)}"
)
return
# Validate that we have at least one valid email address
first_recipient_email = recipient_emails[0]
if not first_recipient_email or not first_recipient_email.strip():
verbose_proxy_logger.warning(
f"Invalid recipient email found for team soft budget alert. event={event.model_dump(exclude_none=True)}"
)
return
verbose_proxy_logger.debug(
f"send_team_soft_budget_alert_email_event: {json.dumps(event.model_dump(exclude_none=True), indent=4, default=str)}"
)
# Get email params using the first recipient email (for template formatting)
# For team alerts with alert_emails, we don't need user_id lookup since we already have email addresses
# Pass user_id=None to prevent _get_email_params from trying to look up email from a potentially None user_id
email_params = await self._get_email_params(
email_event=EmailEvent.soft_budget_crossed,
user_id=None, # Team alerts don't require user_id when alert_emails are provided
user_email=first_recipient_email,
event_message=event.event_message,
)
# Format budget values
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_info = ""
if event.max_budget is not None:
max_budget_info = f"<b>Maximum Budget:</b> ${event.max_budget} <br />"
# Use team alias or generic greeting
team_alias = event.team_alias or "Team"
email_html_content = TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
team_alias=team_alias,
soft_budget=soft_budget_str,
spend=spend_str,
max_budget_info=max_budget_info,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
)
# Send email to all recipients
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
to_email=recipient_emails,
subject=email_params.subject,
html_body=email_html_content,
)
pass
async def send_max_budget_alert_email(self, event: WebhookEvent):
"""
Send email to user when max budget alert threshold is reached
@ -285,15 +364,36 @@ class BaseEmailLogger(CustomLogger):
# - Don't re-alert, if alert already sent
_cache: DualCache = self.internal_usage_cache
# percent of max_budget left to spend
if user_info.max_budget is None and user_info.soft_budget is None:
return
# For soft_budget alerts, check if we've already sent an alert
if type == "soft_budget":
# For team soft budget alerts, we only need team soft_budget to be set
# For other entity types, we need either max_budget or soft_budget
if user_info.event_group == Litellm_EntityType.TEAM:
if user_info.soft_budget is None:
return
# For team soft budget alerts, require alert_emails to be configured
# Team soft budget alerts are sent via metadata.soft_budget_alerting_emails
if user_info.alert_emails is None or len(user_info.alert_emails) == 0:
verbose_proxy_logger.debug(
"Skipping team soft budget email alert: no alert_emails configured",
)
return
else:
# For non-team alerts, require either max_budget or soft_budget
if user_info.max_budget is None and user_info.soft_budget is None:
return
if user_info.soft_budget is not None and user_info.spend >= user_info.soft_budget:
# Generate cache key based on event type and identifier
_id = user_info.token or user_info.user_id or "default_id"
# Use appropriate ID based on event_group to ensure unique cache keys per entity type
if user_info.event_group == Litellm_EntityType.TEAM:
_id = user_info.team_id or "default_id"
elif user_info.event_group == Litellm_EntityType.ORGANIZATION:
_id = user_info.organization_id or "default_id"
elif user_info.event_group == Litellm_EntityType.USER:
_id = user_info.user_id or "default_id"
else:
# For KEY and other types, use token or user_id
_id = user_info.token or user_info.user_id or "default_id"
_cache_key = f"email_budget_alerts:soft_budget_crossed:{_id}"
# Check if we've already sent this alert
@ -318,10 +418,15 @@ class BaseEmailLogger(CustomLogger):
projected_exceeded_date=user_info.projected_exceeded_date,
projected_spend=user_info.projected_spend,
event_group=user_info.event_group,
alert_emails=user_info.alert_emails,
)
try:
await self.send_soft_budget_alert_email(webhook_event)
# Use team-specific function for team alerts, otherwise use standard function
if user_info.event_group == Litellm_EntityType.TEAM:
await self.send_team_soft_budget_alert_email(webhook_event)
else:
await self.send_soft_budget_alert_email(webhook_event)
# Cache the alert to prevent duplicate sends
await _cache.async_set_cache(

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-enterprise"
version = "0.1.29"
version = "0.1.31"
description = "Package for LiteLLM Enterprise features"
authors = ["BerriAI"]
readme = "README.md"
@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "0.1.29"
version = "0.1.31"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-enterprise==",

4
litellm-js/spend-logs/package.json
View file

@ -11,6 +11,8 @@
"tsx": "^4.7.1"
},
"overrides": {
"glob": ">=11.1.0"
"glob": ">=11.1.0",
"tar": ">=7.5.7",
"@isaacs/brace-expansion": ">=5.0.1"
}
}

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.32-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.32.tar.gz vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.33-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.33.tar.gz vendored Normal file
View file

Binary file not shown.

3
litellm-proxy-extras/litellm_proxy_extras/migrations/20260207093506_add_available_on_public_internet_to_mcp_servers/migration.sql Normal file
View file

@ -0,0 +1,3 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "available_on_public_internet" BOOLEAN NOT NULL DEFAULT false;

3
litellm-proxy-extras/litellm_proxy_extras/migrations/20260207110613_add_soft_budget_to_deleted_teams_table/migration.sql Normal file
View file

@ -0,0 +1,3 @@
-- AlterTable
ALTER TABLE "LiteLLM_DeletedTeamTable" ADD COLUMN "soft_budget" DOUBLE PRECISION;

8
litellm-proxy-extras/litellm_proxy_extras/migrations/20260209085821_add_verificationtoken_indexes/migration.sql Normal file
View file

@ -0,0 +1,8 @@
-- CreateIndex
CREATE INDEX "LiteLLM_VerificationToken_user_id_team_id_idx" ON "LiteLLM_VerificationToken"("user_id", "team_id");
-- CreateIndex
CREATE INDEX "LiteLLM_VerificationToken_team_id_idx" ON "LiteLLM_VerificationToken"("team_id");
-- CreateIndex
CREATE INDEX "LiteLLM_VerificationToken_budget_reset_at_expires_idx" ON "LiteLLM_VerificationToken"("budget_reset_at", "expires");

12
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -148,6 +148,7 @@ model LiteLLM_DeletedTeamTable {
members_with_roles Json @default("{}")
metadata Json @default("{}")
max_budget Float?
soft_budget Float?
spend Float @default(0.0)
models String[]
max_parallel_requests Int?
@ -263,6 +264,7 @@ model LiteLLM_MCPServerTable {
token_url String?
registration_url String?
allow_all_keys Boolean @default(false)
available_on_public_internet Boolean @default(false)
}
// Generate Tokens for Proxy
@ -308,6 +310,16 @@ model LiteLLM_VerificationToken {
litellm_budget_table LiteLLM_BudgetTable? @relation(fields: [budget_id], references: [budget_id])
litellm_organization_table LiteLLM_OrganizationTable? @relation(fields: [organization_id], references: [organization_id])
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
// SELECT COUNT(*) FROM (SELECT "public"."LiteLLM_VerificationToken"."token" FROM "public"."LiteLLM_VerificationToken" WHERE ("public"."LiteLLM_VerificationToken"."user_id" = $1 AND ("public"."LiteLLM_VerificationToken"."team_id" IS NULL OR "public"."LiteLLM_VerificationToken"."team_id" <> $2)) OFFSET $3 ) AS "sub"
// SELECT ... FROM "public"."LiteLLM_VerificationToken" WHERE "public"."LiteLLM_VerificationToken"."user_id" = $1 OFFSET $2
@@index([user_id, team_id])
// SELECT ... FROM "public"."LiteLLM_VerificationToken" WHERE "public"."LiteLLM_VerificationToken"."team_id" = $1 OFFSET $2
@@index([team_id])
// SELECT ... FROM "public"."LiteLLM_VerificationToken" WHERE (("public"."LiteLLM_VerificationToken"."expires" IS NULL OR "public"."LiteLLM_VerificationToken"."expires" > $1) AND "public"."LiteLLM_VerificationToken"."budget_reset_at" < $2) OFFSET $3
@@index([budget_reset_at, expires])
}
// Audit table for deleted keys - preserves spend and key information for historical tracking

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-proxy-extras"
version = "0.4.31"
version = "0.4.33"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
authors = ["BerriAI"]
readme = "README.md"
@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "0.4.31"
version = "0.4.33"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-proxy-extras==",

96
litellm/_logging.py
View file

@ -1,9 +1,13 @@
import json
import ast
import logging
import os
import sys
from datetime import datetime
from logging import Formatter
from typing import Any, Dict, Optional
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
set_verbose = False
@ -19,6 +23,67 @@ handler = logging.StreamHandler()
handler.setLevel(numeric_level)
def _try_parse_json_message(message: str) -> Optional[Dict[str, Any]]:
"""
Try to parse a log message as JSON. Returns parsed dict if valid, else None.
Handles messages that are entirely valid JSON (e.g. json.dumps output).
Uses shared safe_json_loads for consistent error handling.
"""
if not message or not isinstance(message, str):
return None
msg_stripped = message.strip()
if not (msg_stripped.startswith("{") or msg_stripped.startswith("[")):
return None
parsed = safe_json_loads(message, default=None)
if parsed is None or not isinstance(parsed, dict):
return None
return parsed
def _try_parse_embedded_python_dict(message: str) -> Optional[Dict[str, Any]]:
"""
Try to find and parse a Python dict repr (e.g. str(d) or repr(d)) embedded in
the message. Handles patterns like:
"get_available_deployment for model: X, Selected deployment: {'model_name': '...', ...} for model: X"
Uses ast.literal_eval for safe parsing. Returns the parsed dict or None.
"""
if not message or not isinstance(message, str) or "{" not in message:
return None
i = 0
while i < len(message):
start = message.find("{", i)
if start == -1:
break
depth = 0
for j in range(start, len(message)):
c = message[j]
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
substr = message[start : j + 1]
try:
result = ast.literal_eval(substr)
if isinstance(result, dict) and len(result) > 0:
return result
except (ValueError, SyntaxError, TypeError):
pass
break
i = start + 1
return None
# Standard LogRecord attribute names - used to identify 'extra' fields.
# Derived at runtime so we automatically include version-specific attrs (e.g. taskName).
def _get_standard_record_attrs() -> frozenset:
"""Standard LogRecord attribute names - excludes extra keys from logger.debug(..., extra={...})."""
return frozenset(logging.LogRecord("", 0, "", 0, "", (), None).__dict__.keys())
_STANDARD_RECORD_ATTRS = _get_standard_record_attrs()
class JsonFormatter(Formatter):
def __init__(self):
super(JsonFormatter, self).__init__()
@ -29,16 +94,31 @@ class JsonFormatter(Formatter):
return dt.isoformat()
def format(self, record):
json_record = {
"message": record.getMessage(),
message_str = record.getMessage()
json_record: Dict[str, Any] = {
"message": message_str,
"level": record.levelname,
"timestamp": self.formatTime(record),
}
# Parse embedded JSON or Python dict repr in message so sub-fields become first-class properties
parsed = _try_parse_json_message(message_str)
if parsed is None:
parsed = _try_parse_embedded_python_dict(message_str)
if parsed is not None:
for key, value in parsed.items():
if key not in json_record:
json_record[key] = value
# Include extra attributes passed via logger.debug("msg", extra={...})
for key, value in record.__dict__.items():
if key not in _STANDARD_RECORD_ATTRS and key not in json_record:
json_record[key] = value
if record.exc_info:
json_record["stacktrace"] = self.formatException(record.exc_info)
return json.dumps(json_record)
return safe_dumps(json_record)
# Function to set up exception handlers for JSON logging
@ -169,15 +249,15 @@ def _initialize_loggers_with_handler(handler: logging.Handler):
def _get_uvicorn_json_log_config():
"""
Generate a uvicorn log_config dictionary that applies JSON formatting to all loggers.
This ensures that uvicorn's access logs, error logs, and all application logs
are formatted as JSON when json_logs is enabled.
"""
json_formatter_class = "litellm._logging.JsonFormatter"
# Use the module-level log_level variable for consistency
uvicorn_log_level = log_level.upper()
log_config = {
"version": 1,
"disable_existing_loggers": False,
@ -222,7 +302,7 @@ def _get_uvicorn_json_log_config():
},
},
}
return log_config

14
litellm/a2a_protocol/__init__.py
View file

@ -39,6 +39,12 @@ Example usage (class-based):
"""
from litellm.a2a_protocol.client import A2AClient
from litellm.a2a_protocol.exceptions import (
A2AAgentCardError,
A2AConnectionError,
A2AError,
A2ALocalhostURLError,
)
from litellm.a2a_protocol.main import (
aget_agent_card,
asend_message,
@ -49,11 +55,19 @@ from litellm.a2a_protocol.main import (
from litellm.types.agents import LiteLLMSendMessageResponse
__all__ = [
# Client
"A2AClient",
# Functions
"asend_message",
"send_message",
"asend_message_streaming",
"aget_agent_card",
"create_a2a_client",
# Response types
"LiteLLMSendMessageResponse",
# Exceptions
"A2AError",
"A2AConnectionError",
"A2AAgentCardError",
"A2ALocalhostURLError",
]

67
litellm/a2a_protocol/card_resolver.py
View file

@ -7,6 +7,7 @@ Extends the A2A SDK's card resolver to support multiple well-known paths.
from typing import TYPE_CHECKING, Any, Dict, Optional
from litellm._logging import verbose_logger
from litellm.constants import LOCALHOST_URL_PATTERNS
if TYPE_CHECKING:
from a2a.types import AgentCard
@ -26,15 +27,61 @@ except ImportError:
pass
def is_localhost_or_internal_url(url: Optional[str]) -> bool:
"""
Check if a URL is a localhost or internal URL.
This detects common development URLs that are accidentally left in
agent cards when deploying to production.
Args:
url: The URL to check
Returns:
True if the URL is localhost/internal
"""
if not url:
return False
url_lower = url.lower()
return any(pattern in url_lower for pattern in LOCALHOST_URL_PATTERNS)
def fix_agent_card_url(agent_card: "AgentCard", base_url: str) -> "AgentCard":
"""
Fix the agent card URL if it contains a localhost/internal address.
Many A2A agents are deployed with agent cards that contain internal URLs
like "http://0.0.0.0:8001/" or "http://localhost:8000/". This function
replaces such URLs with the provided base_url.
Args:
agent_card: The agent card to fix
base_url: The base URL to use as replacement
Returns:
The agent card with the URL fixed if necessary
"""
card_url = getattr(agent_card, "url", None)
if card_url and is_localhost_or_internal_url(card_url):
# Normalize base_url to ensure it ends with /
fixed_url = base_url.rstrip("/") + "/"
agent_card.url = fixed_url
return agent_card
class LiteLLMA2ACardResolver(_A2ACardResolver): # type: ignore[misc]
"""
Custom A2A card resolver that supports multiple well-known paths.
Extends the base A2ACardResolver to try both:
- /.well-known/agent-card.json (standard)
- /.well-known/agent.json (previous/alternative)
"""
async def get_agent_card(
self,
relative_card_path: Optional[str] = None,
@ -42,17 +89,17 @@ class LiteLLMA2ACardResolver(_A2ACardResolver): # type: ignore[misc]
) -> "AgentCard":
"""
Fetch the agent card, trying multiple well-known paths.
First tries the standard path, then falls back to the previous path.
Args:
relative_card_path: Optional path to the agent card endpoint.
If None, tries both well-known paths.
http_kwargs: Optional dictionary of keyword arguments to pass to httpx.get
Returns:
AgentCard from the A2A agent
Raises:
A2AClientHTTPError or A2AClientJSONError if both paths fail
"""
@ -62,13 +109,13 @@ class LiteLLMA2ACardResolver(_A2ACardResolver): # type: ignore[misc]
relative_card_path=relative_card_path,
http_kwargs=http_kwargs,
)
# Try both well-known paths
paths = [
AGENT_CARD_WELL_KNOWN_PATH,
PREV_AGENT_CARD_WELL_KNOWN_PATH,
]
last_error = None
for path in paths:
try:
@ -85,11 +132,11 @@ class LiteLLMA2ACardResolver(_A2ACardResolver): # type: ignore[misc]
)
last_error = e
continue
# If we get here, all paths failed - re-raise the last error
if last_error is not None:
raise last_error
# This shouldn't happen, but just in case
raise Exception(
f"Failed to fetch agent card from {self.base_url}. "

203
litellm/a2a_protocol/exception_mapping_utils.py Normal file
View file

@ -0,0 +1,203 @@
"""
A2A Protocol Exception Mapping Utils.
Maps A2A SDK exceptions to LiteLLM A2A exception types.
"""
from typing import TYPE_CHECKING, Any, Optional
from litellm._logging import verbose_logger
from litellm.a2a_protocol.card_resolver import (
fix_agent_card_url,
is_localhost_or_internal_url,
)
from litellm.a2a_protocol.exceptions import (
A2AAgentCardError,
A2AConnectionError,
A2AError,
A2ALocalhostURLError,
)
from litellm.constants import CONNECTION_ERROR_PATTERNS
if TYPE_CHECKING:
from a2a.client import A2AClient as A2AClientType
# Runtime import
A2A_SDK_AVAILABLE = False
try:
from a2a.client import A2AClient as _A2AClient # type: ignore[no-redef]
A2A_SDK_AVAILABLE = True
except ImportError:
_A2AClient = None # type: ignore[assignment, misc]
class A2AExceptionCheckers:
"""
Helper class for checking various A2A error conditions.
"""
@staticmethod
def is_connection_error(error_str: str) -> bool:
"""
Check if an error string indicates a connection error.
Args:
error_str: The error string to check
Returns:
True if the error indicates a connection issue
"""
if not isinstance(error_str, str):
return False
error_str_lower = error_str.lower()
return any(pattern in error_str_lower for pattern in CONNECTION_ERROR_PATTERNS)
@staticmethod
def is_localhost_url(url: Optional[str]) -> bool:
"""
Check if a URL is a localhost/internal URL.
Args:
url: The URL to check
Returns:
True if the URL is localhost/internal
"""
return is_localhost_or_internal_url(url)
@staticmethod
def is_agent_card_error(error_str: str) -> bool:
"""
Check if an error string indicates an agent card error.
Args:
error_str: The error string to check
Returns:
True if the error is related to agent card fetching/parsing
"""
if not isinstance(error_str, str):
return False
error_str_lower = error_str.lower()
agent_card_patterns = [
"agent card",
"agent-card",
".well-known",
"card not found",
"invalid agent",
]
return any(pattern in error_str_lower for pattern in agent_card_patterns)
def map_a2a_exception(
original_exception: Exception,
card_url: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
) -> Exception:
"""
Map an A2A SDK exception to a LiteLLM A2A exception type.
Args:
original_exception: The original exception from the A2A SDK
card_url: The URL from the agent card (if available)
api_base: The original API base URL
model: The model/agent name
Returns:
A mapped LiteLLM A2A exception
Raises:
A2ALocalhostURLError: If the error is a connection error to a localhost URL
A2AConnectionError: If the error is a general connection error
A2AAgentCardError: If the error is related to agent card issues
A2AError: For other A2A-related errors
"""
error_str = str(original_exception)
# Check for localhost URL connection error (special case - retryable)
if (
card_url
and api_base
and A2AExceptionCheckers.is_localhost_url(card_url)
and A2AExceptionCheckers.is_connection_error(error_str)
):
raise A2ALocalhostURLError(
localhost_url=card_url,
base_url=api_base,
original_error=original_exception,
model=model,
)
# Check for agent card errors
if A2AExceptionCheckers.is_agent_card_error(error_str):
raise A2AAgentCardError(
message=error_str,
url=api_base,
model=model,
)
# Check for general connection errors
if A2AExceptionCheckers.is_connection_error(error_str):
raise A2AConnectionError(
message=error_str,
url=card_url or api_base,
model=model,
)
# Default: wrap in generic A2AError
raise A2AError(
message=error_str,
model=model,
)
def handle_a2a_localhost_retry(
error: A2ALocalhostURLError,
agent_card: Any,
a2a_client: "A2AClientType",
is_streaming: bool = False,
) -> "A2AClientType":
"""
Handle A2ALocalhostURLError by fixing the URL and creating a new client.
This is called when we catch an A2ALocalhostURLError and want to retry
with the corrected URL.
Args:
error: The localhost URL error
agent_card: The agent card object to fix
a2a_client: The current A2A client
is_streaming: Whether this is a streaming request (for logging)
Returns:
A new A2A client with the fixed URL
Raises:
ImportError: If the A2A SDK is not installed
"""
if not A2A_SDK_AVAILABLE or _A2AClient is None:
raise ImportError(
"A2A SDK is required for localhost retry handling. "
"Install it with: pip install a2a"
)
request_type = "streaming " if is_streaming else ""
verbose_logger.warning(
f"A2A {request_type}request to '{error.localhost_url}' failed: {error.original_error}. "
f"Agent card contains localhost/internal URL. "
f"Retrying with base_url '{error.base_url}'."
)
# Fix the agent card URL
fix_agent_card_url(agent_card, error.base_url)
# Create a new client with the fixed agent card (transport caches URL)
return _A2AClient(
httpx_client=a2a_client._transport.httpx_client, # type: ignore[union-attr]
agent_card=agent_card,
)

150
litellm/a2a_protocol/exceptions.py Normal file
View file

@ -0,0 +1,150 @@
"""
A2A Protocol Exceptions.
Custom exception types for A2A protocol operations, following LiteLLM's exception pattern.
"""
from typing import Optional
import httpx
class A2AError(Exception):
"""
Base exception for A2A protocol errors.
Follows the same pattern as LiteLLM's main exceptions.
"""
def __init__(
self,
message: str,
status_code: int = 500,
llm_provider: str = "a2a_agent",
model: Optional[str] = None,
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
):
self.status_code = status_code
self.message = f"litellm.A2AError: {message}"
self.llm_provider = llm_provider
self.model = model
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
self.response = response or httpx.Response(
status_code=self.status_code,
request=httpx.Request(method="POST", url="https://litellm.ai"),
)
super().__init__(self.message)
def __str__(self) -> str:
_message = self.message
if self.num_retries:
_message += f" LiteLLM Retried: {self.num_retries} times"
if self.max_retries:
_message += f", LiteLLM Max Retries: {self.max_retries}"
return _message
def __repr__(self) -> str:
return self.__str__()
class A2AConnectionError(A2AError):
"""
Raised when connection to an A2A agent fails.
This typically occurs when:
- The agent is unreachable
- The agent card contains a localhost/internal URL
- Network issues prevent connection
"""
def __init__(
self,
message: str,
url: Optional[str] = None,
model: Optional[str] = None,
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
):
self.url = url
super().__init__(
message=message,
status_code=503,
llm_provider="a2a_agent",
model=model,
response=response,
litellm_debug_info=litellm_debug_info,
max_retries=max_retries,
num_retries=num_retries,
)
class A2AAgentCardError(A2AError):
"""
Raised when there's an issue with the agent card.
This includes:
- Failed to fetch agent card
- Invalid agent card format
- Missing required fields
"""
def __init__(
self,
message: str,
url: Optional[str] = None,
model: Optional[str] = None,
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
):
self.url = url
super().__init__(
message=message,
status_code=404,
llm_provider="a2a_agent",
model=model,
response=response,
litellm_debug_info=litellm_debug_info,
)
class A2ALocalhostURLError(A2AConnectionError):
"""
Raised when an agent card contains a localhost/internal URL.
Many A2A agents are deployed with agent cards that contain internal URLs
like "http://0.0.0.0:8001/" or "http://localhost:8000/". This error
indicates that the URL needs to be corrected and the request should be retried.
Attributes:
localhost_url: The localhost/internal URL found in the agent card
base_url: The public base URL that should be used instead
original_error: The original connection error that was raised
"""
def __init__(
self,
localhost_url: str,
base_url: str,
original_error: Optional[Exception] = None,
model: Optional[str] = None,
):
self.localhost_url = localhost_url
self.base_url = base_url
self.original_error = original_error
message = (
f"Agent card contains localhost/internal URL '{localhost_url}'. "
f"Retrying with base URL '{base_url}'."
)
super().__init__(
message=message,
url=localhost_url,
model=model,
)

184
litellm/a2a_protocol/main.py
View file

@ -44,6 +44,11 @@ except ImportError:
# Import our custom card resolver that supports multiple well-known paths
from litellm.a2a_protocol.card_resolver import LiteLLMA2ACardResolver
from litellm.a2a_protocol.exception_mapping_utils import (
handle_a2a_localhost_retry,
map_a2a_exception,
)
from litellm.a2a_protocol.exceptions import A2ALocalhostURLError
# Use our custom resolver instead of the default A2A SDK resolver
A2ACardResolver = LiteLLMA2ACardResolver
@ -244,10 +249,50 @@ async def asend_message(
verbose_logger.info(f"A2A send_message request_id={request.id}, agent={agent_name}")
a2a_response = await a2a_client.send_message(request)
# Get agent card URL for localhost retry logic
agent_card = getattr(a2a_client, "_litellm_agent_card", None) or getattr(
a2a_client, "agent_card", None
)
card_url = getattr(agent_card, "url", None) if agent_card else None
# Retry loop: if connection fails due to localhost URL in agent card, retry with fixed URL
a2a_response = None
for _ in range(2): # max 2 attempts: original + 1 retry
try:
a2a_response = await a2a_client.send_message(request)
break # success, exit retry loop
except A2ALocalhostURLError as e:
# Localhost URL error - fix and retry
a2a_client = handle_a2a_localhost_retry(
error=e,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
except Exception as e:
# Map exception - will raise A2ALocalhostURLError if applicable
try:
map_a2a_exception(e, card_url, api_base, model=agent_name)
except A2ALocalhostURLError as localhost_err:
# Localhost URL error - fix and retry
a2a_client = handle_a2a_localhost_retry(
error=localhost_err,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
continue
except Exception:
# Re-raise the mapped exception
raise
verbose_logger.info(f"A2A send_message completed, request_id={request.id}")
# a2a_response is guaranteed to be set if we reach here (loop breaks on success or raises)
assert a2a_response is not None
# Wrap in LiteLLM response type for _hidden_params support
response = LiteLLMSendMessageResponse.from_a2a_response(a2a_response)
@ -307,6 +352,48 @@ def send_message(
)
def _build_streaming_logging_obj(
request: "SendStreamingMessageRequest",
agent_name: str,
agent_id: Optional[str],
litellm_params: Optional[Dict[str, Any]],
metadata: Optional[Dict[str, Any]],
proxy_server_request: Optional[Dict[str, Any]],
) -> Logging:
"""Build logging object for streaming A2A requests."""
start_time = datetime.datetime.now()
model = f"a2a_agent/{agent_name}"
logging_obj = Logging(
model=model,
messages=[{"role": "user", "content": "streaming-request"}],
stream=False,
call_type="asend_message_streaming",
start_time=start_time,
litellm_call_id=str(request.id),
function_id=str(request.id),
)
logging_obj.model = model
logging_obj.custom_llm_provider = "a2a_agent"
logging_obj.model_call_details["model"] = model
logging_obj.model_call_details["custom_llm_provider"] = "a2a_agent"
if agent_id:
logging_obj.model_call_details["agent_id"] = agent_id
_litellm_params = litellm_params.copy() if litellm_params else {}
if metadata:
_litellm_params["metadata"] = metadata
if proxy_server_request:
_litellm_params["proxy_server_request"] = proxy_server_request
logging_obj.litellm_params = _litellm_params
logging_obj.optional_params = _litellm_params
logging_obj.model_call_details["litellm_params"] = _litellm_params
logging_obj.model_call_details["metadata"] = metadata or {}
return logging_obj
async def asend_message_streaming(
a2a_client: Optional["A2AClientType"] = None,
request: Optional["SendStreamingMessageRequest"] = None,
@ -403,55 +490,72 @@ async def asend_message_streaming(
verbose_logger.info(f"A2A send_message_streaming request_id={request.id}")
# Track for logging
start_time = datetime.datetime.now()
stream = a2a_client.send_message_streaming(request)
# Build logging object for streaming completion callbacks
agent_card = getattr(a2a_client, "_litellm_agent_card", None) or getattr(
a2a_client, "agent_card", None
)
card_url = getattr(agent_card, "url", None) if agent_card else None
agent_name = getattr(agent_card, "name", "unknown") if agent_card else "unknown"
model = f"a2a_agent/{agent_name}"
logging_obj = Logging(
model=model,
messages=[{"role": "user", "content": "streaming-request"}],
stream=False, # complete response logging after stream ends
call_type="asend_message_streaming",
start_time=start_time,
litellm_call_id=str(request.id),
function_id=str(request.id),
)
logging_obj.model = model
logging_obj.custom_llm_provider = "a2a_agent"
logging_obj.model_call_details["model"] = model
logging_obj.model_call_details["custom_llm_provider"] = "a2a_agent"
if agent_id:
logging_obj.model_call_details["agent_id"] = agent_id
# Propagate litellm_params for spend logging (includes cost_per_query, etc.)
_litellm_params = litellm_params.copy() if litellm_params else {}
# Merge metadata into litellm_params.metadata (required for proxy cost tracking)
if metadata:
_litellm_params["metadata"] = metadata
if proxy_server_request:
_litellm_params["proxy_server_request"] = proxy_server_request
logging_obj.litellm_params = _litellm_params
logging_obj.optional_params = _litellm_params # used by cost calc
logging_obj.model_call_details["litellm_params"] = _litellm_params
logging_obj.model_call_details["metadata"] = metadata or {}
iterator = A2AStreamingIterator(
stream=stream,
logging_obj = _build_streaming_logging_obj(
request=request,
logging_obj=logging_obj,
agent_name=agent_name,
agent_id=agent_id,
litellm_params=litellm_params,
metadata=metadata,
proxy_server_request=proxy_server_request,
)
async for chunk in iterator:
yield chunk
# Retry loop: if connection fails due to localhost URL in agent card, retry with fixed URL
# Connection errors in streaming typically occur on first chunk iteration
first_chunk = True
for attempt in range(2): # max 2 attempts: original + 1 retry
stream = a2a_client.send_message_streaming(request)
iterator = A2AStreamingIterator(
stream=stream,
request=request,
logging_obj=logging_obj,
agent_name=agent_name,
)
try:
first_chunk = True
async for chunk in iterator:
if first_chunk:
first_chunk = False # connection succeeded
yield chunk
return # stream completed successfully
except A2ALocalhostURLError as e:
# Only retry on first chunk, not mid-stream
if first_chunk and attempt == 0:
a2a_client = handle_a2a_localhost_retry(
error=e,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=True,
)
card_url = agent_card.url if agent_card else None
else:
raise
except Exception as e:
# Only map exception on first chunk
if first_chunk and attempt == 0:
try:
map_a2a_exception(e, card_url, api_base, model=agent_name)
except A2ALocalhostURLError as localhost_err:
# Localhost URL error - fix and retry
a2a_client = handle_a2a_localhost_retry(
error=localhost_err,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=True,
)
card_url = agent_card.url if agent_card else None
continue
except Exception:
# Re-raise the mapped exception
raise
raise
async def create_a2a_client(

33
litellm/anthropic_beta_headers_config.json Normal file
View file

@ -0,0 +1,33 @@
{
"description": "Unsupported Anthropic beta headers for each provider. Headers listed here will be dropped. Headers not listed are passed through as-is.",
"anthropic": [],
"azure_ai": [],
"bedrock_converse": [
"prompt-caching-scope-2026-01-05",
"bash_20250124",
"bash_20241022",
"text_editor_20250124",
"text_editor_20241022",
"compact-2026-01-12",
"advanced-tool-use-2025-11-20",
"web-fetch-2025-09-10",
"code-execution-2025-08-25",
"skills-2025-10-02",
"files-api-2025-04-14",
"fast-mode-2026-02-01"
],
"bedrock": [
"advanced-tool-use-2025-11-20",
"prompt-caching-scope-2026-01-05",
"structured-outputs-2025-11-13",
"web-fetch-2025-09-10",
"code-execution-2025-08-25",
"skills-2025-10-02",
"files-api-2025-04-14",
"fast-mode-2026-02-01",
"mcp-servers-2025-12-04"
],
"vertex_ai": [
"prompt-caching-scope-2026-01-05"
]
}

221
litellm/anthropic_beta_headers_manager.py Normal file
View file

@ -0,0 +1,221 @@
"""
Centralized manager for Anthropic beta headers across different providers.
This module provides utilities to:
1. Load beta header configuration from JSON (lists unsupported headers per provider)
2. Filter out unsupported beta headers
3. Handle provider-specific header name mappings (e.g., advanced-tool-use -> tool-search-tool)
Design:
- JSON config lists UNSUPPORTED headers for each provider
- Headers not in the unsupported list are passed through
- Header mappings allow renaming headers for specific providers
"""
import json
import os
from typing import Dict, List, Optional, Set
from litellm.litellm_core_utils.litellm_logging import verbose_logger
# Cache for the loaded configuration
_BETA_HEADERS_CONFIG: Optional[Dict] = None
def _load_beta_headers_config() -> Dict:
"""
Load the beta headers configuration from JSON file.
Uses caching to avoid repeated file reads.
Returns:
Dict containing the beta headers configuration
"""
global _BETA_HEADERS_CONFIG
if _BETA_HEADERS_CONFIG is not None:
return _BETA_HEADERS_CONFIG
config_path = os.path.join(
os.path.dirname(__file__),
"anthropic_beta_headers_config.json"
)
try:
with open(config_path, "r") as f:
_BETA_HEADERS_CONFIG = json.load(f)
verbose_logger.debug(f"Loaded beta headers config from {config_path}")
return _BETA_HEADERS_CONFIG
except Exception as e:
verbose_logger.error(f"Failed to load beta headers config: {e}")
# Return empty config as fallback
return {
"anthropic": [],
"azure_ai": [],
"bedrock": [],
"bedrock_converse": [],
"vertex_ai": []
}
def get_provider_name(provider: str) -> str:
"""
Resolve provider aliases to canonical provider names.
Args:
provider: Provider name (may be an alias)
Returns:
Canonical provider name
"""
config = _load_beta_headers_config()
aliases = config.get("provider_aliases", {})
return aliases.get(provider, provider)
def filter_and_transform_beta_headers(
beta_headers: List[str],
provider: str,
) -> List[str]:
"""
Filter beta headers based on provider's unsupported list.
This function:
1. Removes headers that are in the provider's unsupported list
2. Passes through all other headers as-is
Note: Header transformations/mappings (e.g., advanced-tool-use -> tool-search-tool)
are handled in each provider's transformation code, not here.
Args:
beta_headers: List of Anthropic beta header values
provider: Provider name (e.g., "anthropic", "bedrock", "vertex_ai")
Returns:
List of filtered beta headers for the provider
"""
if not beta_headers:
return []
config = _load_beta_headers_config()
provider = get_provider_name(provider)
# Get unsupported headers for this provider
unsupported_headers = set(config.get(provider, []))
filtered_headers: Set[str] = set()
for header in beta_headers:
header = header.strip()
# Skip if header is unsupported
if header in unsupported_headers:
verbose_logger.debug(
f"Dropping unsupported beta header '{header}' for provider '{provider}'"
)
continue
# Pass through as-is
filtered_headers.add(header)
return sorted(list(filtered_headers))
def is_beta_header_supported(
beta_header: str,
provider: str,
) -> bool:
"""
Check if a specific beta header is supported by a provider.
Args:
beta_header: The Anthropic beta header value
provider: Provider name
Returns:
True if the header is supported (not in unsupported list), False otherwise
"""
config = _load_beta_headers_config()
provider = get_provider_name(provider)
unsupported_headers = set(config.get(provider, []))
return beta_header not in unsupported_headers
def get_provider_beta_header(
anthropic_beta_header: str,
provider: str,
) -> Optional[str]:
"""
Check if a beta header is supported by a provider.
Note: This does NOT handle header transformations/mappings.
Those are handled in each provider's transformation code.
Args:
anthropic_beta_header: The Anthropic beta header value
provider: Provider name
Returns:
The original header if supported, or None if unsupported
"""
config = _load_beta_headers_config()
provider = get_provider_name(provider)
# Check if unsupported
unsupported_headers = set(config.get(provider, []))
if anthropic_beta_header in unsupported_headers:
return None
return anthropic_beta_header
def update_headers_with_filtered_beta(
headers: dict,
provider: str,
) -> dict:
"""
Update headers dict by filtering and transforming anthropic-beta header values.
Modifies the headers dict in place and returns it.
Args:
headers: Request headers dict (will be modified in place)
provider: Provider name
Returns:
Updated headers dict
"""
existing_beta = headers.get("anthropic-beta")
if not existing_beta:
return headers
# Parse existing beta headers
beta_values = [b.strip() for b in existing_beta.split(",") if b.strip()]
# Filter and transform based on provider
filtered_beta_values = filter_and_transform_beta_headers(
beta_headers=beta_values,
provider=provider,
)
# Update or remove the header
if filtered_beta_values:
headers["anthropic-beta"] = ",".join(filtered_beta_values)
else:
# Remove the header if no values remain
headers.pop("anthropic-beta", None)
return headers
def get_unsupported_headers(provider: str) -> List[str]:
"""
Get all beta headers that are unsupported by a provider.
Args:
provider: Provider name
Returns:
List of unsupported Anthropic beta header names
"""
config = _load_beta_headers_config()
provider = get_provider_name(provider)
return config.get(provider, [])

2
litellm/caching/redis_cache.py
View file

@ -1123,7 +1123,7 @@ class RedisCache(BaseCache):
redis_client = redis_async.Redis(**self.redis_kwargs)
# Test the connection
ping_result = await redis_client.ping()
ping_result = await redis_client.ping() # type: ignore[misc]
# Close the connection
await redis_client.aclose() # type: ignore[attr-defined]

2
litellm/caching/redis_cluster_cache.py
View file

@ -83,7 +83,7 @@ class RedisClusterCache(RedisCache):
)
# Test the connection
ping_result = await redis_client.ping() # type: ignore[attr-defined]
ping_result = await redis_client.ping() # type: ignore[attr-defined, misc]
# Close the connection
await redis_client.aclose() # type: ignore[attr-defined]

37
litellm/constants.py
View file

@ -2,6 +2,8 @@ import os
import sys
from typing import List, Literal
from litellm.litellm_core_utils.env_utils import get_env_int
DEFAULT_HEALTH_CHECK_PROMPT = str(
os.getenv("DEFAULT_HEALTH_CHECK_PROMPT", "test from litellm")
)
@ -81,6 +83,20 @@ MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH = int(
os.getenv("MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH", 150)
)
# MCP OAuth2 Client Credentials Defaults
MCP_OAUTH2_TOKEN_EXPIRY_BUFFER_SECONDS = int(
os.getenv("MCP_OAUTH2_TOKEN_EXPIRY_BUFFER_SECONDS", "60")
)
MCP_OAUTH2_TOKEN_CACHE_MAX_SIZE = int(
os.getenv("MCP_OAUTH2_TOKEN_CACHE_MAX_SIZE", "200")
)
MCP_OAUTH2_TOKEN_CACHE_DEFAULT_TTL = int(
os.getenv("MCP_OAUTH2_TOKEN_CACHE_DEFAULT_TTL", "3600")
)
MCP_OAUTH2_TOKEN_CACHE_MIN_TTL = int(
os.getenv("MCP_OAUTH2_TOKEN_CACHE_MIN_TTL", "10")
)
LITELLM_UI_ALLOW_HEADERS = [
"x-litellm-semantic-filter",
"x-litellm-semantic-filter-tools",
@ -99,6 +115,11 @@ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE = int(
)
)
# Maximum number of callbacks that can be registered
# This prevents callbacks from exponentially growing and consuming CPU resources
# Override with LITELLM_MAX_CALLBACKS env var for large deployments (e.g., many teams with guardrails)
MAX_CALLBACKS = get_env_int("LITELLM_MAX_CALLBACKS", 30)
# Generic fallback for unknown models
DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET = int(
os.getenv("DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET", 128)
@ -306,6 +327,22 @@ DEFAULT_MAX_TOKENS_FOR_TRITON = int(os.getenv("DEFAULT_MAX_TOKENS_FOR_TRITON", 2
#### Networking settings ####
request_timeout: float = float(os.getenv("REQUEST_TIMEOUT", 6000)) # time in seconds
DEFAULT_A2A_AGENT_TIMEOUT: float = float(os.getenv("DEFAULT_A2A_AGENT_TIMEOUT", 6000)) # 10 minutes
# Patterns that indicate a localhost/internal URL in A2A agent cards that should be
# replaced with the original base_url. This is a common misconfiguration where
# developers deploy agents with development URLs in their agent cards.
LOCALHOST_URL_PATTERNS: List[str] = [
"localhost",
"127.0.0.1",
"0.0.0.0",
"[::1]", # IPv6 localhost
]
# Patterns in error messages that indicate a connection failure
CONNECTION_ERROR_PATTERNS: List[str] = [
"connect",
"connection",
"network",
"refused",
]
STREAM_SSE_DONE_STRING: str = "[DONE]"
STREAM_SSE_DATA_PREFIX: str = "data: "
### SPEND TRACKING ###

113
litellm/cost_calculator.py
View file

@ -1,5 +1,6 @@
# What is this?
## File for 'response_cost' calculation in Logging
import logging
import time
from functools import lru_cache
from typing import TYPE_CHECKING, Any, List, Literal, Optional, Tuple, Union, cast
@ -774,10 +775,11 @@ def _apply_cost_discount(
discount_amount = original_cost * discount_percent
final_cost = original_cost - discount_amount
verbose_logger.debug(
f"Applied {discount_percent*100}% discount to {custom_llm_provider}: "
f"${original_cost:.6f} -> ${final_cost:.6f} (saved ${discount_amount:.6f})"
)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
f"Applied {discount_percent*100}% discount to {custom_llm_provider}: "
f"${original_cost:.6f} -> ${final_cost:.6f} (saved ${discount_amount:.6f})"
)
return final_cost, discount_percent, discount_amount
@ -807,17 +809,20 @@ def _apply_cost_margin(
margin_config = None
if custom_llm_provider and custom_llm_provider in litellm.cost_margin_config:
margin_config = litellm.cost_margin_config[custom_llm_provider]
verbose_logger.debug(
f"Found provider-specific margin config for {custom_llm_provider}: {margin_config}"
)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
f"Found provider-specific margin config for {custom_llm_provider}: {margin_config}"
)
elif "global" in litellm.cost_margin_config:
margin_config = litellm.cost_margin_config["global"]
verbose_logger.debug(f"Using global margin config: {margin_config}")
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(f"Using global margin config: {margin_config}")
else:
verbose_logger.debug(
f"No margin config found. Provider: {custom_llm_provider}, "
f"Available configs: {list(litellm.cost_margin_config.keys())}"
)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
f"No margin config found. Provider: {custom_llm_provider}, "
f"Available configs: {list(litellm.cost_margin_config.keys())}"
)
if margin_config is not None:
# Handle different margin config formats
@ -836,11 +841,12 @@ def _apply_cost_margin(
final_cost = original_cost + margin_total_amount
verbose_logger.debug(
f"Applied margin to {custom_llm_provider or 'global'}: "
f"${original_cost:.6f} -> ${final_cost:.6f} "
f"(margin: {margin_percent*100 if margin_percent > 0 else 0}% + ${margin_fixed_amount:.6f} = ${margin_total_amount:.6f})"
)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
f"Applied margin to {custom_llm_provider or 'global'}: "
f"${original_cost:.6f} -> ${final_cost:.6f} "
f"(margin: {margin_percent*100 if margin_percent > 0 else 0}% + ${margin_fixed_amount:.6f} = ${margin_total_amount:.6f})"
)
return final_cost, margin_percent, margin_fixed_amount, margin_total_amount
@ -1021,9 +1027,10 @@ def completion_cost( # noqa: PLR0915
for idx, model in enumerate(potential_model_names):
try:
verbose_logger.debug(
f"selected model name for cost calculation: {model}"
)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
f"selected model name for cost calculation: {model}"
)
if completion_response is not None and (
isinstance(completion_response, BaseModel)
@ -1411,37 +1418,47 @@ def completion_cost( # noqa: PLR0915
# Apply discount from module-level config if configured
original_cost = _final_cost
_final_cost, discount_percent, discount_amount = _apply_cost_discount(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
)
if litellm.cost_discount_config:
_final_cost, discount_percent, discount_amount = _apply_cost_discount(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
)
else:
discount_percent = 0.0
discount_amount = 0.0
# Apply margin from module-level config if configured
(
_final_cost,
margin_percent,
margin_fixed_amount,
margin_total_amount,
) = _apply_cost_margin(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
)
if litellm.cost_margin_config:
(
_final_cost,
margin_percent,
margin_fixed_amount,
margin_total_amount,
) = _apply_cost_margin(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
)
else:
margin_percent = 0.0
margin_fixed_amount = 0.0
margin_total_amount = 0.0
# Store cost breakdown in logging object if available
_store_cost_breakdown_in_logging_obj(
litellm_logging_obj=litellm_logging_obj,
prompt_tokens_cost_usd_dollar=prompt_tokens_cost_usd_dollar,
completion_tokens_cost_usd_dollar=completion_tokens_cost_usd_dollar,
cost_for_built_in_tools_cost_usd_dollar=cost_for_built_in_tools,
total_cost_usd_dollar=_final_cost,
additional_costs=additional_costs,
original_cost=original_cost,
discount_percent=discount_percent,
discount_amount=discount_amount,
margin_percent=margin_percent,
margin_fixed_amount=margin_fixed_amount,
margin_total_amount=margin_total_amount,
)
if litellm_logging_obj is not None:
_store_cost_breakdown_in_logging_obj(
litellm_logging_obj=litellm_logging_obj,
prompt_tokens_cost_usd_dollar=prompt_tokens_cost_usd_dollar,
completion_tokens_cost_usd_dollar=completion_tokens_cost_usd_dollar,
cost_for_built_in_tools_cost_usd_dollar=cost_for_built_in_tools,
total_cost_usd_dollar=_final_cost,
original_cost=original_cost,
additional_costs=additional_costs,
discount_percent=discount_percent,
discount_amount=discount_amount,
margin_percent=margin_percent,
margin_fixed_amount=margin_fixed_amount,
margin_total_amount=margin_total_amount,
)
return _final_cost
except Exception as e:
@ -2116,3 +2133,5 @@ def handle_realtime_stream_cost_calculation(
total_cost = input_cost_per_token + output_cost_per_token
return total_cost

2
litellm/experimental_mcp_client/client.py
View file

@ -209,6 +209,8 @@ class MCPClient:
headers["X-API-Key"] = self._mcp_auth_value
elif self.auth_type == MCPAuth.authorization:
headers["Authorization"] = self._mcp_auth_value
elif self.auth_type == MCPAuth.oauth2:
headers["Authorization"] = f"Bearer {self._mcp_auth_value}"
elif isinstance(self._mcp_auth_value, dict):
headers.update(self._mcp_auth_value)

1
litellm/integrations/custom_guardrail.py
View file

@ -268,6 +268,7 @@ class CustomGuardrail(CustomLogger):
"""
Returns the guardrail(s) to be run from the metadata or root
"""
if "guardrails" in data:
return data["guardrails"]
metadata = data.get("litellm_metadata") or data.get("metadata", {})

31
litellm/integrations/custom_logger.py
View file

@ -664,6 +664,37 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
return final_response
"""
pass
async def async_should_run_chat_completion_agentic_loop(
self,
response: Any,
model: str,
messages: List[Dict],
tools: Optional[List[Dict]],
stream: bool,
custom_llm_provider: str,
kwargs: Dict,
) -> Tuple[bool, Dict]:
"""
Hook to determine if chat completion agentic loop should be executed.
"""
return False, {}
async def async_run_chat_completion_agentic_loop(
self,
tools: Dict,
model: str,
messages: List[Dict],
response: Any,
optional_params: Dict,
logging_obj: "LiteLLMLoggingObj",
stream: bool,
kwargs: Dict,
) -> Any:
"""
Hook to execute chat completion agentic loop based on context from should_run hook.
"""
pass
# Useful helpers for custom logger classes

113
litellm/integrations/datadog/datadog.py
View file

@ -45,7 +45,14 @@ from litellm.llms.custom_httpx.http_handler import (
httpxSpecialProvider,
)
from litellm.types.integrations.base_health_check import IntegrationHealthCheckStatus
from litellm.types.integrations.datadog import *
from litellm.types.integrations.datadog import (
DD_ERRORS,
DD_MAX_BATCH_SIZE,
DataDogStatus,
DatadogInitParams,
DatadogPayload,
DatadogProxyFailureHookJsonMessage,
)
from litellm.types.services import ServiceLoggerPayload, ServiceTypes
from litellm.types.utils import StandardLoggingPayload
@ -85,12 +92,14 @@ class DataDogLogger(
"""
try:
verbose_logger.debug("Datadog: in init datadog logger")
self.is_mock_mode = should_use_datadog_mock()
if self.is_mock_mode:
create_mock_datadog_client()
verbose_logger.debug("[DATADOG MOCK] Datadog logger initialized in mock mode")
verbose_logger.debug(
"[DATADOG MOCK] Datadog logger initialized in mock mode"
)
#########################################################
# Handle datadog_params set as litellm.datadog_params
@ -209,6 +218,96 @@ class DataDogLogger(
)
pass
async def async_post_call_failure_hook(
self,
request_data: dict,
original_exception: Exception,
user_api_key_dict: Any,
traceback_str: Optional[str] = None,
) -> Optional[Any]:
"""
Log proxy-level failures (e.g. 401 auth, DB connection errors) to Datadog.
Ensures failures that occur before or outside the LLM completion flow
(e.g. ConnectError during auth when DB is down) are visible in Datadog
alongside Prometheus.
"""
try:
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
error_information = StandardLoggingPayloadSetup.get_error_information(
original_exception=original_exception,
traceback_str=traceback_str,
)
_code = error_information.get("error_code") or ""
status_code: Optional[int] = None
if _code and str(_code).strip().isdigit():
status_code = int(_code)
# Use project-standard sanitized user context when running in proxy
user_context: Dict[str, Any] = {}
try:
from litellm.proxy.litellm_pre_call_utils import (
LiteLLMProxyRequestSetup,
)
_meta = (
LiteLLMProxyRequestSetup.get_sanitized_user_information_from_key(
user_api_key_dict=user_api_key_dict
)
)
user_context = dict(_meta) if isinstance(_meta, dict) else _meta
except Exception:
# Fallback if proxy not available (e.g. SDK-only): minimal safe fields
if hasattr(user_api_key_dict, "request_route"):
user_context["request_route"] = getattr(
user_api_key_dict, "request_route", None
)
if hasattr(user_api_key_dict, "team_id"):
user_context["team_id"] = getattr(
user_api_key_dict, "team_id", None
)
if hasattr(user_api_key_dict, "user_id"):
user_context["user_id"] = getattr(
user_api_key_dict, "user_id", None
)
if hasattr(user_api_key_dict, "end_user_id"):
user_context["end_user_id"] = getattr(
user_api_key_dict, "end_user_id", None
)
message_payload: DatadogProxyFailureHookJsonMessage = {
"exception": error_information.get("error_message")
or str(original_exception),
"error_class": error_information.get("error_class")
or original_exception.__class__.__name__,
"status_code": status_code,
"traceback": error_information.get("traceback") or "",
"user_api_key_dict": user_context,
}
dd_payload = DatadogPayload(
ddsource=get_datadog_source(),
ddtags=get_datadog_tags(),
hostname=get_datadog_hostname(),
message=safe_dumps(message_payload),
service=get_datadog_service(),
status=DataDogStatus.ERROR,
)
self._add_trace_context_to_payload(dd_payload=dd_payload)
self.log_queue.append(dd_payload)
if len(self.log_queue) >= self.batch_size:
await self.async_send_batch()
except Exception as e:
verbose_logger.exception(
f"Datadog: async_post_call_failure_hook - {str(e)}\n{traceback.format_exc()}"
)
return None
async def async_send_batch(self):
"""
Sends the in memory logs queue to datadog api
@ -230,9 +329,11 @@ class DataDogLogger(
len(self.log_queue),
self.intake_url,
)
if self.is_mock_mode:
verbose_logger.debug("[DATADOG MOCK] Mock mode enabled - API calls will be intercepted")
verbose_logger.debug(
"[DATADOG MOCK] Mock mode enabled - API calls will be intercepted"
)
response = await self.async_send_compressed_data(self.log_queue)
if response.status_code == 413:

24
litellm/integrations/email_templates/templates.py
View file

@ -85,6 +85,30 @@ SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
The LiteLLM team <br />
"""
TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
<img src="{email_logo_url}" alt="LiteLLM Logo" width="150" height="50" />
<p> Hi {team_alias} team member, <br/>
Your LiteLLM team has crossed its <b>soft budget limit of {soft_budget}</b>. <br /> <br />
<b>Current Spend:</b> {spend} <br />
<b>Soft Budget:</b> {soft_budget} <br />
{max_budget_info}
<p style="color: #dc2626; font-weight: 500;">
⚠️ Note: Your API requests will continue to work, but you should monitor your usage closely.
If you reach your maximum budget, requests will be rejected.
</p>
You can view your usage and manage your budget in the <a href="{base_url}">LiteLLM Dashboard</a>. <br /> <br />
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
"""
MAX_BUDGET_ALERT_EMAIL_TEMPLATE = """
<img src="{email_logo_url}" alt="LiteLLM Logo" width="150" height="50" />

235
litellm/integrations/opentelemetry.py
View file

@ -5,6 +5,10 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union, cast
import litellm
from litellm._logging import verbose_logger
from litellm.integrations._types.open_inference import (
OpenInferenceSpanKindValues,
SpanAttributes,
)
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.secret_managers.main import get_secret_bool
@ -17,10 +21,6 @@ from litellm.types.utils import (
StandardCallbackDynamicParams,
StandardLoggingPayload,
)
from litellm.integrations._types.open_inference import (
OpenInferenceSpanKindValues,
SpanAttributes,
)
# OpenTelemetry imports moved to individual functions to avoid import errors when not installed
@ -40,7 +40,9 @@ if TYPE_CHECKING:
Context = Union[_Context, Any]
SpanExporter = Union[_SpanExporter, Any]
UserAPIKeyAuth = Union[_UserAPIKeyAuth, Any]
ManagementEndpointLoggingPayload = Union[_ManagementEndpointLoggingPayload, Any]
ManagementEndpointLoggingPayload = Union[
_ManagementEndpointLoggingPayload, Any
]
else:
Span = Any
Tracer = Any
@ -95,12 +97,16 @@ class OpenTelemetryConfig:
exporter = os.getenv(
"OTEL_EXPORTER_OTLP_PROTOCOL", os.getenv("OTEL_EXPORTER", "console")
)
endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", os.getenv("OTEL_ENDPOINT"))
endpoint = os.getenv(
"OTEL_EXPORTER_OTLP_ENDPOINT", os.getenv("OTEL_ENDPOINT")
)
headers = os.getenv(
"OTEL_EXPORTER_OTLP_HEADERS", os.getenv("OTEL_HEADERS")
) # example: OTEL_HEADERS=x-honeycomb-team=B85YgLm96***"
enable_metrics: bool = (
os.getenv("LITELLM_OTEL_INTEGRATION_ENABLE_METRICS", "false").lower()
os.getenv(
"LITELLM_OTEL_INTEGRATION_ENABLE_METRICS", "false"
).lower()
== "true"
)
enable_events: bool = (
@ -108,7 +114,9 @@ class OpenTelemetryConfig:
== "true"
)
service_name = os.getenv("OTEL_SERVICE_NAME", "litellm")
deployment_environment = os.getenv("OTEL_ENVIRONMENT_NAME", "production")
deployment_environment = os.getenv(
"OTEL_ENVIRONMENT_NAME", "production"
)
model_id = os.getenv("OTEL_MODEL_ID", service_name)
if exporter == "in_memory":
@ -157,7 +165,9 @@ class OpenTelemetry(CustomLogger):
logging.getLogger(__name__)
# Enable OpenTelemetry logging
otel_exporter_logger = logging.getLogger("opentelemetry.sdk.trace.export")
otel_exporter_logger = logging.getLogger(
"opentelemetry.sdk.trace.export"
)
otel_exporter_logger.setLevel(logging.DEBUG)
# init CustomLogger params
@ -253,7 +263,9 @@ class OpenTelemetry(CustomLogger):
# Don't call set_provider to preserve existing context
else:
# Default proxy provider or unknown type, create our own
verbose_logger.debug("OpenTelemetry: Creating new %s", provider_name)
verbose_logger.debug(
"OpenTelemetry: Creating new %s", provider_name
)
provider = create_new_provider_fn()
set_provider_fn(provider)
except Exception as e:
@ -274,7 +286,9 @@ class OpenTelemetry(CustomLogger):
from opentelemetry.trace import SpanKind
def create_tracer_provider():
provider = TracerProvider(resource=self._get_litellm_resource(self.config))
provider = TracerProvider(
resource=self._get_litellm_resource(self.config)
)
provider.add_span_processor(self._get_span_processor())
return provider
@ -388,10 +402,14 @@ class OpenTelemetry(CustomLogger):
def log_failure_event(self, kwargs, response_obj, start_time, end_time):
self._handle_failure(kwargs, response_obj, start_time, end_time)
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
async def async_log_success_event(
self, kwargs, response_obj, start_time, end_time
):
self._handle_success(kwargs, response_obj, start_time, end_time)
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
async def async_log_failure_event(
self, kwargs, response_obj, start_time, end_time
):
self._handle_failure(kwargs, response_obj, start_time, end_time)
async def async_service_success_hook(
@ -588,7 +606,9 @@ class OpenTelemetry(CustomLogger):
if dynamic_headers is not None:
# Create spans using a temporary tracer with dynamic headers
tracer_to_use = self._get_tracer_with_dynamic_headers(dynamic_headers)
tracer_to_use = self._get_tracer_with_dynamic_headers(
dynamic_headers
)
verbose_logger.debug(
"Using dynamic headers for this request: %s", dynamic_headers
)
@ -624,7 +644,9 @@ class OpenTelemetry(CustomLogger):
)
# Create a temporary tracer provider with dynamic headers
temp_provider = TracerProvider(resource=self._get_litellm_resource(self.config))
temp_provider = TracerProvider(
resource=self._get_litellm_resource(self.config)
)
temp_provider.add_span_processor(
self._get_span_processor(dynamic_headers=dynamic_headers)
)
@ -755,7 +777,9 @@ class OpenTelemetry(CustomLogger):
metadata = litellm_params.get("metadata") or {}
generation_name = metadata.get("generation_name")
raw_span_name = generation_name if generation_name else RAW_REQUEST_SPAN_NAME
raw_span_name = (
generation_name if generation_name else RAW_REQUEST_SPAN_NAME
)
otel_tracer: Tracer = self.get_tracer_to_use_for_request(kwargs)
raw_span = otel_tracer.start_span(
@ -780,7 +804,9 @@ class OpenTelemetry(CustomLogger):
}
std_log = kwargs.get("standard_logging_object")
md = getattr(std_log, "metadata", None) or (std_log or {}).get("metadata", {})
md = getattr(std_log, "metadata", None) or (std_log or {}).get(
"metadata", {}
)
for key in [
"user_api_key_hash",
"user_api_key_alias",
@ -802,9 +828,9 @@ class OpenTelemetry(CustomLogger):
common_attrs[f"metadata.{key}"] = str(md[key])
# get hidden params
hidden_params = getattr(std_log, "hidden_params", None) or (std_log or {}).get(
"hidden_params", {}
)
hidden_params = getattr(std_log, "hidden_params", None) or (
std_log or {}
).get("hidden_params", {})
if hidden_params:
common_attrs["hidden_params"] = safe_dumps(hidden_params)
@ -838,7 +864,9 @@ class OpenTelemetry(CustomLogger):
self._record_response_duration_metric(kwargs, end_time, common_attrs)
@staticmethod
def _to_timestamp(val: Optional[Union[datetime, float, str]]) -> Optional[float]:
def _to_timestamp(
val: Optional[Union[datetime, float, str]],
) -> Optional[float]:
"""Convert datetime/float/string to timestamp."""
if val is None:
return None
@ -855,7 +883,9 @@ class OpenTelemetry(CustomLogger):
except ValueError:
return None
def _record_time_to_first_token_metric(self, kwargs: dict, common_attrs: dict):
def _record_time_to_first_token_metric(
self, kwargs: dict, common_attrs: dict
):
"""Record Time to First Token (TTFT) metric for streaming requests."""
optional_params = kwargs.get("optional_params", {})
is_streaming = optional_params.get("stream", False)
@ -868,7 +898,10 @@ class OpenTelemetry(CustomLogger):
api_call_start_time = kwargs.get("api_call_start_time", None)
completion_start_time = kwargs.get("completion_start_time", None)
if api_call_start_time is not None and completion_start_time is not None:
if (
api_call_start_time is not None
and completion_start_time is not None
):
# Convert to timestamps if needed (handles datetime, float, and string)
api_call_start_ts = self._to_timestamp(api_call_start_time)
completion_start_ts = self._to_timestamp(completion_start_time)
@ -876,7 +909,9 @@ class OpenTelemetry(CustomLogger):
if api_call_start_ts is None or completion_start_ts is None:
return # Skip recording if conversion failed
time_to_first_token_seconds = completion_start_ts - api_call_start_ts
time_to_first_token_seconds = (
completion_start_ts - api_call_start_ts
)
self._time_to_first_token_histogram.record(
time_to_first_token_seconds, attributes=common_attrs
)
@ -946,7 +981,9 @@ class OpenTelemetry(CustomLogger):
generation_time_seconds = duration_s
if generation_time_seconds > 0:
time_per_output_token_seconds = generation_time_seconds / completion_tokens
time_per_output_token_seconds = (
generation_time_seconds / completion_tokens
)
self._time_per_output_token_histogram.record(
time_per_output_token_seconds, attributes=common_attrs
)
@ -1007,21 +1044,26 @@ class OpenTelemetry(CustomLogger):
# See: https://github.com/open-telemetry/opentelemetry-python/pull/4676
# TODO: Refactor to use the proper OTEL Logs API instead of directly creating SDK LogRecords
from opentelemetry._logs import SeverityNumber, get_logger, get_logger_provider
from opentelemetry._logs import (
SeverityNumber,
get_logger,
)
try:
from opentelemetry.sdk._logs import LogRecord as SdkLogRecord # type: ignore[attr-defined] # OTEL < 1.39.0
except ImportError:
from opentelemetry.sdk._logs._internal import LogRecord as SdkLogRecord # type: ignore[attr-defined, no-redef] # OTEL >= 1.39.0
# MyPy evaluates both branches of try/except imports and can fail when
# newer OTEL stubs remove/relocate symbols. Gate the typing import so
# only the canonical location is type-checked.
if TYPE_CHECKING:
from opentelemetry.sdk._logs._internal import LogRecord as SdkLogRecord
else:
try:
from opentelemetry.sdk._logs import (
LogRecord as SdkLogRecord, # type: ignore[attr-defined]
)
except ImportError:
from opentelemetry.sdk._logs._internal import LogRecord as SdkLogRecord
otel_logger = get_logger(LITELLM_LOGGER_NAME)
# Get the resource from the logger provider
logger_provider = get_logger_provider()
resource = getattr(
logger_provider, "_resource", None
) or self._get_litellm_resource(self.config)
parent_ctx = span.get_span_context()
provider = (kwargs.get("litellm_params") or {}).get(
"custom_llm_provider", "Unknown"
@ -1030,7 +1072,10 @@ class OpenTelemetry(CustomLogger):
# per-message events
for msg in kwargs.get("messages", []):
role = msg.get("role", "user")
attrs = {"event_name": "gen_ai.content.prompt", "gen_ai.system": provider}
attrs = {
"event_name": "gen_ai.content.prompt",
"gen_ai.system": provider,
}
if role == "tool" and msg.get("id"):
attrs["id"] = msg["id"]
if self.message_logging and msg.get("content"):
@ -1044,7 +1089,6 @@ class OpenTelemetry(CustomLogger):
severity_number=SeverityNumber.INFO,
severity_text="INFO",
body=msg.copy(),
resource=resource,
attributes=attrs,
)
otel_logger.emit(log_record)
@ -1076,7 +1120,6 @@ class OpenTelemetry(CustomLogger):
severity_number=SeverityNumber.INFO,
severity_text="INFO",
body=body,
resource=resource,
attributes=attrs,
)
otel_logger.emit(log_record)
@ -1146,7 +1189,9 @@ class OpenTelemetry(CustomLogger):
value=guardrail_information.get("guardrail_mode"),
)
masked_entity_count = guardrail_information.get("masked_entity_count")
masked_entity_count = guardrail_information.get(
"masked_entity_count"
)
if masked_entity_count is not None:
guardrail_span.set_attribute(
"masked_entity_count", safe_dumps(masked_entity_count)
@ -1173,8 +1218,9 @@ class OpenTelemetry(CustomLogger):
# Decide whether to create a primary span
# Always create if no parent span exists (backward compatibility)
# OR if USE_OTEL_LITELLM_REQUEST_SPAN is explicitly enabled
should_create_primary_span = parent_otel_span is None or get_secret_bool(
"USE_OTEL_LITELLM_REQUEST_SPAN"
should_create_primary_span = (
parent_otel_span is None
or get_secret_bool("USE_OTEL_LITELLM_REQUEST_SPAN")
)
if should_create_primary_span:
@ -1200,7 +1246,9 @@ class OpenTelemetry(CustomLogger):
if parent_otel_span.is_recording():
parent_otel_span.set_status(Status(StatusCode.ERROR))
self.set_attributes(parent_otel_span, kwargs, response_obj)
self._record_exception_on_span(span=parent_otel_span, kwargs=kwargs)
self._record_exception_on_span(
span=parent_otel_span, kwargs=kwargs
)
# Create span for guardrail information
self._create_guardrail_span(kwargs=kwargs, context=_parent_context)
@ -1223,7 +1271,9 @@ class OpenTelemetry(CustomLogger):
2. Sets structured error attributes from StandardLoggingPayloadErrorInformation
"""
try:
from litellm.integrations._types.open_inference import ErrorAttributes
from litellm.integrations._types.open_inference import (
ErrorAttributes,
)
# Get the exception object if available
exception = kwargs.get("exception")
@ -1233,15 +1283,17 @@ class OpenTelemetry(CustomLogger):
span.record_exception(exception)
# Get StandardLoggingPayload for structured error information
standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
"standard_logging_object"
standard_logging_payload: Optional[StandardLoggingPayload] = (
kwargs.get("standard_logging_object")
)
if standard_logging_payload is None:
return
# Extract error_information from StandardLoggingPayload
error_information = standard_logging_payload.get("error_information")
error_information = standard_logging_payload.get(
"error_information"
)
if error_information is None:
# Fallback to error_str if error_information is not available
@ -1331,7 +1383,9 @@ class OpenTelemetry(CustomLogger):
)
pass
def cast_as_primitive_value_type(self, value) -> Union[str, bool, int, float]:
def cast_as_primitive_value_type(
self, value
) -> Union[str, bool, int, float]:
"""
Casts the value to a primitive OTEL type if it is not already a primitive type.
@ -1401,8 +1455,8 @@ class OpenTelemetry(CustomLogger):
optional_params = kwargs.get("optional_params", {})
litellm_params = kwargs.get("litellm_params", {}) or {}
standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
"standard_logging_object"
standard_logging_payload: Optional[StandardLoggingPayload] = (
kwargs.get("standard_logging_object")
)
if standard_logging_payload is None:
raise ValueError("standard_logging_object not found in kwargs")
@ -1424,11 +1478,13 @@ class OpenTelemetry(CustomLogger):
) or (standard_logging_payload or {}).get("hidden_params", {})
if hidden_params:
self.safe_set_attribute(
span=span, key="hidden_params", value=safe_dumps(hidden_params)
span=span,
key="hidden_params",
value=safe_dumps(hidden_params),
)
# Cost breakdown tracking
cost_breakdown: Optional[CostBreakdown] = standard_logging_payload.get(
"cost_breakdown"
cost_breakdown: Optional[CostBreakdown] = (
standard_logging_payload.get("cost_breakdown")
)
if cost_breakdown:
for key, value in cost_breakdown.items():
@ -1504,7 +1560,9 @@ class OpenTelemetry(CustomLogger):
# The unique identifier for the completion.
if response_obj and response_obj.get("id"):
self.safe_set_attribute(
span=span, key="gen_ai.response.id", value=response_obj.get("id")
span=span,
key="gen_ai.response.id",
value=response_obj.get("id"),
)
# The model used to generate the response.
@ -1639,7 +1697,9 @@ class OpenTelemetry(CustomLogger):
"OpenTelemetry logging error in set_attributes %s", str(e)
)
def _cast_as_primitive_value_type(self, value) -> Union[str, bool, int, float]:
def _cast_as_primitive_value_type(
self, value
) -> Union[str, bool, int, float]:
"""
Casts the value to a primitive OTEL type if it is not already a primitive type.
@ -1673,7 +1733,10 @@ class OpenTelemetry(CustomLogger):
if isinstance(messages, str):
# Handle system_instructions passed as a string
return [
{"role": "system", "parts": [{"type": "text", "content": messages}]}
{
"role": "system",
"parts": [{"type": "text", "content": messages}],
}
]
transformed = []
@ -1714,9 +1777,11 @@ class OpenTelemetry(CustomLogger):
message = choice.get("message") or {}
finish_reason = choice.get("finish_reason")
transformed_msg = self._transform_messages_to_otel_semantic_conventions(
[message]
)[0]
transformed_msg = (
self._transform_messages_to_otel_semantic_conventions(
[message]
)[0]
)
if finish_reason:
transformed_msg["finish_reason"] = finish_reason
@ -1728,7 +1793,9 @@ class OpenTelemetry(CustomLogger):
self.set_attributes(span, kwargs, response_obj)
kwargs.get("optional_params", {})
litellm_params = kwargs.get("litellm_params", {}) or {}
custom_llm_provider = litellm_params.get("custom_llm_provider", "Unknown")
custom_llm_provider = litellm_params.get(
"custom_llm_provider", "Unknown"
)
_raw_response = kwargs.get("original_response")
_additional_args = kwargs.get("additional_args", {}) or {}
@ -1741,7 +1808,9 @@ class OpenTelemetry(CustomLogger):
if complete_input_dict and isinstance(complete_input_dict, dict):
for param, val in complete_input_dict.items():
self.safe_set_attribute(
span=span, key=f"llm.{custom_llm_provider}.{param}", value=val
span=span,
key=f"llm.{custom_llm_provider}.{param}",
value=val,
)
#############################################
@ -1773,7 +1842,8 @@ class OpenTelemetry(CustomLogger):
)
except Exception as e:
verbose_logger.exception(
"OpenTelemetry logging error in set_raw_request_attributes %s", str(e)
"OpenTelemetry logging error in set_raw_request_attributes %s",
str(e),
)
def _to_ns(self, dt):
@ -1813,7 +1883,9 @@ class OpenTelemetry(CustomLogger):
)
litellm_params = kwargs.get("litellm_params", {}) or {}
proxy_server_request = litellm_params.get("proxy_server_request", {}) or {}
proxy_server_request = (
litellm_params.get("proxy_server_request", {}) or {}
)
headers = proxy_server_request.get("headers", {}) or {}
traceparent = headers.get("traceparent", None)
_metadata = litellm_params.get("metadata", {}) or {}
@ -1832,7 +1904,10 @@ class OpenTelemetry(CustomLogger):
"OpenTelemetry: Using traceparent header for context propagation"
)
carrier = {"traceparent": traceparent}
return TraceContextTextMapPropagator().extract(carrier=carrier), None
return (
TraceContextTextMapPropagator().extract(carrier=carrier),
None,
)
# Priority 3: Active span from global context (auto-detection)
try:
@ -1960,10 +2035,14 @@ class OpenTelemetry(CustomLogger):
self.OTEL_HEADERS,
)
_split_otel_headers = OpenTelemetry._get_headers_dictionary(self.OTEL_HEADERS)
_split_otel_headers = OpenTelemetry._get_headers_dictionary(
self.OTEL_HEADERS
)
# Normalize endpoint for logs - ensure it points to /v1/logs instead of /v1/traces
normalized_endpoint = self._normalize_otel_endpoint(self.OTEL_ENDPOINT, "logs")
normalized_endpoint = self._normalize_otel_endpoint(
self.OTEL_ENDPOINT, "logs"
)
verbose_logger.debug(
"OpenTelemetry: Log endpoint normalized from %s to %s",
@ -2051,14 +2130,18 @@ class OpenTelemetry(CustomLogger):
self.OTEL_HEADERS,
)
_split_otel_headers = OpenTelemetry._get_headers_dictionary(self.OTEL_HEADERS)
_split_otel_headers = OpenTelemetry._get_headers_dictionary(
self.OTEL_HEADERS
)
normalized_endpoint = self._normalize_otel_endpoint(
self.OTEL_ENDPOINT, "metrics"
)
if self.OTEL_EXPORTER == "console":
exporter = ConsoleMetricExporter()
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
return PeriodicExportingMetricReader(
exporter, export_interval_millis=5000
)
elif (
self.OTEL_EXPORTER == "otlp_http"
@ -2074,7 +2157,9 @@ class OpenTelemetry(CustomLogger):
headers=_split_otel_headers,
preferred_temporality={Histogram: AggregationTemporality.DELTA},
)
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
return PeriodicExportingMetricReader(
exporter, export_interval_millis=5000
)
elif self.OTEL_EXPORTER == "otlp_grpc" or self.OTEL_EXPORTER == "grpc":
try:
@ -2092,7 +2177,9 @@ class OpenTelemetry(CustomLogger):
headers=_split_otel_headers,
preferred_temporality={Histogram: AggregationTemporality.DELTA},
)
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
return PeriodicExportingMetricReader(
exporter, export_interval_millis=5000
)
else:
verbose_logger.warning(
@ -2100,7 +2187,9 @@ class OpenTelemetry(CustomLogger):
self.OTEL_EXPORTER,
)
exporter = ConsoleMetricExporter()
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
return PeriodicExportingMetricReader(
exporter, export_interval_millis=5000
)
def _normalize_otel_endpoint(
self, endpoint: Optional[str], signal_type: str
@ -2171,7 +2260,9 @@ class OpenTelemetry(CustomLogger):
return endpoint
@staticmethod
def _get_headers_dictionary(headers: Optional[Union[str, dict]]) -> Dict[str, str]:
def _get_headers_dictionary(
headers: Optional[Union[str, dict]],
) -> Dict[str, str]:
"""
Convert a string or dictionary of headers into a dictionary of headers.
"""

7
litellm/integrations/posthog.py
View file

@ -17,6 +17,7 @@ from typing import Any, Dict, Optional, Tuple
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.integrations.posthog_mock_client import (
should_use_posthog_mock,
create_mock_posthog_client,
@ -100,7 +101,7 @@ class PostHogLogger(CustomBatchLogger):
response = self.sync_client.post(
url=capture_url,
json=payload,
content=safe_dumps(payload),
headers=headers,
)
response.raise_for_status()
@ -356,7 +357,7 @@ class PostHogLogger(CustomBatchLogger):
response = await self.async_client.post(
url=capture_url,
json=payload,
content=safe_dumps(payload),
headers=headers,
)
response.raise_for_status()
@ -438,7 +439,7 @@ class PostHogLogger(CustomBatchLogger):
response = self.sync_client.post(
url=capture_url,
json=payload,
content=safe_dumps(payload),
headers=headers,
)
response.raise_for_status()

99
litellm/integrations/prometheus.py
View file

@ -1,6 +1,7 @@
# used for /metrics endpoint on LiteLLM Proxy
#### What this does ####
# On success, log events to Prometheus
import asyncio
import os
import sys
from datetime import datetime, timedelta
@ -28,7 +29,10 @@ from litellm.proxy._types import (
UserAPIKeyAuth,
)
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
from litellm.types.integrations.prometheus import (
_sanitize_prometheus_label_name,
_sanitize_prometheus_label_value,
)
from litellm.types.utils import StandardLoggingPayload
if TYPE_CHECKING:
@ -1188,28 +1192,34 @@ class PrometheusLogger(CustomLogger):
_user_spend = _metadata.get("user_api_key_user_spend", None)
_user_max_budget = _metadata.get("user_api_key_user_max_budget", None)
await self._set_api_key_budget_metrics_after_api_request(
user_api_key=user_api_key,
user_api_key_alias=user_api_key_alias,
response_cost=response_cost,
key_max_budget=_api_key_max_budget,
key_spend=_api_key_spend,
)
await self._set_team_budget_metrics_after_api_request(
user_api_team=user_api_team,
user_api_team_alias=user_api_team_alias,
team_spend=_team_spend,
team_max_budget=_team_max_budget,
response_cost=response_cost,
)
await self._set_user_budget_metrics_after_api_request(
user_id=user_id,
user_spend=_user_spend,
user_max_budget=_user_max_budget,
response_cost=response_cost,
results = await asyncio.gather(
self._set_api_key_budget_metrics_after_api_request(
user_api_key=user_api_key,
user_api_key_alias=user_api_key_alias,
response_cost=response_cost,
key_max_budget=_api_key_max_budget,
key_spend=_api_key_spend,
),
self._set_team_budget_metrics_after_api_request(
user_api_team=user_api_team,
user_api_team_alias=user_api_team_alias,
team_spend=_team_spend,
team_max_budget=_team_max_budget,
response_cost=response_cost,
),
self._set_user_budget_metrics_after_api_request(
user_id=user_id,
user_spend=_user_spend,
user_max_budget=_user_max_budget,
response_cost=response_cost,
),
return_exceptions=True,
)
for i, r in enumerate(results):
if isinstance(r, Exception):
verbose_logger.debug(
f"[Non-Blocking] Prometheus: Budget metric lookup {['key', 'team', 'user'][i]} failed: {r}"
)
def _increment_top_level_request_and_spend_metrics(
self,
@ -1269,11 +1279,17 @@ class PrometheusLogger(CustomLogger):
)
self.litellm_remaining_api_key_requests_for_model.labels(
user_api_key, user_api_key_alias, model_group, model_id
_sanitize_prometheus_label_value(user_api_key),
_sanitize_prometheus_label_value(user_api_key_alias),
_sanitize_prometheus_label_value(model_group),
_sanitize_prometheus_label_value(model_id),
).set(remaining_requests)
self.litellm_remaining_api_key_tokens_for_model.labels(
user_api_key, user_api_key_alias, model_group, model_id
_sanitize_prometheus_label_value(user_api_key),
_sanitize_prometheus_label_value(user_api_key_alias),
_sanitize_prometheus_label_value(model_group),
_sanitize_prometheus_label_value(model_id),
).set(remaining_tokens)
def _set_latency_metrics(
@ -1394,14 +1410,14 @@ class PrometheusLogger(CustomLogger):
try:
self.litellm_llm_api_failed_requests_metric.labels(
end_user_id,
user_api_key,
user_api_key_alias,
model,
user_api_team,
user_api_team_alias,
user_id,
standard_logging_payload.get("model_id", ""),
_sanitize_prometheus_label_value(end_user_id),
_sanitize_prometheus_label_value(user_api_key),
_sanitize_prometheus_label_value(user_api_key_alias),
_sanitize_prometheus_label_value(model),
_sanitize_prometheus_label_value(user_api_team),
_sanitize_prometheus_label_value(user_api_team_alias),
_sanitize_prometheus_label_value(user_id),
_sanitize_prometheus_label_value(standard_logging_payload.get("model_id", "")),
).inc()
self.set_llm_deployment_failure_metrics(kwargs)
except Exception as e:
@ -2347,7 +2363,11 @@ class PrometheusLogger(CustomLogger):
increment metric when litellm.Router / load balancing logic places a deployment in cool down
"""
self.litellm_deployment_cooled_down.labels(
litellm_model_name, model_id, api_base, api_provider, exception_status
_sanitize_prometheus_label_value(litellm_model_name),
_sanitize_prometheus_label_value(model_id),
_sanitize_prometheus_label_value(api_base),
_sanitize_prometheus_label_value(api_provider),
_sanitize_prometheus_label_value(exception_status),
).inc()
def increment_callback_logging_failure(
@ -2898,12 +2918,14 @@ class PrometheusLogger(CustomLogger):
max_budget=max_budget,
)
try:
# Note: Setting check_db_only=True bypasses cache and hits DB on every request,
# causing huge latency increase and CPU spikes. Keep check_db_only=False.
user_info = await get_user_object(
user_id=user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
check_db_only=True,
check_db_only=False,
)
except Exception as e:
verbose_logger.debug(
@ -3065,9 +3087,10 @@ def prometheus_label_factory(
# Extract dictionary from Pydantic object
enum_dict = enum_values.model_dump()
# Filter supported labels
# Filter supported labels and sanitize values to prevent breaking
# the Prometheus text format (e.g. U+2028 Line Separator in label values)
filtered_labels = {
label: value
label: _sanitize_prometheus_label_value(value)
for label, value in enum_dict.items()
if label in supported_enum_labels
}
@ -3085,14 +3108,14 @@ def prometheus_label_factory(
# check sanitized key
sanitized_key = _sanitize_prometheus_label_name(key)
if sanitized_key in supported_enum_labels:
filtered_labels[sanitized_key] = value
filtered_labels[sanitized_key] = _sanitize_prometheus_label_value(value)
# Add custom tags if configured
if enum_values.tags is not None:
custom_tag_labels = get_custom_labels_from_tags(enum_values.tags)
for key, value in custom_tag_labels.items():
if key in supported_enum_labels:
filtered_labels[key] = value
filtered_labels[key] = _sanitize_prometheus_label_value(value)
for label in supported_enum_labels:
if label not in filtered_labels:

269
litellm/integrations/websearch_interception/handler.py
View file

@ -17,6 +17,7 @@ from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.websearch_interception.tools import (
get_litellm_web_search_tool,
is_web_search_tool,
is_web_search_tool_chat_completion,
)
from litellm.integrations.websearch_interception.transformation import (
WebSearchTransformation,
@ -48,7 +49,8 @@ class WebSearchInterceptionLogger(CustomLogger):
Args:
enabled_providers: List of LLM providers to enable interception for.
Use LlmProviders enum values (e.g., [LlmProviders.BEDROCK])
Default: [LlmProviders.BEDROCK]
If None or empty list, enables for ALL providers.
Default: None (all providers enabled)
search_tool_name: Name of search tool configured in router's search_tools.
If None, will attempt to use first available search tool.
"""
@ -183,10 +185,10 @@ class WebSearchInterceptionLogger(CustomLogger):
verbose_logger.debug(
f"WebSearchInterception: Pre-request hook called"
f" - custom_llm_provider={custom_llm_provider}"
f" - enabled_providers={self.enabled_providers}"
f" - enabled_providers={self.enabled_providers or 'ALL'}"
)
if custom_llm_provider not in self.enabled_providers:
if self.enabled_providers is not None and custom_llm_provider not in self.enabled_providers:
verbose_logger.debug(
f"WebSearchInterception: Skipping - provider {custom_llm_provider} not in {self.enabled_providers}"
)
@ -245,7 +247,12 @@ class WebSearchInterceptionLogger(CustomLogger):
custom_llm_provider: str,
kwargs: Dict,
) -> Tuple[bool, Dict]:
"""Check if WebSearch tool interception is needed"""
"""
Check if WebSearch tool interception is needed for Anthropic Messages API.
This is the legacy method for Anthropic-style responses.
For chat completions, use async_should_run_chat_completion_agentic_loop instead.
"""
verbose_logger.debug(f"WebSearchInterception: Hook called! provider={custom_llm_provider}, stream={stream}")
verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
@ -253,7 +260,7 @@ class WebSearchInterceptionLogger(CustomLogger):
# Check if provider should be intercepted
# Note: custom_llm_provider is already normalized by get_llm_provider()
# (e.g., "bedrock/invoke/..." -> "bedrock")
if custom_llm_provider not in self.enabled_providers:
if self.enabled_providers is not None and custom_llm_provider not in self.enabled_providers:
verbose_logger.debug(
f"WebSearchInterception: Skipping provider {custom_llm_provider} (not in enabled list: {self.enabled_providers})"
)
@ -267,10 +274,11 @@ class WebSearchInterceptionLogger(CustomLogger):
)
return False, {}
# Detect WebSearch tool_use in response
# Detect WebSearch tool_use in response (Anthropic format)
should_intercept, tool_calls = WebSearchTransformation.transform_request(
response=response,
stream=stream,
response_format="anthropic",
)
if not should_intercept:
@ -288,6 +296,67 @@ class WebSearchInterceptionLogger(CustomLogger):
"tool_calls": tool_calls,
"tool_type": "websearch",
"provider": custom_llm_provider,
"response_format": "anthropic",
}
return True, tools_dict
async def async_should_run_chat_completion_agentic_loop(
self,
response: Any,
model: str,
messages: List[Dict],
tools: Optional[List[Dict]],
stream: bool,
custom_llm_provider: str,
kwargs: Dict,
) -> Tuple[bool, Dict]:
"""
Check if WebSearch tool interception is needed for Chat Completions API.
Similar to async_should_run_agentic_loop but for OpenAI-style chat completions.
"""
verbose_logger.debug(f"WebSearchInterception: Chat completion hook called! provider={custom_llm_provider}, stream={stream}")
verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
# Check if provider should be intercepted
if self.enabled_providers is not None and custom_llm_provider not in self.enabled_providers:
verbose_logger.debug(
f"WebSearchInterception: Skipping provider {custom_llm_provider} (not in enabled list: {self.enabled_providers})"
)
return False, {}
# Check if tools include any web search tool (strict check for chat completions)
has_websearch_tool = any(is_web_search_tool_chat_completion(t) for t in (tools or []))
if not has_websearch_tool:
verbose_logger.debug(
"WebSearchInterception: No litellm_web_search tool in request"
)
return False, {}
# Detect WebSearch tool_calls in response (OpenAI format)
should_intercept, tool_calls = WebSearchTransformation.transform_request(
response=response,
stream=stream,
response_format="openai",
)
if not should_intercept:
verbose_logger.debug(
"WebSearchInterception: No WebSearch tool_calls detected in response"
)
return False, {}
verbose_logger.debug(
f"WebSearchInterception: Detected {len(tool_calls)} WebSearch tool call(s), executing agentic loop"
)
# Return tools dict with tool calls
tools_dict = {
"tool_calls": tool_calls,
"tool_type": "websearch",
"provider": custom_llm_provider,
"response_format": "openai",
}
return True, tools_dict
@ -303,7 +372,11 @@ class WebSearchInterceptionLogger(CustomLogger):
stream: bool,
kwargs: Dict,
) -> Any:
"""Execute agentic loop with WebSearch execution"""
"""
Execute agentic loop with WebSearch execution for Anthropic Messages API.
This is the legacy method for Anthropic-style responses.
"""
tool_calls = tools["tool_calls"]
@ -321,6 +394,41 @@ class WebSearchInterceptionLogger(CustomLogger):
kwargs=kwargs,
)
async def async_run_chat_completion_agentic_loop(
self,
tools: Dict,
model: str,
messages: List[Dict],
response: Any,
optional_params: Dict,
logging_obj: Any,
stream: bool,
kwargs: Dict,
) -> Any:
"""
Execute agentic loop with WebSearch execution for Chat Completions API.
Similar to async_run_agentic_loop but for OpenAI-style chat completions.
"""
tool_calls = tools["tool_calls"]
response_format = tools.get("response_format", "openai")
verbose_logger.debug(
f"WebSearchInterception: Executing chat completion agentic loop for {len(tool_calls)} search(es)"
)
return await self._execute_chat_completion_agentic_loop(
model=model,
messages=messages,
tool_calls=tool_calls,
optional_params=optional_params,
logging_obj=logging_obj,
stream=stream,
kwargs=kwargs,
response_format=response_format,
)
async def _execute_agentic_loop(
self,
model: str,
@ -382,7 +490,8 @@ class WebSearchInterceptionLogger(CustomLogger):
)
# Make follow-up request with search results
follow_up_messages = messages + [assistant_message, user_message]
# Type cast: user_message is a Dict for Anthropic format (default response_format)
follow_up_messages = messages + [assistant_message, cast(Dict, user_message)]
verbose_logger.debug(
"WebSearchInterception: Making follow-up request with search results"
@ -521,6 +630,150 @@ class WebSearchInterceptionLogger(CustomLogger):
)
raise
async def _execute_chat_completion_agentic_loop( # noqa: PLR0915
self,
model: str,
messages: List[Dict],
tool_calls: List[Dict],
optional_params: Dict,
logging_obj: Any,
stream: bool,
kwargs: Dict,
response_format: str = "openai",
) -> Any:
"""Execute litellm.search() and make follow-up chat completion request"""
# Extract search queries from tool_calls
search_tasks = []
for tool_call in tool_calls:
# Handle both Anthropic-style input and OpenAI-style function.arguments
query = None
if "input" in tool_call and isinstance(tool_call["input"], dict):
query = tool_call["input"].get("query")
elif "function" in tool_call:
func = tool_call["function"]
if isinstance(func, dict):
args = func.get("arguments", {})
if isinstance(args, dict):
query = args.get("query")
if query:
verbose_logger.debug(
f"WebSearchInterception: Queuing search for query='{query}'"
)
search_tasks.append(self._execute_search(query))
else:
verbose_logger.warning(
f"WebSearchInterception: Tool call {tool_call.get('id')} has no query"
)
# Add empty result for tools without query
search_tasks.append(self._create_empty_search_result())
# Execute searches in parallel
verbose_logger.debug(
f"WebSearchInterception: Executing {len(search_tasks)} search(es) in parallel"
)
search_results = await asyncio.gather(*search_tasks, return_exceptions=True)
# Handle any exceptions in search results
final_search_results: List[str] = []
for i, result in enumerate(search_results):
if isinstance(result, Exception):
verbose_logger.error(
f"WebSearchInterception: Search {i} failed with error: {str(result)}"
)
final_search_results.append(
f"Search failed: {str(result)}"
)
elif isinstance(result, str):
final_search_results.append(cast(str, result))
else:
verbose_logger.warning(
f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
)
final_search_results.append(str(result))
# Build assistant and tool messages using transformation
assistant_message, tool_messages_or_user = WebSearchTransformation.transform_response(
tool_calls=tool_calls,
search_results=final_search_results,
response_format=response_format,
)
# Make follow-up request with search results
# For OpenAI format, tool_messages_or_user is a list of tool messages
if response_format == "openai":
follow_up_messages = messages + [assistant_message] + cast(List[Dict], tool_messages_or_user)
else:
# For Anthropic format (shouldn't happen in this method, but handle it)
follow_up_messages = messages + [assistant_message, cast(Dict, tool_messages_or_user)]
verbose_logger.debug(
"WebSearchInterception: Making follow-up chat completion request with search results"
)
verbose_logger.debug(
f"WebSearchInterception: Follow-up messages count: {len(follow_up_messages)}"
)
# Use litellm.acompletion for follow-up request
try:
# Remove internal parameters that shouldn't be passed to follow-up request
internal_params = {
'_websearch_interception',
'acompletion',
'litellm_logging_obj',
'custom_llm_provider',
'model_alias_map',
'stream_response',
'custom_prompt_dict',
}
kwargs_for_followup = {
k: v for k, v in kwargs.items()
if not k.startswith('_websearch_interception') and k not in internal_params
}
# Get full model name from kwargs
full_model_name = model
if "custom_llm_provider" in kwargs:
custom_llm_provider = kwargs["custom_llm_provider"]
# Reconstruct full model name with provider prefix if needed
if not model.startswith(custom_llm_provider):
# Check if model already has a provider prefix
if "/" not in model:
full_model_name = f"{custom_llm_provider}/{model}"
verbose_logger.debug(
f"WebSearchInterception: Using model name: {full_model_name}"
)
# Prepare tools for follow-up request (same as original)
tools_param = optional_params.get("tools")
# Remove tools and extra_body from optional_params to avoid issues
# extra_body often contains internal LiteLLM params that shouldn't be forwarded
optional_params_clean = {
k: v for k, v in optional_params.items()
if k not in {"tools", "extra_body", "model_alias_map","stream_response", "custom_prompt_dict" }
}
final_response = await litellm.acompletion(
model=full_model_name,
messages=follow_up_messages,
tools=tools_param,
**optional_params_clean,
**kwargs_for_followup,
)
verbose_logger.debug(
f"WebSearchInterception: Follow-up request completed, response type: {type(final_response)}"
)
return final_response
except Exception as e:
verbose_logger.exception(
f"WebSearchInterception: Follow-up request failed: {str(e)}"
)
raise
async def _create_empty_search_result(self) -> str:
"""Create an empty search result for tool calls without queries"""
return "No search query provided"

56
litellm/integrations/websearch_interception/tools.py
View file

@ -49,12 +49,57 @@ def get_litellm_web_search_tool() -> Dict[str, Any]:
}
def is_web_search_tool_chat_completion(tool: Dict[str, Any]) -> bool:
"""
Check if a tool is a web search tool for Chat Completions API (strict check).
This is a stricter version that ONLY checks for the exact LiteLLM web search tool name.
Use this for Chat Completions API to avoid false positives with user-defined tools.
Detects ONLY:
- LiteLLM standard: name == "litellm_web_search" (Anthropic format)
- OpenAI format: type == "function" with function.name == "litellm_web_search"
Args:
tool: Tool dictionary to check
Returns:
True if tool is exactly the LiteLLM web search tool
Example:
>>> is_web_search_tool_chat_completion({"name": "litellm_web_search"})
True
>>> is_web_search_tool_chat_completion({"type": "function", "function": {"name": "litellm_web_search"}})
True
>>> is_web_search_tool_chat_completion({"name": "web_search"})
False
>>> is_web_search_tool_chat_completion({"name": "WebSearch"})
False
"""
tool_name = tool.get("name", "")
tool_type = tool.get("type", "")
# Check for OpenAI format: {"type": "function", "function": {"name": "litellm_web_search"}}
if tool_type == "function" and "function" in tool:
function_def = tool.get("function", {})
function_name = function_def.get("name", "")
if function_name == LITELLM_WEB_SEARCH_TOOL_NAME:
return True
# Check for LiteLLM standard tool (Anthropic format)
if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME:
return True
return False
def is_web_search_tool(tool: Dict[str, Any]) -> bool:
"""
Check if a tool is a web search tool (native or LiteLLM standard).
Detects:
- LiteLLM standard: name == "litellm_web_search"
- OpenAI format: type == "function" with function.name == "litellm_web_search"
- Anthropic native: type starts with "web_search_" (e.g., "web_search_20250305")
- Claude Code: name == "web_search" with a type field
- Custom: name == "WebSearch" (legacy format)
@ -68,6 +113,8 @@ def is_web_search_tool(tool: Dict[str, Any]) -> bool:
Example:
>>> is_web_search_tool({"name": "litellm_web_search"})
True
>>> is_web_search_tool({"type": "function", "function": {"name": "litellm_web_search"}})
True
>>> is_web_search_tool({"type": "web_search_20250305", "name": "web_search"})
True
>>> is_web_search_tool({"name": "calculator"})
@ -75,8 +122,15 @@ def is_web_search_tool(tool: Dict[str, Any]) -> bool:
"""
tool_name = tool.get("name", "")
tool_type = tool.get("type", "")
# Check for OpenAI format: {"type": "function", "function": {"name": "..."}}
if tool_type == "function" and "function" in tool:
function_def = tool.get("function", {})
function_name = function_def.get("name", "")
if function_name == LITELLM_WEB_SEARCH_TOOL_NAME:
return True
# Check for LiteLLM standard tool
# Check for LiteLLM standard tool (Anthropic format)
if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME:
return True

190
litellm/integrations/websearch_interception/transformation.py
View file

@ -1,10 +1,10 @@
"""
WebSearch Tool Transformation
Transforms between Anthropic tool_use format and LiteLLM search format.
Transforms between Anthropic/OpenAI tool_use format and LiteLLM search format.
"""
from typing import Any, Dict, List, Tuple
import json
from typing import Any, Dict, List, Tuple, Union
from litellm._logging import verbose_logger
from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
@ -17,28 +17,31 @@ class WebSearchTransformation:
Handles transformation between:
- Anthropic tool_use format → LiteLLM search requests
- LiteLLM SearchResponse → Anthropic tool_result format
- OpenAI tool_calls format → LiteLLM search requests
- LiteLLM SearchResponse → Anthropic/OpenAI tool_result format
"""
@staticmethod
def transform_request(
response: Any,
stream: bool,
response_format: str = "anthropic",
) -> Tuple[bool, List[Dict]]:
"""
Transform Anthropic response to extract WebSearch tool calls.
Transform model response to extract WebSearch tool calls.
Detects if response contains WebSearch tool_use blocks and extracts
Detects if response contains WebSearch tool_use/tool_calls blocks and extracts
the search queries for execution.
Args:
response: Model response (dict or AnthropicMessagesResponse)
response: Model response (dict, AnthropicMessagesResponse, or ModelResponse)
stream: Whether response is streaming
response_format: Response format - "anthropic" or "openai" (default: "anthropic")
Returns:
(has_websearch, tool_calls):
has_websearch: True if WebSearch tool_use found
tool_calls: List of tool_use dicts with id, name, input
tool_calls: List of tool_use/tool_calls dicts with id, name, input/function
Note:
Streaming requests are handled by converting stream=True to stream=False
@ -54,8 +57,11 @@ class WebSearchTransformation:
)
return False, []
# Parse non-streaming response
return WebSearchTransformation._detect_from_non_streaming_response(response)
# Parse non-streaming response based on format
if response_format == "openai":
return WebSearchTransformation._detect_from_openai_response(response)
else:
return WebSearchTransformation._detect_from_non_streaming_response(response)
@staticmethod
def _detect_from_non_streaming_response(
@ -114,26 +120,142 @@ class WebSearchTransformation:
return len(tool_calls) > 0, tool_calls
@staticmethod
def _detect_from_openai_response(
response: Any,
) -> Tuple[bool, List[Dict]]:
"""Parse OpenAI-style response for WebSearch tool_calls"""
# Handle both dict and ModelResponse objects
if isinstance(response, dict):
choices = response.get("choices", [])
else:
if not hasattr(response, "choices"):
verbose_logger.debug(
"WebSearchInterception: Response has no choices attribute"
)
return False, []
choices = response.choices or []
if not choices:
verbose_logger.debug(
"WebSearchInterception: Response has empty choices"
)
return False, []
# Get first choice's message
first_choice = choices[0]
if isinstance(first_choice, dict):
message = first_choice.get("message", {})
else:
message = getattr(first_choice, "message", None)
if not message:
verbose_logger.debug(
"WebSearchInterception: First choice has no message"
)
return False, []
# Get tool_calls from message
if isinstance(message, dict):
openai_tool_calls = message.get("tool_calls", [])
else:
openai_tool_calls = getattr(message, "tool_calls", None) or []
if not openai_tool_calls:
verbose_logger.debug(
"WebSearchInterception: Message has no tool_calls"
)
return False, []
# Find all WebSearch tool calls
tool_calls = []
for tool_call in openai_tool_calls:
# Handle both dict and object tool calls
if isinstance(tool_call, dict):
tool_id = tool_call.get("id")
tool_type = tool_call.get("type")
function = tool_call.get("function", {})
function_name = function.get("name") if isinstance(function, dict) else getattr(function, "name", None)
function_arguments = function.get("arguments") if isinstance(function, dict) else getattr(function, "arguments", None)
else:
tool_id = getattr(tool_call, "id", None)
tool_type = getattr(tool_call, "type", None)
function = getattr(tool_call, "function", None)
function_name = getattr(function, "name", None) if function else None
function_arguments = getattr(function, "arguments", None) if function else None
# Check for LiteLLM standard or legacy web search tools
if tool_type == "function" and function_name in (
LITELLM_WEB_SEARCH_TOOL_NAME, "WebSearch", "web_search"
):
# Parse arguments (might be JSON string)
if isinstance(function_arguments, str):
try:
arguments = json.loads(function_arguments)
except json.JSONDecodeError:
verbose_logger.warning(
f"WebSearchInterception: Failed to parse function arguments: {function_arguments}"
)
arguments = {}
else:
arguments = function_arguments or {}
# Convert to internal format (similar to Anthropic)
tool_call_dict = {
"id": tool_id,
"type": "function",
"name": function_name,
"function": {
"name": function_name,
"arguments": arguments,
},
"input": arguments, # For compatibility with Anthropic format
}
tool_calls.append(tool_call_dict)
verbose_logger.debug(
f"WebSearchInterception: Found {function_name} tool_call with id={tool_id}"
)
return len(tool_calls) > 0, tool_calls
@staticmethod
def transform_response(
tool_calls: List[Dict],
search_results: List[str],
) -> Tuple[Dict, Dict]:
response_format: str = "anthropic",
) -> Tuple[Dict, Union[Dict, List[Dict]]]:
"""
Transform LiteLLM search results to Anthropic tool_result format.
Transform LiteLLM search results to Anthropic/OpenAI tool_result format.
Builds the assistant and user messages needed for the agentic loop
Builds the assistant and user/tool messages needed for the agentic loop
follow-up request.
Args:
tool_calls: List of tool_use dicts from transform_request
tool_calls: List of tool_use/tool_calls dicts from transform_request
search_results: List of search result strings (one per tool_call)
response_format: Response format - "anthropic" or "openai" (default: "anthropic")
Returns:
(assistant_message, user_message):
assistant_message: Message with tool_use blocks
user_message: Message with tool_result blocks
(assistant_message, user_or_tool_messages):
For Anthropic: assistant_message with tool_use blocks, user_message with tool_result blocks
For OpenAI: assistant_message with tool_calls, tool_messages list with tool results
"""
if response_format == "openai":
return WebSearchTransformation._transform_response_openai(
tool_calls, search_results
)
else:
return WebSearchTransformation._transform_response_anthropic(
tool_calls, search_results
)
@staticmethod
def _transform_response_anthropic(
tool_calls: List[Dict],
search_results: List[str],
) -> Tuple[Dict, Dict]:
"""Transform to Anthropic format (single user message with tool_result blocks)"""
# Build assistant message with tool_use blocks
assistant_message = {
"role": "assistant",
@ -163,6 +285,40 @@ class WebSearchTransformation:
return assistant_message, user_message
@staticmethod
def _transform_response_openai(
tool_calls: List[Dict],
search_results: List[str],
) -> Tuple[Dict, List[Dict]]:
"""Transform to OpenAI format (assistant with tool_calls, separate tool messages)"""
# Build assistant message with tool_calls
assistant_message = {
"role": "assistant",
"tool_calls": [
{
"id": tc["id"],
"type": "function",
"function": {
"name": tc["name"],
"arguments": json.dumps(tc["input"]) if isinstance(tc["input"], dict) else str(tc["input"]),
},
}
for tc in tool_calls
],
}
# Build separate tool messages (one per tool call)
tool_messages = [
{
"role": "tool",
"tool_call_id": tool_calls[i]["id"],
"content": search_results[i],
}
for i in range(len(tool_calls))
]
return assistant_message, tool_messages
@staticmethod
def format_search_response(result: SearchResponse) -> str:
"""

4
litellm/litellm_core_utils/core_helpers.py
View file

@ -94,8 +94,8 @@ def map_finish_reason(
return "length"
elif finish_reason == "tool_use": # anthropic
return "tool_calls"
elif finish_reason == "content_filtered":
return "content_filter"
elif finish_reason == "compaction":
return "length"
return finish_reason

21
litellm/litellm_core_utils/env_utils.py Normal file
View file

@ -0,0 +1,21 @@
"""
Utility helpers for reading and parsing environment variables.
"""
import os
def get_env_int(env_var: str, default: int) -> int:
"""Parse an environment variable as an integer, falling back to default on invalid values.
Handles empty strings, whitespace, and non-numeric values gracefully
so that misconfiguration doesn't crash the process at import time.
"""
raw = os.getenv(env_var)
if raw is None:
return default
raw = raw.strip()
try:
return int(raw)
except (ValueError, TypeError):
return default

82
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -1,19 +1,48 @@
from typing import Optional
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
_OPTIONAL_KWARGS_KEYS = frozenset({
"azure_ad_token",
"tenant_id",
"client_id",
"client_secret",
"azure_username",
"azure_password",
"azure_scope",
"timeout",
"bucket_name",
"vertex_credentials",
"vertex_project",
"vertex_location",
"vertex_ai_project",
"vertex_ai_location",
"vertex_ai_credentials",
"aws_region_name",
"aws_access_key_id",
"aws_secret_access_key",
"aws_session_token",
"aws_session_name",
"aws_profile_name",
"aws_role_name",
"aws_web_identity_token",
"aws_sts_endpoint",
"aws_external_id",
"aws_bedrock_runtime_endpoint",
"tpm",
"rpm",
})
def _get_base_model_from_litellm_call_metadata(
metadata: Optional[dict],
) -> Optional[str]:
if metadata is None:
return None
if metadata is not None:
model_info = metadata.get("model_info", {})
if model_info is not None:
base_model = model_info.get("base_model", None)
if base_model is not None:
return base_model
model_info = metadata.get("model_info")
if model_info:
return model_info.get("base_model")
return None
@ -66,6 +95,7 @@ def get_litellm_params(
litellm_request_debug: Optional[bool] = None,
**kwargs,
) -> dict:
# Build base dict with explicit parameters (always included)
litellm_params = {
"acompletion": acompletion,
"api_key": api_key,
@ -112,37 +142,15 @@ def get_litellm_params(
"ssl_verify": ssl_verify,
"merge_reasoning_content_in_choices": merge_reasoning_content_in_choices,
"api_version": api_version,
"azure_ad_token": kwargs.get("azure_ad_token"),
"tenant_id": kwargs.get("tenant_id"),
"client_id": kwargs.get("client_id"),
"client_secret": kwargs.get("client_secret"),
"azure_username": kwargs.get("azure_username"),
"azure_password": kwargs.get("azure_password"),
"azure_scope": kwargs.get("azure_scope"),
"max_retries": max_retries,
"timeout": kwargs.get("timeout"),
"bucket_name": kwargs.get("bucket_name"),
"vertex_credentials": kwargs.get("vertex_credentials"),
"vertex_project": kwargs.get("vertex_project"),
"vertex_location": kwargs.get("vertex_location"),
"vertex_ai_project": kwargs.get("vertex_ai_project"),
"vertex_ai_location": kwargs.get("vertex_ai_location"),
"vertex_ai_credentials": kwargs.get("vertex_ai_credentials"),
"use_litellm_proxy": use_litellm_proxy,
"litellm_request_debug": litellm_request_debug,
"aws_region_name": kwargs.get("aws_region_name"),
# AWS credentials for Bedrock/Sagemaker
"aws_access_key_id": kwargs.get("aws_access_key_id"),
"aws_secret_access_key": kwargs.get("aws_secret_access_key"),
"aws_session_token": kwargs.get("aws_session_token"),
"aws_session_name": kwargs.get("aws_session_name"),
"aws_profile_name": kwargs.get("aws_profile_name"),
"aws_role_name": kwargs.get("aws_role_name"),
"aws_web_identity_token": kwargs.get("aws_web_identity_token"),
"aws_sts_endpoint": kwargs.get("aws_sts_endpoint"),
"aws_external_id": kwargs.get("aws_external_id"),
"aws_bedrock_runtime_endpoint": kwargs.get("aws_bedrock_runtime_endpoint"),
"tpm": kwargs.get("tpm"),
"rpm": kwargs.get("rpm"),
}
# Sparse extraction: only add kwargs keys that are actually present
if kwargs:
for key in _OPTIONAL_KWARGS_KEYS:
if key in kwargs:
litellm_params[key] = kwargs[key]
return litellm_params

22
litellm/litellm_core_utils/litellm_logging.py
View file

@ -203,6 +203,10 @@ except Exception as e:
EnterpriseStandardLoggingPayloadSetupVAR = None
_in_memory_loggers: List[Any] = []
_STANDARD_LOGGING_METADATA_KEYS: frozenset = frozenset(
StandardLoggingMetadata.__annotations__.keys()
)
### GLOBAL VARIABLES ###
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
@ -522,7 +526,8 @@ class Logging(LiteLLMLoggingBaseClass):
}
self.litellm_request_debug = litellm_params.get("litellm_request_debug", False)
self.logger_fn = litellm_params.get("logger_fn", None)
verbose_logger.debug(f"self.optional_params: {self.optional_params}")
if _is_debugging_on() or self.litellm_request_debug:
verbose_logger.debug(f"self.optional_params: {self.optional_params}")
self.model_call_details.update(
{
@ -4515,17 +4520,12 @@ class StandardLoggingPayloadSetup:
user_api_key_auth_metadata=None,
)
if isinstance(metadata, dict):
# Filter the metadata dictionary to include only the specified keys
supported_keys = StandardLoggingMetadata.__annotations__.keys()
for key in supported_keys:
if key in metadata:
clean_metadata[key] = metadata[key] # type: ignore
for key in metadata.keys() & _STANDARD_LOGGING_METADATA_KEYS:
clean_metadata[key] = metadata[key] # type: ignore
if metadata.get("user_api_key") is not None:
if is_valid_sha256_hash(str(metadata.get("user_api_key"))):
clean_metadata["user_api_key_hash"] = metadata.get(
"user_api_key"
) # this is the hash
user_api_key = metadata.get("user_api_key")
if user_api_key and isinstance(user_api_key, str) and is_valid_sha256_hash(user_api_key):
clean_metadata["user_api_key_hash"] = user_api_key
_potential_requester_metadata = metadata.get(
"metadata", None
) # check if user passed metadata in the sdk request - e.g. metadata for langsmith logging - https://docs.litellm.ai/docs/observability/langsmith_integration#set-langsmith-fields

8
litellm/litellm_core_utils/logging_callback_manager.py
View file

@ -2,6 +2,7 @@ from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Set, Type, Uni
import litellm
from litellm._logging import verbose_logger
from litellm.constants import MAX_CALLBACKS
from litellm.integrations.additional_logging_utils import AdditionalLoggingUtils
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.generic_api.generic_api_callback import GenericAPILogger
@ -24,9 +25,6 @@ class LoggingCallbackManager:
- Keep a reasonable MAX_CALLBACKS limit (this ensures callbacks don't exponentially grow and consume CPU Resources)
"""
# healthy maximum number of callbacks - unlikely someone needs more than 20
MAX_CALLBACKS = 30
def add_litellm_input_callback(self, callback: Union[CustomLogger, str]):
"""
Add a input callback to litellm.input_callback
@ -155,9 +153,9 @@ class LoggingCallbackManager:
Check if adding another callback would exceed MAX_CALLBACKS
Returns True if safe to add, False if would exceed limit
"""
if len(parent_list) >= self.MAX_CALLBACKS:
if len(parent_list) >= MAX_CALLBACKS:
verbose_logger.warning(
f"Cannot add callback - would exceed MAX_CALLBACKS limit of {self.MAX_CALLBACKS}. Current callbacks: {len(parent_list)}"
f"Cannot add callback - would exceed MAX_CALLBACKS limit of {MAX_CALLBACKS}. Current callbacks: {len(parent_list)}"
)
return False
return True

56
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -1272,3 +1272,59 @@ def parse_tool_call_arguments(
)
raise ValueError(error_message) from e
def split_concatenated_json_objects(raw: str) -> List[Dict[str, Any]]:
"""
Split a string that contains one or more concatenated JSON objects into
a list of parsed dicts.
LLM providers (notably Bedrock Claude Sonnet 4.5) sometimes return
multiple tool-call argument objects concatenated in a single
``arguments`` string, e.g.::
'{"command":["curl",...]}{"command":["curl",...]}{"command":["curl",...]}'
``json.loads()`` fails on this with ``JSONDecodeError: Extra data``.
This helper uses ``json.JSONDecoder.raw_decode()`` to walk the string
and extract each JSON object individually.
Returns
-------
list[dict]
A list of parsed dicts – one per JSON object found. If *raw* is
empty or whitespace-only, an empty list is returned.
Raises
------
json.JSONDecodeError
If the string contains text that cannot be parsed as JSON at all.
"""
import json
raw = raw.strip()
if not raw:
return []
decoder = json.JSONDecoder()
results: List[Dict[str, Any]] = []
idx = 0
length = len(raw)
while idx < length:
# Skip whitespace between objects
while idx < length and raw[idx] in " \t\n\r":
idx += 1
if idx >= length:
break
obj, end_idx = decoder.raw_decode(raw, idx)
if isinstance(obj, dict):
results.append(obj)
else:
# Non-dict JSON value – wrap in empty dict (Bedrock requires
# toolUse.input to be an object).
results.append({})
idx = end_idx
return results

69
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -2190,6 +2190,16 @@ def anthropic_messages_pt( # noqa: PLR0915
while msg_i < len(messages) and messages[msg_i]["role"] == "assistant":
assistant_content_block: ChatCompletionAssistantMessage = messages[msg_i] # type: ignore
# Extract compaction_blocks from provider_specific_fields and add them first
_provider_specific_fields_raw = assistant_content_block.get(
"provider_specific_fields"
)
if isinstance(_provider_specific_fields_raw, dict):
_compaction_blocks = _provider_specific_fields_raw.get("compaction_blocks")
if _compaction_blocks and isinstance(_compaction_blocks, list):
# Add compaction blocks at the beginning of assistant content : https://platform.claude.com/docs/en/build-with-claude/compaction
assistant_content.extend(_compaction_blocks) # type: ignore
thinking_blocks = assistant_content_block.get("thinking_blocks", None)
if (
thinking_blocks is not None
@ -3277,25 +3287,68 @@ def _convert_to_bedrock_tool_call_invoke(
- extract name
- extract id
"""
from litellm.litellm_core_utils.prompt_templates.common_utils import (
split_concatenated_json_objects,
)
try:
_parts_list: List[BedrockContentBlock] = []
for tool in tool_calls:
if "function" in tool:
id = tool["id"]
tool_id = tool["id"]
name = tool["function"].get("name", "")
arguments = tool["function"].get("arguments", "")
arguments_dict = json.loads(arguments) if arguments else {}
# Ensure arguments_dict is always a dict (Bedrock requires toolUse.input to be an object)
# When some providers return arguments: '""' (JSON-encoded empty string), json.loads returns ""
if not isinstance(arguments_dict, dict):
arguments_dict = {}
if not arguments or not arguments.strip():
arguments_dict = {}
else:
arguments_dict = json.loads(arguments)
try:
arguments_dict = json.loads(arguments)
# Ensure arguments_dict is always a dict
# (Bedrock requires toolUse.input to be an object).
# Some providers return arguments: '""' which
# json.loads decodes to a bare string.
if not isinstance(arguments_dict, dict):
arguments_dict = {}
except json.JSONDecodeError:
# The model may return multiple JSON objects
# concatenated in a single arguments string, e.g.
# '{"cmd":"a"}{"cmd":"b"}{"cmd":"c"}'
# Split them and emit one toolUse block per object.
# Fixes: https://github.com/BerriAI/litellm/issues/20543
parsed_objects = split_concatenated_json_objects(
arguments
)
if parsed_objects:
# First object keeps the original tool id.
for obj_idx, obj in enumerate(parsed_objects):
block_id = (
tool_id
if obj_idx == 0
else f"{tool_id}_{obj_idx}"
)
bedrock_tool = BedrockToolUseBlock(
input=obj, name=name, toolUseId=block_id
)
_parts_list.append(
BedrockContentBlock(toolUse=bedrock_tool)
)
# cache_control applies to the whole original
# tool call; attach after the last split block.
if tool.get("cache_control", None) is not None:
_parts_list.append(
BedrockContentBlock(
cachePoint=CachePointBlock(
type="default"
)
)
)
continue
# Fallback: no objects extracted — use empty dict.
arguments_dict = {}
bedrock_tool = BedrockToolUseBlock(
input=arguments_dict, name=name, toolUseId=id
input=arguments_dict, name=name, toolUseId=tool_id
)
bedrock_content_block = BedrockContentBlock(toolUse=bedrock_tool)
_parts_list.append(bedrock_content_block)

58
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -1,6 +1,6 @@
import base64
import time
from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union, cast
from litellm.types.llms.openai import (
ChatCompletionAssistantContentValue,
@ -326,10 +326,22 @@ class ChunkProcessor:
thinking_blocks: List[
Union["ChatCompletionThinkingBlock", "ChatCompletionRedactedThinkingBlock"]
] = []
combined_thinking_text: Optional[str] = None
data: Optional[str] = None
signature: Optional[str] = None
type: Literal["thinking", "redacted_thinking"] = "thinking"
current_thinking_text_parts: List[str] = []
current_signature: Optional[str] = None
def _flush_thinking_block() -> None:
nonlocal current_thinking_text_parts, current_signature
if len(current_thinking_text_parts) > 0 and current_signature:
thinking_blocks.append(
ChatCompletionThinkingBlock(
type="thinking",
thinking="".join(current_thinking_text_parts),
signature=current_signature,
)
)
current_thinking_text_parts = []
current_signature = None
for chunk in chunks:
choices = chunk["choices"]
for choice in choices:
@ -339,33 +351,25 @@ class ChunkProcessor:
for thinking_block in thinking:
thinking_type = thinking_block.get("type", None)
if thinking_type and thinking_type == "redacted_thinking":
type = "redacted_thinking"
data = thinking_block.get("data", None)
_flush_thinking_block()
redacted_data = thinking_block.get("data", None)
if redacted_data:
thinking_blocks.append(
ChatCompletionRedactedThinkingBlock(
type="redacted_thinking",
data=redacted_data,
)
)
else:
type = "thinking"
thinking_text = thinking_block.get("thinking", None)
if thinking_text:
if combined_thinking_text is None:
combined_thinking_text = ""
combined_thinking_text += thinking_text
current_thinking_text_parts.append(thinking_text)
signature = thinking_block.get("signature", None)
if signature:
current_signature = signature
_flush_thinking_block()
if combined_thinking_text and type == "thinking" and signature:
thinking_blocks.append(
ChatCompletionThinkingBlock(
type=type,
thinking=combined_thinking_text,
signature=signature,
)
)
elif data and type == "redacted_thinking":
thinking_blocks.append(
ChatCompletionRedactedThinkingBlock(
type=type,
data=data,
)
)
_flush_thinking_block()
if len(thinking_blocks) > 0:
return thinking_blocks

4
litellm/litellm_core_utils/token_counter.py
View file

@ -706,7 +706,7 @@ def _count_content_list(
if isinstance(c, str):
num_tokens += count_function(c)
elif c["type"] == "text":
num_tokens += count_function(c.get("text", ""))
num_tokens += count_function(str(c.get("text", "")))
elif c["type"] == "image_url":
image_url = c.get("image_url")
num_tokens += _count_image_tokens(
@ -722,7 +722,7 @@ def _count_content_list(
elif c["type"] == "thinking":
# Claude extended thinking content block
# Count the thinking text and skip signature (opaque signature blob)
thinking_text = c.get("thinking", "")
thinking_text = str(c.get("thinking", ""))
if thinking_text:
num_tokens += count_function(thinking_text)
else:

155
litellm/llms/a2a/chat/guardrail_translation/README.md Normal file
View file

@ -0,0 +1,155 @@
# A2A Protocol Guardrail Translation Handler
Handler for processing A2A (Agent-to-Agent) Protocol messages with guardrails.
## Overview
This handler processes A2A JSON-RPC 2.0 input/output by:
1. Extracting text from message parts (`kind: "text"`)
2. Applying guardrails to text content
3. Mapping guardrailed text back to original structure
## A2A Protocol Format
### Input Format (JSON-RPC 2.0)
```json
{
"jsonrpc": "2.0",
"id": "request-id",
"method": "message/send",
"params": {
"message": {
"kind": "message",
"messageId": "...",
"role": "user",
"parts": [
{"kind": "text", "text": "Hello, my SSN is 123-45-6789"}
]
},
"metadata": {
"guardrails": ["block-ssn"]
}
}
}
```
### Output Formats
The handler supports multiple A2A response formats:
**Direct message:**
```json
{
"result": {
"kind": "message",
"parts": [{"kind": "text", "text": "Response text"}]
}
}
```
**Nested message:**
```json
{
"result": {
"message": {
"parts": [{"kind": "text", "text": "Response text"}]
}
}
}
```
**Task with artifacts:**
```json
{
"result": {
"kind": "task",
"artifacts": [
{"parts": [{"kind": "text", "text": "Artifact text"}]}
]
}
}
```
**Task with status message:**
```json
{
"result": {
"kind": "task",
"status": {
"message": {
"parts": [{"kind": "text", "text": "Status message"}]
}
}
}
}
```
**Streaming artifact-update:**
```json
{
"result": {
"kind": "artifact-update",
"artifact": {
"parts": [{"kind": "text", "text": "Streaming text"}]
}
}
}
```
## Usage
The handler is automatically discovered and applied when guardrails are used with A2A endpoints.
### Via LiteLLM Proxy
```bash
curl -X POST 'http://localhost:4000/a2a/my-agent' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer your-api-key' \
-d '{
"jsonrpc": "2.0",
"id": "1",
"method": "message/send",
"params": {
"message": {
"kind": "message",
"messageId": "msg-1",
"role": "user",
"parts": [{"kind": "text", "text": "Hello, my SSN is 123-45-6789"}]
},
"metadata": {
"guardrails": ["block-ssn"]
}
}
}'
```
### Specifying Guardrails
Guardrails can be specified in the A2A request via the `metadata.guardrails` field:
```json
{
"params": {
"message": {...},
"metadata": {
"guardrails": ["block-ssn", "pii-filter"]
}
}
}
```
## Extension
Override these methods to customize behavior:
- `_extract_texts_from_result()`: Custom text extraction from A2A responses
- `_extract_texts_from_parts()`: Custom text extraction from message parts
- `_apply_text_to_path()`: Custom application of guardrailed text
## Call Types
This handler is registered for:
- `CallTypes.send_message`: Synchronous A2A message sending
- `CallTypes.asend_message`: Asynchronous A2A message sending

11
litellm/llms/a2a/chat/guardrail_translation/__init__.py Normal file
View file

@ -0,0 +1,11 @@
"""A2A Protocol handler for Unified Guardrails."""
from litellm.llms.a2a.chat.guardrail_translation.handler import A2AGuardrailHandler
from litellm.types.utils import CallTypes
guardrail_translation_mappings = {
CallTypes.send_message: A2AGuardrailHandler,
CallTypes.asend_message: A2AGuardrailHandler,
}
__all__ = ["guardrail_translation_mappings"]

315
litellm/llms/a2a/chat/guardrail_translation/handler.py Normal file
View file

@ -0,0 +1,315 @@
"""
A2A Protocol Handler for Unified Guardrails
This module provides guardrail translation support for A2A (Agent-to-Agent) Protocol.
It handles both JSON-RPC 2.0 input requests and output responses, extracting text
from message parts and applying guardrails.
A2A Protocol Format:
- Input: JSON-RPC 2.0 with params.message.parts containing text parts
- Output: JSON-RPC 2.0 with result containing message/artifact parts
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.types.utils import GenericGuardrailAPIInputs
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
class A2AGuardrailHandler(BaseTranslation):
"""
Handler for processing A2A Protocol messages with guardrails.
This class provides methods to:
1. Process input messages (pre-call hook) - extracts text from A2A message parts
2. Process output responses (post-call hook) - extracts text from A2A response parts
A2A Message Format:
- Input: params.message.parts[].text (where kind == "text")
- Output: result.message.parts[].text or result.artifacts[].parts[].text
"""
async def process_input_messages(
self,
data: dict,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
) -> Any:
"""
Process A2A input messages by applying guardrails to text content.
Extracts text from A2A message parts and applies guardrails.
Args:
data: The A2A JSON-RPC 2.0 request data
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
Returns:
Modified data with guardrails applied to text content
"""
# A2A request format: { "params": { "message": { "parts": [...] } } }
params = data.get("params", {})
message = params.get("message", {})
parts = message.get("parts", [])
if not parts:
verbose_proxy_logger.debug("A2A: No parts in message, skipping guardrail")
return data
texts_to_check: List[str] = []
text_part_indices: List[int] = [] # Track which parts contain text
# Step 1: Extract text from all text parts
for part_idx, part in enumerate(parts):
if part.get("kind") == "text":
text = part.get("text", "")
if text:
texts_to_check.append(text)
text_part_indices.append(part_idx)
# Step 2: Apply guardrail to all texts in batch
if texts_to_check:
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
# Pass the structured A2A message to guardrails
inputs["structured_messages"] = [message]
# Include agent model info if available
model = data.get("model")
if model:
inputs["model"] = model
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=data,
input_type="request",
logging_obj=litellm_logging_obj,
)
guardrailed_texts = guardrailed_inputs.get("texts", [])
# Step 3: Apply guardrailed text back to original parts
if guardrailed_texts and len(guardrailed_texts) == len(text_part_indices):
for task_idx, part_idx in enumerate(text_part_indices):
parts[part_idx]["text"] = guardrailed_texts[task_idx]
verbose_proxy_logger.debug("A2A: Processed input message: %s", message)
return data
async def process_output_response(
self,
response: Any,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
) -> Any:
"""
Process A2A output response by applying guardrails to text content.
Handles multiple A2A response formats:
- Direct message: {"result": {"kind": "message", "parts": [...]}}
- Nested message: {"result": {"message": {"parts": [...]}}}
- Task with artifacts: {"result": {"kind": "task", "artifacts": [{"parts": [...]}]}}
- Task with status message: {"result": {"kind": "task", "status": {"message": {"parts": [...]}}}}
Args:
response: A2A JSON-RPC 2.0 response dict or object
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
user_api_key_dict: User API key metadata
Returns:
Modified response with guardrails applied to text content
"""
# Handle both dict and Pydantic model responses
if hasattr(response, "model_dump"):
response_dict = response.model_dump()
is_pydantic = True
elif isinstance(response, dict):
response_dict = response
is_pydantic = False
else:
verbose_proxy_logger.warning(
"A2A: Unknown response type %s, skipping guardrail", type(response)
)
return response
result = response_dict.get("result", {})
if not result or not isinstance(result, dict):
verbose_proxy_logger.debug("A2A: No result in response, skipping guardrail")
return response
# Find all text-containing parts in the response
texts_to_check: List[str] = []
# Each mapping is (path_to_parts_list, part_index)
# path_to_parts_list is a tuple of keys to navigate to the parts list
task_mappings: List[Tuple[Tuple[str, ...], int]] = []
# Extract texts from all possible locations
self._extract_texts_from_result(
result=result,
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
if not texts_to_check:
verbose_proxy_logger.debug("A2A: No text content in response")
return response
# Step 2: Apply guardrail to all texts in batch
# Create a request_data dict with response info and user API key metadata
request_data: dict = {"response": response_dict}
# Add user API key metadata with prefixed keys
user_metadata = self.transform_user_api_key_dict_to_metadata(user_api_key_dict)
if user_metadata:
request_data["litellm_metadata"] = user_metadata
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=request_data,
input_type="response",
logging_obj=litellm_logging_obj,
)
guardrailed_texts = guardrailed_inputs.get("texts", [])
# Step 3: Apply guardrailed text back to original response
if guardrailed_texts and len(guardrailed_texts) == len(task_mappings):
for task_idx, (path, part_idx) in enumerate(task_mappings):
self._apply_text_to_path(
result=result,
path=path,
part_idx=part_idx,
text=guardrailed_texts[task_idx],
)
verbose_proxy_logger.debug("A2A: Processed output response")
# Update the original response
if is_pydantic:
# For Pydantic models, we need to update the underlying dict
# and the model will reflect the changes
response_dict["result"] = result
return response
else:
response["result"] = result
return response
def _extract_texts_from_result(
self,
result: Dict[str, Any],
texts_to_check: List[str],
task_mappings: List[Tuple[Tuple[str, ...], int]],
) -> None:
"""
Extract text from all possible locations in an A2A result.
Handles multiple response formats:
1. Direct message with parts: {"parts": [...]}
2. Nested message: {"message": {"parts": [...]}}
3. Task with artifacts: {"artifacts": [{"parts": [...]}]}
4. Task with status message: {"status": {"message": {"parts": [...]}}}
5. Streaming artifact-update: {"artifact": {"parts": [...]}}
"""
# Case 1: Direct parts in result (direct message)
if "parts" in result:
self._extract_texts_from_parts(
parts=result["parts"],
path=("parts",),
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
# Case 2: Nested message
message = result.get("message")
if message and isinstance(message, dict) and "parts" in message:
self._extract_texts_from_parts(
parts=message["parts"],
path=("message", "parts"),
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
# Case 3: Streaming artifact-update (singular artifact)
artifact = result.get("artifact")
if artifact and isinstance(artifact, dict) and "parts" in artifact:
self._extract_texts_from_parts(
parts=artifact["parts"],
path=("artifact", "parts"),
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
# Case 4: Task with status message
status = result.get("status", {})
if isinstance(status, dict):
status_message = status.get("message")
if (
status_message
and isinstance(status_message, dict)
and "parts" in status_message
):
self._extract_texts_from_parts(
parts=status_message["parts"],
path=("status", "message", "parts"),
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
# Case 5: Task with artifacts (plural, array)
artifacts = result.get("artifacts", [])
if artifacts and isinstance(artifacts, list):
for artifact_idx, art in enumerate(artifacts):
if isinstance(art, dict) and "parts" in art:
self._extract_texts_from_parts(
parts=art["parts"],
path=("artifacts", str(artifact_idx), "parts"),
texts_to_check=texts_to_check,
task_mappings=task_mappings,
)
def _extract_texts_from_parts(
self,
parts: List[Dict[str, Any]],
path: Tuple[str, ...],
texts_to_check: List[str],
task_mappings: List[Tuple[Tuple[str, ...], int]],
) -> None:
"""Extract text from message parts."""
for part_idx, part in enumerate(parts):
if part.get("kind") == "text":
text = part.get("text", "")
if text:
texts_to_check.append(text)
task_mappings.append((path, part_idx))
def _apply_text_to_path(
self,
result: Dict[Union[str, int], Any],
path: Tuple[str, ...],
part_idx: int,
text: str,
) -> None:
"""Apply guardrailed text back to the specified path in the result."""
# Navigate to the parts list
current = result
for key in path:
if key.isdigit():
# Array index
current = current[int(key)]
else:
current = current[key]
# Update the text in the part
current[part_idx]["text"] = text

35
litellm/llms/anthropic/chat/handler.py
View file

@ -75,6 +75,7 @@ async def make_call(
logging_obj,
timeout: Optional[Union[float, httpx.Timeout]],
json_mode: bool,
speed: Optional[str] = None,
) -> Tuple[Any, httpx.Headers]:
if client is None:
client = litellm.module_level_aclient
@ -103,6 +104,7 @@ async def make_call(
streaming_response=response.aiter_lines(),
sync_stream=False,
json_mode=json_mode,
speed=speed,
)
# LOGGING
@ -126,6 +128,7 @@ def make_sync_call(
logging_obj,
timeout: Optional[Union[float, httpx.Timeout]],
json_mode: bool,
speed: Optional[str] = None,
) -> Tuple[Any, httpx.Headers]:
if client is None:
client = litellm.module_level_client # re-use a module level client
@ -159,7 +162,7 @@ def make_sync_call(
)
completion_stream = ModelResponseIterator(
streaming_response=response.iter_lines(), sync_stream=True, json_mode=json_mode
streaming_response=response.iter_lines(), sync_stream=True, json_mode=json_mode, speed=speed
)
# LOGGING
@ -213,6 +216,7 @@ class AnthropicChatCompletion(BaseLLM):
logging_obj=logging_obj,
timeout=timeout,
json_mode=json_mode,
speed=optional_params.get("speed") if optional_params else None,
)
streamwrapper = CustomStreamWrapper(
completion_stream=completion_stream,
@ -427,6 +431,7 @@ class AnthropicChatCompletion(BaseLLM):
logging_obj=logging_obj,
timeout=timeout,
json_mode=json_mode,
speed=optional_params.get("speed") if optional_params else None,
)
return CustomStreamWrapper(
completion_stream=completion_stream,
@ -485,13 +490,14 @@ class AnthropicChatCompletion(BaseLLM):
class ModelResponseIterator:
def __init__(
self, streaming_response, sync_stream: bool, json_mode: Optional[bool] = False
self, streaming_response, sync_stream: bool, json_mode: Optional[bool] = False, speed: Optional[str] = None
):
self.streaming_response = streaming_response
self.response_iterator = self.streaming_response
self.content_blocks: List[ContentBlockDelta] = []
self.tool_index = -1
self.json_mode = json_mode
self.speed = speed
# Generate response ID once per stream to match OpenAI-compatible behavior
self.response_id = _generate_id()
@ -512,6 +518,9 @@ class ModelResponseIterator:
# Accumulate web_search_tool_result blocks for multi-turn reconstruction
# See: https://github.com/BerriAI/litellm/issues/17737
self.web_search_results: List[Dict[str, Any]] = []
# Accumulate compaction blocks for multi-turn reconstruction
self.compaction_blocks: List[Dict[str, Any]] = []
def check_empty_tool_call_args(self) -> bool:
"""
@ -538,7 +547,7 @@ class ModelResponseIterator:
def _handle_usage(self, anthropic_usage_chunk: Union[dict, UsageDelta]) -> Usage:
return AnthropicConfig().calculate_usage(
usage_object=cast(dict, anthropic_usage_chunk), reasoning_content=None
usage_object=cast(dict, anthropic_usage_chunk), reasoning_content=None, speed=self.speed
)
def _content_block_delta_helper(self, chunk: dict) -> Tuple[
@ -592,6 +601,12 @@ class ModelResponseIterator:
)
]
provider_specific_fields["thinking_blocks"] = thinking_blocks
elif "content" in content_block["delta"] and content_block["delta"].get("type") == "compaction_delta":
# Handle compaction delta
provider_specific_fields["compaction_delta"] = {
"type": "compaction_delta",
"content": content_block["delta"]["content"]
}
return text, tool_use, thinking_blocks, provider_specific_fields
@ -721,6 +736,20 @@ class ModelResponseIterator:
provider_specific_fields=provider_specific_fields,
)
elif content_block_start["content_block"]["type"] == "compaction":
# Handle compaction blocks
# The full content comes in content_block_start
self.compaction_blocks.append(
content_block_start["content_block"]
)
provider_specific_fields["compaction_blocks"] = (
self.compaction_blocks
)
provider_specific_fields["compaction_start"] = {
"type": "compaction",
"content": content_block_start["content_block"].get("content", "")
}
elif content_block_start["content_block"]["type"].endswith("_tool_result"):
# Handle all tool result types (web_search, bash_code_execution, text_editor, etc.)
content_type = content_block_start["content_block"]["type"]

147
litellm/llms/anthropic/chat/transformation.py
View file

@ -170,9 +170,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
tool_call["caller"] = cast(Dict[str, Any], anthropic_tool_content["caller"]) # type: ignore[typeddict-item]
return tool_call
def _is_claude_opus_4_5(self, model: str) -> bool:
@staticmethod
def _is_claude_opus_4_6(model: str) -> bool:
"""Check if the model is Claude Opus 4.5."""
return "opus-4-5" in model.lower() or "opus_4_5" in model.lower()
return "opus-4-6" in model.lower() or "opus_4_6" in model.lower()
def get_supported_openai_params(self, model: str):
params = [
@ -189,6 +190,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
"response_format",
"user",
"web_search_options",
"speed",
]
if "claude-3-7-sonnet" in model or supports_reasoning(
@ -659,32 +661,38 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
@staticmethod
def _map_reasoning_effort(
reasoning_effort: Optional[Union[REASONING_EFFORT, str]],
reasoning_effort: Optional[Union[REASONING_EFFORT, str]],
model: str,
) -> Optional[AnthropicThinkingParam]:
if reasoning_effort is None:
return None
elif reasoning_effort == "low":
if AnthropicConfig._is_claude_opus_4_6(model):
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
)
elif reasoning_effort == "medium":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
)
elif reasoning_effort == "high":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
)
elif reasoning_effort == "minimal":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET,
type="adaptive",
)
else:
raise ValueError(f"Unmapped reasoning effort: {reasoning_effort}")
if reasoning_effort is None:
return None
elif reasoning_effort == "low":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
)
elif reasoning_effort == "medium":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
)
elif reasoning_effort == "high":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
)
elif reasoning_effort == "minimal":
return AnthropicThinkingParam(
type="enabled",
budget_tokens=DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET,
)
else:
raise ValueError(f"Unmapped reasoning effort: {reasoning_effort}")
def _extract_json_schema_from_response_format(
self, value: Optional[dict]
@ -860,13 +868,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if param == "thinking":
optional_params["thinking"] = value
elif param == "reasoning_effort" and isinstance(value, str):
# For Claude Opus 4.5, map reasoning_effort to output_config
if self._is_claude_opus_4_5(model):
optional_params["output_config"] = {"effort": value}
# For other models, map to thinking parameter
optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
value
reasoning_effort=value, model=model
)
elif param == "web_search_options" and isinstance(value, dict):
hosted_web_search_tool = self.map_web_search_tool(
@ -877,6 +880,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
elif param == "extra_headers":
optional_params["extra_headers"] = value
elif param == "context_management" and isinstance(value, dict):
# Pass through Anthropic-specific context_management parameter
optional_params["context_management"] = value
elif param == "speed" and isinstance(value, str):
# Pass through Anthropic-specific speed parameter for fast mode
optional_params["speed"] = value
## handle thinking tokens
self.update_optional_params_with_thinking_tokens(
@ -1026,9 +1035,37 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if beta_value not in existing_values:
headers["anthropic-beta"] = f"{existing_beta}, {beta_value}"
def _ensure_context_management_beta_header(self, headers: dict) -> None:
beta_value = ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
self._ensure_beta_header(headers, beta_value)
def _ensure_context_management_beta_header(
self, headers: dict, context_management: dict
) -> None:
"""
Add appropriate beta headers based on context_management edits.
- If any edit has type "compact_20260112", add compact-2026-01-12 header
- For all other edits, add context-management-2025-06-27 header
"""
edits = context_management.get("edits", [])
has_compact = False
has_other = False
for edit in edits:
edit_type = edit.get("type", "")
if edit_type == "compact_20260112":
has_compact = True
else:
has_other = True
# Add compact header if any compact edits exist
if has_compact:
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.COMPACT_2026_01_12.value
)
# Add context management header if any other edits exist
if has_other:
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
)
def update_headers_with_optional_anthropic_beta(
self, headers: dict, optional_params: dict
@ -1056,11 +1093,17 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
headers, ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
)
if optional_params.get("context_management") is not None:
self._ensure_context_management_beta_header(headers)
self._ensure_context_management_beta_header(
headers, optional_params["context_management"]
)
if optional_params.get("output_format") is not None:
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value
)
if optional_params.get("speed") == "fast":
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.FAST_MODE_2026_02_01.value
)
return headers
def transform_request(
@ -1225,6 +1268,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
List[ChatCompletionToolCallChunk],
Optional[List[Any]],
Optional[List[Any]],
Optional[List[Any]],
]:
text_content = ""
citations: Optional[List[Any]] = None
@ -1237,6 +1281,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
tool_calls: List[ChatCompletionToolCallChunk] = []
web_search_results: Optional[List[Any]] = None
tool_results: Optional[List[Any]] = None
compaction_blocks: Optional[List[Any]] = None
for idx, content in enumerate(completion_response["content"]):
if content["type"] == "text":
text_content += content["text"]
@ -1278,6 +1323,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
thinking_blocks.append(
cast(ChatCompletionRedactedThinkingBlock, content)
)
## COMPACTION
elif content["type"] == "compaction":
if compaction_blocks is None:
compaction_blocks = []
compaction_blocks.append(content)
## CITATIONS
if content.get("citations") is not None:
@ -1299,13 +1350,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if thinking_content is not None:
reasoning_content += thinking_content
return text_content, citations, thinking_blocks, reasoning_content, tool_calls, web_search_results, tool_results
return text_content, citations, thinking_blocks, reasoning_content, tool_calls, web_search_results, tool_results, compaction_blocks
def calculate_usage(
self,
usage_object: dict,
reasoning_content: Optional[str],
completion_response: Optional[dict] = None,
speed: Optional[str] = None,
) -> Usage:
# NOTE: Sometimes the usage object has None set explicitly for token counts, meaning .get() & key access returns None, and we need to account for this
prompt_tokens = usage_object.get("input_tokens", 0) or 0
@ -1316,6 +1368,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
cache_creation_token_details: Optional[CacheCreationTokenDetails] = None
web_search_requests: Optional[int] = None
tool_search_requests: Optional[int] = None
inference_geo: Optional[str] = None
if "inference_geo" in _usage and _usage["inference_geo"] is not None:
inference_geo = _usage["inference_geo"]
if (
"cache_creation_input_tokens" in _usage
and _usage["cache_creation_input_tokens"] is not None
@ -1399,6 +1455,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if (web_search_requests is not None or tool_search_requests is not None)
else None
),
inference_geo=inference_geo,
speed=speed,
)
return usage
@ -1409,6 +1467,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
model_response: ModelResponse,
json_mode: Optional[bool] = None,
prefix_prompt: Optional[str] = None,
speed: Optional[str] = None,
):
_hidden_params: Dict = {}
_hidden_params["additional_headers"] = process_anthropic_headers(
@ -1442,6 +1501,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
tool_calls,
web_search_results,
tool_results,
compaction_blocks,
) = self.extract_response_content(completion_response=completion_response)
if (
@ -1469,6 +1529,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
provider_specific_fields["tool_results"] = tool_results
if container is not None:
provider_specific_fields["container"] = container
if compaction_blocks is not None:
provider_specific_fields["compaction_blocks"] = compaction_blocks
_message = litellm.Message(
tool_calls=tool_calls,
@ -1477,6 +1539,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
thinking_blocks=thinking_blocks,
reasoning_content=reasoning_content,
)
_message.provider_specific_fields = provider_specific_fields
## HANDLE JSON MODE - anthropic returns single function call
json_mode_message = self._transform_response_for_json_mode(
@ -1501,24 +1564,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
usage_object=completion_response["usage"],
reasoning_content=reasoning_content,
completion_response=completion_response,
speed=speed,
)
setattr(model_response, "usage", usage) # type: ignore
model_response.created = int(time.time())
model_response.model = completion_response["model"]
context_management_response = completion_response.get("context_management")
if context_management_response is not None:
_hidden_params["context_management"] = context_management_response
try:
model_response.__dict__["context_management"] = (
context_management_response
)
except Exception:
pass
model_response._hidden_params = _hidden_params
return model_response
def get_prefix_prompt(self, messages: List[AllMessageValues]) -> Optional[str]:
@ -1580,6 +1633,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
prefix_prompt = self.get_prefix_prompt(messages=messages)
speed = optional_params.get("speed")
model_response = self.transform_parsed_response(
completion_response=completion_response,
@ -1587,6 +1641,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
model_response=model_response,
json_mode=json_mode,
prefix_prompt=prefix_prompt,
speed=speed,
)
return model_response

16
litellm/llms/anthropic/cost_calculation.py
View file

@ -22,10 +22,22 @@ def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
Returns:
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
"""
return generic_cost_per_token(
model=model, usage=usage, custom_llm_provider="anthropic"
model_with_prefix = model
# First, prepend inference_geo if present
if hasattr(usage, "inference_geo") and usage.inference_geo and usage.inference_geo.lower() not in ["global", "not_available"]:
model_with_prefix = f"{usage.inference_geo}/{model_with_prefix}"
# Then, prepend speed if it's "fast"
if hasattr(usage, "speed") and usage.speed == "fast":
model_with_prefix = f"fast/{model_with_prefix}"
prompt_cost, completion_cost = generic_cost_per_token(
model=model_with_prefix, usage=usage, custom_llm_provider="anthropic"
)
return prompt_cost, completion_cost
def get_cost_for_anthropic_web_search(
model_info: Optional["ModelInfo"] = None,

Some files were not shown because too many files have changed in this diff Show more