scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 123

Merge branch 'BerriAI:main' into main

Browse source
This commit is contained in:
mubashir1osmani 2026-03-09 14:51:24 -04:00 • committed by GitHub
commit 200910f7e6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1008 changed files with 36004 additions and 8932 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
.github/workflows/publish_enterprise.yml vendored
View file

@ -19,6 +19,7 @@ jobs:
if: github.repository == 'BerriAI/litellm'
permissions:
contents: write
pull-requests: write
defaults:
run:
working-directory: enterprise
@ -56,14 +57,33 @@ jobs:
- name: Build
run: poetry build
- name: Commit version bump
- name: Commit version bump and create PR
id: create-pr
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
cd ..
BRANCH="bump/enterprise-${{ steps.bump.outputs.new }}"
git checkout -b "$BRANCH"
git add enterprise/pyproject.toml pyproject.toml requirements.txt poetry.lock
git commit -m "bump: litellm-enterprise ${{ steps.bump.outputs.old }} → ${{ steps.bump.outputs.new }}"
git push
git push origin "$BRANCH" --force
gh pr create \
--title "bump: litellm-enterprise ${{ steps.bump.outputs.old }} → ${{ steps.bump.outputs.new }}" \
--body "Version bump for litellm-enterprise. Merge to update main." \
--head "$BRANCH" \
--base main \
|| true
PR_URL=$(gh pr list --head "$BRANCH" --json url -q '.[0].url')
echo "pr_url=$PR_URL" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ github.token }}
- name: Enable auto-merge
run: |
gh pr merge "${{ steps.create-pr.outputs.pr_url }}" --auto --squash
env:
GH_TOKEN: ${{ github.token }}
- name: Publish to PyPI
env:

4
AGENTS.md
View file

@ -212,6 +212,8 @@ When opening issues or pull requests, follow these templates:
Using helpers like `supports_reasoning` (which read from `model_prices_and_context_window.json` / `get_model_info`) allows future model updates to "just work" without code changes.
9. **Never close HTTP/SDK clients on cache eviction**: Do not add `close()`, `aclose()`, or `create_task(close_fn())` inside `LLMClientCache._remove_key()` or any cache eviction path. Evicted clients may still be held by in-flight requests; closing them causes `RuntimeError: Cannot send a request, as the client has been closed.` in production after the cache TTL (1 hour) expires. Connection cleanup is handled at shutdown by `close_litellm_async_clients()`. See PR #22247 for the full incident history.
## HELPFUL RESOURCES
- Main documentation: https://docs.litellm.ai/
@ -249,9 +251,11 @@ The proxy takes ~15-20 seconds to fully start (it runs Prisma migrations on boot
See `CLAUDE.md` and the `Makefile` for standard commands. Key notes:
- `psycopg-binary` must be installed (`poetry run pip install psycopg-binary`) because the pytest-postgresql plugin requires it and the lock file only includes `psycopg` (no binary).
- `openapi-core` must be installed (`poetry run pip install openapi-core`) for the OpenAPI compliance tests in `tests/test_litellm/interactions/`.
- The `--timeout` pytest flag is NOT available; don't pass it.
- Unit tests: `poetry run pytest tests/test_litellm/ -x -vv -n 4`
- Black `--check` may report pre-existing formatting issues; this does not block test runs.
- If `poetry install` fails with "pyproject.toml changed significantly since poetry.lock was last generated", run `poetry lock` first to regenerate the lock file.
### Lint

15
CLAUDE.md
View file

@ -114,4 +114,17 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
### Enterprise Features
- Enterprise-specific code in `enterprise/` directory
- Optional features enabled via environment variables
- Separate licensing and authentication for enterprise features
- Separate licensing and authentication for enterprise features
### HTTP Client Cache Safety
- **Never close HTTP/SDK clients on cache eviction.** `LLMClientCache._remove_key()` must not call `close()`/`aclose()` on evicted clients — they may still be used by in-flight requests. Doing so causes `RuntimeError: Cannot send a request, as the client has been closed.` after the 1-hour TTL expires. Cleanup happens at shutdown via `close_litellm_async_clients()`.
### Troubleshooting: DB schema out of sync after proxy restart
`litellm-proxy-extras` runs `prisma migrate deploy` on startup using **its own** bundled migration files, which may lag behind schema changes in the current worktree. Symptoms: `Unknown column`, `Invalid prisma invocation`, or missing data on new fields.
**Diagnose:** Run `\d "TableName"` in psql and compare against `schema.prisma` — missing columns confirm the issue.
**Fix options:**
1. **Create a Prisma migration** (permanent) — run `prisma migrate dev --name <description>` in the worktree. The generated file will be picked up by `prisma migrate deploy` on next startup.
2. **Apply manually for local dev** — `psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF NOT EXISTS ..."` after each proxy start. Fine for dev, not for production.
3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.

13
Dockerfile
View file

@ -49,7 +49,7 @@ USER root
# Install runtime dependencies (libsndfile needed for audio processing on ARM64)
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
# SECURITY FIX: npm bundles tar, glob, and brace-expansion at multiple nested
# levels inside its dependency tree. `npm install -g <pkg>` only creates a
# SEPARATE global package, it does NOT replace npm's internal copies.
@ -70,7 +70,15 @@ RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
# SECURITY FIX: patch npm's own package.json metadata so scanners see the
# actual installed versions instead of the stale declared dependencies.
find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
npm cache clean --force && \
# Remove the apk-tracked npm so its stale SBOM metadata (tar 7.5.9) is
# no longer visible to image scanners. The globally installed npm@latest
# at /usr/local/lib/node_modules/npm/ remains fully functional.
{ apk del --no-cache npm 2>/dev/null || true; }
WORKDIR /app
# Copy the current directory contents into the container at /app
@ -96,6 +104,7 @@ RUN find /usr/lib -type f -path "*/tornado/test/*" -delete && \
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
[ -n "$GLOBAL" ] || { echo "ERROR: npm root -g returned empty; aborting"; exit 1; } && \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \

2
ci_cd/security_scans.sh
View file

@ -161,6 +161,8 @@ run_grype_scans() {
"GHSA-3ppc-4f35-3m26" # minimatch ReDoS via repeated wildcards - from nodejs_wheel bundled npm, not used in application runtime code
"GHSA-83g3-92jg-28cx" # tar arbitrary file read/write via hardlink - from nodejs_wheel bundled npm, not used in application runtime code
"CVE-2026-25639" # axios - full fix requires 1.x major version bump; pinned to >=0.30.2 to clear other axios CVEs, upgrade to 1.x in follow-up
"CVE-2026-2297" # Python 3.13 SourcelessFileLoader audit hook bypass - no fix available in base image
"GHSA-qffp-2rhf-9h96" # tar hardlink path traversal - from nodejs_wheel bundled npm, not used in application runtime code
)
# Build JSON array of allowlisted CVE IDs for jq

7
docker/Dockerfile.custom_ui
View file

@ -19,7 +19,7 @@ RUN apt-get update && apt-get upgrade -y \
libgnutls30 \
libc6 && \
apt-get install -y nodejs npm && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -36,7 +36,10 @@ RUN apt-get update && apt-get upgrade -y \
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
npm cache clean --force && \
apt-get purge -y npm
# Copy the UI source into the container
COPY ./ui/litellm-dashboard /app/ui/litellm-dashboard

8
docker/Dockerfile.database
View file

@ -50,7 +50,7 @@ USER root
# Install runtime dependencies
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 && \
npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -67,7 +67,10 @@ RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
npm cache clean --force && \
{ apk del --no-cache npm 2>/dev/null || true; }
WORKDIR /app
# Copy the current directory contents into the container at /app
@ -85,6 +88,7 @@ RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ && rm -f *.whl
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
[ -n "$GLOBAL" ] || { echo "ERROR: npm root -g returned empty; aborting"; exit 1; } && \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \

8
docker/Dockerfile.dev
View file

@ -75,7 +75,7 @@ RUN apt-get update && apt-get upgrade -y \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/* \
&& npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 \
&& npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -92,7 +92,10 @@ RUN apt-get update && apt-get upgrade -y \
&& find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done \
&& npm cache clean --force
&& find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null \
&& npm cache clean --force \
&& apt-get purge -y npm
WORKDIR /app
@ -114,6 +117,7 @@ RUN pip install --no-cache-dir *.whl /wheels/* --no-index --find-links=/wheels/
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
[ -n "$GLOBAL" ] || { echo "ERROR: npm root -g returned empty; aborting"; exit 1; } && \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \

8
docker/Dockerfile.non_root
View file

@ -106,7 +106,7 @@ RUN for i in 1 2 3; do \
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor && break || sleep 5; \
done \
&& apk upgrade --no-cache nodejs \
&& npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 \
&& npm install -g npm@latest tar@7.5.10 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@ -123,7 +123,10 @@ RUN for i in 1 2 3; do \
&& find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done \
&& npm cache clean --force
&& find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null \
&& npm cache clean --force \
&& { apk del --no-cache npm 2>/dev/null || true; }
# Copy artifacts from builder
COPY --from=builder /app/requirements.txt /app/requirements.txt
@ -169,6 +172,7 @@ RUN pip install --no-index --find-links=/wheels/ -r requirements.txt && \
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
[ -n "$GLOBAL" ] || { echo "ERROR: npm root -g returned empty; aborting"; exit 1; } && \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \

97
docs/my-website/blog/gpt_5_4/index.md Normal file
View file

@ -0,0 +1,97 @@
---
slug: gpt_5_4
title: "Day 0 Support: GPT-5.4"
date: 2026-03-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
description: "GPT-5.4 model support in LiteLLM"
tags: [openai, gpt-5.4, completion]
hide_table_of_contents: false
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
LiteLLM now supports fully GPT-5.4!
## Docker Image
```bash
docker pull ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch
```
## Usage
<Tabs>
<TabItem value="proxy" label="LiteLLM Proxy">
**1. Setup config.yaml**
```yaml
model_list:
- model_name: gpt-5.4
litellm_params:
model: openai/gpt-5.4
api_key: os.environ/OPENAI_API_KEY
```
**2. Start the proxy**
```bash
docker run -d \
-p 4000:4000 \
-e OPENAI_API_KEY=$OPENAI_API_KEY \
-v $(pwd)/config.yaml:/app/config.yaml \
ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch \
--config /app/config.yaml
```
**3. Test it**
```bash
curl -X POST "http://0.0.0.0:4000/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_KEY" \
-d '{
"model": "gpt-5.4",
"messages": [
{"role": "user", "content": "Write a Python function to check if a number is prime."}
]
}'
```
</TabItem>
<TabItem value="sdk" label="LiteLLM SDK">
```python
from litellm import completion
response = completion(
model="openai/gpt-5.4",
messages=[
{"role": "user", "content": "Write a Python function to check if a number is prime."}
],
)
print(response.choices[0].message.content)
```
</TabItem>
</Tabs>
## Notes
- Restart your container to get the cost tracking for this model.
- Use `/responses` for better model performance.
- GPT-5.4 supports reasoning, function calling, vision, and tool-use — see the [OpenAI provider docs](../../docs/providers/openai) for advanced usage.

1
docs/my-website/docs/a2a.md
View file

@ -20,6 +20,7 @@ Add A2A Agents on LiteLLM AI Gateway, Invoke agents in A2A Protocol, track reque
| Logging | ✅ |
| Load Balancing | ✅ |
| Streaming | ✅ |
| [Iteration Budgets](a2a_iteration_budgets) | ✅ |
:::tip

252
docs/my-website/docs/a2a_agent_headers.md Normal file
View file

@ -0,0 +1,252 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# A2A Agent Authentication Headers
Forward authentication credentials (Bearer tokens, API keys, etc.) from clients to backend A2A agents.
## Overview
When LiteLLM proxies a request to a backend A2A agent, the agent may require its own authentication headers. There are three ways to supply them:
| Method | Who configures | How it works |
|---|---|---|
| **Static headers** | Admin (UI / API) | Always sent, regardless of client request |
| **Forward client headers** | Admin (UI / API) | Header names to extract from client request and forward |
| **Convention-based** | Client (no admin config) | Client sends `x-a2a-{agent_name}-{header}` — automatically routed |
All three methods can be combined. **Static headers always win** on key conflicts.
---
## Method 1 — Static Headers
Admin-configured headers that are always sent to the backend agent. Use this for server-to-server tokens or internal credentials that clients should never see or override.
<Tabs>
<TabItem value="ui" label="UI">
1. Go to **Agents** in the LiteLLM dashboard.
2. Create or edit an agent.
3. Open the **Authentication Headers** panel.
4. Under **Static Headers**, click **Add Static Header** and fill in the header name and value.
</TabItem>
<TabItem value="api" label="REST API">
```bash
curl -X POST http://localhost:4000/v1/agents \
-H "Authorization: Bearer sk-admin" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "my-agent",
"agent_card_params": { ... },
"static_headers": {
"Authorization": "Bearer internal-server-token",
"X-Internal-Service": "litellm-proxy"
}
}'
```
To update an existing agent:
```bash
curl -X PATCH http://localhost:4000/v1/agents/{agent_id} \
-H "Authorization: Bearer sk-admin" \
-H "Content-Type: application/json" \
-d '{
"static_headers": {
"Authorization": "Bearer new-token"
}
}'
```
</TabItem>
</Tabs>
**Client call — no special headers needed:**
```bash
curl -X POST http://localhost:4000/a2a/my-agent \
-H "Authorization: Bearer sk-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0", "id": "1", "method": "message/send",
"params": { "message": { "role": "user", "parts": [{"kind": "text", "text": "Hello"}], "messageId": "msg-1" } }
}'
```
The backend agent receives `Authorization: Bearer internal-server-token` without the client ever knowing the value.
---
## Method 2 — Forward Client Headers
Admin specifies a list of header **names**. When the client sends a request that includes those headers, LiteLLM extracts their values and forwards them to the backend agent. The client controls the values; the admin controls which headers are eligible to be forwarded.
<Tabs>
<TabItem value="ui" label="UI">
1. Go to **Agents** in the LiteLLM dashboard.
2. Create or edit an agent.
3. Open the **Authentication Headers** panel.
4. Under **Forward Client Headers**, type header names and press **Enter** (e.g. `x-api-key`, `Authorization`).
</TabItem>
<TabItem value="api" label="REST API">
```bash
curl -X POST http://localhost:4000/v1/agents \
-H "Authorization: Bearer sk-admin" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "my-agent",
"agent_card_params": { ... },
"extra_headers": ["x-api-key", "x-user-token"]
}'
```
</TabItem>
</Tabs>
**Client call — include the forwarded headers:**
```bash
curl -X POST http://localhost:4000/a2a/my-agent \
-H "Authorization: Bearer sk-client-key" \
-H "x-api-key: user-secret-value" \
-H "Content-Type: application/json" \
-d '{ ... }'
```
The backend agent receives `x-api-key: user-secret-value`.
:::note
Header name matching is **case-insensitive**. If the client sends `X-API-Key` and `extra_headers` lists `x-api-key`, they match.
:::
---
## Method 3 — Convention-Based Forwarding
Clients can forward headers to a specific agent without any admin pre-configuration by using the naming convention:
```
x-a2a-{agent_name_or_id}-{header_name}: value
```
LiteLLM parses these headers automatically and routes them to the matching agent only.
**Examples:**
| Client header sent | Agent name/ID | Forwarded as |
|---|---|---|
| `x-a2a-my-agent-authorization: Bearer tok` | `my-agent` | `authorization: Bearer tok` |
| `x-a2a-my-agent-x-api-key: secret` | `my-agent` | `x-api-key: secret` |
| `x-a2a-abc123-authorization: Bearer tok` | agent ID `abc123` | `authorization: Bearer tok` |
```bash
curl -X POST http://localhost:4000/a2a/my-agent \
-H "Authorization: Bearer sk-client-key" \
-H "x-a2a-my-agent-authorization: Bearer agent-specific-token" \
-H "Content-Type: application/json" \
-d '{ ... }'
```
The `x-a2a-other-agent-authorization` header sent in the same request is **not** forwarded to `my-agent` — it is silently ignored.
:::tip Matches both agent name and agent ID
Both the human-readable name (e.g. `my-agent`) and the UUID (e.g. `abc123-...`) are valid. Use whichever is convenient for the client.
:::
---
## Merge Precedence
When multiple methods supply the same header name, **static headers win**:
```
dynamic (forwarded/convention) → merged ← static (overlays, wins)
```
Example:
| Source | `Authorization` value |
|---|---|
| Client sends (via `extra_headers` or convention) | `Bearer client-token` |
| Admin-configured `static_headers` | `Bearer server-token` |
| **What the backend agent receives** | **`Bearer server-token`** |
This ensures admin-controlled credentials cannot be overridden by client requests.
---
## Combining All Three Methods
```bash
# Register agent with static + forwarded headers
curl -X POST http://localhost:4000/v1/agents \
-H "Authorization: Bearer sk-admin" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "my-agent",
"agent_card_params": { ... },
"static_headers": {
"X-Internal-Token": "secret123"
},
"extra_headers": ["x-user-id"]
}'
# Client call using all three mechanisms
curl -X POST http://localhost:4000/a2a/my-agent \
-H "Authorization: Bearer sk-client-key" \
-H "x-user-id: user-42" \
-H "x-a2a-my-agent-x-request-id: req-abc" \
-H "Content-Type: application/json" \
-d '{ ... }'
```
The backend agent receives:
```
X-Internal-Token: secret123 ← static header (always)
x-user-id: user-42 ← forwarded (in extra_headers)
x-request-id: req-abc ← convention-based (x-a2a-my-agent-*)
X-LiteLLM-Trace-Id: <uuid> ← LiteLLM internal
X-LiteLLM-Agent-Id: <agent-id> ← LiteLLM internal
```
---
## Header Isolation
Each agent invocation uses an isolated HTTP connection. Headers configured for agent A are **never** sent to agent B, even if both agents are running and receiving requests simultaneously.
---
## API Reference
### `POST /v1/agents` / `PATCH /v1/agents/{agent_id}`
| Field | Type | Description |
|---|---|---|
| `static_headers` | `object` | `{"Header-Name": "value"}` — always forwarded |
| `extra_headers` | `string[]` | Header names to extract from client request and forward |
### Agent Response
Both fields are returned in `GET /v1/agents` and `GET /v1/agents/{agent_id}`:
```json
{
"agent_id": "...",
"agent_name": "my-agent",
"static_headers": { "X-Internal-Token": "secret123" },
"extra_headers": ["x-user-id"],
...
}
```
:::caution
`static_headers` values are stored in the database and returned by the API. Treat them as you would any credential — do not store sensitive long-lived tokens here if your API is publicly accessible. Consider using short-lived tokens or environment-injected secrets instead.
:::

188
docs/my-website/docs/a2a_iteration_budgets.md Normal file
View file

@ -0,0 +1,188 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Agent Iteration Budgets
Control runaway costs from agentic loops with per-session iteration and budget caps.
## Overview
When agents run agentic loops, they can make unbounded LLM calls, causing unexpected costs. LiteLLM provides two controls:
| Control | Description |
|---------|-------------|
| **Max Iterations** | Hard cap on the number of LLM calls per session |
| **Max Budget Per Session** | Dollar cap per session (identified by `x-litellm-trace-id`) |
Both controls require a `session_id` (sent via `x-litellm-trace-id` header or `metadata.session_id`) to track calls within a session.
## Trace-ID Enforcement
LiteLLM supports two independent trace-id flags, configured in `litellm_params` on the agent:
| Flag | Description |
|------|-------------|
| `require_trace_id_on_calls_to_agent` | Requires callers invoking this agent to include `x-litellm-trace-id`. Use when the agent should only be called as a sub-agent with a trace context. Returns **400** if missing. |
| `require_trace_id_on_calls_by_agent` | Requires all LLM/MCP calls made **by** this agent (via its virtual key) to include `x-litellm-trace-id`. This is what enables `max_iterations` and `max_budget_per_session` tracking. Returns **400** if missing. |
## Configuring via UI
When creating an agent in the LiteLLM Admin UI:
1. Navigate to the **Agents** tab and click **Add Agent**
2. In the **Agent Settings** step, expand the **Tracing** section
3. Toggle **Require x-litellm-trace-id on calls BY this agent** to enable session tracking
4. Set **Max Iterations** to cap the number of LLM calls per session
5. Set **Max Budget Per Session ($)** to cap spend per session
The trace-id flags are stored on the agent's `litellm_params`. Budget controls (`max_iterations`, `max_budget_per_session`) are stored in the virtual key's metadata.
## Configuring via API
Set trace-id enforcement on the agent itself:
```bash
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent with budget controls",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"litellm_params": {
"require_trace_id_on_calls_to_agent": true,
"require_trace_id_on_calls_by_agent": true
}
}'
```
Budget controls are set on the agent's `litellm_params` (not on individual keys), so they apply across all keys for the agent:
```bash
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent with budget controls",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"litellm_params": {
"require_trace_id_on_calls_by_agent": true,
"max_iterations": 25,
"max_budget_per_session": 5.00
}
}'
```
## How It Works
### Session Tracking
Callers identify their session by including a `session_id` in one of these ways:
- **Header**: `x-litellm-trace-id: my-session-123`
- **Metadata**: `{"metadata": {"session_id": "my-session-123"}}`
### Max Iterations
When `max_iterations` is set in agent `litellm_params`:
- Each LLM call for a session increments a counter
- When the counter exceeds `max_iterations`, the request receives a **429 Too Many Requests**
- Counters expire after 1 hour by default (configurable via `LITELLM_MAX_ITERATIONS_TTL` env var)
### Max Budget Per Session
When `max_budget_per_session` is set in agent `litellm_params`:
- After each successful LLM call, the response cost is accumulated for the session
- Before each call, the accumulated spend is checked against the budget
- When spend exceeds the budget, the request receives a **429 Too Many Requests**
- Session spend counters expire after 1 hour by default (configurable via `LITELLM_MAX_BUDGET_PER_SESSION_TTL` env var)
## Example
Create an agent with max 25 iterations and a $5 budget cap:
<Tabs>
<TabItem value="ui" label="Via UI">
1. Go to **Agents** → **Add Agent**
2. Configure your agent (name, model, etc.)
3. In **Agent Settings**, expand the **Tracing** section
4. Toggle on **Require x-litellm-trace-id on calls BY this agent**
5. Set **Max Iterations** to `25`
6. Set **Max Budget Per Session** to `5.00`
7. Proceed to create a new key for the agent
8. Click **Create Agent**
</TabItem>
<TabItem value="api" label="Via API">
```bash
# 1. Create the agent with trace-id enforcement
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent with budget controls",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"litellm_params": {
"require_trace_id_on_calls_by_agent": true
}
}'
# 2. Create a key for the agent
curl -X POST 'http://localhost:4000/key/generate' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_id": "<agent_id_from_step_1>",
"key_alias": "my-research-agent-key"
}'
```
</TabItem>
</Tabs>
### Making Calls with Session Tracking
```bash
curl -X POST 'http://localhost:4000/chat/completions' \
-H 'Authorization: Bearer sk-agent-key-xxx' \
-H 'x-litellm-trace-id: session-abc-123' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
After 25 calls or $5 spent within this session, subsequent requests will receive:
```json
{
"error": {
"message": "Session budget exceeded for session session-abc-123. Current spend: $5.0032, max_budget_per_session: $5.00.",
"type": "budget_exceeded",
"code": 429
}
}
```
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `LITELLM_MAX_ITERATIONS_TTL` | `3600` (1 hour) | TTL in seconds for session iteration counters |
| `LITELLM_MAX_BUDGET_PER_SESSION_TTL` | `3600` (1 hour) | TTL in seconds for session budget counters |

120
docs/my-website/docs/anthropic_unified/messages_to_responses_mapping.md Normal file
View file

@ -0,0 +1,120 @@
# v1/messages → /responses Parameter Mapping
When you send a request to `/v1/messages` targeting an OpenAI or Azure model, LiteLLM internally routes it through the OpenAI Responses API. This page documents exactly how every parameter gets translated in both directions.
The transformation lives in `litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py`.
## Request: Anthropic → Responses API
### Top-level parameters
| Anthropic (`/v1/messages`) | Responses API | Notes |
|---|---|---|
| `model` | `model` | Passed through as-is |
| `messages` | `input` | Structurally transformed — see the messages section below |
| `system` (string) | `instructions` | Passed as a plain string |
| `system` (list of content blocks) | `instructions` | Text blocks are joined with `\n`; non-text blocks are ignored |
| `max_tokens` | `max_output_tokens` | Renamed |
| `temperature` | `temperature` | Passed through as-is |
| `top_p` | `top_p` | Passed through as-is |
| `tools` | `tools` | Format-translated — see the tools section below |
| `tool_choice` | `tool_choice` | Type-remapped — see the tool_choice section below |
| `thinking` | `reasoning` | Budget tokens mapped to effort level — see the thinking section below |
| `output_format` or `output_config.format` | `text` | Wrapped as `{"format": {"type": "json_schema", "name": "structured_output", "schema": ..., "strict": true}}` |
| `context_management` | `context_management` | Converted from Anthropic dict to OpenAI array format — see the context_management section below |
| `metadata.user_id` | `user` | Extracted from the metadata object and truncated to 64 characters |
| `stop_sequences` | ❌ Not mapped | Dropped silently |
| `top_k` | ❌ Not mapped | Dropped silently |
| `speed` | ❌ Not mapped | Only used to set Anthropic beta headers on the native path |
### How messages get converted
Each Anthropic message is expanded into one or more Responses API input items. The key difference is that `tool_result` and `tool_use` blocks become **top-level items** in the input array rather than being nested inside a message.
| Anthropic message | Responses API input item |
|---|---|
| `user` role, string content | `{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "..."}]}` |
| `user` role, `{"type": "text"}` block | `{"type": "input_text", "text": "..."}` inside a user message |
| `user` role, `{"type": "image", "source": {"type": "base64"}}` | `{"type": "input_image", "image_url": "data:<media_type>;base64,<data>"}` inside a user message |
| `user` role, `{"type": "image", "source": {"type": "url"}}` | `{"type": "input_image", "image_url": "<url>"}` inside a user message |
| `user` role, `{"type": "tool_result"}` block | Top-level `{"type": "function_call_output", "call_id": "...", "output": "..."}` — pulled out of the message entirely |
| `assistant` role, string content | `{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "..."}]}` |
| `assistant` role, `{"type": "text"}` block | `{"type": "output_text", "text": "..."}` inside an assistant message |
| `assistant` role, `{"type": "tool_use"}` block | Top-level `{"type": "function_call", "call_id": "<id>", "name": "...", "arguments": "<JSON string>"}` — pulled out of the message entirely |
| `assistant` role, `{"type": "thinking"}` block | `{"type": "output_text", "text": "<thinking text>"}` inside an assistant message |
### tools
| Anthropic tool | Responses API tool |
|---|---|
| Any tool where `type` starts with `"web_search"` or `name == "web_search"` | `{"type": "web_search_preview"}` |
| All other tools | `{"type": "function", "name": "...", "description": "...", "parameters": <input_schema>}` |
### tool_choice
| Anthropic `tool_choice.type` | Responses API `tool_choice` |
|---|---|
| `"auto"` | `{"type": "auto"}` |
| `"any"` | `{"type": "required"}` |
| `"tool"` | `{"type": "function", "name": "<tool name>"}` |
### thinking → reasoning
The `budget_tokens` value is mapped to a string effort level. `summary` is always set to `"detailed"`.
| `thinking.budget_tokens` | `reasoning.effort` |
|---|---|
| >= 10000 | `"high"` |
| >= 5000 | `"medium"` |
| >= 2000 | `"low"` |
| < 2000 | `"minimal"` |
If `thinking.type` is anything other than `"enabled"`, the `reasoning` field is not sent at all.
### context_management
Anthropic uses a nested dict with an `edits` array. OpenAI uses a flat array of compaction objects.
```
Anthropic input:
{
"edits": [
{
"type": "compact_20260112",
"trigger": {"type": "input_tokens", "value": 150000}
}
]
}
Responses API output:
[
{"type": "compaction", "compact_threshold": 150000}
]
```
## Response: Responses API → Anthropic
When the Responses API reply comes back, LiteLLM converts it into an Anthropic `AnthropicMessagesResponse`.
| Responses API field | Anthropic response field | Notes |
|---|---|---|
| `response.id` | `id` | |
| `response.model` | `model` | Falls back to `"unknown-model"` if missing |
| `ResponseReasoningItem` — `summary[*].text` | `content` block `{"type": "thinking", "thinking": "..."}` | Each non-empty summary text becomes a thinking block |
| `ResponseOutputMessage` — `content[*]` where `type == "output_text"` | `content` block `{"type": "text", "text": "..."}` | |
| `ResponseFunctionToolCall` — `{call_id, name, arguments}` | `content` block `{"type": "tool_use", "id": "...", "name": "...", "input": {...}}` | `arguments` is JSON-parsed back into a dict |
| Any `function_call` present in output | `stop_reason: "tool_use"` | |
| `response.status == "incomplete"` | `stop_reason: "max_tokens"` | Takes precedence over the default |
| Everything else | `stop_reason: "end_turn"` | Default |
| `response.usage.input_tokens` | `usage.input_tokens` | |
| `response.usage.output_tokens` | `usage.output_tokens` | |
| *(hardcoded)* | `type: "message"` | Always set |
| *(hardcoded)* | `role: "assistant"` | Always set |
| *(hardcoded)* | `stop_sequence: null` | Always null on this path |

57
docs/my-website/docs/mcp.md
View file

@ -704,6 +704,63 @@ asyncio.run(main())
[Learn more about customer management →](./proxy/customers)
## Calling the Proxy's /v1/responses Endpoint
When calling your LiteLLM Proxy's `/v1/responses` endpoint to use MCP tools, **always use `server_url: "litellm_proxy"`** in the tools array. This tells the proxy to use its configured MCP servers.
:::important Do not use the full proxy URL
Using `server_url: "https://your-proxy.com/mcp"` is incorrect when the request is already going to the proxy. The proxy needs the literal value `litellm_proxy` to route to its configured MCP servers.
:::
```bash title="Correct: Using litellm_proxy" showLineNumbers
curl --location 'https://your-proxy.com/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--data '{
"model": "gpt-4",
"tools": [
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never"
}
],
"input": "Run available tools",
"tool_choice": "required"
}'
```
### Sending Custom Headers to MCP Servers
To pass custom headers (e.g., API keys, auth tokens) to specific MCP servers, use either:
**Option 1: Request headers** – Add `x-mcp-{server_alias}-{header_name}` to your request headers. The proxy forwards these to the matching MCP server.
```bash
# Send Authorization header to the "weather2" MCP server
--header 'x-mcp-weather2-authorization: Bearer your-token'
# Send custom header to the "github" MCP server
--header 'x-mcp-github-x-api-key: your-api-key'
```
**Option 2: Headers in tool config** – Include a `headers` object in the tool definition. These are merged with request headers.
```json
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never",
"headers": {
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",
"x-mcp-servers": "Zapier_MCP,dev-group",
"x-mcp-weather2-authorization": "Bearer your-weather-api-token"
}
}
```
## Using your MCP with client side credentials
Use this if you want to pass a client side authentication token to LiteLLM to then pass to your MCP to auth to your MCP.

8
docs/my-website/docs/mcp_control.md
View file

@ -323,7 +323,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
{
"type": "mcp",
"server_label": "litellm",
"server_url": "<your-litellm-proxy-base-url>/dev_group/mcp",
"server_url": "litellm_proxy",
"require_approval": "never",
"headers": {
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
@ -335,7 +335,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
}'
```
This example uses URL namespacing to access all servers in the "dev_group" access group.
This example uses the `x-mcp-servers` header to access all servers in the "dev_group" access group. Use `server_url: "litellm_proxy"` when calling the proxy's `/v1/responses` endpoint—do not use the full proxy URL.
</TabItem>
@ -423,7 +423,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
{
"type": "mcp",
"server_label": "litellm",
"server_url": "<your-litellm-proxy-base-url>/mcp/",
"server_url": "litellm_proxy",
"require_approval": "never",
"headers": {
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",
@ -436,7 +436,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
}'
```
This configuration restricts the request to only use tools from the specified MCP servers.
This configuration restricts the request to only use tools from the specified MCP servers. Use `server_url: "litellm_proxy"` when calling the proxy's `/v1/responses` endpoint.
</TabItem>

72
docs/my-website/docs/providers/azure_ai/azure_model_router.md
View file

@ -2,6 +2,32 @@
Azure Model Router is a feature in Azure AI Foundry that automatically routes your requests to the best available model based on your requirements. This allows you to use a single endpoint that intelligently selects the optimal model for each request.
## Quick Start
**Model pattern**: `azure_ai/model_router/<deployment-name>`
```python
import litellm
response = litellm.completion(
model="azure_ai/model_router/model-router", # Replace with your deployment name
messages=[{"role": "user", "content": "Hello!"}],
api_base="https://your-endpoint.cognitiveservices.azure.com/openai/v1/",
api_key="your-api-key",
)
```
**Proxy config** (`config.yaml`):
```yaml
model_list:
- model_name: model-router
litellm_params:
model: azure_ai/model_router/model-router
api_base: https://your-endpoint.cognitiveservices.azure.com/openai/deployments/model-router/chat/completions?api-version=2025-01-01-preview
api_key: your-api-key
```
## Key Features
- **Automatic Model Selection**: Azure Model Router dynamically selects the best model for your request
@ -229,19 +255,51 @@ Cost is tracked based on the actual model used (e.g., `gpt-4.1-nano`), plus a fl
## Cost Tracking
LiteLLM automatically handles cost tracking for Azure Model Router by:
LiteLLM automatically handles cost tracking for Azure Model Router. Understanding how this works helps you interpret spend and debug billing.
1. **Detecting the actual model**: When Azure Model Router routes your request to a specific model (e.g., `gpt-4.1-nano-2025-04-14`), LiteLLM extracts this from the response
2. **Calculating accurate costs**: Costs are calculated based on:
- The actual model used (e.g., `gpt-4.1-nano` token costs)
- Plus a flat infrastructure cost of **$0.14 per million input tokens** for using the Model Router
3. **Streaming support**: Cost tracking works correctly for both streaming and non-streaming requests
### How LiteLLM Calculates Cost
When you use Azure Model Router, LiteLLM computes **two cost components**:
| Component | Description | When Applied |
|-----------|-------------|--------------|
| **Model Cost** | Token-based cost for the actual model that handled the request (e.g., `gpt-5-nano`, `gpt-4.1-nano`) | Always, when Azure returns the model in the response |
| **Router Flat Cost** | $0.14 per million input tokens (Azure AI Foundry infrastructure fee) | When the **request** was made via a model router endpoint |
### Cost Calculation Flow
1. **Request model detection**: LiteLLM records the model you requested (e.g., `azure_ai/model_router/model-router`). If it contains `model_router` or `model-router`, the request is treated as a router request.
2. **Response model extraction**: Azure returns the actual model used in the response (e.g., `gpt-5-nano-2025-08-07`). LiteLLM uses this for the model cost lookup.
3. **Model cost**: LiteLLM looks up the response model in its pricing table and computes cost from prompt tokens and completion tokens.
4. **Router flat cost**: Because the original request was to a model router, LiteLLM adds the flat cost ($0.14 per M input tokens) on top of the model cost.
5. **Total cost**: `Total = Model Cost + Router Flat Cost`
### Configuration Requirements
For cost tracking to work correctly:
- **Use the full pattern**: `azure_ai/model_router/<deployment-name>` (e.g., `azure_ai/model_router/model-router`)
- **Proxy config**: When using the LiteLLM proxy, set `model` in `litellm_params` to the full pattern so the request model is correctly identified as a router
```yaml
# proxy_server_config.yaml
model_list:
- model_name: model-router
litellm_params:
model: azure_ai/model_router/model-router # Required for router cost detection
api_base: https://your-endpoint.cognitiveservices.azure.com/openai/deployments/model-router/chat/completions?api-version=2025-01-01-preview
api_key: your-api-key
```
### Cost Breakdown
When you use Azure Model Router, the total cost includes:
- **Model Cost**: Based on the actual model that handled your request (e.g., `gpt-4.1-nano`)
- **Model Cost**: Based on the actual model that handled your request (e.g., `gpt-5-nano`, `gpt-4.1-nano`)
- **Router Flat Cost**: $0.14 per million input tokens (Azure AI Foundry infrastructure fee)
### Example Response with Cost

157
docs/my-website/docs/providers/bedrock_mantle.md Normal file
View file

@ -0,0 +1,157 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Amazon Bedrock Mantle
[Amazon Bedrock Mantle](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html) is Amazon Bedrock's distributed inference engine (Project Mantle) that exposes an **OpenAI-compatible API** for Bedrock-hosted models.
Use this provider to call Bedrock Mantle models with accurate **AWS Bedrock pricing** instead of OpenAI pricing.
:::tip
**We support ALL Bedrock Mantle models, just set `model=bedrock_mantle/<model-id>` as a prefix when sending litellm requests**
:::
## API Key
```python
# env variable
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-aws-bedrock-api-key"
# optional: override region (defaults to us-east-1)
os.environ['BEDROCK_MANTLE_REGION'] = "us-east-1" # or use AWS_REGION
```
## Supported Models
| Model | Context Window | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|---------------|----------------------|------------------------|
| `openai.gpt-oss-120b` | 131K | $0.15 | $0.60 |
| `openai.gpt-oss-20b` | 131K | $0.075 | $0.30 |
| `openai.gpt-oss-safeguard-120b` | 131K | $0.15 | $0.60 |
| `openai.gpt-oss-safeguard-20b` | 131K | $0.075 | $0.30 |
## Sample Usage
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import os
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
response = completion(
model="bedrock_mantle/openai.gpt-oss-120b",
messages=[{"role": "user", "content": "hello from litellm"}],
)
print(response)
```
</TabItem>
<TabItem value="streaming" label="Streaming">
```python
from litellm import completion
import os
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
response = completion(
model="bedrock_mantle/openai.gpt-oss-120b",
messages=[{"role": "user", "content": "hello from litellm"}],
stream=True,
)
for chunk in response:
print(chunk)
```
</TabItem>
<TabItem value="async" label="Async">
```python
import asyncio
from litellm import acompletion
import os
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
async def main():
response = await acompletion(
model="bedrock_mantle/openai.gpt-oss-120b",
messages=[{"role": "user", "content": "hello from litellm"}],
)
print(response)
asyncio.run(main())
```
</TabItem>
</Tabs>
## Region Configuration
The API base URL is `https://bedrock-mantle.{region}.api.aws/v1`. Region is resolved in this order:
1. `BEDROCK_MANTLE_REGION` env var
2. `AWS_REGION` env var
3. Default: `us-east-1`
**Supported regions:** `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, `eu-west-2`, `eu-central-1`, `eu-south-1`, `eu-north-1`, `ap-northeast-1`, `ap-south-1`, `ap-southeast-3`, `sa-east-1`
```python
import os
os.environ['BEDROCK_MANTLE_REGION'] = "eu-west-1"
# or pass api_base directly
response = completion(
model="bedrock_mantle/openai.gpt-oss-120b",
messages=[{"role": "user", "content": "hello"}],
api_base="https://bedrock-mantle.eu-west-1.api.aws/v1",
)
```
## Usage with LiteLLM Proxy
### 1. Set Bedrock Mantle models on config.yaml
```yaml
model_list:
- model_name: gpt-oss-120b
litellm_params:
model: bedrock_mantle/openai.gpt-oss-120b
api_key: os.environ/BEDROCK_MANTLE_API_KEY
# optional region override:
api_base: "https://bedrock-mantle.us-east-1.api.aws/v1"
- model_name: gpt-oss-20b
litellm_params:
model: bedrock_mantle/openai.gpt-oss-20b
api_key: os.environ/BEDROCK_MANTLE_API_KEY
```
### 2. Start the proxy
```shell
litellm --config /path/to/config.yaml
```
### 3. Send a request
```python
import openai
client = openai.OpenAI(
api_key="anything",
base_url="http://0.0.0.0:4000",
)
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[{"role": "user", "content": "hello from litellm"}],
)
print(response)
```

36
docs/my-website/docs/providers/chatgpt.md
View file

@ -4,12 +4,12 @@ Use ChatGPT Pro/Max subscription models through LiteLLM with OAuth device flow a
| Property | Details |
|-------|-------|
| Description | ChatGPT subscription access (Codex + GPT-5.2 family) via ChatGPT backend API |
| Description | ChatGPT subscription access (Codex + GPT-5.3/5.4 family) via ChatGPT backend API |
| Provider Route on LiteLLM | `chatgpt/` |
| Supported Endpoints | `/responses`, `/chat/completions` (bridged to Responses for supported models) |
| API Reference | https://chatgpt.com |
ChatGPT subscription access is native to the Responses API. Chat Completions requests are bridged to Responses for supported models (for example `chatgpt/gpt-5.2`).
ChatGPT subscription access is native to the Responses API. Chat Completions requests are bridged to Responses for supported models (for example `chatgpt/gpt-5.4`).
Notes:
- The ChatGPT subscription backend rejects token limit fields (`max_tokens`, `max_output_tokens`, `max_completion_tokens`) and `metadata`. LiteLLM strips these fields for this provider.
@ -31,7 +31,7 @@ ChatGPT subscription access uses an OAuth device code flow:
import litellm
response = litellm.responses(
model="chatgpt/gpt-5.2-codex",
model="chatgpt/gpt-5.3-codex",
input="Write a Python hello world"
)
@ -44,7 +44,7 @@ print(response)
import litellm
response = litellm.completion(
model="chatgpt/gpt-5.2",
model="chatgpt/gpt-5.4",
messages=[{"role": "user", "content": "Write a Python hello world"}]
)
@ -55,16 +55,36 @@ print(response)
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: chatgpt/gpt-5.2
- model_name: chatgpt/gpt-5.4
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.2
- model_name: chatgpt/gpt-5.2-codex
model: chatgpt/gpt-5.4
- model_name: chatgpt/gpt-5.4-pro
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.2-codex
model: chatgpt/gpt-5.4-pro
- model_name: chatgpt/gpt-5.3-codex
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.3-codex
- model_name: chatgpt/gpt-5.3-codex-spark
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.3-codex-spark
- model_name: chatgpt/gpt-5.3-instant
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.3-instant
- model_name: chatgpt/gpt-5.3-chat-latest
model_info:
mode: responses
litellm_params:
model: chatgpt/gpt-5.3-chat-latest
```
```bash showLineNumbers title="Start LiteLLM Proxy"

4
docs/my-website/docs/providers/openai.md
View file

@ -192,8 +192,12 @@ os.environ["OPENAI_BASE_URL"] = "https://your_host/v1" # OPTIONAL
| gpt-5.2-2025-12-11 | `response = completion(model="gpt-5.2-2025-12-11", messages=messages)` |
| gpt-5.2-chat-latest | `response = completion(model="gpt-5.2-chat-latest", messages=messages)` |
| gpt-5.3-chat-latest | `response = completion(model="gpt-5.3-chat-latest", messages=messages)` |
| gpt-5.4 | `response = completion(model="gpt-5.4", messages=messages)` |
| gpt-5.4-2026-03-05 | `response = completion(model="gpt-5.4-2026-03-05", messages=messages)` |
| gpt-5.2-pro | `response = completion(model="gpt-5.2-pro", messages=messages)` |
| gpt-5.2-pro-2025-12-11 | `response = completion(model="gpt-5.2-pro-2025-12-11", messages=messages)` |
| gpt-5.4-pro | `response = completion(model="gpt-5.4-pro", messages=messages)` |
| gpt-5.4-pro-2026-03-05 | `response = completion(model="gpt-5.4-pro-2026-03-05", messages=messages)` |
| gpt-5.1 | `response = completion(model="gpt-5.1", messages=messages)` |
| gpt-5.1-codex | `response = completion(model="gpt-5.1-codex", messages=messages)` |
| gpt-5.1-codex-mini | `response = completion(model="gpt-5.1-codex-mini", messages=messages)` |

90
docs/my-website/docs/providers/vertex.md
View file

@ -1472,6 +1472,82 @@ Your WIF credentials JSON file typically looks like this (for AWS federation):
For more details on setting up Workload Identity Federation, see [Google Cloud WIF documentation](https://cloud.google.com/iam/docs/workload-identity-federation).
#### Explicit AWS Credentials for WIF
By default, AWS-based WIF relies on the EC2 instance metadata service to obtain AWS credentials. This works when LiteLLM runs on an EC2 instance or ECS task with an IAM role attached.
If your environment **does not have access to the EC2 metadata service** (e.g., running on-premises, in a container without host networking, or in a different cloud with security restrictions), you can provide explicit AWS credentials directly in the WIF credential JSON file. LiteLLM will use these to authenticate to AWS before performing the GCP token exchange.
Add the `aws_*` keys at the **top level** of your WIF credential JSON (alongside `type`, `audience`, etc.):
```json
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
"subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"environment_id": "aws1",
"region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone",
"url": "http://169.254.169.254/latest/meta-data/iam/security-credentials",
"regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15"
},
"aws_role_name": "arn:aws:iam::123456789012:role/MyWifRole",
"aws_region_name": "us-east-1"
}
```
**Supported `aws_*` parameters:**
| Parameter | Required | Description |
|---|---|---|
| `aws_region_name` | Yes | AWS region for credential verification (e.g. `us-east-1`) |
| `aws_role_name` | No | IAM role ARN for STS AssumeRole |
| `aws_access_key_id` | No | Static AWS access key ID |
| `aws_secret_access_key` | No | Static AWS secret access key |
| `aws_session_token` | No | Temporary session token |
| `aws_profile_name` | No | AWS CLI profile name |
| `aws_session_name` | No | Session name for AssumeRole |
| `aws_web_identity_token` | No | Web identity token for STS |
| `aws_sts_endpoint` | No | Custom STS endpoint URL |
| `aws_external_id` | No | External ID for cross-account AssumeRole |
`aws_region_name` is always required when using explicit AWS credentials. The other parameters follow the same authentication flows as [Bedrock AWS auth](/docs/providers/bedrock#authentication) -- you can use role assumption, static keys, profiles, or web identity tokens.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
response = completion(
model="vertex_ai/gemini-1.5-pro",
messages=[{"role": "user", "content": "Hello!"}],
vertex_credentials="/path/to/wif-credentials-with-aws.json", # WIF JSON with aws_* keys
vertex_project="your-gcp-project-id",
vertex_location="us-central1"
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: gemini-model
litellm_params:
model: vertex_ai/gemini-1.5-pro
vertex_project: your-gcp-project-id
vertex_location: us-central1
vertex_credentials: /path/to/wif-credentials-with-aws.json # WIF JSON with aws_* keys
```
</TabItem>
</Tabs>
When `aws_*` keys are present in the JSON, LiteLLM automatically uses explicit AWS authentication instead of the EC2 metadata service. When they are absent, the standard metadata-based flow is used unchanged.
### **Environment Variables**
You can set:
@ -1687,6 +1763,20 @@ litellm.vertex_location = "us-central1 # Your Location
| gemini-2.5-flash-lite-preview-09-2025 | `completion('gemini-2.5-flash-lite-preview-09-2025', messages)`, `completion('vertex_ai/gemini-2.5-flash-lite-preview-09-2025', messages)` |
| gemini-3.1-flash-lite-preview | `completion('gemini-3.1-flash-lite-preview', messages)`, `completion('vertex_ai/gemini-3.1-flash-lite-preview', messages)` |
## PayGo / Priority Cost Tracking
LiteLLM automatically tracks spend for Vertex AI Gemini models using the correct pricing tier based on the response's `usageMetadata.trafficType`:
| Vertex AI `trafficType` | LiteLLM `service_tier` | Pricing applied |
|-------------------------|-------------------------|-----------------|
| `ON_DEMAND_PRIORITY` | `priority` | PayGo / priority pricing (`input_cost_per_token_priority`, `output_cost_per_token_priority`) |
| `ON_DEMAND` | standard | Default on-demand pricing |
| `FLEX` / `BATCH` | `flex` | Batch/flex pricing |
When you use [Vertex AI PayGo](https://cloud.google.com/vertex-ai/generative-ai/pricing) (on-demand priority) or batch workloads, LiteLLM reads `trafficType` from the response and applies the matching cost per token from the [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). No configuration is required — spend tracking works out of the box for both standard and PayGo requests.
See [Spend Tracking](../proxy/cost_tracking.md) for general cost tracking setup.
## Private Service Connect (PSC) Endpoints
LiteLLM supports Vertex AI models deployed to Private Service Connect (PSC) endpoints, allowing you to use custom `api_base` URLs for private deployments.

72
docs/my-website/docs/proxy/admin_ui_sso.md
View file

@ -41,12 +41,38 @@ After creating the app, copy your **Client ID** and **Client Secret** from the a
Ensure users are assigned to the app in the **Assignments** tab. If Federation Broker Mode is enabled, you may need to disable it to assign users manually.
#### Step 3: Configure Authorization Server Access Policy
#### Step 3: Set Environment Variables
:::warning Important
This step is required. Without an Access Policy for your app, users will get a `no_matching_policy` error when attempting to log in.
Set the following environment variables. The only difference between the two Okta authorization servers is the endpoint URLs:
**Org Authorization Server** (available on all Okta plans, no additional SKU required):
```bash
GENERIC_CLIENT_ID="<your-client-id>"
GENERIC_CLIENT_SECRET="<your-client-secret>"
GENERIC_AUTHORIZATION_ENDPOINT="https://<your-okta-domain>/oauth2/v1/authorize"
GENERIC_TOKEN_ENDPOINT="https://<your-okta-domain>/oauth2/v1/token"
GENERIC_USERINFO_ENDPOINT="https://<your-okta-domain>/oauth2/v1/userinfo"
PROXY_BASE_URL="https://<your-proxy-base-url>"
```
**Custom Authorization Server** (requires the Okta API Access Management SKU):
```bash
GENERIC_CLIENT_ID="<your-client-id>"
GENERIC_CLIENT_SECRET="<your-client-secret>"
GENERIC_AUTHORIZATION_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/authorize"
GENERIC_TOKEN_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/token"
GENERIC_USERINFO_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/userinfo"
PROXY_BASE_URL="https://<your-proxy-base-url>"
```
:::tip
You can find all OAuth endpoints at `https://<your-okta-domain>/.well-known/openid-configuration`
:::
#### Step 3a: Configure Access Policy (Custom Authorization Server only)
If you are using the Custom Authorization Server, you must configure an Access Policy. Without it, users will get a `no_matching_policy` error. Skip this step if you are using the Org Authorization Server.
1. Go to **Security** → **API**
<Image img={require('../../img/okta_security_api.png')} />
@ -62,21 +88,21 @@ This step is required. Without an Access Policy for your app, users will get a `
See [Okta's Access Policy documentation](https://help.okta.com/en-us/content/topics/security/api-access-management/access-policies.htm) for more details.
#### Step 4: Configure LiteLLM Environment Variables
#### Step 4: Configure Okta Security Settings
**GENERIC_CLIENT_STATE** is recommended for Okta to prevent CSRF attacks:
```bash
GENERIC_CLIENT_ID="<your-client-id>"
GENERIC_CLIENT_SECRET="<your-client-secret>"
GENERIC_AUTHORIZATION_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/authorize"
GENERIC_TOKEN_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/token"
GENERIC_USERINFO_ENDPOINT="https://<your-okta-domain>/oauth2/default/v1/userinfo"
GENERIC_CLIENT_STATE="random-string"
PROXY_BASE_URL="https://<your-proxy-base-url>"
```
:::tip
You can find all OAuth endpoints at `https://<your-okta-domain>/.well-known/openid-configuration`
:::
**PKCE (Proof Key for Code Exchange)** — If your Okta application is configured to require PKCE, enable it by setting:
```bash
GENERIC_CLIENT_USE_PKCE="true"
```
LiteLLM will automatically handle PKCE parameter generation and verification during the OAuth flow.
#### Step 5: Test the SSO Flow
@ -91,7 +117,7 @@ You can find all OAuth endpoints at `https://<your-okta-domain>/.well-known/open
|-------|-------|----------|
| `redirect_uri` error | Redirect URI not configured | Add `<proxy_base_url>/sso/callback` to Sign-in redirect URIs in Okta |
| `access_denied` | User not assigned to app | Assign the user in the Assignments tab |
| `no_matching_policy` | Missing Access Policy | Create an Access Policy in the Authorization Server (see Step 3) |
| `no_matching_policy` | Missing Access Policy (Custom Authorization Server only) | Create an Access Policy in the Authorization Server (see Step 3a) |
</TabItem>
<TabItem value="google" label="Google SSO">
@ -456,23 +482,9 @@ PROXY_BASE_URL=http://litellm.platform.com
PROXY_BASE_URL=litellm.platform.com
```
**2. For Okta specifically, ensure GENERIC_CLIENT_STATE is set**
**2. For Okta specifically, ensure `GENERIC_CLIENT_STATE` is set and PKCE is configured if required**
Okta requires the `GENERIC_CLIENT_STATE` parameter:
```bash
GENERIC_CLIENT_STATE="random-string" # Required for Okta
```
### Okta PKCE
If your Okta application is configured to require PKCE (Proof Key for Code Exchange), enable it by setting:
```bash
GENERIC_CLIENT_USE_PKCE="true"
```
This is required when your Okta app settings enforce PKCE for enhanced security. LiteLLM will automatically handle PKCE parameter generation and verification during the OAuth flow.
See [Okta SSO — Step 4: Configure Okta Security Settings](#step-4-configure-okta-security-settings) for details on `GENERIC_CLIENT_STATE` and PKCE configuration.
### Common Configuration Issues

4
docs/my-website/docs/proxy/config_settings.md
View file

@ -199,6 +199,7 @@ router_settings:
| use_chat_completions_url_for_anthropic_messages | boolean | If true, routes OpenAI `/v1/messages` requests through chat/completions instead of the Responses API. Can also be set via env var `LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true`. |
| disable_hf_tokenizer_download | boolean | If true, it defaults to using the openai tokenizer for all models (including huggingface models). |
| enable_json_schema_validation | boolean | If true, enables json schema validation for all requests. |
| enable_key_alias_format_validation | boolean | If true, validates `key_alias` format on `/key/generate` and `/key/update`. Must be 2-255 chars, start/end with alphanumeric, only allow `a-zA-Z0-9_-/.@`. Default `false`. |
| disable_copilot_system_to_assistant | boolean | **DEPRECATED** - GitHub Copilot API supports system prompts. |
### general_settings - Reference
@ -815,6 +816,7 @@ router_settings:
| LITELLM_TOKEN | Access token for LiteLLM integration
| LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES | When set to "true", routes OpenAI /v1/messages requests through chat/completions instead of the Responses API for Anthropic models. Can also be set via `litellm_settings.use_chat_completions_url_for_anthropic_messages`
| LITELLM_USER_AGENT | Custom user agent string for LiteLLM API requests. Used for partner telemetry attribution
| LITELLM_WORKER_STARTUP_HOOKS | Comma-separated list of `module.path:function_name` callables to run in each worker process during startup. Runs early in the worker lifecycle (before config/DB loading). Useful for re-initializing per-process state like [gflags](https://github.com/google/python-gflags). See [Worker Startup Hooks](/proxy/worker_startup_hooks) for details
| LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging
| LITELM_ENVIRONMENT | Environment for LiteLLM Instance. This is currently only logged to DeepEval to determine the environment for DeepEval integration.
| LITELLM_ASYNCIO_QUEUE_MAXSIZE | Maximum size for asyncio queues (e.g. log queues, spend update queues, and cookbook examples such as realtime audio in `nova_sonic_realtime.py`). Bounds in-memory growth to prevent OOM. Default is 1000.
@ -918,6 +920,7 @@ router_settings:
| PRISMA_HEALTH_WATCHDOG_INTERVAL_SECONDS | Interval in seconds for Prisma health watchdog probes. Default is 30
| PRISMA_HEALTH_WATCHDOG_PROBE_TIMEOUT_SECONDS | Timeout in seconds for each Prisma health probe. Default is 5.0
| PRISMA_RECONNECT_COOLDOWN_SECONDS | Cooldown in seconds between Prisma reconnection attempts. Default is 15
| PRISMA_RECONNECT_ESCALATION_THRESHOLD | Number of consecutive reconnect failures before escalating the reconnection strategy. Default is 3
| PRISMA_WATCHDOG_RECONNECT_TIMEOUT_SECONDS | Timeout in seconds for Prisma watchdog-initiated reconnection. Default is 30.0
| PREDIBASE_API_BASE | Base URL for Predibase API
| PRESIDIO_ANALYZER_API_BASE | Base URL for Presidio Analyzer service
@ -940,6 +943,7 @@ router_settings:
| QDRANT_URL | Connection URL for Qdrant database
| QDRANT_VECTOR_SIZE | Vector size for Qdrant operations. Default is 1536
| REDIS_CONNECTION_POOL_TIMEOUT | Timeout in seconds for Redis connection pool. Default is 5
| REDIS_CLUSTER_NODES | JSON-formatted list of Redis cluster startup nodes for Redis Cluster mode. Example: '[{"host": "node1", "port": 6379}]'
| REDIS_HOST | Hostname for Redis server
| REDIS_PASSWORD | Password for Redis service
| REDIS_PORT | Port number for Redis server

2
docs/my-website/docs/proxy/cost_tracking.md
View file

@ -8,6 +8,8 @@ Track spend for keys, users, and teams across 100+ LLMs.
LiteLLM automatically tracks spend for all known models. See our [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
Provider-specific cost tracking (e.g., [Vertex AI PayGo / priority pricing](../providers/vertex.md#paygo--priority-cost-tracking), [Bedrock service tiers](../providers/bedrock.md#usage---service-tier), [Azure base model mapping](./custom_pricing.md#set-base_model-for-cost-tracking-eg-azure-deployments)) is applied automatically when the response includes tier metadata.
:::tip Keep Pricing Data Updated
[Sync model pricing data from GitHub](./sync_models_github.md) to ensure accurate cost tracking.
:::

9
docs/my-website/docs/proxy/custom_pricing.md
View file

@ -104,9 +104,18 @@ There are other keys you can use to specify costs for different scenarios and mo
- `input_cost_per_video_per_second` - Cost per second of video input
- `input_cost_per_video_per_second_above_128k_tokens` - Video cost for large contexts
- `input_cost_per_character` - Character-based pricing for some providers
- `input_cost_per_token_priority` / `output_cost_per_token_priority` - Priority/PayGo pricing (Vertex AI Gemini, Bedrock)
- `input_cost_per_token_flex` / `output_cost_per_token_flex` - Batch/flex pricing
These keys evolve based on how new models handle multimodality. The latest version can be found at [https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json).
### Service Tier / PayGo Pricing (Vertex AI, Bedrock)
For providers that support multiple pricing tiers (e.g., Vertex AI PayGo, Bedrock service tiers), LiteLLM automatically applies the correct cost based on the response:
- **Vertex AI Gemini**: Uses `usageMetadata.trafficType` (`ON_DEMAND_PRIORITY` → priority, `FLEX`/`BATCH` → flex). See [Vertex AI - PayGo / Priority Cost Tracking](../providers/vertex.md#paygo--priority-cost-tracking).
- **Bedrock**: Uses `serviceTier` from the response. See [Bedrock - Usage - Service Tier](../providers/bedrock.md#usage---service-tier).
## Zero-Cost Models (Bypass Budget Checks)
**Use Case**: You have on-premises or free models that should be accessible even when users exceed their budget limits.

16
docs/my-website/docs/proxy/custom_sso.md
View file

@ -121,15 +121,14 @@ Use this if you want to run your own code **after** a user signs on to the LiteL
Make sure the response type follows the `SSOUserDefinedValues` pydantic object. This is used for logging the user into the Admin UI:
```python
from fastapi import Request
from fastapi_sso.sso.base import OpenID
from litellm.proxy._types import LitellmUserRoles, SSOUserDefinedValues
from litellm.proxy.management_endpoints.internal_user_endpoints import (
new_user,
user_info,
)
from litellm.proxy.management_endpoints.team_endpoints import add_new_member
from litellm.proxy import proxy_server
# These imports are available if you need to create users or manage team membership:
# from litellm.proxy.management_endpoints.internal_user_endpoints import new_user
# from litellm.proxy.management_endpoints.team_endpoints import add_new_member
async def custom_sso_handler(userIDPInfo: OpenID) -> SSOUserDefinedValues:
@ -158,8 +157,9 @@ async def custom_sso_handler(userIDPInfo: OpenID) -> SSOUserDefinedValues:
#################################################
# Run your custom code / logic here
# check if user exists in litellm proxy DB
_user_info = await user_info(user_id=userIDPInfo.id)
print("_user_info from litellm DB ", _user_info) # noqa
if proxy_server.prisma_client is not None:
_user_info = await proxy_server.prisma_client.get_data(user_id=userIDPInfo.id)
print("_user_info from litellm DB ", _user_info) # noqa
#################################################
return SSOUserDefinedValues(

2
docs/my-website/docs/proxy/dynamic_rate_limit.md
View file

@ -3,6 +3,8 @@
Prevent projects from gobbling too much tpm/rpm.
**See Also:** [Request Prioritization](../scheduler.md) - Prioritize LLM API requests in high-traffic by adding them to a priority queue.
Dynamically allocate TPM/RPM quota to api keys, based on active keys in that minute. [**See Code**](https://github.com/BerriAI/litellm/blob/9bffa9a48e610cc6886fc2dce5c1815aeae2ad46/litellm/proxy/hooks/dynamic_rate_limiter.py#L125)
## Quick Start Usage

2
docs/my-website/docs/proxy/forward_client_headers.md
View file

@ -112,6 +112,8 @@ general_settings:
forward_llm_provider_auth_headers: true # Enable BYOK
```
For **Claude Code** with `/login` and your own Anthropic key, see [Claude Code BYOK](../tutorials/claude_code_byok.md). Use `ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345"` to pass your LiteLLM key while your Anthropic key (from `/login`) is forwarded as `x-api-key`.
Client request:
```bash
curl -X POST "http://localhost:4000/v1/messages" \

33
docs/my-website/docs/proxy/guardrails/quick_start.md
View file

@ -497,7 +497,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
Run guardrails based on the user-agent header. This is useful for running pre-call checks on OpenWebUI but only masking in logs for Claude CLI.
`default` can be a single mode string or a list of modes.
Both `default` and tag values can be a single mode string or a list of modes.
<Tabs>
<TabItem value="single" label="Single Default Mode">
@ -545,6 +545,29 @@ guardrails:
default_on: true
```
</TabItem>
<TabItem value="tag-list" label="Multiple Tag Modes">
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "guardrails_ai-guard"
litellm_params:
guardrail: guardrails_ai
guard_name: "pii_detect"
mode:
tags:
"User-Agent: claude-cli": ["pre_call", "post_call"] # Run both pre and post call for claude-cli
default: "logging_only" # Default to logging only when no tags match
api_base: os.environ/GUARDRAILS_AI_API_BASE
default_on: true
```
</TabItem>
</Tabs>
@ -669,7 +692,7 @@ guardrails:
Mode Specification
`default` accepts either a single string or a list of strings.
Both `default` and tag values accept either a single string or a list of strings.
```python
from litellm.types.guardrails import Mode
@ -685,6 +708,12 @@ mode = Mode(
tags={"User-Agent: claude-cli": "logging_only"},
default=["pre_call", "post_call"]
)
# Multiple modes on a tag value
mode = Mode(
tags={"User-Agent: claude-cli": ["pre_call", "post_call"]},
default="logging_only"
)
```
### `guardrails` Request Parameter

19
docs/my-website/docs/proxy/prometheus.md
View file

@ -561,9 +561,26 @@ Use these metrics to monitor the health of the DB Transaction Queue. Eg. Monitor
| `litellm_in_memory_spend_update_queue_size` | In-memory aggregate spend values for keys, users, teams, team members, etc.| In-Memory |
| `litellm_redis_spend_update_queue_size` | Redis aggregate spend values for keys, users, teams, etc. | Redis |
#### DB Connection Pool and Engine Health Metrics
Monitor PostgreSQL connection pool utilization and Prisma query engine health. These metrics are collected every 30 seconds by default.
## 🔥 LiteLLM Maintained Grafana Dashboards
| Metric Name | Type | Labels | Description |
|------------------------------------------|---------|---------|-----------------------------------------------------------|
| `litellm_db_pool_connections` | Gauge | `state` | Number of DB connections by state (active, idle, etc.) |
| `litellm_db_pool_lock_waiting_connections` | Gauge | | Number of connections blocked on row/table locks |
| `litellm_db_engine_up` | Gauge | | Whether the Prisma query engine is alive (1=up, 0=down) |
| `litellm_db_engine_restarts_total` | Counter | | Total number of Prisma query engine restarts |
The `state` label values come from PostgreSQL's `pg_stat_activity.state` column: `active`, `idle`, `idle in transaction`, `idle in transaction (aborted)`, `fastpath function call`, `disabled`.
**Prerequisites:** Metrics collection requires both:
- `prometheus_system` in `service_callback` (see [Monitor System Health](#monitor-system-health))
- `PRISMA_HEALTH_WATCHDOG_ENABLED` not set to `false` (default: `true`). If disabled, a warning is logged and no DB metrics are collected.
The collection interval can be configured via the `PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS` environment variable (default: 30, minimum: 5).
## 🔥 LiteLLM Maintained Grafana Dashboards
Link to Grafana Dashboards maintained by LiteLLM

130
docs/my-website/docs/proxy/users.md
View file

@ -10,6 +10,8 @@ import TabItem from '@theme/TabItem';
**Team member budgets**: Set individual spending limits within the team's shared budget
**Agent budgets**: Set rate limits (tpm/rpm) and session-level caps (iterations, dollar budget) on agents [**Jump**](#agents)
***If a key belongs to a team, the team budget is applied, not the user's personal budget.***
:::
@ -420,6 +422,109 @@ Expected response on failure
</Tabs>
### Agents
Set budgets and rate limits on agents registered with LiteLLM's [Agent Gateway](../a2a.md). You can control:
- **Per-agent rate limits**: `tpm_limit` and `rpm_limit` on the agent itself
- **Per-session rate limits**: `session_tpm_limit` and `session_rpm_limit` applied per session
- **Per-session iteration cap**: `max_iterations` in agent `litellm_params`
- **Per-session budget cap**: `max_budget_per_session` in agent `litellm_params`
<Tabs>
<TabItem value="agent-rate-limits" label="Agent Rate Limits">
Set `tpm_limit` and `rpm_limit` on the agent to cap total throughput across all sessions.
```bash
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"tpm_limit": 100000,
"rpm_limit": 100
}'
```
</TabItem>
<TabItem value="session-rate-limits" label="Session Rate Limits">
Set `session_tpm_limit` and `session_rpm_limit` to cap throughput per individual session.
```bash
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"session_tpm_limit": 50000,
"session_rpm_limit": 50
}'
```
</TabItem>
<TabItem value="session-budgets" label="Session Budgets">
Set `max_iterations` and `max_budget_per_session` in agent `litellm_params` to cap individual sessions. Requires `require_trace_id_on_calls_by_agent` so LiteLLM can track calls per session.
```bash
curl -X POST 'http://localhost:4000/v1/agents' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"agent_name": "my-research-agent",
"agent_card_params": {
"name": "my-research-agent",
"description": "A research agent",
"url": "http://my-agent:8080",
"version": "1.0.0"
},
"litellm_params": {
"require_trace_id_on_calls_by_agent": true,
"max_iterations": 25,
"max_budget_per_session": 5.00
}
}'
```
When a session exceeds the limit, requests receive a **429 Too Many Requests** response.
See the [Agent Iteration Budgets](../a2a_iteration_budgets) guide for full details.
</TabItem>
</Tabs>
:::info
You can also update rate limits on existing agents using `PATCH /v1/agents/{agent_id}`:
```bash
curl -X PATCH 'http://localhost:4000/v1/agents/<agent_id>' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"tpm_limit": 200000,
"rpm_limit": 200,
"session_tpm_limit": 50000,
"session_rpm_limit": 50
}'
```
:::
### Customers
Use this to budget `user` passed to `/chat/completions`, **without needing to create a key for every user**
@ -685,6 +790,31 @@ These headers indicate:
- 1 request remaining for the GPT-4 model for key=`sk-ulGNRXWtv7M0lFnnsQk0wQ`
- 179 tokens remaining for the GPT-4 model for key=`sk-ulGNRXWtv7M0lFnnsQk0wQ`
</TabItem>
<TabItem value="per-agent" label="Per Agent">
Set rate limits on agents registered with the [Agent Gateway](../a2a.md).
**Agent-level limits** cap total throughput across all sessions:
```shell
curl -X POST 'http://0.0.0.0:4000/v1/agents' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{"agent_name": "my-agent", "agent_card_params": {"name": "my-agent", "description": "My agent", "url": "http://my-agent:8080", "version": "1.0.0"}, "tpm_limit": 100000, "rpm_limit": 100}'
```
**Session-level limits** cap throughput per individual session:
```shell
curl -X POST 'http://0.0.0.0:4000/v1/agents' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{"agent_name": "my-agent", "agent_card_params": {"name": "my-agent", "description": "My agent", "url": "http://my-agent:8080", "version": "1.0.0"}, "session_tpm_limit": 50000, "session_rpm_limit": 50}'
```
You can also set **max_iterations** (call count cap) and **max_budget_per_session** (dollar cap) per session via `litellm_params`. See [Agent Iteration Budgets](../a2a_iteration_budgets) for details.
</TabItem>
<TabItem value="per-end-user" label="For customers">

155
docs/my-website/docs/proxy/worker_startup_hooks.md Normal file
View file

@ -0,0 +1,155 @@
# Worker Startup Hooks
Use `LITELLM_WORKER_STARTUP_HOOKS` to run custom initialization functions in **each worker process** during proxy startup. This is essential when using multi-worker deployments (`--num_workers > 1`) with libraries that require per-process initialization, such as [gflags](https://github.com/google/python-gflags).
## The Problem
When running the LiteLLM proxy with multiple workers:
```bash
litellm --config config.yaml --num_workers 4
```
Each worker is a **separate process** spawned by uvicorn or gunicorn. Any in-process state initialized in the master process (before `run_server()`) is **not available** in worker processes. This includes:
- [python-gflags](https://github.com/google/python-gflags) (`gflags.FLAGS`)
- [absl-py flags](https://abseil.io/docs/python/guides/flags) (`absl.flags.FLAGS`)
- Custom singleton registries or connection pools
- Any module-level state that requires explicit initialization
## Usage
Set the `LITELLM_WORKER_STARTUP_HOOKS` environment variable to a comma-separated list of `module.path:function_name` callables:
```bash
export LITELLM_WORKER_STARTUP_HOOKS="my_module:my_init_function"
```
Each hook is called **early** in the worker startup lifecycle — before config loading, database setup, or any request handling. Both sync and async functions are supported.
## Example: gflags Initialization
### 1. Define your wrapper module
```python title="my_litellm_wrapper.py"
import gflags
import json
import os
import sys
from typing import Optional, List, Any
def init_gflags(
usage: Optional[Any] = None,
raw_args: Optional[List[str]] = None,
known_only: bool = False,
) -> List[str]:
"""Initialize gflags from command-line arguments."""
try:
gflags.FLAGS.set_gnu_getopt(True)
if raw_args is None:
raw_args = sys.argv
argv = gflags.FLAGS(raw_args, known_only=known_only)
except gflags.Error as e:
if usage is None:
print("%s\nUsage: %s ARGS\n%s" % (e, sys.argv[0], gflags.FLAGS))
else:
print(usage % dict(cmd=sys.argv[0], flags=gflags.FLAGS))
sys.exit(1)
return argv
def init_gflags_for_worker():
"""Re-initialize gflags in each worker process.
Reads the original sys.argv from the GFLAGS_ARGV env var
(set by the master process before starting the proxy).
"""
raw_args = json.loads(os.environ.get("GFLAGS_ARGV", "[]")) or sys.argv
init_gflags(raw_args=raw_args, known_only=True)
```
### 2. Start the proxy
```python title="start_proxy.py"
import json
import os
import sys
from my_litellm_wrapper import init_gflags
# Store sys.argv so workers can re-parse the same flags
os.environ["GFLAGS_ARGV"] = json.dumps(sys.argv)
# Tell LiteLLM to call our hook in each worker
os.environ["LITELLM_WORKER_STARTUP_HOOKS"] = "my_litellm_wrapper:init_gflags_for_worker"
# Initialize gflags in the master process
init_gflags()
# Start the proxy (programmatic invocation)
from litellm.proxy.proxy_cli import run_server
run_server(
["--config", "config.yaml", "--num_workers", "4"],
standalone_mode=False,
)
```
Or via shell:
```bash
export GFLAGS_ARGV='["my_app", "--my_flag=value", "--batch_size=32"]'
export LITELLM_WORKER_STARTUP_HOOKS="my_litellm_wrapper:init_gflags_for_worker"
litellm --config config.yaml --num_workers 4
```
## How It Works
```
Master Process Worker Process (×N)
───────────────── ──────────────────────
1. init_gflags() 3. proxy_startup_event():
2. run_server() → Read LITELLM_WORKER_STARTUP_HOOKS
→ sets env vars → Import & call each hook
→ uvicorn.run(workers=N) (gflags.FLAGS re-initialized ✓)
→ spawns workers ──────────────────► → Continue with config/DB setup
→ Ready to serve requests
```
- Hooks run at the **very beginning** of `proxy_startup_event` (the FastAPI lifespan), before config loading, database connections, or any other initialization.
- Environment variables set in the master process are **inherited** by worker processes (standard Unix fork/spawn behavior).
- If a hook **raises an exception**, the worker fails to start — this is intentional, since missing initialization (e.g., uninitialized gflags) would cause downstream errors.
## Multiple Hooks
Separate multiple hooks with commas:
```bash
export LITELLM_WORKER_STARTUP_HOOKS="my_module:init_gflags,my_module:init_metrics,my_module:init_connections"
```
Hooks are executed **in order**, left to right.
## Async Hooks
Async functions are also supported — they are automatically awaited:
```python
async def init_async_connections():
"""Example async hook for initializing async resources."""
await setup_async_connection_pool()
```
```bash
export LITELLM_WORKER_STARTUP_HOOKS="my_module:init_async_connections"
```
## Reference
| Environment Variable | Description |
|---|---|
| `LITELLM_WORKER_STARTUP_HOOKS` | Comma-separated `module.path:function_name` callables to run in each worker on startup |
The hook format follows the standard Python entry point syntax: `module.path:function_name`, where `module.path` is a dotted Python import path and `function_name` is the name of the callable within that module.

5
docs/my-website/docs/search/index.md
View file

@ -2,7 +2,7 @@
| Feature | Supported |
|---------|-----------|
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `brave`, `google_pse`, `dataforseo`, `firecrawl`, `searxng`, `linkup`, `duckduckgo`, `searchapi` |
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `brave`, `google_pse`, `dataforseo`, `firecrawl`, `searxng`, `linkup`, `duckduckgo`, `searchapi`, `serper` |
| Cost Tracking | ✅ |
| Logging | ✅ |
| Load Balancing | ❌ |
@ -210,7 +210,7 @@ See the [official Perplexity Search documentation](https://docs.perplexity.ai/ap
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string or array | Yes | Search query. Can be a single string or array of strings |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"brave"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, `"searxng"`, `"linkup"`, `"duckduckgo"`, or `"searchapi"` |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"brave"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, `"searxng"`, `"linkup"`, `"duckduckgo"`, `"searchapi"`, or `"serper"` |
| `search_tool_name` | string | Yes (Proxy) | Name of the search tool configured in `config.yaml` |
| `max_results` | integer | No | Maximum number of results to return (1-20). Default: 10 |
| `search_domain_filter` | array | No | List of domains to filter results (max 20 domains) |
@ -276,6 +276,7 @@ The response follows Perplexity's search format with the following structure:
| Firecrawl | `FIRECRAWL_API_KEY` | `firecrawl` |
| SearXNG | `SEARXNG_API_BASE` (required) | `searxng` |
| Linkup | `LINKUP_API_KEY` | `linkup` |
| Serper | `SERPER_API_KEY` | `serper` |
| DuckDuckGo | `DUCKDUCKGO_API_BASE` | `duckduckgo` |
| SearchAPI.io | `SEARCHAPI_API_KEY` | `searchapi` |

77
docs/my-website/docs/search/serper.md Normal file
View file

@ -0,0 +1,77 @@
# Serper Search
**Get API Key:** [https://serper.dev](https://serper.dev)
## LiteLLM Python SDK
```python showLineNumbers title="Serper Search"
import os
from litellm import search
os.environ["SERPER_API_KEY"] = "your-api-key"
response = search(
query="latest AI developments",
search_provider="serper",
max_results=5
)
```
## LiteLLM AI Gateway
### 1. Setup config.yaml
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-5
litellm_params:
model: gpt-5
api_key: os.environ/OPENAI_API_KEY
search_tools:
- search_tool_name: serper-search
litellm_params:
search_provider: serper
api_key: os.environ/SERPER_API_KEY
```
### 2. Start the proxy
```bash
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
### 3. Test the search endpoint
```bash showLineNumbers title="Test Request"
curl http://0.0.0.0:4000/v1/search/serper-search \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"query": "latest AI developments",
"max_results": 5
}'
```
## Provider-specific Parameters
```python showLineNumbers title="Serper Search with Provider-specific Parameters"
import os
from litellm import search
os.environ["SERPER_API_KEY"] = "your-api-key"
response = search(
query="latest tech news",
search_provider="serper",
max_results=10,
# Serper-specific parameters
gl="us", # Country/geolocation code
hl="en", # Language code
autocorrect=False, # Disable autocorrect
tbs="qdr:d", # Time filter: past day ('qdr:h' hour, 'qdr:w' week, 'qdr:m' month)
page=2 # Page number
)
```

123
docs/my-website/docs/tutorials/claude_code_byok.md Normal file
View file

@ -0,0 +1,123 @@
# Claude Code with Bring Your Own Key (BYOK)
Use Claude Code with your own Anthropic API key through the LiteLLM proxy. When you use Claude's `/login` with your Anthropic account, your API key is sent as `x-api-key`. With BYOK enabled, LiteLLM forwards your key to Anthropic instead of using proxy-configured keys — so you pay Anthropic directly while still benefiting from LiteLLM's routing, logging, and guardrails.
## How It Works
1. **Claude Code `/login`** — You sign in with your Anthropic account; Claude Code sends your Anthropic API key as `x-api-key`.
2. **LiteLLM authentication** — You pass your LiteLLM proxy key via `ANTHROPIC_CUSTOM_HEADERS` so the proxy can authenticate and track your usage.
3. **Key forwarding** — With `forward_llm_provider_auth_headers: true`, LiteLLM forwards your `x-api-key` to Anthropic, giving it precedence over any proxy-configured keys.
## Prerequisites
- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
- Anthropic API key (from [console.anthropic.com](https://console.anthropic.com))
- LiteLLM proxy with a virtual key for authentication
## Step 1: Configure LiteLLM Proxy
Enable forwarding of LLM provider auth headers so your Anthropic key takes precedence:
```yaml title="config.yaml"
model_list:
- model_name: claude-sonnet-4-5
litellm_params:
model: anthropic/claude-sonnet-4-5
# No api_key needed — client's key will be used
litellm_settings:
forward_llm_provider_auth_headers: true # Required for BYOK
```
:::info Why `forward_llm_provider_auth_headers`?
By default, LiteLLM strips `x-api-key` from client requests for security. Setting this to `true` allows client-provided provider keys (like your Anthropic key from `/login`) to be forwarded to Anthropic, overriding any proxy-configured keys.
:::
## Step 2: Create a LiteLLM Virtual Key
Create a virtual key in the LiteLLM UI or via API.
```bash
# Example: Create key via API
curl -X POST "http://localhost:4000/key/generate" \
-H "Authorization: Bearer sk-your-master-key" \
-H "Content-Type: application/json" \
-d '{"key_alias": "claude-code-byok", "models": ["claude-sonnet-4-5"]}'
```
## Step 3: Configure Claude Code
Set environment variables so Claude Code uses LiteLLM and sends your LiteLLM key for proxy auth:
```bash
# Point Claude Code to your LiteLLM proxy
export ANTHROPIC_BASE_URL="http://localhost:4000"
# Model name from your config
export ANTHROPIC_MODEL="claude-sonnet-4-5"
# LiteLLM proxy auth — this is added to every request
# Use x-litellm-api-key so the proxy authenticates you; your Anthropic key goes via x-api-key from /login
export ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345"
```
Replace `sk-12345` with your actual LiteLLM virtual key.
:::tip Multiple headers
For multiple headers, use newline-separated values:
```bash
export ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345
x-litellm-user-id: my-user-id"
```
:::
## Step 4: Sign In with Claude Code
1. Launch Claude Code:
```bash
claude
```
2. Use **`/login`** and sign in with your Anthropic account (or use your API key directly).
3. Claude Code will send:
- `x-api-key`: Your Anthropic API key (from `/login`)
- `x-litellm-api-key`: Your LiteLLM key (from `ANTHROPIC_CUSTOM_HEADERS`)
4. LiteLLM authenticates you via `x-litellm-api-key`, then forwards `x-api-key` to Anthropic. Your Anthropic key takes precedence over any proxy-configured key.
## Summary
| Header | Source | Purpose |
|--------|--------|---------|
| `x-api-key` | Claude Code `/login` (Anthropic key) | Sent to Anthropic for API calls |
| `x-litellm-api-key` | `ANTHROPIC_CUSTOM_HEADERS` | Proxy authentication, tracking, rate limits |
## Troubleshooting
### Requests fail with "invalid x-api-key"
- Ensure `forward_llm_provider_auth_headers: true` is set in `litellm_settings` (or `general_settings`).
- Restart the LiteLLM proxy after changing the config.
- Verify you completed `/login` in Claude Code so your Anthropic key is being sent.
### Proxy returns 401
- Check that `ANTHROPIC_CUSTOM_HEADERS` includes `x-litellm-api-key: <your-key>`.
- Ensure the LiteLLM key is valid and has access to the model.
### Proxy key is used instead of my Anthropic key
- Confirm `forward_llm_provider_auth_headers: true` is in your config.
- The setting can be in `litellm_settings` or `general_settings` depending on your config structure.
- Enable debug logging: `LITELLM_LOG=DEBUG` to see which key is being forwarded.
## Related
- [Forward Client Headers](./../proxy/forward_client_headers.md) — Full BYOK and header forwarding docs
- [Claude Code Max Subscription](./claude_code_max_subscription.md) — Using Claude Code with OAuth/Max subscription through LiteLLM

BIN
docs/my-website/img/claude_code_byok_screenshot.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 99 KiB

67
docs/my-website/package-lock.json generated
View file

@ -7449,15 +7449,6 @@
"tslib": "^2.6.2"
}
},
"node_modules/@trysound/sax": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/@trysound/sax/-/sax-0.2.0.tgz",
"integrity": "sha512-L7z9BgrNEcYyUYtF+HaEfiS5ebkh9jXqbszz7pC0hRBPaatV0XjSD3+eHrpqFemQfgwiFF0QPIarnIihIDn7OA==",
"license": "ISC",
"engines": {
"node": ">=10.13.0"
}
},
"node_modules/@types/body-parser": {
"version": "1.19.6",
"resolved": "https://registry.npmjs.org/@types/body-parser/-/body-parser-1.19.6.tgz",
@ -10340,13 +10331,13 @@
}
},
"node_modules/css-tree": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/css-tree/-/css-tree-2.3.1.tgz",
"integrity": "sha512-6Fv1DV/TYw//QF5IzQdqsNDjx/wc8TrMBZsqjL9eW01tWb7R7k/mq+/VXfJCl7SoD5emsJop9cOByJZfs8hYIw==",
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz",
"integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==",
"license": "MIT",
"dependencies": {
"mdn-data": "2.0.30",
"source-map-js": "^1.0.1"
"mdn-data": "2.27.1",
"source-map-js": "^1.2.1"
},
"engines": {
"node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0"
@ -11363,10 +11354,13 @@
}
},
"node_modules/dompurify": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.3.0.tgz",
"integrity": "sha512-r+f6MYR1gGN1eJv0TVQbhA7if/U7P87cdPl3HN5rikqaBSBxLiCb/b9O+2eG0cxz0ghyU+mU1QkbsOwERMYlWQ==",
"version": "3.3.2",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.3.2.tgz",
"integrity": "sha512-6obghkliLdmKa56xdbLOpUZ43pAR6xFy1uOrxBaIDjT+yaRuuybLjGS9eVBoSR/UPU5fq3OXClEHLJNGvbxKpQ==",
"license": "(MPL-2.0 OR Apache-2.0)",
"engines": {
"node": ">=20"
},
"optionalDependencies": {
"@types/trusted-types": "^2.0.7"
}
@ -14704,9 +14698,9 @@
}
},
"node_modules/mdn-data": {
"version": "2.0.30",
"resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.0.30.tgz",
"integrity": "sha512-GaqWWShW4kv/G9IEucWScBx9G1/vsFZZJUO+tD26M8J8z3Kw5RDQjaoZe03YAClgeS/SWPOcb4nkFBTEi5DUEA==",
"version": "2.27.1",
"resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz",
"integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==",
"license": "CC0-1.0"
},
"node_modules/media-typer": {
@ -20409,6 +20403,13 @@
"url": "https://opencollective.com/webpack"
}
},
"node_modules/search-insights": {
"version": "2.17.3",
"resolved": "https://registry.npmjs.org/search-insights/-/search-insights-2.17.3.tgz",
"integrity": "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ==",
"license": "MIT",
"peer": true
},
"node_modules/section-matter": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/section-matter/-/section-matter-1.0.0.tgz",
@ -21381,24 +21382,24 @@
"license": "MIT"
},
"node_modules/svgo": {
"version": "3.3.2",
"resolved": "https://registry.npmjs.org/svgo/-/svgo-3.3.2.tgz",
"integrity": "sha512-OoohrmuUlBs8B8o6MB2Aevn+pRIH9zDALSR+6hhqVfa6fRwG/Qw9VUMSMW9VNg2CFc/MTIfabtdOVl9ODIJjpw==",
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/svgo/-/svgo-4.0.1.tgz",
"integrity": "sha512-XDpWUOPC6FEibaLzjfe0ucaV0YrOjYotGJO1WpF0Zd+n6ZGEQUsSugaoLq9QkEZtAfQIxT42UChcssDVPP3+/w==",
"license": "MIT",
"dependencies": {
"@trysound/sax": "0.2.0",
"commander": "^7.2.0",
"commander": "^11.1.0",
"css-select": "^5.1.0",
"css-tree": "^2.3.1",
"css-tree": "^3.0.1",
"css-what": "^6.1.0",
"csso": "^5.0.5",
"picocolors": "^1.0.0"
"picocolors": "^1.1.1",
"sax": "^1.5.0"
},
"bin": {
"svgo": "bin/svgo"
"svgo": "bin/svgo.js"
},
"engines": {
"node": ">=14.0.0"
"node": ">=16"
},
"funding": {
"type": "opencollective",
@ -21406,12 +21407,12 @@
}
},
"node_modules/svgo/node_modules/commander": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/commander/-/commander-7.2.0.tgz",
"integrity": "sha512-QrWXB+ZQSVPmIWIhtEO9H+gwHaMGYiF5ChvoJ+K9ZGHG/sVsa6yiesAD1GC/x46sET00Xlwo1u49RVVVzvcSkw==",
"version": "11.1.0",
"resolved": "https://registry.npmjs.org/commander/-/commander-11.1.0.tgz",
"integrity": "sha512-yPVavfyCcRhmorC7rWlkHn15b4wDVgVmBA7kV4QVBsF7kv/9TKJAbAXVTxvTnwP8HHKjRCJDClKbciiYS7p0DQ==",
"license": "MIT",
"engines": {
"node": ">= 10"
"node": ">=16"
}
},
"node_modules/tailwind-merge": {

6
docs/my-website/package.json
View file

@ -61,7 +61,7 @@
"mermaid": ">=11.10.0",
"gray-matter": "4.0.3",
"glob": ">=11.1.0",
"tar": ">=7.5.8",
"tar": ">=7.5.10",
"minimatch": ">=10.2.4",
"diff": ">=8.0.3",
"@isaacs/brace-expansion": ">=5.0.1",
@ -93,6 +93,8 @@
"axios": ">=0.30.2",
"webpack": ">=5.94.0",
"serve-static": ">=1.16.0",
"path-to-regexp": ">=0.1.12"
"path-to-regexp": ">=0.1.12",
"dompurify": ">=3.3.2",
"svgo": ">=3.3.3"
}
}

9
docs/my-website/sidebars.js
View file

@ -154,6 +154,7 @@ const sidebars = {
items: [
"tutorials/claude_responses_api",
"tutorials/claude_code_max_subscription",
"tutorials/claude_code_byok",
"tutorials/claude_code_customer_tracking",
"tutorials/claude_code_prompt_cache_routing",
"tutorials/claude_code_websearch",
@ -310,6 +311,7 @@ const sidebars = {
"proxy/master_key_rotations",
"proxy/model_management",
"proxy/prod",
"proxy/worker_startup_hooks",
"proxy/release_cycle",
],
},
@ -538,8 +540,10 @@ const sidebars = {
items: [
"a2a",
"a2a_invoking_agents",
"a2a_agent_headers",
"a2a_cost_tracking",
"a2a_agent_permissions"
"a2a_agent_permissions",
"a2a_iteration_budgets"
],
},
"assistants",
@ -624,6 +628,7 @@ const sidebars = {
items: [
"anthropic_unified/index",
"anthropic_unified/structured_output",
"anthropic_unified/messages_to_responses_mapping",
]
},
"anthropic_count_tokens",
@ -679,6 +684,7 @@ const sidebars = {
"search/firecrawl",
"search/searxng",
"search/linkup",
"search/serper",
]
},
"skills",
@ -795,6 +801,7 @@ const sidebars = {
"providers/bedrock_realtime_with_audio",
"providers/aws_polly",
"providers/bedrock_vector_store",
"providers/bedrock_mantle",
]
},
"providers/litellm_proxy",

4
enterprise/litellm_enterprise/integrations/custom_guardrail.py
View file

@ -50,8 +50,10 @@ class EnterpriseCustomGuardrailHelper:
break
if matched_mode is not None:
# Tag matched: only run if event_type matches the tag's mode value
# Tag matched: only run if event_type matches the tag's mode value(s)
if event_type is not None:
if isinstance(matched_mode, list):
return event_type.value in matched_mode
return event_type.value == matched_mode
return True

22
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -78,8 +78,6 @@ class CheckBatchCost:
"status": {"not_in": ["failed", "expired", "cancelled"]}
}
)
completed_jobs = []
for job in jobs:
# get the model from the job
unified_object_id = job.unified_object_id
@ -237,10 +235,16 @@ class CheckBatchCost:
)
# mark the job as complete
completed_jobs.append(job)
if len(completed_jobs) > 0:
await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={"id": {"in": [job.id for job in completed_jobs]}},
data={"batch_processed": True, "status": "complete"},
)
try:
await self.prisma_client.db.litellm_managedobjecttable.update(
where={"id": job.id},
data={
"batch_processed": True,
"status": "complete",
"file_object": response.model_dump_json(),
},
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to mark job {job.id} complete in DB: {db_err}"
)

2
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-enterprise"
version = "0.1.33"
version = "0.1.34"
description = "Package for LiteLLM Enterprise features"
authors = ["BerriAI"]
readme = "README.md"

4
litellm-js/spend-logs/package.json
View file

@ -12,8 +12,8 @@
},
"overrides": {
"glob": ">=11.1.0",
"tar": ">=7.5.8",
"minimatch": ">=10.2.1",
"tar": ">=7.5.10",
"minimatch": ">=10.2.4",
"diff": ">=8.0.3",
"@isaacs/brace-expansion": ">=5.0.1",
"@babel/traverse": ">=7.23.2",

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.51-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.51.tar.gz vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.52-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.52.tar.gz vendored Normal file
View file

Binary file not shown.

3
litellm-proxy-extras/litellm_proxy_extras/migrations/20260304175016_add_spend_to_agent_table/migration.sql Normal file
View file

@ -0,0 +1,3 @@
-- AlterTable
ALTER TABLE "LiteLLM_AgentsTable" ADD COLUMN "spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

5
litellm-proxy-extras/litellm_proxy_extras/migrations/20260305000000_add_agent_headers/migration.sql Normal file
View file

@ -0,0 +1,5 @@
-- Add static_headers and extra_headers to LiteLLM_AgentsTable
ALTER TABLE "LiteLLM_AgentsTable"
ADD COLUMN IF NOT EXISTS "static_headers" JSONB DEFAULT '{}',
ADD COLUMN IF NOT EXISTS "extra_headers" TEXT[] DEFAULT ARRAY[]::TEXT[];

5
litellm-proxy-extras/litellm_proxy_extras/migrations/20260305000000_add_rate_limits_to_agents/migration.sql Normal file
View file

@ -0,0 +1,5 @@
-- AlterTable
ALTER TABLE "LiteLLM_AgentsTable" ADD COLUMN "tpm_limit" INTEGER;
ALTER TABLE "LiteLLM_AgentsTable" ADD COLUMN "rpm_limit" INTEGER;
ALTER TABLE "LiteLLM_AgentsTable" ADD COLUMN "session_tpm_limit" INTEGER;
ALTER TABLE "LiteLLM_AgentsTable" ADD COLUMN "session_rpm_limit" INTEGER;

3
litellm-proxy-extras/litellm_proxy_extras/migrations/20260306175056_add_configs_override_table/migration.sql Normal file
View file

@ -0,0 +1,3 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "spec_path" TEXT;

57
litellm-proxy-extras/litellm_proxy_extras/migrations/20260306233848_schema_sync/migration.sql Normal file
View file

@ -0,0 +1,57 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "byok_api_key_help_url" TEXT,
ADD COLUMN "byok_description" TEXT[] DEFAULT ARRAY[]::TEXT[],
ADD COLUMN "is_byok" BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN "tool_name_to_description" JSONB DEFAULT '{}',
ADD COLUMN "tool_name_to_display_name" JSONB DEFAULT '{}';
-- CreateTable
CREATE TABLE "LiteLLM_MCPUserCredentials" (
"id" TEXT NOT NULL,
"user_id" TEXT NOT NULL,
"server_id" TEXT NOT NULL,
"credential_b64" TEXT NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "LiteLLM_MCPUserCredentials_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "LiteLLM_JWTKeyMapping" (
"id" TEXT NOT NULL,
"jwt_claim_name" TEXT NOT NULL,
"jwt_claim_value" TEXT NOT NULL,
"token" TEXT NOT NULL,
"description" TEXT,
"is_active" BOOLEAN NOT NULL DEFAULT true,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"created_by" TEXT,
"updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_by" TEXT,
CONSTRAINT "LiteLLM_JWTKeyMapping_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "LiteLLM_ConfigOverrides" (
"config_type" TEXT NOT NULL,
"config_value" JSONB NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMP(3) NOT NULL,
CONSTRAINT "LiteLLM_ConfigOverrides_pkey" PRIMARY KEY ("config_type")
);
-- CreateIndex
CREATE UNIQUE INDEX "LiteLLM_MCPUserCredentials_user_id_server_id_key" ON "LiteLLM_MCPUserCredentials"("user_id", "server_id");
-- CreateIndex
CREATE INDEX "LiteLLM_JWTKeyMapping_jwt_claim_name_jwt_claim_value_is_act_idx" ON "LiteLLM_JWTKeyMapping"("jwt_claim_name", "jwt_claim_value", "is_active");
-- CreateIndex
CREATE UNIQUE INDEX "LiteLLM_JWTKeyMapping_jwt_claim_name_jwt_claim_value_key" ON "LiteLLM_JWTKeyMapping"("jwt_claim_name", "jwt_claim_value");
-- AddForeignKey
ALTER TABLE "LiteLLM_JWTKeyMapping" ADD CONSTRAINT "LiteLLM_JWTKeyMapping_token_fkey" FOREIGN KEY ("token") REFERENCES "LiteLLM_VerificationToken"("token") ON DELETE RESTRICT ON UPDATE CASCADE;

85
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -63,9 +63,16 @@ model LiteLLM_AgentsTable {
agent_name String @unique
litellm_params Json?
agent_card_params Json
static_headers Json? @default("{}")
extra_headers String[] @default([])
agent_access_groups String[] @default([])
object_permission_id String?
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
spend Float @default(0.0)
tpm_limit Int?
rpm_limit Int?
session_tpm_limit Int?
session_rpm_limit Int?
created_at DateTime @default(now()) @map("created_at")
created_by String
updated_at DateTime @default(now()) @updatedAt @map("updated_at")
@ -288,6 +295,8 @@ model LiteLLM_MCPServerTable {
mcp_info Json? @default("{}")
mcp_access_groups String[]
allowed_tools String[] @default([])
tool_name_to_display_name Json? @default("{}")
tool_name_to_description Json? @default("{}")
extra_headers String[] @default([])
static_headers Json? @default("{}")
// Health check status
@ -303,6 +312,21 @@ model LiteLLM_MCPServerTable {
registration_url String?
allow_all_keys Boolean @default(false)
available_on_public_internet Boolean @default(true)
is_byok Boolean @default(false)
byok_description String[] @default([])
byok_api_key_help_url String?
}
// Per-user BYOK credentials for MCP servers
model LiteLLM_MCPUserCredentials {
id String @id @default(uuid())
user_id String
server_id String
credential_b64 String
created_at DateTime @default(now()) @map("created_at")
updated_at DateTime @default(now()) @updatedAt @map("updated_at")
@@unique([user_id, server_id])
}
// Generate Tokens for Proxy
@ -353,6 +377,7 @@ model LiteLLM_VerificationToken {
litellm_organization_table LiteLLM_OrganizationTable? @relation(fields: [organization_id], references: [organization_id])
litellm_project_table LiteLLM_ProjectTable? @relation(fields: [project_id], references: [project_id])
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
jwt_key_mappings LiteLLM_JWTKeyMapping[]
// SELECT COUNT(*) FROM (SELECT "public"."LiteLLM_VerificationToken"."token" FROM "public"."LiteLLM_VerificationToken" WHERE ("public"."LiteLLM_VerificationToken"."user_id" = $1 AND ("public"."LiteLLM_VerificationToken"."team_id" IS NULL OR "public"."LiteLLM_VerificationToken"."team_id" <> $2)) OFFSET $3 ) AS "sub"
// SELECT ... FROM "public"."LiteLLM_VerificationToken" WHERE "public"."LiteLLM_VerificationToken"."user_id" = $1 OFFSET $2
@ -365,6 +390,24 @@ model LiteLLM_VerificationToken {
@@index([budget_reset_at, expires])
}
model LiteLLM_JWTKeyMapping {
id String @id @default(uuid())
jwt_claim_name String // e.g. "sub", "email"
jwt_claim_value String // The claim value to match
token String // Hashed virtual key (FK)
description String?
is_active Boolean @default(true)
created_at DateTime @default(now())
created_by String?
updated_at DateTime @default(now()) @updatedAt
updated_by String?
litellm_verification_token LiteLLM_VerificationToken @relation(fields: [token], references: [token])
@@unique([jwt_claim_name, jwt_claim_value])
@@index([jwt_claim_name, jwt_claim_value, is_active])
}
// Deprecated keys during grace period - allows old key to work until revoke_at
model LiteLLM_DeprecatedVerificationToken {
id String @id @default(uuid())
@ -1019,6 +1062,14 @@ model LiteLLM_UISettings {
updated_at DateTime @updatedAt
}
// Generic config overrides table - one row per config_type
model LiteLLM_ConfigOverrides {
config_type String @id
config_value Json
created_at DateTime @default(now())
updated_at DateTime @updatedAt
}
// Skills table for storing LiteLLM-managed skills
model LiteLLM_SkillsTable {
skill_id String @id @default(uuid())
@ -1077,24 +1128,24 @@ model LiteLLM_PolicyAttachmentTable {
updated_by String?
}
// Global tool registry - auto-discovered from LLM responses; admins set input_policy/output_policy here
// Global tool registry - auto-discovered from LLM responses; admins set input/output policies here
model LiteLLM_ToolTable {
tool_id String @id @default(uuid())
tool_name String @unique // e.g. "huggingface_remote-mcp__dynamic_space"
origin String? // MCP server name or "user_defined"
input_policy String @default("untrusted") // "trusted" | "untrusted" | "blocked"
output_policy String @default("untrusted") // "trusted" | "untrusted"
call_count Int @default(0) // cumulative number of times this tool was seen
assignments Json? @default("{}")
key_hash String? // hash of the virtual key that first called this tool
team_id String? // team that first called this tool
key_alias String? // human-readable alias of the virtual key
user_agent String? // user-agent of the first request that discovered this tool
last_used_at DateTime? // timestamp of the most recent call
created_at DateTime @default(now())
created_by String?
updated_at DateTime @default(now()) @updatedAt
updated_by String?
tool_id String @id @default(uuid())
tool_name String @unique // e.g. "huggingface_remote-mcp__dynamic_space"
origin String? // MCP server name or "user_defined"
input_policy String @default("untrusted") // "trusted" | "untrusted" | "blocked"
output_policy String @default("untrusted") // "trusted" | "untrusted"
call_count Int @default(0) // cumulative number of times this tool was seen
assignments Json? @default("{}")
key_hash String? // hash of the virtual key that first called this tool
team_id String? // team that first called this tool
key_alias String? // human-readable alias of the virtual key
user_agent String? // user-agent of the first request that discovered this tool
last_used_at DateTime? // timestamp of the most recent call
created_at DateTime @default(now())
created_by String?
updated_at DateTime @default(now()) @updatedAt
updated_by String?
@@index([input_policy])
@@index([output_policy])

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-proxy-extras"
version = "0.4.50"
version = "0.4.52"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
authors = ["BerriAI"]
readme = "README.md"
@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "0.4.50"
version = "0.4.52"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-proxy-extras==",

9
litellm/__init__.py
View file

@ -305,6 +305,9 @@ return_response_headers: bool = (
False # get response headers from LLM Api providers - example x-remaining-requests,
)
enable_json_schema_validation: bool = False
enable_key_alias_format_validation: bool = (
False # opt-in validation of key_alias format on /key/generate and /key/update
)
####################
logging: bool = True
enable_loadbalancing_on_batch_endpoints: Optional[bool] = None
@ -593,6 +596,7 @@ minimax_models: Set = set()
aws_polly_models: Set = set()
gigachat_models: Set = set()
llamagate_models: Set = set()
bedrock_mantle_models: Set = set()
def is_bedrock_pricing_only_model(key: str) -> bool:
@ -855,6 +859,8 @@ def add_known_models(model_cost_map: Optional[Dict] = None):
gigachat_models.add(key)
elif value.get("litellm_provider") == "llamagate":
llamagate_models.add(key)
elif value.get("litellm_provider") == "bedrock_mantle":
bedrock_mantle_models.add(key)
add_known_models()
@ -962,6 +968,7 @@ model_list = list(
| ovhcloud_models
| lemonade_models
| docker_model_runner_models
| bedrock_mantle_models
| set(clarifai_models)
)
@ -1065,6 +1072,7 @@ models_by_provider: dict = {
"aws_polly": aws_polly_models,
"gigachat": gigachat_models,
"llamagate": llamagate_models,
"bedrock_mantle": bedrock_mantle_models
}
# mapping for those models which have larger equivalents
@ -1426,6 +1434,7 @@ if TYPE_CHECKING:
from .llms.topaz.image_variations.transformation import TopazImageVariationConfig as TopazImageVariationConfig
from litellm.llms.openai.completion.transformation import OpenAITextCompletionConfig as OpenAITextCompletionConfig
from .llms.groq.chat.transformation import GroqChatConfig as GroqChatConfig
from .llms.bedrock_mantle.chat.transformation import BedrockMantleChatConfig as BedrockMantleChatConfig
from .llms.a2a.chat.transformation import A2AConfig as A2AConfig
from .llms.voyage.embedding.transformation import VoyageEmbeddingConfig as VoyageEmbeddingConfig
from .llms.voyage.embedding.transformation_contextual import VoyageContextualEmbeddingConfig as VoyageContextualEmbeddingConfig

2
litellm/_lazy_imports_registry.py
View file

@ -214,6 +214,7 @@ LLM_CONFIG_NAMES = (
"TopazImageVariationConfig",
"OpenAITextCompletionConfig",
"GroqChatConfig",
"BedrockMantleChatConfig",
"A2AConfig",
"GenAIHubOrchestrationConfig",
"VoyageEmbeddingConfig",
@ -858,6 +859,7 @@ _LLM_CONFIGS_IMPORT_MAP = {
"OpenAITextCompletionConfig",
),
"GroqChatConfig": (".llms.groq.chat.transformation", "GroqChatConfig"),
"BedrockMantleChatConfig": (".llms.bedrock_mantle.chat.transformation", "BedrockMantleChatConfig"),
"A2AConfig": (".llms.a2a.chat.transformation", "A2AConfig"),
"GenAIHubOrchestrationConfig": (
".llms.sap.chat.transformation",

134
litellm/a2a_protocol/main.py
View file

@ -162,6 +162,49 @@ async def _send_message_via_completion_bridge(
return LiteLLMSendMessageResponse.from_dict(response_dict)
async def _execute_a2a_send_with_retry(
a2a_client: Any,
request: Any,
agent_card: Any,
card_url: Optional[str],
api_base: Optional[str],
agent_name: Optional[str],
) -> Any:
"""Send an A2A message with retry logic for localhost URL errors."""
a2a_response = None
for _ in range(2): # max 2 attempts: original + 1 retry
try:
a2a_response = await a2a_client.send_message(request)
break # success, exit retry loop
except A2ALocalhostURLError as e:
a2a_client = handle_a2a_localhost_retry(
error=e,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
except Exception as e:
try:
map_a2a_exception(e, card_url, api_base, model=agent_name)
except A2ALocalhostURLError as localhost_err:
a2a_client = handle_a2a_localhost_retry(
error=localhost_err,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
continue
except Exception:
raise
if a2a_response is None:
raise RuntimeError(
"A2A send_message failed: no response received after retry attempts."
)
return a2a_response
@client
async def asend_message(
a2a_client: Optional["A2AClientType"] = None,
@ -169,6 +212,7 @@ async def asend_message(
api_base: Optional[str] = None,
litellm_params: Optional[Dict[str, Any]] = None,
agent_id: Optional[str] = None,
agent_extra_headers: Optional[Dict[str, str]] = None,
**kwargs: Any,
) -> LiteLLMSendMessageResponse:
"""
@ -250,9 +294,12 @@ async def asend_message(
"Either a2a_client or api_base is required for standard A2A flow"
)
trace_id = trace_id or str(uuid.uuid4())
extra_headers = {"X-LiteLLM-Trace-Id": trace_id}
extra_headers: Dict[str, str] = {"X-LiteLLM-Trace-Id": trace_id}
if agent_id:
extra_headers["X-LiteLLM-Agent-Id"] = agent_id
# Overlay agent-level headers (agent headers take precedence over LiteLLM internal ones)
if agent_extra_headers:
extra_headers.update(agent_extra_headers)
a2a_client = await create_a2a_client(
base_url=api_base, extra_headers=extra_headers
)
@ -279,44 +326,17 @@ async def asend_message(
if getattr(message, "context_id", None) is None:
message.context_id = context_id
# Retry loop: if connection fails due to localhost URL in agent card, retry with fixed URL
a2a_response = None
for _ in range(2): # max 2 attempts: original + 1 retry
try:
a2a_response = await a2a_client.send_message(request)
break # success, exit retry loop
except A2ALocalhostURLError as e:
# Localhost URL error - fix and retry
a2a_client = handle_a2a_localhost_retry(
error=e,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
except Exception as e:
# Map exception - will raise A2ALocalhostURLError if applicable
try:
map_a2a_exception(e, card_url, api_base, model=agent_name)
except A2ALocalhostURLError as localhost_err:
# Localhost URL error - fix and retry
a2a_client = handle_a2a_localhost_retry(
error=localhost_err,
agent_card=agent_card,
a2a_client=a2a_client,
is_streaming=False,
)
card_url = agent_card.url if agent_card else None
continue
except Exception:
# Re-raise the mapped exception
raise
a2a_response = await _execute_a2a_send_with_retry(
a2a_client=a2a_client,
request=request,
agent_card=agent_card,
card_url=card_url,
api_base=api_base,
agent_name=agent_name,
)
verbose_logger.info(f"A2A send_message completed, request_id={request.id}")
# a2a_response is guaranteed to be set if we reach here (loop breaks on success or raises)
assert a2a_response is not None
# Wrap in LiteLLM response type for _hidden_params support
response = LiteLLMSendMessageResponse.from_a2a_response(a2a_response)
@ -418,7 +438,7 @@ def _build_streaming_logging_obj(
return logging_obj
async def asend_message_streaming(
async def asend_message_streaming( # noqa: PLR0915
a2a_client: Optional["A2AClientType"] = None,
request: Optional["SendStreamingMessageRequest"] = None,
api_base: Optional[str] = None,
@ -426,6 +446,7 @@ async def asend_message_streaming(
agent_id: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
proxy_server_request: Optional[Dict[str, Any]] = None,
agent_extra_headers: Optional[Dict[str, str]] = None,
) -> AsyncIterator[Any]:
"""
Async: Send a streaming message to an A2A agent.
@ -507,7 +528,17 @@ async def asend_message_streaming(
raise ValueError(
"Either a2a_client or api_base is required for standard A2A flow"
)
a2a_client = await create_a2a_client(base_url=api_base)
# Mirror the non-streaming path: always include trace and agent-id headers
streaming_extra_headers: Dict[str, str] = {
"X-LiteLLM-Trace-Id": str(request.id),
}
if agent_id:
streaming_extra_headers["X-LiteLLM-Agent-Id"] = agent_id
if agent_extra_headers:
streaming_extra_headers.update(agent_extra_headers)
a2a_client = await create_a2a_client(
base_url=api_base, extra_headers=streaming_extra_headers
)
# Type assertion: a2a_client is guaranteed to be non-None here
assert a2a_client is not None
@ -621,17 +652,30 @@ async def create_a2a_client(
verbose_logger.info(f"Creating A2A client for {base_url}")
# Use LiteLLM's cached httpx client
http_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.A2A,
params={"timeout": timeout},
# Use get_async_httpx_client with per-agent params so that different agents
# (with different extra_headers) get separate cached clients. The params
# dict is hashed into the cache key, keeping agent auth isolated while
# still reusing connections within the same agent.
#
# Only pass params that AsyncHTTPHandler.__init__ accepts (e.g. timeout).
# Use "disable_aiohttp_transport" key for cache-key-only data (it's
# filtered out before reaching the constructor).
_client_params: dict = {"timeout": timeout}
if extra_headers:
# Encode headers into a cache-key-only param so each unique header
# set produces a distinct cache key.
_client_params["disable_aiohttp_transport"] = str(
sorted(extra_headers.items())
)
_async_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.A2AProvider,
params=_client_params,
)
httpx_client = http_handler.client
httpx_client = _async_handler.client
if extra_headers:
httpx_client.headers.update(extra_headers)
verbose_proxy_logger.debug(
f"A2A client created with extra_headers={extra_headers}"
f"A2A client created with extra_headers={list(extra_headers.keys())}"
)
# Resolve agent card

7
litellm/batches/batch_utils.py
View file

@ -198,9 +198,8 @@ async def _get_batch_output_file_content_as_dictionary(
Required for Azure and other providers that need authentication
"""
from litellm.files.main import afile_content
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
)
from litellm.proxy.openai_files_endpoints.common_utils import \
_is_base64_encoded_unified_file_id
if custom_llm_provider == "vertex_ai":
raise ValueError("Vertex AI does not support file content retrieval")
@ -227,7 +226,7 @@ async def _get_batch_output_file_content_as_dictionary(
credentials = _extract_file_access_credentials(litellm_params)
file_content_kwargs.update(credentials)
_file_content = await afile_content(**file_content_kwargs)
_file_content = await afile_content(**file_content_kwargs) # type: ignore[reportArgumentType]
return _get_file_content_as_dictionary(_file_content.content)

3
litellm/batches/main.py
View file

@ -33,6 +33,7 @@ from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import (
CancelBatchRequest,
CreateBatchRequest,
FileExpiresAfter,
RetrieveBatchRequest,
)
from litellm.types.router import GenericLiteLLMParams
@ -219,7 +220,7 @@ def create_batch( # noqa: PLR0915
extra_body=extra_body,
)
if output_expires_after is not None:
_create_batch_request["output_expires_after"] = output_expires_after
_create_batch_request["output_expires_after"] = cast(FileExpiresAfter, output_expires_after)
if model is not None:
provider_config = ProviderConfigManager.get_provider_batches_config(
model=model,

8
litellm/caching/caching.py
View file

@ -166,6 +166,14 @@ class Cache:
None. Cache is set as a litellm param
"""
if type == LiteLLMCacheType.REDIS:
# Check REDIS_CLUSTER_NODES env var if no explicit startup nodes
if not redis_startup_nodes:
_env_cluster_nodes = litellm.get_secret("REDIS_CLUSTER_NODES")
if _env_cluster_nodes is not None and isinstance(
_env_cluster_nodes, str
):
redis_startup_nodes = json.loads(_env_cluster_nodes)
if redis_startup_nodes:
# Only pass GCP parameters if they are provided
cluster_kwargs = {

33
litellm/caching/llm_caching_handler.py
View file

@ -3,36 +3,21 @@ Add the event loop to the cache key, to prevent event loop closed errors.
"""
import asyncio
from typing import Set
from .in_memory_cache import InMemoryCache
class LLMClientCache(InMemoryCache):
# Background tasks must be stored to prevent garbage collection, which would
# trigger "coroutine was never awaited" warnings. See:
# https://docs.python.org/3/library/asyncio-task.html#creating-tasks
# Intentionally shared across all instances as a global task registry.
_background_tasks: Set[asyncio.Task] = set()
"""Cache for LLM HTTP clients (OpenAI, Azure, httpx, etc.).
def _remove_key(self, key: str) -> None:
"""Close async clients before evicting them to prevent connection pool leaks."""
value = self.cache_dict.get(key)
super()._remove_key(key)
if value is not None:
close_fn = getattr(value, "aclose", None) or getattr(value, "close", None)
if close_fn and asyncio.iscoroutinefunction(close_fn):
try:
task = asyncio.get_running_loop().create_task(close_fn())
self._background_tasks.add(task)
task.add_done_callback(self._background_tasks.discard)
except RuntimeError:
pass
elif close_fn and callable(close_fn):
try:
close_fn()
except Exception:
pass
IMPORTANT: This cache intentionally does NOT close clients on eviction.
Evicted clients may still be in use by in-flight requests. Closing them
eagerly causes ``RuntimeError: Cannot send a request, as the client has
been closed.`` errors in production after the TTL (1 hour) expires.
Clients that are no longer referenced will be garbage-collected normally.
For explicit shutdown cleanup, use ``close_litellm_async_clients()``.
"""
def update_cache_key_with_event_loop(self, key):
"""

8
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -1028,12 +1028,16 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
if provider_specific_fields:
tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
# Do NOT emit finish_reason here — response.completed handles the terminal
# finish_reason. Emitting "tool_calls" here would prematurely terminate
# the stream before subsequent tool calls arrive (same fix as #17246 for
# the message-type branch).
return ModelResponseStream(
choices=[
StreamingChoices(
index=0,
delta=Delta(tool_calls=[tool_call_chunk]),
finish_reason="tool_calls",
delta=Delta(),
finish_reason=None,
)
]
)

5
litellm/constants.py
View file

@ -1242,6 +1242,11 @@ X_LITELLM_DISABLE_CALLBACKS = "x-litellm-disable-callbacks"
LITELLM_METADATA_FIELD = "litellm_metadata"
OLD_LITELLM_METADATA_FIELD = "metadata"
LITELLM_TRUNCATED_PAYLOAD_FIELD = "litellm_truncated"
LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE = (
"Truncation is a DB storage safeguard. "
"Full, untruncated data is logged to logging callbacks (OTEL, Datadog, etc.). "
"To increase the truncation limit, set `MAX_STRING_LENGTH_PROMPT_IN_DB` in your env."
)
########################### LiteLLM Proxy Specific Constants ###########################
########################################################################################

10
litellm/cost_calculator.py
View file

@ -272,6 +272,8 @@ def cost_per_token( # noqa: PLR0915
### SERVICE TIER ###
service_tier: Optional[str] = None, # for OpenAI service tier pricing
response: Optional[Any] = None,
### REQUEST MODEL ###
request_model: Optional[str] = None, # original request model for router detection
) -> Tuple[float, float]: # type: ignore
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -520,7 +522,7 @@ def cost_per_token( # noqa: PLR0915
return dashscope_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "azure_ai":
return azure_ai_cost_per_token(
model=model, usage=usage_block, response_time_ms=response_time_ms
model=model, usage=usage_block, response_time_ms=response_time_ms, request_model=request_model
)
else:
model_info = _cached_get_model_info_helper(
@ -1457,6 +1459,11 @@ def completion_cost( # noqa: PLR0915
text=completion_string
)
# Get the original request model for router detection
request_model_for_cost = None
if litellm_logging_obj is not None:
request_model_for_cost = litellm_logging_obj.model
(
prompt_tokens_cost_usd_dollar,
completion_tokens_cost_usd_dollar,
@ -1479,6 +1486,7 @@ def completion_cost( # noqa: PLR0915
rerank_billed_units=rerank_billed_units,
service_tier=service_tier,
response=completion_response,
request_model=request_model_for_cost,
)
# Get additional costs from provider (e.g., routing fees, infrastructure costs)

89
litellm/fine_tuning/main.py
View file

@ -126,6 +126,31 @@ async def acreate_fine_tuning_job(
raise e
def _build_fine_tuning_job_data(model, training_file, hyperparameters, suffix, validation_file, integrations, seed):
return FineTuningJobCreate(
model=model,
training_file=training_file,
hyperparameters=hyperparameters,
suffix=suffix,
validation_file=validation_file,
integrations=integrations,
seed=seed,
)
def _resolve_fine_tuning_timeout(
timeout: Any,
custom_llm_provider: str,
) -> Union[float, httpx.Timeout]:
"""Normalise a raw timeout value to a float (seconds) or httpx.Timeout for fine-tuning calls."""
timeout = timeout or 600.0
if isinstance(timeout, httpx.Timeout):
if not supports_httpx_timeout(custom_llm_provider):
return float(timeout.read or 600)
return timeout
return float(timeout)
@client
def create_fine_tuning_job(
model: str,
@ -164,21 +189,10 @@ def create_fine_tuning_job(
_oai_hyperparameters: Hyperparameters = Hyperparameters(
**hyperparameters
) # Typed Hyperparameters for OpenAI Spec
### TIMEOUT LOGIC ###
timeout = optional_params.timeout or kwargs.get("request_timeout", 600) or 600
# set timeout for 10 minutes by default
if (
timeout is not None
and isinstance(timeout, httpx.Timeout)
and supports_httpx_timeout(custom_llm_provider) is False
):
read_timeout = timeout.read or 600
timeout = read_timeout # default 10 min timeout
elif timeout is not None and not isinstance(timeout, httpx.Timeout):
timeout = float(timeout) # type: ignore
elif timeout is None:
timeout = 600.0
timeout = _resolve_fine_tuning_timeout(
optional_params.timeout or kwargs.get("request_timeout", 600),
custom_llm_provider,
)
# OpenAI
if custom_llm_provider == "openai":
@ -204,19 +218,9 @@ def create_fine_tuning_job(
or os.getenv("OPENAI_API_KEY")
)
create_fine_tuning_job_data = FineTuningJobCreate(
model=model,
training_file=training_file,
hyperparameters=_oai_hyperparameters,
suffix=suffix,
validation_file=validation_file,
integrations=integrations,
seed=seed,
)
create_fine_tuning_job_data_dict = create_fine_tuning_job_data.model_dump(
exclude_none=True
)
create_fine_tuning_job_data_dict = _build_fine_tuning_job_data(
model, training_file, _oai_hyperparameters, suffix, validation_file, integrations, seed,
).model_dump(exclude_none=True)
response = openai_fine_tuning_apis_instance.create_fine_tuning_job(
api_base=api_base,
@ -258,20 +262,10 @@ def create_fine_tuning_job(
# Prepare Azure-specific parameters for extra_body
extra_body = _prepare_azure_extra_body(extra_body, kwargs, azure_specific_hyperparams)
create_fine_tuning_job_data = FineTuningJobCreate(
model=model,
training_file=training_file,
hyperparameters=_oai_hyperparameters,
suffix=suffix,
validation_file=validation_file,
integrations=integrations,
seed=seed,
)
create_fine_tuning_job_data_dict = _build_fine_tuning_job_data(
model, training_file, _oai_hyperparameters, suffix, validation_file, integrations, seed,
).model_dump(exclude_none=True)
create_fine_tuning_job_data_dict = create_fine_tuning_job_data.model_dump(
exclude_none=True
)
# Add extra_body if it has Azure-specific parameters
if extra_body:
create_fine_tuning_job_data_dict["extra_body"] = extra_body
@ -301,18 +295,11 @@ def create_fine_tuning_job(
vertex_credentials = optional_params.vertex_credentials or get_secret_str(
"VERTEXAI_CREDENTIALS"
)
create_fine_tuning_job_data = FineTuningJobCreate(
model=model,
training_file=training_file,
hyperparameters=_oai_hyperparameters,
suffix=suffix,
validation_file=validation_file,
integrations=integrations,
seed=seed,
)
response = vertex_fine_tuning_apis_instance.create_fine_tuning_job(
_is_async=_is_async,
create_fine_tuning_job_data=create_fine_tuning_job_data,
create_fine_tuning_job_data=_build_fine_tuning_job_data(
model, training_file, _oai_hyperparameters, suffix, validation_file, integrations, seed,
),
vertex_credentials=vertex_credentials,
vertex_project=vertex_ai_project,
vertex_location=vertex_ai_location,

16
litellm/integrations/custom_guardrail.py
View file

@ -231,8 +231,14 @@ class CustomGuardrail(CustomLogger):
event_hook, supported_event_hooks
)
elif isinstance(event_hook, Mode):
tag_values_flat: list = []
for v in event_hook.tags.values():
if isinstance(v, list):
tag_values_flat.extend(v)
else:
tag_values_flat.append(v)
_validate_event_hook_list_is_in_supported_event_hooks(
list(event_hook.tags.values()), supported_event_hooks
tag_values_flat, supported_event_hooks
)
if event_hook.default:
default_list = (
@ -466,8 +472,12 @@ class CustomGuardrail(CustomLogger):
if isinstance(self.event_hook, list):
return event_type.value in self.event_hook
if isinstance(self.event_hook, Mode):
if event_type.value in self.event_hook.tags.values():
return True
for tag_value in self.event_hook.tags.values():
if isinstance(tag_value, list):
if event_type.value in tag_value:
return True
elif event_type.value == tag_value:
return True
if self.event_hook.default:
default_list = (
self.event_hook.default

68
litellm/integrations/opentelemetry.py
View file

@ -735,13 +735,10 @@ class OpenTelemetry(CustomLogger):
self._maybe_log_raw_request(
kwargs, response_obj, start_time, end_time, span
)
# Ensure proxy-request parent span is annotated with the actual operation kind
if (
parent_span is not None
and hasattr(parent_span, "name")
and parent_span.name == LITELLM_PROXY_REQUEST_SPAN_NAME
):
self.set_attributes(parent_span, kwargs, response_obj)
# Do NOT duplicate attributes onto the parent proxy-request span.
# The child litellm_request span already carries all attributes;
# copying them to the parent doubles storage and complicates
# search (Issue #4).
else:
# Do not create primary span (keep hierarchy shallow when parent exists)
from opentelemetry.trace import Status, StatusCode
@ -757,8 +754,12 @@ class OpenTelemetry(CustomLogger):
kwargs, response_obj, start_time, end_time, parent_span
)
# 3. Guardrail span
self._create_guardrail_span(kwargs=kwargs, context=ctx)
# 3. Guardrail span — ensure guardrails are always parented to an
# existing span so they never become orphaned root spans (Issue #5).
guardrail_ctx = self._resolve_guardrail_context(
span=span, parent_span=parent_span, fallback_ctx=ctx
)
self._create_guardrail_span(kwargs=kwargs, context=guardrail_ctx)
# 4. Metrics & cost recording
self._record_metrics(kwargs, response_obj, start_time, end_time)
@ -1145,6 +1146,27 @@ class OpenTelemetry(CustomLogger):
)
otel_logger.emit(log_record)
@staticmethod
def _resolve_guardrail_context(
span: Optional[Any],
parent_span: Optional[Any],
fallback_ctx: Optional[Any],
) -> Optional[Any]:
"""
Return a valid OTEL context for guardrail child spans so they are
never orphaned (Issue #5). Priority:
1. The litellm_request span that was just created
2. The parent proxy-request span
3. The original fallback context (may be None — last resort)
"""
from opentelemetry import trace as _trace
if span is not None:
return _trace.set_span_in_context(span)
if parent_span is not None:
return _trace.set_span_in_context(parent_span)
return fallback_ctx
def _create_guardrail_span(
self, kwargs: Optional[dict], context: Optional[Context]
):
@ -1250,6 +1272,7 @@ class OpenTelemetry(CustomLogger):
"USE_OTEL_LITELLM_REQUEST_SPAN"
)
span = None
if should_create_primary_span:
# Span 1: Request sent to litellm SDK
otel_tracer: Tracer = self.get_tracer_to_use_for_request(kwargs)
@ -1275,8 +1298,11 @@ class OpenTelemetry(CustomLogger):
self.set_attributes(parent_otel_span, kwargs, response_obj)
self._record_exception_on_span(span=parent_otel_span, kwargs=kwargs)
# Create span for guardrail information
self._create_guardrail_span(kwargs=kwargs, context=_parent_context)
# Create span for guardrail information — ensure proper parenting (Issue #5)
guardrail_ctx = self._resolve_guardrail_context(
span=span, parent_span=parent_otel_span, fallback_ctx=_parent_context
)
self._create_guardrail_span(kwargs=kwargs, context=guardrail_ctx)
# Do NOT end parent span - it should be managed by its creator
# External spans (from Langfuse, user code, HTTP headers, global context) must not be closed by LiteLLM
@ -1579,12 +1605,20 @@ class OpenTelemetry(CustomLogger):
value=optional_params.get("user"),
)
# The unique identifier for the completion.
if response_obj and response_obj.get("id"):
# The unique identifier for the LLM call.
# Completions have a provider response ID (e.g. "chatcmpl-xxx"),
# but Embeddings and Image-gen responses do not. Fall back to
# the litellm call ID so every call type can be correlated
# across LiteLLM UI, Phoenix traces, and provider logs (Issue #8).
response_id = (
(response_obj.get("id") if response_obj else None)
or standard_logging_payload.get("id")
)
if response_id:
self.safe_set_attribute(
span=span,
key="gen_ai.response.id",
value=response_obj.get("id"),
value=response_id,
)
# The model used to generate the response.
@ -1808,8 +1842,10 @@ class OpenTelemetry(CustomLogger):
def set_raw_request_attributes(self, span: Span, kwargs, response_obj):
try:
self.set_attributes(span, kwargs, response_obj)
kwargs.get("optional_params", {})
# Only set provider-specific raw payload attributes on this span.
# The parent litellm_request span already carries the standard
# gen_ai.* / metadata.* attributes — duplicating them here doubles
# storage and adds noise (Issue #3).
litellm_params = kwargs.get("litellm_params", {}) or {}
custom_llm_provider = litellm_params.get("custom_llm_provider", "Unknown")

91
litellm/integrations/websearch_interception/handler.py
View file

@ -7,6 +7,7 @@ server-side using litellm router's search tools.
"""
import asyncio
import math
from typing import Any, Dict, List, Optional, Tuple, Union, cast
import litellm
@ -481,6 +482,56 @@ class WebSearchInterceptionLogger(CustomLogger):
response_format=response_format,
)
@staticmethod
def _resolve_max_tokens(
optional_params: Dict,
kwargs: Dict,
) -> int:
"""Extract max_tokens and validate against thinking.budget_tokens.
Anthropic API requires ``max_tokens > thinking.budget_tokens``.
If the constraint is violated, auto-adjust to ``budget_tokens + 1024``.
"""
max_tokens: int = optional_params.get(
"max_tokens",
kwargs.get("max_tokens", 1024),
)
thinking_param = optional_params.get("thinking")
if thinking_param and isinstance(thinking_param, dict):
budget_tokens = thinking_param.get("budget_tokens")
if (
budget_tokens is not None
and isinstance(budget_tokens, (int, float))
and math.isfinite(budget_tokens)
and budget_tokens > 0
):
if max_tokens <= budget_tokens:
adjusted = math.ceil(budget_tokens) + 1024
verbose_logger.debug(
"WebSearchInterception: max_tokens=%s <= thinking.budget_tokens=%s, "
"adjusting to %s to satisfy Anthropic API constraint",
max_tokens, budget_tokens, adjusted,
)
max_tokens = adjusted
return max_tokens
@staticmethod
def _prepare_followup_kwargs(kwargs: Dict) -> Dict:
"""Build kwargs for the follow-up call, excluding internal keys.
``litellm_logging_obj`` MUST be excluded so the follow-up call creates
its own ``Logging`` instance via ``function_setup``. Reusing the
initial call's logging object triggers the dedup flag
(``has_logged_async_success``) which silently prevents the initial
call's spend from being recorded — the root cause of the
SpendLog / AWS billing mismatch.
"""
_internal_keys = {'litellm_logging_obj'}
return {
k: v for k, v in kwargs.items()
if not k.startswith('_websearch_interception') and k not in _internal_keys
}
async def _execute_agentic_loop(
self,
model: str,
@ -504,7 +555,7 @@ class WebSearchInterceptionLogger(CustomLogger):
)
search_tasks.append(self._execute_search(query))
else:
verbose_logger.warning(
verbose_logger.debug(
f"WebSearchInterception: Tool call {tool_call['id']} has no query"
)
# Add empty result for tools without query
@ -531,7 +582,7 @@ class WebSearchInterceptionLogger(CustomLogger):
final_search_results.append(cast(str, result))
else:
# Should never happen, but handle for type safety
verbose_logger.warning(
verbose_logger.debug(
f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
)
final_search_results.append(str(result))
@ -557,13 +608,18 @@ class WebSearchInterceptionLogger(CustomLogger):
f"WebSearchInterception: Last message (tool_result): {user_message}"
)
# Correlation context for structured logging
_call_id = (
getattr(logging_obj, "litellm_call_id", None)
or kwargs.get("litellm_call_id", "unknown")
)
full_model_name = model # safe default before try block
# Use anthropic_messages.acreate for follow-up request
try:
# Extract max_tokens from optional params or kwargs
# max_tokens is a required parameter for anthropic_messages.acreate()
max_tokens = anthropic_messages_optional_request_params.get(
"max_tokens",
kwargs.get("max_tokens", 1024) # Default to 1024 if not found
max_tokens = self._resolve_max_tokens(
anthropic_messages_optional_request_params, kwargs
)
verbose_logger.debug(
@ -576,16 +632,10 @@ class WebSearchInterceptionLogger(CustomLogger):
if k != 'max_tokens'
}
# Remove internal websearch interception flags from kwargs before follow-up request
# These flags are used internally and should not be passed to the LLM provider
kwargs_for_followup = {
k: v for k, v in kwargs.items()
if not k.startswith('_websearch_interception')
}
kwargs_for_followup = self._prepare_followup_kwargs(kwargs)
# Get model from logging_obj.model_call_details["agentic_loop_params"]
# This preserves the full model name with provider prefix (e.g., "bedrock/invoke/...")
full_model_name = model
if logging_obj is not None:
agentic_params = logging_obj.model_call_details.get("agentic_loop_params", {})
full_model_name = agentic_params.get("model", model)
@ -609,7 +659,10 @@ class WebSearchInterceptionLogger(CustomLogger):
return final_response
except Exception as e:
verbose_logger.exception(
f"WebSearchInterception: Follow-up request failed: {str(e)}"
"WebSearchInterception: Follow-up request failed "
"[call_id=%s model=%s messages=%d searches=%d]: %s",
_call_id, full_model_name, len(follow_up_messages),
len(final_search_results), str(e),
)
raise
@ -620,7 +673,7 @@ class WebSearchInterceptionLogger(CustomLogger):
try:
from litellm.proxy.proxy_server import llm_router
except ImportError:
verbose_logger.warning(
verbose_logger.debug(
"WebSearchInterception: Could not import llm_router from proxy_server, "
"falling back to direct litellm.asearch() with perplexity"
)
@ -643,7 +696,7 @@ class WebSearchInterceptionLogger(CustomLogger):
f"with provider '{search_provider}'"
)
else:
verbose_logger.warning(
verbose_logger.debug(
f"WebSearchInterception: Search tool '{self.search_tool_name}' not found in router, "
"falling back to first available or perplexity"
)
@ -717,7 +770,7 @@ class WebSearchInterceptionLogger(CustomLogger):
)
search_tasks.append(self._execute_search(query))
else:
verbose_logger.warning(
verbose_logger.debug(
f"WebSearchInterception: Tool call {tool_call.get('id')} has no query"
)
# Add empty result for tools without query
@ -742,7 +795,7 @@ class WebSearchInterceptionLogger(CustomLogger):
elif isinstance(result, str):
final_search_results.append(cast(str, result))
else:
verbose_logger.warning(
verbose_logger.debug(
f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
)
final_search_results.append(str(result))

7
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -561,6 +561,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.GroqChatConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "bedrock_mantle":
(
api_base,
dynamic_api_key,
) = litellm.BedrockMantleChatConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "nvidia_nim":
# nvidia_nim is openai compatible, we just need to set this to custom_openai and have the api_base be https://api.endpoints.anyscale.com/v1
api_base = (

2
litellm/litellm_core_utils/get_supported_openai_params.py
View file

@ -88,6 +88,8 @@ def get_supported_openai_params( # noqa: PLR0915
return litellm.VolcEngineConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "groq":
return litellm.GroqChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "bedrock_mantle":
return litellm.BedrockMantleChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "hosted_vllm":
return litellm.HostedVLLMChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "vllm":

27
litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
View file

@ -2,7 +2,7 @@ import asyncio
import json
import time
import traceback
from typing import Dict, Iterable, List, Literal, Optional, Tuple, Union
from typing import Dict, Iterable, List, Literal, Optional, Tuple, Union, cast
import litellm
from litellm._logging import verbose_logger
@ -13,6 +13,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.types.llms.databricks import DatabricksTool
from litellm.types.llms.openai import (
ChatCompletionThinkingBlock,
ImageURLListItem,
OpenAIModerationResponse,
)
from litellm.types.utils import (
@ -26,13 +27,13 @@ from litellm.types.utils import (
Function,
HiddenParams,
ImageResponse,
PromptTokensDetailsWrapper,
)
from litellm.types.utils import Logprobs as TextCompletionLogprobs
from litellm.types.utils import (
Message,
ModelResponse,
ModelResponseStream,
PromptTokensDetailsWrapper,
RerankResponse,
StreamingChoices,
TextChoices,
@ -52,6 +53,24 @@ _MODEL_RESPONSE_FIELDS: frozenset = frozenset(ModelResponse.model_fields.keys())
}
def _normalize_images_for_message(
images: Optional[List[dict]],
) -> Optional[List[ImageURLListItem]]:
"""
Ensure each image has an 'index' field, as required by ImageURLListItem.
Some providers (e.g. OpenRouter) return images without index.
"""
if not images:
return cast(Optional[List[ImageURLListItem]], images)
normalized: List[ImageURLListItem] = []
for i, img in enumerate(images):
if isinstance(img, dict) and "index" not in img:
normalized.append(cast(ImageURLListItem, {**img, "index": i}))
else:
normalized.append(cast(ImageURLListItem, img))
return normalized
def _safe_convert_created_field(created_value) -> int:
"""
Safely convert a 'created' field value to an integer.
@ -591,7 +610,9 @@ def convert_to_model_response_object( # noqa: PLR0915
reasoning_content=reasoning_content,
thinking_blocks=thinking_blocks,
annotations=choice["message"].get("annotations", None),
images=choice["message"].get("images", None),
images=_normalize_images_for_message(
choice["message"].get("images", None)
),
)
finish_reason = choice.get("finish_reason", None)
if finish_reason is None:

95
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -20,6 +20,7 @@ from typing import (
cast,
)
from litellm import verbose_logger
from litellm.router_utils.batch_utils import InMemoryFile
from litellm.types.llms.openai import (
AllMessageValues,
@ -1278,16 +1279,76 @@ def extract_images_from_message(message: AllMessageValues) -> List[str]:
return images
def _attempt_json_repair(s: str) -> Optional[Any]:
"""
Attempt to repair truncated JSON produced by LLM tool calls.
Handles the most common truncation patterns where the model generates
valid JSON that is cut short (missing closing brackets/braces).
Returns the parsed value on success, or None if repair fails.
"""
import json
stripped = s.rstrip()
if not stripped:
return None
# Track the stack of unmatched openers to respect nesting order
opener_stack: list = []
in_string = False
escape_next = False
for ch in stripped:
if escape_next:
escape_next = False
continue
if ch == "\\":
if in_string:
escape_next = True
continue
if ch == '"':
in_string = not in_string
continue
if in_string:
continue
if ch == "{":
opener_stack.append("}")
elif ch == "[":
opener_stack.append("]")
elif ch in ("}", "]"):
if opener_stack and opener_stack[-1] == ch:
opener_stack.pop()
if not opener_stack:
return None
# Remove trailing comma before we close brackets
candidate = stripped.rstrip(",")
# Close in reverse order of opening (respects nesting)
candidate += "".join(reversed(opener_stack))
try:
return json.loads(candidate)
except json.JSONDecodeError:
pass
return None
def parse_tool_call_arguments(
arguments: Optional[str],
tool_name: Optional[str] = None,
context: Optional[str] = None,
) -> Dict[str, Any]:
) -> Any:
"""
Parse tool call arguments from a JSON string.
This function handles malformed JSON gracefully by raising a ValueError
with context about what failed and what the problematic input was.
When the JSON is malformed (e.g. truncated by the model), this function
attempts a lightweight repair (closing unmatched brackets/braces) before
raising an error. A warning is logged whenever repair succeeds so that
callers are aware the arguments were not perfectly formed.
Args:
arguments: The JSON string containing tool arguments, or None.
@ -1295,19 +1356,34 @@ def parse_tool_call_arguments(
context: Optional context string (e.g., "Anthropic Messages API").
Returns:
Parsed arguments as a dictionary. Returns empty dict if arguments is None or empty.
Parsed arguments (usually a dict, but may be any JSON-deserializable
type such as list, str, int, float, or None). Returns empty dict if
arguments is None or empty.
Raises:
ValueError: If the arguments string is not valid JSON.
ValueError: If the arguments string is not valid JSON and cannot be repaired.
"""
import json
if not arguments:
if not arguments or not arguments.strip():
return {}
try:
return json.loads(arguments)
except json.JSONDecodeError as e:
except json.JSONDecodeError as original_error:
repaired = _attempt_json_repair(arguments)
if repaired is not None:
verbose_logger.warning(
"Repaired truncated tool call arguments for tool '%s' (%s). "
"Original (%d chars): %.200s%s",
tool_name or "<unknown>",
context or "unknown context",
len(arguments),
arguments,
"..." if len(arguments) > 200 else "",
)
return repaired
error_parts = ["Failed to parse tool call arguments"]
if tool_name:
@ -1316,10 +1392,11 @@ def parse_tool_call_arguments(
error_parts.append(f"({context})")
error_message = (
" ".join(error_parts) + f". Error: {str(e)}. Arguments: {arguments}"
" ".join(error_parts)
+ f". Error: {str(original_error)}. Arguments: {arguments}"
)
raise ValueError(error_message) from e
raise ValueError(error_message) from original_error
def split_concatenated_json_objects(raw: str) -> List[Dict[str, Any]]:

10
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -1035,9 +1035,13 @@ def convert_to_anthropic_tool_invoke_xml(tool_calls: list) -> str:
parsed_args = parse_tool_call_arguments(
tool_arguments, tool_name=tool_name, context="Anthropic XML tool invoke"
)
parameters = "".join(
f"<{param}>{val}</{param}>\n" for param, val in parsed_args.items()
)
if isinstance(parsed_args, dict):
parameters = "".join(
f"<{param}>{val}</{param}>\n"
for param, val in parsed_args.items()
)
else:
parameters = f"<result>{parsed_args}</result>\n"
invokes += (
"<invoke>\n"
f"<tool_name>{tool_name}</tool_name>\n"

6
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -476,13 +476,15 @@ class ChunkProcessor:
"prompt_tokens_details": prompt_tokens_details,
}
def count_reasoning_tokens(self, response: ModelResponse) -> int:
reasoning_tokens = 0
def count_reasoning_tokens(self, response: ModelResponse) -> Optional[int]:
reasoning_tokens: Optional[int] = None
for choice in response.choices:
if (
hasattr(cast(Choices, choice).message, "reasoning_content")
and cast(Choices, choice).message.reasoning_content is not None
):
if reasoning_tokens is None:
reasoning_tokens = 0
reasoning_tokens += token_counter(
text=cast(Choices, choice).message.reasoning_content,
count_response_tokens=True,

9
litellm/litellm_core_utils/streaming_handler.py
View file

@ -1099,7 +1099,14 @@ class CustomStreamWrapper:
and self.custom_llm_provider in litellm._custom_providers
):
if self.received_finish_reason is not None:
if "provider_specific_fields" not in chunk:
_chunk_has_content = isinstance(chunk, dict) and (
bool(chunk.get("text", ""))
or chunk.get("tool_use") is not None
)
if not _chunk_has_content and (
not isinstance(chunk, dict)
or "provider_specific_fields" not in chunk
):
raise StopIteration
anthropic_response_obj: GChunk = cast(GChunk, chunk)
completion_obj["content"] = anthropic_response_obj["text"]

22
litellm/llms/anthropic/chat/transformation.py
View file

@ -194,6 +194,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
"web_search_options",
"speed",
"context_management",
"cache_control",
]
if (
@ -316,6 +317,11 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
else:
result[key] = value
# Anthropic requires additionalProperties=false for object schemas
# See: https://docs.anthropic.com/en/docs/build-with-claude/structured-outputs
if result.get("type") == "object" and "additionalProperties" not in result:
result["additionalProperties"] = False
return result
def get_json_schema_from_pydantic_object(
@ -769,6 +775,19 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if json_schema is None:
return None
# Resolve $ref/$defs before filtering — Anthropic doesn't support
# external schema references (e.g., /$defs/CalendarEvent).
import copy
from litellm.litellm_core_utils.prompt_templates.common_utils import (
unpack_defs,
)
json_schema = copy.deepcopy(json_schema)
defs = json_schema.pop("$defs", json_schema.pop("definitions", {}))
if defs:
unpack_defs(json_schema, defs)
# Filter out unsupported fields for Anthropic's output_format API
filtered_schema = self.filter_anthropic_output_schema(json_schema)
@ -1031,6 +1050,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
elif param == "speed" and isinstance(value, str):
# Pass through Anthropic-specific speed parameter for fast mode
optional_params["speed"] = value
elif param == "cache_control" and isinstance(value, dict):
# Pass through top-level cache_control for automatic prompt caching
optional_params["cache_control"] = value
## handle thinking tokens
self.update_optional_params_with_thinking_tokens(

4
litellm/llms/anthropic/skills/transformation.py
View file

@ -77,8 +77,8 @@ class AnthropicSkillsConfig(BaseSkillsAPIConfig):
api_base = AnthropicModelInfo.get_api_base()
if skill_id:
return f"{api_base}/v1/skills/{skill_id}?beta=true"
return f"{api_base}/v1/{endpoint}?beta=true"
return f"{api_base}/v1/skills/{skill_id}"
return f"{api_base}/v1/{endpoint}"
def transform_create_skill_request(
self,

6
litellm/llms/azure/batches/handler.py
View file

@ -35,7 +35,7 @@ class AzureBatchesAPI(BaseAzureLLM):
create_batch_data: CreateBatchRequest,
azure_client: Union[AsyncAzureOpenAI, AsyncOpenAI],
) -> LiteLLMBatch:
response = await azure_client.batches.create(**create_batch_data)
response = await azure_client.batches.create(**create_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
def create_batch(
@ -73,7 +73,7 @@ class AzureBatchesAPI(BaseAzureLLM):
return self.acreate_batch( # type: ignore
create_batch_data=create_batch_data, azure_client=azure_client
)
response = cast(Union[AzureOpenAI, OpenAI], azure_client).batches.create(**create_batch_data)
response = cast(Union[AzureOpenAI, OpenAI], azure_client).batches.create(**create_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
async def aretrieve_batch(
@ -81,7 +81,7 @@ class AzureBatchesAPI(BaseAzureLLM):
retrieve_batch_data: RetrieveBatchRequest,
client: Union[AsyncAzureOpenAI, AsyncOpenAI],
) -> LiteLLMBatch:
response = await client.batches.retrieve(**retrieve_batch_data)
response = await client.batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
def retrieve_batch(

35
litellm/llms/azure/chat/gpt_5_transformation.py
View file

@ -15,6 +15,21 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
GPT5_SERIES_ROUTE = "gpt5_series/"
@classmethod
def _supports_reasoning_effort_level(cls, model: str, level: str) -> bool:
"""Override to handle gpt5_series/ prefix used for Azure routing.
The parent class calls ``_supports_factory(model, custom_llm_provider=None)``
which fails to resolve ``gpt5_series/gpt-5.1`` to the correct Azure model
entry. Strip the prefix and prepend ``azure/`` so the lookup finds
``azure/gpt-5.1`` in model_prices_and_context_window.json.
"""
if model.startswith(cls.GPT5_SERIES_ROUTE):
model = "azure/" + model[len(cls.GPT5_SERIES_ROUTE) :]
elif not model.startswith("azure/"):
model = "azure/" + model
return super()._supports_reasoning_effort_level(model, level)
@classmethod
def is_model_gpt_5_model(cls, model: str) -> bool:
"""Check if the Azure model string refers to a gpt-5 variant.
@ -28,8 +43,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
def get_supported_openai_params(self, model: str) -> List[str]:
"""Get supported parameters for Azure OpenAI GPT-5 models.
Azure OpenAI GPT-5.2 models support logprobs, unlike OpenAI's GPT-5.
This overrides the parent class to add logprobs support back for gpt-5.2.
Azure OpenAI GPT-5.2/5.4 models support logprobs, unlike OpenAI's GPT-5.
This overrides the parent class to add logprobs support back for gpt-5.2+.
Reference:
- Tested with Azure OpenAI GPT-5.2 (api-version: 2025-01-01-preview)
@ -43,10 +58,10 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
if "tool_choice" not in params:
params.append("tool_choice")
# Only gpt-5.2 has been verified to support logprobs on Azure.
# Only gpt-5.2+ has been verified to support logprobs on Azure.
# The base OpenAI class includes logprobs for gpt-5.1+, but Azure
# hasn't verified support for gpt-5.1, so remove them unless gpt-5.2.
if self.is_model_gpt_5_1_model(model) and not self.is_model_gpt_5_2_model(model):
# hasn't verified support for gpt-5.1, so remove them unless gpt-5.2/5.4+.
if self._supports_reasoning_effort_level(model, "none") and not self.is_model_gpt_5_2_model(model):
params = [p for p in params if p not in ["logprobs", "top_logprobs"]]
elif self.is_model_gpt_5_2_model(model):
azure_supported_params = ["logprobs", "top_logprobs"]
@ -67,11 +82,11 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
or optional_params.get("reasoning_effort")
)
# gpt-5.1 supports reasoning_effort='none', but other gpt-5 models don't
# gpt-5.1/5.2/5.4 support reasoning_effort='none', but other gpt-5 models don't
# See: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning
is_gpt_5_1 = self.is_model_gpt_5_1_model(model)
supports_none = self._supports_reasoning_effort_level(model, "none")
if reasoning_effort_value == "none" and not is_gpt_5_1:
if reasoning_effort_value == "none" and not supports_none:
if litellm.drop_params is True or (
drop_params is not None and drop_params is True
):
@ -101,8 +116,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
drop_params=drop_params,
)
# Only drop reasoning_effort='none' for non-gpt-5.1 models
if result.get("reasoning_effort") == "none" and not is_gpt_5_1:
# Only drop reasoning_effort='none' for models that don't support it
if result.get("reasoning_effort") == "none" and not supports_none:
result.pop("reasoning_effort")
return result

52
litellm/llms/azure_ai/anthropic/messages_transformation.py
View file

@ -1,7 +1,7 @@
"""
Azure Anthropic messages transformation config - extends AnthropicMessagesConfig with Azure authentication
"""
from typing import TYPE_CHECKING, Any, List, Optional, Tuple
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
@ -114,3 +114,53 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
return api_base
def _remove_scope_from_cache_control(
self, anthropic_messages_request: Dict
) -> None:
"""
Remove `scope` field from cache_control for Azure AI Foundry.
Azure AI Foundry's Anthropic endpoint does not support the `scope` field
(e.g., "global" for cross-request caching). Only `type` and `ttl` are supported.
Processes both `system` and `messages` content blocks.
"""
def _sanitize(cache_control: Any) -> None:
if isinstance(cache_control, dict):
cache_control.pop("scope", None)
def _process_content_list(content: list) -> None:
for item in content:
if isinstance(item, dict) and "cache_control" in item:
_sanitize(item["cache_control"])
if "system" in anthropic_messages_request:
system = anthropic_messages_request["system"]
if isinstance(system, list):
_process_content_list(system)
if "messages" in anthropic_messages_request:
for message in anthropic_messages_request["messages"]:
if isinstance(message, dict) and "content" in message:
content = message["content"]
if isinstance(content, list):
_process_content_list(content)
def transform_anthropic_messages_request(
self,
model: str,
messages: List[Dict],
anthropic_messages_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Dict:
anthropic_messages_request = super().transform_anthropic_messages_request(
model=model,
messages=messages,
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
litellm_params=litellm_params,
headers=headers,
)
self._remove_scope_from_cache_control(anthropic_messages_request)
return anthropic_messages_request

30
litellm/llms/azure_ai/cost_calculator.py
View file

@ -61,7 +61,10 @@ def calculate_azure_model_router_flat_cost(model: str, prompt_tokens: int) -> fl
def cost_per_token(
model: str, usage: Usage, response_time_ms: Optional[float] = 0.0
model: str,
usage: Usage,
response_time_ms: Optional[float] = 0.0,
request_model: Optional[str] = None,
) -> Tuple[float, float]:
"""
Calculate the cost per token for Azure AI models.
@ -71,9 +74,10 @@ def cost_per_token(
- Plus the cost of the actual model used (handled by generic_cost_per_token)
Args:
model: str, the model name without provider prefix
model: str, the model name without provider prefix (from response)
usage: LiteLLM Usage block
response_time_ms: Optional response time in milliseconds
request_model: Optional[str], the original request model name (to detect router usage)
Returns:
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
@ -84,7 +88,13 @@ def cost_per_token(
"""
prompt_cost = 0.0
completion_cost = 0.0
# Determine if this was a model router request
# Check both the response model and the request model
is_router_request = _is_azure_model_router(model) or (
request_model is not None and _is_azure_model_router(request_model)
)
# Calculate base cost using generic cost calculator
# This may raise an exception if the model is not in the cost map
try:
@ -103,19 +113,21 @@ def cost_per_token(
verbose_logger.debug(
f"Azure AI Model Router: model '{model}' not in cost map, calculating routing flat cost only. Error: {e}"
)
# Add flat cost for Azure Model Router
# The flat cost is defined in model_prices_and_context_window.json for azure_ai/model_router
if _is_azure_model_router(model):
router_flat_cost = calculate_azure_model_router_flat_cost(model, usage.prompt_tokens)
if is_router_request:
# Use the request model for flat cost calculation if available, otherwise use response model
router_model_for_calc = request_model if request_model else model
router_flat_cost = calculate_azure_model_router_flat_cost(router_model_for_calc, usage.prompt_tokens)
if router_flat_cost > 0:
verbose_logger.debug(
f"Azure AI Model Router flat cost: ${router_flat_cost:.6f} "
f"({usage.prompt_tokens} tokens × ${router_flat_cost / usage.prompt_tokens:.9f}/token)"
)
# Add flat cost to prompt cost
prompt_cost += router_flat_cost
return prompt_cost, completion_cost

187
litellm/llms/bedrock/chat/agentcore/transformation.py
View file

@ -334,24 +334,67 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
"""
Parse direct JSON response (non-streaming).
JSON response structure:
{
"result": {
"role": "assistant",
"content": [{"text": "..."}]
}
}
Supports multiple agent response schemas:
1. {"result": {"role": "assistant", "content": [{"text": "..."}]}} - standard AgentCore
2. {"response": [{"text": "..."}]} - Strands agent format
3. {"result": "plain text"} or {"response": "plain text"} - simple string
4. Fallback: raw JSON as content string
"""
result = response_json.get("result", {})
# Guard: if json.loads() returned a non-dict (e.g. array or primitive),
# skip strategy matching and fall back to raw JSON string
if not isinstance(response_json, dict):
verbose_logger.warning(
"AgentCore: JSON response is not a dict. "
"Returning raw JSON as content."
)
return AgentCoreParsedResponse(
content=json.dumps(response_json),
usage=None,
final_message=None,
)
# Extract content using the same helper as SSE parsing
content = self._extract_content_from_message(result) # type: ignore
# Strategy 1: {"result": {"content": [{"text": "..."}]}} - standard AgentCore format
if "result" in response_json and isinstance(response_json["result"], dict):
result = response_json["result"]
content = self._extract_content_from_message(result) # type: ignore
return AgentCoreParsedResponse(
content=content,
usage=None,
final_message=result, # type: ignore
)
# JSON responses don't include usage data
# Strategy 2: {"response": [{"text": "..."}]} - Strands agent content blocks
if "response" in response_json and isinstance(
response_json["response"], list
):
content = self._extract_content_from_message(
{"content": response_json["response"]} # type: ignore
)
return AgentCoreParsedResponse(
content=content,
usage=None,
final_message=None,
)
# Strategy 3: string values - {"result": "text"} or {"response": "text"}
for key in ("result", "response"):
val = response_json.get(key)
if isinstance(val, str):
return AgentCoreParsedResponse(
content=val,
usage=None,
final_message=None,
)
# Strategy 4: fallback - return raw JSON as content
verbose_logger.warning(
f"AgentCore: Could not extract content from JSON response keys "
f"{list(response_json.keys())}. Returning raw JSON as content."
)
return AgentCoreParsedResponse(
content=content,
content=json.dumps(response_json),
usage=None,
final_message=result, # type: ignore
final_message=None,
)
def _get_parsed_response(
@ -589,7 +632,64 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
additional_args={"complete_input_dict": data},
)
# Wrap the generator in CustomStreamWrapper
# Check if response is JSON (agent used sync return) instead of SSE
content_type = response.headers.get("content-type", "").lower()
if "application/json" in content_type:
verbose_logger.debug(
"AgentCore streaming: received JSON response instead of SSE, "
"converting to single-chunk stream"
)
try:
body = response.read()
response_json = json.loads(body)
except (json.JSONDecodeError, Exception) as e:
raise BedrockError(
status_code=response.status_code,
message=f"AgentCore: Failed to read/parse JSON response body: {e}",
)
parsed = self._parse_json_response(response_json)
def _json_as_sync_stream():
# Content chunk
content_chunk = ModelResponseStream(
id=f"chatcmpl-{uuid.uuid4()}",
created=0,
model=model,
object="chat.completion.chunk",
)
content_chunk.choices = [
StreamingChoices(
finish_reason=None,
index=0,
delta=Delta(content=parsed["content"], role="assistant"),
)
]
yield content_chunk
# Stop sentinel chunk (matches SSE path convention)
stop_chunk = ModelResponseStream(
id=f"chatcmpl-{uuid.uuid4()}",
created=0,
model=model,
object="chat.completion.chunk",
)
stop_chunk.choices = [
StreamingChoices(
finish_reason="stop",
index=0,
delta=Delta(),
)
]
yield stop_chunk
return CustomStreamWrapper(
completion_stream=_json_as_sync_stream(),
model=model,
custom_llm_provider="bedrock",
logging_obj=logging_obj,
)
# SSE stream (text/event-stream or default) - use existing SSE parser
return CustomStreamWrapper(
completion_stream=self._stream_agentcore_response_sync(response, model),
model=model,
@ -746,7 +846,64 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
additional_args={"complete_input_dict": data},
)
# Wrap the async generator in CustomStreamWrapper
# Check if response is JSON (agent used sync return) instead of SSE
content_type = response.headers.get("content-type", "").lower()
if "application/json" in content_type:
verbose_logger.debug(
"AgentCore streaming: received JSON response instead of SSE, "
"converting to single-chunk stream"
)
try:
body = await response.aread()
response_json = json.loads(body)
except (json.JSONDecodeError, Exception) as e:
raise BedrockError(
status_code=response.status_code,
message=f"AgentCore: Failed to read/parse JSON response body: {e}",
)
parsed = self._parse_json_response(response_json)
async def _json_as_async_stream() -> AsyncGenerator[ModelResponseStream, None]:
# Content chunk
content_chunk = ModelResponseStream(
id=f"chatcmpl-{uuid.uuid4()}",
created=0,
model=model,
object="chat.completion.chunk",
)
content_chunk.choices = [
StreamingChoices(
finish_reason=None,
index=0,
delta=Delta(content=parsed["content"], role="assistant"),
)
]
yield content_chunk
# Stop sentinel chunk (matches SSE path convention)
stop_chunk = ModelResponseStream(
id=f"chatcmpl-{uuid.uuid4()}",
created=0,
model=model,
object="chat.completion.chunk",
)
stop_chunk.choices = [
StreamingChoices(
finish_reason="stop",
index=0,
delta=Delta(),
)
]
yield stop_chunk
return CustomStreamWrapper(
completion_stream=_json_as_async_stream(),
model=model,
custom_llm_provider="bedrock",
logging_obj=logging_obj,
)
# SSE stream (text/event-stream or default) - use existing SSE parser
return CustomStreamWrapper(
completion_stream=self._stream_agentcore_response(response, model),
model=model,

14
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -6,7 +6,10 @@ from litellm.llms.anthropic.chat.transformation import AnthropicConfig
from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation import (
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
from litellm.llms.bedrock.common_utils import (
get_anthropic_beta_from_headers,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
@ -105,9 +108,18 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
_anthropic_request.pop("stream", None)
# Bedrock Invoke doesn't support output_format parameter
_anthropic_request.pop("output_format", None)
# Bedrock Invoke doesn't support output_config parameter
# Fixes: https://github.com/BerriAI/litellm/issues/22797
_anthropic_request.pop("output_config", None)
if "anthropic_version" not in _anthropic_request:
_anthropic_request["anthropic_version"] = self.anthropic_version
# Remove `custom` field from tools (Bedrock doesn't support it)
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
# Ref: https://github.com/BerriAI/litellm/issues/22847
remove_custom_field_from_tools(_anthropic_request)
tools = optional_params.get("tools")
tool_search_used = self.is_tool_search_used(tools)
programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools)

21
litellm/llms/bedrock/common_utils.py
View file

@ -49,6 +49,27 @@ def get_cached_model_info():
return _get_model_info
def remove_custom_field_from_tools(request_body: dict) -> None:
"""
Remove ``custom`` field from each tool in the request body.
Claude Code (v2.1.69+) sends ``custom: {defer_loading: true}`` on tool
definitions, which Anthropic's API accepts but Bedrock rejects with
``"Extra inputs are not permitted"``.
Args:
request_body: The request dictionary to modify in-place.
Ref: https://github.com/BerriAI/litellm/issues/22847
"""
tools = request_body.get("tools")
if not tools or not isinstance(tools, list):
return
for tool in tools:
if isinstance(tool, dict):
tool.pop("custom", None)
class AmazonBedrockGlobalConfig:
def __init__(self):
pass

4
litellm/llms/bedrock/image_edit/stability_transformation.py
View file

@ -22,7 +22,6 @@ API Reference: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parame
"""
import base64
import json
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple
import httpx
@ -285,8 +284,6 @@ class BedrockStabilityImageEditConfig(BaseImageEditConfig):
"""
try:
response_data = raw_response.json()
with open("response_data.json", "w") as f:
json.dump(response_data, f)
except Exception as e:
raise self.get_error_class(
error_message=f"Error parsing Bedrock Stability response: {e}",
@ -396,4 +393,3 @@ class BedrockStabilityImageEditConfig(BaseImageEditConfig):
headers["Content-Type"] = "application/json"
return headers

57
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -26,6 +26,7 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
from litellm.llms.bedrock.common_utils import (
get_anthropic_beta_from_headers,
is_claude_4_5_on_bedrock,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
@ -118,10 +119,13 @@ class AmazonAnthropicClaudeMessagesConfig(
self, anthropic_messages_request: Dict, model: Optional[str] = None
) -> None:
"""
Remove `ttl` field from cache_control in messages.
Bedrock doesn't support the ttl field in cache_control.
Remove unsupported fields from cache_control for Bedrock.
Update: Bedock supports `5m` and `1h` for Claude 4.5 models.
Bedrock only supports `type` and `ttl` in cache_control. It does NOT support:
- `scope` (e.g., "global") - always removed
- `ttl` - removed for older models; Claude 4.5+ supports "5m" and "1h"
Processes both `system` and `messages` content blocks.
Args:
anthropic_messages_request: The request dictionary to modify in-place
@ -131,23 +135,36 @@ class AmazonAnthropicClaudeMessagesConfig(
if model:
is_claude_4_5 = self._is_claude_4_5_on_bedrock(model)
def _sanitize_cache_control(cache_control: dict) -> None:
if not isinstance(cache_control, dict):
return
# Bedrock doesn't support scope (e.g., "global" for cross-request caching)
cache_control.pop("scope", None)
# Remove ttl for models that don't support it
if "ttl" in cache_control:
ttl = cache_control["ttl"]
if is_claude_4_5 and ttl in ["5m", "1h"]:
return
cache_control.pop("ttl", None)
def _process_content_list(content: list) -> None:
for item in content:
if isinstance(item, dict) and "cache_control" in item:
_sanitize_cache_control(item["cache_control"])
# Process system (list of content blocks)
if "system" in anthropic_messages_request:
system = anthropic_messages_request["system"]
if isinstance(system, list):
_process_content_list(system)
# Process messages
if "messages" in anthropic_messages_request:
for message in anthropic_messages_request["messages"]:
if isinstance(message, dict) and "content" in message:
content = message["content"]
if isinstance(content, list):
for item in content:
if isinstance(item, dict) and "cache_control" in item:
cache_control = item["cache_control"]
if (
isinstance(cache_control, dict)
and "ttl" in cache_control
):
ttl = cache_control["ttl"]
if is_claude_4_5 and ttl in ["5m", "1h"]:
continue
cache_control.pop("ttl", None)
_process_content_list(content)
def _supports_extended_thinking_on_bedrock(self, model: str) -> bool:
"""
@ -402,6 +419,16 @@ class AmazonAnthropicClaudeMessagesConfig(
anthropic_messages_request=anthropic_messages_request,
)
# 5b. Strip `output_config` — Bedrock Invoke doesn't support it
# Fixes: https://github.com/BerriAI/litellm/issues/22797
anthropic_messages_request.pop("output_config", None)
# 5a. Remove `custom` field from tools (Bedrock doesn't support it)
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
# Ref: https://github.com/BerriAI/litellm/issues/22847
remove_custom_field_from_tools(anthropic_messages_request)
# 6. AUTO-INJECT beta headers based on features used
anthropic_model_info = AnthropicModelInfo()
tools = anthropic_messages_optional_request_params.get("tools")

80
litellm/llms/bedrock_mantle/chat/transformation.py Normal file
View file

@ -0,0 +1,80 @@
"""
Amazon Bedrock Mantle - OpenAI-compatible inference engine in Amazon Bedrock.
API docs: https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html
Base URL: https://bedrock-mantle.{region}.api.aws/v1
Auth: AWS Bedrock API key as Bearer token (set via BEDROCK_MANTLE_API_KEY env var)
or region-aware key via BEDROCK_MANTLE_{REGION}_API_KEY.
"""
from typing import Iterator, AsyncIterator, Any, Optional, Tuple, Union
import litellm
from litellm._logging import verbose_logger
from litellm.secret_managers.main import get_secret_str
from ...openai_like.chat.transformation import OpenAILikeChatConfig
BEDROCK_MANTLE_DEFAULT_REGION = "us-east-1"
class BedrockMantleChatConfig(OpenAILikeChatConfig):
"""
Transformation config for Amazon Bedrock Mantle OpenAI-compatible API.
"""
@property
def custom_llm_provider(self) -> Optional[str]:
return "bedrock_mantle"
@classmethod
def get_config(cls):
return super().get_config()
def _get_openai_compatible_provider_info(
self, api_base: Optional[str], api_key: Optional[str]
) -> Tuple[Optional[str], Optional[str]]:
region = (
get_secret_str("BEDROCK_MANTLE_REGION")
or get_secret_str("AWS_REGION")
or BEDROCK_MANTLE_DEFAULT_REGION
)
api_base = (
api_base
or get_secret_str("BEDROCK_MANTLE_API_BASE")
or f"https://bedrock-mantle.{region}.api.aws/v1"
)
dynamic_api_key = api_key or get_secret_str("BEDROCK_MANTLE_API_KEY")
return api_base, dynamic_api_key
def get_supported_openai_params(self, model: str) -> list:
base_params = super().get_supported_openai_params(model)
try:
if litellm.supports_reasoning(
model=model, custom_llm_provider=self.custom_llm_provider
):
if "reasoning_effort" not in base_params:
base_params.append("reasoning_effort")
except Exception as e:
verbose_logger.debug(
f"BedrockMantleChatConfig: error checking reasoning support: {e}"
)
return base_params
def get_model_response_iterator(
self,
streaming_response: Union[Iterator[str], AsyncIterator[str], Any],
sync_stream: bool,
json_mode: Optional[bool] = False,
) -> Any:
from litellm.llms.openai.chat.gpt_transformation import (
OpenAIChatCompletionStreamingHandler,
)
return OpenAIChatCompletionStreamingHandler(
streaming_response=streaming_response,
sync_stream=sync_stream,
json_mode=json_mode,
)

5
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -4454,8 +4454,11 @@ class BaseLLMHTTPHandler:
return agentic_response
except Exception as e:
_call_id = getattr(logging_obj, "litellm_call_id", "unknown")
verbose_logger.exception(
f"LiteLLM.AgenticHookError: Exception in agentic completion hooks: {str(e)}"
"LiteLLM.AgenticHookError: Exception in agentic completion hooks "
"[call_id=%s model=%s]: %s",
_call_id, model, str(e),
)
# Check if we need to convert response to fake stream

115
litellm/llms/openai/chat/gpt_5_transformation.py
View file

@ -1,12 +1,30 @@
"""Support for OpenAI gpt-5 model family."""
from typing import Optional
from typing import Optional, Union
import litellm
from litellm.utils import _supports_factory
from .gpt_transformation import OpenAIGPTConfig
def _normalize_reasoning_effort_for_chat_completion(
value: Union[str, dict, None],
) -> Optional[str]:
"""Convert reasoning_effort to the string format expected by OpenAI chat completion API.
The chat completion API expects a simple string: 'none', 'low', 'medium', 'high', or 'xhigh'.
Config/deployments may pass the Responses API format: {'effort': 'high', 'summary': 'detailed'}.
"""
if value is None:
return None
if isinstance(value, str):
return value
if isinstance(value, dict) and "effort" in value:
return value["effort"]
return None
class OpenAIGPT5Config(OpenAIGPTConfig):
"""Configuration for gpt-5 models including GPT-5-Codex variants.
@ -40,41 +58,31 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
"""Check if the model is specifically a GPT-5 Codex variant."""
return "gpt-5-codex" in model
@classmethod
def is_model_gpt_5_1_codex_max_model(cls, model: str) -> bool:
"""Check if the model is the gpt-5.1-codex-max variant."""
model_name = model.split("/")[-1] # handle provider prefixes
return model_name == "gpt-5.1-codex-max"
@classmethod
def is_model_gpt_5_1_model(cls, model: str) -> bool:
"""Check if the model is a gpt-5.1 or gpt-5.2 chat variant.
gpt-5.1/5.2 support temperature when reasoning_effort="none",
unlike base gpt-5 which only supports temperature=1. Excludes
pro variants which keep stricter knobs and gpt-5.2-chat variants
which only support temperature=1.
"""
model_name = model.split("/")[-1]
is_gpt_5_1 = model_name.startswith("gpt-5.1")
is_gpt_5_2 = (
model_name.startswith("gpt-5.2")
and "pro" not in model_name
and not model_name.startswith("gpt-5.2-chat")
)
return is_gpt_5_1 or is_gpt_5_2
@classmethod
def is_model_gpt_5_2_pro_model(cls, model: str) -> bool:
"""Check if the model is the gpt-5.2-pro snapshot/alias."""
model_name = model.split("/")[-1]
return model_name.startswith("gpt-5.2-pro")
@classmethod
def is_model_gpt_5_2_model(cls, model: str) -> bool:
"""Check if the model is a gpt-5.2 variant (including pro)."""
model_name = model.split("/")[-1]
return model_name.startswith("gpt-5.2")
return model_name.startswith("gpt-5.2") or model_name.startswith("gpt-5.4")
@classmethod
def is_model_gpt_5_4_model(cls, model: str) -> bool:
"""Check if the model is a gpt-5.4 variant (including pro)."""
model_name = model.split("/")[-1]
return model_name.startswith("gpt-5.4")
@classmethod
def _supports_reasoning_effort_level(cls, model: str, level: str) -> bool:
"""Check if the model supports a specific reasoning_effort level.
Looks up ``supports_{level}_reasoning_effort`` in the model map via
the shared ``_supports_factory`` helper.
Returns False for unknown models (safe fallback).
"""
return _supports_factory(
model=model,
custom_llm_provider=None,
key=f"supports_{level}_reasoning_effort",
)
def get_supported_openai_params(self, model: str) -> list:
if self.is_model_gpt_5_search_model(model):
@ -114,7 +122,7 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
]
# gpt-5.1/5.2 support logprobs, top_p, top_logprobs when reasoning_effort="none"
if not self.is_model_gpt_5_1_model(model):
if not self._supports_reasoning_effort_level(model, "none"):
non_supported_params.extend(["logprobs", "top_p", "top_logprobs"])
return [
@ -142,21 +150,28 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
drop_params=drop_params,
)
reasoning_effort = (
# Normalize reasoning_effort: chat completion API expects a string, not a dict
# (e.g. {'effort': 'high', 'summary': 'detailed'} -> 'high')
raw_reasoning_effort = (
non_default_params.get("reasoning_effort")
or optional_params.get("reasoning_effort")
)
normalized = _normalize_reasoning_effort_for_chat_completion(raw_reasoning_effort)
if raw_reasoning_effort is not None and normalized is not None:
if "reasoning_effort" in non_default_params:
non_default_params["reasoning_effort"] = normalized
if "reasoning_effort" in optional_params:
optional_params["reasoning_effort"] = normalized
reasoning_effort = normalized or raw_reasoning_effort
if reasoning_effort is not None and reasoning_effort == "xhigh":
if not (
self.is_model_gpt_5_1_codex_max_model(model)
or self.is_model_gpt_5_2_model(model)
):
if not self._supports_reasoning_effort_level(model, "xhigh"):
if litellm.drop_params or drop_params:
non_default_params.pop("reasoning_effort", None)
else:
raise litellm.utils.UnsupportedParamsError(
message=(
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max and gpt-5.2 models."
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max, gpt-5.2, and gpt-5.4+ models."
),
status_code=400,
)
@ -170,8 +185,20 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
"max_tokens"
)
# gpt-5.4: function calls not supported when reasoning_effort != "none"
# Drop reasoning_effort when tools are present (small minority of volume)
if self.is_model_gpt_5_4_model(model):
has_tools = bool(
non_default_params.get("tools") or optional_params.get("tools")
)
if has_tools and reasoning_effort not in (None, "none"):
non_default_params.pop("reasoning_effort", None)
optional_params.pop("reasoning_effort", None)
reasoning_effort = None
# gpt-5.1/5.2 support logprobs, top_p, top_logprobs only when reasoning_effort="none"
if self.is_model_gpt_5_1_model(model):
supports_none = self._supports_reasoning_effort_level(model, "none")
if supports_none:
sampling_params = ["logprobs", "top_logprobs", "top_p"]
has_sampling = any(p in non_default_params for p in sampling_params)
if has_sampling and reasoning_effort not in (None, "none"):
@ -181,7 +208,7 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
else:
raise litellm.utils.UnsupportedParamsError(
message=(
"gpt-5.1/5.2 only support logprobs, top_p, top_logprobs when "
"gpt-5.1/5.2/5.4 only support logprobs, top_p, top_logprobs when "
"reasoning_effort='none'. Current reasoning_effort='{}'. "
"To drop unsupported params set `litellm.drop_params = True`"
).format(reasoning_effort),
@ -191,10 +218,8 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
if "temperature" in non_default_params:
temperature_value: Optional[float] = non_default_params.pop("temperature")
if temperature_value is not None:
is_gpt_5_1 = self.is_model_gpt_5_1_model(model)
# gpt-5.1 supports any temperature when reasoning_effort="none" (or not specified, as it defaults to "none")
if is_gpt_5_1 and (reasoning_effort == "none" or reasoning_effort is None):
# models supporting reasoning_effort="none" also support flexible temperature
if supports_none and (reasoning_effort == "none" or reasoning_effort is None):
optional_params["temperature"] = temperature_value
elif temperature_value == 1:
optional_params["temperature"] = temperature_value

5
litellm/llms/openai/chat/o_series_transformation.py
View file

@ -131,7 +131,10 @@ class OpenAIOSeriesConfig(OpenAIGPTConfig):
def is_model_o_series_model(self, model: str) -> bool:
model = model.split("/")[-1] # could be "openai/o3" or "o3"
return model.startswith(("o1", "o3", "o4")) and model in litellm.open_ai_chat_completion_models
return (
len(model) > 1 and model[0] == "o" and model[1].isdigit()
and model in litellm.open_ai_chat_completion_models
)
@overload
def _transform_messages(

8
litellm/llms/openai/openai.py
View file

@ -1938,7 +1938,7 @@ class OpenAIBatchesAPI(BaseLLM):
create_batch_data: CreateBatchRequest,
openai_client: AsyncOpenAI,
) -> LiteLLMBatch:
response = await openai_client.batches.create(**create_batch_data)
response = await openai_client.batches.create(**create_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
def create_batch(
@ -1974,7 +1974,7 @@ class OpenAIBatchesAPI(BaseLLM):
return self.acreate_batch( # type: ignore
create_batch_data=create_batch_data, openai_client=openai_client
)
response = cast(OpenAI, openai_client).batches.create(**create_batch_data)
response = cast(OpenAI, openai_client).batches.create(**create_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
@ -1984,7 +1984,7 @@ class OpenAIBatchesAPI(BaseLLM):
openai_client: AsyncOpenAI,
) -> LiteLLMBatch:
verbose_logger.debug("retrieving batch, args= %s", retrieve_batch_data)
response = await openai_client.batches.retrieve(**retrieve_batch_data)
response = await openai_client.batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
def retrieve_batch(
@ -2020,7 +2020,7 @@ class OpenAIBatchesAPI(BaseLLM):
return self.aretrieve_batch( # type: ignore
retrieve_batch_data=retrieve_batch_data, openai_client=openai_client
)
response = cast(OpenAI, openai_client).batches.retrieve(**retrieve_batch_data)
response = cast(OpenAI, openai_client).batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
return LiteLLMBatch(**response.model_dump())
async def acancel_batch(

4
litellm/llms/openrouter/image_edit/transformation.py
View file

@ -91,9 +91,9 @@ class OpenRouterImageEditConfig(BaseImageEditConfig):
if key == "size":
if "image_config" not in mapped_params:
mapped_params["image_config"] = {}
mapped_params["image_config"]["aspect_ratio"] = self._map_size_to_aspect_ratio(value)
mapped_params["image_config"]["aspect_ratio"] = self._map_size_to_aspect_ratio(cast(str, value))
elif key == "quality":
image_size = self._map_quality_to_image_size(value)
image_size = self._map_quality_to_image_size(cast(str, value))
if image_size:
if "image_config" not in mapped_params:
mapped_params["image_config"] = {}

6
litellm/llms/searchapi/search/transformation.py
View file

@ -3,7 +3,7 @@ Calls SearchAPI.io's Google Search API endpoint.
SearchAPI.io API Reference: https://www.searchapi.io/docs/google
"""
from typing import Dict, List, Literal, Optional, TypedDict, Union
from typing import Dict, List, Literal, Optional, TypedDict, Union, cast
from urllib.parse import urlencode
import httpx
@ -159,12 +159,12 @@ class SearchAPIConfig(BaseSearchConfig):
domains = optional_params["search_domain_filter"]
if isinstance(domains, list) and len(domains) > 0:
result_data["q"] = self._append_domain_filters(
result_data["q"], domains
str(result_data["q"]), domains
)
if "country" in optional_params:
# Map to gl parameter
result_data["gl"] = optional_params["country"].lower()
result_data["gl"] = cast(str, optional_params["country"]).lower()
# Pass through all other SearchAPI.io-specific parameters
for param, value in optional_params.items():

6
litellm/llms/serper/search/__init__.py Normal file
View file

@ -0,0 +1,6 @@
"""
Serper Search API module.
"""
from litellm.llms.serper.search.transformation import SerperSearchConfig
__all__ = ["SerperSearchConfig"]

167
litellm/llms/serper/search/transformation.py Normal file
View file

@ -0,0 +1,167 @@
"""
Calls Serper's /search endpoint to search Google.
Serper API Reference: https://serper.dev
"""
from typing import Dict, List, Optional, TypedDict, Union
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.search.transformation import (
BaseSearchConfig,
SearchResponse,
SearchResult,
)
from litellm.secret_managers.main import get_secret_str
class _SerperSearchRequestRequired(TypedDict):
"""Required fields for Serper Search API request."""
q: str # Required - search query
class SerperSearchRequest(_SerperSearchRequestRequired, total=False):
"""
Serper Search API request format.
Based on: https://serper.dev
"""
num: int # Optional - number of results to return, default 10
page: int # Optional - page number (default 1)
gl: str # Optional - country/geolocation code (e.g., "us", "gb")
hl: str # Optional - language code (e.g., "en", "de")
location: str # Optional - specific location for search targeting
autocorrect: bool # Optional - enable autocorrect (default True)
tbs: str # Optional - time-based search filter (e.g., "qdr:h", "qdr:d", "qdr:w")
class SerperSearchConfig(BaseSearchConfig):
SERPER_API_BASE = "https://google.serper.dev"
@staticmethod
def ui_friendly_name() -> str:
return "Serper"
def validate_environment(
self,
headers: Dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
**kwargs,
) -> Dict:
"""
Validate environment and return headers.
"""
api_key = api_key or get_secret_str("SERPER_API_KEY")
if not api_key:
raise ValueError("SERPER_API_KEY is not set. Set `SERPER_API_KEY` environment variable.")
headers["X-API-KEY"] = api_key
headers["Content-Type"] = "application/json"
return headers
def get_complete_url(
self,
api_base: Optional[str],
optional_params: dict,
data: Optional[Union[Dict, List[Dict]]] = None,
**kwargs,
) -> str:
"""
Get complete URL for Search endpoint.
"""
api_base = api_base or get_secret_str("SERPER_API_BASE") or self.SERPER_API_BASE
api_base = api_base.rstrip("/")
if not api_base.endswith("/search"):
api_base = f"{api_base}/search"
return api_base
def transform_search_request(
self,
query: Union[str, List[str]],
optional_params: dict,
**kwargs,
) -> Dict:
"""
Transform Search request to Serper API format.
Args:
query: Search query (string or list of strings). Serper only supports single string queries.
optional_params: Optional parameters for the request
- max_results: Maximum number of search results -> maps to `num`
- search_domain_filter: List of domains -> appended as site: clauses to `q`
- country: Country code filter (e.g., 'US', 'GB') -> maps to `gl` (lowercased)
Returns:
Dict with typed request data following SerperSearchRequest spec
"""
if isinstance(query, list):
query = " ".join(query)
request_data: SerperSearchRequest = {
"q": query,
}
if "max_results" in optional_params:
request_data["num"] = optional_params["max_results"]
if "country" in optional_params:
request_data["gl"] = optional_params["country"].lower()
if "search_domain_filter" in optional_params:
domains = optional_params["search_domain_filter"]
if isinstance(domains, list) and len(domains) > 0:
domain_clauses = " OR ".join(f"site:{d}" for d in domains)
request_data["q"] = f"({request_data['q']}) ({domain_clauses})"
# Convert to dict before dynamic key assignments
result_data = dict(request_data)
# pass through all other parameters as-is
for param, value in optional_params.items():
if param not in self.get_supported_perplexity_optional_params() and param not in result_data:
result_data[param] = value
return result_data
def transform_search_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
**kwargs,
) -> SearchResponse:
"""
Transform Serper API response to LiteLLM unified SearchResponse format.
Serper -> LiteLLM mappings:
- organic[].title -> SearchResult.title
- organic[].link -> SearchResult.url
- organic[].snippet -> SearchResult.snippet
- organic[].date -> SearchResult.date (optional, not always present)
Args:
raw_response: Raw httpx response from Serper API
logging_obj: Logging object for tracking
Returns:
SearchResponse with standardized format
"""
response_json = raw_response.json()
results = []
for result in response_json.get("organic", []):
search_result = SearchResult(
title=result.get("title", ""),
url=result.get("link", ""),
snippet=result.get("snippet", ""),
date=result.get("date"),
last_updated=None,
)
results.append(search_result)
return SearchResponse(
results=results,
object="search",
)

52
litellm/llms/vertex_ai/aws_credentials_supplier.py Normal file
View file

@ -0,0 +1,52 @@
"""
Custom AWS Security Credentials Supplier for Vertex AI WIF.
Wraps boto3/botocore credentials so that google-auth can use them
for the AWS-to-GCP Workload Identity Federation token exchange
without hitting the EC2 instance metadata service.
Requires google-auth >= 2.29.0.
"""
from typing import Callable
from google.auth import aws
class AwsCredentialsSupplier(aws.AwsSecurityCredentialsSupplier):
"""
Supplies AWS credentials to google-auth's aws.Credentials for WIF
token exchange.
This bypasses the default metadata-based credential retrieval,
allowing WIF to work in environments where EC2 metadata is blocked.
Accepts a credentials_provider callable that is invoked on every
get_aws_security_credentials() call, so that refreshed/rotated
credentials are picked up automatically (important for temporary
STS tokens).
"""
def __init__(self, credentials_provider: Callable, aws_region: str):
"""
Args:
credentials_provider: A zero-arg callable that returns a
botocore.credentials.Credentials object (with access_key,
secret_key, and token attributes).
aws_region: The AWS region string (e.g. "us-east-1").
"""
self._credentials_provider = credentials_provider
self._region = aws_region
def get_aws_security_credentials(self, context, request):
"""Return current AWS credentials for the GCP token exchange."""
current = self._credentials_provider()
return aws.AwsSecurityCredentials(
access_key_id=current.access_key,
secret_access_key=current.secret_key,
session_token=current.token,
)
def get_aws_region(self, context, request):
"""Return the AWS region for credential verification."""
return self._region

36
litellm/llms/vertex_ai/common_utils.py
View file

@ -571,38 +571,14 @@ def _filter_anyof_fields(schema_dict: Dict[str, Any]) -> Dict[str, Any]:
return schema_dict
def _is_any_type_schema(schema: dict) -> bool:
"""
Detect schemas that represent "any JSON value" (no type constraints).
In JSON Schema, an empty schema {} means "any value is valid".
Schemas with only metadata keys (title, description, default, examples)
but no type-constraining keywords also represent "any type".
Gemini's Schema proto uses TYPE_UNSPECIFIED (0) as default,
so omitting the type field is valid and means "any type".
"""
type_constraining_keys = {
"type",
"properties",
"items",
"anyOf",
"oneOf",
"allOf",
"enum",
"required",
"$ref",
"$schema",
}
return not any(key in type_constraining_keys for key in schema.keys())
def process_items(schema, depth=0):
if depth > DEFAULT_MAX_RECURSE_DEPTH:
raise ValueError(
f"Max depth of {DEFAULT_MAX_RECURSE_DEPTH} exceeded while processing schema. Please check the schema for excessive nesting."
)
if isinstance(schema, dict):
if "items" in schema and schema["items"] == {}:
schema["items"] = {"type": "object"}
for key, value in schema.items():
if isinstance(value, dict):
process_items(value, depth + 1)
@ -701,8 +677,9 @@ def convert_anyof_null_to_nullable(schema, depth=0):
# remove null type
anyof.remove(atype)
contains_null = True
elif isinstance(atype, dict) and _is_any_type_schema(atype):
pass # preserve "any type" semantics — don't coerce to object
elif "type" not in atype and len(atype) == 0:
# Handle empty object case
atype["type"] = "object"
if len(anyof) == 0:
# Edge case: response schema with only null type present is invalid in Vertex AI
@ -737,8 +714,7 @@ def add_object_type(schema):
# Gemini requires all function parameters to be type OBJECT
# Handle case where schema has no properties and no type (e.g. tools with no arguments)
if "type" not in schema and "anyOf" not in schema and "oneOf" not in schema and "allOf" not in schema:
if not _is_any_type_schema(schema):
schema["type"] = "object"
schema["type"] = "object"
properties = schema.get("properties", None)
if properties is not None:

2
litellm/llms/vertex_ai/gemini/transformation.py
View file

@ -595,6 +595,8 @@ def _transform_request_body(
safety_settings: Optional[List[SafetSettingsConfig]] = optional_params.pop(
"safety_settings", None
) # type: ignore
# Drop output_config as it's not supported by Vertex AI
optional_params.pop("output_config", None)
config_fields = GenerationConfig.__annotations__.keys()
# If the LiteLLM client sends Gemini-supported parameter "labels", add it

5
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -800,9 +800,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
GeminiThinkingConfig with thinkingLevel and includeThoughts
"""
# Check if this is gemini-3-flash which supports MINIMAL thinking level
# Covers gemini-3-flash, gemini-3-flash-preview, gemini-3.1-flash, gemini-3.1-flash-lite-preview, etc.
is_gemini3flash = model and (
"gemini-3-flash-preview" in model.lower()
or "gemini-3-flash" in model.lower()
"gemini-3-flash" in model.lower()
or "gemini-3.1-flash" in model.lower()
)
is_gemini31pro = model and (
"gemini-3.1-pro-preview" in model.lower()

125
litellm/llms/vertex_ai/vertex_ai_aws_wif.py Normal file
View file

@ -0,0 +1,125 @@
"""
AWS Workload Identity Federation (WIF) auth for Vertex AI.
Handles explicit AWS credentials for GCP WIF token exchange,
bypassing the EC2 instance metadata service.
When aws_* keys are present in the WIF credential JSON, this module
uses BaseAWSLLM to obtain AWS credentials and wraps them in a custom
AwsSecurityCredentialsSupplier for google-auth.
"""
from typing import Dict
GOOGLE_IMPORT_ERROR_MESSAGE = (
"Google Cloud SDK not found. Install it with: pip install 'litellm[google]' "
"or pip install google-cloud-aiplatform"
)
# AWS params recognized in WIF credential JSON for explicit auth.
# These match the kwargs accepted by BaseAWSLLM.get_credentials().
_AWS_CREDENTIAL_KEYS = frozenset({
"aws_access_key_id",
"aws_secret_access_key",
"aws_session_token",
"aws_region_name",
"aws_session_name",
"aws_profile_name",
"aws_role_name",
"aws_web_identity_token",
"aws_sts_endpoint",
"aws_external_id",
})
class VertexAIAwsWifAuth:
"""
Handles AWS-to-GCP Workload Identity Federation credential creation
for Vertex AI, using explicit AWS credentials rather than EC2 metadata.
"""
@staticmethod
def extract_aws_params(json_obj: dict) -> Dict[str, str]:
"""
Extract LiteLLM-specific aws_* keys from a WIF credential JSON dict.
Returns a dict of {param_name: value} for any recognized aws_* keys
found in the JSON. Returns empty dict if none are present.
"""
return {
key: json_obj[key]
for key in _AWS_CREDENTIAL_KEYS
if key in json_obj
}
@staticmethod
def credentials_from_explicit_aws(json_obj, aws_params, scopes):
"""
Create GCP credentials using explicit AWS credentials for WIF.
Uses BaseAWSLLM to obtain AWS credentials (via STS AssumeRole, profile,
static keys, etc.), then wraps them in a custom AwsSecurityCredentialsSupplier
so that google-auth bypasses the EC2 metadata service.
Args:
json_obj: The WIF credential JSON dict (contains audience, token_url, etc.)
aws_params: Dict of aws_* params extracted from json_obj
scopes: OAuth scopes for the GCP credentials
"""
try:
from google.auth import aws
except ImportError:
raise ImportError(GOOGLE_IMPORT_ERROR_MESSAGE)
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.vertex_ai.aws_credentials_supplier import (
AwsCredentialsSupplier,
)
# Validate region first — required for the GCP token exchange.
# Check before get_credentials() to avoid unnecessary AWS API calls
# (e.g. STS AssumeRole) on misconfiguration.
aws_region = aws_params.get("aws_region_name")
if not aws_region:
raise ValueError(
"aws_region_name is required in the WIF credential JSON "
"when using explicit AWS authentication. Add "
'"aws_region_name": "<your-region>" to your credential file.'
)
# Build a credentials provider that re-resolves AWS creds on each call.
# This ensures rotated/refreshed STS tokens are picked up during
# long-running processes when google-auth refreshes the GCP token.
base_aws = BaseAWSLLM()
aws_params_copy = dict(aws_params) # avoid mutating caller's dict
def _get_aws_credentials():
return base_aws.get_credentials(**aws_params_copy)
# Create the custom supplier with a lazy credentials provider
supplier = AwsCredentialsSupplier(
credentials_provider=_get_aws_credentials,
aws_region=aws_region,
)
# Build kwargs for aws.Credentials — forward optional fields from JSON
creds_kwargs = dict(
audience=json_obj.get("audience"),
subject_token_type=json_obj.get("subject_token_type"),
token_url=json_obj.get("token_url"),
credential_source=None, # Not using metadata endpoints
aws_security_credentials_supplier=supplier,
service_account_impersonation_url=json_obj.get(
"service_account_impersonation_url"
),
)
# Forward universe_domain if present (defaults to googleapis.com)
if "universe_domain" in json_obj:
creds_kwargs["universe_domain"] = json_obj["universe_domain"]
creds = aws.Credentials(**creds_kwargs)
if scopes and hasattr(creds, "requires_scopes") and creds.requires_scopes:
creds = creds.with_scopes(scopes)
return creds

Some files were not shown because too many files have changed in this diff Show more