scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-08 03:08:45 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 115

Merge pull request #19638 from BerriAI/main

Browse source 
merge main in stagin 1 22 26
This commit is contained in:
Sameer Kankute 2026-01-23 14:54:17 +05:30 • committed by GitHub
commit 12463809bd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
168 changed files with 13145 additions and 1181 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.github/workflows/create_daily_staging_branch.yml vendored
View file

@ -2,7 +2,7 @@ name: Create Daily Staging Branch
on:
schedule:
- cron: '0 0 * * *' # Runs daily at midnight UTC
- cron: '0 0,12 * * *' # Runs every 12 hours at midnight and noon UTC
workflow_dispatch: # Allow manual trigger
jobs:
@ -24,7 +24,7 @@ jobs:
git config user.email "github-actions[bot]@users.noreply.github.com"
# Generate branch name with MM_DD_YYYY format
BRANCH_NAME="litellm_staging_$(date +'%m_%d_%Y')"
BRANCH_NAME="litellm_oss_staging_$(date +'%m_%d_%Y')"
echo "Creating branch: $BRANCH_NAME"
# Fetch all branches

8
README.md
View file

@ -258,6 +258,14 @@ LiteLLM Performance: **8ms P95 latency** at 1k RPS (See benchmarks [here](https:
Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+).
## OSS Adopters
<img width="250" height="104" alt="Stripe wordmark - Blurple - Small" src="https://github.com/user-attachments/assets/f7296d4f-9fbd-460d-9d05-e4df31697c4b" />
<img width="250" height="69" alt="download__1_-removebg-preview" src="https://github.com/user-attachments/assets/0be4bd8a-7cfa-48d3-9090-f415fe948280" />
## Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers))
| Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` |

1
ci_cd/security_scans.sh
View file

@ -137,6 +137,7 @@ run_grype_scans() {
"CVE-2019-1010025" # glibc pthread heap address leak - awaiting patched Wolfi glibc build
"CVE-2026-22184" # zlib untgz buffer overflow - untgz unused + no fixed Wolfi build yet
"GHSA-58pv-8j8x-9vj2" # jaraco.context path traversal - setuptools vendored only (v5.3.0), not used in application code (using v6.1.0+)
"GHSA-r6q2-hw4h-h46w" # node-tar not used by application runtime, Linux-only container, not affect by macOS APFS-specific exploit
)
# Build JSON array of allowlisted CVE IDs for jq

136
cookbook/ai_coding_tool_guides/claude_code_quickstart/guide.md
View file

@ -97,17 +97,75 @@ export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
## Step 5: Use Claude Code
Start Claude Code and it will automatically use your configured models:
### Choosing Your Model
You have two options for specifying which model Claude Code uses:
#### Option 1: Command Line / Session Model Selection
Specify the model directly when starting Claude Code or during a session:
```bash
# Claude Code will use the models configured in your LiteLLM proxy
claude
# Or specify a model if you have multiple configured
# Specify model at startup
claude --model claude-3-5-sonnet-20241022
claude --model claude-3-5-haiku-20241022
# Or change model during a session
/model claude-3-5-haiku-20241022
```
This method uses the exact model you specify.
#### Option 2: Environment Variables
Configure default models using environment variables:
```bash
# Tell Claude Code which models to use by default
export ANTHROPIC_DEFAULT_SONNET_MODEL=claude-3-5-sonnet-20241022
export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-3-5-haiku-20241022
export ANTHROPIC_DEFAULT_OPUS_MODEL=claude-opus-3-5-20240229
claude # Will use the models specified above
```
**Note:** Claude Code may cache the model from a previous session. If environment variables don't take effect, use Option 1 to explicitly set the model.
**Important:** The `model_name` in your LiteLLM config must match what Claude Code requests (either from env vars or command line).
### Using 1M Context Window
Claude Code supports extended context (1 million tokens) using the `[1m]` suffix with Claude 4+ models:
```bash
# Use Sonnet 4.5 with 1M context (requires quotes for shell)
claude --model 'claude-sonnet-4-5-20250929[1m]'
# Inside a Claude Code session (no quotes needed)
/model claude-sonnet-4-5-20250929[1m]
```
**Important:** When using `--model` with `[1m]` in the shell, you must use quotes to prevent the shell from interpreting the brackets.
Alternatively, set as default with environment variables:
```bash
export ANTHROPIC_DEFAULT_SONNET_MODEL='claude-sonnet-4-5-20250929[1m]'
claude
```
**How it works:**
- Claude Code strips the `[1m]` suffix before sending to LiteLLM
- Claude Code automatically adds the header `anthropic-beta: context-1m-2025-08-07`
- Your LiteLLM config should **NOT** include `[1m]` in model names
**Verify 1M context is active:**
```bash
/context
# Should show: 21k/1000k tokens (2%)
```
**Pricing:** Models using 1M context have different pricing. Input tokens above 200k are charged at a higher rate.
## Troubleshooting
Common issues and solutions:
@ -123,18 +181,25 @@ Common issues and solutions:
- Ensure the `ANTHROPIC_AUTH_TOKEN` matches your LiteLLM master key
**Model not found:**
- Ensure the model name in Claude Code matches exactly with your `config.yaml`
- Check LiteLLM logs for detailed error messages
- Check what model Claude Code is requesting in LiteLLM logs
- Ensure your `config.yaml` has a matching `model_name` entry
- If using environment variables, verify they're set: `echo $ANTHROPIC_DEFAULT_SONNET_MODEL`
**1M context not working (showing 200k instead of 1000k):**
- Verify you're using the `[1m]` suffix: `/model your-model-name[1m]`
- Check LiteLLM logs for the header `context-1m-2025-08-07` in the request
- Ensure your model supports 1M context (only certain Claude models do)
- Your LiteLLM config should **NOT** include `[1m]` in the `model_name`
## Using Multiple Models and Providers
Expand your configuration to support multiple providers and models:
You can configure LiteLLM to route to any supported provider. Here's an example with multiple providers:
```yaml
model_list:
# OpenAI models
- model_name: codex-mini
litellm_params:
litellm_params:
model: openai/codex-mini
api_key: os.environ/OPENAI_API_KEY
api_base: https://api.openai.com/v1
@ -156,7 +221,7 @@ model_list:
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-5-haiku-20241022
litellm_params:
model: anthropic/claude-3-5-haiku-20241022
@ -174,19 +239,54 @@ litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
```
**Note:** The `model_name` can be anything you choose. Claude Code will request whatever model you specify (via env vars or command line), and LiteLLM will route to the `model` configured in `litellm_params`.
Switch between models seamlessly:
```bash
# Use Claude for complex reasoning
claude --model claude-3-5-sonnet-20241022
# Use environment variables to set defaults
export ANTHROPIC_DEFAULT_SONNET_MODEL=claude-3-5-sonnet-20241022
export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-3-5-haiku-20241022
# Use Haiku for fast responses
claude --model claude-3-5-haiku-20241022
# Use Bedrock deployment
claude --model claude-bedrock
# Or specify directly
claude --model claude-3-5-sonnet-20241022 # Complex reasoning
claude --model claude-3-5-haiku-20241022 # Fast responses
claude --model claude-bedrock # Bedrock deployment
```
## Default Models Used by Claude Code
If you **don't** set environment variables, Claude Code uses these default model names:
| Purpose | Default Model Name (v2.1.14) |
|---------|------------------------------|
| Main model | `claude-sonnet-4-5-20250929` |
| Light tasks (subagents, summaries) | `claude-haiku-4-5-20251001` |
| Planning mode | `claude-opus-4-5-20251101` |
Your LiteLLM config should include these model names if you want Claude Code to work without setting environment variables:
```yaml
model_list:
- model_name: claude-sonnet-4-5-20250929
litellm_params:
# Can be any provider - Anthropic, Bedrock, Vertex AI, etc.
model: anthropic/claude-sonnet-4-5-20250929
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-haiku-4-5-20251001
litellm_params:
model: anthropic/claude-haiku-4-5-20251001
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-opus-4-5-20251101
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
**Warning:** These default model names may change with new Claude Code versions. Check LiteLLM proxy logs for "model not found" errors to identify what Claude Code is requesting.
## Additional Resources
- [LiteLLM Documentation](https://docs.litellm.ai/)

1
docker/Dockerfile.non_root
View file

@ -15,6 +15,7 @@ USER root
RUN for i in 1 2 3; do \
apk add --no-cache \
python3 \
python3-dev \
py3-pip \
clang \
llvm \

0
docs/my-website/docs/anthropic_unified.md → docs/my-website/docs/anthropic_unified/index.md
View file

294
docs/my-website/docs/anthropic_unified/structured_output.md Normal file
View file

@ -0,0 +1,294 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Structured Output /v1/messages
Use LiteLLM to call Anthropic's structured output feature via the `/v1/messages` endpoint.
## Supported Providers
| Provider | Supported | Notes |
|----------|-----------|-------|
| Anthropic | ✅ | Native support |
| Azure AI (Anthropic models) | ✅ | Claude models on Azure AI |
| Bedrock (Converse Anthropic models) | ✅ | Claude models via Bedrock Converse API |
| Bedrock (Invoke Anthropic models) | ✅ | Claude models via Bedrock Invoke API |
## Usage
### LiteLLM Proxy Server
<Tabs>
<TabItem value="anthropic" label="Anthropic">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-5-20250514
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl http://localhost:4000/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "claude-sonnet",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract the key information from this email: John Smith (john@example.com) is interested in our Enterprise plan and wants to schedule a demo for next Tuesday at 2pm."
}
],
"output_format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"plan_interest": {"type": "string"},
"demo_requested": {"type": "boolean"}
},
"required": ["name", "email", "plan_interest", "demo_requested"],
"additionalProperties": false
}
}
}'
```
</TabItem>
<TabItem value="azure_ai" label="Azure AI (Anthropic)">
1. Setup config.yaml
```yaml
model_list:
- model_name: azure-claude-sonnet
litellm_params:
model: azure_ai/claude-sonnet-4-5-20250514
api_key: os.environ/AZURE_AI_API_KEY
api_base: https://your-endpoint.inference.ai.azure.com
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl http://localhost:4000/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "azure-claude-sonnet",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract the key information from this email: John Smith (john@example.com) is interested in our Enterprise plan and wants to schedule a demo for next Tuesday at 2pm."
}
],
"output_format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"plan_interest": {"type": "string"},
"demo_requested": {"type": "boolean"}
},
"required": ["name", "email", "plan_interest", "demo_requested"],
"additionalProperties": false
}
}
}'
```
</TabItem>
<TabItem value="bedrock" label="Bedrock (Converse)">
1. Setup config.yaml
```yaml
model_list:
- model_name: bedrock-claude-sonnet
litellm_params:
model: bedrock/global.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-west-2
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl http://localhost:4000/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "bedrock-claude-sonnet",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract the key information from this email: John Smith (john@example.com) is interested in our Enterprise plan and wants to schedule a demo for next Tuesday at 2pm."
}
],
"output_format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"plan_interest": {"type": "string"},
"demo_requested": {"type": "boolean"}
},
"required": ["name", "email", "plan_interest", "demo_requested"],
"additionalProperties": false
}
}
}'
```
</TabItem>
<TabItem value="bedrock_invoke" label="Bedrock (Invoke)">
1. Setup config.yaml
```yaml
model_list:
- model_name: bedrock-claude-invoke
litellm_params:
model: bedrock/invoke/global.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-west-2
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl http://localhost:4000/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "bedrock-claude-invoke",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract the key information from this email: John Smith (john@example.com) is interested in our Enterprise plan and wants to schedule a demo for next Tuesday at 2pm."
}
],
"output_format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"plan_interest": {"type": "string"},
"demo_requested": {"type": "boolean"}
},
"required": ["name", "email", "plan_interest", "demo_requested"],
"additionalProperties": false
}
}
}'
```
</TabItem>
</Tabs>
## Example Response
```json
{
"id": "msg_01XFDUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"content": [
{
"type": "text",
"text": "{\"name\":\"John Smith\",\"email\":\"john@example.com\",\"plan_interest\":\"Enterprise\",\"demo_requested\":true}"
}
],
"model": "claude-sonnet-4-5-20250514",
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 75,
"output_tokens": 28
}
}
```
## Request Format
### output_format
The `output_format` parameter specifies the structured output format.
```json
{
"output_format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"field_name": {"type": "string"},
"another_field": {"type": "integer"}
},
"required": ["field_name", "another_field"],
"additionalProperties": false
}
}
}
```
#### Fields
- **type** (string): Must be `"json_schema"`
- **schema** (object): A JSON Schema object defining the expected output structure
- **type** (string): The root type, typically `"object"`
- **properties** (object): Defines the fields and their types
- **required** (array): List of required field names
- **additionalProperties** (boolean): Set to `false` to enforce strict schema adherence

2
docs/my-website/docs/completion/input.md
View file

@ -199,6 +199,8 @@ messages=[{"role": "user", "content": [
- `include_usage` *boolean (optional)* - If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value.
- `stop`: *string/ array/ null (optional)* - Up to 4 sequences where the API will stop generating further tokens.
**Note**: OpenAI supports a maximum of 4 stop sequences. If you provide more than 4, LiteLLM will automatically truncate the list to the first 4 elements. To disable this automatic truncation, set `litellm.disable_stop_sequence_limit = True`.
- `max_completion_tokens`: *integer (optional)* - An upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens.

33
docs/my-website/docs/guides/security_settings.md
View file

@ -187,4 +187,37 @@ export AIOHTTP_TRUST_ENV='True'
```
</TabItem>
</Tabs>
## 7. Per-Service SSL Verification
LiteLLM allows you to override SSL verification settings for specific services or provider calls. This is useful when different services (e.g., an internal guardrail vs. a public LLM provider) require different CA certificates.
### Bedrock (SDK)
You can pass `ssl_verify` directly in the `completion` call.
```python
import litellm
response = litellm.completion(
model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
messages=[{"role": "user", "content": "hi"}],
ssl_verify="path/to/bedrock_cert.pem" # Or False to disable
)
```
### AIM Guardrail (Proxy)
You can configure `ssl_verify` per guardrail in your `config.yaml`.
```yaml
guardrails:
- guardrail_name: aim-protected-app
litellm_params:
guardrail: aim
ssl_verify: "/path/to/aim_cert.pem" # Use specific cert for AIM
```
### Priority Logic
LiteLLM resolves `ssl_verify` using the following priority:
1. **Explicit Parameter**: Passed in `completion()` or guardrail config.
2. **Environment Variable**: `SSL_VERIFY` environment variable.
3. **Global Setting**: `litellm.ssl_verify` setting.
4. **System Standard**: `SSL_CERT_FILE` environment variable.

6
docs/my-website/docs/observability/opentelemetry_integration.md
View file

@ -63,6 +63,8 @@ OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_HEADERS="api-key=key,other-config-value=value"
```
> Note: OTLP gRPC requires `grpcio`. Install via `pip install "litellm[grpc]"` (or `grpcio`).
</TabItem>
<TabItem value="laminar" label="Log to Laminar">
@ -73,6 +75,8 @@ OTEL_ENDPOINT="https://api.lmnr.ai:8443"
OTEL_HEADERS="authorization=Bearer <project-api-key>"
```
> Note: OTLP gRPC requires `grpcio`. Install via `pip install "litellm[grpc]"` (or `grpcio`).
</TabItem>
</Tabs>
@ -128,4 +132,4 @@ If you don't see traces landing on your integration, set `OTEL_DEBUG="True"` in
export OTEL_DEBUG="True"
```
This will emit any logging issues to the console.
This will emit any logging issues to the console.

2
docs/my-website/docs/observability/phoenix_integration.md
View file

@ -73,6 +73,8 @@ environment_variables:
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the HTTP endpoint
```
> Note: If you set the gRPC endpoint, install `grpcio` via `pip install "litellm[grpc]"` (or `grpcio`).
2. Start the proxy
```bash

4
docs/my-website/docs/observability/signoz.md
View file

@ -99,6 +99,8 @@ OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=openai \
opentelemetry-instrument <your_run_command>
```
> Note: OTLP gRPC requires `grpcio`. Install via `pip install "litellm[grpc]"` (or `grpcio`).
> 📌 Note: We're using `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=openai` in the run command to disable the OpenAI instrumentor for tracing. This avoids conflicts with LiteLLM's native telemetry/instrumentation, ensuring that telemetry is captured exclusively through LiteLLM's built-in instrumentation.
- **`<service_name>`** is the name of your service
@ -362,6 +364,8 @@ export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
```
> Note: OTLP gRPC requires `grpcio`. Install via `pip install "litellm[grpc]"` (or `grpcio`).
- Set the `<region>` to match your SigNoz Cloud [region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint)
- Replace `<your_ingestion_key>` with your SigNoz [ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/)

45
docs/my-website/docs/pass_through/vertex_ai.md
View file

@ -461,3 +461,48 @@ generateContent();
</TabItem>
</Tabs>
### Using Anthropic Beta Features on Vertex AI
When using Anthropic models via Vertex AI passthrough (e.g., Claude on Vertex), you can enable Anthropic beta features like extended context windows.
The `anthropic-beta` header is automatically forwarded to Vertex AI when calling Anthropic models.
```bash
curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet:rawPredict \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-H "anthropic-beta: context-1m-2025-08-07" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 500
}'
```
### Forwarding Custom Headers with `x-pass-` Prefix
You can forward any custom header to the provider by prefixing it with `x-pass-`. The prefix is stripped before the header is sent to the provider.
For example:
- `x-pass-anthropic-beta: value` becomes `anthropic-beta: value`
- `x-pass-custom-header: value` becomes `custom-header: value`
This is useful when you need to send provider-specific headers that aren't in the default allowlist.
```bash
curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet:rawPredict \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-H "x-pass-anthropic-beta: context-1m-2025-08-07" \
-H "x-pass-custom-feature: enabled" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 500
}'
```
:::info
The `x-pass-` prefix works for all LLM pass-through endpoints, not just Vertex AI.
:::

140
docs/my-website/docs/providers/gmi.md Normal file
View file

@ -0,0 +1,140 @@
# GMI Cloud
## Overview
| Property | Details |
|-------|-------|
| Description | GMI Cloud is a GPU cloud infrastructure provider offering access to top AI models including Claude, GPT, DeepSeek, Gemini, and more through OpenAI-compatible APIs. |
| Provider Route on LiteLLM | `gmi/` |
| Link to Provider Doc | [GMI Cloud Docs ↗](https://docs.gmicloud.ai) |
| Base URL | `https://api.gmi-serving.com/v1` |
| Supported Operations | [`/chat/completions`](#sample-usage), [`/models`](#supported-models) |
<br />
## What is GMI Cloud?
GMI Cloud is a venture-backed digital infrastructure company ($82M+ funding) providing:
- **Top-tier GPU Access**: NVIDIA H100 GPUs for AI workloads
- **Multiple AI Models**: Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, and more
- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK
- **Global Infrastructure**: Data centers in US (Colorado) and APAC (Taiwan)
## Required Variables
```python showLineNumbers title="Environment Variables"
os.environ["GMI_API_KEY"] = "" # your GMI Cloud API key
```
Get your GMI Cloud API key from [console.gmicloud.ai](https://console.gmicloud.ai).
## Usage - LiteLLM Python SDK
### Non-streaming
```python showLineNumbers title="GMI Cloud Non-streaming Completion"
import os
import litellm
from litellm import completion
os.environ["GMI_API_KEY"] = "" # your GMI Cloud API key
messages = [{"content": "What is the capital of France?", "role": "user"}]
# GMI Cloud call
response = completion(
model="gmi/deepseek-ai/DeepSeek-V3.2",
messages=messages
)
print(response)
```
### Streaming
```python showLineNumbers title="GMI Cloud Streaming Completion"
import os
import litellm
from litellm import completion
os.environ["GMI_API_KEY"] = "" # your GMI Cloud API key
messages = [{"content": "Write a short poem about AI", "role": "user"}]
# GMI Cloud call with streaming
response = completion(
model="gmi/anthropic/claude-sonnet-4.5",
messages=messages,
stream=True
)
for chunk in response:
print(chunk)
```
## Usage - LiteLLM Proxy Server
### 1. Save key in your environment
```bash
export GMI_API_KEY=""
```
### 2. Start the proxy
```yaml
model_list:
- model_name: deepseek-v3
litellm_params:
model: gmi/deepseek-ai/DeepSeek-V3.2
api_key: os.environ/GMI_API_KEY
- model_name: claude-sonnet
litellm_params:
model: gmi/anthropic/claude-sonnet-4.5
api_key: os.environ/GMI_API_KEY
```
## Supported Models
| Model | Model ID | Context Length |
|-------|----------|----------------|
| Claude Opus 4.5 | `gmi/anthropic/claude-opus-4.5` | 409K |
| Claude Sonnet 4.5 | `gmi/anthropic/claude-sonnet-4.5` | 409K |
| Claude Sonnet 4 | `gmi/anthropic/claude-sonnet-4` | 409K |
| Claude Opus 4 | `gmi/anthropic/claude-opus-4` | 409K |
| GPT-5.2 | `gmi/openai/gpt-5.2` | 409K |
| GPT-5.1 | `gmi/openai/gpt-5.1` | 409K |
| GPT-5 | `gmi/openai/gpt-5` | 409K |
| GPT-4o | `gmi/openai/gpt-4o` | 131K |
| GPT-4o-mini | `gmi/openai/gpt-4o-mini` | 131K |
| DeepSeek V3.2 | `gmi/deepseek-ai/DeepSeek-V3.2` | 163K |
| DeepSeek V3 0324 | `gmi/deepseek-ai/DeepSeek-V3-0324` | 163K |
| Gemini 3 Pro | `gmi/google/gemini-3-pro-preview` | 1M |
| Gemini 3 Flash | `gmi/google/gemini-3-flash-preview` | 1M |
| Kimi K2 Thinking | `gmi/moonshotai/Kimi-K2-Thinking` | 262K |
| MiniMax M2.1 | `gmi/MiniMaxAI/MiniMax-M2.1` | 196K |
| Qwen3-VL 235B | `gmi/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` | 262K |
| GLM-4.7 | `gmi/zai-org/GLM-4.7-FP8` | 202K |
## Supported OpenAI Parameters
GMI Cloud supports all standard OpenAI-compatible parameters:
| Parameter | Type | Description |
|-----------|------|-------------|
| `messages` | array | **Required**. Array of message objects with 'role' and 'content' |
| `model` | string | **Required**. Model ID from available models |
| `stream` | boolean | Optional. Enable streaming responses |
| `temperature` | float | Optional. Sampling temperature |
| `top_p` | float | Optional. Nucleus sampling parameter |
| `max_tokens` | integer | Optional. Maximum tokens to generate |
| `frequency_penalty` | float | Optional. Penalize frequent tokens |
| `presence_penalty` | float | Optional. Penalize tokens based on presence |
| `stop` | string/array | Optional. Stop sequences |
| `response_format` | object | Optional. JSON mode with `{"type": "json_object"}` |
## Additional Resources
- [GMI Cloud Website](https://www.gmicloud.ai)
- [GMI Cloud Documentation](https://docs.gmicloud.ai)
- [GMI Cloud Console](https://console.gmicloud.ai)

1
docs/my-website/docs/proxy/config_settings.md
View file

@ -178,6 +178,7 @@ router_settings:
| turn_off_message_logging | boolean | If true, prevents messages and responses from being logged to callbacks, but request metadata will still be logged. Useful for privacy/compliance when handling sensitive data [Proxy Logging](logging) |
| modify_params | boolean | If true, allows modifying the parameters of the request before it is sent to the LLM provider |
| enable_preview_features | boolean | If true, enables preview features - e.g. Azure O1 Models with streaming support.|
| LITELLM_DISABLE_STOP_SEQUENCE_LIMIT | Disable validation for stop sequence limit (default: 4) |
| redact_user_api_key_info | boolean | If true, redacts information about the user api key from logs [Proxy Logging](logging#redacting-userapikeyinfo) |
| mcp_aliases | object | Maps friendly aliases to MCP server names for easier tool access. Only the first alias for each server is used. [MCP Aliases](../mcp#mcp-aliases) |
| langfuse_default_tags | array of strings | Default tags for Langfuse Logging. Use this if you want to control which LiteLLM-specific fields are logged as tags by the LiteLLM proxy. By default LiteLLM Proxy logs no LiteLLM-specific fields as tags. [Further docs](./logging#litellm-specific-tags-on-langfuse---cache_hit-cache_key) |

22
docs/my-website/docs/proxy/custom_pricing.md
View file

@ -127,6 +127,28 @@ model_list:
base_model: azure/gpt-4-1106-preview
```
### OpenAI Models with Dated Versions
`base_model` is also useful when OpenAI returns a dated model name in the response that differs from your configured model name.
**Example**: You configure custom pricing for `gpt-4o-mini-audio-preview`, but OpenAI returns `gpt-4o-mini-audio-preview-2024-12-17` in the response. Since LiteLLM uses the response model name for pricing lookup, your custom pricing won't be applied.
**Solution** ✅: Set `base_model` to the key you want LiteLLM to use for pricing lookup.
```yaml
model_list:
- model_name: my-audio-model
litellm_params:
model: openai/gpt-4o-mini-audio-preview
api_key: os.environ/OPENAI_API_KEY
model_info:
base_model: gpt-4o-mini-audio-preview # 👈 Used for pricing lookup
input_cost_per_token: 0.0000006
output_cost_per_token: 0.0000024
input_cost_per_audio_token: 0.00001
output_cost_per_audio_token: 0.00002
```
## Debugging

1
docs/my-website/docs/proxy/guardrails/aim_security.md
View file

@ -46,6 +46,7 @@ guardrails:
mode: [pre_call, post_call] # "During_call" is also available
api_key: os.environ/AIM_API_KEY
api_base: os.environ/AIM_API_BASE # Optional, use only when using a self-hosted Aim Outpost
ssl_verify: False # Optional, set to False to disable SSL verification or a string path to a custom CA bundle
```
Under the `api_key`, insert the API key you were issued. The key can be found in the guard's page.

283
docs/my-website/docs/proxy/guardrails/guardrail_policies.md Normal file
View file

@ -0,0 +1,283 @@
# [Beta] Guardrail Policies
Use policies to group guardrails and control which ones run for specific teams, keys, or models.
## Why use policies?
- Enable/disable specific guardrails for teams, keys, or models
- Group guardrails into a single policy
- Inherit from existing policies and override what you need
## Quick Start
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
# 1. Define your guardrails
guardrails:
- guardrail_name: pii_masking
litellm_params:
guardrail: presidio
mode: pre_call
- guardrail_name: prompt_injection
litellm_params:
guardrail: lakera
mode: pre_call
api_key: os.environ/LAKERA_API_KEY
# 2. Create a policy
policies:
my-policy:
guardrails:
add:
- pii_masking
- prompt_injection
# 3. Attach the policy
policy_attachments:
- policy: my-policy
scope: "*" # apply to all requests
```
Response headers show what ran:
```
x-litellm-applied-policies: my-policy
x-litellm-applied-guardrails: pii_masking,prompt_injection
```
## Add guardrails for a specific team
:::info
✨ Enterprise only feature for team/key-based policy attachments. [Get a free trial](https://www.litellm.ai/enterprise#trial)
:::
You have a global baseline, but want to add extra guardrails for a specific team.
```yaml showLineNumbers title="config.yaml"
policies:
global-baseline:
guardrails:
add:
- pii_masking
finance-team-policy:
inherit: global-baseline
guardrails:
add:
- strict_compliance_check
- audit_logger
policy_attachments:
- policy: global-baseline
scope: "*"
- policy: finance-team-policy
teams:
- finance # team alias from /team/new
```
Now the `finance` team gets `pii_masking` + `strict_compliance_check` + `audit_logger`, while everyone else just gets `pii_masking`.
## Remove guardrails for a specific team
:::info
✨ Enterprise only feature for team/key-based policy attachments. [Get a free trial](https://www.litellm.ai/enterprise#trial)
:::
You have guardrails running globally, but want to disable some for a specific team (e.g., internal testing).
```yaml showLineNumbers title="config.yaml"
policies:
global-baseline:
guardrails:
add:
- pii_masking
- prompt_injection
internal-team-policy:
inherit: global-baseline
guardrails:
remove:
- pii_masking # don't need PII masking for internal testing
policy_attachments:
- policy: global-baseline
scope: "*"
- policy: internal-team-policy
teams:
- internal-testing # team alias from /team/new
```
Now the `internal-testing` team only gets `prompt_injection`, while everyone else gets both guardrails.
## Inheritance
Start with a base policy and build on it:
```yaml showLineNumbers title="config.yaml"
policies:
base:
guardrails:
add:
- pii_masking
- toxicity_filter
strict:
inherit: base
guardrails:
add:
- prompt_injection
relaxed:
inherit: base
guardrails:
remove:
- toxicity_filter
```
What you get:
- `base` → `[pii_masking, toxicity_filter]`
- `strict` → `[pii_masking, toxicity_filter, prompt_injection]`
- `relaxed` → `[pii_masking]`
## Model Conditions
Run guardrails only for specific models:
```yaml showLineNumbers title="config.yaml"
policies:
gpt4-safety:
guardrails:
add:
- strict_content_filter
condition:
model: "gpt-4.*" # regex - matches gpt-4, gpt-4-turbo, gpt-4o
bedrock-compliance:
guardrails:
add:
- audit_logger
condition:
model: # exact match list
- bedrock/claude-3
- bedrock/claude-2
```
## Attachments
Policies don't do anything until you attach them. Attachments tell LiteLLM *where* to apply each policy.
**Global** - runs on every request:
```yaml showLineNumbers title="config.yaml"
policy_attachments:
- policy: default
scope: "*"
```
**Team-specific** (uses team alias from `/team/new`):
```yaml showLineNumbers title="config.yaml"
policy_attachments:
- policy: hipaa-compliance
teams:
- healthcare-team # team alias
- medical-research # team alias
```
**Key-specific** (uses key alias from `/key/generate`, wildcards supported):
```yaml showLineNumbers title="config.yaml"
policy_attachments:
- policy: internal-testing
keys:
- "dev-*" # key alias pattern
- "test-*" # key alias pattern
```
## Config Reference
### `policies`
```yaml
policies:
<policy-name>:
description: ...
inherit: ...
guardrails:
add: [...]
remove: [...]
condition:
model: ...
```
| Field | Type | Description |
|-------|------|-------------|
| `description` | `string` | Optional. What this policy does. |
| `inherit` | `string` | Optional. Parent policy to inherit guardrails from. |
| `guardrails.add` | `list[string]` | Guardrails to enable. |
| `guardrails.remove` | `list[string]` | Guardrails to disable (useful with inheritance). |
| `condition.model` | `string` or `list[string]` | Optional. Only apply when model matches. Supports regex. |
### `policy_attachments`
```yaml
policy_attachments:
- policy: ...
scope: ...
teams: [...]
keys: [...]
```
| Field | Type | Description |
|-------|------|-------------|
| `policy` | `string` | **Required.** Name of the policy to attach. |
| `scope` | `string` | Use `"*"` to apply globally. |
| `teams` | `list[string]` | Team aliases (from `/team/new`). |
| `keys` | `list[string]` | Key aliases (from `/key/generate`). Supports `*` wildcard. |
### Response Headers
| Header | Description |
|--------|-------------|
| `x-litellm-applied-policies` | Policies that matched this request |
| `x-litellm-applied-guardrails` | Guardrails that actually ran |
## How it works
Example config:
```yaml showLineNumbers title="config.yaml"
policies:
base:
guardrails:
add: [pii_masking]
finance-policy:
inherit: base
guardrails:
add: [audit_logger]
policy_attachments:
- policy: base
scope: "*"
- policy: finance-policy
teams: [finance]
```
```mermaid
flowchart TD
A["Request with team_alias='finance'"] --> B["Matches policies: base, finance-policy"]
B --> C["Resolves guardrails: pii_masking, audit_logger"]
```
1. Request comes in with `team_alias='finance'`
2. Matches `base` (via `scope: "*"`) and `finance-policy` (via `teams: [finance]`)
3. Resolves guardrails: `base` adds `pii_masking`, `finance-policy` inherits and adds `audit_logger`
4. Final guardrails: `pii_masking`, `audit_logger`

6
docs/my-website/docs/proxy/guardrails/quick_start.md
View file

@ -203,8 +203,12 @@ Your response headers will include `x-litellm-applied-guardrails` with the guard
x-litellm-applied-guardrails: aporia-pre-guard
```
### Guardrail Policies
Need more control? Use [Guardrail Policies](./guardrail_policies.md) to:
- Group guardrails into reusable policies
- Enable/disable guardrails for specific teams, keys, or models
- Inherit from existing policies and override specific guardrails
## **Using Guardrails Client Side**

2
docs/my-website/docs/proxy/logging.md
View file

@ -982,6 +982,8 @@ OTEL_ENDPOINT="http:/0.0.0.0:4317"
OTEL_HEADERS="x-honeycomb-team=<your-api-key>" # Optional
```
> Note: OTLP gRPC requires `grpcio`. Install via `pip install "litellm[grpc]"` (or `grpcio`).
Add `otel` as a callback on your `litellm_config.yaml`
```shell

55
docs/my-website/docs/search/brave.md Normal file
View file

@ -0,0 +1,55 @@
# Brave Search
Get started by creating a free API key via https://brave.com/search/api/.
For documentation on other parameters supported by the Brave Search API, visit https://api-dashboard.search.brave.com/api-reference/web/search.
## LiteLLM Python SDK
```python showLineNumbers title="Brave Search"
import os
from litellm import search
os.environ["BRAVE_API_KEY"] = "BSATzx..."
response = search(
query="Brave browser features",
search_provider="brave",
max_results=5
)
```
## LiteLLM AI Gateway
### 1. Setup config.yaml
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
search_tools:
- search_tool_name: brave-search
litellm_params:
search_provider: brave
api_key: os.environ/BRAVE_API_KEY
```
### 2. Start the proxy
```bash
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
### 3. Test the search endpoint
```bash showLineNumbers title="Test Request"
curl http://0.0.0.0:4000/v1/search/brave-search \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{ "query": "Brave browser features", "max_results": 5 }'
```

10
docs/my-website/docs/search/index.md
View file

@ -2,7 +2,7 @@
| Feature | Supported |
|---------|-----------|
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `google_pse`, `dataforseo`, `firecrawl`, `searxng`, `linkup` |
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `brave`, `google_pse`, `dataforseo`, `firecrawl`, `searxng`, `linkup` |
| Cost Tracking | ✅ |
| Logging | ✅ |
| Load Balancing | ❌ |
@ -162,6 +162,11 @@ search_tools:
search_provider: exa_ai
api_key: os.environ/EXA_API_KEY
- search_tool_name: my-search
litellm_params:
search_provider: brave
api_key: os.environ/BRAVE_API_KEY
router_settings:
routing_strategy: simple-shuffle # or 'least-busy', 'latency-based-routing'
```
@ -205,7 +210,7 @@ See the [official Perplexity Search documentation](https://docs.perplexity.ai/ap
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string or array | Yes | Search query. Can be a single string or array of strings |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, `"searxng"`, or `"linkup"` |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"brave"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, `"searxng"`, or `"linkup"` |
| `search_tool_name` | string | Yes (Proxy) | Name of the search tool configured in `config.yaml` |
| `max_results` | integer | No | Maximum number of results to return (1-20). Default: 10 |
| `search_domain_filter` | array | No | List of domains to filter results (max 20 domains) |
@ -264,6 +269,7 @@ The response follows Perplexity's search format with the following structure:
| Perplexity AI | `PERPLEXITYAI_API_KEY` | `perplexity` |
| Tavily | `TAVILY_API_KEY` | `tavily` |
| Exa AI | `EXA_API_KEY` | `exa_ai` |
| Brave Search | `BRAVE_API_KEY` | `brave` |
| Parallel AI | `PARALLEL_AI_API_KEY` | `parallel_ai` |
| Google PSE | `GOOGLE_PSE_API_KEY`, `GOOGLE_PSE_ENGINE_ID` | `google_pse` |
| DataForSEO | `DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD` | `dataforseo` |

79
docs/my-website/docs/tutorials/claude_responses_api.md
View file

@ -37,18 +37,22 @@ Create a secure configuration using environment variables:
```yaml
model_list:
# Claude models
- model_name: claude-3-5-sonnet-20241022
# Configure the models you want to use
- model_name: claude-sonnet-4-5-20250929
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-5-haiku-20241022
litellm_params:
model: anthropic/claude-3-5-haiku-20241022
model: anthropic/claude-sonnet-4-5-20250929
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-haiku-4-5-20251001
litellm_params:
model: anthropic/claude-haiku-4-5-20251001
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-opus-4-5-20251101
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
```
@ -60,6 +64,10 @@ export ANTHROPIC_API_KEY="your-anthropic-api-key"
export LITELLM_MASTER_KEY="sk-1234567890" # Generate a secure key
```
:::tip
Alternatively, you can store `ANTHROPIC_API_KEY` in a `.env` file in your proxy directory. LiteLLM will automatically load it when starting.
:::
### 2. Start proxy
```bash
@ -111,15 +119,55 @@ export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
### 5. Use Claude Code
Start Claude Code and it will automatically use your configured models:
Start Claude Code with the model you want to use:
```bash
# Claude Code will use the models configured in your LiteLLM proxy
claude
# Specify model at startup
claude --model claude-sonnet-4-5-20250929
# Or specify a model if you have multiple configured
claude --model claude-3-5-sonnet-20241022
claude --model claude-3-5-haiku-20241022
# Or specify a different model
claude --model claude-haiku-4-5-20251001
claude --model claude-opus-4-5-20251101
# Or change model during a session
claude
/model claude-sonnet-4-5-20250929
```
Alternatively, set default models with environment variables:
```bash
export ANTHROPIC_DEFAULT_SONNET_MODEL=claude-sonnet-4-5-20250929
export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-haiku-4-5-20251001
export ANTHROPIC_DEFAULT_OPUS_MODEL=claude-opus-4-5-20251101
claude
```
### Using 1M Context Window
Claude Code supports extended context (1 million tokens) using the `[1m]` suffix:
```bash
# Use Sonnet with 1M context (requires quotes in shell)
claude --model 'claude-sonnet-4-5-20250929[1m]'
# Inside a Claude Code session (no quotes needed)
/model claude-sonnet-4-5-20250929[1m]
```
:::warning
**Important:** When using `--model` with `[1m]` in the shell, you must use quotes to prevent the shell from interpreting the brackets.
:::
**How it works:**
- Claude Code strips the `[1m]` suffix before sending to LiteLLM
- Claude Code automatically adds the header `anthropic-beta: context-1m-2025-08-07`
- Your LiteLLM config should **NOT** include `[1m]` in model names
**Verify 1M context is active:**
```bash
/context
# Should show: 21k/1000k tokens (2%)
```
Example conversation:
@ -140,6 +188,7 @@ Common issues and solutions:
**Model not found:**
- Ensure the model name in Claude Code matches exactly with your `config.yaml`
- Use `--model` flag or environment variables to specify the model
- Check LiteLLM logs for detailed error messages
## Using Bedrock/Vertex AI/Azure Foundry Models

301
docs/my-website/docs/tutorials/opencode_integration.md Normal file
View file

@ -0,0 +1,301 @@
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# OpenCode Quickstart
This tutorial shows how to connect OpenCode to your existing LiteLLM instance and switch between models.
:::info
This integration allows you to use any LiteLLM supported model through OpenCode with centralized authentication, usage tracking, and cost controls.
:::
<br />
### Video Walkthrough
<iframe width="840" height="500" src="https://www.loom.com/embed/00791498f1d84e4ba6d7476bd2e1442f" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
## Prerequisites
- LiteLLM already configured and running (e.g., http://localhost:4000)
- LiteLLM API key
## Installation
### Step 1: Install OpenCode
Choose your preferred installation method:
<Tabs>
<TabItem value="curl" label="One-line install (recommended)">
```bash
curl -fsSL https://opencode.ai/install | bash
```
</TabItem>
<TabItem value="npm" label="NPM">
```bash
npm install -g opencode-ai
```
</TabItem>
<TabItem value="homebrew" label="Homebrew">
```bash
brew install sst/tap/opencode
```
</TabItem>
</Tabs>
Verify installation:
```bash
opencode --version
```
### Step 2: Configure LiteLLM Provider
Create your OpenCode configuration file. You can place this in different locations depending on your needs:
**Configuration locations:**
- **Global**: `~/.config/opencode/opencode.json` (applies to all projects)
- **Project**: `opencode.json` in your project root (project-specific settings)
- **Custom**: Set `OPENCODE_CONFIG` environment variable
Create `~/.config/opencode/opencode.json` (global config):
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"litellm": {
"npm": "@ai-sdk/openai-compatible",
"name": "LiteLLM",
"options": {
"baseURL": "http://localhost:4000/v1"
},
"models": {
"gpt-4": {
"name": "GPT-4"
},
"claude-3-5-sonnet-20241022": {
"name": "Claude 3.5 Sonnet"
},
"deepseek-chat": {
"name": "DeepSeek Chat"
}
}
}
}
}
```
:::tip
The keys in the "models" object (e.g., "gpt-4", "claude-3-5-sonnet-20241022") should match the `model_name` values from your LiteLLM configuration. The "name" field provides a friendly display name that will appear as an alias in OpenCode.
:::
### Step 3: Connect to LiteLLM Provider
Launch OpenCode:
```bash
opencode
```
Add your API key:
```bash
/connect
```
Then:
- **Enter provider name**: `LiteLLM` (must match the "name" field in your config)
- **Enter your LiteLLM API key**: Your LiteLLM master key or virtual key
### Step 4: Switch Between Models
In OpenCode, run:
```bash
/models
```
Select any model from your LiteLLM configuration. OpenCode will route all requests through your LiteLLM instance.
## Advanced Configuration
### Model Parameters
You can customize model parameters like context limits:
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"litellm": {
"npm": "@ai-sdk/openai-compatible",
"name": "LiteLLM",
"options": {
"baseURL": "http://localhost:4000/v1"
},
"models": {
"gpt-4": {
"name": "GPT-4",
"limit": {
"context": 128000,
"output": 4096
}
},
"claude-3-5-sonnet-20241022": {
"name": "Claude 3.5 Sonnet",
"limit": {
"context": 200000,
"output": 8192
}
}
}
}
}
}
```
### Multi-Provider Setup
You can configure multiple LiteLLM instances or mix with other providers:
<Tabs>
<TabItem value="multi-litellm" label="Multiple LiteLLM Instances">
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"litellm-prod": {
"npm": "@ai-sdk/openai-compatible",
"name": "LiteLLM Production",
"options": {
"baseURL": "https://your-prod-instance.com/v1"
},
"models": {
"gpt-4": {
"name": "GPT-4 (Production)"
}
}
},
"litellm-dev": {
"npm": "@ai-sdk/openai-compatible",
"name": "LiteLLM Development",
"options": {
"baseURL": "http://localhost:4000/v1"
},
"models": {
"gpt-4": {
"name": "GPT-4 (Development)"
}
}
}
}
}
```
</TabItem>
<TabItem value="mixed-providers" label="Mixed Providers">
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"litellm": {
"npm": "@ai-sdk/openai-compatible",
"name": "LiteLLM",
"options": {
"baseURL": "http://localhost:4000/v1"
},
"models": {
"gpt-4": {
"name": "GPT-4 via LiteLLM"
},
"claude-3-5-sonnet-20241022": {
"name": "Claude 3.5 Sonnet via LiteLLM"
}
}
},
"openai": {
"npm": "@ai-sdk/openai",
"name": "OpenAI Direct",
"models": {
"gpt-4o": {
"name": "GPT-4o (Direct)"
}
}
}
}
}
```
</TabItem>
</Tabs>
## Example LiteLLM Configuration
Here's an example LiteLLM `config.yaml` that works well with OpenCode:
```yaml
model_list:
# OpenAI models
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
# Anthropic models
- model_name: claude-3-5-sonnet-20241022
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
# DeepSeek models
- model_name: deepseek-chat
litellm_params:
model: deepseek/deepseek-chat
api_key: os.environ/DEEPSEEK_API_KEY
```
## Troubleshooting
**OpenCode not connecting:**
- Verify your LiteLLM proxy is running: `curl http://localhost:4000/health`
- Check that the `baseURL` in your OpenCode config matches your LiteLLM instance
- Ensure the provider name in `/connect` matches exactly with your config
**Authentication errors:**
- Verify your LiteLLM API key is correct
- Check that your LiteLLM instance has authentication properly configured
- Ensure your API key has access to the models you're trying to use
**Model not found:**
- Ensure the model names in OpenCode config match your LiteLLM `model_name` values
- Check LiteLLM logs for detailed error messages
- Verify the models are properly configured in your LiteLLM instance
**Configuration not loading:**
- Check the config file path and permissions
- Validate JSON syntax using a JSON validator
- Ensure the `$schema` URL is accessible
## Tips
- Add more models to the config as needed - they'll appear in `/models`
- Use project-specific configs for different codebases with different model requirements
- Monitor your LiteLLM proxy logs to see OpenCode requests in real-time

50
docs/my-website/package-lock.json generated
View file

@ -180,7 +180,6 @@
"resolved": "https://registry.npmjs.org/@algolia/client-search/-/client-search-5.44.0.tgz",
"integrity": "sha512-/FRKUM1G4xn3vV8+9xH1WJ9XknU8rkBGlefruq9jDhYUAvYozKimhrmC2pRqw/RyHhPivmgZCRuC8jHP8piz4Q==",
"license": "MIT",
"peer": true,
"dependencies": {
"@algolia/client-common": "5.44.0",
"@algolia/requester-browser-xhr": "5.44.0",
@ -328,7 +327,6 @@
"resolved": "https://registry.npmjs.org/@babel/core/-/core-7.28.5.tgz",
"integrity": "sha512-e7jT4DxYvIDLk1ZHmU/m/mB19rex9sv0c2ftBtjSBv+kVM/902eh0fINUzD7UwLLNR+jU585GxUJ8/EBfAM5fw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/code-frame": "^7.27.1",
"@babel/generator": "^7.28.5",
@ -2163,7 +2161,6 @@
}
],
"license": "MIT",
"peer": true,
"engines": {
"node": ">=18"
},
@ -2186,7 +2183,6 @@
}
],
"license": "MIT",
"peer": true,
"engines": {
"node": ">=18"
}
@ -2296,7 +2292,6 @@
"resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-7.1.0.tgz",
"integrity": "sha512-8sLjZwK0R+JlxlYcTuVnyT2v+htpdrjDOKuMcOVdYjt52Lh8hWRYpxBPoKx/Zg+bcjc3wx6fmQevMmUztS/ccA==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@ -2718,7 +2713,6 @@
"resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-7.1.0.tgz",
"integrity": "sha512-8sLjZwK0R+JlxlYcTuVnyT2v+htpdrjDOKuMcOVdYjt52Lh8hWRYpxBPoKx/Zg+bcjc3wx6fmQevMmUztS/ccA==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@ -3595,7 +3589,6 @@
"resolved": "https://registry.npmjs.org/@docusaurus/plugin-content-docs/-/plugin-content-docs-3.8.1.tgz",
"integrity": "sha512-oByRkSZzeGNQByCMaX+kif5Nl2vmtj2IHQI2fWjCfCootsdKZDPFLonhIp5s3IGJO7PLUfe0POyw0Xh/RrGXJA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@docusaurus/core": "3.8.1",
"@docusaurus/logger": "3.8.1",
@ -4634,7 +4627,6 @@
"resolved": "https://registry.npmjs.org/@mdx-js/react/-/react-3.1.1.tgz",
"integrity": "sha512-f++rKLQgUVYDAtECQ6fn/is15GkEH9+nZPM3MS0RcxVqoTfawHvDlSCH7JbMhAM6uJ32v3eXLvLmLvjGu7PTQw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/mdx": "^2.0.0"
},
@ -7191,7 +7183,6 @@
"resolved": "https://registry.npmjs.org/@svgr/core/-/core-8.1.0.tgz",
"integrity": "sha512-8QqtOQT5ACVlmsvKOJNEaWmRPmcojMOzCz4Hs2BGG/toAp/K38LcsMRyLp349glq5AzJbCEeimEoxaX6v/fLrA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/core": "^7.21.3",
"@svgr/babel-preset": "8.1.0",
@ -7849,7 +7840,6 @@
"resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.6.tgz",
"integrity": "sha512-p/jUvulfgU7oKtj6Xpk8cA2Y1xKTtICGpJYeJXz2YVO2UcvjQgeRMLDGfDeqeRW2Ta+0QNFwcc8X3GH8SxZz6w==",
"license": "MIT",
"peer": true,
"dependencies": {
"csstype": "^3.2.2"
}
@ -8274,7 +8264,6 @@
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz",
"integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==",
"license": "MIT",
"peer": true,
"bin": {
"acorn": "bin/acorn"
},
@ -8354,7 +8343,6 @@
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.17.1.tgz",
"integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.3",
"fast-uri": "^3.0.1",
@ -8400,7 +8388,6 @@
"resolved": "https://registry.npmjs.org/algoliasearch/-/algoliasearch-5.44.0.tgz",
"integrity": "sha512-f8IpsbdQjzTjr/4mJ/jv5UplrtyMnnciGax6/B0OnLCs2/GJTK13O4Y7Ff1AvJVAaztanH+m5nzPoUq6EAy+aA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@algolia/abtesting": "1.10.0",
"@algolia/client-abtesting": "5.44.0",
@ -9077,7 +9064,6 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"baseline-browser-mapping": "^2.8.25",
"caniuse-lite": "^1.0.30001754",
@ -9413,7 +9399,6 @@
"resolved": "https://registry.npmjs.org/chevrotain/-/chevrotain-11.0.3.tgz",
"integrity": "sha512-ci2iJH6LeIkvP9eJW6gpueU8cnZhv85ELY8w8WiFtNjMHA5ad6pQLaJo9mEly/9qUyCpvqX8/POVUTf18/HFdw==",
"license": "Apache-2.0",
"peer": true,
"dependencies": {
"@chevrotain/cst-dts-gen": "11.0.3",
"@chevrotain/gast": "11.0.3",
@ -10177,7 +10162,6 @@
"resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-7.1.0.tgz",
"integrity": "sha512-8sLjZwK0R+JlxlYcTuVnyT2v+htpdrjDOKuMcOVdYjt52Lh8hWRYpxBPoKx/Zg+bcjc3wx6fmQevMmUztS/ccA==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@ -10497,7 +10481,6 @@
"resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.33.1.tgz",
"integrity": "sha512-iJc4TwyANnOGR1OmWhsS9ayRS3s+XQ185FmuHObThD+5AeJCakAAbWv8KimMTt08xCCLNgneQwFp+JRJOr9qGQ==",
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10"
}
@ -10907,7 +10890,6 @@
"resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
"integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==",
"license": "ISC",
"peer": true,
"engines": {
"node": ">=12"
}
@ -12164,7 +12146,6 @@
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@ -14192,15 +14173,15 @@
}
},
"node_modules/lodash": {
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==",
"version": "4.17.23",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.23.tgz",
"integrity": "sha512-LgVTMpQtIopCi79SJeDiP0TfWi5CNEc/L/aRdTh3yIvmZXTnheWpKjSZhnvMl8iXbC1tFg9gdHHDMLoV7CnG+w==",
"license": "MIT"
},
"node_modules/lodash-es": {
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.21.tgz",
"integrity": "sha512-mKnC+QJ9pWVzv+C4/U3rRsHapFfHvQFoFB92e52xeyGMcX6/OlIl78je1u8vePzYZSkkogMPJ2yjxxsb89cxyw==",
"version": "4.17.23",
"resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.23.tgz",
"integrity": "sha512-kVI48u3PZr38HdYz98UmfPnXl2DXrpdctLrFLCd3kOx1xUkOmpFPx7gCWWM5MPkL/fD8zb+Ph0QzjGFs4+hHWg==",
"license": "MIT"
},
"node_modules/lodash.debounce": {
@ -17044,7 +17025,6 @@
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@ -17665,7 +17645,6 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"nanoid": "^3.3.11",
"picocolors": "^1.1.1",
@ -18569,7 +18548,6 @@
"resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-7.1.0.tgz",
"integrity": "sha512-8sLjZwK0R+JlxlYcTuVnyT2v+htpdrjDOKuMcOVdYjt52Lh8hWRYpxBPoKx/Zg+bcjc3wx6fmQevMmUztS/ccA==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@ -19496,7 +19474,6 @@
"resolved": "https://registry.npmjs.org/react/-/react-19.2.0.tgz",
"integrity": "sha512-tmbWg6W31tQLeB5cdIBOicJDJRR2KzXsV7uSK9iNfLWQ5bIZfxuPEHp7M8wiHyHnn0DD1i7w3Zmin0FtkrwoCQ==",
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@ -19506,7 +19483,6 @@
"resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.0.tgz",
"integrity": "sha512-UlbRu4cAiGaIewkPyiRGJk0imDN2T3JjieT6spoL2UeSf5od4n5LB/mQ4ejmxhCFT1tYe8IvaFulzynWovsEFQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"scheduler": "^0.27.0"
},
@ -19590,7 +19566,6 @@
"resolved": "https://registry.npmjs.org/@docusaurus/react-loadable/-/react-loadable-6.0.0.tgz",
"integrity": "sha512-YMMxTUQV/QFSnbgrP3tjDzLHRg7vsbMn8e9HAa8o/1iXoiomo48b7sk/kkmWEuWNDPJVlKSJRB6Y2fHqdJk+SQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/react": "*"
},
@ -19692,7 +19667,6 @@
"resolved": "https://registry.npmjs.org/react-router/-/react-router-5.3.4.tgz",
"integrity": "sha512-Ys9K+ppnJah3QuaRiLxk+jDWOR1MekYQrlytiXxC1RyfbdsZkS5pvKAzCCr031xHixZwpnsYNT5xysdFHQaYsA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/runtime": "^7.12.13",
"history": "^4.9.0",
@ -20481,13 +20455,6 @@
"url": "https://opencollective.com/webpack"
}
},
"node_modules/search-insights": {
"version": "2.17.3",
"resolved": "https://registry.npmjs.org/search-insights/-/search-insights-2.17.3.tgz",
"integrity": "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ==",
"license": "MIT",
"peer": true
},
"node_modules/section-matter": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/section-matter/-/section-matter-1.0.0.tgz",
@ -21711,8 +21678,7 @@
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"license": "0BSD",
"peer": true
"license": "0BSD"
},
"node_modules/tunnel-agent": {
"version": "0.6.0",
@ -22099,7 +22065,6 @@
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@ -22451,7 +22416,6 @@
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.103.0.tgz",
"integrity": "sha512-HU1JOuV1OavsZ+mfigY0j8d1TgQgbZ6M+J75zDkpEAwYeXjWSqrGJtgnPblJjd/mAyTNQ7ygw0MiKOn6etz8yw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/eslint-scope": "^3.7.7",
"@types/estree": "^1.0.8",

3
docs/my-website/package.json
View file

@ -62,6 +62,7 @@
"gray-matter": "4.0.3",
"glob": ">=11.1.0",
"node-forge": ">=1.3.2",
"mdast-util-to-hast": ">=13.2.1"
"mdast-util-to-hast": ">=13.2.1",
"lodash-es": ">=4.17.23"
}
}

13
docs/my-website/sidebars.js
View file

@ -42,6 +42,7 @@ const sidebars = {
label: "Guardrails",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/guardrail_policies",
"proxy/guardrails/guardrail_load_balancing",
{
type: "category",
@ -129,6 +130,7 @@ const sidebars = {
"tutorials/claude_code_plugin_marketplace",
]
},
"tutorials/opencode_integration",
"tutorials/cost_tracking_coding",
"tutorials/cursor_integration",
"tutorials/github_copilot_integration",
@ -517,7 +519,14 @@ const sidebars = {
"mcp_troubleshoot",
]
},
"anthropic_unified",
{
type: "category",
label: "/v1/messages",
items: [
"anthropic_unified/index",
"anthropic_unified/structured_output",
]
},
"anthropic_count_tokens",
"moderation",
"ocr",
@ -563,6 +572,7 @@ const sidebars = {
"search/perplexity",
"search/tavily",
"search/exa_ai",
"search/brave",
"search/parallel_ai",
"search/google_pse",
"search/dataforseo",
@ -719,6 +729,7 @@ const sidebars = {
"providers/galadriel",
"providers/github",
"providers/github_copilot",
"providers/gmi",
"providers/chatgpt",
"providers/gradient_ai",
"providers/groq",

7
litellm/__init__.py
View file

@ -377,6 +377,9 @@ priority_reservation: Optional[
Dict[str, Union[float, "PriorityReservationDict"]]
] = None
# priority_reservation_settings is lazy-loaded via __getattr__
# Only declare for type checking - at runtime __getattr__ handles it
if TYPE_CHECKING:
priority_reservation_settings: Optional["PriorityReservationSettings"] = None
######## Networking Settings ########
@ -392,6 +395,9 @@ force_ipv4: bool = (
False # when True, litellm will force ipv4 for all LLM requests. Some users have seen httpx ConnectionError when using ipv6.
)
####### STOP SEQUENCE LIMIT #######
disable_stop_sequence_limit: bool = False # when True, stop sequence limit is disabled
#### RETRIES ####
num_retries: Optional[int] = None # per model endpoint
max_fallbacks: Optional[int] = None
@ -1270,6 +1276,7 @@ def set_global_gitlab_config(config: Dict[str, Any]) -> None:
if TYPE_CHECKING:
from litellm.types.utils import ModelInfo as _ModelInfoType
from litellm.types.utils import PriorityReservationSettings
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.caching.caching import Cache

3
litellm/batches/main.py
View file

@ -404,6 +404,7 @@ def _handle_retrieve_batch_providers_without_provider_config(
_retrieve_batch_request: RetrieveBatchRequest,
_is_async: bool,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm", "anthropic"] = "openai",
logging_obj: Optional[Any] = None,
):
api_base: Optional[str] = None
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
@ -499,6 +500,7 @@ def _handle_retrieve_batch_providers_without_provider_config(
vertex_credentials=vertex_credentials,
timeout=timeout,
max_retries=optional_params.max_retries,
logging_obj=logging_obj,
)
elif custom_llm_provider == "anthropic":
api_base = (
@ -662,6 +664,7 @@ def retrieve_batch(
_retrieve_batch_request=_retrieve_batch_request,
_is_async=_is_async,
timeout=timeout,
logging_obj=litellm_logging_obj,
)
except Exception as e:

16
litellm/constants.py
View file

@ -1067,7 +1067,7 @@ known_tokenizer_config = {
}
OPENAI_FINISH_REASONS = ["stop", "length", "function_call", "content_filter", "null"]
OPENAI_FINISH_REASONS = ["stop", "length", "function_call", "content_filter", "null", "finish_reason_unspecified", "malformed_function_call", "guardrail_intervened", "eos"]
HUMANLOOP_PROMPT_CACHE_TTL_SECONDS = int(
os.getenv("HUMANLOOP_PROMPT_CACHE_TTL_SECONDS", 60)
) # 1 minute
@ -1122,6 +1122,20 @@ BEDROCK_AGENT_RUNTIME_PASS_THROUGH_ROUTES = [
"generateQuery/",
"optimize-prompt/",
]
# Headers that are safe to forward from incoming requests to Vertex AI
# Using an allowlist approach for security - only forward headers we explicitly trust
ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS = {
"anthropic-beta", # Required for Anthropic features like extended context windows
"content-type", # Required for request body parsing
}
# Prefix for headers that should be forwarded to the provider with the prefix stripped
# e.g., 'x-pass-anthropic-beta: value' becomes 'anthropic-beta: value'
# Works for all LLM pass-through endpoints (Vertex AI, Anthropic, Bedrock, etc.)
PASS_THROUGH_HEADER_PREFIX = "x-pass-"
BASE_MCP_ROUTE = "/mcp"
BATCH_STATUS_POLL_INTERVAL_SECONDS = int(

8
litellm/exceptions.py
View file

@ -142,11 +142,13 @@ class BadRequestError(openai.BadRequestError): # type: ignore
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
# Use response if it's a valid httpx.Response with a request, otherwise use minimal error response
# Note: We check _request (not .request property) to avoid RuntimeError when _request is None
if (
response is not None
and isinstance(response, httpx.Response)
and hasattr(response, "request")
and response.request is not None
and hasattr(response, "_request")
and getattr(response, "_request", None) is not None
):
self.response = response
else:
@ -467,6 +469,7 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
provider_specific_fields: Optional[dict] = None,
body: Optional[dict] = None,
):
self.status_code = 400
self.message = "litellm.ContentPolicyViolationError: {}".format(message)
@ -480,6 +483,7 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
llm_provider=self.llm_provider, # type: ignore
response=response,
litellm_debug_info=self.litellm_debug_info,
body=body,
) # Call the base class constructor with the parameters it needs
def __str__(self):

3
litellm/google_genai/adapters/transformation.py
View file

@ -2,7 +2,6 @@ import json
from typing import Any, AsyncIterator, Dict, Iterator, List, Optional, Union, cast
from litellm import verbose_logger
from litellm.litellm_core_utils.json_validation_rule import normalize_tool_schema
from litellm.types.llms.openai import (
AllMessageValues,
@ -771,6 +770,8 @@ class GoogleGenAIAdapter:
"content_filter": "SAFETY",
"tool_calls": "STOP",
"function_call": "STOP",
"finish_reason_unspecified": "FINISH_REASON_UNSPECIFIED",
"malformed_function_call": "MALFORMED_FUNCTION_CALL",
}
return mapping.get(finish_reason, "STOP")

50
litellm/integrations/opentelemetry.py
View file

@ -1829,12 +1829,6 @@ class OpenTelemetry(CustomLogger):
return None, None
def _get_span_processor(self, dynamic_headers: Optional[dict] = None):
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter as OTLPSpanExporterGRPC,
)
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter as OTLPSpanExporterHTTP,
)
from opentelemetry.sdk.trace.export import (
BatchSpanProcessor,
ConsoleSpanExporter,
@ -1872,6 +1866,16 @@ class OpenTelemetry(CustomLogger):
or self.OTEL_EXPORTER == "http/protobuf"
or self.OTEL_EXPORTER == "http/json"
):
try:
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter as OTLPSpanExporterHTTP,
)
except ImportError as exc:
raise ImportError(
"OpenTelemetry OTLP HTTP exporter is not available. Install "
"`opentelemetry-exporter-otlp` to enable OTLP HTTP."
) from exc
verbose_logger.debug(
"OpenTelemetry: intiializing http exporter. Value of OTEL_EXPORTER: %s",
self.OTEL_EXPORTER,
@ -1885,6 +1889,16 @@ class OpenTelemetry(CustomLogger):
),
)
elif self.OTEL_EXPORTER == "otlp_grpc" or self.OTEL_EXPORTER == "grpc":
try:
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter as OTLPSpanExporterGRPC,
)
except ImportError as exc:
raise ImportError(
"OpenTelemetry OTLP gRPC exporter is not available. Install "
"`opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`)."
) from exc
verbose_logger.debug(
"OpenTelemetry: intiializing grpc exporter. Value of OTEL_EXPORTER: %s",
self.OTEL_EXPORTER,
@ -1961,9 +1975,15 @@ class OpenTelemetry(CustomLogger):
endpoint=normalized_endpoint, headers=_split_otel_headers
)
elif self.OTEL_EXPORTER == "otlp_grpc" or self.OTEL_EXPORTER == "grpc":
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
OTLPLogExporter,
)
try:
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
OTLPLogExporter,
)
except ImportError as exc:
raise ImportError(
"OpenTelemetry OTLP gRPC log exporter is not available. Install "
"`opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`)."
) from exc
verbose_logger.debug(
"OpenTelemetry: Using gRPC log exporter. Value of OTEL_EXPORTER: %s, endpoint: %s",
@ -2026,9 +2046,15 @@ class OpenTelemetry(CustomLogger):
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
elif self.OTEL_EXPORTER == "otlp_grpc" or self.OTEL_EXPORTER == "grpc":
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
OTLPMetricExporter,
)
try:
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
OTLPMetricExporter,
)
except ImportError as exc:
raise ImportError(
"OpenTelemetry OTLP gRPC metric exporter is not available. Install "
"`opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`)."
) from exc
exporter = OTLPMetricExporter(
endpoint=normalized_endpoint,

45
litellm/integrations/prometheus.py
View file

@ -409,6 +409,19 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric("litellm_cached_tokens_metric"),
)
# User and Team count metrics
self.litellm_total_users_metric = self._gauge_factory(
"litellm_total_users",
"Total number of users in LiteLLM",
labelnames=[],
)
self.litellm_teams_count_metric = self._gauge_factory(
"litellm_teams_count",
"Total number of teams in LiteLLM",
labelnames=[],
)
except Exception as e:
print_verbose(f"Got exception on init prometheus client {str(e)}")
raise e
@ -2344,6 +2357,38 @@ class PrometheusLogger(CustomLogger):
await self._initialize_team_budget_metrics()
await self._initialize_api_key_budget_metrics()
await self._initialize_user_budget_metrics()
await self._initialize_user_and_team_count_metrics()
async def _initialize_user_and_team_count_metrics(self):
"""
Initialize user and team count metrics by querying the database.
Updates:
- litellm_total_users: Total count of users in the database
- litellm_teams_count: Total count of teams in the database
"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
verbose_logger.debug(
"Prometheus: skipping user/team count metrics initialization, DB not initialized"
)
return
try:
# Get total user count
total_users = await prisma_client.db.litellm_usertable.count()
self.litellm_total_users_metric.set(total_users)
verbose_logger.debug(f"Prometheus: set litellm_total_users to {total_users}")
# Get total team count
total_teams = await prisma_client.db.litellm_teamtable.count()
self.litellm_teams_count_metric.set(total_teams)
verbose_logger.debug(f"Prometheus: set litellm_teams_count to {total_teams}")
except Exception as e:
verbose_logger.exception(
f"Error initializing user/team count metrics: {str(e)}"
)
async def _set_key_list_budget_metrics(
self, keys: List[Union[str, UserAPIKeyAuth]]

9
litellm/integrations/websearch_interception/handler.py
View file

@ -413,6 +413,13 @@ class WebSearchInterceptionLogger(CustomLogger):
if k != 'max_tokens'
}
# Remove internal websearch interception flags from kwargs before follow-up request
# These flags are used internally and should not be passed to the LLM provider
kwargs_for_followup = {
k: v for k, v in kwargs.items()
if not k.startswith('_websearch_interception')
}
# Get model from logging_obj.model_call_details["agentic_loop_params"]
# This preserves the full model name with provider prefix (e.g., "bedrock/invoke/...")
full_model_name = model
@ -428,7 +435,7 @@ class WebSearchInterceptionLogger(CustomLogger):
messages=follow_up_messages,
model=full_model_name,
**optional_params_without_max_tokens,
**kwargs,
**kwargs_for_followup,
)
verbose_logger.debug(
f"WebSearchInterception: Follow-up request completed, response type: {type(final_response)}"

6
litellm/litellm_core_utils/core_helpers.py
View file

@ -79,9 +79,11 @@ def map_finish_reason(
elif finish_reason == "eos_token" or finish_reason == "stop_sequence":
return "stop"
elif (
finish_reason == "FINISH_REASON_UNSPECIFIED" or finish_reason == "STOP"
finish_reason == "FINISH_REASON_UNSPECIFIED"
): # vertex ai - got from running `print(dir(response_obj.candidates[0].finish_reason))`: ['FINISH_REASON_UNSPECIFIED', 'MAX_TOKENS', 'OTHER', 'RECITATION', 'SAFETY', 'STOP',]
return "stop"
return "finish_reason_unspecified"
elif finish_reason == "MALFORMED_FUNCTION_CALL":
return "malformed_function_call"
elif finish_reason == "SAFETY" or finish_reason == "RECITATION": # vertex ai
return "content_filter"
elif finish_reason == "STOP": # vertex ai

9
litellm/litellm_core_utils/default_encoding.py
View file

@ -15,6 +15,13 @@ except (ImportError, AttributeError):
__name__, "litellm_core_utils/tokenizers"
)
# Check if the directory is writable. If not, use /tmp as a fallback.
# This is especially important for non-root Docker environments where the package directory is read-only.
is_non_root = os.getenv("LITELLM_NON_ROOT", "").lower() == "true"
if not os.access(filename, os.W_OK) and is_non_root:
filename = "/tmp/tiktoken_cache"
os.makedirs(filename, exist_ok=True)
os.environ["TIKTOKEN_CACHE_DIR"] = os.getenv(
"CUSTOM_TIKTOKEN_CACHE_DIR", filename
) # use local copy of tiktoken b/c of - https://github.com/BerriAI/litellm/issues/1071
@ -36,5 +43,5 @@ for attempt in range(_max_retries):
# Last attempt, re-raise the exception
raise
# Exponential backoff with jitter to reduce collision probability
delay = _retry_delay * (2 ** attempt) + random.uniform(0, 0.1)
delay = _retry_delay * (2**attempt) + random.uniform(0, 0.1)
time.sleep(delay)

26
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -142,7 +142,14 @@ def get_error_message(error_obj) -> Optional[str]:
if hasattr(error_obj, "body"):
_error_obj_body = getattr(error_obj, "body")
if isinstance(_error_obj_body, dict):
return _error_obj_body.get("message")
# OpenAI-style: {"message": "...", "type": "...", ...}
if _error_obj_body.get("message"):
return _error_obj_body.get("message")
# Azure-style: {"error": {"message": "...", ...}}
nested_error = _error_obj_body.get("error")
if isinstance(nested_error, dict):
return nested_error.get("message")
# If all else fails, return None
return None
@ -2044,6 +2051,20 @@ def exception_type( # type: ignore # noqa: PLR0915
else:
message = str(original_exception)
# Azure OpenAI (especially Images) often nests error details under
# body["error"]. Detect content policy violations using the structured
# payload in addition to string matching.
azure_error_code: Optional[str] = None
try:
body_dict = getattr(original_exception, "body", None) or {}
if isinstance(body_dict, dict):
if isinstance(body_dict.get("error"), dict):
azure_error_code = body_dict["error"].get("code") # type: ignore[index]
else:
azure_error_code = body_dict.get("code")
except Exception:
azure_error_code = None
if "Internal server error" in error_str:
exception_mapping_worked = True
raise litellm.InternalServerError(
@ -2072,7 +2093,8 @@ def exception_type( # type: ignore # noqa: PLR0915
response=getattr(original_exception, "response", None),
)
elif (
ExceptionCheckers.is_azure_content_policy_violation_error(error_str)
azure_error_code == "content_policy_violation"
or ExceptionCheckers.is_azure_content_policy_violation_error(error_str)
):
exception_mapping_worked = True
from litellm.llms.azure.exception_mapping import (

29
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -1462,7 +1462,7 @@ def convert_to_gemini_tool_call_invoke(
)
def convert_to_gemini_tool_call_result(
def convert_to_gemini_tool_call_result( # noqa: PLR0915
message: Union[ChatCompletionToolMessage, ChatCompletionFunctionMessage],
last_message_with_tool_calls: Optional[dict],
) -> Union[VertexPartType, List[VertexPartType]]:
@ -1529,6 +1529,33 @@ def convert_to_gemini_tool_call_result(
verbose_logger.warning(
f"Failed to process image in tool response: {e}"
)
elif content_type in ("file", "input_file"):
# Extract file for inline_data (for tool results with PDF, audio, video, etc.)
file_data = content.get("file_data", "")
if not file_data:
file_content = content.get("file", {})
file_data = (
file_content.get("file_data", "")
if isinstance(file_content, dict)
else file_content
if isinstance(file_content, str)
else ""
)
if file_data:
# Convert file to base64 blob format for Gemini
try:
file_obj = convert_to_anthropic_image_obj(
file_data, format=None
)
inline_data = BlobType(
data=file_obj["data"],
mime_type=file_obj["media_type"],
)
except Exception as e:
verbose_logger.warning(
f"Failed to process file in tool response: {e}"
)
name: Optional[str] = message.get("name", "") # type: ignore
# Recover name from last message with tool calls

99
litellm/litellm_core_utils/streaming_handler.py
View file

@ -1571,6 +1571,90 @@ class CustomStreamWrapper:
)
return chunk
def _add_mcp_list_tools_to_first_chunk(self, chunk: ModelResponseStream) -> ModelResponseStream:
"""
Add mcp_list_tools from _hidden_params to the first chunk's delta.provider_specific_fields.
This method checks if MCP metadata with mcp_list_tools is stored in _hidden_params
and adds it to the first chunk's delta.provider_specific_fields.
"""
try:
# Check if MCP metadata should be added to first chunk
if not hasattr(self, "_hidden_params") or not self._hidden_params:
return chunk
mcp_metadata = self._hidden_params.get("mcp_metadata")
if not mcp_metadata or not isinstance(mcp_metadata, dict):
return chunk
# Only add mcp_list_tools to first chunk (not tool_calls or tool_results)
mcp_list_tools = mcp_metadata.get("mcp_list_tools")
if not mcp_list_tools:
return chunk
# Add mcp_list_tools to delta.provider_specific_fields
if hasattr(chunk, "choices") and chunk.choices:
for choice in chunk.choices:
if isinstance(choice, StreamingChoices) and hasattr(choice, "delta") and choice.delta:
# Get existing provider_specific_fields or create new dict
provider_fields = (
getattr(choice.delta, "provider_specific_fields", None) or {}
)
# Add only mcp_list_tools to first chunk
provider_fields["mcp_list_tools"] = mcp_list_tools
# Set the provider_specific_fields
setattr(choice.delta, "provider_specific_fields", provider_fields)
except Exception as e:
from litellm._logging import verbose_logger
verbose_logger.exception(
f"Error adding MCP list tools to first chunk: {str(e)}"
)
return chunk
def _add_mcp_metadata_to_final_chunk(self, chunk: ModelResponseStream) -> ModelResponseStream:
"""
Add MCP metadata from _hidden_params to the final chunk's delta.provider_specific_fields.
This method checks if MCP metadata is stored in _hidden_params and adds it to
the chunk's delta.provider_specific_fields, similar to how RAG adds search results.
"""
try:
# Check if MCP metadata should be added to final chunk
if not hasattr(self, "_hidden_params") or not self._hidden_params:
return chunk
mcp_metadata = self._hidden_params.get("mcp_metadata")
if not mcp_metadata:
return chunk
# Add MCP metadata to delta.provider_specific_fields
if hasattr(chunk, "choices") and chunk.choices:
for choice in chunk.choices:
if isinstance(choice, StreamingChoices) and hasattr(choice, "delta") and choice.delta:
# Get existing provider_specific_fields or create new dict
provider_fields = (
getattr(choice.delta, "provider_specific_fields", None) or {}
)
# Add MCP metadata
if isinstance(mcp_metadata, dict):
provider_fields.update(mcp_metadata)
# Set the provider_specific_fields
setattr(choice.delta, "provider_specific_fields", provider_fields)
except Exception as e:
from litellm._logging import verbose_logger
verbose_logger.exception(
f"Error adding MCP metadata to final chunk: {str(e)}"
)
return chunk
def cache_streaming_response(self, processed_chunk, cache_hit: bool):
"""
Caches the streaming response
@ -1687,6 +1771,12 @@ class CustomStreamWrapper:
)
# HANDLE STREAM OPTIONS
self.chunks.append(response)
# Add mcp_list_tools to first chunk if present
if not self.sent_first_chunk:
response = self._add_mcp_list_tools_to_first_chunk(response)
self.sent_first_chunk = True
if hasattr(
response, "usage"
): # remove usage from chunk, only send on final chunk
@ -1712,6 +1802,8 @@ class CustomStreamWrapper:
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
response._hidden_params["usage"] = usage
# Add MCP metadata to final chunk if present
response = self._add_mcp_metadata_to_final_chunk(response)
# RETURN RESULT
return response
@ -1852,6 +1944,11 @@ class CustomStreamWrapper:
input=self.response_uptil_now, model=self.model
)
self.chunks.append(processed_chunk)
# Add mcp_list_tools to first chunk if present
if not self.sent_first_chunk:
processed_chunk = self._add_mcp_list_tools_to_first_chunk(processed_chunk)
self.sent_first_chunk = True
if hasattr(
processed_chunk, "usage"
): # remove usage from chunk, only send on final chunk
@ -1884,6 +1981,8 @@ class CustomStreamWrapper:
processed_chunk
)
)
# Add MCP metadata to final chunk if present (after hooks)
processed_chunk = self._add_mcp_metadata_to_final_chunk(processed_chunk)
return processed_chunk
raise StopAsyncIteration

8
litellm/llms/anthropic/experimental_pass_through/adapters/handler.py
View file

@ -45,6 +45,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
output_format: Optional[Dict] = None,
extra_kwargs: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Prepare kwargs for litellm.completion/acompletion"""
@ -76,6 +77,8 @@ class LiteLLMMessagesToCompletionTransformationHandler:
request_data["top_k"] = top_k
if top_p is not None:
request_data["top_p"] = top_p
if output_format:
request_data["output_format"] = output_format
openai_request = ANTHROPIC_ADAPTER.translate_completion_input_params(
request_data
@ -130,6 +133,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
output_format: Optional[Dict] = None,
**kwargs,
) -> Union[AnthropicMessagesResponse, AsyncIterator]:
"""Handle non-Anthropic models asynchronously using the adapter"""
@ -148,6 +152,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools=tools,
top_k=top_k,
top_p=top_p,
output_format=output_format,
extra_kwargs=kwargs,
)
)
@ -189,6 +194,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
output_format: Optional[Dict] = None,
_is_async: bool = False,
**kwargs,
) -> Union[
@ -212,6 +218,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools=tools,
top_k=top_k,
top_p=top_p,
output_format=output_format,
**kwargs,
)
@ -230,6 +237,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
tools=tools,
top_k=top_k,
top_p=top_p,
output_format=output_format,
extra_kwargs=kwargs,
)
)

48
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -172,7 +172,7 @@ class LiteLLMAnthropicMessagesAdapter:
"""
Which anthropic params, we need to translate to the openai format.
"""
return ["messages", "metadata", "system", "tool_choice", "tools", "thinking"]
return ["messages", "metadata", "system", "tool_choice", "tools", "thinking", "output_format"]
def translate_anthropic_messages_to_openai( # noqa: PLR0915
self,
@ -554,6 +554,42 @@ class LiteLLMAnthropicMessagesAdapter:
return new_tools
def translate_anthropic_output_format_to_openai(
self, output_format: Any
) -> Optional[Dict[str, Any]]:
"""
Translate Anthropic's output_format to OpenAI's response_format.
Anthropic output_format: {"type": "json_schema", "schema": {...}}
OpenAI response_format: {"type": "json_schema", "json_schema": {"name": "...", "schema": {...}}}
Args:
output_format: Anthropic output_format dict with 'type' and 'schema'
Returns:
OpenAI-compatible response_format dict, or None if invalid
"""
if not isinstance(output_format, dict):
return None
output_type = output_format.get("type")
if output_type != "json_schema":
return None
schema = output_format.get("schema")
if not schema:
return None
# Convert to OpenAI response_format structure
return {
"type": "json_schema",
"json_schema": {
"name": "structured_output",
"schema": schema,
"strict": True,
},
}
def translate_anthropic_to_openai(
self, anthropic_message_request: AnthropicMessagesRequest
) -> ChatCompletionRequest:
@ -636,6 +672,16 @@ class LiteLLMAnthropicMessagesAdapter:
if reasoning_effort:
new_kwargs["reasoning_effort"] = reasoning_effort
## CONVERT OUTPUT_FORMAT to RESPONSE_FORMAT
if "output_format" in anthropic_message_request:
output_format = anthropic_message_request["output_format"]
if output_format:
response_format = self.translate_anthropic_output_format_to_openai(
output_format=output_format
)
if response_format:
new_kwargs["response_format"] = response_format
translatable_params = self.translatable_anthropic_params()
for k, v in anthropic_message_request.items():
if k not in translatable_params: # pass remaining params as is

51
litellm/llms/anthropic/experimental_pass_through/architecture.md Normal file
View file

@ -0,0 +1,51 @@
# Anthropic Messages Pass-Through Architecture
## Request Flow
```mermaid
flowchart TD
A[litellm.anthropic.messages.acreate] --> B{Provider?}
B -->|anthropic| C[AnthropicMessagesConfig]
B -->|azure_ai| D[AzureAnthropicMessagesConfig]
B -->|bedrock invoke| E[BedrockAnthropicMessagesConfig]
B -->|vertex_ai| F[VertexAnthropicMessagesConfig]
B -->|Other providers| G[LiteLLMAnthropicMessagesAdapter]
C --> H[Direct Anthropic API]
D --> I[Azure AI Foundry API]
E --> J[Bedrock Invoke API]
F --> K[Vertex AI API]
G --> L[translate_anthropic_to_openai]
L --> M[litellm.completion]
M --> N[Provider API]
N --> O[translate_openai_response_to_anthropic]
O --> P[Anthropic Response Format]
H --> P
I --> P
J --> P
K --> P
```
## Adapter Flow (Non-Native Providers)
```mermaid
sequenceDiagram
participant User
participant Handler as anthropic_messages_handler
participant Adapter as LiteLLMAnthropicMessagesAdapter
participant LiteLLM as litellm.completion
participant Provider as Provider API
User->>Handler: Anthropic Messages Request
Handler->>Adapter: translate_anthropic_to_openai()
Note over Adapter: messages, tools, thinking,<br/>output_format → response_format
Adapter->>LiteLLM: OpenAI Format Request
LiteLLM->>Provider: Provider-specific Request
Provider->>LiteLLM: Provider Response
LiteLLM->>Adapter: OpenAI Format Response
Adapter->>Handler: translate_openai_response_to_anthropic()
Handler->>User: Anthropic Messages Response
```

22
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -42,6 +42,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
"tool_choice",
"thinking",
"context_management",
"output_format",
# TODO: Add Anthropic `metadata` support
# "metadata",
]
@ -169,27 +170,32 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
) -> dict:
"""
Auto-inject anthropic-beta headers based on features used.
Handles:
- context_management: adds 'context-management-2025-06-27'
- tool_search: adds provider-specific tool search header
- output_format: adds 'structured-outputs-2025-11-13'
Args:
headers: Request headers dict
optional_params: Optional parameters including tools, context_management
optional_params: Optional parameters including tools, context_management, output_format
custom_llm_provider: Provider name for looking up correct tool search header
"""
beta_values: set = set()
# Get existing beta headers if any
existing_beta = headers.get("anthropic-beta")
if existing_beta:
beta_values.update(b.strip() for b in existing_beta.split(","))
# Check for context management
if optional_params.get("context_management") is not None:
beta_values.add(ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value)
# Check for structured outputs
if optional_params.get("output_format") is not None:
beta_values.add(ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value)
# Check for tool search tools
tools = optional_params.get("tools")
if tools:
@ -198,8 +204,8 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
# Use provider-specific tool search header
tool_search_header = get_tool_search_beta_header(custom_llm_provider)
beta_values.add(tool_search_header)
if beta_values:
headers["anthropic-beta"] = ",".join(sorted(beta_values))
return headers

75
litellm/llms/azure/exception_mapping.py
View file

@ -1,4 +1,4 @@
from typing import Optional
from typing import Any, Dict, Optional, Tuple
from litellm.exceptions import ContentPolicyViolationError
@ -18,27 +18,76 @@ class AzureOpenAIExceptionMapping:
"""
Create a content policy violation error
"""
azure_error, inner_error = AzureOpenAIExceptionMapping._extract_azure_error(
original_exception
)
# Prefer the provider message/type/code when present.
provider_message = (
azure_error.get("message")
if isinstance(azure_error, dict)
else None
) or message
provider_type = (
azure_error.get("type") if isinstance(azure_error, dict) else None
)
provider_code = (
azure_error.get("code") if isinstance(azure_error, dict) else None
)
# Keep the OpenAI-style body fields populated so downstream (proxy + SDK)
# can surface `type` / `code` correctly.
openai_style_body: Dict[str, Any] = {
"message": provider_message,
"type": provider_type or "invalid_request_error",
"code": provider_code or "content_policy_violation",
"param": None,
}
raise ContentPolicyViolationError(
message=f"AzureException - {message}",
message=provider_message,
llm_provider="azure",
model=model,
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
provider_specific_fields={
"innererror": AzureOpenAIExceptionMapping._get_innererror_from_exception(
original_exception
)
# Preserve legacy key for backward compatibility.
"innererror": inner_error,
# Prefer Azure's current naming.
"inner_error": inner_error,
# Include the full Azure error object for clients that want it.
"azure_error": azure_error or None,
},
body=openai_style_body,
)
@staticmethod
def _get_innererror_from_exception(original_exception: Exception) -> Optional[dict]:
def _extract_azure_error(
original_exception: Exception,
) -> Tuple[Dict[str, Any], Optional[dict]]:
"""Extract Azure OpenAI error payload and inner error details.
Azure error formats can vary by endpoint/version. Common shapes:
- {"innererror": {...}} (legacy)
- {"error": {"code": "...", "message": "...", "type": "...", "inner_error": {...}}}
- {"code": "...", "message": "...", "type": "..."} (already flattened)
"""
Azure OpenAI returns the innererror in the body of the exception
This method extracts the innererror from the exception
"""
innererror = None
body_dict = getattr(original_exception, "body", None) or {}
if isinstance(body_dict, dict):
innererror = body_dict.get("innererror")
return innererror
if not isinstance(body_dict, dict):
return {}, None
# Some SDKs place the payload under "error".
azure_error: Dict[str, Any]
if isinstance(body_dict.get("error"), dict):
azure_error = body_dict.get("error", {}) # type: ignore[assignment]
else:
azure_error = body_dict
inner_error = (
azure_error.get("inner_error")
or azure_error.get("innererror")
or body_dict.get("innererror")
or body_dict.get("inner_error")
)
return azure_error, inner_error

67
litellm/llms/bedrock/base_aws_llm.py
View file

@ -74,40 +74,20 @@ class BaseAWSLLM:
"aws_external_id",
]
def _get_ssl_verify(self):
def _get_ssl_verify(self, ssl_verify: Optional[Union[bool, str]] = None):
"""
Get SSL verification setting for boto3 clients.
This ensures that custom CA certificates are properly used for all AWS API calls,
including STS and Bedrock services.
Returns:
Union[bool, str]: SSL verification setting - False to disable, True to enable,
or a string path to a CA bundle file
"""
import litellm
from litellm.secret_managers.main import str_to_bool
from litellm.llms.custom_httpx.http_handler import get_ssl_verify
# Check environment variable first (highest priority)
ssl_verify = os.getenv("SSL_VERIFY", litellm.ssl_verify)
# Convert string "False"/"True" to boolean
if isinstance(ssl_verify, str):
# Check if it's a file path
if os.path.exists(ssl_verify):
return ssl_verify
# Otherwise try to convert to boolean
ssl_verify_bool = str_to_bool(ssl_verify)
if ssl_verify_bool is not None:
ssl_verify = ssl_verify_bool
# Check SSL_CERT_FILE environment variable for custom CA bundle
if ssl_verify is True or ssl_verify == "True":
ssl_cert_file = os.getenv("SSL_CERT_FILE")
if ssl_cert_file and os.path.exists(ssl_cert_file):
return ssl_cert_file
return ssl_verify
return get_ssl_verify(ssl_verify=ssl_verify)
def get_cache_key(self, credential_args: Dict[str, Optional[str]]) -> str:
"""
@ -130,6 +110,7 @@ class BaseAWSLLM:
aws_web_identity_token: Optional[str] = None,
aws_sts_endpoint: Optional[str] = None,
aws_external_id: Optional[str] = None,
ssl_verify: Optional[Union[bool, str]] = None,
):
"""
Return a boto3.Credentials object
@ -198,7 +179,11 @@ class BaseAWSLLM:
)
# create cache key for non-expiring auth flows
args = {k: v for k, v in locals().items() if k.startswith("aws_")}
args = {
k: v
for k, v in locals().items()
if k.startswith("aws_") or k == "ssl_verify"
}
cache_key = self.get_cache_key(args)
_cached_credentials = self.iam_cache.get_cache(cache_key)
@ -262,6 +247,7 @@ class BaseAWSLLM:
aws_role_name=aws_role_name,
aws_session_name=aws_session_name,
aws_external_id=aws_external_id,
ssl_verify=ssl_verify,
)
elif aws_profile_name is not None: ### CHECK SESSION ###
@ -576,6 +562,7 @@ class BaseAWSLLM:
aws_region_name: Optional[str],
aws_sts_endpoint: Optional[str],
aws_external_id: Optional[str] = None,
ssl_verify: Optional[Union[bool, str]] = None,
) -> Tuple[Credentials, Optional[int]]:
"""
Authenticate with AWS Web Identity Token
@ -604,7 +591,7 @@ class BaseAWSLLM:
"sts",
region_name=aws_region_name,
endpoint_url=sts_endpoint,
verify=self._get_ssl_verify(),
verify=self._get_ssl_verify(ssl_verify),
)
# https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html
@ -649,6 +636,7 @@ class BaseAWSLLM:
region: str,
web_identity_token_file: str,
aws_external_id: Optional[str] = None,
ssl_verify: Optional[Union[bool, str]] = None,
) -> dict:
"""Handle cross-account role assumption for IRSA."""
import boto3
@ -661,7 +649,9 @@ class BaseAWSLLM:
# Create an STS client without credentials
with tracer.trace("boto3.client(sts) for manual IRSA"):
sts_client = boto3.client("sts", region_name=region, verify=self._get_ssl_verify())
sts_client = boto3.client(
"sts", region_name=region, verify=self._get_ssl_verify(ssl_verify)
)
# Manually assume the IRSA role with the session name
verbose_logger.debug(
@ -684,7 +674,7 @@ class BaseAWSLLM:
aws_access_key_id=irsa_creds["AccessKeyId"],
aws_secret_access_key=irsa_creds["SecretAccessKey"],
aws_session_token=irsa_creds["SessionToken"],
verify=self._get_ssl_verify(),
verify=self._get_ssl_verify(ssl_verify),
)
# Get current caller identity for debugging
@ -717,13 +707,16 @@ class BaseAWSLLM:
aws_session_name: str,
region: str,
aws_external_id: Optional[str] = None,
ssl_verify: Optional[Union[bool, str]] = None,
) -> dict:
"""Handle same-account role assumption for IRSA."""
import boto3
verbose_logger.debug("Same account role assumption, using automatic IRSA")
with tracer.trace("boto3.client(sts) with automatic IRSA"):
sts_client = boto3.client("sts", region_name=region, verify=self._get_ssl_verify())
sts_client = boto3.client(
"sts", region_name=region, verify=self._get_ssl_verify(ssl_verify)
)
# Get current caller identity for debugging
try:
@ -778,6 +771,7 @@ class BaseAWSLLM:
aws_role_name: str,
aws_session_name: str,
aws_external_id: Optional[str] = None,
ssl_verify: Optional[Union[bool, str]] = None,
) -> Tuple[Credentials, Optional[int]]:
"""
Authenticate with AWS Role
@ -820,10 +814,15 @@ class BaseAWSLLM:
region,
web_identity_token_file,
aws_external_id,
ssl_verify=ssl_verify,
)
else:
sts_response = self._handle_irsa_same_account(
aws_role_name, aws_session_name, region, aws_external_id
aws_role_name,
aws_session_name,
region,
aws_external_id,
ssl_verify=ssl_verify,
)
return self._extract_credentials_and_ttl(sts_response)
@ -846,7 +845,9 @@ class BaseAWSLLM:
# This allows the web identity token to work automatically
if aws_access_key_id is None and aws_secret_access_key is None:
with tracer.trace("boto3.client(sts)"):
sts_client = boto3.client("sts", verify=self._get_ssl_verify())
sts_client = boto3.client(
"sts", verify=self._get_ssl_verify(ssl_verify)
)
else:
with tracer.trace("boto3.client(sts)"):
sts_client = boto3.client(
@ -854,7 +855,7 @@ class BaseAWSLLM:
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
verify=self._get_ssl_verify(),
verify=self._get_ssl_verify(ssl_verify),
)
assume_role_params = {

87
litellm/llms/bedrock/chat/invoke_handler.py
View file

@ -197,7 +197,12 @@ async def make_call(
try:
if client is None:
client = get_async_httpx_client(
llm_provider=litellm.LlmProviders.BEDROCK
llm_provider=litellm.LlmProviders.BEDROCK,
params={"ssl_verify": logging_obj.litellm_params.get("ssl_verify")}
if logging_obj
and logging_obj.litellm_params
and logging_obj.litellm_params.get("ssl_verify")
else None,
) # Create a new client if none provided
response = await client.post(
@ -286,7 +291,13 @@ def make_sync_call(
):
try:
if client is None:
client = _get_httpx_client(params={})
client = _get_httpx_client(
params={"ssl_verify": logging_obj.litellm_params.get("ssl_verify")}
if logging_obj
and logging_obj.litellm_params
and logging_obj.litellm_params.get("ssl_verify")
else None
)
response = client.post(
api_base,
@ -323,16 +334,22 @@ def make_sync_call(
sync_stream=True,
json_mode=json_mode,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
completion_stream = decoder.iter_bytes(
response.iter_bytes(chunk_size=stream_chunk_size)
)
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
model=model,
sync_stream=True,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
completion_stream = decoder.iter_bytes(
response.iter_bytes(chunk_size=stream_chunk_size)
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
completion_stream = decoder.iter_bytes(
response.iter_bytes(chunk_size=stream_chunk_size)
)
# LOGGING
logging_obj.post_call(
@ -612,12 +629,16 @@ class BedrockLLM(BaseAWSLLM):
outputText = completion_response["generation"]
elif provider == "openai":
# OpenAI imported models use OpenAI Chat Completions format
if "choices" in completion_response and len(completion_response["choices"]) > 0:
if (
"choices" in completion_response
and len(completion_response["choices"]) > 0
):
choice = completion_response["choices"][0]
if "message" in choice:
outputText = choice["message"].get("content")
elif "text" in choice: # fallback for completion format
outputText = choice["text"]
# Set finish reason
if "finish_reason" in choice:
model_response.choices[0].finish_reason = map_finish_reason(
@ -697,7 +718,10 @@ class BedrockLLM(BaseAWSLLM):
## CALCULATING USAGE - bedrock returns usage in the headers
# Skip if usage was already set (e.g., from JSON response for OpenAI provider)
if not hasattr(model_response, "usage") or getattr(model_response, "usage", None) is None:
if (
not hasattr(model_response, "usage")
or getattr(model_response, "usage", None) is None
):
bedrock_input_tokens = response.headers.get(
"x-amzn-bedrock-input-token-count", None
)
@ -780,6 +804,7 @@ class BedrockLLM(BaseAWSLLM):
) # https://bedrock-runtime.{region_name}.amazonaws.com
aws_web_identity_token = optional_params.pop("aws_web_identity_token", None)
aws_sts_endpoint = optional_params.pop("aws_sts_endpoint", None)
ssl_verify = optional_params.pop("ssl_verify", None)
### SET REGION NAME ###
if aws_region_name is None:
@ -810,6 +835,7 @@ class BedrockLLM(BaseAWSLLM):
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
ssl_verify=ssl_verify,
)
### SET RUNTIME ENDPOINT ###
@ -961,8 +987,7 @@ class BedrockLLM(BaseAWSLLM):
# Filter to only supported OpenAI params
filtered_params = {
k: v for k, v in inference_params.items()
if k in supported_params
k: v for k, v in inference_params.items() if k in supported_params
}
# OpenAI uses messages format, not prompt
@ -1075,7 +1100,9 @@ class BedrockLLM(BaseAWSLLM):
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
completion_stream = decoder.iter_bytes(
response.iter_bytes(chunk_size=stream_chunk_size)
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
@ -1343,9 +1370,7 @@ class AWSEventStreamDecoder:
dict,
Optional[
List[
Union[
ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock
]
Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]
]
],
]:
@ -1354,9 +1379,7 @@ class AWSEventStreamDecoder:
provider_specific_fields: dict = {}
thinking_blocks: Optional[
List[
Union[
ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock
]
Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]
]
] = None
@ -1369,9 +1392,7 @@ class AWSEventStreamDecoder:
response_tool_name=_response_tool_name
)
self.tool_calls_index = (
0
if self.tool_calls_index is None
else self.tool_calls_index + 1
0 if self.tool_calls_index is None else self.tool_calls_index + 1
)
tool_use = {
"id": start_obj["toolUse"]["toolUseId"],
@ -1405,9 +1426,7 @@ class AWSEventStreamDecoder:
Optional[str],
Optional[
List[
Union[
ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock
]
Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]
]
],
]:
@ -1418,9 +1437,7 @@ class AWSEventStreamDecoder:
reasoning_content: Optional[str] = None
thinking_blocks: Optional[
List[
Union[
ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock
]
Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]
]
] = None
@ -1456,8 +1473,16 @@ class AWSEventStreamDecoder:
and len(thinking_blocks) > 0
and reasoning_content is None
):
reasoning_content = "" # set to non-empty string to ensure consistency with Anthropic
return text, tool_use, provider_specific_fields, reasoning_content, thinking_blocks
reasoning_content = (
"" # set to non-empty string to ensure consistency with Anthropic
)
return (
text,
tool_use,
provider_specific_fields,
reasoning_content,
thinking_blocks,
)
def _handle_converse_stop_event(
self, index: int
@ -1505,9 +1530,11 @@ class AWSEventStreamDecoder:
content_block_index = int(chunk_data.get("contentBlockIndex", 0))
if "start" in chunk_data:
start_obj = ContentBlockStartEvent(**chunk_data["start"])
tool_use, provider_specific_fields, thinking_blocks = (
self._handle_converse_start_event(start_obj)
)
(
tool_use,
provider_specific_fields,
thinking_blocks,
) = self._handle_converse_start_event(start_obj)
elif "delta" in chunk_data:
delta_obj = ContentBlockDeltaEvent(**chunk_data["delta"])
(

78
litellm/llms/bedrock/common_utils.py
View file

@ -1,3 +1,5 @@
from __future__ import annotations
"""
Common utilities used across bedrock chat/embedding/image generation
"""
@ -34,7 +36,7 @@ _get_model_info = None
def get_cached_model_info():
"""
Lazy import and cache get_model_info to avoid circular imports.
This function is used by bedrock transformation classes that need get_model_info
but cannot import it at module level due to circular import issues.
The function is cached after first use to avoid performance impact.
@ -42,6 +44,7 @@ def get_cached_model_info():
global _get_model_info
if _get_model_info is None:
from litellm import get_model_info
_get_model_info = get_model_info
return _get_model_info
@ -135,33 +138,15 @@ def add_custom_header(headers):
def _get_bedrock_client_ssl_verify() -> Union[bool, str]:
"""
Get SSL verification setting for Bedrock client.
Returns the SSL verification setting which can be:
- True: Use default SSL verification
- False: Disable SSL verification
- str: Path to a custom CA bundle file
"""
from litellm.secret_managers.main import str_to_bool
ssl_verify: Union[bool, str, None] = os.getenv("SSL_VERIFY", litellm.ssl_verify)
# Convert string "False"/"True" to boolean
if isinstance(ssl_verify, str):
# Check if it's a file path
if os.path.exists(ssl_verify):
return ssl_verify # Keep the file path
# Otherwise try to convert to boolean
ssl_verify_bool = str_to_bool(ssl_verify)
if ssl_verify_bool is not None:
ssl_verify = ssl_verify_bool
# Check SSL_CERT_FILE environment variable for custom CA bundle
if ssl_verify is True or ssl_verify == "True":
ssl_cert_file = os.getenv("SSL_CERT_FILE")
if ssl_cert_file and os.path.exists(ssl_cert_file):
return ssl_cert_file
return ssl_verify if ssl_verify is not None else True
from litellm.llms.custom_httpx.http_handler import get_ssl_verify
return get_ssl_verify()
def init_bedrock_client(
@ -287,7 +272,7 @@ def init_bedrock_client(
"sts",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
verify=ssl_verify
verify=ssl_verify,
)
sts_response = sts_client.assume_role(
@ -426,7 +411,7 @@ def strip_bedrock_routing_prefix(model: str) -> str:
def strip_bedrock_throughput_suffix(model: str) -> str:
""" Strip throughput tier suffixes from Bedrock model names. """
"""Strip throughput tier suffixes from Bedrock model names."""
import re
# Pattern matches model:version:throughput where throughput is like 51k, 18k, etc.
@ -500,6 +485,22 @@ class BedrockModelInfo(BaseLLMModelInfo):
) -> List[str]:
return []
# def get_provider_info(self, model: str) -> Optional[ProviderSpecificModelInfo]:
# """
# Handles Bedrock throughput suffixes like ":28k", ":51k".
# """
# import re
# overrides: ProviderSpecificModelInfo = {}
# # Parse context window suffix (e.g., :28k, :51k)
# match = re.search(r":(\d+)k$", model)
# if match:
# throughput_value = int(match.group(1)) * 1000
# overrides["max_input_tokens"] = throughput_value
# return overrides if overrides else None
def get_token_counter(self) -> Optional[BaseTokenCounter]:
"""
Factory method to create a Bedrock token counter.
@ -532,12 +533,29 @@ class BedrockModelInfo(BaseLLMModelInfo):
@staticmethod
def get_bedrock_route(
model: str,
) -> Literal["converse", "invoke", "converse_like", "agent", "agentcore", "async_invoke", "openai"]:
) -> Literal[
"converse",
"invoke",
"converse_like",
"agent",
"agentcore",
"async_invoke",
"openai",
]:
"""
Get the bedrock route for the given model.
"""
route_mappings: Dict[
str, Literal["invoke", "converse_like", "converse", "agent", "agentcore", "async_invoke", "openai"]
str,
Literal[
"invoke",
"converse_like",
"converse",
"agent",
"agentcore",
"async_invoke",
"openai",
],
] = {
"invoke/": "invoke",
"converse_like/": "converse_like",
@ -645,10 +663,10 @@ class BedrockModelInfo(BaseLLMModelInfo):
def get_bedrock_chat_config(model: str):
"""
Helper function to get the appropriate Bedrock chat config based on model and route.
Args:
model: The model name/identifier
Returns:
The appropriate Bedrock config class instance
"""
@ -667,11 +685,13 @@ def get_bedrock_chat_config(model: str):
from litellm.llms.bedrock.chat.invoke_agent.transformation import (
AmazonInvokeAgentConfig,
)
return AmazonInvokeAgentConfig()
elif bedrock_route == "agentcore":
from litellm.llms.bedrock.chat.agentcore.transformation import (
AmazonAgentCoreConfig,
)
return AmazonAgentCoreConfig()
# Handle provider-specific configs

67
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -235,6 +235,63 @@ class AmazonAnthropicClaudeMessagesConfig(
if "opus-4" in model.lower() or "opus_4" in model.lower():
beta_set.add("tool-search-tool-2025-10-19")
def _convert_output_format_to_inline_schema(
self,
output_format: Dict,
anthropic_messages_request: Dict,
) -> None:
"""
Convert Anthropic output_format to inline schema in message content.
Bedrock Invoke doesn't support the output_format parameter, so we embed
the schema directly into the user message content as text instructions.
This approach adds the schema to the last user message, instructing the model
to respond in the specified JSON format.
Args:
output_format: The output_format dict with 'type' and 'schema'
anthropic_messages_request: The request dict to modify in-place
Ref: https://aws.amazon.com/blogs/machine-learning/structured-data-response-with-amazon-bedrock-prompt-engineering-and-tool-use/
"""
import json
# Extract schema from output_format
schema = output_format.get("schema")
if not schema:
return
# Get messages from the request
messages = anthropic_messages_request.get("messages", [])
if not messages:
return
# Find the last user message
last_user_message_idx = None
for idx in range(len(messages) - 1, -1, -1):
if messages[idx].get("role") == "user":
last_user_message_idx = idx
break
if last_user_message_idx is None:
return
last_user_message = messages[last_user_message_idx]
content = last_user_message.get("content", [])
# Ensure content is a list
if isinstance(content, str):
content = [{"type": "text", "text": content}]
last_user_message["content"] = content
# Add schema as text content to the message
schema_text = {
"type": "text",
"text": json.dumps(schema)
}
content.append(schema_text)
def transform_anthropic_messages_request(
self,
model: str,
@ -271,8 +328,16 @@ class AmazonAnthropicClaudeMessagesConfig(
# 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it)
self._remove_ttl_from_cache_control(anthropic_messages_request)
# 5. Convert `output_format` to inline schema (Bedrock invoke doesn't support output_format)
output_format = anthropic_messages_request.pop("output_format", None)
if output_format:
self._convert_output_format_to_inline_schema(
output_format=output_format,
anthropic_messages_request=anthropic_messages_request,
)
# 5. AUTO-INJECT beta headers based on features used
# 6. AUTO-INJECT beta headers based on features used
anthropic_model_info = AnthropicModelInfo()
tools = anthropic_messages_optional_request_params.get("tools")
messages_typed = cast(List[AllMessageValues], messages)

7
litellm/llms/brave/search/__init__.py Normal file
View file

@ -0,0 +1,7 @@
"""
Brave Search API module.
"""
from litellm.llms.brave.search.transformation import BraveSearchConfig
__all__ = ["BraveSearchConfig"]

307
litellm/llms/brave/search/transformation.py Normal file
View file

@ -0,0 +1,307 @@
"""
Brave Search /web/search endpoint.
Documentation: https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
"""
from __future__ import annotations
from datetime import datetime, timezone
from dateutil import parser
from typing import Dict, List, Literal, Optional, TypedDict, Union
import httpx
import re
_ISO_YMD = re.compile(r"^\s*\d{4}[-/]\d{1,2}[-/]\d{1,2}\s*$")
_UNIX_TIMESTAMP = re.compile(r"^\s*-?\d+(\.\d+)?\s*$")
BRAVE_SECTIONS = ["web", "discussions", "faqs", "faq", "news", "videos"]
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.search.transformation import (
BaseSearchConfig,
SearchResponse,
SearchResult,
)
from litellm.secret_managers.main import get_secret_str
def to_yyyy_mm_dd(
s: Union[str, int, float, None],
*,
dayfirst: bool = False,
yearfirst: bool = False,
) -> Optional[str]:
"""
Convert a string/int/float to YYYY-MM-DD; return None if parsing fails.
"""
if not s:
return None
s = str(s).strip()
# Handle Unix timestamps (seconds or milliseconds).
if _UNIX_TIMESTAMP.match(s):
try:
ts_float = float(s)
# Treat large values as milliseconds.
if ts_float > 1e11 or ts_float < -1e11:
ts_float /= 1000.0
return datetime.fromtimestamp(ts_float, tz=timezone.utc).date().isoformat()
except Exception:
return None
# If it looks like YYYY-M-D (ISO-ish), force yearfirst to avoid surprises.
try:
if _ISO_YMD.match(s):
dt = parser.parse(s, yearfirst=True, dayfirst=False, fuzzy=True)
else:
dt = parser.parse(s, yearfirst=yearfirst, dayfirst=dayfirst, fuzzy=True)
return dt.date().isoformat()
except Exception:
return None
class _BraveSearchRequestRequired(TypedDict):
"""Required fields for Brave Search API request."""
q: str # Required - search query
class BraveSearchRequest(_BraveSearchRequestRequired, total=False):
"""
Brave Search API request format.
Based on: https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
"""
count: int # Optional - number of web results to return (Brave max is 20)
offset: int # Optional - pagination offset
country: str # Optional - two-letter ISO country code
search_lang: str # Optional - language to bias results
ui_lang: str # Optional - language for UI strings
freshness: str # Optional - Brave freshness window (e.g., "pd", "pw", "pm")
safesearch: str # Optional - "off" | "moderate" | "strict"
spellcheck: str # Optional - "strict" | "moderate" | "off"
text_decorations: bool # Optional - enable/disable text decorations
result_filter: str # Optional - e.g., "web"
units: str # Optional - measurement units
goggles_id: str # Optional - Brave Goggles id
goggles: str # Optional - Brave Goggles DSL
extra_snippets: bool # Optional - request extra snippets
summary: bool # Optional - include summary block
enable_rich_callback: bool # Optional - structured result blocks
include_fetch_metadata: bool # Optional - include fetch metadata
operators: bool # Optional - enable advanced operators
class BraveSearchConfig(BaseSearchConfig):
BRAVE_API_BASE = "https://api.search.brave.com/res/v1/web/search"
@staticmethod
def ui_friendly_name() -> str:
return "Brave Search"
def get_http_method(self) -> Literal["GET", "POST"]:
"""
Brave Search API uses GET requests for search.
"""
return "GET"
def validate_environment(
self,
headers: Dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
**kwargs,
) -> Dict:
"""
Validate environment and return headers.
"""
api_key = api_key or get_secret_str("BRAVE_API_KEY")
if not api_key:
raise ValueError(
"BRAVE_API_KEY is not set. Set `BRAVE_API_KEY` environment variable."
)
headers["X-Subscription-Token"] = api_key
headers["Accept"] = "application/json"
headers["Accept-Encoding"] = "gzip"
headers["Content-Type"] = "application/json"
return headers
def get_complete_url(
self,
api_base: Optional[str],
optional_params: dict,
data: Optional[Union[Dict, List[Dict]]] = None,
**kwargs,
) -> str:
"""
Get complete URL for Search endpoint with query parameters.
The Brave Search API uses GET requests and therefore needs the request
body (data) to construct query parameters in the URL.
"""
from urllib.parse import urlencode
api_base = api_base or get_secret_str("BRAVE_API_BASE") or self.BRAVE_API_BASE
# Build query parameters from the transformed request body
if data and isinstance(data, dict) and "_brave_params" in data:
params = data["_brave_params"]
query_string = urlencode(params, doseq=True)
return f"{api_base}?{query_string}"
return api_base
def transform_search_request(
self,
query: Union[str, List[str]],
optional_params: dict,
api_key: Optional[str] = None,
search_engine_id: Optional[str] = None,
**kwargs,
) -> Dict:
"""
Transform Search request to Brave Search API format.
Transforms Perplexity unified spec parameters:
- query → q (same)
- max_results → count
- search_domain_filter → q (append domain filters)
- country → country
- max_tokens_per_page → (not applicable, ignored)
All other Brave Search API-specific parameters are passed through as-is.
Args:
query: Search query (string or list of strings). Brave Search API supports single string queries.
optional_params: Optional parameters for the request
Returns:
Dict with typed request data following Brave Search API spec
"""
if isinstance(query, list):
# Brave Search API only supports single string queries
query = " ".join(query)
request_data: BraveSearchRequest = {
"q": query,
}
# Only include "include_fetch_metadata" if it is not explicitly set to False
# This parameter results (more often than not) in a timestamp which we can use for last_updated
if (
"include_fetch_metadata" in optional_params
and optional_params["include_fetch_metadata"] is False
):
request_data["include_fetch_metadata"] = False
else:
request_data["include_fetch_metadata"] = True
# Transform unified spec parameters to Brave Search API format
if "max_results" in optional_params:
# Brave Search API supports 1-20 results per /web/search request
num_results = min(optional_params["max_results"], 20)
request_data["count"] = num_results
if "search_domain_filter" in optional_params:
# Convert to multiple "site:domain" clauses, joined by OR
domains = optional_params["search_domain_filter"]
if isinstance(domains, list) and len(domains) > 0:
request_data["q"] = self._append_domain_filters(
request_data["q"], domains
)
# Convert to dict before dynamic key assignments
result_data = dict(request_data)
# Pass through all other parameters as-is
for param, value in optional_params.items():
if (
param not in self.get_supported_perplexity_optional_params()
and param not in result_data
):
result_data[param] = value
# Store params in special key for URL building (Brave Search API uses GET not POST)
# Return a wrapper dict that stores params for get_complete_url to use
return {
"_brave_params": result_data,
}
@staticmethod
def _append_domain_filters(query: str, domains: List[str]) -> str:
"""
Add site: filters to emulate domain restriction in Brave.
"""
domain_clauses = [f"site:{domain}" for domain in domains]
domain_query = " OR ".join(domain_clauses)
return f"({query}) AND ({domain_query})"
def transform_search_response(
self,
raw_response: httpx.Response,
logging_obj: Optional[LiteLLMLoggingObj],
**kwargs,
) -> SearchResponse:
"""
Transform Brave Search API response to LiteLLM unified SearchResponse format.
"""
response_json = raw_response.json()
# Transform results to SearchResult objects
results: List[SearchResult] = []
query_params = raw_response.request.url.params if raw_response.request else {}
sections_to_process = self._sections_from_params(dict(query_params))
max_results = max(1, min(int(query_params.get("count", 20)), 20))
for section in sections_to_process:
for result in response_json.get(section, {}).get("results", []):
# Because the `max_results`/`count` parameters do not affect
# the number of "discussion", "faq", "news", or "videos"
# results, we need to manually limit the number of results
# returned when an explicit limit has been provided.
if len(results) >= max_results:
break
title = result.get("title", "")
url = result.get("url", "")
snippet = result.get("description", "")
date = to_yyyy_mm_dd(result.get("page_age") or result.get("age"))
last_updated = to_yyyy_mm_dd(
result.get("fetched_content_timestamp", "")
)
search_result = SearchResult(
title=title,
url=url,
snippet=snippet,
date=date,
last_updated=last_updated,
)
results.append(search_result)
return SearchResponse(
results=results,
object="search",
)
@staticmethod
def _sections_from_params(query_params: dict) -> List[str]:
"""
Returns a list of sections the user has requested via the Brave Search
API's `result_filter` parameter. If no `result_filter` parameter is
provided, returns all sections.
"""
raw_filter = query_params.get("result_filter")
requested_filters: List[str] = []
if raw_filter and isinstance(raw_filter, str):
requested_filters = [part.strip() for part in raw_filter.split(",")]
sections = [s.lower() for s in requested_filters if s.lower() in BRAVE_SECTIONS]
return sections or BRAVE_SECTIONS

57
litellm/llms/custom_httpx/http_handler.py
View file

@ -154,6 +154,45 @@ def _create_ssl_context(
return custom_ssl_context
def get_ssl_verify(
ssl_verify: Optional[Union[bool, str]] = None,
) -> Union[bool, str]:
"""
Common utility to resolve the SSL verification setting.
Prioritizes:
1. Passed-in ssl_verify
2. os.environ["SSL_VERIFY"]
3. litellm.ssl_verify
4. os.environ["SSL_CERT_FILE"] (if ssl_verify is True)
Returns:
Union[bool, str]: The resolved SSL verification setting (bool or path to CA bundle)
"""
from litellm.secret_managers.main import str_to_bool
if ssl_verify is None:
ssl_verify = os.getenv("SSL_VERIFY", litellm.ssl_verify)
# Convert string "False"/"True" to boolean if applicable
if isinstance(ssl_verify, str):
# If it's a file path, return it directly
if os.path.exists(ssl_verify):
return ssl_verify
# Otherwise, check if it's a boolean string
ssl_verify_bool = str_to_bool(ssl_verify)
if ssl_verify_bool is not None:
ssl_verify = ssl_verify_bool
# If SSL verification is enabled, check for SSL_CERT_FILE override
if ssl_verify is True:
ssl_cert_file = os.getenv("SSL_CERT_FILE")
if ssl_cert_file and os.path.exists(ssl_cert_file):
return ssl_cert_file
return ssl_verify if ssl_verify is not None else True
def get_ssl_configuration(
ssl_verify: Optional[VerifyTypes] = None,
) -> Union[bool, str, ssl.SSLContext]:
@ -182,20 +221,12 @@ def get_ssl_configuration(
Returns:
Union[bool, str, ssl.SSLContext]: Appropriate SSL configuration
"""
from litellm.secret_managers.main import str_to_bool
if isinstance(ssl_verify, ssl.SSLContext):
# If ssl_verify is already an SSLContext, return it directly
return ssl_verify
# Get ssl_verify from environment or litellm settings if not provided
if ssl_verify is None:
ssl_verify = os.getenv("SSL_VERIFY", litellm.ssl_verify)
ssl_verify_bool = (
str_to_bool(ssl_verify) if isinstance(ssl_verify, str) else ssl_verify
)
if ssl_verify_bool is not None:
ssl_verify = ssl_verify_bool
# Get resolved ssl_verify
ssl_verify = get_ssl_verify(ssl_verify=ssl_verify)
ssl_security_level = os.getenv("SSL_SECURITY_LEVEL", litellm.ssl_security_level)
ssl_ecdh_curve = os.getenv("SSL_ECDH_CURVE", litellm.ssl_ecdh_curve)
@ -822,9 +853,9 @@ class AsyncHTTPHandler:
if AIOHTTP_CONNECTOR_LIMIT > 0:
transport_connector_kwargs["limit"] = AIOHTTP_CONNECTOR_LIMIT
if AIOHTTP_CONNECTOR_LIMIT_PER_HOST > 0:
transport_connector_kwargs["limit_per_host"] = (
AIOHTTP_CONNECTOR_LIMIT_PER_HOST
)
transport_connector_kwargs[
"limit_per_host"
] = AIOHTTP_CONNECTOR_LIMIT_PER_HOST
return LiteLLMAiohttpTransport(
client=lambda: ClientSession(

7
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -25,6 +25,7 @@ from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response impo
_handle_invalid_parallel_tool_calls,
_should_convert_tool_call_to_json_mode,
)
from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.litellm_core_utils.prompt_templates.common_utils import get_tool_call_names
from litellm.litellm_core_utils.prompt_templates.image_handling import (
async_convert_url_to_base64,
@ -586,8 +587,10 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
enhancements=None,
)
translated_choice.finish_reason = self._get_finish_reason(
translated_message, choice["finish_reason"]
translated_choice.finish_reason = map_finish_reason(
self._get_finish_reason(
translated_message, choice["finish_reason"]
)
)
transformed_choices.append(translated_choice)

4
litellm/llms/openai_like/providers.json
View file

@ -72,6 +72,10 @@
"max_completion_tokens": "max_tokens"
}
},
"gmi": {
"base_url": "https://api.gmi-serving.com/v1",
"api_key_env": "GMI_API_KEY"
},
"sarvam": {
"base_url": "https://api.sarvam.ai/v1",
"api_key_env": "SARVAM_API_KEY",

46
litellm/llms/vertex_ai/batches/handler.py
View file

@ -142,6 +142,7 @@ class VertexAIBatchPrediction(VertexLLM):
vertex_location: Optional[str],
timeout: Union[float, httpx.Timeout],
max_retries: Optional[int],
logging_obj: Optional[Any] = None,
) -> Union[LiteLLMBatch, Coroutine[Any, Any, LiteLLMBatch]]:
sync_handler = _get_httpx_client()
@ -187,8 +188,30 @@ class VertexAIBatchPrediction(VertexLLM):
return self._async_retrieve_batch(
api_base=api_base,
headers=headers,
logging_obj=logging_obj,
)
# Log the request using logging_obj if available
if logging_obj is not None:
from litellm.litellm_core_utils.litellm_logging import Logging
if isinstance(logging_obj, Logging):
logging_obj.pre_call(
input="",
api_key="",
additional_args={
"complete_input_dict": {},
"api_base": api_base,
"headers": headers,
"request_str": (
f"\nGET Request Sent from LiteLLM:\n"
f"curl -X GET \\\n"
f"{api_base} \\\n"
f"-H 'Authorization: Bearer ***REDACTED***' \\\n"
f"-H 'Content-Type: application/json; charset=utf-8'\n"
),
},
)
response = sync_handler.get(
url=api_base,
headers=headers,
@ -207,10 +230,33 @@ class VertexAIBatchPrediction(VertexLLM):
self,
api_base: str,
headers: Dict[str, str],
logging_obj: Optional[Any] = None,
) -> LiteLLMBatch:
client = get_async_httpx_client(
llm_provider=litellm.LlmProviders.VERTEX_AI,
)
# Log the request using logging_obj if available
if logging_obj is not None:
from litellm.litellm_core_utils.litellm_logging import Logging
if isinstance(logging_obj, Logging):
logging_obj.pre_call(
input="",
api_key="",
additional_args={
"complete_input_dict": {},
"api_base": api_base,
"headers": headers,
"request_str": (
f"\nGET Request Sent from LiteLLM:\n"
f"curl -X GET \\\n"
f"{api_base} \\\n"
f"-H 'Authorization: Bearer ***REDACTED***' \\\n"
f"-H 'Content-Type: application/json; charset=utf-8'\n"
),
},
)
response = await client.get(
url=api_base,
headers=headers,

2
litellm/llms/vertex_ai/gemini/transformation.py
View file

@ -591,7 +591,7 @@ def _transform_request_body(
data["toolConfig"] = tool_choice
if safety_settings is not None:
data["safetySettings"] = safety_settings
if generation_config is not None:
if generation_config is not None and len(generation_config) > 0:
data["generationConfig"] = generation_config
if cached_content is not None:
data["cachedContent"] = cached_content

4
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -1199,7 +1199,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
and what it means
"""
return {
"FINISH_REASON_UNSPECIFIED": "stop", # openai doesn't have a way of representing this
"FINISH_REASON_UNSPECIFIED": "finish_reason_unspecified",
"STOP": "stop",
"MAX_TOKENS": "length",
"SAFETY": "content_filter",
@ -1209,7 +1209,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"BLOCKLIST": "content_filter",
"PROHIBITED_CONTENT": "content_filter",
"SPII": "content_filter",
"MALFORMED_FUNCTION_CALL": "stop", # openai doesn't have a way of representing this
"MALFORMED_FUNCTION_CALL": "malformed_function_call", # openai doesn't have a way of representing this
"IMAGE_SAFETY": "content_filter",
}

5
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py
View file

@ -117,4 +117,9 @@ class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, Vert
anthropic_messages_request.pop(
"model", None
) # do not pass model in request body to vertex ai
anthropic_messages_request.pop(
"output_format", None
) # do not pass output_format in request body to vertex ai - vertex ai does not support output_format as yet
return anthropic_messages_request

13
litellm/main.py
View file

@ -148,6 +148,7 @@ from litellm.utils import (
validate_and_fix_openai_messages,
validate_and_fix_openai_tools,
validate_chat_completion_tool_choice,
validate_openai_optional_params
)
from ._logging import verbose_logger
@ -599,9 +600,8 @@ async def acompletion( # noqa: PLR0915
ctx = contextvars.copy_context()
func_with_context = partial(ctx.run, func)
# Wrap with timeout if specified
if timeout is not None:
timeout_value = float(timeout) if not isinstance(timeout, (int, float)) else timeout
if timeout is not None and isinstance(timeout, (int, float)):
timeout_value = float(timeout)
init_response = await asyncio.wait_for(
loop.run_in_executor(None, func_with_context),
timeout=timeout_value
@ -616,8 +616,8 @@ async def acompletion( # noqa: PLR0915
response = ModelResponse(**init_response)
response = init_response
elif asyncio.iscoroutine(init_response):
if timeout is not None:
timeout_value = float(timeout) if not isinstance(timeout, (int, float)) else timeout
if timeout is not None and isinstance(timeout, (int, float)):
timeout_value = float(timeout)
response = await asyncio.wait_for(init_response, timeout=timeout_value)
else:
response = await init_response
@ -1115,6 +1115,9 @@ def completion( # type: ignore # noqa: PLR0915
tools = validate_and_fix_openai_tools(tools=tools)
# validate tool_choice
tool_choice = validate_chat_completion_tool_choice(tool_choice=tool_choice)
# validate optional params
stop = validate_openai_optional_params(stop=stop)
######### unpacking kwargs #####################
args = locals()

375
litellm/model_prices_and_context_window_backup.json
View file

@ -1312,6 +1312,9 @@
"supports_function_calling": true
},
"azure_ai/claude-haiku-4-5": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_creation_input_token_cost_above_1hr": 2e-06,
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1330,6 +1333,9 @@
"supports_vision": true
},
"azure_ai/claude-opus-4-5": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1348,6 +1354,9 @@
"supports_vision": true
},
"azure_ai/claude-opus-4-1": {
"cache_creation_input_token_cost": 1.875e-05,
"cache_creation_input_token_cost_above_1hr": 3e-05,
"cache_read_input_token_cost": 1.5e-06,
"input_cost_per_token": 1.5e-05,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1366,6 +1375,9 @@
"supports_vision": true
},
"azure_ai/claude-sonnet-4-5": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -16094,6 +16106,181 @@
"output_cost_per_token": 0.0,
"output_vector_size": 2560
},
"gmi/anthropic/claude-opus-4.5": {
"input_cost_per_token": 5e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-sonnet-4.5": {
"input_cost_per_token": 3e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-sonnet-4": {
"input_cost_per_token": 3e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-opus-4": {
"input_cost_per_token": 1.5e-05,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 7.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/openai/gpt-5.2": {
"input_cost_per_token": 1.75e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.4e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-5.1": {
"input_cost_per_token": 1.25e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-5": {
"input_cost_per_token": 1.25e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-4o": {
"input_cost_per_token": 2.5e-06,
"litellm_provider": "gmi",
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/openai/gpt-4o-mini": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "gmi",
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/deepseek-ai/DeepSeek-V3.2": {
"input_cost_per_token": 2.8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 163840,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_function_calling": true
},
"gmi/deepseek-ai/DeepSeek-V3-0324": {
"input_cost_per_token": 2.8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 163840,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 8.8e-07,
"supports_function_calling": true
},
"gmi/google/gemini-3-pro-preview": {
"input_cost_per_token": 2e-06,
"litellm_provider": "gmi",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/google/gemini-3-flash-preview": {
"input_cost_per_token": 5e-07,
"litellm_provider": "gmi",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 3e-06,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/moonshotai/Kimi-K2-Thinking": {
"input_cost_per_token": 8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 262144,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"gmi/MiniMaxAI/MiniMax-M2.1": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gmi",
"max_input_tokens": 196608,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"gmi/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gmi",
"max_input_tokens": 262144,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.4e-06,
"supports_vision": true
},
"gmi/zai-org/GLM-4.7-FP8": {
"input_cost_per_token": 4e-07,
"litellm_provider": "gmi",
"max_input_tokens": 202752,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 2e-06
},
"google.gemma-3-12b-it": {
"input_cost_per_token": 9e-08,
"litellm_provider": "bedrock_converse",
@ -16863,14 +17050,14 @@
"supports_vision": true
},
"gpt-4o-audio-preview": {
"input_cost_per_audio_token": 0.0001,
"input_cost_per_audio_token": 4e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 0.0002,
"output_cost_per_audio_token": 8e-05,
"output_cost_per_token": 1e-05,
"supports_audio_input": true,
"supports_audio_output": true,
@ -16880,14 +17067,14 @@
"supports_tool_choice": true
},
"gpt-4o-audio-preview-2024-10-01": {
"input_cost_per_audio_token": 0.0001,
"input_cost_per_audio_token": 4e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 0.0002,
"output_cost_per_audio_token": 8e-05,
"output_cost_per_token": 1e-05,
"supports_audio_input": true,
"supports_audio_output": true,
@ -16930,6 +17117,186 @@
"supports_system_messages": true,
"supports_tool_choice": true
},
"gpt-audio": {
"input_cost_per_audio_token": 3.2e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 6.4e-05,
"output_cost_per_token": 1e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-2025-08-28": {
"input_cost_per_audio_token": 3.2e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 6.4e-05,
"output_cost_per_token": 1e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini-2025-10-06": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini-2025-12-15": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-4o-mini": {
"cache_read_input_token_cost": 7.5e-08,
"cache_read_input_token_cost_priority": 1.25e-07,

16
litellm/passthrough/utils.py
View file

@ -3,6 +3,8 @@ from urllib.parse import parse_qs
import httpx
from litellm.constants import PASS_THROUGH_HEADER_PREFIX
class BasePassthroughUtils:
@staticmethod
@ -27,7 +29,11 @@ class BasePassthroughUtils:
forward_headers: Optional[bool] = False,
):
"""
Helper to forward headers from original request
Helper to forward headers from original request.
Also handles 'x-pass-' prefixed headers which are always forwarded
with the prefix stripped, regardless of forward_headers setting.
e.g., 'x-pass-anthropic-beta: value' becomes 'anthropic-beta: value'
"""
if forward_headers is True:
# Header We Should NOT forward
@ -36,6 +42,14 @@ class BasePassthroughUtils:
# Combine request headers with custom headers
headers = {**request_headers, **headers}
# Always process x-pass- prefixed headers (strip prefix and forward)
for header_name, header_value in request_headers.items():
if header_name.lower().startswith(PASS_THROUGH_HEADER_PREFIX):
# Strip the 'x-pass-' prefix to get the actual header name
actual_header_name = header_name[len(PASS_THROUGH_HEADER_PREFIX) :]
headers[actual_header_name] = header_value
return headers
class CommonUtils:

26
litellm/proxy/common_utils/callback_utils.py
View file

@ -380,6 +380,11 @@ def get_logging_caching_headers(request_data: Dict) -> Optional[Dict]:
_metadata["applied_guardrails"]
)
if "applied_policies" in _metadata:
headers["x-litellm-applied-policies"] = ",".join(
_metadata["applied_policies"]
)
if "semantic-similarity" in _metadata:
headers["x-litellm-semantic-similarity"] = str(_metadata["semantic-similarity"])
@ -406,6 +411,27 @@ def add_guardrail_to_applied_guardrails_header(
request_data["metadata"] = _metadata
def add_policy_to_applied_policies_header(
request_data: Dict, policy_name: Optional[str]
):
"""
Add a policy name to the applied_policies list in request metadata.
This is used to track which policies were applied to a request,
similar to how applied_guardrails tracks guardrails.
"""
if policy_name is None:
return
_metadata = request_data.get("metadata", None) or {}
if "applied_policies" in _metadata:
if policy_name not in _metadata["applied_policies"]:
_metadata["applied_policies"].append(policy_name)
else:
_metadata["applied_policies"] = [policy_name]
# Ensure metadata is set back to request_data (important when metadata didn't exist)
request_data["metadata"] = _metadata
def add_guardrail_response_to_standard_logging_object(
litellm_logging_obj: Optional["LiteLLMLogging"],
guardrail_response: StandardLoggingGuardrailInformation,

16
litellm/proxy/guardrails/guardrail_hooks/aim/aim.py
View file

@ -43,8 +43,10 @@ class AimGuardrail(CustomGuardrail):
def __init__(
self, api_key: Optional[str] = None, api_base: Optional[str] = None, **kwargs
):
ssl_verify = kwargs.pop("ssl_verify", None)
self.async_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.GuardrailCallback
llm_provider=httpxSpecialProvider.GuardrailCallback,
params={"ssl_verify": ssl_verify} if ssl_verify is not None else None,
)
self.api_key = api_key or os.environ.get("AIM_API_KEY")
if not self.api_key:
@ -116,9 +118,7 @@ class AimGuardrail(CustomGuardrail):
elif action_type == "block_action":
self._handle_block_action(res["analysis_result"], required_action)
elif action_type == "anonymize_action":
return self._anonymize_request(
res, data
)
return self._anonymize_request(res, data)
else:
verbose_proxy_logger.error(f"Aim: {action_type} action")
return data
@ -132,9 +132,7 @@ class AimGuardrail(CustomGuardrail):
)
raise HTTPException(status_code=400, detail=detection_message)
def _anonymize_request(
self, res: Any, data: dict
) -> dict:
def _anonymize_request(self, res: Any, data: dict) -> dict:
verbose_proxy_logger.info("Aim: anonymize action")
redacted_chat = res.get("redacted_chat")
if not redacted_chat:
@ -179,7 +177,9 @@ class AimGuardrail(CustomGuardrail):
redacted_chat = res.get("redacted_chat", None)
if action_type and action_type == "anonymize_action" and redacted_chat:
return {"redacted_output": redacted_chat["all_redacted_messages"][-1]["content"]}
return {
"redacted_output": redacted_chat["all_redacted_messages"][-1]["content"]
}
return {"redacted_output": output}
def _handle_block_action_on_output(

4
litellm/proxy/guardrails/guardrail_hooks/prompt_security/__init__.py
View file

@ -10,7 +10,9 @@ if TYPE_CHECKING:
def initialize_guardrail(litellm_params: "LitellmParams", guardrail: "Guardrail"):
import litellm
from litellm.proxy.guardrails.guardrail_hooks.prompt_security import PromptSecurityGuardrail
from litellm.proxy.guardrails.guardrail_hooks.prompt_security import (
PromptSecurityGuardrail,
)
_prompt_security_callback = PromptSecurityGuardrail(
api_base=litellm_params.api_base,

731
litellm/proxy/guardrails/guardrail_hooks/prompt_security/prompt_security.py
View file

@ -1,41 +1,58 @@
import asyncio
import base64
import os
import re
from typing import TYPE_CHECKING, Any, AsyncGenerator, Optional, Type, Union
from typing import TYPE_CHECKING, Any, List, Literal, Optional, Type
from fastapi import HTTPException
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.utils import (
Choices,
Delta,
EmbeddingResponse,
ImageResponse,
ModelResponse,
ModelResponseStream,
)
from litellm.types.utils import GenericGuardrailAPIInputs
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel
class PromptSecurityGuardrailMissingSecrets(Exception):
pass
class PromptSecurityGuardrail(CustomGuardrail):
def __init__(self, api_key: Optional[str] = None, api_base: Optional[str] = None, user: Optional[str] = None, system_prompt: Optional[str] = None, **kwargs):
self.async_handler = get_async_httpx_client(llm_provider=httpxSpecialProvider.GuardrailCallback)
def __init__(
self,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
user: Optional[str] = None,
system_prompt: Optional[str] = None,
check_tool_results: Optional[bool] = None,
**kwargs,
):
self.async_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.GuardrailCallback
)
self.api_key = api_key or os.environ.get("PROMPT_SECURITY_API_KEY")
self.api_base = api_base or os.environ.get("PROMPT_SECURITY_API_BASE")
self.user = user or os.environ.get("PROMPT_SECURITY_USER")
self.system_prompt = system_prompt or os.environ.get("PROMPT_SECURITY_SYSTEM_PROMPT")
self.system_prompt = system_prompt or os.environ.get(
"PROMPT_SECURITY_SYSTEM_PROMPT"
)
# Configure whether to check tool/function results for indirect prompt injection
# Default: False (Filter out tool/function messages)
# True: Transform to "other" role and send to API
if check_tool_results is None:
check_tool_results_env = os.environ.get(
"PROMPT_SECURITY_CHECK_TOOL_RESULTS", "false"
).lower()
self.check_tool_results = check_tool_results_env in ("true", "1", "yes")
else:
self.check_tool_results = check_tool_results
if not self.api_key or not self.api_base:
msg = (
"Couldn't get Prompt Security api base or key, "
@ -43,40 +60,316 @@ class PromptSecurityGuardrail(CustomGuardrail):
"or pass them as parameters to the guardrail in the config file"
)
raise PromptSecurityGuardrailMissingSecrets(msg)
# Configuration for file sanitization
self.max_poll_attempts = 30 # Maximum number of polling attempts
self.poll_interval = 2 # Seconds between polling attempts
super().__init__(**kwargs)
async def async_pre_call_hook(
async def apply_guardrail(
self,
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: dict,
call_type: str,
) -> Union[Exception, str, dict, None]:
return await self.call_prompt_security_guardrail(data)
async def async_moderation_hook(
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: str,
) -> Union[Exception, str, dict, None]:
await self.call_prompt_security_guardrail(data)
return data
async def sanitize_file_content(self, file_data: bytes, filename: str) -> dict:
inputs: GenericGuardrailAPIInputs,
request_data: dict,
input_type: Literal["request", "response"],
logging_obj: Optional["LiteLLMLoggingObj"] = None,
) -> GenericGuardrailAPIInputs:
"""
Sanitize file content using Prompt Security API
Apply Prompt Security guardrail to the given inputs.
This method is called by LiteLLM's guardrail framework for ALL endpoints:
- /chat/completions
- /responses
- /messages (Anthropic)
- /embeddings
- /image/generations
- /audio/transcriptions
- /rerank
- MCP server
- and more...
Args:
inputs: Dictionary containing:
- texts: List of texts to check
- images: Optional list of image URLs
- tool_calls: Optional list of tool calls
- structured_messages: Optional full message structure
request_data: The original request data
input_type: "request" for input checking, "response" for output checking
logging_obj: Optional logging object
Returns:
The inputs (potentially modified if action is "modify")
Raises:
HTTPException: If content is blocked by Prompt Security
"""
texts = inputs.get("texts", [])
images = inputs.get("images", [])
structured_messages = inputs.get("structured_messages", [])
# Resolve user API key alias from request metadata
user_api_key_alias = self._resolve_key_alias_from_request_data(request_data)
verbose_proxy_logger.debug(
"Prompt Security Guardrail: apply_guardrail called with input_type=%s, "
"texts=%d, images=%d, structured_messages=%d",
input_type,
len(texts),
len(images),
len(structured_messages),
)
if input_type == "request":
return await self._apply_guardrail_on_request(
inputs=inputs,
texts=texts,
images=images,
structured_messages=structured_messages,
request_data=request_data,
user_api_key_alias=user_api_key_alias,
)
else: # response
return await self._apply_guardrail_on_response(
inputs=inputs,
texts=texts,
user_api_key_alias=user_api_key_alias,
)
async def _apply_guardrail_on_request(
self,
inputs: GenericGuardrailAPIInputs,
texts: List[str],
images: List[str],
structured_messages: list,
request_data: dict,
user_api_key_alias: Optional[str],
) -> GenericGuardrailAPIInputs:
"""Handle request-side guardrail checks."""
# If we have structured messages, use them (they contain role information)
# Otherwise, convert texts to simple user messages
if structured_messages:
messages = list(structured_messages)
else:
messages = [{"role": "user", "content": text} for text in texts]
# Process any embedded files/images in messages
messages = await self.process_message_files(
messages, user_api_key_alias=user_api_key_alias
)
# Also process standalone images from inputs
if images:
await self._process_standalone_images(images, user_api_key_alias)
# Filter messages by role for the API call
filtered_messages = self.filter_messages_by_role(messages)
if not filtered_messages:
verbose_proxy_logger.debug(
"Prompt Security Guardrail: No messages to check after filtering"
)
return inputs
# Call Prompt Security API
headers = self._build_headers(user_api_key_alias)
payload = {
"messages": filtered_messages,
"user": user_api_key_alias or self.user,
"system_prompt": self.system_prompt,
}
self._log_api_request(
method="POST",
url=f"{self.api_base}/api/protect",
headers=headers,
payload={"messages_count": len(filtered_messages)},
)
response = await self.async_handler.post(
f"{self.api_base}/api/protect",
headers=headers,
json=payload,
)
response.raise_for_status()
res = response.json()
self._log_api_response(
url=f"{self.api_base}/api/protect",
status_code=response.status_code,
payload={"result": res.get("result")},
)
result = res.get("result", {}).get("prompt", {})
if result is None:
return inputs
action = result.get("action")
violations = result.get("violations", [])
if action == "block":
raise HTTPException(
status_code=400,
detail="Blocked by Prompt Security, Violations: "
+ ", ".join(violations),
)
elif action == "modify":
# Extract modified texts from modified_messages
modified_messages = result.get("modified_messages", [])
modified_texts = self._extract_texts_from_messages(modified_messages)
if modified_texts:
inputs["texts"] = modified_texts
return inputs
async def _apply_guardrail_on_response(
self,
inputs: GenericGuardrailAPIInputs,
texts: List[str],
user_api_key_alias: Optional[str],
) -> GenericGuardrailAPIInputs:
"""Handle response-side guardrail checks."""
if not texts:
return inputs
# Combine all texts for response checking
combined_text = "\n".join(texts)
headers = self._build_headers(user_api_key_alias)
payload = {
"response": combined_text,
"user": user_api_key_alias or self.user,
"system_prompt": self.system_prompt,
}
self._log_api_request(
method="POST",
url=f"{self.api_base}/api/protect",
headers=headers,
payload={"response_length": len(combined_text)},
)
response = await self.async_handler.post(
f"{self.api_base}/api/protect",
headers=headers,
json=payload,
)
response.raise_for_status()
res = response.json()
self._log_api_response(
url=f"{self.api_base}/api/protect",
status_code=response.status_code,
payload={"result": res.get("result")},
)
result = res.get("result", {}).get("response", {})
if result is None:
return inputs
action = result.get("action")
violations = result.get("violations", [])
if action == "block":
raise HTTPException(
status_code=400,
detail="Blocked by Prompt Security, Violations: "
+ ", ".join(violations),
)
elif action == "modify":
modified_text = result.get("modified_text")
if modified_text is not None:
# If we combined multiple texts, return the modified version as single text
# The framework will handle distributing it back
inputs["texts"] = [modified_text]
return inputs
def _extract_texts_from_messages(self, messages: list) -> List[str]:
"""Extract text content from messages."""
texts = []
for message in messages:
content = message.get("content")
if isinstance(content, str):
texts.append(content)
elif isinstance(content, list):
for item in content:
if isinstance(item, dict) and item.get("type") == "text":
text = item.get("text")
if text:
texts.append(text)
return texts
async def _process_standalone_images(
self, images: List[str], user_api_key_alias: Optional[str]
) -> None:
"""Process standalone images from inputs (data URLs)."""
for image_url in images:
if image_url.startswith("data:"):
try:
header, encoded = image_url.split(",", 1)
file_data = base64.b64decode(encoded)
mime_type = header.split(";")[0].split(":")[1]
extension = mime_type.split("/")[-1]
filename = f"image.{extension}"
result = await self.sanitize_file_content(
file_data, filename, user_api_key_alias=user_api_key_alias
)
if result.get("action") == "block":
violations = result.get("violations", [])
raise HTTPException(
status_code=400,
detail=f"Image blocked by Prompt Security. Violations: {', '.join(violations)}",
)
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.error(f"Error processing image: {str(e)}")
@staticmethod
def _resolve_key_alias_from_request_data(request_data: dict) -> Optional[str]:
"""Resolve user API key alias from request_data metadata."""
# Check litellm_metadata first (set by guardrail framework)
litellm_metadata = request_data.get("litellm_metadata", {})
if litellm_metadata:
alias = litellm_metadata.get("user_api_key_alias")
if alias:
return alias
# Then check regular metadata
metadata = request_data.get("metadata", {})
if metadata:
alias = metadata.get("user_api_key_alias")
if alias:
return alias
return None
async def sanitize_file_content(
self,
file_data: bytes,
filename: str,
user_api_key_alias: Optional[str] = None,
) -> dict:
"""
Sanitize file content using Prompt Security API.
Returns: dict with keys 'action', 'content', 'metadata'
"""
headers = {'APP-ID': self.api_key}
headers = {"APP-ID": self.api_key}
if user_api_key_alias:
headers["X-LiteLLM-Key-Alias"] = user_api_key_alias
self._log_api_request(
method="POST",
url=f"{self.api_base}/api/sanitizeFile",
headers=headers,
payload=f"file upload: {filename}",
)
# Step 1: Upload file for sanitization
files = {'file': (filename, file_data)}
files = {"file": (filename, file_data)}
upload_response = await self.async_handler.post(
f"{self.api_base}/api/sanitizeFile",
headers=headers,
@ -85,16 +378,32 @@ class PromptSecurityGuardrail(CustomGuardrail):
upload_response.raise_for_status()
upload_result = upload_response.json()
job_id = upload_result.get("jobId")
self._log_api_response(
url=f"{self.api_base}/api/sanitizeFile",
status_code=upload_response.status_code,
payload={"jobId": job_id},
)
if not job_id:
raise HTTPException(status_code=500, detail="Failed to get jobId from Prompt Security")
verbose_proxy_logger.debug(f"File sanitization started with jobId: {job_id}")
raise HTTPException(
status_code=500, detail="Failed to get jobId from Prompt Security"
)
verbose_proxy_logger.debug(
"Prompt Security Guardrail: File sanitization started with jobId=%s", job_id
)
# Step 2: Poll for results
for attempt in range(self.max_poll_attempts):
await asyncio.sleep(self.poll_interval)
self._log_api_request(
method="GET",
url=f"{self.api_base}/api/sanitizeFile",
headers=headers,
payload={"jobId": job_id},
)
poll_response = await self.async_handler.get(
f"{self.api_base}/api/sanitizeFile",
headers=headers,
@ -102,11 +411,20 @@ class PromptSecurityGuardrail(CustomGuardrail):
)
poll_response.raise_for_status()
result = poll_response.json()
self._log_api_response(
url=f"{self.api_base}/api/sanitizeFile",
status_code=poll_response.status_code,
payload={"jobId": job_id, "status": result.get("status")},
)
status = result.get("status")
if status == "done":
verbose_proxy_logger.debug(f"File sanitization completed: {result}")
verbose_proxy_logger.debug(
"Prompt Security Guardrail: File sanitization completed for jobId=%s",
job_id,
)
return {
"action": result.get("metadata", {}).get("action", "allow"),
"content": result.get("content"),
@ -114,70 +432,92 @@ class PromptSecurityGuardrail(CustomGuardrail):
"violations": result.get("metadata", {}).get("violations", []),
}
elif status == "in progress":
verbose_proxy_logger.debug(f"File sanitization in progress (attempt {attempt + 1}/{self.max_poll_attempts})")
verbose_proxy_logger.debug(
"Prompt Security Guardrail: File sanitization in progress (attempt %d/%d)",
attempt + 1,
self.max_poll_attempts,
)
continue
else:
raise HTTPException(status_code=500, detail=f"Unexpected sanitization status: {status}")
raise HTTPException(
status_code=500, detail=f"Unexpected sanitization status: {status}"
)
raise HTTPException(status_code=408, detail="File sanitization timeout")
async def _process_image_url_item(self, item: dict) -> dict:
async def _process_image_url_item(
self, item: dict, user_api_key_alias: Optional[str]
) -> dict:
"""Process and sanitize image_url items."""
image_url_data = item.get("image_url", {})
url = image_url_data.get("url", "") if isinstance(image_url_data, dict) else image_url_data
url = (
image_url_data.get("url", "")
if isinstance(image_url_data, dict)
else image_url_data
)
if not url.startswith("data:"):
return item
try:
header, encoded = url.split(",", 1)
file_data = base64.b64decode(encoded)
mime_type = header.split(";")[0].split(":")[1]
extension = mime_type.split("/")[-1]
filename = f"image.{extension}"
sanitization_result = await self.sanitize_file_content(file_data, filename)
sanitization_result = await self.sanitize_file_content(
file_data, filename, user_api_key_alias=user_api_key_alias
)
action = sanitization_result.get("action")
if action == "block":
violations = sanitization_result.get("violations", [])
raise HTTPException(
status_code=400,
detail=f"File blocked by Prompt Security. Violations: {', '.join(violations)}"
detail=f"File blocked by Prompt Security. Violations: {', '.join(violations)}",
)
if action == "modify":
sanitized_content = sanitization_result.get("content", "")
if sanitized_content:
sanitized_encoded = base64.b64encode(sanitized_content.encode()).decode()
sanitized_encoded = base64.b64encode(
sanitized_content.encode()
).decode()
sanitized_url = f"{header},{sanitized_encoded}"
if isinstance(image_url_data, dict):
image_url_data["url"] = sanitized_url
else:
item["image_url"] = sanitized_url
verbose_proxy_logger.info("File content modified by Prompt Security")
verbose_proxy_logger.info(
"File content modified by Prompt Security"
)
return item
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.error(f"Error sanitizing image file: {str(e)}")
raise HTTPException(status_code=500, detail=f"File sanitization failed: {str(e)}")
raise HTTPException(
status_code=500, detail=f"File sanitization failed: {str(e)}"
)
async def _process_document_item(self, item: dict) -> dict:
async def _process_document_item(
self, item: dict, user_api_key_alias: Optional[str]
) -> dict:
"""Process and sanitize document/file items."""
doc_data = item.get("document") or item.get("file") or item
if isinstance(doc_data, dict):
url = doc_data.get("url", "")
doc_content = doc_data.get("data", "")
else:
url = doc_data if isinstance(doc_data, str) else ""
doc_content = ""
if not (url.startswith("data:") or doc_content):
return item
try:
header = ""
if url.startswith("data:"):
@ -186,8 +526,12 @@ class PromptSecurityGuardrail(CustomGuardrail):
mime_type = header.split(";")[0].split(":")[1]
else:
file_data = base64.b64decode(doc_content)
mime_type = doc_data.get("mime_type", "application/pdf") if isinstance(doc_data, dict) else "application/pdf"
mime_type = (
doc_data.get("mime_type", "application/pdf")
if isinstance(doc_data, dict)
else "application/pdf"
)
if "pdf" in mime_type:
filename = "document.pdf"
elif "word" in mime_type or "docx" in mime_type:
@ -197,185 +541,186 @@ class PromptSecurityGuardrail(CustomGuardrail):
else:
extension = mime_type.split("/")[-1]
filename = f"document.{extension}"
verbose_proxy_logger.info(f"Sanitizing document: {filename}")
sanitization_result = await self.sanitize_file_content(file_data, filename)
sanitization_result = await self.sanitize_file_content(
file_data, filename, user_api_key_alias=user_api_key_alias
)
action = sanitization_result.get("action")
if action == "block":
violations = sanitization_result.get("violations", [])
raise HTTPException(
status_code=400,
detail=f"Document blocked by Prompt Security. Violations: {', '.join(violations)}"
detail=f"Document blocked by Prompt Security. Violations: {', '.join(violations)}",
)
if action == "modify":
sanitized_content = sanitization_result.get("content", "")
if sanitized_content:
sanitized_encoded = base64.b64encode(
sanitized_content if isinstance(sanitized_content, bytes) else sanitized_content.encode()
sanitized_content
if isinstance(sanitized_content, bytes)
else sanitized_content.encode()
).decode()
if url.startswith("data:") and header:
sanitized_url = f"{header},{sanitized_encoded}"
if isinstance(doc_data, dict):
doc_data["url"] = sanitized_url
elif isinstance(doc_data, dict):
doc_data["data"] = sanitized_encoded
verbose_proxy_logger.info("Document content modified by Prompt Security")
verbose_proxy_logger.info(
"Document content modified by Prompt Security"
)
return item
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.error(f"Error sanitizing document: {str(e)}")
raise HTTPException(status_code=500, detail=f"Document sanitization failed: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Document sanitization failed: {str(e)}"
)
async def process_message_files(self, messages: list) -> list:
async def process_message_files(
self, messages: list, user_api_key_alias: Optional[str] = None
) -> list:
"""Process messages and sanitize any file content (images, documents, PDFs, etc.)."""
processed_messages = []
for message in messages:
content = message.get("content")
if not isinstance(content, list):
processed_messages.append(message)
continue
processed_content = []
for item in content:
if isinstance(item, dict):
item_type = item.get("type")
if item_type == "image_url":
item = await self._process_image_url_item(item)
item = await self._process_image_url_item(
item, user_api_key_alias
)
elif item_type in ["document", "file"]:
item = await self._process_document_item(item)
item = await self._process_document_item(
item, user_api_key_alias
)
processed_content.append(item)
processed_message = message.copy()
processed_message["content"] = processed_content
processed_messages.append(processed_message)
return processed_messages
async def call_prompt_security_guardrail(self, data: dict) -> dict:
def filter_messages_by_role(self, messages: list) -> list:
"""Filter messages to only include standard OpenAI/Anthropic roles.
messages = data.get("messages", [])
# First, sanitize any files in the messages
messages = await self.process_message_files(messages)
Behavior depends on check_tool_results flag:
- False (default): Filters out tool/function roles completely
- True: Transforms tool/function to "other" role and includes them
def good_msg(msg):
content = msg.get('content', '')
# Handle both string and list content types
if isinstance(content, str):
if content.startswith('### '):
return False
if '"follow_ups": [' in content:
return False
return True
This allows checking tool results for indirect prompt injection when enabled.
"""
supported_roles = ["system", "user", "assistant"]
filtered_messages = []
transformed_count = 0
filtered_count = 0
messages = list(filter(lambda msg: good_msg(msg), messages))
for message in messages:
role = message.get("role", "")
if role in supported_roles:
filtered_messages.append(message)
else:
if self.check_tool_results:
transformed_message = {
"role": "other",
**{
key: value
for key, value in message.items()
if key != "role"
},
}
filtered_messages.append(transformed_message)
transformed_count += 1
verbose_proxy_logger.debug(
"Prompt Security Guardrail: Transformed message from role '%s' to 'other'",
role,
)
else:
filtered_count += 1
verbose_proxy_logger.debug(
"Prompt Security Guardrail: Filtered message with role '%s'",
role,
)
data["messages"] = messages
if transformed_count > 0:
verbose_proxy_logger.debug(
"Prompt Security Guardrail: Transformed %d tool/function messages to 'other' role",
transformed_count,
)
# Then, run the regular prompt security check
headers = { 'APP-ID': self.api_key, 'Content-Type': 'application/json' }
response = await self.async_handler.post(
f"{self.api_base}/api/protect",
headers=headers,
json={"messages": messages, "user": self.user, "system_prompt": self.system_prompt},
)
response.raise_for_status()
res = response.json()
result = res.get("result", {}).get("prompt", {})
if result is None: # prompt can exist but be with value None!
return data
action = result.get("action")
violations = result.get("violations", [])
if action == "block":
raise HTTPException(status_code=400, detail="Blocked by Prompt Security, Violations: " + ", ".join(violations))
elif action == "modify":
data["messages"] = result.get("modified_messages", [])
return data
if filtered_count > 0:
verbose_proxy_logger.debug(
"Prompt Security Guardrail: Filtered %d messages (%d -> %d messages)",
filtered_count,
len(messages),
len(filtered_messages),
)
async def call_prompt_security_guardrail_on_output(self, output: str) -> dict:
response = await self.async_handler.post(
f"{self.api_base}/api/protect",
headers = { 'APP-ID': self.api_key, 'Content-Type': 'application/json' },
json = { "response": output, "user": self.user, "system_prompt": self.system_prompt }
)
response.raise_for_status()
res = response.json()
result = res.get("result", {}).get("response", {})
if result is None: # prompt can exist but be with value None!
return {}
violations = result.get("violations", [])
return { "action": result.get("action"), "modified_text": result.get("modified_text"), "violations": violations }
return filtered_messages
async def async_post_call_success_hook(
def _build_headers(self, user_api_key_alias: Optional[str] = None) -> dict:
headers = {"APP-ID": self.api_key, "Content-Type": "application/json"}
if user_api_key_alias:
headers["X-LiteLLM-Key-Alias"] = user_api_key_alias
return headers
@staticmethod
def _redact_headers(headers: dict) -> dict:
return {
name: ("REDACTED" if name.lower() == "app-id" else value)
for name, value in headers.items()
}
def _log_api_request(
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
response: Union[Any, ModelResponse, EmbeddingResponse, ImageResponse],
) -> Any:
if (isinstance(response, ModelResponse) and response.choices and isinstance(response.choices[0], Choices)):
content = response.choices[0].message.content or ""
ret = await self.call_prompt_security_guardrail_on_output(content)
violations = ret.get("violations", [])
if ret.get("action") == "block":
raise HTTPException(status_code=400, detail="Blocked by Prompt Security, Violations: " + ", ".join(violations))
elif ret.get("action") == "modify":
response.choices[0].message.content = ret.get("modified_text")
return response
method: str,
url: str,
headers: dict,
payload: Any,
) -> None:
verbose_proxy_logger.debug(
"Prompt Security request %s %s headers=%s payload=%s",
method,
url,
self._redact_headers(headers),
payload,
)
async def async_post_call_streaming_iterator_hook(
def _log_api_response(
self,
user_api_key_dict: UserAPIKeyAuth,
response,
request_data: dict,
) -> AsyncGenerator[ModelResponseStream, None]:
buffer: str = ""
WINDOW_SIZE = 250 # Adjust window size as needed
url: str,
status_code: int,
payload: Any,
) -> None:
verbose_proxy_logger.debug(
"Prompt Security response %s status=%s payload=%s",
url,
status_code,
payload,
)
async for item in response:
if not isinstance(item, ModelResponseStream) or not item.choices or len(item.choices) == 0:
yield item
continue
choice = item.choices[0]
if choice.delta and choice.delta.content:
buffer += choice.delta.content
if choice.finish_reason or len(buffer) >= WINDOW_SIZE:
if buffer:
if not choice.finish_reason and re.search(r'\s', buffer):
chunk, buffer = re.split(r'(?=\s\S*$)', buffer, 1)
else:
chunk, buffer = buffer,''
ret = await self.call_prompt_security_guardrail_on_output(chunk)
violations = ret.get("violations", [])
if ret.get("action") == "block":
from litellm.proxy.proxy_server import StreamingCallbackError
raise StreamingCallbackError("Blocked by Prompt Security, Violations: " + ", ".join(violations))
elif ret.get("action") == "modify":
chunk = ret.get("modified_text")
if choice.delta:
choice.delta.content = chunk
else:
choice.delta = Delta(content=chunk)
yield item
@staticmethod
def get_config_model() -> Optional[Type["GuardrailConfigModel"]]:
from litellm.types.proxy.guardrails.guardrail_hooks.prompt_security import (
PromptSecurityGuardrailConfigModel,
)
return PromptSecurityGuardrailConfigModel
return PromptSecurityGuardrailConfigModel

35
litellm/proxy/hooks/dynamic_rate_limiter_v3.py
View file

@ -4,7 +4,7 @@ Dynamic rate limiter v3 - Saturation-aware priority-based rate limiting
import os
from datetime import datetime
from typing import Callable, Dict, List, Optional, Union
from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Union
from fastapi import HTTPException
@ -24,6 +24,25 @@ from litellm.proxy.utils import InternalUsageCache
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import CallTypesLiteral
if TYPE_CHECKING:
from litellm.types.utils import PriorityReservationSettings
def _get_priority_settings() -> "PriorityReservationSettings":
"""
Get the priority reservation settings, guaranteed to be non-None.
The settings are lazy-loaded in litellm.__init__ and always return an instance.
This helper provides proper type narrowing for mypy.
"""
settings = litellm.priority_reservation_settings
if settings is None:
# This should never happen due to lazy loading, but satisfy mypy
from litellm.types.utils import PriorityReservationSettings
return PriorityReservationSettings()
return settings
class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
"""
@ -60,7 +79,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
def _get_saturation_check_cache_ttl(self) -> int:
"""Get the configurable TTL for local cache when reading saturation values."""
return litellm.priority_reservation_settings.saturation_check_cache_ttl
return _get_priority_settings().saturation_check_cache_ttl
async def _get_saturation_value_from_cache(
self,
@ -91,7 +110,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
self, priority: Optional[str], model_info: Optional[ModelGroupInfo] = None
) -> float:
"""Get the weight for a given priority from litellm.priority_reservation"""
weight: float = litellm.priority_reservation_settings.default_priority
weight: float = _get_priority_settings().default_priority
if (
litellm.priority_reservation is None
or priority not in litellm.priority_reservation
@ -201,7 +220,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
priority_key = f"{model}:{priority}"
else:
# No explicit priority: share the default_priority pool with ALL other default keys
priority_weight = litellm.priority_reservation_settings.default_priority
priority_weight = _get_priority_settings().default_priority
# Use shared key for all default-priority requests
priority_key = f"{model}:default_pool"
@ -418,9 +437,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
"""
import json
saturation_threshold = (
litellm.priority_reservation_settings.saturation_threshold
)
saturation_threshold = _get_priority_settings().saturation_threshold
should_enforce_priority = saturation >= saturation_threshold
# Build ALL descriptors upfront
@ -593,9 +610,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
# STEP 1: Check current saturation level
saturation = await self._check_model_saturation(model, model_group_info)
saturation_threshold = (
litellm.priority_reservation_settings.saturation_threshold
)
saturation_threshold = _get_priority_settings().saturation_threshold
verbose_proxy_logger.debug(
f"[Dynamic Rate Limiter] Model={model}, Saturation={saturation:.1%}, "

116
litellm/proxy/litellm_pre_call_utils.py
View file

@ -1082,13 +1082,20 @@ async def add_litellm_data_to_request( # noqa: PLR0915
if disabled_callbacks and isinstance(disabled_callbacks, list):
data["litellm_disabled_callbacks"] = disabled_callbacks
# Guardrails
# Guardrails from key/team metadata
move_guardrails_to_metadata(
data=data,
_metadata_variable_name=_metadata_variable_name,
user_api_key_dict=user_api_key_dict,
)
# Guardrails from policy engine
add_guardrails_from_policy_engine(
data=data,
metadata_variable_name=_metadata_variable_name,
user_api_key_dict=user_api_key_dict,
)
# Team Model Aliases
_update_model_if_team_alias_exists(
data=data,
@ -1314,6 +1321,7 @@ def move_guardrails_to_metadata(
- If guardrails set on API Key metadata then sets guardrails on request metadata
- If guardrails not set on API key, then checks request metadata
- Adds guardrails from policy engine based on team/key/model context
"""
# Check key-level guardrails
_add_guardrails_from_key_or_team_metadata(
@ -1323,6 +1331,15 @@ def move_guardrails_to_metadata(
metadata_variable_name=_metadata_variable_name,
)
#########################################################################################
# Add guardrails from policy engine based on team/key/model context
#########################################################################################
add_guardrails_from_policy_engine(
data=data,
metadata_variable_name=_metadata_variable_name,
user_api_key_dict=user_api_key_dict,
)
#########################################################################################
# User's might send "guardrails" in the request body, we need to add them to the request metadata.
# Since downstream logic requires "guardrails" to be in the request metadata
@ -1351,6 +1368,103 @@ def move_guardrails_to_metadata(
] = request_body_guardrail_config
def add_guardrails_from_policy_engine(
data: dict,
metadata_variable_name: str,
user_api_key_dict: UserAPIKeyAuth,
) -> None:
"""
Add guardrails from the policy engine based on request context.
This function:
1. Gets matching policies based on team_alias, key_alias, and model
2. Resolves guardrails from matching policies (including inheritance)
3. Adds guardrails to request metadata
4. Tracks applied policies in metadata for response headers
Args:
data: The request data to update
metadata_variable_name: The name of the metadata field in data
user_api_key_dict: The user's API key authentication info
"""
from litellm._logging import verbose_proxy_logger
from litellm.proxy.common_utils.callback_utils import (
add_policy_to_applied_policies_header,
)
from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
from litellm.types.proxy.policy_engine import PolicyMatchContext
registry = get_policy_registry()
verbose_proxy_logger.debug(
f"Policy engine: registry initialized={registry.is_initialized()}, "
f"policy_count={len(registry.get_all_policies())}"
)
if not registry.is_initialized():
verbose_proxy_logger.debug("Policy engine not initialized, skipping policy matching")
return
# Build context from request
context = PolicyMatchContext(
team_alias=user_api_key_dict.team_alias,
key_alias=user_api_key_dict.key_alias,
model=data.get("model"),
)
verbose_proxy_logger.debug(
f"Policy engine: matching policies for context team_alias={context.team_alias}, "
f"key_alias={context.key_alias}, model={context.model}"
)
# Get matching policies via attachments
matching_policy_names = PolicyMatcher.get_matching_policies(context=context)
verbose_proxy_logger.debug(f"Policy engine: matched policies via attachments: {matching_policy_names}")
if not matching_policy_names:
return
# Filter to only policies whose conditions match the context
applied_policy_names = PolicyMatcher.get_policies_with_matching_conditions(
policy_names=matching_policy_names,
context=context,
)
verbose_proxy_logger.debug(f"Policy engine: applied policies (conditions matched): {applied_policy_names}")
# Track applied policies in metadata for response headers
for policy_name in applied_policy_names:
add_policy_to_applied_policies_header(
request_data=data, policy_name=policy_name
)
# Resolve guardrails from matching policies
resolved_guardrails = PolicyResolver.resolve_guardrails_for_context(context=context)
verbose_proxy_logger.debug(f"Policy engine: resolved guardrails: {resolved_guardrails}")
if not resolved_guardrails:
return
# Add resolved guardrails to request metadata
if metadata_variable_name not in data:
data[metadata_variable_name] = {}
existing_guardrails = data[metadata_variable_name].get("guardrails", [])
if not isinstance(existing_guardrails, list):
existing_guardrails = []
# Combine existing guardrails with policy-resolved guardrails (no duplicates)
combined = set(existing_guardrails)
combined.update(resolved_guardrails)
data[metadata_variable_name]["guardrails"] = list(combined)
verbose_proxy_logger.debug(
f"Policy engine: added guardrails to request metadata: {list(combined)}"
)
def add_provider_specific_headers_to_request(
data: dict,
headers: dict,

79
litellm/proxy/management_endpoints/common_utils.py
View file

@ -1,5 +1,7 @@
from typing import Any, Dict, Optional, Union
from typing import Any, Dict, Optional, Union, TYPE_CHECKING
from litellm._logging import verbose_proxy_logger
from litellm.caching import DualCache
from litellm.proxy._types import (
KeyRequestBase,
LiteLLM_ManagementEndpoint_MetadataFields,
@ -11,6 +13,9 @@ from litellm.proxy._types import (
)
from litellm.proxy.utils import _premium_user_check
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient, ProxyLogging
def _user_has_admin_view(user_api_key_dict: UserAPIKeyAuth) -> bool:
return (
@ -31,6 +36,78 @@ def _is_user_team_admin(
return False
async def _user_has_admin_privileges(
user_api_key_dict: UserAPIKeyAuth,
prisma_client: Optional["PrismaClient"] = None,
user_api_key_cache: Optional["DualCache"] = None,
proxy_logging_obj: Optional["ProxyLogging"] = None,
) -> bool:
"""
Check if user has admin privileges (proxy admin, team admin, or org admin).
Args:
user_api_key_dict: User API key authentication object
prisma_client: Prisma client for database operations
user_api_key_cache: Cache for user API keys
proxy_logging_obj: Proxy logging object
Returns:
True if user is proxy admin, team admin for any team, or org admin for any organization
"""
# Check if user is proxy admin
if user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN:
return True
# If no database connection, can't check team/org admin status
if prisma_client is None or user_api_key_dict.user_id is None:
return False
# Get user object to check team and org admin status
from litellm.caching import DualCache as DualCacheImport
from litellm.proxy.auth.auth_checks import get_user_object
try:
user_obj = await get_user_object(
user_id=user_api_key_dict.user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache or DualCacheImport(),
user_id_upsert=False,
proxy_logging_obj=proxy_logging_obj,
)
if user_obj is None:
return False
# Check if user is org admin for any organization
if user_obj.organization_memberships is not None:
for membership in user_obj.organization_memberships:
if membership.user_role == LitellmUserRoles.ORG_ADMIN.value:
return True
# Check if user is team admin for any team
if user_obj.teams is not None and len(user_obj.teams) > 0:
# Get all teams user is in
teams = await prisma_client.db.litellm_teamtable.find_many(
where={"team_id": {"in": user_obj.teams}}
)
for team in teams:
team_obj = LiteLLM_TeamTable(**team.model_dump())
if _is_user_team_admin(
user_api_key_dict=user_api_key_dict, team_obj=team_obj
):
return True
except Exception as e:
# If there's an error checking, default to False for security
verbose_proxy_logger.debug(
f"Error checking admin privileges for user {user_api_key_dict.user_id}: {e}"
)
return False
return False
def _set_object_metadata_field(
object_data: Union[
LiteLLM_TeamTable,

72
litellm/proxy/management_endpoints/cost_tracking_settings.py
View file

@ -10,7 +10,7 @@ PATCH /config/cost_margin_config - Update cost margin configuration
POST /cost/estimate - Estimate cost for a given model and token counts
"""
from typing import Dict, Union
from typing import Dict, Optional, Tuple, Union
from fastapi import APIRouter, Depends, HTTPException
@ -29,6 +29,52 @@ from litellm.types.utils import LlmProvidersSet
router = APIRouter()
def _resolve_model_for_cost_lookup(model: str) -> Tuple[str, Optional[str]]:
"""
Resolve a model name (which may be a router alias/model_group) to the
underlying litellm model name for cost lookup.
Args:
model: The model name from the request (could be a router alias like 'e-model-router'
or an actual model name like 'azure_ai/gpt-4')
Returns:
Tuple of (resolved_model_name, custom_llm_provider)
- resolved_model_name: The actual model name to use for cost lookup
- custom_llm_provider: The provider if resolved from router, None otherwise
"""
from litellm.proxy.proxy_server import llm_router
custom_llm_provider: Optional[str] = None
# Try to resolve from router if available
if llm_router is not None:
try:
# Get deployments for this model name (handles aliases, wildcards, etc.)
deployments = llm_router.get_model_list(model_name=model)
if deployments and len(deployments) > 0:
# Get the first deployment's litellm model
first_deployment = deployments[0]
litellm_params = first_deployment.get("litellm_params", {})
resolved_model = litellm_params.get("model")
if resolved_model:
verbose_proxy_logger.debug(
f"Resolved model '{model}' to '{resolved_model}' from router"
)
# Extract custom_llm_provider if present
custom_llm_provider = litellm_params.get("custom_llm_provider")
return resolved_model, custom_llm_provider
except Exception as e:
verbose_proxy_logger.debug(
f"Could not resolve model '{model}' from router: {e}"
)
# Return original model if not resolved
return model, custom_llm_provider
def _calculate_period_costs(
num_requests, cost_per_request, input_cost, output_cost, margin_cost
):
@ -413,12 +459,18 @@ async def estimate_cost(
```
"""
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.utils import Usage
from litellm.utils import ModelResponse
from litellm.types.utils import ModelResponse, Usage
# Resolve model name (handles router aliases like 'e-model-router' -> 'azure_ai/gpt-4')
resolved_model, resolved_provider = _resolve_model_for_cost_lookup(request.model)
verbose_proxy_logger.debug(
f"Cost estimate: request.model='{request.model}' resolved to '{resolved_model}'"
)
# Create a mock response with usage for completion_cost
mock_response = ModelResponse(
model=request.model,
model=resolved_model,
usage=Usage(
prompt_tokens=request.input_tokens,
completion_tokens=request.output_tokens,
@ -428,7 +480,7 @@ async def estimate_cost(
# Create a logging object to capture cost breakdown
litellm_logging_obj = LiteLLMLoggingObj(
model=request.model,
model=resolved_model,
messages=[],
stream=False,
call_type="completion",
@ -441,14 +493,14 @@ async def estimate_cost(
try:
cost_per_request = completion_cost(
completion_response=mock_response,
model=request.model,
model=resolved_model,
litellm_logging_obj=litellm_logging_obj,
)
except Exception as e:
raise HTTPException(
status_code=404,
detail={
"error": f"Could not calculate cost for model '{request.model}': {str(e)}"
"error": f"Could not calculate cost for model '{request.model}' (resolved to '{resolved_model}'): {str(e)}"
},
)
@ -461,7 +513,7 @@ async def estimate_cost(
# Get model info for per-token pricing display
try:
model_info = litellm.get_model_info(model=request.model)
model_info = litellm.get_model_info(model=resolved_model)
input_cost_per_token = model_info.get("input_cost_per_token")
output_cost_per_token = model_info.get("output_cost_per_token")
custom_llm_provider = model_info.get("litellm_provider")
@ -470,6 +522,10 @@ async def estimate_cost(
output_cost_per_token = None
custom_llm_provider = None
# Use provider from router resolution if not found in model_info
if custom_llm_provider is None and resolved_provider is not None:
custom_llm_provider = resolved_provider
# Calculate daily and monthly costs
daily_cost, daily_input_cost, daily_output_cost, daily_margin_cost = (
_calculate_period_costs(

259
litellm/proxy/management_endpoints/policy_endpoints.py Normal file
View file

@ -0,0 +1,259 @@
"""
POLICY MANAGEMENT
All /policy management endpoints
/policy/validate - Validate a policy configuration
/policy/list - List all loaded policies
/policy/info - Get information about a specific policy
"""
from fastapi import APIRouter, Depends, HTTPException, Request
from litellm._logging import verbose_proxy_logger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.management_helpers.utils import management_endpoint_wrapper
from litellm.types.proxy.policy_engine import (
PolicyGuardrailsResponse,
PolicyInfoResponse,
PolicyListResponse,
PolicyMatchContext,
PolicyScopeResponse,
PolicySummaryItem,
PolicyTestResponse,
PolicyValidateRequest,
PolicyValidationResponse,
)
router = APIRouter()
@router.post(
"/policy/validate",
tags=["policy management"],
dependencies=[Depends(user_api_key_auth)],
response_model=PolicyValidationResponse,
)
@management_endpoint_wrapper
async def validate_policy(
request: Request,
data: PolicyValidateRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
) -> PolicyValidationResponse:
"""
Validate a policy configuration before applying it.
Checks:
- All referenced guardrails exist in the guardrail registry
- All non-wildcard team aliases exist in the database
- All non-wildcard key aliases exist in the database
- Inheritance chains are valid (no cycles, parents exist)
- Scope patterns are syntactically valid
Returns:
- valid: True if the policy configuration is valid (no blocking errors)
- errors: List of blocking validation errors
- warnings: List of non-blocking validation warnings
Example request:
```json
{
"policies": {
"global-baseline": {
"guardrails": {
"add": ["pii_blocker", "phi_blocker"]
},
"scope": {
"teams": ["*"],
"keys": ["*"],
"models": ["*"]
}
},
"healthcare-compliance": {
"inherit": "global-baseline",
"guardrails": {
"add": ["hipaa_audit"]
},
"scope": {
"teams": ["healthcare-team"]
}
}
}
}
```
"""
from litellm.proxy.policy_engine.policy_validator import PolicyValidator
from litellm.proxy.proxy_server import prisma_client
verbose_proxy_logger.debug(
f"Validating policy configuration with {len(data.policies)} policies"
)
validator = PolicyValidator(prisma_client=prisma_client)
result = await validator.validate_policy_config(
data.policies,
validate_db=prisma_client is not None,
)
return result
@router.get(
"/policy/list",
tags=["policy management"],
dependencies=[Depends(user_api_key_auth)],
response_model=PolicyListResponse,
)
@management_endpoint_wrapper
async def list_policies(
request: Request,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
) -> PolicyListResponse:
"""
List all loaded policies with their resolved guardrails.
Returns information about each policy including:
- Inheritance configuration
- Scope (teams, keys, models)
- Guardrails to add/remove
- Resolved guardrails (after inheritance)
- Inheritance chain
"""
from litellm.proxy.policy_engine.init_policies import get_policies_summary
summary = get_policies_summary()
return PolicyListResponse(
policies={
name: PolicySummaryItem(
inherit=data.get("inherit"),
scope=PolicyScopeResponse(**data.get("scope", {})),
guardrails=PolicyGuardrailsResponse(**data.get("guardrails", {})),
resolved_guardrails=data.get("resolved_guardrails", []),
inheritance_chain=data.get("inheritance_chain", []),
)
for name, data in summary.get("policies", {}).items()
},
total_count=summary.get("total_count", 0),
)
@router.get(
"/policy/info/{policy_name}",
tags=["policy management"],
dependencies=[Depends(user_api_key_auth)],
response_model=PolicyInfoResponse,
)
@management_endpoint_wrapper
async def get_policy_info(
request: Request,
policy_name: str,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
) -> PolicyInfoResponse:
"""
Get detailed information about a specific policy.
Returns:
- Policy configuration
- Resolved guardrails (after inheritance)
- Inheritance chain
"""
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
registry = get_policy_registry()
if not registry.is_initialized():
raise HTTPException(
status_code=404,
detail="Policy engine not initialized. No policies loaded.",
)
policy = registry.get_policy(policy_name)
if policy is None:
raise HTTPException(
status_code=404,
detail=f"Policy '{policy_name}' not found",
)
resolved = PolicyResolver.resolve_policy_guardrails(
policy_name=policy_name, policies=registry.get_all_policies()
)
return PolicyInfoResponse(
policy_name=policy_name,
inherit=policy.inherit,
scope=PolicyScopeResponse(
teams=[],
keys=[],
models=[],
),
guardrails=PolicyGuardrailsResponse(
add=policy.guardrails.get_add(),
remove=policy.guardrails.get_remove(),
),
resolved_guardrails=resolved.guardrails,
inheritance_chain=resolved.inheritance_chain,
)
@router.post(
"/policy/test",
tags=["policy management"],
dependencies=[Depends(user_api_key_auth)],
response_model=PolicyTestResponse,
)
@management_endpoint_wrapper
async def test_policy_matching(
request: Request,
context: PolicyMatchContext,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
) -> PolicyTestResponse:
"""
Test which policies would match a given request context.
This is useful for debugging and understanding policy behavior.
Request body:
```json
{
"team_alias": "healthcare-team",
"key_alias": "my-api-key",
"model": "gpt-4"
}
```
Returns:
- matching_policies: List of policy names that match
- resolved_guardrails: Final list of guardrails that would be applied
"""
from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
registry = get_policy_registry()
if not registry.is_initialized():
return PolicyTestResponse(
context=context,
matching_policies=[],
resolved_guardrails=[],
message="Policy engine not initialized. No policies loaded.",
)
policies = registry.get_all_policies()
# Get matching policies
matching_policy_names = PolicyMatcher.get_matching_policies(context=context)
# Resolve guardrails
resolved_guardrails = PolicyResolver.resolve_guardrails_for_context(
context=context, policies=policies
)
return PolicyTestResponse(
context=context,
matching_policies=matching_policy_names,
resolved_guardrails=resolved_guardrails,
)

51
litellm/proxy/pass_through_endpoints/architecture.md Normal file
View file

@ -0,0 +1,51 @@
# Pass-Through Endpoints Architecture
## Why Pass-Through Endpoints Transform Requests
Even "pass-through" endpoints must perform essential transformations. The request **body** passes through unchanged, but:
```mermaid
sequenceDiagram
participant Client
participant Proxy as LiteLLM Proxy
participant Provider as LLM Provider
Client->>Proxy: POST /vertex_ai/v1/projects/.../generateContent
Note over Client,Proxy: Headers: Authorization: Bearer sk-litellm-key
Note over Client,Proxy: Body: { "contents": [...] }
rect rgb(240, 240, 240)
Note over Proxy: 1. URL Construction
Note over Proxy: Build: https://us-central1-aiplatform.googleapis.com/...
end
rect rgb(240, 240, 240)
Note over Proxy: 2. Auth Header Replacement
Note over Proxy: Replace litellm key → provider credentials
end
Proxy->>Provider: POST https://us-central1-aiplatform.googleapis.com/...
Note over Proxy,Provider: Headers: Authorization: Bearer ya29.google-oauth...
Note over Proxy,Provider: Body: { "contents": [...] } ← UNCHANGED
Provider-->>Proxy: Response
rect rgb(240, 240, 240)
Note over Proxy: 3. Logging (async, optional)
Note over Proxy: Parse response → calculate cost → log
end
Proxy-->>Client: Response (unchanged)
```
## Essential Transformations
- **URL Construction** - Build correct provider URL (e.g., regional endpoints for Vertex AI, Bedrock)
- **Auth Header Replacement** - Swap LiteLLM virtual key for actual provider credentials
- **Logging** (optional) - Parse response to extract usage and calculate cost
## What Does NOT Change
- Request body
- Response body
- Provider-specific parameters

36
litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
View file

@ -17,7 +17,10 @@ from starlette.websockets import WebSocketState
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.constants import BEDROCK_AGENT_RUNTIME_PASS_THROUGH_ROUTES
from litellm.constants import (
ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS,
BEDROCK_AGENT_RUNTIME_PASS_THROUGH_ROUTES,
)
from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
from litellm.proxy._types import *
from litellm.proxy.auth.route_checks import RouteChecks
@ -1369,24 +1372,24 @@ def get_vertex_base_url(vertex_location: Optional[str]) -> str:
return f"https://{vertex_location}-aiplatform.googleapis.com/"
def add_incoming_headers(request: Request, auth_header: str) -> dict:
def get_vertex_ai_allowed_incoming_headers(request: Request) -> dict:
"""
Build headers from incoming request, preserving headers like anthropic-beta,
while removing headers that should not be forwarded and adding authorization.
Extract only the allowed headers from incoming request for Vertex AI pass-through.
Uses an allowlist approach for security - only forwards headers we explicitly trust.
This prevents accidentally forwarding sensitive headers like the LiteLLM auth token.
Args:
request: The FastAPI request object
auth_header: The authorization token to add
Returns:
dict: Headers dictionary with authorization added
dict: Headers dictionary with only allowed headers
"""
headers = dict(request.headers) or {}
# Remove headers that should not be forwarded
headers.pop("content-length", None)
headers.pop("host", None)
# Add/override the Authorization header
headers["Authorization"] = f"Bearer {auth_header}"
incoming_headers = dict(request.headers) or {}
headers = {}
for header_name in ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS:
if header_name in incoming_headers:
headers[header_name] = incoming_headers[header_name]
return headers
@ -1533,12 +1536,9 @@ async def _prepare_vertex_auth_headers(
api_base="",
)
# Start with incoming request headers to preserve headers like anthropic-beta
headers = dict(request.headers) or {}
# Remove headers that should not be forwarded
headers.pop("content-length", None)
headers.pop("host", None)
# Add/override the Authorization header
# Use allowlist approach - only forward specific safe headers
headers = get_vertex_ai_allowed_incoming_headers(request)
# Add the Authorization header with vendor credentials
headers["Authorization"] = f"Bearer {auth_header}"
if base_target_url is not None:

6
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py
View file

@ -396,7 +396,7 @@ class AnthropicPassthroughLoggingHandler:
# Add batch-specific metadata to indicate this is a pending batch job
litellm_model_response.choices = [Choices(
finish_reason="batch_pending",
finish_reason="stop",
index=0,
message={
"role": "assistant",
@ -438,7 +438,7 @@ class AnthropicPassthroughLoggingHandler:
# Add error-specific metadata
litellm_model_response.choices = [Choices(
finish_reason="batch_error",
finish_reason="stop",
index=0,
message={
"role": "assistant",
@ -472,7 +472,7 @@ class AnthropicPassthroughLoggingHandler:
# Add error-specific metadata
litellm_model_response.choices = [Choices(
finish_reason="batch_error",
finish_reason="stop",
index=0,
message={
"role": "assistant",

6
litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
View file

@ -619,7 +619,7 @@ class VertexPassthroughLoggingHandler:
# Add batch-specific metadata to indicate this is a pending batch job
litellm_model_response.choices = [Choices(
finish_reason="batch_pending",
finish_reason="stop",
index=0,
message={
"role": "assistant",
@ -661,7 +661,7 @@ class VertexPassthroughLoggingHandler:
# Add error-specific metadata
litellm_model_response.choices = [Choices(
finish_reason="batch_error",
finish_reason="stop",
index=0,
message={
"role": "assistant",
@ -695,7 +695,7 @@ class VertexPassthroughLoggingHandler:
# Add error-specific metadata
litellm_model_response.choices = [Choices(
finish_reason="batch_error",
finish_reason="stop",
index=0,
message={
"role": "assistant",

60
litellm/proxy/policy_engine/__init__.py Normal file
View file

@ -0,0 +1,60 @@
"""
LiteLLM Policy Engine
The Policy Engine allows administrators to define policies that combine guardrails
with scoping rules. Policies can target specific teams, API keys, and models using
wildcard patterns, and support inheritance from base policies.
Configuration structure:
- `policies`: Define WHAT guardrails to apply (with inheritance and conditions)
- `policy_attachments`: Define WHERE policies apply (teams, keys, models)
Example:
```yaml
policies:
global-baseline:
description: "Base guardrails for all requests"
guardrails:
add: [pii_blocker]
gpt4-safety:
inherit: global-baseline
description: "Extra safety for GPT-4"
guardrails:
add: [toxicity_filter]
condition:
model: "gpt-4.*" # regex pattern
policy_attachments:
- policy: global-baseline
scope: "*"
- policy: gpt4-safety
scope: "*"
```
"""
from litellm.proxy.policy_engine.attachment_registry import (
AttachmentRegistry,
get_attachment_registry,
)
from litellm.proxy.policy_engine.condition_evaluator import ConditionEvaluator
from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
from litellm.proxy.policy_engine.policy_registry import (
PolicyRegistry,
get_policy_registry,
)
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
from litellm.proxy.policy_engine.policy_validator import PolicyValidator
__all__ = [
# Registries
"PolicyRegistry",
"get_policy_registry",
"AttachmentRegistry",
"get_attachment_registry",
# Core components
"PolicyMatcher",
"PolicyResolver",
"PolicyValidator",
"ConditionEvaluator",
]

54
litellm/proxy/policy_engine/architecture.md Normal file
View file

@ -0,0 +1,54 @@
# Policy Engine Architecture
## Overview
The Policy Engine allows administrators to define policies that combine guardrails with scoping rules. Policies can target specific teams, API keys, and models using wildcard patterns, and support inheritance from base policies.
## Architecture Diagram
```mermaid
flowchart TD
subgraph Config["config.yaml"]
PC[policies config]
end
subgraph PolicyEngine["Policy Engine"]
PR[PolicyRegistry]
PV[PolicyValidator]
PM[PolicyMatcher]
PRe[PolicyResolver]
end
subgraph Request["Incoming Request"]
CTX[Context: team_alias, key_alias, model]
end
subgraph Output["Output"]
GR[Guardrails to Apply]
end
PC -->|load| PR
PC -->|validate| PV
PV -->|errors/warnings| PR
CTX -->|match| PM
PM -->|matching policies| PRe
PR -->|policies| PM
PR -->|policies| PRe
PRe -->|resolve inheritance + add/remove| GR
```
## Components
| Component | File | Description |
|-----------|------|-------------|
| **PolicyRegistry** | `policy_registry.py` | In-memory singleton store for parsed policies |
| **PolicyValidator** | `policy_validator.py` | Validates configs (guardrails, inheritance, teams/keys/models) |
| **PolicyMatcher** | `policy_matcher.py` | Matches request context against policy scopes |
| **PolicyResolver** | `policy_resolver.py` | Resolves final guardrails via inheritance chain |
## Flow
1. **Startup**: `init_policies()` loads policies from config, validates, and populates `PolicyRegistry`
2. **Request**: `PolicyMatcher` finds policies matching the request's team/key/model
3. **Resolution**: `PolicyResolver` traverses inheritance and applies add/remove to get final guardrails

206
litellm/proxy/policy_engine/attachment_registry.py Normal file
View file

@ -0,0 +1,206 @@
"""
Attachment Registry - Manages policy attachments from YAML config.
Attachments define WHERE policies apply, separate from the policy definitions.
This allows the same policy to be attached to multiple scopes.
"""
from typing import Any, Dict, List, Optional
from litellm._logging import verbose_proxy_logger
from litellm.types.proxy.policy_engine import (
PolicyAttachment,
PolicyMatchContext,
)
class AttachmentRegistry:
"""
In-memory registry for storing and managing policy attachments.
Attachments define the relationship between policies and their scopes.
A single policy can have multiple attachments (applied to different scopes).
Example YAML:
```yaml
attachments:
- policy: global-baseline
scope: "*"
- policy: healthcare-compliance
teams: [healthcare-team]
- policy: dev-safety
keys: ["dev-key-*"]
```
"""
def __init__(self):
self._attachments: List[PolicyAttachment] = []
self._initialized: bool = False
def load_attachments(self, attachments_config: List[Dict[str, Any]]) -> None:
"""
Load attachments from a configuration list.
Args:
attachments_config: List of attachment dictionaries from YAML.
"""
self._attachments = []
for attachment_data in attachments_config:
try:
attachment = self._parse_attachment(attachment_data)
self._attachments.append(attachment)
verbose_proxy_logger.debug(
f"Loaded attachment for policy: {attachment.policy}"
)
except Exception as e:
verbose_proxy_logger.error(
f"Error loading attachment: {str(e)}"
)
raise ValueError(f"Invalid attachment: {str(e)}") from e
self._initialized = True
verbose_proxy_logger.info(f"Loaded {len(self._attachments)} policy attachments")
def _parse_attachment(self, attachment_data: Dict[str, Any]) -> PolicyAttachment:
"""
Parse an attachment from raw configuration data.
Args:
attachment_data: Raw attachment configuration
Returns:
Parsed PolicyAttachment object
"""
return PolicyAttachment(
policy=attachment_data.get("policy", ""),
scope=attachment_data.get("scope"),
teams=attachment_data.get("teams"),
keys=attachment_data.get("keys"),
models=attachment_data.get("models"),
)
def get_attached_policies(self, context: PolicyMatchContext) -> List[str]:
"""
Get list of policy names attached to the given context.
Args:
context: The request context to match against
Returns:
List of policy names that are attached to matching scopes
"""
from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
attached_policies: List[str] = []
for attachment in self._attachments:
scope = attachment.to_policy_scope()
if PolicyMatcher.scope_matches(scope=scope, context=context):
if attachment.policy not in attached_policies:
attached_policies.append(attachment.policy)
verbose_proxy_logger.debug(
f"Attachment matched: policy={attachment.policy}, "
f"context=(team={context.team_alias}, key={context.key_alias}, model={context.model})"
)
return attached_policies
def is_policy_attached(
self, policy_name: str, context: PolicyMatchContext
) -> bool:
"""
Check if a specific policy is attached to the given context.
Args:
policy_name: Name of the policy to check
context: The request context to match against
Returns:
True if the policy is attached to a matching scope
"""
attached = self.get_attached_policies(context)
return policy_name in attached
def get_all_attachments(self) -> List[PolicyAttachment]:
"""
Get all loaded attachments.
Returns:
List of all PolicyAttachment objects
"""
return self._attachments.copy()
def get_attachments_for_policy(self, policy_name: str) -> List[PolicyAttachment]:
"""
Get all attachments for a specific policy.
Args:
policy_name: Name of the policy
Returns:
List of attachments for the policy
"""
return [a for a in self._attachments if a.policy == policy_name]
def is_initialized(self) -> bool:
"""
Check if the registry has been initialized with attachments.
Returns:
True if attachments have been loaded, False otherwise
"""
return self._initialized
def clear(self) -> None:
"""
Clear all attachments from the registry.
"""
self._attachments = []
self._initialized = False
def add_attachment(self, attachment: PolicyAttachment) -> None:
"""
Add a single attachment.
Args:
attachment: PolicyAttachment object to add
"""
self._attachments.append(attachment)
verbose_proxy_logger.debug(f"Added attachment for policy: {attachment.policy}")
def remove_attachments_for_policy(self, policy_name: str) -> int:
"""
Remove all attachments for a specific policy.
Args:
policy_name: Name of the policy
Returns:
Number of attachments removed
"""
original_count = len(self._attachments)
self._attachments = [a for a in self._attachments if a.policy != policy_name]
removed_count = original_count - len(self._attachments)
if removed_count > 0:
verbose_proxy_logger.debug(
f"Removed {removed_count} attachment(s) for policy: {policy_name}"
)
return removed_count
# Global singleton instance
_attachment_registry: Optional[AttachmentRegistry] = None
def get_attachment_registry() -> AttachmentRegistry:
"""
Get the global AttachmentRegistry singleton.
Returns:
The global AttachmentRegistry instance
"""
global _attachment_registry
if _attachment_registry is None:
_attachment_registry = AttachmentRegistry()
return _attachment_registry

111
litellm/proxy/policy_engine/condition_evaluator.py Normal file
View file

@ -0,0 +1,111 @@
"""
Condition Evaluator - Evaluates policy conditions.
Supports model-based conditions with exact match or regex patterns.
"""
import re
from typing import List, Optional, Union
from litellm._logging import verbose_proxy_logger
from litellm.types.proxy.policy_engine import (
PolicyCondition,
PolicyMatchContext,
)
class ConditionEvaluator:
"""
Evaluates policy conditions against request context.
Supports model conditions with:
- Exact string match: "gpt-4"
- Regex pattern: "gpt-4.*"
- List of values: ["gpt-4", "gpt-4-turbo"]
"""
@staticmethod
def evaluate(
condition: Optional[PolicyCondition],
context: PolicyMatchContext,
) -> bool:
"""
Evaluate a policy condition against a request context.
Args:
condition: The condition to evaluate (None = always matches)
context: The request context with team, key, model
Returns:
True if condition matches, False otherwise
"""
# No condition means always matches
if condition is None:
return True
# Check model condition
if condition.model is not None:
if not ConditionEvaluator._evaluate_model_condition(
condition=condition.model,
model=context.model,
):
verbose_proxy_logger.debug(
f"Condition failed: model={context.model} did not match {condition.model}"
)
return False
return True
@staticmethod
def _evaluate_model_condition(
condition: Union[str, List[str]],
model: Optional[str],
) -> bool:
"""
Evaluate a model condition.
Args:
condition: String (exact or regex) or list of strings
model: The model name to check
Returns:
True if model matches condition, False otherwise
"""
if model is None:
return False
# Handle list of values
if isinstance(condition, list):
return any(
ConditionEvaluator._matches_pattern(pattern, model)
for pattern in condition
)
# Single value - check as pattern
return ConditionEvaluator._matches_pattern(condition, model)
@staticmethod
def _matches_pattern(pattern: str, value: str) -> bool:
"""
Check if value matches pattern (exact match or regex).
Args:
pattern: Pattern to match (exact string or regex)
value: Value to check
Returns:
True if matches, False otherwise
"""
# First try exact match
if pattern == value:
return True
# Try as regex pattern
try:
if re.fullmatch(pattern, value):
return True
except re.error:
# Invalid regex, treat as literal string (already checked above)
pass
return False

276
litellm/proxy/policy_engine/init_policies.py Normal file
View file

@ -0,0 +1,276 @@
"""
Policy Initialization - Loads policies from config and validates on startup.
Configuration structure:
- policies: Define WHAT guardrails to apply (with inheritance and conditions)
- policy_attachments: Define WHERE policies apply (teams, keys, models)
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from litellm._logging import verbose_proxy_logger
from litellm.proxy.policy_engine.attachment_registry import get_attachment_registry
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
from litellm.proxy.policy_engine.policy_validator import PolicyValidator
from litellm.types.proxy.policy_engine import PolicyValidationResponse
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient
# ANSI color codes for terminal output
_green_color_code = "\033[92m"
_blue_color_code = "\033[94m"
_yellow_color_code = "\033[93m"
_reset_color_code = "\033[0m"
def _print_policies_on_startup(
policies_config: Dict[str, Any],
policy_attachments_config: Optional[List[Dict[str, Any]]] = None,
) -> None:
"""
Print loaded policies to console on startup (similar to model list).
"""
import sys
print( # noqa: T201
f"{_green_color_code}\nLiteLLM Policy Engine: Loaded {len(policies_config)} policies{_reset_color_code}\n"
)
sys.stdout.flush()
for policy_name, policy_data in policies_config.items():
guardrails = policy_data.get("guardrails", {})
inherit = policy_data.get("inherit")
condition = policy_data.get("condition")
description = policy_data.get("description")
guardrails_add = guardrails.get("add", []) if isinstance(guardrails, dict) else []
guardrails_remove = guardrails.get("remove", []) if isinstance(guardrails, dict) else []
inherit_str = f" (inherits: {inherit})" if inherit else ""
print( # noqa: T201
f"{_blue_color_code} - {policy_name}{inherit_str}{_reset_color_code}"
)
if description:
print(f" description: {description}") # noqa: T201
if guardrails_add:
print(f" guardrails.add: {guardrails_add}") # noqa: T201
if guardrails_remove:
print(f" guardrails.remove: {guardrails_remove}") # noqa: T201
if condition:
model_condition = condition.get("model") if isinstance(condition, dict) else None
if model_condition:
print(f" condition.model: {model_condition}") # noqa: T201
# Print attachments
if policy_attachments_config:
print( # noqa: T201
f"\n{_yellow_color_code}Policy Attachments: {len(policy_attachments_config)} attachment(s){_reset_color_code}"
)
for attachment in policy_attachments_config:
policy = attachment.get("policy", "unknown")
scope = attachment.get("scope")
teams = attachment.get("teams")
keys = attachment.get("keys")
models = attachment.get("models")
scope_parts = []
if scope == "*":
scope_parts.append("scope=* (global)")
if teams:
scope_parts.append(f"teams={teams}")
if keys:
scope_parts.append(f"keys={keys}")
if models:
scope_parts.append(f"models={models}")
scope_str = ", ".join(scope_parts) if scope_parts else "all"
print(f" - {policy} -> {scope_str}") # noqa: T201
else:
print( # noqa: T201
f"\n{_yellow_color_code}Warning: No policy_attachments configured. Policies will not be applied to any requests.{_reset_color_code}"
)
print() # noqa: T201
sys.stdout.flush()
async def init_policies(
policies_config: Dict[str, Any],
policy_attachments_config: Optional[List[Dict[str, Any]]] = None,
prisma_client: Optional["PrismaClient"] = None,
validate_db: bool = True,
fail_on_error: bool = True,
) -> PolicyValidationResponse:
"""
Initialize policies from configuration.
This function:
1. Parses the policy configuration
2. Validates policies (guardrails exist, teams/keys exist in DB)
3. Loads policies into the global registry
4. Loads attachments into the attachment registry (if provided)
Args:
policies_config: Dictionary mapping policy names to policy definitions
policy_attachments_config: Optional list of policy attachment configurations
prisma_client: Optional Prisma client for database validation
validate_db: Whether to validate team/key aliases against database
fail_on_error: If True, raise exception on validation errors
Returns:
PolicyValidationResponse with validation results
Raises:
ValueError: If fail_on_error is True and validation errors are found
"""
verbose_proxy_logger.info(f"Initializing {len(policies_config)} policies...")
# Print policies to console on startup
_print_policies_on_startup(policies_config, policy_attachments_config)
# Get the global registries
policy_registry = get_policy_registry()
attachment_registry = get_attachment_registry()
# Create validator
validator = PolicyValidator(prisma_client=prisma_client)
# Validate the configuration
validation_result = await validator.validate_policy_config(
policies_config,
validate_db=validate_db,
)
# Log validation results
if validation_result.errors:
for error in validation_result.errors:
verbose_proxy_logger.error(
f"Policy validation error in '{error.policy_name}': "
f"[{error.error_type}] {error.message}"
)
if validation_result.warnings:
for warning in validation_result.warnings:
verbose_proxy_logger.warning(
f"Policy validation warning in '{warning.policy_name}': "
f"[{warning.error_type}] {warning.message}"
)
# Fail if there are errors and fail_on_error is True
if not validation_result.valid and fail_on_error:
error_messages = [
f"[{e.policy_name}] {e.message}" for e in validation_result.errors
]
raise ValueError(
f"Policy validation failed with {len(validation_result.errors)} error(s):\n"
+ "\n".join(error_messages)
)
# Load policies into registry (even with warnings)
try:
policy_registry.load_policies(policies_config)
verbose_proxy_logger.info(
f"Successfully loaded {len(policies_config)} policies"
)
except Exception as e:
verbose_proxy_logger.error(f"Failed to load policies: {str(e)}")
raise
# Load attachments if provided
if policy_attachments_config:
try:
attachment_registry.load_attachments(policy_attachments_config)
verbose_proxy_logger.info(
f"Successfully loaded {len(policy_attachments_config)} policy attachments"
)
except Exception as e:
verbose_proxy_logger.error(f"Failed to load policy attachments: {str(e)}")
raise
return validation_result
def init_policies_sync(
policies_config: Dict[str, Any],
policy_attachments_config: Optional[List[Dict[str, Any]]] = None,
fail_on_error: bool = True,
) -> None:
"""
Synchronous version of init_policies (without DB validation).
Use this when async is not available or DB validation is not needed.
Args:
policies_config: Dictionary mapping policy names to policy definitions
policy_attachments_config: Optional list of policy attachment configurations
fail_on_error: If True, raise exception on validation errors
"""
import asyncio
# Run the async function without DB validation
try:
loop = asyncio.get_event_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(
init_policies(
policies_config=policies_config,
policy_attachments_config=policy_attachments_config,
prisma_client=None,
validate_db=False,
fail_on_error=fail_on_error,
)
)
def get_policies_summary() -> Dict[str, Any]:
"""
Get a summary of loaded policies for debugging/display.
Returns:
Dictionary with policy information
"""
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
policy_registry = get_policy_registry()
attachment_registry = get_attachment_registry()
if not policy_registry.is_initialized():
return {"initialized": False, "policies": {}, "attachments": []}
resolved = PolicyResolver.get_all_resolved_policies()
summary: Dict[str, Any] = {
"initialized": True,
"policy_count": len(resolved),
"attachment_count": len(attachment_registry.get_all_attachments()),
"policies": {},
"attachments": [],
}
for policy_name, resolved_policy in resolved.items():
policy = policy_registry.get_policy(policy_name)
summary["policies"][policy_name] = {
"inherit": policy.inherit if policy else None,
"description": policy.description if policy else None,
"guardrails_add": policy.guardrails.get_add() if policy else [],
"guardrails_remove": policy.guardrails.get_remove() if policy else [],
"condition": policy.condition.model_dump() if policy and policy.condition else None,
"resolved_guardrails": resolved_policy.guardrails,
"inheritance_chain": resolved_policy.inheritance_chain,
}
# Add attachment info
for attachment in attachment_registry.get_all_attachments():
summary["attachments"].append({
"policy": attachment.policy,
"scope": attachment.scope,
"teams": attachment.teams,
"keys": attachment.keys,
"models": attachment.models,
})
return summary

168
litellm/proxy/policy_engine/policy_matcher.py Normal file
View file

@ -0,0 +1,168 @@
"""
Policy Matcher - Matches requests against policy attachments.
Uses existing wildcard pattern matching helpers to determine which policies
apply to a given request based on team alias, key alias, and model.
Policies are matched via policy_attachments which define WHERE each policy applies.
"""
from typing import Dict, List, Optional
from litellm._logging import verbose_proxy_logger
from litellm.proxy.auth.route_checks import RouteChecks
from litellm.types.proxy.policy_engine import Policy, PolicyMatchContext, PolicyScope
class PolicyMatcher:
"""
Matches incoming requests against policy attachments.
Supports wildcard patterns:
- "*" matches everything
- "prefix-*" matches anything starting with "prefix-"
Uses policy_attachments to determine which policies apply to a request.
"""
@staticmethod
def matches_pattern(value: Optional[str], patterns: List[str]) -> bool:
"""
Check if a value matches any of the given patterns.
Uses the existing RouteChecks._route_matches_wildcard_pattern helper.
Args:
value: The value to check (e.g., team alias, key alias, model)
patterns: List of patterns to match against
Returns:
True if value matches any pattern, False otherwise
"""
# If no value provided, only match if patterns include "*"
if value is None:
return "*" in patterns
for pattern in patterns:
# Use existing wildcard pattern matching helper
if RouteChecks._route_matches_wildcard_pattern(
route=value, pattern=pattern
):
return True
return False
@staticmethod
def scope_matches(scope: PolicyScope, context: PolicyMatchContext) -> bool:
"""
Check if a policy scope matches the given context.
A scope matches if ALL of its fields match:
- teams matches context.team_alias
- keys matches context.key_alias
- models matches context.model
Args:
scope: The policy scope to check
context: The request context
Returns:
True if scope matches context, False otherwise
"""
# Check teams
if not PolicyMatcher.matches_pattern(context.team_alias, scope.get_teams()):
return False
# Check keys
if not PolicyMatcher.matches_pattern(context.key_alias, scope.get_keys()):
return False
# Check models
if not PolicyMatcher.matches_pattern(context.model, scope.get_models()):
return False
return True
@staticmethod
def get_matching_policies(
context: PolicyMatchContext,
) -> List[str]:
"""
Get list of policy names that match the given context via attachments.
Args:
context: The request context to match against
Returns:
List of policy names that match the context
"""
from litellm.proxy.policy_engine.attachment_registry import (
get_attachment_registry,
)
registry = get_attachment_registry()
if not registry.is_initialized():
verbose_proxy_logger.debug(
"AttachmentRegistry not initialized, returning empty list"
)
return []
return registry.get_attached_policies(context)
@staticmethod
def get_matching_policies_from_registry(
context: PolicyMatchContext,
) -> List[str]:
"""
Get list of policy names that match the given context from the global registry.
Args:
context: The request context to match against
Returns:
List of policy names that match the context
"""
return PolicyMatcher.get_matching_policies(context=context)
@staticmethod
def get_policies_with_matching_conditions(
policy_names: List[str],
context: PolicyMatchContext,
policies: Optional[Dict[str, Policy]] = None,
) -> List[str]:
"""
Filter policies to only those whose conditions match the context.
A policy's condition matches if:
- The policy has no condition (condition is None), OR
- The policy's condition evaluates to True for the given context
Args:
policy_names: List of policy names to filter
context: The request context to evaluate conditions against
policies: Dictionary of all policies (if None, uses global registry)
Returns:
List of policy names whose conditions match the context
"""
from litellm.proxy.policy_engine.condition_evaluator import ConditionEvaluator
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
if policies is None:
registry = get_policy_registry()
if not registry.is_initialized():
return []
policies = registry.get_all_policies()
matching_policies = []
for policy_name in policy_names:
policy = policies.get(policy_name)
if policy is None:
continue
# Policy matches if it has no condition OR condition evaluates to True
if policy.condition is None or ConditionEvaluator.evaluate(
policy.condition, context
):
matching_policies.append(policy_name)
return matching_policies

196
litellm/proxy/policy_engine/policy_registry.py Normal file
View file

@ -0,0 +1,196 @@
"""
Policy Registry - In-memory storage for policies.
Handles storing, retrieving, and managing policies.
Policies define WHAT guardrails to apply. WHERE they apply is defined
by policy_attachments (see AttachmentRegistry).
"""
from typing import Any, Dict, List, Optional
from litellm._logging import verbose_proxy_logger
from litellm.types.proxy.policy_engine import (
Policy,
PolicyCondition,
PolicyGuardrails,
)
class PolicyRegistry:
"""
In-memory registry for storing and managing policies.
This is a singleton that holds all loaded policies and provides
methods to access them.
Policies define WHAT guardrails to apply:
- Base guardrails via guardrails.add/remove
- Inheritance via inherit field
- Conditional guardrails via condition.model
"""
def __init__(self):
self._policies: Dict[str, Policy] = {}
self._initialized: bool = False
def load_policies(self, policies_config: Dict[str, Any]) -> None:
"""
Load policies from a configuration dictionary.
Args:
policies_config: Dictionary mapping policy names to policy definitions.
This is the raw config from the YAML file.
"""
self._policies = {}
for policy_name, policy_data in policies_config.items():
try:
policy = self._parse_policy(policy_name, policy_data)
self._policies[policy_name] = policy
verbose_proxy_logger.debug(f"Loaded policy: {policy_name}")
except Exception as e:
verbose_proxy_logger.error(
f"Error loading policy '{policy_name}': {str(e)}"
)
raise ValueError(f"Invalid policy '{policy_name}': {str(e)}") from e
self._initialized = True
verbose_proxy_logger.info(f"Loaded {len(self._policies)} policies")
def _parse_policy(self, policy_name: str, policy_data: Dict[str, Any]) -> Policy:
"""
Parse a policy from raw configuration data.
Args:
policy_name: Name of the policy
policy_data: Raw policy configuration
Returns:
Parsed Policy object
"""
# Parse guardrails
guardrails_data = policy_data.get("guardrails", {})
if isinstance(guardrails_data, dict):
guardrails = PolicyGuardrails(
add=guardrails_data.get("add"),
remove=guardrails_data.get("remove"),
)
else:
# Handle legacy format where guardrails might be a list
guardrails = PolicyGuardrails(add=guardrails_data if guardrails_data else None)
# Parse condition (simple model-based condition)
condition = None
condition_data = policy_data.get("condition")
if condition_data:
condition = PolicyCondition(model=condition_data.get("model"))
return Policy(
inherit=policy_data.get("inherit"),
description=policy_data.get("description"),
guardrails=guardrails,
condition=condition,
)
def get_policy(self, policy_name: str) -> Optional[Policy]:
"""
Get a policy by name.
Args:
policy_name: Name of the policy to retrieve
Returns:
Policy object if found, None otherwise
"""
return self._policies.get(policy_name)
def get_all_policies(self) -> Dict[str, Policy]:
"""
Get all loaded policies.
Returns:
Dictionary mapping policy names to Policy objects
"""
return self._policies.copy()
def get_policy_names(self) -> List[str]:
"""
Get list of all policy names.
Returns:
List of policy names
"""
return list(self._policies.keys())
def has_policy(self, policy_name: str) -> bool:
"""
Check if a policy exists.
Args:
policy_name: Name of the policy to check
Returns:
True if policy exists, False otherwise
"""
return policy_name in self._policies
def is_initialized(self) -> bool:
"""
Check if the registry has been initialized with policies.
Returns:
True if policies have been loaded, False otherwise
"""
return self._initialized
def clear(self) -> None:
"""
Clear all policies from the registry.
"""
self._policies = {}
self._initialized = False
def add_policy(self, policy_name: str, policy: Policy) -> None:
"""
Add or update a single policy.
Args:
policy_name: Name of the policy
policy: Policy object to add
"""
self._policies[policy_name] = policy
verbose_proxy_logger.debug(f"Added/updated policy: {policy_name}")
def remove_policy(self, policy_name: str) -> bool:
"""
Remove a policy by name.
Args:
policy_name: Name of the policy to remove
Returns:
True if policy was removed, False if it didn't exist
"""
if policy_name in self._policies:
del self._policies[policy_name]
verbose_proxy_logger.debug(f"Removed policy: {policy_name}")
return True
return False
# Global singleton instance
_policy_registry: Optional[PolicyRegistry] = None
def get_policy_registry() -> PolicyRegistry:
"""
Get the global PolicyRegistry singleton.
Returns:
The global PolicyRegistry instance
"""
global _policy_registry
if _policy_registry is None:
_policy_registry = PolicyRegistry()
return _policy_registry

227
litellm/proxy/policy_engine/policy_resolver.py Normal file
View file

@ -0,0 +1,227 @@
"""
Policy Resolver - Resolves final guardrail list from policies.
Handles:
- Inheritance chain resolution (inherit with add/remove)
- Applying add/remove guardrails
- Evaluating model conditions
- Combining guardrails from multiple matching policies
"""
from typing import Dict, List, Optional, Set
from litellm._logging import verbose_proxy_logger
from litellm.types.proxy.policy_engine import (
Policy,
PolicyMatchContext,
ResolvedPolicy,
)
class PolicyResolver:
"""
Resolves the final list of guardrails from policies.
Handles:
- Inheritance chains with add/remove operations
- Model-based conditions
"""
@staticmethod
def resolve_inheritance_chain(
policy_name: str,
policies: Dict[str, Policy],
visited: Optional[Set[str]] = None,
) -> List[str]:
"""
Get the inheritance chain for a policy (from root to policy).
Args:
policy_name: Name of the policy
policies: Dictionary of all policies
visited: Set of visited policies (for cycle detection)
Returns:
List of policy names from root ancestor to the given policy
"""
if visited is None:
visited = set()
if policy_name in visited:
verbose_proxy_logger.warning(
f"Circular inheritance detected for policy '{policy_name}'"
)
return []
policy = policies.get(policy_name)
if policy is None:
return []
visited.add(policy_name)
if policy.inherit:
parent_chain = PolicyResolver.resolve_inheritance_chain(
policy_name=policy.inherit, policies=policies, visited=visited
)
return parent_chain + [policy_name]
return [policy_name]
@staticmethod
def resolve_policy_guardrails(
policy_name: str,
policies: Dict[str, Policy],
context: Optional[PolicyMatchContext] = None,
) -> ResolvedPolicy:
"""
Resolve the final guardrails for a single policy, including inheritance.
This method:
1. Resolves the inheritance chain
2. Applies add/remove from each policy in the chain
3. Evaluates model conditions (if context provided)
Args:
policy_name: Name of the policy to resolve
policies: Dictionary of all policies
context: Optional request context for evaluating conditions
Returns:
ResolvedPolicy with final guardrails list
"""
from litellm.proxy.policy_engine.condition_evaluator import ConditionEvaluator
inheritance_chain = PolicyResolver.resolve_inheritance_chain(
policy_name=policy_name, policies=policies
)
# Start with empty set of guardrails
guardrails: Set[str] = set()
# Apply each policy in the chain (from root to leaf)
for chain_policy_name in inheritance_chain:
policy = policies.get(chain_policy_name)
if policy is None:
continue
# Check if policy condition matches (if context provided)
if context is not None and policy.condition is not None:
if not ConditionEvaluator.evaluate(
condition=policy.condition,
context=context,
):
verbose_proxy_logger.debug(
f"Policy '{chain_policy_name}' condition did not match, skipping guardrails"
)
continue
# Add guardrails from guardrails.add
for guardrail in policy.guardrails.get_add():
guardrails.add(guardrail)
# Remove guardrails from guardrails.remove
for guardrail in policy.guardrails.get_remove():
guardrails.discard(guardrail)
return ResolvedPolicy(
policy_name=policy_name,
guardrails=list(guardrails),
inheritance_chain=inheritance_chain,
)
@staticmethod
def resolve_guardrails_for_context(
context: PolicyMatchContext,
policies: Optional[Dict[str, Policy]] = None,
) -> List[str]:
"""
Resolve the final list of guardrails for a request context.
This:
1. Finds all policies that match the context via policy_attachments
2. Resolves each policy's guardrails (including inheritance)
3. Evaluates model conditions
4. Combines all guardrails (union)
Args:
context: The request context
policies: Dictionary of all policies (if None, uses global registry)
Returns:
List of guardrail names to apply
"""
from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
if policies is None:
registry = get_policy_registry()
if not registry.is_initialized():
return []
policies = registry.get_all_policies()
# Get matching policies via attachments
matching_policy_names = PolicyMatcher.get_matching_policies(context=context)
if not matching_policy_names:
verbose_proxy_logger.debug(
f"No policies match context: team_alias={context.team_alias}, "
f"key_alias={context.key_alias}, model={context.model}"
)
return []
# Resolve each matching policy and combine guardrails
all_guardrails: Set[str] = set()
for policy_name in matching_policy_names:
resolved = PolicyResolver.resolve_policy_guardrails(
policy_name=policy_name,
policies=policies,
context=context,
)
all_guardrails.update(resolved.guardrails)
verbose_proxy_logger.debug(
f"Policy '{policy_name}' contributes guardrails: {resolved.guardrails}"
)
result = list(all_guardrails)
verbose_proxy_logger.debug(
f"Final guardrails for context: {result}"
)
return result
@staticmethod
def get_all_resolved_policies(
policies: Optional[Dict[str, Policy]] = None,
context: Optional[PolicyMatchContext] = None,
) -> Dict[str, ResolvedPolicy]:
"""
Resolve all policies and return their final guardrails.
Useful for debugging and displaying policy configurations.
Args:
policies: Dictionary of all policies (if None, uses global registry)
context: Optional context for evaluating conditions
Returns:
Dictionary mapping policy names to ResolvedPolicy objects
"""
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
if policies is None:
registry = get_policy_registry()
if not registry.is_initialized():
return {}
policies = registry.get_all_policies()
resolved: Dict[str, ResolvedPolicy] = {}
for policy_name in policies:
resolved[policy_name] = PolicyResolver.resolve_policy_guardrails(
policy_name=policy_name,
policies=policies,
context=context,
)
return resolved

348
litellm/proxy/policy_engine/policy_validator.py Normal file
View file

@ -0,0 +1,348 @@
"""
Policy Validator - Validates policy configurations.
Validates:
- Guardrail names exist in the guardrail registry
- Non-wildcard team aliases exist in the database
- Non-wildcard key aliases exist in the database
- Non-wildcard model names exist in the router or match a wildcard route
- Inheritance chains are valid (no cycles, parents exist)
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
from litellm._logging import verbose_proxy_logger
from litellm.types.proxy.policy_engine import (
Policy,
PolicyValidationError,
PolicyValidationErrorType,
PolicyValidationResponse,
)
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient
from litellm.router import Router
class PolicyValidator:
"""
Validates policy configurations against actual data.
"""
def __init__(
self,
prisma_client: Optional["PrismaClient"] = None,
llm_router: Optional["Router"] = None,
):
"""
Initialize the validator.
Args:
prisma_client: Optional Prisma client for database validation
llm_router: Optional LLM router for model validation
"""
self.prisma_client = prisma_client
self.llm_router = llm_router
@staticmethod
def is_wildcard_pattern(pattern: str) -> bool:
"""
Check if a pattern contains wildcards.
Args:
pattern: The pattern to check
Returns:
True if the pattern contains wildcard characters
"""
return "*" in pattern or "?" in pattern
def get_available_guardrails(self) -> Set[str]:
"""
Get set of available guardrail names from the guardrail registry.
Returns:
Set of guardrail names
"""
try:
from litellm.proxy.guardrails.guardrail_registry import (
IN_MEMORY_GUARDRAIL_HANDLER,
)
guardrails = IN_MEMORY_GUARDRAIL_HANDLER.list_in_memory_guardrails()
return {g.get("guardrail_name", "") for g in guardrails if g.get("guardrail_name")}
except Exception as e:
verbose_proxy_logger.warning(
f"Could not get guardrails from registry: {str(e)}"
)
return set()
async def check_team_alias_exists(self, team_alias: str) -> bool:
"""
Check if a specific team alias exists in the database.
Args:
team_alias: The team alias to check
Returns:
True if the team alias exists
"""
if self.prisma_client is None:
return True # Can't validate without DB, assume valid
try:
team = await self.prisma_client.db.litellm_teamtable.find_first(
where={"team_alias": team_alias},
)
return team is not None
except Exception as e:
verbose_proxy_logger.warning(
f"Could not check team alias '{team_alias}': {str(e)}"
)
return True # Assume valid on error
async def check_key_alias_exists(self, key_alias: str) -> bool:
"""
Check if a specific key alias exists in the database.
Args:
key_alias: The key alias to check
Returns:
True if the key alias exists
"""
if self.prisma_client is None:
return True # Can't validate without DB, assume valid
try:
key = await self.prisma_client.db.litellm_verificationtoken.find_first(
where={"key_alias": key_alias},
)
return key is not None
except Exception as e:
verbose_proxy_logger.warning(
f"Could not check key alias '{key_alias}': {str(e)}"
)
return True # Assume valid on error
def check_model_exists(self, model: str) -> bool:
"""
Check if a model exists in the router or matches a wildcard pattern.
Args:
model: The model name to check
Returns:
True if the model exists or matches a pattern in the router
"""
if self.llm_router is None:
return True # Can't validate without router, assume valid
try:
# Check if model is in router's model names
if model in self.llm_router.model_names:
return True
# Check if model matches any pattern via pattern router
if hasattr(self.llm_router, "pattern_router"):
pattern_deployments = self.llm_router.pattern_router.get_deployments_by_pattern(
model=model
)
if pattern_deployments:
return True
return False
except Exception as e:
verbose_proxy_logger.warning(
f"Could not check model '{model}': {str(e)}"
)
return True # Assume valid on error
def _validate_inheritance_chain(
self,
policy_name: str,
policies: Dict[str, Policy],
visited: Optional[Set[str]] = None,
max_depth: int = 100,
) -> List[PolicyValidationError]:
"""
Validate the inheritance chain for a policy.
Checks for:
- Parent policy exists
- No circular inheritance
- Max depth not exceeded
Args:
policy_name: Name of the policy to validate
policies: All policies
visited: Set of already visited policy names (for cycle detection)
max_depth: Maximum recursion depth to prevent infinite loops
Returns:
List of validation errors
"""
errors: List[PolicyValidationError] = []
# Prevent infinite recursion
if max_depth <= 0:
errors.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.CIRCULAR_INHERITANCE,
message=f"Inheritance chain too deep (exceeded max depth of 100)",
field="inherit",
)
)
return errors
if visited is None:
visited = set()
if policy_name in visited:
errors.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.CIRCULAR_INHERITANCE,
message=f"Circular inheritance detected: {' -> '.join(visited)} -> {policy_name}",
field="inherit",
)
)
return errors
policy = policies.get(policy_name)
if policy is None:
return errors
if policy.inherit:
if policy.inherit not in policies:
errors.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.INVALID_INHERITANCE,
message=f"Parent policy '{policy.inherit}' not found",
field="inherit",
value=policy.inherit,
)
)
else:
# Recursively check parent with decremented depth
visited.add(policy_name)
errors.extend(
self._validate_inheritance_chain(
policy.inherit, policies, visited, max_depth - 1
)
)
return errors
async def validate_policies(
self,
policies: Dict[str, Policy],
validate_db: bool = True,
) -> PolicyValidationResponse:
"""
Validate a set of policies.
Args:
policies: Dictionary mapping policy names to Policy objects
validate_db: Whether to validate against database (teams, keys)
Returns:
PolicyValidationResponse with errors and warnings
"""
errors: List[PolicyValidationError] = []
warnings: List[PolicyValidationError] = []
# Get available guardrails
available_guardrails = self.get_available_guardrails()
for policy_name, policy in policies.items():
# Validate guardrails
for guardrail in policy.guardrails.get_add():
if available_guardrails and guardrail not in available_guardrails:
errors.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.INVALID_GUARDRAIL,
message=f"Guardrail '{guardrail}' not found in guardrail registry",
field="guardrails.add",
value=guardrail,
)
)
for guardrail in policy.guardrails.get_remove():
if available_guardrails and guardrail not in available_guardrails:
warnings.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.INVALID_GUARDRAIL,
message=f"Guardrail '{guardrail}' in remove list not found in guardrail registry",
field="guardrails.remove",
value=guardrail,
)
)
# Note: Team, key, and model validation is done via policy_attachments
# Policies no longer have scope - attachments define where policies apply
# Validate inheritance
inheritance_errors = self._validate_inheritance_chain(
policy_name=policy_name, policies=policies
)
errors.extend(inheritance_errors)
return PolicyValidationResponse(
valid=len(errors) == 0,
errors=errors,
warnings=warnings,
)
async def validate_policy_config(
self,
policy_config: Dict[str, Any],
validate_db: bool = True,
) -> PolicyValidationResponse:
"""
Validate a raw policy configuration dictionary.
This parses the config and then validates it.
Args:
policy_config: Raw policy configuration from YAML
validate_db: Whether to validate against database
Returns:
PolicyValidationResponse with errors and warnings
"""
from litellm.proxy.policy_engine.policy_registry import PolicyRegistry
# First, try to parse the policies
errors: List[PolicyValidationError] = []
policies: Dict[str, Policy] = {}
temp_registry = PolicyRegistry()
for policy_name, policy_data in policy_config.items():
try:
policy = temp_registry._parse_policy(policy_name, policy_data)
policies[policy_name] = policy
except Exception as e:
errors.append(
PolicyValidationError(
policy_name=policy_name,
error_type=PolicyValidationErrorType.INVALID_SYNTAX,
message=f"Failed to parse policy: {str(e)}",
)
)
# If there were parsing errors, return early
if errors:
return PolicyValidationResponse(
valid=False,
errors=errors,
warnings=[],
)
# Validate the parsed policies
return await self.validate_policies(policies, validate_db=validate_db)

129
litellm/proxy/proxy_config.yaml
View file

@ -1,42 +1,101 @@
model_list:
# Anthropic direct
- model_name: anthropic-claude
- model_name: "*"
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
# Azure AI Anthropic
- model_name: azure-ai-claude
model: "*"
- model_name: "gpt-4"
litellm_params:
model: azure_ai/claude-3-5-sonnet
api_base: https://krish-mh44t553-eastus2.services.ai.azure.com/
api_key: os.environ/AZURE_ANTHROPIC_API_KEY
# Azure AI Anthropic (alternate endpoint format)
- model_name: claude-4.5-haiku
model: "gpt-4"
api_key: os.environ/OPENAI_API_KEY
- model_name: "gpt-3.5-turbo"
litellm_params:
model: anthropic/claude-haiku-4-5
api_base: https://krish-mh44t553-eastus2.services.ai.azure.com/anthropic/v1/messages
api_version: "2023-06-01"
api_key: os.environ/AZURE_ANTHROPIC_API_KEY
# Search Tools Configuration - Define search providers for WebSearch interception
# search_tools:
# - search_tool_name: "my-perplexity-search"
# litellm_params:
# search_provider: "perplexity" # Can be: perplexity, brave, etc.
litellm_settings:
callbacks: ["websearch_interception"]
# WebSearch Interception - Automatically intercepts and executes WebSearch tool calls
# for models that don't natively support web search (e.g., Bedrock/Claude)
websearch_interception_params:
enabled_providers: ["bedrock"] # List of providers to enable interception for
search_tool_name: "my-perplexity-search" # Optional: Name of search tool from search_tools config
model: "gpt-3.5-turbo"
api_key: os.environ/OPENAI_API_KEY
general_settings:
store_prompts_in_spend_logs: true
forward_client_headers_to_llm_api: true
master_key: sk-1234
# ───────────────────────────────────────────────
# POLICIES - Define WHAT guardrails to apply
# ───────────────────────────────────────────────
#
# Policies define guardrails with:
# - inherit: Inherit guardrails from another policy
# - description: Human-readable description
# - guardrails.add: Add guardrails (on top of inherited)
# - guardrails.remove: Remove guardrails (from inherited)
# - condition.model: Model pattern (exact or regex) for when guardrails apply
#
policies:
# Global baseline policy
global-baseline:
description: "Base guardrails for all requests"
guardrails:
add:
- pii_blocker
# Healthcare policy - inherits from global-baseline
healthcare-compliance:
inherit: global-baseline
description: "HIPAA compliance for healthcare teams"
guardrails:
add:
- hipaa_audit
# Dev policy - inherits but removes PII blocker for testing
internal-dev:
inherit: global-baseline
description: "Relaxed policy for internal development"
guardrails:
add:
- toxicity_filter
remove:
- pii_blocker
# Policy with model condition (regex pattern)
gpt4-safety:
description: "Extra safety for GPT-4 models"
guardrails:
add:
- toxicity_filter
condition:
model: "gpt-4.*" # regex: matches gpt-4, gpt-4-turbo, gpt-4o, etc.
# Policy with model condition (exact match list)
bedrock-compliance:
description: "Compliance for Bedrock models"
guardrails:
add:
- strict_pii_blocker
condition:
model: ["bedrock/claude-3", "bedrock/claude-2"] # exact matches
# ───────────────────────────────────────────────
# POLICY ATTACHMENTS - Define WHERE policies apply
# ───────────────────────────────────────────────
#
# Attachments are REQUIRED to make policies active.
# A policy without an attachment will not be applied.
#
policy_attachments:
# Global attachment - applies to all requests
- policy: global-baseline
scope: "*"
# Team-specific attachment
- policy: healthcare-compliance
teams:
- healthcare-team
- medical-research
# Key pattern attachment
- policy: internal-dev
keys:
- "dev-key-*"
- "test-key-*"
# Model-specific policies (attached globally, condition filters by model)
- policy: gpt4-safety
scope: "*"
- policy: bedrock-compliance
scope: "*"

143
litellm/proxy/proxy_server.py
View file

@ -203,13 +203,13 @@ from litellm.proxy.agent_endpoints.endpoints import router as agent_endpoints_ro
from litellm.proxy.analytics_endpoints.analytics_endpoints import (
router as analytics_router,
)
from litellm.proxy.anthropic_endpoints.claude_code_endpoints import (
claude_code_marketplace_router,
)
from litellm.proxy.anthropic_endpoints.endpoints import router as anthropic_router
from litellm.proxy.anthropic_endpoints.skills_endpoints import (
router as anthropic_skills_router,
)
from litellm.proxy.anthropic_endpoints.claude_code_endpoints import (
claude_code_marketplace_router,
)
from litellm.proxy.auth.auth_checks import (
ExperimentalUIJWTToken,
get_team_object,
@ -334,6 +334,7 @@ from litellm.proxy.management_endpoints.model_management_endpoints import (
from litellm.proxy.management_endpoints.organization_endpoints import (
router as organization_router,
)
from litellm.proxy.management_endpoints.policy_endpoints import router as policy_router
from litellm.proxy.management_endpoints.router_settings_endpoints import (
router as router_settings_router,
)
@ -1115,8 +1116,13 @@ try:
# In development, we restructure directly in _experimental/out.
# In non-root Docker, we restructure in /var/lib/litellm/ui.
try:
_restructure_ui_html_files(ui_path)
verbose_proxy_logger.info(f"Restructured UI directory: {ui_path}")
if is_non_root and ui_path == "/var/lib/litellm/ui":
verbose_proxy_logger.info(
f"Skipping runtime UI restructuring for non-root Docker. UI at {ui_path} is pre-restructured."
)
else:
_restructure_ui_html_files(ui_path)
verbose_proxy_logger.info(f"Restructured UI directory: {ui_path}")
except PermissionError as e:
verbose_proxy_logger.exception(
f"Permission error while restructuring UI directory {ui_path}: {e}"
@ -2732,6 +2738,14 @@ class ProxyConfig:
for k, v in router_settings.items():
if k in available_args:
router_params[k] = v
elif k == "health_check_interval":
raise ValueError(
f"'{k}' is NOT a valid router_settings parameter. Please move it to 'general_settings'."
)
else:
verbose_proxy_logger.warning(
f"Key '{k}' is not a valid argument for Router.__init__(). Ignoring this key."
)
router = litellm.Router(
**router_params,
assistants_config=assistants_config,
@ -2757,6 +2771,13 @@ class ProxyConfig:
llm_router=router,
)
# Policy Engine settings
await self._init_policy_engine(
config=config,
prisma_client=prisma_client,
llm_router=router,
)
## Prompt settings
prompts: Optional[List[Dict]] = None
if config is not None:
@ -2817,6 +2838,45 @@ class ProxyConfig:
)
pass
async def _init_policy_engine(
self,
config: Optional[dict],
prisma_client: Optional["PrismaClient"],
llm_router: Optional["Router"],
):
"""
Initialize the policy engine from config.
Args:
config: The proxy configuration dictionary
prisma_client: Optional Prisma client for DB validation
llm_router: Optional LLM router for model validation
"""
from litellm.proxy.policy_engine.init_policies import init_policies
from litellm.proxy.policy_engine.policy_validator import PolicyValidator
if config is None:
verbose_proxy_logger.debug("Policy engine: config is None, skipping")
return
policies_config = config.get("policies", None)
if not policies_config:
verbose_proxy_logger.debug("Policy engine: no policies in config, skipping")
return
policy_attachments_config = config.get("policy_attachments", None)
verbose_proxy_logger.info(f"Policy engine: found {len(policies_config)} policies in config")
# Initialize policies
await init_policies(
policies_config=policies_config,
policy_attachments_config=policy_attachments_config,
prisma_client=prisma_client,
validate_db=prisma_client is not None,
fail_on_error=True,
)
def _load_alerting_settings(self, general_settings: dict):
"""
Initialize alerting settings
@ -5069,6 +5129,7 @@ async def model_list(
only_model_access_groups: Optional[bool] = False,
include_metadata: Optional[bool] = False,
fallback_type: Optional[str] = None,
scope: Optional[str] = None,
):
"""
Use `/model/info` - to get detailed model information, example - pricing, mode, etc.
@ -5079,14 +5140,85 @@ async def model_list(
- include_metadata: Include additional metadata in the response with fallback information
- fallback_type: Type of fallbacks to include ("general", "context_window", "content_policy")
Defaults to "general" when include_metadata=true
- scope: Optional scope parameter. Currently only accepts "expand".
When scope=expand is passed, proxy admins, team admins, and org admins
will receive all proxy models as if they are a proxy admin.
"""
global llm_model_list, general_settings, llm_router, prisma_client, user_api_key_cache, proxy_logging_obj
from litellm.proxy.management_endpoints.common_utils import (
_user_has_admin_privileges,
)
from litellm.proxy.utils import (
create_model_info_response,
get_available_models_for_user,
)
# Validate scope parameter if provided
if scope is not None and scope != "expand":
raise HTTPException(
status_code=400,
detail=f"Invalid scope parameter. Only 'expand' is currently supported. Received: {scope}",
)
# Check if scope=expand is requested and user has admin privileges
should_expand_scope = False
if scope == "expand":
should_expand_scope = await _user_has_admin_privileges(
user_api_key_dict=user_api_key_dict,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
# If scope=expand and user has admin privileges, return all proxy models
if should_expand_scope:
# Get all proxy models as if user is a proxy admin
if llm_router is None:
proxy_model_list = []
model_access_groups = {}
else:
proxy_model_list = llm_router.get_model_names()
model_access_groups = llm_router.get_model_access_groups()
# Include model access groups if requested
if include_model_access_groups:
proxy_model_list = list(set(proxy_model_list + list(model_access_groups.keys())))
# Get complete model list including wildcard routes if requested
from litellm.proxy.auth.model_checks import get_complete_model_list
all_models = get_complete_model_list(
key_models=[],
team_models=[],
proxy_model_list=proxy_model_list,
user_model=None,
infer_model_from_keys=False,
return_wildcard_routes=return_wildcard_routes or False,
llm_router=llm_router,
model_access_groups=model_access_groups,
include_model_access_groups=include_model_access_groups or False,
only_model_access_groups=only_model_access_groups or False,
)
# Build response data with all proxy models
model_data = []
for model in all_models:
model_info = create_model_info_response(
model_id=model,
provider="openai",
include_metadata=include_metadata or False,
fallback_type=fallback_type,
llm_router=llm_router,
)
model_data.append(model_info)
return dict(
data=model_data,
object="list",
)
# Otherwise, use the normal behavior (current implementation)
# Get available models for the user
all_models = await get_available_models_for_user(
user_api_key_dict=user_api_key_dict,
@ -10569,6 +10701,7 @@ app.include_router(cloudzero_router)
app.include_router(caching_router)
app.include_router(analytics_router)
app.include_router(guardrails_router)
app.include_router(policy_router)
app.include_router(search_tool_management_router)
app.include_router(prompts_router)
app.include_router(callback_management_endpoints_router)

457
litellm/responses/mcp/chat_completions_handler.py
View file

@ -5,6 +5,7 @@ from typing import (
List,
Optional,
Union,
cast,
)
from litellm.responses.mcp.litellm_proxy_mcp_handler import (
@ -15,7 +16,70 @@ from litellm.types.utils import ModelResponse
from litellm.utils import CustomStreamWrapper
async def acompletion_with_mcp(
def _add_mcp_metadata_to_response(
response: Union[ModelResponse, CustomStreamWrapper],
openai_tools: Optional[List],
tool_calls: Optional[List] = None,
tool_results: Optional[List] = None,
) -> None:
"""
Add MCP metadata to response's provider_specific_fields.
This function adds MCP-related information to the response so that
clients can access which tools were available, which were called, and
what results were returned.
For ModelResponse: adds to choices[].message.provider_specific_fields
For CustomStreamWrapper: stores in _hidden_params and automatically adds to
final chunk's delta.provider_specific_fields via CustomStreamWrapper._add_mcp_metadata_to_final_chunk()
"""
if isinstance(response, CustomStreamWrapper):
# For streaming, store MCP metadata in _hidden_params
# CustomStreamWrapper._add_mcp_metadata_to_final_chunk() will automatically
# add it to the final chunk's delta.provider_specific_fields
if not hasattr(response, "_hidden_params"):
response._hidden_params = {}
mcp_metadata = {}
if openai_tools:
mcp_metadata["mcp_list_tools"] = openai_tools
if tool_calls:
mcp_metadata["mcp_tool_calls"] = tool_calls
if tool_results:
mcp_metadata["mcp_call_results"] = tool_results
if mcp_metadata:
response._hidden_params["mcp_metadata"] = mcp_metadata
return
if not isinstance(response, ModelResponse):
return
if not hasattr(response, "choices") or not response.choices:
return
# Add MCP metadata to all choices' messages
for choice in response.choices:
message = getattr(choice, "message", None)
if message is not None:
# Get existing provider_specific_fields or create new dict
provider_fields = (
getattr(message, "provider_specific_fields", None) or {}
)
# Add MCP metadata
if openai_tools:
provider_fields["mcp_list_tools"] = openai_tools
if tool_calls:
provider_fields["mcp_tool_calls"] = tool_calls
if tool_results:
provider_fields["mcp_call_results"] = tool_results
# Set the provider_specific_fields
setattr(message, "provider_specific_fields", provider_fields)
async def acompletion_with_mcp( # noqa: PLR0915
model: str,
messages: List,
tools: Optional[List] = None,
@ -103,12 +167,375 @@ async def acompletion_with_mcp(
# If not auto-executing, just make the call with transformed tools
if not should_auto_execute:
return await litellm_acompletion(**base_call_args)
response = await litellm_acompletion(**base_call_args)
if isinstance(response, (ModelResponse, CustomStreamWrapper)):
_add_mcp_metadata_to_response(
response=response,
openai_tools=openai_tools,
)
return response
# For auto-execute: disable streaming for initial call
# For auto-execute: handle streaming vs non-streaming differently
stream = kwargs.get("stream", False)
mock_tool_calls = base_call_args.pop("mock_tool_calls", None)
if stream:
# Streaming mode: make initial call with streaming, collect chunks, detect tool calls
initial_call_args = dict(base_call_args)
initial_call_args["stream"] = True
if mock_tool_calls is not None:
initial_call_args["mock_tool_calls"] = mock_tool_calls
# Make initial streaming call
initial_stream = await litellm_acompletion(**initial_call_args)
if not isinstance(initial_stream, CustomStreamWrapper):
# Not a stream, return as-is
if isinstance(initial_stream, ModelResponse):
_add_mcp_metadata_to_response(
response=initial_stream,
openai_tools=openai_tools,
)
return initial_stream
# Create a custom async generator that collects chunks and handles tool execution
from litellm.main import stream_chunk_builder
from litellm.types.utils import ModelResponseStream
class MCPStreamingIterator:
"""Custom iterator that collects chunks, detects tool calls, and adds MCP metadata to final chunk."""
def __init__(self, stream_wrapper, messages, tool_server_map, user_api_key_auth,
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, raw_headers,
litellm_call_id, litellm_trace_id, openai_tools, base_call_args):
self.stream_wrapper = stream_wrapper
self.messages = messages
self.tool_server_map = tool_server_map
self.user_api_key_auth = user_api_key_auth
self.mcp_auth_header = mcp_auth_header
self.mcp_server_auth_headers = mcp_server_auth_headers
self.oauth2_headers = oauth2_headers
self.raw_headers = raw_headers
self.litellm_call_id = litellm_call_id
self.litellm_trace_id = litellm_trace_id
self.openai_tools = openai_tools
self.base_call_args = base_call_args
self.collected_chunks: List[ModelResponseStream] = []
self.tool_calls: Optional[List] = None
self.tool_results: Optional[List] = None
self.complete_response: Optional[ModelResponse] = None
self.stream_exhausted = False
self.tool_execution_done = False
self.follow_up_stream = None
self.follow_up_iterator = None
self.follow_up_exhausted = False
async def __aiter__(self):
return self
def _add_mcp_list_tools_to_chunk(self, chunk: ModelResponseStream) -> ModelResponseStream:
"""Add mcp_list_tools to the first chunk."""
from litellm.types.utils import StreamingChoices, add_provider_specific_fields
if not self.openai_tools:
return chunk
if hasattr(chunk, "choices") and chunk.choices:
for choice in chunk.choices:
if isinstance(choice, StreamingChoices) and hasattr(choice, "delta") and choice.delta:
# Get existing provider_specific_fields or create new dict
existing_fields = getattr(choice.delta, "provider_specific_fields", None) or {}
provider_fields = dict(existing_fields) # Create a copy to avoid mutating the original
# Add only mcp_list_tools to first chunk
provider_fields["mcp_list_tools"] = self.openai_tools
# Use add_provider_specific_fields to ensure proper setting
# This function handles Pydantic model attribute setting correctly
add_provider_specific_fields(choice.delta, provider_fields)
return chunk
def _add_mcp_tool_metadata_to_final_chunk(self, chunk: ModelResponseStream) -> ModelResponseStream:
"""Add mcp_tool_calls and mcp_call_results to the final chunk."""
from litellm.types.utils import StreamingChoices, add_provider_specific_fields
if hasattr(chunk, "choices") and chunk.choices:
for choice in chunk.choices:
if isinstance(choice, StreamingChoices) and hasattr(choice, "delta") and choice.delta:
# Get existing provider_specific_fields or create new dict
# Access the attribute directly to handle Pydantic model attributes correctly
existing_fields = {}
if hasattr(choice.delta, "provider_specific_fields"):
attr_value = getattr(choice.delta, "provider_specific_fields", None)
if attr_value is not None:
# Create a copy to avoid mutating the original
existing_fields = dict(attr_value) if isinstance(attr_value, dict) else {}
provider_fields = existing_fields
# Add tool_calls and tool_results if available
if self.tool_calls:
provider_fields["mcp_tool_calls"] = self.tool_calls
if self.tool_results:
provider_fields["mcp_call_results"] = self.tool_results
# Use add_provider_specific_fields to ensure proper setting
# This function handles Pydantic model attribute setting correctly
add_provider_specific_fields(choice.delta, provider_fields)
return chunk
async def __anext__(self):
# Phase 1: Collect and yield initial stream chunks
if not self.stream_exhausted:
# Get the iterator from the stream wrapper
if not hasattr(self, '_stream_iterator'):
self._stream_iterator = self.stream_wrapper.__aiter__()
# Add mcp_list_tools to the first chunk (available from the start)
_add_mcp_metadata_to_response(
response=self.stream_wrapper,
openai_tools=self.openai_tools,
)
try:
chunk = await self._stream_iterator.__anext__()
self.collected_chunks.append(chunk)
# Add mcp_list_tools to the first chunk
if len(self.collected_chunks) == 1:
chunk = self._add_mcp_list_tools_to_chunk(chunk)
# Check if this is the final chunk (has finish_reason)
is_final = (
hasattr(chunk, "choices")
and chunk.choices
and hasattr(chunk.choices[0], "finish_reason")
and chunk.choices[0].finish_reason is not None
)
if is_final:
# This is the final chunk, mark stream as exhausted
self.stream_exhausted = True
# Process tool calls after we've collected all chunks
await self._process_tool_calls()
# Apply MCP metadata (tool_calls and tool_results) to final chunk
chunk = self._add_mcp_tool_metadata_to_final_chunk(chunk)
# If we have tool results, prepare follow-up call immediately
if self.tool_results and self.complete_response:
await self._prepare_follow_up_call()
return chunk
except StopAsyncIteration:
self.stream_exhausted = True
# Process tool calls after stream is exhausted
await self._process_tool_calls()
# If we have chunks, yield the final one with metadata
if self.collected_chunks:
final_chunk = self.collected_chunks[-1]
final_chunk = self._add_mcp_tool_metadata_to_final_chunk(final_chunk)
# If we have tool results, prepare follow-up call
if self.tool_results and self.complete_response:
await self._prepare_follow_up_call()
return final_chunk
# Phase 2: Yield follow-up stream chunks if available
if self.follow_up_stream and not self.follow_up_exhausted:
if not self.follow_up_iterator:
self.follow_up_iterator = self.follow_up_stream.__aiter__()
from litellm._logging import verbose_logger
verbose_logger.debug("Follow-up stream iterator created")
try:
chunk = await self.follow_up_iterator.__anext__()
from litellm._logging import verbose_logger
verbose_logger.debug(f"Follow-up chunk yielded: {chunk}")
return chunk
except StopAsyncIteration:
self.follow_up_exhausted = True
from litellm._logging import verbose_logger
verbose_logger.debug("Follow-up stream exhausted")
# After follow-up stream is exhausted, check if we need to raise StopAsyncIteration
raise StopAsyncIteration
# If we're here and follow_up_stream is None but we expected it, log a warning
if self.stream_exhausted and self.tool_results and self.complete_response and self.follow_up_stream is None:
from litellm._logging import verbose_logger
verbose_logger.warning(
"Follow-up stream was not created despite having tool results"
)
raise StopAsyncIteration
async def _process_tool_calls(self):
"""Process tool calls after streaming completes."""
if self.tool_execution_done:
return
self.tool_execution_done = True
if not self.collected_chunks:
return
# Build complete response from chunks
complete_response = stream_chunk_builder(
chunks=self.collected_chunks,
messages=self.messages,
)
if isinstance(complete_response, ModelResponse):
self.complete_response = complete_response
# Extract tool calls from complete response
self.tool_calls = LiteLLM_Proxy_MCP_Handler._extract_tool_calls_from_chat_response(
response=complete_response
)
if self.tool_calls:
# Execute tool calls
self.tool_results = await LiteLLM_Proxy_MCP_Handler._execute_tool_calls(
tool_server_map=self.tool_server_map,
tool_calls=self.tool_calls,
user_api_key_auth=self.user_api_key_auth,
mcp_auth_header=self.mcp_auth_header,
mcp_server_auth_headers=self.mcp_server_auth_headers,
oauth2_headers=self.oauth2_headers,
raw_headers=self.raw_headers,
litellm_call_id=self.litellm_call_id,
litellm_trace_id=self.litellm_trace_id,
)
async def _prepare_follow_up_call(self):
"""Prepare and initiate follow-up call with tool results."""
if self.follow_up_stream is not None:
return # Already prepared
if not self.tool_results or not self.complete_response:
return
# Create follow-up messages with tool results
follow_up_messages = LiteLLM_Proxy_MCP_Handler._create_follow_up_messages_for_chat(
original_messages=self.messages,
response=self.complete_response,
tool_results=self.tool_results,
)
# Make follow-up call with streaming
follow_up_call_args = dict(self.base_call_args)
follow_up_call_args["messages"] = follow_up_messages
follow_up_call_args["stream"] = True
# Ensure follow-up call doesn't trigger MCP handler again
follow_up_call_args["_skip_mcp_handler"] = True
# Import litellm here to ensure we get the patched version
# This ensures the patch works correctly in tests
import litellm
follow_up_response = await litellm.acompletion(**follow_up_call_args)
# Ensure follow-up response is a CustomStreamWrapper
if isinstance(follow_up_response, CustomStreamWrapper):
self.follow_up_stream = follow_up_response
from litellm._logging import verbose_logger
verbose_logger.debug("Follow-up stream created successfully")
else:
# Unexpected response type - log and set to None
from litellm._logging import verbose_logger
verbose_logger.warning(
f"Follow-up response is not a CustomStreamWrapper: {type(follow_up_response)}"
)
self.follow_up_stream = None
# Create the custom iterator
iterator = MCPStreamingIterator(
stream_wrapper=initial_stream,
messages=messages,
tool_server_map=tool_server_map,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_call_id=kwargs.get("litellm_call_id"),
litellm_trace_id=kwargs.get("litellm_trace_id"),
openai_tools=openai_tools,
base_call_args=base_call_args,
)
# Create a wrapper class that delegates to our custom iterator
# We'll use a simple approach: just replace the __aiter__ method
class MCPStreamWrapper(CustomStreamWrapper):
def __init__(self, original_wrapper, custom_iterator):
# Initialize with the same parameters as original wrapper
super().__init__(
completion_stream=None,
model=getattr(original_wrapper, "model", "unknown"),
logging_obj=getattr(original_wrapper, "logging_obj", None),
custom_llm_provider=getattr(original_wrapper, "custom_llm_provider", None),
stream_options=getattr(original_wrapper, "stream_options", None),
make_call=getattr(original_wrapper, "make_call", None),
_response_headers=getattr(original_wrapper, "_response_headers", None),
)
self._original_wrapper = original_wrapper
self._custom_iterator = custom_iterator
# Copy important attributes from original wrapper
if hasattr(original_wrapper, "_hidden_params"):
self._hidden_params = original_wrapper._hidden_params
# For synchronous iteration, we need to run the async iterator
self._sync_iterator = None
self._sync_loop = None
def __aiter__(self):
return self._custom_iterator
def __iter__(self):
# For synchronous iteration, create a sync wrapper
if self._sync_iterator is None:
import asyncio
try:
self._sync_loop = asyncio.get_event_loop()
except RuntimeError:
self._sync_loop = asyncio.new_event_loop()
asyncio.set_event_loop(self._sync_loop)
self._sync_iterator = _SyncIteratorWrapper(self._custom_iterator, self._sync_loop)
return self._sync_iterator
def __next__(self):
# Delegate to sync iterator
if self._sync_iterator is None:
self.__iter__()
return next(self._sync_iterator)
def __getattr__(self, name):
# Delegate all other attributes to original wrapper
return getattr(self._original_wrapper, name)
# Helper class to wrap async iterator for sync iteration
class _SyncIteratorWrapper:
def __init__(self, async_iterator, loop):
self._async_iterator = async_iterator
self._loop = loop
self._iterator = None
def __iter__(self):
return self
def __next__(self):
if self._iterator is None:
# __aiter__ might be async, so we need to await it
aiter_result = self._async_iterator.__aiter__()
if hasattr(aiter_result, '__await__'):
# It's a coroutine, await it
self._iterator = self._loop.run_until_complete(aiter_result)
else:
# It's already an iterator
self._iterator = aiter_result
try:
return self._loop.run_until_complete(self._iterator.__anext__())
except StopAsyncIteration:
raise StopIteration
return cast(CustomStreamWrapper, MCPStreamWrapper(initial_stream, iterator))
# Non-streaming mode: use existing logic
initial_call_args = dict(base_call_args)
initial_call_args["stream"] = False
if mock_tool_calls is not None:
@ -126,11 +553,10 @@ async def acompletion_with_mcp(
)
if not tool_calls:
# No tool calls, return response or retry with streaming if needed
if stream:
retry_args = dict(base_call_args)
retry_args["stream"] = stream
return await litellm_acompletion(**retry_args)
_add_mcp_metadata_to_response(
response=initial_response,
openai_tools=openai_tools,
)
return initial_response
# Execute tool calls
@ -147,6 +573,11 @@ async def acompletion_with_mcp(
)
if not tool_results:
_add_mcp_metadata_to_response(
response=initial_response,
openai_tools=openai_tools,
tool_calls=tool_calls,
)
return initial_response
# Create follow-up messages with tool results
@ -161,4 +592,12 @@ async def acompletion_with_mcp(
follow_up_call_args["messages"] = follow_up_messages
follow_up_call_args["stream"] = stream
return await litellm_acompletion(**follow_up_call_args)
response = await litellm_acompletion(**follow_up_call_args)
if isinstance(response, (ModelResponse, CustomStreamWrapper)):
_add_mcp_metadata_to_response(
response=response,
openai_tools=openai_tools,
tool_calls=tool_calls,
tool_results=tool_results,
)
return response

6
litellm/responses/mcp/litellm_proxy_mcp_handler.py
View file

@ -775,6 +775,12 @@ class LiteLLM_Proxy_MCP_Handler:
first_choice, "message", None
):
message_to_append = first_choice.message.model_dump(exclude_none=True)
# Ensure tool_calls have arguments field (required by OpenAI API)
if message_to_append.get("tool_calls"):
for tool_call in message_to_append["tool_calls"]:
if isinstance(tool_call, dict) and "function" in tool_call:
if "arguments" not in tool_call["function"]:
tool_call["function"]["arguments"] = "{}"
except Exception:
verbose_logger.exception("Failed to convert assistant message for MCP flow")

1
litellm/types/llms/anthropic.py
View file

@ -359,6 +359,7 @@ class AnthropicMessagesRequestOptionalParams(TypedDict, total=False):
mcp_servers: Optional[List[AnthropicMcpServerTool]]
context_management: Optional[Dict[str, Any]]
container: Optional[Dict[str, Any]] # Container config with skills for code execution
output_format: Optional[AnthropicOutputSchema] # Structured outputs support
class AnthropicMessagesRequest(AnthropicMessagesRequestOptionalParams, total=False):

14
litellm/types/llms/openai.py
View file

@ -71,7 +71,7 @@ from openai.types.responses.response_create_params import (
ToolParam,
)
from openai.types.responses.response_function_tool_call import ResponseFunctionToolCall
from pydantic import BaseModel, ConfigDict, Discriminator, PrivateAttr
from pydantic import BaseModel, ConfigDict, Discriminator, PrivateAttr, field_validator
from typing_extensions import Annotated, Dict, Required, TypedDict, override
from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
@ -1199,6 +1199,16 @@ class ResponsesAPIResponse(BaseLiteLLMOpenAIResponseObject):
# Define private attributes using PrivateAttr
_hidden_params: dict = PrivateAttr(default_factory=dict)
@field_validator("usage", mode="before")
@classmethod
def validate_usage(cls, value):
"""Convert usage dict to ResponseAPIUsage object if needed"""
if value is None:
return value
if isinstance(value, dict):
return ResponseAPIUsage(**value)
return value
@property
def output_text(self) -> str:
"""
@ -2002,7 +2012,7 @@ class OpenAIBatchResult(TypedDict, total=False):
OpenAIChatCompletionFinishReason = Literal[
"stop", "content_filter", "function_call", "tool_calls", "length"
"stop", "content_filter", "function_call", "tool_calls", "length", "guardrail_intervened", "eos", "finish_reason_unspecified", "malformed_function_call" # last 2 are vertex ai specific, guardrail_intervened is bedrock specific
]

2
litellm/types/llms/vertex_ai.py
View file

@ -396,6 +396,8 @@ class Candidates(TypedDict, total=False):
"BLOCKLIST",
"PROHIBITED_CONTENT",
"SPII",
"MALFORMED_FUNCTION_CALL",
"IMAGE_SAFETY",
]
safetyRatings: List[SafetyRatings]
citationMetadata: CitationMetadata

36
litellm/types/policy_engine.py Normal file
View file

@ -0,0 +1,36 @@
"""
Type definitions for the LiteLLM Policy Engine.
This module re-exports types from litellm.types.proxy.policy_engine for backward compatibility.
The canonical location for these types is litellm/types/proxy/policy_engine/.
"""
# Re-export all types from the new location
from litellm.types.proxy.policy_engine import ( # Policy types; Validation types; Resolver types
Policy,
PolicyConfig,
PolicyGuardrails,
PolicyMatchContext,
PolicyScope,
PolicyValidateRequest,
PolicyValidationError,
PolicyValidationErrorType,
PolicyValidationResponse,
ResolvedPolicy,
)
__all__ = [
# Policy types
"Policy",
"PolicyConfig",
"PolicyGuardrails",
"PolicyScope",
# Validation types
"PolicyValidateRequest",
"PolicyValidationError",
"PolicyValidationErrorType",
"PolicyValidationResponse",
# Resolver types
"PolicyMatchContext",
"ResolvedPolicy",
]

61
litellm/types/proxy/policy_engine/__init__.py Normal file
View file

@ -0,0 +1,61 @@
"""
Type definitions for the LiteLLM Policy Engine.
The Policy Engine allows administrators to define policies that combine guardrails
with scoping rules. Policies can target specific teams, API keys, and models using
wildcard patterns, and support inheritance from base policies.
Configuration:
- `policies`: Define WHAT guardrails to apply (with inheritance and conditions)
- `policy_attachments`: Define WHERE policies apply (teams, keys, models)
"""
from litellm.types.proxy.policy_engine.policy_types import (
Policy,
PolicyAttachment,
PolicyCondition,
PolicyConfig,
PolicyGuardrails,
PolicyScope,
)
from litellm.types.proxy.policy_engine.resolver_types import (
PolicyGuardrailsResponse,
PolicyInfoResponse,
PolicyListResponse,
PolicyMatchContext,
PolicyScopeResponse,
PolicySummaryItem,
PolicyTestResponse,
ResolvedPolicy,
)
from litellm.types.proxy.policy_engine.validation_types import (
PolicyValidateRequest,
PolicyValidationError,
PolicyValidationErrorType,
PolicyValidationResponse,
)
__all__ = [
# Policy types
"Policy",
"PolicyConfig",
"PolicyGuardrails",
"PolicyScope",
"PolicyCondition",
"PolicyAttachment",
# Validation types
"PolicyValidateRequest",
"PolicyValidationError",
"PolicyValidationErrorType",
"PolicyValidationResponse",
# Resolver types
"PolicyMatchContext",
"ResolvedPolicy",
# API Response types
"PolicyGuardrailsResponse",
"PolicyInfoResponse",
"PolicyListResponse",
"PolicyScopeResponse",
"PolicySummaryItem",
"PolicyTestResponse",
]

299
litellm/types/proxy/policy_engine/policy_types.py Normal file
View file

@ -0,0 +1,299 @@
"""
Core policy type definitions.
Policy Engine Configuration:
```yaml
policies:
global-baseline:
description: "Base guardrails for all requests"
guardrails:
add: [pii_blocker]
healthcare-compliance:
inherit: global-baseline
guardrails:
add: [hipaa_audit]
condition:
model: "gpt-4" # exact match or regex pattern
policy_attachments:
- policy: global-baseline
scope: "*"
- policy: healthcare-compliance
teams: [healthcare-team]
```
Key concepts:
- `policies`: Define WHAT guardrails to apply (with inheritance via `inherit` and `guardrails.add`/`remove`)
- `policy_attachments`: Define WHERE policies apply (teams, keys, models)
- `condition`: Optional model condition for when guardrails apply
"""
from typing import Any, Dict, List, Optional, Union
from pydantic import BaseModel, ConfigDict, Field
# ─────────────────────────────────────────────────────────────────────────────
# Policy Condition
# ─────────────────────────────────────────────────────────────────────────────
class PolicyCondition(BaseModel):
"""
Condition for when a policy's guardrails apply.
Currently supports model-based conditions with exact match or regex.
Example YAML:
```yaml
condition:
model: "gpt-4" # exact match
model: "gpt-4.*" # regex pattern
model: ["gpt-4", "gpt-4-turbo"] # list of exact matches
```
"""
model: Optional[Union[str, List[str]]] = Field(
default=None,
description="Model name(s) to match. Can be exact string, regex pattern, or list.",
)
model_config = ConfigDict(extra="forbid")
# ─────────────────────────────────────────────────────────────────────────────
# Policy Scope (used internally by attachments)
# ─────────────────────────────────────────────────────────────────────────────
class PolicyScope(BaseModel):
"""
Defines the scope for matching requests.
Used internally by PolicyAttachment to define WHERE a policy applies.
Scope Fields:
| Field | What it matches | Wildcard support |
|--------|-----------------|----------------------|
| teams | Team aliases | *, healthcare-* |
| keys | Key aliases | *, dev-key-* |
| models | Model names | *, bedrock/*, gpt-* |
If a field is None or empty, it defaults to matching everything (["*"]).
A request must match ALL specified scope fields for the attachment to apply.
"""
teams: Optional[List[str]] = Field(
default=None,
description="Team aliases or wildcard patterns. Use '*' for all teams.",
)
keys: Optional[List[str]] = Field(
default=None,
description="Key aliases or wildcard patterns. Use '*' for all keys.",
)
models: Optional[List[str]] = Field(
default=None,
description="Model names or wildcard patterns. Use '*' for all models.",
)
model_config = ConfigDict(extra="forbid")
def get_teams(self) -> List[str]:
"""Returns teams list, defaulting to ['*'] if not specified."""
return self.teams if self.teams else ["*"]
def get_keys(self) -> List[str]:
"""Returns keys list, defaulting to ['*'] if not specified."""
return self.keys if self.keys else ["*"]
def get_models(self) -> List[str]:
"""Returns models list, defaulting to ['*'] if not specified."""
return self.models if self.models else ["*"]
# ─────────────────────────────────────────────────────────────────────────────
# Policy Guardrails
# ─────────────────────────────────────────────────────────────────────────────
class PolicyGuardrails(BaseModel):
"""
Defines guardrails to add or remove in a policy.
- `add`: List of guardrail names to add (on top of inherited guardrails)
- `remove`: List of guardrail names to remove (from inherited guardrails)
This supports the inheritance pattern where child policies can:
- Add new guardrails on top of parent's guardrails
- Remove specific guardrails inherited from parent
"""
add: Optional[List[str]] = Field(
default=None,
description="Guardrail names to add to this policy.",
)
remove: Optional[List[str]] = Field(
default=None,
description="Guardrail names to remove (typically from inherited policy).",
)
model_config = ConfigDict(extra="forbid")
def get_add(self) -> List[str]:
"""Returns add list, defaulting to empty list if not specified."""
return self.add if self.add else []
def get_remove(self) -> List[str]:
"""Returns remove list, defaulting to empty list if not specified."""
return self.remove if self.remove else []
# ─────────────────────────────────────────────────────────────────────────────
# Policy
# ─────────────────────────────────────────────────────────────────────────────
class Policy(BaseModel):
"""
A policy that defines WHAT guardrails to apply.
Policies define guardrails but NOT where they apply - that's done via policy_attachments.
Policies can inherit from other policies using the `inherit` field.
When inheriting:
- Guardrails from `guardrails.add` are added to the inherited guardrails
- Guardrails from `guardrails.remove` are removed from the inherited guardrails
Policies can have a `condition` for model-based guardrail application.
Example configuration:
```yaml
policies:
global-baseline:
description: "Base guardrails for all requests"
guardrails:
add:
- pii_blocker
- phi_blocker
healthcare-compliance:
inherit: global-baseline
description: "HIPAA compliance for healthcare"
guardrails:
add:
- hipaa_audit
gpt4-safety:
description: "Extra safety for GPT-4 models"
guardrails:
add:
- toxicity_filter
condition:
model: "gpt-4.*" # regex pattern
policy_attachments:
- policy: global-baseline
scope: "*"
- policy: healthcare-compliance
teams: [healthcare-team]
- policy: gpt4-safety
scope: "*"
```
"""
inherit: Optional[str] = Field(
default=None,
description="Name of the parent policy to inherit from.",
)
description: Optional[str] = Field(
default=None,
description="Human-readable description of the policy.",
)
guardrails: PolicyGuardrails = Field(
default_factory=PolicyGuardrails,
description="Guardrails configuration with add/remove lists.",
)
condition: Optional[PolicyCondition] = Field(
default=None,
description="Optional condition for when this policy's guardrails apply.",
)
model_config = ConfigDict(extra="forbid")
# ─────────────────────────────────────────────────────────────────────────────
# Policy Attachments
# ─────────────────────────────────────────────────────────────────────────────
class PolicyAttachment(BaseModel):
"""
Attaches a policy to a scope - defines WHERE a policy applies.
Attachments are REQUIRED to make policies active. A policy without
an attachment will not be applied to any requests.
Example YAML:
```yaml
policy_attachments:
- policy: global-baseline
scope: "*" # applies to all requests
- policy: healthcare-compliance
teams: [healthcare-team, medical-research]
- policy: dev-safety
keys: ["dev-key-*", "test-key-*"]
- policy: gpt4-specific
models: ["gpt-4", "gpt-4-turbo"]
```
"""
policy: str = Field(
description="Name of the policy to attach.",
)
scope: Optional[str] = Field(
default=None,
description="Use '*' for global scope (applies to all requests).",
)
teams: Optional[List[str]] = Field(
default=None,
description="Team aliases or patterns this attachment applies to.",
)
keys: Optional[List[str]] = Field(
default=None,
description="Key aliases or patterns this attachment applies to.",
)
models: Optional[List[str]] = Field(
default=None,
description="Model names or patterns this attachment applies to.",
)
model_config = ConfigDict(extra="forbid")
def is_global(self) -> bool:
"""Check if this is a global attachment (scope='*')."""
return self.scope == "*"
def to_policy_scope(self) -> PolicyScope:
"""Convert attachment to a PolicyScope for matching."""
if self.is_global():
return PolicyScope(teams=["*"], keys=["*"], models=["*"])
return PolicyScope(
teams=self.teams,
keys=self.keys,
models=self.models,
)
class PolicyConfig(BaseModel):
"""
Root configuration for all policies.
Maps policy names to their Policy definitions.
"""
policies: Dict[str, Policy] = Field(
default_factory=dict,
description="Map of policy names to Policy objects.",
)
model_config = ConfigDict(extra="forbid")

110
litellm/types/proxy/policy_engine/resolver_types.py Normal file
View file

@ -0,0 +1,110 @@
"""
Policy resolver type definitions.
These types are used for matching requests to policies and resolving
the final guardrails list.
"""
from typing import Dict, List, Optional
from pydantic import BaseModel, ConfigDict, Field
class PolicyMatchContext(BaseModel):
"""
Context used to match a request against policies.
Contains the team alias, key alias, and model from the incoming request.
"""
team_alias: Optional[str] = Field(
default=None,
description="Team alias from the request.",
)
key_alias: Optional[str] = Field(
default=None,
description="API key alias from the request.",
)
model: Optional[str] = Field(
default=None,
description="Model name from the request.",
)
model_config = ConfigDict(extra="forbid")
class ResolvedPolicy(BaseModel):
"""
Result of resolving a policy with its inheritance chain.
Contains the final list of guardrails after applying all add/remove operations.
"""
policy_name: str = Field(description="Name of the resolved policy.")
guardrails: List[str] = Field(
default_factory=list,
description="Final list of guardrail names to apply.",
)
inheritance_chain: List[str] = Field(
default_factory=list,
description="List of policy names in the inheritance chain (from root to this policy).",
)
model_config = ConfigDict(extra="forbid")
# ─────────────────────────────────────────────────────────────────────────────
# API Response Types
# ─────────────────────────────────────────────────────────────────────────────
class PolicyScopeResponse(BaseModel):
"""Scope configuration for a policy."""
teams: List[str] = Field(default_factory=list)
keys: List[str] = Field(default_factory=list)
models: List[str] = Field(default_factory=list)
class PolicyGuardrailsResponse(BaseModel):
"""Guardrails configuration for a policy."""
add: List[str] = Field(default_factory=list)
remove: List[str] = Field(default_factory=list)
class PolicyInfoResponse(BaseModel):
"""Response for /policy/info/{policy_name} endpoint."""
policy_name: str
inherit: Optional[str] = None
scope: PolicyScopeResponse
guardrails: PolicyGuardrailsResponse
resolved_guardrails: List[str]
inheritance_chain: List[str]
class PolicySummaryItem(BaseModel):
"""Summary of a single policy for list endpoint."""
inherit: Optional[str] = None
scope: PolicyScopeResponse
guardrails: PolicyGuardrailsResponse
resolved_guardrails: List[str]
inheritance_chain: List[str]
class PolicyListResponse(BaseModel):
"""Response for /policy/list endpoint."""
policies: Dict[str, PolicySummaryItem]
total_count: int
class PolicyTestResponse(BaseModel):
"""Response for /policy/test endpoint."""
context: PolicyMatchContext
matching_policies: List[str]
resolved_guardrails: List[str]
message: Optional[str] = None

80
litellm/types/proxy/policy_engine/validation_types.py Normal file
View file

@ -0,0 +1,80 @@
"""
Policy validation type definitions.
These types are used for validating policy configurations and returning
validation results.
"""
from enum import Enum
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, ConfigDict, Field
class PolicyValidationErrorType(str, Enum):
"""Types of validation errors that can occur."""
INVALID_GUARDRAIL = "invalid_guardrail"
INVALID_TEAM = "invalid_team"
INVALID_KEY = "invalid_key"
INVALID_MODEL = "invalid_model"
INVALID_INHERITANCE = "invalid_inheritance"
CIRCULAR_INHERITANCE = "circular_inheritance"
INVALID_SCOPE = "invalid_scope"
INVALID_SYNTAX = "invalid_syntax"
class PolicyValidationError(BaseModel):
"""
Represents a validation error or warning for a policy.
"""
policy_name: str = Field(description="Name of the policy with the issue.")
error_type: PolicyValidationErrorType = Field(
description="Type of validation error."
)
message: str = Field(description="Human-readable error message.")
field: Optional[str] = Field(
default=None,
description="Specific field that caused the error (e.g., 'guardrails.add', 'scope.teams').",
)
value: Optional[str] = Field(
default=None,
description="The invalid value that caused the error.",
)
model_config = ConfigDict(extra="forbid")
class PolicyValidationResponse(BaseModel):
"""
Response from policy validation.
- `valid`: True if no blocking errors were found
- `errors`: List of blocking errors (prevent policy from being applied)
- `warnings`: List of non-blocking warnings (policy can still be applied)
"""
valid: bool = Field(description="True if the policy configuration is valid.")
errors: List[PolicyValidationError] = Field(
default_factory=list,
description="List of blocking validation errors.",
)
warnings: List[PolicyValidationError] = Field(
default_factory=list,
description="List of non-blocking validation warnings.",
)
model_config = ConfigDict(extra="forbid")
class PolicyValidateRequest(BaseModel):
"""
Request body for the /policy/validate endpoint.
"""
policies: Dict[str, Any] = Field(
description="Policy configuration to validate. Map of policy names to policy definitions."
)
model_config = ConfigDict(extra="forbid")

4
litellm/types/utils.py
View file

@ -46,6 +46,7 @@ from .llms.openai import (
FineTuningJob,
ImageURLListItem,
OpenAIChatCompletionChunk,
OpenAIChatCompletionFinishReason,
OpenAIFileObject,
OpenAIRealtimeStreamList,
ResponsesAPIResponse,
@ -1254,7 +1255,7 @@ class Delta(SafeAttributeModel, OpenAIObject):
class Choices(SafeAttributeModel, OpenAIObject):
finish_reason: str
finish_reason: OpenAIChatCompletionFinishReason
index: int
message: Message
logprobs: Optional[Union[ChoiceLogprobs, Any]] = None
@ -3119,6 +3120,7 @@ class SearchProviders(str, Enum):
TAVILY = "tavily"
PARALLEL_AI = "parallel_ai"
EXA_AI = "exa_ai"
BRAVE = "brave"
GOOGLE_PSE = "google_pse"
DATAFORSEO = "dataforseo"
FIRECRAWL = "firecrawl"

40
litellm/utils.py
View file

@ -4649,7 +4649,9 @@ def add_provider_specific_params_to_optional_params(
else:
for k in passed_params.keys():
if k not in openai_params and passed_params[k] is not None:
if _should_drop_param(k=k, additional_drop_params=additional_drop_params):
if _should_drop_param(
k=k, additional_drop_params=additional_drop_params
):
continue
optional_params[k] = passed_params[k]
return optional_params
@ -5777,6 +5779,14 @@ def get_model_info(model: str, custom_llm_provider: Optional[str] = None) -> Mod
custom_llm_provider=custom_llm_provider,
)
provider_info = get_provider_info(
model=model, custom_llm_provider=custom_llm_provider
)
if provider_info:
for key, value in provider_info.items():
if value is not None:
_model_info[key] = value # type: ignore
verbose_logger.debug(f"model_info: {_model_info}")
returned_model_info = ModelInfo(
@ -7697,6 +7707,27 @@ def validate_chat_completion_tool_choice(
f"Invalid tool choice, tool_choice={tool_choice}. Got={type(tool_choice)}. Expecting str, or dict. Please ensure tool_choice follows the OpenAI tool_choice spec"
)
def validate_openai_optional_params(
stop: Optional[Union[str, List[str]]] = None,
**kwargs
) -> Optional[Union[str, List[str]]]:
"""
Validates and fixes OpenAI optional parameters.
Args:
stop: Stop sequences (string or list of strings)
**kwargs: Additional optional parameters
Returns:
Validated stop parameter (truncated to 4 elements if needed)
"""
if stop is not None and isinstance(stop, list) and not litellm.disable_stop_sequence_limit:
# Truncate to 4 elements if more are provided as openai only supports up to 4 stop sequences
if len(stop) > 4:
stop = stop[:4]
return stop
class ProviderConfigManager:
# Dictionary mapping for O(1) provider lookup
@ -8153,7 +8184,10 @@ class ProviderConfigManager:
# Note: GPT models (gpt-3.5, gpt-4, gpt-5, etc.) support temperature parameter
# O-series models (o1, o3) do not contain "gpt" and have different parameter restrictions
is_gpt_model = model and "gpt" in model.lower()
is_o_series = model and ("o_series" in model.lower() or (supports_reasoning(model) and not is_gpt_model))
is_o_series = model and (
"o_series" in model.lower()
or (supports_reasoning(model) and not is_gpt_model)
)
is_o_series = model and (
"o_series" in model.lower()
@ -8654,6 +8688,7 @@ class ProviderConfigManager:
"""
from litellm.llms.dataforseo.search.transformation import DataForSEOSearchConfig
from litellm.llms.exa_ai.search.transformation import ExaAISearchConfig
from litellm.llms.brave.search.transformation import BraveSearchConfig
from litellm.llms.firecrawl.search.transformation import FirecrawlSearchConfig
from litellm.llms.google_pse.search.transformation import GooglePSESearchConfig
from litellm.llms.linkup.search.transformation import LinkupSearchConfig
@ -8669,6 +8704,7 @@ class ProviderConfigManager:
SearchProviders.TAVILY: TavilySearchConfig,
SearchProviders.PARALLEL_AI: ParallelAISearchConfig,
SearchProviders.EXA_AI: ExaAISearchConfig,
SearchProviders.BRAVE: BraveSearchConfig,
SearchProviders.GOOGLE_PSE: GooglePSESearchConfig,
SearchProviders.DATAFORSEO: DataForSEOSearchConfig,
SearchProviders.FIRECRAWL: FirecrawlSearchConfig,

375
model_prices_and_context_window.json
View file

@ -1312,6 +1312,9 @@
"supports_function_calling": true
},
"azure_ai/claude-haiku-4-5": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_creation_input_token_cost_above_1hr": 2e-06,
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1330,6 +1333,9 @@
"supports_vision": true
},
"azure_ai/claude-opus-4-5": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1348,6 +1354,9 @@
"supports_vision": true
},
"azure_ai/claude-opus-4-1": {
"cache_creation_input_token_cost": 1.875e-05,
"cache_creation_input_token_cost_above_1hr": 3e-05,
"cache_read_input_token_cost": 1.5e-06,
"input_cost_per_token": 1.5e-05,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -1366,6 +1375,9 @@
"supports_vision": true
},
"azure_ai/claude-sonnet-4-5": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 200000,
@ -16094,6 +16106,181 @@
"output_cost_per_token": 0.0,
"output_vector_size": 2560
},
"gmi/anthropic/claude-opus-4.5": {
"input_cost_per_token": 5e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-sonnet-4.5": {
"input_cost_per_token": 3e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-sonnet-4": {
"input_cost_per_token": 3e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/anthropic/claude-opus-4": {
"input_cost_per_token": 1.5e-05,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 7.5e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/openai/gpt-5.2": {
"input_cost_per_token": 1.75e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.4e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-5.1": {
"input_cost_per_token": 1.25e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-5": {
"input_cost_per_token": 1.25e-06,
"litellm_provider": "gmi",
"max_input_tokens": 409600,
"max_output_tokens": 32000,
"max_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true
},
"gmi/openai/gpt-4o": {
"input_cost_per_token": 2.5e-06,
"litellm_provider": "gmi",
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/openai/gpt-4o-mini": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "gmi",
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/deepseek-ai/DeepSeek-V3.2": {
"input_cost_per_token": 2.8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 163840,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_function_calling": true
},
"gmi/deepseek-ai/DeepSeek-V3-0324": {
"input_cost_per_token": 2.8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 163840,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 8.8e-07,
"supports_function_calling": true
},
"gmi/google/gemini-3-pro-preview": {
"input_cost_per_token": 2e-06,
"litellm_provider": "gmi",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/google/gemini-3-flash-preview": {
"input_cost_per_token": 5e-07,
"litellm_provider": "gmi",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 3e-06,
"supports_function_calling": true,
"supports_vision": true
},
"gmi/moonshotai/Kimi-K2-Thinking": {
"input_cost_per_token": 8e-07,
"litellm_provider": "gmi",
"max_input_tokens": 262144,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"gmi/MiniMaxAI/MiniMax-M2.1": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gmi",
"max_input_tokens": 196608,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"gmi/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gmi",
"max_input_tokens": 262144,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.4e-06,
"supports_vision": true
},
"gmi/zai-org/GLM-4.7-FP8": {
"input_cost_per_token": 4e-07,
"litellm_provider": "gmi",
"max_input_tokens": 202752,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 2e-06
},
"google.gemma-3-12b-it": {
"input_cost_per_token": 9e-08,
"litellm_provider": "bedrock_converse",
@ -16863,14 +17050,14 @@
"supports_vision": true
},
"gpt-4o-audio-preview": {
"input_cost_per_audio_token": 0.0001,
"input_cost_per_audio_token": 4e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 0.0002,
"output_cost_per_audio_token": 8e-05,
"output_cost_per_token": 1e-05,
"supports_audio_input": true,
"supports_audio_output": true,
@ -16880,14 +17067,14 @@
"supports_tool_choice": true
},
"gpt-4o-audio-preview-2024-10-01": {
"input_cost_per_audio_token": 0.0001,
"input_cost_per_audio_token": 4e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 0.0002,
"output_cost_per_audio_token": 8e-05,
"output_cost_per_token": 1e-05,
"supports_audio_input": true,
"supports_audio_output": true,
@ -16930,6 +17117,186 @@
"supports_system_messages": true,
"supports_tool_choice": true
},
"gpt-audio": {
"input_cost_per_audio_token": 3.2e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 6.4e-05,
"output_cost_per_token": 1e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-2025-08-28": {
"input_cost_per_audio_token": 3.2e-05,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 6.4e-05,
"output_cost_per_token": 1e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini-2025-10-06": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-audio-mini-2025-12-15": {
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses",
"/v1/realtime",
"/v1/batch"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"gpt-4o-mini": {
"cache_read_input_token_cost": 7.5e-08,
"cache_read_input_token_cost_priority": 1.25e-07,

24
poetry.lock generated
View file

@ -1,4 +1,4 @@
# This file is automatically @generated by Poetry 2.1.4 and should not be changed by hand.
# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand.
[[package]]
name = "a2a-sdk"
@ -902,7 +902,7 @@ files = [
{file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"},
{file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"},
]
markers = {main = "(extra == \"utils\" or extra == \"semantic-router\" or platform_system == \"Windows\") and python_version < \"3.14\" and (sys_platform == \"win32\" or platform_system == \"Windows\" or extra == \"semantic-router\") or (extra == \"utils\" and sys_platform == \"win32\" or platform_system == \"Windows\") and python_version >= \"3.14\"", dev = "platform_system == \"Windows\" or sys_platform == \"win32\"", proxy-dev = "platform_system == \"Windows\""}
markers = {main = "platform_system == \"Windows\" or sys_platform == \"win32\" and python_version < \"3.14\" and (extra == \"utils\" or extra == \"semantic-router\") or sys_platform == \"win32\" and extra == \"utils\" or python_version < \"3.14\" and extra == \"semantic-router\"", dev = "platform_system == \"Windows\" or sys_platform == \"win32\"", proxy-dev = "platform_system == \"Windows\""}
[[package]]
name = "coloredlogs"
@ -2204,6 +2204,8 @@ files = [
{file = "greenlet-3.2.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c2ca18a03a8cfb5b25bc1cbe20f3d9a4c80d8c3b13ba3df49ac3961af0b1018d"},
{file = "greenlet-3.2.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9fe0a28a7b952a21e2c062cd5756d34354117796c6d9215a87f55e38d15402c5"},
{file = "greenlet-3.2.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8854167e06950ca75b898b104b63cc646573aa5fef1353d4508ecdd1ee76254f"},
{file = "greenlet-3.2.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f47617f698838ba98f4ff4189aef02e7343952df3a615f847bb575c3feb177a7"},
{file = "greenlet-3.2.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:af41be48a4f60429d5cad9d22175217805098a9ef7c40bfef44f7669fb9d74d8"},
{file = "greenlet-3.2.4-cp310-cp310-win_amd64.whl", hash = "sha256:73f49b5368b5359d04e18d15828eecc1806033db5233397748f4ca813ff1056c"},
{file = "greenlet-3.2.4-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:96378df1de302bc38e99c3a9aa311967b7dc80ced1dcc6f171e99842987882a2"},
{file = "greenlet-3.2.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1ee8fae0519a337f2329cb78bd7a8e128ec0f881073d43f023c7b8d4831d5246"},
@ -2213,6 +2215,8 @@ files = [
{file = "greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2523e5246274f54fdadbce8494458a2ebdcdbc7b802318466ac5606d3cded1f8"},
{file = "greenlet-3.2.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:1987de92fec508535687fb807a5cea1560f6196285a4cde35c100b8cd632cc52"},
{file = "greenlet-3.2.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:55e9c5affaa6775e2c6b67659f3a71684de4c549b3dd9afca3bc773533d284fa"},
{file = "greenlet-3.2.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c9c6de1940a7d828635fbd254d69db79e54619f165ee7ce32fda763a9cb6a58c"},
{file = "greenlet-3.2.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:03c5136e7be905045160b1b9fdca93dd6727b180feeafda6818e6496434ed8c5"},
{file = "greenlet-3.2.4-cp311-cp311-win_amd64.whl", hash = "sha256:9c40adce87eaa9ddb593ccb0fa6a07caf34015a29bf8d344811665b573138db9"},
{file = "greenlet-3.2.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3b67ca49f54cede0186854a008109d6ee71f66bd57bb36abd6d0a0267b540cdd"},
{file = "greenlet-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ddf9164e7a5b08e9d22511526865780a576f19ddd00d62f8a665949327fde8bb"},
@ -2222,6 +2226,8 @@ files = [
{file = "greenlet-3.2.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b3812d8d0c9579967815af437d96623f45c0f2ae5f04e366de62a12d83a8fb0"},
{file = "greenlet-3.2.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:abbf57b5a870d30c4675928c37278493044d7c14378350b3aa5d484fa65575f0"},
{file = "greenlet-3.2.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:20fb936b4652b6e307b8f347665e2c615540d4b42b3b4c8a321d8286da7e520f"},
{file = "greenlet-3.2.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ee7a6ec486883397d70eec05059353b8e83eca9168b9f3f9a361971e77e0bcd0"},
{file = "greenlet-3.2.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:326d234cbf337c9c3def0676412eb7040a35a768efc92504b947b3e9cfc7543d"},
{file = "greenlet-3.2.4-cp312-cp312-win_amd64.whl", hash = "sha256:a7d4e128405eea3814a12cc2605e0e6aedb4035bf32697f72deca74de4105e02"},
{file = "greenlet-3.2.4-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:1a921e542453fe531144e91e1feedf12e07351b1cf6c9e8a3325ea600a715a31"},
{file = "greenlet-3.2.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd3c8e693bff0fff6ba55f140bf390fa92c994083f838fece0f63be121334945"},
@ -2231,6 +2237,8 @@ files = [
{file = "greenlet-3.2.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23768528f2911bcd7e475210822ffb5254ed10d71f4028387e5a99b4c6699671"},
{file = "greenlet-3.2.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:00fadb3fedccc447f517ee0d3fd8fe49eae949e1cd0f6a611818f4f6fb7dc83b"},
{file = "greenlet-3.2.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:d25c5091190f2dc0eaa3f950252122edbbadbb682aa7b1ef2f8af0f8c0afefae"},
{file = "greenlet-3.2.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6e343822feb58ac4d0a1211bd9399de2b3a04963ddeec21530fc426cc121f19b"},
{file = "greenlet-3.2.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca7f6f1f2649b89ce02f6f229d7c19f680a6238af656f61e0115b24857917929"},
{file = "greenlet-3.2.4-cp313-cp313-win_amd64.whl", hash = "sha256:554b03b6e73aaabec3745364d6239e9e012d64c68ccd0b8430c64ccc14939a8b"},
{file = "greenlet-3.2.4-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:49a30d5fda2507ae77be16479bdb62a660fa51b1eb4928b524975b3bde77b3c0"},
{file = "greenlet-3.2.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:299fd615cd8fc86267b47597123e3f43ad79c9d8a22bebdce535e53550763e2f"},
@ -2238,6 +2246,8 @@ files = [
{file = "greenlet-3.2.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b4a1870c51720687af7fa3e7cda6d08d801dae660f75a76f3845b642b4da6ee1"},
{file = "greenlet-3.2.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:061dc4cf2c34852b052a8620d40f36324554bc192be474b9e9770e8c042fd735"},
{file = "greenlet-3.2.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44358b9bf66c8576a9f57a590d5f5d6e72fa4228b763d0e43fee6d3b06d3a337"},
{file = "greenlet-3.2.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2917bdf657f5859fbf3386b12d68ede4cf1f04c90c3a6bc1f013dd68a22e2269"},
{file = "greenlet-3.2.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:015d48959d4add5d6c9f6c5210ee3803a830dce46356e3bc326d6776bde54681"},
{file = "greenlet-3.2.4-cp314-cp314-win_amd64.whl", hash = "sha256:e37ab26028f12dbb0ff65f29a8d3d44a765c61e729647bf2ddfbbed621726f01"},
{file = "greenlet-3.2.4-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:b6a7c19cf0d2742d0809a4c05975db036fdff50cd294a93632d6a310bf9ac02c"},
{file = "greenlet-3.2.4-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:27890167f55d2387576d1f41d9487ef171849ea0359ce1510ca6e06c8bece11d"},
@ -2247,6 +2257,8 @@ files = [
{file = "greenlet-3.2.4-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c9913f1a30e4526f432991f89ae263459b1c64d1608c0d22a5c79c287b3c70df"},
{file = "greenlet-3.2.4-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:b90654e092f928f110e0007f572007c9727b5265f7632c2fa7415b4689351594"},
{file = "greenlet-3.2.4-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:81701fd84f26330f0d5f4944d4e92e61afe6319dcd9775e39396e39d7c3e5f98"},
{file = "greenlet-3.2.4-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:28a3c6b7cd72a96f61b0e4b2a36f681025b60ae4779cc73c1535eb5f29560b10"},
{file = "greenlet-3.2.4-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:52206cd642670b0b320a1fd1cbfd95bca0e043179c1d8a045f2c6109dfe973be"},
{file = "greenlet-3.2.4-cp39-cp39-win32.whl", hash = "sha256:65458b409c1ed459ea899e939f0e1cdb14f58dbc803f2f93c5eab5694d32671b"},
{file = "greenlet-3.2.4-cp39-cp39-win_amd64.whl", hash = "sha256:d2e685ade4dafd447ede19c31277a224a239a0a1a4eca4e6390efedf20260cfb"},
{file = "greenlet-3.2.4.tar.gz", hash = "sha256:0dca0d95ff849f9a364385f36ab49f50065d76964944638be9691e1832e9f86d"},
@ -2344,6 +2356,7 @@ files = [
{file = "grpcio-1.76.0-cp39-cp39-win_amd64.whl", hash = "sha256:acab0277c40eff7143c2323190ea57b9ee5fd353d8190ee9652369fae735668a"},
{file = "grpcio-1.76.0.tar.gz", hash = "sha256:7be78388d6da1a25c0d5ec506523db58b18be22d9c37d8d3a32c08be4987bd73"},
]
markers = {main = "extra == \"extra-proxy\" or extra == \"grpc\""}
[package.dependencies]
typing-extensions = ">=4.12,<5.0"
@ -2376,7 +2389,7 @@ description = "WSGI HTTP Server for UNIX"
optional = true
python-versions = ">=3.7"
groups = ["main"]
markers = "extra == \"proxy\" or (extra == \"mlflow\" or extra == \"proxy\") and platform_system != \"Windows\" and python_version >= \"3.10\""
markers = "extra == \"proxy\" or (extra == \"proxy\" or extra == \"mlflow\") and platform_system != \"Windows\" and python_version >= \"3.10\""
files = [
{file = "gunicorn-23.0.0-py3-none-any.whl", hash = "sha256:ec400d38950de4dfd418cff8328b2c8faed0edb0d517d3394e457c317908ca4d"},
{file = "gunicorn-23.0.0.tar.gz", hash = "sha256:f014447a0101dc57e294f6c18ca6b40227a4c90e9bdb586042628030cba004ec"},
@ -3847,7 +3860,7 @@ description = "Fundamental package for array computing in Python"
optional = true
python-versions = ">=3.9"
groups = ["main"]
markers = "python_version >= \"3.10\" and python_version < \"3.12\" and (extra == \"extra-proxy\" or extra == \"semantic-router\" or extra == \"mlflow\") or python_version == \"3.9\" and (extra == \"extra-proxy\" or extra == \"semantic-router\")"
markers = "(python_version >= \"3.10\" or extra == \"extra-proxy\" or extra == \"semantic-router\") and python_version < \"3.12\" and (extra == \"extra-proxy\" or extra == \"semantic-router\" or extra == \"mlflow\")"
files = [
{file = "numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0"},
{file = "numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a"},
@ -7983,6 +7996,7 @@ type = ["pytest-mypy"]
[extras]
caching = ["diskcache"]
extra-proxy = ["a2a-sdk", "azure-identity", "azure-keyvault-secrets", "google-cloud-iam", "google-cloud-kms", "prisma", "redisvl", "resend"]
grpc = ["grpcio", "grpcio"]
mlflow = ["mlflow"]
proxy = ["PyJWT", "apscheduler", "azure-identity", "azure-storage-blob", "backoff", "boto3", "cryptography", "fastapi", "fastapi-sso", "gunicorn", "litellm-enterprise", "litellm-proxy-extras", "mcp", "orjson", "polars", "pynacl", "python-multipart", "pyyaml", "rich", "rq", "soundfile", "uvicorn", "uvloop", "websockets"]
semantic-router = ["semantic-router"]
@ -7991,4 +8005,4 @@ utils = ["numpydoc"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.9,<4.0"
content-hash = "3a929b2e1dc2b85edcf78f93b0c15eda2bf0cdf8d3e0e30778fc63178c650e40"
content-hash = "f6a98e687d478db6e30274a4cf70391960775cbf648da0783558444da3a662ea"

35
provider_endpoints_support.json
View file

@ -759,6 +759,23 @@
"search": true
}
},
"brave": {
"display_name": "Brave Search (`brave`)",
"url": "https://docs.litellm.ai/docs/search/brave",
"endpoints": {
"chat_completions": false,
"messages": false,
"responses": false,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"search": true
}
},
"empower": {
"display_name": "Empower (`empower`)",
"url": "https://docs.litellm.ai/docs/providers/empower",
@ -955,6 +972,24 @@
"interactions": true
}
},
"gmi": {
"display_name": "GMI Cloud (`gmi`)",
"url": "https://docs.litellm.ai/docs/providers/gmi_cloud",
"endpoints": {
"chat_completions": true,
"messages": true,
"responses": true,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"a2a": true,
"interactions": true
}
},
"vertex_ai": {
"display_name": "Google - Vertex AI (`vertex_ai`)",
"url": "https://docs.litellm.ai/docs/providers/vertex",

Some files were not shown because too many files have changed in this diff Show more