scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 86

Merge branch 'main' into litellm_ui_callback_fix

Browse source
This commit is contained in:
yuneng-jiang 2025-11-10 16:50:58 -08:00 • committed by GitHub
commit 5853dbafc8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
467 changed files with 16232 additions and 4765 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.circleci/config.yml
View file

@ -532,7 +532,7 @@ jobs:
command: |
pwd
ls
python -m pytest -vv tests/router_unit_tests --cov=litellm --cov-report=xml -x -s -v --junitxml=test-results/junit.xml --durations=5
python -m pytest -vv tests/router_unit_tests --cov=litellm --cov-report=xml -x -s --junitxml=test-results/junit.xml --durations=5
no_output_timeout: 120m
- run:
name: Rename the coverage files
@ -1164,7 +1164,7 @@ jobs:
command: |
pwd
ls
python -m pytest -vv tests/test_litellm --cov=litellm --cov-report=xml -s -v --junitxml=test-results/junit-litellm.xml --durations=10 -n 8
python -m pytest -vv tests/test_litellm --cov=litellm --cov-report=xml -v --junitxml=test-results/junit-litellm.xml --durations=10 -n 8
no_output_timeout: 120m
- run:
name: Rename the coverage files
@ -1396,7 +1396,7 @@ jobs:
command: |
pwd
ls
python -m pytest -vv tests/image_gen_tests --cov=litellm --cov-report=xml -x -s -v --junitxml=test-results/junit.xml --durations=5
python -m pytest -vv tests/image_gen_tests --cov=litellm --cov-report=xml -x -v --junitxml=test-results/junit.xml --durations=5
no_output_timeout: 120m
- run:
name: Rename the coverage files

3
.circleci/requirements.txt
View file

@ -15,4 +15,5 @@ fastapi-sso==0.16.0
uvloop==0.21.0
mcp==1.10.1 # for MCP server
semantic_router==0.1.10 # for auto-routing with litellm
fastuuid==0.12.0
fastuuid==0.12.0
responses==0.25.7 # for proxy client tests

4
Dockerfile
View file

@ -65,6 +65,10 @@ COPY --from=builder /wheels/ /wheels/
# Install the built wheel using pip; again using a wildcard if it's the only file
RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ && rm -f *.whl && rm -rf /wheels
# Remove test files and keys from dependencies
RUN find /usr/lib -type f -path "*/tornado/test/*" -delete && \
find /usr/lib -type d -path "*/tornado/test" -delete
# Install semantic_router and aurelio-sdk using script
RUN chmod +x docker/install_auto_router.sh && ./docker/install_auto_router.sh

3
docker/Dockerfile.dev
View file

@ -57,6 +57,9 @@ USER root
# Install only runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
libssl3 \
libatomic1 \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app

4
docker/Dockerfile.non_root
View file

@ -69,6 +69,10 @@ RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ \
&& rm -f *.whl \
&& rm -rf /wheels
# Remove test files and keys from dependencies
RUN find /usr/lib -type f -path "*/tornado/test/*" -delete && \
find /usr/lib -type d -path "*/tornado/test" -delete
# Install semantic_router and aurelio-sdk using script
RUN chmod +x docker/install_auto_router.sh && ./docker/install_auto_router.sh

83
docs/my-website/docs/benchmarks.md
View file

@ -48,47 +48,6 @@ In these tests the baseline latency characteristics are measured against a fake-
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## LiteLLM vs Portkey Performance Comparison
**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up
### Multi-Instance (4×) Performance
| Metric | Portkey (no DB) | LiteLLM (with DB) |
| ------------------- | --------------- | ----------------- |
| **Total Requests** | 293,796 | 312,405 |
| **Failed Requests** | 0 | 0 |
| **Median Latency** | 100 ms | 100 ms |
| **p95 Latency** | 230 ms | 150 ms |
| **p99 Latency** | 500 ms | 240 ms |
| **Average Latency** | 123 ms | 111 ms |
| **Current RPS** | 1,170.9 | 1,170 |
### Technical Insights
**Portkey**
**Pros**
* Low memory footprint
* Stable latency with minimal spikes
**Cons**
* CPU utilization capped around ~40%, indicating underutilization of available compute resources
* Experienced three I/O timeout outages
**LiteLLM**
**Pros**
* Fully utilizes available CPU capacity
* Strong connection handling and low latency after initial warm-up spikes
**Cons**
* High memory usage during initialization and per request
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
@ -163,6 +122,48 @@ class MyUser(HttpUser):
```
## LiteLLM vs Portkey Performance Comparison
**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up
### Multi-Instance (4×) Performance
| Metric | Portkey (no DB) | LiteLLM (with DB) |
| ------------------- | --------------- | ----------------- |
| **Total Requests** | 293,796 | 312,405 |
| **Failed Requests** | 0 | 0 |
| **Median Latency** | 100 ms | 100 ms |
| **p95 Latency** | 230 ms | 150 ms |
| **p99 Latency** | 500 ms | 240 ms |
| **Average Latency** | 123 ms | 111 ms |
| **Current RPS** | 1,170.9 | 1,170 |
### Technical Insights
**Portkey**
**Pros**
* Low memory footprint
* Stable latency with minimal spikes
**Cons**
* CPU utilization capped around ~40%, indicating underutilization of available compute resources
* Experienced three I/O timeout outages
**LiteLLM**
**Pros**
* Fully utilizes available CPU capacity
* Strong connection handling and low latency after initial warm-up spikes
**Cons**
* High memory usage during initialization and per request
## Logging Callbacks

70
docs/my-website/docs/completion/image_generation_chat.md
View file

@ -15,16 +15,22 @@ Supported Providers:
- Google AI Studio (`gemini`)
- Vertex AI (`vertex_ai/`)
LiteLLM will standardize the `image` response in the assistant message for models that support image generation during chat completions.
LiteLLM will standardize the `images` response in the assistant message for models that support image generation during chat completions.
```python title="Example response from litellm"
"message": {
...
"content": "Here's the image you requested:",
"image": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
"detail": "auto"
}
"images": [
{
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
"detail": "auto"
},
"index": 0,
"type": "image_url"
}
]
}
```
@ -47,7 +53,7 @@ response = completion(
)
print(response.choices[0].message.content) # Text response
print(response.choices[0].message.image) # Image data
print(response.choices[0].message.images) # List of image objects
```
</TabItem>
@ -103,10 +109,16 @@ curl http://0.0.0.0:4000/v1/chat/completions \
"message": {
"content": "Here's the image you requested:",
"role": "assistant",
"image": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
"detail": "auto"
}
"images": [
{
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
"detail": "auto"
},
"index": 0,
"type": "image_url"
}
]
}
}
],
@ -141,8 +153,8 @@ response = completion(
)
for chunk in response:
if hasattr(chunk.choices[0].delta, "image") and chunk.choices[0].delta.image is not None:
print("Generated image:", chunk.choices[0].delta.image["url"])
if hasattr(chunk.choices[0].delta, "images") and chunk.choices[0].delta.images is not None:
print("Generated image:", chunk.choices[0].delta.images[0]["image_url"]["url"])
break
```
@ -175,7 +187,7 @@ data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1723323084
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1723323084,"model":"gemini/gemini-2.5-flash-image-preview","choices":[{"index":0,"delta":{"content":"Here's the image you requested:"},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1723323084,"model":"gemini/gemini-2.5-flash-image-preview","choices":[{"index":0,"delta":{"image":{"url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...","detail":"auto"}},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1723323084,"model":"gemini/gemini-2.5-flash-image-preview","choices":[{"index":0,"delta":{"images":[{"image_url":{"url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...","detail":"auto"},"index":0,"type":"image_url"}]},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1723323084,"model":"gemini/gemini-2.5-flash-image-preview","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
@ -200,8 +212,8 @@ async def generate_image():
)
print(response.choices[0].message.content) # Text response
print(response.choices[0].message.image) # Image data
print(response.choices[0].message.images) # List of image objects
return response
# Run the async function
@ -215,18 +227,28 @@ asyncio.run(generate_image())
| Google AI Studio | `gemini/gemini-2.5-flash-image-preview` |
| Vertex AI | `vertex_ai/gemini-2.5-flash-image-preview` |
## Spec
## Spec
The `image` field in the response follows this structure:
The `images` field in the response follows this structure:
```python
"image": {
"url": "data:image/png;base64,<base64_encoded_image>",
"detail": "auto"
}
"images": [
{
"image_url": {
"url": "data:image/png;base64,<base64_encoded_image>",
"detail": "auto"
},
"index": 0,
"type": "image_url"
}
]
```
- `url` - str: Base64 encoded image data in data URI format
- `detail` - str: Image detail level (always "auto" for generated images)
- `images` - List[ImageURLListItem]: Array of generated images
- `image_url` - ImageURLObject: Container for image data
- `url` - str: Base64 encoded image data in data URI format
- `detail` - str: Image detail level (always "auto" for generated images)
- `index` - int: Index of the image in the response
- `type` - str: Type identifier (always "image_url")
The image is returned as a base64-encoded data URI that can be directly used in HTML `<img>` tags or saved to a file.
The images are returned as base64-encoded data URIs that can be directly used in HTML `<img>` tags or saved to files.

79
docs/my-website/docs/exception_mapping.md
View file

@ -112,6 +112,85 @@ except openai.APITimeoutError as e:
print(f"should_retry: {should_retry}")
```
## Advanced
### Accessing Provider-Specific Error Details
LiteLLM exceptions include a `provider_specific_fields` attribute that contains additional error information specific to each provider. This is particularly useful for Azure OpenAI, which provides detailed content filtering information.
#### Azure OpenAI - Content Policy Violation Inner Error Access
When Azure OpenAI returns content policy violations, you can access the detailed content filtering results through the `innererror` field:
```python
import litellm
from litellm.exceptions import ContentPolicyViolationError
try:
response = litellm.completion(
model="azure/gpt-4",
messages=[
{
"role": "user",
"content": "Some content that might violate policies"
}
]
)
except ContentPolicyViolationError as e:
# Access Azure-specific error details
if e.provider_specific_fields and "innererror" in e.provider_specific_fields:
innererror = e.provider_specific_fields["innererror"]
# Access content filter results
content_filter_result = innererror.get("content_filter_result", {})
print(f"Content filter code: {innererror.get('code')}")
print(f"Hate filtered: {content_filter_result.get('hate', {}).get('filtered')}")
print(f"Violence severity: {content_filter_result.get('violence', {}).get('severity')}")
print(f"Sexual content filtered: {content_filter_result.get('sexual', {}).get('filtered')}")
```
**Example Response Structure:**
When calling the LiteLLM proxy, content policy violations will return detailed filtering information:
```json
{
"error": {
"message": "litellm.ContentPolicyViolationError: AzureException - The response was filtered due to the prompt triggering Azure OpenAI's content management policy...",
"type": null,
"param": null,
"code": "400",
"provider_specific_fields": {
"innererror": {
"code": "ResponsibleAIPolicyViolation",
"content_filter_result": {
"hate": {
"filtered": true,
"severity": "high"
},
"jailbreak": {
"filtered": false,
"detected": false
},
"self_harm": {
"filtered": false,
"severity": "safe"
},
"sexual": {
"filtered": false,
"severity": "safe"
},
"violence": {
"filtered": true,
"severity": "medium"
}
}
}
}
}
}
## Details
To see how it's implemented - [check out the code](https://github.com/BerriAI/litellm/blob/a42c197e5a6de56ea576c73715e6c7c6b19fa249/litellm/utils.py#L1217)

15
docs/my-website/docs/moderation.md
View file

@ -22,10 +22,19 @@ response = moderation(
For `/moderations` endpoint, there is **no need to specify `model` in the request or on the litellm config.yaml**
Start litellm proxy server
1. Setup config.yaml
```yaml
model_list:
- model_name: text-moderation-stable
litellm_params:
model: openai/omni-moderation-latest
```
2. Start litellm proxy server
```
litellm
litellm --config /path/to/config.yaml
```
@ -41,7 +50,7 @@ client = OpenAI(api_key="<proxy-api-key>", base_url="http://0.0.0.0:4000")
response = client.moderations.create(
input="hello from litellm",
model="text-moderation-stable" # optional, defaults to `omni-moderation-latest`
model="text-moderation-stable"
)
print(response)

29
docs/my-website/docs/observability/datadog.md
View file

@ -56,12 +56,32 @@ litellm_settings:
**Step 2**: Set Required env variables for datadog
#### Direct API
Send logs directly to Datadog API:
```shell
DD_API_KEY="5f2d0f310***********" # your datadog API Key
DD_SITE="us5.datadoghq.com" # your datadog base url
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to differentiate dev vs. prod deployments
```
#### Via DataDog Agent
Send logs through a local DataDog agent (useful for containerized environments):
```shell
DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
```
When `DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
- Centralized log shipping in containerized environments
- Reducing direct API calls from multiple services
- Leveraging agent-side processing and filtering
**Step 3**: Start the proxy, make a test request
Start proxy
@ -169,8 +189,10 @@ LiteLLM supports customizing the following Datadog environment variables
| Environment Variable | Description | Default Value | Required |
|---------------------|-------------|---------------|----------|
| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes |
| `DD_API_KEY` | Your Datadog API key for authentication (required for direct API, optional for agent) | None | Conditional* |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") (required for direct API) | None | Conditional* |
| `DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
@ -178,3 +200,6 @@ LiteLLM supports customizing the following Datadog environment variables
| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No |
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
\* **Required when using Direct API** (default): `DD_API_KEY` and `DD_SITE` are required
\* **Optional when using DataDog Agent**: Set `DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required

7
docs/my-website/docs/projects/Softgen Normal file
View file

@ -0,0 +1,7 @@
# Softgen
`Softgen` is an AI-powered platform that builds full-stack web apps from your plain instructions.
LiteLLM helps `Softgen` users to choose and use different LLMs.
- [Softgen](https://softgen.ai)
- [Academy](hhttps://academy.softgen.ai)

68
docs/my-website/docs/providers/anthropic.md
View file

@ -953,7 +953,7 @@ except Exception as e:
s/o @[Shekhar Patnaik](https://www.linkedin.com/in/patnaikshekhar) for requesting this!
### Anthropic Hosted Tools (Computer, Text Editor, Web Search)
### Anthropic Hosted Tools (Computer, Text Editor, Web Search, Memory)
<Tabs>
@ -1183,6 +1183,72 @@ curl http://0.0.0.0:4000/v1/chat/completions \
</Tabs>
</TabItem>
<TabItem value="memory" label="Memory">
:::info
The Anthropic Memory tool is currently in beta.
:::
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
tools = [{
"type": "memory_20250818",
"name": "memory"
}]
model = "claude-sonnet-4-5-20250929"
messages = [{"role": "user", "content": "Please remember that my favorite color is blue."}]
response = completion(
model=model,
messages=messages,
tools=tools,
)
print(response)
```
</TabItem>
<TabItem value="proxy" label="Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-memory-model
litellm_params:
model: anthropic/claude-sonnet-4-5-20250929
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_KEY" \
-d '{
"model": "claude-memory-model",
"messages": [{"role": "user", "content": "Please remember that my favorite color is blue."}],
"tools": [{"type": "memory_20250818", "name": "memory"}]
}'
```
</TabItem>
</Tabs>
</TabItem>
</Tabs>

18
docs/my-website/docs/providers/azure/videos.md
View file

@ -25,7 +25,6 @@ LiteLLM supports Azure OpenAI's video generation models including Sora with full
import os
os.environ["AZURE_OPENAI_API_KEY"] = "your-azure-api-key"
os.environ["AZURE_OPENAI_API_BASE"] = "https://your-resource.openai.azure.com/"
os.environ["AZURE_OPENAI_API_VERSION"] = "2024-02-15-preview"
```
### Basic Usage
@ -37,7 +36,6 @@ import time
os.environ["AZURE_OPENAI_API_KEY"] = "your-azure-api-key"
os.environ["AZURE_OPENAI_API_BASE"] = "https://your-resource.openai.azure.com/"
os.environ["AZURE_OPENAI_API_VERSION"] = "2024-02-15-preview"
# Generate video
response = video_generation(
@ -53,8 +51,7 @@ print(f"Initial Status: {response.status}")
# Check status until video is ready
while True:
status_response = video_status(
video_id=response.id,
custom_llm_provider="azure"
video_id=response.id
)
print(f"Current Status: {status_response.status}")
@ -69,8 +66,7 @@ while True:
# Download video content when ready
video_bytes = video_content(
video_id=response.id,
custom_llm_provider="azure"
video_id=response.id
)
# Save to file
@ -87,7 +83,6 @@ Here's how to call Azure video generation models with the LiteLLM Proxy Server
```bash
export AZURE_OPENAI_API_KEY="your-azure-api-key"
export AZURE_OPENAI_API_BASE="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_VERSION="2024-02-15-preview"
```
### 2. Start the proxy
@ -102,7 +97,6 @@ model_list:
model: azure/sora-2
api_key: os.environ/AZURE_OPENAI_API_KEY
api_base: os.environ/AZURE_OPENAI_API_BASE
api_version: "2024-02-15-preview"
```
</TabItem>
@ -211,8 +205,7 @@ general_settings:
```python
# Download video content
video_bytes = video_content(
video_id="video_1234567890",
model="azure/sora-2"
video_id="video_1234567890"
)
# Save to file
@ -243,8 +236,7 @@ def generate_and_download_video(prompt):
# Step 3: Download video
video_bytes = litellm.video_content(
video_id=video_id,
custom_llm_provider="azure"
video_id=video_id
)
# Step 4: Save to file
@ -264,9 +256,9 @@ video_file = generate_and_download_video(
```python
# Video editing with reference image
response = litellm.video_remix(
video_id="video_456",
prompt="Make the cat jump higher",
input_reference=open("path/to/image.jpg", "rb"), # Reference image as file object
custom_llm_provider="azure"
seconds="8"
)

6
docs/my-website/docs/providers/fireworks_ai.md
View file

@ -204,7 +204,7 @@ from litellm import completion
import os
os.environ["FIREWORKS_AI_API_KEY"] = "YOUR_API_KEY"
os.environ["FIREWORKS_AI_API_BASE"] = "https://audio-prod.us-virginia-1.direct.fireworks.ai/v1"
os.environ["FIREWORKS_AI_API_BASE"] = "https://audio-prod.api.fireworks.ai/v1"
completion = litellm.completion(
model="fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct",
@ -343,7 +343,7 @@ from litellm import transcription
import os
os.environ["FIREWORKS_AI_API_KEY"] = "YOUR_API_KEY"
os.environ["FIREWORKS_AI_API_BASE"] = "https://audio-prod.us-virginia-1.direct.fireworks.ai/v1"
os.environ["FIREWORKS_AI_API_BASE"] = "https://audio-prod.api.fireworks.ai/v1"
response = transcription(
model="fireworks_ai/whisper-v3",
@ -363,7 +363,7 @@ model_list:
- model_name: whisper-v3
litellm_params:
model: fireworks_ai/whisper-v3
api_base: https://audio-prod.us-virginia-1.direct.fireworks.ai/v1
api_base: https://audio-prod.api.fireworks.ai/v1
api_key: os.environ/FIREWORKS_API_KEY
model_info:
mode: audio_transcription

2
docs/my-website/docs/providers/gemini.md
View file

@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem';
| Provider Route on LiteLLM | `gemini/` |
| Provider Doc | [Google AI Studio ↗](https://aistudio.google.com/) |
| API Endpoint for Provider | https://generativelanguage.googleapis.com |
| Supported OpenAI Endpoints | `/chat/completions`, [`/embeddings`](../embedding/supported_embedding#gemini-ai-embedding-models), `/completions` |
| Supported OpenAI Endpoints | `/chat/completions`, [`/embeddings`](../embedding/supported_embedding#gemini-ai-embedding-models), `/completions`, [`/videos`](./gemini/videos.md) |
| Pass-through Endpoint | [Supported](../pass_through/google_ai_studio.md) |
<br />

409
docs/my-website/docs/providers/gemini/videos.md Normal file
View file

@ -0,0 +1,409 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Gemini Video Generation (Veo)
LiteLLM supports Google's Veo video generation models through a unified API interface.
| Property | Details |
|-------|-------|
| Description | Google's Veo AI video generation models |
| Provider Route on LiteLLM | `gemini/` |
| Supported Models | `veo-3.0-generate-preview`, `veo-3.1-generate-preview` |
| Cost Tracking | ✅ Duration-based pricing |
| Logging Support | ✅ Full request/response logging |
| Proxy Server Support | ✅ Full proxy integration with virtual keys |
| Spend Management | ✅ Budget tracking and rate limiting |
| Link to Provider Doc | [Google Veo Documentation ↗](https://ai.google.dev/gemini-api/docs/video) |
## Quick Start
### Required API Keys
```python
import os
os.environ["GEMINI_API_KEY"] = "your-google-api-key"
# OR
os.environ["GOOGLE_API_KEY"] = "your-google-api-key"
```
### Basic Usage
```python
from litellm import video_generation, video_status, video_content
import os
import time
os.environ["GEMINI_API_KEY"] = "your-google-api-key"
# Step 1: Generate video
response = video_generation(
model="gemini/veo-3.0-generate-preview",
prompt="A cat playing with a ball of yarn in a sunny garden"
)
print(f"Video ID: {response.id}")
print(f"Initial Status: {response.status}") # "processing"
# Step 2: Poll for completion
while True:
status_response = video_status(
video_id=response.id
)
print(f"Current Status: {status_response.status}")
if status_response.status == "completed":
break
elif status_response.status == "failed":
print("Video generation failed")
break
time.sleep(10) # Wait 10 seconds before checking again
# Step 3: Download video content
video_bytes = video_content(
video_id=response.id
)
# Save to file
with open("generated_video.mp4", "wb") as f:
f.write(video_bytes)
print("Video downloaded successfully!")
```
## Supported Models
| Model Name | Description | Max Duration | Status |
|------------|-------------|--------------|--------|
| veo-3.0-generate-preview | Veo 3.0 video generation | 8 seconds | Preview |
| veo-3.1-generate-preview | Veo 3.1 video generation | 8 seconds | Preview |
## Video Generation Parameters
LiteLLM automatically maps OpenAI-style parameters to Veo's format:
| OpenAI Parameter | Veo Parameter | Description | Example |
|------------------|---------------|-------------|---------|
| `prompt` | `prompt` | Text description of the video | "A cat playing" |
| `size` | `aspectRatio` | Video dimensions → aspect ratio | "1280x720" → "16:9" |
| `seconds` | `durationSeconds` | Duration in seconds | "8" → 8 |
| `input_reference` | `image` | Reference image to animate | File object or path |
| `model` | `model` | Model to use | "gemini/veo-3.0-generate-preview" |
### Size to Aspect Ratio Mapping
LiteLLM automatically converts size dimensions to Veo's aspect ratio format:
- `"1280x720"`, `"1920x1080"` → `"16:9"` (landscape)
- `"720x1280"`, `"1080x1920"` → `"9:16"` (portrait)
### Supported Veo Parameters
Based on Veo's API:
- **prompt** (required): Text description with optional audio cues
- **aspectRatio**: `"16:9"` (default) or `"9:16"`
- **resolution**: `"720p"` (default) or `"1080p"` (Veo 3.1 only, 16:9 aspect ratio only)
- **durationSeconds**: Video length (max 8 seconds for most models)
- **image**: Reference image for animation
- **negativePrompt**: What to exclude from the video (Veo 3.1)
- **referenceImages**: Style and content references (Veo 3.1 only)
## Complete Workflow Example
```python
import litellm
import time
def generate_and_download_veo_video(
prompt: str,
output_file: str = "video.mp4",
size: str = "1280x720",
seconds: str = "8"
):
"""
Complete workflow for Veo video generation.
Args:
prompt: Text description of the video
output_file: Where to save the video
size: Video dimensions (e.g., "1280x720" for 16:9)
seconds: Duration in seconds
Returns:
bool: True if successful
"""
print(f"🎬 Generating video: {prompt}")
# Step 1: Initiate generation
response = litellm.video_generation(
model="gemini/veo-3.0-generate-preview",
prompt=prompt,
size=size, # Maps to aspectRatio
seconds=seconds # Maps to durationSeconds
)
video_id = response.id
print(f"✓ Video generation started (ID: {video_id})")
# Step 2: Wait for completion
max_wait_time = 600 # 10 minutes
start_time = time.time()
while time.time() - start_time < max_wait_time:
status_response = litellm.video_status(video_id=video_id)
if status_response.status == "completed":
print("✓ Video generation completed!")
break
elif status_response.status == "failed":
print("✗ Video generation failed")
return False
print(f"⏳ Status: {status_response.status}")
time.sleep(10)
else:
print("✗ Timeout waiting for video generation")
return False
# Step 3: Download video
print("⬇️ Downloading video...")
video_bytes = litellm.video_content(video_id=video_id)
with open(output_file, "wb") as f:
f.write(video_bytes)
print(f"✓ Video saved to {output_file}")
return True
# Use it
generate_and_download_veo_video(
prompt="A serene lake at sunset with mountains in the background",
output_file="sunset_lake.mp4"
)
```
## Async Usage
```python
from litellm import avideo_generation, avideo_status, avideo_content
import asyncio
async def async_video_workflow():
# Generate video
response = await avideo_generation(
model="gemini/veo-3.0-generate-preview",
prompt="A cat playing with a ball of yarn"
)
# Poll for completion
while True:
status = await avideo_status(video_id=response.id)
if status.status == "completed":
break
await asyncio.sleep(10)
# Download content
video_bytes = await avideo_content(video_id=response.id)
with open("video.mp4", "wb") as f:
f.write(video_bytes)
# Run it
asyncio.run(async_video_workflow())
```
## LiteLLM Proxy Usage
### Configuration
Add Veo models to your `config.yaml`:
```yaml
model_list:
- model_name: veo-3
litellm_params:
model: gemini/veo-3.0-generate-preview
api_key: os.environ/GEMINI_API_KEY
```
Start the proxy:
```bash
litellm --config config.yaml
# Server running on http://0.0.0.0:4000
```
### Making Requests
<Tabs>
<TabItem value="curl" label="Curl">
```bash
# Step 1: Generate video
curl --location 'http://0.0.0.0:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
"model": "veo-3",
"prompt": "A cat playing with a ball of yarn in a sunny garden"
}'
# Response: {"id": "gemini::operations/generate_12345::...", "status": "processing", ...}
# Step 2: Check status
curl --location 'http://localhost:4000/v1/videos/{video_id}' \
--header 'x-litellm-api-key: sk-1234'
# Step 3: Download video (when status is "completed")
curl --location 'http://localhost:4000/v1/videos/{video_id}/content' \
--header 'x-litellm-api-key: sk-1234' \
--output video.mp4
```
</TabItem>
<TabItem value="python" label="Python SDK">
```python
import litellm
litellm.api_base = "http://0.0.0.0:4000"
litellm.api_key = "sk-1234"
# Generate video
response = litellm.video_generation(
model="veo-3",
prompt="A cat playing with a ball of yarn in a sunny garden"
)
# Check status
import time
while True:
status = litellm.video_status(video_id=response.id)
if status.status == "completed":
break
time.sleep(10)
# Download video
video_bytes = litellm.video_content(video_id=response.id)
with open("video.mp4", "wb") as f:
f.write(video_bytes)
```
</TabItem>
</Tabs>
## Cost Tracking
LiteLLM automatically tracks costs for Veo video generation:
```python
response = litellm.video_generation(
model="gemini/veo-3.0-generate-preview",
prompt="A beautiful sunset"
)
# Cost is calculated based on video duration
# Veo pricing: ~$0.10 per second (estimated)
# Default video duration: ~5 seconds
# Estimated cost: ~$0.50
```
## Differences from OpenAI Video API
| Feature | OpenAI (Sora) | Gemini (Veo) |
|---------|---------------|--------------|
| Reference Images | ✅ Supported | ❌ Not supported |
| Size Control | ✅ Supported | ❌ Not supported |
| Duration Control | ✅ Supported | ❌ Not supported |
| Video Remix/Edit | ✅ Supported | ❌ Not supported |
| Video List | ✅ Supported | ❌ Not supported |
| Prompt-based Generation | ✅ Supported | ✅ Supported |
| Async Operations | ✅ Supported | ✅ Supported |
## Error Handling
```python
from litellm import video_generation, video_status, video_content
from litellm.exceptions import APIError, Timeout
try:
response = video_generation(
model="gemini/veo-3.0-generate-preview",
prompt="A beautiful landscape"
)
# Poll with timeout
max_attempts = 60 # 10 minutes (60 * 10s)
for attempt in range(max_attempts):
status = video_status(video_id=response.id)
if status.status == "completed":
video_bytes = video_content(video_id=response.id)
with open("video.mp4", "wb") as f:
f.write(video_bytes)
break
elif status.status == "failed":
raise APIError("Video generation failed")
time.sleep(10)
else:
raise Timeout("Video generation timed out")
except APIError as e:
print(f"API Error: {e}")
except Timeout as e:
print(f"Timeout: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
```
## Best Practices
1. **Always poll for completion**: Veo video generation is asynchronous and can take several minutes
2. **Set reasonable timeouts**: Allow at least 5-10 minutes for video generation
3. **Handle failures gracefully**: Check for `failed` status and implement retry logic
4. **Use descriptive prompts**: More detailed prompts generally produce better results
5. **Store video IDs**: Save the operation ID/video ID to resume polling if your application restarts
## Troubleshooting
### Video generation times out
```python
# Increase polling timeout
max_wait_time = 900 # 15 minutes instead of 10
```
### Video not found when downloading
```python
# Make sure video is completed before downloading
status = video_status(video_id=video_id)
if status.status != "completed":
print("Video not ready yet!")
```
### API key errors
```python
# Verify your API key is set
import os
print(os.environ.get("GEMINI_API_KEY"))
# Or pass it explicitly
response = video_generation(
model="gemini/veo-3.0-generate-preview",
prompt="...",
api_key="your-api-key-here"
)
```
## See Also
- [OpenAI Video Generation](../openai/videos.md)
- [Azure Video Generation](../azure/videos.md)
- [Vertex AI Video Generation](../vertex_ai/videos.md)
- [Video Generation API Reference](/docs/videos)
- [Veo Pass-through Endpoints](/docs/pass_through/google_ai_studio#example-4-video-generation-with-veo)

118
docs/my-website/docs/providers/openai/videos.md
View file

@ -36,7 +36,6 @@ print(f"Status: {response.status}")
# Download video content when ready
video_bytes = video_content(
video_id=response.id,
model="sora-2"
)
# Save to file
@ -44,6 +43,113 @@ with open("generated_video.mp4", "wb") as f:
f.write(video_bytes)
```
## **LiteLLM Proxy Usage**
LiteLLM provides OpenAI API compatible video endpoints for complete video generation workflow:
- `/videos/generations` - Generate new videos
- `/videos/remix` - Edit existing videos with reference images
- `/videos/status` - Check video generation status
- `/videos/retrieval` - Download completed videos
**Setup**
Add this to your litellm proxy config.yaml
```yaml
model_list:
- model_name: sora-2
litellm_params:
model: openai/sora-2
api_key: os.environ/OPENAI_API_KEY
```
Start litellm
```bash
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
Test video generation request
```bash
curl --location 'http://localhost:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--data '{
"model": "sora-2",
"prompt": "A beautiful sunset over the ocean"
}'
```
Test video status request
```bash
# Using custom-llm-provider header
curl --location 'http://localhost:4000/v1/videos/video_id' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: openai'
```
Test video retrieval request
```bash
# Using custom-llm-provider header
curl --location 'http://localhost:4000/v1/videos/video_id/content' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: openai' \
--output video.mp4
# Or using query parameter
curl --location 'http://localhost:4000/v1/videos/video_id/content?custom_llm_provider=openai' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--output video.mp4
```
Test video remix request
```bash
# Using custom_llm_provider in request body
curl --location --request POST 'http://localhost:4000/v1/videos/video_id/remix' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--data '{
"prompt": "New remix instructions",
"custom_llm_provider": "openai"
}'
# Or using custom-llm-provider header
curl --location --request POST 'http://localhost:4000/v1/videos/video_id/remix' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: openai' \
--data '{
"prompt": "New remix instructions"
}'
```
Test OpenAI video generation request
```bash
curl http://localhost:4000/v1/videos \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "sora-2",
"prompt": "A cat playing with a ball of yarn in a sunny garden",
"seconds": "8",
"size": "720x1280"
}'
```
## Supported Models
| Model Name | Description | Max Duration | Supported Sizes |
@ -64,8 +170,7 @@ with open("generated_video.mp4", "wb") as f:
```python
# Download video content
video_bytes = video_content(
video_id="video_1234567890",
custom_llm_provider="openai" # Or use model="sora-2"
video_id="video_1234567890"
)
# Save to file
@ -96,8 +201,7 @@ def generate_and_download_video(prompt):
# Step 3: Download video
video_bytes = litellm.video_content(
video_id=video_id,
custom_llm_provider="openai"
video_id=video_id
)
# Step 4: Save to file
@ -112,6 +216,7 @@ video_file = generate_and_download_video(
)
```
## Video Editing with Reference Images
```python
@ -133,8 +238,7 @@ from litellm.exceptions import BadRequestError, AuthenticationError
try:
response = video_generation(
prompt="A cat playing with a ball of yarn",
model="sora-2"
prompt="A cat playing with a ball of yarn"
)
except AuthenticationError as e:
print(f"Authentication failed: {e}")

268
docs/my-website/docs/providers/vertex_ai/videos.md Normal file
View file

@ -0,0 +1,268 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Vertex AI Video Generation (Veo)
LiteLLM supports Vertex AI's Veo video generation models using the unified OpenAI video API surface.
| Property | Details |
|-------|-------|
| Description | Google Cloud Vertex AI Veo video generation models |
| Provider Route on LiteLLM | `vertex_ai/` |
| Supported Models | `veo-2.0-generate-001`, `veo-3.0-generate-preview`, `veo-3.0-fast-generate-preview`, `veo-3.1-generate-preview`, `veo-3.1-fast-generate-preview` |
| Cost Tracking | ✅ Duration-based pricing |
| Logging Support | ✅ Full request/response logging |
| Proxy Server Support | ✅ Full proxy integration with virtual keys |
| Spend Management | ✅ Budget tracking and rate limiting |
| Link to Provider Doc | [Vertex AI Veo Documentation ↗](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/veo-video-generation) |
## Quick Start
### Required Environment Setup
```python
import json
import os
os.environ["VERTEXAI_PROJECT"] = "your-gcp-project-id"
os.environ["VERTEXAI_LOCATION"] = "us-central1"
# Option 1: Point to a service account file
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/service_account.json"
# Option 2: Store the service account JSON directly
with open("/path/to/service_account.json", "r", encoding="utf-8") as f:
os.environ["VERTEXAI_CREDENTIALS"] = f.read()
```
### Basic Usage
```python
from litellm import video_generation, video_status, video_content
import json
import os
import time
with open("/path/to/service_account.json", "r", encoding="utf-8") as f:
vertex_credentials = f.read()
response = video_generation(
model="vertex_ai/veo-3.0-generate-preview",
prompt="A cat playing with a ball of yarn in a sunny garden",
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
seconds="8",
size="1280x720",
)
print(f"Video ID: {response.id}")
print(f"Initial Status: {response.status}")
# Poll for completion
while True:
status = video_status(
video_id=response.id,
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
print(f"Current Status: {status.status}")
if status.status == "completed":
break
if status.status == "failed":
raise RuntimeError("Video generation failed")
time.sleep(10)
# Download the rendered video
video_bytes = video_content(
video_id=response.id,
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
with open("generated_video.mp4", "wb") as f:
f.write(video_bytes)
```
## Supported Models
| Model Name | Description | Max Duration | Status |
|------------|-------------|--------------|--------|
| veo-2.0-generate-001 | Veo 2.0 video generation | 5 seconds | GA |
| veo-3.0-generate-preview | Veo 3.0 high quality | 8 seconds | Preview |
| veo-3.0-fast-generate-preview | Veo 3.0 fast generation | 8 seconds | Preview |
| veo-3.1-generate-preview | Veo 3.1 high quality | 10 seconds | Preview |
| veo-3.1-fast-generate-preview | Veo 3.1 fast | 10 seconds | Preview |
## Video Generation Parameters
LiteLLM converts OpenAI-style parameters to Veo's API shape automatically:
| OpenAI Parameter | Vertex AI Parameter | Description | Example |
|------------------|---------------------|-------------|---------|
| `prompt` | `instances[].prompt` | Text description of the video | "A cat playing" |
| `size` | `parameters.aspectRatio` | Converted to `16:9` or `9:16` | "1280x720" → `16:9` |
| `seconds` | `parameters.durationSeconds` | Clip length in seconds | "8" → `8` |
| `input_reference` | `instances[].image` | Reference image for animation | `open("image.jpg", "rb")` |
| Provider-specific params | `extra_body` | Forwarded to Vertex API | `{"negativePrompt": "blurry"}` |
### Size to Aspect Ratio Mapping
- `1280x720`, `1920x1080` → `16:9`
- `720x1280`, `1080x1920` → `9:16`
- Unknown sizes default to `16:9`
## Async Usage
```python
from litellm import avideo_generation, avideo_status, avideo_content
import asyncio
import json
with open("/path/to/service_account.json", "r", encoding="utf-8") as f:
vertex_credentials = f.read()
async def workflow():
response = await avideo_generation(
model="vertex_ai/veo-3.1-generate-preview",
prompt="Slow motion water droplets splashing into a pool",
seconds="10",
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
while True:
status = await avideo_status(
video_id=response.id,
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
if status.status == "completed":
break
if status.status == "failed":
raise RuntimeError("Video generation failed")
await asyncio.sleep(10)
video_bytes = await avideo_content(
video_id=response.id,
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
with open("veo_water.mp4", "wb") as f:
f.write(video_bytes)
asyncio.run(workflow())
```
## LiteLLM Proxy Usage
Add Veo models to your `config.yaml`:
```yaml
model_list:
- model_name: veo-3
litellm_params:
model: vertex_ai/veo-3.0-generate-preview
vertex_project: os.environ/VERTEXAI_PROJECT
vertex_location: os.environ/VERTEXAI_LOCATION
vertex_credentials: os.environ/VERTEXAI_CREDENTIALS
```
Start the proxy and make requests:
<Tabs>
<TabItem value="curl" label="Curl">
```bash
# Step 1: Generate video
curl --location 'http://0.0.0.0:4000/videos' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
"model": "veo-3",
"prompt": "Aerial shot over a futuristic city at sunrise",
"seconds": "8"
}'
# Step 2: Poll status
curl --location 'http://localhost:4000/v1/videos/{video_id}' \
--header 'x-litellm-api-key: sk-1234'
# Step 3: Download video
curl --location 'http://localhost:4000/v1/videos/{video_id}/content' \
--header 'x-litellm-api-key: sk-1234' \
--output video.mp4
```
</TabItem>
<TabItem value="python" label="Python SDK">
```python
import litellm
litellm.api_base = "http://0.0.0.0:4000"
litellm.api_key = "sk-1234"
response = litellm.video_generation(
model="veo-3",
prompt="Aerial shot over a futuristic city at sunrise",
)
status = litellm.video_status(video_id=response.id)
while status.status not in ["completed", "failed"]:
status = litellm.video_status(video_id=response.id)
if status.status == "completed":
content = litellm.video_content(video_id=response.id)
with open("veo_city.mp4", "wb") as f:
f.write(content)
```
</TabItem>
</Tabs>
## Cost Tracking
LiteLLM records the duration returned by Veo so you can apply duration-based pricing.
```python
with open("/path/to/service_account.json", "r", encoding="utf-8") as f:
vertex_credentials = f.read()
response = video_generation(
model="vertex_ai/veo-2.0-generate-001",
prompt="Flowers blooming in fast forward",
seconds="5",
vertex_project="your-gcp-project-id",
vertex_location="us-central1",
vertex_credentials=vertex_credentials,
)
print(response.usage) # {"duration_seconds": 5.0}
```
## Troubleshooting
- **`vertex_project is required`**: set `VERTEXAI_PROJECT` env var or pass `vertex_project` in the request.
- **`Permission denied`**: ensure the service account has the `Vertex AI User` role and the correct region enabled.
- **Video stuck in `processing`**: Veo operations are long-running. Continue polling every 10–15 seconds up to ~10 minutes.
## See Also
- [OpenAI Video Generation](../openai/videos.md)
- [Azure Video Generation](../azure/videos.md)
- [Gemini Video Generation](../gemini/videos.md)
- [Video Generation API Reference](/docs/videos)

19
docs/my-website/docs/proxy/config_settings.md
View file

@ -399,6 +399,8 @@ router_settings:
| AZURE_COMPUTER_USE_INPUT_COST_PER_1K_TOKENS | Input cost per 1K tokens for Azure Computer Use service
| AZURE_COMPUTER_USE_OUTPUT_COST_PER_1K_TOKENS | Output cost per 1K tokens for Azure Computer Use service
| AZURE_DEFAULT_RESPONSES_API_VERSION | Version of the Azure Default Responses API being used. Default is "preview"
| AZURE_DOCUMENT_INTELLIGENCE_API_VERSION | API version for Azure Document Intelligence service
| AZURE_DOCUMENT_INTELLIGENCE_DEFAULT_DPI | Default DPI (dots per inch) setting for Azure Document Intelligence service
| AZURE_TENANT_ID | Tenant ID for Azure Active Directory
| AZURE_USERNAME | Username for Azure services, use in conjunction with AZURE_PASSWORD for azure ad token with basic username/password workflow
| AZURE_PASSWORD | Password for Azure services, use in conjunction with AZURE_USERNAME for azure ad token with basic username/password workflow
@ -429,6 +431,12 @@ router_settings:
| CLOUDZERO_MAX_FETCHED_DATA_RECORDS | Maximum number of data records to fetch from CloudZero
| CLOUDZERO_TIMEZONE | Timezone for date handling (default: UTC)
| CONFIG_FILE_PATH | File path for configuration file
| CYBERARK_ACCOUNT | CyberArk account name for secret management
| CYBERARK_API_BASE | Base URL for CyberArk API
| CYBERARK_API_KEY | API key for CyberArk secret management service
| CYBERARK_CLIENT_CERT | Path to client certificate for CyberArk authentication
| CYBERARK_CLIENT_KEY | Path to client key for CyberArk authentication
| CYBERARK_USERNAME | Username for CyberArk authentication
| CONFIDENT_API_KEY | API key for DeepEval integration
| CUSTOM_TIKTOKEN_CACHE_DIR | Custom directory for Tiktoken cache
| CONFIDENT_API_KEY | API key for Confident AI (Deepeval) Logging service
@ -452,6 +460,8 @@ router_settings:
| DD_BASE_URL | Base URL for Datadog integration
| DATADOG_BASE_URL | (Alternative to DD_BASE_URL) Base URL for Datadog integration
| _DATADOG_BASE_URL | (Alternative to DD_BASE_URL) Base URL for Datadog integration
| DD_AGENT_HOST | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API
| DD_AGENT_PORT | Port of DataDog agent for log intake. Default is 10518
| DD_API_KEY | API key for Datadog integration
| DD_SITE | Site URL for Datadog (e.g., datadoghq.com)
| DD_SOURCE | Source identifier for Datadog logs
@ -470,6 +480,7 @@ router_settings:
| DEFAULT_FAILURE_THRESHOLD_PERCENT | Threshold percentage of failures to cool down a deployment. Default is 0.5 (50%)
| DEFAULT_FLUSH_INTERVAL_SECONDS | Default interval in seconds for flushing operations. Default is 5
| DEFAULT_HEALTH_CHECK_INTERVAL | Default interval in seconds for health checks. Default is 300 (5 minutes)
| DEFAULT_HEALTH_CHECK_PROMPT | Default prompt used during health checks for non-image models. Default is "test from litellm"
| DEFAULT_IMAGE_HEIGHT | Default height for images. Default is 300
| DEFAULT_IMAGE_TOKEN_COUNT | Default token count for images. Default is 250
| DEFAULT_IMAGE_WIDTH | Default width for images. Default is 300
@ -496,6 +507,7 @@ router_settings:
| DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH | Default minimal reasoning effort thinking budget for Gemini 2.5 Flash. Default is 512
| DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE | Default minimal reasoning effort thinking budget for Gemini 2.5 Flash Lite. Default is 512
| DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO | Default minimal reasoning effort thinking budget for Gemini 2.5 Pro. Default is 512
| DEFAULT_REDIS_MAJOR_VERSION | Default Redis major version to assume when version cannot be determined. Default is 7
| DEFAULT_REDIS_SYNC_INTERVAL | Default Redis synchronization interval in seconds. Default is 1
| DEFAULT_REPLICATE_GPU_PRICE_PER_SECOND | Default price per second for Replicate GPU. Default is 0.001400
| DEFAULT_REPLICATE_POLLING_DELAY_SECONDS | Default delay in seconds for Replicate polling. Default is 1
@ -507,6 +519,7 @@ router_settings:
| DEFAULT_SLACK_ALERTING_THRESHOLD | Default threshold for Slack alerting. Default is 300
| DEFAULT_SOFT_BUDGET | Default soft budget for LiteLLM proxy keys. Default is 50.0
| DEFAULT_TRIM_RATIO | Default ratio of tokens to trim from prompt end. Default is 0.75
| DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS | Default duration for video generation in seconds in google. Default is 8
| DIRECT_URL | Direct URL for service endpoint
| DISABLE_ADMIN_UI | Toggle to disable the admin UI
| DISABLE_AIOHTTP_TRANSPORT | Flag to disable aiohttp transport. When this is set to True, litellm will use httpx instead of aiohttp. **Default is False**
@ -581,9 +594,14 @@ router_settings:
| HEROKU_API_KEY | API key for Heroku services
| HF_API_BASE | Base URL for Hugging Face API
| HCP_VAULT_ADDR | Address for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_APPROLE_MOUNT_PATH | Mount path for AppRole authentication in [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault). Default is "approle"
| HCP_VAULT_APPROLE_ROLE_ID | Role ID for AppRole authentication in [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_APPROLE_SECRET_ID | Secret ID for AppRole authentication in [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_CLIENT_CERT | Path to client certificate for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_CLIENT_KEY | Path to client key for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_MOUNT_NAME | Mount name for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_NAMESPACE | Namespace for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_PATH_PREFIX | Path prefix for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_TOKEN | Token for [Hashicorp Vault Secret Manager](../secret.md#hashicorp-vault)
| HCP_VAULT_CERT_ROLE | Role for [Hashicorp Vault Secret Manager Auth](../secret.md#hashicorp-vault)
| HELICONE_API_KEY | API key for Helicone service
@ -650,6 +668,7 @@ router_settings:
| LITELLM_OTEL_INTEGRATION_ENABLE_METRICS | Optionally enable emantic metrics for OTEL
| LITELLM_MASTER_KEY | Master key for proxy authentication
| LITELLM_MODE | Operating mode for LiteLLM (e.g., production, development)
| LITELLM_NON_ROOT | Flag to run LiteLLM in non-root mode for enhanced security in Docker containers
| LITELLM_RATE_LIMIT_WINDOW_SIZE | Rate limit window size for LiteLLM. Default is 60
| LITELLM_SALT_KEY | Salt key for encryption in LiteLLM
| LITELLM_SSL_CIPHERS | SSL/TLS cipher configuration for faster handshakes. Controls cipher suite preferences for OpenSSL connections.

455
docs/my-website/docs/proxy/guardrails/litellm_content_filter.md Normal file
View file

@ -0,0 +1,455 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# LiteLLM Content Filter
**Built-in guardrail** for detecting and filtering sensitive information using regex patterns and keyword matching. No external dependencies required.
## Overview
| Property | Details |
|----------|---------|
| Description | On-device guardrail for detecting and filtering sensitive information using regex patterns and keyword matching. Built into LiteLLM with no external dependencies. |
| Guardrail Name | `litellm_content_filter` |
| Detection Methods | Prebuilt regex patterns, custom regex, keyword matching |
| Actions | `BLOCK` (reject request), `MASK` (redact content) |
| Supported Modes | `pre_call`, `post_call`, `during_call` (streaming) |
| Performance | Fast - runs locally, no external API calls |
## Quick Start
## LiteLLM UI
### Step 1: Select LiteLLM Content Filter
Click "Add New Guardrail" and select "LiteLLM Content Filter" as your guardrail provider.
<Image img={require('../../../img/create_guard.gif')} alt="Select LiteLLM Content Filter" />
### Step 2: Configure Pattern Detection
Select the prebuilt entities you want to block or mask. In this example, we select "Email" to detect and block email addresses.
If you need to block a custom entity, you can add a custom regex pattern by clicking "Add custom regex".
<Image img={require('../../../img/add_Guard2.gif')} alt="Select prebuilt entities or add custom regex" />
### Step 3: Add Blocked Keywords
Enter specific keywords you want to block. This is useful if you have policies to block certain words or phrases.
<Image img={require('../../../img/create_guard3.gif')} alt="Add blocked keywords" />
### Step 4: Test Your Guardrail
After creating the guardrail, navigate to "Test Playground" to test it. Select the guardrail you just created.
Test examples:
- **Blocked keyword test**: Entering "hi blue" will trigger the block since we set "blue" as a blocked keyword
- **Pattern detection test**: Entering "Hi ishaan@berri.ai" will trigger the email pattern detector
<Image img={require('../../../img/add_guard5.gif')} alt="Test guardrail in playground" />
## LiteLLM Config.yaml Setup
### Step 1: Define Guardrails in config.yaml
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "content-filter-pre"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
# Prebuilt patterns for common PII
patterns:
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
# Custom blocked keywords
blocked_words:
- keyword: "confidential"
action: "BLOCK"
description: "Sensitive internal information"
```
### Step 2: Start LiteLLM Gateway
```shell
litellm --config config.yaml
```
### Step 3: Test Request
<Tabs>
<TabItem label="SSN Blocked" value="ssn-blocked">
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "My SSN is 123-45-6789"}
],
"guardrails": ["content-filter-pre"]
}'
```
**Response: HTTP 400 Error**
```json
{
"error": {
"message": {
"error": "Content blocked: us_ssn pattern detected",
"pattern": "us_ssn"
},
"code": "400"
}
}
```
</TabItem>
<TabItem label="Email Masked" value="email-masked">
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Contact me at john@example.com"}
],
"guardrails": ["content-filter-pre"]
}'
```
The request is sent to the LLM with the email masked:
```
Contact me at [EMAIL_REDACTED]
```
</TabItem>
</Tabs>
## Configuration
### Supported Modes
- **`pre_call`** - Run before LLM call, filters input messages
- **`post_call`** - Run after LLM call, filters output responses
- **`during_call`** - Run during streaming, filters each chunk in real-time
### Actions
- **`BLOCK`** - Reject the request with HTTP 400 error
- **`MASK`** - Replace sensitive content with redaction tags (e.g., `[EMAIL_REDACTED]`)
## Prebuilt Patterns
### Available Patterns
| Pattern Name | Description | Example |
|-------------|-------------|---------|
| `us_ssn` | US Social Security Numbers | `123-45-6789` |
| `email` | Email addresses | `user@example.com` |
| `phone` | Phone numbers | `+1-555-123-4567` |
| `visa` | Visa credit cards | `4532-1234-5678-9010` |
| `mastercard` | Mastercard credit cards | `5425-2334-3010-9903` |
| `amex` | American Express cards | `3782-822463-10005` |
| `aws_access_key` | AWS access keys | `AKIAIOSFODNN7EXAMPLE` |
| `aws_secret_key` | AWS secret keys | `wJalrXUtnFEMI/K7MDENG/bPxRfi...` |
| `github_token` | GitHub tokens | `ghp_16C7e42F292c6912E7710c838347Ae178B4a` |
### Using Prebuilt Patterns
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "pii-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
patterns:
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
- pattern_type: "prebuilt"
pattern_name: "aws_access_key"
action: "BLOCK"
```
## Custom Regex Patterns
Define your own regex patterns for domain-specific sensitive data:
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "custom-patterns"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
patterns:
# Custom employee ID format
- pattern_type: "regex"
pattern: '\b[A-Z]{3}-\d{4}\b'
name: "employee_id"
action: "MASK"
# Custom project code format
- pattern_type: "regex"
pattern: 'PROJECT-\d{6}'
name: "project_code"
action: "BLOCK"
```
## Keyword Filtering
Block or mask specific keywords:
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "keyword-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
blocked_words:
- keyword: "confidential"
action: "BLOCK"
description: "Internal confidential information"
- keyword: "proprietary"
action: "MASK"
description: "Proprietary company data"
- keyword: "secret_project"
action: "BLOCK"
```
### Loading Keywords from File
For large keyword lists, use a YAML file:
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "keyword-file-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
blocked_words_file: "/path/to/sensitive_keywords.yaml"
```
```yaml showLineNumbers title="sensitive_keywords.yaml"
blocked_words:
- keyword: "project_apollo"
action: "BLOCK"
description: "Confidential project codename"
- keyword: "internal_api"
action: "MASK"
description: "Internal API references"
- keyword: "customer_database"
action: "BLOCK"
description: "Protected database name"
```
## Streaming Support
Content filter works with streaming responses by checking each chunk:
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "streaming-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "during_call" # Check each streaming chunk
patterns:
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
```
```python
import openai
client = openai.OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000"
)
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Tell me about yourself"}],
stream=True,
extra_body={"guardrails": ["streaming-filter"]}
)
for chunk in response:
print(chunk.choices[0].delta.content)
# Emails automatically masked in real-time
```
## Customizing Redaction Tags
When using the `MASK` action, sensitive content is replaced with redaction tags. You can customize how these tags appear.
### Default Behavior
**Patterns:** Each pattern type gets its own tag based on the pattern name
```
Input: "My email is john@example.com and SSN is 123-45-6789"
Output: "My email is [EMAIL_REDACTED] and SSN is [US_SSN_REDACTED]"
```
**Keywords:** All keywords use the same generic tag
```
Input: "This is confidential and proprietary information"
Output: "This is [KEYWORD_REDACTED] and [KEYWORD_REDACTED] information"
```
### Customizing Tags
Use `pattern_redaction_format` and `keyword_redaction_tag` to change the redaction format:
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "custom-redaction"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
pattern_redaction_format: "***{pattern_name}***" # Use {pattern_name} placeholder
keyword_redaction_tag: "***REDACTED***"
patterns:
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "MASK"
blocked_words:
- keyword: "confidential"
action: "MASK"
```
**Output:**
```
Input: "Email john@example.com, SSN 123-45-6789, confidential data"
Output: "Email ***EMAIL***, SSN ***US_SSN***, ***REDACTED*** data"
```
**Key Points:**
- `pattern_redaction_format` must include `{pattern_name}` placeholder
- Pattern names are automatically uppercased (e.g., `email` → `EMAIL`)
- `keyword_redaction_tag` is a fixed string (no placeholders)
## Use Cases
### 1. PII Protection
Block or mask personally identifiable information before sending to LLMs:
```yaml
patterns:
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
```
### 2. Credential Detection
Prevent API keys and secrets from being exposed:
```yaml
patterns:
- pattern_type: "prebuilt"
pattern_name: "aws_access_key"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "github_token"
action: "BLOCK"
```
### 3. Sensitive Internal Data Protection
Block or mask references to confidential internal projects, codenames, or proprietary information:
```yaml
blocked_words:
- keyword: "project_titan"
action: "BLOCK"
description: "Confidential project codename"
- keyword: "internal_api"
action: "MASK"
description: "Internal system references"
```
For large lists of sensitive terms, use a file:
```yaml
blocked_words_file: "/path/to/sensitive_terms.yaml"
```
### 4. Compliance
Ensure regulatory compliance by filtering sensitive data types:
```yaml
patterns:
- pattern_type: "prebuilt"
pattern_name: "visa"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
```
## Troubleshooting
### Pattern Not Matching
**Issue:** Regex pattern isn't detecting expected content
**Solution:** Test your regex pattern:
```python
import re
pattern = r'\b[A-Z]{3}-\d{4}\b'
test_text = "Employee ID: ABC-1234"
print(re.search(pattern, test_text)) # Should match
```
### Multiple Pattern Matches
**Issue:** Text contains multiple sensitive patterns
**Solution:** First matching pattern/keyword is processed. Order patterns by priority:
```yaml
patterns:
# Most critical first
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
# Less critical
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
```

73
docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md
View file

@ -4,12 +4,12 @@ import TabItem from '@theme/TabItem';
# PANW Prisma AIRS
LiteLLM supports PANW Prisma AIRS (AI Runtime Security) guardrails via the [Prisma AIRS Scan API](https://pan.dev/prisma-airs/api/airuntimesecurity/scan-sync-request/). This integration provides **Security-as-Code** for AI applications using Palo Alto Networks' AI security platform.
LiteLLM supports PANW Prisma AIRS (AI Runtime Security) guardrails via the [Prisma AIRS Scan API](https://pan.dev/prisma-airs/api/airuntimesecurity/airuntimesecurityapi//). This integration provides **Security-as-Code** for AI applications using Palo Alto Networks' AI security platform.
## Features
- ✅ **Real-time prompt injection detection**
- ✅ **Malicious content filtering**
- ✅ **Malicious URL detection**
- ✅ **Data loss prevention (DLP)**
- ✅ **Sensitive content masking** - Automatically mask PII, credit cards, SSNs instead of blocking
- ✅ **Comprehensive threat detection** for AI models and datasets
@ -17,6 +17,7 @@ LiteLLM supports PANW Prisma AIRS (AI Runtime Security) guardrails via the [Pris
- ✅ **Synchronous scanning** with immediate response
- ✅ **Configurable security profiles**
- ✅ **Streaming support** - Real-time masking for streaming responses
- ✅ **Multi-turn conversation tracking** - Automatic session grouping in Prisma AIRS SCM logs
- ✅ **Fail-closed security** - Blocks requests if PANW API is unavailable (maximum security)
## Quick Start
@ -237,6 +238,74 @@ You can override guardrail settings on a per-request basis using the `metadata`
- **Note:** If your API key is not linked to a profile, you must provide `profile_name` or `profile_id`
:::
## Multi-Turn Conversation Tracking
PANW Prisma AIRS automatically tracks multi-turn conversations using LiteLLM's `litellm_trace_id`. This enables you to:
- **Group related requests** - All requests in a conversation share the same AI Session ID in Prisma AIRS SCM logs
- **Track conversation context** - See the full history of prompts and responses for a user session
- **Analyze attack patterns** - Identify sophisticated multi-turn attacks across conversation history
### How It Works
LiteLLM automatically generates a unique `litellm_trace_id` for each conversation session. The PANW guardrail uses this as the PANW transaction ID (which maps to "AI Session ID" in Strata Cloud Manager):
```
Conversation Session: litellm_trace_id = "abc-123-def-456"
Turn 1 (User): "What's the capital of France?"
→ Scan ID: scan_001 | Prisma AIRS AI Session ID: abc-123-def-456
Turn 2 (Assistant): "Paris is the capital of France."
→ Scan ID: scan_002 | Prisma AIRS AI Session ID: abc-123-def-456
Turn 3 (User): "What's the population?"
→ Scan ID: scan_003 | Prisma AIRS AI Session ID: abc-123-def-456
Turn 4 (Assistant): "Paris has approximately 2.1 million residents."
→ Scan ID: scan_004 | Prisma AIRS AI Session ID: abc-123-def-456
```
All scans appear under the same AI Session ID in Prisma AIRS logs, making it easy to:
- Review complete conversation history (all 4 turns grouped together)
- Identify patterns across multiple turns
- Correlate security events within a session
- Track the flow of user prompts and AI responses
### Session Tracking
LiteLLM automatically generates a unique `litellm_trace_id` for each request, which the PANW guardrail uses as the AI Session ID in Strata Cloud Manager. All prompt and response scans for a request are automatically grouped under the same session.
#### Custom Session IDs (Per-App Tracking)
You can provide your own `litellm_trace_id` to track sessions on a per-app or per-conversation basis:
```bash
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "capital of France"}],
"litellm_trace_id": "my-app-session-123", # Custom AI Session ID
"metadata": {
"profile_name": "dev-allow-all-profile", # Override security profile
"user_ip": "192.168.1.1", # Track user IP
"app_name": "eng" # Custom app identifier
},
"guardrails": ["panw-prisma-airs-pre-guard", "panw-prisma-airs-post-guard"]
}'
```
**Result in PANW SCM:**
- AI Session ID: `my-app-session-123`
- All prompt and response scans will be grouped under this custom session ID
- Perfect for tracking multi-turn conversations or per-application sessions
:::tip Viewing Sessions in Prisma AIRS SCM Logs
In Strata Cloud Manager, navigate to **AI Runtime > Sessions** to view all AI Session IDs and their associated scans. Click on a session to see the complete conversation history with security analysis.
:::
## Environment Variables
```bash

46
docs/my-website/docs/proxy/guardrails/test_playground.md Normal file
View file

@ -0,0 +1,46 @@
import Image from '@theme/IdealImage';
# Guardrail Testing Playground
Test and compare multiple guardrails in real-time with an interactive playground interface.
<Image img={require('../../../img/guardrail_playground.png')} alt="Guardrail Test Playground" />
## How to Use the Guardrail Testing Playground
The Guardrail Testing Playground allows you to quickly test and compare the behavior of different guardrails with sample inputs.
### Steps to Test Guardrails
1. **Navigate to the Guardrails Section**
- Open the LiteLLM Admin UI
- Go to the **Guardrails** section
2. **Open Test Playground**
- Click on the **Test Playground** tab at the top of the page
3. **Select Guardrails to Test**
- Check the guardrails you want to compare
- You can select multiple guardrails to see how they each respond to the same input
4. **Enter Your Input**
- Type or paste your test input in the text area
- This could be a prompt, message, or any text you want to validate against the guardrails
5. **Run the Test**
- Click the **Test guardrails** button (or press Enter)
6. **View Results**
- See the output from each selected guardrail
- Compare how different guardrails handle the same input
- Results will show whether the input passed or was blocked by each guardrail
## Use Cases
This is ideal for **Security Teams** & **LiteLLM Admins** evaluating guardrail solutions.
This brings the following benefits for LiteLLM users:
- **Compare guardrail responses**: test the same prompt across multiple providers (Lakera, Noma AI, Bedrock Guardrails, etc.) simultaneously.
- **Validate configurations**: verify your guardrails catch the threats you care about before production deployment.

7
docs/my-website/docs/proxy/health.md
View file

@ -106,6 +106,13 @@ model_list:
mode: image_generation # 👈 ADD THIS
```
#### Custom Health Check Prompt
By default, health checks use the prompt `"test from litellm"`. You can customize this prompt globally by setting an environment variable, or per-model via config:
```bash
DEFAULT_HEALTH_CHECK_PROMPT="this is a test prompt"
```
### Text Completion Models

89
docs/my-website/docs/secret_managers/hashicorp_vault.md
View file

@ -16,21 +16,28 @@ import Image from '@theme/IdealImage';
|---------|----------|-------------|
| Reading Secrets | ✅ | Read secrets e.g `OPENAI_API_KEY` |
| Writing Secrets | ✅ | Store secrets e.g `Virtual Keys` |
| Authentication Methods to Hashicorp Vault | ✅ | AppRole, TLS Certificate, Token |
Read secrets from [Hashicorp Vault](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2)
**Step 1.** Add Hashicorp Vault details in your environment
LiteLLM supports two methods of authentication:
LiteLLM supports three methods of authentication:
1. TLS cert authentication - `HCP_VAULT_CLIENT_CERT` and `HCP_VAULT_CLIENT_KEY`
2. Token authentication - `HCP_VAULT_TOKEN`
1. AppRole authentication (recommended) - `HCP_VAULT_APPROLE_ROLE_ID` and `HCP_VAULT_APPROLE_SECRET_ID`
2. TLS cert authentication - `HCP_VAULT_CLIENT_CERT` and `HCP_VAULT_CLIENT_KEY`
3. Token authentication - `HCP_VAULT_TOKEN`
```bash
HCP_VAULT_ADDR="https://test-cluster-public-vault-0f98180c.e98296b2.z1.hashicorp.cloud:8200"
HCP_VAULT_NAMESPACE="admin"
# Authentication via TLS cert
# Authentication via AppRole (recommended)
HCP_VAULT_APPROLE_ROLE_ID="your-role-id"
HCP_VAULT_APPROLE_SECRET_ID="your-secret-id"
HCP_VAULT_APPROLE_MOUNT_PATH="approle" # OPTIONAL. defaults to "approle"
# OR - Authentication via TLS cert
HCP_VAULT_CLIENT_CERT="path/to/client.pem"
HCP_VAULT_CLIENT_KEY="path/to/client.key"
@ -64,6 +71,80 @@ $ litellm --config /path/to/config.yaml
[Quick Test Proxy](../proxy/user_keys)
## Authentication Methods
LiteLLM supports three authentication methods for Hashicorp Vault, with the following priority:
1. **AppRole** - Recommended for production applications
2. **TLS Certificate** - For certificate-based authentication
3. **Token** - Direct token authentication
### 1. AppRole Authentication
To set up AppRole authentication:
1. Enable AppRole auth in Vault:
```bash
vault auth enable approle
```
2. Create a policy and role for LiteLLM:
```bash
# Create a policy file (litellm-policy.hcl)
path "secret/data/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Apply the policy
vault policy write litellm-policy litellm-policy.hcl
# Create an AppRole
vault write auth/approle/role/litellm \
token_policies="litellm-policy" \
token_ttl=32d \
token_max_ttl=32d
```
3. Get your Role ID and Secret ID:
```bash
# Get Role ID
vault read auth/approle/role/litellm/role-id
# Generate Secret ID
vault write -f auth/approle/role/litellm/secret-id
```
4. Set the environment variables:
```bash
export HCP_VAULT_APPROLE_ROLE_ID="your-role-id"
export HCP_VAULT_APPROLE_SECRET_ID="your-secret-id"
```
### 2. TLS Certificate Authentication
TLS Certificate authentication uses client certificates for mutual TLS authentication with Vault.
**Environment Variables:**
```bash
export HCP_VAULT_CLIENT_CERT="path/to/client.pem"
export HCP_VAULT_CLIENT_KEY="path/to/client.key"
export HCP_VAULT_CERT_ROLE="your-cert-role" # Optional
```
**How it works:**
- LiteLLM uses the client certificate and key for mutual TLS authentication
- Vault validates the certificate and issues a temporary token
- The token is cached for the duration of its lease
### 3. Token Authentication
Direct token authentication uses a static Vault token.
**Environment Variables:**
```bash
export HCP_VAULT_TOKEN="hvs.CAESIG52gL6ljBSdmq*****"
```
## How it works
**Reading Secrets**

92
docs/my-website/docs/videos.md
View file

@ -9,7 +9,7 @@ Fallbacks | ✅ (Between supported models) |
| Guardrails Support | ✅ Content moderation and safety checks |
| Proxy Server Support | ✅ Full proxy integration with virtual keys |
| Spend Management | ✅ Budget tracking and rate limiting |
| Supported Providers | `openai`, `azure` |
| Supported Providers | `openai`, `azure`, `gemini`, `vertex_ai` |
:::tip
@ -41,8 +41,7 @@ print(f"Initial Status: {response.status}")
# Check status until video is ready
while True:
status_response = video_status(
video_id=response.id,
custom_llm_provider="openai"
video_id=response.id
)
print(f"Current Status: {status_response.status}")
@ -57,8 +56,7 @@ while True:
# Download video content when ready
video_bytes = video_content(
video_id=response.id,
custom_llm_provider="openai"
video_id=response.id
)
# Save to file
@ -88,8 +86,7 @@ async def test_async_video():
# Check status until video is ready
while True:
status_response = await avideo_status(
video_id=response.id,
custom_llm_provider="openai"
video_id=response.id
)
print(f"Current Status: {status_response.status}")
@ -104,8 +101,7 @@ async def test_async_video():
# Download video content when ready
video_bytes = await avideo_content(
video_id=response.id,
custom_llm_provider="openai"
video_id=response.id
)
# Save to file
@ -120,21 +116,27 @@ asyncio.run(test_async_video())
```python
from litellm import video_status
# Check the status of a video generation
status_response = video_status(
video_id="video_1234567890",
custom_llm_provider="openai"
video_id="video_1234567890"
)
print(f"Video Status: {status_response.status}")
print(f"Created At: {status_response.created_at}")
print(f"Model: {status_response.model}")
```
# Possible status values:
# - "queued": Video is in the queue
# - "processing": Video is being generated
# - "completed": Video is ready for download
# - "failed": Video generation failed
### List Videos
For listing videos, you need to specify the provider since there's no video_id to decode from:
```python
from litellm import video_list
# List videos from OpenAI
videos = video_list(custom_llm_provider="openai")
for video in videos:
print(f"Video ID: {video['id']}")
```
### Video Generation with Reference Image
@ -207,7 +209,7 @@ print(f"Video ID: {response.id}")
LiteLLM provides OpenAI API compatible video endpoints for complete video generation workflow:
- `/videos/generations` - Generate new videos
- `/videos` - Generate new videos
- `/videos/remix` - Edit existing videos with reference images
- `/videos/status` - Check video generation status
- `/videos/retrieval` - Download completed videos
@ -227,7 +229,6 @@ model_list:
model: azure/sora-2
api_key: os.environ/AZURE_OPENAI_API_KEY
api_base: os.environ/AZURE_OPENAI_API_BASE
api_version: "2024-02-15-preview"
```
Start litellm
@ -253,31 +254,14 @@ curl --location 'http://localhost:4000/v1/videos' \
Test video status request
```bash
# Using custom-llm-provider header
curl --location 'http://localhost:4000/v1/videos/video_id' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: azure'
# Or using query parameter
curl --location 'http://localhost:4000/v1/videos/video_id?custom_llm_provider=azure' \
--header 'Accept: application/json' \
curl --location 'http://localhost:4000/v1/videos/{video_id}' \
--header 'x-litellm-api-key: sk-1234'
```
Test video retrieval request
```bash
# Using custom-llm-provider header
curl --location 'http://localhost:4000/v1/videos/video_id/content' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: openai' \
--output video.mp4
# Or using query parameter
curl --location 'http://localhost:4000/v1/videos/video_id/content?custom_llm_provider=openai' \
--header 'Accept: application/json' \
curl --location 'http://localhost:4000/v1/videos/{video_id}/content' \
--header 'x-litellm-api-key: sk-1234' \
--output video.mp4
```
@ -285,27 +269,27 @@ curl --location 'http://localhost:4000/v1/videos/video_id/content?custom_llm_pro
Test video remix request
```bash
# Using custom_llm_provider in request body
curl --location --request POST 'http://localhost:4000/v1/videos/video_id/remix' \
--header 'Accept: application/json' \
curl --location --request POST 'http://localhost:4000/v1/videos/{video_id}/remix' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--data '{
"prompt": "New remix instructions",
"custom_llm_provider": "azure"
}'
# Or using custom-llm-provider header
curl --location --request POST 'http://localhost:4000/v1/videos/video_id/remix' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: azure' \
--data '{
"prompt": "New remix instructions"
}'
```
Test video list request (requires custom_llm_provider)
```bash
# Note: video_list requires custom_llm_provider since there's no video_id to decode from
curl --location 'http://localhost:4000/v1/videos?custom_llm_provider=openai' \
--header 'x-litellm-api-key: sk-1234'
# Or using header
curl --location 'http://localhost:4000/v1/videos' \
--header 'x-litellm-api-key: sk-1234' \
--header 'custom-llm-provider: azure'
```
Test Azure video generation request
```bash
@ -618,4 +602,6 @@ The response follows OpenAI's video generation format with the following structu
| Provider | Link to Usage |
|-------------|--------------------|
| OpenAI | [Usage](providers/openai/videos) |
| Azure | [Usage](providers/azure/videos) |
| Azure | [Usage](providers/azure/videos) |
| Gemini | [Usage](providers/gemini/videos) |
| Vertex AI | [Usage](providers/vertex_ai/videos) |

BIN
docs/my-website/img/add_Guard2.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2.4 MiB

BIN
docs/my-website/img/add_guard5.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 5 MiB

BIN
docs/my-website/img/create_guard.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2.9 MiB

BIN
docs/my-website/img/create_guard3.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.8 MiB

BIN
docs/my-website/img/guardrail_playground.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 603 KiB

BIN
docs/my-website/img/release_notes/built_in_guard.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 552 KiB

6
docs/my-website/release_notes/v1.79.1-stable/index.md
View file

@ -1,5 +1,5 @@
---
title: "[Preview] v1.79.1-stable - FAL AI Support"
title: "v1.79.1-stable - Guardrail Playground"
slug: "v1-79-1"
date: 2025-11-01T10:00:00
authors:
@ -27,7 +27,7 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.0-stable
ghcr.io/berriai/litellm:v1.79.1-stable
```
</TabItem>
@ -151,7 +151,7 @@ pip install litellm==1.80.0
#### Features
- **[Container API](../../docs/container_api)**
- **[Container API](../../docs/containers)**
- Add end-to-end OpenAI Container API support to LiteLLM SDK - [PR #16136](https://github.com/BerriAI/litellm/pull/16136)
- Add proxy support for container APIs - [PR #16049](https://github.com/BerriAI/litellm/pull/16049)
- Add logging support for Container API - [PR #16049](https://github.com/BerriAI/litellm/pull/16049)

444
docs/my-website/release_notes/v1.79.3-stable/index.md Normal file
View file

@ -0,0 +1,444 @@
---
title: "[Preview] v1.79.3-stable - Built-in Guardrails on AI Gateway"
slug: "v1-79-3"
date: 2025-11-08T10:00:00
authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false
---
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
## Deploy this version
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.79.3.rc.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.79.3.rc.1
```
</TabItem>
</Tabs>
---
## Key Highlights
- **LiteLLM Custom Guardrail** - Built-in guardrail with UI configuration support
- **Performance Improvements** - `/responses` API 19× Lower Median Latency
- **Veo3 Video Generation (Vertex AI + Google AI Studio)** - Use OpenAI Video API to generate videos with Vertex AI and Google AI Studio Veo3 models
---
### Built-in Guardrails on AI Gateway
<Image
img={require('../../img/release_notes/built_in_guard.png')}
style={{width: '100%', display: 'block', margin: '2rem auto'}}
/>
<br/>
This release introduces built-in guardrails for LiteLLM AI Gateway, allowing you to enforce protections without depending on an external guardrail API.
- **Blocking Keywords** - Block known sensitive keywords like "litellm", "python", etc.
- **Pattern Detection** - Block known sensitive patterns like emails, Social Security Numbers, API keys, etc.
- **Custom Regex Patterns** - Define custom regex patterns for your specific use case.
Get started with the built-in guardrails on AI Gateway [here](https://docs.litellm.ai/docs/proxy/guardrails/litellm_content_filter).
---
### Performance – `/responses` 19× Lower Median Latency
This update significantly improves `/responses` latency by integrating our internal network management for connection handling, eliminating per-request setup overhead.
#### Results
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Median latency | 3,600 ms | **190 ms** | **−95% (~19× faster)** |
| p95 latency | 4,300 ms | **280 ms** | −93% |
| p99 latency | 4,600 ms | **590 ms** | −87% |
| Average latency | 3,571 ms | **208 ms** | −94% |
| RPS | 231 | **1,059** | +358% |
#### Test Setup
| Category | Specification |
|----------|---------------|
| **Load Testing** | Locust: 1,000 concurrent users, 500 ramp-up |
| **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances |
| **Database** | PostgreSQL (Redis unused) |
| **Configuration** | [config.yaml](https://gist.github.com/AlexsanderHamir/550791675fd752befcac6a9e44024652) |
| **Load Script** | [no_cache_hits.py](https://gist.github.com/AlexsanderHamir/99d673bf74cdd81fd39f59fa9048f2e8) |
---
## New Models / Updated Models
#### New Model Support
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
| Azure | `azure/gpt-5-pro` | 272K | $15.00 | $120.00 | Responses API, reasoning, vision, PDF input |
| Azure | `azure/gpt-image-1-mini` | - | - | - | Image generation - per pixel pricing |
| Azure | `azure/container` | - | - | - | Container API - $0.03/session |
| OpenAI | `openai/container` | - | - | - | Container API - $0.03/session |
| Cohere | `cohere/embed-v4.0` | 128K | $0.12 | - | Embeddings with image input support |
| Gemini | `gemini/gemini-live-2.5-flash-preview-native-audio-09-2025` | 1M | $0.30 | $2.00 | Native audio, vision, web search |
| Vertex AI | `vertex_ai/minimaxai/minimax-m2-maas` | 196K | $0.30 | $1.20 | Function calling, tool choice |
| NVIDIA | `nvidia/nemotron-nano-9b-v2` | - | - | - | Chat completions |
#### OCR Models
| Provider | Model | Cost Per Page | Features |
| -------- | ----- | ------------- | -------- |
| Azure AI | `azure_ai/doc-intelligence/prebuilt-read` | $0.0015 | Document reading |
| Azure AI | `azure_ai/doc-intelligence/prebuilt-layout` | $0.01 | Layout analysis |
| Azure AI | `azure_ai/doc-intelligence/prebuilt-document` | $0.01 | Document processing |
| Vertex AI | `vertex_ai/mistral-ocr-2505` | $0.0005 | OCR processing |
#### Search Models
| Provider | Model | Pricing | Features |
| -------- | ----- | ------- | -------- |
| Firecrawl | `firecrawl/search` | Tiered: $0.00166-$0.0166/query | 10-100 results per query |
| SearXNG | `searxng/search` | Free | Open-source metasearch |
#### Features
- **[Azure](../../docs/providers/azure)**
- Add Azure GPT-5-Pro Responses API support with reasoning capabilities - [PR #16235](https://github.com/BerriAI/litellm/pull/16235)
- Add gpt-image-1-mini pricing for Azure with quality tiers (low/medium/high) - [PR #16182](https://github.com/BerriAI/litellm/pull/16182)
- Add support for returning Azure Content Policy error information when exceptions from Azure OpenAI occur - [PR #16231](https://github.com/BerriAI/litellm/pull/16231)
- Fix Azure GPT-5 incorrectly routed to O-series config (temperature parameter unsupported) - [PR #16246](https://github.com/BerriAI/litellm/pull/16246)
- Fix Azure doesn't accept extra body param - [PR #16116](https://github.com/BerriAI/litellm/pull/16116)
- Fix Azure DALL-E-3 health check content policy violation by using safe default prompt - [PR #16329](https://github.com/BerriAI/litellm/pull/16329)
- **[Bedrock](../../docs/providers/bedrock)**
- Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - [PR #15850](https://github.com/BerriAI/litellm/pull/15850)
- Fix: Filter AWS authentication params from Bedrock InvokeModel request body - [PR #16315](https://github.com/BerriAI/litellm/pull/16315)
- Fix Bedrock proxy adding name to file content, breaks when cache_control in use - [PR #16275](https://github.com/BerriAI/litellm/pull/16275)
- Fix global.anthropic.claude-haiku-4-5-20251001-v1:0 supports_reasoning flag and update pricing - [PR #16263](https://github.com/BerriAI/litellm/pull/16263)
- **[Gemini (Google AI Studio + Vertex AI)](../../docs/providers/gemini)**
- Add gemini live audio model cost in model map - [PR #16183](https://github.com/BerriAI/litellm/pull/16183)
- Fix translation problem with Gemini parallel tool calls - [PR #16194](https://github.com/BerriAI/litellm/pull/16194)
- Fix: Send Gemini API key via x-goog-api-key header with custom api_base - [PR #16085](https://github.com/BerriAI/litellm/pull/16085)
- Fix image_config.aspect_ratio not working for gemini-2.5-flash-image - [PR #15999](https://github.com/BerriAI/litellm/pull/15999)
- Fix Gemini minimal reasoning env overrides disabling thoughts - [PR #16347](https://github.com/BerriAI/litellm/pull/16347)
- Fix cache_read_input_token_cost for gemini-2.5-flash - [PR #16354](https://github.com/BerriAI/litellm/pull/16354)
- **[Anthropic](../../docs/providers/anthropic)**
- Fix Anthropic token counting for VertexAI - [PR #16171](https://github.com/BerriAI/litellm/pull/16171)
- Fix anthropic-adapter: properly translate Anthropic image format to OpenAI - [PR #16202](https://github.com/BerriAI/litellm/pull/16202)
- Enable automated prompt caching message format for Claude on Databricks - [PR #16200](https://github.com/BerriAI/litellm/pull/16200)
- Add support for Anthropic Memory Tool - [PR #16115](https://github.com/BerriAI/litellm/pull/16115)
- Propagate cache creation/read token costs for model info to fix Anthropic long context cost calculations - [PR #16376](https://github.com/BerriAI/litellm/pull/16376)
- **[Vertex AI](../../docs/providers/vertex_ai)**
- Add Vertex MiniMAX m2 model support - [PR #16373](https://github.com/BerriAI/litellm/pull/16373)
- Correctly map 429 Resource Exhausted to RateLimitError - [PR #16363](https://github.com/BerriAI/litellm/pull/16363)
- Add `vertex_credentials` support to `litellm.rerank()` for Vertex AI - [PR #16266](https://github.com/BerriAI/litellm/pull/16266)
- **[Databricks](../../docs/providers/databricks)**
- Fix databricks streaming - [PR #16368](https://github.com/BerriAI/litellm/pull/16368)
- **[Deepgram](../../docs/providers/deepgram)**
- Return the diarized transcript when it's required in the request - [PR #16133](https://github.com/BerriAI/litellm/pull/16133)
- **[Fireworks](../../docs/providers/fireworks_ai)**
- Update Fireworks audio endpoints to new `api.fireworks.ai` domains - [PR #16346](https://github.com/BerriAI/litellm/pull/16346)
- **[Cohere](../../docs/providers/cohere)**
- Add cohere embed-v4.0 model support - [PR #16358](https://github.com/BerriAI/litellm/pull/16358)
- **[Watsonx](../../docs/providers/watsonx)**
- Support `reasoning_effort` for watsonx chat models - [PR #16261](https://github.com/BerriAI/litellm/pull/16261)
- **[OpenAI](../../docs/providers/openai)**
- Remove automatic summary from reasoning_effort transformation - [PR #16210](https://github.com/BerriAI/litellm/pull/16210)
- **[XAI](../../docs/providers/xai)**
- Remove Grok 4 Models Reasoning Effort Parameter - [PR #16265](https://github.com/BerriAI/litellm/pull/16265)
- **[Hosted VLLM](../../docs/providers/vllm)**
- Fix HostedVLLMRerankConfig will not be used - [PR #16352](https://github.com/BerriAI/litellm/pull/16352)
#### New Provider Support
- **[Bedrock Agentcore](../../docs/providers/bedrock)**
- Add Bedrock Agentcore as a provider on LiteLLM Python SDK and LiteLLM AI Gateway - [PR #16252](https://github.com/BerriAI/litellm/pull/16252)
---
## LLM API Endpoints
#### Features
- **[OCR API](../../docs/ocr)**
- Add VertexAI OCR provider support + cost tracking - [PR #16216](https://github.com/BerriAI/litellm/pull/16216)
- Add Azure AI Doc Intelligence OCR support - [PR #16219](https://github.com/BerriAI/litellm/pull/16219)
- **[Search API](../../docs/search)**
- Add firecrawl search API support with tiered pricing - [PR #16257](https://github.com/BerriAI/litellm/pull/16257)
- Add searxng search API provider - [PR #16259](https://github.com/BerriAI/litellm/pull/16259)
- **[Responses API](../../docs/response_api)**
- Support responses API streaming in langfuse otel - [PR #16153](https://github.com/BerriAI/litellm/pull/16153)
- Pass extra_body parameters to provider in Responses API requests - [PR #16320](https://github.com/BerriAI/litellm/pull/16320)
- **[Container API](../../docs/container_api)**
- Add E2E Container API Support - [PR #16136](https://github.com/BerriAI/litellm/pull/16136)
- Update container documentation to be similar to others - [PR #16327](https://github.com/BerriAI/litellm/pull/16327)
- **[Video Generation API](../../docs/video_generation)**
- Add Vertex and Gemini Videos API with Cost Tracking + UI support - [PR #16323](https://github.com/BerriAI/litellm/pull/16323)
- Add `custom_llm_provider` support for video endpoints (non-generation) - [PR #16121](https://github.com/BerriAI/litellm/pull/16121)
- **[Audio API](../../docs/audio)**
- Add gpt-4o-transcribe cost tracking - [PR #16412](https://github.com/BerriAI/litellm/pull/16412)
- **[Vector Stores](../../docs/vector_stores)**
- Milvus - search vector store support + support multi-part form data on passthrough - [PR #16035](https://github.com/BerriAI/litellm/pull/16035)
- Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - [PR #16160](https://github.com/BerriAI/litellm/pull/16160)
- Milvus - Passthrough API support - adds create + read vector store support via passthrough API's - [PR #16170](https://github.com/BerriAI/litellm/pull/16170)
- **[Embeddings API](../../docs/embedding/supported_embedding)**
- Use valid CallTypes enum value in embeddings endpoint - [PR #16328](https://github.com/BerriAI/litellm/pull/16328)
- **[Rerank API](../../docs/rerank)**
- Generalize tiered pricing in generic cost calculator - [PR #16150](https://github.com/BerriAI/litellm/pull/16150)
#### Bugs
- **General**
- Fix index field not populated in streaming mode with n>1 and tool calls - [PR #15962](https://github.com/BerriAI/litellm/pull/15962)
- Pass aws_region_name in litellm_params - [PR #16321](https://github.com/BerriAI/litellm/pull/16321)
- Add `retry-after` header support for errors `502`, `503`, `504` - [PR #16288](https://github.com/BerriAI/litellm/pull/16288)
---
## Management Endpoints / UI
#### Features
- **Virtual Keys**
- UI - Delete Team Member with friction - [PR #16167](https://github.com/BerriAI/litellm/pull/16167)
- UI - Litellm test key audio support - [PR #16251](https://github.com/BerriAI/litellm/pull/16251)
- UI - Test Key Page Revert Model To Single Select - [PR #16390](https://github.com/BerriAI/litellm/pull/16390)
- **Models + Endpoints**
- UI - Add Model Existing Credentials Improvement - [PR #16166](https://github.com/BerriAI/litellm/pull/16166)
- UI - Add Azure AD Token field and Azure API Key optional - [PR #16331](https://github.com/BerriAI/litellm/pull/16331)
- UI - Fixed Label for vLLM in Model Create Flow - [PR #16285](https://github.com/BerriAI/litellm/pull/16285)
- UI - Include Model Access Group Models on Team Models Table - [PR #16298](https://github.com/BerriAI/litellm/pull/16298)
- Fix /model_group/info Returning Entire Model List for SSO Users - [PR #16296](https://github.com/BerriAI/litellm/pull/16296)
- Litellm non root docker Model Hub Table fix - [PR #16282](https://github.com/BerriAI/litellm/pull/16282)
- **Guardrails**
- UI - Fix regression where Guardrail Entity Could not be selected and entity was not displayed - [PR #16165](https://github.com/BerriAI/litellm/pull/16165)
- UI - Guardrail Info Page Show PII Config - [PR #16164](https://github.com/BerriAI/litellm/pull/16164)
- Change guardrail_information to list type - [PR #16127](https://github.com/BerriAI/litellm/pull/16127)
- UI - LiteLLM Guardrail - ensure you can see UI Friendly name for PII Patterns - [PR #16382](https://github.com/BerriAI/litellm/pull/16382)
- UI - Guardrails - LiteLLM Content Filter, Allow Viewing/Editing Content Filter Settings - [PR #16383](https://github.com/BerriAI/litellm/pull/16383)
- UI - Guardrails - allow updating guardrails through UI. Ensure litellm_params actually get updated in memory - [PR #16384](https://github.com/BerriAI/litellm/pull/16384)
- **SSO Settings**
- Support dot notation on ui sso - [PR #16135](https://github.com/BerriAI/litellm/pull/16135)
- UI - Prevent trailing slash in sso proxy base url input - [PR #16244](https://github.com/BerriAI/litellm/pull/16244)
- UI - SSO Proxy Base URL input validation and remove normalizing / - [PR #16332](https://github.com/BerriAI/litellm/pull/16332)
- UI - Surface SSO Create errors on create flow - [PR #16369](https://github.com/BerriAI/litellm/pull/16369)
- **Usage & Analytics**
- UI - Tag Usage Top Model Table View and Label Fix - [PR #16249](https://github.com/BerriAI/litellm/pull/16249)
- UI - Litellm usage date picker - [PR #16264](https://github.com/BerriAI/litellm/pull/16264)
- **Cache Settings**
- UI - Cache Settings Redis Add Semantic Cache Settings - [PR #16398](https://github.com/BerriAI/litellm/pull/16398)
#### Bugs
- **General**
- UI - Remove encoding_format in request for embedding models - [PR #16367](https://github.com/BerriAI/litellm/pull/16367)
- UI - Revert Changes for Test Key Multiple Model Select - [PR #16372](https://github.com/BerriAI/litellm/pull/16372)
- UI - Various Small Issues - [PR #16406](https://github.com/BerriAI/litellm/pull/16406)
---
## AI Integrations
### Logging
- **[Langfuse](../../docs/proxy/logging#langfuse)**
- Fix langfuse input tokens logic for cached tokens - [PR #16203](https://github.com/BerriAI/litellm/pull/16203)
- **[Opik](../../docs/proxy/logging#opik)**
- Fix the bug with not incorrect attachment to existing trace & refactor - [PR #15529](https://github.com/BerriAI/litellm/pull/15529)
- **[S3](../../docs/proxy/logging#s3)**
- S3 logger, add support for ssl_verify when using minio logger - [PR #16211](https://github.com/BerriAI/litellm/pull/16211)
- Strip base64 in s3 - [PR #16157](https://github.com/BerriAI/litellm/pull/16157)
- Add allowing Key based prefix to s3 path - [PR #16237](https://github.com/BerriAI/litellm/pull/16237)
- Add Prometheus metric to track callback logging failures in S3 - [PR #16209](https://github.com/BerriAI/litellm/pull/16209)
- **[OpenTelemetry](../../docs/proxy/logging#opentelemetry)**
- OTEL - Log Cost Breakdown on OTEL Logger - [PR #16334](https://github.com/BerriAI/litellm/pull/16334)
- **[DataDog](../../docs/proxy/logging#datadog)**
- Add DD Agent Host support for `datadog` callback - [PR #16379](https://github.com/BerriAI/litellm/pull/16379)
### Guardrails
- **[Noma](../../docs/proxy/guardrails)**
- Revert Noma Apply Guardrail implementation - [PR #16214](https://github.com/BerriAI/litellm/pull/16214)
- Litellm noma guardrail support images - [PR #16199](https://github.com/BerriAI/litellm/pull/16199)
- **[PANW Prisma AIRS](../../docs/proxy/guardrails)**
- PANW prisma airs guardrail deduplication and enhanced session tracking - [PR #16273](https://github.com/BerriAI/litellm/pull/16273)
- **[LiteLLM Custom Guardrail](../../docs/proxy/guardrails)**
- Add LiteLLM Gateway built in guardrail - [PR #16338](https://github.com/BerriAI/litellm/pull/16338)
- UI - Allow configuring LiteLLM Custom Guardrail - [PR #16339](https://github.com/BerriAI/litellm/pull/16339)
- Bug Fix: Content Filter Guard - [PR #16414](https://github.com/BerriAI/litellm/pull/16414)
### Secret Managers
- **[CyberArk](../../docs/secret_managers)**
- Add CyberArk Secrets Manager Integration - [PR #16278](https://github.com/BerriAI/litellm/pull/16278)
- Cyber Ark - Add Key Rotations support - [PR #16289](https://github.com/BerriAI/litellm/pull/16289)
- **[HashiCorp Vault](../../docs/secret_managers)**
- Add configurable mount name and path prefix for HashiCorp Vault - [PR #16253](https://github.com/BerriAI/litellm/pull/16253)
- Secret Manager - Hashicorp, add auth via approle - [PR #16374](https://github.com/BerriAI/litellm/pull/16374)
- **[AWS Secrets Manager](../../docs/secret_managers)**
- Add tags and descriptions support to aws secrets manager - [PR #16224](https://github.com/BerriAI/litellm/pull/16224)
- **[Custom Secret Manager](../../docs/secret_managers)**
- Add Custom Secret Manager - Allow users to define and write a custom secret manager - [PR #16297](https://github.com/BerriAI/litellm/pull/16297)
- **General**
- Email Notifications - Ensure Users get Key Rotated Email - [PR #16292](https://github.com/BerriAI/litellm/pull/16292)
- Fix verify ssl on sts boto3 - [PR #16313](https://github.com/BerriAI/litellm/pull/16313)
---
## Spend Tracking, Budgets and Rate Limiting
- **Cost Tracking**
- Fix OpenAI Responses API streaming tests usage field names and cost calculation - [PR #16236](https://github.com/BerriAI/litellm/pull/16236)
---
## MCP Gateway
- **Configuration**
- Configure static mcp header - [PR #16179](https://github.com/BerriAI/litellm/pull/16179)
- Persist mcp credentials in db - [PR #16308](https://github.com/BerriAI/litellm/pull/16308)
## Performance / Loadbalancing / Reliability improvements
- **Memory Leak Fixes**
- Resolve memory accumulation caused by Pydantic 2.11+ deprecation warnings - [PR #16110](https://github.com/BerriAI/litellm/pull/16110)
- **Session Management**
- Add shared_session support to responses API - [PR #16260](https://github.com/BerriAI/litellm/pull/16260)
- **Error Handling**
- Gracefully handle connection closed errors during streaming - [PR #16294](https://github.com/BerriAI/litellm/pull/16294)
- Handle None values in daily spend sort key - [PR #16245](https://github.com/BerriAI/litellm/pull/16245)
- **Configuration**
- Remove minimum validation for cache control injection index - [PR #16149](https://github.com/BerriAI/litellm/pull/16149)
- Improve clearing logic - only remove unvisited endpoints - [PR #16400](https://github.com/BerriAI/litellm/pull/16400)
- **Redis**
- Handle float redis_version from AWS ElastiCache Valkey - [PR #16207](https://github.com/BerriAI/litellm/pull/16207)
- **Hooks**
- Add parallel execution handling in during_call_hook - [PR #16279](https://github.com/BerriAI/litellm/pull/16279)
- **Infrastructure**
- Install runtime node for prisma - [PR #16410](https://github.com/BerriAI/litellm/pull/16410)
---
## Documentation Updates
- **Provider Documentation**
- Docs - v1.79.1 - [PR #16163](https://github.com/BerriAI/litellm/pull/16163)
- Fix broken link on model_management.md - [PR #16217](https://github.com/BerriAI/litellm/pull/16217)
- Fix image generation response format - use 'images' array instead of 'image' object - [PR #16378](https://github.com/BerriAI/litellm/pull/16378)
- **General Documentation**
- Add minimum resource requirement for production - [PR #16146](https://github.com/BerriAI/litellm/pull/16146)
- Add benchmark comparison with other AI gateways - [PR #16248](https://github.com/BerriAI/litellm/pull/16248)
- LiteLLM content filter guard documentation - [PR #16413](https://github.com/BerriAI/litellm/pull/16413)
- Fix typo of the word orginal - [PR #16255](https://github.com/BerriAI/litellm/pull/16255)
- **Security**
- Remove tornado test files (including test.key), fixes Python 3.13 security issues - [PR #16342](https://github.com/BerriAI/litellm/pull/16342)
---
## New Contributors
* @steve-gore-snapdocs made their first contribution in [PR #16149](https://github.com/BerriAI/litellm/pull/16149)
* @timbmg made their first contribution in [PR #16120](https://github.com/BerriAI/litellm/pull/16120)
* @Nivg made their first contribution in [PR #16202](https://github.com/BerriAI/litellm/pull/16202)
* @pablobgar made their first contribution in [PR #16194](https://github.com/BerriAI/litellm/pull/16194)
* @AlanPonnachan made their first contribution in [PR #16150](https://github.com/BerriAI/litellm/pull/16150)
* @Chesars made their first contribution in [PR #16236](https://github.com/BerriAI/litellm/pull/16236)
* @bowenliang123 made their first contribution in [PR #16255](https://github.com/BerriAI/litellm/pull/16255)
* @dean-zavad made their first contribution in [PR #16199](https://github.com/BerriAI/litellm/pull/16199)
* @alexkuzmik made their first contribution in [PR #15529](https://github.com/BerriAI/litellm/pull/15529)
* @Granine made their first contribution in [PR #16281](https://github.com/BerriAI/litellm/pull/16281)
* @Oodapow made their first contribution in [PR #16279](https://github.com/BerriAI/litellm/pull/16279)
* @jgoodyear made their first contribution in [PR #16275](https://github.com/BerriAI/litellm/pull/16275)
* @Qanpi made their first contribution in [PR #16321](https://github.com/BerriAI/litellm/pull/16321)
* @ShimonMimoun made their first contribution in [PR #16313](https://github.com/BerriAI/litellm/pull/16313)
* @andriykislitsyn made their first contribution in [PR #16288](https://github.com/BerriAI/litellm/pull/16288)
* @reckless-huang made their first contribution in [PR #16263](https://github.com/BerriAI/litellm/pull/16263)
* @chenmoneygithub made their first contribution in [PR #16368](https://github.com/BerriAI/litellm/pull/16368)
* @stembe-digitalex made their first contribution in [PR #16354](https://github.com/BerriAI/litellm/pull/16354)
* @jfcherng made their first contribution in [PR #16352](https://github.com/BerriAI/litellm/pull/16352)
* @xingyaoww made their first contribution in [PR #16246](https://github.com/BerriAI/litellm/pull/16246)
* @emerzon made their first contribution in [PR #16373](https://github.com/BerriAI/litellm/pull/16373)
* @wwwillchen made their first contribution in [PR #16376](https://github.com/BerriAI/litellm/pull/16376)
* @fabriciojoc made their first contribution in [PR #16203](https://github.com/BerriAI/litellm/pull/16203)
* @jroberts2600 made their first contribution in [PR #16273](https://github.com/BerriAI/litellm/pull/16273)
---
## Full Changelog
**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.79.1-nightly...v1.79.2.rc.1)**

5
docs/my-website/sidebars.js
View file

@ -31,6 +31,7 @@ const sidebars = {
label: "Guardrails",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/test_playground",
...[
"adding_provider/adding_guardrail_support",
"proxy/guardrails/aim_security",
@ -41,6 +42,7 @@ const sidebars = {
"proxy/guardrails/ibm_guardrails",
"proxy/guardrails/grayswan",
"proxy/guardrails/lasso_security",
"proxy/guardrails/litellm_content_filter",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
"proxy/guardrails/model_armor",
@ -477,6 +479,7 @@ const sidebars = {
label: "Vertex AI",
items: [
"providers/vertex",
"providers/vertex_ai/videos",
"providers/vertex_partner",
"providers/vertex_self_deployed",
"providers/vertex_image",
@ -489,6 +492,7 @@ const sidebars = {
label: "Google AI Studio",
items: [
"providers/gemini",
"providers/gemini/videos",
"providers/google_ai_studio/files",
"providers/google_ai_studio/image_gen",
"providers/google_ai_studio/realtime",
@ -780,6 +784,7 @@ const sidebars = {
"projects/GPTLocalhost",
"projects/HolmesGPT",
"projects/Railtracks",
"projects/Softgen",
],
},
"extras/code_quality",

13
enterprise/enterprise_hooks/aporia_ai.py
View file

@ -8,6 +8,8 @@
import os
import sys
from litellm.types.utils import CallTypesLiteral
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
@ -166,16 +168,7 @@ class AporiaGuardrail(CustomGuardrail):
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
):
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_to_applied_guardrails_header,

14
enterprise/enterprise_hooks/google_text_moderation.py
View file

@ -6,14 +6,13 @@
# +-----------------------------------------------+
# Thank you users! We ❤️ you! - Krrish & Ishaan
from typing import Literal
from fastapi import HTTPException
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.utils import CallTypesLiteral
class _ENTERPRISE_GoogleTextModeration(CustomLogger):
@ -89,16 +88,7 @@ class _ENTERPRISE_GoogleTextModeration(CustomLogger):
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
):
"""
- Calls Google's Text Moderation API

13
enterprise/enterprise_hooks/openai_moderation.py
View file

@ -12,7 +12,6 @@ sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import sys
from typing import Literal
from fastapi import HTTPException
@ -20,6 +19,7 @@ import litellm
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.utils import CallTypesLiteral
class _ENTERPRISE_OpenAI_Moderation(CustomLogger):
@ -35,16 +35,7 @@ class _ENTERPRISE_OpenAI_Moderation(CustomLogger):
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
):
text = ""
if "messages" in data and isinstance(data["messages"], list):

13
enterprise/litellm_enterprise/enterprise_callbacks/llama_guard.py
View file

@ -23,7 +23,7 @@ import litellm
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.utils import Choices, ModelResponse
from litellm.types.utils import CallTypesLiteral, Choices, ModelResponse
class _ENTERPRISE_LlamaGuard(CustomLogger):
@ -98,16 +98,7 @@ class _ENTERPRISE_LlamaGuard(CustomLogger):
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
):
"""
- Calls the Llama Guard Endpoint

12
enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py
View file

@ -17,6 +17,7 @@ from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import CallTypesLiteral
from litellm.utils import get_formatted_prompt
@ -120,16 +121,7 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
):
"""
- Calls the LLM Guard Endpoint

14
enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py
View file

@ -31,6 +31,7 @@ from litellm.types.integrations.pagerduty import (
PagerDutyRequestBody,
)
from litellm.types.utils import (
CallTypesLiteral,
StandardLoggingPayload,
StandardLoggingPayloadErrorInformation,
)
@ -142,18 +143,7 @@ class PagerDutyAlerting(SlackAlerting):
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: dict,
call_type: Literal[
"completion",
"text_completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"pass_through_endpoint",
"rerank",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
) -> Optional[Union[Exception, str, dict]]:
"""
Example of detecting hanging requests by waiting a given threshold.

24
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -36,6 +36,7 @@ from litellm.types.llms.openai import (
OpenAIFilesPurpose,
)
from litellm.types.utils import (
CallTypesLiteral,
LiteLLMBatch,
LiteLLMFineTuningJob,
LLMResponseTypes,
@ -272,28 +273,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: Dict,
call_type: Literal[
"completion",
"text_completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"pass_through_endpoint",
"rerank",
"acreate_batch",
"aretrieve_batch",
"acreate_file",
"afile_list",
"afile_delete",
"afile_content",
"acreate_fine_tuning_job",
"aretrieve_fine_tuning_job",
"alist_fine_tuning_jobs",
"acancel_fine_tuning_job",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
) -> Union[Exception, str, Dict, None]:
"""
- Detect litellm_proxy/ file_id

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.2-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.2.tar.gz vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.3-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
litellm-proxy-extras/dist/litellm_proxy_extras-0.4.3.tar.gz vendored Normal file
View file

Binary file not shown.

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20251104220043_add_credentials_to_mcp_servers/migration.sql Normal file
View file

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "credentials" JSONB DEFAULT '{}';

1
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -174,6 +174,7 @@ model LiteLLM_MCPServerTable {
url String?
transport String @default("sse")
auth_type String?
credentials Json? @default("{}")
created_at DateTime? @default(now()) @map("created_at")
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-proxy-extras"
version = "0.4.1"
version = "0.4.3"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
authors = ["BerriAI"]
readme = "README.md"
@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "0.4.1"
version = "0.4.3"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-proxy-extras==",

139
litellm/__init__.py
View file

@ -4,7 +4,9 @@ import warnings
warnings.filterwarnings("ignore", message=".*conflict with protected namespace.*")
# Suppress Pydantic 2.11+ deprecation warning about accessing model_fields on instances
# This warning can accumulate during streaming and cause memory leaks
warnings.filterwarnings("ignore", message=".*Accessing the.*attribute on the instance is deprecated.*")
warnings.filterwarnings(
"ignore", message=".*Accessing the.*attribute on the instance is deprecated.*"
)
### INIT VARIABLES #######################
import threading
import os
@ -32,7 +34,7 @@ from litellm.types.utils import (
all_litellm_params as _litellm_completion_params,
CredentialItem,
PriorityReservationDict,
) # maintain backwards compatibility for root param
) # maintain backwards compatibility for root param.
from litellm._logging import (
set_verbose,
_turn_on_debug,
@ -179,22 +181,22 @@ prometheus_initialize_budget_metrics: Optional[bool] = False
require_auth_for_metrics_endpoint: Optional[bool] = False
argilla_batch_size: Optional[int] = None
datadog_use_v1: Optional[bool] = False # if you want to use v1 datadog logged payload.
gcs_pub_sub_use_v1: Optional[bool] = (
False # if you want to use v1 gcs pubsub logged payload
)
generic_api_use_v1: Optional[bool] = (
False # if you want to use v1 generic api logged payload
)
gcs_pub_sub_use_v1: Optional[
bool
] = False # if you want to use v1 gcs pubsub logged payload
generic_api_use_v1: Optional[
bool
] = False # if you want to use v1 generic api logged payload
argilla_transformation_object: Optional[Dict[str, Any]] = None
_async_input_callback: List[Union[str, Callable, CustomLogger]] = (
[]
) # internal variable - async custom callbacks are routed here.
_async_success_callback: List[Union[str, Callable, CustomLogger]] = (
[]
) # internal variable - async custom callbacks are routed here.
_async_failure_callback: List[Union[str, Callable, CustomLogger]] = (
[]
) # internal variable - async custom callbacks are routed here.
_async_input_callback: List[
Union[str, Callable, CustomLogger]
] = [] # internal variable - async custom callbacks are routed here.
_async_success_callback: List[
Union[str, Callable, CustomLogger]
] = [] # internal variable - async custom callbacks are routed here.
_async_failure_callback: List[
Union[str, Callable, CustomLogger]
] = [] # internal variable - async custom callbacks are routed here.
pre_call_rules: List[Callable] = []
post_call_rules: List[Callable] = []
turn_off_message_logging: Optional[bool] = False
@ -202,18 +204,18 @@ log_raw_request_response: bool = False
redact_messages_in_exceptions: Optional[bool] = False
redact_user_api_key_info: Optional[bool] = False
filter_invalid_headers: Optional[bool] = False
add_user_information_to_llm_headers: Optional[bool] = (
None # adds user_id, team_id, token hash (params from StandardLoggingMetadata) to request headers
)
add_user_information_to_llm_headers: Optional[
bool
] = None # adds user_id, team_id, token hash (params from StandardLoggingMetadata) to request headers
store_audit_logs = False # Enterprise feature, allow users to see audit logs
### end of callbacks #############
email: Optional[str] = (
None # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
)
token: Optional[str] = (
None # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
)
email: Optional[
str
] = None # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
token: Optional[
str
] = None # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
telemetry = True
max_tokens: int = DEFAULT_MAX_TOKENS # OpenAI Defaults
drop_params = bool(os.getenv("LITELLM_DROP_PARAMS", False))
@ -269,9 +271,9 @@ use_client: bool = False
ssl_verify: Union[str, bool] = True
ssl_security_level: Optional[str] = None
ssl_certificate: Optional[str] = None
ssl_ecdh_curve: Optional[str] = (
None # Set to 'X25519' to disable PQC and improve performance
)
ssl_ecdh_curve: Optional[
str
] = None # Set to 'X25519' to disable PQC and improve performance
disable_streaming_logging: bool = False
disable_token_counter: bool = False
disable_add_transform_inline_image_block: bool = False
@ -317,24 +319,20 @@ enable_loadbalancing_on_batch_endpoints: Optional[bool] = None
enable_caching_on_provider_specific_optional_params: bool = (
False # feature-flag for caching on optional params - e.g. 'top_k'
)
caching: bool = (
False # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
)
caching_with_models: bool = (
False # # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
)
cache: Optional[Cache] = (
None # cache object <- use this - https://docs.litellm.ai/docs/caching
)
caching: bool = False # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
caching_with_models: bool = False # # Not used anymore, will be removed in next MAJOR release - https://github.com/BerriAI/litellm/discussions/648
cache: Optional[
Cache
] = None # cache object <- use this - https://docs.litellm.ai/docs/caching
default_in_memory_ttl: Optional[float] = None
default_redis_ttl: Optional[float] = None
default_redis_batch_cache_expiry: Optional[float] = None
model_alias_map: Dict[str, str] = {}
model_group_settings: Optional["ModelGroupSettings"] = None
max_budget: float = 0.0 # set the max budget across all providers
budget_duration: Optional[str] = (
None # proxy only - resets budget after fixed duration. You can set duration as seconds ("30s"), minutes ("30m"), hours ("30h"), days ("30d").
)
budget_duration: Optional[
str
] = None # proxy only - resets budget after fixed duration. You can set duration as seconds ("30s"), minutes ("30m"), hours ("30h"), days ("30d").
default_soft_budget: float = (
DEFAULT_SOFT_BUDGET # by default all litellm proxy keys have a soft budget of 50.0
)
@ -343,15 +341,11 @@ forward_traceparent_to_llm_provider: bool = False
_current_cost = 0.0 # private variable, used if max budget is set
error_logs: Dict = {}
add_function_to_prompt: bool = (
False # if function calling not supported by api, append function call details to system prompt
)
add_function_to_prompt: bool = False # if function calling not supported by api, append function call details to system prompt
client_session: Optional[httpx.Client] = None
aclient_session: Optional[httpx.AsyncClient] = None
model_fallbacks: Optional[List] = None # Deprecated for 'litellm.fallbacks'
model_cost_map_url: str = (
"https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
)
model_cost_map_url: str = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
suppress_debug_info = False
dynamodb_table_name: Optional[str] = None
s3_callback_params: Optional[Dict] = None
@ -381,9 +375,7 @@ prometheus_metrics_config: Optional[List] = None
disable_add_prefix_to_prompt: bool = (
False # used by anthropic, to disable adding prefix to prompt
)
disable_copilot_system_to_assistant: bool = (
False # If false (default), converts all 'system' role messages to 'assistant' for GitHub Copilot compatibility. Set to true to disable this behavior.
)
disable_copilot_system_to_assistant: bool = False # If false (default), converts all 'system' role messages to 'assistant' for GitHub Copilot compatibility. Set to true to disable this behavior.
public_model_groups: Optional[List[str]] = None
public_model_groups_links: Dict[str, str] = {}
#### REQUEST PRIORITIZATION #######
@ -394,17 +386,13 @@ priority_reservation_settings: "PriorityReservationSettings" = (
######## Networking Settings ########
use_aiohttp_transport: bool = (
True # Older variable, aiohttp is now the default. use disable_aiohttp_transport instead.
)
use_aiohttp_transport: bool = True # Older variable, aiohttp is now the default. use disable_aiohttp_transport instead.
aiohttp_trust_env: bool = False # set to true to use HTTP_ Proxy settings
disable_aiohttp_transport: bool = False # Set this to true to use httpx instead
disable_aiohttp_trust_env: bool = (
False # When False, aiohttp will respect HTTP(S)_PROXY env vars
)
force_ipv4: bool = (
False # when True, litellm will force ipv4 for all LLM requests. Some users have seen httpx ConnectionError when using ipv6.
)
force_ipv4: bool = False # when True, litellm will force ipv4 for all LLM requests. Some users have seen httpx ConnectionError when using ipv6.
module_level_aclient = AsyncHTTPHandler(
timeout=request_timeout, client_alias="module level aclient"
)
@ -418,13 +406,13 @@ fallbacks: Optional[List] = None
context_window_fallbacks: Optional[List] = None
content_policy_fallbacks: Optional[List] = None
allowed_fails: int = 3
num_retries_per_request: Optional[int] = (
None # for the request overall (incl. fallbacks + model retries)
)
num_retries_per_request: Optional[
int
] = None # for the request overall (incl. fallbacks + model retries)
####### SECRET MANAGERS #####################
secret_manager_client: Optional[Any] = (
None # list of instantiated key management clients - e.g. azure kv, infisical, etc.
)
secret_manager_client: Optional[
Any
] = None # list of instantiated key management clients - e.g. azure kv, infisical, etc.
_google_kms_resource_name: Optional[str] = None
_key_management_system: Optional[KeyManagementSystem] = None
_key_management_settings: KeyManagementSettings = KeyManagementSettings()
@ -434,9 +422,9 @@ output_parse_pii: bool = False
from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map
model_cost = get_model_cost_map(url=model_cost_map_url)
cost_discount_config: Dict[str, float] = (
{}
) # Provider-specific cost discounts {"vertex_ai": 0.05} = 5% discount
cost_discount_config: Dict[
str, float
] = {} # Provider-specific cost discounts {"vertex_ai": 0.05} = 5% discount
custom_prompt_dict: Dict[str, dict] = {}
check_provider_endpoint = False
@ -492,6 +480,7 @@ vertex_deepseek_models: Set = set()
vertex_ai_ai21_models: Set = set()
vertex_mistral_models: Set = set()
vertex_openai_models: Set = set()
vertex_minimax_models: Set = set()
ai21_models: Set = set()
ai21_chat_models: Set = set()
nlp_cloud_models: Set = set()
@ -652,6 +641,9 @@ def add_known_models():
elif value.get("litellm_provider") == "vertex_ai-openai_models":
key = key.replace("vertex_ai/", "")
vertex_openai_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-minimax_models":
key = key.replace("vertex_ai/", "")
vertex_minimax_models.add(key)
elif value.get("litellm_provider") == "ai21":
if value.get("mode") == "chat":
ai21_chat_models.add(key)
@ -907,7 +899,8 @@ models_by_provider: dict = {
| vertex_anthropic_models
| vertex_vision_models
| vertex_language_models
| vertex_deepseek_models,
| vertex_deepseek_models
| vertex_minimax_models,
"ai21": ai21_models,
"bedrock": bedrock_models | bedrock_converse_models,
"petals": petals_models,
@ -1105,6 +1098,7 @@ from .llms.azure_ai.rerank.transformation import AzureAIRerankConfig
from .llms.infinity.rerank.transformation import InfinityRerankConfig
from .llms.jina_ai.rerank.transformation import JinaAIRerankConfig
from .llms.deepinfra.rerank.transformation import DeepinfraRerankConfig
from .llms.hosted_vllm.rerank.transformation import HostedVLLMRerankConfig
from .llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
from .llms.vertex_ai.rerank.transformation import VertexAIRerankConfig
from .llms.clarifai.chat.transformation import ClarifaiConfig
@ -1345,6 +1339,7 @@ from .exceptions import (
NotFoundError,
RateLimitError,
ServiceUnavailableError,
BadGatewayError,
OpenAIError,
ContextWindowExceededError,
ContentPolicyViolationError,
@ -1400,12 +1395,12 @@ from .types.llms.custom_llm import CustomLLMItem
from .types.utils import GenericStreamingChunk
custom_provider_map: List[CustomLLMItem] = []
_custom_providers: List[str] = (
[]
) # internal helper util, used to track names of custom providers
disable_hf_tokenizer_download: Optional[bool] = (
None # disable huggingface tokenizer download. Defaults to openai clk100
)
_custom_providers: List[
str
] = [] # internal helper util, used to track names of custom providers
disable_hf_tokenizer_download: Optional[
bool
] = None # disable huggingface tokenizer download. Defaults to openai clk100
global_disable_no_log_param: bool = False
### CLI UTILITIES ###

5
litellm/constants.py
View file

@ -1,6 +1,7 @@
import os
from typing import List, Literal
DEFAULT_HEALTH_CHECK_PROMPT = str(os.getenv("DEFAULT_HEALTH_CHECK_PROMPT", "test from litellm"))
AZURE_DEFAULT_RESPONSES_API_VERSION = str(
os.getenv("AZURE_DEFAULT_RESPONSES_API_VERSION", "preview")
)
@ -219,7 +220,7 @@ REDIS_SOCKET_TIMEOUT = float(os.getenv("REDIS_SOCKET_TIMEOUT", 0.1))
REDIS_CONNECTION_POOL_TIMEOUT = int(os.getenv("REDIS_CONNECTION_POOL_TIMEOUT", 5))
# Default Redis major version to assume when version cannot be determined
# Using 7 as it's the modern version that supports LPOP with count parameter
DEFAULT_REDIS_MAJOR_VERSION = 7
DEFAULT_REDIS_MAJOR_VERSION = int(os.getenv("DEFAULT_REDIS_MAJOR_VERSION", 7))
NON_LLM_CONNECTION_TIMEOUT = int(
os.getenv("NON_LLM_CONNECTION_TIMEOUT", 15)
) # timeout for adjacent services (e.g. jwt auth)
@ -279,6 +280,8 @@ ANTHROPIC_WEB_SEARCH_TOOL_MAX_USES = {
DEFAULT_IMAGE_ENDPOINT_MODEL = "dall-e-2"
DEFAULT_VIDEO_ENDPOINT_MODEL = "sora-2"
DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS = int(os.getenv("DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS", 8))
### DATAFORSEO CONSTANTS ###
DEFAULT_DATAFORSEO_LOCATION_CODE = int(
os.getenv("DEFAULT_DATAFORSEO_LOCATION_CODE", 2250)

124
litellm/cost_calculator.py
View file

@ -17,6 +17,9 @@ from litellm.constants import (
from litellm.litellm_core_utils.llm_cost_calc.tool_call_cost_tracking import (
StandardBuiltInToolCostTracking,
)
from litellm.litellm_core_utils.llm_cost_calc.usage_object_transformation import (
TranscriptionUsageObjectTransformation,
)
from litellm.litellm_core_utils.llm_cost_calc.utils import (
CostCalculatorUtils,
_generic_cost_per_character,
@ -81,6 +84,8 @@ from litellm.types.utils import (
LlmProvidersSet,
ModelInfo,
StandardBuiltInToolsParams,
TranscriptionUsageDurationObject,
TranscriptionUsageTokensObject,
Usage,
VectorStoreSearchResponse,
)
@ -319,20 +324,32 @@ def cost_per_token( # noqa: PLR0915
usage=usage_block, model=model, custom_llm_provider=custom_llm_provider
)
elif call_type == "atranscription" or call_type == "transcription":
return openai_cost_per_second(
model=model,
custom_llm_provider=custom_llm_provider,
duration=audio_transcription_file_duration,
)
if model == "gpt-4o-mini-transcribe":
return openai_cost_per_token(
model=model,
usage=usage_block,
service_tier=service_tier,
)
else:
return openai_cost_per_second(
model=model,
custom_llm_provider=custom_llm_provider,
duration=audio_transcription_file_duration,
)
elif call_type == "search" or call_type == "asearch":
# Search providers use per-query pricing
from litellm.search import search_provider_cost_per_query
return search_provider_cost_per_query(
model=model,
custom_llm_provider=custom_llm_provider,
number_of_queries=number_of_queries or 1,
optional_params=response._hidden_params if response and hasattr(response, "_hidden_params") else None
optional_params=(
response._hidden_params
if response and hasattr(response, "_hidden_params")
else None
),
)
elif custom_llm_provider == "vertex_ai":
cost_router = google_cost_router(
@ -509,16 +526,18 @@ def _select_model_name_for_cost_calc(
else:
return_model = model
if base_model is not None:
elif base_model is not None:
return_model = base_model
if completion_response_model is None and hidden_params is not None:
elif completion_response_model is None and hidden_params is not None:
if (
hidden_params.get("model", None) is not None
and len(hidden_params["model"]) > 0
):
return_model = hidden_params.get("model", model)
if hidden_params is not None and hidden_params.get("region_name", None) is not None:
elif (
hidden_params is not None and hidden_params.get("region_name", None) is not None
):
region_name = hidden_params.get("region_name", None)
if return_model is None and completion_response_model is not None:
@ -573,6 +592,19 @@ def _get_usage_object(
return ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(
usage_obj
)
elif TranscriptionUsageObjectTransformation.is_transcription_usage_object(
usage_obj
):
return (
TranscriptionUsageObjectTransformation.transform_transcription_usage_object(
cast(
Union[
TranscriptionUsageDurationObject, TranscriptionUsageTokensObject
],
usage_obj,
)
)
)
elif isinstance(usage_obj, dict):
return Usage(**usage_obj)
elif isinstance(usage_obj, BaseModel):
@ -586,8 +618,12 @@ def _get_usage_object(
def _is_known_usage_objects(usage_obj):
"""Returns True if the usage obj is a known Usage type"""
return isinstance(usage_obj, litellm.Usage) or isinstance(
usage_obj, ResponseAPIUsage
return (
isinstance(usage_obj, litellm.Usage)
or isinstance(usage_obj, ResponseAPIUsage)
or TranscriptionUsageObjectTransformation.is_transcription_usage_object(
usage_obj
)
)
@ -827,6 +863,22 @@ def completion_cost( # noqa: PLR0915
_usage = ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(
_usage
).model_dump()
elif TranscriptionUsageObjectTransformation.is_transcription_usage_object(
_usage
):
tr_usage = TranscriptionUsageObjectTransformation.transform_transcription_usage_object(
cast(
Union[
TranscriptionUsageDurationObject,
TranscriptionUsageTokensObject,
],
_usage,
)
)
if tr_usage is not None:
_usage = tr_usage.model_dump()
else:
_usage = _usage
# get input/output tokens from completion_response
prompt_tokens = _usage.get("prompt_tokens", 0)
@ -853,15 +905,6 @@ def completion_cost( # noqa: PLR0915
"custom_llm_provider", custom_llm_provider or None
)
region_name = hidden_params.get("region_name", region_name)
size = hidden_params.get("optional_params", {}).get(
"size", "1024-x-1024"
) # openai default
quality = hidden_params.get("optional_params", {}).get(
"quality", "standard"
) # openai default
n = hidden_params.get("optional_params", {}).get(
"n", 1
) # openai default
else:
if model is None:
raise ValueError(
@ -888,7 +931,9 @@ def completion_cost( # noqa: PLR0915
str(e)
)
)
if CostCalculatorUtils._call_type_has_image_response(call_type):
if CostCalculatorUtils._call_type_has_image_response(
call_type
) and isinstance(completion_response, ImageResponse):
### IMAGE GENERATION COST CALCULATION ###
return CostCalculatorUtils.route_image_generation_cost_calculator(
model=model,
@ -906,27 +951,32 @@ def completion_cost( # noqa: PLR0915
or call_type == CallTypes.avideo_remix.value
):
### VIDEO GENERATION COST CALCULATION ###
if completion_response is not None and hasattr(completion_response, 'usage'):
usage_obj = completion_response.usage
usage_obj = getattr(completion_response, "usage", None)
if completion_response is not None and usage_obj:
# Handle both dict and Pydantic Usage object
if isinstance(usage_obj, dict):
duration_seconds = usage_obj.get('duration_seconds', None)
duration_seconds = usage_obj.get("duration_seconds", None)
else:
duration_seconds = getattr(usage_obj, 'duration_seconds', None)
duration_seconds = getattr(
usage_obj, "duration_seconds", None
)
if duration_seconds is not None:
# Calculate cost based on video duration using video-specific cost calculation
from litellm.llms.openai.cost_calculation import video_generation_cost
from litellm.llms.openai.cost_calculation import (
video_generation_cost,
)
return video_generation_cost(
model=model,
duration_seconds=duration_seconds,
custom_llm_provider=custom_llm_provider
custom_llm_provider=custom_llm_provider,
)
# Fallback to default video cost calculation if no duration available
return default_video_cost_calculator(
model=model,
duration_seconds=0.0, # Default to 0 if no duration available
custom_llm_provider=custom_llm_provider
custom_llm_provider=custom_llm_provider,
)
elif (
call_type == CallTypes.speech.value
@ -1460,13 +1510,13 @@ def default_video_cost_calculator(
model_name_without_custom_llm_provider = model.replace(
f"{custom_llm_provider}/", ""
)
base_model_name = f"{custom_llm_provider}/{model_name_without_custom_llm_provider}"
base_model_name = (
f"{custom_llm_provider}/{model_name_without_custom_llm_provider}"
)
verbose_logger.debug(
f"Looking up cost for video model: {base_model_name}"
)
verbose_logger.debug(f"Looking up cost for video model: {base_model_name}")
model_without_provider = model.split('/')[-1]
model_without_provider = model.split("/")[-1]
# Try model with provider first, fall back to base model name
cost_info: Optional[dict] = None
@ -1480,7 +1530,7 @@ def default_video_cost_calculator(
if _model is not None and _model in litellm.model_cost:
cost_info = litellm.model_cost[_model]
break
# If still not found, try with custom_llm_provider prefix
if cost_info is None and custom_llm_provider:
prefixed_model = f"{custom_llm_provider}/{model}"
@ -1495,12 +1545,12 @@ def default_video_cost_calculator(
video_cost_per_second = cost_info.get("output_cost_per_video_per_second")
if video_cost_per_second is not None:
return video_cost_per_second * duration_seconds
# Fallback to general output cost per second
output_cost_per_second = cost_info.get("output_cost_per_second")
if output_cost_per_second is not None:
return output_cost_per_second * duration_seconds
# If no cost information found, return 0
verbose_logger.info(
f"No cost information found for video model {model}. Please add pricing to model_prices_and_context_window.json"

76
litellm/exceptions.py
View file

@ -450,6 +450,7 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
llm_provider,
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
provider_specific_fields: Optional[dict] = None,
):
self.status_code = 400
self.message = "litellm.ContentPolicyViolationError: {}".format(message)
@ -458,6 +459,8 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
self.litellm_debug_info = litellm_debug_info
request = httpx.Request(method="POST", url="https://api.openai.com/v1")
self.response = httpx.Response(status_code=400, request=request)
self.provider_specific_fields = provider_specific_fields
super().__init__(
message=self.message,
model=self.model, # type: ignore
@ -465,16 +468,18 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
response=self.response,
litellm_debug_info=self.litellm_debug_info,
) # Call the base class constructor with the parameters it needs
def __str__(self):
_message = self.message
if self.num_retries:
_message += f" LiteLLM Retried: {self.num_retries} times"
if self.max_retries:
_message += f", LiteLLM Max Retries: {self.max_retries}"
return _message
return self._transform_error_to_string()
def __repr__(self):
return self._transform_error_to_string()
def _transform_error_to_string(self) -> str:
"""
Transform the error to a string
"""
_message = self.message
if self.num_retries:
_message += f" LiteLLM Retried: {self.num_retries} times"
@ -501,8 +506,62 @@ class ServiceUnavailableError(openai.APIStatusError): # type: ignore
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
_response_headers = (
getattr(response, "headers", None) if response is not None else None
)
self.response = httpx.Response(
status_code=self.status_code,
headers=_response_headers,
request=httpx.Request(
method="POST",
url=" https://cloud.google.com/vertex-ai/",
),
)
super().__init__(
self.message, response=self.response, body=None
) # Call the base class constructor with the parameters it needs
def __str__(self):
_message = self.message
if self.num_retries:
_message += f" LiteLLM Retried: {self.num_retries} times"
if self.max_retries:
_message += f", LiteLLM Max Retries: {self.max_retries}"
return _message
def __repr__(self):
_message = self.message
if self.num_retries:
_message += f" LiteLLM Retried: {self.num_retries} times"
if self.max_retries:
_message += f", LiteLLM Max Retries: {self.max_retries}"
return _message
class BadGatewayError(openai.APIStatusError): # type: ignore
def __init__(
self,
message,
llm_provider,
model,
response: Optional[httpx.Response] = None,
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
):
self.status_code = 502
self.message = "litellm.BadGatewayError: {}".format(message)
self.llm_provider = llm_provider
self.model = model
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
_response_headers = (
getattr(response, "headers", None) if response is not None else None
)
self.response = httpx.Response(
status_code=self.status_code,
headers=_response_headers,
request=httpx.Request(
method="POST",
url=" https://cloud.google.com/vertex-ai/",
@ -547,8 +606,12 @@ class InternalServerError(openai.InternalServerError): # type: ignore
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
_response_headers = (
getattr(response, "headers", None) if response is not None else None
)
self.response = httpx.Response(
status_code=self.status_code,
headers=_response_headers,
request=httpx.Request(
method="POST",
url=" https://cloud.google.com/vertex-ai/",
@ -754,6 +817,7 @@ LITELLM_EXCEPTION_TYPES = [
ContentPolicyViolationError,
InternalServerError,
ServiceUnavailableError,
BadGatewayError,
APIError,
APIConnectionError,
APIResponseValidationError,

119
litellm/integrations/custom_logger.py
View file

@ -8,7 +8,6 @@ from typing import (
AsyncGenerator,
Dict,
List,
Literal,
Optional,
Tuple,
Union,
@ -24,6 +23,7 @@ from litellm.types.llms.openai import AllMessageValues, ChatCompletionRequest
from litellm.types.utils import (
AdapterCompletionStreamWrapper,
CallTypes,
CallTypesLiteral,
LLMResponseTypes,
ModelResponse,
ModelResponseStream,
@ -65,12 +65,11 @@ _BASE64_INLINE_PATTERN = re.compile(
class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callback#callback-class
# Class variables or attributes
def __init__(
self,
self,
turn_off_message_logging: bool = False,
# deprecated param, use `turn_off_message_logging` instead
message_logging: bool = True,
**kwargs
**kwargs,
) -> None:
"""
Args:
@ -259,7 +258,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
) -> Optional[Any]:
"""
Allow modifying streaming chunks just before they're returned to the user.
This is called for each streaming chunk in the response.
"""
pass
@ -330,18 +329,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: dict,
call_type: Literal[
"completion",
"text_completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"pass_through_endpoint",
"rerank",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
) -> Optional[
Union[Exception, str, dict]
]: # raise exception if invalid, return a str for the user to receive - if rejected, or return a modified dictionary for passing into litellm
@ -380,16 +368,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
call_type: Literal[
"completion",
"embeddings",
"image_generation",
"moderation",
"audio_transcription",
"responses",
"mcp_call",
"anthropic_messages",
],
call_type: CallTypesLiteral,
) -> Any:
pass
@ -473,7 +452,6 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
# MCP TOOL CALL HOOKS
#########################################################
async def async_post_mcp_tool_call_hook(
self, kwargs, response_obj: MCPPostCallResponseObject, start_time, end_time
) -> Optional[MCPPostCallResponseObject]:
@ -557,33 +535,32 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
if LITELLM_METADATA_FIELD in request_kwargs:
return LITELLM_METADATA_FIELD
return OLD_LITELLM_METADATA_FIELD
def redact_standard_logging_payload_from_model_call_details(
self, model_call_details: Dict
) -> Dict:
"""
Only redacts messages and responses when self.turn_off_message_logging is True
By default, self.turn_off_message_logging is False and this does nothing.
Return a redacted deepcopy of the provided logging payload.
This is useful for logging payloads that contain sensitive information.
"""
from copy import copy
from litellm import Choices, Message, ModelResponse
from litellm.types.utils import LiteLLMCommonStrings
turn_off_message_logging: bool = getattr(self, "turn_off_message_logging", False)
if turn_off_message_logging is False:
return model_call_details
# Only make a shallow copy of the top-level dict to avoid deepcopy issues
# with complex objects like AuthenticationError that may be present
model_call_details_copy = copy(model_call_details)
redacted_str = LiteLLMCommonStrings.redacted_by_litellm.value
redacted_str = "redacted-by-litellm"
standard_logging_object = model_call_details.get("standard_logging_object")
if standard_logging_object is None:
return model_call_details_copy
@ -592,20 +569,40 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
standard_logging_object_copy = copy(standard_logging_object)
if standard_logging_object_copy.get("messages") is not None:
standard_logging_object_copy["messages"] = [Message(content=redacted_str).model_dump()]
standard_logging_object_copy["messages"] = [
Message(content=redacted_str).model_dump()
]
if standard_logging_object_copy.get("response") is not None:
model_response = ModelResponse(
choices=[Choices(message=Message(content=redacted_str))]
)
model_response_dict = model_response.model_dump()
standard_logging_object_copy["response"] = model_response_dict
response = standard_logging_object_copy["response"]
# Check if this is a ResponsesAPIResponse (has "output" field)
if isinstance(response, dict) and "output" in response:
# Make a copy to avoid modifying the original
from copy import deepcopy
response_copy = deepcopy(response)
# Redact content in output array
if isinstance(response_copy.get("output"), list):
for output_item in response_copy["output"]:
if isinstance(output_item, dict) and "content" in output_item:
if isinstance(output_item["content"], list):
# Redact text in content items
for content_item in output_item["content"]:
if isinstance(content_item, dict) and "text" in content_item:
content_item["text"] = redacted_str
standard_logging_object_copy["response"] = response_copy
else:
# Standard ModelResponse format
model_response = ModelResponse(
choices=[Choices(message=Message(content=redacted_str))]
)
model_response_dict = model_response.model_dump()
standard_logging_object_copy["response"] = model_response_dict
model_call_details_copy["standard_logging_object"] = standard_logging_object_copy
model_call_details_copy["standard_logging_object"] = (
standard_logging_object_copy
)
return model_call_details_copy
async def get_proxy_server_request_from_cold_storage_with_object_key(
self,
object_key: str,
@ -643,7 +640,9 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
verbose_logger.debug(f"Error in handle_callback_failure for {callback_name}: {str(e)}")
async def _strip_base64_from_messages(
self, payload: "StandardLoggingPayload", max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
self,
payload: "StandardLoggingPayload",
max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER,
) -> "StandardLoggingPayload":
"""
Removes or redacts base64-encoded file data (e.g., PDFs, images, audio)
@ -692,7 +691,9 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
verbose_logger.debug(f"[CustomLogger] Stripping base64 from {len(messages)} messages")
if messages:
payload["messages"] = self._process_messages(messages=messages, max_depth=max_depth)
payload["messages"] = self._process_messages(
messages=messages, max_depth=max_depth
)
total_items = 0
for m in payload.get("messages", []) or []:
@ -705,9 +706,13 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
f"[CustomLogger] Completed base64 strip; retained {total_items} content items"
)
return payload
def _redact_base64(self, value: Any, depth: int = 0, max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER) -> Any:
def _redact_base64(
self,
value: Any,
depth: int = 0,
max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER,
) -> Any:
"""Recursively redact inline base64 from any nested structure with a max recursion depth limit."""
if depth > max_depth:
verbose_logger.warning(
@ -724,10 +729,16 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
return value
if isinstance(value, list):
return [self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth) for v in value]
return [
self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth)
for v in value
]
if isinstance(value, dict):
return {k: self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth) for k, v in value.items()}
return {
k: self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth)
for k, v in value.items()
}
return value
@ -750,10 +761,14 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
cleaned: List[Any] = []
for c in contents:
if self._should_keep_content(content=c):
cleaned.append(self._redact_base64(value=c, max_depth=max_depth))
cleaned.append(
self._redact_base64(value=c, max_depth=max_depth)
)
msg["content"] = cleaned
else:
msg["content"] = self._redact_base64(value=contents, max_depth=max_depth)
msg["content"] = self._redact_base64(
value=contents, max_depth=max_depth
)
for key, val in list(msg.items()):
if key != "content":

109
litellm/integrations/datadog/datadog.py
View file

@ -17,7 +17,6 @@ import asyncio
import datetime
import os
import traceback
from litellm._uuid import uuid
from datetime import datetime as datetimeObj
from typing import Any, Dict, List, Optional, Union
@ -26,6 +25,7 @@ from httpx import Response
import litellm
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.llms.custom_httpx.http_handler import (
_get_httpx_client,
@ -60,17 +60,19 @@ class DataDogLogger(
"""
Initializes the datadog logger, checks if the correct env variables are set
Required environment variables:
Required environment variables (Direct API):
`DD_API_KEY` - your datadog api key
`DD_SITE` - your datadog site, example = `"us5.datadoghq.com"`
Optional environment variables (DataDog Agent):
`DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
Note: If DD_AGENT_HOST is set, logs will be sent to the agent instead of directly to DataDog API.
In this case, DD_API_KEY and DD_SITE are not required (agent handles authentication).
"""
try:
verbose_logger.debug("Datadog: in init datadog logger")
# check if the correct env variables are set
if os.getenv("DD_API_KEY", None) is None:
raise Exception("DD_API_KEY is not set, set 'DD_API_KEY=<>")
if os.getenv("DD_SITE", None) is None:
raise Exception("DD_SITE is not set in .env, set 'DD_SITE=<>")
#########################################################
# Handle datadog_params set as litellm.datadog_params
@ -81,21 +83,16 @@ class DataDogLogger(
self.async_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)
self.DD_API_KEY = os.getenv("DD_API_KEY")
self.intake_url = (
f"https://http-intake.logs.{os.getenv('DD_SITE')}/api/v2/logs"
)
###################################
# OPTIONAL -only used for testing
dd_base_url: Optional[str] = (
os.getenv("_DATADOG_BASE_URL")
or os.getenv("DATADOG_BASE_URL")
or os.getenv("DD_BASE_URL")
)
if dd_base_url is not None:
self.intake_url = f"{dd_base_url}/api/v2/logs"
###################################
# Configure DataDog endpoint (Agent or Direct API)
dd_agent_host = os.getenv("DD_AGENT_HOST")
if dd_agent_host:
self._configure_dd_agent(dd_agent_host=dd_agent_host)
else:
self._configure_dd_direct_api()
# Optional override for testing
self._apply_dd_base_url_override()
self.sync_client = _get_httpx_client()
asyncio.create_task(self.periodic_flush())
self.flush_lock = asyncio.Lock()
@ -123,6 +120,47 @@ class DataDogLogger(
dict_datadog_params = DatadogInitParams(**litellm.datadog_params).model_dump()
return dict_datadog_params
def _configure_dd_agent(self, dd_agent_host: str) -> None:
"""
Configure DataDog Agent for log forwarding
Args:
dd_agent_host: Hostname or IP of DataDog agent
"""
dd_agent_port = os.getenv("DD_AGENT_PORT", "10518") # default port for logs
self.intake_url = f"http://{dd_agent_host}:{dd_agent_port}/api/v2/logs"
self.DD_API_KEY = os.getenv("DD_API_KEY") # Optional when using agent
verbose_logger.debug(f"Datadog: Using DD Agent at {self.intake_url}")
def _configure_dd_direct_api(self) -> None:
"""
Configure direct DataDog API connection
Raises:
Exception: If required environment variables are not set
"""
if os.getenv("DD_API_KEY", None) is None:
raise Exception("DD_API_KEY is not set, set 'DD_API_KEY=<>")
if os.getenv("DD_SITE", None) is None:
raise Exception("DD_SITE is not set in .env, set 'DD_SITE=<>")
self.DD_API_KEY = os.getenv("DD_API_KEY")
self.intake_url = (
f"https://http-intake.logs.{os.getenv('DD_SITE')}/api/v2/logs"
)
def _apply_dd_base_url_override(self) -> None:
"""
Apply base URL override for testing purposes
"""
dd_base_url: Optional[str] = (
os.getenv("_DATADOG_BASE_URL")
or os.getenv("DATADOG_BASE_URL")
or os.getenv("DD_BASE_URL")
)
if dd_base_url is not None:
self.intake_url = f"{dd_base_url}/api/v2/logs"
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
"""
Async Log success events to Datadog
@ -226,12 +264,16 @@ class DataDogLogger(
end_time=end_time,
)
# Build headers
headers = {}
# Add API key if available (required for direct API, optional for agent)
if self.DD_API_KEY:
headers["DD-API-KEY"] = self.DD_API_KEY
response = self.sync_client.post(
url=self.intake_url,
json=dd_payload, # type: ignore
headers={
"DD-API-KEY": self.DD_API_KEY,
},
headers=headers,
)
response.raise_for_status()
@ -342,14 +384,21 @@ class DataDogLogger(
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
compressed_data = gzip.compress(safe_dumps(data).encode("utf-8"))
# Build headers
headers = {
"Content-Encoding": "gzip",
"Content-Type": "application/json",
}
# Add API key if available (required for direct API, optional for agent)
if self.DD_API_KEY:
headers["DD-API-KEY"] = self.DD_API_KEY
response = await self.async_client.post(
url=self.intake_url,
data=compressed_data, # type: ignore
headers={
"DD-API-KEY": self.DD_API_KEY,
"Content-Encoding": "gzip",
"Content-Type": "application/json",
},
headers=headers,
)
return response

11
litellm/integrations/opentelemetry.py
View file

@ -10,6 +10,7 @@ from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.types.services import ServiceLoggerPayload
from litellm.types.utils import (
ChatCompletionMessageToolCall,
CostBreakdown,
Function,
StandardCallbackDynamicParams,
StandardLoggingPayload,
@ -1076,6 +1077,16 @@ class OpenTelemetry(CustomLogger):
self.safe_set_attribute(
span=span, key="hidden_params", value=safe_dumps(hidden_params)
)
# Cost breakdown tracking
cost_breakdown: Optional[CostBreakdown] = standard_logging_payload.get("cost_breakdown")
if cost_breakdown:
for key, value in cost_breakdown.items():
if value is not None:
self.safe_set_attribute(
span=span,
key=f"gen_ai.cost.{key}",
value=value,
)
#############################################
########## LLM Request Attributes ###########
#############################################

4
litellm/integrations/opik/utils.py
View file

@ -1,7 +1,7 @@
import configparser
import os
import time
from typing import Any, Dict, Final, List, Optional
from typing import Any, Dict, Final, List, Optional, Tuple
CONFIG_FILE_PATH_DEFAULT: Final[str] = "~/.opik.config"
@ -106,7 +106,7 @@ def _remove_nulls(x: Dict[str, Any]) -> Dict[str, Any]:
def get_traces_and_spans_from_payload(
payload: List[Dict[str, Any]]
) -> tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
"""
Separate traces and spans from payload.

119
litellm/litellm_core_utils/audio_utils/utils.py
View file

@ -4,6 +4,7 @@ Utils used for litellm.transcription() and litellm.atranscription()
import os
from dataclasses import dataclass
from typing import Optional
from litellm.types.files import get_file_mime_type_from_extension
from litellm.types.utils import FileTypes
@ -13,12 +14,13 @@ from litellm.types.utils import FileTypes
class ProcessedAudioFile:
"""
Processed audio file data.
Attributes:
file_content: The binary content of the audio file
filename: The filename (extracted or generated)
content_type: The MIME type of the audio file
"""
file_content: bytes
filename: str
content_type: str
@ -27,61 +29,63 @@ class ProcessedAudioFile:
def process_audio_file(audio_file: FileTypes) -> ProcessedAudioFile:
"""
Common utility function to process audio files for audio transcription APIs.
Handles various input types:
- File paths (str, os.PathLike)
- Raw bytes/bytearray
- Tuples (filename, content, optional content_type)
- File-like objects with read() method
Args:
audio_file: The audio file input in various formats
Returns:
ProcessedAudioFile: Structured data with file content, filename, and content type
Raises:
ValueError: If audio_file type is unsupported or content cannot be extracted
"""
file_content = None
filename = None
if isinstance(audio_file, (bytes, bytearray)):
# Raw bytes
filename = 'audio.wav'
filename = "audio.wav"
file_content = bytes(audio_file)
elif isinstance(audio_file, (str, os.PathLike)):
# File path or PathLike
file_path = str(audio_file)
with open(file_path, 'rb') as f:
with open(file_path, "rb") as f:
file_content = f.read()
filename = file_path.split('/')[-1]
filename = file_path.split("/")[-1]
elif isinstance(audio_file, tuple):
# Tuple format: (filename, content, content_type) or (filename, content)
if len(audio_file) >= 2:
filename = audio_file[0] or 'audio.wav'
filename = audio_file[0] or "audio.wav"
content = audio_file[1]
if isinstance(content, (bytes, bytearray)):
file_content = bytes(content)
elif isinstance(content, (str, os.PathLike)):
# File path or PathLike
with open(str(content), 'rb') as f:
with open(str(content), "rb") as f:
file_content = f.read()
elif hasattr(content, 'read'):
elif hasattr(content, "read"):
# File-like object
file_content = content.read()
if hasattr(content, 'seek'):
if hasattr(content, "seek"):
content.seek(0)
else:
raise ValueError(f"Unsupported content type in tuple: {type(content)}")
else:
raise ValueError("Tuple must have at least 2 elements: (filename, content)")
elif hasattr(audio_file, 'read') and not isinstance(audio_file, (str, bytes, bytearray, tuple, os.PathLike)):
elif hasattr(audio_file, "read") and not isinstance(
audio_file, (str, bytes, bytearray, tuple, os.PathLike)
):
# File-like object (IO) - check this after all other types
filename = getattr(audio_file, 'name', 'audio.wav')
filename = getattr(audio_file, "name", "audio.wav")
file_content = audio_file.read() # type: ignore
# Reset file pointer if possible
if hasattr(audio_file, 'seek'):
if hasattr(audio_file, "seek"):
audio_file.seek(0) # type: ignore
else:
raise ValueError(f"Unsupported audio_file type: {type(audio_file)}")
@ -90,20 +94,18 @@ def process_audio_file(audio_file: FileTypes) -> ProcessedAudioFile:
raise ValueError("Could not extract file content from audio_file")
# Determine content type using LiteLLM's file type utilities
content_type = 'audio/wav' # Default fallback
content_type = "audio/wav" # Default fallback
if filename:
try:
# Extract extension from filename
extension = filename.split('.')[-1].lower() if '.' in filename else 'wav'
extension = filename.split(".")[-1].lower() if "." in filename else "wav"
content_type = get_file_mime_type_from_extension(extension)
except ValueError:
# If extension is not recognized, fallback to audio/wav
content_type = 'audio/wav'
content_type = "audio/wav"
return ProcessedAudioFile(
file_content=file_content,
filename=filename,
content_type=content_type
file_content=file_content, filename=filename, content_type=content_type
)
@ -134,3 +136,74 @@ def get_audio_file_for_health_check() -> FileTypes:
pwd = os.path.dirname(os.path.realpath(__file__))
file_path = os.path.join(pwd, "audio_health_check.wav")
return open(file_path, "rb")
def calculate_request_duration(file: FileTypes) -> Optional[float]:
"""
Calculate audio duration from file content.
Args:
file: The audio file (can be file path, bytes, or file-like object)
Returns:
Duration in seconds, or None if extraction fails or soundfile is not available
"""
try:
import soundfile as sf
except ImportError:
# soundfile not available, cannot extract duration
return None
try:
import io
# Handle different file input types
file_content: Optional[bytes] = None
if isinstance(file, (bytes, bytearray)):
# Raw bytes
file_content = bytes(file)
elif isinstance(file, (str, os.PathLike)):
# File path
with open(str(file), "rb") as f:
file_content = f.read()
elif isinstance(file, tuple):
# Tuple format: (filename, content, optional content_type)
if len(file) >= 2:
content = file[1]
if isinstance(content, bytes):
file_content = content
elif hasattr(content, "read") and not isinstance(
content, (str, os.PathLike)
):
# File-like object in tuple
current_pos = getattr(content, "tell", lambda: None)()
# Seek to start to ensure we read the entire content
if hasattr(content, "seek"):
content.seek(0)
file_content = content.read()
if current_pos is not None and hasattr(content, "seek"):
content.seek(current_pos)
elif hasattr(file, "read") and not isinstance(file, tuple):
# File-like object (including BytesIO)
current_position = file.tell() if hasattr(file, "tell") else None
# Seek to start to ensure we read the entire content
if hasattr(file, "seek"):
file.seek(0)
file_content = file.read()
# Reset file position if possible
if current_position is not None and hasattr(file, "seek"):
file.seek(current_position)
if file_content is None or not isinstance(file_content, bytes):
return None
# Extract duration using soundfile
file_object = io.BytesIO(file_content)
with sf.SoundFile(file_object) as audio:
duration = len(audio) / audio.samplerate
return duration
except Exception:
# Silently fail if duration extraction fails
return None

89
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -12,6 +12,7 @@ from ..exceptions import (
APIConnectionError,
APIError,
AuthenticationError,
BadGatewayError,
BadRequestError,
ContentPolicyViolationError,
ContextWindowExceededError,
@ -43,16 +44,16 @@ class ExceptionCheckers:
"""
if not isinstance(error_str, str):
return False
if "429" in error_str or "rate limit" in error_str.lower():
return True
#######################################
# Mistral API returns this error string
#########################################
if "service tier capacity exceeded" in error_str.lower():
return True
return False
@staticmethod
@ -73,6 +74,24 @@ class ExceptionCheckers:
if substring in _error_str_lowercase:
return True
return False
@staticmethod
def is_azure_content_policy_violation_error(error_str: str) -> bool:
"""
Check if an error string indicates a content policy violation error.
"""
known_exception_substrings = [
"invalid_request_error",
"content_policy_violation",
"the response was filtered due to the prompt triggering azure openai's content management",
"your task failed as a result of our safety system",
"the model produced invalid content",
"content_filter_policy",
]
for substring in known_exception_substrings:
if substring in error_str.lower():
return True
return False
def get_error_message(error_obj) -> Optional[str]:
@ -507,6 +526,15 @@ def exception_type( # type: ignore # noqa: PLR0915
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
)
elif original_exception.status_code == 502:
exception_mapping_worked = True
raise BadGatewayError(
message=f"BadGatewayError: {exception_provider} - {message}",
model=model,
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
raise ServiceUnavailableError(
@ -637,6 +665,15 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"AnthropicException - {error_str}. Handle with `litellm.InternalServerError`.",
llm_provider="anthropic",
model=model,
response=getattr(original_exception, "response", None),
)
elif original_exception.status_code == 502:
exception_mapping_worked = True
raise BadGatewayError(
message=f"AnthropicException BadGatewayError - {error_str}",
llm_provider="anthropic",
model=model,
response=getattr(original_exception, "response", None),
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
@ -644,6 +681,15 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"AnthropicException - {error_str}. Handle with `litellm.ServiceUnavailableError`.",
llm_provider="anthropic",
model=model,
response=getattr(original_exception, "response", None),
)
elif original_exception.status_code == 504: # gateway timeout error
exception_mapping_worked = True
raise Timeout(
message=f"AnthropicException Timeout - {error_str}",
model=model,
llm_provider="anthropic",
exception_status_code=original_exception.status_code,
)
elif custom_llm_provider == "replicate":
if "Incorrect authentication token" in error_str:
@ -1260,6 +1306,7 @@ def exception_type( # type: ignore # noqa: PLR0915
elif (
"429 Quota exceeded" in error_str
or "Quota exceeded for" in error_str
or "Resource exhausted" in error_str
or "IndexError: list index out of range" in error_str
or "429 Unable to submit request because the service is temporarily out of capacity."
in error_str
@ -1992,26 +2039,19 @@ def exception_type( # type: ignore # noqa: PLR0915
response=getattr(original_exception, "response", None),
)
elif (
(
"invalid_request_error" in error_str
and "content_policy_violation" in error_str
)
or (
"The response was filtered due to the prompt triggering Azure OpenAI's content management"
in error_str
)
or "Your task failed as a result of our safety system" in error_str
or "The model produced invalid content" in error_str
or "content_filter_policy" in error_str
ExceptionCheckers.is_azure_content_policy_violation_error(error_str)
):
exception_mapping_worked = True
raise ContentPolicyViolationError(
message=f"litellm.ContentPolicyViolationError: AzureException - {message}",
llm_provider="azure",
model=model,
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
from litellm.llms.azure.exception_mapping import (
AzureOpenAIExceptionMapping,
)
raise AzureOpenAIExceptionMapping.create_content_policy_violation_error(
message=message,
model=model,
extra_information=extra_information,
original_exception=original_exception,
)
elif "invalid_request_error" in error_str:
exception_mapping_worked = True
raise BadRequestError(
@ -2089,6 +2129,15 @@ def exception_type( # type: ignore # noqa: PLR0915
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
)
elif original_exception.status_code == 502:
exception_mapping_worked = True
raise BadGatewayError(
message=f"AzureException BadGatewayError - {message}",
model=model,
llm_provider="azure",
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
raise ServiceUnavailableError(

1
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -120,5 +120,6 @@ def get_litellm_params(
"vertex_project": kwargs.get("vertex_project"),
"use_litellm_proxy": use_litellm_proxy,
"litellm_request_debug": litellm_request_debug,
"aws_region_name": kwargs.get("aws_region_name"),
}
return litellm_params

5
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -97,6 +97,7 @@ class HealthCheckHelpers:
"audio_speech",
"audio_transcription",
"image_generation",
"video_generation",
"rerank",
"realtime",
"batch",
@ -159,6 +160,10 @@ class HealthCheckHelpers:
**_filter_model_params(model_params=model_params),
prompt=prompt,
),
"video_generation": lambda: litellm.avideo_generation(
**_filter_model_params(model_params=model_params),
prompt=prompt or "test video generation",
),
"rerank": lambda: litellm.arerank(
**_filter_model_params(model_params=model_params),
query=prompt or "",

426
litellm/litellm_core_utils/litellm_logging.py
View file

@ -308,9 +308,9 @@ class Logging(LiteLLMLoggingBaseClass):
self.litellm_trace_id: str = litellm_trace_id or str(uuid.uuid4())
self.function_id = function_id
self.streaming_chunks: List[Any] = [] # for generating complete stream response
self.sync_streaming_chunks: List[
Any
] = [] # for generating complete stream response
self.sync_streaming_chunks: List[Any] = (
[]
) # for generating complete stream response
self.log_raw_request_response = log_raw_request_response
# Initialize dynamic callbacks
@ -686,9 +686,9 @@ class Logging(LiteLLMLoggingBaseClass):
if anthropic_cache_control_logger := AnthropicCacheControlHook.get_custom_logger_for_anthropic_cache_control_hook(
non_default_params
):
self.model_call_details[
"prompt_integration"
] = anthropic_cache_control_logger.__class__.__name__
self.model_call_details["prompt_integration"] = (
anthropic_cache_control_logger.__class__.__name__
)
return anthropic_cache_control_logger
#########################################################
@ -700,9 +700,9 @@ class Logging(LiteLLMLoggingBaseClass):
internal_usage_cache=None,
llm_router=None,
)
self.model_call_details[
"prompt_integration"
] = vector_store_custom_logger.__class__.__name__
self.model_call_details["prompt_integration"] = (
vector_store_custom_logger.__class__.__name__
)
# Add to global callbacks so post-call hooks are invoked
if (
vector_store_custom_logger
@ -762,9 +762,9 @@ class Logging(LiteLLMLoggingBaseClass):
model
): # if model name was changes pre-call, overwrite the initial model call name with the new one
self.model_call_details["model"] = model
self.model_call_details["litellm_params"][
"api_base"
] = self._get_masked_api_base(additional_args.get("api_base", ""))
self.model_call_details["litellm_params"]["api_base"] = (
self._get_masked_api_base(additional_args.get("api_base", ""))
)
def pre_call(self, input, api_key, model=None, additional_args={}): # noqa: PLR0915
# Log the exact input to the LLM API
@ -793,10 +793,10 @@ class Logging(LiteLLMLoggingBaseClass):
try:
# [Non-blocking Extra Debug Information in metadata]
if turn_off_message_logging is True:
_metadata[
"raw_request"
] = "redacted by litellm. \
_metadata["raw_request"] = (
"redacted by litellm. \
'litellm.turn_off_message_logging=True'"
)
else:
curl_command = self._get_request_curl_command(
api_base=additional_args.get("api_base", ""),
@ -807,32 +807,32 @@ class Logging(LiteLLMLoggingBaseClass):
_metadata["raw_request"] = str(curl_command)
# split up, so it's easier to parse in the UI
self.model_call_details[
"raw_request_typed_dict"
] = RawRequestTypedDict(
raw_request_api_base=str(
additional_args.get("api_base") or ""
),
raw_request_body=self._get_raw_request_body(
additional_args.get("complete_input_dict", {})
),
raw_request_headers=self._get_masked_headers(
additional_args.get("headers", {}) or {},
ignore_sensitive_headers=True,
),
error=None,
self.model_call_details["raw_request_typed_dict"] = (
RawRequestTypedDict(
raw_request_api_base=str(
additional_args.get("api_base") or ""
),
raw_request_body=self._get_raw_request_body(
additional_args.get("complete_input_dict", {})
),
raw_request_headers=self._get_masked_headers(
additional_args.get("headers", {}) or {},
ignore_sensitive_headers=True,
),
error=None,
)
)
except Exception as e:
self.model_call_details[
"raw_request_typed_dict"
] = RawRequestTypedDict(
error=str(e),
self.model_call_details["raw_request_typed_dict"] = (
RawRequestTypedDict(
error=str(e),
)
)
_metadata[
"raw_request"
] = "Unable to Log \
_metadata["raw_request"] = (
"Unable to Log \
raw request: {}".format(
str(e)
str(e)
)
)
if getattr(self, "logger_fn", None) and callable(self.logger_fn):
try:
@ -1133,13 +1133,13 @@ class Logging(LiteLLMLoggingBaseClass):
for callback in callbacks:
try:
if isinstance(callback, CustomLogger):
response: Optional[
MCPPostCallResponseObject
] = await callback.async_post_mcp_tool_call_hook(
kwargs=kwargs,
response_obj=post_mcp_tool_call_response_obj,
start_time=start_time,
end_time=end_time,
response: Optional[MCPPostCallResponseObject] = (
await callback.async_post_mcp_tool_call_hook(
kwargs=kwargs,
response_obj=post_mcp_tool_call_response_obj,
start_time=start_time,
end_time=end_time,
)
)
######################################################################
# if any of the callbacks modify the response, use the modified response
@ -1243,6 +1243,7 @@ class Logging(LiteLLMLoggingBaseClass):
used for consistent cost calculation across response headers + logging integrations.
"""
if isinstance(result, BaseModel) and hasattr(result, "_hidden_params"):
hidden_params = getattr(result, "_hidden_params", {})
if (
@ -1302,9 +1303,9 @@ class Logging(LiteLLMLoggingBaseClass):
verbose_logger.debug(
f"response_cost_failure_debug_information: {debug_info}"
)
self.model_call_details[
"response_cost_failure_debug_information"
] = debug_info
self.model_call_details["response_cost_failure_debug_information"] = (
debug_info
)
return None
try:
@ -1330,9 +1331,9 @@ class Logging(LiteLLMLoggingBaseClass):
verbose_logger.debug(
f"response_cost_failure_debug_information: {debug_info}"
)
self.model_call_details[
"response_cost_failure_debug_information"
] = debug_info
self.model_call_details["response_cost_failure_debug_information"] = (
debug_info
)
return None
@ -1461,6 +1462,51 @@ class Logging(LiteLLMLoggingBaseClass):
)
return logging_result
def _process_hidden_params_and_response_cost(
self,
logging_result,
start_time,
end_time,
):
hidden_params = getattr(logging_result, "_hidden_params", {})
if hidden_params:
if self.model_call_details.get("litellm_params") is not None:
self.model_call_details["litellm_params"].setdefault("metadata", {})
if self.model_call_details["litellm_params"]["metadata"] is None:
self.model_call_details["litellm_params"]["metadata"] = {}
self.model_call_details["litellm_params"]["metadata"]["hidden_params"] = getattr(logging_result, "_hidden_params", {}) # type: ignore
if "response_cost" in hidden_params:
self.model_call_details["response_cost"] = hidden_params["response_cost"]
else:
self.model_call_details["response_cost"] = self._response_cost_calculator(result=logging_result)
self.model_call_details["standard_logging_object"] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=logging_result,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
def _transform_usage_objects(self, result):
if isinstance(result, ResponsesAPIResponse):
result = result.model_copy()
transformed_usage = ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(result.usage)
setattr(result, "usage", transformed_usage.model_dump() if hasattr(transformed_usage, "model_dump") else dict(transformed_usage))
if (standard_logging_payload := self.model_call_details.get("standard_logging_object")) is not None:
standard_logging_payload["response"] = result.model_dump() if hasattr(result, "model_dump") else dict(result)
elif isinstance(result, TranscriptionResponse):
from litellm.litellm_core_utils.llm_cost_calc.usage_object_transformation import (
TranscriptionUsageObjectTransformation,
)
result = result.model_copy()
transformed_usage = TranscriptionUsageObjectTransformation.transform_transcription_usage_object(result.usage) # type: ignore
setattr(result, "usage", transformed_usage)
return result
def _success_handler_helper_fn(
self,
result=None,
@ -1476,82 +1522,24 @@ class Logging(LiteLLMLoggingBaseClass):
end_time = datetime.datetime.now()
if self.completion_start_time is None:
self.completion_start_time = end_time
self.model_call_details[
"completion_start_time"
] = self.completion_start_time
self.model_call_details["completion_start_time"] = self.completion_start_time
self.model_call_details["log_event_type"] = "successful_api_call"
self.model_call_details["end_time"] = end_time
self.model_call_details["cache_hit"] = cache_hit
if self.call_type == CallTypes.anthropic_messages.value:
result = self._handle_anthropic_messages_response_logging(result=result)
elif (
self.call_type == CallTypes.generate_content.value
or self.call_type == CallTypes.agenerate_content.value
):
result = self._handle_non_streaming_google_genai_generate_content_response_logging(
result=result
)
## if model in model cost map - log the response cost
## else set cost to None
elif self.call_type == CallTypes.generate_content.value or self.call_type == CallTypes.agenerate_content.value:
result = self._handle_non_streaming_google_genai_generate_content_response_logging(result=result)
logging_result = self.normalize_logging_result(result=result)
if (
standard_logging_object is None
and result is not None
and self.stream is not True
):
if self._is_recognized_call_type_for_logging(
logging_result=logging_result
):
## HIDDEN PARAMS ##
hidden_params = getattr(logging_result, "_hidden_params", {})
if hidden_params:
# add to metadata for logging
if self.model_call_details.get("litellm_params") is not None:
self.model_call_details["litellm_params"].setdefault(
"metadata", {}
)
if (
self.model_call_details["litellm_params"]["metadata"]
is None
):
self.model_call_details["litellm_params"][
"metadata"
] = {}
self.model_call_details["litellm_params"]["metadata"][ # type: ignore
"hidden_params"
] = getattr(
logging_result, "_hidden_params", {}
)
## RESPONSE COST - Only calculate if not in hidden_params ##
if "response_cost" in hidden_params:
self.model_call_details["response_cost"] = hidden_params[
"response_cost"
]
else:
self.model_call_details[
"response_cost"
] = self._response_cost_calculator(result=logging_result)
## STANDARDIZED LOGGING PAYLOAD
self.model_call_details[
"standard_logging_object"
] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=logging_result,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
if standard_logging_object is None and result is not None and self.stream is not True:
if self._is_recognized_call_type_for_logging(logging_result=logging_result):
self._process_hidden_params_and_response_cost(logging_result=logging_result, start_time=start_time, end_time=end_time)
elif isinstance(result, dict) or isinstance(result, list):
## STANDARDIZED LOGGING PAYLOAD
self.model_call_details[
"standard_logging_object"
] = get_standard_logging_object_payload(
self.model_call_details["standard_logging_object"] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=result,
start_time=start_time,
@ -1561,31 +1549,13 @@ class Logging(LiteLLMLoggingBaseClass):
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
elif standard_logging_object is not None:
self.model_call_details[
"standard_logging_object"
] = standard_logging_object
else: # streaming chunks + image gen.
self.model_call_details["standard_logging_object"] = standard_logging_object
else:
self.model_call_details["response_cost"] = None
## RESPONSES API USAGE OBJECT TRANSFORMATION ##
# MAP RESPONSES API USAGE OBJECT TO LITELLM USAGE OBJECT
if isinstance(result, ResponsesAPIResponse):
result = result.model_copy()
setattr(
result,
"usage",
ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(
result.usage
),
)
if (
litellm.max_budget
and self.stream is False
and result is not None
and isinstance(result, dict)
and "content" in result
):
result = self._transform_usage_objects(result=result)
if litellm.max_budget and self.stream is False and result is not None and isinstance(result, dict) and "content" in result:
time_diff = (end_time - start_time).total_seconds()
float_diff = float(time_diff)
litellm._current_cost += litellm.completion_cost(
@ -1625,7 +1595,7 @@ class Logging(LiteLLMLoggingBaseClass):
or isinstance(logging_result, OCRResponse) # OCR
or isinstance(logging_result, dict)
and logging_result.get("object") == "vector_store.search_results.page"
or isinstance(logging_result, VideoObject)
or isinstance(logging_result, VideoObject)
or isinstance(logging_result, ContainerObject)
or (self.call_type == CallTypes.call_mcp_tool.value)
):
@ -1719,23 +1689,23 @@ class Logging(LiteLLMLoggingBaseClass):
verbose_logger.debug(
"Logging Details LiteLLM-Success Call streaming complete"
)
self.model_call_details[
"complete_streaming_response"
] = complete_streaming_response
self.model_call_details[
"response_cost"
] = self._response_cost_calculator(result=complete_streaming_response)
self.model_call_details["complete_streaming_response"] = (
complete_streaming_response
)
self.model_call_details["response_cost"] = (
self._response_cost_calculator(result=complete_streaming_response)
)
## STANDARDIZED LOGGING PAYLOAD
self.model_call_details[
"standard_logging_object"
] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=complete_streaming_response,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
self.model_call_details["standard_logging_object"] = (
get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=complete_streaming_response,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
)
callbacks = self.get_combined_callback_list(
dynamic_success_callbacks=self.dynamic_success_callbacks,
@ -2063,10 +2033,10 @@ class Logging(LiteLLMLoggingBaseClass):
)
else:
if self.stream and complete_streaming_response:
self.model_call_details[
"complete_response"
] = self.model_call_details.get(
"complete_streaming_response", {}
self.model_call_details["complete_response"] = (
self.model_call_details.get(
"complete_streaming_response", {}
)
)
result = self.model_call_details["complete_response"]
openMeterLogger.log_success_event(
@ -2105,10 +2075,10 @@ class Logging(LiteLLMLoggingBaseClass):
)
else:
if self.stream and complete_streaming_response:
self.model_call_details[
"complete_response"
] = self.model_call_details.get(
"complete_streaming_response", {}
self.model_call_details["complete_response"] = (
self.model_call_details.get(
"complete_streaming_response", {}
)
)
result = self.model_call_details["complete_response"]
@ -2251,9 +2221,9 @@ class Logging(LiteLLMLoggingBaseClass):
if complete_streaming_response is not None:
print_verbose("Async success callbacks: Got a complete streaming response")
self.model_call_details[
"async_complete_streaming_response"
] = complete_streaming_response
self.model_call_details["async_complete_streaming_response"] = (
complete_streaming_response
)
try:
if self.model_call_details.get("cache_hit", False) is True:
@ -2264,10 +2234,10 @@ class Logging(LiteLLMLoggingBaseClass):
model_call_details=self.model_call_details
)
# base_model defaults to None if not set on model_info
self.model_call_details[
"response_cost"
] = self._response_cost_calculator(
result=complete_streaming_response
self.model_call_details["response_cost"] = (
self._response_cost_calculator(
result=complete_streaming_response
)
)
verbose_logger.debug(
@ -2280,16 +2250,16 @@ class Logging(LiteLLMLoggingBaseClass):
self.model_call_details["response_cost"] = None
## STANDARDIZED LOGGING PAYLOAD
self.model_call_details[
"standard_logging_object"
] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=complete_streaming_response,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
self.model_call_details["standard_logging_object"] = (
get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj=complete_streaming_response,
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="success",
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
)
callbacks = self.get_combined_callback_list(
dynamic_success_callbacks=self.dynamic_async_success_callbacks,
@ -2478,26 +2448,24 @@ class Logging(LiteLLMLoggingBaseClass):
def _handle_callback_failure(self, callback: Any):
"""
Handle callback logging failures by incrementing Prometheus metrics.
Works for both sync and async contexts since Prometheus counter increment is synchronous.
Args:
callback: The callback that failed
"""
try:
callback_name = self._get_callback_name(callback)
all_callbacks = litellm.logging_callback_manager._get_all_callbacks()
for callback_obj in all_callbacks:
if hasattr(callback_obj, 'increment_callback_logging_failure'):
if hasattr(callback_obj, "increment_callback_logging_failure"):
callback_obj.increment_callback_logging_failure(callback_name=callback_name) # type: ignore
break # Only increment once
except Exception as e:
verbose_logger.debug(
f"Error in _handle_callback_failure: {str(e)}"
)
verbose_logger.debug(f"Error in _handle_callback_failure: {str(e)}")
def _failure_handler_helper_fn(
self, exception, traceback_exception, start_time=None, end_time=None
@ -2527,18 +2495,18 @@ class Logging(LiteLLMLoggingBaseClass):
## STANDARDIZED LOGGING PAYLOAD
self.model_call_details[
"standard_logging_object"
] = get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj={},
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="failure",
error_str=str(exception),
original_exception=exception,
standard_built_in_tools_params=self.standard_built_in_tools_params,
self.model_call_details["standard_logging_object"] = (
get_standard_logging_object_payload(
kwargs=self.model_call_details,
init_response_obj={},
start_time=start_time,
end_time=end_time,
logging_obj=self,
status="failure",
error_str=str(exception),
original_exception=exception,
standard_built_in_tools_params=self.standard_built_in_tools_params,
)
)
return start_time, end_time
@ -2975,12 +2943,12 @@ class Logging(LiteLLMLoggingBaseClass):
"""
if isinstance(cb, str):
return cb
if hasattr(cb, "__class__"):
return cb.__class__.__name__
if hasattr(cb, "__name__"):
return cb.__name__
if hasattr(cb, "__func__"):
return cb.__func__.__name__
if hasattr(cb, "__class__"):
return cb.__class__.__name__
return str(cb)
def _is_internal_litellm_proxy_callback(self, cb) -> bool:
@ -3036,13 +3004,19 @@ class Logging(LiteLLMLoggingBaseClass):
elif isinstance(result, ResponseCompletedEvent):
## return unified Usage object
if isinstance(result.response.usage, ResponseAPIUsage):
transformed_usage = (
ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(
result.response.usage
)
)
# Set as dict instead of Usage object so model_dump() serializes it correctly
setattr(
result.response,
"usage",
(
ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(
result.response.usage
)
transformed_usage.model_dump()
if hasattr(transformed_usage, "model_dump")
else dict(transformed_usage)
),
)
return result.response
@ -3443,9 +3417,9 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
endpoint=arize_config.endpoint,
)
os.environ[
"OTEL_EXPORTER_OTLP_TRACES_HEADERS"
] = f"space_id={arize_config.space_key},api_key={arize_config.api_key}"
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = (
f"space_id={arize_config.space_key},api_key={arize_config.api_key}"
)
for callback in _in_memory_loggers:
if (
isinstance(callback, ArizeLogger)
@ -3469,9 +3443,9 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
# auth can be disabled on local deployments of arize phoenix
if arize_phoenix_config.otlp_auth_headers is not None:
os.environ[
"OTEL_EXPORTER_OTLP_TRACES_HEADERS"
] = arize_phoenix_config.otlp_auth_headers
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = (
arize_phoenix_config.otlp_auth_headers
)
for callback in _in_memory_loggers:
if (
@ -3603,9 +3577,9 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
exporter="otlp_http",
endpoint="https://langtrace.ai/api/trace",
)
os.environ[
"OTEL_EXPORTER_OTLP_TRACES_HEADERS"
] = f"api_key={os.getenv('LANGTRACE_API_KEY')}"
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = (
f"api_key={os.getenv('LANGTRACE_API_KEY')}"
)
for callback in _in_memory_loggers:
if (
isinstance(callback, OpenTelemetry)
@ -4305,10 +4279,10 @@ class StandardLoggingPayloadSetup:
for key in StandardLoggingHiddenParams.__annotations__.keys():
if key in hidden_params:
if key == "additional_headers":
clean_hidden_params[
"additional_headers"
] = StandardLoggingPayloadSetup.get_additional_headers(
hidden_params[key]
clean_hidden_params["additional_headers"] = (
StandardLoggingPayloadSetup.get_additional_headers(
hidden_params[key]
)
)
else:
clean_hidden_params[key] = hidden_params[key] # type: ignore
@ -4369,7 +4343,7 @@ class StandardLoggingPayloadSetup:
s3_object_key = get_s3_object_key(
s3_path=s3_path, # Use actual s3_path from logger configuration
team_alias_prefix="", # Don't split by team alias for cold storage
prefix="", # Don't split by team alias for cold storage
start_time=start_time,
s3_file_name=s3_file_name,
)
@ -4871,9 +4845,9 @@ def scrub_sensitive_keys_in_metadata(litellm_params: Optional[dict]):
):
for k, v in metadata["user_api_key_metadata"].items():
if k == "logging": # prevent logging user logging keys
cleaned_user_api_key_metadata[
k
] = "scrubbed_by_litellm_for_sensitive_keys"
cleaned_user_api_key_metadata[k] = (
"scrubbed_by_litellm_for_sensitive_keys"
)
else:
cleaned_user_api_key_metadata[k] = v

38
litellm/litellm_core_utils/llm_cost_calc/usage_object_transformation.py Normal file
View file

@ -0,0 +1,38 @@
from typing import Any, Optional, Union
from litellm.types.utils import (
PromptTokensDetailsWrapper,
TranscriptionUsageDurationObject,
TranscriptionUsageTokensObject,
Usage,
)
class TranscriptionUsageObjectTransformation:
@staticmethod
def is_transcription_usage_object(
usage_object: Any,
) -> bool:
return isinstance(usage_object, TranscriptionUsageDurationObject) or isinstance(
usage_object, TranscriptionUsageTokensObject
)
@staticmethod
def transform_transcription_usage_object(
usage_object: Union[
TranscriptionUsageDurationObject, TranscriptionUsageTokensObject
],
) -> Optional[Usage]:
if isinstance(usage_object, TranscriptionUsageDurationObject):
return None
elif isinstance(usage_object, TranscriptionUsageTokensObject):
return Usage(
prompt_tokens=usage_object.input_tokens,
completion_tokens=usage_object.output_tokens,
total_tokens=usage_object.total_tokens,
prompt_tokens_details=PromptTokensDetailsWrapper(
text_tokens=usage_object.input_token_details.text_tokens,
audio_tokens=usage_object.input_token_details.audio_tokens,
),
)
return None

52
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -1,7 +1,7 @@
# What is this?
## Helper utilities for cost_per_token()
from typing import Any, Literal, Optional, Tuple, TypedDict, cast
from typing import Literal, Optional, Tuple, TypedDict, cast
import litellm
from litellm._logging import verbose_logger
@ -118,21 +118,21 @@ def _generic_cost_per_character(
def _get_service_tier_cost_key(base_key: str, service_tier: Optional[str]) -> str:
"""
Get the appropriate cost key based on service tier.
Args:
base_key: The base cost key (e.g., "input_cost_per_token")
service_tier: The service tier ("flex", "priority", or None for standard)
Returns:
str: The cost key to use (e.g., "input_cost_per_token_flex" or "input_cost_per_token")
"""
if service_tier is None:
return base_key
# Only use service tier specific keys for "flex" and "priority"
if service_tier.lower() in [ServiceTier.FLEX.value, ServiceTier.PRIORITY.value]:
return f"{base_key}_{service_tier.lower()}"
# For any other service tier, use standard pricing
return base_key
@ -152,15 +152,15 @@ def _get_token_base_cost(
# Get service tier aware cost keys
input_cost_key = _get_service_tier_cost_key("input_cost_per_token", service_tier)
output_cost_key = _get_service_tier_cost_key("output_cost_per_token", service_tier)
cache_creation_cost_key = _get_service_tier_cost_key("cache_creation_input_token_cost", service_tier)
cache_read_cost_key = _get_service_tier_cost_key("cache_read_input_token_cost", service_tier)
prompt_base_cost = cast(
float, _get_cost_per_unit(model_info, input_cost_key)
cache_creation_cost_key = _get_service_tier_cost_key(
"cache_creation_input_token_cost", service_tier
)
completion_base_cost = cast(
float, _get_cost_per_unit(model_info, output_cost_key)
cache_read_cost_key = _get_service_tier_cost_key(
"cache_read_input_token_cost", service_tier
)
prompt_base_cost = cast(float, _get_cost_per_unit(model_info, input_cost_key))
completion_base_cost = cast(float, _get_cost_per_unit(model_info, output_cost_key))
cache_creation_cost = cast(
float, _get_cost_per_unit(model_info, cache_creation_cost_key)
)
@ -168,9 +168,7 @@ def _get_token_base_cost(
float,
_get_cost_per_unit(model_info, "cache_creation_input_token_cost_above_1hr"),
)
cache_read_cost = cast(
float, _get_cost_per_unit(model_info, cache_read_cost_key)
)
cache_read_cost = cast(float, _get_cost_per_unit(model_info, cache_read_cost_key))
## CHECK IF ABOVE THRESHOLD
threshold: Optional[float] = None
@ -278,7 +276,7 @@ def _get_cost_per_unit(
verbose_logger.exception(
f"litellm.litellm_core_utils.llm_cost_calc.utils.py::calculate_cost_per_component(): Exception occured - {cost_per_unit}\nDefaulting to 0.0"
)
# If the service tier key doesn't exist or is None, try to fall back to the standard key
if cost_per_unit is None:
# Check if any service tier suffix exists in the cost key using ServiceTier enum
@ -286,7 +284,7 @@ def _get_cost_per_unit(
suffix = f"_{service_tier.value}"
if suffix in cost_key:
# Extract the base key by removing the matched suffix
base_key = cost_key.replace(suffix, '')
base_key = cost_key.replace(suffix, "")
fallback_cost = model_info.get(base_key)
if isinstance(fallback_cost, float):
return fallback_cost
@ -300,7 +298,7 @@ def _get_cost_per_unit(
f"litellm.litellm_core_utils.llm_cost_calc.utils.py::_get_cost_per_unit(): Exception occured - {fallback_cost}\nDefaulting to 0.0"
)
break # Only try the first matching suffix
return default_value
@ -495,7 +493,10 @@ def _calculate_input_cost(
def generic_cost_per_token(
model: str, usage: Usage, custom_llm_provider: str, service_tier: Optional[str] = None
model: str,
usage: Usage,
custom_llm_provider: str,
service_tier: Optional[str] = None,
) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -547,7 +548,9 @@ def generic_cost_per_token(
cache_creation_cost,
cache_creation_cost_above_1hr,
cache_read_cost,
) = _get_token_base_cost(model_info=model_info, usage=usage, service_tier=service_tier)
) = _get_token_base_cost(
model_info=model_info, usage=usage, service_tier=service_tier
)
prompt_cost = _calculate_input_cost(
prompt_tokens_details=prompt_tokens_details,
@ -631,7 +634,7 @@ class CostCalculatorUtils:
@staticmethod
def route_image_generation_cost_calculator(
model: str,
completion_response: Any,
completion_response: ImageResponse,
custom_llm_provider: Optional[str] = None,
quality: Optional[str] = None,
n: Optional[int] = None,
@ -658,6 +661,13 @@ class CostCalculatorUtils:
cost_calculator as vertex_ai_image_cost_calculator,
)
if size is None:
size = completion_response.size or "1024-x-1024"
if quality is None:
quality = completion_response.quality or "standard"
if n is None:
n = len(completion_response.data) if completion_response.data else 0
if custom_llm_provider == litellm.LlmProviders.VERTEX_AI.value:
if isinstance(completion_response, ImageResponse):
return vertex_ai_image_cost_calculator(

20
litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
View file

@ -37,6 +37,8 @@ from litellm.types.utils import (
TextChoices,
TextCompletionResponse,
TranscriptionResponse,
TranscriptionUsageDurationObject,
TranscriptionUsageTokensObject,
Usage,
)
@ -684,6 +686,24 @@ def convert_to_model_response_object( # noqa: PLR0915
if key in response_object:
setattr(model_response_object, key, response_object[key])
if "usage" in response_object and response_object["usage"] is not None:
tr_usage_object: Optional[
Union[
TranscriptionUsageDurationObject, TranscriptionUsageTokensObject
]
] = None
if response_object["usage"].get("type", None) == "duration":
tr_usage_object = TranscriptionUsageDurationObject(
**response_object["usage"]
)
elif response_object["usage"].get("type", None) == "tokens":
tr_usage_object = TranscriptionUsageTokensObject(
**response_object["usage"]
)
if tr_usage_object is not None:
setattr(model_response_object, "usage", tr_usage_object)
if hidden_params is not None:
model_response_object._hidden_params = hidden_params

6
litellm/litellm_core_utils/redact_messages.py
View file

@ -7,17 +7,17 @@
#
# Thank you users! We ❤️ you! - Krrish & Ishaan
import asyncio
import copy
from typing import TYPE_CHECKING, Any, Optional
import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.secret_managers.main import str_to_bool
from litellm.types.utils import StandardCallbackDynamicParams
from litellm.litellm_core_utils.core_helpers import (
get_metadata_variable_name_from_kwargs,
)
import asyncio
from litellm.secret_managers.main import str_to_bool
from litellm.types.utils import StandardCallbackDynamicParams
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import (

2
litellm/litellm_core_utils/safe_json_dumps.py
View file

@ -49,4 +49,4 @@ def safe_dumps(data: Any, max_depth: int = DEFAULT_MAX_RECURSE_DEPTH) -> str:
return "Unserializable Object"
safe_data = _serialize(data, set(), 0)
return json.dumps(safe_data, default=str)
return json.dumps(safe_data, default=str)

59
litellm/llms/anthropic/chat/transformation.py
View file

@ -82,9 +82,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
to pass metadata to anthropic, it's {"user_id": "any-relevant-information"}
"""
max_tokens: Optional[int] = (
DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS # anthropic requires a default value (Opus, Sonnet, and Haiku have the same default)
)
max_tokens: Optional[
int
] = DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS # anthropic requires a default value (Opus, Sonnet, and Haiku have the same default)
stop_sequences: Optional[list] = None
temperature: Optional[int] = None
top_p: Optional[int] = None
@ -118,7 +118,6 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
return super().get_config()
def get_supported_openai_params(self, model: str):
params = [
"stream",
"stop",
@ -465,11 +464,11 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if mcp_servers:
optional_params["mcp_servers"] = mcp_servers
if param == "tool_choice" or param == "parallel_tool_calls":
_tool_choice: Optional[AnthropicMessagesToolChoice] = (
self._map_tool_choice(
tool_choice=non_default_params.get("tool_choice"),
parallel_tool_use=non_default_params.get("parallel_tool_calls"),
)
_tool_choice: Optional[
AnthropicMessagesToolChoice
] = self._map_tool_choice(
tool_choice=non_default_params.get("tool_choice"),
parallel_tool_use=non_default_params.get("parallel_tool_calls"),
)
if _tool_choice is not None:
@ -517,6 +516,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
self._add_tools_to_optional_params(
optional_params=optional_params, tools=[hosted_web_search_tool]
)
elif param == "extra_headers":
optional_params["extra_headers"] = value
## handle thinking tokens
self.update_optional_params_with_thinking_tokens(
@ -575,9 +576,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
text=system_message_block["content"],
)
if "cache_control" in system_message_block:
anthropic_system_message_content["cache_control"] = (
system_message_block["cache_control"]
)
anthropic_system_message_content[
"cache_control"
] = system_message_block["cache_control"]
anthropic_system_message_list.append(
anthropic_system_message_content
)
@ -591,9 +592,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
)
if "cache_control" in _content:
anthropic_system_message_content["cache_control"] = (
_content["cache_control"]
)
anthropic_system_message_content[
"cache_control"
] = _content["cache_control"]
anthropic_system_message_list.append(
anthropic_system_message_content
@ -641,13 +642,25 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
)
return tools
def update_headers_with_optional_anthropic_beta(self, headers: dict, optional_params: dict) -> dict:
def update_headers_with_optional_anthropic_beta(
self, headers: dict, optional_params: dict
) -> dict:
"""Update headers with optional anthropic beta."""
_tools = optional_params.get("tools", [])
for tool in _tools:
if tool.get("type", None) and tool.get("type").startswith(ANTHROPIC_HOSTED_TOOLS.WEB_FETCH.value):
headers["anthropic-beta"] = ANTHROPIC_BETA_HEADER_VALUES.WEB_FETCH_2025_09_10.value
if tool.get("type", None) and tool.get("type").startswith(
ANTHROPIC_HOSTED_TOOLS.WEB_FETCH.value
):
headers[
"anthropic-beta"
] = ANTHROPIC_BETA_HEADER_VALUES.WEB_FETCH_2025_09_10.value
elif tool.get("type", None) and tool.get("type").startswith(
ANTHROPIC_HOSTED_TOOLS.MEMORY.value
):
headers[
"anthropic-beta"
] = ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
return headers
def transform_request(
@ -685,7 +698,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
llm_provider="anthropic",
)
headers = self.update_headers_with_optional_anthropic_beta(headers=headers, optional_params=optional_params)
headers = self.update_headers_with_optional_anthropic_beta(
headers=headers, optional_params=optional_params
)
# Separate system prompt from rest of message
anthropic_system_message_list = self.translate_system_message(messages=messages)
@ -764,7 +779,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
return _message
def extract_response_content(self, completion_response: dict) -> Tuple[
def extract_response_content(
self, completion_response: dict
) -> Tuple[
str,
Optional[List[Any]],
Optional[

51
litellm/llms/azure/azure.py
View file

@ -36,6 +36,7 @@ from .common_utils import (
process_azure_headers,
select_azure_base_url_or_endpoint,
)
from .image_generation import get_azure_image_generation_config
class AzureOpenAIAssistantsAPIConfig:
@ -1011,7 +1012,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
async def aimage_generation(
self,
data: dict,
model_response: ModelResponse,
model_response: Optional[ImageResponse],
azure_client_params: dict,
api_key: str,
input: list,
@ -1020,6 +1021,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
client=None,
timeout=None,
) -> litellm.ImageResponse:
response: Optional[dict] = None
try:
# response = await azure_client.images.generate(**data, timeout=timeout)
@ -1052,21 +1054,38 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
data=data,
headers=headers,
)
response = httpx_response.json()
stringified_response = response
## LOGGING
logging_obj.post_call(
input=input,
api_key=api_key,
additional_args={"complete_input_dict": data},
original_response=stringified_response,
)
return convert_to_model_response_object( # type: ignore
response_object=stringified_response,
model_response_object=model_response,
response_type="image_generation",
provider_config = get_azure_image_generation_config(
data.get("model", "dall-e-2")
)
if provider_config is not None:
return provider_config.transform_image_generation_response(
model=data.get("model", "dall-e-2"),
raw_response=httpx_response,
model_response=model_response or ImageResponse(),
logging_obj=logging_obj,
request_data=data,
optional_params=data,
litellm_params=data,
encoding=litellm.encoding,
)
else:
response = httpx_response.json()
stringified_response = response
## LOGGING
logging_obj.post_call(
input=input,
api_key=api_key,
additional_args={"complete_input_dict": data},
original_response=stringified_response,
)
return convert_to_model_response_object( # type: ignore
response_object=stringified_response,
model_response_object=model_response,
response_type="image_generation",
)
except Exception as e:
## LOGGING
logging_obj.post_call(
@ -1124,9 +1143,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
if api_key is None and azure_ad_token_provider is not None:
azure_ad_token = azure_ad_token_provider()
if azure_ad_token:
headers.pop(
"api-key", None
)
headers.pop("api-key", None)
headers["Authorization"] = f"Bearer {azure_ad_token}"
# init AzureOpenAI Client

42
litellm/llms/azure/exception_mapping.py Normal file
View file

@ -0,0 +1,42 @@
from typing import Optional
from litellm.exceptions import ContentPolicyViolationError
class AzureOpenAIExceptionMapping:
"""
Class for creating Azure OpenAI specific exceptions
"""
@staticmethod
def create_content_policy_violation_error(
message: str,
model: str,
extra_information: str,
original_exception: Exception,
) -> ContentPolicyViolationError:
"""
Create a content policy violation error
"""
raise ContentPolicyViolationError(
message=f"litellm.ContentPolicyViolationError: AzureException - {message}",
llm_provider="azure",
model=model,
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
provider_specific_fields={
"innererror": AzureOpenAIExceptionMapping._get_innererror_from_exception(original_exception)
},
)
@staticmethod
def _get_innererror_from_exception(original_exception: Exception) -> Optional[dict]:
"""
Azure OpenAI returns the innererror in the body of the exception
This method extracts the innererror from the exception
"""
innererror = None
body_dict = getattr(original_exception, "body", None) or {}
if isinstance(body_dict, dict):
innererror = body_dict.get("innererror")
return innererror

8
litellm/llms/azure_ai/rerank/transformation.py
View file

@ -18,7 +18,12 @@ class AzureAIRerankConfig(CohereRerankConfig):
Azure AI Rerank - Follows the same Spec as Cohere Rerank
"""
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
if api_base is None:
raise ValueError(
"Azure AI API Base is required. api_base=None. Set in call or via `AZURE_AI_API_BASE` env var."
@ -32,6 +37,7 @@ class AzureAIRerankConfig(CohereRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
if api_key is None:
api_key = get_secret_str("AZURE_AI_API_KEY") or litellm.azure_key

8
litellm/llms/base_llm/rerank/transformation.py
View file

@ -23,6 +23,7 @@ class BaseRerankConfig(ABC):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
pass
@ -50,7 +51,12 @@ class BaseRerankConfig(ABC):
return model_response
@abstractmethod
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
"""
OPTIONAL

8
litellm/llms/base_llm/videos/transformation.py
View file

@ -92,10 +92,11 @@ class BaseVideoConfig(ABC):
self,
model: str,
prompt: str,
api_base: str,
video_create_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict, RequestFiles]:
) -> Tuple[Dict, RequestFiles, str]:
pass
@abstractmethod
@ -104,6 +105,8 @@ class BaseVideoConfig(ABC):
model: str,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
pass
@ -154,6 +157,7 @@ class BaseVideoConfig(ABC):
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
pass
@ -181,6 +185,7 @@ class BaseVideoConfig(ABC):
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> Dict[str,str]:
pass
@ -229,6 +234,7 @@ class BaseVideoConfig(ABC):
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
pass

1
litellm/llms/bedrock/common_utils.py
View file

@ -237,6 +237,7 @@ def init_bedrock_client(
"sts",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
verify=ssl_verify
)
sts_response = sts_client.assume_role(

8
litellm/llms/cohere/rerank/transformation.py
View file

@ -20,7 +20,12 @@ class CohereRerankConfig(BaseRerankConfig):
def __init__(self) -> None:
pass
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
if api_base:
# Remove trailing slashes and ensure clean base URL
api_base = api_base.rstrip("/")
@ -72,6 +77,7 @@ class CohereRerankConfig(BaseRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
if api_key is None:
api_key = (

7
litellm/llms/cohere/rerank_v2/transformation.py
View file

@ -12,7 +12,12 @@ class CohereRerankV2Config(CohereRerankConfig):
def __init__(self) -> None:
pass
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
if api_base:
# Remove trailing slashes and ensure clean base URL
api_base = api_base.rstrip("/")

112
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -921,11 +921,13 @@ class BaseLLMHTTPHandler:
api_key=api_key,
headers=headers or {},
model=model,
optional_params=optional_rerank_params,
)
api_base = provider_config.get_complete_url(
api_base=api_base,
model=model,
optional_params=optional_rerank_params,
)
data = provider_config.transform_rerank_request(
@ -2009,6 +2011,9 @@ class BaseLLMHTTPHandler:
headers=headers,
)
if extra_body:
data.update(extra_body)
## LOGGING
logging_obj.pre_call(
input=input,
@ -2135,6 +2140,9 @@ class BaseLLMHTTPHandler:
headers=headers,
)
if extra_body:
data.update(extra_body)
## LOGGING
logging_obj.pre_call(
input=input,
@ -4091,7 +4099,7 @@ class BaseLLMHTTPHandler:
or {},
model=model,
)
if extra_headers:
headers.update(extra_headers)
@ -4101,12 +4109,13 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
data, files = video_generation_provider_config.transform_video_create_request(
data, files, api_base = video_generation_provider_config.transform_video_create_request(
model=model,
prompt=prompt,
video_create_optional_request_params=video_generation_optional_request_params,
litellm_params=litellm_params,
headers=headers,
api_base=api_base,
)
## LOGGING
@ -4132,8 +4141,8 @@ class BaseLLMHTTPHandler:
timeout=timeout,
)
# --- END MOCK VIDEO RESPONSE ---
else:
# Use JSON content type for POST requests without files
response = sync_httpx_client.post(
url=api_base,
headers=headers,
@ -4151,6 +4160,8 @@ class BaseLLMHTTPHandler:
model=model,
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
request_data=data,
)
async def async_video_generation_handler(
@ -4198,9 +4209,10 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
data, files = video_generation_provider_config.transform_video_create_request(
data, files, api_base = video_generation_provider_config.transform_video_create_request(
model=model,
prompt=prompt,
api_base=api_base,
video_create_optional_request_params=video_generation_optional_request_params,
litellm_params=litellm_params,
headers=headers,
@ -4218,7 +4230,7 @@ class BaseLLMHTTPHandler:
)
try:
# Use JSON when no files, otherwise use form data with files
#Use JSON when no files, otherwise use form data with files
if files is None or len(files) == 0:
response = await async_httpx_client.post(
url=api_base,
@ -4245,6 +4257,8 @@ class BaseLLMHTTPHandler:
model=model,
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
request_data=data,
)
###### VIDEO CONTENT HANDLER ######
@ -4300,7 +4314,7 @@ class BaseLLMHTTPHandler:
)
# Transform the request using the provider config
url, params = video_content_provider_config.transform_video_content_request(
url, data = video_content_provider_config.transform_video_content_request(
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
@ -4308,12 +4322,21 @@ class BaseLLMHTTPHandler:
)
try:
# Make the GET request to download content
response = sync_httpx_client.get(
url=url,
headers=headers,
params=params,
)
# Use POST if params contains data (e.g., Vertex AI fetchPredictOperation)
# Otherwise use GET (e.g., OpenAI video content download)
if data:
response = sync_httpx_client.post(
url=url,
headers=headers,
json=data,
)
else:
# Otherwise it's a GET request with query params
response = sync_httpx_client.get(
url=url,
headers=headers,
params=data,
)
# Transform the response using the provider config
return video_content_provider_config.transform_video_content_response(
@ -4366,7 +4389,7 @@ class BaseLLMHTTPHandler:
)
# Transform the request using the provider config
url, params = video_content_provider_config.transform_video_content_request(
url, data = video_content_provider_config.transform_video_content_request(
video_id=video_id,
api_base=api_base,
litellm_params=litellm_params,
@ -4374,12 +4397,21 @@ class BaseLLMHTTPHandler:
)
try:
# Make the GET request to download content
response = await async_httpx_client.get(
url=url,
headers=headers,
params=params,
)
# Use POST if params contains data (e.g., Vertex AI fetchPredictOperation)
# Otherwise use GET (e.g., OpenAI video content download)
if data:
response = await async_httpx_client.post(
url=url,
headers=headers,
json=data,
)
else:
# Otherwise it's a GET request with query params
response = await async_httpx_client.get(
url=url,
headers=headers,
params=data,
)
# Transform the response using the provider config
return video_content_provider_config.transform_video_content_response(
@ -4484,6 +4516,7 @@ class BaseLLMHTTPHandler:
return video_remix_provider_config.transform_video_remix_response(
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
except Exception as e:
@ -4565,6 +4598,7 @@ class BaseLLMHTTPHandler:
return video_remix_provider_config.transform_video_remix_response(
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
except Exception as e:
@ -4700,6 +4734,7 @@ class BaseLLMHTTPHandler:
return video_list_provider_config.transform_video_list_response(
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
except Exception as e:
@ -4855,17 +4890,29 @@ class BaseLLMHTTPHandler:
"api_base": url,
"headers": headers,
"video_id": video_id,
"data": data,
},
)
try:
response = sync_httpx_client.get(
url=url,
headers=headers,
)
# Use POST if data is provided (e.g., Vertex AI fetchPredictOperation)
# Otherwise use GET (e.g., OpenAI video status)
if data:
response = sync_httpx_client.post(
url=url,
headers=headers,
json=data,
)
else:
response = sync_httpx_client.get(
url=url,
headers=headers,
)
return video_status_provider_config.transform_video_status_retrieve_response(
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
except Exception as e:
@ -4929,17 +4976,28 @@ class BaseLLMHTTPHandler:
"api_base": url,
"headers": headers,
"video_id": video_id,
"data": data,
},
)
try:
response = await async_httpx_client.get(
url=url,
headers=headers,
)
# Use POST if data is provided (e.g., Vertex AI fetchPredictOperation)
# Otherwise use GET (e.g., OpenAI video status)
if data:
response = await async_httpx_client.post(
url=url,
headers=headers,
json=data,
)
else:
response = await async_httpx_client.get(
url=url,
headers=headers,
)
return video_status_provider_config.transform_video_status_retrieve_response(
raw_response=response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
except Exception as e:

2
litellm/llms/databricks/chat/transformation.py
View file

@ -640,7 +640,7 @@ class DatabricksChatResponseIterator(BaseModelResponseIterator):
for _tc in tool_calls:
if _tc.get("function", {}).get("arguments") == "{}":
_tc["function"]["arguments"] = "" # avoid invalid json
if isinstance(choice["delta"]["content"], list) and (
if isinstance(choice["delta"].get("content"), list) and (
content := choice["delta"]["content"]
):
if citations := content[0].get("citations"):

8
litellm/llms/deepinfra/rerank/transformation.py
View file

@ -28,7 +28,12 @@ class DeepinfraRerankConfig(BaseRerankConfig):
Deepinfra Rerank - Follows the same Spec as Cohere Rerank
"""
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
"""
Constructs the complete DeepInfra inference endpoint URL for rerank.
@ -63,6 +68,7 @@ class DeepinfraRerankConfig(BaseRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
if api_key is None:
api_key = get_secret_str("DEEPINFRA_API_KEY")

5
litellm/llms/gemini/videos/__init__.py Normal file
View file

@ -0,0 +1,5 @@
# Gemini Video Generation Support
from .transformation import GeminiVideoConfig
__all__ = ["GeminiVideoConfig"]

523
litellm/llms/gemini/videos/transformation.py Normal file
View file

@ -0,0 +1,523 @@
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union
import base64
import httpx
from httpx._types import RequestFiles
from litellm.types.videos.main import VideoCreateOptionalRequestParams, VideoObject
from litellm.types.router import GenericLiteLLMParams
from litellm.secret_managers.main import get_secret_str
from litellm.types.videos.utils import (
encode_video_id_with_provider,
extract_original_video_id,
)
from litellm.images.utils import ImageEditRequestUtils
import litellm
from litellm.types.llms.gemini import GeminiLongRunningOperationResponse, GeminiVideoGenerationInstance, GeminiVideoGenerationParameters, GeminiVideoGenerationRequest
from litellm.constants import DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
from ...base_llm.videos.transformation import BaseVideoConfig as _BaseVideoConfig
from ...base_llm.chat.transformation import BaseLLMException as _BaseLLMException
LiteLLMLoggingObj = _LiteLLMLoggingObj
BaseVideoConfig = _BaseVideoConfig
BaseLLMException = _BaseLLMException
else:
LiteLLMLoggingObj = Any
BaseVideoConfig = Any
BaseLLMException = Any
def _convert_image_to_gemini_format(image_file) -> Dict[str, str]:
"""
Convert image file to Gemini format with base64 encoding and MIME type.
Args:
image_file: File-like object opened in binary mode (e.g., open("path", "rb"))
Returns:
Dict with bytesBase64Encoded and mimeType
"""
mime_type = ImageEditRequestUtils.get_image_content_type(image_file)
if hasattr(image_file, 'seek'):
image_file.seek(0)
image_bytes = image_file.read()
base64_encoded = base64.b64encode(image_bytes).decode("utf-8")
return {
"bytesBase64Encoded": base64_encoded,
"mimeType": mime_type
}
class GeminiVideoConfig(BaseVideoConfig):
"""
Configuration class for Gemini (Veo) video generation.
Veo uses a long-running operation model:
1. POST to :predictLongRunning returns operation name
2. Poll operation until done=true
3. Extract video URI from response
4. Download video using file API
"""
def __init__(self):
super().__init__()
def get_supported_openai_params(self, model: str) -> list:
"""
Get the list of supported OpenAI parameters for Veo video generation.
Veo supports minimal parameters compared to OpenAI.
"""
return [
"model",
"prompt",
"input_reference",
"seconds",
"size"
]
def map_openai_params(
self,
video_create_optional_params: VideoCreateOptionalRequestParams,
model: str,
drop_params: bool,
) -> Dict[str, Any]:
"""
Map OpenAI-style parameters to Veo format.
Mappings:
- prompt → prompt
- input_reference → image
- size → aspectRatio (e.g., "1280x720" → "16:9")
- seconds → durationSeconds (defaults to 4 seconds if not provided)
All other params are passed through as-is to support Gemini-specific parameters.
"""
mapped_params: Dict[str, Any] = {}
# Get supported OpenAI params (exclude "model" and "prompt" which are handled separately)
supported_openai_params = self.get_supported_openai_params(model)
openai_params_to_map = {
param for param in supported_openai_params
if param not in {"model", "prompt"}
}
# Map input_reference to image
if "input_reference" in video_create_optional_params:
mapped_params["image"] = video_create_optional_params["input_reference"]
# Map size to aspectRatio
if "size" in video_create_optional_params:
size = video_create_optional_params["size"]
if size is not None:
aspect_ratio = self._convert_size_to_aspect_ratio(size)
if aspect_ratio:
mapped_params["aspectRatio"] = aspect_ratio
# Map seconds to durationSeconds, default to 4 seconds (matching OpenAI)
if "seconds" in video_create_optional_params:
seconds = video_create_optional_params["seconds"]
try:
duration = int(seconds) if isinstance(seconds, str) else seconds
if duration is not None:
mapped_params["durationSeconds"] = duration
except (ValueError, TypeError):
# If conversion fails, use default
pass
# Pass through any other params that weren't mapped (Gemini-specific params)
for key, value in video_create_optional_params.items():
if key not in openai_params_to_map and key not in mapped_params:
mapped_params[key] = value
return mapped_params
def _convert_size_to_aspect_ratio(self, size: str) -> Optional[str]:
"""
Convert OpenAI size format to Veo aspectRatio format.
https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-videos
Supported aspect ratios: 9:16 (portrait), 16:9 (landscape)
"""
if not size:
return None
aspect_ratio_map = {
"1280x720": "16:9",
"1920x1080": "16:9",
"720x1280": "9:16",
"1080x1920": "9:16",
}
return aspect_ratio_map.get(size, "16:9")
def validate_environment(
self,
headers: dict,
model: str,
api_key: Optional[str] = None,
) -> dict:
"""
Validate environment and add Gemini API key to headers.
Gemini uses x-goog-api-key header for authentication.
"""
api_key = (
api_key
or litellm.api_key
or get_secret_str("GOOGLE_API_KEY")
or get_secret_str("GEMINI_API_KEY")
)
if not api_key:
raise ValueError(
"GEMINI_API_KEY or GOOGLE_API_KEY is required for Veo video generation. "
"Set it via environment variable or pass it as api_key parameter."
)
headers.update({
"x-goog-api-key": api_key,
"Content-Type": "application/json",
})
return headers
def get_complete_url(
self,
model: str,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for Veo video generation.
For video creation: returns full URL with :predictLongRunning
For status/delete: returns base URL only
"""
if api_base is None:
api_base = get_secret_str("GEMINI_API_BASE") or "https://generativelanguage.googleapis.com"
if not model or model == "":
return api_base.rstrip('/')
model_name = model.replace("gemini/", "")
url = f"{api_base.rstrip('/')}/v1beta/models/{model_name}:predictLongRunning"
return url
def transform_video_create_request(
self,
model: str,
prompt: str,
api_base: str,
video_create_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict, RequestFiles, str]:
"""
Transform the video creation request for Veo API.
Veo expects:
{
"instances": [
{
"prompt": "A cat playing with a ball of yarn"
}
],
"parameters": {
"aspectRatio": "16:9",
"durationSeconds": 8,
"resolution": "720p"
}
}
"""
instance = GeminiVideoGenerationInstance(prompt=prompt)
params_copy = video_create_optional_request_params.copy()
if "image" in params_copy and params_copy["image"] is not None:
image_data = _convert_image_to_gemini_format(params_copy["image"])
params_copy["image"] = image_data
parameters = GeminiVideoGenerationParameters(**params_copy)
request_body_obj = GeminiVideoGenerationRequest(
instances=[instance],
parameters=parameters
)
request_data = request_body_obj.model_dump(exclude_none=True)
return request_data, [], api_base
def transform_video_create_response(
self,
model: str,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
"""
Transform the Veo video creation response.
Veo returns:
{
"name": "operations/generate_1234567890",
"metadata": {...},
"done": false,
"error": {...}
}
We return this as a VideoObject with:
- id: operation name (used for polling)
- status: "processing"
- usage: includes duration_seconds for cost calculation
"""
response_data = raw_response.json()
# Parse response using Pydantic model for type safety
try:
operation_response = GeminiLongRunningOperationResponse(**response_data)
except Exception as e:
raise ValueError(f"Failed to parse operation response: {e}")
operation_name = operation_response.name
if not operation_name:
raise ValueError(f"No operation name in Veo response: {response_data}")
if custom_llm_provider:
video_id = encode_video_id_with_provider(operation_name, custom_llm_provider, model)
else:
video_id = operation_name
video_obj = VideoObject(
id=video_id,
object="video",
status="processing",
model=model,
)
usage_data = {}
if request_data:
parameters = request_data.get("parameters", {})
duration = parameters.get("durationSeconds") or DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
if duration is not None:
try:
usage_data["duration_seconds"] = float(duration)
except (ValueError, TypeError):
pass
video_obj.usage = usage_data
return video_obj
def transform_video_status_retrieve_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Transform the video status retrieve request for Veo API.
Veo polls operations at:
GET https://generativelanguage.googleapis.com/v1beta/{operation_name}
"""
operation_name = extract_original_video_id(video_id)
url = f"{api_base.rstrip('/')}/v1beta/{operation_name}"
params: Dict[str, Any] = {}
return url, params
def transform_video_status_retrieve_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""
Transform the Veo operation status response.
Veo returns:
{
"name": "operations/generate_1234567890",
"done": false # or true when complete
}
When done=true:
{
"name": "operations/generate_1234567890",
"done": true,
"response": {
"generateVideoResponse": {
"generatedSamples": [
{
"video": {
"uri": "files/abc123..."
}
}
]
}
}
}
"""
response_data = raw_response.json()
# Parse response using Pydantic model for type safety
operation_response = GeminiLongRunningOperationResponse(**response_data)
operation_name = operation_response.name
is_done = operation_response.done
if custom_llm_provider:
video_id = encode_video_id_with_provider(operation_name, custom_llm_provider, None)
else:
video_id = operation_name
video_obj = VideoObject(
id=video_id,
object="video",
status="processing" if not is_done else "completed"
)
return video_obj
def transform_video_content_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Transform the video content request for Veo API.
For Veo, we need to:
1. Get operation status to extract video URI
2. Return download URL for the video
"""
operation_name = extract_original_video_id(video_id)
status_url = f"{api_base.rstrip('/')}/v1beta/{operation_name}"
client = litellm.module_level_client
status_response = client.get(url=status_url, headers=headers)
status_response.raise_for_status()
response_data = status_response.json()
operation_response = GeminiLongRunningOperationResponse(**response_data)
if not operation_response.done:
raise ValueError(
"Video generation is not complete yet. "
"Please check status with video_status() before downloading."
)
if not operation_response.response:
raise ValueError("No response data in completed operation")
generated_samples = operation_response.response.generateVideoResponse.generatedSamples
download_url = generated_samples[0].video.uri
params: Dict[str, Any] = {}
return download_url, params
def transform_video_content_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
) -> bytes:
"""
Transform the Veo video content download response.
Returns the video bytes directly.
"""
return raw_response.content
def transform_video_remix_request(
self,
video_id: str,
prompt: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Video remix is not supported by Veo API.
"""
raise NotImplementedError(
"Video remix is not supported by Google Veo. "
"Please use video_generation() to create new videos."
)
def transform_video_remix_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""Video remix is not supported."""
raise NotImplementedError("Video remix is not supported by Google Veo.")
def transform_video_list_request(
self,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
after: Optional[str] = None,
limit: Optional[int] = None,
order: Optional[str] = None,
extra_query: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Video list is not supported by Veo API.
"""
raise NotImplementedError(
"Video list is not supported by Google Veo. "
"Use the operations endpoint directly if you need to list operations."
)
def transform_video_list_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> Dict[str, str]:
"""Video list is not supported."""
raise NotImplementedError("Video list is not supported by Google Veo.")
def transform_video_delete_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Video delete is not supported by Veo API.
"""
raise NotImplementedError(
"Video delete is not supported by Google Veo. "
"Videos are automatically cleaned up by Google."
)
def transform_video_delete_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
) -> VideoObject:
"""Video delete is not supported."""
raise NotImplementedError("Video delete is not supported by Google Veo.")
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
) -> BaseLLMException:
from ..common_utils import GeminiError
return GeminiError(
status_code=status_code,
message=error_message,
headers=headers,
)

10
litellm/llms/hosted_vllm/rerank/transformation.py
View file

@ -37,7 +37,12 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
def __init__(self) -> None:
pass
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
if api_base:
# Remove trailing slashes and ensure clean base URL
api_base = api_base.rstrip("/")
@ -91,6 +96,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
if api_key is None:
api_key = get_secret_str("HOSTED_VLLM_API_KEY") or "fake-api-key"
@ -150,7 +156,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
f"Error parsing response: {raw_response.text}, status_code={raw_response.status_code}"
)
return RerankResponse(**raw_response_json)
return self._transform_response(raw_response_json)
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]

8
litellm/llms/huggingface/rerank/transformation.py
View file

@ -60,7 +60,12 @@ class HuggingFaceRerankConfig(BaseRerankConfig):
else:
return "https://api-inference.huggingface.co"
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
"""
Get the complete URL for the API call, including the /rerank suffix if necessary.
"""
@ -117,6 +122,7 @@ class HuggingFaceRerankConfig(BaseRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
api_base: Optional[str] = None,
) -> dict:
# Get API credentials

8
litellm/llms/infinity/rerank/transformation.py
View file

@ -26,7 +26,12 @@ from ..common_utils import InfinityError
class InfinityRerankConfig(CohereRerankConfig):
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
if api_base is None:
raise ValueError("api_base is required for Infinity rerank")
# Remove trailing slashes and ensure clean base URL
@ -40,6 +45,7 @@ class InfinityRerankConfig(CohereRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
if api_key is None:
api_key = (

13
litellm/llms/jina_ai/rerank/transformation.py
View file

@ -55,7 +55,12 @@ class JinaAIRerankConfig(BaseRerankConfig):
**optional_params,
))
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
base_path = "/v1/rerank"
if api_base is None:
@ -122,7 +127,11 @@ class JinaAIRerankConfig(BaseRerankConfig):
) # Return response
def validate_environment(
self, headers: Dict, model: str, api_key: Optional[str] = None
self,
headers: Dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> Dict:
if api_key is None:
raise ValueError(

8
litellm/llms/nvidia_nim/rerank/transformation.py
View file

@ -55,7 +55,12 @@ class NvidiaNimRerankConfig(BaseRerankConfig):
def __init__(self) -> None:
pass
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
"""
Construct the Nvidia NIM rerank URL.
@ -131,6 +136,7 @@ class NvidiaNimRerankConfig(BaseRerankConfig):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> dict:
"""
Validate that the Nvidia NIM API key is present.

24
litellm/llms/openai/cost_calculation.py
View file

@ -18,7 +18,9 @@ def cost_router(call_type: CallTypes) -> Literal["cost_per_token", "cost_per_sec
return "cost_per_token"
def cost_per_token(model: str, usage: Usage, service_tier: Optional[str] = None) -> Tuple[float, float]:
def cost_per_token(
model: str, usage: Usage, service_tier: Optional[str] = None
) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -31,7 +33,10 @@ def cost_per_token(model: str, usage: Usage, service_tier: Optional[str] = None)
"""
## CALCULATE INPUT COST
return generic_cost_per_token(
model=model, usage=usage, custom_llm_provider="openai", service_tier=service_tier
model=model,
usage=usage,
custom_llm_provider="openai",
service_tier=service_tier,
)
# ### Non-cached text tokens
# non_cached_text_tokens = usage.prompt_tokens
@ -92,6 +97,7 @@ def cost_per_second(
Returns:
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
"""
## GET MODEL INFO
model_info = get_model_info(
model=model, custom_llm_provider=custom_llm_provider or "openai"
@ -123,18 +129,16 @@ def cost_per_second(
def video_generation_cost(
model: str,
duration_seconds: float,
custom_llm_provider: Optional[str] = None
model: str, duration_seconds: float, custom_llm_provider: Optional[str] = None
) -> float:
"""
Calculates the cost for video generation based on duration in seconds.
Input:
- model: str, the model name without provider prefix
- duration_seconds: float, the duration of the generated video in seconds
- custom_llm_provider: str, the custom llm provider
Returns:
float - total_cost_in_usd
"""
@ -142,7 +146,7 @@ def video_generation_cost(
model_info = get_model_info(
model=model, custom_llm_provider=custom_llm_provider or "openai"
)
# Check for video-specific cost per second
video_cost_per_second = model_info.get("output_cost_per_video_per_second")
if video_cost_per_second is not None:
@ -150,7 +154,7 @@ def video_generation_cost(
f"For model={model} - output_cost_per_video_per_second: {video_cost_per_second}; duration: {duration_seconds}"
)
return video_cost_per_second * duration_seconds
# Fallback to general output cost per second
output_cost_per_second = model_info.get("output_cost_per_second")
if output_cost_per_second is not None:
@ -158,7 +162,7 @@ def video_generation_cost(
f"For model={model} - output_cost_per_second: {output_cost_per_second}; duration: {duration_seconds}"
)
return output_cost_per_second * duration_seconds
# If no cost information found, return 0
verbose_logger.warning(
f"No cost information found for video model {model}. Please add pricing to model_prices_and_context_window.json"

51
litellm/llms/openai/image_generation/dall_e_2_transformation.py
View file

@ -1,9 +1,16 @@
from typing import List
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageResponse
from litellm.utils import convert_to_model_response_object
if TYPE_CHECKING:
from litellm.litellm_core_utils.logging import Logging as LiteLLMLoggingObj
class DallE2ImageGenerationConfig(BaseImageGenerationConfig):
@ -36,3 +43,45 @@ class DallE2ImageGenerationConfig(BaseImageGenerationConfig):
)
return optional_params
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: "LiteLLMLoggingObj",
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
response = raw_response.json()
stringified_response = response
## LOGGING
logging_obj.post_call(
input=request_data.get("prompt", ""),
api_key=api_key,
additional_args={"complete_input_dict": request_data},
original_response=stringified_response,
)
image_response: ImageResponse = convert_to_model_response_object( # type: ignore
response_object=stringified_response,
model_response_object=model_response,
response_type="image_generation",
)
# set optional params
image_response.size = optional_params.get(
"size", "1024x1024"
) # default is always 1024x1024
image_response.quality = optional_params.get(
"quality", "standard"
) # always standard for dall-e-2
image_response.output_format = optional_params.get(
"output_format", "png"
) # always png for dall-e-2
return image_response

51
litellm/llms/openai/image_generation/dall_e_3_transformation.py
View file

@ -1,9 +1,16 @@
from typing import List
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageResponse
from litellm.utils import convert_to_model_response_object
if TYPE_CHECKING:
from litellm.litellm_core_utils.logging import Logging as LiteLLMLoggingObj
class DallE3ImageGenerationConfig(BaseImageGenerationConfig):
@ -36,3 +43,45 @@ class DallE3ImageGenerationConfig(BaseImageGenerationConfig):
)
return optional_params
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: "LiteLLMLoggingObj",
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
response = raw_response.json()
stringified_response = response
## LOGGING
logging_obj.post_call(
input=request_data.get("prompt", ""),
api_key=api_key,
additional_args={"complete_input_dict": request_data},
original_response=stringified_response,
)
image_response: ImageResponse = convert_to_model_response_object( # type: ignore
response_object=stringified_response,
model_response_object=model_response,
response_type="image_generation",
)
# set optional params
image_response.size = optional_params.get(
"size", "1024x1024"
) # default is always 1024x1024
image_response.quality = optional_params.get(
"quality", "hd"
) # always hd for dall-e-3
image_response.output_format = optional_params.get(
"output_format", "png"
) # always png for dall-e-3
return image_response

51
litellm/llms/openai/image_generation/gpt_transformation.py
View file

@ -1,9 +1,16 @@
from typing import List
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageResponse
from litellm.utils import convert_to_model_response_object
if TYPE_CHECKING:
from litellm.litellm_core_utils.logging import Logging as LiteLLMLoggingObj
class GPTImageGenerationConfig(BaseImageGenerationConfig):
@ -45,3 +52,45 @@ class GPTImageGenerationConfig(BaseImageGenerationConfig):
)
return optional_params
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: "LiteLLMLoggingObj",
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
response = raw_response.json()
stringified_response = response
## LOGGING
logging_obj.post_call(
input=request_data.get("prompt", ""),
api_key=api_key,
additional_args={"complete_input_dict": request_data},
original_response=stringified_response,
)
image_response: ImageResponse = convert_to_model_response_object( # type: ignore
response_object=stringified_response,
model_response_object=model_response,
response_type="image_generation",
)
# set optional params
image_response.size = optional_params.get(
"size", "1024x1024"
) # default is always 1024x1024
image_response.quality = optional_params.get(
"quality", "high"
) # always hd for dall-e-3
image_response.output_format = optional_params.get(
"response_format", "png"
) # always png for dall-e-3
return image_response

1
litellm/llms/openai/transcriptions/handler.py
View file

@ -213,6 +213,7 @@ class OpenAIAudioTranscription(OpenAIChatCompletion):
# Extract the actual model from data instead of hardcoding "whisper-1"
actual_model = data.get("model", "whisper-1")
hidden_params = {"model": actual_model, "custom_llm_provider": "openai"}
return convert_to_model_response_object(response_object=stringified_response, model_response_object=model_response, hidden_params=hidden_params, response_type="audio_transcription") # type: ignore
except Exception as e:
## LOGGING

71
litellm/llms/openai/videos/transformation.py
View file

@ -9,6 +9,7 @@ from litellm.types.llms.openai import CreateVideoRequest
from litellm.types.router import GenericLiteLLMParams
from litellm.secret_managers.main import get_secret_str
from litellm.types.videos.main import VideoObject
from litellm.types.videos.utils import encode_video_id_with_provider, extract_original_video_id
import litellm
from litellm.llms.openai.image_edit.transformation import ImageEditRequestUtils
if TYPE_CHECKING:
@ -94,17 +95,18 @@ class OpenAIVideoConfig(BaseVideoConfig):
self,
model: str,
prompt: str,
api_base: str,
video_create_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict, RequestFiles]:
) -> Tuple[Dict, RequestFiles, str]:
"""
Transform the video creation request for OpenAI API.
"""
# Remove model and extra_headers from optional params as they're handled separately
video_create_optional_request_params = {
k: v for k, v in video_create_optional_request_params.items()
if k not in ["model", "extra_headers"]
if k not in ["model", "extra_headers", "prompt"]
}
# Create the request data
@ -129,26 +131,24 @@ class OpenAIVideoConfig(BaseVideoConfig):
image=_input_reference,
field_name="input_reference",
)
# Convert to dict for JSON serialization
return data_without_files, files_list
return data_without_files, files_list, api_base
def transform_video_create_response(
self,
model: str,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
"""
Transform the OpenAI video creation response.
"""
"""Transform the OpenAI video creation response."""
response_data = raw_response.json()
# Transform the response data
video_obj = VideoObject(**response_data) # type: ignore[arg-type]
# Create usage object with duration information for cost calculation
# Video generation API doesn't provide usage, so we create one with duration
if custom_llm_provider and video_obj.id:
video_obj.id = encode_video_id_with_provider(video_obj.id, custom_llm_provider, model)
usage_data = {}
if video_obj:
if hasattr(video_obj, 'seconds') and video_obj.seconds:
@ -156,9 +156,7 @@ class OpenAIVideoConfig(BaseVideoConfig):
usage_data["duration_seconds"] = float(video_obj.seconds)
except (ValueError, TypeError):
pass
# Create the response
video_obj.usage = usage_data
return video_obj
@ -175,11 +173,13 @@ class OpenAIVideoConfig(BaseVideoConfig):
OpenAI API expects the following request:
- GET /v1/videos/{video_id}/content
"""
original_video_id = extract_original_video_id(video_id)
# Construct the URL for video content download
url = f"{api_base.rstrip('/')}/{video_id}/content"
url = f"{api_base.rstrip('/')}/{original_video_id}/content"
# Add video_id as query parameter
params = {"video_id": video_id}
params = {"video_id": original_video_id}
return url, params
@ -198,8 +198,10 @@ class OpenAIVideoConfig(BaseVideoConfig):
OpenAI API expects the following request:
- POST /v1/videos/{video_id}/remix
"""
original_video_id = extract_original_video_id(video_id)
# Construct the URL for video remix
url = f"{api_base.rstrip('/')}/{video_id}/remix"
url = f"{api_base.rstrip('/')}/{original_video_id}/remix"
# Prepare the request data
data = {"prompt": prompt}
@ -215,17 +217,14 @@ class OpenAIVideoConfig(BaseVideoConfig):
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
) -> bytes:
"""
Transform the OpenAI video content download response.
Returns raw video content as bytes.
"""
# For video content download, return the raw content as bytes
"""Transform the OpenAI video content download response."""
return raw_response.content
def transform_video_remix_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""
Transform the OpenAI video remix response.
@ -235,6 +234,9 @@ class OpenAIVideoConfig(BaseVideoConfig):
# Transform the response data
video_obj = VideoObject(**response_data) # type: ignore[arg-type]
if custom_llm_provider and video_obj.id:
video_obj.id = encode_video_id_with_provider(video_obj.id, custom_llm_provider, None)
# Create usage object with duration information for cost calculation
# Video remix API doesn't provide usage, so we create one with duration
usage_data = {}
@ -287,8 +289,20 @@ class OpenAIVideoConfig(BaseVideoConfig):
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> Dict[str,str]:
return raw_response.json()
response_data = raw_response.json()
if custom_llm_provider and "data" in response_data:
for video_obj in response_data.get("data", []):
if isinstance(video_obj, dict) and "id" in video_obj:
video_obj["id"] = encode_video_id_with_provider(
video_obj["id"],
custom_llm_provider,
video_obj.get("model")
)
return response_data
def transform_video_delete_request(
self,
@ -303,8 +317,10 @@ class OpenAIVideoConfig(BaseVideoConfig):
OpenAI API expects the following request:
- DELETE /v1/videos/{video_id}
"""
original_video_id = extract_original_video_id(video_id)
# Construct the URL for video delete
url = f"{api_base.rstrip('/')}/{video_id}"
url = f"{api_base.rstrip('/')}/{original_video_id}"
# No data needed for DELETE request
data: Dict[str, Any] = {}
@ -336,8 +352,11 @@ class OpenAIVideoConfig(BaseVideoConfig):
"""
Transform the OpenAI video retrieve request.
"""
# Extract the original video_id (remove provider encoding if present)
original_video_id = extract_original_video_id(video_id)
# For video retrieve, we just need to construct the URL
url = f"{api_base.rstrip('/')}/{video_id}"
url = f"{api_base.rstrip('/')}/{original_video_id}"
# No additional data needed for GET request
data: Dict[str, Any] = {}
@ -348,6 +367,7 @@ class OpenAIVideoConfig(BaseVideoConfig):
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""
Transform the OpenAI video retrieve response.
@ -355,6 +375,9 @@ class OpenAIVideoConfig(BaseVideoConfig):
response_data = raw_response.json()
# Transform the response data
video_obj = VideoObject(**response_data) # type: ignore[arg-type]
if custom_llm_provider and video_obj.id:
video_obj.id = encode_video_id_with_provider(video_obj.id, custom_llm_provider, None)
return video_obj

1
litellm/llms/perplexity/chat/__init__.py
View file

@ -1 +0,0 @@
"""Perplexity chat completion transformations."""

133
litellm/llms/perplexity/chat/transformation.py
View file

@ -1,32 +1,25 @@
"""Translate from OpenAI's `/v1/chat/completions` to Perplexity's `/v1/chat/completions`."""
"""
Translate from OpenAI's `/v1/chat/completions` to Perplexity's `/v1/chat/completions`
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, List, Optional, Tuple
from typing import Any, List, Optional, Tuple
import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import ModelResponse, PromptTokensDetailsWrapper, Usage
if TYPE_CHECKING:
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAnnotation,
ChatCompletionAnnotationURLCitation,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import Usage, PromptTokensDetailsWrapper
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.types.utils import ModelResponse
from litellm.types.llms.openai import ChatCompletionAnnotation
from litellm.types.llms.openai import ChatCompletionAnnotationURLCitation
class PerplexityChatConfig(OpenAIGPTConfig):
"""Configuration for Perplexity chat completions."""
@property
def custom_llm_provider(self) -> str | None:
"""Return the custom LLM provider name."""
def custom_llm_provider(self) -> Optional[str]:
return "perplexity"
def _get_openai_compatible_provider_info(
@ -40,38 +33,6 @@ class PerplexityChatConfig(OpenAIGPTConfig):
)
return api_base, dynamic_api_key
def validate_environment(
self,
headers: dict,
model: str,
messages: list,
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
"""Validate Perplexity environment and set headers."""
# Get API key from environment if not provided
if api_key is None:
_, api_key = self._get_openai_compatible_provider_info(
api_base=api_base, api_key=api_key
)
# Validate API key is present
if api_key is None:
raise ValueError(
"The api_key client option must be set either by passing api_key to the client or by setting the PERPLEXITY_API_KEY environment variable"
)
# Set authorization header
headers["Authorization"] = f"Bearer {api_key}"
# Ensure Content-Type is set to application/json
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
return headers
def get_supported_openai_params(self, model: str) -> list:
"""
Perplexity supports a subset of OpenAI params
@ -111,8 +72,7 @@ class PerplexityChatConfig(OpenAIGPTConfig):
return base_openai_params
def transform_response( # noqa: PLR0913
def transform_response(
self,
model: str,
raw_response: httpx.Response,
@ -122,11 +82,10 @@ class PerplexityChatConfig(OpenAIGPTConfig):
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
json_mode: Optional[bool] = None,
) -> ModelResponse:
"""Transform Perplexity response to standard format."""
# Call the parent transform_response first to handle the standard transformation
model_response = super().transform_response(
model=model,
@ -145,29 +104,28 @@ class PerplexityChatConfig(OpenAIGPTConfig):
# Extract and enhance usage with Perplexity-specific fields
try:
raw_response_json = raw_response.json()
self.add_cost_to_usage(model_response, raw_response_json)
self._enhance_usage_with_perplexity_fields(
model_response, raw_response_json,
model_response, raw_response_json
)
self._add_citations_as_annotations(model_response, raw_response_json)
except (ValueError, TypeError, KeyError) as e:
except Exception as e:
verbose_logger.debug(f"Error extracting Perplexity-specific usage fields: {e}")
return model_response
def _enhance_usage_with_perplexity_fields(
self, model_response: ModelResponse, raw_response_json: dict,
def _enhance_usage_with_perplexity_fields(
self, model_response: ModelResponse, raw_response_json: dict
) -> None:
"""Extract citation tokens and search queries from Perplexity API response.
Add them to the usage object using standard LiteLLM fields.
"""
Extract citation tokens and search queries from Perplexity API response
and add them to the usage object using standard LiteLLM fields.
"""
if not hasattr(model_response, "usage") or model_response.usage is None:
# Create a usage object if it doesn't exist (when usage was None)
model_response.usage = Usage( # type: ignore[attr-defined]
prompt_tokens=0,
completion_tokens=0,
total_tokens=0,
total_tokens=0
)
usage = model_response.usage # type: ignore[attr-defined]
@ -188,7 +146,7 @@ class PerplexityChatConfig(OpenAIGPTConfig):
# Extract search queries count from usage or response metadata
# Perplexity might include this in the usage object or as separate metadata
perplexity_usage = raw_response_json.get("usage", {})
# Try to extract search queries from usage field first, then root level
num_search_queries = perplexity_usage.get("num_search_queries")
if num_search_queries is None:
@ -197,18 +155,18 @@ class PerplexityChatConfig(OpenAIGPTConfig):
num_search_queries = perplexity_usage.get("search_queries")
if num_search_queries is None:
num_search_queries = raw_response_json.get("search_queries")
# Create or update prompt_tokens_details to include web search requests and citation tokens
if citation_tokens > 0 or (
num_search_queries is not None and num_search_queries > 0
):
if usage.prompt_tokens_details is None:
usage.prompt_tokens_details = PromptTokensDetailsWrapper()
# Store citation tokens count for cost calculation
if citation_tokens > 0:
usage.citation_tokens = citation_tokens
setattr(usage, "citation_tokens", citation_tokens)
# Store search queries count in the standard web_search_requests field
if num_search_queries is not None and num_search_queries > 0:
usage.prompt_tokens_details.web_search_requests = num_search_queries
@ -290,35 +248,4 @@ class PerplexityChatConfig(OpenAIGPTConfig):
if citations:
setattr(model_response, "citations", citations)
if search_results:
setattr(model_response, "search_results", search_results)
def add_cost_to_usage(self, model_response: ModelResponse, raw_response_json: dict) -> None:
"""Add the cost to the usage object."""
try:
usage_data = raw_response_json.get("usage")
if usage_data:
# Try different possible cost field locations
response_cost = None
# Check if cost is directly in usage (flat structure)
if "total_cost" in usage_data:
response_cost = usage_data["total_cost"]
# Check if cost is nested (cost.total_cost structure)
elif "cost" in usage_data and isinstance(usage_data["cost"], dict):
response_cost = usage_data["cost"].get("total_cost")
# Check if cost is a simple value
elif "cost" in usage_data:
response_cost = usage_data["cost"]
if response_cost is not None:
# Store cost in hidden params for the cost calculator to use
if not hasattr(model_response, "_hidden_params"):
model_response._hidden_params = {}
if "additional_headers" not in model_response._hidden_params:
model_response._hidden_params["additional_headers"] = {}
model_response._hidden_params["additional_headers"][
"llm_provider-x-litellm-response-cost"
] = float(response_cost)
except (ValueError, TypeError, KeyError) as e:
verbose_logger.debug(f"Error adding cost to usage: {e}")
# If we can't extract cost, continue without it - don't fail the response
setattr(model_response, "search_results", search_results)

41
litellm/llms/vertex_ai/rerank/transformation.py
View file

@ -27,19 +27,40 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase):
def __init__(self) -> None:
super().__init__()
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[Dict] = None,
) -> str:
"""
Get the complete URL for the Vertex AI Discovery Engine ranking API
"""
# Get project ID from environment or litellm config
# Try to get project ID from optional_params first (e.g., vertex_project parameter)
params = optional_params or {}
# Get credentials to extract project ID if needed
vertex_credentials = self.get_vertex_ai_credentials(params.copy())
vertex_project = self.get_vertex_ai_project(params.copy())
# Use _ensure_access_token to extract project_id from credentials
# This is the same method used in vertex embeddings
_, vertex_project = self._ensure_access_token(
credentials=vertex_credentials,
project_id=vertex_project,
custom_llm_provider="vertex_ai",
)
# Fallback to environment or litellm config
project_id = (
get_secret_str("VERTEXAI_PROJECT")
vertex_project
or get_secret_str("VERTEXAI_PROJECT")
or litellm.vertex_project
)
if not project_id:
raise ValueError(
"Vertex AI project ID is required. Please set 'VERTEXAI_PROJECT' or 'litellm.vertex_project'"
"Vertex AI project ID is required. Please set 'VERTEXAI_PROJECT', 'litellm.vertex_project', or pass 'vertex_project' parameter"
)
return f"https://discoveryengine.googleapis.com/v1/projects/{project_id}/locations/global/rankingConfigs/default_ranking_config:rank"
@ -49,13 +70,15 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase):
headers: dict,
model: str,
api_key: Optional[str] = None,
optional_params: Optional[Dict] = None,
) -> dict:
"""
Validate and set up authentication for Vertex AI Discovery Engine API
"""
# Get credentials and project info
vertex_credentials = self.get_vertex_ai_credentials({})
vertex_project = self.get_vertex_ai_project({})
# Get credentials and project info from optional_params (which contains vertex_credentials, etc.)
litellm_params = optional_params or {}
vertex_credentials = self.get_vertex_ai_credentials(litellm_params)
vertex_project = self.get_vertex_ai_project(litellm_params)
# Get access token using the base class method
access_token, project_id = self._ensure_access_token(
@ -218,10 +241,12 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase):
"""
Map Cohere rerank params to Vertex AI format
"""
return {
result = {
"query": query,
"documents": documents,
"top_n": top_n,
"return_documents": return_documents,
}
result.update(non_default_params)
return result

3
litellm/llms/vertex_ai/vertex_ai_partner_models/main.py
View file

@ -38,6 +38,7 @@ class PartnerModelPrefixes(str, Enum):
CLAUDE_PREFIX = "claude"
QWEN_PREFIX = "qwen"
GPT_OSS_PREFIX = "openai/gpt-oss-"
MINIMAX_PREFIX = "minimaxai/"
class VertexAIPartnerModels(VertexBase):
@ -62,6 +63,7 @@ class VertexAIPartnerModels(VertexBase):
or model.startswith(PartnerModelPrefixes.CLAUDE_PREFIX)
or model.startswith(PartnerModelPrefixes.QWEN_PREFIX)
or model.startswith(PartnerModelPrefixes.GPT_OSS_PREFIX)
or model.startswith(PartnerModelPrefixes.MINIMAX_PREFIX)
):
return True
return False
@ -73,6 +75,7 @@ class VertexAIPartnerModels(VertexBase):
PartnerModelPrefixes.DEEPSEEK_PREFIX,
PartnerModelPrefixes.QWEN_PREFIX,
PartnerModelPrefixes.GPT_OSS_PREFIX,
PartnerModelPrefixes.MINIMAX_PREFIX,
]
if any(provider in model for provider in OPENAI_LIKE_VERTEX_PROVIDERS):
return True

10
litellm/llms/vertex_ai/videos/__init__.py Normal file
View file

@ -0,0 +1,10 @@
"""
Vertex AI Video Generation Module
This module provides support for Vertex AI's Veo video generation API.
"""
from .transformation import VertexAIVideoConfig
__all__ = ["VertexAIVideoConfig"]

597
litellm/llms/vertex_ai/videos/transformation.py Normal file
View file

@ -0,0 +1,597 @@
"""
Vertex AI Video Generation Transformation
Handles transformation of requests/responses for Vertex AI's Veo video generation API.
Based on: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/veo-video-generation
"""
import base64
import time
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union
import httpx
from httpx._types import RequestFiles
from litellm.llms.base_llm.videos.transformation import BaseVideoConfig
from litellm.llms.vertex_ai.common_utils import (
_convert_vertex_datetime_to_openai_datetime,
)
from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
from litellm.types.router import GenericLiteLLMParams
from litellm.types.videos.main import VideoCreateOptionalRequestParams, VideoObject
from litellm.types.videos.utils import (
encode_video_id_with_provider,
extract_original_video_id,
)
from litellm.images.utils import ImageEditRequestUtils
from litellm.constants import DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
from litellm.llms.base_llm.chat.transformation import (
BaseLLMException as _BaseLLMException,
)
LiteLLMLoggingObj = _LiteLLMLoggingObj
BaseLLMException = _BaseLLMException
else:
LiteLLMLoggingObj = Any
BaseLLMException = Any
def _convert_image_to_vertex_format(image_file) -> Dict[str, str]:
"""
Convert image file to Vertex AI format with base64 encoding and MIME type.
Args:
image_file: File-like object opened in binary mode (e.g., open("path", "rb"))
Returns:
Dict with bytesBase64Encoded and mimeType
"""
mime_type = ImageEditRequestUtils.get_image_content_type(image_file)
if hasattr(image_file, "seek"):
image_file.seek(0)
image_bytes = image_file.read()
base64_encoded = base64.b64encode(image_bytes).decode("utf-8")
return {"bytesBase64Encoded": base64_encoded, "mimeType": mime_type}
class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
"""
Configuration class for Vertex AI (Veo) video generation.
Veo uses a long-running operation model:
1. POST to :predictLongRunning returns operation name
2. Poll operation using :fetchPredictOperation until done=true
3. Extract video data (base64) from response
"""
def __init__(self):
BaseVideoConfig.__init__(self)
VertexBase.__init__(self)
@staticmethod
def extract_model_from_operation_name(operation_name: str) -> Optional[str]:
"""
Extract the model name from a Vertex AI operation name.
Args:
operation_name: Operation name in format:
projects/PROJECT/locations/LOCATION/publishers/google/models/MODEL/operations/OPERATION_ID
Returns:
Model name (e.g., "veo-2.0-generate-001") or None if extraction fails
"""
parts = operation_name.split("/")
# Model is at index 7 in the operation name format
if len(parts) >= 8:
return parts[7]
return None
def get_supported_openai_params(self, model: str) -> list:
"""
Get the list of supported OpenAI parameters for Veo video generation.
Veo supports minimal parameters compared to OpenAI.
"""
return ["model", "prompt", "input_reference", "seconds", "size"]
def map_openai_params(
self,
video_create_optional_params: VideoCreateOptionalRequestParams,
model: str,
drop_params: bool,
) -> Dict[str, Any]:
"""
Map OpenAI-style parameters to Veo format.
Mappings:
- prompt → prompt (in instances)
- input_reference → image (in instances)
- size → aspectRatio (e.g., "1280x720" → "16:9")
- seconds → durationSeconds (defaults to 4 seconds if not provided)
"""
mapped_params: Dict[str, Any] = {}
# Map input_reference to image (will be processed in transform_video_create_request)
if "input_reference" in video_create_optional_params:
mapped_params["image"] = video_create_optional_params["input_reference"]
# Map size to aspectRatio
if "size" in video_create_optional_params:
size = video_create_optional_params["size"]
if size is not None:
aspect_ratio = self._convert_size_to_aspect_ratio(size)
if aspect_ratio:
mapped_params["aspectRatio"] = aspect_ratio
# Map seconds to durationSeconds, default to 4 seconds (matching OpenAI)
if "seconds" in video_create_optional_params:
seconds = video_create_optional_params["seconds"]
try:
duration = int(seconds) if isinstance(seconds, str) else seconds
if duration is not None:
mapped_params["durationSeconds"] = duration
except (ValueError, TypeError):
# If conversion fails, use default
pass
return mapped_params
def _convert_size_to_aspect_ratio(self, size: str) -> Optional[str]:
"""
Convert OpenAI size format to Veo aspectRatio format.
Supported aspect ratios: 9:16 (portrait), 16:9 (landscape)
"""
if not size:
return None
aspect_ratio_map = {
"1280x720": "16:9",
"1920x1080": "16:9",
"720x1280": "9:16",
"1080x1920": "9:16",
}
return aspect_ratio_map.get(size, "16:9")
def validate_environment(
self,
headers: Dict,
model: str,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
litellm_params: Optional[dict] = None,
**kwargs,
) -> Dict:
"""
Validate environment and return headers for Vertex AI OCR.
Vertex AI uses Bearer token authentication with access token from credentials.
"""
# Extract Vertex AI parameters using safe helpers from VertexBase
# Use safe_get_* methods that don't mutate litellm_params dict
litellm_params = litellm_params or {}
vertex_project = VertexBase.safe_get_vertex_ai_project(litellm_params=litellm_params)
vertex_credentials = VertexBase.safe_get_vertex_ai_credentials(litellm_params=litellm_params)
# Get access token from Vertex credentials
access_token, project_id = self.get_access_token(
credentials=vertex_credentials,
project_id=vertex_project,
)
headers = {
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json",
**headers,
}
return headers
def get_complete_url(
self,
model: str,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for Veo video generation.
Returns URL for :predictLongRunning endpoint:
https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/publishers/google/models/MODEL:predictLongRunning
"""
vertex_project = VertexBase.safe_get_vertex_ai_project(litellm_params)
vertex_location = VertexBase.safe_get_vertex_ai_location(litellm_params)
if not vertex_project:
raise ValueError(
"vertex_project is required for Vertex AI video generation. "
"Set it via environment variable VERTEXAI_PROJECT or pass as parameter."
)
# Default to us-central1 if no location specified
vertex_location = vertex_location or "us-central1"
# Extract model name (remove vertex_ai/ prefix if present)
model_name = model.replace("vertex_ai/", "")
# Construct the URL
if api_base:
base_url = api_base.rstrip("/")
else:
base_url = f"https://{vertex_location}-aiplatform.googleapis.com"
url = f"{base_url}/v1/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model_name}"
return url
def transform_video_create_request(
self,
model: str,
prompt: str,
api_base: str,
video_create_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict, RequestFiles, str]:
"""
Transform the video creation request for Veo API.
Veo expects:
{
"instances": [
{
"prompt": "A cat playing with a ball of yarn",
"image": {
"bytesBase64Encoded": "...",
"mimeType": "image/jpeg"
}
}
],
"parameters": {
"aspectRatio": "16:9",
"durationSeconds": 8
}
}
"""
# Build instance with prompt
instance_dict: Dict[str, Any] = {"prompt": prompt}
params_copy = video_create_optional_request_params.copy()
# Check if user wants to provide full instance dict
if "instances" in params_copy and isinstance(params_copy["instances"], dict):
# Replace/merge with user-provided instance
instance_dict.update(params_copy["instances"])
params_copy.pop("instances")
elif "image" in params_copy and params_copy["image"] is not None:
image_data = _convert_image_to_vertex_format(params_copy["image"])
instance_dict["image"] = image_data
params_copy.pop("image")
# Build request data directly (TypedDict doesn't have model_dump)
request_data: Dict[str, Any] = {"instances": [instance_dict]}
# Only add parameters if there are any
if params_copy:
request_data["parameters"] = params_copy
# Append :predictLongRunning endpoint to api_base
url = f"{api_base}:predictLongRunning"
# No files needed - everything is in JSON
return request_data, [], url
def transform_video_create_response(
self,
model: str,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
request_data: Optional[Dict] = None,
) -> VideoObject:
"""
Transform the Veo video creation response.
Veo returns:
{
"name": "projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL/operations/OPERATION_ID"
}
We return this as a VideoObject with:
- id: operation name (used for polling)
- status: "processing"
- usage: includes duration_seconds for cost calculation
"""
response_data = raw_response.json()
operation_name = response_data.get("name")
if not operation_name:
raise ValueError(f"No operation name in Veo response: {response_data}")
if custom_llm_provider:
video_id = encode_video_id_with_provider(
operation_name, custom_llm_provider, model
)
else:
video_id = operation_name
video_obj = VideoObject(
id=video_id,
object="video",
status="processing",
model=model
)
usage_data = {}
if request_data:
parameters = request_data.get("parameters", {})
duration = parameters.get("durationSeconds") or DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
if duration is not None:
try:
usage_data["duration_seconds"] = float(duration)
except (ValueError, TypeError):
pass
video_obj.usage = usage_data
return video_obj
def transform_video_status_retrieve_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Transform the video status retrieve request for Veo API.
Veo polls operations using :fetchPredictOperation endpoint with POST request.
"""
operation_name = extract_original_video_id(video_id)
model = self.extract_model_from_operation_name(operation_name)
if not model:
raise ValueError(
f"Invalid operation name format: {operation_name}. "
"Expected format: projects/PROJECT/locations/LOCATION/publishers/google/models/MODEL/operations/OPERATION_ID"
)
# Construct the full URL including model ID
# URL format: https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/publishers/google/models/MODEL:fetchPredictOperation
# Strip trailing slashes from api_base and append model
url = f"{api_base.rstrip('/')}/{model}:fetchPredictOperation"
# Request body contains the operation name
params = {"operationName": operation_name}
return url, params
def transform_video_status_retrieve_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""
Transform the Veo operation status response.
Veo returns:
{
"name": "projects/.../operations/OPERATION_ID",
"done": false # or true when complete
}
When done=true:
{
"name": "projects/.../operations/OPERATION_ID",
"done": true,
"response": {
"@type": "type.googleapis.com/cloud.ai.large_models.vision.GenerateVideoResponse",
"raiMediaFilteredCount": 0,
"videos": [
{
"bytesBase64Encoded": "...",
"mimeType": "video/mp4"
}
]
}
}
"""
response_data = raw_response.json()
operation_name = response_data.get("name", "")
is_done = response_data.get("done", False)
error_data = response_data.get("error")
# Extract model from operation name
model = self.extract_model_from_operation_name(operation_name)
if custom_llm_provider:
video_id = encode_video_id_with_provider(
operation_name, custom_llm_provider, model
)
else:
video_id = operation_name
# Convert createTime to Unix timestamp
create_time_str = response_data.get("metadata", {}).get("createTime")
if create_time_str:
try:
created_at = _convert_vertex_datetime_to_openai_datetime(
create_time_str
)
except Exception:
created_at = int(time.time())
else:
created_at = int(time.time())
if error_data:
status = "failed"
elif is_done:
status = "completed"
else:
status = "processing"
video_obj = VideoObject(
id=video_id,
object="video",
status=status,
model=model,
created_at=created_at,
error=error_data,
)
return video_obj
def transform_video_content_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Transform the video content request for Veo API.
For Veo, we need to:
1. Poll the operation status to ensure it's complete
2. Extract the base64 video data from the response
3. Return it for decoding
Since we need to make an HTTP call here, we'll use the same fetchPredictOperation
approach as status retrieval.
"""
return self.transform_video_status_retrieve_request(video_id, api_base, litellm_params, headers)
def transform_video_content_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
) -> bytes:
"""
Transform the Veo video content download response.
Extracts the base64 encoded video from the response and decodes it to bytes.
"""
response_data = raw_response.json()
if not response_data.get("done", False):
raise ValueError(
"Video generation is not complete yet. "
"Please check status with video_status() before downloading."
)
try:
video_response = response_data.get("response", {})
videos = video_response.get("videos", [])
if not videos or len(videos) == 0:
raise ValueError("No video data found in completed operation")
# Get the first video
video_data = videos[0]
base64_encoded = video_data.get("bytesBase64Encoded")
if not base64_encoded:
raise ValueError("No base64 encoded video data found")
# Decode base64 to bytes
video_bytes = base64.b64decode(base64_encoded)
return video_bytes
except (KeyError, IndexError) as e:
raise ValueError(f"Failed to extract video data: {e}")
def transform_video_remix_request(
self,
video_id: str,
prompt: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Video remix is not supported by Veo API.
"""
raise NotImplementedError(
"Video remix is not supported by Vertex AI Veo. "
"Please use video_generation() to create new videos."
)
def transform_video_remix_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> VideoObject:
"""Video remix is not supported."""
raise NotImplementedError("Video remix is not supported by Vertex AI Veo.")
def transform_video_list_request(
self,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
after: Optional[str] = None,
limit: Optional[int] = None,
order: Optional[str] = None,
extra_query: Optional[Dict[str, Any]] = None,
) -> Tuple[str, Dict]:
"""
Video list is not supported by Veo API.
"""
raise NotImplementedError(
"Video list is not supported by Vertex AI Veo. "
"Use the operations endpoint directly if you need to list operations."
)
def transform_video_list_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Optional[str] = None,
) -> Dict[str, str]:
"""Video list is not supported."""
raise NotImplementedError("Video list is not supported by Vertex AI Veo.")
def transform_video_delete_request(
self,
video_id: str,
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[str, Dict]:
"""
Video delete is not supported by Veo API.
"""
raise NotImplementedError(
"Video delete is not supported by Vertex AI Veo. "
"Videos are automatically cleaned up by Google."
)
def transform_video_delete_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
) -> VideoObject:
"""Video delete is not supported."""
raise NotImplementedError("Video delete is not supported by Vertex AI Veo.")
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
) -> BaseLLMException:
from litellm.llms.vertex_ai.common_utils import VertexAIError
return VertexAIError(
status_code=status_code,
message=error_message,
headers=headers,
)

5
litellm/llms/xai/responses/__init__.py
View file

@ -1,5 +0,0 @@
# XAI Responses API
from .transformation import XAIResponsesAPIConfig
__all__ = ["XAIResponsesAPIConfig"]

2
litellm/llms/xai/responses/transformation.py
View file

@ -85,7 +85,7 @@ class XAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
# XAI supports code_interpreter but doesn't use the container field
# Keep only the type field
verbose_logger.debug(
f"XAI: Transforming code_interpreter tool, removing container field"
"XAI: Transforming code_interpreter tool, removing container field"
)
transformed_tools.append({"type": "code_interpreter"})
else:

58
litellm/main.py
View file

@ -65,7 +65,10 @@ from litellm.constants import (
)
from litellm.exceptions import LiteLLMUnknownProvider
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.audio_utils.utils import get_audio_file_for_health_check
from litellm.litellm_core_utils.audio_utils.utils import (
calculate_request_duration,
get_audio_file_for_health_check,
)
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.get_provider_specific_headers import (
ProviderSpecificHeaderUtils,
@ -2033,36 +2036,11 @@ def completion( # type: ignore # noqa: PLR0915
logging.post_call(
input=messages, api_key=api_key, original_response=response
)
elif custom_llm_provider == "perplexity":
response = base_llm_http_handler.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
api_key=api_key,
api_base=api_base,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
shared_session=shared_session,
timeout=timeout,
client=client,
custom_llm_provider=custom_llm_provider,
encoding=encoding,
stream=stream,
provider_config=provider_config,
)
## LOGGING - Call after response has been processed by transform_response
logging.post_call(
input=messages, api_key=api_key, original_response=response
)
elif (
model in litellm.open_ai_chat_completion_models
or custom_llm_provider == "custom_openai"
or custom_llm_provider == "deepinfra"
or custom_llm_provider == "perplexity"
or custom_llm_provider == "nvidia_nim"
or custom_llm_provider == "cerebras"
or custom_llm_provider == "baseten"
@ -5431,6 +5409,7 @@ async def atranscription(*args, **kwargs) -> TranscriptionResponse:
model = args[0] if len(args) > 0 else kwargs["model"]
### PASS ARGS TO Image Generation ###
kwargs["atranscription"] = True
file = kwargs.get("file", None)
custom_llm_provider = None
try:
# Use a partial function to pass your keyword arguments
@ -5459,6 +5438,20 @@ async def atranscription(*args, **kwargs) -> TranscriptionResponse:
raise ValueError(
f"Invalid response from transcription provider, expected TranscriptionResponse, but got {type(response)}"
)
# Calculate and add duration if response is missing it
if (
response is not None
and not isinstance(response, Coroutine)
and file is not None
):
# Check if response is missing duration
existing_duration = getattr(response, "duration", None)
if existing_duration is None:
calculated_duration = calculate_request_duration(file)
if calculated_duration is not None:
setattr(response, "duration", calculated_duration)
return response
except Exception as e:
custom_llm_provider = custom_llm_provider or "openai"
@ -5669,6 +5662,16 @@ def transcription(
headers={},
provider_config=provider_config,
)
# Calculate and add duration if response is missing it
if response is not None and not isinstance(response, Coroutine):
# Check if response is missing duration
existing_duration = getattr(response, "duration", None)
if existing_duration is None:
calculated_duration = calculate_request_duration(file)
if calculated_duration is not None:
setattr(response, "duration", calculated_duration)
if response is None:
raise ValueError("Unmapped provider passed in. Unable to get the response.")
return response
@ -6026,6 +6029,7 @@ async def ahealth_check(
"audio_speech",
"audio_transcription",
"image_generation",
"video_generation",
"batch",
"rerank",
"realtime",

Some files were not shown because too many files have changed in this diff Show more