scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-07 08:26:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36

Merge remote-tracking branch 'origin' into litellm_sso_role_mapping

Browse source
This commit is contained in:
yuneng-jiang 2025-12-19 11:03:30 -08:00
commit 02355f602c
373 changed files with 26159 additions and 2449 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
.circleci/config.yml
View file

@ -657,7 +657,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2108,7 +2108,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2250,7 +2250,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2390,7 +2390,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2551,7 +2551,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2664,7 +2664,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -2800,7 +2800,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -3032,7 +3032,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14
@ -3549,7 +3549,7 @@ jobs:
docker run -d \
--name postgres-db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_PASSWORD=test-postgres \
-e POSTGRES_DB=circle_test \
-p 5432:5432 \
postgres:14

84
.gitguardian.yaml Normal file
View file

@ -0,0 +1,84 @@
version: 2
secret:
# Exclude files and paths by globbing
ignored_paths:
- "**/*.whl"
- "**/*.pyc"
- "**/__pycache__/**"
- "**/node_modules/**"
- "**/dist/**"
- "**/build/**"
- "**/.git/**"
- "**/venv/**"
- "**/.venv/**"
# Large data/metadata files that don't need scanning
- "**/model_prices_and_context_window*.json"
- "**/*_metadata/*.txt"
- "**/tokenizers/*.json"
- "**/tokenizers/*"
- "miniconda.sh"
# Build outputs and static assets
- "litellm/proxy/_experimental/out/**"
- "ui/litellm-dashboard/public/**"
- "**/swagger/*.js"
- "**/*.woff"
- "**/*.woff2"
- "**/*.avif"
- "**/*.webp"
# Test data files
- "**/tests/**/data_map.txt"
- "tests/**/*.txt"
# Documentation and other non-code files
- "docs/**"
- "**/*.md"
- "**/*.lock"
- "poetry.lock"
- "package-lock.json"
# Ignore security incidents with the SHA256 of the occurrence (false positives)
ignored_matches:
# === Current detected false positives (SHA-based) ===
# gcs_pub_sub_body - folder name, not a password
- name: GCS pub/sub test folder name
match: 75f377c456eede69e5f6e47399ccee6016a2a93cc5dd11db09cc5b1359ae569a
# os.environ/APORIA_API_KEY_1 - environment variable reference
- name: Environment variable reference APORIA_API_KEY_1
match: e2ddeb8b88eca97a402559a2be2117764e11c074d86159ef9ad2375dea188094
# os.environ/APORIA_API_KEY_2 - environment variable reference
- name: Environment variable reference APORIA_API_KEY_2
match: 09aa39a29e050b86603aa55138af1ff08fb86a4582aa965c1bd0672e1575e052
# oidc/circleci_v2/ - test authentication path, not a secret
- name: OIDC CircleCI test path
match: feb3475e1f89a65b7b7815ac4ec597e18a9ec1847742ad445c36ca617b536e15
# text-davinci-003 - OpenAI model identifier, not a secret
- name: OpenAI model identifier text-davinci-003
match: c489000cf6c7600cee0eefb80ad0965f82921cfb47ece880930eb7e7635cf1f1
# === Preventive patterns for test keys (pattern-based) ===
# Test API keys (124 instances across 45 files)
- name: Test API keys with sk-test prefix
match: sk-test-
# Mock API keys
- name: Mock API keys with sk-mock prefix
match: sk-mock-
# Fake API keys
- name: Fake API keys with sk-fake prefix
match: sk-fake-
# Generic test API key patterns
- name: Test API key patterns
match: test-api-key

2
.github/workflows/locustfile.py vendored
View file

@ -8,7 +8,7 @@ class MyUser(HttpUser):
def chat_completion(self):
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer sk-8N1tLOOyH8TIxwOLahhIVg",
"Authorization": "Bearer sk-test-load-test-key-123",
# Include any additional headers you may need for authentication, etc.
}

4
.github/workflows/publish-migrations.yml vendored
View file

@ -20,7 +20,7 @@ jobs:
env:
POSTGRES_DB: temp_db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_PASSWORD: test-postgres
ports:
- 5432:5432
options: >-
@ -35,7 +35,7 @@ jobs:
env:
POSTGRES_DB: shadow_db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_PASSWORD: test-postgres
ports:
- 5433:5432
options: >-

428
README.md
View file

@ -2,16 +2,16 @@
🚅 LiteLLM
</h1>
<p align="center">
<p align="center">Call 100+ LLMs in OpenAI format. [Bedrock, Azure, OpenAI, VertexAI, Anthropic, Groq, etc.]
</p>
<p align="center">
<a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render"></a>
<a href="https://railway.app/template/HLP0Ub?referralCode=jch2ME">
<img src="https://railway.app/button.svg" alt="Deploy on Railway">
</a>
</p>
<p align="center">Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]
<br>
</p>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (LLM Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://docs.litellm.ai/docs/enterprise"target="_blank">Enterprise Tier</a></h4>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://docs.litellm.ai/docs/enterprise"target="_blank">Enterprise Tier</a></h4>
<h4 align="center">
<a href="https://pypi.org/project/litellm/" target="_blank">
<img src="https://img.shields.io/pypi/v/litellm.svg" alt="PyPI Version">
@ -30,27 +30,17 @@
</a>
</h4>
LiteLLM manages:
<img width="2688" height="1600" alt="Group 7154 (1)" src="https://github.com/user-attachments/assets/c5ee0412-6fb5-4fb6-ab5b-bafae4209ca6" />
- Translate inputs to provider's `completion`, `embedding`, and `image_generation` endpoints
- [Consistent output](https://docs.litellm.ai/docs/completion/output), text responses will always be available at `['choices'][0]['message']['content']`
- Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - [Router](https://docs.litellm.ai/docs/routing)
- Set Budgets & Rate limits per project, api key, model [LiteLLM Proxy Server (LLM Gateway)](https://docs.litellm.ai/docs/simple_proxy)
LiteLLM Performance: **8ms P95 latency** at 1k RPS (See benchmarks [here](https://docs.litellm.ai/docs/benchmarks))
## Use LiteLLM for
[**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://github.com/BerriAI/litellm?tab=readme-ov-file#litellm-proxy-server-llm-gateway---docs) <br>
[**Jump to Supported LLM Providers**](https://docs.litellm.ai/docs/providers)
<details open>
<summary><b>LLMs</b> - Call 100+ LLMs (Python SDK + AI Gateway)</summary>
🚨 **Stable Release:** Use docker images with the `-stable` tag. These have undergone 12 hour load tests, before being published. [More information about the release cycle here](https://docs.litellm.ai/docs/proxy/release_cycle)
[**All Supported Endpoints**](https://docs.litellm.ai/docs/supported_endpoints) - `/chat/completions`, `/responses`, `/embeddings`, `/images`, `/audio`, `/batches`, `/rerank`, `/a2a`, `/messages` and more.
Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+).
# Usage ([**Docs**](https://docs.litellm.ai/docs/))
<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Getting_Started.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
### Python SDK
```shell
pip install litellm
@ -60,249 +50,214 @@ pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# openai call
response = completion(model="openai/gpt-4o", messages=messages)
# anthropic call
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=messages)
print(response)
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
```
### Response (OpenAI Format)
### AI Gateway (Proxy Server)
```json
{
"id": "chatcmpl-1214900a-6cdd-4148-b663-b5e2f642b4de",
"created": 1751494488,
"model": "claude-sonnet-4-20250514",
"object": "chat.completion",
"system_fingerprint": null,
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Hello! I'm doing well, thank you for asking. I'm here and ready to help with whatever you'd like to discuss or work on. How are you doing today?",
"role": "assistant",
"tool_calls": null,
"function_call": null
}
}
],
"usage": {
"completion_tokens": 39,
"prompt_tokens": 13,
"total_tokens": 52,
"completion_tokens_details": null,
"prompt_tokens_details": {
"audio_tokens": null,
"cached_tokens": 0
},
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0
}
}
```
> **Note:** LiteLLM also supports the [Responses API](https://docs.litellm.ai/docs/response_api) (`litellm.responses()`)
Call any model supported by a provider, with `model=<provider_name>/<model_name>`. There might be provider-specific details here, so refer to [provider docs for more information](https://docs.litellm.ai/docs/providers)
## Async ([Docs](https://docs.litellm.ai/docs/completion/stream#async-completion))
```python
from litellm import acompletion
import asyncio
async def test_get_response():
user_message = "Hello, how are you?"
messages = [{"content": user_message, "role": "user"}]
response = await acompletion(model="openai/gpt-4o", messages=messages)
return response
response = asyncio.run(test_get_response())
print(response)
```
## Streaming ([Docs](https://docs.litellm.ai/docs/completion/stream))
LiteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
```python
from litellm import completion
messages = [{"content": "Hello, how are you?", "role": "user"}]
# gpt-4o
response = completion(model="openai/gpt-4o", messages=messages, stream=True)
for part in response:
print(part.choices[0].delta.content or "")
# claude sonnet 4
response = completion('anthropic/claude-sonnet-4-20250514', messages, stream=True)
for part in response:
print(part)
```
### Response chunk (OpenAI Format)
```json
{
"id": "chatcmpl-fe575c37-5004-4926-ae5e-bfbc31f356ca",
"created": 1751494808,
"model": "claude-sonnet-4-20250514",
"object": "chat.completion.chunk",
"system_fingerprint": null,
"choices": [
{
"finish_reason": null,
"index": 0,
"delta": {
"provider_specific_fields": null,
"content": "Hello",
"role": "assistant",
"function_call": null,
"tool_calls": null,
"audio": null
},
"logprobs": null
}
],
"provider_specific_fields": null,
"stream_options": null,
"citations": null
}
```
## Logging Observability ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
LiteLLM exposes pre defined callbacks to send data to Lunary, MLflow, Langfuse, DynamoDB, s3 Buckets, Helicone, Promptlayer, Traceloop, Athina, Slack
```python
from litellm import completion
## set env variables for logging tools (when using MLflow, no API key set up is required)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["HELICONE_API_KEY"] = "your-helicone-auth-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"
os.environ["OPENAI_API_KEY"] = "your-openai-key"
# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "athina", "helicone"] # log input/output to lunary, langfuse, supabase, athina, helicone etc
#openai call
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
```
# LiteLLM Proxy Server (LLM Gateway) - ([Docs](https://docs.litellm.ai/docs/simple_proxy))
Track spend + Load Balance across multiple projects
[Hosted Proxy](https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy)
The proxy provides:
1. [Hooks for auth](https://docs.litellm.ai/docs/proxy/virtual_keys#custom-auth)
2. [Hooks for logging](https://docs.litellm.ai/docs/proxy/logging#step-1---create-your-custom-litellm-callback-class)
3. [Cost tracking](https://docs.litellm.ai/docs/proxy/virtual_keys#tracking-spend)
4. [Rate Limiting](https://docs.litellm.ai/docs/proxy/users#set-rate-limits)
## 📖 Proxy Endpoints - [Swagger Docs](https://litellm-api.up.railway.app/)
## Quick Start Proxy - CLI
[**Getting Started - E2E Tutorial**](https://docs.litellm.ai/docs/proxy/docker_quick_start) - Setup virtual keys, make your first request
```shell
pip install 'litellm[proxy]'
litellm --model gpt-4o
```
### Step 1: Start litellm proxy
```shell
$ litellm --model huggingface/bigcode/starcoder
#INFO: Proxy running on http://0.0.0.0:4000
```
### Step 2: Make ChatCompletions Request to Proxy
> [!IMPORTANT]
> 💡 [Use LiteLLM Proxy with Langchain (Python, JS), OpenAI SDK (Python, JS) Anthropic SDK, Mistral SDK, LlamaIndex, Instructor, Curl](https://docs.litellm.ai/docs/proxy/user_keys)
```python
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
import openai
print(response)
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
```
## Proxy Key Management ([Docs](https://docs.litellm.ai/docs/proxy/virtual_keys))
[**Docs: LLM Providers**](https://docs.litellm.ai/docs/providers)
Connect the proxy with a Postgres DB to create proxy keys
</details>
<details>
<summary><b>Agents</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>
[**Supported Providers**](https://docs.litellm.ai/docs/a2a#add-a2a-agents) - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
### Python SDK - A2A Protocol
```python
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
```
### AI Gateway (Proxy Server)
**Step 1.** [Add your Agent to the AI Gateway](https://docs.litellm.ai/docs/a2a#adding-your-agent)
**Step 2.** Call Agent via A2A SDK
```python
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"} # LiteLLM Virtual Key
async with httpx.AsyncClient(headers=headers) as httpx_client:
resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
```
[**Docs: A2A Agent Gateway**](https://docs.litellm.ai/docs/a2a)
</details>
<details>
<summary><b>MCP Tools</b> - Connect MCP servers to any LLM (Python SDK + AI Gateway)</summary>
### Python SDK - MCP Bridge
```python
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
```
### AI Gateway - MCP Gateway
**Step 1.** [Add your MCP Server to the AI Gateway](https://docs.litellm.ai/docs/mcp#adding-your-mcp)
**Step 2.** Call MCP tools via `/chat/completions`
```bash
# Get the code
git clone https://github.com/BerriAI/litellm
# Go to folder
cd litellm
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env
# Start
docker compose up
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
```
### Use with Cursor IDE
UI on `/ui` on your proxy server
![ui_3](https://github.com/BerriAI/litellm/assets/29436595/47c97d5e-b9be-4839-b28c-43d7f4f10033)
Set budgets and rate limits across multiple projects
`POST /key/generate`
### Request
```shell
curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'
```
### Expected Response
```shell
```json
{
"key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
"expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
```
[**Docs: MCP Gateway**](https://docs.litellm.ai/docs/mcp)
</details>
---
## How to use LiteLLM
You can use LiteLLM through either the Proxy Server or Python SDK. Both gives you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
<table style={{width: '100%', tableLayout: 'fixed'}}>
<thead>
<tr>
<th style={{width: '14%'}}></th>
<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/simple_proxy">LiteLLM AI Gateway</a></strong></th>
<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/">LiteLLM Python SDK</a></strong></th>
</tr>
</thead>
<tbody>
<tr>
<td style={{width: '14%'}}><strong>Use Case</strong></td>
<td style={{width: '43%'}}>Central service (LLM Gateway) to access multiple LLMs</td>
<td style={{width: '43%'}}>Use LiteLLM directly in your Python code</td>
</tr>
<tr>
<td style={{width: '14%'}}><strong>Who Uses It?</strong></td>
<td style={{width: '43%'}}>Gen AI Enablement / ML Platform Teams</td>
<td style={{width: '43%'}}>Developers building LLM projects</td>
</tr>
<tr>
<td style={{width: '14%'}}><strong>Key Features</strong></td>
<td style={{width: '43%'}}>Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management</td>
<td style={{width: '43%'}}>Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - <a href="https://docs.litellm.ai/docs/routing">Router</a>, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)</td>
</tr>
</tbody>
</table>
LiteLLM Performance: **8ms P95 latency** at 1k RPS (See benchmarks [here](https://docs.litellm.ai/docs/benchmarks))
[**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://docs.litellm.ai/docs/simple_proxy) <br>
[**Jump to Supported LLM Providers**](https://docs.litellm.ai/docs/providers)
**Stable Release:** Use docker images with the `-stable` tag. These have undergone 12 hour load tests, before being published. [More information about the release cycle here](https://docs.litellm.ai/docs/proxy/release_cycle)
Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+).
## Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers))
| Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` |
@ -311,6 +266,7 @@ curl 'http://0.0.0.0:4000/key/generate' \
| [AI21 (`ai21`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | |
| [AI21 Chat (`ai21_chat`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | |
| [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | | | | | | | |
| [Amazon Nova](https://docs.litellm.ai/docs/providers/amazon_nova) | ✅ | ✅ | ✅ | | | | | | | |
| [Anthropic (`anthropic`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| [Anthropic Text (`anthropic_text`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| [Anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | ✅ | ✅ | | | | | | | |

40
ci_cd/TEST_KEY_PATTERNS.md Normal file
View file

@ -0,0 +1,40 @@
# Test Key Patterns Standard
Standard patterns for test/mock keys and credentials in the LiteLLM codebase to avoid triggering secret detection.
## How GitGuardian Works
GitGuardian uses **machine learning and entropy analysis**, not just pattern matching:
- **Low entropy** values (like `sk-1234`, `postgres`) are automatically ignored
- **High entropy** values (realistic-looking secrets) trigger detection
- **Context-aware** detection understands code syntax like `os.environ["KEY"]`
## Recommended Test Key Patterns
### Option 1: Low Entropy Values (Simplest)
These won't trigger GitGuardian's ML detector:
```python
api_key = "sk-1234"
api_key = "sk-12345"
database_password = "postgres"
token = "test123"
```
### Option 2: High Entropy with Test Prefixes
If you need realistic-looking test keys with high entropy, use these prefixes:
```python
api_key = "sk-test-abc123def456ghi789..." # OpenAI-style test key
api_key = "sk-mock-1234567890abcdef1234..." # Mock key
api_key = "sk-fake-xyz789uvw456rst123..." # Fake key
token = "test-api-key-with-high-entropy"
```
## Configured Ignore Patterns
These patterns are in `.gitguardian.yaml` for high-entropy test keys:
- `sk-test-*` - OpenAI-style test keys
- `sk-mock-*` - Mock API keys
- `sk-fake-*` - Fake API keys
- `test-api-key` - Generic test tokens

53
ci_cd/security_scans.sh
View file

@ -26,6 +26,56 @@ install_grype() {
echo "Grype installed successfully"
}
# Function to install ggshield
install_ggshield() {
echo "Installing ggshield..."
pip3 install --upgrade pip
pip3 install ggshield
echo "ggshield installed successfully"
}
# Function to run secret detection scans
run_secret_detection() {
echo "Running secret detection scans..."
if ! command -v ggshield &> /dev/null; then
install_ggshield
fi
# Check if GITGUARDIAN_API_KEY is set (required for CI/CD)
if [ -z "$GITGUARDIAN_API_KEY" ]; then
echo "Warning: GITGUARDIAN_API_KEY environment variable is not set."
echo "ggshield requires a GitGuardian API key to scan for secrets."
echo "Please set GITGUARDIAN_API_KEY in your CI/CD environment variables."
exit 1
fi
echo "Scanning codebase for secrets..."
echo "Note: Large codebases may take several minutes due to API rate limits (50 requests/minute on free plan)"
echo "ggshield will automatically handle rate limits and retry as needed."
echo "Binary files, cache files, and build artifacts are excluded via .gitguardian.yaml"
# Use --recursive for directory scanning and auto-confirm if prompted
# .gitguardian.yaml will automatically exclude binary files, wheel files, etc.
# GITGUARDIAN_API_KEY environment variable will be used for authentication
echo y | ggshield secret scan path . --recursive || {
echo ""
echo "=========================================="
echo "ERROR: Secret Detection Failed"
echo "=========================================="
echo "ggshield has detected secrets in the codebase."
echo "Please review discovered secrets above, revoke any actively used secrets"
echo "from underlying systems and make changes to inject secrets dynamically at runtime."
echo ""
echo "For more information, see: https://docs.gitguardian.com/secrets-detection/"
echo "=========================================="
echo ""
exit 1
}
echo "Secret detection scans completed successfully"
}
# Function to run Trivy scans
run_trivy_scans() {
echo "Running Trivy scans..."
@ -158,6 +208,9 @@ main() {
install_trivy
install_grype
echo "Running secret detection scans..."
run_secret_detection
echo "Running filesystem vulnerability scans..."
run_trivy_scans

2
cookbook/LiteLLM_PromptLayer.ipynb vendored
View file

@ -39,7 +39,7 @@
"import os\n",
"os.environ['OPENAI_API_KEY'] = \"\"\n",
"os.environ['REPLICATE_API_TOKEN'] = \"\"\n",
"os.environ['PROMPTLAYER_API_KEY'] = \"pl_4ea2bb00a4dca1b8a70cebf2e9e11564\"\n",
"os.environ['PROMPTLAYER_API_KEY'] = \"test-promptlayer-key-123\"\n",
"\n",
"# Set Promptlayer as a success callback\n",
"litellm.success_callback =['promptlayer']\n",

278
cookbook/Migrating_to_LiteLLM_Proxy_from_OpenAI_Azure_OpenAI.ipynb vendored
View file

@ -1,21 +1,10 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "kccfk0mHZ4Ad"
},
"source": [
"# Migrating to LiteLLM Proxy from OpenAI/Azure OpenAI\n",
"\n",
@ -32,29 +21,26 @@
"To pass provider-specific args, [go here](https://docs.litellm.ai/docs/completion/provider_specific_params#proxy-usage)\n",
"\n",
"To drop unsupported params (E.g. frequency_penalty for bedrock with librechat), [go here](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage)\n"
],
"metadata": {
"id": "kccfk0mHZ4Ad"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nmSClzCPaGH6"
},
"source": [
"## /chat/completion\n",
"\n"
],
"metadata": {
"id": "nmSClzCPaGH6"
}
]
},
{
"cell_type": "markdown",
"source": [
"### OpenAI Python SDK"
],
"metadata": {
"id": "_vqcjwOVaKpO"
}
},
"source": [
"### OpenAI Python SDK"
]
},
{
"cell_type": "code",
@ -94,15 +80,20 @@
},
{
"cell_type": "markdown",
"source": [
"## Function Calling"
],
"metadata": {
"id": "AqkyKk9Scxgj"
}
},
"source": [
"## Function Calling"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wDg10VqLczE1"
},
"outputs": [],
"source": [
"from openai import OpenAI\n",
"client = OpenAI(\n",
@ -139,24 +130,24 @@
")\n",
"\n",
"print(completion)\n"
],
"metadata": {
"id": "wDg10VqLczE1"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Azure OpenAI Python SDK"
],
"metadata": {
"id": "YYoxLloSaNWW"
}
},
"source": [
"### Azure OpenAI Python SDK"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yA1XcgowaSRy"
},
"outputs": [],
"source": [
"import openai\n",
"client = openai.AzureOpenAI(\n",
@ -184,24 +175,24 @@
")\n",
"\n",
"print(response)"
],
"metadata": {
"id": "yA1XcgowaSRy"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Langchain Python"
],
"metadata": {
"id": "yl9qhDvnaTpL"
}
},
"source": [
"### Langchain Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5MUZgSquaW5t"
},
"outputs": [],
"source": [
"from langchain.chat_models import ChatOpenAI\n",
"from langchain.prompts.chat import (\n",
@ -239,24 +230,22 @@
"response = chat(messages)\n",
"\n",
"print(response)"
],
"metadata": {
"id": "5MUZgSquaW5t"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Curl"
],
"metadata": {
"id": "B9eMgnULbRaz"
}
},
"source": [
"### Curl"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VWCCk5PFcmhS"
},
"source": [
"\n",
"\n",
@ -280,22 +269,24 @@
"}'\n",
"```\n",
"\n"
],
"metadata": {
"id": "VWCCk5PFcmhS"
}
]
},
{
"cell_type": "markdown",
"source": [
"### LlamaIndex"
],
"metadata": {
"id": "drBAm2e1b6xe"
}
},
"source": [
"### LlamaIndex"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d0bZcv8fb9mL"
},
"outputs": [],
"source": [
"import os, dotenv\n",
"\n",
@ -326,24 +317,24 @@
"query_engine = index.as_query_engine()\n",
"response = query_engine.query(\"What did the author do growing up?\")\n",
"print(response)\n"
],
"metadata": {
"id": "d0bZcv8fb9mL"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Langchain JS"
],
"metadata": {
"id": "xypvNdHnb-Yy"
}
},
"source": [
"### Langchain JS"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "R55mK2vCcBN2"
},
"outputs": [],
"source": [
"import { ChatOpenAI } from \"@langchain/openai\";\n",
"\n",
@ -359,24 +350,24 @@
"const message = await model.invoke(\"Hi there!\");\n",
"\n",
"console.log(message);\n"
],
"metadata": {
"id": "R55mK2vCcBN2"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### OpenAI JS"
],
"metadata": {
"id": "nC4bLifCcCiW"
}
},
"source": [
"### OpenAI JS"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MICH8kIMcFpg"
},
"outputs": [],
"source": [
"const { OpenAI } = require('openai');\n",
"\n",
@ -398,24 +389,24 @@
"}\n",
"\n",
"main();\n"
],
"metadata": {
"id": "MICH8kIMcFpg"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Anthropic SDK"
],
"metadata": {
"id": "D1Q07pEAcGTb"
}
},
"source": [
"### Anthropic SDK"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qBjFcAvgcI3t"
},
"outputs": [],
"source": [
"import os\n",
"\n",
@ -423,7 +414,7 @@
"\n",
"client = Anthropic(\n",
" base_url=\"http://localhost:4000\", # proxy endpoint\n",
" api_key=\"sk-s4xN1IiLTCytwtZFJaYQrA\", # litellm proxy virtual key\n",
" api_key=\"sk-test-proxy-key-123\", # litellm proxy virtual key (example)\n",
")\n",
"\n",
"message = client.messages.create(\n",
@ -437,33 +428,33 @@
" model=\"claude-3-opus-20240229\",\n",
")\n",
"print(message.content)"
],
"metadata": {
"id": "qBjFcAvgcI3t"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"## /embeddings"
],
"metadata": {
"id": "dFAR4AJGcONI"
}
},
"source": [
"## /embeddings"
]
},
{
"cell_type": "markdown",
"source": [
"### OpenAI Python SDK"
],
"metadata": {
"id": "lgNoM281cRzR"
}
},
"source": [
"### OpenAI Python SDK"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NY3DJhPfcQhA"
},
"outputs": [],
"source": [
"import openai\n",
"from openai import OpenAI\n",
@ -478,24 +469,24 @@
")\n",
"\n",
"print(response)\n"
],
"metadata": {
"id": "NY3DJhPfcQhA"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Langchain Embeddings"
],
"metadata": {
"id": "hmbg-DW6cUZs"
}
},
"source": [
"### Langchain Embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lX2S8Nl1cWVP"
},
"outputs": [],
"source": [
"from langchain.embeddings import OpenAIEmbeddings\n",
"\n",
@ -526,24 +517,22 @@
"\n",
"print(f\"TITAN EMBEDDINGS\")\n",
"print(query_result[:5])"
],
"metadata": {
"id": "lX2S8Nl1cWVP"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"### Curl Request"
],
"metadata": {
"id": "oqGbWBCQcYfd"
}
},
"source": [
"### Curl Request"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7rkIMV9LcdwQ"
},
"source": [
"\n",
"\n",
@ -556,10 +545,21 @@
" }'\n",
"```\n",
"\n"
],
"metadata": {
"id": "7rkIMV9LcdwQ"
}
]
}
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

4
docker/Dockerfile.alpine
View file

@ -34,8 +34,8 @@ RUN pip wheel --no-cache-dir --wheel-dir=/wheels/ -r requirements.txt
# Runtime stage
FROM $LITELLM_RUNTIME_IMAGE AS runtime
# Update dependencies and clean up
RUN apk upgrade --no-cache
# Update dependencies and clean up, install libsndfile for audio processing
RUN apk upgrade --no-cache && apk add --no-cache libsndfile
WORKDIR /app

2
docs/my-website/blog/anthropic_opus_4_5_and_advanced_features/index.md
View file

@ -6,7 +6,7 @@ authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1765411200&v=beta&t=c8396f--_lH6Fb_pVvx_jGholPfcl0bvwmNynbNdnII
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/

2
docs/my-website/blog/gemini_3/index.md
View file

@ -6,7 +6,7 @@ authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1765411200&v=beta&t=c8396f--_lH6Fb_pVvx_jGholPfcl0bvwmNynbNdnII
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/

254
docs/my-website/blog/gemini_3_flash/index.md Normal file
View file

@ -0,0 +1,254 @@
---
slug: gemini_3_flash
title: "DAY 0 Support: Gemini 3 Flash on LiteLLM"
date: 2025-12-17T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Gemini 3 Flash Day 0 Support
LiteLLM now supports `gemini-3-flash-preview` and all the new API changes along with it.
:::note
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
## Deploy this version
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.80.8-stable.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.80.8.post1
```
</TabItem>
</Tabs>
## What's New
### 1. New Thinking Levels: `thinkingLevel` with MINIMAL & MEDIUM
Gemini 3 Flash introduces granular thinking control with `thinkingLevel` instead of `thinkingBudget`.
- **MINIMAL**: Ultra-lightweight thinking for fast responses
- **MEDIUM**: Balanced thinking for complex reasoning
- **HIGH**: Maximum reasoning depth
LiteLLM automatically maps the OpenAI `reasoning_effort` parameter to Gemini's `thinkingLevel`, so you can use familiar `reasoning_effort` values (`minimal`, `low`, `medium`, `high`) without changing your code!
### 2. Thought Signatures
Like `gemini-3-pro`, this model also includes thought signatures for tool calls. LiteLLM handles signature extraction and embedding internally. [Learn more about thought signatures](../gemini_3/index.md#thought-signatures).
**Edge Case Handling**: If thought signatures are missing in the request, LiteLLM adds a dummy signature ensuring the API call doesn't break
---
## Supported Endpoints
LiteLLM provides **full end-to-end support** for Gemini 3 Flash on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
- Function calling with thought signatures
- Multi-turn conversations
- All Gemini 3-specific features
- Converstion of provider specific thinking related param to thinkingLevel
## Quick Start
<Tabs>
<TabItem value="sdk" label="SDK">
**Basic Usage with MEDIUM thinking (NEW)**
```python
from litellm import completion
# No need to make any changes to your code as we map openai reasoning param to thinkingLevel
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "Solve this complex math problem: 25 * 4 + 10"}],
reasoning_effort="medium", # NEW: MEDIUM thinking level
)
print(response.choices[0].message.content)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
**1. Setup config.yaml**
```yaml
model_list:
- model_name: gemini-3-flash
litellm_params:
model: gemini/gemini-3-flash-preview
api_key: os.environ/GEMINI_API_KEY
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
```
**3. Call with MEDIUM thinking**
```bash
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-d '{
"model": "gemini-3-flash",
"messages": [{"role": "user", "content": "Complex reasoning task"}],
"reasoning_effort": "medium"
}'
``'
</TabItem>
</Tabs>
---
## All `reasoning_effort` Levels
<Tabs>
<TabItem value="minimal" label="MINIMAL">
**Ultra-fast, minimal reasoning**
```python
from litellm import completion
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "What's 2+2?"}],
reasoning_effort="minimal",
)
```
</TabItem>
<TabItem value="low" label="LOW">
**Simple instruction following**
```python
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "Write a haiku about coding"}],
reasoning_effort="low",
)
```
</TabItem>
<TabItem value="medium" label="MEDIUM (NEW)">
**Balanced reasoning for complex tasks** ✨
```python
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "Analyze this dataset and find patterns"}],
reasoning_effort="medium", # NEW!
)
```
</TabItem>
<TabItem value="high" label="HIGH">
**Maximum reasoning depth**
```python
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "Prove this mathematical theorem"}],
reasoning_effort="high",
)
```
</TabItem>
</Tabs>
---
## Key Features
✅ **Thinking Levels**: MINIMAL, LOW, MEDIUM, HIGH
✅ **Thought Signatures**: Track reasoning with unique identifiers
✅ **Seamless Integration**: Works with existing OpenAI-compatible client
✅ **Backward Compatible**: Gemini 2.5 models continue using `thinkingBudget`
---
## Installation
```bash
pip install litellm --upgrade
```
```python
import litellm
from litellm import completion
response = completion(
model="gemini/gemini-3-flash-preview",
messages=[{"role": "user", "content": "Your question here"}],
reasoning_effort="medium", # Use MEDIUM thinking
)
print(response)
```
:::note
If using this model via vertex_ai, keep the location as global as this is the only supported location as of now.
:::
## `reasoning_effort` Mapping for Gemini 3+
| reasoning_effort | thinking_level |
|------------------|----------------|
| `minimal` | `minimal` |
| `low` | `low` |
| `medium` | `medium` |
| `high` | `high` |
| `disable` | `minimal` |
| `none` | `minimal` |

2
docs/my-website/docs/benchmarks.md
View file

@ -172,7 +172,7 @@ class MyUser(HttpUser):
## Logging Callbacks
### [GCS Bucket Logging](https://docs.litellm.ai/docs/proxy/bucket)
### [GCS Bucket Logging](https://docs.litellm.ai/docs/observability/gcs_bucket_integration)
Using GCS Bucket has **no impact on latency, RPS compared to Basic Litellm Proxy**

2
docs/my-website/docs/image_edits.md
View file

@ -16,7 +16,7 @@ LiteLLM provides image editing functionality that maps to OpenAI's `/images/edit
| Supported operations | Create image edits | Single and multiple images supported |
| Supported LiteLLM SDK Versions | 1.63.8+ | Gemini support requires 1.79.3+ |
| Supported LiteLLM Proxy Versions | 1.71.1+ | Gemini support requires 1.79.3+ |
| Supported LLM providers | **OpenAI**, **Gemini (Google AI Studio)**, **Vertex AI** | Gemini supports the new `gemini-2.5-flash-image` family. Vertex AI supports both Gemini and Imagen models. |
| Supported LLM providers | **OpenAI**, **Gemini (Google AI Studio)**, **Vertex AI**, **Stability AI**, **AWS Bedrock (Stability)** | Gemini supports the new `gemini-2.5-flash-image` family. Vertex AI supports both Gemini and Imagen models. Stability AI and Bedrock Stability support various image editing operations. |
#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)

238
docs/my-website/docs/observability/azure_sentinel.md Normal file
View file

@ -0,0 +1,238 @@
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Azure Sentinel
<Image img={require('../../img/sentinel.png')} />
LiteLLM supports logging to Azure Sentinel via the Azure Monitor Logs Ingestion API. Azure Sentinel uses Log Analytics workspaces for data storage, so logs sent to the workspace will be available in Sentinel for security monitoring and analysis.
## Azure Sentinel Integration
| Feature | Details |
|---------|---------|
| **What is logged** | [StandardLoggingPayload](../proxy/logging_spec) |
| **Events** | Success + Failure |
| **Product Link** | [Azure Sentinel](https://learn.microsoft.com/en-us/azure/sentinel/overview) |
| **API Reference** | [Logs Ingestion API](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) |
We will use the `--config` to set `litellm.callbacks = ["azure_sentinel"]` this will log all successful and failed LLM calls to Azure Sentinel.
**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `callbacks`
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
litellm_settings:
callbacks: ["azure_sentinel"] # logs llm success + failure logs to Azure Sentinel
```
**Step 2**: Set Up Azure Resources
Before using the Logs Ingestion API, you need to set up the following in Azure:
1. **Create a Log Analytics Workspace** (if you don't have one)
2. **Create a Custom Table** in your Log Analytics workspace (e.g., `LiteLLM_CL`)
3. **Create a Data Collection Rule (DCR)** with:
- Stream declaration matching your data structure
- Transformation to map data to your custom table
- Access granted to your app registration
4. **Register an Application** in Microsoft Entra ID (Azure AD) with:
- Client ID
- Client Secret
- Permissions to write to the DCR
For detailed setup instructions, see the [Microsoft documentation on Logs Ingestion API](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview).
**Step 3**: Set Required Environment Variables
Set the following environment variables with your Azure credentials:
```shell showLineNumbers title="Environment Variables"
# Required: Data Collection Rule (DCR) configuration
AZURE_SENTINEL_DCR_IMMUTABLE_ID="dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # DCR Immutable ID from Azure portal
AZURE_SENTINEL_STREAM_NAME="Custom-LiteLLM_CL_CL" # Stream name from your DCR
AZURE_SENTINEL_ENDPOINT="https://your-dcr-endpoint.eastus-1.ingest.monitor.azure.com" # DCR logs ingestion endpoint (NOT the DCE endpoint)
# Required: OAuth2 Authentication (App Registration)
AZURE_SENTINEL_TENANT_ID="your-tenant-id" # Azure Tenant ID
AZURE_SENTINEL_CLIENT_ID="your-client-id" # Application (client) ID
AZURE_SENTINEL_CLIENT_SECRET="your-client-secret" # Client secret value
```
**Note**: The `AZURE_SENTINEL_ENDPOINT` should be the DCR's logs ingestion endpoint (found in the DCR Overview page), NOT the Data Collection Endpoint (DCE). The DCR endpoint is associated with your specific DCR and looks like: `https://your-dcr-endpoint.{region}-1.ingest.monitor.azure.com`
**Step 4**: Start the proxy and make a test request
Start proxy
```shell showLineNumbers title="Start Proxy"
litellm --config config.yaml --debug
```
Test Request
```shell showLineNumbers title="Test Request"
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
"metadata": {
"your-custom-metadata": "custom-field",
}
}'
```
**Step 5**: View logs in Azure Sentinel
1. Navigate to your Azure Sentinel workspace in the Azure portal
2. Go to "Logs" and query your custom table (e.g., `LiteLLM_CL`)
3. Run a query like:
```kusto showLineNumbers title="KQL Query"
LiteLLM_CL
| where TimeGenerated > ago(1h)
| project TimeGenerated, model, status, total_tokens, response_cost
| order by TimeGenerated desc
```
You should see following logs in Azure Workspace.
<Image img={require('../../img/sentinel.png')} />
## Environment Variables
| Environment Variable | Description | Default Value | Required |
|---------------------|-------------|---------------|----------|
| `AZURE_SENTINEL_DCR_IMMUTABLE_ID` | Data Collection Rule (DCR) Immutable ID | None | ✅ Yes |
| `AZURE_SENTINEL_ENDPOINT` | DCR logs ingestion endpoint URL (from DCR Overview page) | None | ✅ Yes |
| `AZURE_SENTINEL_STREAM_NAME` | Stream name from DCR (e.g., "Custom-LiteLLM_CL_CL") | "Custom-LiteLLM" | ❌ No |
| `AZURE_SENTINEL_TENANT_ID` | Azure Tenant ID for OAuth2 authentication | None (falls back to `AZURE_TENANT_ID`) | ✅ Yes |
| `AZURE_SENTINEL_CLIENT_ID` | Application (client) ID for OAuth2 authentication | None (falls back to `AZURE_CLIENT_ID`) | ✅ Yes |
| `AZURE_SENTINEL_CLIENT_SECRET` | Client secret for OAuth2 authentication | None (falls back to `AZURE_CLIENT_SECRET`) | ✅ Yes |
## How It Works
The Azure Sentinel integration uses the [Azure Monitor Logs Ingestion API](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) to send logs to your Log Analytics workspace. The integration:
- Authenticates using OAuth2 client credentials flow with your app registration
- Sends logs to the Data Collection Rule (DCR) endpoint
- Batches logs for efficient transmission
- Sends logs in the [StandardLoggingPayload](../proxy/logging_spec) format
- Automatically handles both success and failure events
- Caches OAuth2 tokens and refreshes them automatically
Logs sent to the Log Analytics workspace are automatically available in Azure Sentinel for security monitoring, threat detection, and analysis.
## Azure Sentinel Setup Guide
Follow this step-by-step guide to set up Azure Sentinel with LiteLLM.
### Step 1: Create a Log Analytics Workspace
1. Navigate to [https://portal.azure.com/#home](https://portal.azure.com/#home)
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/5659f6f5-a166-4b26-a991-73352274e3bb/ascreenshot.jpeg?tl_px=0,210&br_px=2618,1673&force_format=jpeg&q=100&width=1120.0)
2. Search for "Log Analytics workspaces" and click "Create"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/a827ba10-a391-486a-a36a-51816c6255de/ascreenshot.jpeg?tl_px=0,0&br_px=2618,1463&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=21,106)
3. Enter a name for your workspace (e.g., "litellm-sentinel-prod")
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/943458f1-fd4c-47dd-a273-ea5a04734ed9/ascreenshot.jpeg?tl_px=0,420&br_px=2618,1884&force_format=jpeg&q=100&width=1120.0)
4. Click "Review + Create"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/c54828fb-f895-4eb7-b810-cacf437617bd/ascreenshot.jpeg?tl_px=0,420&br_px=2618,1884&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=40,564)
### Step 2: Create a Custom Table
1. Go to your Log Analytics workspace and click "Tables"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/72d65f70-75c0-471f-95e9-947c72e173cc/ascreenshot.jpeg?tl_px=0,142&br_px=2618,1605&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=330,277)
2. Click "Create" → "New custom log (Direct Ingest)"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/863ad29b-2c3a-4b7c-9a6b-36d3a76c9f32/ascreenshot.jpeg?tl_px=0,0&br_px=2618,1463&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=526,146)
3. Enter a table name (e.g., "LITELLM_PROD_CL")
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/ef2f1c52-aa36-46a1-91e6-9bd868891b15/ascreenshot.jpeg?tl_px=0,0&br_px=2618,1463&force_format=jpeg&q=100&width=1120.0)
### Step 3: Create a Data Collection Rule (DCR)
1. Click "Create a new data collection rule"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/f2abc0d3-8be8-4057-9290-946d10cfd183/ascreenshot.jpeg?tl_px=0,420&br_px=2618,1884&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=264,404)
2. Enter a name for the DCR (e.g., "litellm-prod")
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/79bbebdc-e4d9-46ff-a270-1930619050a1/ascreenshot.jpeg?tl_px=0,8&br_px=2618,1471&force_format=jpeg&q=100&width=1120.0)
3. Select a Data Collection Endpoint
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/f3112e9a-551e-415c-a7f9-55aad801bc8a/ascreenshot.jpeg?tl_px=0,420&br_px=2618,1884&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=332,480)
4. Upload the sample JSON file for schema (use the [example_standard_logging_payload.json](https://github.com/BerriAI/litellm/blob/main/litellm/integrations/azure_sentinel/example_standard_logging_payload.json) file)
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/703c0762-840a-4f1f-a60f-876dc24b7a03/ascreenshot.jpeg?tl_px=0,0&br_px=2618,1463&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=518,272)
5. Click "Next" and then "Create"
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/0bca0200-5c64-4fbd-8061-9308aa6656b8/ascreenshot.jpeg?tl_px=0,420&br_px=2618,1884&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=128,560)
### Step 4: Get the DCR Immutable ID and Logs Ingestion Endpoint
1. Go to "Data Collection Rules" and select your DCR
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/11c06a0d-584f-4d22-b36e-9c338d43812c/ascreenshot.jpeg?tl_px=0,0&br_px=2618,1463&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=94,258)
2. Copy the **DCR Immutable ID** (starts with `dcr-`)
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/cd0ad69a-4d95-4b6a-9533-7720908ba809/ascreenshot.jpeg?tl_px=1160,92&br_px=2618,907&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=530,277)
3. Copy the **Logs Ingestion Endpoint** URL
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/3d3752ed-08ea-4490-8c98-a97d33947ea7/ascreenshot.jpeg?tl_px=1160,464&br_px=2618,1279&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=532,277)
### Step 5: Get the Stream Name
1. Click "JSON View" in the DCR
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/fd8a5504-4769-4f23-983e-520f256ee308/ascreenshot.jpeg?tl_px=1160,0&br_px=2618,814&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=965,257)
2. Find the **Stream Name** in the `streamDeclarations` section (e.g., "Custom-LITELLM_PROD_CL_CL")
![](https://ajeuwbhvhr.cloudimg.io/https://colony-recorder.s3.amazonaws.com/files/2025-12-17/a4052b32-2028-4d12-8930-bfcdf6f47652/ascreenshot.jpeg?tl_px=405,270&br_px=2115,1225&force_format=jpeg&q=100&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=523,277)
### Step 6: Register an App and Grant Permissions
1. Go to **Microsoft Entra ID** → **App registrations** → **New registration**
2. Create a new app and note the **Client ID** and **Tenant ID**
3. Go to **Certificates & secrets** → Create a new client secret and copy the **Secret Value**
4. Go back to your DCR → **Access Control (IAM)** → **Add role assignment**
5. Assign the **"Monitoring Metrics Publisher"** role to your app registration
### Summary: Where to Find Each Value
| Environment Variable | Where to Find It |
|---------------------|------------------|
| `AZURE_SENTINEL_DCR_IMMUTABLE_ID` | DCR Overview page → Immutable ID (starts with `dcr-`) |
| `AZURE_SENTINEL_ENDPOINT` | DCR Overview page → Logs Ingestion Endpoint |
| `AZURE_SENTINEL_STREAM_NAME` | DCR JSON View → `streamDeclarations` section |
| `AZURE_SENTINEL_TENANT_ID` | App Registration → Overview → Directory (tenant) ID |
| `AZURE_SENTINEL_CLIENT_ID` | App Registration → Overview → Application (client) ID |
| `AZURE_SENTINEL_CLIENT_SECRET` | App Registration → Certificates & secrets → Secret Value |
For more details, refer to the [Microsoft Logs Ingestion API documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview).

2
docs/my-website/docs/oidc.md
View file

@ -106,7 +106,7 @@ model_list:
aws_region_name: us-west-2
aws_session_name: "my-test-session"
aws_role_name: "arn:aws:iam::335785316107:role/litellm-github-unit-tests-circleci"
aws_web_identity_token: "oidc/circleci_v2/"
aws_web_identity_token: "oidc/example-provider/"
```
#### Amazon IAM Role Configuration for CircleCI v2 -> Bedrock

53
docs/my-website/docs/providers/openai/responses_api.md
View file

@ -623,6 +623,58 @@ display(styled_df)
</TabItem>
</Tabs>
## Function Calling
```python showLineNumbers title="Function Calling with Parallel Tool Calls"
import litellm
import json
tools = [
{
"type": "function",
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
]
# Step 1: Request with tools (parallel_tool_calls=True allows multiple calls)
response = litellm.responses(
model="openai/gpt-4o",
input=[{"role": "user", "content": "What's the weather in Paris and Tokyo?"}],
tools=tools,
parallel_tool_calls=True, # Defaults = True
)
# Step 2: Execute tool calls and collect results
tool_results = []
for output in response.output:
if output.type == "function_call":
result = {"temperature": 15, "condition": "sunny"} # Your function logic here
tool_results.append({
"type": "function_call_output",
"call_id": output.call_id,
"output": json.dumps(result)
})
# Step 3: Send results back
final_response = litellm.responses(
model="openai/gpt-4o",
input=tool_results,
tools=tools,
)
print(final_response.output)
```
Set `parallel_tool_calls=False` to ensure zero or one tool is called per turn. [More details](https://platform.openai.com/docs/guides/function-calling#parallel-function-calling).
## Free-form Function Calling
<Tabs>
@ -633,7 +685,6 @@ display(styled_df)
import litellm
response = litellm.responses(
response = client.responses.create(
model="gpt-5-mini",
input="Please use the code_exec tool to calculate the area of a circle with radius equal to the number of 'r's in strawberry",
text={"format": {"type": "text"}},

284
docs/my-website/docs/providers/stability.md
View file

@ -8,7 +8,7 @@ https://stability.ai/
| Description | Stability AI creates open AI models for image, video, audio, and 3D generation. Known for Stable Diffusion. |
| Provider Route on LiteLLM | `stability/` |
| Link to Provider Doc | [Stability AI API ↗](https://platform.stability.ai/docs/api-reference) |
| Supported Operations | [`/images/generations`](#image-generation) |
| Supported Operations | [`/images/generations`](#image-generation), [`/images/edits`](#image-editing) |
LiteLLM supports Stability AI Image Generation calls via the Stability AI REST API (not via Bedrock).
@ -169,13 +169,285 @@ Stability AI returns images in base64 format. The response is OpenAI-compatible:
}
```
## Comparing with Bedrock
## Image Editing
Stability AI supports various image editing operations including inpainting, upscaling, outpainting, background removal, and more.
### Usage - LiteLLM Python SDK
#### Inpainting (Edit with Mask)
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Inpainting - edit specific areas using a mask
response = image_edit(
model="stability/stable-image-inpaint-v1:0",
image=open("original_image.png", "rb"),
mask=open("mask_image.png", "rb"),
prompt="Add a beautiful sunset in the masked area",
size="1024x1024",
)
print(response)
```
#### Image Upscaling
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Conservative upscaling - preserves details
response = image_edit(
model="stability/stable-conservative-upscale-v1:0",
image=open("low_res_image.png", "rb"),
prompt="Upscale this image while preserving details",
)
# Creative upscaling - adds creative details
response = image_edit(
model="stability/stable-creative-upscale-v1:0",
image=open("low_res_image.png", "rb"),
prompt="Upscale and enhance with creative details",
creativity=0.3, # 0-0.35, higher = more creative
)
# Fast upscaling - quick upscaling
response = image_edit(
model="stability/stable-fast-upscale-v1:0",
image=open("low_res_image.png", "rb"),
prompt="Quickly upscale this image",
)
print(response)
```
#### Image Outpainting
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Extend image beyond its borders
response = image_edit(
model="stability/stable-outpaint-v1:0",
image=open("original_image.png", "rb"),
prompt="Extend this landscape with mountains",
left=100, # Pixels to extend on the left
right=100, # Pixels to extend on the right
up=50, # Pixels to extend on top
down=50, # Pixels to extend on bottom
)
print(response)
```
#### Background Removal
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Remove background from image
response = image_edit(
model="stability/stable-image-remove-background-v1:0",
image=open("portrait.png", "rb"),
prompt="Remove the background",
)
print(response)
```
#### Search and Replace
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Search and replace objects in image
response = image_edit(
model="stability/stable-image-search-replace-v1:0",
image=open("scene.png", "rb"),
prompt="A red sports car",
search_prompt="blue sedan", # What to replace
)
# Search and recolor
response = image_edit(
model="stability/stable-image-search-recolor-v1:0",
image=open("scene.png", "rb"),
prompt="Make it golden yellow",
select_prompt="the car", # What to recolor
)
print(response)
```
#### Image Control (Sketch/Structure)
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Control with sketch
response = image_edit(
model="stability/stable-image-control-sketch-v1:0",
image=open("sketch.png", "rb"),
prompt="Turn this sketch into a realistic photo",
control_strength=0.7, # 0-1, higher = more control
)
# Control with structure
response = image_edit(
model="stability/stable-image-control-structure-v1:0",
image=open("structure_reference.png", "rb"),
prompt="Generate image following this structure",
control_strength=0.7,
)
print(response)
```
#### Erase Objects
```python showLineNumbers
from litellm import image_edit
import os
os.environ['STABILITY_API_KEY'] = "your-api-key"
# Erase objects from image
response = image_edit(
model="stability/stable-image-erase-object-v1:0",
image=open("scene.png", "rb"),
mask=open("object_mask.png", "rb"), # Mask the object to erase
prompt="Remove the object",
)
print(response)
```
### Supported Image Edit Models
| Model Name | Function Call | Description |
|------------|---------------|-------------|
| stable-image-inpaint-v1:0 | `image_edit(model="stability/stable-image-inpaint-v1:0", ...)` | Inpainting with mask |
| stable-conservative-upscale-v1:0 | `image_edit(model="stability/stable-conservative-upscale-v1:0", ...)` | Conservative upscaling |
| stable-creative-upscale-v1:0 | `image_edit(model="stability/stable-creative-upscale-v1:0", ...)` | Creative upscaling |
| stable-fast-upscale-v1:0 | `image_edit(model="stability/stable-fast-upscale-v1:0", ...)` | Fast upscaling |
| stable-outpaint-v1:0 | `image_edit(model="stability/stable-outpaint-v1:0", ...)` | Extend image borders |
| stable-image-remove-background-v1:0 | `image_edit(model="stability/stable-image-remove-background-v1:0", ...)` | Remove background |
| stable-image-search-replace-v1:0 | `image_edit(model="stability/stable-image-search-replace-v1:0", ...)` | Search and replace objects |
| stable-image-search-recolor-v1:0 | `image_edit(model="stability/stable-image-search-recolor-v1:0", ...)` | Search and recolor |
| stable-image-control-sketch-v1:0 | `image_edit(model="stability/stable-image-control-sketch-v1:0", ...)` | Control with sketch |
| stable-image-control-structure-v1:0 | `image_edit(model="stability/stable-image-control-structure-v1:0", ...)` | Control with structure |
| stable-image-erase-object-v1:0 | `image_edit(model="stability/stable-image-erase-object-v1:0", ...)` | Erase objects |
| stable-image-style-guide-v1:0 | `image_edit(model="stability/stable-image-style-guide-v1:0", ...)` | Apply style guide |
| stable-style-transfer-v1:0 | `image_edit(model="stability/stable-style-transfer-v1:0", ...)` | Transfer style |
### Usage - LiteLLM Proxy Server
#### 1. Setup config.yaml
```yaml showLineNumbers
model_list:
- model_name: stability-inpaint
litellm_params:
model: stability/stable-image-inpaint-v1:0
api_key: os.environ/STABILITY_API_KEY
model_info:
mode: image_edit
- model_name: stability-upscale
litellm_params:
model: stability/stable-conservative-upscale-v1:0
api_key: os.environ/STABILITY_API_KEY
model_info:
mode: image_edit
general_settings:
master_key: sk-1234
```
#### 2. Start the proxy
```bash showLineNumbers
litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000
```
#### 3. Test it
```bash showLineNumbers
curl -X POST "http://0.0.0.0:4000/v1/images/edits" \
-H "Authorization: Bearer sk-1234" \
-F "model=stability-inpaint" \
-F "image=@original_image.png" \
-F "mask=@mask_image.png" \
-F "prompt=Add a beautiful garden in the masked area"
```
## AWS Bedrock (Stability)
LiteLLM also supports Stability AI models via AWS Bedrock. This is useful if you're already using AWS infrastructure.
### Usage - Bedrock Stability
```python showLineNumbers
from litellm import image_edit
import os
# Set AWS credentials
os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-key"
os.environ["AWS_REGION_NAME"] = "us-east-1"
# Bedrock Stability inpainting
response = image_edit(
model="bedrock/us.stability.stable-image-inpaint-v1:0",
image=open("original_image.png", "rb"),
mask=open("mask_image.png", "rb"),
prompt="Add flowers in the masked area",
size="1024x1024",
)
print(response)
```
### Supported Bedrock Stability Models
All Stability AI image edit models are available via Bedrock with the `bedrock/` prefix:
| Direct API Model | Bedrock Model | Description |
|------------------|---------------|-------------|
| stability/stable-image-inpaint-v1:0 | bedrock/us.stability.stable-image-inpaint-v1:0 | Inpainting |
| stability/stable-conservative-upscale-v1:0 | bedrock/stability.stable-conservative-upscale-v1:0 | Conservative upscaling |
| stability/stable-creative-upscale-v1:0 | bedrock/stability.stable-creative-upscale-v1:0 | Creative upscaling |
| stability/stable-fast-upscale-v1:0 | bedrock/stability.stable-fast-upscale-v1:0 | Fast upscaling |
| stability/stable-outpaint-v1:0 | bedrock/stability.stable-outpaint-v1:0 | Outpainting |
| stability/stable-image-remove-background-v1:0 | bedrock/stability.stable-image-remove-background-v1:0 | Remove background |
| stability/stable-image-search-replace-v1:0 | bedrock/stability.stable-image-search-replace-v1:0 | Search and replace |
| stability/stable-image-search-recolor-v1:0 | bedrock/stability.stable-image-search-recolor-v1:0 | Search and recolor |
| stability/stable-image-control-sketch-v1:0 | bedrock/stability.stable-image-control-sketch-v1:0 | Control with sketch |
| stability/stable-image-control-structure-v1:0 | bedrock/stability.stable-image-control-structure-v1:0 | Control with structure |
| stability/stable-image-erase-object-v1:0 | bedrock/stability.stable-image-erase-object-v1:0 | Erase objects |
**Note:** Bedrock model IDs may use `us.stability.*` or `stability.*` prefix depending on the region and model.
## Comparing Routes
LiteLLM supports Stability AI models via two routes:
| Route | Provider | Use Case |
|-------|----------|----------|
| `stability/` | Stability AI Direct API | Direct access, all latest models |
| `bedrock/stability.*` | AWS Bedrock | AWS integration, enterprise features |
| Route | Provider | Use Case | Image Generation | Image Editing |
|-------|----------|----------|------------------|---------------|
| `stability/` | Stability AI Direct API | Direct access, all latest models | ✅ | ✅ |
| `bedrock/stability.*` | AWS Bedrock | AWS integration, enterprise features | ✅ | ✅ |
Use `stability/` for direct API access. Use `bedrock/stability.*` if you're already using AWS Bedrock.

7
docs/my-website/docs/providers/vertex_ocr.md
View file

@ -140,7 +140,7 @@ with open("document.pdf", "rb") as f:
pdf_base64 = base64.b64encode(f.read()).decode()
response = litellm.ocr(
model="vertex_ai/mistral-ocr-2505",
model="vertex_ai/mistral-ocr-2505", # This doesn't work for deepseek
document={
"type": "document_url",
"document_url": f"data:application/pdf;base64,{pdf_base64}"
@ -219,7 +219,7 @@ print(f"Cost: ${response._hidden_params.get('response_cost', 0)}")
## Important Notes
:::info URL Conversion
Vertex AI OCR endpoints don't have internet access. LiteLLM automatically converts public URLs to base64 data URIs before sending requests to Vertex AI.
Vertex AI Mistral OCR endpoints don't have internet access. LiteLLM automatically converts public URLs to base64 data URIs before sending requests to Vertex AI.
:::
:::tip Regional Availability
@ -227,11 +227,14 @@ Mistral OCR is available in multiple regions. Specify `vertex_location` to use a
- `us-central1` (default)
- `europe-west1`
- `asia-southeast1`
Deepseek OCR is only available in global region.
:::
## Supported Models
- `mistral-ocr-2505` - Latest Mistral OCR model on Vertex AI
- `deepseek-ocr-maas` - Lates Deepseek OCR model on Vertex AI
Use the Vertex AI provider prefix: `vertex_ai/<model-name>`

22
docs/my-website/docs/proxy/alerting.md
View file

@ -215,16 +215,16 @@ general_settings:
alerting: ["slack"]
alerting_threshold: 0.0001 # (Seconds) set an artificially low threshold for testing alerting
alert_to_webhook_url: {
"llm_exceptions": "https://hooks.slack.com/services/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",
"llm_too_slow": "https://hooks.slack.com/services/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",
"llm_requests_hanging": "https://hooks.slack.com/services/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",
"budget_alerts": "https://hooks.slack.com/services/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",
"db_exceptions": "https://hooks.slack.com/services/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",
"daily_reports": "https://hooks.slack.com/services/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",
"spend_reports": "https://hooks.slack.com/services/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",
"cooldown_deployment": "https://hooks.slack.com/services/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",
"new_model_added": "https://hooks.slack.com/services/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",
"outage_alerts": "https://hooks.slack.com/services/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",
"llm_exceptions": "example-slack-webhook-url",
"llm_too_slow": "example-slack-webhook-url",
"llm_requests_hanging": "example-slack-webhook-url",
"budget_alerts": "example-slack-webhook-url",
"db_exceptions": "example-slack-webhook-url",
"daily_reports": "example-slack-webhook-url",
"spend_reports": "example-slack-webhook-url",
"cooldown_deployment": "example-slack-webhook-url",
"new_model_added": "example-slack-webhook-url",
"outage_alerts": "example-slack-webhook-url",
}
litellm_settings:
@ -399,7 +399,7 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \
{
"spend": 1, # the spend for the 'event_group'
"max_budget": 0, # the 'max_budget' set for the 'event_group'
"token": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"token": "example-api-key-123",
"user_id": "default_user_id",
"team_id": null,
"user_email": null,

12
docs/my-website/docs/proxy/config_settings.md
View file

@ -346,6 +346,7 @@ router_settings:
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Currently supported: 'router_budget_limiting', 'prompt_caching' |
| ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) |
| guardrail_list | List[GuardrailTypedDict] | List of guardrail configurations for guardrail load balancing. Enables load balancing across multiple guardrail deployments with the same guardrail_name. [Further Docs](./guardrails/guardrail_load_balancing.md) |
### environment variables - Reference
@ -413,6 +414,12 @@ router_settings:
| AZURE_FEDERATED_TOKEN_FILE | File path to Azure federated token
| AZURE_FILE_SEARCH_COST_PER_GB_PER_DAY | Cost per GB per day for Azure File Search service
| AZURE_SCOPE | For EntraID Auth, Scope for Azure services, defaults to "https://cognitiveservices.azure.com/.default"
| AZURE_SENTINEL_DCR_IMMUTABLE_ID | Immutable ID of the Data Collection Rule for Azure Sentinel logging
| AZURE_SENTINEL_STREAM_NAME | Stream name for Azure Sentinel logging
| AZURE_SENTINEL_CLIENT_SECRET | Client secret for Azure Sentinel authentication
| AZURE_SENTINEL_ENDPOINT | Endpoint for Azure Sentinel logging
| AZURE_SENTINEL_TENANT_ID | Tenant ID for Azure Sentinel authentication
| AZURE_SENTINEL_CLIENT_ID | Client ID for Azure Sentinel authentication
| AZURE_KEY_VAULT_URI | URI for Azure Key Vault
| AZURE_OPERATION_POLLING_TIMEOUT | Timeout in seconds for Azure operation polling
| AZURE_STORAGE_ACCOUNT_KEY | The Azure Storage Account Key to use for Authentication to Azure Blob Storage logging
@ -541,6 +548,8 @@ router_settings:
| DOCS_TITLE | Title of the documentation pages
| DOCS_URL | The path to the Swagger API documentation. **By default this is "/"**
| EMAIL_LOGO_URL | URL for the logo used in emails
| EMAIL_BUDGET_ALERT_TTL | Time-to-live for email budget alerts in seconds
| EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE | Maximum spend percentage for triggering email budget alerts
| EMAIL_SUPPORT_CONTACT | Support contact email address
| EMAIL_SIGNATURE | Custom HTML footer/signature for all emails. Can include HTML tags for formatting and links.
| EMAIL_SUBJECT_INVITATION | Custom subject template for invitation emails.
@ -596,6 +605,8 @@ router_settings:
| GREENSCALE_ENDPOINT | Endpoint URL for Greenscale service
| GRAYSWAN_API_BASE | Base URL for GraySwan API. Default is https://api.grayswan.ai
| GRAYSWAN_API_KEY | API key for GraySwan Cygnal service
| GRAYSWAN_REASONING_MODE | Reasoning mode for GraySwan guardrail
| GRAYSWAN_VIOLATION_THRESHOLD | Violation threshold for GraySwan guardrail
| GOOGLE_APPLICATION_CREDENTIALS | Path to Google Cloud credentials JSON file
| GOOGLE_CLIENT_ID | Client ID for Google OAuth
| GOOGLE_CLIENT_SECRET | Client secret for Google OAuth
@ -825,6 +836,7 @@ router_settings:
| SMTP_TLS | Flag to enable or disable TLS for SMTP connections
| SMTP_USERNAME | Username for SMTP authentication (do not set if SMTP does not require auth)
| SENDGRID_API_KEY | API key for SendGrid email service
| RESEND_API_KEY | API key for Resend email service
| SENDGRID_SENDER_EMAIL | Email address used as the sender in SendGrid email transactions
| SPEND_LOGS_URL | URL for retrieving spend logs
| SPEND_LOG_CLEANUP_BATCH_SIZE | Number of logs deleted per batch during cleanup. Default is 1000

6
docs/my-website/docs/proxy/cost_tracking.md
View file

@ -722,7 +722,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end
```shell
[
{
"api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"api_key": "example-api-key-123",
"total_cost": 0.3201286305151999,
"total_input_tokens": 36.0,
"total_output_tokens": 1593.0,
@ -766,7 +766,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end
```shell
[
{
"api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"api_key": "example-api-key-123",
"total_cost": 0.00013132,
"total_input_tokens": 105.0,
"total_output_tokens": 872.0,
@ -1151,7 +1151,7 @@ curl -X GET "http://0.0.0.0:4000/spend/logs?request_id=<your-call-id" \ # e.g.:
"request_id": "chatcmpl-9ZKMURhVYSi9D6r6PJ9vLcayIK0Vm",
"call_type": "acompletion",
"metadata": {
"user_api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"user_api_key": "example-api-key-123",
"user_api_key_alias": null,
"spend_logs_metadata": { # 👈 LOGGED CUSTOM METADATA
"hello": "world"

27
docs/my-website/docs/proxy/custom_auth.md
View file

@ -9,6 +9,7 @@ You can now override the default api key auth.
Make sure the response type follows the `UserAPIKeyAuth` pydantic object. This is used by for logging usage specific to that user key.
```python
from fastapi import Request
from litellm.proxy._types import UserAPIKeyAuth
async def user_api_key_auth(request: Request, api_key: str) -> UserAPIKeyAuth:
@ -114,6 +115,29 @@ UserAPIKeyAuth(
)
```
### Object Permission Example (MCP, agents, etc.)
```python
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
def _server_id(name: str) -> str:
server = global_mcp_server_manager.get_mcp_server_by_name(name)
if not server:
raise ValueError(f"Unknown MCP server '{name}'")
return server.server_id
object_permission = LiteLLM_ObjectPermissionTable(
mcp_servers=[_server_id("deepwiki"), _server_id("everything")], # MCP servers this key is allowed to use
mcp_tool_permissions={"deepwiki": ["search", "read_doc"]}, # optional per-server tool allow-list
)
UserAPIKeyAuth(
object_permission=object_permission,
)
```
### Advanced Configuration
```python
UserAPIKeyAuth(
@ -139,6 +163,7 @@ UserAPIKeyAuth(
### Complete Example
```python
from fastapi import Request
from datetime import datetime, timedelta
from litellm.proxy._types import UserAPIKeyAuth, LitellmUserRoles
@ -333,4 +358,4 @@ async def user_api_key_auth(
except Exception:
raise Exception("Invalid API key")
```
```

2
docs/my-website/docs/proxy/customers.md
View file

@ -103,7 +103,7 @@ Expected Response
{
"spend": 0.0011120000000000001, # 👈 SPEND
"max_budget": null,
"token": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"token": "example-api-key-123",
"customer_id": "krrish12", # 👈 CUSTOMER ID
"user_id": null,
"team_id": null,

2
docs/my-website/docs/proxy/enterprise.md
View file

@ -29,7 +29,7 @@ Features:
- **Spend Tracking & Data Exports**
- ✅ [Set USD Budgets Spend for Custom Tags](./provider_budget_routing#-tag-budgets)
- ✅ [Set Model budgets for Virtual Keys](./users#-virtual-key-model-specific)
- ✅ [Exporting LLM Logs to GCS Bucket, Azure Blob Storage](./proxy/bucket#🪣-logging-gcs-s3-buckets)
- ✅ [Exporting LLM Logs to GCS Bucket, Azure Blob Storage](../observability/gcs_bucket_integration)
- ✅ [`/spend/report` API endpoint](cost_tracking.md#✨-enterprise-api-endpoints-to-get-spend)
- **Control Guardrails per API Key/Team**
- **Custom Branding**

351
docs/my-website/docs/proxy/guardrails/guardrail_load_balancing.md Normal file
View file

@ -0,0 +1,351 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Guardrail Load Balancing
Load balance guardrail requests across multiple guardrail deployments. This is useful when you have rate limits on guardrail providers (e.g., AWS Bedrock Guardrails) and want to distribute requests across multiple accounts or regions.
## How It Works
```mermaid
flowchart LR
subgraph LiteLLM Gateway
Router[Router]
G1[Guardrail Instance A]
G2[Guardrail Instance B]
G3[Guardrail Instance N]
end
Client[Client Request] --> Router
Router -->|Round Robin / Weighted| G1
Router -->|Round Robin / Weighted| G2
Router -->|Round Robin / Weighted| G3
G1 --> AWS1[AWS Account 1]
G2 --> AWS2[AWS Account 2]
G3 --> AWSN[AWS Account N]
```
When you define multiple guardrails with the **same `guardrail_name`**, LiteLLM automatically load balances requests across them using the router's load balancing strategy.
## Why Use Guardrail Load Balancing?
| Use Case | Benefit |
|----------|---------|
| **AWS Bedrock Rate Limits** | Bedrock Guardrails have per-account rate limits. Distribute across multiple AWS accounts to increase throughput |
| **Multi-Region Redundancy** | Deploy guardrails across regions for failover and lower latency |
| **Cost Optimization** | Spread usage across accounts with different pricing tiers or credits |
| **A/B Testing** | Test different guardrail configurations with weighted distribution |
## Quick Start
### 1. Define Multiple Guardrails with Same Name
Define multiple guardrail entries with the **same `guardrail_name`** but different configurations:
<Tabs>
<TabItem value="bedrock" label="Bedrock Guardrails">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY
guardrails:
# First Bedrock guardrail - AWS Account 1
- guardrail_name: "content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "pre_call"
guardrailIdentifier: "abc123"
guardrailVersion: "1"
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID_1
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY_1
aws_region_name: "us-east-1"
# Second Bedrock guardrail - AWS Account 2
- guardrail_name: "content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "pre_call"
guardrailIdentifier: "def456"
guardrailVersion: "1"
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID_2
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY_2
aws_region_name: "us-west-2"
```
</TabItem>
<TabItem value="custom" label="Custom Guardrails">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY
guardrails:
# First custom guardrail instance
- guardrail_name: "pii-filter"
litellm_params:
guardrail: custom_guardrail.PIIFilterA
mode: "pre_call"
# Second custom guardrail instance
- guardrail_name: "pii-filter"
litellm_params:
guardrail: custom_guardrail.PIIFilterB
mode: "pre_call"
```
</TabItem>
<TabItem value="aporia" label="Aporia Guardrails">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY
guardrails:
# First Aporia instance
- guardrail_name: "toxicity-filter"
litellm_params:
guardrail: aporia
mode: "pre_call"
api_key: os.environ/APORIA_API_KEY_1
api_base: os.environ/APORIA_API_BASE_1
# Second Aporia instance
- guardrail_name: "toxicity-filter"
litellm_params:
guardrail: aporia
mode: "pre_call"
api_key: os.environ/APORIA_API_KEY_2
api_base: os.environ/APORIA_API_BASE_2
```
</TabItem>
</Tabs>
### 2. Start LiteLLM Gateway
```bash showLineNumbers title="Start proxy"
litellm --config config.yaml --detailed_debug
```
### 3. Make Requests
Requests using the guardrail will be automatically load balanced:
```bash showLineNumbers title="Test request"
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello, how are you?"}],
"guardrails": ["content-filter"]
}'
```
## Weighted Load Balancing
Assign weights to distribute traffic unevenly across guardrail instances:
```yaml showLineNumbers title="config.yaml - Weighted distribution"
guardrails:
# 80% of traffic
- guardrail_name: "content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "pre_call"
guardrailIdentifier: "primary-guard"
guardrailVersion: "1"
weight: 8 # Higher weight = more traffic
# 20% of traffic
- guardrail_name: "content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "pre_call"
guardrailIdentifier: "secondary-guard"
guardrailVersion: "1"
weight: 2 # Lower weight = less traffic
```
## Bedrock Guardrails - Multi-Account Setup
AWS Bedrock Guardrails have rate limits per account. Here's how to set up load balancing across multiple AWS accounts:
### Architecture
```mermaid
flowchart TB
subgraph LiteLLM["LiteLLM Gateway"]
LB[Load Balancer]
end
subgraph AWS1["AWS Account 1 (us-east-1)"]
BG1[Bedrock Guardrail]
end
subgraph AWS2["AWS Account 2 (us-west-2)"]
BG2[Bedrock Guardrail]
end
subgraph AWS3["AWS Account 3 (eu-west-1)"]
BG3[Bedrock Guardrail]
end
Client[Client] --> LiteLLM
LB --> BG1
LB --> BG2
LB --> BG3
```
### Configuration
```yaml showLineNumbers title="config.yaml - Multi-account Bedrock"
model_list:
- model_name: claude-3
litellm_params:
model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
guardrails:
# AWS Account 1 - US East
- guardrail_name: "bedrock-content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "during_call"
guardrailIdentifier: "guard-us-east"
guardrailVersion: "DRAFT"
aws_access_key_id: os.environ/AWS_ACCESS_KEY_1
aws_secret_access_key: os.environ/AWS_SECRET_KEY_1
aws_region_name: "us-east-1"
# AWS Account 2 - US West
- guardrail_name: "bedrock-content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "during_call"
guardrailIdentifier: "guard-us-west"
guardrailVersion: "DRAFT"
aws_access_key_id: os.environ/AWS_ACCESS_KEY_2
aws_secret_access_key: os.environ/AWS_SECRET_KEY_2
aws_region_name: "us-west-2"
# AWS Account 3 - EU West
- guardrail_name: "bedrock-content-filter"
litellm_params:
guardrail: bedrock/guardrail
mode: "during_call"
guardrailIdentifier: "guard-eu-west"
guardrailVersion: "DRAFT"
aws_access_key_id: os.environ/AWS_ACCESS_KEY_3
aws_secret_access_key: os.environ/AWS_SECRET_KEY_3
aws_region_name: "eu-west-1"
```
### Test Multi-Account Setup
```bash showLineNumbers title="Run multiple requests to verify load balancing"
# Run 10 requests - they will be distributed across accounts
for i in {1..10}; do
curl -s -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "claude-3",
"messages": [{"role": "user", "content": "Hello"}],
"guardrails": ["bedrock-content-filter"]
}' &
done
wait
```
Check proxy logs to verify requests are distributed across different AWS accounts.
## Custom Guardrails Example
Create two custom guardrail classes for load balancing:
```python showLineNumbers title="custom_guardrail.py"
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.proxy._types import UserAPIKeyAuth
from litellm.caching.caching import DualCache
class PIIFilterA(CustomGuardrail):
"""PII Filter Instance A"""
async def async_pre_call_hook(
self,
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: dict,
call_type: str,
):
print("PIIFilterA processing request")
# Your PII filtering logic here
return data
class PIIFilterB(CustomGuardrail):
"""PII Filter Instance B"""
async def async_pre_call_hook(
self,
user_api_key_dict: UserAPIKeyAuth,
cache: DualCache,
data: dict,
call_type: str,
):
print("PIIFilterB processing request")
# Your PII filtering logic here
return data
```
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "pii-filter"
litellm_params:
guardrail: custom_guardrail.PIIFilterA
mode: "pre_call"
- guardrail_name: "pii-filter"
litellm_params:
guardrail: custom_guardrail.PIIFilterB
mode: "pre_call"
```
## Verifying Load Balancing
Enable detailed debug logging to verify load balancing is working:
```bash showLineNumbers title="Start with debug logging"
litellm --config config.yaml --detailed_debug
```
You should see logs indicating which guardrail instance is selected:
```
Selected guardrail deployment: bedrock/guardrail (guard-us-east)
Selected guardrail deployment: bedrock/guardrail (guard-us-west)
Selected guardrail deployment: bedrock/guardrail (guard-eu-west)
...
```
## Related
- [Guardrails Quick Start](./quick_start.md)
- [Bedrock Guardrails](./bedrock.md)
- [Custom Guardrails](./custom_guardrail.md)
- [Load Balancing for LLM Calls](../load_balancing.md)

11
docs/my-website/docs/proxy/guardrails/lakera_ai.md
View file

@ -29,6 +29,13 @@ guardrails:
mode: "pre_call"
api_key: os.environ/LAKERA_API_KEY
api_base: os.environ/LAKERA_API_BASE
- guardrail_name: "lakera-monitor"
litellm_params:
guardrail: lakera_v2
mode: "pre_call"
on_flagged: "monitor" # Log violations but don't block
api_key: os.environ/LAKERA_API_KEY
api_base: os.environ/LAKERA_API_BASE
```
@ -144,6 +151,7 @@ guardrails:
# breakdown: Optional[bool] = True,
# metadata: Optional[Dict] = None,
# dev_info: Optional[bool] = True,
# on_flagged: Optional[str] = "block", # "block" or "monitor"
```
- `api_base`: (Optional[str]) The base of the Lakera integration. Defaults to `https://api.lakera.ai`
@ -153,3 +161,6 @@ guardrails:
- `breakdown`: (Optional[bool]) When true the response will return a breakdown list of the detectors that were run, as defined in the policy, and whether each of them detected something or not.
- `metadata`: (Optional[Dict]) Metadata tags can be attached to screening requests as an object that can contain any arbitrary key-value pairs.
- `dev_info`: (Optional[bool]) When true the response will return an object with developer information about the build of Lakera Guard.
- `on_flagged`: (Optional[str]) Action to take when content is flagged. Defaults to `"block"`.
- `"block"`: Raises an HTTP 400 exception when violations are detected (default behavior)
- `"monitor"`: Logs violations but allows the request to proceed. Useful for tuning security policies without blocking legitimate requests.

415
docs/my-website/docs/proxy/guardrails/litellm_content_filter.md
View file

@ -3,10 +3,12 @@ import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# LiteLLM Content Filter
# LiteLLM Content Filter (Built-in Guardrails)
**Built-in guardrail** for detecting and filtering sensitive information using regex patterns and keyword matching. No external dependencies required.
**When to use?** Good for cases which do not require an ML model to detect sensitive information.
## Overview
| Property | Details |
@ -56,6 +58,44 @@ Test examples:
### Step 1: Define Guardrails in config.yaml
<Tabs>
<TabItem label="Harmful Content Detection" value="harmful">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "harmful-content-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
# Enable harmful content categories
categories:
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_illegal_weapons"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
```
</TabItem>
<TabItem label="PII Protection" value="pii">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-3.5-turbo
@ -86,6 +126,48 @@ guardrails:
description: "Sensitive internal information"
```
</TabItem>
<TabItem label="Combined" value="combined">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "comprehensive-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
# Harmful content categories
categories:
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "high"
# PII patterns
patterns:
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
# Custom keywords
blocked_words:
- keyword: "confidential"
action: "BLOCK"
```
</TabItem>
</Tabs>
### Step 2: Start LiteLLM Gateway
```shell
@ -175,7 +257,7 @@ Contact me at [EMAIL_REDACTED]
| `amex` | American Express cards | `3782-822463-10005` |
| `aws_access_key` | AWS access keys | `AKIAIOSFODNN7EXAMPLE` |
| `aws_secret_key` | AWS secret keys | `wJalrXUtnFEMI/K7MDENG/bPxRfi...` |
| `github_token` | GitHub tokens | `ghp_16C7e42F292c6912E7710c838347Ae178B4a` |
| `github_token` | GitHub tokens | `example-github-token-123` |
### Using Prebuilt Patterns
@ -310,6 +392,85 @@ for chunk in response:
# Emails automatically masked in real-time
```
## Image Content Filtering
Content filter can analyze images by generating descriptions and applying filters to the text descriptions.
:::warning
This can introduce significant latency to the request - depending on the speed of the vision-capable model.
This is because, each request containing images will be sent to the vision-capable model to generate a description.
:::
### Configuration
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4-vision
litellm_params:
model: openai/gpt-4-vision-preview
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "image-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
image_model: "gpt-4-vision" # value is `model_name` of the vision-capable model
# Apply same filters to image descriptions
categories:
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
patterns:
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
```
### How It Works
1. Image is sent to the vision model to generate a text description
2. Content filters are applied to the description
3. If harmful content is detected, request is blocked with context about the image
**Example:**
```python
import openai
client = openai.OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000"
)
response = client.chat.completions.create(
model="gpt-4-vision",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
]
}],
extra_body={"guardrails": ["image-filter"]}
)
```
If the image description contains filtered content, you'll get:
```json
{
"error": "Content blocked: harmful_violence category keyword 'weapon' detected (severity: high) (Image description): The image shows..."
}
```
## Customizing Redaction Tags
When using the `MASK` action, sensitive content is replaced with redaction tags. You can customize how these tags appear.
@ -363,9 +524,171 @@ Output: "Email ***EMAIL***, SSN ***US_SSN***, ***REDACTED*** data"
- Pattern names are automatically uppercased (e.g., `email` → `EMAIL`)
- `keyword_redaction_tag` is a fixed string (no placeholders)
## Content Categories
Prebuilt categories use **keyword matching** to detect harmful content, bias, and inappropriate advice. Keywords are matched with word boundaries (single words) or as substrings (multi-word phrases), case-insensitive.
### Available Categories
| Category | Description |
|----------|-------------|
| **Harmful Content** | |
| `harmful_self_harm` | Self-harm, suicide, eating disorders |
| `harmful_violence` | Violence, criminal planning, attacks |
| `harmful_illegal_weapons` | Illegal weapons, explosives, dangerous materials |
| **Bias Detection** | |
| `bias_gender` | Gender-based discrimination, stereotypes |
| `bias_sexual_orientation` | LGBTQ+ discrimination, homophobia, transphobia |
| `bias_racial` | Racial/ethnic discrimination, stereotypes |
| `bias_religious` | Religious discrimination, stereotypes |
| **Denied Advice** | |
| `denied_financial_advice` | Personalized financial advice, investment recommendations |
| `denied_medical_advice` | Medical advice, diagnosis, treatment recommendations |
| `denied_legal_advice` | Legal advice, representation, legal strategy |
:::info Bias Detection Considerations
Bias detection is **complex and context-dependent**. Rule-based systems catch explicit discriminatory language but may generate false positives on legitimate discussions. Start with **high severity thresholds** and test thoroughly. For mission-critical bias detection, consider combining with AI-based guardrails (e.g., HiddenLayer, Lakera).
:::
### Configuration
```yaml showLineNumbers title="config.yaml"
guardrails:
- guardrail_name: "content-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
categories:
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium" # Blocks medium+ severity
- category: "bias_gender"
enabled: true
action: "BLOCK"
severity_threshold: "high" # Only explicit discrimination
- category: "denied_financial_advice"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
```
**Severity Thresholds:**
- `"high"` - Only blocks high severity items
- `"medium"` - Blocks medium and high severity (default)
- `"low"` - Blocks all severity levels
### Custom Category Files
Override default categories with custom keyword lists:
```yaml showLineNumbers title="config.yaml"
categories:
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
category_file: "/path/to/custom.yaml"
```
```yaml showLineNumbers title="custom.yaml"
category_name: "harmful_self_harm"
description: "Custom self-harm detection"
default_action: "BLOCK"
keywords:
- keyword: "suicide"
severity: "high"
- keyword: "harm myself"
severity: "high"
exceptions:
- "suicide prevention"
- "mental health"
```
## Use Cases
### 1. PII Protection
### 1. Harmful Content Detection
Block or detect requests containing harmful, illegal, or dangerous content:
```yaml
categories:
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "high"
- category: "harmful_illegal_weapons"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
```
### 2. Bias and Discrimination Detection
Detect and block biased, discriminatory, or hateful content across multiple dimensions:
```yaml
categories:
# Gender-based discrimination
- category: "bias_gender"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
# LGBTQ+ discrimination
- category: "bias_sexual_orientation"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
# Racial/ethnic discrimination
- category: "bias_racial"
enabled: true
action: "BLOCK"
severity_threshold: "high" # Only explicit to reduce false positives
# Religious discrimination
- category: "bias_religious"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
```
**Sensitivity Tuning:**
For bias detection, severity thresholds are critical to balance safety and legitimate discourse:
```yaml
# Conservative (low false positives, may miss subtle bias)
categories:
- category: "bias_racial"
severity_threshold: "high" # Only blocks explicit discriminatory language
# Balanced (recommended)
categories:
- category: "bias_gender"
severity_threshold: "medium" # Blocks stereotypes and explicit discrimination
# Strict (high safety, may have more false positives)
categories:
- category: "bias_sexual_orientation"
severity_threshold: "low" # Blocks all potentially problematic content
```
### 3. PII Protection
Block or mask personally identifiable information before sending to LLMs:
```yaml
@ -409,10 +732,64 @@ For large lists of sensitive terms, use a file:
blocked_words_file: "/path/to/sensitive_terms.yaml"
```
### 4. Compliance
### 4. Safe AI for Consumer Applications
Combining harmful content and bias detection for consumer-facing AI:
```yaml
guardrails:
- guardrail_name: "safe-consumer-ai"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
categories:
# Harmful content - strict
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
# Bias detection - balanced
- category: "bias_gender"
enabled: true
action: "BLOCK"
severity_threshold: "high" # Avoid blocking legitimate gender discussions
- category: "bias_sexual_orientation"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "bias_racial"
enabled: true
action: "BLOCK"
severity_threshold: "high" # Education and news may discuss race
```
**Perfect for:**
- Chatbots and virtual assistants
- Educational AI tools
- Customer service AI
- Content generation platforms
- Public-facing AI applications
### 5. Compliance
Ensure regulatory compliance by filtering sensitive data types:
```yaml
# Categories checked first (high priority)
# Category keywords are matched first
categories:
- category: "harmful_self_harm"
severity_threshold: "high"
# Then regex patterns
patterns:
- pattern_type: "prebuilt"
pattern_name: "visa"
@ -422,34 +799,4 @@ patterns:
action: "BLOCK"
```
## Troubleshooting
### Pattern Not Matching
**Issue:** Regex pattern isn't detecting expected content
**Solution:** Test your regex pattern:
```python
import re
pattern = r'\b[A-Z]{3}-\d{4}\b'
test_text = "Employee ID: ABC-1234"
print(re.search(pattern, test_text)) # Should match
```
### Multiple Pattern Matches
**Issue:** Text contains multiple sensitive patterns
**Solution:** First matching pattern/keyword is processed. Order patterns by priority:
```yaml
patterns:
# Most critical first
- pattern_type: "prebuilt"
pattern_name: "us_ssn"
action: "BLOCK"
# Less critical
- pattern_type: "prebuilt"
pattern_name: "email"
action: "MASK"
```

4
docs/my-website/docs/proxy/guardrails/pillar_security.md
View file

@ -790,7 +790,7 @@ curl -X POST "http://localhost:4000/v1/chat/completions" \
"messages": [
{
"role": "user",
"content": "Generate python code that accesses my Github repo using this PAT: ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8"
"content": "Generate python code that accesses my Github repo using this PAT: example-github-token-123"
}
],
"max_tokens": 50
@ -815,7 +815,7 @@ curl -X POST "http://localhost:4000/v1/chat/completions" \
"type": "github_token",
"start_idx": 66,
"end_idx": 106,
"evidence": "ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8",
"evidence": "example-github-token-123",
}
]
}

7
docs/my-website/docs/proxy/guardrails/quick_start.md
View file

@ -69,6 +69,13 @@ guardrails:
- `during_call` Run **during** LLM call, on **input** Same as `pre_call` but runs in parallel as LLM call. Response not returned until guardrail check completes
- A list of the above values to run multiple modes, e.g. `mode: [pre_call, post_call]`
### Load Balancing Guardrails
Need to distribute guardrail requests across multiple accounts or regions? See [Guardrail Load Balancing](./guardrail_load_balancing.md) for details on:
- Load balancing across multiple AWS Bedrock accounts (useful for rate limit management)
- Weighted distribution across guardrail instances
- Multi-region guardrail deployments
## 2. Start LiteLLM Gateway

5
docs/my-website/docs/proxy/logging.md
View file

@ -16,6 +16,7 @@ Log Proxy input, output, and exceptions using:
- Custom Callbacks - Custom code and API endpoints
- Langsmith
- DataDog
- Azure Sentinel
- DynamoDB
- etc.
@ -1574,6 +1575,10 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
👉 Go here for using [Datadog LLM Observability](../observability/datadog) with LiteLLM Proxy
## [Azure Sentinel](../observability/azure_sentinel)
👉 Go here for using [Azure Sentinel](../observability/azure_sentinel) with LiteLLM Proxy
## Lunary
#### Step1: Install dependencies and set your environment variables

2
docs/my-website/docs/proxy/multiple_admins.md
View file

@ -89,7 +89,7 @@ curl -X POST 'http://0.0.0.0:4000/team/update' \
"id": "bd136c28-edd0-4cb6-b963-f35464cf6f5a",
"updated_at": "2024-06-08 23:41:14.793",
"changed_by": "krrish@berri.ai", # 👈 CHANGED BY
"changed_by_api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b",
"changed_by_api_key": "example-api-key-123",
"action": "updated",
"table_name": "LiteLLM_TeamTable",
"object_id": "8bf18b11-7f52-4717-8e1f-7c65f9d01e52",

2
docs/my-website/docs/proxy/prod.md
View file

@ -33,7 +33,7 @@ litellm_settings:
Set slack webhook url in your env
```shell
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/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"
export SLACK_WEBHOOK_URL="example-slack-webhook-url"
```
Turn off FASTAPI's default info logs

2
docs/my-website/docs/proxy/quick_start.md
View file

@ -400,7 +400,7 @@ from anthropic import Anthropic
client = Anthropic(
base_url="http://localhost:4000", # proxy endpoint
api_key="sk-s4xN1IiLTCytwtZFJaYQrA", # litellm proxy virtual key
api_key="sk-test-proxy-key-123", # litellm proxy virtual key (example)
)
message = client.messages.create(

2
docs/my-website/docs/proxy/user_keys.md
View file

@ -285,7 +285,7 @@ from anthropic import Anthropic
client = Anthropic(
base_url="http://localhost:4000", # proxy endpoint
api_key="sk-s4xN1IiLTCytwtZFJaYQrA", # litellm proxy virtual key
api_key="sk-test-proxy-key-123", # litellm proxy virtual key (example)
)
message = client.messages.create(

2
docs/my-website/docs/response_api.md
View file

@ -4,7 +4,7 @@ import TabItem from '@theme/TabItem';
# /responses
LiteLLM provides a BETA endpoint in the spec of [OpenAI's `/responses` API](https://platform.openai.com/docs/api-reference/responses)
LiteLLM provides an endpoint in the spec of [OpenAI's `/responses` API](https://platform.openai.com/docs/api-reference/responses)
Requests to /chat/completions may be bridged here automatically when the provider lacks support for that endpoint. The model’s default `mode` determines how bridging works.(see `model_prices_and_context_window`)

5
docs/my-website/docs/search/index.md
View file

@ -2,7 +2,7 @@
| Feature | Supported |
|---------|-----------|
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `google_pse`, `dataforseo`, `firecrawl`, `searxng` |
| Supported Providers | `perplexity`, `tavily`, `parallel_ai`, `exa_ai`, `google_pse`, `dataforseo`, `firecrawl`, `searxng`, `linkup` |
| Cost Tracking | ✅ |
| Logging | ✅ |
| Load Balancing | ❌ |
@ -205,7 +205,7 @@ See the [official Perplexity Search documentation](https://docs.perplexity.ai/ap
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string or array | Yes | Search query. Can be a single string or array of strings |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, or `"searxng"` |
| `search_provider` | string | Yes (SDK) | The search provider to use: `"perplexity"`, `"tavily"`, `"parallel_ai"`, `"exa_ai"`, `"google_pse"`, `"dataforseo"`, `"firecrawl"`, `"searxng"`, or `"linkup"` |
| `search_tool_name` | string | Yes (Proxy) | Name of the search tool configured in `config.yaml` |
| `max_results` | integer | No | Maximum number of results to return (1-20). Default: 10 |
| `search_domain_filter` | array | No | List of domains to filter results (max 20 domains) |
@ -269,6 +269,7 @@ The response follows Perplexity's search format with the following structure:
| DataForSEO | `DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD` | `dataforseo` |
| Firecrawl | `FIRECRAWL_API_KEY` | `firecrawl` |
| SearXNG | `SEARXNG_API_BASE` (required) | `searxng` |
| Linkup | `LINKUP_API_KEY` | `linkup` |
See the individual provider documentation for detailed setup instructions and provider-specific parameters.

152
docs/my-website/docs/search/linkup.md Normal file
View file

@ -0,0 +1,152 @@
# Linkup Search
**Get API Key:** [https://linkup.so](https://linkup.so)
## LiteLLM Python SDK
```python showLineNumbers title="Linkup Search"
import os
from litellm import search
os.environ["LINKUP_API_KEY"] = "..."
response = search(
query="latest AI developments",
search_provider="linkup",
max_results=5
)
```
## LiteLLM AI Gateway
### 1. Setup config.yaml
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
search_tools:
- search_tool_name: linkup-search
litellm_params:
search_provider: linkup
api_key: os.environ/LINKUP_API_KEY
```
### 2. Start the proxy
```bash
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
### 3. Test the search endpoint
```bash showLineNumbers title="Test Request"
curl http://0.0.0.0:4000/v1/search/linkup-search \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"query": "latest AI developments",
"max_results": 5
}'
```
## Provider-specific Parameters
```python showLineNumbers title="Linkup Search with Provider-specific Parameters"
import os
from litellm import search
os.environ["LINKUP_API_KEY"] = "..."
response = search(
query="machine learning research",
search_provider="linkup",
max_results=10,
# Linkup-specific parameters
depth="deep", # "standard" (faster) or "deep" (more comprehensive)
outputType="searchResults", # "searchResults", "sourcedAnswer", or "structured"
includeSources=True, # Include sources in response
includeImages=True, # Include images in results
fromDate="2024-01-01", # Start date filter (YYYY-MM-DD)
toDate="2024-12-31", # End date filter (YYYY-MM-DD)
includeDomains=["arxiv.org", "nature.com"], # Domains to search (max 100)
excludeDomains=["wikipedia.com"], # Domains to exclude
includeInlineCitations=True, # Include inline citations in sourcedAnswer
)
```
## Features
Linkup provides powerful web search with context retrieval capabilities:
### Search Depth
Control the precision and speed of your search:
- `standard` - Returns results faster
- `deep` - Takes longer but yields more comprehensive results
### Output Types
Choose how results are formatted:
- `searchResults` - Returns a list of search results with URLs and content
- `sourcedAnswer` - Returns an AI-generated answer with sources
- `structured` - Returns results in a custom JSON schema format
### Date Filtering
Filter results by date range:
```python
response = search(
query="AI developments",
search_provider="linkup",
fromDate="2024-06-01",
toDate="2024-12-31"
)
```
### Domain Filtering
Include or exclude specific domains:
```python
response = search(
query="research papers",
search_provider="linkup",
includeDomains=["arxiv.org", "nature.com", "ieee.org"],
excludeDomains=["wikipedia.com"]
)
```
### Structured Output
Get results in a custom JSON schema format:
```python
response = search(
query="Microsoft 2024 revenue",
search_provider="linkup",
outputType="structured",
structuredOutputSchema='{"type": "object", "properties": {"revenue": {"type": "string"}, "year": {"type": "string"}}}'
)
```
## Response Format
Linkup returns results in the following format:
```json
{
"results": [
{
"type": "text",
"name": "Microsoft 2024 Annual Report",
"url": "https://www.microsoft.com/investor/reports/ar24/index.html",
"content": "Highlights from fiscal year 2024..."
}
]
}
```
LiteLLM transforms this to the standard `SearchResponse` format:
- `results[].name` → `SearchResult.title`
- `results[].url` → `SearchResult.url`
- `results[].content` → `SearchResult.snippet`

24
docs/my-website/docs/secret_managers/hashicorp_vault.md
View file

@ -197,3 +197,27 @@ When a Virtual Key is Created / Deleted on LiteLLM, LiteLLM will automatically c
LiteLLM stores secret under the `prefix_for_stored_virtual_keys` path (default: `litellm/`)
<Image img={require('../../img/hcorp_virtual_key.png')} />
### Team-specific overrides
When running the LiteLLM proxy you can override the Vault location per team. Use the [Team-Level Secret Manager Settings](./overview.md#team-level-secret-manager-settings) flow in the dashboard and configure the panel shown below:
<Image img={require('../../img/secret_manager_hashicorp_vault_settings.png')} />
Use the following structure for the JSON payload:
```json
{
"namespace": "teams/team-a",
"mount": "kv-prod",
"path_prefix": "virtual-keys",
"data": "password"
}
```
- `namespace` – overrides the `X-Vault-Namespace` header.
- `mount` – which KV engine mount to use (defaults to `secret`).
- `path_prefix` – additional path segments between the mount and the secret name.
- `data` – the field name inside the KV payload (defaults to `key`).
Whenever LiteLLM stores or deletes virtual keys for that team, these overrides are applied so you can keep each team’s credentials in its own namespace, mount, or field layout without changing the global Vault configuration.

29
docs/my-website/docs/secret_managers/overview.md
View file

@ -1,3 +1,5 @@
import Image from '@theme/IdealImage';
# Secret Managers Overview
:::info
@ -45,3 +47,30 @@ general_settings:
primary_secret_name: "litellm_secrets" # OPTIONAL. Read multiple keys from one JSON secret on AWS Secret Manager
```
## Team-Level Secret Manager Settings
Team-level secret manager settings let every team bring their own key-management configuration. These settings are used when creating virtual keys tied to the team.
Follow these steps to configure it:
1. **Create a team**
Open the Teams page and click `Create Team` to launch the modal.
<Image img={require('../../img/secret_manager_settings_create_team.png')} />
2. **Expand Additional Settings**
Use the `Additional Settings` toggle to reveal the advanced configuration panel.
<Image img={require('../../img/secret_manager_settings_additional_settings.png')} />
3. **Configure the Secret Manager**
In the `Secret Manager Settings` panel, paste the provider-specific JSON. Refer to each provider page (AWS, Azure, Google, Hashicorp, etc.) for the supported keys/values. JSON is required today, but we plan to add a more UI-friendly editor.
<Image img={require('../../img/secret_manager_settings.png')} />
4. **Create the team**
Review the inputs and click `Create Team` to save.
<Image img={require('../../img/secret_manager_settings_create_button.png')} />
Once saved, LiteLLM will use this configuration.

2
docs/my-website/docusaurus.config.js
View file

@ -8,7 +8,7 @@ const darkCodeTheme = require('prism-react-renderer/themes/dracula');
const inkeepConfig = {
baseSettings: {
apiKey: "0cb9c9916ec71bfe0e53c9d7f83ff046daee3fa9ef318f6a",
apiKey: "test-inkeep-api-key-123",
organizationDisplayName: 'liteLLM',
primaryBrandColor: '#4965f5',
theme: {

BIN
docs/my-website/img/secret_manager_hashicorp_vault_settings.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 107 KiB

BIN
docs/my-website/img/secret_manager_settings.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 680 KiB

BIN
docs/my-website/img/secret_manager_settings_additional_settings.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 683 KiB

BIN
docs/my-website/img/secret_manager_settings_create_button.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 691 KiB

BIN
docs/my-website/img/secret_manager_settings_create_team.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 524 KiB

BIN
docs/my-website/img/sentinel.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 918 KiB

5
docs/my-website/sidebars.js
View file

@ -42,6 +42,7 @@ const sidebars = {
label: "Guardrails",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/guardrail_load_balancing",
{
type: "category",
"label": "Contributing to Guardrails",
@ -52,6 +53,7 @@ const sidebars = {
]
},
"proxy/guardrails/test_playground",
"proxy/guardrails/litellm_content_filter",
...[
"proxy/guardrails/aim_security",
"proxy/guardrails/onyx_security",
@ -63,7 +65,6 @@ const sidebars = {
"proxy/guardrails/grayswan",
"proxy/guardrails/hiddenlayer",
"proxy/guardrails/lasso_security",
"proxy/guardrails/litellm_content_filter",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
"proxy/guardrails/model_armor",
@ -544,6 +545,7 @@ const sidebars = {
"search/dataforseo",
"search/firecrawl",
"search/searxng",
"search/linkup",
]
},
"skills",
@ -669,6 +671,7 @@ const sidebars = {
"providers/ai21",
"providers/aiml",
"providers/aleph_alpha",
"providers/amazon_nova",
"providers/anyscale",
"providers/baseten",
"providers/bytez",

BIN
enterprise/dist/litellm_enterprise-0.1.26-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.26.tar.gz vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.27-py3-none-any.whl vendored Normal file
View file

Binary file not shown.

BIN
enterprise/dist/litellm_enterprise-0.1.27.tar.gz vendored Normal file
View file

Binary file not shown.

240
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
View file

@ -5,7 +5,7 @@ Base class for sending emails to user after creating keys or invite links
import json
import os
from typing import List, Optional
from typing import List, Literal, Optional
from litellm_enterprise.types.enterprise_callbacks.send_emails import (
EmailEvent,
@ -15,6 +15,7 @@ from litellm_enterprise.types.enterprise_callbacks.send_emails import (
)
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.email_templates.email_footer import EMAIL_FOOTER
from litellm.integrations.email_templates.key_created_email import (
@ -26,9 +27,17 @@ from litellm.integrations.email_templates.key_rotated_email import (
from litellm.integrations.email_templates.user_invitation_email import (
USER_INVITATION_EMAIL_TEMPLATE,
)
from litellm.proxy._types import InvitationNew, UserAPIKeyAuth, WebhookEvent
from litellm.integrations.email_templates.templates import (
MAX_BUDGET_ALERT_EMAIL_TEMPLATE,
SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
)
from litellm.proxy._types import CallInfo, InvitationNew, UserAPIKeyAuth, WebhookEvent
from litellm.secret_managers.main import get_secret_bool
from litellm.types.integrations.slack_alerting import LITELLM_LOGO_URL
from litellm.constants import (
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
EMAIL_BUDGET_ALERT_TTL,
)
class BaseEmailLogger(CustomLogger):
@ -40,6 +49,21 @@ class BaseEmailLogger(CustomLogger):
EmailEvent.virtual_key_rotated: "LiteLLM: {event_message}",
}
def __init__(
self,
internal_usage_cache: Optional[DualCache] = None,
**kwargs,
):
"""
Initialize BaseEmailLogger
Args:
internal_usage_cache: DualCache instance for preventing duplicate alerts
**kwargs: Additional arguments passed to CustomLogger
"""
super().__init__(**kwargs)
self.internal_usage_cache = internal_usage_cache or DualCache()
async def send_user_invitation_email(self, event: WebhookEvent):
"""
Send email to user after inviting them to the team
@ -154,6 +178,218 @@ class BaseEmailLogger(CustomLogger):
)
pass
async def send_soft_budget_alert_email(self, event: WebhookEvent):
"""
Send email to user when soft budget is crossed
"""
email_params = await self._get_email_params(
email_event=EmailEvent.soft_budget_crossed, # Reuse existing event type for subject template
user_id=event.user_id,
user_email=event.user_email,
event_message=event.event_message,
)
verbose_proxy_logger.debug(
f"send_soft_budget_alert_email_event: {json.dumps(event.model_dump(exclude_none=True), indent=4, default=str)}"
)
# Format budget values
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_info = ""
if event.max_budget is not None:
max_budget_info = f"<b>Maximum Budget:</b> ${event.max_budget} <br />"
email_html_content = SOFT_BUDGET_ALERT_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
recipient_email=email_params.recipient_email,
soft_budget=soft_budget_str,
spend=spend_str,
max_budget_info=max_budget_info,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
)
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
to_email=[email_params.recipient_email],
subject=email_params.subject,
html_body=email_html_content,
)
pass
async def send_max_budget_alert_email(self, event: WebhookEvent):
"""
Send email to user when max budget alert threshold is reached
"""
email_params = await self._get_email_params(
email_event=EmailEvent.max_budget_alert,
user_id=event.user_id,
user_email=event.user_email,
event_message=event.event_message,
)
verbose_proxy_logger.debug(
f"send_max_budget_alert_email_event: {json.dumps(event.model_dump(exclude_none=True), indent=4, default=str)}"
)
# Format budget values
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_str = f"${event.max_budget}" if event.max_budget is not None else "N/A"
# Calculate percentage and alert threshold
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
alert_threshold_str = f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}" if event.max_budget is not None else "N/A"
email_html_content = MAX_BUDGET_ALERT_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
recipient_email=email_params.recipient_email,
percentage=percentage,
spend=spend_str,
max_budget=max_budget_str,
alert_threshold=alert_threshold_str,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
)
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
to_email=[email_params.recipient_email],
subject=email_params.subject,
html_body=email_html_content,
)
pass
async def budget_alerts(
self,
type: Literal[
"token_budget",
"soft_budget",
"max_budget_alert",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
user_info: CallInfo,
):
"""
Send a budget alert via email
Args:
type: The type of budget alert to send
user_info: The user info to send the alert for
"""
## PREVENTITIVE ALERTING ##
# - Alert once within 24hr period
# - Cache this information
# - Don't re-alert, if alert already sent
_cache: DualCache = self.internal_usage_cache
# percent of max_budget left to spend
if user_info.max_budget is None and user_info.soft_budget is None:
return
# For soft_budget alerts, check if we've already sent an alert
if type == "soft_budget":
if user_info.soft_budget is not None and user_info.spend >= user_info.soft_budget:
# Generate cache key based on event type and identifier
_id = user_info.token or user_info.user_id or "default_id"
_cache_key = f"email_budget_alerts:soft_budget_crossed:{_id}"
# Check if we've already sent this alert
result = await _cache.async_get_cache(key=_cache_key)
if result is None:
# Create WebhookEvent for soft budget alert
event_message = f"Soft Budget Crossed - Total Soft Budget: ${user_info.soft_budget}"
webhook_event = WebhookEvent(
event="soft_budget_crossed",
event_message=event_message,
spend=user_info.spend,
max_budget=user_info.max_budget,
soft_budget=user_info.soft_budget,
token=user_info.token,
customer_id=user_info.customer_id,
user_id=user_info.user_id,
team_id=user_info.team_id,
team_alias=user_info.team_alias,
organization_id=user_info.organization_id,
user_email=user_info.user_email,
key_alias=user_info.key_alias,
projected_exceeded_date=user_info.projected_exceeded_date,
projected_spend=user_info.projected_spend,
event_group=user_info.event_group,
)
try:
await self.send_soft_budget_alert_email(webhook_event)
# Cache the alert to prevent duplicate sends
await _cache.async_set_cache(
key=_cache_key,
value="SENT",
ttl=EMAIL_BUDGET_ALERT_TTL,
)
except Exception as e:
verbose_proxy_logger.error(
f"Error sending soft budget alert email: {e}",
exc_info=True,
)
return
# For max_budget_alert, check if we've already sent an alert
if type == "max_budget_alert":
if user_info.max_budget is not None and user_info.spend is not None:
alert_threshold = user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
# Only alert if we've crossed the threshold but haven't exceeded max_budget yet
if user_info.spend >= alert_threshold and user_info.spend < user_info.max_budget:
# Generate cache key based on event type and identifier
_id = user_info.token or user_info.user_id or "default_id"
_cache_key = f"email_budget_alerts:max_budget_alert:{_id}"
# Check if we've already sent this alert
result = await _cache.async_get_cache(key=_cache_key)
if result is None:
# Calculate percentage
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
# Create WebhookEvent for max budget alert
event_message = f"Max Budget Alert - {percentage}% of Maximum Budget Reached"
webhook_event = WebhookEvent(
event="max_budget_alert",
event_message=event_message,
spend=user_info.spend,
max_budget=user_info.max_budget,
soft_budget=user_info.soft_budget,
token=user_info.token,
customer_id=user_info.customer_id,
user_id=user_info.user_id,
team_id=user_info.team_id,
team_alias=user_info.team_alias,
organization_id=user_info.organization_id,
user_email=user_info.user_email,
key_alias=user_info.key_alias,
projected_exceeded_date=user_info.projected_exceeded_date,
projected_spend=user_info.projected_spend,
event_group=user_info.event_group,
)
try:
await self.send_max_budget_alert_email(webhook_event)
# Cache the alert to prevent duplicate sends
await _cache.async_set_cache(
key=_cache_key,
value="SENT",
ttl=EMAIL_BUDGET_ALERT_TTL,
)
except Exception as e:
verbose_proxy_logger.error(
f"Error sending max budget alert email: {e}",
exc_info=True,
)
return
async def _get_email_params(
self,
email_event: EmailEvent,

3
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/resend_email.py
View file

@ -19,7 +19,8 @@ RESEND_API_ENDPOINT = "https://api.resend.com/emails"
class ResendEmailLogger(BaseEmailLogger):
def __init__(self):
def __init__(self, internal_usage_cache=None, **kwargs):
super().__init__(internal_usage_cache=internal_usage_cache, **kwargs)
self.async_httpx_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)

3
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/sendgrid_email.py
View file

@ -27,7 +27,8 @@ class SendGridEmailLogger(BaseEmailLogger):
- SENDGRID_API_KEY
"""
def __init__(self):
def __init__(self, internal_usage_cache=None, **kwargs):
super().__init__(internal_usage_cache=internal_usage_cache, **kwargs)
self.async_httpx_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)

3
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/smtp_email.py
View file

@ -21,7 +21,8 @@ class SMTPEmailLogger(BaseEmailLogger):
- SMTP_SENDER_EMAIL
"""
def __init__(self):
def __init__(self, internal_usage_cache=None, **kwargs):
super().__init__(internal_usage_cache=internal_usage_cache, **kwargs)
verbose_logger.debug("SMTP Email Logger initialized....")
async def send_email(

110
enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py Normal file
View file

@ -0,0 +1,110 @@
"""
Polls LiteLLM_ManagedObjectTable to check if the response is complete.
Cost tracking is handled automatically by litellm.aget_responses().
"""
from typing import TYPE_CHECKING
import litellm
from litellm._logging import verbose_proxy_logger
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
class CheckResponsesCost:
def __init__(
self,
proxy_logging_obj: "ProxyLogging",
prisma_client: "PrismaClient",
llm_router: "Router",
):
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
async def check_responses_cost(self):
"""
Check if background responses are complete and track their cost.
- Get all status="queued" or "in_progress" and file_purpose="response" jobs
- Query the provider to check if response is complete
- Cost is automatically tracked by litellm.aget_responses()
- Mark completed/failed/cancelled responses as complete in the database
"""
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"status": {"in": ["queued", "in_progress"]},
"file_purpose": "response",
}
)
verbose_proxy_logger.debug(f"Found {len(jobs)} response jobs to check")
completed_jobs = []
for job in jobs:
unified_object_id = job.unified_object_id
try:
from litellm.proxy.hooks.responses_id_security import (
ResponsesIDSecurity,
)
# Get the stored response object to extract model information
stored_response = job.file_object
model_name = stored_response.get("model", None)
# Decrypt the response ID
responses_id_security, _, _ = ResponsesIDSecurity()._decrypt_response_id(unified_object_id)
# Prepare metadata with model information for cost tracking
litellm_metadata = {
"user_api_key_user_id": job.created_by or "default-user-id",
}
# Add model information if available
if model_name:
litellm_metadata["model"] = model_name
litellm_metadata["model_group"] = model_name # Use same value for model_group
response = await litellm.aget_responses(
response_id=responses_id_security,
litellm_metadata=litellm_metadata,
)
verbose_proxy_logger.debug(
f"Response {unified_object_id} status: {response.status}, model: {model_name}"
)
except Exception as e:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} due to error: {e}"
)
continue
# Check if response is in a terminal state
if response.status == "completed":
verbose_proxy_logger.info(
f"Response {unified_object_id} is complete. Cost automatically tracked by aget_responses."
)
completed_jobs.append(job)
elif response.status in ["failed", "cancelled"]:
verbose_proxy_logger.info(
f"Response {unified_object_id} has status {response.status}, marking as complete"
)
completed_jobs.append(job)
# Mark completed jobs in the database
if len(completed_jobs) > 0:
await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={"id": {"in": [job.id for job in completed_jobs]}},
data={"status": "completed"},
)
verbose_proxy_logger.info(
f"Marked {len(completed_jobs)} response jobs as completed"
)

15
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -23,7 +23,9 @@ from litellm.proxy._types import (
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_batch_id_from_unified_batch_id,
get_content_type_from_file_object,
get_model_id_from_unified_batch_id,
normalize_mime_type_for_provider,
)
from litellm.types.llms.openai import (
AllMessageValues,
@ -33,6 +35,7 @@ from litellm.types.llms.openai import (
FileObject,
OpenAIFileObject,
OpenAIFilesPurpose,
ResponsesAPIResponse,
)
from litellm.types.utils import (
CallTypesLiteral,
@ -41,10 +44,6 @@ from litellm.types.utils import (
LLMResponseTypes,
SpecialEnums,
)
from litellm.proxy.openai_files_endpoints.common_utils import (
get_content_type_from_file_object,
normalize_mime_type_for_provider,
)
if TYPE_CHECKING:
from litellm.types.llms.openai import HttpxBinaryResponseContent
@ -133,10 +132,10 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
async def store_unified_object_id(
self,
unified_object_id: str,
file_object: Union[LiteLLMBatch, LiteLLMFineTuningJob],
file_object: Union[LiteLLMBatch, LiteLLMFineTuningJob, "ResponsesAPIResponse"],
litellm_parent_otel_span: Optional[Span],
model_object_id: str,
file_purpose: Literal["batch", "fine-tune"],
file_purpose: Literal["batch", "fine-tune", "response"],
user_api_key_dict: UserAPIKeyAuth,
) -> None:
verbose_logger.info(
@ -946,7 +945,9 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
# File is stored in a storage backend, download and convert to base64
try:
from litellm.llms.base_llm.files.storage_backend_factory import get_storage_backend
from litellm.llms.base_llm.files.storage_backend_factory import (
get_storage_backend,
)
storage_backend_name = db_file.storage_backend
storage_url = db_file.storage_url

4
enterprise/litellm_enterprise/types/enterprise_callbacks/send_emails.py
View file

@ -36,6 +36,8 @@ class EmailEvent(str, enum.Enum):
virtual_key_created = "Virtual Key Created"
new_user_invitation = "New User Invitation"
virtual_key_rotated = "Virtual Key Rotated"
soft_budget_crossed = "Soft Budget Crossed"
max_budget_alert = "Max Budget Alert"
class EmailEventSettings(BaseModel):
event: EmailEvent
@ -51,6 +53,8 @@ class DefaultEmailSettings(BaseModel):
EmailEvent.virtual_key_created: True, # On by default
EmailEvent.new_user_invitation: True, # On by default
EmailEvent.virtual_key_rotated: True, # On by default
EmailEvent.soft_budget_crossed: True, # On by default
EmailEvent.max_budget_alert: True, # On by default
}
)
def to_dict(self) -> Dict[str, bool]:

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-enterprise"
version = "0.1.25"
version = "0.1.27"
description = "Package for LiteLLM Enterprise features"
authors = ["BerriAI"]
readme = "README.md"
@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "0.1.25"
version = "0.1.27"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-enterprise==",

18
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -727,4 +727,22 @@ model LiteLLM_UISettings {
ui_settings Json
created_at DateTime @default(now())
updated_at DateTime @updatedAt
}
// Skills table for storing LiteLLM-managed skills
model LiteLLM_SkillsTable {
skill_id String @id @default(uuid())
display_title String?
description String?
instructions String? // The skill instructions/prompt (from SKILL.md)
source String @default("custom") // "custom" or "anthropic"
latest_version String?
file_content Bytes? // Binary content of the skill files (zip)
file_name String? // Original filename
file_type String? // MIME type (e.g., "application/zip")
metadata Json? @default("{}")
created_at DateTime @default(now())
created_by String?
updated_at DateTime @default(now()) @updatedAt
updated_by String?
}

15
litellm/__init__.py
View file

@ -134,6 +134,7 @@ _custom_logger_compatible_callbacks_literal = Literal[
"weave_otel",
"pagerduty",
"humanloop",
"azure_sentinel",
"gcs_pubsub",
"agentops",
"anthropic_cache_control_hook",
@ -557,6 +558,8 @@ ovhcloud_embedding_models: Set = set()
lemonade_models: Set = set()
docker_model_runner_models: Set = set()
amazon_nova_models: Set = set()
stability_models: Set = set()
github_copilot_models: Set = set()
def is_bedrock_pricing_only_model(key: str) -> bool:
@ -801,6 +804,10 @@ def add_known_models():
docker_model_runner_models.add(key)
elif value.get("litellm_provider") == "amazon_nova":
amazon_nova_models.add(key)
elif value.get("litellm_provider") == "stability":
stability_models.add(key)
elif value.get("litellm_provider") == "github_copilot":
github_copilot_models.add(key)
add_known_models()
@ -1003,6 +1010,8 @@ models_by_provider: dict = {
"lemonade": lemonade_models,
"clarifai": clarifai_models,
"amazon_nova": amazon_nova_models,
"stability": stability_models,
"github_copilot": github_copilot_models,
}
# mapping for those models which have larger equivalents
@ -1194,9 +1203,9 @@ from .llms.bedrock.chat.invoke_transformations.amazon_openai_transformation impo
AmazonBedrockOpenAIConfig,
)
from .llms.bedrock.image.amazon_stability1_transformation import AmazonStabilityConfig
from .llms.bedrock.image.amazon_stability3_transformation import AmazonStability3Config
from .llms.bedrock.image.amazon_nova_canvas_transformation import AmazonNovaCanvasConfig
from .llms.bedrock.image_generation.amazon_stability1_transformation import AmazonStabilityConfig
from .llms.bedrock.image_generation.amazon_stability3_transformation import AmazonStability3Config
from .llms.bedrock.image_generation.amazon_nova_canvas_transformation import AmazonNovaCanvasConfig
from .llms.bedrock.embed.amazon_titan_g1_transformation import AmazonTitanG1Config
from .llms.bedrock.embed.amazon_titan_multimodal_transformation import (
AmazonTitanMultimodalEmbeddingG1Config,

5
litellm/anthropic_interface/messages/__init__.py
View file

@ -37,6 +37,7 @@ async def acreate(
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
container: Optional[Dict] = None,
**kwargs
) -> Union[AnthropicMessagesResponse, AsyncIterator]:
"""
@ -56,6 +57,7 @@ async def acreate(
tools (List[Dict], optional): List of tool definitions
top_k (int, optional): Top K sampling parameter
top_p (float, optional): Nucleus sampling parameter
container (Dict, optional): Container config with skills for code execution
**kwargs: Additional arguments
Returns:
@ -75,6 +77,7 @@ async def acreate(
tools=tools,
top_k=top_k,
top_p=top_p,
container=container,
**kwargs,
)
@ -93,6 +96,7 @@ def create(
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
container: Optional[Dict] = None,
**kwargs
) -> Union[
AnthropicMessagesResponse,
@ -135,5 +139,6 @@ def create(
tools=tools,
top_k=top_k,
top_p=top_p,
container=container,
**kwargs,
)

71
litellm/caching/redis_cache.py
View file

@ -10,6 +10,7 @@ Has 4 primary methods:
import ast
import asyncio
import hashlib
import inspect
import json
import time
@ -145,9 +146,17 @@ class RedisCache(BaseCache):
except Exception:
pass
### ASYNC HEALTH PING ###
self._setup_health_pings()
if litellm.default_redis_ttl is not None:
super().__init__(default_ttl=int(litellm.default_redis_ttl))
else:
super().__init__() # defaults to 60s
def _setup_health_pings(self):
"""Setup async and sync health pings for Redis."""
# ASYNC HEALTH PING
try:
# asyncio.get_running_loop().create_task(self.ping())
_ = asyncio.get_running_loop().create_task(self.ping())
except Exception as e:
if "no running event loop" in str(e):
@ -159,8 +168,9 @@ class RedisCache(BaseCache):
"Error connecting to Async Redis client - {}".format(str(e)),
extra={"error": str(e)},
)
self._handle_async_ping_error(e)
### SYNC HEALTH PING ###
# SYNC HEALTH PING
try:
if hasattr(self.redis_client, "ping"):
self.redis_client.ping() # type: ignore
@ -168,11 +178,53 @@ class RedisCache(BaseCache):
verbose_logger.error(
"Error connecting to Sync Redis client", extra={"error": str(e)}
)
self._handle_sync_ping_error(e)
if litellm.default_redis_ttl is not None:
super().__init__(default_ttl=int(litellm.default_redis_ttl))
else:
super().__init__() # defaults to 60s
def _handle_async_ping_error(self, e: Exception):
"""Handle async ping error with service failure hook."""
try:
loop = asyncio.get_running_loop()
start_time = time.time()
end_time = start_time
loop.create_task(
self.service_logger_obj.async_service_failure_hook(
service=ServiceTypes.REDIS,
duration=end_time - start_time,
error=e,
call_type="redis_async_ping",
)
)
except Exception:
pass
def _handle_sync_ping_error(self, e: Exception):
"""Handle sync ping error with service failure hook."""
try:
loop = asyncio.get_running_loop()
start_time = time.time()
end_time = start_time
loop.create_task(
self.service_logger_obj.async_service_failure_hook(
service=ServiceTypes.REDIS,
duration=end_time - start_time,
error=e,
call_type="redis_sync_ping",
)
)
except Exception:
pass
def _get_async_client_cache_key(self) -> str:
"""
Generate a cache key for the async Redis client based on connection parameters.
This ensures different Redis configurations use different cached clients.
"""
# Create a stable representation of redis_kwargs for hashing
# Sort keys to ensure consistent hash regardless of parameter order
sorted_kwargs = sorted(self.redis_kwargs.items())
kwargs_str = json.dumps(sorted_kwargs, sort_keys=True)
kwargs_hash = hashlib.sha256(kwargs_str.encode()).hexdigest()[:16]
return f"async-redis-client-{kwargs_hash}"
def init_async_client(
self,
@ -181,7 +233,8 @@ class RedisCache(BaseCache):
from .._redis import get_redis_async_client, get_redis_connection_pool
cached_client = in_memory_llm_clients_cache.get_cache(key="async-redis-client")
cache_key = self._get_async_client_cache_key()
cached_client = in_memory_llm_clients_cache.get_cache(key=cache_key)
if cached_client is not None:
redis_async_client = cast(
Union[async_redis_client, async_redis_cluster_client], cached_client
@ -193,7 +246,7 @@ class RedisCache(BaseCache):
connection_pool=self.async_redis_conn_pool, **self.redis_kwargs
)
in_memory_llm_clients_cache.set_cache(
key="async-redis-client", value=redis_async_client
key=cache_key, value=redis_async_client
)
self.redis_async_client = redis_async_client # type: ignore

63
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -167,24 +167,28 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
)
elif role == "tool":
# Convert tool message to function call output format
# Transform content to responses format (handles str, list, and other types)
# _convert_content_to_responses_format always returns List[Dict[str, Any]]
# The Responses API expects 'output' to be a string, not a list
if content is None:
transformed_output: list[dict[str, Any]] = []
elif isinstance(content, (str, list)):
transformed_output = self._convert_content_to_responses_format(
content, "tool"
)
output_str = ""
elif isinstance(content, str):
output_str = content
elif isinstance(content, list):
# If content is a list, extract text parts and join them
text_parts = []
for item in content:
if isinstance(item, str):
text_parts.append(item)
elif isinstance(item, dict) and item.get("type") == "text":
text_parts.append(item.get("text", ""))
output_str = " ".join(text_parts) if text_parts else str(content)
else:
# Fallback: convert unexpected types to string first
transformed_output = self._convert_content_to_responses_format(
str(content), "tool"
)
# Fallback: convert unexpected types to string
output_str = str(content)
input_items.append(
{
"type": "function_call_output",
"call_id": tool_call_id,
"output": transformed_output,
"output": output_str,
}
)
elif role == "assistant" and tool_calls and isinstance(tool_calls, list):
@ -345,6 +349,11 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
index = 0
reasoning_content: Optional[str] = None
# Collect all tool calls to put them in a single choice
# (Chat Completions API expects all tool calls in one message)
accumulated_tool_calls: List[Dict[str, Any]] = []
tool_call_index = 0
for item in output_items:
if isinstance(item, ResponseReasoningItem):
for summary_item in item.summary:
@ -378,20 +387,10 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
tool_call_dict = LiteLLMCompletionResponsesConfig.convert_response_function_tool_call_to_chat_completion_tool_call(
tool_call_item=item,
index=index,
index=tool_call_index,
)
msg = Message(
content=None,
tool_calls=[tool_call_dict],
reasoning_content=reasoning_content,
)
choices.append(
Choices(message=msg, finish_reason="tool_calls", index=index)
)
reasoning_content = None # flush reasoning content
index += 1
accumulated_tool_calls.append(tool_call_dict)
tool_call_index += 1
elif isinstance(item, dict) and handle_raw_dict_callback is not None:
# Handle raw dict responses (e.g., from GPT-5 Codex)
@ -401,6 +400,18 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
else:
pass # don't fail request if item in list is not supported
# If we accumulated tool calls, create a single choice with all of them
if accumulated_tool_calls:
msg = Message(
content=None,
tool_calls=accumulated_tool_calls,
reasoning_content=reasoning_content,
)
choices.append(
Choices(message=msg, finish_reason="tool_calls", index=index)
)
reasoning_content = None
return choices
def transform_response( # noqa: PLR0915
@ -492,7 +503,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _convert_content_str_to_input_text(
self, content: str, role: str
) -> Dict[str, Any]:
if role == "user" or role == "system":
if role == "user" or role == "system" or role == "tool":
return {"type": "input_text", "text": content}
else:
return {"type": "output_text", "text": content}

3
litellm/constants.py
View file

@ -313,6 +313,8 @@ DD_TRACER_STREAMING_CHUNK_YIELD_RESOURCE = os.getenv(
"DD_TRACER_STREAMING_CHUNK_YIELD_RESOURCE", "streaming.chunk.yield"
)
EMAIL_BUDGET_ALERT_TTL = int(os.getenv("EMAIL_BUDGET_ALERT_TTL", 24 * 60 * 60)) # 24 hours in seconds
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE = float(os.getenv("EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE", 0.8)) # 80% of max budget
############### LLM Provider Constants ###############
### ANTHROPIC CONSTANTS ###
ANTHROPIC_SKILLS_API_BETA_VERSION = "skills-2025-10-02"
@ -890,6 +892,7 @@ BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"qwen2",
"twelvelabs",
"openai",
"stability",
]
BEDROCK_EMBEDDING_PROVIDERS_LITERAL = Literal[

65
litellm/images/main.py
View file

@ -33,6 +33,7 @@ from litellm.main import (
base_llm_aiohttp_handler,
base_llm_http_handler,
bedrock_image_generation,
bedrock_image_edit,
openai_chat_completions,
openai_image_variations,
)
@ -670,7 +671,7 @@ def image_variation(
@client
def image_edit(
def image_edit( # noqa: PLR0915
image: Union[FileTypes, List[FileTypes]],
prompt: str,
model: Optional[str] = None,
@ -695,6 +696,29 @@ def image_edit(
"""
local_vars = locals()
try:
openai_params = [
"user",
"request_timeout",
"api_base",
"api_version",
"api_key",
"deployment_id",
"organization",
"base_url",
"default_headers",
"timeout",
"max_retries",
"n",
"quality",
"size",
"style",
"async_call",
]
litellm_params_list = all_litellm_params
default_params = openai_params + litellm_params_list
non_default_params = {
k: v for k, v in kwargs.items() if k not in default_params
} # model-specific params - pass them straight to the model/provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id", None)
_is_async = kwargs.pop("async_call", False) is True
@ -788,13 +812,14 @@ def image_edit(
image_edit_optional_params: ImageEditOptionalRequestParams = (
_get_ImageEditRequestUtils().get_requested_image_edit_optional_param(local_vars)
)
# Get optional parameters for the responses API
image_edit_request_params: Dict = (
_get_ImageEditRequestUtils().get_optional_params_image_edit(
model=model,
image_edit_provider_config=image_edit_provider_config,
image_edit_optional_params=image_edit_optional_params,
drop_params=kwargs.get("drop_params"),
additional_drop_params=kwargs.get("additional_drop_params"),
)
)
@ -810,6 +835,42 @@ def image_edit(
custom_llm_provider=custom_llm_provider,
)
# Route bedrock to its specific handler (AWS signing required)
if custom_llm_provider == "bedrock":
if model is None:
raise Exception("Model needs to be set for bedrock")
image_edit_request_params.update(non_default_params)
return bedrock_image_edit.image_edit( # type: ignore
model=model,
image=images,
prompt=prompt,
timeout=timeout,
logging_obj=litellm_logging_obj,
optional_params=image_edit_request_params,
model_response=ImageResponse(),
aimage_edit=_is_async,
client=kwargs.get("client"),
api_base=kwargs.get("api_base"),
extra_headers=extra_headers,
api_key=kwargs.get("api_key"),
)
elif custom_llm_provider == "stability":
image_edit_request_params.update(non_default_params)
return base_llm_http_handler.image_edit_handler(
model=model,
image=images,
prompt=prompt,
image_edit_provider_config=image_edit_provider_config,
image_edit_optional_request_params=image_edit_request_params,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
logging_obj=litellm_logging_obj,
extra_headers=extra_headers,
extra_body=extra_body,
timeout=timeout or DEFAULT_REQUEST_TIMEOUT,
_is_async=_is_async,
client=kwargs.get("client"),
)
# Call the handler with _is_async flag instead of directly calling the async handler
return base_llm_http_handler.image_edit_handler(
model=model,

41
litellm/images/utils.py
View file

@ -1,5 +1,5 @@
from io import BufferedReader, BytesIO
from typing import Any, Dict, cast, get_type_hints
from typing import Any, Dict, List, Optional, cast, get_type_hints
import litellm
from litellm.litellm_core_utils.token_counter import get_image_type
@ -14,41 +14,53 @@ class ImageEditRequestUtils:
model: str,
image_edit_provider_config: BaseImageEditConfig,
image_edit_optional_params: ImageEditOptionalRequestParams,
drop_params: Optional[bool] = None,
additional_drop_params: Optional[List[str]] = None,
) -> Dict:
"""
Get optional parameters for the image edit API.
Args:
params: Dictionary of all parameters
model: The model name
image_edit_provider_config: The provider configuration for image edit API
image_edit_optional_params: The optional parameters for the image edit API
drop_params: If True, silently drop unsupported parameters instead of raising
additional_drop_params: List of additional parameter names to drop
Returns:
A dictionary of supported parameters for the image edit API
"""
# Remove None values and internal parameters
# Get supported parameters for the model
supported_params = image_edit_provider_config.get_supported_openai_params(model)
# Check for unsupported parameters
should_drop = litellm.drop_params is True or drop_params is True
filtered_optional_params = dict(image_edit_optional_params)
if additional_drop_params:
for param in additional_drop_params:
filtered_optional_params.pop(param, None)
unsupported_params = [
param
for param in image_edit_optional_params
for param in filtered_optional_params
if param not in supported_params
]
if unsupported_params:
raise litellm.UnsupportedParamsError(
model=model,
message=f"The following parameters are not supported for model {model}: {', '.join(unsupported_params)}",
)
if should_drop:
for param in unsupported_params:
filtered_optional_params.pop(param, None)
else:
raise litellm.UnsupportedParamsError(
model=model,
message=f"The following parameters are not supported for model {model}: {', '.join(unsupported_params)}",
)
# Map parameters to provider-specific format
mapped_params = image_edit_provider_config.map_openai_params(
image_edit_optional_params=image_edit_optional_params,
image_edit_optional_params=cast(
ImageEditOptionalRequestParams, filtered_optional_params
),
model=model,
drop_params=litellm.drop_params,
drop_params=should_drop,
)
return mapped_params
@ -70,7 +82,6 @@ class ImageEditRequestUtils:
filtered_params = {
k: v for k, v in params.items() if k in valid_keys and v is not None
}
return cast(ImageEditOptionalRequestParams, filtered_params)
@staticmethod

4
litellm/integrations/SlackAlerting/budget_alert_types.py
View file

@ -77,8 +77,9 @@ class ProjectedLimitExceededAlert(BaseBudgetAlertType):
def get_budget_alert_type(
type: Literal[
"token_budget",
"soft_budget",
"user_budget",
"soft_budget",
"max_budget_alert",
"team_budget",
"organization_budget",
"proxy_budget",
@ -91,6 +92,7 @@ def get_budget_alert_type(
"proxy_budget": ProxyBudgetAlert(),
"soft_budget": SoftBudgetAlert(),
"user_budget": UserBudgetAlert(),
"max_budget_alert": TokenBudgetAlert(),
"team_budget": TeamBudgetAlert(),
"organization_budget": OrganizationBudgetAlert(),
"token_budget": TokenBudgetAlert(),

3
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -531,8 +531,9 @@ class SlackAlerting(CustomBatchLogger):
self,
type: Literal[
"token_budget",
"soft_budget",
"user_budget",
"soft_budget",
"max_budget_alert",
"team_budget",
"organization_budget",
"proxy_budget",

42
litellm/integrations/arize/arize_phoenix.py
View file

@ -1,12 +1,10 @@
import os
from typing import TYPE_CHECKING, Any, Optional, Union
from datetime import datetime
from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
from litellm.integrations.arize._utils import ArizeOTELAttributes
from litellm.types.integrations.arize_phoenix import ArizePhoenixConfig
from litellm.types.services import ServiceLoggerPayload
from litellm.integrations.opentelemetry import OpenTelemetry
if TYPE_CHECKING:
@ -35,13 +33,19 @@ class ArizePhoenixLogger(OpenTelemetry):
@staticmethod
def set_arize_phoenix_attributes(span: Span, kwargs, response_obj):
_utils.set_attributes(span, kwargs, response_obj, ArizeOTELAttributes)
# Set project name on the span for all traces to go to custom Phoenix projects
config = ArizePhoenixLogger.get_arize_phoenix_config()
if config.project_name:
from litellm.integrations.opentelemetry_utils.base_otel_llm_obs_attributes import safe_set_attribute
safe_set_attribute(span, "openinference.project.name", config.project_name)
return
@staticmethod
def get_arize_phoenix_config() -> ArizePhoenixConfig:
"""
Retrieves the Arize Phoenix configuration based on environment variables.
Returns:
ArizePhoenixConfig: A Pydantic model containing Arize Phoenix configuration.
"""
@ -95,7 +99,7 @@ class ArizePhoenixLogger(OpenTelemetry):
"PHOENIX_API_KEY must be set when using Phoenix Cloud (app.phoenix.arize.com)."
)
project_name = os.environ.get("PHOENIX_PROJECT_NAME", "litellm-project")
project_name = os.environ.get("PHOENIX_PROJECT_NAME", "default")
return ArizePhoenixConfig(
otlp_auth_headers=otlp_auth_headers,
@ -103,34 +107,8 @@ class ArizePhoenixLogger(OpenTelemetry):
endpoint=endpoint,
project_name=project_name,
)
async def async_service_success_hook(
self,
payload: ServiceLoggerPayload,
parent_otel_span: Optional[Span] = None,
start_time: Optional[Union[datetime, float]] = None,
end_time: Optional[Union[datetime, float]] = None,
event_metadata: Optional[dict] = None,
):
pass # suppress additional spans
async def async_service_failure_hook(
self,
payload: ServiceLoggerPayload,
error: Optional[str] = "",
parent_otel_span: Optional[Span] = None,
start_time: Optional[Union[datetime, float]] = None,
end_time: Optional[Union[float, datetime]] = None,
event_metadata: Optional[dict] = None,
):
pass # suppress additional spans
def create_litellm_proxy_request_started_span(
self,
start_time: datetime,
headers: dict,
):
pass # suppress additional spans
## cannot suppress additional proxy server spans, removed previous methods.
async def async_health_check(self):

4
litellm/integrations/azure_sentinel/__init__.py Normal file
View file

@ -0,0 +1,4 @@
from litellm.integrations.azure_sentinel.azure_sentinel import AzureSentinelLogger
__all__ = ["AzureSentinelLogger"]

304
litellm/integrations/azure_sentinel/azure_sentinel.py Normal file
View file

@ -0,0 +1,304 @@
"""
Azure Sentinel Integration - sends logs to Azure Log Analytics using Logs Ingestion API
Azure Sentinel uses Log Analytics workspaces for data storage. This integration sends
LiteLLM logs to the Log Analytics workspace using the Azure Monitor Logs Ingestion API.
Reference API: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview
`async_log_success_event` - used by litellm proxy to send logs to Azure Sentinel
`async_log_failure_event` - used by litellm proxy to send failure logs to Azure Sentinel
For batching specific details see CustomBatchLogger class
"""
import asyncio
import os
import traceback
from typing import List, Optional
from litellm._logging import verbose_logger
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.types.utils import StandardLoggingPayload
class AzureSentinelLogger(CustomBatchLogger):
"""
Logger that sends LiteLLM logs to Azure Sentinel via Azure Monitor Logs Ingestion API
"""
def __init__(
self,
dcr_immutable_id: Optional[str] = None,
stream_name: Optional[str] = None,
endpoint: Optional[str] = None,
tenant_id: Optional[str] = None,
client_id: Optional[str] = None,
client_secret: Optional[str] = None,
**kwargs,
):
"""
Initialize Azure Sentinel logger using Logs Ingestion API
Args:
dcr_immutable_id (str, optional): Data Collection Rule (DCR) Immutable ID.
If not provided, will use AZURE_SENTINEL_DCR_IMMUTABLE_ID env var.
stream_name (str, optional): Stream name from DCR (e.g., "Custom-LiteLLM").
If not provided, will use AZURE_SENTINEL_STREAM_NAME env var or default to "Custom-LiteLLM".
endpoint (str, optional): Data Collection Endpoint (DCE) or DCR ingestion endpoint.
If not provided, will use AZURE_SENTINEL_ENDPOINT env var.
tenant_id (str, optional): Azure Tenant ID for OAuth2 authentication.
If not provided, will use AZURE_SENTINEL_TENANT_ID or AZURE_TENANT_ID env var.
client_id (str, optional): Azure Client ID (Application ID) for OAuth2 authentication.
If not provided, will use AZURE_SENTINEL_CLIENT_ID or AZURE_CLIENT_ID env var.
client_secret (str, optional): Azure Client Secret for OAuth2 authentication.
If not provided, will use AZURE_SENTINEL_CLIENT_SECRET or AZURE_CLIENT_SECRET env var.
"""
self.async_httpx_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)
self.dcr_immutable_id = (
dcr_immutable_id or os.getenv("AZURE_SENTINEL_DCR_IMMUTABLE_ID")
)
self.stream_name = stream_name or os.getenv(
"AZURE_SENTINEL_STREAM_NAME", "Custom-LiteLLM"
)
self.endpoint = endpoint or os.getenv("AZURE_SENTINEL_ENDPOINT")
self.tenant_id = tenant_id or os.getenv("AZURE_SENTINEL_TENANT_ID") or os.getenv(
"AZURE_TENANT_ID"
)
self.client_id = client_id or os.getenv("AZURE_SENTINEL_CLIENT_ID") or os.getenv(
"AZURE_CLIENT_ID"
)
self.client_secret = (
client_secret
or os.getenv("AZURE_SENTINEL_CLIENT_SECRET")
or os.getenv("AZURE_CLIENT_SECRET")
)
if not self.dcr_immutable_id:
raise ValueError(
"AZURE_SENTINEL_DCR_IMMUTABLE_ID is required. Set it as an environment variable or pass dcr_immutable_id parameter."
)
if not self.endpoint:
raise ValueError(
"AZURE_SENTINEL_ENDPOINT is required. Set it as an environment variable or pass endpoint parameter."
)
if not self.tenant_id:
raise ValueError(
"AZURE_SENTINEL_TENANT_ID or AZURE_TENANT_ID is required. Set it as an environment variable or pass tenant_id parameter."
)
if not self.client_id:
raise ValueError(
"AZURE_SENTINEL_CLIENT_ID or AZURE_CLIENT_ID is required. Set it as an environment variable or pass client_id parameter."
)
if not self.client_secret:
raise ValueError(
"AZURE_SENTINEL_CLIENT_SECRET or AZURE_CLIENT_SECRET is required. Set it as an environment variable or pass client_secret parameter."
)
# Build API endpoint: {Endpoint}/dataCollectionRules/{DCR Immutable ID}/streams/{Stream Name}?api-version=2023-01-01
self.api_endpoint = (
f"{self.endpoint.rstrip('/')}/dataCollectionRules/{self.dcr_immutable_id}/streams/{self.stream_name}?api-version=2023-01-01"
)
# OAuth2 scope for Azure Monitor
self.oauth_scope = "https://monitor.azure.com/.default"
self.oauth_token: Optional[str] = None
self.oauth_token_expires_at: Optional[float] = None
self.flush_lock = asyncio.Lock()
super().__init__(**kwargs, flush_lock=self.flush_lock)
asyncio.create_task(self.periodic_flush())
self.log_queue: List[StandardLoggingPayload] = []
async def _get_oauth_token(self) -> str:
"""
Get OAuth2 Bearer token for Azure Monitor Logs Ingestion API
Returns:
Bearer token string
"""
# Check if we have a valid cached token
import time
if (
self.oauth_token
and self.oauth_token_expires_at
and time.time() < self.oauth_token_expires_at - 60
): # Refresh 60 seconds before expiry
return self.oauth_token
# Get new token using client credentials flow
assert self.tenant_id is not None, "tenant_id is required"
assert self.client_id is not None, "client_id is required"
assert self.client_secret is not None, "client_secret is required"
token_url = f"https://login.microsoftonline.com/{self.tenant_id}/oauth2/v2.0/token"
token_data = {
"client_id": self.client_id,
"client_secret": self.client_secret,
"scope": self.oauth_scope,
"grant_type": "client_credentials",
}
response = await self.async_httpx_client.post(
url=token_url,
data=token_data,
headers={"Content-Type": "application/x-www-form-urlencoded"},
)
if response.status_code != 200:
raise Exception(
f"Failed to get OAuth2 token: {response.status_code} - {response.text}"
)
token_response = response.json()
self.oauth_token = token_response.get("access_token")
expires_in = token_response.get("expires_in", 3600)
if not self.oauth_token:
raise Exception("OAuth2 token response did not contain access_token")
# Cache token expiry time
import time
self.oauth_token_expires_at = time.time() + expires_in
return self.oauth_token
async def async_log_success_event(
self, kwargs, response_obj, start_time, end_time
):
"""
Async Log success events to Azure Sentinel
- Gets StandardLoggingPayload from kwargs
- Adds to batch queue
- Flushes based on CustomBatchLogger settings
Raises:
Raises a NON Blocking verbose_logger.exception if an error occurs
"""
try:
verbose_logger.debug(
"Azure Sentinel: Logging - Enters logging function for model %s", kwargs
)
standard_logging_payload = kwargs.get("standard_logging_object", None)
if standard_logging_payload is None:
verbose_logger.warning(
"Azure Sentinel: standard_logging_object not found in kwargs"
)
return
self.log_queue.append(standard_logging_payload)
if len(self.log_queue) >= self.batch_size:
await self.async_send_batch()
except Exception as e:
verbose_logger.exception(
f"Azure Sentinel Layer Error - {str(e)}\n{traceback.format_exc()}"
)
pass
async def async_log_failure_event(
self, kwargs, response_obj, start_time, end_time
):
"""
Async Log failure events to Azure Sentinel
- Gets StandardLoggingPayload from kwargs
- Adds to batch queue
- Flushes based on CustomBatchLogger settings
Raises:
Raises a NON Blocking verbose_logger.exception if an error occurs
"""
try:
verbose_logger.debug(
"Azure Sentinel: Logging - Enters failure logging function for model %s",
kwargs,
)
standard_logging_payload = kwargs.get("standard_logging_object", None)
if standard_logging_payload is None:
verbose_logger.warning(
"Azure Sentinel: standard_logging_object not found in kwargs"
)
return
self.log_queue.append(standard_logging_payload)
if len(self.log_queue) >= self.batch_size:
await self.async_send_batch()
except Exception as e:
verbose_logger.exception(
f"Azure Sentinel Layer Error - {str(e)}\n{traceback.format_exc()}"
)
pass
async def async_send_batch(self):
"""
Sends the batch of logs to Azure Monitor Logs Ingestion API
Raises:
Raises a NON Blocking verbose_logger.exception if an error occurs
"""
try:
if not self.log_queue:
return
verbose_logger.debug(
"Azure Sentinel - about to flush %s events", len(self.log_queue)
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
# Get OAuth2 token
bearer_token = await self._get_oauth_token()
# Convert log queue to JSON array format expected by Logs Ingestion API
# Each log entry should be a JSON object in the array
body = safe_dumps(self.log_queue)
# Set headers for Logs Ingestion API
headers = {
"Authorization": f"Bearer {bearer_token}",
"Content-Type": "application/json",
}
# Send the request
response = await self.async_httpx_client.post(
url=self.api_endpoint, data=body.encode("utf-8"), headers=headers
)
if response.status_code not in [200, 204]:
verbose_logger.error(
"Azure Sentinel API error: status_code=%s, response=%s",
response.status_code,
response.text,
)
raise Exception(
f"Failed to send logs to Azure Sentinel: {response.status_code} - {response.text}"
)
verbose_logger.debug(
"Azure Sentinel: Response from API status_code: %s",
response.status_code,
)
except Exception as e:
verbose_logger.exception(
f"Azure Sentinel Error sending batch API - {str(e)}\n{traceback.format_exc()}"
)
finally:
self.log_queue.clear()

179
litellm/integrations/azure_sentinel/example_standard_logging_payload.json Normal file
View file

@ -0,0 +1,179 @@
{
"id": "chatcmpl-2299b6a2-82a3-465a-b47c-04e685a2227f",
"trace_id": "97311c60-9a61-4f48-a814-70139ee57868",
"call_type": "acompletion",
"cache_hit": null,
"stream": true,
"status": "success",
"custom_llm_provider": "openai",
"saved_cache_cost": 0.0,
"startTime": 1766000068.28466,
"endTime": 1766000070.07935,
"completionStartTime": 1766000070.07935,
"response_time": 1.79468512535095,
"model": "gpt-4o",
"metadata": {
"user_api_key_hash": null,
"user_api_key_alias": null,
"user_api_key_team_id": null,
"user_api_key_org_id": null,
"user_api_key_user_id": null,
"user_api_key_team_alias": null,
"user_api_key_user_email": null,
"spend_logs_metadata": null,
"requester_ip_address": null,
"requester_metadata": null,
"user_api_key_end_user_id": null,
"prompt_management_metadata": null,
"applied_guardrails": [],
"mcp_tool_call_metadata": null,
"vector_store_request_metadata": null,
"guardrail_information": null
},
"cache_key": null,
"response_cost": 0.00022500000000000002,
"total_tokens": 30,
"prompt_tokens": 10,
"completion_tokens": 20,
"request_tags": [],
"end_user": "",
"api_base": "",
"model_group": "",
"model_id": "",
"requester_ip_address": null,
"messages": [
{
"role": "user",
"content": "Hello, world!"
}
],
"response": {
"id": "chatcmpl-2299b6a2-82a3-465a-b47c-04e685a2227f",
"created": 1742855151,
"model": "gpt-4o",
"object": "chat.completion",
"system_fingerprint": null,
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "hi",
"role": "assistant",
"tool_calls": null,
"function_call": null,
"provider_specific_fields": null
}
}
],
"usage": {
"completion_tokens": 20,
"prompt_tokens": 10,
"total_tokens": 30,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
},
"model_parameters": {},
"hidden_params": {
"model_id": null,
"cache_key": null,
"api_base": "https://api.openai.com",
"response_cost": 0.00022500000000000002,
"additional_headers": {},
"litellm_overhead_time_ms": null,
"batch_models": null,
"litellm_model_name": "gpt-4o"
},
"model_map_information": {
"model_map_key": "gpt-4o",
"model_map_value": {
"key": "gpt-4o",
"max_tokens": 16384,
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"input_cost_per_token": 2.5e-06,
"cache_creation_input_token_cost": null,
"cache_read_input_token_cost": 1.25e-06,
"input_cost_per_character": null,
"input_cost_per_token_above_128k_tokens": null,
"input_cost_per_query": null,
"input_cost_per_second": null,
"input_cost_per_audio_token": null,
"input_cost_per_token_batches": 1.25e-06,
"output_cost_per_token_batches": 5e-06,
"output_cost_per_token": 1e-05,
"output_cost_per_audio_token": null,
"output_cost_per_character": null,
"output_cost_per_token_above_128k_tokens": null,
"output_cost_per_character_above_128k_tokens": null,
"output_cost_per_second": null,
"output_cost_per_image": null,
"output_vector_size": null,
"litellm_provider": "openai",
"mode": "chat",
"supports_system_messages": true,
"supports_response_schema": true,
"supports_vision": true,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_assistant_prefill": false,
"supports_prompt_caching": true,
"supports_audio_input": false,
"supports_audio_output": false,
"supports_pdf_input": false,
"supports_embedding_image_input": false,
"supports_native_streaming": null,
"supports_web_search": true,
"search_context_cost_per_query": {
"search_context_size_low": 0.03,
"search_context_size_medium": 0.035,
"search_context_size_high": 0.05
},
"tpm": null,
"rpm": null,
"supported_openai_params": [
"frequency_penalty",
"logit_bias",
"logprobs",
"top_logprobs",
"max_tokens",
"max_completion_tokens",
"modalities",
"prediction",
"n",
"presence_penalty",
"seed",
"stop",
"stream",
"stream_options",
"temperature",
"top_p",
"tools",
"tool_choice",
"function_call",
"functions",
"max_retries",
"extra_headers",
"parallel_tool_calls",
"audio",
"response_format",
"user"
]
}
},
"error_str": null,
"error_information": {
"error_code": "",
"error_class": "",
"llm_provider": "",
"traceback": "",
"error_message": ""
},
"response_cost_failure_debug_info": null,
"guardrail_information": null,
"standard_built_in_tools_params": {
"web_search_options": null,
"file_search": null
}
}

24
litellm/integrations/custom_guardrail.py
View file

@ -240,6 +240,28 @@ class CustomGuardrail(CustomLogger):
return metadata["disable_global_guardrail"]
return False
def _is_valid_response_type(self, result: Any) -> bool:
"""
Check if result is a valid LLMResponseTypes instance.
Safely handles TypedDict types which don't support isinstance checks.
For non-LiteLLM responses (like passthrough httpx.Response), returns True
to allow them through.
"""
if result is None:
return False
try:
# Try isinstance check on valid types that support it
response_types = get_args(LLMResponseTypes)
return isinstance(result, response_types)
except TypeError as e:
# TypedDict types don't support isinstance checks
# In this case, we can't validate the type, so we allow it through
if "TypedDict" in str(e):
return True
raise
def get_guardrail_from_metadata(
self, data: dict
) -> Union[List[str], List[Dict[str, DynamicGuardrailParams]]]:
@ -342,7 +364,7 @@ class CustomGuardrail(CustomLogger):
response=response,
)
if result is None or not isinstance(result, get_args(LLMResponseTypes)):
if not self._is_valid_response_type(result):
return response
return result

48
litellm/integrations/email_templates/templates.py
View file

@ -60,3 +60,51 @@ USER_INVITED_EMAIL_TEMPLATE = """
Best, <br />
The LiteLLM team <br />
"""
SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
<img src="{email_logo_url}" alt="LiteLLM Logo" width="150" height="50" />
<p> Hi {recipient_email}, <br/>
Your LiteLLM API key has crossed its <b>soft budget limit of {soft_budget}</b>. <br /> <br />
<b>Current Spend:</b> {spend} <br />
<b>Soft Budget:</b> {soft_budget} <br />
{max_budget_info}
<p style="color: #dc2626; font-weight: 500;">
⚠️ Note: Your API requests will continue to work, but you should monitor your usage closely.
If you reach your maximum budget, requests will be rejected.
</p>
You can view your usage and manage your budget in the <a href="{base_url}">LiteLLM Dashboard</a>. <br /> <br />
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
"""
MAX_BUDGET_ALERT_EMAIL_TEMPLATE = """
<img src="{email_logo_url}" alt="LiteLLM Logo" width="150" height="50" />
<p> Hi {recipient_email}, <br/>
Your LiteLLM API key has reached <b>{percentage}% of its maximum budget</b>. <br /> <br />
<b>Current Spend:</b> {spend} <br />
<b>Maximum Budget:</b> {max_budget} <br />
<b>Alert Threshold:</b> {alert_threshold} ({percentage}%) <br />
<p style="color: #dc2626; font-weight: 500;">
⚠️ Warning: You are approaching your maximum budget limit.
Once you reach your maximum budget of {max_budget}, all API requests will be rejected.
</p>
You can view your usage and manage your budget in the <a href="{base_url}">LiteLLM Dashboard</a>. <br /> <br />
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
"""

2
litellm/integrations/gcs_bucket/Readme.md
View file

@ -8,5 +8,5 @@ This folder contains the GCS Bucket Logging integration for LiteLLM Gateway.
- `gcs_bucket_base.py`: This file contains the GCSBucketBase class which handles Authentication for GCS Buckets
## Further Reading
- [Doc setting up GCS Bucket Logging on LiteLLM Proxy (Gateway)](https://docs.litellm.ai/docs/proxy/bucket)
- [Doc setting up GCS Bucket Logging on LiteLLM Proxy (Gateway)](https://docs.litellm.ai/docs/observability/gcs_bucket_integration)
- [Doc on Key / Team Based logging with GCS](https://docs.litellm.ai/docs/proxy/team_logging)

5
litellm/integrations/langfuse/langfuse_prompt_management.py
View file

@ -294,6 +294,11 @@ class LangfusePromptManagement(LangFuseLogger, PromptManagementBase, CustomLogge
self.async_log_success_event, kwargs, response_obj, start_time, end_time
)
def log_failure_event(self, kwargs, response_obj, start_time, end_time):
return run_async_function(
self.async_log_failure_event, kwargs, response_obj, start_time, end_time
)
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
standard_callback_dynamic_params = kwargs.get(
"standard_callback_dynamic_params"

5
litellm/integrations/opentelemetry.py
View file

@ -1994,10 +1994,7 @@ class OpenTelemetry(CustomLogger):
"""
Create a span for the received proxy server request.
"""
# don't create proxy parent spans for arize phoenix - [TODO]: figure out a better way to handle this
if self.callback_name == "arize_phoenix":
return None
return self.tracer.start_span(
name="Received Proxy Server Request",
start_time=self._to_ns(start_time),

13
litellm/integrations/prometheus.py
View file

@ -815,7 +815,20 @@ class PrometheusLogger(CustomLogger):
user_api_key_auth_metadata: Optional[dict] = standard_logging_payload[
"metadata"
].get("user_api_key_auth_metadata")
# Include top-level metadata fields (excluding nested dictionaries)
# This allows accessing fields like requester_ip_address from top-level metadata
top_level_metadata = standard_logging_payload.get("metadata", {})
top_level_fields: Dict[str, Any] = {}
if isinstance(top_level_metadata, dict):
top_level_fields = {
k: v
for k, v in top_level_metadata.items()
if not isinstance(v, dict) # Exclude nested dicts to avoid conflicts
}
combined_metadata: Dict[str, Any] = {
**top_level_fields, # Include top-level fields first
**(_requester_metadata if _requester_metadata else {}),
**(user_api_key_auth_metadata if user_api_key_auth_metadata else {}),
}

2
litellm/interactions/http_handler.py
View file

@ -4,7 +4,6 @@ HTTP Handler for Interactions API requests.
This module handles the HTTP communication for the Google Interactions API.
"""
import json
from typing import (
Any,
AsyncIterator,
@ -18,7 +17,6 @@ from typing import (
import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.constants import request_timeout
from litellm.interactions.streaming_iterator import (
InteractionsAPIStreamingIterator,

4
litellm/interactions/streaming_iterator.py
View file

@ -8,11 +8,10 @@ from the Google Interactions API, similar to the responses API streaming iterato
import asyncio
import json
from datetime import datetime
from typing import Any, Dict, Iterator, Optional
from typing import Any, Dict, Optional
import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.constants import STREAM_SSE_DONE_STRING
from litellm.litellm_core_utils.asyncify import run_async_function
@ -22,7 +21,6 @@ from litellm.litellm_core_utils.llm_response_utils.get_api_base import get_api_b
from litellm.litellm_core_utils.thread_pool_executor import executor
from litellm.llms.base_llm.interactions.transformation import BaseInteractionsAPIConfig
from litellm.types.interactions import (
InteractionsAPIResponse,
InteractionsAPIStreamingResponse,
)
from litellm.utils import CustomStreamWrapper

8
litellm/litellm_core_utils/api_route_to_call_types.py
View file

@ -5,10 +5,12 @@ This dictionary maps each API endpoint to the CallTypes that can be used for tha
Each route can have both async (prefixed with 'a') and sync call types.
"""
from typing import List, Optional
from litellm.types.utils import API_ROUTE_TO_CALL_TYPES, CallTypes
def get_call_types_for_route(route: str) -> list:
def get_call_types_for_route(route: str) -> Optional[List[CallTypes]]:
"""
Get the list of CallTypes for a given API route.
@ -16,9 +18,9 @@ def get_call_types_for_route(route: str) -> list:
route: API route path (e.g., "/chat/completions")
Returns:
List of CallTypes for that route, or empty list if route not found
List of CallTypes for that route, or None if route not found
"""
return API_ROUTE_TO_CALL_TYPES.get(route, [])
return API_ROUTE_TO_CALL_TYPES.get(route, None)
def get_routes_for_call_type(call_type: CallTypes) -> list:

26
litellm/litellm_core_utils/get_model_cost_map.py
View file

@ -18,14 +18,15 @@ def get_model_cost_map(url: str) -> dict:
os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False)
or os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False) == "True"
):
import importlib.resources
from importlib.resources import files
import json
with importlib.resources.open_text(
"litellm", "model_prices_and_context_window_backup.json"
) as f:
content = json.load(f)
return content
content = json.loads(
files("litellm")
.joinpath("model_prices_and_context_window_backup.json")
.read_text(encoding="utf-8")
)
return content
try:
response = httpx.get(
@ -35,11 +36,12 @@ def get_model_cost_map(url: str) -> dict:
content = response.json()
return content
except Exception:
import importlib.resources
from importlib.resources import files
import json
with importlib.resources.open_text(
"litellm", "model_prices_and_context_window_backup.json"
) as f:
content = json.load(f)
return content
content = json.loads(
files("litellm")
.joinpath("model_prices_and_context_window_backup.json")
.read_text(encoding="utf-8")
)
return content

17
litellm/litellm_core_utils/litellm_logging.py
View file

@ -127,6 +127,7 @@ from litellm.utils import _get_base_model_from_metadata, executor, print_verbose
from ..integrations.argilla import ArgillaLogger
from ..integrations.arize.arize_phoenix import ArizePhoenixLogger
from ..integrations.athina import AthinaLogger
from ..integrations.azure_sentinel.azure_sentinel import AzureSentinelLogger
from ..integrations.azure_storage.azure_storage import AzureBlobStorageLogger
from ..integrations.custom_prompt_management import CustomPromptManagement
from ..integrations.datadog.datadog import DataDogLogger
@ -917,9 +918,11 @@ class Logging(LiteLLMLoggingBaseClass):
raw_request_body=self._get_raw_request_body(
additional_args.get("complete_input_dict", {})
),
# NOTE: setting ignore_sensitive_headers to True will cause
# the Authorization header to be leaked when calls to the health
# endpoint are made and fail.
raw_request_headers=self._get_masked_headers(
additional_args.get("headers", {}) or {},
ignore_sensitive_headers=True,
),
error=None,
)
@ -3548,6 +3551,14 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
_datadog_llm_obs_logger = DataDogLLMObsLogger()
_in_memory_loggers.append(_datadog_llm_obs_logger)
return _datadog_llm_obs_logger # type: ignore
elif logging_integration == "azure_sentinel":
for callback in _in_memory_loggers:
if isinstance(callback, AzureSentinelLogger):
return callback # type: ignore
_azure_sentinel_logger = AzureSentinelLogger()
_in_memory_loggers.append(_azure_sentinel_logger)
return _azure_sentinel_logger # type: ignore
elif logging_integration == "gcs_bucket":
for callback in _in_memory_loggers:
if isinstance(callback, GCSBucketLogger):
@ -4052,6 +4063,10 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
for callback in _in_memory_loggers:
if isinstance(callback, DataDogLLMObsLogger):
return callback
elif logging_integration == "azure_sentinel":
for callback in _in_memory_loggers:
if isinstance(callback, AzureSentinelLogger):
return callback
elif logging_integration == "gcs_bucket":
for callback in _in_memory_loggers:
if isinstance(callback, GCSBucketLogger):

2
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -674,7 +674,7 @@ class CostCalculatorUtils:
from litellm.llms.azure_ai.image_generation.cost_calculator import (
cost_calculator as azure_ai_image_cost_calculator,
)
from litellm.llms.bedrock.image.cost_calculator import (
from litellm.llms.bedrock.image_generation.cost_calculator import (
cost_calculator as bedrock_image_cost_calculator,
)
from litellm.llms.gemini.image_generation.cost_calculator import (

23
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -1572,6 +1572,21 @@ def convert_to_gemini_tool_call_result(
return _part
def _sanitize_anthropic_tool_use_id(tool_use_id: str) -> str:
"""
Sanitize tool_use_id to match Anthropic's required pattern: ^[a-zA-Z0-9_-]+$
Anthropic requires tool_use_id to only contain alphanumeric characters, underscores, and hyphens.
This function replaces any invalid characters with underscores.
"""
# Replace any character that's not alphanumeric, underscore, or hyphen with underscore
sanitized = re.sub(r'[^a-zA-Z0-9_-]', '_', tool_use_id)
# Ensure it's not empty (fallback to a default if needed)
if not sanitized:
sanitized = "tool_use_id"
return sanitized
def convert_to_anthropic_tool_result(
message: Union[ChatCompletionToolMessage, ChatCompletionFunctionMessage],
) -> AnthropicMessagesToolResultParam:
@ -1639,18 +1654,22 @@ def convert_to_anthropic_tool_result(
if message["role"] == "tool":
tool_message: ChatCompletionToolMessage = message
tool_call_id: str = tool_message["tool_call_id"]
# Sanitize tool_use_id to match Anthropic's pattern requirement: ^[a-zA-Z0-9_-]+$
sanitized_tool_use_id = _sanitize_anthropic_tool_use_id(tool_call_id)
# We can't determine from openai message format whether it's a successful or
# error call result so default to the successful result template
anthropic_tool_result = AnthropicMessagesToolResultParam(
type="tool_result", tool_use_id=tool_call_id, content=anthropic_content
type="tool_result", tool_use_id=sanitized_tool_use_id, content=anthropic_content
)
if message["role"] == "function":
function_message: ChatCompletionFunctionMessage = message
tool_call_id = function_message.get("tool_call_id") or str(uuid.uuid4())
# Sanitize tool_use_id to match Anthropic's pattern requirement: ^[a-zA-Z0-9_-]+$
sanitized_tool_use_id = _sanitize_anthropic_tool_use_id(tool_call_id)
anthropic_tool_result = AnthropicMessagesToolResultParam(
type="tool_result", tool_use_id=tool_call_id, content=anthropic_content
type="tool_result", tool_use_id=sanitized_tool_use_id, content=anthropic_content
)
if anthropic_tool_result is None:

65
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -43,7 +43,6 @@ if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
AnthropicResponseTextBlock,
)
@ -253,20 +252,39 @@ class AnthropicMessagesHandler(BaseTranslation):
task_mappings: List[Tuple[int, Optional[int]]] = []
# Track (content_index, None) for each text
response_content = response.get("content", [])
# Handle both dict and object responses
response_content: List[Any] = []
if isinstance(response, dict):
response_content = response.get("content", []) or []
elif hasattr(response, "content"):
content = getattr(response, "content", None)
response_content = content or []
else:
response_content = []
if not response_content:
return response
# Step 1: Extract all text content and tool calls from response
for content_idx, content_block in enumerate(response_content):
# Check if this is a text or tool_use block by checking the 'type' field
if isinstance(content_block, dict) and content_block.get("type") in [
"text",
"tool_use",
]:
# Cast to dict to handle the union type properly
# Handle both dict and Pydantic object content blocks
block_dict: Dict[str, Any] = {}
if isinstance(content_block, dict):
block_type = content_block.get("type")
block_dict = cast(Dict[str, Any], content_block)
elif hasattr(content_block, "type"):
block_type = getattr(content_block, "type", None)
# Convert Pydantic object to dict for processing
if hasattr(content_block, "model_dump"):
block_dict = content_block.model_dump()
else:
block_dict = {"type": block_type, "text": getattr(content_block, "text", None)}
else:
continue
if block_type in ["text", "tool_use"]:
self._extract_output_text_and_images(
content_block=cast(Dict[str, Any], content_block),
content_block=block_dict,
content_idx=content_idx,
texts_to_check=texts_to_check,
images_to_check=images_to_check,
@ -530,7 +548,11 @@ class AnthropicMessagesHandler(BaseTranslation):
Override this method to customize text content detection.
"""
response_content = response.get("content", [])
if isinstance(response, dict):
response_content = response.get("content", [])
else:
response_content = getattr(response, "content", None) or []
if not response_content:
return False
for content_block in response_content:
@ -590,7 +612,16 @@ class AnthropicMessagesHandler(BaseTranslation):
mapping = task_mappings[task_idx]
content_idx = cast(int, mapping[0])
response_content = response.get("content", [])
# Handle both dict and object responses
response_content: List[Any] = []
if isinstance(response, dict):
response_content = response.get("content", []) or []
elif hasattr(response, "content"):
content = getattr(response, "content", None)
response_content = content or []
else:
continue
if not response_content:
continue
@ -601,7 +632,11 @@ class AnthropicMessagesHandler(BaseTranslation):
content_block = response_content[content_idx]
# Verify it's a text block and update the text field
if isinstance(content_block, dict) and content_block.get("type") == "text":
# Cast to dict to handle the union type properly for assignment
content_block = cast("AnthropicResponseTextBlock", content_block)
content_block["text"] = guardrail_response
# Handle both dict and Pydantic object content blocks
if isinstance(content_block, dict):
if content_block.get("type") == "text":
cast(Dict[str, Any], content_block)["text"] = guardrail_response
elif hasattr(content_block, "type") and getattr(content_block, "type", None) == "text":
# Update Pydantic object's text attribute
if hasattr(content_block, "text"):
content_block.text = guardrail_response

5
litellm/llms/anthropic/chat/handler.py
View file

@ -692,12 +692,15 @@ class ModelResponseIterator:
text = content_block_start["content_block"]["text"]
elif content_block_start["content_block"]["type"] == "tool_use" or content_block_start["content_block"]["type"] == "server_tool_use":
self.tool_index += 1
# Some server_tool_use blocks (e.g. web_search) may omit `input` at start;
# default to {} to avoid KeyError and let deltas populate arguments.
tool_input = content_block_start["content_block"].get("input", {})
tool_use = ChatCompletionToolCallChunk(
id=content_block_start["content_block"]["id"],
type="function",
function=ChatCompletionToolCallFunctionChunk(
name=content_block_start["content_block"]["name"],
arguments=str(content_block_start["content_block"]["input"]),
arguments=str(tool_input),
),
index=self.tool_index,
)

41
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -169,7 +169,7 @@ class LiteLLMAnthropicMessagesAdapter:
"""
Which anthropic params, we need to translate to the openai format.
"""
return ["messages", "metadata", "system", "tool_choice", "tools"]
return ["messages", "metadata", "system", "tool_choice", "tools", "thinking"]
def translate_anthropic_messages_to_openai( # noqa: PLR0915
self,
@ -420,6 +420,35 @@ class LiteLLMAnthropicMessagesAdapter:
return new_messages
def translate_anthropic_thinking_to_openai(
self, thinking: Dict[str, Any]
) -> Optional[str]:
"""
Translate Anthropic's thinking parameter to OpenAI's reasoning_effort.
Anthropic thinking format: {'type': 'enabled'|'disabled', 'budget_tokens': int}
OpenAI reasoning_effort: 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'default'
"""
if not isinstance(thinking, dict):
return None
thinking_type = thinking.get("type", "disabled")
if thinking_type == "disabled":
return None
elif thinking_type == "enabled":
budget_tokens = thinking.get("budget_tokens", 0)
if budget_tokens >= 10000:
return "high"
elif budget_tokens >= 5000:
return "medium"
elif budget_tokens >= 2000:
return "low"
else:
return "minimal"
return None
def translate_anthropic_tool_choice_to_openai(
self, tool_choice: AnthropicMessagesToolChoice
) -> ChatCompletionToolChoiceValues:
@ -529,6 +558,16 @@ class LiteLLMAnthropicMessagesAdapter:
tools=cast(List[AllAnthropicToolsValues], tools)
)
## CONVERT THINKING
if "thinking" in anthropic_message_request:
thinking = anthropic_message_request["thinking"]
if thinking:
reasoning_effort = self.translate_anthropic_thinking_to_openai(
thinking=cast(Dict[str, Any], thinking)
)
if reasoning_effort:
new_kwargs["reasoning_effort"] = reasoning_effort
translatable_params = self.translatable_anthropic_params()
for k, v in anthropic_message_request.items():
if k not in translatable_params: # pass remaining params as is

4
litellm/llms/anthropic/experimental_pass_through/messages/handler.py
View file

@ -119,6 +119,7 @@ def anthropic_messages_handler(
tools: Optional[List[Dict]] = None,
top_k: Optional[int] = None,
top_p: Optional[float] = None,
container: Optional[Dict] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
client: Optional[AsyncHTTPHandler] = None,
@ -131,6 +132,9 @@ def anthropic_messages_handler(
]:
"""
Makes Anthropic `/v1/messages` API calls In the Anthropic API Spec
Args:
container: Container config with skills for code execution
"""
from litellm.types.utils import LlmProviders

2
litellm/llms/azure/realtime/handler.py
View file

@ -94,7 +94,7 @@ class AzureOpenAIRealtime(AzureChatCompletion):
ssl_context = get_shared_realtime_ssl_context()
async with websockets.connect( # type: ignore
url,
extra_headers={
additional_headers={
"api-key": api_key, # type: ignore
},
max_size=REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES,

12
litellm/llms/bedrock/base_aws_llm.py
View file

@ -357,6 +357,18 @@ class BaseAWSLLM:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="openai"
)
elif provider == "qwen2" and "qwen2/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="qwen2"
)
elif provider == "qwen3" and "qwen3/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="qwen3"
)
elif provider == "stability" and "stability/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="stability"
)
return model_id
@staticmethod

10
litellm/llms/bedrock/image_edit/__init__.py Normal file
View file

@ -0,0 +1,10 @@
"""
Bedrock Image Edit Module
Handles image edit operations for Bedrock stability models.
"""
from .handler import BedrockImageEdit
__all__ = ["BedrockImageEdit"]

310
litellm/llms/bedrock/image_edit/handler.py Normal file
View file

@ -0,0 +1,310 @@
"""
Bedrock Image Edit Handler
Handles image edit requests for Bedrock stability models.
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any, Optional, Union
import httpx
from pydantic import BaseModel
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LitellmLogging
from litellm.llms.bedrock.image_edit.stability_transformation import (
BedrockStabilityImageEditConfig,
)
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
_get_httpx_client,
get_async_httpx_client,
)
from litellm.types.utils import ImageResponse
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import BedrockError
if TYPE_CHECKING:
from botocore.awsrequest import AWSPreparedRequest
else:
AWSPreparedRequest = Any
class BedrockImageEditPreparedRequest(BaseModel):
"""
Internal/Helper class for preparing the request for bedrock image edit
"""
endpoint_url: str
prepped: AWSPreparedRequest
body: bytes
data: dict
class BedrockImageEdit(BaseAWSLLM):
"""
Bedrock Image Edit handler
"""
@classmethod
def get_config_class(cls, model: str | None):
if BedrockStabilityImageEditConfig._is_stability_edit_model(model):
return BedrockStabilityImageEditConfig
else:
raise ValueError(f"Unsupported model for bedrock image edit: {model}")
def image_edit(
self,
model: str,
image: list,
prompt: str,
model_response: ImageResponse,
optional_params: dict,
logging_obj: LitellmLogging,
timeout: Optional[Union[float, httpx.Timeout]],
aimage_edit: bool = False,
api_base: Optional[str] = None,
extra_headers: Optional[dict] = None,
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
api_key: Optional[str] = None,
):
prepared_request = self._prepare_request(
model=model,
image=image,
prompt=prompt,
optional_params=optional_params,
api_base=api_base,
extra_headers=extra_headers,
logging_obj=logging_obj,
api_key=api_key,
)
if aimage_edit is True:
return self.async_image_edit(
prepared_request=prepared_request,
timeout=timeout,
model=model,
logging_obj=logging_obj,
prompt=prompt,
model_response=model_response,
client=(
client
if client is not None and isinstance(client, AsyncHTTPHandler)
else None
),
)
if client is None or not isinstance(client, HTTPHandler):
client = _get_httpx_client()
try:
response = client.post(url=prepared_request.endpoint_url, headers=prepared_request.prepped.headers, data=prepared_request.body) # type: ignore
response.raise_for_status()
except httpx.HTTPStatusError as err:
error_code = err.response.status_code
raise BedrockError(status_code=error_code, message=err.response.text)
except httpx.TimeoutException:
raise BedrockError(status_code=408, message="Timeout error occurred.")
### FORMAT RESPONSE TO OPENAI FORMAT ###
model_response = self._transform_response_dict_to_openai_response(
model_response=model_response,
model=model,
logging_obj=logging_obj,
prompt=prompt,
response=response,
data=prepared_request.data,
)
return model_response
async def async_image_edit(
self,
prepared_request: BedrockImageEditPreparedRequest,
timeout: Optional[Union[float, httpx.Timeout]],
model: str,
logging_obj: LitellmLogging,
prompt: str,
model_response: ImageResponse,
client: Optional[AsyncHTTPHandler] = None,
) -> ImageResponse:
"""
Asynchronous handler for bedrock image edit
"""
async_client = client or get_async_httpx_client(
llm_provider=litellm.LlmProviders.BEDROCK,
params={"timeout": timeout},
)
try:
response = await async_client.post(url=prepared_request.endpoint_url, headers=prepared_request.prepped.headers, data=prepared_request.body) # type: ignore
response.raise_for_status()
except httpx.HTTPStatusError as err:
error_code = err.response.status_code
raise BedrockError(status_code=error_code, message=err.response.text)
except httpx.TimeoutException:
raise BedrockError(status_code=408, message="Timeout error occurred.")
### FORMAT RESPONSE TO OPENAI FORMAT ###
model_response = self._transform_response_dict_to_openai_response(
model=model,
logging_obj=logging_obj,
prompt=prompt,
response=response,
data=prepared_request.data,
model_response=model_response,
)
return model_response
def _prepare_request(
self,
model: str,
image: list,
prompt: str,
optional_params: dict,
api_base: Optional[str],
extra_headers: Optional[dict],
logging_obj: LitellmLogging,
api_key: Optional[str],
) -> BedrockImageEditPreparedRequest:
"""
Prepare the request body, headers, and endpoint URL for the Bedrock Image Edit API
Args:
model (str): The model to use for the image edit
image (list): The images to edit
prompt (str): The prompt for the edit
optional_params (dict): The optional parameters for the image edit
api_base (Optional[str]): The base URL for the Bedrock API
extra_headers (Optional[dict]): The extra headers to include in the request
logging_obj (LitellmLogging): The logging object to use for logging
api_key (Optional[str]): The API key to use
Returns:
BedrockImageEditPreparedRequest: The prepared request object
"""
boto3_credentials_info = self._get_boto_credentials_from_optional_params(
optional_params, model
)
# Use the existing ARN-aware provider detection method
bedrock_provider = self.get_bedrock_invoke_provider(model)
### SET RUNTIME ENDPOINT ###
modelId = self.get_bedrock_model_id(
model=model,
provider=bedrock_provider,
optional_params=optional_params,
)
_, proxy_endpoint_url = self.get_runtime_endpoint(
api_base=api_base,
aws_bedrock_runtime_endpoint=boto3_credentials_info.aws_bedrock_runtime_endpoint,
aws_region_name=boto3_credentials_info.aws_region_name,
)
proxy_endpoint_url = f"{proxy_endpoint_url}/model/{modelId}/invoke"
data = self._get_request_body(
model=model,
image=image,
prompt=prompt,
optional_params=optional_params,
)
# Make POST Request
body = json.dumps(data).encode("utf-8")
headers = {"Content-Type": "application/json"}
if extra_headers is not None:
headers = {"Content-Type": "application/json", **extra_headers}
prepped = self.get_request_headers(
credentials=boto3_credentials_info.credentials,
aws_region_name=boto3_credentials_info.aws_region_name,
extra_headers=extra_headers,
endpoint_url=proxy_endpoint_url,
data=body,
headers=headers,
api_key=api_key,
)
## LOGGING
logging_obj.pre_call(
input=prompt,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": proxy_endpoint_url,
"headers": prepped.headers,
},
)
return BedrockImageEditPreparedRequest(
endpoint_url=proxy_endpoint_url,
prepped=prepped,
body=body,
data=data,
)
def _get_request_body(
self,
model: str,
image: list,
prompt: str,
optional_params: dict,
) -> dict:
"""
Get the request body for the Bedrock Image Edit API
Checks the model/provider and transforms the request body accordingly
Returns:
dict: The request body to use for the Bedrock Image Edit API
"""
config_class = self.get_config_class(model=model)
config_instance = config_class()
request_body = config_instance.transform_image_edit_request(
model=model,
prompt=prompt,
image=image[0] if image else None,
image_edit_optional_request_params=optional_params,
litellm_params={},
headers={},
)
return dict(request_body)
def _transform_response_dict_to_openai_response(
self,
model_response: ImageResponse,
model: str,
logging_obj: LitellmLogging,
prompt: str,
response: httpx.Response,
data: dict,
) -> ImageResponse:
"""
Transforms the Image Edit response from Bedrock to OpenAI format
"""
## LOGGING
if logging_obj is not None:
logging_obj.post_call(
input=prompt,
api_key="",
original_response=response.text,
additional_args={"complete_input_dict": data},
)
verbose_logger.debug("raw model_response: %s", response.text)
response_dict = response.json()
if response_dict is None:
raise ValueError("Error in response object format, got None")
config_class = self.get_config_class(model=model)
config_instance = config_class()
model_response = config_instance.transform_image_edit_response(
model=model,
raw_response=response,
logging_obj=logging_obj,
)
return model_response

377
litellm/llms/bedrock/image_edit/stability_transformation.py Normal file
View file

@ -0,0 +1,377 @@
"""
Bedrock Stability AI Image Edit Transformation
Handles transformation between OpenAI-compatible format and Bedrock Stability AI Image Edit API format.
Supported models:
- stability.stable-conservative-upscale-v1:0
- stability.stable-creative-upscale-v1:0
- stability.stable-fast-upscale-v1:0
- stability.stable-outpaint-v1:0
- stability.stable-image-control-sketch-v1:0
- stability.stable-image-control-structure-v1:0
- stability.stable-image-erase-object-v1:0
- stability.stable-image-inpaint-v1:0
- stability.stable-image-remove-background-v1:0
- stability.stable-image-search-recolor-v1:0
- stability.stable-image-search-replace-v1:0
- stability.stable-image-style-guide-v1:0
- stability.stable-style-transfer-v1:0
API Reference: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html
"""
import json
import base64
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple
import httpx
from litellm.llms.base_llm.image_edit.transformation import BaseImageEditConfig
from litellm.types.images.main import ImageEditOptionalRequestParams
from litellm.types.router import GenericLiteLLMParams
from litellm.types.llms.stability import (
OPENAI_SIZE_TO_STABILITY_ASPECT_RATIO,
)
from litellm.types.utils import FileTypes, ImageObject, ImageResponse
from litellm.utils import get_model_info
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class BedrockStabilityImageEditConfig(BaseImageEditConfig):
"""
Configuration for Bedrock Stability AI image edit.
Supports all Stability image edit operations through Bedrock.
"""
@classmethod
def _is_stability_edit_model(cls, model: Optional[str] = None) -> bool:
"""
Returns True if the model is a Bedrock Stability edit model.
Bedrock Stability edit models follow this pattern:
stability.stable-conservative-upscale-v1:0
stability.stable-creative-upscale-v1:0
stability.stable-fast-upscale-v1:0
stability.stable-outpaint-v1:0
stability.stable-image-inpaint-v1:0
stability.stable-image-erase-object-v1:0
etc.
"""
if model:
model_lower = model.lower()
if "stability." in model_lower and any([
"upscale" in model_lower,
"outpaint" in model_lower,
"inpaint" in model_lower,
"erase" in model_lower,
"remove-background" in model_lower,
"search-recolor" in model_lower,
"search-replace" in model_lower,
"control-sketch" in model_lower,
"control-structure" in model_lower,
"style-guide" in model_lower,
"style-transfer" in model_lower,
]):
return True
return False
def get_supported_openai_params(
self, model: str
) -> list:
"""
Return list of OpenAI params supported by Bedrock Stability.
"""
return [
"n", # Number of images (Stability always returns 1, we can loop)
"size", # Maps to aspect_ratio
"response_format", # b64_json or url (Stability only returns b64)
"mask",
]
def map_openai_params(
self,
image_edit_optional_params: ImageEditOptionalRequestParams,
model: str,
drop_params: bool,
) -> Dict:
"""
Map OpenAI parameters to Bedrock Stability parameters.
OpenAI -> Stability mappings:
- size -> aspect_ratio
- n -> (handled separately, Stability returns 1 image per request)
"""
supported_params = self.get_supported_openai_params(model)
# Define mapping from OpenAI params to Stability params
param_mapping = {
"size": "aspect_ratio",
# "n" and "response_format" are handled separately
}
# Create a copy to not mutate original - convert TypedDict to regular dict
mapped_params: Dict[str, Any] = dict(image_edit_optional_params)
for k, v in image_edit_optional_params.items():
if k in param_mapping:
# Map param if mapping exists and value is valid
if k == "size" and v in OPENAI_SIZE_TO_STABILITY_ASPECT_RATIO:
mapped_params[param_mapping[k]] = OPENAI_SIZE_TO_STABILITY_ASPECT_RATIO[v] # type: ignore
# Don't copy "size" itself to final dict
elif k == "n":
# Store for logic but do not add to outgoing params
mapped_params["_n"] = v
elif k == "response_format":
# Only b64 supported at Stability; store for postprocessing
mapped_params["_response_format"] = v
elif k not in supported_params:
if not drop_params:
raise ValueError(
f"Parameter {k} is not supported for model {model}. "
f"Supported parameters are {supported_params}. "
f"Set drop_params=True to drop unsupported parameters."
)
# Otherwise, param will simply be dropped
else:
# param is supported and not mapped, keep as-is
continue
# Remove OpenAI params that have been mapped unless they're in stability
for mapped in ["size", "n", "response_format"]:
if mapped in mapped_params:
del mapped_params[mapped]
return mapped_params
def transform_image_edit_request(
self,
model: str,
prompt: str,
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict, Any]:
"""
Transform OpenAI-style request to Bedrock Stability request format.
Returns the request body dict that will be JSON-encoded by the handler.
"""
# Build Bedrock Stability request
data: Dict[str, Any] = {
"prompt": prompt,
"output_format": "png", # Default to PNG
}
# Convert image to base64
image_b64: str
if hasattr(image, 'read') and callable(getattr(image, 'read', None)):
# File-like object (e.g., BufferedReader from open())
image_bytes = image.read() # type: ignore
image_b64 = base64.b64encode(image_bytes).decode('utf-8') # type: ignore
elif isinstance(image, bytes):
# Raw bytes
image_b64 = base64.b64encode(image).decode('utf-8')
elif isinstance(image, str):
# Already a base64 string
image_b64 = image
else:
# Try to handle as bytes
image_b64 = base64.b64encode(bytes(image)).decode('utf-8') # type: ignore
data["image"] = image_b64
# Add optional params (already mapped in map_openai_params)
for key, value in image_edit_optional_request_params.items(): # type: ignore
# Skip internal params (prefixed with _)
if key.startswith("_") or value is None:
continue
# File-like optional params (mask, init_image, style_image, etc.)
if key in ["mask", "init_image", "style_image"]:
# Handle case where value might be in a list
file_value = value
if isinstance(value, list) and len(value) > 0:
file_value = value[0]
if hasattr(file_value, 'read') and callable(getattr(file_value, 'read', None)):
file_bytes = file_value.read() # type: ignore
elif isinstance(file_value, bytes):
file_bytes = file_value
elif isinstance(file_value, str):
# Already a base64 string
data[key] = file_value
continue
else:
file_bytes = file_value # type: ignore
if isinstance(file_bytes, bytes):
file_b64 = base64.b64encode(file_bytes).decode('utf-8')
else:
file_b64 = str(file_bytes)
data[key] = file_b64
continue
# Supported text fields
if key in [
"negative_prompt",
"aspect_ratio",
"seed",
"output_format",
"model",
"mode",
"strength",
"style_preset",
"creativity",
"control_strength",
"grow_mask",
"left",
"right",
"up",
"down",
"select_prompt",
"search_prompt",
"fidelity",
"composition_fidelity",
"style_strength",
"change_strength",
]:
data[key] = value # type: ignore
return data, {}
def transform_image_edit_response(
self,
model: str,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform Bedrock Stability response to OpenAI-compatible ImageResponse.
Bedrock returns: {"images": ["base64..."], "finish_reasons": [null], "seeds": [123]}
OpenAI expects: {"data": [{"b64_json": "base64..."}], "created": timestamp}
"""
try:
response_data = raw_response.json()
with open("response_data.json", "w") as f:
json.dump(response_data, f)
except Exception as e:
raise self.get_error_class(
error_message=f"Error parsing Bedrock Stability response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
# Check for errors in response
if "errors" in response_data:
raise self.get_error_class(
error_message=f"Bedrock Stability error: {response_data['errors']}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
# Check finish_reasons
finish_reasons = response_data.get("finish_reasons", [])
if finish_reasons and finish_reasons[0]:
raise self.get_error_class(
error_message=f"Bedrock Stability error: {finish_reasons[0]}",
status_code=400,
headers=raw_response.headers,
)
model_response = ImageResponse()
if not model_response.data:
model_response.data = []
# Extract images from response
images = response_data.get("images", [])
if images:
for image_b64 in images:
if image_b64:
model_response.data.append(
ImageObject(
b64_json=image_b64,
url=None,
revised_prompt=None,
)
)
if not hasattr(model_response, "_hidden_params"):
model_response._hidden_params = {}
if "additional_headers" not in model_response._hidden_params:
model_response._hidden_params["additional_headers"] = {}
# Set cost based on model
model_info = get_model_info(model, custom_llm_provider="bedrock")
cost_per_image = model_info.get("output_cost_per_image", 0)
if cost_per_image is not None:
model_response._hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] = float(cost_per_image)
return model_response
def use_multipart_form_data(self) -> bool:
"""
Bedrock Stability uses JSON format, not multipart/form-data.
"""
return False
def get_complete_url(
self,
model: str,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for the Bedrock Image Edit API.
For Bedrock, this is handled by the handler which constructs the endpoint URL
based on the model ID and AWS region. This method is required by the base class
but the actual URL construction happens in BedrockImageEdit.image_edit().
Returns a placeholder - the real endpoint is constructed in the handler.
"""
# Bedrock URLs are constructed in the handler using boto3
# This is a placeholder for the abstract method requirement
return "bedrock://image-edit"
def validate_environment(
self,
headers: dict,
model: str,
api_key: Optional[str] = None,
) -> dict:
"""
Validate environment for Bedrock Stability image edit.
For Bedrock, AWS credentials are managed by the BaseAWSLLM class.
This method validates that headers are properly set up.
Args:
headers: The request headers to validate/update
model: The model name being used
api_key: Optional API key (not used for Bedrock, which uses AWS credentials)
Returns:
Updated headers dict
"""
if headers is None:
headers = {}
# Bedrock uses AWS credentials, not API keys
# Headers are set up by the handler's get_request_headers() method
# This just ensures basic headers are present
if "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
return headers

0
litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py → litellm/llms/bedrock/image_generation/amazon_nova_canvas_transformation.py
View file

0
litellm/llms/bedrock/image/amazon_stability1_transformation.py → litellm/llms/bedrock/image_generation/amazon_stability1_transformation.py
View file

0
litellm/llms/bedrock/image/amazon_stability3_transformation.py → litellm/llms/bedrock/image_generation/amazon_stability3_transformation.py
View file

Some files were not shown because too many files have changed in this diff Show more